Chunking and Context Management for Web-Retrieved Content
How you split web documents for LLMs shapes every retrieval decision that follows.
A developer calling a search or contents API gets back a raw document, often several thousand words of uneven prose, tables, and navigation debris, and the very next decision, how to split that document into pieces an LLM can actually use, shapes everything that happens afterward. That decision is chunking, and it belongs with the architecture of the retrieval pipeline, not with the preprocessing scripts that run quietly before the "real" work starts.
Chunking is an architectural decision, not a preprocessing detail
Web documents arrive large, noisy, and built from inconsistent structure. LLMs work inside a fixed context window. Chunking is the bridge between those two facts, and nothing downstream, not the embedding model, not the vector database, not the LLM doing the final reasoning, can recover information that a bad cut point threw away. Research from Wrocław University of Science and Technology treats chunking as one of the central tasks in retrieval-augmented generation systems, and its findings make clear that chunking introduces a range of consequences that are easy to overlook when it is treated as a minor step. Missing any one of the three leaves the rest of the pipeline working with damaged material no matter how well it is built.
How a web-retrieval pipeline delivers content to a chunker
Chunking sits at a specific point in a two-stage pipeline. The quality of that extraction step carries real weight. A chunker fed messy, half-parsed HTML inherits all of that mess and has no way to tell a heading from a footer. A chunker fed clean Markdown with headings and section breaks intact can treat those existing boundaries as natural places to cut, already giving the output a measure of coherence before any chunking logic runs. You.com's Contents API is built to produce exactly this kind of output, clean and structured enough that it slots directly into a chunking strategy without extra cleanup work. This matters beyond a single API call, too. Get the chunking wrong, and the error does not happen once. It happens on every retrieval call the agent makes for the rest of its run.
Fixed-size chunking's strengths and its persistence as the default
Fixed-size chunking splits text at a consistent character or token count, with no attention to what the content actually says at the cut point. For uniform, record-like content, product listings, log entries, database exports, this works fine, because every record looks roughly the same and there is no topic boundary to respect. For the heterogeneous prose and structured data that web retrieval actually returns, it is a poor default. The Wrocław University evaluation confirms that fixed-size and semantic chunking have long served as the two standard approaches, even as the field has produced a growing list of newer methods that claim to outperform them. None of that makes fixed-size chunking a mistake on its own. The same Wrocław University research backs up that trade-off directly: more computationally expensive chunking methods do not always produce meaningful gains in effectiveness, while they reliably add computational overhead. So the decision rule is narrower than "always upgrade": use fixed-size chunking when documents are uniform in structure and latency is the binding constraint, and treat it with suspicion as a universal default for the kind of varied content a live web crawl actually returns.
Semantic chunking: cutting at topic boundaries rather than token counts
Semantic chunking cuts text where topic similarity drops, rather than at a fixed offset, letting it respect the actual shape of an argument or a narrative as it cuts through it. On prose-heavy web content, that produces more coherent chunks and better retrieval matches than fixed-size splitting delivers. But the computational cost of measuring similarity across a document is not free, and the research does not support treating semantic chunking as a universal upgrade. That distinction matters directly for web retrieval, since a single page pulled from a contents API might contain long explanatory prose sitting right next to a table or a code sample, and treating the whole page with one chunking method will shortchange one part of it. The practical rule: reach for semantic splitting when the corpus is long-form prose and precision is the metric that matters most, and skip it when the corpus is uniform records, code, or structured data, or when p99 latency is what the system is actually optimizing for.
Chunk size is task-specific, and using the wrong size for the task is a common production error
Chunk size does not have a single correct answer. It depends on the task the retrieval system is performing, and applying the wrong size undermines accuracy no matter which chunking strategy produced the chunk. Two pressures bound the useful range from both directions. Below roughly 100 tokens, a chunk often lacks enough signal to generate an accurate embedding, since there simply is not enough text for the model to characterize. Above roughly 800 tokens, a chunk tends to blend multiple subtopics together, and the resulting embedding becomes a blurred average of several ideas. For most English-language technical content pulled from the web, a reasonable starting point is around 512 tokens with 50 to 100 tokens of overlap between chunks, and that number should be treated as a hypothesis to test against recall@k. A natural objection follows from the rise of million-token context windows: if a model can ingest an entire document at once, does chunk size even matter anymore? Model accuracy degrades as the amount of context grows, well before the window limit is reached, so pouring a huge amount of poorly sized text into a large window leaves the precision loss intact. It just moves it further downstream.
Contextual retrieval and late chunking: the leading approaches for web-retrieved content in production
Neither fixed-size nor plain semantic chunking solves the failure mode that matters most for web content: a chunk that reads perfectly inside its original document can turn into nonsense the moment it is pulled out on its own, because its subject or framing lived somewhere else on the page. Two techniques address this directly, and both earn their place in a production pipeline built around live web retrieval. Contextual retrieval, introduced by Anthropic, prepends a short LLM-generated summary, roughly 50 to 100 tokens, to each chunk before it gets embedded, giving the embedding model the document-level frame that the chunk alone would lack. Airbyte's guidance describes this method and notes that pairing contextual retrieval with BM25 keyword search and a reranking step cuts down top-20 retrieval failures substantially in Anthropic's own evaluations. Late chunking runs the logic in the opposite direction: a long-context model embeds the entire document first, and only after that does the pipeline pool the resulting token-level vectors into chunk-sized segments, so every chunk's final embedding already carries the context of the full document beyond just its own local text. Cost is the obvious concern with generating a custom context for every chunk, and prompt caching is what makes contextual retrieval workable at scale: the source document gets cached once, each chunk's context generation reuses that cache, and the one-time cost comes out to around $1.02 per million document tokens. For web-retrieved content specifically, these two techniques matter because live web documents show up with unpredictable structure, uneven density of references, and no guarantee that any single passage can stand on its own, so restoring context improves retrieval accuracy on exactly this kind of content. That is exactly the condition under which the extra step of restoring context pays for itself.
Context budget management: keeping retrieval cost from consuming the agent's working memory
Even chunks built well create a second problem once an agent starts making multiple retrieval calls in sequence: every chunk pulled into context is a chunk of token budget that is no longer available for tool output, intermediate reasoning, or planning the next step. The arithmetic adds up fast. Five chunks at 1,000 tokens each, repeated across three retrieval steps, already consumes 15,000 tokens before the agent has produced a single line of reasoning, and most agentic systems run well beyond three retrieval steps in a real task. A handful of concrete techniques keep that budget under control. Chunking itself is the first and strongest lever, since loading only the relevant section of a document rather than the whole thing avoids the problem at its source. Giving the agent tool access to query a database or an API directly keeps structured data out of context entirely, reserving the window for reasoning. This pressure is highest in multi-step agents built for financial research, regulatory compliance, or market intelligence, which happen to be exactly the use cases where web-retrieved content is most valuable and most abundant, and therefore where a careless context budget breaks down fastest.
How poor chunking produces hallucination and missed answers
Bad chunking produces two distinct failure modes, and both end with the same outcome: a wrong or unsupported answer. In the first, the chunk containing the right information never gets retrieved at all, because it was buried inside a chunk so large or so mixed in topic that its vector did not match the query closely enough. The CRAG Comprehensive RAG Benchmark adds a sharper data point here: for a substantial share of test questions, the correct ground truth is present somewhere in the web retrieval results, but never makes it into the prompt because of context window limits; the pipeline's chunking and ranking choices failed to get it in front of the model even though the right information exists on the web. The objection that larger context windows make all of this moot does not hold up well against that evidence, since model accuracy measurably degrades as the amount of context grows, even before the window's hard limit is reached. A bigger window stuffed with mixed-signal chunks does not improve grounding. It spreads the model's attention thinner across more material, which is the opposite of what grounding needs.
Grounding quality depends on source quality, not just retrieval precision
A pipeline that chunks and retrieves perfectly still runs into the oldest limit of open-web research: the content it is built on might simply be wrong. A system can cite its sources with complete accuracy and still produce an incorrect answer if the cited source itself got the facts wrong, and no amount of chunking sophistication changes that. This is the real distinction between retrieval built on a curated, vetted corpus and retrieval built on the open web. Curated corpora trade coverage for trustworthiness, since someone has already filtered the material for reliability before it enters the system. Open web search trades that vetting for breadth, pulling in orders of magnitude more content at the cost of uneven quality across sources. No chunking method resolves that trade-off, because it sits at the level of source selection, not text segmentation, so it has to be managed in the architecture. The practical answer is not to avoid open-web retrieval, which would throw away its real advantage in coverage and freshness, but to build the pipeline so source provenance stays visible at every step: answers that come with citations and traceable links back to their originating pages, so a person or a downstream system can weigh the credibility of the source directly rather than trusting the retrieval layer to have made that judgment silently. You.com's approach to cited, benchmark-verified retrieval treats source transparency as a production requirement.
Choosing a chunking strategy for your pipeline: a decision framework
Three variables decide the right chunking strategy: the structure of the content, the nature of the retrieval task, and the latency budget available to the system. Those three combine predictably enough to produce a working decision rule. For uniform, record-like web content, product pages following a repeating schema or structured listings, fixed-size chunking at 200 to 400 tokens with low overlap does the job at low computational cost. For long-form prose such as news articles, research reports, or regulatory filings, semantic splitting to locate topic boundaries, paired with contextual retrieval to restore self-containment at the chunk level, produces far better results than either method alone. For code and technical documentation with clear hierarchical structure, boundary-aware chunking that respects function, class, or section divisions in the 50 to 200 token range outperforms any method that ignores that structure in favor of a fixed token count. Most real web pages mix these formats on a single page, so a hybrid approach, semantic splitting for coherence combined with contextual retrieval or late chunking for self-containment, tends to serve general-purpose retrieval best, and that combination should still be checked against recall@k before it goes into production. Across every one of these paths, the same discipline applies: validate the choice against recall@k on a sample that actually represents the retrieval task at hand, rather than trusting a vendor default or an intuition about what should work. The spread between the best and worst chunking strategy on the same embedding model and the same corpus is large enough to be the single biggest factor in end-to-end accuracy, which makes measurement the real deliverable here, not the strategy name on its own. You.com's Web Search API and Contents API supply the extraction layer that produces this content, delivering clean, structured, real-time web content at low latency so the chunker is never working from a degraded source regardless of which strategy gets chosen. For teams that would rather not build and tune this whole stack themselves, the Research API and Finance Research API handle chunking, retrieval, and source reconciliation as a managed layer, built on the same measurement discipline this framework asks every developer to apply on their own.
