When building RAG pipelines or web agents, the quality of input data is paramount. Every page fetched consumes tokens, filling the context window and increasing query costs. A recent study of 20 real web pages demonstrated that using raw HTML is highly inefficient, as the median page contains approximately 128,000 tokens, which can easily exhaust a standard context window. This necessitates a more intelligent approach to data ingestion for LLMs.
The analysis reveals that converting pages into LLM-oriented Markdown reduces token counts by 98% compared to raw HTML. Even when compared to simple full-page text conversion, optimized Markdown provides a 63% reduction in tokens. This significant saving is achieved by stripping away navigation menus, footers, cookie notices, and teasers, which typically clutter the model's context and offer no value for the actual reasoning or retrieval tasks performed by the AI.
The primary performance gain comes from main-content extraction, which provides a median saving of 51%. Additional cleanup of the remaining text adds roughly 9% more efficiency. It is important to note that these savings vary by page type; portal-style pages with extensive menus show much higher optimization potential than clean articles, where the savings are smaller but still significant for maintaining the overall stability and performance of the system.
Using raw HTML not only increases costs but also degrades model output quality. Excessive noise in the context forces the LLM to process irrelevant information, which can lead to hallucinations or missed details. Furthermore, redundant text in the vector index creates unnecessary competition for the relevant passages during retrieval, which negatively impacts the accuracy and relevance of the information retrieved by the pipeline during the query phase.
From a financial perspective, token savings are substantial. At a cost of one dollar per million input tokens, optimization saves approximately eight dollars per thousand pages processed. Since RAG pipelines pay for tokens during embedding and agents pay again every time a page enters the context window, these costs accumulate rapidly, making high-quality data preprocessing an essential investment for any production-grade AI project or automated web-scraping system.
Developers must carefully select their content extraction tools. It is crucial to verify that the chosen parser does not strip away too much useful information, as aggressive extraction can be worse than insufficient cleaning. We recommend performing spot checks on the output of your pipelines to ensure you achieve the optimal balance between brevity and data completeness, ensuring the model receives the most valuable information for its specific tasks.
🔗 Original Source: Read Original Article