HTML to Markdown: What Benchmarks Forget

A look at what actually survives

TL/DR: Stripping HTML down to plain markdown is widely described as cutting tokens by about 80%, or roughly 5x. On ten pages I measured, the cuts ranged from 1.4x to 734x. The biggest cuts were not tightly packed pages. They were pages missing their prices, specs, and comparison values. Ranking the best sections against the question, instead of sending the whole page, did not fix it.

The problem, in plain terms

When an AI agent reads a web page, it reads the whole thing. Every word costs money. Tokens are the unit your model bill is measured in, and a token is roughly three quarters of a word.

A modern web page is mostly not words. It is markup, scripts, styling, and hidden data the browser uses to assemble what you see. Send all of that to a model and you pay for all of it.

The standard fix is to strip the page down to just the text, in a simple format called markdown. Nearly every tool that does this reports a similar result. Cloudflare cites a blog post going from 16,180 tokens to 3,150, an 80% cut. Apify actors advertise 60 to 77%. Open-source tools report around 80%. The numbers cluster tightly enough that "about 5x" has become the working assumption.

I set out to test something else: instead of sending the whole cleaned page, what if you rank the sections against the question being asked and send only the best few? That approach has a name in current usage, context engineering, and the idea is that less but better-chosen content beats more.

I wanted to know whether it beat 5x by enough to be worth building. It doesn't. But the reason surprised me, and it turned out to matter more than the feature I was testing.

How the ten pages were tested

I picked pages for variety instead of convenience: a documentation page, a SaaS pricing page, a news article, a product roundup, a Wikipedia article, a GitHub README, a product page, a recipe, a Paul Graham essay (hand-written HTML with almost no clutter), and my own site.

For each page I wrote one question first, before running anything, and phrased it the way an outsider would. If a page had a heading called "Compatibility," my question was "can I run this inside Cursor," not "what is the compatibility." Reusing a page's own words makes the results look better than they are.

I also wrote down the correct answer and which section held it, by reading the full pages myself, before any code ran. That is the answer key.

For the conversion step I used trafilatura, a widely used open-source extractor. To be clear about what this is and isn't: I measured that tool on those pages. I did not test Cloudflare's converter, or any vendor's, and nothing here should be read as a measurement of one.

Finding one: the 5x baseline didn't show up

The middle value is 62x. The range runs from 1.4x to 734x, a spread of more than 500 to 1.

The familiar 4x to 5x figure appears nowhere on this list. The closest is the Python docs at 4.3x, and it is closest because it is the most old-fashioned page in the set: plain HTML, very little framework code.

The essay at the bottom explains the pattern. Paul Graham writes his HTML by hand, so there is almost nothing to strip and conversion barely helps. The pages at the top are modern JavaScript applications where most of the download is hidden data. Strip that and the number looks enormous.

None of the published figures are wrong, exactly. They are measurements of the pages someone chose, and those were mostly clean article pages from clean publishing systems. The number just does not generalize, and it is being quoted as though it does.

Finding two: the biggest cuts were deletion, not compression

Look at the Sentry row again. 542,996 tokens down to 740.

Sentry's pricing page has a plan comparison chart. After conversion, that chart survives as about thirty bare row labels. "SSO: SAML2" and so on. No values. The checkmarks showing which plan includes which feature are drawn as images, and images are not text, so they are gone. The Enterprise tier does not appear at all.

740 tokens is not a compressed pricing page. It is a pricing page with the pricing taken out.

The ThermoWorks product page fails more quietly. A 580x reduction, and the product's price appears nowhere in the output, because the site fills it in with JavaScript after the page loads. Nothing crashed. The tool ran fine and produced clean, readable text about a kitchen thermometer that costs an unknown amount of money.

My own site: 10.3x, and the entire pricing table is missing. I wrote that page. I had no idea.

The Tom's Guide roundup was the one that got my attention. Every "Specifications" and "Reasons to buy" block converts to an empty heading. So when I asked which laptop to get if I wanted to swap out parts later, the top-ranked section came back with a confident recommendation and not one number in it. My own test script scored that a pass, because the right product name was there. It was the correct answer with all the evidence removed.

The failure isn't the one I expected

Going in, I expected a different problem: number kept, warning label dropped. My site shows a large dollar figure for total token savings, sitting right next to text explaining that tokens are counted while dollars are estimated. I assumed conversion would keep the impressive figure and discard the caveat.

It didn't. The caveat sits 116 characters after the number, so they travel together no matter how I sliced the page. I tried chunk sizes from 1500 characters down to 300 and could not separate them. That is luck about how that particular block is written, not evidence the method is safe.

The real failure runs the other way: the number gets dropped and the prose stays!

That is much harder to catch. Truncated output looks truncated. This looks finished. You get smooth, confident, well-organized text with the quantities quietly missing, and nothing anywhere signals the loss. An agent reading it cannot tell it is answering from a summary with no evidence in it.

Finding three: ranking the best sections didn't rescue it

This was the feature I actually set out to test. Rank the sections against the question, return only the top few, skip the rest.

Against raw HTML the numbers look spectacular, up to 1,440x. But any real product converts the page first, so the fair question is what ranking adds on top of conversion. Below, "k" is how many sections you send back, and "recall" is how often the correct answer was actually in them.

Cheap and wrong, or correct and barely cheaper. Sending three sections found nothing that sending one didn't. Every accuracy gain arrived at ranks four and five, which is exactly where the savings disappear. No setting gave both.

Two pages went backwards. On the news article and the product page, sending the top five sections cost more than sending the whole converted page. Short pages break into few chunks, and the chunks overlap, so you ship some text twice.

The most useful row was my own site, because it is the one page where I had a hand-written version to compare against. My hand-written markdown: 1,464 tokens, and it answers all three test questions. Ranked retrieval at five sections: 1,468 to 1,702 tokens, and it answers two. A human-written full page beat the automated approach on cost and accuracy at the same time. Small sample, and I wrote both the page and the summary, so weigh it accordingly.

One miss was structural rather than fixable. I asked Wikipedia's transformer article, "if it reads all the words at once how does it know which came first." The correct section is titled "Positional encoding." It ranked 12th out of 43. The words in my question pull toward a different section about tokenization, and a small ranking model has no way to connect that phrasing to that term. Catching it would mean sending twelve sections, about a third of the page.

What I'd take away

A compression ratio without a fidelity check is only half a measurement. Plenty of tools publish reduction numbers. As far as I can find, none publish what share of prices, specs, or table values survive. Those are two different questions, and only one is being asked.

Test on the pages you actually care about. The spread here was more than 500 to 1. A single headline multiple means very little across mixed page types, including mine.

If your content matters commercially, check whether it survives. Run an extractor on your own product or pricing page and search the output for your price. That takes about five minutes. I did it expecting to confirm something and found a hole in a page I wrote myself.

If you publish for agents, a hand-written version is still hard to beat. It won on both measures against every automated approach I tried, on the one page where I could make the comparison.

One last thing worth logging. Eight of my page fetches were refused, including one that returned a normal HTTP 200 along with 1.09 MB of a completely different page. If you are building anything that fetches pages from a client, that is a number to discover early...

The pages, the code, the answer key, and the raw downloads are all saved, so these numbers run against stored bytes rather than a web that keeps changing. Disclosure: I build token-efficiency tooling. That is why I went looking for the 5x baseline, and why I am irritated it isn't there.

J. Gravelle
jCodeMunch.com