Every few months, another model ships with a bigger context window.
1M tokens. 2M. Someone's already at 10M.
The pitch is always the same:
Feed it more, get better answers.
I ran into the opposite while building a retrieval system designed to enforce coding guidelines before code generation:
More context in, worse output out.
Turns out, that's not a fluke.
It's a consequence of how transformers work.
The Problem: Context Rot
Dump a 300-page PDF into a prompt and you'd expect the model to have "read the whole thing."
What actually happens is degradation.
Studies from Chroma and Anthropic show a similar pattern: as the surrounding context grows, reasoning accuracy on the actual task can drop.
Not because the model literally forgot the tokens. They're technically still in the context.
The problem is that the model is reasoning over a much noisier signal.
More context doesn't automatically mean more useful context.
Lost in the Middle
There's a well-documented effect where models are better at retrieving information from the beginning or end of a long prompt than from the middle.
If the fact you need is buried somewhere in the middle of a large document, retrieval and reasoning accuracy can drop significantly.
And that's not a rounding error.
It's the difference between:
- a demo that impresses a stakeholder, and
- a production system that quietly gives the wrong answer.
The model might technically have access to the information.
That doesn't mean it can reliably use it.
Needle in a Haystack Is the Wrong Benchmark
A lot of the "look how big our context window is" marketing leans heavily on needle-in-a-haystack tests.
The setup is simple:
- Take a huge document.
- Hide one specific fact somewhere inside it.
- Ask the model to find that fact.
That's useful for measuring retrieval.
But it doesn't tell us much about reasoning across a large context.
Real-world tasks are usually more like:
Find three relevant pieces of information from three different sections, understand how they relate, and use them to make a decision.
A model can ace a needle-in-a-haystack benchmark and still fall apart when it needs to synthesize multiple pieces of information scattered across the context.
Finding one needle isn't the same as understanding the haystack.
Why This Happens: It's Architectural, Not a Bug
This comes down to the attention mechanism itself.
At a high level, tokens attend to other tokens to determine what's relevant.
As the amount of input grows, there are more relationships to reason over.
The model now has to distinguish the useful signal from an increasingly large amount of surrounding information.
There's no simple patch coming for this.
It's not necessarily a broken mechanism.
It's the mechanism doing what it was designed to do, just at a scale where the signal-to-noise ratio starts working against you.
And that's an important distinction.
The problem isn't simply:
"The model isn't smart enough."
The problem can be:
"We're giving the model too much information to reason over reliably."
What Actually Helps (and Where It Breaks)
RAG is the obvious mitigation.
Instead of dumping everything into the prompt, filter the data before it reaches the model.
Retrieve only the information that's likely to matter.
It works.
But RAG has its own failure mode.
Fetch the wrong chunk, or fetch a pile of "almost relevant" chunks, and you've recreated the same dilution problem you were trying to avoid.
I hit this directly while building a guideline-retrieval system.
A surprising amount of the engineering work wasn't the retrieval itself.
It was ranking.
The system needed to distinguish:
- highly relevant chunks
- partially relevant chunks
- superficially similar chunks
- completely irrelevant chunks
Because if you return ten "almost relevant" chunks alongside the one chunk that actually matters, you've made the model's job harder again.
Retrieval quality isn't just about finding relevant information.
It's also about not returning information that looks relevant but isn't.
Context Engineering and Recursive Models
Newer approaches like context engineering and recursive language models are trying to address the problem from a different angle.
Instead of asking:
"How do we fit more tokens into the context window?"
the question becomes:
"How should the model ingest, organize, compress, and reason over information?"
That's a much more interesting direction.
These approaches are still relatively early, but they're worth watching because they challenge the assumption that the solution to context limitations is simply more context.
The Takeaway
Sam Altman has talked about AI eventually having perfect context over your entire life.
Maybe that happens.
But there's a real gap right now between that promise and what a transformer can reliably reason over, even with tens of thousands of tokens.
If you're building anything RAG-dependent or long-context-dependent, that gap is where a lot of your bugs are going to come from.
Not necessarily because the model is "not smart enough."
But because the architecture is doing exactly what it's built to do at a scale where more information can actually make the answer worse.
The question isn't:
How much context can the model handle?
It's:
How much context can the model reliably use?
Those are very different numbers.
If you're fighting this in production, what has actually moved the needle for you?
Better chunking? Re-ranking? Context compression? Something else?
