Context windows and model choice: how much your AI can actually remember
aieducation
Context window size shapes what tasks a model can handle, how much it costs, and when retrieval beats cramming everything in.
The context window is the amount of text a model can read in one shot — prompt, documents, prior turns, and tool output combined. It's a quieter dial than model size or thinking effort, but it decides which tasks are even possible.
What "context window" really means
If a model advertises a 200K-token window, that's the ceiling for input + output combined. Hidden thinking tokens eat into it too on reasoning models. Hit the ceiling and the oldest content drops or the request fails.
Why bigger isn't automatically better
Three trade-offs:
- Cost scales with tokens read. A 200K-token prompt costs ~200x a 1K-token prompt at the same per-token rate.
- Attention degrades with distance. Models often recall the start and end of a long context better than the middle — the "lost in the middle" effect.
- Latency grows roughly linearly with input size.
When a large window earns its keep
- Multi-document synthesis where chunking would lose cross-references.
- Long legal, clinical, or financial documents where every clause matters.
- Agent workflows accumulating tool output over many steps.
When retrieval beats a bigger window
For knowledge-base Q&A, pulling the 5 most relevant passages into a smaller window usually outperforms dumping the full corpus into a 1M-token model — cheaper, faster, and often more accurate.
How to choose
Measure actual token usage on representative tasks. If 95% of requests fit in 32K, paying for a 1M-token model is waste. If 20% spill over, either upgrade the window or add retrieval.
For the companion dimensions, see our overview of how AI thinks and reasoning models vs chat models.