Context windows and model choice: how much your AI can actually remember
aieducation
Why context window size is a third dial — alongside model size and reasoning effort — and how to pick the right one for long documents and agents.
Size and thinking are the two obvious dials. There's a third that quietly decides whether a model is right for your job: the context window — how much text the model can hold in working memory at once.
What the context window really contains
The window holds everything the model sees in one turn: the system prompt, the user question, any retrieved documents, prior conversation, tool outputs, and — for reasoning models — the hidden chain of thought. A nominal 200K window shrinks fast once you load a 50-page contract, a system prompt, and 10,000 reasoning tokens.
Why bigger windows aren't free
Three costs scale with window usage:
- Price per request — input tokens are cheap individually but add up at scale.
- Latency — larger contexts take longer to process, even before the model starts writing.
- Accuracy decay — most models get worse at finding specific facts buried in the middle of very long contexts, a pattern often called "lost in the middle."
Matching window size to the task
- 8K–32K: chat, short rewrites, single-document Q&A.
- 100K–200K: contract review, codebase navigation, multi-document synthesis.
- 1M+: whole-repository analysis, long agent trajectories, video transcripts.
For most enterprise workloads, retrieval into a mid-size window beats stuffing everything into a huge one — it's cheaper and more accurate.
When to pair big windows with reasoning
Agentic workflows need both: room to hold tool outputs and enough reasoning budget to plan the next step. If you cap one too tight, the other can't compensate. The context windows explainer goes deeper on sizing retrieval chunks.
For enterprise architecture guidance, explore Silverberry's AI Consulting Services.