Inside the context window: how AI decides what to keep and what to drop

The context window is the single most underappreciated constraint in generative AI. Every token the app wants the model to consider — system rules, retrieved documents, chat history, your prompt — has to fit inside it. Understanding how apps manage that budget tells you a lot about why one tool feels sharp and another feels forgetful.

What a token actually costs you

Tokens are the atoms of context. A page of English is roughly 500 tokens. A 128K-token window sounds huge until you add a 40-page policy PDF, a 60-turn chat, user memory, and tool definitions. Suddenly the model has 2K tokens left for your actual question.

The three strategies apps use when the window fills

Truncation. Drop the oldest turns. Simple, but the model forgets early constraints you set.

Summarization. Compress earlier history into a short recap. Preserves gist, loses specifics — and the summary itself can introduce errors.

Retrieval. Keep history in a database and fetch only the passages that match the current question. More complex, far more scalable.

Attention is not uniform

Even within the window, models don't weight every token equally. Research on "lost in the middle" shows accuracy dips for information buried between the start and end of a long input. Good apps put critical instructions and the question itself at the boundaries.

Why bigger windows aren't always better

Larger windows cost more per call, run slower, and encourage lazy context design — dumping instead of curating. A 32K window with disciplined retrieval often beats a 1M window stuffed with noise.

What leaders should ask vendors

• What's the effective context size after system prompts and tools?
• How is history compressed or retrieved?
• Can we inspect what was in the window for a given answer?

For the retrieval side of this story, read hybrid search and RAG. For platform-level controls, see the Silverberry AI Platform.