Tenant Isolation and Cache Safety in Multi-Tenant LLM Systems
aiengineering
Why every KV cache, embedding, and prompt cache key needs a tenant ID — and the five-layer defense that keeps context from leaking across customers.
Multi-tenant LLM systems fail quietly. A shared prompt cache with the wrong key can serve Customer A's data inside Customer B's answer, and no error will fire.
Where cross-tenant leakage hides
- Prompt caches keyed only on prompt text
- Semantic caches where embeddings collide across tenants
- Vector indexes without tenant filters at query time
- Agent scratchpads persisted in a shared store
- LLM-as-judge caches replaying another tenant's reasoning
The isolation contract
Every cache key, index shard, and log line carries a tenant ID. Not as a metadata field — as part of the primary key. If tenant ID is ever optional, isolation is already broken.
Five-layer defense against prompt injection and context bleed
- Input sanitization — strip instructions from retrieved content before it reaches the prompt.
- Structural separation — put untrusted content inside typed delimiters the model is trained to treat as data, not instructions.
- Capability gating — tools available to the model are scoped by tenant and role, checked server-side.
- Output validation — parse, schema-check, and reject any tool call that references resources outside the tenant.
- Audit logging — every retrieval, tool call, and cache hit written with tenant ID for after-the-fact review.
Verification
Run a red-team suite that impersonates Tenant A and asks for Tenant B's data across every entry point: chat, agent tool, retrieval, and cache warm-up. Any single leak is a stop-ship.
For how this looks in a shipped product, see Governance Inside SNAP: Tenant Isolation, Audit, and Access Control.