How The KV Cache Stores Conversation Context

The umbrella retailer tests its customer assistant with one refund question. The model fits on its accelerator and the answer appears. Then the service opens to many customers with longer policy documents. Memory fills, even though the model's learned weights have not changed. What grew? Lesson twelve: KV cache and long-context memory. By the end, you should be able to explain what the cache reuses, estimate its size under stated assumptions, and diagnose why a one-user fit test can fail in deployment.

We'll take three steps. First, distinguish learned weights from use-time context state. Second, build a cache estimate one factor at a time. Third, test what longer conversations and more simultaneous users do to the service. Our guiding question is: what else has to fit while a response is being generated? In the previous lesson, prefill processed the input and decode generated the answer through repeated steps. During attention, a model combines information from positions in its context. Here we only need the serving idea: some numerical representations from earlier positions can be stored and reused when later tokens are generated.

A key-value cache, or KV cache, stores selected key and value representations under the model's architecture. Keys help determine which earlier contextual information is relevant. Values carry information that can be combined into a later representation. They are numerical arrays, not a saved list of literal customer questions and answers. Without this reuse, a naive process could repeat more of the earlier-position work at every new generation step. Caching can save that repeated calculation, but the stored arrays occupy memory. As the retailer's assistant produces more output, there are more positions whose state may need to be retained. Keep two kinds of information separate.

Learned weights are the model parameters established by training. The KV cache is use-time state for the current context. Passing the model-fit test tells us that weights fit under that test; it does not prove every live request and its temporary state will fit. Suppose the retailer attaches a long refund policy to each customer request. A longer cached sequence generally means more key and value state under a conventional implementation. If thirty-two independent customers are active at once, the service can hold many such sequences. Neither change requires adding a single learned weight. The cache does not have one universal size.

The model's layer count, the number and grouping of key-value heads, the size and numerical format of each representation, the number of cached tokens, and the number of simultaneous sequences all matter. Shared prefixes, cache layout, and offloading can change where and how much memory is required. We should estimate for the actual architecture and serving policy. The basic growth direction is still useful. More context positions or more independent active sequences usually add state. A nominal maximum context length is therefore a product promise with capacity and cost consequences. Advertising that every customer can provide a much longer document may reduce how many customers the same hardware can serve together.

We will use a teaching approximation. Cache bytes equal two times the layer count, times cached tokens, times key-value heads, times head dimension, times bytes per value, times batch size. The leading two counts both keys and values. A layer is a stage in the model. Heads organize attention-related processing. Head dimension is the number of values in each stored representation. Batch size here counts equal-length sequences in the simplified example.

Before using a formula, name its boundary. This estimates the stored key and value arrays for the stated arrangement. It does not estimate model weights, all temporary workspace, allocator overhead, other serving buffers, or movement when a cache is offloaded. It is not a universal measurement of device memory. The factors have units that tell us what the answer means. Layers and heads are counts. Cached tokens count positions. Head dimension gives values per head.

Multiply by bytes per value, and the result is bytes. If you change the numerical format, you change bytes per value, but a smaller number does not automatically preserve quality or speed. Now use this hypothetical configuration: thirty-two layers, eight key-value heads, a head dimension of one hundred twenty-eight, eight thousand one hundred ninety-two cached tokens, two bytes per value, and one sequence. First calculate values for one cached token. Two, for keys and values, times thirty-two layers, times eight heads, times one hundred twenty-eight values per head equals sixty-five thousand five hundred thirty-six values. At two bytes each, that is one hundred thirty-one thousand seventy-two bytes for one token's stored state across those layers.

Now multiply by eight thousand one hundred ninety-two tokens. The result is one billion seventy-three million seven hundred forty-one thousand eight hundred twenty-four bytes. That is exactly one gibibyte, written one GiB, where a GiB is one thousand seventy-three million seven hundred forty-one thousand eight hundred twenty-four bytes. We are using binary memory units here, so do not silently swap in a decimal gigabyte. This is a worked cache estimate, not a claim about the retailer's real model. The example makes the scale visible and shows exactly which assumptions produced it. If the model has a different head arrangement or representation format, recalculate instead of carrying over the one-GiB result.

Under those same simplified assumptions, thirty-two independent sequences of the same cached length take thirty-two times one GiB: thirty-two GiB of key-value state. The weights have not grown. The active use-time state has. A one-sequence fit test that ignored this growth could give a false sense of deployed capacity. Real requests will not all have identical lengths. Some customers ask one sentence; others bring a long policy or a continuing conversation. Output tokens can extend the cached sequence during generation.

Sharing a common prefix may avoid some duplication where the system safely supports it. Offloading may move some state elsewhere, but then the movement path and its latency matter. The simple multiplication tells us a baseline, not every production allocation. If the retailer sees out-of-memory failures only during busy periods, inspect simultaneous sequence count, input and output lengths, cache allocation and temporary workspace. Look at actual memory measurements under the real request mix. A model can be small enough in weights yet too large as a whole deployed workload. Caching is a trade.

It can save repeated computation, but it consumes capacity and must be managed. If a cache entry is evicted, later work may need to be recomputed or fetched, changing performance. If state persists or is shared, the service must respect customer permissions and keep data appropriately isolated. A cache is part of the application design, not a free performance switch. Quantizing selected cached state can reduce its stored bytes, with possible quality and implementation trade-offs. A different layout or grouping may change how the hardware moves and uses state. These choices need end-to-end evaluation: answer quality, first-token time, streaming pace, memory, and cost. A smaller cache is only useful if the resulting service still meets its purpose.

For the retailer, the decision is operational. A long-context setting may improve some customer tasks, yet use more memory per active request. The team can choose a context policy, serving pool or capacity target with those costs visible, rather than treating the maximum context number as free. Why might a model fit in a one-user test but run out of memory after deployment? The one-user test may have counted weights and one small context. Deployment adds concurrent sequences, longer input contexts, generated output positions, temporary workspace and cache state. The working set grows even while learned weights stay fixed. Measure that full request mix before sizing capacity.

Try the numerical check. In this simplified configuration, one eight-thousand-one-hundred-ninety-two-token sequence uses one GiB of key-value state. What do thirty-two independent equal-length sequences use, and what remains unchanged? Thirty-two such sequences use thirty-two GiB under those same assumptions. The model weights remain the same size. This is cache state only; total deployment memory also includes weights and other working space. Sharing, formats and layout can change an actual implementation's footprint.

The KV cache stores selected numerical key and value representations for reuse during later generation. It is use-time state, separate from learned weights. The simplified estimate counts keys and values across layers, tokens, heads, dimensions, bytes and simultaneous sequences. In the worked example, one sequence takes one GiB of cache state, and thirty-two equal independent sequences take thirty-two GiB. Those figures have stated assumptions and do not include the whole service. For the retailer, a real capacity test includes long requests, output growth, concurrency and temporary workspace, alongside answer quality. In Lesson thirteen, we'll see how batching, arrivals and queues connect this per-request memory to throughput and customer waiting time.