Main Point
Budget the actual formatted input and output allowance, test how the model uses relevant evidence, and calculate temporary state from the architecture. These records answer different parts of the request’s feasibility and usefulness question.
Key Points
- A provider’s accounting convention determines which limits apply.
- Documents and history share the remaining allowance with any additional formatting or margin.
- Relevant evidence needs to be found, interpreted and associated with the right task.
- KV payload depends on layers, retained positions, requests, KV heads, head dimension and precision.
- Query-head count and KV-head count can differ in a grouped architecture.
- A full-grid pair-count comparison does not determine end-to-end latency.
1. Budget a defined request
Use a supplied hypothetical combined limit of 16,000 tokens. This is a classroom convention, not a named provider’s specification.
| Allocation | Tokens |
|---|---|
| Instructions | 1,000 |
| Tool definitions | 1,000 |
| Current question | 500 |
| Reserved output | 2,000 |
| Remaining documents and history | 11,500 |
`16,000 − 1,000 − 1,000 − 500 − 2,000 = 11,500`
The 500 allocation belongs to the current question. Additional formatting and any chosen margin still consume capacity. Count the final representation using the matched tokenizer/version/template and the actual service’s input/output convention. Reserving 2,000 output tokens does not require the model to generate all of them or establish answer quality.
2. Test use of the evidence
The museum’s fictional packet includes a room closure that the answer must associate with the correct date. Keeping the packet within its allowance does not test that association, source support or conflicting notices.
For a controlled placement test, move the same relevant notice among the other supplied material while holding the question and scoring rule fixed. Compare beginning, middle and end placements, then record answers and support. Observed results remain blank until an actual authorized test.
Liu and colleagues’ *Lost in the Middle*, published in February 2024, studied multi-document question answering and key-value retrieval. Its tested models often performed differently as relevant information moved, with weaker use in the middle of long contexts. This motivates a test design; it supplies no result for the museum or every later model. Use representative questions and current applicable evidence when judging the actual task.
3. Define the cache payload
For a simplified retained KV payload:
`bytes = 2 × layers × retained positions × requests × KV heads × head dimension × bytes per stored value`
The 1st factor counts keys and values. The final factor counts bytes for each numerical value. Count the actual retained positions and implementation, rather than assume every reserved output slot has already been processed or allocated in the same way.
The supplied configuration has 32 layers, 8,192 positions, 1 request, 8 KV heads, 128 coordinates per head and 2 bytes per value:
`2 × 32 × 8,192 × 1 × 8 × 128 × 2 = 1,073,741,824 bytes = 1 GiB`
A GiB contains `1,024³` bytes. The same amount is approximately 1.074 GB in decimal units. Keep the unit with the result.
This is KV payload only. Weight storage, other activations, workspace, metadata, padding and allocation effects require additional accounting. No device-fit or runtime measurement is supplied.
4. Inspect the KV heads and sharing assumptions
With every other factor unchanged, 32 KV heads instead of 8 give `32/8=4` times the payload: 4 GiB for 1 request. 2 independently stored such request payloads give 8 GiB. Supported sharing, sharding, replication and precision can change actual physical memory, so record the implementation rather than silently changing assumptions.
Ainslie and colleagues’ 2023 grouped-query attention work describes query-head groups sharing key/value heads. The KV-head factor therefore need not equal the number of query heads. This is an architectural choice associated with a trained configuration. The paper’s conversion/training and performance results are outside this classroom memory calculation.
5. Interpret the full-grid comparison
A mathematical full attention score grid has a query-position axis and a key-position axis. Doubling both lengths multiplies its entries by 4. Causal masks restrict permitted relationships; optimized implementations can avoid storing the full score array, and architectural variants change the work.
The 4× entry count establishes the stated geometric comparison. Actual memory use and latency also depend on kernels, other computation and serving conditions. An end-to-end response-delay claim requires measurement.
6. Use the 3 records together
The token budget tells the museum how much defined material fits. Evidence-use tests examine whether the answer uses the right notice and resolves the task. The cache calculation identifies a temporary-state payload under explicit architecture assumptions. A useful service needs a satisfactory combination of all these conditions.
The next lesson examines another distinction in architectural resource counts: a mixture-of-experts model’s total parameters and the subset active for a token.
7. Learning Checks
Attempt each question before opening its explanation. The worksheet leaves calculations and observations blank.
M14-L12-Q01
A hypothetical combined input-and-reserved-output allowance is 16,000 tokens. Instructions use 1,000, tool definitions use 1,000, the current question uses 500, and output reserves 2,000. How much remains for documents and history before additional formatting or margin? If the packet fits, what further checks are needed before trusting its opening-hours answer?
Your calculation and explanation: ____________________
Show explanation after your attempt
Subtract 1,000+1,000+500+2,000=4,500 from 16,000, leaving 11,500 tokens for documents and history. Additional formatting and margin must still fit under the stated accounting rule. Then inspect applicable evidence, room/date associations, conflicts and source support; test representative questions and evidence placements, and check actual resource availability. Fit addresses capacity while the task and configuration still need evaluation.
M14-L12-P01
A supplied cache payload is 1 GiB for 8 KV heads and 1 request. Keep layers, positions, coordinates and precision fixed, use 32 KV heads, and retain 2 such requests with their cache payloads stored separately. What is the combined payload? Does it guarantee that the complete service fits on an 8 GiB device? If sequence length doubles, does a 4× full-score-grid entry count establish 4× actual response delay? State the assumptions and limits.
Your calculation and explanation: ____________________
Show explanation after your attempt
The head change contributes 32/8=4 and the 2 separately stored request payloads contribute another factor of 2. The combined cache payload is 8 GiB under the supplied assumptions. The service also needs weights, other activations, workspace, metadata and implementation-specific allocation, so device fit requires more evidence. Doubling both dimensions of a mathematical full score grid gives 4× entries. Actual latency depends on permitted/computed relationships, implementation and other serving work; measure it rather than assert that multiplier.
Sources
- Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. *Lost in the Middle: How Language Models Use Long Contexts*. TACL 12:157–173, February 2024 (publication month on authoritative PDF page 1). Primary paper; publication record. DOI: 10.1162/tacl_a_00638. Retrieved 30 October 2026. Historical position-controlled question-answering/retrieval study; no current-model or museum outcome adopted.
- Ainslie, Joshua, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón and Sumit Sanghai. *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*. Original preprint 22 May 2023; v3, 23 December 2023; accepted at EMNLP 2023. Primary paper, section 2.2 and Figure 2; version history. Retrieved 30 October 2026. Used for query groups sharing KV heads, with no benchmark or automatic checkpoint-conversion claim.
- Hugging Face. Transformers v4.57.1: Caching, “Attention matrices”. Version-specific documentation; page publication date unstated, associated release 14 October 2025. Retrieved 30 October 2026. Per-layer computed KV state and retained positions; not a universal memory constant. All numerical configurations in this lesson are supplied classroom inputs.