How AI Generates 1 Token At A Time

Suppose the museum assistant begins displaying an answer quickly, but the visitor still waits for the complete explanation. The 1st visible words give us evidence that generation has started. What computation remains, and how will the application know whether the answer finished? Follow the known prompt into processing,

then follow a selected token back into the next input. We will connect that loop to streaming and the reason a response stops. Keep the trained checkpoint fixed for this lesson. The request contains its instructions, approved museum information and conversation

text in the chosen format. Ordinary inference computes with that input and the saved parameters. The parameter-learning signal from the previous lesson belongs to a different operation. For an inspectable trace, we will stipulate classroom sequence units instead of claim a production tokenizer result.

The initial prompt-processing phase is often called prefill. The prompt positions are already known, so their computations can be processed together under the model's causal mask. The last prompt position produces scores for the 1st new token. Earlier positions still have

their own information restrictions. This 1st pass also provides intermediate attention states that a suitable implementation can retain for subsequent steps. Use the supplied prefix The exhibition. Suppose the selection rule chooses opens. Append that selected unit, giving The exhibition opens.

The next computation can now select on. Append it, and the next input is The exhibition opens on. A further selection gives Monday. These are supplied choices for tracing the dependency. The later computation uses the extended input; our trace supplies no hidden,

prewritten complete answer or measured museum response. A key-value cache can avoid recomputing earlier attention keys and values. Each layer retains the relevant states for positions already processed. When the newly selected token enters the next computation, its keys and values join the earlier cache, and that

computation produces scores for the following token. Cached states depend on the earlier prefix, positions and model configuration. Matching only the last visible word is insufficient evidence that 2 requests can share them. A dated serving-system example makes the retained-state mechanism concrete.

Kwon and colleagues' September 2023 paper introduced PagedAttention and the vLLM serving system. It separated processing the known prompt from sequential generation and managed the growing key-value state. We use that historical design to inspect the dependency and memory problem. Its reported throughput results belong to its

tested models and workloads; this lesson supplies no speed measurement for our museum service. Caching removes some repeated computation while using memory for the retained state. It leaves the dependency between ordinary autoregressive steps: a later step needs earlier selected output. A long answer therefore requires

further work after the 1st token. Queueing, request processing, batching and delivery also affect observed delay. Record time to 1st output and time to completion separately before deciding what made a service feel faster. Streaming sends portions of the generated output while the process continues.

A displayed chunk can contain several tokens, and token pieces may need decoding or buffering before they form readable text. The application might display Museum hours followed later by an exception for Saturday. A staff member or automated workflow needs a way to distinguish that partial draft from the complete response.

Collect the required result and check its fields, evidence and exceptions before using it for an action. Generation can stop for different reasons. A model may select a configured end-of-sequence token. A maximum-output setting can stop a response before its sentence

or required fields are complete. An application can also enforce a stopping string, cancellation or another limit. Record the actual stopping condition under the implementation's contract. Even a normal end token leaves the application responsible for checking whether the result satisfies the task.

A useful trace therefore keeps several identities together: the exact checkpoint, tokenizer, formatted prompt, selection settings and output limit. Record the selected sequence, whether caching was used, 1st-output and completion times, and the stopping reason. A cache-sharing decision needs the relevant prefix and configuration to match

under the engine's supported rules. These records help separate a processing problem, a delivery delay and an incomplete result. Our worksheet leaves observation fields blank until an actual authorized run. Pause at the supplied trace. The prefix is The exhibition, and a baseline trace selects opens,

then on, then Monday. What full input is available immediately before the computation that selects Monday? In a separate run with the same supplied 1st choices, the maximum number of new tokens is 2. What output can that run deliver, and what does reaching its

cap tell us about completion? Why can a longer answer require further work after an early 1st token? Immediately before selecting Monday, the input contains The exhibition opens on. In the separate capped run, the 2 new units are opens and on. Reaching that output cap

can leave an unfinished continuation; it supplies no proof that the requested answer is complete. Each remaining selected unit requires its dependent step; that work continues after the 1st output. Inspect the stopping reason and the actual task requirements.

This reasoning follows the stipulated units and choices, rather than a production token-count measurement. Try a rail-service transfer. 2 requests end with the same word, but their earlier text specifies different routes and dates. A colleague proposes reusing the complete key-value cache solely because that final word matches.

What must be checked before accepting that reuse? The service also streams an answer that ends at a length limit. How would you decide whether it is ready to show as a final travel answer? Pause and make a brief plan.

The last word alone cannot establish compatible cached state. Earlier context and positions influence the retained keys and values. Check the relevant exact prefix, model configuration and the engine's supported reuse rules; process the new input when that match is absent.

For the travel answer, inspect the actual stopping reason, required fields and source-supported route and date. A length-limited fragment may need completion or repair before final use. The supplied scenario includes no observed journey or model result. The museum's early visible words now

have a useful interpretation: prompt processing and an initial selection have occurred, while later selections may still be pending. The application follows the growing sequence, temporary cache and stopping condition, then validates the complete task output. You can trace that dependency and diagnose a truncated response.

Next we will inspect the selection rules that choose among candidate tokens, including how temperature and candidate filters change the resulting sequence.