Read the video transcript
A customer asks the umbrella retailer's assistant about a refund. One service begins showing an answer quickly but continues slowly. Another pauses longer before the first word appears, then finishes the reply sooner. Which one is faster? The answer depends on which part of the customer's wait we mean. Lesson eleven: Prefill and decode create different inference work. By the end, you should be able to trace the input and output phases of a generated reply, explain two user-facing timing measures, and calculate how the same hundred-token response can rank two services differently.
We'll follow three steps. First, see what happens before the first visible output. Second, follow repeated output generation. Third, compare first-token delay with completion time. Our guiding question is: when does the answer begin, and how quickly does the rest arrive? An autoregressive language model generates output in a sequence. Later output depends on the earlier context and on tokens it has already generated. A token is a piece of text handled by the model; it is not necessarily a full word.
The customer's request has an input phase. The service first receives the question and any relevant context, such as the retailer's policy or a long document. Prefill processes the input tokens and establishes numerical representations and state that later generation will use. That is not the model reading a document one word at a time like a person. The implementation organizes numerical transformations across positions according to the model's architecture and serving strategy. A long input can make prefill substantial. Some work across the input can benefit from parallel execution, though the details depend on the model and system. Before prefill, the request may also wait in a queue, be prepared by application software, and travel across a network.
The customer does not see a first answer token until these relevant stages and an initial generation step have occurred. After the input is prepared, the service begins decode: generating subsequent output tokens through repeated steps. Each step uses earlier context and generated state. Under the basic autoregressive process, the next token generally cannot be finalized before the preceding generation step supplies what it needs. That dependence makes decode different from simply processing all output positions at once. More devices do not automatically remove the sequence. The model may repeatedly access weights and cached context. Batching several users' requests can change how the device is used.
Those details can affect the interval between visible pieces of the reply. Do not turn these tendencies into fixed labels such as “prefill is always compute-bound” or “decode is always memory-bound.” Input length, output length, batch size, model design and software vary. Profile the actual service and its memory movement and calculation stages. For the retailer, a short customer question and a long attached policy document create different input work. A two-sentence reply and a page-long explanation create different output work. The stages have to be measured separately. Time to first token measures the delay until the first visible token, under a defined start and end boundary.
It can include queueing, preprocessing, relevant transport, prefill and the initial generation step. If we omit the queue from one measurement but include it in another, the figures are not directly comparable. Inter-token latency is the time between later tokens in the stream. A service may begin quickly, then leave long gaps as it continues. Another may begin later and stream smoothly. The first-token measure captures the initial wait; the later-token measure captures the pace after the answer starts. A performance dashboard needs both. One summary number can hide the cause of slowness and the customer experience.
When a customer says “it takes too long to respond,” ask whether the first visible answer is late, the continuation is slow, or both. The requested result matters too. A clear short answer may meet the customer's need with fewer generated tokens. An unnecessarily long reply needs more decode steps and may hold serving state longer, affecting time and cost. That does not mean cutting necessary explanation. It means output policy and task quality belong in capacity planning. A tool lookup or a longer reasoning chain can add work outside a simple final-token count. The retailer's assistant might ask an inventory system for a purchase record or validate a policy condition before sending a final reply.
Those waits belong in the whole-service measurement even if the language model's output rate is unchanged. The two phases also interact with memory. Prefill establishes relevant context state. Decode uses earlier context and generated state as each token arrives. Long inputs and many simultaneous users can increase what must fit, while repeated generation can increase movement. Lesson twelve will look more closely at the KV cache involved in that reuse. The practical point is to keep the service stages visible. “Tokens per second” alone does not explain why the first token took a long time,
and first-token delay alone does not describe how long a complete answer takes. Consider two hypothetical configurations and a hundred-token output. Configuration A takes two seconds before the first visible token, then produces fifty tokens per second. At that simplified constant rate, a hundred tokens take about two seconds to stream. Total time is about four seconds. Configuration B begins after half a second, then produces twenty tokens per second. A hundred tokens at twenty per second take about five seconds to stream. Add the half-second initial delay, and completion is about five and a half seconds.
B feels more responsive at the beginning: the customer sees text after half a second instead of two. A finishes the complete hundred-token answer sooner: about four seconds instead of five and a half. Neither figure alone captures both experiences. We are using approximate constant rates and a defined hundred-token output to make the comparison. Real streams vary from token to token. The first token is at the boundary between initial and streaming phases, so exact accounting can differ slightly. These rounded totals are enough to show why a dashboard needs more than one speed number. Suppose the retailer's first-token delay grows during busy periods while inter-token gaps stay similar.
A queue, preprocessing stage, transport path or prefill workload may have changed. It would be premature to buy faster decode hardware without locating the delay. Suppose the first token arrives promptly but the rest of a long answer crawls. Then inspect decode steps, weight and cache movement, batching and the chosen output length. A single “average response time” can blur those possibilities. The service target should match the customer's task. A short answer may make first-token time especially noticeable. A detailed report may make completion time more important.
Both still need an explicit quality standard; a fast but incorrect refund explanation is not a good result. Why should a performance dashboard include both first-token delay and inter-token measures? Input processing and initial waiting affect when the answer begins. Decode pacing affects how the later output arrives. A single speed figure can hide which stage is slow and what the customer experiences. Measure the boundaries separately and connect them to the whole request. One more check. For the hundred-token answer, which configuration begins sooner, and which finishes sooner under the simplified rates?
B begins sooner, after half a second, while A begins after two seconds. A then streams fifty tokens per second and finishes in about four seconds total. B streams twenty per second and finishes in about five and a half. Responsiveness at the start and total completion are different rankings. Prefill handles input context and establishes state. Decode builds output through repeated dependent steps. The first visible token follows queueing, preparation, transport, prefill and initial generation under the defined boundary. Later-token intervals tell us how the rest of the answer feels.
Output length, other tools and simultaneous requests can change full service time and cost. For the retailer, a useful dashboard shows both the initial wait and continuing pace, alongside quality and completion. In Lesson twelve, we'll examine the KV cache that stores selected earlier-context information for reuse during generation, and why long contexts and many users change memory needs.