Read the video transcript
The umbrella retailer improves its assistant's aggregate output. Across all customers, the serving system now produces more tokens each second. Yet some customers say their answers start later and arrive less smoothly. How can the service produce more while feeling slower? Lesson thirteen: Batching, concurrency, and queueing. By the end, you should be able to distinguish total throughput from an individual's wait, explain what happens when requests arrive faster than sustainable service, and use a simple rate-times-time estimate without mistaking it for a burst forecast.
We'll follow three steps. First, see what batching changes. Second, trace arrivals, active work and the queue. Third, choose service controls and measures that match the customer's task. Our guiding question is: who benefits when the system handles more work at once, and who may wait? Batching groups work for execution. In model serving, the retailer's assistant may process several customer sequences together on an accelerator. That can make better use of loaded model weights and hardware.
More total output may be completed per unit time. That total is aggregate throughput. An individual customer experiences latency: time from a defined request start to a defined visible result. The customer may first wait for a group to form or for access to hardware. Later, the answer may stream with gaps between tokens. The higher total rate says nothing by itself about those individual waits. Picture a bus that carries more people per trip. The transport system can move more people per hour, while one rider waits longer for the bus to depart.
The serving system is more complicated, but the distinction is the same: total completed work and one person's elapsed time answer different questions. We need both measures. From Lesson twelve, each active sequence also carries context state such as a key-value cache. Grouping more sequences can use device capacity more efficiently, but it can raise the memory needed at once. A batch choice is therefore a compute, memory and service-target choice together. Define the counts carefully. Arrival rate is new requests per unit time, such as customer questions per second. Concurrency is work active at the same time under a stated boundary.
Service capacity is how much work this system can complete under its current conditions. A request waiting in a queue may be counted in a broader in-system measure but is not necessarily executing on the accelerator. Say which boundary you use. If customers arrive faster than the service can sustainably complete requests, the queue grows unless the system rejects some work, delays it elsewhere, or demand changes. A brief spike can be absorbed for a time, but unlimited waiting is not a solution. The retailer needs a policy for what enters and what the customer sees. Even if the average arrival rate is lower than average capacity, bursts can make a queue.
A promotion might bring many questions in one minute. Some requests take longer because they include documents or need more output. Long jobs can hold resources that short jobs need. An average can hide that uneven experience. This is why measuring only average latency misses the slow tail. A ninety-fifth-percentile latency is a threshold under a particular measurement definition: roughly ninety-five percent of observed requests fall at or below it, and roughly five percent take longer. It does not promise that every customer gets that time, and its meaning depends on which requests and time window were measured.
A fixed batch can group several sequences and process them together until the group is done. If one customer's answer is much longer, some capacity may be underused while the others wait for the group to finish. Continuous batching lets sequences enter or leave an execution group as they progress. That can reduce this particular waste. But continuous batching is an operating method, not a guarantee of good service. The system must track each sequence's state and memory, schedule work, and protect latency targets. A group that is efficient for overall output may still create unacceptable first-token or later-token delays for a customer. Test the actual request mix rather than assuming the feature name proves the outcome.
For the retailer, imagine short refund-policy questions arriving alongside long exchanges about a complicated purchase. The long exchange may need more cached state and more decode steps. A scheduler can decide how to combine them, but its decisions change who waits. The right policy depends on the product's quality and response targets. Admission control decides which requests enter service under available capacity and policy. Quotas and limits can constrain a user's or application's load. Routing can send requests to a suitable model or serving pool. These are choices about the service, not changes to the customer's underlying question.
Autoscaling can add capacity as demand grows, or remove it when demand falls. New model-serving capacity may need time to load model weights and initialize. A sudden burst can arrive before the additional capacity is ready. The retailer cannot assume an autoscaling setting makes the first minutes of a promotion instantaneously safe. These controls have customer effects. Rejecting requests may be preferable to silently creating an extreme queue if the service can explain what happened and offer a retry. Routing to a smaller model may change answer quality. Limits can protect one customer from another's surge, but the policy needs to match the product promise.
Measure the chosen behavior, including failure and waiting, not only completed-request speed. Consider a simplified concurrency estimate. Suppose ten requests arrive each second, and the average time in the system is two seconds. Under a suitable stable condition with matching boundaries, average in-system work is arrival rate times average time: ten per second times two seconds equals about twenty requests in the system on average. Now keep the arrival rate at ten per second but let average time in the system rise to ten seconds. Ten per second times ten seconds equals about one hundred requests in the system on average. The rate did not change; longer time means more work present at once.
That presence may include work waiting as well as executing, depending on the defined boundary. The units help: requests per second multiplied by seconds gives requests. This is a rate-times-time relationship for a suitable steady state. It is not a rule that the retailer needs exactly twenty or one hundred accelerator slots, and it does not predict a peak burst. The request lengths, queues, resource usage and time window all matter. One way this estimate helps is as a consistency check. If a dashboard reports the same arrival rate but a much longer average in-system time, we should expect more average work somewhere in the system.
Then investigate where it is: queued, running, waiting on a tool, or waiting for memory and hardware. Higher concurrency can require more memory and hardware capacity. Better batching may lower the resource cost per completed answer, yet slower responses can reduce task completion or what customers are willing to pay. The cheapest configuration per token may not be the best retailer service. Set quality and latency targets before maximizing aggregate throughput. Does the customer need an immediate first sign of progress? Is a complete answer within a certain time more important? What happens at the ninety-fifth percentile, during a promotion, or when a long request arrives?
The answer should be tied to the task, not a single convenient benchmark. A useful dashboard can show arrival rate, completed work, queue time, first-token delay, inter-token pace, total response time and a latency distribution, all with clear boundaries. Some metrics may improve while others worsen. The team should identify the trade-off rather than calling every higher token rate a product improvement. Can higher tokens per second make a product feel slower? Yes. Batching and scheduling may raise aggregate throughput while increasing an individual's queue or inter-token delay. Compare the relevant distribution of customer waits, not only the total stream rate.
Ask when an answer first appears and how the later text arrives. Now check the stable-system estimate. At ten arrivals per second, what is average in-system work when average time is two seconds? What is it when average time becomes ten seconds, under the same suitable steady-state assumptions? About twenty requests at two seconds, and about one hundred at ten seconds. Requests per second times seconds gives requests. These are average in-system amounts with consistent boundaries, not the number of simultaneous accelerator executions and not a prediction of a peak burst.
Batching can raise total work completed per unit time while individual customers wait for grouping, resources or later output. Concurrency, arrivals and service capacity describe different quantities. Queues can grow under sustained overload and still appear during bursts below average capacity. Continuous batching, admission control, routing and autoscaling offer ways to operate the service, each with memory, quality and timing consequences. The retailer should judge them against the customer's actual task and the slow tail of response times. In Lesson fourteen, we'll examine speculative decoding: a different serving procedure that proposes and verifies tokens in an effort to reduce sequential waiting.