Read the video transcript
The umbrella retailer's stock-forecast model can produce a forecast quickly. Its team wants to improve the model using new examples. They look at the time for one forecast and multiply that by the number of examples to estimate training cost. That estimate misses much of what learning requires. Lesson ten: Training has more work than forward prediction. By the end, you should be able to trace the added backward and update work, explain an activation-memory trade-off, and calculate why four devices need not make a training step four times faster. We'll follow three steps.
First, compare one forward prediction with one training step. Second, list the extra state and distinguish two meanings of checkpoint. Third, see how distributed work and coordination affect useful progress. Our guiding question is: what has to happen to improve the model, beyond producing its current output? A forward pass takes inputs and the model's current parameters and computes outputs. For the retailer, that could be a forecast made from recent sales information. During inference, the system uses the current model to produce an output. That is one part of a training step, not the whole step.
Training compares the output with an example's target under a chosen objective. A loss is a number representing the mismatch or penalty under that objective. Backward computation then works back through the model to calculate and propagate gradients, which inform how selected parameters should change. An optimizer applies an update according to its procedure. The sequence is forward output, loss, backward gradients, and parameter update. The backward and optimizer stages add numerical work. They also need memory for intermediate and update-related information. A benchmark measuring only forward inference cannot tell the retailer the complete training step time or memory requirement.
We also should not imply that one lower loss value guarantees a better deployed model. The retailer will need evaluation on relevant new cases. This lesson is about the physical work and time to carry out training, not the full evaluation procedure. Activations are intermediate values produced as information passes through a model. To calculate gradients, training may need some of those values again during the backward stage. Keeping every useful intermediate value in memory can require substantial capacity. Activation checkpointing is a trade-off. The procedure retains less selected intermediate state during the forward pass, then recomputes needed values when backward work reaches them.
Less state is stored at once, but some calculation is repeated. It can reduce memory use while increasing computation and time. The name checkpoint can cause confusion. A training checkpoint saved to persistent storage is a recovery artifact. If a long job is interrupted, the saved state can help it resume. Activation checkpointing concerns runtime intermediate values and recomputation inside a step. A recovery checkpoint concerns the job's progress across failures. They solve different problems even though both use the same word.
For the retailer, the distinction matters when someone says, “We added checkpoints and saved memory.” Ask which kind. Writing a recovery file does not automatically reduce the live activations; retaining fewer activations is the memory trade-off described here. Optimizer state can also occupy substantial memory. Some update procedures keep additional values for each parameter so they can decide the next change. The required amount depends on the chosen method, not just the number of saved weights. The number format can differ by role. Weights, intermediate accumulations, gradients and optimizer state need not all use the same precision.
As we saw in Lesson four, formats can change memory footprint and numerical behavior. A capacity estimate must reflect the actual procedure and implementation. There is no reliable universal multiplier such as “training memory is exactly three times model weights.” That slogan ignores activation size, the optimizer, formats, batch size and software choices. The retailer should inventory the actual state and measure peak use under the intended training step. Distributed training uses multiple devices. There is more than one way to divide the job. In data parallelism, devices hold copies of the model and work on different examples.
Their update information must be coordinated so the copies continue in a consistent training process. Tensor parallelism divides selected calculations or numerical arrays across devices. Pipeline parallelism assigns different parts of the model to stages, so work flows from one stage to the next. Expert parallelism distributes selected expert components in a model that uses them. These names describe the division of work, not a promise that any division is fast. Each division creates communication needs. Devices may synchronize updates or transfer intermediate values. The pattern affects which network paths and topology are useful.
Here, the diagnostic point is that adding devices can make more arithmetic available while also adding waiting, coordination and failure complexity. The retailer should measure a complete training step, including transfers and synchronization, rather than add device peak rates on paper. Consider a simple idealized example. Imagine a training workload of one hundred work units. Eighty units can be perfectly divided among devices. Twenty units must remain serial: those parts still happen in sequence. With one resource, the total is eighty plus twenty, or one hundred units. With four equal resources and perfect division of the parallel portion, eighty divided by four becomes twenty.
The serial twenty is still twenty. The new total is twenty plus twenty, or forty units. Speedup is original time divided by new time under the same work-unit scale. One hundred divided by forty is two point five. Four resources produced an ideal two-point-five-fold improvement, not fourfold. Real distributed work also communicates and synchronizes. That added time can make the achieved speedup smaller than two point five. The example is not an exact forecast for the retailer's training run.
It isolates why the serial portion remains and why summed arithmetic capacity does not equal end-to-end step speed. Hardware time is only part of training cost. The retailer also prepares data, tries experiments, evaluates results, saves recovery checkpoints, handles interruptions and pays for engineering. A more expensive configuration might shorten useful training time, or it might mostly increase idle and coordination cost. A device can be reserved for a job while the job is stalled. Allocation or utilization figures need to distinguish paid capacity from actual progress through completed steps and new model state that can be evaluated. A stopped job can consume money without producing a better validated model. The comparison should use complete step time and useful progress toward a model that meets the task target.
If adding devices reduces arithmetic time but adds enough communication or failures, the overall project may not improve. Measure the full process before treating a higher peak throughput number as economic value. Why can activation checkpointing reduce training memory while increasing computation? It keeps less selected intermediate activation state during the forward work and recomputes needed values for gradients during backward work. The saved runtime memory is exchanged for repeated calculation. This is different from saving a persistent training checkpoint so a failed job can restart. One more check. If eighty work units are perfectly parallelizable and twenty are serial, what ideal total remains with four equal resources?
What is the speedup from the original one hundred units? The parallel portion falls from eighty to twenty. Add the unchanged serial twenty, and the total is forty. One hundred divided by forty is two point five. Communication and synchronization could lower the achieved speedup further. A forward prediction uses current parameters to make an output. Training adds loss, backward gradients, optimizer updates, retained or recomputed activations, and update state. Those requirements depend on the chosen procedure and formats, so a forward-only benchmark cannot price the full job.
Activation checkpointing trades runtime memory for recomputation. A recovery checkpoint saves training progress for interruption. Distributed work can divide examples, calculations, model stages or experts, but communication and serial work remain. In this ideal case, four resources gave a two-point-five-fold speedup. In Lesson eleven, we'll return to serving a language model. Processing the input and generating output tokens create different work and different parts of the customer's wait.