Read the video transcript
The umbrella retailer's nightly forecast performs a great deal of numerical work. A seller offers an accelerator with twice the peak arithmetic rate. It sounds as if the forecast should run much faster. But the retailer's current job repeatedly moves values from memory. If that movement already sets the smaller limit, paying for more arithmetic capacity may change little. Lesson nine: Arithmetic intensity and the roofline. By the end, you should be able to calculate operations per byte, compare a compute ceiling with a memory-movement ceiling, and explain why a higher ceiling is not a guarantee of achieved speed.
We'll take three steps. First, measure work relative to bytes moved. Second, calculate the two simplified ceilings. Third, decide what kind of change is worth testing, including its engineering cost. The guiding question is: for this particular workload and memory boundary, which resource sets the lower ceiling? Arithmetic intensity is useful numerical operations divided by bytes moved across a specified memory boundary. Its unit is operations per byte. We already know that operation counts need a convention and that byte traffic needs an interface.
Keep both labels in the ratio. Suppose a stage of the retailer's forecast does two trillion counted operations and moves one trillion bytes across a chosen external-memory boundary. Two trillion divided by one trillion gives two operations per byte. This is an illustrative way to reach the starting intensity in this example. It does not mean every stage of the retailer's service has the same ratio. Low intensity means many bytes cross the boundary for each operation. High intensity means the work can use loaded information for more calculations before bringing additional bytes across that boundary. Reuse can raise the ratio by reducing movement while keeping useful work.
Change the boundary and the intensity can change. Counting bytes between a cache and a processor is different from counting bytes between external memory and the chip. A useful roofline calculation states which movement path it describes. Otherwise a work-per-byte number floats without a physical meaning. The roofline model uses two simplified upper bounds on a numerical workload's rate. One is the relevant peak compute throughput: how many counted operations per second the hardware can perform under the matching conditions. The other is memory bandwidth times arithmetic intensity. Check the units.
Bandwidth is bytes per second. Intensity is operations per byte. Multiply them, cancel bytes, and the result is operations per second. It tells us the highest operation rate that movement could support under this simplified picture if each moved byte supplies the stated amount of work. The smaller of the two ceilings is the roofline bound. If memory can supply only enough data for two trillion operations per second while arithmetic hardware could do one hundred trillion, two trillion is the smaller limit. If the situation reverses, compute is the smaller limit.
This is a first approximation and an upper bound, not an exact forecast. It is valuable because it identifies which type of resource a proposed upgrade addresses. Use hypothetical hardware: relevant compute peak of one hundred trillion operations per second and memory bandwidth of one trillion bytes per second at the chosen boundary. Suppose the retailer's workload has intensity of two operations per byte. Multiply one trillion bytes per second by two operations per byte. The movement-related ceiling is two trillion operations per second. The compute ceiling is one hundred trillion operations per second.
The smaller is two trillion. Under this simplified model, the workload is memory-bound at that intensity. Memory-bound does not mean the device has no arithmetic hardware or that it spends exactly a known fraction of time idle. It means movement sets the tighter of these two particular upper bounds. The retailer still has to measure actual performance and check other limits. If someone divided the workload's operation count by the one-hundred-trillion compute peak and announced an elapsed time, that estimate would ignore the lower movement ceiling. It would be the wrong rate for this work-per-byte pattern.
Now suppose the retailer buys a device with twice the relevant compute peak: two hundred trillion operations per second. Keep bandwidth at one trillion bytes per second and intensity at two operations per byte. The movement ceiling remains two trillion operations per second. The smaller bound is still two trillion. In this simplified analysis, buying more peak arithmetic did not raise the ceiling. Try a different intervention. Improve reuse so the workload performs twenty operations for each byte moved across the same boundary. The memory ceiling becomes one trillion bytes per second times twenty operations per byte, or twenty trillion operations per second.
That is a higher ceiling than two trillion, while still below the original one-hundred-trillion compute peak. This does not promise the retailer a tenfold speedup. The numbers are bounds, and other effects may prevent the program from reaching twenty trillion. It shows the mechanism: reuse or reduced movement changes the work-per-byte ratio that was constraining the original comparison. Representation, operation fusion and workload organization might also change traffic or work arrangement. The retailer would test whether one of those changes raises useful performance at acceptable quality and engineering cost. The ridge point is where the two roofline ceilings meet. Divide the compute peak by memory bandwidth.
For the original hardware, one hundred trillion operations per second divided by one trillion bytes per second gives one hundred operations per byte. The per-second units cancel, leaving the intensity unit. Below one hundred operations per byte, the bandwidth-times-intensity line is the lower of the two ceilings in this example. Above one hundred, the one-hundred-trillion compute ceiling is lower. A roofline diagram draws that rising movement line and flat compute line, with the ridge where they meet. You can reason through the comparison without memorizing the graph. Real workloads can land below either ceiling. Small dimensions and kernel launch overhead can limit work.
The instruction mix may differ from the peak's ideal operation. Devices may communicate. Serial dependencies, queues and thermal behavior can reduce achieved rate. Choosing a cache-level boundary instead of external memory can also produce a different roofline picture. So the roofline is a diagnostic lens that needs measured operations, traffic and relevant hardware rates. It does not prove that a code change will attain the bound. Suppose measurement confirms low arithmetic intensity at the external-memory boundary. The retailer can investigate keeping loaded values for more calculations, changing representation to move fewer bytes, fusing suitable operations,
or reorganizing the workload. These are technical candidates, not an automatic prescription. Each has a cost. A small software change that safely improves reuse may be valuable. A difficult rewrite could cost more engineering time than the benefit justifies. A different hardware design may be simpler for this workload, or more expensive overall. The retailer needs to compare the expected and measured improvement against purchase cost, energy, quality and development effort. The mechanism tells us what to try first.
The economic decision asks whether the qualified task becomes better enough to pay for that change. Keep those questions linked but distinct. Use the original hardware for this checkpoint. At fifty operations per byte, what is the roofline ceiling? What about one hundred fifty operations per byte? At fifty, one trillion bytes per second times fifty operations per byte gives fifty trillion operations per second. That is below the one-hundred-trillion compute peak, so fifty trillion is the simplified ceiling. At one hundred fifty, the movement ceiling would be one hundred fifty trillion, but compute tops out at one hundred trillion.
The smaller compute ceiling is one hundred trillion. Now return to the seller's proposal. If intensity stays at two operations per byte, does doubling compute peak from one hundred to two hundred trillion operations per second raise this simplified ceiling? No. Bandwidth times intensity still gives two trillion operations per second. That remains smaller than either compute peak. The purchase might affect other properties, but extra peak arithmetic alone does not change this roofline limit under the stated assumptions. Arithmetic intensity connects numerical work with bytes moved across a named boundary.
The roofline compares relevant peak arithmetic against bandwidth times that intensity, then takes the smaller upper bound. In our hypothetical example, two operations per byte produced a two-trillion-operation-per-second movement ceiling despite a one-hundred-trillion compute peak. The ridge point was one hundred operations per byte. Changing reuse from two to twenty raised the movement ceiling; simply doubling compute peak did not. Actual performance can still be lower because of software, communication, queues and other limits. In Lesson ten, we'll compare training with a forward prediction. That changes the kinds of numerical work and state the system must handle, so we'll revisit resource estimates with the learning process attached.