Read the video transcript
The umbrella retailer trains a demand forecast across several accelerators. Each device can calculate quickly, yet the training step spends more time waiting for devices to exchange information than doing useful computation. Buying faster arithmetic hardware sounds attractive. Would it fix this particular delay? Lesson fifteen: Communication, scale-up, scale-out, and rack systems. By the end, you should be able to name the path and message pattern that a multi-device job needs, explain why a peak link rate is not a full performance answer, and compare the compute-versus-communication timing options in our example. We'll take three steps.
First, map the links and boundaries between devices. Second, distinguish communication patterns and their waits. Third, work the timing example and connect the rack to power and cooling. Our guiding question is: where does the information have to travel for useful work to continue? When a model or training job uses multiple devices, some information must move between them. Communication bandwidth is a data rate across a specified link or system boundary. Communication latency is the delay for a message or operation. Topology is how devices and links connect.
The job's communication pattern decides which property matters most. A large continuous transfer may care strongly about effective bandwidth. Many small exchanges can suffer from startup and message latency, even when the advertised maximum rate is high. If several transfers compete for the same route, contention can lower the rate they actually achieve. We must name the endpoints and path before comparing numbers. For the retailer's training job, a device may need partial results from other devices before the next synchronized step. The wait is not visible from one accelerator's compute rating. We need a trace of the whole step: computation, messages, dependencies and idle intervals.
Scale-up generally connects devices into a closely integrated system. Scale-out connects systems or nodes across a broader network. These are architectural distinctions, not fixed distance or speed guarantees. Each product defines its own boundaries and routes. A job might cross an inside-system path for one exchange and a broader network path for another. A rack-scale arrangement coordinates processors, networking, power, cooling and software across a larger unit. The rack is more than a container for accelerator cards. It is a physical and operational system that must deliver power, carry data, reject heat and recover from faults.
Examples of ecosystem-specific paths include NVLink, AMD interconnect arrangements, TPU inter-chip communication and AWS accelerator networking. Their names are not interchangeable units of performance. When comparing designs, specify endpoints, topology, effective throughput, message behavior, software support and failure domains needed by the workload. Suppose the retailer is offered a high link-rate figure. Ask: between which devices? Is it one direction, one link, the sum of many links, or the effective rate the job sees?
Can the software use it for this communication pattern? A label or maximum does not answer those questions. A collective communication operation involves multiple participants. In all-reduce, participants combine values and obtain an aggregate under the operation. This can appear when a distributed training job combines information such as gradients. In all-gather, participants collect pieces distributed across them. In all-to-all, participants exchange different pieces with one another, as can happen in some expert-routing designs. Those patterns place different demands on the system.
A single large file moving from device A to device B is a point-to-point transfer. It does not tell us how well a many-participant all-to-all or all-reduce will perform. Message sizes, ordering, software and topology can change the result. A synchronized job may wait for its slowest participant. If one route is congested or one device takes longer, others can sit idle. Some computation and communication may overlap, but that depends on the algorithm and implementation. A high peak link rate does not prove a short end-to-end training step. For the retailer, the useful question is which exchange holds up the next forecast-training step.
Trace where devices send partial results, when they can compute again, and whether data movement overlaps useful calculation. This identifies the actual boundary to improve. Closer integration and faster paths can improve selected communication, but they concentrate power and cooling needs. A rack-scale system has to deliver enough electricity at the needed density, provide appropriate redundancy and remove the heat produced by its equipment. If the facility cannot support those requirements, a fast network design alone cannot produce the planned capacity. That is an ecosystem connection. Chip and interconnect choices affect the building, its electrical path and its thermal path.
Here, remember to include them when evaluating a larger compute arrangement. The retailer's business case should therefore include more than the accelerator price. Useful step time, networking, software, power delivery, cooling and downtime all affect the cost of training. Faster links can create upstream infrastructure needs, and a faster card can sit idle if communication remains the bottleneck. Consider a simplified training step with four seconds of useful computation and six seconds waiting for communication. Add them, and the step takes ten seconds under the assumed non-overlapping trace. What if computation becomes twice as fast while communication stays the same? Four seconds of compute become two.
Add the unchanged six-second communication wait: the step takes eight seconds. We saved two seconds, not half of the whole step. What if communication time is halved while computation stays four seconds? Six seconds become three. Four plus three gives seven seconds. Under these specific assumptions, reducing communication saves three seconds and beats the faster-compute option by one second. This is a diagnosis example, not a universal purchasing rule. If computation and communication overlap in the real job, or a software change alters the pattern, the simple sums change.
Measure actual traces before choosing an intervention. The lesson is to improve the part that controls useful progress, rather than choosing the component with the most impressive isolated peak. “More network bandwidth” is incomplete because we do not yet know the message size, latency sensitivity, topology, contention, collective pattern, software path or overlap. A workload that sends many small messages may need a different improvement from one moving large blocks. A synchronized all-to-all across nodes may stress a different route from an exchange inside one tightly connected system. Describe the job and the observed bottleneck. How many endpoints participate?
Which collective is used? What is the distribution of message sizes? Which path is congested? Can useful computation continue during the transfer? Which failure domain must the system tolerate? Then test effective step time, not just a link specification. For the retailer's forecast training, a trace that shows long all-reduce waits points to a particular investigation. It does not automatically say the fix is one brand of interconnect.
The software implementation, data placement and system topology may be as important as hardware rate. Why is “more network bandwidth” an incomplete requirement? Useful communication depends on message size, latency, topology, contention, collective pattern, software and overlap with computation. Name the workload and measured bottleneck. Peak rate at an unspecified boundary does not establish the training step time. One more check. In the simplified ten-second step, what happens if computation doubles in speed? What happens if communication time is halved, with the other stage unchanged and no overlap?
Faster computation makes four seconds become two; two plus six is eight seconds. Halving communication makes six become three; four plus three is seven seconds. The second change saves more here because communication occupied more of the original measured step. Multi-device work depends on specific communication paths. Bandwidth is a rate, latency is delay, and topology determines how participants connect. Scale-up, scale-out and rack-scale designs place those paths in different system boundaries. Collective operations can create waits that a point-to-point peak figure will not predict. For the retailer, the ten-second trace shows why step-level evidence matters: faster compute alone leaves a six-second communication wait.
The complete system also needs power, cooling and reliable operation. Choose improvements by the useful work they unlock, with costs attached. In Lesson sixteen, we'll ask which of the system's busy time actually counts as useful output and value.