How GPUs Run Many Calculations At Once

An umbrella retailer serves many shops. At the end of a busy day, it has a long list of umbrellas sold at each shop and another list of umbrellas already reserved. It wants to combine matching values before planning stock. Many pairs can be handled independently. But another task follows a chain of audit records, where each record tells the system where to look next. Giving both tasks a chip with many calculation units will not help them in the same way. Lesson two: GPUs and parallel work. By the end, you should be able to spot work that can run concurrently, recognize a serial dependency,

and explain why a GPU's impressive peak rate or core count may not predict the retailer's useful result. We'll take three steps. First, look at the shape of the work. Second, see what a GPU is organized to do and what can hold it back. Third, include data movement, software and cost when judging the complete service. Our guiding question is: can this work be divided into many pieces that have the information they need? GPU stands for graphics processing unit. A digital picture is made from many small picture elements called pixels.

Drawing or changing an image often calls for related calculations across many pixels. That need encouraged designs that can carry out large amounts of similar work concurrently. Parallelism means multiple pieces of work can make progress at the same time. It is useful when a job can be divided and the pieces can be supplied with the data they need. The number of available calculation units is only part of that condition. Return to the retailer's two lists. For each shop, take the number sold and the number reserved, then add those matching values. Shop one might have sold twenty and reserved five, giving twenty-five.

Shop two might have sold twelve and reserved three, giving fifteen. The calculation for shop two does not need the result for shop one. If there are many shops and the values are available, the pairs offer many independent pieces of work. This addition is a simple teaching example, not the whole of an AI model. A neural network performs many related numerical transformations. Some of that structured mathematics can also be divided across many calculation units. That is why GPUs became useful beyond graphics. The exact matrix arithmetic comes in Lesson three.

Now consider the retailer's audit search. The first record contains the location of the next record. The system must read the first before it knows where the second is. The second gives the location of the third. That location is an address: a way of identifying where information is stored. This is a serial dependency. The next step needs the exact result of the previous one. Extra calculation units may be available, but they cannot all follow this particular chain before the next address is known.

A better design may sometimes reorganize the work, but merely adding more workers does not remove the dependency we have described. The two-list calculation and the linked search can live in the same application. Calling the whole retailer program “parallel” hides the difference. Ask which portion has independent pieces, how many there are, and whether the information for those pieces is ready. Then ask what must wait. Those questions are more useful than looking at a single device number. A GPU organizes its execution resources and memory for high throughput on suitable workloads. Throughput means how much work is completed in a period of time.

A GPU program uses threads, which are units of programmed work. Groups of threads execute under the device's scheduling and hardware rules. You do not need to learn those rules in detail here. The practical point is that many available units need a regular enough stream of useful operations and data. Picture the retailer processing many store records. The device can be given groups of related numerical tasks. If every group can read its values and perform similar calculations, a large amount of work can proceed. Our picture of many workers is only an analogy for that organization; a real GPU has specific scheduling and memory behavior.

Now change the workload. Some shops need an extra exception check, while others follow the ordinary path. If threads scheduled to execute together take different branches, some execution lanes may sit idle while the paths are handled. This is called branch divergence. Some store records may be scattered across memory, making data slower to supply. If an operation is tiny, the effort of arranging and launching it may be large compared with the work itself. The GPU can have abundant arithmetic capacity while parts of the program wait or take inefficient paths. This does not mean every branch is disastrous or every scattered read makes a GPU unsuitable.

It means the actual program and its data layout determine how much of the available device can be used. The shop needs measurements of its own workload. In Lesson one, a CPU core was a processing unit capable of running a sequence of instructions. A CPU is designed for broad programs and complex control. A GPU has a different organization for running many suitable pieces of work. The word core appears in descriptions of both, but it does not identify identical units. Scheduling, instruction execution and memory arrangements differ across CPUs, GPUs and specialized accelerators. So a claim that one device has more cores than another is usually a poor comparison by itself.

We would not decide how quickly the retailer finishes forecasts by counting workers without asking what each worker can do, how the work is divided, and whether materials reach them. A fairer test uses comparable conditions: the same task definition, input size, required accuracy, and service target. Measure completed useful work and the stages that take time. The CPU can still handle request logic and preparation while the GPU works on suitable numerical portions. Some GPUs also have specialized engines for selected numerical patterns. Tensor Cores are NVIDIA's name for engines that accelerate supported matrix operations. A matrix is an organized table of numbers; a tensor is a broader name for an arranged collection of numbers.

The engines have requirements about the pattern, representation and dimensions of the work. Not every calculation takes that fast path. Suppose part of the retailer's model uses a supported matrix operation. That part may run very quickly. But the application still has to prepare its numerical data, move that data to the right place, launch a GPU program, and collect the result. A small program launched on the GPU is often called a kernel. Other operations may not use Tensor Cores at all. The software running the model also needs support for the suitable operation; otherwise it may use a slower path.

A peak Tensor Core rate describes a qualified hardware capability under specified conditions. It is not the speed of the whole forecasting service. If the retailer spends time reshaping store records, moving values, launching many small kernels, or waiting for other steps, the measured application rate can be far below the peak. This is the same whole-task lesson we saw with the CPU, now applied to a highly parallel device. Lesson three will show why matrices supply a useful shape for this hardware. Later lessons will make us check number representation, memory movement and what a quoted operation rate actually counts. For now, keep the boundary clear: supported fast arithmetic is one stage of a complete request.

What does the retailer buy when it adds parallel hardware? It needs forecasts that arrive in time and meet its quality requirement. It also pays for equipment, electricity and the software that keeps the system working. A device with unused arithmetic capacity can be expensive even when its brochure number is impressive. That is an economic connection, not an argument that a GPU is always a bad purchase. If the retailer has enough suitable work and meets its service target at an acceptable total cost, the parallel device can be valuable. If the work is tiny, irregular or dominated by waiting, buying more peak capacity may pay for idle potential. Software can change the payoff without adding a chip.

It may arrange the store values so that the needed data is supplied more regularly. It may combine small operations to reduce repeated launch or movement overhead when the calculation permits. These changes still require measurement; an arrangement that helps one workload may not help another. The practical comparison is completed, qualified forecasts per unit of time, at the required quality and acceptable cost and energy. Those are outcomes of the complete service. Peak arithmetic, raw core count and the mere presence of Tensor Cores are clues to investigate, not a verdict. Why can a very parallel chip perform poorly on a particular program? Think of the retailer's work before hearing the answer.

The program may contain serial dependencies, irregular memory access, or branches that send threads scheduled together down different paths. Small operations and launch overhead, software support, and data movement can also limit useful throughput. Our linked audit search waits for the previous address. Scattered store records may arrive late. An unsupported operation will not gain the advertised specialized rate. We would profile the actual application and its waiting paths, rather than compare raw core counts. One more check. Which task offers many independent pieces: adding corresponding values in the two long lists,

or following the audit chain where each record reveals the next address? The list additions can proceed independently once the pairs are available. The audit chain has to reveal each next location in sequence. More simultaneous workers help the first task more directly. They cannot guess the next link in the chain. The useful question was whether work can be divided into many supplied pieces. GPUs grew from workloads with many related graphics calculations and can also help with structured neural-network mathematics. Their threads and specialized engines offer capacity, but dependencies, branches, data access,

unsupported operations and overhead determine how much a program can use. For this retailer, the worthwhile result is timely, accurate forecasts at an acceptable cost and energy use. Measure that result and the stages behind it. In Lesson three, we'll take a small matrix operation apart so you can see exactly what numerical work suitable hardware may accelerate, and why an operation count is not the same as elapsed time.