Why AI Uses Matrix Calculations

An umbrella retailer uses the same accelerator for two jobs. Overnight, it runs a large batch of numerical forecasts. During the day, a live customer question may need a small model calculation, a record from another system, and a reply across the network. The chip specification is the same in both cases. Yet the time and cost to complete each useful job can be very different. Lesson three: Matrix operations connect learning to hardware. By the end, you should be able to make a small matrix product, explain what the common operation count means, and say why that count alone cannot predict elapsed time.

We'll take three steps. First, build four outputs from rows of inputs and columns of weights. Second, count the numerical work using a stated convention. Third, follow the whole task path to see what the count leaves out. The guiding question is: what work does the model request, and what else must happen before a useful result arrives? A weighted sum took input values, multiplied each by a weight, and added the products. For example, three inputs can be combined with three weights to make one output. Matrix multiplication organizes many related combinations of that kind.

A matrix is a rectangular arrangement of numbers in rows and columns. For our retailer, imagine two store records, each with three numerical inputs. We will use deliberately simple whole numbers so you can follow the arithmetic. The first store's row is two, one, three. The second store's row is one, four, two. This input matrix has two rows and three columns, so we call its shape two by three. Now suppose the model has two sets of weights, each meant to form a different intermediate output score. The first weight column is one, two, two.

The second is two, one, three. Placing those columns side by side makes a weight matrix with three rows and two columns: three weights for each of two outputs. These numbers are invented for the exercise. The scores will show the calculation, not a validated demand forecast or an automatic business decision. The inner sizes match: three input values per store and three weights per output. For every store row, we can combine the values with each weight column. Two store rows times two output columns gives four output positions. The product therefore has two rows and two columns.

Neural networks repeatedly use related numerical transformations like this, alongside other kinds of work. The repeated structure is one reason specialized hardware can help. It does not mean every operation in a neural network, much less the entire application, is a matrix multiplication. Let's calculate the top-left output. Use the first store row, two, one, three, and the first weight column, one, two, two. Multiply matching positions: two times one is two; one times two is two; three times two is six. Add those products. Two plus two plus six is ten.

That is the first store's first output score. For the first store's second score, keep its row but use the second weight column, two, one, three. Two times two is four. One times one is one. Three times three is nine. Together they make fourteen. Repeat the same method for the second store's row, one, four, two. With the first weight column, one times one plus four times two plus two times two gives one plus eight plus four, or thirteen.

With the second column, one times two plus four times one plus two times three gives two plus four plus six, or twelve. Our output matrix has two rows: ten and fourteen for the first store, thirteen and twelve for the second. Each output used one input row and one weight column. Four outputs came from twelve matching input-weight pairs. We have not changed the meaning of those outputs by calculating them quickly; they remain invented intermediate scores. A multiply-accumulate takes a pair of values, multiplies them, and adds the product into a running total. In our calculation of ten, we multiplied three pairs and accumulated their contributions. Some hardware can fuse a multiplication and addition into one instruction, called a fused multiply-add.

An instruction is a command the processor executes. Performance reports often use a different counting unit: a FLOP, meaning a floating-point operation. Floating point is one way computers represent numerical values. The representation details come in the next lesson. Under a common multiply-add convention, each pair contributes two counted floating-point operations: one multiplication and one addition. A fused instruction can therefore be one instruction while still counting as two operations under that convention. We had four outputs, with three pairs for each output. Four times three is twelve pairs.

Two counted operations per pair gives twenty-four FLOPs. The common high-level formula says the same thing for larger matrices. If the first matrix is m rows by k columns and the second is k rows by n columns, the approximate count is two times m times n times k. Here, m is two, n is two, and k is three. Two times two times two times three is twenty-four. There is a counting detail worth keeping straight. If you write each output as three products and add only those three products together, you can count three multiplications and two additions per output. Across four outputs, that is twelve multiplications and eight additions, or twenty scalar steps.

The standard twenty-four-FLOP convention instead counts a multiply and an accumulation for each of the three pairs, including adding the first product to a starting total of zero. The answer values are unchanged. The convention differs. Before comparing performance figures, check that the operation counts use the same convention. Our example values are whole numbers for easy listening. The FLOP convention describes a corresponding floating-point computation. It is a workload description, not a stopwatch reading.

Larger matrix dimensions can expose more independent combinations and let hardware reuse values. In our small example, each store row is used with two weight columns, and each weight column is used with two store rows. At larger scale, arranging such reuse can matter because repeatedly moving the same values may be expensive. But a tiny matrix operation can finish so little arithmetic that setting it up takes a noticeable share of the total. Software must launch and coordinate work. If a model consists of many small operations, that overhead can keep it far below an advertised peak matrix rate. Shape is not the only qualifier. The numerical representation and the exact dimensions can determine whether the device has a fast supported path.

A framework, meaning software that selects and runs model operations, may choose a fast kernel for one shape or data type and a slower kernel for another. A kernel is a program launched on the accelerator. Both versions may produce the correct output, while their performance differs. This creates a portability problem. Moving a model to another device can require software and data-layout work to preserve efficiency, even when the model runs correctly there. Test the actual shapes, formats and software path. A vendor's peak rate does not certify the retailer's particular matrix operation.

Return to the retailer's two jobs. The overnight forecast uses large matrix operations on a batch of store data. If that data can be reused and the software selects efficient operations, the accelerator may spend much of its time on suitable numerical work. The live customer question can have a very different path. The service may fetch a purchase record, wait for a network response, perform a small model operation, and send an answer. Retrieval and network time are part of what the customer waits for. The small calculation may also pay launch and coordination overhead. Both applications use the same accelerator, but the accelerator's numerical count describes only part of the live task.

We also have to be careful with cost per task. A large batch may use the device efficiently but still consume time and energy. A live request may use little arithmetic but occupy a service that must meet a response deadline. The useful comparison needs the full task, the required quality, elapsed time and resources paid for. It cannot be settled by operation count alone. If two versions of the retailer's model have the same approximate operation count, could they still take different time? Yes. One may run many operations at once while another has dependencies.

One may find its values nearby while another must move them. One may use a supported fast kernel, while another falls back to a slower path. Starting and coordinating small jobs adds overhead. Some stages must synchronize: wait for other stages to finish before continuing. Actual clock behavior can also vary. These are separate effects on elapsed time. The operation count tells us how much of a specified kind of arithmetic the workload describes under our chosen convention. It does not say how efficiently a device executes it or how much waiting surrounds it.

For the retailer, we would record matrix shapes and number formats, identify the software kernel, measure data movement and launches, and time the whole service. That turns an impressive hardware claim into a testable diagnosis. It also keeps the nightly batch and the customer-facing request from being mistaken for the same workload. First, a shape check. What size is the output when a two-by-three matrix multiplies a three-by-two matrix? How many output values does that give? It is a two-by-two output matrix, with four values.

Each of two input rows combines with each of two weight columns. Each output draws on three matching pairs. Why doesn't the common count of twenty-four floating-point operations tell us how long the calculation or complete task will take? Because the count is only one description of work. Parallel execution, data movement, the selected supported kernel, launch and coordination overhead, synchronization, and actual clock behavior all affect elapsed time. The larger application can also wait on retrieval or a network call. We must measure the relevant path, not turn a count into an invented duration.

We built four outputs by combining each store row with each weight column. Three matching pairs per output gave twelve pairs, or twenty-four counted floating-point operations under the common multiply-add convention. We also saw why a different literal scalar-add count can be correct under its own stated convention. The operation count describes the model's numerical workload. Whether it runs quickly depends on shape, reuse, data movement, software support and overhead; whether the customer gets a timely useful answer depends on the rest of the service too. In Lesson four, we will look at how numbers are represented in bits and bytes. That representation changes how much data must fit and move, and it can affect which hardware path is available.