How Much Memory Does An AI Model Need?

The umbrella retailer is considering a large customer assistant. Its saved model weights alone are estimated at one hundred forty gigabytes. A seller offers devices with eighty gigabytes of memory each. Would two devices solve the problem? The arithmetic says one hundred sixty gigabytes in total, but a running service needs more than saved weights, and memory on separate devices does not combine by magic. Lesson seven: Memory capacity, what must fit. By the end, you should be able to name the main kinds of model state that occupy memory, explain why a weight-only estimate can be too small,

and assess an illustrative one-, two-, and four-device capacity comparison. We'll follow three steps. First, inventory what inference and training store. Second, test a complete working set against capacity, including peak use and simultaneous requests. Third, compare ways to place or shrink that state. Our guiding question is: what must be in memory at the same time for this task to run? Memory capacity is the amount of information a particular memory resource can hold. The word particular matters.

A device's eighty gigabytes is its own nominal capacity, not automatically a shared shelf available to every other device. When the assistant performs inference, it applies its saved model to a request. The weights are learned parameter values used in calculations. They are important, but they are only one part of the working set: the information that must be available while the task runs. Activations are intermediate values produced as information passes through the model. A language model may keep an attention cache, often called a KV cache, so selected information from earlier tokens can be reused as it continues a response. The request and result need input and output buffers.

Some operations need workspace, temporary room for their method. The serving software itself also reserves memory and has overhead. Picture the retailer loading a shipment of weights onto a work floor. The staff still need room to open customer requests, hold intermediate work, keep useful earlier context, and pack the reply. Counting the shipment alone does not measure the floor area needed while the service is busy. The analogy is about capacity; actual computer memory has placement and allocation rules we must check. Training has further memory roles. As the learning procedure compares outputs with its objective and updates parameters, it uses gradients.

A gradient provides information for a parameter update. An optimizer is the update procedure; its state keeps information that the chosen procedure needs across updates. Training can also retain activations and other structures needed to compute those updates. The exact amounts depend on the model and training method. We should not invent a fixed multiplier such as “training always takes four times the weights.” Instead, list the required state for the actual procedure and measure its peak. The distinction is useful even if the retailer only plans to serve requests. Weight storage, inference activations, attention cache, gradients and optimizer state have different jobs.

A diagram that treats every byte as “the model” makes it harder to diagnose which change will help. The working set is not fixed by the model name. Batch size changes how many pieces of work are processed together. Context length changes how much input a language model must handle. More simultaneous customers can increase active request state. Architecture, number format and serving software also affect what is kept and how it is placed. The retailer might test one short question and see the assistant fit. Then it opens the service to many customers sending long conversations.

The weights are unchanged, but attention state, buffers and workspace can grow. A weights-only calculation cannot guarantee the busy service will fit. An out-of-memory error means a required allocation could not be satisfied under the system's conditions. That is different from a model running slowly while it fits comfortably. Some memory may be reserved by software, and a temporary peak can be higher than the steady use seen in a quiet moment. Free space can also be fragmented into pieces that do not satisfy a particular allocation. Placement across devices adds another constraint. A nominal capacity figure does not describe all those conditions.

We need the peak working set for the intended batch, context, concurrency and implementation. Then we need to know where each part can actually live. The answer may change when the serving design changes. Consider a hypothetical model with seventy billion parameters, each stored at two bytes. Seventy billion times two is one hundred forty billion bytes. In decimal units, that is one hundred forty gigabytes for weights alone, before activations, caches, buffers, workspace and runtime overhead. One device with eighty gigabytes cannot hold that two-byte weight representation by itself. It is short even before the other state arrives.

Two nominal eighty-gigabyte devices provide one hundred sixty gigabytes in aggregate. Subtract the one hundred forty-gigabyte weight payload and only twenty gigabytes of nominal aggregate capacity remain. But the weights have to be divided and placed with software support. Each device also needs its own usable space for the work assigned to it. We cannot claim the complete service fits merely because one hundred sixty exceeds one hundred forty. Four such devices provide three hundred twenty gigabytes of nominal total capacity. That may create more room for the working set and more ways to distribute it. It also introduces communication among devices, software requirements and cost.

More total memory is not a complete deployment design. The capacity arithmetic answers a narrow question: what is obviously impossible, and which configurations might be worth testing? It does not certify a running service. One strategy is sharding: split selected model or runtime state across devices. That can make a larger working set feasible, but the pieces must communicate when a calculation needs information from another device. Offloading moves selected information to a different level of memory or storage. It may relieve a scarce fast-memory capacity, while adding movement and delay when the information is needed again. Compression changes how information is represented.

The quantization we studied in Lesson five can shrink selected values, but it can add metadata, require a supported kernel, and alter result quality. None of these strategies is a free increase in capacity. Each trades one constraint for possible movement, communication, software work or quality consequences. For the retailer, the right comparison depends on its actual request pattern. A method that allows the model to load but makes each customer wait too long is not automatically useful. A method that saves memory but causes unacceptable answers also fails the service goal. Capacity determines which designs are possible and how many requests can coexist. A memory-rich device might hold more state locally, reduce the number of devices needed, or let more customer sessions run at once.

It may also cost more. Four smaller devices, two larger devices, or a compressed design can have different communication, power, software and quality costs. The retailer should compare them under the same workload and quality target. It needs a configuration that fits at peak and delivers qualifying answers at an acceptable cost. The device with the largest memory number is not always the cheapest useful system. Nor is the configuration with the smallest weight payload always the fastest. First establish feasibility; then measure whole-service results. Why is parameter count times bytes per parameter only a lower-level estimate of memory need?

It estimates the weights under a chosen representation. It leaves out activations, attention caches, input and output buffers, workspace and runtime overhead. Training adds gradients, optimizer state and other structures. Batch size, context length, concurrency and implementation determine the complete working set. One more check. Do two nominal eighty-gigabyte devices guarantee a fit for one hundred forty gigabytes of weights plus the running assistant? No. The total nominal capacity is one hundred sixty gigabytes, but the weights must be placed and shared through supported software. Other state needs space, possibly at a temporary peak, and each device has its own allocations.

The sum makes the design a candidate for measurement, not a guaranteed fit. Memory capacity asks what can be held, not how fast calculations run. The retailer must account for weights and all request-time state, plus different structures if it trains. A successful small test may not predict peak use with longer contexts or more customers. One eighty-gigabyte device cannot hold the one-hundred-forty-gigabyte weight-only example. Two devices offer one hundred sixty nominal gigabytes, and four offer three hundred twenty, but placement, extra state, communication and cost decide whether the full service works. In Lesson eight, we'll separate this fit question from memory bandwidth: how quickly information can move across a specified path.

A model can fit comfortably and still leave processors waiting for data.