How Quantization Makes AI Models Smaller

An umbrella retailer wants its customer assistant to use less memory. A supplier says the model can be made “four-bit.” That may sound like a complete answer: smaller numbers, fewer devices, lower cost. But which numbers are smaller? What extra information is needed to use them? And will the assistant still answer customers well and on time? Lesson five: Quantization and its actual trade-offs. By the end, you should be able to explain how values are mapped into fewer levels, calculate a theoretical weight payload,

and identify the evidence needed before adopting a compact model. We'll follow three steps. First, identify the values being changed. Second, separate a simple bit-count calculation from the actual memory and software path. Third, compare quality and operating results for the retailer's task. Our guiding question is: what did we compress, and did the complete service improve? Quantization maps values into a smaller set of representable levels. If several original values must share one stored level, some detail is lost.

The numerical values used later can therefore change. Consider a classroom example. Suppose the allowed levels are tenths and a weight is zero point one three seven. A rounding rule might store it as zero point one. The difference is zero point zero three seven. That might be harmless for one calculation or matter after many calculations. We cannot tell from the small difference alone. Real quantization schemes are more involved than rounding every number to the nearest tenth.

They can use scales, zero points, groups of values and other structures. The simple example tells us what kind of change to look for: the stored representation stands in for an original value under a defined mapping. We then test how the changed values affect the actual task. In Lesson four, we compared bit counts and floating-point formats. Quantization is a particular representation intervention. Saying “four bits” names a size per stored code under some scheme. It does not specify the mapping, what part of the model uses it, or whether the hardware can run it efficiently. The retailer's assistant has several kinds of numerical state.

Weights are the model parameters learned or selected before a request. Weight quantization changes how those parameters are stored and used; it does not remove parameters just because each one uses fewer bits. Activations are values produced during a model calculation. Activation quantization changes selected values as the model runs. A system may quantize weights while leaving some activations in a different format, or choose another mixture after testing. For language generation, a KV cache stores selected attention information from earlier tokens so it can be reused while the response continues. Quantizing that cache changes a use-time memory store, not the saved weights. We will learn its growth and limits in Lesson twelve.

For now, the distinction is enough: weights, activations and the KV cache occupy different parts of the workload and can be treated differently. So a description such as “a four-bit model” is incomplete. Are the weights four-bit? Which ones? Are activations or cache values also compact? What stays at higher precision? The answers affect memory, speed and possible quality loss. A useful specification names the component and method.

A compact stored code is not always the full numerical value used in a computation. A scale can relate the code to an effective value. In a simple scheme, the code one with a scale of one tenth can stand for one tenth. A zero point may shift where zero falls in the code range. Other schemes make different choices. A model can also divide values into groups, giving different groups their own scales. That may preserve useful detail better than one scale for everything, but the scales and other metadata need storage. The code payload and the supporting information travel together.

Packing matters too. Two four-bit codes can fit in one byte in an ideal count. Real storage and fast execution may require padding, alignment or a particular layout. A supported kernel, meaning the program that actually runs the operation on the device, may accept some arrangements and reject or slowly handle others. This is why parameter count times bits per value is a useful payload estimate, not a complete deployment footprint. We have not yet counted scales, metadata, temporary values, activations, the KV cache, software buffers or other runtime state. We also have not shown that the desired kernel runs efficiently on the selected hardware. Use a hypothetical large model: seventy billion parameters.

Suppose each saved weight occupies two bytes. Seventy billion times two gives one hundred forty billion bytes. In decimal units, where one gigabyte is one billion bytes, the weight payload is one hundred forty gigabytes before other state. Now suppose each weight could be represented with four bits. Eight bits make a byte, so four bits are half a byte in the ideal packed count. Seventy billion times half a byte gives thirty-five billion bytes, or thirty-five decimal gigabytes. The theoretical weight payload is one quarter of the two-byte version. That is a significant arithmetic difference.

It is still not proof that a thirty-five-gigabyte device can run the complete model. Actual weights may require scales and metadata. Packing and layout can change the footprint. During a request, activations, cache values and other temporary state also occupy memory. Some values may stay in a different format. The service must measure its actual peak and steady memory use. The same caution applies to speed. Moving fewer weight bytes may help when weight traffic is the limiting stage.

The bit-count calculation does not tell us how quickly the chosen device unpacks values, whether an efficient kernel is supported, or how long customers wait for the whole answer. There are different procedures for creating a quantized model. Post-training quantization applies after a training phase. Some methods use representative examples to calibrate how compact levels should be chosen or scaled. The examples should reflect the intended use well enough to reveal important behavior. Quantization-aware training accounts for selected quantization effects while the model is being learned or adjusted. It can have a different preparation cost and may achieve different quality from a conversion done after training.

Neither label guarantees success. The method, model, data, software support and task all matter. For the retailer, testing should include ordinary customer questions and difficult cases where an incorrect answer would be costly. We should compare the compact assistant with the original under the same task definition. A small change in an average score could hide a severe failure in a narrow but important group of requests. The goal is not merely to produce smaller files; it is to deliver qualifying service. Compact weights may reduce memory traffic. They may allow the assistant to use fewer devices, serve more requests at once, or consume less energy.

Supported hardware can sometimes compute efficiently on the compact format. These are possible benefits to measure. They are not automatic. If unpacking, extra metadata or an incompatible kernel adds enough overhead, latency may not fall. If the retailer's customer reply waits mainly on an inventory tool, a network call or a queue, reducing weight bytes will not remove that wait. A smaller model can also lose answer quality, prompting more human review or failures to correct. The economic comparison is cost per qualifying task. Count the assistant responses that actually meet the required quality and service time, then include hardware, energy,

engineering and evaluation costs. If compression changes the deployment design, compare the before and after systems as they really operate. A lower raw memory number is one input to that decision. What evidence would support a decision to quantize this retailer's assistant? We would compare measured task quality and severe failures, actual memory use, latency and throughput, power, device and software compatibility, and total cost under the intended workload. We would ask whether the service still meets its target, not merely whether a file became smaller. Those measurements should be attached to the specific representation and components changed.

One more question. The four-bit weight arithmetic gave thirty-five gigabytes. Does that prove one thirty-five-gigabyte device can run the whole model? No. Thirty-five decimal gigabytes is a theoretical payload for the weights alone under ideal four-bit packing. Scales, metadata, layout and runtime state add requirements. A particular device and software path also need to support the actual scheme. Measure the full deployment footprint before choosing capacity. Quantization maps values into fewer representable levels.

The intervention may target weights, activations or the KV cache, and each target changes a different part of the system. Compact codes can need scales, metadata and supported kernels, so a nominal bit count is not a complete specification. For seventy billion weights, two bytes each give one hundred forty decimal gigabytes of weight payload. Four bits each give a theoretical thirty-five gigabytes. The useful decision also needs actual memory, speed, energy, quality, compatibility and total cost per qualifying task. In Lesson six, we'll examine FLOPS and peak numerical rates. As with a bit count, a large advertised number will need units, assumptions and a measured workload before it can guide a decision.