Model Parameters And Activations Explained

A colleague sees a calculation change from 7 to 11 and says the model has learned something new. Here are the supplied classroom rules: multiply the input by a weight of 2, then add a bias of 1. The 1st input is 3; the next is 5.

What changed in the computation, and what would count as a change to the model itself? The weight and bias define the transformation. In a trained model, training establishes learned numerical values such as these. We call those values parameters. Here they are supplied so

that we can inspect their roles. With input 3, the calculation is 2 times 3 plus 1, giving 7. With input 5, the same parameters give 11. Changing the input changes the computed result while the weight and bias remain fixed. An activation is a value produced

while the model computes for an input. Larger networks produce many intermediate activations before their final output. The simple rule gives us a small version of that distinction: 2 and 1 define the transformation, while 7 and 11 are results for particular inputs. A checkpoint stores the chosen learned state.

A log can separately record inputs or outputs; keeping such a record does not turn an output into a learned parameter. Training crosses a different boundary. Suppose a separate update process produces a new weight of 3 while the bias stays 1. The same input 5 now produces 16.

The transformation has changed, and a saved new checkpoint can carry that change into later requests. We have supplied the changed weight as a teaching condition; no training experiment has been run. To investigate a real change, identify the trained versions and update process rather than

infer learning from a different answer alone. The embedding row from the previous lesson belongs to the learned state. Other learned matrices and values transform representations as they pass through layers. The actual vectors computed for a particular input depend on those values and that input.

In attention, which we will develop next, learned transformations and request-specific representations will both participate. Keeping their roles clear helps us follow the operation without treating every number on a diagram as the same kind of object. A parameter count reports learned scalar

values under a model’s counting convention. It is separate from the number of training documents, vocabulary entries or facts a person might list. The embedding-table calculation counted 1 matrix; the full model includes other transformations. Capacity and resource needs can change with size, while useful task performance also depends

on data, training, architecture and adaptation. Record the exact model configuration alongside the headline count. Learned associations can contribute to an answer without a normal database row for each claim. Exact source tracing and a targeted update therefore need their own method.

A real dated model card illustrates what those identities look like. Meta’s Llama 3.1 release was dated 23 July 2024 and described pretrained and instruction-tuned models in 8B, 70B and 405B sizes. The card describes a static model trained on an offline dataset.

These are saved trained configurations. An application supplying a new document can change the current request without establishing a new checkpoint. The size labels help identify the model family; they leave the storage format and complete runtime plan to inspect. Now use a defined storage

assumption for a classroom calculation. At 2 bytes per parameter, 8 billion parameters require 16 billion bytes for their values, or 16 GB in decimal units. A 70-billion-parameter collection at the same 2 bytes per value requires 140 GB. We are multiplying supplied rounded counts by a chosen representation size.

These are weight-only estimates, not measurements of a particular downloaded checkpoint or a promise that a named model fits on a device. A running service needs room for more than the weight values. It can need input-dependent activations, attention caches, working buffers, metadata and operating overhead.

A longer request or a different batch can change working-state requirements while the checkpoint stays fixed. Training can also require gradients and optimizer state used in updating the parameters. The exact implementation determines the complete memory requirement. We will calculate a

defined attention-cache example later. This gives the team a practical record to keep. Identify the checkpoint and stored representation, then specify the request length, batch and runtime conditions. Separate the weight calculation from the remaining memory allowance.

If the same checkpoint receives another sentence, inspect the changed input and computed states. If the checkpoint itself changed, inspect the training or adaptation history. Each explanation points to a different intervention and a different test. Pause for a checkpoint. Our rule uses weight

2 and bias 1. Input changes from 3 to 5, and the output changes from 7 to 11. Has that shown a parameter update? Separately, under the supplied 2-byte assumption, what does the 140 GB estimate for 70 billion parameters cover? Finish your explanation before continuing.

The provided weight and bias stayed fixed, so the different outputs demonstrate the rule computing for different inputs. They do not show a parameter update. The 140 GB counts the parameter values under the stated decimal storage assumption. It leaves activations, caches, working space and other overhead outside the estimate.

A complete runtime plan needs those conditions and resources too. For a separate practice task, a checkpoint is described by a rounded count of 10 billion parameters. Use 2 bytes per parameter to estimate weight storage in decimal GB. A colleague proposes a runtime with

exactly that much memory and claims the calculation proves it will fit. Explain what remains to be checked. Pause and use the blank worksheet. The calculation gives 20 billion bytes, or 20 GB, for the specified weight values. The count and format still need confirmation for the actual checkpoint, and the runtime

needs its other working state and overhead. The weight calculation is a component of the plan. It cannot by itself establish that the complete workload will fit or meet its operating target. We can now explain the opening change without assuming new learning: the input changed,

the supplied parameters stayed fixed, and the computation produced another result. We can also separate a saved model’s weight values from request-specific working state and a complete memory requirement. Next, we will use that distinction to follow attention: how 1 sequence position combines information from allowed positions around it.