What actually changes?
The shop has a cost example and a way to measure error. Now it wants the model to learn from that example. What exactly changes between a prediction that is too low and a better one?
This is Module one, Lesson four: A parameter update you can calculate. By the end, you should be able to follow one complete training step, explain what a negative gradient means in this example, calculate the new parameter and loss, and state why a successful single step does not prove a reliable model.
We'll take a deliberately small route: predict, measure the loss, use its local direction to update one parameter, and predict again. Every number will stay visible so the mechanism is easy to inspect.
Calculate the initial prediction and loss
Use a simpler model than the one in Lesson one. Prediction equals w times x, with no fixed starting charge. The letter x names the input, and w is the parameter the learning procedure may change.
We have one training observation. Its input x is two, and its target y is six. Think of those as small teaching numbers, rather than a claim about the shop's real delivery prices. Start with w equal to one. The prediction is one times two, which equals two.
The prediction is below the target of six. With our convention, prediction minus target, the signed error is two minus six, or negative four. We use squared-error loss: negative four multiplied by negative four equals sixteen.
These numbers tell us the current rule's penalty for this one observation. The training question is how to change w so the next prediction moves closer to six.
Inspect a local change
Because x is positive two, increasing w increases this model's prediction. If w were two, prediction would be four. If w were three, prediction would be six. We can see that moving upward might help here. A general training procedure needs a way to find a useful direction even when the model has many parameters and the answer is not obvious by inspection.
Calculate the derivative
A derivative describes how one quantity changes as another quantity changes near a particular value. Here, we ask how the squared loss changes when w changes a little. The relevant formula is two times the prediction error times x. You don't need to derive it to follow the teaching example; we'll substitute the numbers and interpret the result.
The prediction error is negative four. Multiply two times negative four times the input two. The result is negative sixteen. This is the derivative of the loss with respect to w at our current value of w. With one parameter, the gradient is just that one derivative. With many parameters, a gradient collects corresponding change information for them.
What does negative sixteen mean? Near w equal to one, a small increase in w is expected to lower this loss. It is local information about this specific model and loss. It does not mean negative numbers are always good or that every parameter in every model should rise.
Scale the gradient-guided step
Gradient descent changes a parameter in the direction opposite the gradient. We need a learning rate to set the size of that change relative to the calculated gradient. Let the learning rate be zero point one.
The update rule says: new w equals old w minus learning rate times gradient. Substitute our numbers. New w equals one minus zero point one times negative sixteen. Zero point one times negative sixteen is negative one point six. Subtracting negative one point six is adding one point six. New w is two point six.
Keep the input and target fixed
Notice the sequence. We did not change the input or the target. We used them to calculate an error, then a gradient. The update changed w, the parameter inside the rule. Training is the stage in which this change happens. A later prediction with the saved w uses the parameter rather than automatically updating it.
Recalculate prediction and loss
Use the same input x equal to two with the new w equal to two point six. Prediction is two point six times two, or five point two. The target is six, so signed error is five point two minus six, or negative zero point eight.
Square negative zero point eight. The new loss is zero point six four. For this one observation, the update lowered squared loss from sixteen to zero point six four. We can observe the improvement directly; we did not merely assume that following a gradient would help.
Limit the conclusion to the evidence
The prediction has moved toward the target but has not reached it. A further step might help, and repeated learning steps are common. Yet if our only observation is x equal to two and y equal to six, we have learned very little about other inputs. A model could fit one row beautifully and be poor for other distances. The later generalization lesson addresses that gap.
Compare step sizes
The learning rate is not a promise of progress. Imagine taking a tiny step along a route. It may require many steps to reach a useful place. Take a huge step and you can pass the place you wanted, perhaps repeatedly bouncing from one side to the other.
For this simple example, if the learning rate were one instead of zero point one, the update would be one minus one times negative sixteen, or seventeen. Prediction would become thirty-four for input two, far above the target six. The squared error would now be twenty-eight squared, or seven hundred eighty-four. The same direction with an excessive step made this observation much worse.
This extreme comparison makes the mechanism visible; it is not a recommended tuning value. A suitable scale depends on how data are represented, which loss is used, the model, the optimizer and the training conditions. Teams monitor actual behavior rather than assuming that one learning rate works everywhere.
Extend the work to many parameters
Our model had only w. A larger model has many parameters. Its output may pass through several interacting computations, so changing one value can affect the loss indirectly. Training calculates relevant gradients and uses an optimizer to choose parameter updates.
Account for optimizer state
Different optimizers follow different update rules. Some keep additional state from earlier steps, such as running summaries, to guide later changes. Those extra values consume memory and computation during training. A deployed model making predictions may have different resource needs from the same model while it is being trained.
Connect the training loop
The basic loop still has recognizable parts: examples supply inputs and targets; the model predicts; an objective produces loss; gradient information guides an optimizer; parameters change; the process is repeated. We have calculated one visible instance of that loop rather than treating learning as a mysterious event.
Check your understanding
In our worked example the gradient was negative sixteen. Why did gradient descent increase w instead of decrease it?
Gradient descent subtracts the gradient. Subtracting a negative value adds its magnitude. Here, the negative derivative said that a small increase in w should lower the chosen loss locally, so the update moved w from one to two point six.
The loss fell from sixteen to zero point six four on this observation. What can we conclude about future deliveries?
Only that this particular update improved this observation under this loss. We still need data from other situations and an evaluation that tests the intended use. A good single-row training result does not establish the right learning rate generally, strong generalization, or a useful business decision.
Review the complete update
We started with a prediction of two for a target of six and squared loss of sixteen. The derivative was negative sixteen. Using a learning rate of zero point one, gradient descent changed w from one to two point six. The new prediction was five point two and the new loss was zero point six four.
The mechanism connects data and objective to a parameter change. Its local success remains limited evidence. On the ecosystem map, repeated updates create compute and memory work; people still choose the task, data, objective and evaluation standard.
Continue to batches and epochs
Lesson five extends this one-row calculation to batches of examples and repeated passes through a dataset. We'll separate a single update, a batch, and an epoch so those words describe actual work rather than a vague idea of more training.