What did the training score show?
The delivery shop's model predicts the costs in its training table almost perfectly. Then the shop opens a new service area, and the estimates are poor. What did the strong training score actually show?
This is Module one, Lesson six: Generalization is the goal beyond the examples. By the end, you should be able to distinguish generalization, overfitting and underfitting, compare a model with a simple baseline, and explain why a falling training loss can be misleading.
We'll compare the model's behavior on familiar and new deliveries, inspect the signals it learned, then decide what evidence and remedies fit the diagnosis.
Seek useful performance on relevant new cases
Generalization means useful performance on new examples under relevant conditions. The word relevant matters. If the shop will quote next month's city deliveries, we want evidence on cases resembling those future jobs, including conditions that matter to the intended use.
Training performance tells us how the rule behaves on examples that helped shape its parameters. A model can memorize those examples or learn accidental patterns and still look excellent there. A new route, season or customer can expose the weakness. We need separate evidence from situations the model did not use to fit itself.
This is a property of the whole prediction setup, not a magic trait we infer from model size. Inputs, targets, population and decision time all affect whether a learned relationship transfers.
Inspect an unstable shortcut
Overfitting occurs when a model adapts too closely to its particular training examples, including noise and coincidences, and performs poorly on relevant new examples. Imagine that customer names happen to line up with distance in the original records. The model learns that a certain name predicts an expensive route. In the new service area, names and routes are associated differently. The shortcut fails.
The model may have optimized its loss correctly. The problem is that the signal it used was not stable for the job. Merely asking for a lower training loss could make it rely on the coincidence even more.
Distinguish underfitting
Underfitting is different. The model fails to capture a useful relationship even in the data available. A distance-only straight line may be too simple if costs differ by vehicle or rise nonlinearly with congestion. Both training and independent evaluation may show sizable errors.
Adding complexity can sometimes help underfitting. The same added capacity can make overfitting easier if the data are limited or noisy. Diagnose first, then choose a remedy.
Identify noise in the target
Noise is variation or error that does not represent a stable, useful relationship for the task. A fuel receipt entered incorrectly, two depots defining cost differently, or an unpredictable delay can create variation in the observed target. A highly flexible model may fit some of it, reducing training loss without improving new quotes.
Define capacity without treating it as trust
Capacity is the range of patterns a model structure can represent. More capacity can capture a real relationship missed by a simple rule. It can also represent misleading quirks. Neither a small nor a large model earns trust by its complexity. We need to observe what it learns and how it performs under the conditions that matter.
Check whether a signal can transfer
The name shortcut suggests a practical question: would the apparent relationship still make sense when the shop changes geography, marketing or customer mix? If the answer is doubtful, inspect performance when those conditions change, and reconsider the feature.
Shape learning with regularization
Regularization is a family name for methods that constrain or shape learning to discourage unwanted fitting behavior. In one formulation, a penalty can favor smaller or simpler parameter values. That does not automatically make a model good, but it changes what solutions the learning procedure prefers.
Use distinct interventions carefully
Early stopping ends training when performance on independent validation data stops improving, even if training loss continues falling. Data augmentation creates meaningful variations of training examples. For images, a carefully chosen change of lighting may represent a real condition; for the delivery shop, invented weather variations would need valid cost outcomes, not just labels guessed by wishful thinking.
Choose remedies from the diagnosis
These interventions solve different problems. If the target itself is inconsistently defined, a penalty on parameters cannot repair the label process. If the new service area differs sharply from training, useful new examples or a changed task definition may matter more. A remedy should follow the evidence.
Establish a simple baseline
A baseline is a simple alternative that a proposed model should beat under the same evaluation conditions. The shop could predict average delivery cost or use its simple distance rule. For demand forecasting, repeating last week's quantity might be a baseline. For classification, predicting the common category can expose a misleadingly high accuracy score.
Compare matched demand cases
Consider a demand model with average error of five units on a held-out sample. A simple seasonal rule has average error of four units on that same sample and measure. The complex model has not demonstrated an advantage on that metric. It may have another useful property, such as better behavior on high-value items, but that property needs its own evidence.
Match the evaluation conditions
Comparison should use the same population, time period and error definition. Otherwise, a model can appear to win because the test was easier. Sophistication, higher development cost and more parameters are not substitutes for a matched comparison.
Check your understanding
Could training loss fall while validation loss rises?
Yes. The model may be fitting its training examples more closely while performing worse on separate relevant examples. That pattern can indicate overfitting. It is also worth checking whether the data populations, measurements or pipeline differ before deciding on a remedy.
Suppose a complex model scores five units of error and a seasonal baseline scores four on the same held-out demand cases. Which has demonstrated better performance on that measure?
The seasonal baseline. A complex model could still have an advantage on another outcome, but it must be shown rather than assumed.
Connect fit, generalization and evidence
Training fit is an intermediate result. Generalization concerns relevant new examples. Overfitting captures quirks of the training cases; underfitting misses useful relationships even there. Noise, capacity and data coverage affect the diagnosis, and a simple baseline keeps comparison honest.
Continue to precise evidence roles
The organization ultimately needs useful decisions, not just a low training loss. Lesson seven will make the evidence roles precise: which examples drive learning, which guide development, and which remain fresh enough to support an independent test.