Does the test represent the claim?
The shop tests a model on delivery messages and reports excellent performance. Later it learns that messages from the same customers appeared in both training and testing. Does the result show that the system will work for entirely new customers?
This is Module one, Lesson seven: Training, validation, and test sets. By the end, you should be able to name the different jobs of the three sets, choose a split that matches a deployment question, spot transformation leakage, and explain why repeated test-driven tuning weakens an independent claim.
We'll begin with the roles of the data, then choose boundaries around time and identity, and finally ask how fresh the test evidence really is.
Give each set a distinct job
The training set supplies examples that drive parameter updates. The model fits to these examples under its objective. The validation set helps the team compare model designs, adjust settings and make development choices. The test set is held for an independent assessment after those choices, provided it has not repeatedly shaped them.
Track practical roles, not file names
These are roles, not magical labels attached to three files. If an engineer studies test errors and changes the model to fix them, the test examples have joined the development process in practice. The file may still be named test, but its evidence is no longer fresh in the same way.
We separate roles because a model can improve on data it has encountered during development. An independent set helps estimate how it may behave on relevant new cases. The set must represent the claim we want to make.
Choose relevant boundaries
A random split assigns examples to sets through a random procedure. It can be useful when the examples are suitably independent and drawn from conditions like those future use will face. But delivery records have structure: the same customer may order often, routes repeat, and conditions change with time and place.
Suppose messages from a customer have a recognizable writing style. If that customer appears in both training and test sets, the model might use the style as a clue. That could be relevant for predicting future messages from the same known customers. It does not establish performance for entirely new customers.
An entity split can separate customers, devices or other units across sets. A temporal split can train on earlier periods and evaluate later ones. A geographic split can test transfer to another location. Choose the boundary that matches the real question. If the shop will serve new customers in a new region next month, both identity and time may need attention.
Respect the prediction-time boundary
Forecasting makes the time boundary especially clear. A model asked to predict next week's demand must not learn from observations recorded the following week. Randomly mixing weeks into training and testing can place future information on the wrong side of the evaluation.
Prevent transformation leakage
Leakage can also enter through data preparation. Suppose the team calculates an average and spread for normalization using every row before splitting. The calculation has seen information from future test examples. Even if the model never directly reads their targets, the evaluation pipeline has gained information unavailable in the intended deployment sequence.
Fit a transformation using the appropriate development data, then apply that saved transformation to validation or test examples. Preserve the pipeline and its timing. The exact rule depends on the evaluation design, but the principle is consistent: do not let information from the future or from a supposedly independent set shape the model before the assessment.
Recognize test-driven development
Imagine that the first test run reveals failures on ten messages. The team reads those examples, adds special cases, retests, and repeats until the score rises. The team has learned from the test. That set is now useful as a regression suite for known failures, but it is weaker evidence about broader unseen messages.
Account for benchmark familiarity
Benchmark examples can also find their way into training data or public development discussions. A model may perform well on a familiar benchmark without transferring to the intended workplace. The remedy is not to pretend the old test is untouched. Label its role honestly and obtain fresh evidence for the broader claim.
Keep a development ledger
Keep a record of development choices and what information influenced them. That record helps reviewers interpret a score. It also makes a later assessment more meaningful because the team can specify which decisions were complete before the new evidence was observed.
Interpret an observed percentage
Suppose a model answers nineteen of twenty independent test cases correctly. The observed proportion is nineteen divided by twenty, or ninety-five percent. That is a description of those twenty cases. It does not establish exactly ninety-five percent performance for all future use.
Assess coverage and consequences
The estimate can vary with a small sample. The twenty cases may omit a rare but serious failure, come from one location or repeat the same type of question. Dependence between cases can further reduce what the count tells us. Coverage, sample design and consequences determine how much testing is enough.
For a low-stakes workflow, an early small test may guide exploration. For an action with substantial consequences, a broader and better designed assessment may be needed before deployment. The confidence of the claim should match the evidence, not the neatness of the percentage.
Check your understanding
The shop wants a model for entirely new customers. Why might a random split of messages overstate transfer if the same customer appears on both sides?
The model could learn customer-specific style or habits that reappear in the test set. That would be a different task from handling a new customer. An entity split can test that stated question more directly.
Why is the test set not simply a third place to tune the model?
Because tuning based on its results uses information from it to make development choices. It loses some independence as final evidence. We can keep it as a known-case regression suite and use fresh, relevant data for a new independent assessment.
Connect the evaluation design
Training examples update parameters; validation examples guide development; a genuinely held-out test estimates behavior after those choices. The split must match time, people, devices or places involved in real use. Transformations and repeated inspection can leak information even when the file names look correct.
Continue to classification outcomes
Now that we can design a more honest evaluation, Lesson eight will examine a common classification task. We'll count the true and false outcomes of a fraud alert system so a single accuracy score cannot hide the mistakes that matter.