Data For Pretraining, Fine-Tuning And Testing

Our fictional company has 85 approved order examples. It wants to predict, at checkout, whether an order will be returned within 30 days. Which examples may help the model learn, and which must remain independent to judge it? A sensible-looking split is only useful when we know the prediction time, target and deployment claim. Follow the roles before interpreting a score.

Pretraining supports broad initial learning under an objective, such as learning patterns in language. Post-training changes selected behavior later, including instruction following or task performance. Fine-tuning a pretrained model on specialized examples is 1 form of later training. Evaluation instead measures behavior under specified conditions. These are roles in a process.

The same question-and-answer format could appear in a training collection or a test; its file format cannot establish its role. Hugging Face's Transformers documentation illustrates fine-tuning a pretrained classifier with separate training and evaluation data. That is a specific software workflow, not evidence that our 85 examples are enough or that training will improve this business.

Record which examples and transformations influence learning, which help development choices, and which remain reserved for the final claim. Use the explicit hypothetical split from the course project. January through March orders for customer groups C001 through C016 supply 68 training examples. April orders for entirely different groups C017 through C020 supply 17 final-evaluation examples.

No customer group crosses the partition. The intended claim is return risk at checkout for later orders from customers unseen in training. Both time and customer novelty change together; this 1 test cannot isolate their separate effects. The project makes the arithmetic reproducible. 12 training groups have 4 orders each and 4 groups have 5:

48 plus 20 equals 68. 3 evaluation groups have 4 each and 1 has 5: 12 plus 5 equals 17. These counts describe a synthetic partition. They establish no label accuracy, useful performance, sufficient sample size or absence of leakage. The final 17 outcomes must mature over the 30-day horizon before scoring.

Only information available by checkout belongs in inputs for this claim. A return observed later can be a target once its horizon matures. A later delivery status or revised label must not slip backward into checkout features. Scikit-learn's version 1.7 documentation explains leakage as using information unavailable

at prediction time in model building. Its examples motivate auditing preprocessing as well as the obvious input fields. Keep development separate from final testing. Suppose a team inspects the 17 outcomes and changes its labels, features, prompts, thresholds or model choice to improve that result. Those cases have now influenced development.

Reporting them again as an untouched final test would overstate independence. Use a separate development-validation process for iteration, then freeze the candidate and assess the reserved final set. The project supplies no particular development sample count or measured score. Contamination can also enter through near-duplicate answers, related customer episodes, retrieved test answers or

a public benchmark already encountered during training. Separate directory names cannot prevent these information paths. Trace source families, groups, timing and development access. If you cannot establish an independence claim, state that limitation and design a better test rather than presenting the same number with greater confidence. Training mixtures create another choice. Sources, languages, domains and formats

contribute under selected proportions or sampling rules. More material from 1 source can change what the model practices, but it gives no guaranteed proportional gain. Version the dataset, transformations, split rule and label guide together. A model comparison becomes difficult to interpret if its test set changes silently.

Our checkout predictor also remains separate from the assistant's request-time order lookup and policy retrieval. Penedo and colleagues' FineWeb study, first released in June 2024, documented pretraining-data curation and experiments varying filtering and deduplication choices. They released curation code and the models trained in those experiments. The evidence comes from comparing defined

choices on tested models and benchmarks. It does not make 1 filter a universal guarantee. For our smaller task, preserve the data and transformation versions so a comparison can reveal what changed, and keep evaluation independent of the decisions being compared. Here is the checkpoint. Why preserve a final test set while using a development set?

Think about what happens when repeated decisions are made after looking at the same outcomes. Repeated tuning can adapt choices to that set. A separate final set supports a more credible assessment of the frozen candidate under its stated claim. Independence still needs checks for groups, timing, duplicates and information access; the name final does not provide them.

For practice, classify data roles and challenge a proposed checkout feature and a reused test. The worksheet binds the same project split and gives a separate key. Our 68 and 17 remain teaching counts, with no model score supplied. Before claiming performance, name what was learned from, what guided decisions, what was held out,

and which future situation the test represents. Next we will ask what changes when some examples are generated rather than independently observed.