Labels, Annotation And Ground Truth Explained

2 careful reviewers assign different labels to the same return. One says eligible; the other says ineligible. Before deciding who made a mistake, our fictional company has to ask what each label means. We will follow 1 return through a task definition, a reference check, a disagreement and a recorded decision. The goal is a target that a model can learn and an evaluator can interpret.

A label is a target or annotation attached to an example. It might be a category, an amount, an answer or a preference between answers. Annotation is the process of adding that information. For our returns classifier, the word eligible needs a defined question: eligible under the policy applying to the purchase, eligible under today's policy,

or accepted by a human agent exercising discretion? Those are different targets even when their labels use the same word. Here are the supplied fictional rules. Purchases before June 1 use version A, which allows an unused item to be returned within 30 elapsed days. Purchases from June 1 use version B,

which allows 14 elapsed days. Ignore other exceptions for this small example. Order O901 was purchased on May 20 and its unused item was returned 21 elapsed days later. Under the purchase-date rule it is eligible under A. Applying B instead would produce ineligible. The disagreement can come from choosing different policies rather than reading the dates carelessly.

Ground truth names the reference used to judge correctness. A source transaction, a later observed outcome, an expert decision or a reviewed answer can serve as a reference under a specified process. Each has limits. An agent's discretionary acceptance tells us what the agent did; it does not automatically establish eligibility under the written rule.

Define the task and reference before treating historical behavior as the ideal answer. An annotation guide should make that definition usable. State the policy-selection rule, the elapsed-time convention, required facts, categories, examples and escalation path. If the purchase date or item condition is missing, our guide sends the case for review instead of forcing eligible or ineligible.

Reviewers need access only to information authorized for the task. The guide and its version should remain attached to the resulting labels. Have reviewers apply the guide independently to a trial set. When they disagree, inspect their reasoning and the supplied facts. For O901, an adjudicator can identify the A-versus-B selection difference and apply the agreed purchase-date rule. Record the original annotations,

reason and resolved label. If the guide changes, identify which earlier cases require re-examination. A majority vote without the policy check could simply repeat the same misunderstanding. Northcutt, Athalye and Mueller's 2021 study investigated test-label errors across 10 commonly used datasets. Algorithms flagged possible errors, then people checked those candidates.

A flag was a reason to inspect, rather than an established correction. Their analysis showed that corrected test labels could change model comparisons. A test label is part of the measuring instrument: an incorrect reference can reward the wrong response. In our policy case, keep the original annotation, the independent reference and the

adjudication reason inspectable before changing it. Amazon's Ground Truth documentation describes consolidating annotations from multiple workers and recommends instructions with difficult examples. These are documented workflow mechanisms, not proof that agreement equals correctness. The service documentation also says it is closed to new customers; no service purchase or use is proposed here.

Agreement measures tell us about consistency under a process. Shared incorrect instructions can yield consistent mistakes. Preference data asks another kind of question: which answer is better under stated criteria? Suppose 1 response is polished but applies policy B to O901, while another is awkward but selects A correctly. A fluency-only criterion could prefer the wrong policy answer.

Define accuracy, source use, completeness and style as appropriate to the task, and inspect the evidence. Hugging Face's versioned TRL documentation shows chosen and rejected response fields as 1 preference-data representation; those names do not establish that the chosen response is factually correct. Timing also matters. A return outcome observed 21 days after purchase can

be a target for a prediction made at checkout. It cannot be an input claimed to be available at checkout. Preserve when each feature became available and when the target matured. Clear labels do not repair a test that leaks future information into the inputs. Here is the checkpoint.

Why might 2 carefully annotated datasets still disagree? Think about the question, policy, population, time boundary and review process before judging either collection. They can define different tasks, use different policies or populations, apply different time boundaries, or resolve disagreements differently. Compare those conventions before declaring one universally wrong. In our case, the purchase-date and

current-policy questions can legitimately produce different labels; the task definition determines which is appropriate. For practice, classify a new return under supplied version rules, decide what to do with missing information, and challenge a proposed preference label. Keep original observations separate from decisions and preserve the guide version. Our reviewers' disagreement became useful

when it exposed the target's definition. Next we will place examples into training and evaluation roles, where both meaning and independence must be preserved.