Data: From Business Records To Model Inputs · Lesson 6

How To Trace Data Through A Pipeline

Trace an output through its actual source versions and transformations. Observations can locate a failed boundary; further evidence is needed to establish its cause and verify recovery.

Study And Save Progress

Watch The Lesson

6 min 10 sec

Read the video transcript

A customer asks whether order O101 can be returned. In this fictional case, the policy says to use the version applicable on the purchase date: V1 covers purchases through May 31, 2026, and V2 covers purchases from June 1. O101 was purchased on June 10. The assistant answers on June 20 using a paragraph from V1.

V2 applies to this order under the supplied rule. Where did the path break? A data pipeline is a sequence or network of steps that moves information from a source toward a use. It might collect order events, check fields and identifiers, change formats, combine records, store results, and make selected information available to an application.

Each step should have a clear job and an owner. Databricks describes 1 organizing pattern called medallion architecture. Bronze retains raw ingested data, silver contains validated or refined information, and gold supports business-oriented outputs. These names describe the vendor’s pattern. A useful architecture can have different stages. Choose responsibilities that serve the task rather

than adding layers for their own sake. For the orders assistant, an intake stage can retain authorized source events and policy versions. A refinement stage checks order identifiers, resolves the meaning of timestamps, normalizes currency while retaining the original amount, and quarantines records with unresolved links. A use-oriented stage can provide authorized order

summaries and policy passages prepared for retrieval. Each stage needs clear permissions and a record of what happened. Lineage records how sources, transformations, and outputs relate. Follow request Q17 backward. Its trace identifies retrieved chunk C1 from policy V1 and the index entry that supplied it.

The index available to that request still contains V1 for the applicable passage. Upstream evidence shows that the authorized source V2 is present and that extraction X2 already contains the permitted V2 passage, including the relevant exception. The source version and its extraction have reached this point; the request-facing index has not advanced to them.

That evidence locates the stale boundary between the available V2 extraction and the index used by Q17. It does not yet explain whether an indexing job failed, a publication step was missed, or the request reached an old index. The operator needs the relevant job and routing evidence to distinguish those causes.

We have 1 trace of a failed update path, rather than a reason to blame every upstream stage. A platform can capture some technical lineage for supported work. Databricks’ Unity Catalog is an example, with documented requirements and limitations. External assets and relationships can also be registered under its supported mechanisms. Business applicability and permission still need their own evidence.

The platform’s lineage capability does not establish that our fictional request trace has been captured automatically or that the source was authorized for every use. Netflix's March 25, 2019 engineering account describes collecting lineage from system logs, query plans and workflow metadata, then combining it with table and job-owner information. Its portal used those relationships to

expose upstream dependencies and downstream writers. The authors also describe coverage gaps and changing engine formats. That operating example shows why a graph needs maintained evidence. In our policy trace, an unrecorded publication or routing step remains a question for the operator; an attractive graph cannot supply a missing relationship. The responsible operator can investigate that boundary and propose refreshing or publishing

the intended index from extraction X2. A rerun test should inspect the index version and the exact passage returned for O101, including its relevant exception. It should trace the new request back to V2 and check the existing access rules. Those are expected tests; no running service has passed them in this lesson.

Keep a historical control as well. An order purchased on May 20 belongs under V1 according to the same fictional applicability rule. Its date-qualified lookup should still retrieve V1. A question explicitly asking about the current policy can follow a different selection rule. Retain effective-date metadata and test both paths. Selecting the newest version everywhere

could replace 1 mistake with another. Validation gates decide what to accept, quarantine, or retain with a warning. Quietly deleting every difficult record can hide a broken source and distort the population. Monitor counts, error reasons, retries, freshness and responsible owners. An unresolved customer link should remain available for permitted

investigation while the affected downstream use is held. Here is the checkpoint. After a wrong answer, what should lineage help the team find? Pause and decide. It should show which source versions and transformations contributed to the answer, so the team can locate where to correct or reproduce the path.

In Q17, the version evidence narrows the failure to the boundary between the available V2 extraction and the V1 index used by the request. The technical cause and successful recovery remain to be verified. Try the transfer task: trace 1 report value from its source through a calculation to the output, and identify a business decision or external step

that the technical platform might not capture. The worksheet leaves space for your reasoning, with a separate answer key. Our policy case shows why the pipeline is part of product reliability. Find the evidenced broken boundary, investigate its cause, and test the affected result before resuming the use. Next we will examine why data

quality depends on the particular task.

Main Point

Trace an output through its actual source versions and transformations. Observations can locate a failed boundary; further evidence is needed to establish its cause and verify recovery.

Key Points

Pipeline stages need a clear purpose, owner, permissions and evidence of what happened.
Databricks' bronze/silver/gold pattern is one vendor organizing example.
Lineage connects sources, transformations and outputs, with platform-specific capture limits.
Applicable versions depend on the stated question and rule. A current-policy lookup and a purchase-date lookup can need different versions.
Diagnose from observed trace evidence, then specify a rerun test; a proposed repair is not an executed success.

1. The question determines the policy

All identifiers, dates, policies and trace observations below are hypothetical teaching inputs. The supplied rule selects policy by purchase date: V1 applies through 31 May 2026; V2 applies from 1 June 2026. O101 was purchased on 10 June. Its 20 June lookup therefore requires V2. This conclusion follows from the rule and date, rather than the mere fact that V2 is newer.

A 20 May purchase is a historical control requiring V1 under the same rule. A question explicitly about today's policy has a different time target. Preserve the policy identity, applicability window and question context rather than assuming newest always wins.

2. Give each stage a responsibility

A data pipeline moves information through stages toward a use. For this service, authorized intake retains relevant source events and policy versions. Refinement checks identifiers, timestamp meaning and permitted transformations, preserving original values where appropriate. A use-oriented stage supplies authorized summaries and retrieval passages.

Databricks' medallion guidance describes raw bronze, validated/refined silver and business-oriented gold layers. Its GCP documentation, updated 11 September 2026, calls this a recommended pattern rather than a requirement. These are vendor design terms, not evidence that every architecture needs exactly three layers or achieves a particular quality outcome. [DB-MEDALLION]

3. Follow one observed failed boundary

The following are the complete supplied observations for this exercise:

Q17's response trace identifies chunk C1, policy V1.
The request-facing index used by Q17 contains V1 for this passage.
The authorized source V2 is present with the given effective-date metadata.
Permitted extraction X2 contains the V2 passage and the relevant exception.

The stale boundary is between available extraction X2/V2 and the index state used by Q17. The extraction is already present, so “extract the source” does not address the demonstrated discrepancy. The observations do not distinguish a failed indexing job, missed publication, or routing to an old index. Obtain the appropriate job, publication and routing evidence before naming the cause.

The source/extraction permission is a given in this exercise; technical lineage alone does not establish it. Trace identity also does not prove that the extracted passage is substantively correct beyond the supplied evidence.

4. Define recovery evidence

The responsible operator investigates the boundary and may propose rebuilding or publishing the intended index from X2. A new request, Q18, is a planned test, with its result still blank. Inspect its target index version, returned policy/chunk and relevant exception; confirm applicability to O101 and the existing access rules. It should trace to V2 if this proposed recovery works.

Also test the 20 May purchase control: it should retain the V1 path under the supplied rule. Do not globally remove every historical version. Record failures and uncertainty rather than marking the pipeline recovered before the tests run.

5. What a platform can capture

Databricks Unity Catalog's AWS documentation, updated 26 October 2026, describes automatic lineage for supported queries and assets, requirements, limitations and external metadata relationships. That platform behavior does not prove complete capture of this fictional request-to-chunk chain. Business applicability, unsupported external steps and rights require appropriate additional evidence. [DB-LINEAGE]

Validation and quarantine should preserve a permitted investigation path, counts and failure reasons. Silently discarding difficult records can hide a failed source bridge. An affected use can remain held while its owner investigates. More stages or a larger model do not themselves fix the observed stale boundary.

6. Check and transfer

Learning Check Q01 applies lineage reasoning to the full supplied trace. Q02–Q04 test update capture, quarantine and the vendor-pattern limit. Practice P01 transfers source → transformation → output reasoning to a report value. Use the blank worksheet before the separate key.

7. A dated operating example

Netflix's March 25, 2019 engineering account describes collecting lineage from system logs, query plans and workflow metadata, then combining it with table and job-owner information. Its portal used those relationships to expose upstream dependencies and downstream writers. The authors also describe coverage gaps and changing engine formats. That operating example shows why a graph needs maintained evidence. In our policy trace, an unrecorded publication or routing step remains a question for the operator; an attractive graph cannot supply a missing relationship.

8. Bringing the lesson together

Q17 points to a stale extraction-to-index boundary. You can now identify the evidence for that conclusion, distinguish unresolved causes and specify both a corrected-request test and a historical-policy control.

Learning Checks

Attempt each answer before expanding the reasoning.

M13-L06-Q01

List the steps to trace backward from a wrong policy answer.

Show reasoning

Trace answer/request → retrieval entry/index → chunk → extraction version → source/applicability metadata, retaining owners and relevant routing/publication records. Q17 used C1/V1 while authorized source V2 and permitted X2/V2 already exist. The stale boundary lies between the available X2/V2 extraction and the index used by Q17. A failed indexing job, missed publication or old-index routing requires further evidence.

V2 applies to O101 because its 10 June purchase falls in the given from-1-June window. A 20 May control requires V1. Newest version alone is insufficient.

Expected Q18 evidence: inspect target index identity/version, returned chunk/policy, relevant exception and access rules for O101; verify its trace reaches V2. Test the historical purchase under V1 separately. These are planned checks, with no actual pass supplied.

M13-L06-Q02

A late correction changes a delivery date but pipeline only copies new records. What failed?

Show reasoning

Update/change capture and refresh; inspect pipeline owner, retry/refresh and freshness checks.

M13-L06-Q03

Why quarantine an unresolved customer link?

Show reasoning

Preserve it for investigation, prevent silent downstream use and keep error/count/resolution evidence.

M13-L06-Q04

What do bronze/silver/gold describe and why is the pattern not mandatory?

Show reasoning

A Databricks pattern for raw/refined/business-oriented layers; architectures can use different stages.

Sources

DB-MEDALLION: Databricks, GCP documentation, “What is the medallion lakehouse architecture?”, updated 11 September 2026. https://docs.databricks.com/gcp/en/lakehouse/medallion. Retrieved 29 October 2026. Vendor-specific design guidance.
DB-LINEAGE: Databricks, AWS documentation, “Lineage in Unity Catalog,” updated 26 October 2026. https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage. Retrieved 29 October 2026. Supported technical behavior with requirements/limitations; no complete business-lineage or rights guarantee.

Di Lin, Girish Lingappa, Jitender Aswani; Netflix Technology Blog. Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency. Published 2019-03-25. https://netflixtechblog.com/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977 (retrieved 2026-10-29). Historical company implementation, no measured error reduction claimed.

Learner Materials

Download Lesson 6 Learner Materials

Use the supplied fictional cases and templates for practice.

Practice

Write your answer, compare it with the model answer and decide what to revisit. These written checkpoints are self-assessed.

List the steps to trace backward from a wrong policy answer.

Show The Model Answer

Contributingpath/identities;V2appliesbyJune10purchase;source/extractionV2present/indexusedV1;causeunknown;expectedrerun+historicalcontrol.

Compare your reasoning with the model answer. These written checks are self-assessed.

A late correction changes a delivery date but pipeline only copies new records. What failed?

Show The Model Answer

Update/change capture and refresh; inspect pipeline owner, retry/refresh and freshness checks.

Compare your reasoning with the model answer. These written checks are self-assessed.

Why quarantine an unresolved customer link?

Show The Model Answer

Preserve it for investigation, prevent silent downstream use and keep error/count/resolution evidence.

Compare your reasoning with the model answer. These written checks are self-assessed.

What do bronze/silver/gold describe and why is the pattern not mandatory?

Show The Model Answer

A Databricks pattern for raw/refined/business-oriented layers; architectures can use different stages.

Compare your reasoning with the model answer. These written checks are self-assessed.

Open Practice And Save Your Work