Why this matters
Lineage answers two questions: where did this figure come from, and what breaks if this table changes. Without it, you ask whoever has been at the company longest.
Before you start
Lineage is a graph, and direction matters
Upstream is where data comes from, downstream is what depends on it. Both directions answer different questions. Upstream answers "can I trust this", because it shows you the sources and transformations behind a figure. Downstream answers "what will I break", which is the question to ask before changing or deprecating anything.
Column level lineage is where the real answers are
Table level lineage tells you that a report depends on an orders table. Column level lineage tells you that the revenue figure on that report is derived from a specific column, through a specific transformation. When two dashboards disagree about revenue, table lineage will not resolve it and column lineage usually will.
Lineage is collected, so it can have gaps
Lineage is built by an agent parsing queries, pipeline definitions and transformation code. Where that parsing succeeds the graph is reliable. Where a transformation happens somewhere the agent cannot see, in a spreadsheet or an undocumented script, the graph will show a break. A gap in the graph marks an unmanaged step in the path.
The interactive demo
Work through the demo below. It runs in your browser with nothing to install. Watch for:
- The basics of how data lineage is represented
- Configuring the Lineage Agent
- Navigating table level and column level lineage
This demo is also available on its own at the Learning Center.
After this module you should be able to
- Trace an asset upstream to its sources and downstream to its consumers
- Use column level lineage to explain how a specific figure is derived
- Recognize a gap in a lineage graph and understand what it implies