11 KiB
Estimated reading time: 10–12 minutes
Definition
Data Lineage is the ability to understand where data comes from, how it moves, how it changes, and where it is ultimately used.
It answers questions like:
Where did this number come from?
Which source system produced it?
What transformations were applied?
Which reports depend on it?
A simple lineage path might look like:
JW Player
↓
Raw Viewing Events
↓
Bronze
↓
Silver
↓
Gold
↓
[[Fact Table]]
↓
[[Semantic Layer]]
↓
Power BI Report
Data Lineage documents that path.
Simple Mental Model
Think:
Data Lineage = The family tree and travel history of data.
It tells you:
Where it started
↓
What happened to it
↓
Where it ended up
For a KPI, lineage lets you trace backward:
Executive Dashboard
↓
Total Watch Hours
↓
Semantic Layer
↓
fact_viewing.watch_seconds
↓
Gold
↓
Silver
↓
Bronze
↓
JW Player
How It Works
Imagine a Power BI dashboard displays:
Total Watch Hours: 2.4 million
Someone questions the number.
Without Data Lineage, the investigation might look like:
Ask Power BI analyst
↓
Check report logic
↓
Ask data engineer
↓
Check SQL
↓
Ask platform team
↓
Check source data
Nobody has a clear view of the full chain.
With Data Lineage, you can trace:
Power BI
↓
Metric: Total Watch Hours
↓
[[Semantic Layer]]
↓
SUM(watch_seconds) / 3600
↓
fact_viewing
↓
Gold transformation
↓
Silver viewing events
↓
Bronze raw events
↓
JW Player API
Now the investigation is much more structured.
Example
Suppose your company changes video providers.
You move from:
JW Player
to:
New Video Platform
You want to know:
Which dashboards will be affected?
Data Lineage should help answer:
JW Player
↓
Viewing Pipeline
↓
fact_viewing
↓
Semantic Layer
↓
Total Watch Hours
↓
Executive Dashboard
↓
Content Performance Dashboard
↓
Marketing Dashboard
This is sometimes called impact analysis.
Instead of discovering broken reports after migration, you can identify dependencies beforehand.
How It Fits Into the Bigger Picture
Now the architecture you've been learning connects like this:
Source Systems
↓
[[ETL vs ELT]]
↓
[[Medallion Architecture]]
↓
Bronze
↓
Silver
↓
Gold
↓
[[Fact Table]]
+
[[Dimension Table]]
↓
[[Star Schema]]
or
[[Snowflake Schema]]
↓
[[Semantic Layer]]
↓
Power BI
Data Lineage cuts across the entire architecture.
It does not represent one layer.
It describes the connections between all the layers.
Conceptually:
SOURCE → TRANSFORMATION → MODEL → METRIC → REPORT
That entire chain is lineage.
Technical Lineage vs Business Lineage
Data Lineage can exist at different levels.
Technical Lineage
Focuses on systems, tables, columns, and transformations.
Example:
jw_events.watch_time_ms
↓
silver_viewing.watch_seconds
↓
fact_viewing.watch_seconds
↓
semantic_model.total_watch_hours
This is useful for:
-
Data engineers
-
Developers
-
Analysts
-
Troubleshooting
Business Lineage
Focuses on business meaning.
Example:
JW Player Viewing Events
↓
Validated Viewing Data
↓
Viewing Fact
↓
Total Watch Hours
↓
Executive Audience Dashboard
This is easier for:
-
Managers
-
Executives
-
Data owners
-
Business stakeholders
A mature data environment often needs both.
Column-Level Lineage
Lineage can become very detailed.
Imagine the final metric:
Total Watch Hours
comes from:
watch_time_ms
in the source system.
The transformation could be:
JW Player
watch_time_ms
↓
divide by 1000
↓
watch_seconds
↓
divide by 3600
↓
Total Watch Hours
Column-level lineage tracks that exact transformation.
This is valuable when someone asks:
Why is the number different from the source platform?
You can see whether:
-
Units were converted
-
Filters were applied
-
Records were excluded
-
Duplicates were removed
-
Business rules were introduced
Data Lineage and Data Quality
Imagine a dashboard suddenly shows:
Total Watch Hours
↓ 40%
The first question might be:
Did audience actually drop?
But the problem could be:
JW Player
↓
API changed
↓
Pipeline failed
↓
Silver missing records
↓
Gold incomplete
↓
Dashboard shows lower number
Without lineage, a technical issue can look like a business event.
With lineage, the team can trace the problem upstream.
This connects directly to Data Quality.
A number can be mathematically correct based on the data it received, while the underlying data is incomplete.
Data Lineage and Data Governance
Data Governance asks questions such as:
Who owns this data?
What does this KPI mean?
Who can access it?
Can we trust it?
Data Lineage helps answer:
Where did it come from?
Together:
[[Data Governance]]
+
[[Data Lineage]]
+
[[Semantic Layer]]
=
More trustworthy analytics
Governance without lineage can define what a metric should mean without proving how it was produced.
Lineage without governance can show the technical path without explaining who owns the definition.
You usually need both.
My Company / Real-World Context
Imagine the CEO asks:
Why does the Power BI dashboard show 1.8 million watch hours while JW Player shows 2.1 million?
A weak process might involve several days of investigation.
You ask:
-
Power BI analyst
-
Data engineer
-
JW Player team
-
Vendor
-
Data Governance
Each person knows one piece.
With proper lineage, you could trace:
JW Player Raw Metric
2.1M hours
↓
Bronze
↓
Remove invalid events
↓
Silver
↓
Exclude internal traffic
↓
Gold
↓
Apply governed viewing definition
↓
Semantic Layer
↓
Power BI
1.8M hours
Now the difference may actually be intentional.
The source platform might count all activity.
Your governed KPI might exclude:
-
Internal testing
-
Bots
-
Invalid sessions
-
Duplicate events
The numbers are different because they answer different questions.
Lineage helps explain why.
Another Real-World Example: Vendor Migration
Imagine your company wants to replace JW Player.
Before signing the new vendor, you should understand what depends on the current one.
Lineage could reveal:
JW Player
├── Viewing Analytics
│ ├── Executive Dashboard
│ ├── Content Dashboard
│ └── Audience Report
│
├── Advertising Metrics
│ └── Revenue Dashboard
│
└── Content Metadata
├── Website
└── Mobile App
Now the migration is no longer:
Replace JW Player.
It is:
Replace JW Player while preserving all downstream dependencies.
That is a much more accurate project scope.
CTO Perspective
Data Lineage is important for a CTO because modern organizations have interconnected systems.
A small source change can create unexpected downstream effects.
The CTO-level questions are:
"If this source changes, what breaks?"
"If this KPI is wrong, can we trace it back to the source?"
"Do we know which reports depend on this dataset?"
"Can we explain how an executive number was produced?"
Lineage is also important for:
-
Auditing
-
Compliance
-
Security
-
Data migrations
-
Vendor changes
-
Troubleshooting
-
Impact analysis
A mature organization should not depend entirely on one employee's memory to understand how critical data moves.
The architecture itself should be documented or discoverable.
Questions to Ask
-
What is the original source of this data?
-
Which pipelines transform it?
-
Where does it move through Bronze, Silver, and Gold?
-
Which Fact Table and Dimension Table depend on it?
-
Which metrics in the Semantic Layer use it?
-
Which reports consume those metrics?
-
What happens downstream if this source changes?
-
Can we trace a KPI back to the source field?
-
Are transformations documented?
-
Who owns each stage of the data flow?
-
Is lineage captured automatically or manually?
-
How current is our lineage documentation?
Meeting Scenario
Situation:
A vendor announces that an API field your company uses will be removed in 60 days.
The discussion immediately becomes:
"The data team needs to update the API integration."
Possible response:
"Before we treat this only as an integration change, can we trace the lineage of that field? I want to know which transformations, metrics, and reports depend on it so we understand the full impact."
The team discovers:
API Field
↓
Silver Transformation
↓
fact_viewing
↓
Completion Rate
↓
Three Power BI Dashboards
↓
Monthly Executive Report
You could then respond:
"Good. Let's treat this as a dependency change across the full lineage, not just an API ticket. We need to update the source mapping, validate the transformation, confirm the KPI calculation, and test the affected reports before the vendor removes the field."
You have turned a narrow technical task into a properly scoped change-management problem.
Key Takeaways
-
Data Lineage shows where data originates, how it changes, and where it is consumed.
-
Lineage crosses the entire data architecture rather than belonging to one layer.
-
Technical lineage focuses on tables, columns, and transformations.
-
Business lineage focuses on business concepts, metrics, and reports.
-
Column-level lineage can explain exactly how a KPI was calculated.
-
Lineage supports troubleshooting and Data Quality investigations.
-
Lineage enables impact analysis before source systems or vendors change.
-
Data Governance defines ownership and meaning; Data Lineage explains the path.
-
A CTO should be able to ask: "If this changes, what downstream systems and decisions are affected?"
Related Concepts
-
Data Governance
-
Data Quality
-
Semantic Layer
-
Medallion Architecture
-
ETL vs ELT
-
Fact Table
-
Dimension Table
-
Star Schema
-
Data Pipeline
-
Metadata
-
Data Catalog
-
Impact Analysis
The natural next lesson is Lesson 10: Data Governance.md. Given that this is already part of your actual role, it will be interesting because we'll compare what Data Governance formally means against what you're currently doing in practice, including ownership, stewardship, KPI definitions, quality, lineage, and accountability.