vault backup: 2026-07-20 21:03:03

.obsidian/workspace.json

@@ -11,10 +11,14 @@
             "id": "579a503273b0c5ec",
             "type": "leaf",
             "state": {
-              "type": "empty",
+              "type": "markdown",
-              "state": {},
+              "state": {
+                "file": "10 Knowledge/CTO Academy/Data/Data Lineage.md",
+                "mode": "source",
+                "source": false
+              },
               "icon": "lucide-file",
-              "title": "New tab"
+              "title": "Data Lineage"
             }
           }
         ]
@@ -192,8 +196,10 @@
       "omnisearch:Omnisearch": false
     }
   },
-  "active": "579a503273b0c5ec",
+  "active": "92c89fdf4eb1874e",
   "lastOpenFiles": [
+    "10 Knowledge/CTO Academy/Data/Quizzes",
+    "10 Knowledge/CTO Academy/Data/Data Lineage.md",
     "10 Knowledge/CTO Academy/Data/Semantic Layer.md",
     "10 Knowledge/CTO Academy/Data/Snowflake Schema.md",
     "10 Knowledge/CTO Academy/Data/Star Schema.md",

10 Knowledge/CTO Academy/Data/Data Lineage.md

@@ -0,0 +1,673 @@
+**Estimated reading time:** 10–12 minutes
+
+## Definition
+
+**Data Lineage** is the ability to understand where data comes from, how it moves, how it changes, and where it is ultimately used.
+
+It answers questions like:
+
+> Where did this number come from?
+
+> Which source system produced it?
+
+> What transformations were applied?
+
+> Which reports depend on it?
+
+A simple lineage path might look like:
+
+```text
+JW Player
+    ↓
+Raw Viewing Events
+    ↓
+Bronze
+    ↓
+Silver
+    ↓
+Gold
+    ↓
+[[Fact Table]]
+    ↓
+[[Semantic Layer]]
+    ↓
+Power BI Report
+```
+
+Data Lineage documents that path.
+
+---
+
+## Simple Mental Model
+
+Think:
+
+> **Data Lineage = The family tree and travel history of data.**
+
+It tells you:
+
+```text
+Where it started
+↓
+What happened to it
+↓
+Where it ended up
+```
+
+For a KPI, lineage lets you trace backward:
+
+```text
+Executive Dashboard
+        ↓
+Total Watch Hours
+        ↓
+Semantic Layer
+        ↓
+fact_viewing.watch_seconds
+        ↓
+Gold
+        ↓
+Silver
+        ↓
+Bronze
+        ↓
+JW Player
+```
+
+---
+
+## How It Works
+
+Imagine a Power BI dashboard displays:
+
+> Total Watch Hours: 2.4 million
+
+Someone questions the number.
+
+Without Data Lineage, the investigation might look like:
+
+```text
+Ask Power BI analyst
+↓
+Check report logic
+↓
+Ask data engineer
+↓
+Check SQL
+↓
+Ask platform team
+↓
+Check source data
+```
+
+Nobody has a clear view of the full chain.
+
+With Data Lineage, you can trace:
+
+```text
+Power BI
+↓
+Metric: Total Watch Hours
+↓
+[[Semantic Layer]]
+↓
+SUM(watch_seconds) / 3600
+↓
+fact_viewing
+↓
+Gold transformation
+↓
+Silver viewing events
+↓
+Bronze raw events
+↓
+JW Player API
+```
+
+Now the investigation is much more structured.
+
+---
+
+## Example
+
+Suppose your company changes video providers.
+
+You move from:
+
+```text
+JW Player
+```
+
+to:
+
+```text
+New Video Platform
+```
+
+You want to know:
+
+> Which dashboards will be affected?
+
+Data Lineage should help answer:
+
+```text
+JW Player
+    ↓
+Viewing Pipeline
+    ↓
+fact_viewing
+    ↓
+Semantic Layer
+    ↓
+Total Watch Hours
+    ↓
+Executive Dashboard
+    ↓
+Content Performance Dashboard
+    ↓
+Marketing Dashboard
+```
+
+This is sometimes called **impact analysis**.
+
+Instead of discovering broken reports after migration, you can identify dependencies beforehand.
+
+---
+
+## How It Fits Into the Bigger Picture
+
+Now the architecture you've been learning connects like this:
+
+```text
+Source Systems
+      ↓
+[[ETL vs ELT]]
+      ↓
+[[Medallion Architecture]]
+      ↓
+Bronze
+      ↓
+Silver
+      ↓
+Gold
+      ↓
+[[Fact Table]]
++
+[[Dimension Table]]
+      ↓
+[[Star Schema]]
+or
+[[Snowflake Schema]]
+      ↓
+[[Semantic Layer]]
+      ↓
+Power BI
+```
+
+[[Data Lineage]] cuts **across the entire architecture**.
+
+It does not represent one layer.
+
+It describes the connections between all the layers.
+
+Conceptually:
+
+```text
+SOURCE → TRANSFORMATION → MODEL → METRIC → REPORT
+```
+
+That entire chain is lineage.
+
+---
+
+## Technical Lineage vs Business Lineage
+
+Data Lineage can exist at different levels.
+
+### Technical Lineage
+
+Focuses on systems, tables, columns, and transformations.
+
+Example:
+
+```text
+jw_events.watch_time_ms
+↓
+silver_viewing.watch_seconds
+↓
+fact_viewing.watch_seconds
+↓
+semantic_model.total_watch_hours
+```
+
+This is useful for:
+
+- Data engineers
+    
+- Developers
+    
+- Analysts
+    
+- Troubleshooting
+    
+
+### Business Lineage
+
+Focuses on business meaning.
+
+Example:
+
+```text
+JW Player Viewing Events
+↓
+Validated Viewing Data
+↓
+Viewing Fact
+↓
+Total Watch Hours
+↓
+Executive Audience Dashboard
+```
+
+This is easier for:
+
+- Managers
+    
+- Executives
+    
+- Data owners
+    
+- Business stakeholders
+    
+
+A mature data environment often needs both.
+
+---
+
+## Column-Level Lineage
+
+Lineage can become very detailed.
+
+Imagine the final metric:
+
+```text
+Total Watch Hours
+```
+
+comes from:
+
+```text
+watch_time_ms
+```
+
+in the source system.
+
+The transformation could be:
+
+```text
+JW Player
+watch_time_ms
+      ↓
+divide by 1000
+      ↓
+watch_seconds
+      ↓
+divide by 3600
+      ↓
+Total Watch Hours
+```
+
+Column-level lineage tracks that exact transformation.
+
+This is valuable when someone asks:
+
+> Why is the number different from the source platform?
+
+You can see whether:
+
+- Units were converted
+    
+- Filters were applied
+    
+- Records were excluded
+    
+- Duplicates were removed
+    
+- Business rules were introduced
+    
+
+---
+
+## Data Lineage and Data Quality
+
+Imagine a dashboard suddenly shows:
+
+```text
+Total Watch Hours
+↓ 40%
+```
+
+The first question might be:
+
+> Did audience actually drop?
+
+But the problem could be:
+
+```text
+JW Player
+↓
+API changed
+↓
+Pipeline failed
+↓
+Silver missing records
+↓
+Gold incomplete
+↓
+Dashboard shows lower number
+```
+
+Without lineage, a technical issue can look like a business event.
+
+With lineage, the team can trace the problem upstream.
+
+This connects directly to [[Data Quality]].
+
+A number can be mathematically correct based on the data it received, while the underlying data is incomplete.
+
+---
+
+## Data Lineage and Data Governance
+
+[[Data Governance]] asks questions such as:
+
+> Who owns this data?
+
+> What does this KPI mean?
+
+> Who can access it?
+
+> Can we trust it?
+
+Data Lineage helps answer:
+
+> Where did it come from?
+
+Together:
+
+```text
+[[Data Governance]]
+      +
+[[Data Lineage]]
+      +
+[[Semantic Layer]]
+      =
+More trustworthy analytics
+```
+
+Governance without lineage can define what a metric should mean without proving how it was produced.
+
+Lineage without governance can show the technical path without explaining who owns the definition.
+
+You usually need both.
+
+---
+
+## My Company / Real-World Context
+
+Imagine the CEO asks:
+
+> Why does the Power BI dashboard show 1.8 million watch hours while JW Player shows 2.1 million?
+
+A weak process might involve several days of investigation.
+
+You ask:
+
+- Power BI analyst
+    
+- Data engineer
+    
+- JW Player team
+    
+- Vendor
+    
+- Data Governance
+    
+
+Each person knows one piece.
+
+With proper lineage, you could trace:
+
+```text
+JW Player Raw Metric
+2.1M hours
+      ↓
+Bronze
+      ↓
+Remove invalid events
+      ↓
+Silver
+      ↓
+Exclude internal traffic
+      ↓
+Gold
+      ↓
+Apply governed viewing definition
+      ↓
+Semantic Layer
+      ↓
+Power BI
+1.8M hours
+```
+
+Now the difference may actually be intentional.
+
+The source platform might count all activity.
+
+Your governed KPI might exclude:
+
+- Internal testing
+    
+- Bots
+    
+- Invalid sessions
+    
+- Duplicate events
+    
+
+The numbers are different because they answer different questions.
+
+Lineage helps explain why.
+
+---
+
+## Another Real-World Example: Vendor Migration
+
+Imagine your company wants to replace JW Player.
+
+Before signing the new vendor, you should understand what depends on the current one.
+
+Lineage could reveal:
+
+```text
+JW Player
+├── Viewing Analytics
+│   ├── Executive Dashboard
+│   ├── Content Dashboard
+│   └── Audience Report
+│
+├── Advertising Metrics
+│   └── Revenue Dashboard
+│
+└── Content Metadata
+    ├── Website
+    └── Mobile App
+```
+
+Now the migration is no longer:
+
+> Replace JW Player.
+
+It is:
+
+> Replace JW Player while preserving all downstream dependencies.
+
+That is a much more accurate project scope.
+
+---
+
+## CTO Perspective
+
+Data Lineage is important for a CTO because modern organizations have interconnected systems.
+
+A small source change can create unexpected downstream effects.
+
+The CTO-level questions are:
+
+> **"If this source changes, what breaks?"**
+
+> **"If this KPI is wrong, can we trace it back to the source?"**
+
+> **"Do we know which reports depend on this dataset?"**
+
+> **"Can we explain how an executive number was produced?"**
+
+Lineage is also important for:
+
+- Auditing
+    
+- Compliance
+    
+- Security
+    
+- Data migrations
+    
+- Vendor changes
+    
+- Troubleshooting
+    
+- Impact analysis
+    
+
+A mature organization should not depend entirely on one employee's memory to understand how critical data moves.
+
+The architecture itself should be documented or discoverable.
+
+---
+
+### Questions to Ask
+
+- What is the original source of this data?
+    
+- Which pipelines transform it?
+    
+- Where does it move through Bronze, Silver, and Gold?
+    
+- Which [[Fact Table|Fact Tables]] and [[Dimension Table|Dimension Tables]] depend on it?
+    
+- Which metrics in the [[Semantic Layer]] use it?
+    
+- Which reports consume those metrics?
+    
+- What happens downstream if this source changes?
+    
+- Can we trace a KPI back to the source field?
+    
+- Are transformations documented?
+    
+- Who owns each stage of the data flow?
+    
+- Is lineage captured automatically or manually?
+    
+- How current is our lineage documentation?
+    
+
+---
+
+## Meeting Scenario
+
+**Situation:**
+
+A vendor announces that an API field your company uses will be removed in 60 days.
+
+The discussion immediately becomes:
+
+> "The data team needs to update the API integration."
+
+**Possible response:**
+
+> "Before we treat this only as an integration change, can we trace the lineage of that field? I want to know which transformations, metrics, and reports depend on it so we understand the full impact."
+
+The team discovers:
+
+```text
+API Field
+↓
+Silver Transformation
+↓
+fact_viewing
+↓
+Completion Rate
+↓
+Three Power BI Dashboards
+↓
+Monthly Executive Report
+```
+
+You could then respond:
+
+> "Good. Let's treat this as a dependency change across the full lineage, not just an API ticket. We need to update the source mapping, validate the transformation, confirm the KPI calculation, and test the affected reports before the vendor removes the field."
+
+You have turned a narrow technical task into a properly scoped change-management problem.
+
+---
+
+## Key Takeaways
+
+- **Data Lineage** shows where data originates, how it changes, and where it is consumed.
+    
+- Lineage crosses the entire data architecture rather than belonging to one layer.
+    
+- Technical lineage focuses on tables, columns, and transformations.
+    
+- Business lineage focuses on business concepts, metrics, and reports.
+    
+- Column-level lineage can explain exactly how a KPI was calculated.
+    
+- Lineage supports troubleshooting and [[Data Quality]] investigations.
+    
+- Lineage enables impact analysis before source systems or vendors change.
+    
+- [[Data Governance]] defines ownership and meaning; Data Lineage explains the path.
+    
+- A CTO should be able to ask: **"If this changes, what downstream systems and decisions are affected?"**
+    
+
+## Related Concepts
+
+- [[Data Governance]]
+    
+- [[Data Quality]]
+    
+- [[Semantic Layer]]
+    
+- [[Medallion Architecture]]
+    
+- [[ETL vs ELT]]
+    
+- [[Fact Table]]
+    
+- [[Dimension Table]]
+    
+- [[Star Schema]]
+    
+- [[Data Pipeline]]
+    
+- [[Metadata]]
+    
+- [[Data Catalog]]
+    
+- [[Impact Analysis]]
+    
+
+The natural next lesson is **Lesson 10: `Data Governance.md`**. Given that this is already part of your actual role, it will be interesting because we'll compare what **Data Governance formally means** against what you're currently doing in practice, including ownership, stewardship, KPI definitions, quality, lineage, and accountability.