vault backup: 2026-07-18 23:06:13

10 Knowledge/CTO Academy/Data/ETL vs ELT.md

@@ -0,0 +1,757 @@
+## How Data Moves Through the Architecture
+
+**Time: ~10–15 minutes**
+
+Today we're connecting [[Medallion Architecture]] and [[Data Lake vs Data Warehouse vs Lakehouse]]
+.
+
+You now understand:
+
+```text
+Lesson 1
+Bronze → Silver → Gold
+Data maturity
+
+Lesson 2
+Data Lake → Data Warehouse → Lakehouse
+Data architecture
+
+Lesson 3
+ETL / ELT
+How data gets there and gets transformed
+```
+
+By the end of this lesson, when someone says:
+
+> "We're using an ELT pipeline."
+
+you should understand exactly what they mean and why they might have chosen it.
+
+---
+
+# 1. The simplest definition
+
+Both ETL and ELT describe processes for moving and transforming data.
+
+The letters mean:
+
+```text
+E = Extract
+T = Transform
+L = Load
+```
+
+The difference is **the order**.
+
+### ETL
+
+```text
+Extract
+   ↓
+Transform
+   ↓
+Load
+```
+
+### ELT
+
+```text
+Extract
+   ↓
+Load
+   ↓
+Transform
+```
+
+That tiny difference represents an important architectural decision.
+
+---
+
+# 2. ETL
+
+## Extract → Transform → Load
+
+Imagine we want data from JW Player.
+
+We first:
+
+### Extract
+
+Get the data from JW Player.
+
+```text
+JW Player API
+      ↓
+Raw viewing data
+```
+
+Then:
+
+### Transform
+
+Before putting it into our final destination, we:
+
+- remove duplicates
+    
+- fix dates
+    
+- standardize IDs
+    
+- remove invalid records
+    
+- calculate fields
+    
+- join datasets
+    
+
+Then:
+
+### Load
+
+We put the transformed data into the destination.
+
+```text
+JW Player
+    ↓
+EXTRACT
+    ↓
+Raw data
+    ↓
+TRANSFORM
+    ↓
+Clean data
+    ↓
+LOAD
+    ↓
+Data Warehouse
+```
+
+The important idea:
+
+> **The transformation happens BEFORE the data reaches its destination.**
+
+---
+
+# 3. Why was ETL traditionally popular?
+
+Historically, data warehouses were:
+
+- expensive
+    
+- highly structured
+    
+- limited in storage
+    
+- optimized for specific analytical workloads
+    
+
+You didn't want to dump everything into them.
+
+You wanted to clean the data first.
+
+Think of it like moving into a small apartment.
+
+You don't bring everything you've accumulated over 20 years.
+
+You:
+
+```text
+Pack
+↓
+Sort
+↓
+Throw things away
+↓
+Move
+```
+
+That's ETL.
+
+You decide what is useful **before loading it into the destination**.
+
+---
+
+# 4. ELT
+
+Modern cloud platforms changed things.
+
+Storage became much cheaper.
+
+Compute became much more scalable.
+
+So organizations started saying:
+
+> Why don't we load the raw data first and transform it afterward?
+
+That's:
+
+## Extract → Load → Transform
+
+Example:
+
+```text
+JW Player
+    ↓
+EXTRACT
+    ↓
+LOAD
+    ↓
+Data Lake / Warehouse
+    ↓
+TRANSFORM
+    ↓
+Cleaned data
+```
+
+The key difference:
+
+> **We preserve the raw data in the destination before transforming it.**
+
+---
+
+# 5. Your media-company example
+
+Imagine you receive **100 million JW Player events**.
+
+With traditional ETL:
+
+```text
+JW Player
+     ↓
+Extract
+     ↓
+Transformation Server
+     ↓
+Clean / Aggregate
+     ↓
+Data Warehouse
+```
+
+You might only load the processed results.
+
+Perhaps:
+
+```text
+Show A → 50,000 views
+Show B → 25,000 views
+```
+
+But what happens six months later when someone asks:
+
+> "Can we calculate average viewing duration differently?"
+
+Potential problem.
+
+Maybe the original detailed data wasn't preserved.
+
+---
+
+With ELT:
+
+```text
+JW Player
+     ↓
+Extract
+     ↓
+Data Lake
+     ↓
+RAW DATA STORED
+     ↓
+Transform
+     ↓
+Silver
+     ↓
+Transform
+     ↓
+Gold
+```
+
+Now you still have the original events.
+
+You can go back.
+
+Reprocess.
+
+Create new metrics.
+
+Investigate problems.
+
+That's a major advantage.
+
+---
+
+# 6. Connecting this to Medallion Architecture
+
+This is where everything we've learned starts connecting.
+
+A modern architecture might look like:
+
+```text
+SOURCE SYSTEMS
+
+GA4
+JW Player
+CRM
+Finance
+OTT
+Apps
+
+      │
+      │ EXTRACT
+      ▼
+
+┌─────────────────┐
+│     BRONZE      │
+│                 │
+│    RAW DATA     │
+└────────┬────────┘
+         │
+         │ TRANSFORM
+         ▼
+┌─────────────────┐
+│     SILVER      │
+│                 │
+│   CLEAN DATA    │
+└────────┬────────┘
+         │
+         │ TRANSFORM
+         ▼
+┌─────────────────┐
+│      GOLD       │
+│                 │
+│ BUSINESS READY  │
+└────────┬────────┘
+         │
+         ▼
+
+      Power BI
+```
+
+This architecture is conceptually closer to **ELT**.
+
+Why?
+
+Because we:
+
+```text
+Extract
+↓
+Load raw data
+↓
+Transform afterward
+```
+
+The transformations create increasingly refined datasets.
+
+---
+
+# 7. ETL vs ELT
+
+Here's your executive-level comparison:
+
+||ETL|ELT|
+|---|---|---|
+|Order|Extract → Transform → Load|Extract → Load → Transform|
+|Raw data stored first|Not necessarily|Usually|
+|Transformation location|Before destination|In/around destination platform|
+|Flexibility|Lower|Higher|
+|Reprocessing|Potentially harder|Easier if raw data retained|
+|Traditional warehouses|Common|Less traditional|
+|Modern cloud data platforms|Used|Very common|
+
+Neither is automatically better.
+
+Architecture depends on:
+
+- data volume
+    
+- cost
+    
+- security
+    
+- privacy
+    
+- latency
+    
+- technology
+    
+- business requirements
+    
+
+---
+
+# 8. A subtle but important point
+
+People often use the word:
+
+> **Pipeline**
+
+A data pipeline is simply an automated process that moves and/or transforms data.
+
+For example:
+
+```text
+JW Player API
+     ↓
+Pipeline
+     ↓
+Bronze
+```
+
+Another pipeline might do:
+
+```text
+Bronze
+     ↓
+Pipeline
+     ↓
+Silver
+```
+
+Another:
+
+```text
+Silver
+     ↓
+Pipeline
+     ↓
+Gold
+```
+
+So your entire data architecture might contain dozens or hundreds of pipelines.
+
+Some run:
+
+```text
+Every 5 minutes
+```
+
+Some:
+
+```text
+Hourly
+```
+
+Some:
+
+```text
+Nightly
+```
+
+Some:
+
+```text
+Real-time
+```
+
+---
+
+# 9. Batch vs Streaming
+
+This is related, but it's a separate concept we'll study later.
+
+For now, just understand:
+
+### Batch
+
+Data moves periodically.
+
+```text
+Every night at 2 AM:
+
+Yesterday's JW Player data
+        ↓
+Pipeline
+        ↓
+Data Lake
+```
+
+### Streaming
+
+Data continuously flows.
+
+```text
+User watches video
+       ↓
+Event
+       ↓
+Pipeline
+       ↓
+Data Platform
+```
+
+Think:
+
+> **Batch = packages of data periodically.**
+
+> **Streaming = continuous flow of events.**
+
+This distinction can become important for a media company.
+
+---
+
+# 10. Where does Azure Synapse fit?
+
+Remember Lesson 2:
+
+Synapse is a **platform**.
+
+A company could potentially use Synapse capabilities to orchestrate pipelines that:
+
+```text
+Extract
+   ↓
+Load
+   ↓
+Transform
+```
+
+For example:
+
+```text
+JW Player
+    ↓
+Synapse Pipeline
+    ↓
+Azure Data Lake
+    ↓
+Bronze
+    ↓
+Transformation
+    ↓
+Silver
+    ↓
+Transformation
+    ↓
+Gold
+    ↓
+Power BI
+```
+
+Again, your company's actual architecture might differ.
+
+When you're back from vacation, that's something worth mapping.
+
+---
+
+# 11. CTO perspective
+
+Imagine an architect says:
+
+> "We're moving from ETL to ELT."
+
+Don't just respond:
+
+> "Okay."
+
+You want to understand **why**.
+
+You could ask:
+
+> "What's driving the change?"
+
+Then:
+
+> "Are we preserving all raw source data?"
+
+> "Where are transformations executed?"
+
+> "What's the impact on compute costs?"
+
+> "How are transformations tested?"
+
+> "How do we handle schema changes from source systems?"
+
+> "Can we reprocess historical data if business rules change?"
+
+These questions reveal the architectural consequences.
+
+---
+
+# 12. Meeting scenario
+
+Imagine your analyst says:
+
+> "We need to change the way we calculate total viewing hours, but we'd have to reload everything."
+
+Instead of immediately discussing implementation, you could ask:
+
+> **"Do we still have the original raw viewing events in our Bronze layer?"**
+
+If yes:
+
+> **"Then can we reprocess the affected data through Silver and regenerate the Gold dataset using the new business rule?"**
+
+Now you're thinking architecturally.
+
+You understand:
+
+```text
+Bronze
+Original truth
+     ↓
+Silver
+Clean data
+     ↓
+Gold
+Business interpretation
+```
+
+If the **business rule** changes, you may not need new source data.
+
+You may need to **reprocess existing data**.
+
+That's one reason retaining raw data can be valuable.
+
+---
+
+# 13. Another CTO-level issue: Garbage In, Garbage Out
+
+ELT has a potential downside.
+
+Because storing raw data is easy, organizations sometimes end up with:
+
+```text
+Data Lake
+    ↓
+More Data
+    ↓
+More Data
+    ↓
+More Data
+    ↓
+Nobody knows what anything means
+```
+
+This is sometimes jokingly called a:
+
+> **Data Swamp**
+
+A Data Lake without proper:
+
+- governance
+    
+- metadata
+    
+- ownership
+    
+- documentation
+    
+- lineage
+    
+- quality controls
+    
+
+can become difficult to use.
+
+This connects directly to your role in **Data Governance**.
+
+The technology doesn't create trustworthy data.
+
+Governance does.
+
+---
+
+# 14. Your executive communication exercise
+
+Someone says in a meeting:
+
+> "We should move everything to ELT because it's the modern approach."
+
+Your response could be:
+
+> **"Before we decide on ETL versus ELT, I'd like to understand the problem we're solving. If the goal is to preserve raw data and give us more flexibility to reprocess it as business requirements change, ELT may make sense. But we should also understand the impact on storage, compute, governance and data quality before changing the architecture."**
+
+Notice the pattern again:
+
+**Clarify the problem → acknowledge the option → identify tradeoffs → guide the decision.**
+
+You don't need to be the data engineer.
+
+You need to make sure the organization is solving the right problem.
+
+---
+
+# Your Cheat Sheet
+
+Remember this:
+
+> **ETL:** Clean it, then load it.
+
+> **ELT:** Load it, then clean it.
+
+> **Pipeline:** Automated movement/transformation of data.
+
+> **Batch:** Move data periodically.
+
+> **Streaming:** Move data continuously.
+
+And the connection between our first three lessons:
+
+```text
+        SOURCE SYSTEMS
+              │
+              │
+         Extract / Load
+              │
+              ▼
+          🥉 BRONZE
+           Raw Data
+              │
+          Transform
+              │
+              ▼
+          🥈 SILVER
+          Clean Data
+              │
+          Transform
+              │
+              ▼
+          🥇 GOLD
+       Business Data
+              │
+              ▼
+           Power BI
+```
+
+That's the mental model I want you to keep.
+
+## Add to Obsidian
+
+Create:
+
+```text
+Knowledge
+└── CTO Academy
+    └── Data
+        └── ETL vs ELT.md
+```
+
+At the bottom:
+
+```markdown
+## Related Concepts
+
+- [[Medallion Architecture]]
+- [[Data Lake]]
+- [[Data Warehouse]]
+- [[Lakehouse]]
+- [[Azure Synapse]]
+- [[Data Pipeline]]
+- [[Batch Processing]]
+- [[Stream Processing]]
+- [[Data Governance]]
+- [[Data Lineage]]
+```
+
+### Next: Lesson 4
+
+I think we're ready to move **inside the Gold layer**.
+
+**Fact Tables & Dimension Tables.**
+
+You've encountered these concepts before when we were playing with Metabase and discussing grain/granularité, so Lesson 4 should connect something you already partially understand with the architecture we're building now:
+
+**Source → ELT → Bronze → Silver → Gold → Fact/Dimensions → Power BI.**
+
+That's when this whole data architecture should really start clicking together.