11 KiB
How Data Moves Through the Architecture
Time: ~10–15 minutes
Today we're connecting Medallion Architecture and Data Lake vs Data Warehouse vs Lakehouse .
You now understand:
Lesson 1
Bronze → Silver → Gold
Data maturity
Lesson 2
Data Lake → Data Warehouse → Lakehouse
Data architecture
Lesson 3
ETL / ELT
How data gets there and gets transformed
By the end of this lesson, when someone says:
"We're using an ELT pipeline."
you should understand exactly what they mean and why they might have chosen it.
1. The simplest definition
Both ETL and ELT describe processes for moving and transforming data.
The letters mean:
E = Extract
T = Transform
L = Load
The difference is the order.
ETL
Extract
↓
Transform
↓
Load
ELT
Extract
↓
Load
↓
Transform
That tiny difference represents an important architectural decision.
2. ETL
Extract → Transform → Load
Imagine we want data from JW Player.
We first:
Extract
Get the data from JW Player.
JW Player API
↓
Raw viewing data
Then:
Transform
Before putting it into our final destination, we:
-
remove duplicates
-
fix dates
-
standardize IDs
-
remove invalid records
-
calculate fields
-
join datasets
Then:
Load
We put the transformed data into the destination.
JW Player
↓
EXTRACT
↓
Raw data
↓
TRANSFORM
↓
Clean data
↓
LOAD
↓
Data Warehouse
The important idea:
The transformation happens BEFORE the data reaches its destination.
3. Why was ETL traditionally popular?
Historically, data warehouses were:
-
expensive
-
highly structured
-
limited in storage
-
optimized for specific analytical workloads
You didn't want to dump everything into them.
You wanted to clean the data first.
Think of it like moving into a small apartment.
You don't bring everything you've accumulated over 20 years.
You:
Pack
↓
Sort
↓
Throw things away
↓
Move
That's ETL.
You decide what is useful before loading it into the destination.
4. ELT
Modern cloud platforms changed things.
Storage became much cheaper.
Compute became much more scalable.
So organizations started saying:
Why don't we load the raw data first and transform it afterward?
That's:
Extract → Load → Transform
Example:
JW Player
↓
EXTRACT
↓
LOAD
↓
Data Lake / Warehouse
↓
TRANSFORM
↓
Cleaned data
The key difference:
We preserve the raw data in the destination before transforming it.
5. Your media-company example
Imagine you receive 100 million JW Player events.
With traditional ETL:
JW Player
↓
Extract
↓
Transformation Server
↓
Clean / Aggregate
↓
Data Warehouse
You might only load the processed results.
Perhaps:
Show A → 50,000 views
Show B → 25,000 views
But what happens six months later when someone asks:
"Can we calculate average viewing duration differently?"
Potential problem.
Maybe the original detailed data wasn't preserved.
With ELT:
JW Player
↓
Extract
↓
Data Lake
↓
RAW DATA STORED
↓
Transform
↓
Silver
↓
Transform
↓
Gold
Now you still have the original events.
You can go back.
Reprocess.
Create new metrics.
Investigate problems.
That's a major advantage.
6. Connecting this to Medallion Architecture
This is where everything we've learned starts connecting.
A modern architecture might look like:
SOURCE SYSTEMS
GA4
JW Player
CRM
Finance
OTT
Apps
│
│ EXTRACT
▼
┌─────────────────┐
│ BRONZE │
│ │
│ RAW DATA │
└────────┬────────┘
│
│ TRANSFORM
▼
┌─────────────────┐
│ SILVER │
│ │
│ CLEAN DATA │
└────────┬────────┘
│
│ TRANSFORM
▼
┌─────────────────┐
│ GOLD │
│ │
│ BUSINESS READY │
└────────┬────────┘
│
▼
Power BI
This architecture is conceptually closer to ELT.
Why?
Because we:
Extract
↓
Load raw data
↓
Transform afterward
The transformations create increasingly refined datasets.
7. ETL vs ELT
Here's your executive-level comparison:
| ETL | ELT | |
|---|---|---|
| Order | Extract → Transform → Load | Extract → Load → Transform |
| Raw data stored first | Not necessarily | Usually |
| Transformation location | Before destination | In/around destination platform |
| Flexibility | Lower | Higher |
| Reprocessing | Potentially harder | Easier if raw data retained |
| Traditional warehouses | Common | Less traditional |
| Modern cloud data platforms | Used | Very common |
Neither is automatically better.
Architecture depends on:
-
data volume
-
cost
-
security
-
privacy
-
latency
-
technology
-
business requirements
8. A subtle but important point
People often use the word:
Pipeline
A data pipeline is simply an automated process that moves and/or transforms data.
For example:
JW Player API
↓
Pipeline
↓
Bronze
Another pipeline might do:
Bronze
↓
Pipeline
↓
Silver
Another:
Silver
↓
Pipeline
↓
Gold
So your entire data architecture might contain dozens or hundreds of pipelines.
Some run:
Every 5 minutes
Some:
Hourly
Some:
Nightly
Some:
Real-time
9. Batch vs Streaming
This is related, but it's a separate concept we'll study later.
For now, just understand:
Batch
Data moves periodically.
Every night at 2 AM:
Yesterday's JW Player data
↓
Pipeline
↓
Data Lake
Streaming
Data continuously flows.
User watches video
↓
Event
↓
Pipeline
↓
Data Platform
Think:
Batch = packages of data periodically.
Streaming = continuous flow of events.
This distinction can become important for a media company.
10. Where does Azure Synapse fit?
Remember Lesson 2:
Synapse is a platform.
A company could potentially use Synapse capabilities to orchestrate pipelines that:
Extract
↓
Load
↓
Transform
For example:
JW Player
↓
Synapse Pipeline
↓
Azure Data Lake
↓
Bronze
↓
Transformation
↓
Silver
↓
Transformation
↓
Gold
↓
Power BI
Again, your company's actual architecture might differ.
When you're back from vacation, that's something worth mapping.
11. CTO perspective
Imagine an architect says:
"We're moving from ETL to ELT."
Don't just respond:
"Okay."
You want to understand why.
You could ask:
"What's driving the change?"
Then:
"Are we preserving all raw source data?"
"Where are transformations executed?"
"What's the impact on compute costs?"
"How are transformations tested?"
"How do we handle schema changes from source systems?"
"Can we reprocess historical data if business rules change?"
These questions reveal the architectural consequences.
12. Meeting scenario
Imagine your analyst says:
"We need to change the way we calculate total viewing hours, but we'd have to reload everything."
Instead of immediately discussing implementation, you could ask:
"Do we still have the original raw viewing events in our Bronze layer?"
If yes:
"Then can we reprocess the affected data through Silver and regenerate the Gold dataset using the new business rule?"
Now you're thinking architecturally.
You understand:
Bronze
Original truth
↓
Silver
Clean data
↓
Gold
Business interpretation
If the business rule changes, you may not need new source data.
You may need to reprocess existing data.
That's one reason retaining raw data can be valuable.
13. Another CTO-level issue: Garbage In, Garbage Out
ELT has a potential downside.
Because storing raw data is easy, organizations sometimes end up with:
Data Lake
↓
More Data
↓
More Data
↓
More Data
↓
Nobody knows what anything means
This is sometimes jokingly called a:
Data Swamp
A Data Lake without proper:
-
governance
-
metadata
-
ownership
-
documentation
-
lineage
-
quality controls
can become difficult to use.
This connects directly to your role in Data Governance.
The technology doesn't create trustworthy data.
Governance does.
14. Your executive communication exercise
Someone says in a meeting:
"We should move everything to ELT because it's the modern approach."
Your response could be:
"Before we decide on ETL versus ELT, I'd like to understand the problem we're solving. If the goal is to preserve raw data and give us more flexibility to reprocess it as business requirements change, ELT may make sense. But we should also understand the impact on storage, compute, governance and data quality before changing the architecture."
Notice the pattern again:
Clarify the problem → acknowledge the option → identify tradeoffs → guide the decision.
You don't need to be the data engineer.
You need to make sure the organization is solving the right problem.
Your Cheat Sheet
Remember this:
ETL: Clean it, then load it.
ELT: Load it, then clean it.
Pipeline: Automated movement/transformation of data.
Batch: Move data periodically.
Streaming: Move data continuously.
And the connection between our first three lessons:
SOURCE SYSTEMS
│
│
Extract / Load
│
▼
🥉 BRONZE
Raw Data
│
Transform
│
▼
🥈 SILVER
Clean Data
│
Transform
│
▼
🥇 GOLD
Business Data
│
▼
Power BI
That's the mental model I want you to keep.
Add to Obsidian
Create:
Knowledge
└── CTO Academy
└── Data
└── ETL vs ELT.md
At the bottom:
## Related Concepts
- [[Medallion Architecture]]
- [[Data Lake]]
- [[Data Warehouse]]
- [[Lakehouse]]
- [[Azure Synapse]]
- [[Data Pipeline]]
- [[Batch Processing]]
- [[Stream Processing]]
- [[Data Governance]]
- [[Data Lineage]]
Next: Lesson 4
I think we're ready to move inside the Gold layer.
Fact Tables & Dimension Tables.
You've encountered these concepts before when we were playing with Metabase and discussing grain/granularité, so Lesson 4 should connect something you already partially understand with the architecture we're building now:
Source → ELT → Bronze → Silver → Gold → Fact/Dimensions → Power BI.
That's when this whole data architecture should really start clicking together.