SecondBrain/10 Knowledge/CTO Academy/Data/ETL vs ELT.md

## How Data Moves Through the Architecture

**Time: ~10–15 minutes**

Today we're connecting [[Medallion Architecture]] and [[Data Lake vs Data Warehouse vs Lakehouse]]
.

You now understand:

```text
Lesson 1
Bronze → Silver → Gold
Data maturity

Lesson 2
Data Lake → Data Warehouse → Lakehouse
Data architecture

Lesson 3
ETL / ELT
How data gets there and gets transformed
```

By the end of this lesson, when someone says:

> "We're using an ELT pipeline."

you should understand exactly what they mean and why they might have chosen it.

---

# 1. The simplest definition

Both ETL and ELT describe processes for moving and transforming data.

The letters mean:

```text
E = Extract
T = Transform
L = Load
```

The difference is **the order**.

### ETL

```text
Extract
   ↓
Transform
   ↓
Load
```

### ELT

```text
Extract
   ↓
Load
   ↓
Transform
```

That tiny difference represents an important architectural decision.

---

# 2. ETL

## Extract → Transform → Load

Imagine we want data from JW Player.

We first:

### Extract

Get the data from JW Player.

```text
JW Player API
      ↓
Raw viewing data
```

Then:

### Transform

Before putting it into our final destination, we:

- remove duplicates
    
- fix dates
    
- standardize IDs
    
- remove invalid records
    
- calculate fields
    
- join datasets
    

Then:

### Load

We put the transformed data into the destination.

```text
JW Player
    ↓
EXTRACT
    ↓
Raw data
    ↓
TRANSFORM
    ↓
Clean data
    ↓
LOAD
    ↓
Data Warehouse
```

The important idea:

> **The transformation happens BEFORE the data reaches its destination.**

---

# 3. Why was ETL traditionally popular?

Historically, data warehouses were:

- expensive
    
- highly structured
    
- limited in storage
    
- optimized for specific analytical workloads
    

You didn't want to dump everything into them.

You wanted to clean the data first.

Think of it like moving into a small apartment.

You don't bring everything you've accumulated over 20 years.

You:

```text
Pack
↓
Sort
↓
Throw things away
↓
Move
```

That's ETL.

You decide what is useful **before loading it into the destination**.

---

# 4. ELT

Modern cloud platforms changed things.

Storage became much cheaper.

Compute became much more scalable.

So organizations started saying:

> Why don't we load the raw data first and transform it afterward?

That's:

## Extract → Load → Transform

Example:

```text
JW Player
    ↓
EXTRACT
    ↓
LOAD
    ↓
Data Lake / Warehouse
    ↓
TRANSFORM
    ↓
Cleaned data
```

The key difference:

> **We preserve the raw data in the destination before transforming it.**

---

# 5. Your media-company example

Imagine you receive **100 million JW Player events**.

With traditional ETL:

```text
JW Player
     ↓
Extract
     ↓
Transformation Server
     ↓
Clean / Aggregate
     ↓
Data Warehouse
```

You might only load the processed results.

Perhaps:

```text
Show A → 50,000 views
Show B → 25,000 views
```

But what happens six months later when someone asks:

> "Can we calculate average viewing duration differently?"

Potential problem.

Maybe the original detailed data wasn't preserved.

---

With ELT:

```text
JW Player
     ↓
Extract
     ↓
Data Lake
     ↓
RAW DATA STORED
     ↓
Transform
     ↓
Silver
     ↓
Transform
     ↓
Gold
```

Now you still have the original events.

You can go back.

Reprocess.

Create new metrics.

Investigate problems.

That's a major advantage.

---

# 6. Connecting this to Medallion Architecture

This is where everything we've learned starts connecting.

A modern architecture might look like:

```text
SOURCE SYSTEMS

GA4
JW Player
CRM
Finance
OTT
Apps

      │
      │ EXTRACT
      ▼

┌─────────────────┐
│     BRONZE      │
│                 │
│    RAW DATA     │
└────────┬────────┘
         │
         │ TRANSFORM
         ▼
┌─────────────────┐
│     SILVER      │
│                 │
│   CLEAN DATA    │
└────────┬────────┘
         │
         │ TRANSFORM
         ▼
┌─────────────────┐
│      GOLD       │
│                 │
│ BUSINESS READY  │
└────────┬────────┘
         │
         ▼

      Power BI
```

This architecture is conceptually closer to **ELT**.

Why?

Because we:

```text
Extract
↓
Load raw data
↓
Transform afterward
```

The transformations create increasingly refined datasets.

---

# 7. ETL vs ELT

Here's your executive-level comparison:

||ETL|ELT|
|---|---|---|
|Order|Extract → Transform → Load|Extract → Load → Transform|
|Raw data stored first|Not necessarily|Usually|
|Transformation location|Before destination|In/around destination platform|
|Flexibility|Lower|Higher|
|Reprocessing|Potentially harder|Easier if raw data retained|
|Traditional warehouses|Common|Less traditional|
|Modern cloud data platforms|Used|Very common|

Neither is automatically better.

Architecture depends on:

- data volume
    
- cost
    
- security
    
- privacy
    
- latency
    
- technology
    
- business requirements
    

---

# 8. A subtle but important point

People often use the word:

> **Pipeline**

A data pipeline is simply an automated process that moves and/or transforms data.

For example:

```text
JW Player API
     ↓
Pipeline
     ↓
Bronze
```

Another pipeline might do:

```text
Bronze
     ↓
Pipeline
     ↓
Silver
```

Another:

```text
Silver
     ↓
Pipeline
     ↓
Gold
```

So your entire data architecture might contain dozens or hundreds of pipelines.

Some run:

```text
Every 5 minutes
```

Some:

```text
Hourly
```

Some:

```text
Nightly
```

Some:

```text
Real-time
```

---

# 9. Batch vs Streaming

This is related, but it's a separate concept we'll study later.

For now, just understand:

### Batch

Data moves periodically.

```text
Every night at 2 AM:

Yesterday's JW Player data
        ↓
Pipeline
        ↓
Data Lake
```

### Streaming

Data continuously flows.

```text
User watches video
       ↓
Event
       ↓
Pipeline
       ↓
Data Platform
```

Think:

> **Batch = packages of data periodically.**

> **Streaming = continuous flow of events.**

This distinction can become important for a media company.

---

# 10. Where does Azure Synapse fit?

Remember Lesson 2:

Synapse is a **platform**.

A company could potentially use Synapse capabilities to orchestrate pipelines that:

```text
Extract
   ↓
Load
   ↓
Transform
```

For example:

```text
JW Player
    ↓
Synapse Pipeline
    ↓
Azure Data Lake
    ↓
Bronze
    ↓
Transformation
    ↓
Silver
    ↓
Transformation
    ↓
Gold
    ↓
Power BI
```

Again, your company's actual architecture might differ.

When you're back from vacation, that's something worth mapping.

---

# 11. CTO perspective

Imagine an architect says:

> "We're moving from ETL to ELT."

Don't just respond:

> "Okay."

You want to understand **why**.

You could ask:

> "What's driving the change?"

Then:

> "Are we preserving all raw source data?"

> "Where are transformations executed?"

> "What's the impact on compute costs?"

> "How are transformations tested?"

> "How do we handle schema changes from source systems?"

> "Can we reprocess historical data if business rules change?"

These questions reveal the architectural consequences.

---

# 12. Meeting scenario

Imagine your analyst says:

> "We need to change the way we calculate total viewing hours, but we'd have to reload everything."

Instead of immediately discussing implementation, you could ask:

> **"Do we still have the original raw viewing events in our Bronze layer?"**

If yes:

> **"Then can we reprocess the affected data through Silver and regenerate the Gold dataset using the new business rule?"**

Now you're thinking architecturally.

You understand:

```text
Bronze
Original truth
     ↓
Silver
Clean data
     ↓
Gold
Business interpretation
```

If the **business rule** changes, you may not need new source data.

You may need to **reprocess existing data**.

That's one reason retaining raw data can be valuable.

---

# 13. Another CTO-level issue: Garbage In, Garbage Out

ELT has a potential downside.

Because storing raw data is easy, organizations sometimes end up with:

```text
Data Lake
    ↓
More Data
    ↓
More Data
    ↓
More Data
    ↓
Nobody knows what anything means
```

This is sometimes jokingly called a:

> **Data Swamp**

A Data Lake without proper:

- governance
    
- metadata
    
- ownership
    
- documentation
    
- lineage
    
- quality controls
    

can become difficult to use.

This connects directly to your role in **Data Governance**.

The technology doesn't create trustworthy data.

Governance does.

---

# 14. Your executive communication exercise

Someone says in a meeting:

> "We should move everything to ELT because it's the modern approach."

Your response could be:

> **"Before we decide on ETL versus ELT, I'd like to understand the problem we're solving. If the goal is to preserve raw data and give us more flexibility to reprocess it as business requirements change, ELT may make sense. But we should also understand the impact on storage, compute, governance and data quality before changing the architecture."**

Notice the pattern again:

**Clarify the problem → acknowledge the option → identify tradeoffs → guide the decision.**

You don't need to be the data engineer.

You need to make sure the organization is solving the right problem.

---

# Your Cheat Sheet

Remember this:

> **ETL:** Clean it, then load it.

> **ELT:** Load it, then clean it.

> **Pipeline:** Automated movement/transformation of data.

> **Batch:** Move data periodically.

> **Streaming:** Move data continuously.

And the connection between our first three lessons:

```text
        SOURCE SYSTEMS
              │
              │
         Extract / Load
              │
              ▼
          🥉 BRONZE
           Raw Data
              │
          Transform
              │
              ▼
          🥈 SILVER
          Clean Data
              │
          Transform
              │
              ▼
          🥇 GOLD
       Business Data
              │
              ▼
           Power BI
```

That's the mental model I want you to keep.

## Add to Obsidian

Create:

```text
Knowledge
└── CTO Academy
    └── Data
        └── ETL vs ELT.md
```

At the bottom:

```markdown
## Related Concepts

- [[Medallion Architecture]]
- [[Data Lake]]
- [[Data Warehouse]]
- [[Lakehouse]]
- [[Azure Synapse]]
- [[Data Pipeline]]
- [[Batch Processing]]
- [[Stream Processing]]
- [[Data Governance]]
- [[Data Lineage]]
```

### Next: Lesson 4

I think we're ready to move **inside the Gold layer**.

**Fact Tables & Dimension Tables.**

You've encountered these concepts before when we were playing with Metabase and discussing grain/granularité, so Lesson 4 should connect something you already partially understand with the architecture we're building now:

**Source → ELT → Bronze → Silver → Gold → Fact/Dimensions → Power BI.**

That's when this whole data architecture should really start clicking together.