amadou/SecondBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vault backup: 2026-07-18 23:06:13

Browse Source
This commit is contained in:
amadou
2026-07-18 23:06:13 +02:00
parent 0013290871
commit fda2206f43
24 changed files with 3116 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
10 Knowledge/CTO Academy/Data/Medallion Architecture.md
+405
View File
@@ -0,0 +1,405 @@
## Bronze, Silver & Gold
**Time: ~10 minutes**
### 1. Definition
**Medallion Architecture** is a way of organizing data into layers based on how processed and trustworthy it is.
The three common layers are:
```text
DATA SOURCES
↓
┌──────────────┐
│ BRONZE │ Raw data
└──────────────┘
↓
┌──────────────┐
│ SILVER │ Cleaned & standardized
└──────────────┘
↓
┌──────────────┐
│ GOLD │ Business-ready
└──────────────┘
↓
Power BI / Analytics / KPIs / Applications
```
The fundamental idea is:
> **Raw → Clean → Business-ready**
---
# 2. Why does this exist?
Imagine your company collects data from:
```text
GA4
JW Player
Cable/linear TV
Advertising
CRM
Finance
HR
CMS
Mobile apps
OTT apps
```
Every system has different formats, naming conventions and levels of quality.
You don't want your Power BI analysts independently cleaning and interpreting all of that data every time they create a report.
Instead, you progressively transform the data.
---
# 🥉 Bronze: "What did we receive?"
Bronze is the **raw data**.
Ideally, you preserve the data approximately as it arrived from the source.
Example from JW Player:
```text
user_id video_id play_time timestamp
82372 VID123 134 2026-07-18 12:34
82373 VID555 NULL 2026-07-18 12:35
82372 VID123 134 2026-07-18 12:34
```
Notice:
- NULL values
- duplicates
- potentially incorrect data
That's okay.
Bronze isn't necessarily supposed to be beautiful.
### Why keep it?
Because if your transformation is wrong, you can go back to the original data.
Think:
> **Bronze = source of historical truth about what we received.**
---
# 🥈 Silver: "What data can we trust technically?"
Now we clean it.
We might:
- Remove duplicates
- Standardize dates
- Handle NULL values
- Validate IDs
- Standardize country codes
- Join related datasets
- Correct data types
Our data becomes:
```text
user_id video_id watch_seconds date
82372 VID123 134 2026-07-18
```
Now it's consistent.
But here's an important distinction:
**Silver doesn't necessarily understand the business.**
It understands the **data**.
Think:
> **Silver = clean and standardized data.**
---
# 🥇 Gold: "What does the business need?"
This is where your original question comes in.
Gold data is designed for **business consumption**.
Instead of millions of video events, you might have:
```text
content viewers watch_hours
Show A 152,000 47,500
Show B 89,000 31,200
Show C 67,000 19,800
```
Or:
```text
month streaming_hours unique_viewers
January 1,500,000 320,000
February 1,700,000 350,000
March 1,850,000 380,000
```
Now Power BI can consume this directly.
Think:
> **Gold = data organized around business questions.**
---
# 3. Your TV-company example
Let's imagine your CEO asks:
> "How many people watched our content last month across all digital platforms?"
The architecture could look like this:
```text
JW Player ──────────┐
│
GA4 ────────────────┤
│
Mobile App ─────────┤
↓
BRONZE
Raw source data
↓
SILVER
Clean IDs / Remove duplicates
Standardize timestamps
Match content identifiers
↓
GOLD
Monthly Content Audience
↓
Power BI
↓
CEO
```
Your CEO should never have to understand Bronze.
Your Power BI developer ideally shouldn't repeatedly rebuild Silver transformations.
The Gold layer should provide trusted business-ready datasets.
---
# 4. The CTO perspective
Here's where **your role** becomes important.
You don't necessarily need to know how your data engineer writes the transformation.
You need to ask questions like:
> **Who owns the Gold layer?**
> **Who defines the business rules?**
> **How do we know these KPIs are correct?**
> **Can we trace a Gold KPI back to its original source?**
> **What happens when the source changes?**
> **Are Power BI analysts calculating KPIs independently, or consuming governed metrics?**
That last question is particularly relevant to your Data Governance responsibilities.
Imagine:
```text
Power BI Analyst A
"Active User = logged-in user"
Power BI Analyst B
"Active User = user with a session"
Marketing
"Active User = user who watched content"
```
Now you have three numbers.
A proper Gold layer might establish:
```text
dim_user
fact_viewing
fact_sessions
fact_subscriptions
```
and governed KPI definitions determine how those datasets are interpreted.
We'll get into **fact and dimension tables** in another lesson.
---
# 5. Something important: Medallion ≠ Star Schema
You will eventually hear both terms.
They're related, but they're answering different questions.
**Medallion architecture asks:**
> How processed is the data?
```text
Bronze → Silver → Gold
```
**Star schema asks:**
> How should analytical data be modeled?
```text
Dimension
|
Dimension - Fact - Dimension
|
Dimension
```
A **Gold-layer dataset could be modeled as a star schema**.
That's why these concepts often appear in the same conversation.
---
# 6. Common mistake
A company sometimes says:
> "We have a Gold layer."
But what they really have is:
```text
Raw Data
↓
A bunch of SQL
↓
Power BI
```
The word "Gold" doesn't automatically mean the data is trustworthy.
A CTO should care about:
- Ownership
- Documentation
- Data quality
- Business definitions
- Lineage
- Testing
We'll cover **Data Lineage** separately because it's another term you're very likely to encounter.
---
# 7. Meeting scenario
Imagine your data analyst says:
> "We should create this KPI in the Gold layer."
You could respond:
> **"That makes sense. Before we do that, I want to make sure we agree on the business definition of the KPI and who owns that definition. Once that's established, we can expose the governed metric through the Gold layer so Power BI and other consumers use the same calculation."**
That's a strong manager/CTO answer.
You're not pretending to know how they'll implement it.
You're focusing on **governance, consistency and business value**.
---
# 8. Your cheat sheet
Remember only this:
> 🥉 **Bronze:** What did we receive?
>
> 🥈 **Silver:** What data is technically clean and trustworthy?
>
> 🥇 **Gold:** What does the business need?
>
> **Medallion architecture:** Raw → Clean → Business-ready
>
> **Your CTO question:** Who owns it, who defines it, and can we trust it?
---
## Add this to Obsidian
I'd create:
```text
Knowledge
└── CTO Academy
└── Data
└── Medallion Architecture.md
```
At the bottom:
```markdown
## Related Concepts
- [[Data Warehouse]]
- [[Data Lake vs Data Warehouse vs Lakehouse]]
- [[Lakehouse]]
- [[ETL]]
- [[ELT]]
- [[Star Schema]]
- [[Fact Table]]
- [[Dimension Table]]
- [[Data Lineage]]
- [[Data Governance]]
- [[Semantic Layer]]
```
### One question for you before we consider today's lesson complete:
Your Power BI team currently receives data from multiple sources and creates reports.
**Where do you think your current environment sits today?**
Do you think you have something resembling Bronze → Silver → Gold, or are your analysts mostly taking source data and doing the cleaning/business logic directly inside Power BI?
My answer:
We definitely have a bronze, silver and gold. Power Bi is only used to create the reports so it consumes the gold layer. I do know there's azure synapse, data lake, data warehouses, so all of these should be for bronze and silver layers.