vault backup: 2026-07-23 07:56:59
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
**MD file:** `Business Glossary.md`
|
||||
**Estimated reading time:** 9–11 minutes
|
||||
|
||||
# Business Glossary
|
||||
|
||||
## Definition
|
||||
|
||||
A **Business Glossary** is a governed collection of business terms and their agreed definitions.
|
||||
|
||||
Its purpose is to make sure important words mean the same thing across the organization.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **A Business Glossary is the organization's dictionary.**
|
||||
|
||||
Each important term can include a definition, owner, status, exclusions, and related metrics.
|
||||
|
||||
## How It Works
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Term: Unique Viewer
|
||||
Definition: A distinct viewer identity with at least one valid content viewing event during the selected period.
|
||||
Owner: Digital Product
|
||||
Exclusions: Internal traffic, automated traffic, invalid playback events
|
||||
```
|
||||
|
||||
The technical implementation may live in [[Semantic Layer]], but the meaning is documented in the glossary.
|
||||
|
||||
## Example
|
||||
|
||||
Without a glossary:
|
||||
|
||||
```text
|
||||
Marketing: Subscriber = email newsletter subscriber
|
||||
Product: Subscriber = paid platform subscriber
|
||||
Finance: Subscriber = billable customer
|
||||
```
|
||||
|
||||
All can make sense in context. The problem is when a dashboard simply says "Subscribers."
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Data Owner]]
|
||||
↓
|
||||
Approves business meaning
|
||||
↓
|
||||
[[Business Glossary]]
|
||||
↓
|
||||
Documents meaning
|
||||
↓
|
||||
[[Semantic Layer]]
|
||||
↓
|
||||
Implements governed metrics
|
||||
```
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Useful governed terms could include:
|
||||
|
||||
- Session
|
||||
- Viewer
|
||||
- Unique Viewer
|
||||
- Audience
|
||||
- Video Start
|
||||
- Completion Rate
|
||||
- Watch Hour
|
||||
- Active User
|
||||
- Subscriber
|
||||
- Registered User
|
||||
- OTT
|
||||
- Connected TV
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Many reporting disagreements are vocabulary disagreements disguised as technical problems.
|
||||
|
||||
A CTO can often improve the discussion by asking:
|
||||
|
||||
> **Are we actually using the same definition?**
|
||||
|
||||
Start with terms tied to executive decisions, KPIs, financial impact, or repeated disputes.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Which terms repeatedly cause confusion?
|
||||
- Does each critical term have an agreed definition?
|
||||
- Who owns it?
|
||||
- Is it connected to [[Semantic Layer]]?
|
||||
- How are changes approved?
|
||||
- Can employees easily discover the glossary?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Two reports show 400,000 and 520,000 subscribers.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "Before comparing the numbers, let's confirm what each report means by Subscriber. If one includes free registered accounts and the other only paid accounts, both numbers may be correct but the labels are misleading."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- A Business Glossary is the organization's governed dictionary.
|
||||
- It documents agreed business definitions and ownership.
|
||||
- [[Semantic Layer]] implements definitions; the glossary documents meaning.
|
||||
- Many data disagreements are actually definition disagreements.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Semantic Layer]]
|
||||
- [[Data Governance]]
|
||||
- [[Data Owner]]
|
||||
- [[Data Steward]]
|
||||
- [[Metadata]]
|
||||
- [[Data Catalog]]
|
||||
- [[KPI]]
|
||||
- [[Metric]]
|
||||
@@ -0,0 +1,106 @@
|
||||
**MD file:** `Data Catalog.md`
|
||||
**Estimated reading time:** 10–12 minutes
|
||||
|
||||
# Data Catalog
|
||||
|
||||
## Definition
|
||||
|
||||
A **Data Catalog** is a searchable inventory of an organization's data assets and their [[Metadata]].
|
||||
|
||||
It helps people discover what data exists, what it means, where it comes from, who owns it, and whether it is trusted.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **A Data Catalog is Google for your organization's data.**
|
||||
|
||||
## How It Works
|
||||
|
||||
A Data Catalog may inventory:
|
||||
|
||||
- Databases
|
||||
- Tables
|
||||
- Columns
|
||||
- Files
|
||||
- Dashboards
|
||||
- Reports
|
||||
- Metrics
|
||||
- Models
|
||||
- Pipelines
|
||||
- APIs
|
||||
|
||||
An entry might show owner, status, description, grain, source, refresh, quality, and downstream use.
|
||||
|
||||
## Example
|
||||
|
||||
Search:
|
||||
|
||||
`Revenue`
|
||||
|
||||
Results:
|
||||
|
||||
```text
|
||||
Net Advertising Revenue [Certified]
|
||||
Subscription Revenue [Certified]
|
||||
Gross Advertising Revenue
|
||||
Revenue Forecast [Restricted]
|
||||
```
|
||||
|
||||
Each result includes an owner and business definition.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Metadata]]
|
||||
↓
|
||||
Collected and organized in
|
||||
↓
|
||||
[[Data Catalog]]
|
||||
↓
|
||||
Datasets + Metrics + Reports + Owners
|
||||
```
|
||||
|
||||
The catalog may surface [[Data Lineage]], [[Data Quality]], [[Business Glossary]], [[Data Owner]], and [[Data Steward]] information.
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
A new employee should be able to search `Video Completion Rate` and see its definition, owner, dataset, source system, refresh, and dashboards that use it.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Buying a catalog tool does not create governance. A catalog full of stale descriptions is only expensive documentation.
|
||||
|
||||
The hard part is the operating model: who maintains it, who certifies assets, what "trusted" means, and how deprecation works.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Can employees discover trusted data without knowing who created it?
|
||||
- How do we mark certified vs experimental datasets?
|
||||
- Who maintains catalog entries?
|
||||
- Does the catalog show [[Data Lineage]]?
|
||||
- Does it expose [[Data Quality]] status?
|
||||
- Are people actually using it?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Someone proposes an expensive enterprise Data Catalog because "every mature organization needs one."
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "Before selecting a platform, let's define the problem. Are we struggling with discoverability, ownership, duplicated datasets, lineage, business definitions, or all of these? Then we can determine the lightest solution that meets the need."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- A Data Catalog is a searchable inventory of data assets and metadata.
|
||||
- It supports discoverability, self-service, and governance.
|
||||
- A tool alone does not create governance.
|
||||
- Operating model and adoption matter more than the interface.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Metadata]]
|
||||
- [[Data Governance]]
|
||||
- [[Data Lineage]]
|
||||
- [[Data Quality]]
|
||||
- [[Business Glossary]]
|
||||
- [[Data Owner]]
|
||||
- [[Data Steward]]
|
||||
@@ -0,0 +1,128 @@
|
||||
**MD file:** `Data Contract.md`
|
||||
**Estimated reading time:** 11–13 minutes
|
||||
|
||||
# Data Contract
|
||||
|
||||
## Definition
|
||||
|
||||
A **Data Contract** is an explicit agreement between a data producer and data consumers about the structure, meaning, quality, and reliability of shared data.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **A Data Contract is an API contract for data.**
|
||||
|
||||
A producer should not unexpectedly change data that downstream consumers rely on.
|
||||
|
||||
## How It Works
|
||||
|
||||
A contract might specify:
|
||||
|
||||
```text
|
||||
Dataset: Viewing Events
|
||||
Owner: Streaming Platform
|
||||
|
||||
Required fields:
|
||||
event_id: string
|
||||
content_id: string
|
||||
event_time: timestamp
|
||||
watch_time_ms: integer
|
||||
platform: string
|
||||
|
||||
Freshness:
|
||||
Available within 15 minutes
|
||||
|
||||
Quality:
|
||||
event_id must be unique
|
||||
content_id NULL rate < 0.5%
|
||||
|
||||
Breaking changes:
|
||||
60-day notice required
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
Without a contract:
|
||||
|
||||
```text
|
||||
watch_time_ms renamed
|
||||
↓
|
||||
Pipeline breaks
|
||||
↓
|
||||
Gold stops updating
|
||||
↓
|
||||
Executive dashboard stale
|
||||
```
|
||||
|
||||
With a contract:
|
||||
|
||||
```text
|
||||
Proposed breaking change
|
||||
↓
|
||||
Consumers notified
|
||||
↓
|
||||
[[Impact Analysis]]
|
||||
↓
|
||||
Migration planned
|
||||
↓
|
||||
Consumers migrate
|
||||
↓
|
||||
Old field removed
|
||||
```
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
Contracts can define [[Metadata]], expected [[Data Quality]], ownership, and change rules.
|
||||
|
||||
[[Data Lineage]] shows who depends on the contract.
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
If analytics depends on external fields such as `content_id`, `watch_time`, `device`, and `event_time`, important assumptions should be explicit in vendor or internal integration agreements.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Data Contracts are about managing dependencies and accountability.
|
||||
|
||||
Ask:
|
||||
|
||||
> **What are downstream teams allowed to rely on, and how do producers communicate breaking changes?**
|
||||
|
||||
Use stronger contracts where data is critical, highly reused, cross-team, or vendor-controlled.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Who produces this data?
|
||||
- Who consumes it?
|
||||
- What fields and semantics are guaranteed?
|
||||
- What quality and freshness are expected?
|
||||
- What counts as a breaking change?
|
||||
- How much notice is required?
|
||||
- Can [[Data Lineage]] identify affected consumers?
|
||||
- Who approves contract changes?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** An internal application team wants to rename analytics fields during a release.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "Before changing the fields, let's identify downstream dependencies and treat this as a data-interface change. We need to agree on the new contract, provide a migration period, and verify affected metrics and reports before removing the old fields."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- A Data Contract defines expectations between producers and consumers.
|
||||
- It can cover schema, meaning, freshness, quality, ownership, and change management.
|
||||
- Contracts reduce unexpected downstream breakage.
|
||||
- [[Data Lineage]] helps identify affected consumers.
|
||||
- Treat important shared data interfaces with similar discipline to APIs.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Lineage]]
|
||||
- [[Data Quality]]
|
||||
- [[Metadata]]
|
||||
- [[Data Governance]]
|
||||
- [[Impact Analysis]]
|
||||
- [[Data Observability]]
|
||||
- [[Schema Evolution]]
|
||||
- [[API Contract]]
|
||||
@@ -0,0 +1,105 @@
|
||||
**MD file:** `Data Observability.md`
|
||||
**Estimated reading time:** 10–12 minutes
|
||||
|
||||
# Data Observability
|
||||
|
||||
## Definition
|
||||
|
||||
**Data Observability** is the ability to understand the health of data systems and quickly detect, diagnose, and resolve data problems.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **Monitoring tells you that a pipeline ran. Data Observability tells you whether the data produced by that pipeline still makes sense.**
|
||||
|
||||
## How It Works
|
||||
|
||||
Common signals include:
|
||||
|
||||
- Freshness
|
||||
- Volume
|
||||
- Schema changes
|
||||
- Distribution changes
|
||||
- Quality-rule violations
|
||||
- Downstream impact through lineage
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
NULL platform_id
|
||||
Normal: 0.2%
|
||||
Today: 46%
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
A pipeline reports success but processes only 3.2M events when the normal range is 8M–10M.
|
||||
|
||||
Observability should alert before executives consume the resulting dashboard.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Data Quality]]
|
||||
Rules / expectations
|
||||
+
|
||||
[[Data Lineage]]
|
||||
Dependencies
|
||||
+
|
||||
Operational telemetry
|
||||
↓
|
||||
[[Data Observability]]
|
||||
↓
|
||||
Detect + diagnose + alert
|
||||
```
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Useful signals could include:
|
||||
|
||||
- Viewing events received/hour
|
||||
- Missing content ID percentage
|
||||
- Unknown platform percentage
|
||||
- Time since last successful load
|
||||
- Source-to-Gold processing delay
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Ask:
|
||||
|
||||
> **How do we know our data is broken before a business user tells us?**
|
||||
|
||||
Avoid alerting on everything. Prioritize executive KPIs, revenue, compliance, and critical operational dashboards.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- How quickly do we detect stale or incomplete data?
|
||||
- Do we monitor volume and distribution, not just pipeline status?
|
||||
- Which critical datasets have automated checks?
|
||||
- Can alerts show downstream impact through [[Data Lineage]]?
|
||||
- Who owns incident response?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Executives repeatedly discover broken dashboards before the data team does.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "The recurring issue isn't only data quality; it's detection. We need observability on critical data products so freshness, volume, schema, and key quality conditions are checked automatically."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- Data Observability detects and diagnoses unhealthy data systems.
|
||||
- A successful pipeline does not guarantee healthy data.
|
||||
- Freshness, volume, schema, distributions, and quality are common signals.
|
||||
- [[Data Lineage]] helps identify downstream impact.
|
||||
- Prioritize observability based on business criticality.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Quality]]
|
||||
- [[Data Lineage]]
|
||||
- [[Data Governance]]
|
||||
- [[Metadata]]
|
||||
- [[Data Contract]]
|
||||
- [[Monitoring]]
|
||||
- [[Data Incident]]
|
||||
@@ -0,0 +1,104 @@
|
||||
**MD file:** `Data Owner.md`
|
||||
**Estimated reading time:** 8–10 minutes
|
||||
|
||||
# Data Owner
|
||||
|
||||
## Definition
|
||||
|
||||
A **Data Owner** is the person or business role accountable for a data domain, dataset, or important business definition.
|
||||
|
||||
The Data Owner has authority over meaning, acceptable use, access, quality expectations, retention, and important business rules.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **The Data Owner decides what the data means and what should happen with it.**
|
||||
|
||||
IT may host the system. Data Engineering may build the pipeline. An analyst may build the dashboard. None of them automatically owns the business meaning.
|
||||
|
||||
## How It Works
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Employee Data
|
||||
Business owner: HR
|
||||
IT: operates systems
|
||||
Data team: pipelines and reporting
|
||||
```
|
||||
|
||||
Ownership is accountability, not technical administration.
|
||||
|
||||
## Example
|
||||
|
||||
Marketing and Product disagree on `Active User`.
|
||||
|
||||
The data team should not silently choose the definition because it owns Power BI.
|
||||
|
||||
A Data Owner should approve the business definition, which can then flow into [[Business Glossary]] and [[Semantic Layer]].
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Data Governance]]
|
||||
↓
|
||||
[[Data Owner]]
|
||||
Accountability / decision authority
|
||||
↓
|
||||
[[Data Steward]]
|
||||
Day-to-day quality and maintenance
|
||||
```
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Potential ownership could look like:
|
||||
|
||||
- Advertising Revenue → Finance / Advertising business leader
|
||||
- Employee Data → HR
|
||||
- Content Metadata → Programming / Content
|
||||
- Streaming Audience KPI → agreed business owner
|
||||
|
||||
Your Data Governance team can facilitate and implement without owning every business concept.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
A common failure is confusing system ownership with data ownership.
|
||||
|
||||
A useful question is:
|
||||
|
||||
> **Who has the authority to make the final business decision about this data?**
|
||||
|
||||
If nobody can answer, ownership is unclear.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Who is accountable for this data?
|
||||
- Who approves its business definition?
|
||||
- Who approves access?
|
||||
- Who establishes acceptable quality?
|
||||
- Is the owner a business role or merely the person who built the system?
|
||||
- Is ownership documented?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Marketing and Product ask your data team to decide the definition of Active User.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "Data can quantify the options and explain the consequences, but this is ultimately a business-definition decision. We need an accountable owner who can approve the definition, after which we'll implement it consistently."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- A Data Owner is accountable for business meaning and appropriate management.
|
||||
- Ownership is not the same as building or hosting the system.
|
||||
- Business departments should normally own business meaning.
|
||||
- [[Data Governance]] should make ownership explicit.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Steward]]
|
||||
- [[Data Governance]]
|
||||
- [[Business Glossary]]
|
||||
- [[Metadata]]
|
||||
- [[Data Catalog]]
|
||||
- [[Semantic Layer]]
|
||||
- [[KPI]]
|
||||
@@ -0,0 +1,152 @@
|
||||
**MD file:** `Data Quality.md`
|
||||
**Estimated reading time:** 12–15 minutes
|
||||
|
||||
# Data Quality
|
||||
|
||||
## Definition
|
||||
|
||||
**Data Quality** describes whether data is fit for the business purpose for which it is being used.
|
||||
|
||||
High-quality data is not simply data that exists or data that successfully loaded into a database. It must be sufficiently accurate, complete, consistent, timely, valid, and unique for the decisions being made from it.
|
||||
|
||||
[[Data Governance]] establishes the expectations and accountability for quality. Data Quality measures whether the data actually meets those expectations.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **Data Governance defines what "good" means. Data Quality tells us whether today's data is actually good.**
|
||||
|
||||
A pipeline can run successfully and still produce poor-quality data.
|
||||
|
||||
```text
|
||||
Pipeline status: SUCCESS
|
||||
↓
|
||||
Rows loaded: 1,000,000
|
||||
↓
|
||||
Half the platform values are NULL
|
||||
↓
|
||||
Technically successful
|
||||
Business quality: BAD
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
Common dimensions of Data Quality include:
|
||||
|
||||
### Accuracy
|
||||
Does the data correctly represent reality?
|
||||
|
||||
### Completeness
|
||||
Is required data present?
|
||||
|
||||
### Consistency
|
||||
Is the same concept represented the same way across systems?
|
||||
|
||||
### Timeliness
|
||||
Is the data available when the business needs it?
|
||||
|
||||
### Validity
|
||||
Does the data follow expected rules?
|
||||
|
||||
### Uniqueness
|
||||
Are duplicate records creating false counts?
|
||||
|
||||
## Example
|
||||
|
||||
Suppose Power BI shows:
|
||||
|
||||
> Yesterday's watch hours dropped by 35%.
|
||||
|
||||
Before concluding that audience behavior changed, the team checks quality:
|
||||
|
||||
```text
|
||||
Source events received yesterday: 6.4M
|
||||
Normal daily range: 9M–10M
|
||||
```
|
||||
|
||||
The dashboard calculation may be correct. The real problem is incomplete source data.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
Source
|
||||
↓
|
||||
[[ETL vs ELT]]
|
||||
↓
|
||||
[[Medallion Architecture]]
|
||||
↓
|
||||
Bronze
|
||||
↓
|
||||
Silver ← quality checks often become critical here
|
||||
↓
|
||||
Gold
|
||||
↓
|
||||
[[Semantic Layer]]
|
||||
↓
|
||||
Power BI
|
||||
```
|
||||
|
||||
[[Data Lineage]] helps locate where a quality problem entered the chain.
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
For a media company, useful quality checks could include:
|
||||
|
||||
- Daily viewing-event volume
|
||||
- Percentage of missing content IDs
|
||||
- Duplicate event rate
|
||||
- Percentage of unknown platforms
|
||||
- Missing demographic percentage
|
||||
- Data refresh completion time
|
||||
- Difference between source-system totals and governed analytics totals
|
||||
|
||||
A high "Unknown" percentage is not automatically poor quality. The key question is whether it is expected due to consent, identity availability, or platform limitations, or caused by a collection problem.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Important datasets can have explicit expectations such as:
|
||||
|
||||
```text
|
||||
Daily refresh completed by 07:00
|
||||
NULL content_id < 0.5%
|
||||
Duplicate events < 0.1%
|
||||
Source-vs-ingested row variance < 2%
|
||||
```
|
||||
|
||||
The objective is not perfect data. The objective is quality appropriate to the business decision and risk.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- What quality dimensions matter for this dataset?
|
||||
- What thresholds define acceptable quality?
|
||||
- Who owns those thresholds?
|
||||
- Are quality checks automated?
|
||||
- Who gets alerted when a check fails?
|
||||
- Can we trace the issue through [[Data Lineage]]?
|
||||
- What business decisions are affected when quality fails?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** An executive says, "The dashboard must be wrong. Audience dropped 30% yesterday."
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "Before we interpret this as an audience change, let's validate the quality of the underlying data. I want to confirm source volume, pipeline completeness, and whether any upstream fields changed before we conclude that the business actually moved."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- Data Quality means data is fit for its intended business purpose.
|
||||
- Accuracy, completeness, consistency, timeliness, validity, and uniqueness are common quality dimensions.
|
||||
- A successful pipeline can still produce poor-quality data.
|
||||
- [[Data Governance]] defines expectations; Data Quality measures whether they are met.
|
||||
- [[Data Lineage]] helps identify where problems originated.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Governance]]
|
||||
- [[Data Lineage]]
|
||||
- [[Data Observability]]
|
||||
- [[Metadata]]
|
||||
- [[Data Contract]]
|
||||
- [[Medallion Architecture]]
|
||||
- [[Semantic Layer]]
|
||||
- [[KPI]]
|
||||
@@ -0,0 +1,91 @@
|
||||
**MD file:** `Data Steward.md`
|
||||
**Estimated reading time:** 8–10 minutes
|
||||
|
||||
# Data Steward
|
||||
|
||||
## Definition
|
||||
|
||||
A **Data Steward** is responsible for the day-to-day care, quality, definition, and proper use of data within a governed domain.
|
||||
|
||||
While the [[Data Owner]] is accountable and has decision authority, the steward helps make governance operational.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **Owner = accountable. Steward = keeps it working properly day to day.**
|
||||
|
||||
## How It Works
|
||||
|
||||
A Data Steward may:
|
||||
|
||||
- Review quality issues
|
||||
- Maintain definitions
|
||||
- Validate classifications
|
||||
- Coordinate corrections
|
||||
- Update metadata
|
||||
- Resolve routine questions
|
||||
- Escalate policy decisions to the Data Owner
|
||||
|
||||
## Example
|
||||
|
||||
If the organization defines Roku, Apple TV, Android TV, and Fire TV as `Connected TV`, the owner approves that standard.
|
||||
|
||||
The steward helps detect inconsistent classifications and ensures future records follow the standard.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Data Governance]]
|
||||
↓
|
||||
[[Data Owner]]
|
||||
↓
|
||||
[[Data Steward]]
|
||||
↓
|
||||
[[Data Quality]] + [[Metadata]] + standards
|
||||
```
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
You may not need formal "Data Steward" job titles. Stewardship can be assigned to existing roles such as analysts or operations staff.
|
||||
|
||||
The key is that someone is clearly responsible for routine health and consistency.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Governance structures can look great on PowerPoint while nobody actually performs the work.
|
||||
|
||||
Ask:
|
||||
|
||||
> **Who notices when this data becomes wrong, and what are they expected to do?**
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Who performs day-to-day stewardship?
|
||||
- What quality issues do they monitor?
|
||||
- What can they decide themselves?
|
||||
- What must be escalated to the [[Data Owner]]?
|
||||
- Is stewardship part of their real workload?
|
||||
- Who updates [[Metadata]] and [[Business Glossary]] entries?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Stewardship is assigned on paper, but data classifications remain inconsistent.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "We have stewardship assigned, but the recurring issue suggests the operational responsibility isn't clear. Let's define what the steward monitors, what triggers action, and when the owner needs to be involved."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- Data Stewards perform day-to-day governance.
|
||||
- Owners are accountable; stewards operationalize standards.
|
||||
- Stewardship does not require a dedicated job title.
|
||||
- Governance should be lightweight enough to actually operate.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Owner]]
|
||||
- [[Data Governance]]
|
||||
- [[Data Quality]]
|
||||
- [[Metadata]]
|
||||
- [[Business Glossary]]
|
||||
- [[Data Catalog]]
|
||||
@@ -0,0 +1,127 @@
|
||||
**MD file:** `KPI.md`
|
||||
**Estimated reading time:** 11–13 minutes
|
||||
|
||||
# KPI
|
||||
|
||||
## Definition
|
||||
|
||||
A **KPI**, or **Key Performance Indicator**, is a metric selected because it indicates progress toward an important business objective.
|
||||
|
||||
Every KPI is a metric. Not every metric is a KPI.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **Metric = something we can measure. KPI = a measurement important enough to tell us whether we are succeeding.**
|
||||
|
||||
## How It Works
|
||||
|
||||
A strong KPI connects:
|
||||
|
||||
```text
|
||||
Business Objective
|
||||
↓
|
||||
Desired Outcome
|
||||
↓
|
||||
KPI
|
||||
↓
|
||||
Target
|
||||
↓
|
||||
Actions / Decisions
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
If the strategic objective is streaming growth, executive KPIs might include:
|
||||
|
||||
- Monthly Active Viewers
|
||||
- Total Watch Hours
|
||||
- Viewer Retention
|
||||
- Streaming Revenue
|
||||
|
||||
Other metrics can still matter operationally without being KPIs.
|
||||
|
||||
## Leading vs Lagging Indicators
|
||||
|
||||
**Lagging indicators** measure outcomes after they happen, such as revenue or churn.
|
||||
|
||||
**Leading indicators** may provide earlier signals, such as weekly engagement or trial activation.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
```text
|
||||
[[Fact Table]]
|
||||
Raw measures
|
||||
↓
|
||||
[[Semantic Layer]]
|
||||
Governed metrics
|
||||
↓
|
||||
[[Business Glossary]]
|
||||
Agreed meaning
|
||||
↓
|
||||
KPI
|
||||
↓
|
||||
Power BI
|
||||
↓
|
||||
Management decision
|
||||
```
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Potential KPIs could include:
|
||||
|
||||
- Monthly Active Viewers
|
||||
- Total Watch Hours
|
||||
- Viewer Retention
|
||||
- Video Start Success Rate
|
||||
- Advertising Revenue
|
||||
- WCAG Compliance Rate
|
||||
- SLA Compliance
|
||||
|
||||
If everything is "key," nothing is key.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
Bad executive dashboards display whatever is easy to measure.
|
||||
|
||||
Good dashboards display what is important to manage.
|
||||
|
||||
A CTO should challenge output metrics that do not demonstrate business value.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- What business objective does this KPI represent?
|
||||
- Why is it "key"?
|
||||
- Who owns it?
|
||||
- What is the target?
|
||||
- What decision changes when this KPI changes?
|
||||
- Is it leading or lagging?
|
||||
- Can we trust the underlying [[Data Quality]]?
|
||||
- Are we measuring output or outcome?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** A department proposes 18 more metrics for the executive dashboard.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "These may all be useful metrics, but which of them directly indicate whether we're achieving our strategic objectives, and what decisions would leadership make differently based on them?"
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- Every KPI is a metric, but not every metric is a KPI.
|
||||
- KPIs should connect directly to business objectives.
|
||||
- Leading indicators provide earlier signals; lagging indicators measure outcomes.
|
||||
- Executive dashboards should focus attention.
|
||||
- CTOs should distinguish output from outcome.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Metric]]
|
||||
- [[Measure]]
|
||||
- [[Semantic Layer]]
|
||||
- [[Business Glossary]]
|
||||
- [[Data Governance]]
|
||||
- [[Data Quality]]
|
||||
- [[Data Lineage]]
|
||||
- [[Leading Indicator]]
|
||||
- [[Lagging Indicator]]
|
||||
@@ -0,0 +1,103 @@
|
||||
**MD file:** `Master Data Management.md`
|
||||
**Estimated reading time:** 11–13 minutes
|
||||
|
||||
# Master Data Management
|
||||
|
||||
## Definition
|
||||
|
||||
**Master Data Management (MDM)** is the discipline of creating and maintaining consistent, authoritative representations of important business entities across systems.
|
||||
|
||||
Common master-data domains include Customer, Product, Employee, Supplier, Location, and Content.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **MDM answers: "What is the trusted identity of this thing across the company?"**
|
||||
|
||||
## How It Works
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
CMS: Show ID = 784
|
||||
JW Player: Media ID = ABC91
|
||||
Advertising: Content ID = 55129
|
||||
Analytics: Content Key = 9031
|
||||
```
|
||||
|
||||
These may all refer to the same program.
|
||||
|
||||
MDM establishes mappings and an authoritative representation.
|
||||
|
||||
## Example
|
||||
|
||||
```text
|
||||
CMS
|
||||
"The Great Show - S1E1"
|
||||
|
||||
Video Platform
|
||||
"great_show_ep01"
|
||||
|
||||
Programming
|
||||
Program ID 81817
|
||||
|
||||
Advertising
|
||||
Asset 500992
|
||||
```
|
||||
|
||||
A governed master-content entity can map all of them to one canonical identity.
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
MDM is related to [[Dimension Table]] and [[Data Governance]], but it is broader than analytics.
|
||||
|
||||
Master Data can serve operational and analytical systems.
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Content is a likely MDM challenge in a media organization because the same content may have identifiers in CMS, video, OTT, linear scheduling, advertising, and analytics systems.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
MDM initiatives can become enormous and expensive.
|
||||
|
||||
Start with:
|
||||
|
||||
> **Which business entity is causing enough inconsistency that centralized management would create meaningful value?**
|
||||
|
||||
Do not start with purchasing an MDM platform.
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Which entities exist in multiple systems?
|
||||
- Which system is authoritative for each attribute?
|
||||
- How are duplicates resolved?
|
||||
- How are identifiers mapped?
|
||||
- Who is the [[Data Owner]]?
|
||||
- Do we need full MDM tooling or a simpler governed approach?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** Your team spends significant time manually mapping content IDs across systems.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "We clearly have a master-data problem around content identity. Let's define which system owns each attribute and the canonical identity we need before deciding what technology is required."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- MDM manages authoritative identities across systems.
|
||||
- It addresses inconsistency and duplication.
|
||||
- Master Data is broader than analytical [[Dimension Table|Dimension Tables]].
|
||||
- [[Data Governance]] determines ownership and authority.
|
||||
- Solve the entity problem before choosing the platform.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Governance]]
|
||||
- [[Data Owner]]
|
||||
- [[Dimension Table]]
|
||||
- [[Natural Key]]
|
||||
- [[Surrogate Key]]
|
||||
- [[Metadata]]
|
||||
- [[Data Quality]]
|
||||
- [[Entity Resolution]]
|
||||
@@ -0,0 +1,123 @@
|
||||
**MD file:** `Metadata.md`
|
||||
**Estimated reading time:** 9–11 minutes
|
||||
|
||||
# Metadata
|
||||
|
||||
## Definition
|
||||
|
||||
**Metadata** is data that describes other data.
|
||||
|
||||
It provides context needed to understand, discover, govern, and operate datasets.
|
||||
|
||||
## Simple Mental Model
|
||||
|
||||
> **Data is the book. Metadata is the title, author, table of contents, publication date, and description on the back.**
|
||||
|
||||
## How It Works
|
||||
|
||||
Metadata can be grouped into several categories.
|
||||
|
||||
### Technical Metadata
|
||||
Table names, columns, data types, keys, schemas.
|
||||
|
||||
### Business Metadata
|
||||
Business definitions, owners, descriptions, certified use.
|
||||
|
||||
### Operational Metadata
|
||||
Refresh time, pipeline duration, row counts, success/failure.
|
||||
|
||||
### Governance Metadata
|
||||
Owner, steward, classification, retention, access rules.
|
||||
|
||||
## Example
|
||||
|
||||
Without metadata:
|
||||
|
||||
```text
|
||||
tbl_vw_agg_final_v2
|
||||
dt
|
||||
uid
|
||||
cnt
|
||||
sec
|
||||
src
|
||||
```
|
||||
|
||||
With metadata:
|
||||
|
||||
```text
|
||||
Name: Daily Viewing Aggregate
|
||||
Purpose: Daily content consumption reporting
|
||||
Source: JW Player viewing events
|
||||
Grain: One row per user/content/day
|
||||
Owner: Audience team
|
||||
Refresh: Daily 05:00
|
||||
sec: Total valid watch seconds
|
||||
```
|
||||
|
||||
## How It Fits Into the Bigger Picture
|
||||
|
||||
[[Data Lineage]] uses metadata to understand relationships.
|
||||
|
||||
[[Data Governance]] uses metadata to record ownership and classification.
|
||||
|
||||
[[Data Quality]] uses metadata to record rules and results.
|
||||
|
||||
[[Data Catalog]] makes metadata searchable.
|
||||
|
||||
## My Company / Real-World Context
|
||||
|
||||
Useful metadata could identify which audience dataset is certified, who owns it, how often it refreshes, and which source systems feed it.
|
||||
|
||||
## CTO Perspective
|
||||
|
||||
A mature organization moves knowledge from:
|
||||
|
||||
```text
|
||||
People's heads
|
||||
↓
|
||||
Documented metadata
|
||||
↓
|
||||
Searchable organizational knowledge
|
||||
```
|
||||
|
||||
The useful question is:
|
||||
|
||||
> **Is the metadata complete, current, discoverable, and actually used?**
|
||||
|
||||
### Questions to Ask
|
||||
|
||||
- Who owns this dataset?
|
||||
- What business purpose does it serve?
|
||||
- What is its [[Grain]]?
|
||||
- Where does it come from?
|
||||
- How often is it refreshed?
|
||||
- Is it certified for business use?
|
||||
- Can users discover this information without asking the original developer?
|
||||
|
||||
## Meeting Scenario
|
||||
|
||||
**Situation:** A new analyst finds four datasets that appear to contain audience information.
|
||||
|
||||
**Possible response:**
|
||||
|
||||
> "If the only way to know which dataset is trusted is tribal knowledge, that's a scalability problem. Let's make the owner, purpose, status, grain, source, and refresh information discoverable as metadata."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- Metadata is data about data.
|
||||
- Technical metadata explains structure.
|
||||
- Business metadata explains meaning.
|
||||
- Operational metadata explains processing and freshness.
|
||||
- Governance metadata explains ownership and controls.
|
||||
- [[Data Catalog]] makes metadata discoverable.
|
||||
|
||||
## Related Concepts
|
||||
|
||||
- [[Data Catalog]]
|
||||
- [[Data Governance]]
|
||||
- [[Data Lineage]]
|
||||
- [[Data Quality]]
|
||||
- [[Business Glossary]]
|
||||
- [[Data Owner]]
|
||||
- [[Data Steward]]
|
||||
- [[Grain]]
|
||||
@@ -0,0 +1,197 @@
|
||||
# CTO Academy - Data
|
||||
|
||||
This is the canonical reading order for the Data module.
|
||||
|
||||
## Progress
|
||||
|
||||
- [ ] Lesson 01 - [[Medallion Architecture]]
|
||||
- [ ] Lesson 02 - [[Data Lake vs Data Warehouse vs Lakehouse]]
|
||||
- [ ] Lesson 03 - [[ETL vs ELT]]
|
||||
- [ ] Lesson 04 - [[Fact Table]]
|
||||
- [ ] Lesson 05 - [[Dimension Table]]
|
||||
- [ ] Lesson 06 - [[Star Schema]]
|
||||
- [ ] Lesson 07 - [[Snowflake Schema]]
|
||||
- [ ] Lesson 08 - [[Semantic Layer]]
|
||||
- [ ] Lesson 09 - [[Data Lineage]]
|
||||
- [ ] Lesson 10 - [[Data Governance]]
|
||||
- [ ] Lesson 11 - [[Data Quality]]
|
||||
- [ ] Lesson 12 - [[Metadata]]
|
||||
- [ ] Lesson 13 - [[Data Catalog]]
|
||||
- [ ] Lesson 14 - [[Data Owner]]
|
||||
- [ ] Lesson 15 - [[Data Steward]]
|
||||
- [ ] Lesson 16 - [[Business Glossary]]
|
||||
- [ ] Lesson 17 - [[KPI]]
|
||||
- [ ] Lesson 18 - [[Master Data Management]]
|
||||
- [ ] Lesson 19 - [[Data Observability]]
|
||||
- [ ] Lesson 20 - [[Data Contract]]
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 - Data Architecture Foundations
|
||||
|
||||
### Lesson 01 - [[Medallion Architecture]]
|
||||
|
||||
Understand the Bronze → Silver → Gold model and how data matures through the platform.
|
||||
|
||||
### Lesson 02 - [[Data Lake vs Data Warehouse vs Lakehouse]]
|
||||
|
||||
Understand the major analytical storage architectures and when each is appropriate.
|
||||
|
||||
### Lesson 03 - [[ETL vs ELT]]
|
||||
|
||||
Understand where transformation happens in the data pipeline and why modern architectures increasingly use ELT.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 - Analytical Data Modeling
|
||||
|
||||
### Lesson 04 - [[Fact Table]]
|
||||
|
||||
Understand how business events and measurements are represented.
|
||||
|
||||
### Lesson 05 - [[Dimension Table]]
|
||||
|
||||
Understand how descriptive context is attached to business events.
|
||||
|
||||
### Lesson 06 - [[Star Schema]]
|
||||
|
||||
Understand the standard dimensional model used by analytical and BI systems.
|
||||
|
||||
### Lesson 07 - [[Snowflake Schema]]
|
||||
|
||||
Understand normalization of dimensions and the trade-off between simplicity and reduced duplication.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 - Business Meaning and Trust
|
||||
|
||||
### Lesson 08 - [[Semantic Layer]]
|
||||
|
||||
Understand how organizations create shared and reusable business definitions.
|
||||
|
||||
### Lesson 09 - [[Data Lineage]]
|
||||
|
||||
Understand where data comes from, how it changes, where it goes, and how to perform impact analysis.
|
||||
|
||||
### Lesson 10 - [[Data Governance]]
|
||||
|
||||
Understand the people, policies, ownership, standards, and processes that make organizational data trustworthy.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 - Operationalizing Data Governance
|
||||
|
||||
### Lesson 11 - [[Data Quality]]
|
||||
|
||||
Understand how accuracy, completeness, consistency, timeliness, validity, and uniqueness are measured.
|
||||
|
||||
### Lesson 12 - [[Metadata]]
|
||||
|
||||
Understand the information that describes datasets, fields, ownership, refresh schedules, and business meaning.
|
||||
|
||||
### Lesson 13 - [[Data Catalog]]
|
||||
|
||||
Understand how organizations make datasets, metrics, lineage, ownership, and metadata discoverable.
|
||||
|
||||
### Lesson 14 - [[Data Owner]]
|
||||
|
||||
Understand who is accountable for the business meaning and appropriate management of data.
|
||||
|
||||
### Lesson 15 - [[Data Steward]]
|
||||
|
||||
Understand who handles the day-to-day operational responsibilities of governed data.
|
||||
|
||||
### Lesson 16 - [[Business Glossary]]
|
||||
|
||||
Understand how an organization establishes shared definitions for important business concepts.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 - Managing Data as a Business Asset
|
||||
|
||||
### Lesson 17 - [[KPI]]
|
||||
|
||||
Understand the difference between measures, metrics, KPIs, outputs, outcomes, and leading/lagging indicators.
|
||||
|
||||
### Lesson 18 - [[Master Data Management]]
|
||||
|
||||
Understand how organizations establish authoritative identities for important entities across systems.
|
||||
|
||||
### Lesson 19 - [[Data Observability]]
|
||||
|
||||
Understand how data teams detect broken, stale, incomplete, or abnormal data before business users discover it.
|
||||
|
||||
### Lesson 20 - [[Data Contract]]
|
||||
|
||||
Understand how producers and consumers establish explicit expectations for shared data.
|
||||
|
||||
---
|
||||
|
||||
# Quizzes
|
||||
|
||||
- [[Quiz 01 - Data Foundations]]
|
||||
|
||||
---
|
||||
|
||||
# Mental Model
|
||||
|
||||
The technical path:
|
||||
|
||||
Source Systems
|
||||
↓
|
||||
[[ETL vs ELT]]
|
||||
↓
|
||||
[[Medallion Architecture]]
|
||||
↓
|
||||
Bronze → Silver → Gold
|
||||
↓
|
||||
[[Fact Table]] + [[Dimension Table]]
|
||||
↓
|
||||
[[Star Schema]] / [[Snowflake Schema]]
|
||||
↓
|
||||
[[Semantic Layer]]
|
||||
↓
|
||||
Power BI
|
||||
↓
|
||||
Business Decisions
|
||||
|
||||
The governance surrounding it:
|
||||
|
||||
[[Data Governance]]
|
||||
├── [[Data Owner]]
|
||||
├── [[Data Steward]]
|
||||
├── [[Business Glossary]]
|
||||
├── [[Metadata]]
|
||||
├── [[Data Catalog]]
|
||||
├── [[Data Quality]]
|
||||
├── [[Data Lineage]]
|
||||
└── [[Data Contract]]
|
||||
|
||||
The operational trust layer:
|
||||
|
||||
[[Data Quality]]
|
||||
+
|
||||
[[Data Lineage]]
|
||||
+
|
||||
[[Data Observability]]
|
||||
↓
|
||||
Can we trust the data right now?
|
||||
|
||||
---
|
||||
|
||||
# CTO Questions to Remember
|
||||
|
||||
When presented with data, ask:
|
||||
|
||||
1. What business question are we trying to answer?
|
||||
2. What does one row represent? → [[Fact Table]]
|
||||
3. What is the [[Grain]]?
|
||||
4. What dimensions provide context? → [[Dimension Table]]
|
||||
5. What is the business definition? → [[Business Glossary]]
|
||||
6. Who owns that definition? → [[Data Owner]]
|
||||
7. Is the definition implemented consistently? → [[Semantic Layer]]
|
||||
8. Where did the data come from? → [[Data Lineage]]
|
||||
9. Can we trust it? → [[Data Quality]]
|
||||
10. How would we know if it broke? → [[Data Observability]]
|
||||
11. What happens downstream if the source changes? → [[Data Lineage]]
|
||||
12. What are producers expected to guarantee? → [[Data Contract]]
|
||||
Reference in New Issue
Block a user