vault backup: 2026-07-23 07:56:59

10 Knowledge/CTO Academy/Data/Data Quality.md

@@ -0,0 +1,152 @@
+**MD file:** `Data Quality.md`  
+**Estimated reading time:** 12–15 minutes
+
+# Data Quality
+
+## Definition
+
+**Data Quality** describes whether data is fit for the business purpose for which it is being used.
+
+High-quality data is not simply data that exists or data that successfully loaded into a database. It must be sufficiently accurate, complete, consistent, timely, valid, and unique for the decisions being made from it.
+
+[[Data Governance]] establishes the expectations and accountability for quality. Data Quality measures whether the data actually meets those expectations.
+
+## Simple Mental Model
+
+> **Data Governance defines what "good" means. Data Quality tells us whether today's data is actually good.**
+
+A pipeline can run successfully and still produce poor-quality data.
+
+```text
+Pipeline status: SUCCESS
+        ↓
+Rows loaded: 1,000,000
+        ↓
+Half the platform values are NULL
+        ↓
+Technically successful
+Business quality: BAD
+```
+
+## How It Works
+
+Common dimensions of Data Quality include:
+
+### Accuracy
+Does the data correctly represent reality?
+
+### Completeness
+Is required data present?
+
+### Consistency
+Is the same concept represented the same way across systems?
+
+### Timeliness
+Is the data available when the business needs it?
+
+### Validity
+Does the data follow expected rules?
+
+### Uniqueness
+Are duplicate records creating false counts?
+
+## Example
+
+Suppose Power BI shows:
+
+> Yesterday's watch hours dropped by 35%.
+
+Before concluding that audience behavior changed, the team checks quality:
+
+```text
+Source events received yesterday: 6.4M
+Normal daily range: 9M–10M
+```
+
+The dashboard calculation may be correct. The real problem is incomplete source data.
+
+## How It Fits Into the Bigger Picture
+
+```text
+Source
+  ↓
+[[ETL vs ELT]]
+  ↓
+[[Medallion Architecture]]
+  ↓
+Bronze
+  ↓
+Silver ← quality checks often become critical here
+  ↓
+Gold
+  ↓
+[[Semantic Layer]]
+  ↓
+Power BI
+```
+
+[[Data Lineage]] helps locate where a quality problem entered the chain.
+
+## My Company / Real-World Context
+
+For a media company, useful quality checks could include:
+
+- Daily viewing-event volume
+- Percentage of missing content IDs
+- Duplicate event rate
+- Percentage of unknown platforms
+- Missing demographic percentage
+- Data refresh completion time
+- Difference between source-system totals and governed analytics totals
+
+A high "Unknown" percentage is not automatically poor quality. The key question is whether it is expected due to consent, identity availability, or platform limitations, or caused by a collection problem.
+
+## CTO Perspective
+
+Important datasets can have explicit expectations such as:
+
+```text
+Daily refresh completed by 07:00
+NULL content_id < 0.5%
+Duplicate events < 0.1%
+Source-vs-ingested row variance < 2%
+```
+
+The objective is not perfect data. The objective is quality appropriate to the business decision and risk.
+
+### Questions to Ask
+
+- What quality dimensions matter for this dataset?
+- What thresholds define acceptable quality?
+- Who owns those thresholds?
+- Are quality checks automated?
+- Who gets alerted when a check fails?
+- Can we trace the issue through [[Data Lineage]]?
+- What business decisions are affected when quality fails?
+
+## Meeting Scenario
+
+**Situation:** An executive says, "The dashboard must be wrong. Audience dropped 30% yesterday."
+
+**Possible response:**
+
+> "Before we interpret this as an audience change, let's validate the quality of the underlying data. I want to confirm source volume, pipeline completeness, and whether any upstream fields changed before we conclude that the business actually moved."
+
+## Key Takeaways
+
+- Data Quality means data is fit for its intended business purpose.
+- Accuracy, completeness, consistency, timeliness, validity, and uniqueness are common quality dimensions.
+- A successful pipeline can still produce poor-quality data.
+- [[Data Governance]] defines expectations; Data Quality measures whether they are met.
+- [[Data Lineage]] helps identify where problems originated.
+
+## Related Concepts
+
+- [[Data Governance]]
+- [[Data Lineage]]
+- [[Data Observability]]
+- [[Metadata]]
+- [[Data Contract]]
+- [[Medallion Architecture]]
+- [[Semantic Layer]]
+- [[KPI]]