150 lines
4.1 KiB
Markdown
150 lines
4.1 KiB
Markdown
**Estimated reading time:** 12–15 minutes
|
||
|
||
## Definition
|
||
|
||
**Data Quality** describes whether data is fit for the business purpose for which it is being used.
|
||
|
||
High-quality data is not simply data that exists or data that successfully loaded into a database. It must be sufficiently accurate, complete, consistent, timely, valid, and unique for the decisions being made from it.
|
||
|
||
[[Data Governance]] establishes the expectations and accountability for quality. Data Quality measures whether the data actually meets those expectations.
|
||
|
||
## Simple Mental Model
|
||
|
||
> **Data Governance defines what "good" means. Data Quality tells us whether today's data is actually good.**
|
||
|
||
A pipeline can run successfully and still produce poor-quality data.
|
||
|
||
```text
|
||
Pipeline status: SUCCESS
|
||
↓
|
||
Rows loaded: 1,000,000
|
||
↓
|
||
Half the platform values are NULL
|
||
↓
|
||
Technically successful
|
||
Business quality: BAD
|
||
```
|
||
|
||
## How It Works
|
||
|
||
Common dimensions of Data Quality include:
|
||
|
||
### Accuracy
|
||
Does the data correctly represent reality?
|
||
|
||
### Completeness
|
||
Is required data present?
|
||
|
||
### Consistency
|
||
Is the same concept represented the same way across systems?
|
||
|
||
### Timeliness
|
||
Is the data available when the business needs it?
|
||
|
||
### Validity
|
||
Does the data follow expected rules?
|
||
|
||
### Uniqueness
|
||
Are duplicate records creating false counts?
|
||
|
||
## Example
|
||
|
||
Suppose Power BI shows:
|
||
|
||
> Yesterday's watch hours dropped by 35%.
|
||
|
||
Before concluding that audience behavior changed, the team checks quality:
|
||
|
||
```text
|
||
Source events received yesterday: 6.4M
|
||
Normal daily range: 9M–10M
|
||
```
|
||
|
||
The dashboard calculation may be correct. The real problem is incomplete source data.
|
||
|
||
## How It Fits Into the Bigger Picture
|
||
|
||
```text
|
||
Source
|
||
↓
|
||
[[ETL vs ELT]]
|
||
↓
|
||
[[Medallion Architecture]]
|
||
↓
|
||
Bronze
|
||
↓
|
||
Silver ← quality checks often become critical here
|
||
↓
|
||
Gold
|
||
↓
|
||
[[Semantic Layer]]
|
||
↓
|
||
Power BI
|
||
```
|
||
|
||
[[Data Lineage]] helps locate where a quality problem entered the chain.
|
||
|
||
## My Company / Real-World Context
|
||
|
||
For a media company, useful quality checks could include:
|
||
|
||
- Daily viewing-event volume
|
||
- Percentage of missing content IDs
|
||
- Duplicate event rate
|
||
- Percentage of unknown platforms
|
||
- Missing demographic percentage
|
||
- Data refresh completion time
|
||
- Difference between source-system totals and governed analytics totals
|
||
|
||
A high "Unknown" percentage is not automatically poor quality. The key question is whether it is expected due to consent, identity availability, or platform limitations, or caused by a collection problem.
|
||
|
||
## CTO Perspective
|
||
|
||
Important datasets can have explicit expectations such as:
|
||
|
||
```text
|
||
Daily refresh completed by 07:00
|
||
NULL content_id < 0.5%
|
||
Duplicate events < 0.1%
|
||
Source-vs-ingested row variance < 2%
|
||
```
|
||
|
||
The objective is not perfect data. The objective is quality appropriate to the business decision and risk.
|
||
|
||
### Questions to Ask
|
||
|
||
- What quality dimensions matter for this dataset?
|
||
- What thresholds define acceptable quality?
|
||
- Who owns those thresholds?
|
||
- Are quality checks automated?
|
||
- Who gets alerted when a check fails?
|
||
- Can we trace the issue through [[Data Lineage]]?
|
||
- What business decisions are affected when quality fails?
|
||
|
||
## Meeting Scenario
|
||
|
||
**Situation:** An executive says, "The dashboard must be wrong. Audience dropped 30% yesterday."
|
||
|
||
**Possible response:**
|
||
|
||
> "Before we interpret this as an audience change, let's validate the quality of the underlying data. I want to confirm source volume, pipeline completeness, and whether any upstream fields changed before we conclude that the business actually moved."
|
||
|
||
## Key Takeaways
|
||
|
||
- Data Quality means data is fit for its intended business purpose.
|
||
- Accuracy, completeness, consistency, timeliness, validity, and uniqueness are common quality dimensions.
|
||
- A successful pipeline can still produce poor-quality data.
|
||
- [[Data Governance]] defines expectations; Data Quality measures whether they are met.
|
||
- [[Data Lineage]] helps identify where problems originated.
|
||
|
||
## Related Concepts
|
||
|
||
- [[Data Governance]]
|
||
- [[Data Lineage]]
|
||
- [[Data Observability]]
|
||
- [[Metadata]]
|
||
- [[Data Contract]]
|
||
- [[Medallion Architecture]]
|
||
- [[Semantic Layer]]
|
||
- [[KPI]]
|