4.1 KiB
Estimated reading time: 12–15 minutes
Definition
Data Quality describes whether data is fit for the business purpose for which it is being used.
High-quality data is not simply data that exists or data that successfully loaded into a database. It must be sufficiently accurate, complete, consistent, timely, valid, and unique for the decisions being made from it.
Data Governance establishes the expectations and accountability for quality. Data Quality measures whether the data actually meets those expectations.
Simple Mental Model
Data Governance defines what "good" means. Data Quality tells us whether today's data is actually good.
A pipeline can run successfully and still produce poor-quality data.
Pipeline status: SUCCESS
↓
Rows loaded: 1,000,000
↓
Half the platform values are NULL
↓
Technically successful
Business quality: BAD
How It Works
Common dimensions of Data Quality include:
Accuracy
Does the data correctly represent reality?
Completeness
Is required data present?
Consistency
Is the same concept represented the same way across systems?
Timeliness
Is the data available when the business needs it?
Validity
Does the data follow expected rules?
Uniqueness
Are duplicate records creating false counts?
Example
Suppose Power BI shows:
Yesterday's watch hours dropped by 35%.
Before concluding that audience behavior changed, the team checks quality:
Source events received yesterday: 6.4M
Normal daily range: 9M–10M
The dashboard calculation may be correct. The real problem is incomplete source data.
How It Fits Into the Bigger Picture
Source
↓
[[ETL vs ELT]]
↓
[[Medallion Architecture]]
↓
Bronze
↓
Silver ← quality checks often become critical here
↓
Gold
↓
[[Semantic Layer]]
↓
Power BI
Data Lineage helps locate where a quality problem entered the chain.
My Company / Real-World Context
For a media company, useful quality checks could include:
- Daily viewing-event volume
- Percentage of missing content IDs
- Duplicate event rate
- Percentage of unknown platforms
- Missing demographic percentage
- Data refresh completion time
- Difference between source-system totals and governed analytics totals
A high "Unknown" percentage is not automatically poor quality. The key question is whether it is expected due to consent, identity availability, or platform limitations, or caused by a collection problem.
CTO Perspective
Important datasets can have explicit expectations such as:
Daily refresh completed by 07:00
NULL content_id < 0.5%
Duplicate events < 0.1%
Source-vs-ingested row variance < 2%
The objective is not perfect data. The objective is quality appropriate to the business decision and risk.
Questions to Ask
- What quality dimensions matter for this dataset?
- What thresholds define acceptable quality?
- Who owns those thresholds?
- Are quality checks automated?
- Who gets alerted when a check fails?
- Can we trace the issue through Data Lineage?
- What business decisions are affected when quality fails?
Meeting Scenario
Situation: An executive says, "The dashboard must be wrong. Audience dropped 30% yesterday."
Possible response:
"Before we interpret this as an audience change, let's validate the quality of the underlying data. I want to confirm source volume, pipeline completeness, and whether any upstream fields changed before we conclude that the business actually moved."
Key Takeaways
- Data Quality means data is fit for its intended business purpose.
- Accuracy, completeness, consistency, timeliness, validity, and uniqueness are common quality dimensions.
- A successful pipeline can still produce poor-quality data.
- Data Governance defines expectations; Data Quality measures whether they are met.
- Data Lineage helps identify where problems originated.
Related Concepts
- Data Governance
- Data Lineage
- Data Observability
- Metadata
- Data Contract
- Medallion Architecture
- Semantic Layer
- KPI