2.5 KiB
Estimated reading time: 10–12 minutes
Definition
Data Observability is the ability to understand the health of data systems and quickly detect, diagnose, and resolve data problems.
Simple Mental Model
Monitoring tells you that a pipeline ran. Data Observability tells you whether the data produced by that pipeline still makes sense.
How It Works
Common signals include:
- Freshness
- Volume
- Schema changes
- Distribution changes
- Quality-rule violations
- Downstream impact through lineage
Example:
NULL platform_id
Normal: 0.2%
Today: 46%
Example
A pipeline reports success but processes only 3.2M events when the normal range is 8M–10M.
Observability should alert before executives consume the resulting dashboard.
How It Fits Into the Bigger Picture
[[Data Quality]]
Rules / expectations
+
[[Data Lineage]]
Dependencies
+
Operational telemetry
↓
[[Data Observability]]
↓
Detect + diagnose + alert
My Company / Real-World Context
Useful signals could include:
- Viewing events received/hour
- Missing content ID percentage
- Unknown platform percentage
- Time since last successful load
- Source-to-Gold processing delay
CTO Perspective
Ask:
How do we know our data is broken before a business user tells us?
Avoid alerting on everything. Prioritize executive KPIs, revenue, compliance, and critical operational dashboards.
Questions to Ask
- How quickly do we detect stale or incomplete data?
- Do we monitor volume and distribution, not just pipeline status?
- Which critical datasets have automated checks?
- Can alerts show downstream impact through Data Lineage?
- Who owns incident response?
Meeting Scenario
Situation: Executives repeatedly discover broken dashboards before the data team does.
Possible response:
"The recurring issue isn't only data quality; it's detection. We need observability on critical data products so freshness, volume, schema, and key quality conditions are checked automatically."
Key Takeaways
- Data Observability detects and diagnoses unhealthy data systems.
- A successful pipeline does not guarantee healthy data.
- Freshness, volume, schema, distributions, and quality are common signals.
- Data Lineage helps identify downstream impact.
- Prioritize observability based on business criticality.
Related Concepts
- Data Quality
- Data Lineage
- Data Governance
- Metadata
- Data Contract
- Monitoring
- Data Incident