106 lines
2.6 KiB
Markdown
106 lines
2.6 KiB
Markdown
**MD file:** `Data Observability.md`
|
||
**Estimated reading time:** 10–12 minutes
|
||
|
||
# Data Observability
|
||
|
||
## Definition
|
||
|
||
**Data Observability** is the ability to understand the health of data systems and quickly detect, diagnose, and resolve data problems.
|
||
|
||
## Simple Mental Model
|
||
|
||
> **Monitoring tells you that a pipeline ran. Data Observability tells you whether the data produced by that pipeline still makes sense.**
|
||
|
||
## How It Works
|
||
|
||
Common signals include:
|
||
|
||
- Freshness
|
||
- Volume
|
||
- Schema changes
|
||
- Distribution changes
|
||
- Quality-rule violations
|
||
- Downstream impact through lineage
|
||
|
||
Example:
|
||
|
||
```text
|
||
NULL platform_id
|
||
Normal: 0.2%
|
||
Today: 46%
|
||
```
|
||
|
||
## Example
|
||
|
||
A pipeline reports success but processes only 3.2M events when the normal range is 8M–10M.
|
||
|
||
Observability should alert before executives consume the resulting dashboard.
|
||
|
||
## How It Fits Into the Bigger Picture
|
||
|
||
```text
|
||
[[Data Quality]]
|
||
Rules / expectations
|
||
+
|
||
[[Data Lineage]]
|
||
Dependencies
|
||
+
|
||
Operational telemetry
|
||
↓
|
||
[[Data Observability]]
|
||
↓
|
||
Detect + diagnose + alert
|
||
```
|
||
|
||
## My Company / Real-World Context
|
||
|
||
Useful signals could include:
|
||
|
||
- Viewing events received/hour
|
||
- Missing content ID percentage
|
||
- Unknown platform percentage
|
||
- Time since last successful load
|
||
- Source-to-Gold processing delay
|
||
|
||
## CTO Perspective
|
||
|
||
Ask:
|
||
|
||
> **How do we know our data is broken before a business user tells us?**
|
||
|
||
Avoid alerting on everything. Prioritize executive KPIs, revenue, compliance, and critical operational dashboards.
|
||
|
||
### Questions to Ask
|
||
|
||
- How quickly do we detect stale or incomplete data?
|
||
- Do we monitor volume and distribution, not just pipeline status?
|
||
- Which critical datasets have automated checks?
|
||
- Can alerts show downstream impact through [[Data Lineage]]?
|
||
- Who owns incident response?
|
||
|
||
## Meeting Scenario
|
||
|
||
**Situation:** Executives repeatedly discover broken dashboards before the data team does.
|
||
|
||
**Possible response:**
|
||
|
||
> "The recurring issue isn't only data quality; it's detection. We need observability on critical data products so freshness, volume, schema, and key quality conditions are checked automatically."
|
||
|
||
## Key Takeaways
|
||
|
||
- Data Observability detects and diagnoses unhealthy data systems.
|
||
- A successful pipeline does not guarantee healthy data.
|
||
- Freshness, volume, schema, distributions, and quality are common signals.
|
||
- [[Data Lineage]] helps identify downstream impact.
|
||
- Prioritize observability based on business criticality.
|
||
|
||
## Related Concepts
|
||
|
||
- [[Data Quality]]
|
||
- [[Data Lineage]]
|
||
- [[Data Governance]]
|
||
- [[Metadata]]
|
||
- [[Data Contract]]
|
||
- [[Monitoring]]
|
||
- [[Data Incident]]
|