SecondBrain/10 Knowledge/CTO Academy/Data/Data Observability.md

Estimated reading time: 10–12 minutes

Definition

Data Observability is the ability to understand the health of data systems and quickly detect, diagnose, and resolve data problems.

Simple Mental Model

Monitoring tells you that a pipeline ran. Data Observability tells you whether the data produced by that pipeline still makes sense.

How It Works

Common signals include:

• Freshness
• Volume
• Schema changes
• Distribution changes
• Quality-rule violations
• Downstream impact through lineage

Example:

NULL platform_id
Normal: 0.2%
Today: 46%

Example

A pipeline reports success but processes only 3.2M events when the normal range is 8M–10M.

Observability should alert before executives consume the resulting dashboard.

How It Fits Into the Bigger Picture

[[Data Quality]]
Rules / expectations
       +
[[Data Lineage]]
Dependencies
       +
Operational telemetry
       ↓
[[Data Observability]]
       ↓
Detect + diagnose + alert

My Company / Real-World Context

Useful signals could include:

• Viewing events received/hour
• Missing content ID percentage
• Unknown platform percentage
• Time since last successful load
• Source-to-Gold processing delay

CTO Perspective

Ask:

How do we know our data is broken before a business user tells us?

Avoid alerting on everything. Prioritize executive KPIs, revenue, compliance, and critical operational dashboards.

Questions to Ask

• How quickly do we detect stale or incomplete data?
• Do we monitor volume and distribution, not just pipeline status?
• Which critical datasets have automated checks?
• Can alerts show downstream impact through Data Lineage?
• Who owns incident response?

Meeting Scenario

Situation: Executives repeatedly discover broken dashboards before the data team does.

Possible response:

"The recurring issue isn't only data quality; it's detection. We need observability on critical data products so freshness, volume, schema, and key quality conditions are checked automatically."

Key Takeaways

• Data Observability detects and diagnoses unhealthy data systems.
• A successful pipeline does not guarantee healthy data.
• Freshness, volume, schema, distributions, and quality are common signals.
• Data Lineage helps identify downstream impact.
• Prioritize observability based on business criticality.

Related Concepts

• Data Quality
• Data Lineage
• Data Governance
• Metadata
• Data Contract
• Monitoring
• Data Incident