In brief: A data pipeline is correct when business meaning survives capture, transformation, replay and consumption.
The problem
Batch extraction misses short-lived states, CDC logs contain duplicates and schema changes break downstream jobs. Dashboards then disagree with production while personal data spreads into stores with unclear ownership and retention.
Why the simple answer is not enough
CDC exposes storage-level changes, not ready-made business facts. Consumers need keys, ordering and semantic context to reconstruct state correctly.
Exactly-once processing claims rarely include the warehouse, object store and BI tool together. Deterministic transforms and idempotent loads are the practical guarantee.
System flow
committed log→CDC
ordered change stream→Transform
versioned contracts→Warehouse
quality-gated models
Architecture decisions
Keep an immutable landing zone
Retain raw, access-controlled changes with offsets so transformations can be rebuilt after logic errors.
Version data contracts
Classify additive, breaking and semantic changes; require owners and compatibility checks before source deployment.
Publish quality gates
Do not mark a model ready when freshness or invariants fail. Serving stale last-known-good data can be safer than publishing corruption.
Going deeper
Late data needs an explicit policy
Event-time windows may close before data arrives. Decide when to update history, issue corrections or preserve previously reported numbers.
Lineage is an incident tool
Given a wrong dashboard value, operators must trace source rows, transform versions and affected downstream models quickly.
Implementation path
- Capture a stable source offset and initial snapshot watermark.
- Land immutable raw events before applying versioned transformations.
- Run freshness, completeness, uniqueness and business-invariant checks at each published model.
Technical example: Idempotent merge with source position
read changes after checkpoint.offset
deduplicate by sourcePartition + sourceOffset
transform using contractVersion
MERGE target ON businessKey
commit target and checkpoint atomically where possibleIf target and checkpoint cannot commit together, make the merge idempotent and replay from an earlier safe offset.
Failure modes to design for
- Snapshot and CDC overlap creates duplicate rows.
- A source column changes meaning without changing type.
- A delete event is dropped and personal data remains downstream.
What to observe
| Signal | Why it matters |
|---|---|
| End-to-end freshness | Measures delay from source commit to trusted model. |
| Rejected records and invariant failures | Makes corruption visible before consumers use it. |
| Offset progress and replay volume | Shows stalls and recovery cost. |
How to validate the design
- Run snapshot and live CDC concurrently and reconcile keys.
- Replay the same range twice and prove identical output.
- Introduce additive, breaking and semantic schema changes in contract tests.
Safe rollout plan
Begin with one dataset in parallel with the current batch path. Compare row-level samples and business totals, then switch one consumer. Preserve raw events and checkpoints until a full replay has been tested.
Production checklist
- Every dataset has owner, contract, lineage and retention.
- Consumers know whether records are inserts, updates, deletes or late corrections.
- Sensitive fields are minimised or tokenised before broad access.
Takeaway
Reliable analytics depends on reproducible lineage and data contracts, not just moving more bytes faster.