Ordering, deduplication and replay in message-driven systems

In brief: A broker can preserve order inside a partition; the business must define which messages truly need that same order.

The problem

Global ordering destroys parallelism, retries introduce duplicates, and late events can overwrite newer state. Replaying months of events through current consumers may repeat external side effects or apply changed business meaning.

Why the simple answer is not enough

OrderCreated and OrderCancelled require order for one order, not across every customer. Partitioning by order ID preserves the useful guarantee and retains scale.

Event time, ingestion time and processing time differ. Business rules must name which time controls windows and conflict resolution.

System flow

Architecture decisions

Version the aggregate

A monotonic aggregate version lets consumers detect duplicates, gaps and stale events.

Park gaps without blocking everything

Retry or quarantine one aggregate while unrelated partition work continues where the broker permits.

Build replay-safe consumers

Separate pure state projection from email, payment or webhook activities, and gate those effects during replay.

Going deeper

Exactly once is an outcome

Broker transactions may narrow duplicates, but external databases and APIs still require idempotent business keys.

Schema and meaning both evolve

A syntactically compatible event can change meaning. Preserve semantic version notes and test old fixtures against new consumers.

Implementation path

1. Choose a partition key that matches the business ordering boundary.
2. Carry event ID, aggregate ID, aggregate version and occurred time.
3. Separate state rebuilding from external side effects during replay.

Technical example: Version-aware consumer

if inbox.contains(event.id): ACK
current = projection.version(event.aggregateId)
if event.version <= current: ACK
if event.version > current + 1: park(event)
transaction(apply(event), inbox.add(event.id))
ACK

A parked gap needs an alert and recovery policy; waiting forever is silent data loss.

Failure modes to design for

• A producer resets aggregate versions after data migration.
• A replay sends customer notifications again.
• One hot aggregate saturates a partition.

What to observe

How to validate the design

• Shuffle, duplicate and delay a representative event sequence.
• Replay production-like history into an isolated projection.
• Create a hot partition and verify other keys maintain service objectives.

Safe rollout plan

Add event metadata and observation before enforcing versions. Introduce inbox deduplication for one consumer, then a replay-only environment. Run a full historical rehearsal before making replay part of incident recovery.

Production checklist

• Ordering requirements are documented per event stream.
• Consumers tolerate duplicate and delayed delivery.
• Replay has a destination, rate limit and side-effect policy.

Takeaway

Message correctness comes from explicit business order and deterministic consumption, not broker promises alone.