In brief: An event-driven design is valuable only when a committed business change cannot silently lose its event.
The problem
Writing to the database and publishing to a broker are two separate operations. A crash between them leaves either missing events or events for data that never committed. Retries then introduce duplicates and out-of-order delivery.
Why the simple answer is not enough
A broker decouples time and deployment, but it does not create atomicity with the application database. Treating publish success as guaranteed after commit creates invisible data loss.
Events also become long-lived interfaces. Naming, versioning, retention and ownership deserve the same discipline as public APIs.
System flow
validate command→Database
business data plus outbox→Relay
publish with retry→Consumers
idempotent side effects
Architecture decisions
Commit intent before delivery
Store the event beside the business update. A relay can retry publication until the broker confirms it.
Design idempotent consumers
Record processed event IDs or use an inbox table in the same transaction as the consumer side effect.
Preserve per-aggregate order
Partition by aggregate ID and reject stale aggregate versions instead of assuming global ordering.
Going deeper
Events are facts, not commands in disguise
OrderCreated is owned by the order domain and describes what happened. SendEmailToCustomer couples the producer to another service's implementation. Consumers should decide their own reaction.
Compatibility needs governance
Use additive schema changes, stable meaning and contract tests. Keep sensitive fields out of events unless every retention and access path is approved.
Implementation path
- Define events as durable facts in past tense, not remote procedure calls.
- Write the aggregate change and outbox row in one database transaction.
- Publish asynchronously and make every consumer deduplicate by event ID.
Technical example: Atomic order creation and event publication
BEGIN
INSERT INTO orders (...) VALUES (...)
INSERT INTO outbox(id, aggregate_id, type, payload)
VALUES (:eventId, :orderId, 'OrderCreated.v1', :payload)
COMMIT
relay.publish_unpublished(limit = 500)Mark an outbox row published only after broker acknowledgement. A lease lets another relay recover abandoned batches.
Failure modes to design for
- The relay crashes after publish but before marking the row, so the event is delivered again.
- A poison event blocks a partition; quarantine it with context and a controlled replay path.
- A consumer deploys an incompatible schema and misinterprets retained events.
What to observe
| Signal | Why it matters |
|---|---|
| Oldest unpublished outbox age | Shows how long committed work has been invisible downstream. |
| Consumer lag by event type | Separates a slow business flow from general broker health. |
| Duplicate and replay rate | Reveals unstable relays or consumers that are not truly idempotent. |
How to validate the design
- Kill the API after commit and prove the relay still publishes.
- Kill the relay after broker acknowledgement and prove duplicate delivery has one business effect.
- Replay retained events against the next consumer version before deployment.
Safe rollout plan
Start with one non-critical side effect in compare-only mode. Reconcile synchronous and event-driven results, then move acknowledgement to after database commit. Add replay tooling and ownership dashboards before adding more consumers.
Production checklist
- Delivery is at least once; exactly-once business effects come from idempotency.
- Schema compatibility is checked before producers deploy.
- Event age and consumer lag are user-impact signals, not broker trivia.
Takeaway
Reliable event-driven systems make the transaction boundary, ownership and retry semantics explicit.