In brief: When one business action crosses services, reliability comes from a durable workflow rather than a distributed lock.
The problem
Payment, inventory, fulfilment and notification cannot share one database transaction. Requests time out while work continues, retries repeat side effects, and a naïve rollback cannot undo an email, shipment or external payment.
Why the simple answer is not enough
Two-phase commit is rarely available across SaaS dependencies and would couple their availability even if it were. A durable orchestrator accepts partial completion as a normal state.
Compensation is not always reversal. Refunding a payment after a captured fee may produce a different financial outcome and requires an explicit policy.
System flow
start workflow→State machine
persist each transition→Activities
idempotent service calls→Compensation
business recovery
Architecture decisions
Orchestrate important flows
A central workflow makes sequence, deadlines and support actions observable. Choreography remains useful for independent reactions.
Persist before acting
Record the intended transition before invoking an activity so a restart can decide what remains.
Define semantic idempotency
Use the order or operation identity, not a random retry request ID, so the same business action resolves to one result.
Going deeper
State machines are product models
Pending, confirmed, cancelled and requires-review must have precise customer-visible meaning. Support and finance need the same vocabulary as engineering.
Human work belongs in the design
Some exceptions cannot be safely automated. A manual task needs ownership, deadline, evidence and a resume action that remains idempotent.
Implementation path
- Model business states, deadlines and terminal outcomes before implementing activities.
- Give every activity a stable operation ID and idempotent result lookup.
- Use compensation as a business action with its own failure and audit trail.
Technical example: Durable order workflow
workflow.save(state = 'RESERVING', version = 4)
reservation = inventory.reserve(orderId, operationId)
workflow.save(state = 'CHARGING', reservationId)
charge = payments.charge(orderId, operationId)
workflow.save(state = 'CONFIRMED', chargeId)On replay, activity results are looked up by operation ID. The workflow must not recalculate nondeterministic values such as current time without recording them.
Failure modes to design for
- The payment succeeds but its response is lost.
- Compensation fails and needs a later retry or human decision.
- Two workflow instances race for the same business aggregate.
What to observe
| Signal | Why it matters |
|---|---|
| Workflow age by state | Finds stuck or overdue customer journeys. |
| Activity attempts and latency | Identifies unstable dependencies and retry amplification. |
| Compensation and manual-review rate | Shows both technical and business exception cost. |
How to validate the design
- Drop every activity response after success and prove the workflow converges.
- Restart the orchestrator at every persisted state.
- Force compensation to fail and verify alerting plus manual resolution.
Safe rollout plan
Mirror one existing business flow into a workflow without controlling side effects. Compare histories, then move one activity at a time. Keep a bounded fallback and reconciliation report until the new path survives dependency and restart drills.
Production checklist
- Workflow history is durable and replay deterministic.
- Timeouts express business deadlines, not arbitrary HTTP limits.
- Operators can inspect, retry, compensate or escalate safely.
Takeaway
A Saga succeeds by making partial progress visible and recoverable, not by pretending distribution is atomic.