In brief: A backup is not a recovery capability until the team has restored it, verified it and timed the process.
The problem
Replicas copy accidental deletion and corruption. Backups may be encrypted with unavailable keys, restore procedures omit queues or object storage, and a multi-region diagram says nothing about how DNS, identity and data become consistent.
Why the simple answer is not enough
Different capabilities need different objectives. Losing five minutes of analytics may be acceptable while losing a confirmed payment is not.
Regional failover is a consistency change. Before promoting another writer, the old writer must be fenced or split brain will create conflicts that recovery cannot easily reconcile.
System flow
backup and replication→Detect
declare incident→Recover
restore in dependency order→Verify
data and business checks
Architecture decisions
Tier data by business impact
Choose replication, log shipping, snapshots and retention from the required RPO rather than applying the most expensive pattern everywhere.
Recover dependencies in order
Identity, secrets, database, broker, object storage and applications form a graph. Automate and test that order.
Verify business invariants
Row counts are insufficient. Check balances, tenant isolation, event continuity and object references after restore.
Going deeper
RPO must include asynchronous systems
Database recovery alone can leave broker offsets, search indexes and object metadata inconsistent. Define which are rebuilt and which are authoritative.
Recovery needs organisational capacity
Named incident roles, communication templates, vendor contacts and decision authority are part of RTO. Technology cannot compensate for an approval nobody can obtain.
Implementation path
- Define RPO and RTO per business capability, not once for the whole platform.
- Map every stateful dependency, key and configuration needed to restore it.
- Automate restoration into an isolated environment and run scheduled recovery exercises.
Technical example: Recovery verification gate
restore(backupId, isolatedEnvironment)
assert pointInTime >= incidentTime - RPO
assert invariant('tenant_isolation')
assert invariant('ledger_balanced')
assert syntheticJourney('sign-in-to-publish')
approve traffic cutoverKeep the restored environment isolated until integrity and security checks pass; do not expose an uncertain copy to customers.
Failure modes to design for
- Backups succeed but restore fails because a format or key changed.
- DNS cutover completes while clients keep long-lived connections to the old region.
- Failback overwrites writes accepted during disaster mode.
What to observe
| Signal | Why it matters |
|---|---|
| Last verified recovery point | Measures real data protection, not only backup job success. |
| Restore stage duration | Shows whether the runbook can meet RTO. |
| Replication lag and write fencing | Indicate data-loss and split-brain risk. |
How to validate the design
- Restore a randomly selected backup without using the original environment.
- Lose a region while synthetic writes continue and measure accepted data.
- Perform failback and reconcile every write made in disaster mode.
Safe rollout plan
Automate restore for one critical data store, then add dependencies until a full business journey works. Run quarterly exercises with rotating leaders, record actual RPO and RTO, and turn every manual surprise into code or a tested checklist.
Production checklist
- Backups are immutable, encrypted and stored outside the primary failure boundary.
- Restore credentials and encryption keys survive the same disaster.
- Failover has a fenced single writer and a documented failback.
Takeaway
Resilience is demonstrated by timed recovery evidence and reconciliation, not by the existence of replicas.