Scale and deploy without downtime

In brief: Most downtime happens during change, not because the fleet is permanently too small.

The problem

A new instance can start receiving traffic before its dependencies are ready. An old instance can be stopped while requests are still in flight. Average CPU hides both failures.

Why the simple answer is not enough

High availability is not just a replica count. During a rolling update, new instances need time to load configuration, establish pools, and warm caches. At the same time, old instances need time to stop receiving traffic and finish their requests.

A deployment is safe only when the orchestrator, the load balancer, and the application agree on lifecycle state. Liveness answers whether a process should be restarted. Readiness answers whether it should receive new traffic. Mixing the two creates restart loops and lost capacity.

System flow

Architecture decisions

Scale before saturation

Track p95/p99 latency, active concurrency, queue age, and dependency saturation. Add capacity early enough to cover image pull, process startup, readiness delay, and cache warm-up.

Make readiness meaningful but local

Check the configuration and the critical local dependencies you need to serve traffic. Do not tie readiness to every downstream service, or a single remote outage can remove replicas that are otherwise perfectly usable.

Drain before termination

On SIGTERM, mark the instance unready and wait for the load balancer to catch up. Reject new background work, finish in-flight requests, and then close database and message connections within a bounded grace period.

Going deeper

Probes encode lifecycle, not general health

A startup probe protects slow initialisation from liveness restarts. Liveness should answer whether the process can still make progress, and it is usually local: a database outage should not restart every application replica at once. Readiness is allowed to change as an instance drains or loses a genuinely mandatory dependency. Where you can, keep probe handlers independent of overloaded request pools, and measure their latency. A probe that always returns success is useless, but one that recursively checks the entire dependency graph can amplify an outage by removing all serving capacity.

Capacity planning must account for change

Normal traffic plus a rolling update needs surge headroom. Calculate the minimum healthy replicas from peak concurrency, safe concurrency per instance, and the loss of one failure domain. Then choose maxUnavailable and maxSurge so that starting and draining instances never push ready capacity below that minimum. Autoscaling needs stabilisation windows so it does not react to the temporary metrics of the rollout itself. For queue consumers, coordinate termination with leases and visibility timeouts. For WebSocket or streaming traffic, define reconnect behaviour, because the usual HTTP drain assumptions no longer hold.

Implementation path

1. Scale on saturation, p99 latency, and queue age rather than a single average metric.
2. Keep liveness simple. Use readiness to decide whether an instance may receive traffic.
3. On shutdown, stop new traffic, finish in-flight work, and only then close dependencies.

Technical example: Application shutdown state machine

RUNNING  --SIGTERM-->  DRAINING
DRAINING: readiness = false
DRAINING: stop claiming queue jobs
DRAINING: wait loadBalancerPropagation
DRAINING: await inFlightRequests == 0
DRAINING: close database and broker pools
DRAINING  --deadline-->  TERMINATED

terminationGrace >= propagation + maxRequest + cleanup

The shutdown deadline must be longer than your normal maximum request time, but still finite. Long-running exports belong in asynchronous jobs with lease recovery, not in HTTP requests held open indefinitely.

Failure modes to design for

• Readiness reports healthy before migrations, pools, or mandatory configuration are ready. Use startup probes and a clearly ordered initialisation barrier.
• The load balancer keeps sending traffic after readiness changes. Measure propagation time and add it to the pre-stop delay and the termination grace period.
• Autoscaling and a rolling update remove capacity at the same time. Derive maxUnavailable, surge capacity, and disruption budgets from the same minimum healthy-capacity model.

What to observe

How to validate the design

• Send real requests continuously throughout a rolling update, and assert that success rate, p99, and connection resets all stay within objectives.
• Delay startup dependencies past the normal warm-up to prove that startup probes prevent restart loops without serving traffic too early.
• Terminate an instance with both short and long requests in flight, and verify the load balancer stops assigning new requests before the deadline.
• Remove one availability zone during a deployment, and confirm that disruption budgets preserve the minimum healthy capacity.
• Force a bad release signal, and measure whether the automatic pause or rollback completes within the error budget.

Safe rollout plan

Start by instrumenting lifecycle states and in-flight requests without changing termination behaviour. Measure real load-balancer propagation and request duration, then set a conservative grace period. Enable graceful drain for one small deployment and inspect forced exits before you expand. Add canary analysis based on user-facing SLOs, then automatic pause, and only add automatic rollback once the alerts are trusted. Tune autoscaling and disruption budgets with controlled load tests. Zero downtime is a property you observe across the whole traffic lifecycle, not a checkbox on the deployment manifest.

Production checklist

• A readiness check that verifies too much can flap and remove healthy capacity.
• Include warm-up time in the scaling threshold.
• Verify rolling updates under realistic concurrency before production.

Takeaway

Zero-downtime delivery is a traffic lifecycle: prepare, admit, drain, and only then terminate.