Scaling and deploying without dropping a request

Most of the outages we have debugged did not happen because we were short on servers. They happened at the exact moment the system added or removed a server. An instance that had just booted was already being sent requests by the load balancer while its connection pool was not open yet. An old instance received a stop signal while it still had a few dozen requests in flight and a batch of data not yet written to the database. Both of those situations can happen while CPU stays low and the replica count still matches the configuration.

Put another way: the infrastructure may still have headroom, and users still get a 502. The previous article — Load handling for SaaS: why auto scaling cannot save a bad design — answers the question what to scale first. This one answers the two remaining questions, and they are harder: what to measure to know the system is about to be overloaded, and how to scale and deploy without dropping a request.

1. Downtime is born the moment you change something

A system that sits still is relatively easy to keep stable. The problem is that a real system never sits still: you deploy a new version several times a week, auto scaling adds and removes replicas as load moves, a node gets a kernel update and is drained, a container is rescheduled onto another machine. Every one of those events cuts across a process lifecycle, and that is when requests are most likely to be lost.

There are two basic mistakes, and they are symmetric.

Mistake one — taking traffic too early. The moment the Node process opens port 3000, the orchestrator can consider it done and start routing requests to it. But an open port does not mean ready to serve: the Prisma connection pool may not be open, Redis may not be connected, the in-process cache is empty, and V8 has not warmed up the hot code paths. The first requests that reach that instance will be noticeably slower, or fail outright because the pool has no connection.

Mistake two — leaving the rotation too late. The orchestrator sends SIGTERM, the process calls process.exit() immediately, and the requests still being handled turn into connection resets. The load balancer does not know yet that the instance is gone, so it keeps sending new requests for a few more seconds. Users see a 502.

Being ready to serve is not just a matter of "the process is alive or dead". At startup, an instance needs time to prepare. At shutdown, it also needs time to leave the load balancer and finish the work in flight. Skip either of these two phases and you will lose requests on every deploy — sometimes just few enough errors that they hide in the noise.

2. What to measure to know the system is about to buckle

You can split metrics into two groups. The first group is cause metrics: CPU, RAM, disk bandwidth. They tell you how resources are being used. The second group is effect metrics: p99 latency, error rate, queue depth. They tell you how users and downstream systems are being affected.

The common mistake is to alert only on the first group. For a typical SaaS API, most of a request's time is spent waiting on Postgres or Redis. CPU can sit below 20% while p99 climbs to 8 seconds, purely because requests are queued waiting for an exhausted connection pool. The process is not busy computing; it is waiting. And waiting does not raise CPU.

Two widely used frameworks help you not miss anything:

• USE — for resources (CPU, connection pool, threads, disk). For each resource, ask three questions: Utilization (what percentage is in use), Saturation (how much work is queued waiting for it), Errors (is it rejecting anything).
• RED — for services (each route, each queue). Three questions: Rate (how many requests per second), Errors (what percentage fails), Duration (the distribution of handling time — and here it must be percentiles, not the mean).

The crux is the word Saturation. Utilization tells you how much of a resource is in use. Saturation tells you how much work is queued waiting for that resource. A connection pool at 100% utilization with no request waiting is fine: the pool is being used to its full capacity, exactly as designed. That same pool at 100% utilization with 60 requests waiting for a connection is a sign the system is about to be overloaded. That number 60 is the thing worth alerting on.

In Z-EDU we have a saturation metric that is extremely direct and extremely cheap. Learning progress is written through a write-behind mechanism: the video player's pings touch only Redis, and a worker flushes batches down to Postgres on a cycle. The /health endpoint returns pendingProgress — the number of progress records waiting to be written to the database. Normally that number hovers around 0, because the worker keeps draining the backlog. If it rises and then sits still at a high value, the worker is almost certainly dead or stuck.

// health.controller.ts — a health check has to say something useful
@Get('/health')
async health() {
  const [pendingProgress, auditWaiting, auditFailed] = await Promise.all([
    this.progressBuffer.size(),            // number of Redis keys waiting to flush
    this.auditQueue.getWaitingCount(),     // zedu-audit
    this.auditQueue.getFailedCount(),
  ]);

  return {
    status: 'ok',
    pendingProgress,   // ~0 is normal. Rising and then flat => the worker is dead.
    auditWaiting,
    auditFailed,
    workerEnabled: process.env.WORKER_ENABLED !== '0',
  };
}

What is notable is that when the worker dies, CPU does not go up; often it goes down. Requests per second do not change. The HTTP error rate stays at 0, because the API still accepts pings and returns 200. The infrastructure dashboard still looks green while data quietly piles up in Redis. With an asynchronous architecture, you have to alert on the backlog metric itself, not only on CPU or HTTP errors.

3. Why p99 is what should worry you, not the mean

Suppose a route has a mean latency of 120ms. That sounds fine. But the mean is a sum divided by a count — it cannot tell "every request takes 120ms" apart from "99 requests take 60ms and 1 request takes 6 seconds". Those two systems show the same number on the dashboard and are entirely different in real life.

Percentiles are more honest. p99 = 6 seconds means that out of 100 requests, roughly 1 will take 6 seconds or more. If you think "only 1%, still acceptable", multiply it out the way a real page actually works.

A page in a real application rarely calls exactly one API. The course detail page fetches the course information, the lesson list, the learner's progress, and the user's permissions — call it 8 calls. The probability that all 8 land outside the tail is 0.99 to the eighth power, roughly 92%. Which means about 8% of page loads will hit at least one slow request. The "1% slow" figure at the request layer has just become "8% of users see a slow page" at the experience layer. And a page is only as fast as its slowest call.

The tail is not a rare exception. For a UI that fires many APIs in parallel, the tail is the common experience. This is why every serious SLO is written in percentiles, never in means.

This matters even more when you scale. The tail is where the system reacts first when it starts to saturate. The mean can still look pretty while p99 has already tripled. If you only watch the mean, you will always hear the news late — usually after the customer reports the problem.

4. Health check: liveness is not readiness

This is where many systems inflict failures on themselves, so it is worth spelling out.

Liveness answers the question: does this process need to be killed and restarted? It should only check things a restart can fix, for example whether the event loop is still turning or whether the process is deadlocked.

Readiness answers a different question: should this instance receive requests right now? It is allowed to check dependencies. Answering "not ready" is perfectly normal during warm-up or during a shutdown.

The classic mistake is letting the liveness probe call down into the database. It sounds reasonable: "if the API cannot talk to the DB, what is it alive for?" But the consequences are ugly. Postgres blips for 3 seconds during a failover, liveness fails on every instance at once, and the orchestrator kills them all and restarts them simultaneously. A wave of new instances all open connections at once to a database that has just recovered, knocking the database over again. A 3-second incident can turn into 15 minutes purely because the health check was put in the wrong place.

The rule: liveness only checks what a restart can fix. If the DB is in trouble, restarting the API fixes nothing; it only makes things worse. External dependencies belong to readiness, and readiness does not kill the process — it merely pulls the process out of the traffic rotation.

// health.controller.ts — two endpoints, two purposes, do not merge them
@Controller()
export class HealthController {
  private ready = false;   // on when warm-up finishes, off when shutdown begins

  markReady()    { this.ready = true; }
  markDraining() { this.ready = false; }

  // LIVENESS: only proves the event loop is turning. NEVER touch DB/Redis.
  @Get('/livez')
  livez() {
    return { status: 'ok', uptime: process.uptime() };
  }

  // READINESS: allowed to touch dependencies. Failure => out of the LB, NOT killed.
  @Get('/readyz')
  async readyz(@Res({ passthrough: true }) res: Response) {
    if (!this.ready) {
      res.status(503);
      return { status: 'draining_or_warming' };
    }
    try {
      await this.prisma.$queryRaw`SELECT 1`;   // is the pool open yet?
      await this.redis.ping();
      return { status: 'ok' };
    } catch (err) {
      res.status(503);
      return { status: 'deps_unavailable' };
    }
  }
}

At the configuration layer, two things need attention: a grace period at startup so that failures while the process is still booting are not counted, and the number of consecutive failures before you act so that you do not overreact to a single blip.

# Docker Swarm — HEALTHCHECK decides whether the container may receive traffic.
# Point it at READINESS, not liveness, and give start_period enough room to warm up.
healthcheck:
  test: ["CMD", "node", "-e", "fetch('http://localhost:3000/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
  interval: 5s        # ask every 5s
  timeout: 3s         # slower than 3s counts as broken
  retries: 3          # 3 consecutive failures before concluding — blip protection
  start_period: 30s   # boot grace period: failures in the first 30s are NOT counted

5. Scaling out without downtime: warm-up and the stateless condition

Adding an instance sounds like a very safe operation. In practice it has two traps.

Trap one: the instance is still cold. A freshly started Node process has almost nothing ready: the Prisma connection pool is empty, the Redis handshake is not finished, the in-process cache holds no data, V8 has not JIT-compiled the hot code paths. If the load balancer sends it traffic from the very first second, the requests hitting the new instance will be considerably slower. The paradox is that you scale out because p99 is bad, and right after scaling out, p99 can be even worse for the first few seconds. Plenty of teams see this and mistakenly conclude that "scaling does not help".

The fix is to make readiness the gate: the instance only reports ready after it has warmed itself up. Do not rely on a guessed sleep; rely on work that has genuinely finished.

// main.ts — only report READY after actually being ready to serve
const app = await NestFactory.create(AppModule, { bufferLogs: true });
app.setGlobalPrefix('api/v1');

await app.listen(3000);          // the port is open, but /readyz still returns 503

// Warm-up: do exactly the work the first request would have to do.
const health = app.get(HealthController);
await warmUp(app);
health.markReady();              // only from this second does /readyz return 200
logger.log('ready to serve traffic');

async function warmUp(app: INestApplication): Promise<void> {
  const prisma = app.get(PrismaService);
  const redis  = app.get(RedisService);

  // 1. Force the pool to open connections, instead of making the first request wait for TCP + TLS.
  await Promise.all(
    Array.from({ length: 5 }, () => prisma.$queryRaw`SELECT 1`),
  );
  // 2. Handshake with Redis.
  await redis.ping();
  // 3. Preload the few config keys that nearly every request reads.
  await app.get(SettingsService).preload();
}

Trap two, the more dangerous one: the process is not really stateless. Horizontal scaling silently assumes that every instance is equivalent. A request must produce the same result no matter which instance it lands on. That only holds when the process keeps no important state in its own RAM.

The clearest example is the rate limit. If the per-IP counter lives in process RAM, say in a Map<string, number>, then with 3 replicas each IP can call 3 times the limit you think you set. Worse, the effective limit becomes random, because it depends on which instance the load balancer picks. Adding an instance at that point is not just scaling, it is quietly loosening a protection. Z-EDU's rate limit counts per IP through Redis, so the limit is a global number that does not change no matter how many replicas run.

The same principle applies to everything else: business data lives in Postgres, files live in RustFS (S3-compatible; the browser PUTs directly through a presigned URL, so no temp file is needed on the instance's disk), and transient state and high-speed data live in Redis. The test is simple: if you kill a random instance while it is under load, does anyone lose data or get an error? If the answer is "some people get logged out", "an in-progress upload is corrupted", or "a counter goes wrong", then the system is not stateless enough to auto scale safely.

6. Scaling in and redeploying without dropping a request

This is the harder half, and also the half more often forgotten. Removing a replica, a rolling update, a node being drained: all three lead to the same event. The process receives SIGTERM and has only a finite window to stop cleanly.

The correct sequence has four steps, and the order is the whole point:

1. Mark readiness = false immediately. The load balancer needs a few health check cycles to notice and stop sending new requests. During that window, the process must still serve normally. It is not yet allowed to reject requests; it is only announcing that it is about to leave the traffic rotation.
2. Wait for in-flight requests to finish (drain). Stop accepting new connections and let the requests in progress run to completion. There is a deadline: past it, you cut them off, but the deadline must be wider than the handling time of the slowest request.
3. Close the queue consumers and wait for running jobs. See below — this is the step most often skipped.
4. Close the DB/Redis connections, and only then exit. Closing the pool while requests are still running means actively turning them into 500 errors.

Step 1 is very often skipped, and it is the cause of many 502s at deploy time. Between the moment the process closes its listener and the moment the load balancer knows it, there is always a lag, usually exactly a few health check cycles. Requests arriving during that lag can be routed to an instance that is shutting down. That is why you must announce "I am not ready" first, wait for the load balancer to update, and only then close the listener.

// main.ts + shutdown.service.ts — graceful shutdown IN THE RIGHT ORDER
app.enableShutdownHooks();       // Nest calls onModuleDestroy/onApplicationShutdown on SIGTERM

@Injectable()
export class ShutdownService implements OnApplicationShutdown {
  constructor(
    private readonly health: HealthController,
    private readonly progressFlusher: ProgressFlusher,
    @InjectQueue('zedu-audit') private readonly auditQueue: Queue,
  ) {}

  async onApplicationShutdown(signal?: string): Promise<void> {
    this.logger.log(`${signal} received, starting drain`);

    // (a) Leave the load balancer FIRST. /readyz returns 503 from this second on.
    this.health.markDraining();

    // (b) Wait for the LB to notice: interval * retries + a safety margin.
    //     While waiting, requests still in flight ARE served normally.
    await sleep(15_000);

    // (c) Stop accepting new connections, wait for running requests (Nest does this on close()).
    //     (d) Flush the remaining write-behind batch still in the Redis buffer — do not leave data behind.
    await this.progressFlusher.flushNow();

    await this.auditQueue.close();
    this.logger.log('drain complete, exiting');
  }
}

For a queue worker, the rule is stricter still. If a running job is killed halfway, that write batch can be lost or only half applied. BullMQ already has the right mechanism: worker.close() stops accepting new jobs and waits for the running job to complete. If the job is not done by the time stop-grace-period expires, the job is still in the queue and will be picked up by another instance once the lock expires — provided your handler is idempotent. BullMQ guarantees at-least-once, not exactly-once. That is why Z-EDU's progress merge rule lives in SQL (INSERT ... ON CONFLICT with GREATEST): a batch that runs a second time still cannot push progress backward.

// worker.shutdown.ts — the worker must wait for the running job, not be killed mid-flight
@Injectable()
export class WorkerShutdown implements OnApplicationShutdown {
  constructor(private readonly workers: Worker[]) {}

  async onApplicationShutdown(): Promise<void> {
    // close(false) = stop taking NEW jobs, WAIT for the running job to finish.
    // close(true)  = force stop now -> the unfinished job goes back to the queue and reruns later.
    await Promise.all(this.workers.map((w) => w.close(/* force */ false)));
  }
}

// The prerequisite that makes (2) safe: an idempotent handler.
// A job that runs a second time must produce the same result as the first.

Finally there is the orchestrator layer. All that effort in the code is worthless if Swarm kills the container after 10 seconds while the application needs 30 seconds to drain. Three groups of settings decide this.

# docker-compose.yml (Swarm) — a rolling update that drops no request
services:
  api:
    image: registry.example/zedu:${TAG}
    environment:
      API_ENABLED: "1"
      WORKER_ENABLED: "0"
    healthcheck:
      test: ["CMD", "node", "/app/healthcheck.js"]   # points at /readyz
      interval: 5s
      retries: 3
      start_period: 30s
    stop_signal: SIGTERM
    # MUST be > the drain time in code (15s waiting for the LB + the slowest request).
    # When it expires Swarm sends SIGKILL, so set it wider than the real drain time.
    stop_grace_period: 45s
    deploy:
      replicas: 3
      update_config:
        order: start-first     # bring up the NEW instance, wait for it to be healthy, THEN take the old one down
        parallelism: 1         # one replica at a time: always enough capacity left to serve
        delay: 10s             # pause between steps, letting the system settle
        failure_action: rollback
        monitor: 60s           # watch for 60s after each step; roll back automatically on failure
      rollback_config:
        order: start-first
        parallelism: 1

  worker:
    image: registry.example/zedu:${TAG}    # THE SAME image, only a different role
    environment:
      API_ENABLED: "0"
      WORKER_ENABLED: "1"
    # The worker needs more room than the API: it must wait for the running job to finish.
    stop_grace_period: 90s
    deploy:
      replicas: 2
      update_config:
        order: stop-first      # the worker takes NO traffic from the LB -> no overlap needed
        parallelism: 1

Two details are worth pausing on. order: start-first for the API means the new instance is brought up and must be healthy before the old one is taken down. That way serving capacity never drops below the designed level during a deploy. The price is that for a few seconds, two versions of the code run side by side. That forces both versions to be compatible with the same database schema. So migrations must be additive: add the column first, read and write both the old and the new path, then drop the old column in a later deploy.

The worker, by contrast, uses order: stop-first: it takes no traffic from the load balancer, so there is no reason to overlap; take the old one down, bring the new one up — simpler and cheaper on resources. And the worker's stop_grace_period must be wider than the API's, because a job can run far longer than a request.

7. Self-defense under overload: load shedding and fail-open

Auto scaling is not instantaneous. From the moment a metric crosses the threshold to the moment a new instance is ready to serve can take tens of seconds, sometimes minutes. In that gap, the system has to protect itself by actively refusing some of the work.

That is load shedding. Once a system is overloaded, trying to serve everything usually makes everything slow. Requests queue, clients time out, clients retry, the queue grows further. Shedding 5% of requests early with a 429 can keep the other 95% fast; trying to hold on to 100% sometimes makes all 100% fail. Failing fast and clearly is usually better than hanging for a long time and then timing out.

A rate limit is planned load shedding. Z-EDU counts the rate limit per IP through Redis — mandatory, because as section 5 said, an in-RAM counter breaks the moment a second replica appears. But that raises a new question, and this one has no universally correct answer: what do you do when Redis dies?

There are only two options. Fail-closed: if you cannot check the limit, reject the request. Fail-open: if you cannot check it, let it through. Z-EDU chose fail-open, and the reason is pragmatic: Redis is a dependency of the rate limit, not of the business. With fail-closed, a supporting component could turn into a single point of failure for the entire system: learners cannot get into their courses just because a counter cannot count. This is consistent with the rest of the system: when Redis has trouble, a cache miss goes straight to the DB (slower, but still correct), and write-behind writes straight to the DB.

Fail-open where the rate limit protects performance. Fail-closed where it protects money or accounts — login and forgot-password (the rate limit is exactly what blocks brute force), sending OTP/SMS/email (each request costs real money), and calling a third-party API with a quota. In those places, rejecting is far safer than opening the gate.

The trade-off has to be stated plainly, because fail-open is not free: for the entire duration of a Redis incident, the system has essentially no rate limit. So every switch into fail-open must emit a metric and raise an alert. A silent fail-open is very dangerous: you think you have a layer of protection while it may have been off for days.

8. Auto scaling: which metric to scale on, and the flapping trap

Only now do we get to auto scaling, and it is the last step, not the first. Three principles.

One: do not scale on CPU for an I/O-bound system. As section 2 said, an API waiting on the DB can sit under 20% CPU while p99 is eight seconds. Auto scaling on CPU in that situation will not trigger, and you will sit staring at a green dashboard while the phone rings. Scale on a metric that reflects the outstanding work: requests waiting, queue depth, the p99 latency of an important route.

Two: scale up fast, scale down slow. The two directions are asymmetric in the cost of being wrong. Adding an instance you did not need costs money; removing one just as load is about to rise loses requests. So: a sensitive scale-up threshold with a short observation window (a minute or two); a strict scale-down threshold with a long observation window (ten minutes or more), and only one replica removed at a time.

Three: beware of flapping. This is a very easy loop to fall into. Load rises → scale up → the new instance is still cold, not warmed up → p99 temporarily gets worse → the autoscaler thinks there is still a shortage → scale up again. Or the other direction: scale down → load piles onto the remaining instances → metrics cross the threshold → scale up → load spreads out → metrics fall → scale down. The system keeps adding and removing replicas nonstop, and every cycle brings another round of drain and warm-up risk.

There are three techniques to use together to avoid flapping: a cooldown after each scaling action, long enough to exceed the warm-up time; a buffer zone between the scale-up and scale-down thresholds, for example scale up when p99 > 800ms but only scale down when p99 < 300ms; and an observation window long enough that a 30-second traffic spike does not trigger scaling on its own.

One advantage in Z-EDU's design: the BullMQ worker runs in the same process as the API but can be turned off with WORKER_ENABLED=0, and the whole monorepo builds into a single image. To scale the worker independently of the API you only have to stand up another service using that same image with different environment variables — no code change, no rebuild. This matters because the two roles saturate on two entirely different metrics: the API saturates on p99 and the number of connections waiting, while the worker saturates on queue depth and pendingProgress. Forcing them to scale on one shared rule gets both of them wrong.

9. Checklist: what the system must have before you turn on auto scaling

If a single line is still unticked, auto scaling will be an incident generator rather than an incident preventer — because it turns adding and removing instances from a rare manual action into something that happens automatically, daily, while you are asleep.

• Liveness and readiness are two separate endpoints. Liveness does not touch DB/Redis. Readiness does, and a readiness failure does not kill the process.
• Readiness has a real warm-up. The instance reports ready only after the pool is open, Redis has shaken hands, and the configuration is loaded — not based on a guessed sleep.
• Graceful shutdown in the right order: readiness = false → wait for the load balancer to update → drain in-flight requests → flush the pending write batch → close the connections → exit.
• stop_grace_period is larger than the real drain time. When it expires it is SIGKILL, no negotiation. The worker needs more room than the API.
• The worker closes with close() and waits for the running job, and the handler is idempotent so a job returned to the queue and run a second time still produces the right result.
• The process is genuinely stateless: kill a random instance under load and nobody loses anything. Every counter, session, and distributed lock lives in Redis, not in process RAM.
• Rolling update uses start-first + parallelism: 1, and migrations are additive so that two versions of the code can run side by side for a few seconds without breaking the schema.
• Alerts are built on saturation and effect metrics — p99, queue depth, pendingProgress, requests waiting for a connection — not only on CPU and RAM.
• The limits of the layer that CANNOT scale have been worked out: does replica count × pool size per replica exceed Postgres's max_connections? This is the most common way auto scaling takes the database down by itself.
• The fail-open/fail-closed decision is written down for each dependency, and a metric fires every time the system drops into a degraded mode.

Conclusion

Zero downtime is not a feature you switch on in an infrastructure control panel. It is the sum of a series of small, tedious details, every one of which is easy to forget: a readiness endpoint separated from liveness, a warm-up function that runs before reporting ready, fifteen seconds of waiting for the load balancer to notice you are withdrawing, one stop_grace_period number set correctly, one worker.close(false) instead of process.exit().

None of those details impress anyone when you show them off. But added together, they are the difference between a system you can deploy in the middle of peak hours and a system you only dare deploy at two in the morning. And the final measure is simple enough: if you do not dare press deploy when you have the most users online, you do not have zero downtime — you just have luck.