Load handling for SaaS: why auto scaling cannot save a bad design

When a system starts to slow down, the first reflex of many teams is to open the infrastructure dashboard and raise the replica count. It is easy, it is fast, and in some cases it is the right call. The problem is everything else: adding an instance does not fix the root cause, it only makes a suboptimal design run in parallel more times.

1. Auto scaling cannot fix a bad design

Take a real example from Z-EDU, before we fixed it. The video player sent a progress ping every 15 seconds for each learner. The old code path hit the database with 4 queries per ping; the 20 pings of one session came to roughly 80 queries down to Postgres. Multiply that by the number of learners watching at the same time, and the API is no longer just serving HTTP, it is continuously pushing queries down to the DB.

Now suppose the API starts to slow down and you double the instance count. What happens? Each instance still generates 80 queries per viewing session. You have just doubled your capacity to push queries at a single Postgres. Latency does not drop, p99 gets worse, the connection pool runs dry, and now you have twice as many processes fighting over the same resource that is hard to scale horizontally. This is the crux that many articles about "scalability" skip over:

Auto scaling only replicates the tier you scale. It does not replicate Postgres, it does not replicate Redis, and it does not replicate disk bandwidth. If the bottleneck sits in a tier you cannot replicate, adding an instance only tightens that bottleneck, faster.

The right order is the reverse. Before asking "how many instances do we need", ask "how much work does each request actually have to do". Most of the slow APIs we have debugged share the same shape: they do far too much unnecessary work right on the synchronous request path. There are two ways to take that work out: keep the request from reaching the API at all with a cache, and do not do the side work right then with a queue. Once those two things are done, auto scaling starts to make sense, because what you are replicating is a lean process rather than a process that keeps piling pressure onto the DB.

2. Move some work off the request path

The tier that absorbs most of Z-EDU's load is not the API, and it is not Redis. It is the Cache-Control header set by the @HttpCache decorator on public routes. The CDN/Traefik returns the response right at the edge, and the request never reaches NestJS. A request served at the edge costs the backend almost nothing: no connection, no CPU, no Prisma query. No optimization inside the code beats not having to run that code.

But caching at the edge is a double-edged sword, and we nearly paid for it. Blanket-caching every @Public route sounds reasonable until GET /exams/public/attempts/:id, one candidate's exam attempt, gets held by the proxy and can be served to somebody else. The lesson: edge caching must be opt-in per route, never a default applied to a group. "Public" in authentication means no JWT required; "public" in caching means everyone sees the same thing. These two concepts are not the same.

The second tier is Redis via CacheService.wrap. What matters about wrap is not just that it stores the result, but that it collapses concurrent misses on the same key into a single computation (single-flight). When a hot key expires, hundreds of concurrent requests do not all crash down onto the DB; only one request goes and computes, and the rest wait for that result. This is the mechanism that prevents cache stampede, and it matters precisely at the moment the cache expires. We covered the details of the three cache tiers, version-based invalidation, and why we do not use SCAN/KEYS separately in Three-layer caching for multi-tenant SaaS.

3. Separating the worker from the API: BullMQ and the "same image, different role" principle

Caching handles the read side. The write side needs a different mechanism: a queue. Z-EDU uses BullMQ with two queues that have clear roles: zedu-maintenance (progress cleanup, periodic jobs) and zedu-audit (admin action logs).

The audit queue is the clearest example of "work that does not belong on the request path". Previously, each admin action wrote 2 extra log queries inside the request. The user clicked "Save" and had to wait for the log write to finish before getting a response. Nobody reads the log in the next 200ms, so making the request wait for it is a bad trade. Move it to a queue, the request returns sooner, and the worker writes afterward.

// audit.service.ts — the log leaves the request path
@Injectable()
export class AuditService {
  constructor(@InjectQueue('zedu-audit') private readonly queue: Queue) {}

  // Called from the controller. No awaiting the DB, only awaiting the enqueue.
  async record(entry: AuditEntry): Promise<void> {
    await this.queue.add('write', entry, {
      removeOnComplete: 1000,
      removeOnFail: 5000,
      attempts: 3,
      backoff: { type: 'exponential', delay: 1000 },
    });
  }
}

// audit.processor.ts — runs in the worker, does NOT run if WORKER_ENABLED=0
@Processor('zedu-audit')
export class AuditProcessor extends WorkerHost {
  async process(job: Job<AuditEntry>): Promise<void> {
    // Batched write, idempotent on requestId so a retry does not duplicate rows.
    await this.prisma.auditLog.createMany({
      data: [toRow(job.data)],
      skipDuplicates: true,
    });
  }
}

Three details in that snippet are worth spelling out, because without them a queue turns from a solution into a new source of incidents:

• attempts + backoff: jobs will fail. The DB restarts, the network blips. Without a retry you lose data silently, which is far worse than a loud 500.
• skipDuplicates / idempotency: BullMQ guarantees at-least-once, not exactly-once. A job can run twice. If the handler is not idempotent, a retry creates duplicate rows. Same principle as write-behind for progress: the merge rule lives in SQL, so a late or concurrent batch still cannot make progress go backwards.
• removeOnComplete/removeOnFail: without cleanup, your Redis will fill up with the history of successful jobs. This mistake is very easy to make after a few quiet months in production.

Now for the part that matters operationally. The Z-EDU worker runs in the same process as the API, and can be switched off with the environment variable WORKER_ENABLED=0. That sounds like an ugly compromise — why not split it into its own service and keep things clean? Because splitting it means two codebases, two build pipelines, two images, and two sets of configuration to keep in sync. For a small team, that is a real cost, paid daily.

What we do instead: one image, multiple roles. The same artifact, with the role decided by environment variables at runtime.

// main.ts — the role is decided at runtime, not at build time
const WORKER_ENABLED = process.env.WORKER_ENABLED !== '0';
const API_ENABLED = process.env.API_ENABLED !== '0';

const app = await NestFactory.create(AppModule, { bufferLogs: true });

if (API_ENABLED) {
  app.setGlobalPrefix('api/v1');
  await app.listen(3000);
} else {
  await app.init(); // worker only: no HTTP port opened
}

// AppModule only loads the *Processor classes when WORKER_ENABLED,
// so an API-only process registers no BullMQ consumer at all.

On a normal day, a single service runs both roles: simple, with fewer things to watch. When the queue starts backing up, we stand up an extra worker-only service using that very same image, differing only in the environment variables. No code change, no rebuild, no redeploy of the API.

# docker-compose.yml (Swarm) — same image, different role
services:
  api:
    image: registry.example/zedu:${TAG}
    environment:
      API_ENABLED: "1"
      WORKER_ENABLED: "0"   # API only: serves HTTP only
    deploy:
      replicas: 3
      update_config: { order: start-first, parallelism: 1 }

  worker:
    image: registry.example/zedu:${TAG}   # THAT same image
    environment:
      API_ENABLED: "0"
      WORKER_ENABLED: "1"   # worker only: consumes the queue only
    deploy:
      replicas: 2           # scales independently of api

The real value of this design is that scaling becomes an easier decision. No architecture change, no new PR to merge; just change the replica count. In exchange, the development team has to keep the discipline: every difference between the two roles lives only in environment variables, with no branching on hostname or build flags in the code. Break that, and the image is no longer a single artifact, and the benefit disappears.

A limit worth stating plainly: this model does not give you process-level fault isolation when the roles run together. A worker eating RAM will affect the API in the same process. That is exactly why WORKER_ENABLED exists — it is the valve you open when load rises, not decoration.

4. Horizontal scaling only works when state lives outside

There is a precondition that auto scaling silently assumes but does not check for you: every instance must be equivalent. Whichever instance a request lands on, the result is the same. That only holds when the API process keeps no important state in its own RAM.

In Z-EDU, state is pushed outside deliberately: sessions/JWT need no server-side storage, business data lives in Postgres, files live in RustFS (S3-compatible, the client PUTs directly via a presigned URL), and temporary, high-speed state lives in Redis. The Node process is something you can kill at any moment.

The clearest example is rate limiting. If the per-IP request counter lives in the process's RAM — something like a Map<string, number> — then with 3 replicas, each IP is allowed to call 3 times the limit you think you set. Worse, the effective limit becomes random, depending on where the load balancer throws the request. Our rate limit counts per IP through Redis, so the limit is a global number, independent of the instance count.

Here is the simple test to run before turning on auto scaling: if you randomly kill an instance while it is under load, does anyone lose anything? If the answer is "some people get logged out", "some in-flight uploads break", or "the counters are wrong" — then you are not stateless yet, and adding instances will create errors rather than reduce load.

5. Rate limiting and the fail-open decision

Moving the rate limit into Redis solves the global counting problem, but it raises a new question: what do you do when Redis has an incident? This question has no single right answer for every system.

You only have two options. Fail-closed: if you cannot check the limit, reject the request. Fail-open: if you cannot check it, let it through. Z-EDU chose fail-open.

// rate-limit.guard.ts — a dead Redis must not be allowed to take down the API
async canActivate(ctx: ExecutionContext): Promise<boolean> {
  const req = ctx.switchToHttp().getRequest();
  const key = `rl:${routeOf(ctx)}:${clientIp(req)}`;

  try {
    const hits = await this.redis.incr(key);
    if (hits === 1) await this.redis.expire(key, WINDOW_SEC);
    if (hits > LIMIT) throw new ThrottlerException();
    return true;
  } catch (err) {
    if (err instanceof ThrottlerException) throw err; // over the limit: still blocked
    // Redis is not responding -> FAIL-OPEN, with a log so the alert fires.
    this.logger.error({ err }, 'rate-limit backend down, failing open');
    this.metrics.increment('ratelimit.fail_open');
    return true;
  }
}

The reasoning is very practical: Redis is a dependency of the rate limit, not of the business logic. If Redis fails and we fail-closed, a supporting component could make the whole system reject real users. Same philosophy as the rest of Z-EDU: if Redis fails, a cache miss goes straight to the DB (slower, but correct), and write-behind writes straight to the DB. Redis is an acceleration layer, not the source of truth.

But the trade-off has to be said out loud, because this is not a free choice: for as long as Redis is broken, the system has essentially no rate limit. An attacker could try to overload Redis first and then exploit that gap. For Z-EDU, where sensitive endpoints are additionally protected by JWT, RBAC, and server-side limits, this trade-off is acceptable.

Where is the line? Fail-closed when the rate limit is the only security control on that path. Concretely: the login and forgot-password endpoints (the rate limit is precisely what stops brute force), the OTP/SMS/email sending endpoints (each request costs real money), and endpoints calling a third party with a quota. In those places, rejecting a request is far safer than opening the door. The practical rule: fail-open where the rate limit protects performance, fail-closed where it protects money or accounts.

6. Watch the right metric: low CPU does not mean healthy

When you push work into a queue, you also push some of the errors out of sight. Before, a failure could make the request return a 500 and you knew immediately. After the worker is split out, the API can still return 200, the dashboard can still be green, and the data quietly does not get written. This is the price of an asynchronous architecture, and the way you pay it is by watching the right place.

In Z-EDU, /health returns pendingProgress: the number of progress records waiting to be written down to the DB. Normally this number hovers around 0, because the worker flushes down to Postgres every 5 seconds. If it rises and stays at a high level, the worker is almost certainly dead or stuck. That is the metric we build alerts on.

// health.controller.ts — a health check that actually says something useful
@Get('/health')
async health() {
  const [pendingProgress, auditWaiting, auditFailed] = await Promise.all([
    this.progressBuffer.size(),                 // Redis keys awaiting flush
    this.auditQueue.getWaitingCount(),
    this.auditQueue.getFailedCount(),
  ]);

  return {
    status: 'ok',
    pendingProgress,   // normally ~0. Rises & stays high => the worker is dead.
    auditWaiting,
    auditFailed,
    workerEnabled: process.env.WORKER_ENABLED !== '0',
  };
}

From this comes a general principle, and it is the most valuable thing in this whole article:

Alert on the metrics that reflect the backlog of work: queue depth, processing lag, the number of records waiting to be written. Low CPU with a swelling queue is still a system in trouble: it is not busy, because it has stopped working.

CPU is a cause metric, not an effect metric. It is only useful when the load is compute-bound. For a typical SaaS system — I/O-bound, spending most of its time waiting on the DB and Redis — CPU can sit below 20% while p99 latency is 8 seconds because the connection pool has run dry. CPU-based auto scaling in that situation will not trigger, and you will sit staring at a green dashboard while users are calling you on the phone.

7. Deployment: one image, one domain, multiple roles

Z-EDU is a pnpm + Turborepo monorepo, deployed through GitLab CI down to Portainer Swarm. The whole monorepo builds into one image, and the system is served on one domain with routing at the proxy tier: /api goes to the NestJS API, /admin to the Next.js admin, / to the Next.js web app.

The benefit is not that it is "architecturally elegant" — it is that the number of things that can drift apart goes down. A single image tag describes the state of the whole system; a rollback is changing one TAG variable. There is no scenario where API tag v1.4.2 runs alongside admin tag v1.3.9 and breaks because the API contracts have diverged. One domain means no CORS, no cross-site cookie configuration, and no configuration layer left to forget.

The trade-off you pay: the image is larger than the minimum (a worker-only process still carries the Next.js bundle), and every small change rebuilds everything. At Z-EDU's scale, a few minutes of CI is far cheaper than a few hours of debugging version drift. This is a conditional trade-off — if you have 40 services and 8 teams, the answer is different. Choose based on your real scale, not on the blog post of a company with 5,000 engineers.

8. Checklist before turning on auto scaling

If a single line is still unticked, auto scaling will make things worse, not better.

• Hot read routes are cached at the edge, opt-in one route at a time, and you have verified yourself that no route serves one user's private data through a shared cache.
• The Redis cache uses single-flight so that an expiring hot key does not drag a burst of queries down onto the DB all at once.
• Side work is off the request path — logging, sending mail, and report aggregation live in a queue, not between the request and the response.
• The process is genuinely stateless: kill a random instance under load and nobody loses anything. Every counter, session, and distributed lock lives in Redis, not in process RAM.
• You have accounted for the limit of the tier you CANNOT scale: the Postgres connection pool. Multiply the replica count by the pool size per replica — does that number exceed max_connections? This is the most common way auto scaling takes down the DB by itself.
• Jobs are idempotent and have retries: BullMQ is at-least-once. Running twice must give the same result.
• Alerts are built on backlog metrics (pendingProgress, queue depth, number of failed jobs), not only on CPU and memory.
• The fail-open/fail-closed decision is written down for every dependency, with a metric fired when you drop into degraded mode — a silent fail-open is a dangerous fail-open.
• You measured before scaling: you know exactly whether a request is slow because it waits on the DB, waits on the network, or because of CPU. Those three causes need three different treatments, and auto scaling fixes only one of the three for you.

Conclusion

Auto scaling is a useful operational tool, and we still use it. But it is the last step, not the first. It answers the question "with the same amount of work, how do we handle more requests" — it does not answer the question "why does each request have to do so much work in the first place".

The right order: take the unnecessary work off the request path (edge caching, a queue for side work), push state out of the process, measure where the real bottleneck is, and only then replicate. Follow that order and you will often find you need fewer instances than you thought. Do it backwards and you will have a bigger infrastructure bill and a system just as slow as before — except now it is slow in several places at once.