In brief: Extra instances multiply capacity. They do not fix an expensive request path.
The problem
When a request sends email, builds reports, or processes media synchronously, every server you add also adds database connections and contention. CPU can look healthy while users wait behind a growing queue.
Why the simple answer is not enough
Auto scaling only reacts once a signal crosses a threshold. So if a single request synchronously handles email delivery, media conversion, and report generation, every API replica you add also raises database connections and downstream contention.
The first fix is therefore architectural, not operational. Shorten the synchronous critical path, turn deferred work into measurable queue units, and apply backpressure before a dependency reaches saturation.
System flow
validate and save→API
return quickly→Queue
absorb bursts→Workers
process side work
Architecture decisions
Define the synchronous contract
Commit only the state needed to acknowledge the user's action. Publish side effects through a transactional outbox, so the API can never save business data and then quietly lose the job.
Make workers safe to retry
Use one idempotency key per business operation, store attempt state, and separate transient errors from permanent validation failures. At-least-once delivery must never create duplicate invoices or emails.
Scale from demand and saturation
API replicas can track concurrency and latency. Workers should track queue age, arrival rate, and service rate. Bound worker concurrency by what the database and external APIs can actually handle.
Going deeper
Backpressure is part of the API contract
A queue is not infinite capacity. It converts immediate overload into delayed work. So define the maximum acceptable queue age, and derive your admission rules from it. When predicted wait exceeds the objective, reject or degrade low-priority work rather than accept a promise the system cannot keep. Return Retry-After or a resumable status resource instead of letting requests time out ambiguously. Keep customer-facing priority separate from technical fairness: per-tenant quotas stop one bulk import from consuming every worker, while weighted scheduling still reserves capacity for small interactive jobs.
Exactly-once is a business effect, not a broker feature
Most brokers give you at-least-once delivery, especially around crashes. You achieve a single business effect with a durable idempotency record and a state transition committed in the same transaction as the output. The key should identify the business operation, not a delivery attempt. For external side effects that cannot join the transaction, store an intent, call the provider with its own idempotency key, and reconcile any uncertain outcome. A timeout is not proof of failure. Querying provider status before you retry is what prevents duplicate charges, messages, or exports.
Implementation path
- Measure the request, then strip out any work the user does not need to wait for.
- Make queued jobs idempotent so a retry is always safe.
- Scale workers from queue depth and oldest-job age, not host CPU alone.
Technical example: Queue capacity and scaling signal
arrivalRate = jobsAdded / windowSeconds
serviceRate = jobsCompleted / windowSeconds
backlogTime = queueDepth / max(serviceRate, 1)
desiredWorkers = ceil(arrivalRate / jobsPerWorker)
desiredWorkers = clamp(desiredWorkers, minWorkers, maxWorkers)
if backlogTime > objectiveSeconds:
desiredWorkers = min(desiredWorkers + burstStep, maxWorkers)Little's Law only holds when every measurement uses the same window and the system is reasonably stable. During bursts, queue age stays the guardrail closest to what users actually feel.
Failure modes to design for
- The queue accepts work faster than the database can commit it. Adding workers makes the outage worse, so concurrency limits and admission control have to protect the database.
- A deployment kills workers mid-job. Rely on visibility timeouts, heartbeat extension, and idempotent handlers so abandoned work is reclaimed safely.
- A third-party API rate-limits one job type. Separate queues or concurrency pools keep that one dependency from blocking unrelated work.
What to observe
| Signal | Why it matters |
|---|---|
| Oldest job age | Tracks the delay users actually feel, and catches a slow drain even when the queue is small. |
| Arrival versus completion rate | Shows whether the backlog is temporary or mathematically guaranteed to keep growing. |
| Worker utilisation and job p95 | Separates a genuine capacity shortage from slow or skewed jobs. |
| DB pool saturation and external throttles | Sets the safe ceiling; scaling past it just relocates the bottleneck. |
How to validate the design
- Replay every job several times, and verify that business rows, notifications, and external calls each stay singular.
- Push arrival rate above service rate, and confirm admission control kicks in before the database saturates.
- Kill workers at each transition—before claim, during processing, and after commit—to exercise lease recovery.
- Throttle one external dependency, and prove that isolated queues preserve latency for unrelated job classes.
- Measure recovery time after a long outage, with the full backlog plus normal incoming traffic.
Safe rollout plan
Move one side effect at a time behind the outbox and queue. For an initial cohort, keep the synchronous result as the reference while the asynchronous path runs in compare-only mode. Next, acknowledge after commit and expose job status to users, but keep a bounded synchronous fallback for incident response. Set queue-age alerts before you raise traffic. Then raise worker limits gradually, verifying the database, broker, and third-party budgets at every step. An autoscaler without those ceilings can turn a recoverable backlog into a dependency outage.
Production checklist
- Queues add resilience, but they also mean work completes eventually rather than immediately.
- A growing backlog should trigger an alert before users notice the delay.
- Protect the database with connection and concurrency limits.
Takeaway
Auto scaling pays off once the system has clear units of work and reliable demand signals to scale on.