Multi-layer DDoS defense: from infrastructure to application

"Can the system withstand DDoS?" is a question we get often in technical conversations with customers. The honest answer is not just "yes" or "no", but: which kind of DDoS are you talking about? A 20 Gbps flood of junk traffic on the link, a bot holding thousands of open connections while sending almost no data, and a script calling the heaviest search endpoint 50 times per second can all take a system out of service. But those three cannot be blocked by the same mechanism. Each one has to be handled at a different layer.

This article walks through every defense layer we actually run: the OpenResty configuration at the edge, cache, the limiter inside the API gateway, and then the rate limit sitting in the application itself. More importantly, it also spells out the trade-offs behind those choices, including a few lessons that only surfaced after running in production.

1. Three kinds of attack, three different places to block them

The principle running through all of it is simple: the earlier you block, the cheaper it is. A request rejected right at the edge costs almost nothing. But if that request slips into the application, it can occupy a connection, force the system to verify a JWT, and then pull in a few Postgres queries. And Postgres is usually the hardest tier to replicate. So the defense has to be stacked in layers: whichever layer can block it most cheaply blocks it there, and only the rest travels on to the next layer.

One thing has to be said plainly: a volumetric attack cannot be blocked by software running on the very server under attack. Once the link is saturated, no amount of clever nginx configuration matters, because the packets never even reach the server to be rejected. That layer belongs to the infrastructure provider, the CDN, or a scrubbing service sitting in front. What a development team controls better are the other two groups: connection resource exhaustion and application-layer attacks. The rest of this article focuses on those two.

2. The edge layer: the cheapest limits, placed right at the door

Every request entering the system goes through OpenResty first. This is the right place for the simplest limits: rate limits, connection count limits, timeouts, and body size. These limits do not understand the business domain, but they are cheap and they protect everything behind them. The configuration below is running for the very page you are reading:

# Per-IP ceiling: ~5 requests/second, allowing a short burst of 20 requests
limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=5r/s;
limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

# Tighten timeouts — cut off Slowloris-style junk connections
client_header_timeout 10s;
client_body_timeout   10s;
send_timeout          10s;
reset_timedout_connection on;

# Static pages: no reason to accept a large body
client_max_body_size 2m;
large_client_header_buffers 4 8k;

limit_req_status  429;
limit_conn_status 429;
server_tokens off;

These three groups of limits handle three different problems, and the one most often forgotten is the timeout. limit_req blocks whoever calls too fast. limit_conn blocks whoever opens too many parallel connections. And timeouts block the kind of attack that barely does anything at all: Slowloris. It opens a lot of connections, and each connection sends only a trickle of header bytes to keep the state "active". This attack consumes almost no bandwidth, so the traffic graph can still look normal. But it takes up all the connection slots, leaving real users with no room to get in. What handles it is not a rate limit but an explicit timeout: client_header_timeout 10s means that if the headers are not fully sent within 10 seconds, the connection is closed.

A small detail well worth changing is limit_req_status 429. By default nginx usually returns 503 when a limit is exceeded. A 503 is easily read as "the server is broken", when the real problem is "the client is calling too fast". A 429, paired with Retry-After, describes the situation more accurately: the client knows how long to wait, and your monitoring system does not falsely alert that the backend is down.

A limit placed at the edge protects everything behind it, including the code you have not written yet. A limit placed in the application only protects the exact route you remembered to attach it to.

3. The cheapest request is the request that never reaches the backend

Before talking more about blocking, we need to talk about absorbing. In practice, the layer that helps us handle load the most is not the rate limit, it is cache.

In Z-EDU, public routes get a Cache-Control header attached via the @HttpCache decorator. That lets the CDN/Traefik return the response right at the edge, so the request never has to reach NestJS. When traffic to the course listing page spikes, whether from real people or bots, most of the load is absorbed here and the backend barely notices. As a DDoS defense, this is a very valuable layer: it shifts the pressure from the application to the cache, and cache is far cheaper and easier to scale.

But cache only helps when the request hits the cache. An attacker can add random parameters to the URL to force requests into cache misses and push the load down to the database. That is where the second layer comes in: CacheService.wrap collapses misses on the same key into a single computation (single-flight). If 100 requests miss the same key, only one request goes and computes the result, and the other 99 wait for it. This mechanism was originally built to prevent cache stampede when the cache expires in the middle of peak hours, but it is also very useful against cache-busting. We wrote separately about the three cache layers in Three-layer caching for multi-tenant SaaS.

4. The gateway: one number for every route is the wrong number

In Z-Cloud Workspace, every request goes through an API gateway running on OpenResty, with a rate limit written in Lua. This is where you can see very clearly why you should not set one shared limit for the whole API.

The problem is that endpoints do not have the same "price". Reading a file list may take only a few milliseconds. Calling an AI model costs CPU and money paid to the provider. Logging in pulls in Keycloak and a password hash that is designed to be slow. If you set a shared limit loose enough for reads, it will be far too loose for login. If you set it tight enough for login, ordinary users will constantly hit 429 while using lighter features.

So the gateway classifies each request into a group and applies a separate limit to that group:

The last row only exists because the system is multi-tenant (many businesses sharing one system, with fully isolated data). The limits above count per user. Even so, a business with three hundred employees, or a buggy script running under a legitimate account, can still consume all the shared resources without any single user exceeding their individual limit. tenant_total is a ceiling for the whole tenant, so that one customer's incident does not drag the other customers down with it. In a multi-tenant system, the "noisy neighbor" is usually more common than a targeted attack, and it tends to come from a broken integration rather than from a bad actor.

Count with a sliding window, not a fixed window

The simplest way to count is a fixed window: one counter per minute, reset when the minute ends. This approach has a hole right at the time boundary. With a limit of 10 requests/minute, a client can send 10 requests at second 59 and another 10 requests at second 61. In reality that is 20 requests in a few seconds, but on paper nothing was violated.

So the gateway uses a sliding window. It takes the request count in the current window and adds the still-valid portion of the previous window, weighted by elapsed time. The whole calculation runs inside one atomic Lua script in Redis: read, compute, and increment the counter in a single call. This detail matters a lot because the gateway has multiple replicas. If two replicas read and then write the same key, the limit can be silently exceeded exactly when load is highest, which is exactly when you need the rate limit to be most accurate.

Key on identity, not just on IP

IP is not always a good key. A whole office can reach the Internet through a single IP; a mobile carrier can also put a great many subscribers behind one NAT address. If you only block by IP, one user doing too much can get their entire company a 429. Conversely, if a botnet has thousands of IPs, a per-IP limit stops almost nothing.

So the gateway prefers to count by identity when identity can be determined. Authenticated requests are counted by a key derived from the user session, hashed, with no raw token stored in Redis. Only when the user cannot be identified does the system fall back to IP. The practical result is that a whole office sharing one IP still gets a separate limit per person, while an attacker who keeps rotating IPs still cannot escape the limit tied to the account being used.

5. In the application: block before authentication even runs

The gateway is not the last stop. In Z-Auto, the NestJS API still has its own rate limit. The most important point here is the execution order: RateLimitGuard is registered as the first global guard, meaning it runs before the JWT authentication guard.

// app.module.ts — this order is meaningful
providers: [
  { provide: APP_GUARD, useClass: RateLimitGuard }, // ← first
  { provide: APP_GUARD, useClass: JwtAuthGuard },
  { provide: APP_GUARD, useClass: RolesGuard },
]

If you reverse this order, every junk request still has to go through JWT signature verification, possibly pulling in a user query, and only then gets rejected for exceeding the limit. You still return 429, but you have spent exactly the kind of resource the attack wanted to exhaust. With the counter placed first, the 121st request from an IP can be dropped after a single INCR in Redis, without ever touching business code.

The counter uses Redis, so every API replica shares one limit. A counter held in each process's RAM cannot do that: if you run 4 replicas, the effective limit will be 4 times the number you wrote in the code. The limit structure is split into three layers: a dedicated group for sensitive endpoints, a default group for public routes, and a default group for authenticated routes.

There are two small but very practical details. First, limits can be adjusted while the system is running: an administrator edits them right in the admin screen, the override is stored in Redis, and every API replica picks up the new value within seconds without a redeploy. During an incident, being able to adjust in 10 seconds is very different from having to rebuild and redeploy. Second, the log records only one warning line at the moment an IP crosses the threshold, and nothing more for each request blocked afterwards. With tens of thousands of requests a minute, excessive logging can turn itself into a new incident: a full disk, a clogged logging pipeline, or a slowdown of the application itself.

Two kinds of route are exempt from the rate limit, and the reasons are clear. /health is exempt because the load balancer calls it constantly; if it counted toward the limit, the system could end up blocking its own monitoring tooling. Payment webhooks are also exempt, because returning 429 here can cause real damage: the payment gateway reports a successful transaction, but the system rejects it for being "too fast", so the customer has paid and the order is never recorded. Endpoints like this must be protected by verifying the sender's signature, not by a rate limit.

A rate limit is a tool for blocking volume. For endpoints where losing one request means losing money, the protection must be authentication, not a limit.

6. The last layer: when the attacker does not need many requests

Up to this point, most of the measures rest on the assumption that "attack = many requests". But some kinds of abuse do not need many requests; each request just needs to be expensive enough. This is where a pure rate limit is not enough, and the protection layer has to understand the specific business domain.

On the Z-Auto consultation request form, the first layer is a honeypot: an input field hidden from real users. Real people do not fill in this field, but bots usually fill in every field they see. If that field has a value, the API returns { ok: true } but writes nothing to the database. This does not tell the bot it has been detected, and it costs no Redis, so it keeps working even when the other counters have a problem.

The second layer is a limit on a business key, not just on IP. The number of leads from one IP is limited, but the number of leads on the same phone number is limited too. Rotating IPs is fairly easy for a bot; rotating real phone numbers is far more expensive. The general principle: count on what is hardest for the attacker to fake, not just on what is easiest for you to measure.

The third layer protects something very easy to forget in the age of AI: the bill. The AI assistant on the storefront calls a language model, and every call costs real money. An attacker does not need to take the system down; they just need to call steadily all night, and the next morning you get an unpleasant invoice. So the assistant is guarded at three layers: a per-IP limit, a daily ceiling tied to each visitor via an httpOnly cookie, and the showroom's credit ledger. Every call has to deduct credit, and when credit runs out it stops. The order matters a lot: when a limit is exceeded, the system replies with a canned message before calling the model. If you block after already calling the model, you have only blocked the response, and the cost has already been incurred.

7. The hardest question: what if Redis dies?

All the counters above depend on Redis. So there is one question you cannot dodge: when Redis does not answer, is the next request let through or blocked?

There is no right answer for every case. There are only two kinds of risk, and you have to pick the one you can live with. Fail-closed means that if you cannot check the limit, you block the request. This is safer in terms of protection, but a Redis incident can turn into a system-wide incident: users who did nothing wrong are rejected anyway. Fail-open means that if you cannot check, you let the request through. The system keeps serving, but during that window the limiting layer is essentially useless.

We did not pick one approach for every endpoint. The default is fail-open, because for most APIs, letting a few minutes of unlimited traffic through is still less harmful than rejecting all your real customers over an incident in a secondary piece of infrastructure. But the auth_sensitive group at the gateway is fail-closed. When Redis is unreachable, the login endpoint is not thrown wide open but switches to a local counter in the gateway's shared memory. That counter is less accurate because each gateway replica counts on its own, so the effective limit will be looser than configured. But when it comes to user passwords, "less accurate" is still far better than no limit at all.

Fail-open is a reasonable choice. Silent fail-open is not. If your limiter turns itself off and nobody knows, you do not have a degraded mode — you have a hole.

So the state of Redis is a tracked metric, not an assumption. The rate limit report returns redisHealthy alongside the numbers, and this is the kind of alert you have to build before you need it.

8. Two lessons that only come at a price

A limiter in the wrong place is worse than none

The Traefik-layer rate limit configuration for Z-Auto has been written, and it is currently disabled on purpose. The reason: the system runs on Docker Swarm, and with Swarm's default routing mode, Traefik sees the IP of the ingress layer for every client, not the user's real IP. Turning on a per-IP rate limit in that state means every user in the world is lumped into a single counter. The result is not "the attacker is blocked" — it is the system returning 429 to its own customers, looking exactly like a successful DDoS. We are leaving it off until the network configuration lets Traefik see the real IP.

The lesson generalizes far beyond Swarm: a limiter is only correct when the key it counts on is correct. Before turning on any per-IP limit, answer one question: is the IP this layer sees the user's IP, or the IP of the proxy standing right in front of it?

Trusting a client header is opening the door yourself

That question has an equally dangerous flip side. Behind a reverse proxy, the client's real IP lives in the X-Forwarded-For header. But a header is something the client sends itself — so if your application reads X-Forwarded-For blindly, anyone can neutralize your entire rate limit by making up a new IP for every request. The limiter still runs, the graphs still look good, and it blocks nothing at all.

X-Forwarded-For is only trustworthy when the proxy in front of it is yours and no request has any path around that proxy. On the same principle, the Z-Cloud Workspace gateway strips every internal trust header from incoming requests before doing anything else:

-- strip-trust-headers.lua — runs at the start of every request
ngx.req.clear_header("X-Gateway-Verified")
ngx.req.clear_header("X-Verified-User-Id")
ngx.req.clear_header("X-Verified-User-Email")
ngx.req.clear_header("X-Tenant-Code")
-- …

The services behind it trust these headers to know who the caller is — they are set by the gateway after it has verified the JWT. If the gateway does not strip them first, a client only has to send X-Verified-User-Id along to become anyone. The principle: every header the internal system trusts must be stripped at the entrance and only set again by the layer that verified it. This goes beyond the scope of DDoS, but it has the same root — the boundary between "data supplied by the client" and "facts the system has verified for itself".

9. If you do not measure it, you do not know you are under attack

A rate limit with no numbers behind it is a rate limit you will not dare to change: tighten it and you fear blocking real customers, loosen it and you fear a breach. So every limit check is recorded (best-effort, off the request path), and the admin screen can answer three questions:

• The block rate per limit group. A group with a consistently high block rate is usually not a sign of an attack — it is a sign the limit is set too tight, and the people being blocked are your customers.
• The list of most-blocked IPs, kept in a size-bounded set. This bound is deliberate: a list of "every IP ever blocked" would grow without limit exactly when you are under attack — meaning the attack tracker itself becomes a way to exhaust memory.
• Whether Redis is alive. Because, as said above, this is the answer to "is the limiting layer actually working".

This shares a philosophy with observability in our other systems: alerts have to be built on metrics that say something about business state, not just on CPU and memory. A well-designed L7 attack can take your system out of service while CPU stays at normal levels.

10. Checklist

If you are reviewing your own system, these are the questions worth answering, in order:

• Have you tightened the timeouts? This is the cheapest thing and the most often forgotten. Without it, a trickling connection can hold your slot indefinitely.
• Is there a ceiling on concurrent connections per IP? A rate limit and a connection limit block two different things.
• Is body size capped per route? An upload endpoint needs to accept 500MB; a login endpoint does not — and if it accepts 500MB too, you have just handed the attacker a way to exhaust memory.
• Does the limit distinguish expensive routes from cheap ones? A single number for every endpoint is always both too tight and too loose.
• Where does the counter live? In process RAM, and your real limit is being multiplied by the number of replicas.
• Can the counting key be spoofed? If it comes from a header the client sends itself, the answer is yes.
• Have you written down fail-open or fail-closed for each group, and is there an alert when you drop into the degraded mode?
• Which endpoints cost money on every call? Those endpoints need a cost ceiling, not just a rate ceiling — and they must block before the cost is incurred.
• Do the logs fill the disk when you are under attack? Write one line when the threshold is crossed, not one per request.

Conclusion

None of the layers above "stops DDoS" on its own. An edge configuration will not save you from a script calling the search API. An in-application rate limit will not save you from Slowloris. And nothing in this article will save you from a bandwidth flood that is large enough — that layer is outside the scope of a software team, and anyone who says otherwise is selling you something.

What actually works is accepting that this is a multi-layer problem: tighten timeouts and connection limits at the edge because they are cheap; absorb with cache because a request that never reaches the backend costs nothing; classify routes by their real price at the gateway; put the counter ahead of authentication in the application; and for the endpoints that spend money, block at the cost layer, not just at the rate layer. Finally, write down explicitly what happens when Redis is down, and build alerts on that very decision.

Most of the "DDoS incidents" we have untangled turned out not to come from a bad actor: a customer integration stuck in a loop, a crawler bot that does not read robots.txt, or a cron job calling its own API back exponentially. The defense layers in this article handle those situations with the same mechanisms used to block a real attack. That is why they are worth building, even when you think nobody has any reason to attack you.