Three-layer caching for multi-tenant SaaS: CDN, Redis and version-based invalidation

Z-EDU is the LMS we built for education centers. The backend runs NestJS 11 for the REST API at /api/v1, Prisma on PostgreSQL, Redis, and Next.js 16 for the web and admin apps, all inside one pnpm + Turborepo monorepo, deployed through GitLab CI down to Portainer Swarm. The traffic is heavily skewed: most of it is reads, such as the center's landing page, the course catalog, and public tests; writes are fewer but must be exactly right, such as submissions and grades.

The first instinct is easy to understand: add Redis, wrap a few heavy queries, and call caching done. We thought that too, until we learned three lessons the hard way, one of which almost became a security incident. This is a write-up of the three cache layers we actually run, and of the risks that live between them.

1. Why adding Redis alone is not enough

Redis is fast, but look closely at the path of a request that hits the Redis cache: it still goes through the CDN, the reverse proxy, into the Node process, through middleware, parses the JWT, builds the tenant context, runs the authorization guard, and only then reaches the controller to read Redis, serialize JSON, and respond. You save PostgreSQL time, but you still pay nearly the full cost of the request lifecycle inside Node: the event loop, sockets, middleware, and the resources of the API process.

When a center runs an enrollment campaign and a few thousand people open the landing page within a minute, the thing that saturates first is not necessarily PostgreSQL. The API workers may well be the bottleneck. If the request still travels the full application lifecycle, Redis merely shifts pressure from the database to the API. To actually shed load, you have to stop the request before it ever touches the API. That is the job of Cache-Control.

The best cache layer is the one where the request never reaches your application at all. Every layer inside it is only a safety net for the traffic that slips through.

So we stack the caches from the outside in. Each layer has its own job and its own trade-off:

2. Layer 1 — Cache-Control at the edge, and the incident worth retelling

In Z-EDU the edge layer is driven by a single decorator: @HttpCache. It only attaches metadata; an interceptor reads that metadata and sets the Cache-Control header on the response. No decorator means no cache — the default is no-store.

// http-cache.decorator.ts
export interface HttpCacheOptions {
  maxAge: number;              // seconds — for the browser
  sMaxAge?: number;            // seconds — for shared CDN/proxy caches
  staleWhileRevalidate?: number;
  visibility?: 'public' | 'private';
  vary?: string[];
}

export const HTTP_CACHE = 'http_cache';
export const HttpCache = (opts: HttpCacheOptions) => SetMetadata(HTTP_CACHE, opts);

// Usage: opt in only on routes that are truly public and identical for every user
@Public()
@HttpCache({ maxAge: 60, sMaxAge: 300, staleWhileRevalidate: 600, visibility: 'public', vary: ['Accept-Encoding'] })
@Get('landing/:tenantSlug')
getLanding(@Param('tenantSlug') slug: string) {
  return this.landingService.get(slug);
}

Our first version did not work like this. It tried to be "smarter": a global interceptor saw that a route carried @Public() and automatically attached Cache-Control: public. The reasoning sounded sensible: @Public() means no login required; if no login is required then surely everyone sees the same thing; therefore it can be cached publicly.

That reasoning is wrong on one important point. Z-EDU has the route GET /exams/public/attempts/:id: a candidate takes a public test without creating an account, receives an id for that attempt, and uses that same id to read their own attempt back. The route carries @Public() because it does not require a JWT. But the response is still one specific person's data.

The global interceptor saw @Public() and attached Cache-Control: public, s-maxage=300 to the response. A shared proxy kept one candidate's attempt. The next person hitting the same URL could get that content back. This is no longer a cache display bug; this is a data leak.

@Public() answers the question "do you need a token to call this?". It does not answer the question "is this response the same for everyone?". Those are two different questions, and only the second one decides whether something may be cached publicly.

The lesson became a rule in the repo: edge caching must be opted into per route, never inferred from another decorator. Adding one @HttpCache line per route is far cheaper than the risk of serving private data to the wrong person. We accept a little repetition to make the system safe by default.

public, private, and the role of Vary

Three easily confused concepts need to be kept apart. Cache-Control: public lets shared caches — a CDN, a corporate proxy, or an ISP proxy — store a copy and replay it to other people. private lets only the user's own browser cache store it; the CDN must skip it. And no-store means it is not stored anywhere at all. For the attempts/:id route the right choice is no-store, because this is one person's private data.

Vary is the second piece. It tells the cache which headers the response depends on. If the response changes with Accept-Language and you forget Vary: Accept-Language, a Vietnamese-speaking user can get the English page left behind by the previous visitor. For multi-tenant SaaS (many businesses sharing one system, with fully isolated data) the risk is larger: if the tenant is identified by a header but the cache does not Vary on that header, the CDN can serve center A's content to center B. The safer approach is to put the tenant in the path or the hostname, because that is the part the CDN certainly includes in the cache key.

3. Layer 2 — Redis and single-flight against cache stampede

Not every route can be cached at the edge. A student's progress summary is personal data, it has to pass the guard, and the CDN must never touch it — yet it is a heavy query and it gets called constantly. That is layer two's territory: Redis, behind a thin CacheService.

The heart of CacheService is not get and set. The important part is wrap, specifically its single-flight behavior. Picture a hot cache key, say the course catalog of a large center, which takes about half a second to rebuild. When the TTL expires right at peak hour, hundreds of concurrent requests see a cache miss. Without a collapsing mechanism, all of those hundreds run the heavy query against the DB. PostgreSQL takes a burst of load, the connection pool drains, latency climbs, and then other hot keys start expiring too. This is cache stampede, also known as the thundering herd.

The fix is to collapse duplicate misses on the same key into one: only the first request goes down to the DB, and the rest wait on that exact promise.

@Injectable()
export class CacheService {
  // Map of "in-flight" loads — lives in process memory
  private readonly inflight = new Map<string, Promise<unknown>>();

  async wrap<T>(key: string, ttlSec: number, loader: () => Promise<T>): Promise<T> {
    const cached = await this.redis.get(key).catch(() => null); // Redis error -> treat as a miss
    if (cached) return JSON.parse(cached) as T;

    // Another request is already loading this exact key -> ride along, do not hit the DB
    const flying = this.inflight.get(key);
    if (flying) return flying as Promise<T>;

    const task = loader()
      .then(async (value) => {
        // set is best-effort: a dead Redis must not break the response
        await this.redis
          .set(key, JSON.stringify(value), 'EX', ttlSec)
          .catch(() => undefined);
        return value;
      })
      .finally(() => this.inflight.delete(key));

    this.inflight.set(key, task);
    return task;
  }
}

The limit is worth stating plainly: inflight is a Map in process memory, so it only collapses cache misses within a single API instance. With four replicas, when the key expires, the worst case is still four queries down to the DB instead of one. But four is a very long way from four hundred. For us that reduction is good enough, and it buys a simpler system: no distributed lock, no deadlock risk, no scenario where a process dies while holding the lock and leaves the others waiting forever. A distributed lock on Redis could collapse this down to exactly one query, but the price is a new layer of operational complexity. We take the simple solution and only upgrade when the numbers force us to.

4. Layer 3 — Version-based invalidation, and why KEYS/SCAN is a trap

Here is the problem: an admin edits a course description. Dozens of cache keys relate to that center's public content — the catalog, the landing page, the course detail, the list of tests. How do you delete all of them?

The first answer most people reach for is a pattern scan: KEYS zedu:pub:*, then delete. Do not do that. KEYS is an O(N) command running on Redis's single thread: it walks the entire keyspace, and during that time Redis serves almost nothing else. With a few million keys, one admin editing one course description can push other commands into timeouts. SCAN is better because it breaks the walk into batches, but it still traverses the keyspace, costs many round trips, and gives you no clear completion point. Using SCAN to purge cache turns what should be an instant operation into an unreliable background job.

What we do instead is never delete the old keys directly. Instead, we embed a version number in the cache key itself. When a whole "region" needs invalidation, we just increment that number: every old key instantly becomes a key nobody asks for anymore, and it dies on its own when the TTL expires.

@Injectable()
export class CacheVersionService {
  // in-process micro-cache: avoids one Redis round trip on EVERY request
  private readonly local = new Map<string, { v: number; exp: number }>();
  private static readonly MICRO_TTL_MS = 2000; // <= 2s

  async get(scope: string): Promise<number> {
    const now = Date.now();
    const hit = this.local.get(scope);
    if (hit && hit.exp > now) return hit.v;

    // Redis down -> v = 0. The cache keeps working, it just misses bump commands.
    const raw = await this.redis.get(`ver:${scope}`).catch(() => null);
    const v = Number(raw ?? 0) || 0;
    this.local.set(scope, { v, exp: now + CacheVersionService.MICRO_TTL_MS });
    return v;
  }

  async bump(scope: string): Promise<void> {
    await this.redis.incr(`ver:${scope}`).catch(() => undefined);
    this.local.delete(scope); // this instance sees it now; the others within <= 2s
  }
}

// The cache key carries the version, scoped by region + tenant
async function publicCourseList(tenantId: string) {
  const v = await this.versions.get(`pub:${tenantId}`);
  return this.cache.wrap(
    `zedu:v${v}:pub:${tenantId}:courses`,
    300,
    () => this.prisma.course.findMany({ where: { tenantId, published: true } }),
  );
}

// Admin edits public content -> the tenant's whole pub region becomes garbage
async function onPublicContentChanged(tenantId: string) {
  await this.versions.bump(`pub:${tenantId}`);
}

A few things are worth calling out. First, the version is scoped per tenant, for example pub:${tenantId}, so one center editing content does not cool down the caches of the other centers. In multi-tenant SaaS, if an invalidation region spans the whole system, one customer editing a lot of content can drag down performance for everyone else.

Second, bump is a single INCR — O(1), non-blocking, indifferent to how big the keyspace is. The cost of invalidation no longer depends on the number of keys to delete. That is exactly why we chose this approach.

Third, the micro-cache. If every request had to ask Redis "what is the current version?", we would have added a round trip back onto the hot path, the very thing the cache exists to remove. So the version is held in process memory for at most 2 seconds. The trade-off is that after an admin hits Save, another instance may still serve the old version for up to 2 seconds. For the public content of an LMS that is an acceptable delay, and it gives us a clear promise: an edit shows up in no more than 2 seconds.

Finally, the real cost of the version model: old keys do not disappear immediately, they sit in Redis until the TTL expires. You trade a little memory for O(1) invalidation, with no keyspace scan and no stalling of Redis. If maxmemory-policy is set to allkeys-lru, that garbage can even get evicted on its own under memory pressure. For us this is a reasonable trade-off.

5. When Redis dies: fail-open

Notice a detail that repeats throughout all the code above: every Redis call has an error branch. That is not sloppiness, it is an architectural decision.

Redis in Z-EDU is not the source of data. The source of truth is PostgreSQL. Redis is only an optimization layer. So when Redis fails, the correct behavior is not to return a 500 to the user, but to fail-open: everything goes straight to the DB. The system gets slower and the DB works harder, but the results stay correct. Students can still view lessons and still submit exams. We apply the same principle to rate limiting: the per-IP counters live in Redis, and if Redis fails the rate limit fails open rather than rejecting every request.

A cache is an optimization, not the real source. If your system stops working when the cache dies, what you have is not a cache — it is a database without a backup.

This principle has one very important precondition: the DB must survive the load without the cache. Fail-open only means something if PostgreSQL can still take the full traffic, however slowly, when Redis disappears. If you are caching because the DB is already overloaded, fail-open just converts a Redis incident into a PostgreSQL incident. The test is simple and you should run it before production runs it for you: turn Redis off in staging under production-like load and see whether the system stays up. If it does not, the cache has quietly become a mandatory component; at that point fix the DB first, do not just pile on more cache.

Here layer one saves us once again. Because Cache-Control pushes most of the public read traffic out to the edge, the CDN keeps serving cached content even while both Redis and the API are struggling. Three independent layers means they fail independently — and that is the deeper reason to keep them apart instead of collapsing everything into one place.

6. The checklist we ended up with

• Measure before you cache. If the bottleneck is the number of API workers rather than the DB, adding Redis will only make you feel better, it will not save you. Edge caching will.
• Edge caching must be opted into per route. Never infer a cache policy from another decorator (such as @Public). The default is no-store.
• Ask "is this response the same for everyone?" before you type public. If the answer is "no" or "not sure", use private or no-store.
• Everything that affects the response body must be in the cache key — through the URL, the hostname, or Vary. In a multi-tenant system, put the tenant in the path or the hostname; do not bet on the Vary configuration always being right.
• Every hot key must go through wrap (single-flight). A hot key with no stampede protection is a time bomb set to go off exactly when the TTL expires.
• Invalidate with versions, not with KEYS/SCAN. INCR is O(1); scanning the keyspace is not, and KEYS also blocks all of Redis. Scope versions per tenant so one tenant does not cool down another tenant's cache.
• Fail-open, and rehearse it. Every Redis call has a fallback branch down to the DB. Turn Redis off in staging under real load — if the system goes down, your cache has quietly become the source of truth.

Conclusion

Three layers, three separate jobs. Cache-Control carries the largest share of the read load, but it is also the layer that demands the most care, because a response already sent cannot be taken back. Redis with wrap shields PostgreSQL and prevents many requests from slamming the DB at once when the cache expires. The version key allows near-instant invalidation without scanning the keyspace. Underneath all of it lies a single principle: PostgreSQL is the source of truth, and everything else only makes the system faster.

The attempts/:id incident is the one we keep retelling to new hires. It shows that caching is not only a performance problem; put one header in the wrong place and it becomes a security problem. In multi-tenant SaaS, the distance between "fast" and "serving one person's data to another" is sometimes a single word: public.