Multi-tenant SaaS security: where data actually leaks

When people talk about SaaS security, they usually picture an attacker probing for vulnerabilities. But in the multi-tenant systems we operate (many businesses sharing one system, with fully isolated data), the near-leaks almost always come from very ordinary things: a developer who writes correct business logic but forgets the WHERE tenantId condition, a route that gets caching turned on "to make it fast," or a permission check that lives only in the browser. No sophisticated attack technique is involved. The problem is that the system's defaults are not safe enough.

This article walks through the places multi-tenant data leaks most easily: authorization, cache, uploads, the client clock, rate-limit, and identity. Every one of them involves a trade-off. We will state those trade-offs plainly rather than promise "absolute safety," because that does not exist in real operations.

1. Many businesses: every control must live on the server

Z-Auto is a car showroom management platform. Each showroom is a tenant, with its own vehicle inventory, CRM customers, deposit contracts, and test-drive schedule. Z-Cloud Workspace is also a multi-tenant system, made up of 11 applications sharing a single identity layer. In both systems, a single query missing its tenant condition is enough for showroom A's data to show up on showroom B's screen.

The three most common sources of leaks we run into are:

• Queries missing tenantId — usually on a new endpoint, written in a hurry, copied from an endpoint that did not need a tenant filter.
• Wrongly cached routes — private data held by a proxy and served to someone else. We tell a real incident in section 3.
• Permission checks in the frontend — the button is hidden, but the API still answers anyone who calls it with curl.

The browser is the user's environment, not yours. Everything that runs there — permission checks, file type checks, countdown timers — is a hint about the experience, not a security control. A control only exists when it runs on a machine you control.

The rule we repeat in every code review: if a decision affects money, scores, access rights, or another tenant's data, that decision must be recomputed on the server, whether or not the client already computed it.

2. Authorization: a single source of truth

In Z-EDU, the entire permission list lives in packages/config/src/permissions.ts: one file shared by the NestJS backend and the Next.js frontend inside a pnpm monorepo. No permission constant is declared anywhere else. The reason is pragmatic: when permissions are defined in two places, sooner or later they drift apart. And when they drift, the error tends to widen access rather than narrow it.

The backend blocks with the @RequirePermission and @PermissionScope decorators. The guard reads the metadata, compares it against the user's actual permissions taken from the JWT/DB, and most importantly: it constrains the data scope to the tenant.

// permissions.ts — the single source of truth, shared by both BE and FE
export const PERMISSIONS = {
  COURSE_UPDATE: 'course:update',
  ATTEMPT_READ:  'attempt:read',
} as const;

// courses.controller.ts
@Controller('courses')
export class CoursesController {
  @Patch(':id')
  @RequirePermission(PERMISSIONS.COURSE_UPDATE)
  @PermissionScope('tenant')            // the guard requires a tenantId in the context
  async update(
    @CurrentUser() user: AuthUser,      // user.tenantId comes from the token, NOT from the body
    @Param('id') id: string,
    @Body() dto: UpdateCourseDto,
  ) {
    // The tenantId condition sits inside the where clause, not in a check after reading.
    // updateMany + count = 0 -> does not exist OR does not belong to this tenant: both return 404.
    const res = await this.prisma.course.updateMany({
      where: { id, tenantId: user.tenantId },
      data: dto,
    });
    if (res.count === 0) throw new NotFoundException();
    return this.prisma.course.findFirst({ where: { id, tenantId: user.tenantId } });
  }
}

Two details are worth noting. First, tenantId comes from the authenticated token, never from the body or the query string — if the client can send a tenantId, the client can pick a tenant. Second, when a record does not belong to the current tenant, we return 404, not 403. Returning 403 inadvertently confirms "this ID exists, you just cannot see it" — a small leak channel, but enough to enumerate IDs.

The frontend still receives the permission list, but only uses it to show or hide menus and buttons. That is experience, not security. Hiding the "Delete contract" button keeps users from clicking it by mistake; it does not stop anyone from opening DevTools, reading the JavaScript bundle, finding the endpoint, and calling it directly. If the API does not block on its own, a hidden button protects nothing.

3. Cache: where data leaks without anyone touching the code

This is one of the incidents we remember best. It did not come from business logic, it came from cache configuration.

Z-EDU has three cache layers. The outermost one is Cache-Control, set through the @HttpCache decorator on public routes, so the CDN/Traefik answers directly and the request never even reaches the API. This is the main load-absorbing layer, and precisely because it works so well, we once intended to switch it on quickly for every @Public route.

The problem is that @Public in our system only means "a JWT is not required," it does not mean "everyone sees the same thing." Some @Public routes still return one person's private data, protected by a token in the URL or by the context of an exam session. The textbook example is GET /exams/public/attempts/:id, that is, one candidate's attempt. If that route were tagged with Cache-Control: public, a proxy could hold the response and serve it to the next person hitting the same URL. The data leaks even though the business code is correct.

// WRONG: sweep every @Public route and slap caching on all of them
@Public()
@HttpCache({ public: true, maxAge: 60 })   // <-- a disaster if the route returns personal data
@Get('exams/public/attempts/:id')
getAttempt(@Param('id') id: string) { /* ONE candidate's attempt */ }


// RIGHT: opt in per route, and draw a clear line between public and private
@Public()
@HttpCache({ public: true, maxAge: 300 })  // content that looks the same to everyone
@Get('exams/public/:slug')
getExam(@Param('slug') slug: string) { /* a public exam */ }

@Public()
@NoStore()                                 // Cache-Control: no-store, private
@Get('exams/public/attempts/:id')
getAttempt(@Param('id') id: string) { /* default: NO caching */ }

Two concepts must be kept strictly apart:

• Cache-Control: public — any shared cache (CDN, reverse proxy) is allowed to store the response and replay it to someone else. Use it only for data that every anonymous visitor sees identically.
• Cache-Control: private — only that user's browser cache may store it. The CDN must not keep it. Use it for data tied to one person.
• no-store — stored nowhere at all. Use it for genuinely sensitive material.

Vary is a supporting tool: it tells the cache which header the response depends on (for example Vary: Authorization, Vary: Accept-Language), so two requests differing in that header must become two different cache entries. But do not treat Vary as a safety net. It is easy to forget, and if the data is distinguished by a token in the URL then Vary saves nothing. The more durable rule is no caching by default, opt in one route at a time, and whoever adds @HttpCache must be able to answer the question: "is this response identical for every anonymous visitor?" If there is any hesitation, do not cache.

The trade-off here is obvious: opting in per route means you miss a few caching opportunities and take on extra load. We accept that trade-off, because the price of the opposite direction is serving one candidate's exam attempt to another candidate.

4. File uploads: do not trust the client's Content-Type

In Z-EDU, the browser PUTs files straight to S3 (RustFS, S3-compatible) through a presigned URL. The API is not in the byte path — it only signs the URL and, more importantly, re-checks at the registration step.

Why not trust the Content-Type the client sends? Because it is just a string in an HTTP header, declared by the client itself. Anyone can send Content-Type: image/png with an executable as the payload, or declare a small Content-Length at registration time and then upload a 5GB file. Size and file type only mean something when they are read from the object already sitting on storage, that is, from a source the client cannot rewrite. That is what HeadObject is for.

@Post('uploads/presign')
@RequirePermission(PERMISSIONS.FILE_UPLOAD)
async presign(@CurrentUser() user: AuthUser, @Body() dto: PresignDto) {
  const key = `t/${user.tenantId}/${randomUUID()}`;   // the key always carries the tenant prefix
  const url = await this.s3.getSignedUrl('putObject', {
    Key: key, Expires: 300,                            // the signed URL lives for 5 minutes
  });
  return { key, url };
}

@Post('uploads/commit')
@RequirePermission(PERMISSIONS.FILE_UPLOAD)
async commit(@CurrentUser() user: AuthUser, @Body() dto: CommitDto) {
  if (!dto.key.startsWith(`t/${user.tenantId}/`)) throw new ForbiddenException();

  // The truth lives here: read the metadata FROM STORAGE, not from what the client declared.
  const head = await this.s3.headObject({ Key: dto.key });

  if (head.ContentLength > MAX_SIZE)                  throw new BadRequestException('File too large');
  if (!ALLOWED_TYPES.includes(head.ContentType ?? '')) throw new BadRequestException('Invalid file type');

  return this.files.register({ key: dto.key, tenantId: user.tenantId, size: head.ContentLength });
}

If the client lies, the object may still land on S3, but it will not be registered in the DB. It is just junk, and the periodic cleanup job deletes it later. The attacker burns their own bandwidth, and the system never grants the right to use the file.

The download direction works the same way: the bucket is not public. Files are served through a short-lived signed URL, issued after the API has checked that the user has read permission on that file and that the file belongs to the right tenant. A short-lived URL is not perfect security — while it is still valid, anyone holding the URL can download the file. The trade-off is in the lifetime: too short and the link dies mid-transfer on a slow network, too long and the leak window widens. A few minutes is a reasonable balance for most cases.

One caveat: HeadObject reads the ContentType that S3 stored, and that value still originates from the upload. It stops the lie at the registration step, but it is not yet a check of the file's actual content. For more sensitive data, the next layer should be reading magic bytes or virus scanning in a worker. This is defense proportional to risk, not a single on/off switch.

5. Do not trust the client's clock

In Z-EDU, when a candidate starts a test, the server fixes expiresAt at that very moment and stores it on the attempt record. The client receives that timestamp and does exactly one thing with it: count down to it for the user to see. On submission, the server compares the time the request arrived against the stored expiresAt — it does not read any time field the client sends up. Submissions more than 60 seconds late are rejected.

Why the 60-second grace period? Because the clock on a client machine can drift, the network can be slow, and the request still travels through several proxy layers. If we rejected strictly by the millisecond, an honest candidate on a poor connection could lose their work. The 60-second number is a deliberate trade-off: accept a small window that could be abused in order to avoid punishing real users unfairly. What matters most is that this decision lives on the server, bounded and measurable, rather than depending on the browser's Date.now().

More broadly: every timestamp in the system is stored in UTC, converted to UTC+7 only for display and when generating class schedules or closing reporting periods. Mixing time zones in the storage layer is a slow-burning source of bugs, and in business logic that computes deadlines or closes the books, a time zone bug has consequences no different from a security hole.

The general rule: anything that decides money, scores, or permissions is computed on the server. The client is allowed to know the outcome; it is not allowed to decide the outcome.

6. Rate-limit and the fail-open philosophy

Z-EDU rate-limits by IP and counts through Redis. When Redis has an incident, we choose fail-open: the request goes through instead of being blocked. The reason is that the rate limit here exists to reduce abuse, not to authenticate identity. If it were fail-closed, a failure in a supporting layer could make the whole API turn away real users. The same philosophy shows up elsewhere in Z-EDU: if Redis fails, the cache falls straight through to the DB (slower, but correct), and progress writes switch to writing straight to the DB (you may lose a few seconds of video position, but you do not lose scores).

async consume(ip: string): Promise<boolean> {
  try {
    const n = await this.redis.incr(`rl:${ip}`);
    if (n === 1) await this.redis.expire(`rl:${ip}`, WINDOW_SEC);
    return n <= LIMIT;
  } catch (err) {
    // Redis is down -> FAIL-OPEN. Do not take the API down over a supporting layer.
    this.logger.error({ err }, 'rate-limit backend down, failing open');
    this.metrics.increment('ratelimit.fail_open');   // this metric must have an alert on it
    return true;
  }
}

But fail-open is a conscious trade-off, not a harmless default. It trades away part of your abuse resistance for availability. While Redis is broken, someone can send more requests than usual. We accept that because the price of the API refusing to serve an entire tenant is higher. The counterweight is to count how often you fail open and build an alert on that metric. It is the silent fail-open that is dangerous.

The boundary is very clear: fail-open for mechanisms that protect performance, fail-closed for mechanisms that decide permissions. If the permission-checking service cannot answer, the answer must be "deny," not "let it through." If the payment gateway cannot confirm a transaction, the order must stay pending and must not be marked paid. If a JWT cannot be verified because the public key could not be fetched, the request must be rejected. The question that classifies it: "If this component goes silent, which is worse — wrongly blocking a real user, or wrongly letting through someone with no permission?" The answer decides the fail direction.

7. Centralized identity: lock in one place, cut off every right

Z-Cloud Workspace has 11 integrated applications — chat, mail, docs, files, tasks, calendar, meetings, base, workflow, board, contacts. If each application managed its own accounts, then when an employee leaves, the IT team would have to remember to lock the account in 11 places. In practice they will forget at least one — and the one they forget is usually the one with the most sensitive data.

We solve this with SSO through Keycloak/OIDC: one login for every application and, more importantly, a single place to revoke. Disabling the account in Keycloak cuts off access across all 11 apps. This is the benefit people usually overlook when talking about SSO — they tend to mention the convenience of "log in once," while the more important security value lies in "revoke once."

The trade-off is that centralized identity creates a shared dependency point. If Keycloak has an incident, new users cannot log in. On the other hand, active sessions keep running until their token expires. That makes token lifetime a parameter worth thinking through carefully: long tokens mean revocation is delayed, short tokens mean the system depends more on the IdP's availability. There is no free option, only the option that fits your risk model.

Over on Z-Auto, each showroom has its own storefront on a subdomain, or points its own domain at it (a custom domain). This adds one more surface to be careful about: the tenant is determined from the host. Resolving a host into a tenant must be done on the server and must look the host up in the list of registered domains, rather than inferring it by slicing the string. And once you have a tenantId from the host for the public pages, administrative data must still take its tenantId from the logged-in user's token — the two sources must not be mixed. The CDN/WAF layer sitting in front of the API Gateway (OpenResty) in Z-Cloud Workspace plays a similar role: it is a filtering layer, not an authorization layer.

Finally, chat in Z-Cloud Workspace is end-to-end encrypted (E2E) — meaning that even we, as the operator, cannot read the content. This is a design choice with a price of its own: you lose server-side search, and you lose the ability to recover when a user loses their key. We accept that price for the chat channel because the privacy of internal conversations is worth more.

8. A multi-tenant security checklist

This is the list we use when reviewing a new endpoint in a multi-tenant system. It is deliberately kept short, because a checklist that is too long usually does not get read all the way through.

• tenantId always comes from the authenticated token, never from a body, query, or header the client controls.
• The tenant condition sits inside the where clause, not in a check performed after the record has been read out. Return 404 instead of 403 so you do not confirm that an ID exists.
• No caching by default; @HttpCache is opt-in per route. Only set public when the response is identical for every anonymous visitor. A route that returns personal data — even a @Public route — is always private or no-store.
• One source of truth for permissions (such as permissions.ts), the backend blocks with a guard; the frontend only shows and hides UI.
• Do not trust metadata the client declares: re-check file size and type with HeadObject after the upload; object keys carry the tenant prefix; serve downloads through a short-lived signed URL.
• Do not trust the client's clock: expiry timestamps are fixed on the server and stored in UTC; every computation of money, scores, or permissions is redone on the server.
• Classify the fail direction correctly: rate-limit and cache fail-open (with a metric and an alert); authorization, authentication, and payment fail-closed.
• Centralized identity: one place to revoke. Leaving the organization means one lock, and every application is cut off.
• Write tests for the attack path, not just the happy path: every endpoint should have at least one test that uses tenant A's token against tenant B's resource and expects a 404.

Conclusion

No system is absolutely safe. The more realistic thing we pursue is this: make the safe choice the default. Caching is off out of the box and must be turned on by hand. The guard requires the scope to be declared. File metadata is always re-read from storage. Timestamps are always fixed on the server.

Because a data leak in multi-tenant SaaS usually does not come from a brilliant attack. It usually comes from a good developer in a hurry, at the end of the working day. The system's safe defaults are what still protects you at moments like that.