Building production-grade SaaS infrastructure

A SaaS platform is one system operated for many customers who must never see, slow down or break each other. Adding a tenant_id column gets an application to a demonstration. Operating it for paying customers requires answers to harder questions: how tenants are isolated, how one tenant's load is contained, how schema changes ship without downtime, and how a single customer's data is restored.

Our default is a pooled architecture with isolation enforced in the database and at the platform edge, plus the option to place specific tenants on dedicated infrastructure. It keeps the cost and operational effort of a shared system while leaving a path for customers whose requirements justify their own stack.

Tenancy is usually described as one choice between silo, where each tenant has dedicated resources, and pool, where all tenants share them. In practice the choice is made separately for compute, database, cache, queue and storage, and a mixed result is normal.

In a pooled model, a missing WHERE clause is a data breach. Relying on every developer to remember the filter in every query is not a control. We resolve the tenant once, at the edge, from the authenticated token and never from a request parameter or header the client controls, and then enforce it below the application code.

PostgreSQL row-level security is the mechanism we favour. The application sets the tenant for the transaction, and policies on each table restrict every statement to that tenant's rows, including queries someone forgot to filter. The application's database role must not own the tables or carry the BYPASSRLS attribute, or the policies are silently skipped. With a transaction-mode connection pooler, the tenant setting has to be scoped to the transaction so that it cannot leak to the next request on the same connection.

The same context has to reach everything else that holds data: cache keys prefixed by tenant, object storage paths with scoped access, queue messages that carry the tenant, search indexes that filter on it, and log lines that record it.

Shared infrastructure means shared capacity. One tenant running a large import or a badly written integration can exhaust database connections or fill a queue, and every other tenant experiences an outage they did nothing to cause. A global rate limit does not help, because it protects the platform by penalising everyone equally.

The application plane is the product that tenants use. The control plane is the system that manages tenants: onboarding and provisioning, identity configuration, plans and entitlements, metering for billing, placement of a tenant on pooled or dedicated infrastructure, and the operational tools used by support staff.

Keeping them apart has two benefits. Onboarding becomes an automated, repeatable workflow instead of a checklist, which matters as soon as a silo tenant needs infrastructure created for it. The application plane can also be built to keep serving when the control plane is unavailable, by caching the tenant configuration and entitlements it needs. A billing or provisioning fault should stop new sign-ups, not existing customers. A product with a handful of tenants can defer most of this, provided tenant metadata already sits in one place and not across configuration files.

A shared database cannot be taken offline for a schema change, and during a rolling deployment the old and new versions of the application run against the same schema at once. Every migration therefore has to be compatible with the version before it. The pattern is expand and contract: add the new structure, write to both, backfill, switch reads, and remove the old structure in a later release.

The details matter on large tables. In PostgreSQL an index is created with CREATE INDEX CONCURRENTLY, a constraint is added as NOT VALID and validated separately, and a lock timeout is set so that a migration waiting behind a long transaction fails fast instead of blocking every query queued behind it. Backfills run in small batches with pauses, watching replication lag. Renaming a column in place is never a single step.

This takes more releases than a direct change. For an early product with a maintenance window its customers accept, a brief planned outage is a legitimate trade for speed. Once customers span time zones, that option disappears.

Point-in-time recovery on a pooled database restores every tenant to the same moment. That is the right tool for a platform-wide disaster and the wrong one for the common request: one customer deleted something important and wants it back. Rolling everyone else back is not an option.

The capability has to be designed. A typical approach restores the backup to a separate instance, extracts the rows for the one tenant, and merges them into production with a tool that understands the data model and its foreign keys. Soft deletion with a retention period handles many cases before a restore is needed. The same tenant-scoped export is what makes offboarding and verifiable deletion possible. Restores should be rehearsed on a schedule, because an untested backup is an assumption.

With many customers on one deployment, a bad release reaches all of them at once. We prefer progressive delivery: feature flags evaluated per tenant, a first ring of internal and low-risk tenants, and automated rollback when error budget burn rises. The cost is flag hygiene, since stale flags accumulate into untested combinations, so each flag needs an owner and a removal date.