Designing systems for millions of requests

Serving millions of requests is less about raw throughput than about behaviour at the limits. A stateless service behind a load balancer scales by adding instances. What takes design effort is the state behind it, the latency of the slowest requests, and the way the system degrades when more work arrives than it can complete.

Our position is that overload behaviour should be designed before optimisation begins. A system that sheds excess load and stays responsive at its ceiling is more valuable than one with a higher ceiling that collapses when it is reached.

One million requests a day is an average of about twelve per second, which a single well-configured server handles comfortably. The number that drives design is the peak, not the daily total, together with the ratio of reads to writes and the cost of each request. A system receiving a thousand requests per second of cached reads is a simpler problem than one receiving fifty per second of transactional writes that each touch several tables.

Writing these figures down, even as estimates, prevents the two common errors: building a distributed architecture for a load that one database would carry, and discovering late that the write path has a hard limit nobody measured.

Application instances should hold no state that another instance would need. Sessions live in a shared store or in a signed token, uploads go to object storage, and scheduled work is coordinated through a lock or a queue instead of running on whichever node happens to start it. Any instance can then serve any request, and instances can be added, removed or replaced during a deployment without users noticing.

Statelessness moves the problem; it does not remove it. The state now sits in the database, the cache and the queue, and those are the components that determine how far the system scales.

Caching works in layers, each removing load from the one behind it: a CDN for static assets and cacheable responses, an in-process cache for hot reference data, and a distributed cache such as Redis for shared computed results. The closer to the user a response is served, the less infrastructure it consumes.

Every layer introduces a consistency question and a failure mode. The one that causes outages is the stampede: a popular key expires, hundreds of requests miss at once, and all of them run the same expensive query against the database.

Read replicas are the first step for a read-heavy workload. They are also asynchronous, so a user who writes and immediately reads from a replica may not see their own change. The usual remedies are to route reads that follow a write to the primary for a short period, or to track the replication position the session requires.

Replicas do nothing for write volume. When a single primary can no longer absorb the writes or hold the data, the options are partitioning within one database or sharding across several. The partition key is the decision that is hard to reverse. A good key spreads load evenly and keeps the queries that matter inside one partition. A poor key creates hot partitions, where one large customer or one busy time range receives most of the traffic, and cross-partition queries that have to fan out and merge.

Sharding should be late in the sequence. Indexing, query tuning, connection pooling, archiving cold data and a larger instance are all cheaper and reversible. We would move to sharding when vertical headroom is running out, not as an early statement of intent.

Every system has a maximum rate of work. The design question is what happens to requests beyond it. With no limits, they accumulate in thread pools, connection pools and unbounded queues, latency rises for everyone, clients time out and retry, and the retries add load to a system that is already failing. That feedback loop is how a brief spike becomes a prolonged outage.

Backpressure breaks the loop by making limits explicit and propagating them upstream. Queues are bounded. Every outbound call has a timeout shorter than the caller's own deadline. Retries use exponential backoff with jitter and a retry budget, so they cannot multiply traffic. Circuit breakers stop calls to a dependency that is clearly failing.

A queue smooths bursts; it does not add capacity. If arrivals exceed the processing rate for long enough, the queue only delays the failure and serves stale work when it recovers. When the limit is reached, the system should reject early and cheaply with a 429 or 503 and a Retry-After header, and it should shed by priority: a payment confirmation is kept, a recommendations panel is dropped.

Averages mislead once a request fans out. If a page calls one hundred backends in parallel and each has a one per cent chance of being slow, roughly 63 per cent of page loads will include at least one slow call. The user experiences the slowest dependency, so the p99 of each service becomes the typical experience of the whole.

Capacity is a measured quantity. We load-test with a realistic mix of operations and data volumes, using an open workload model in which requests arrive at a fixed rate whatever the response time. A closed model, where each virtual user waits for a response before sending the next request, slows itself down as the system struggles and hides the very latency it should reveal.

The test should continue past the knee of the curve until something breaks, because the purpose is to learn which component fails first and whether the system recovers unaided when load falls. A system that needs a manual restart after overload has a defect worth fixing before launch.