Backend systems that stay correct under load and partial failure

The dual-write problem appears whenever a service updates its database and publishes a message. If the two operations are separate, a crash between them leaves the database changed and the event unpublished, or the reverse. The transactional outbox removes the gap: the event is inserted into an outbox table in the same local transaction as the state change, and a relay, commonly change data capture with Debezium, publishes it to Kafka afterwards. Delivery becomes at least once, which is why consumers must deduplicate.

A business process that spans services, such as reserve stock, take payment and create shipment, cannot hold one ACID transaction across them. A saga models it as a sequence of local transactions, each with a compensating action that semantically undoes it. Orchestration puts the sequence in one coordinator, built on a workflow engine such as Temporal or a state machine in the database, which makes the process visible and testable. Choreography lets services react to each other's events, which reduces coupling for short flows and becomes hard to reason about for long ones.

Idempotency makes retries safe. The client sends a unique key with each mutating request; the server stores the key with the result inside the same transaction as the effect and returns the stored result when the key is seen again. Combined with deadlines propagated through the call chain and a retry budget that prevents retry storms, this lets the system recover from timeouts without duplicating side effects.

A synchronous call couples the availability and latency of the caller to the callee: if a request path crosses five services, its availability is the product of all five. It remains the right choice when the caller needs the answer to continue, for example a balance check before authorising a payment. Everything that can happen after the response, such as notifications, projections, search indexing and analytics, is better driven by events.

Queues and streams differ in what they retain. A queue hands each message to one consumer and removes it on acknowledgement, which suits task distribution with per-message retry. A Kafka topic retains the log for a configured period regardless of consumption, so new consumers can replay history, several consumer groups can read independently and ordering is guaranteed within a partition. The partition key therefore determines both ordering and parallelism, and choosing it is a design decision, typically the aggregate identifier such as account or order.

Eventual consistency is a product decision as much as a technical one. Read models updated from events lag the write side, so the interface must either read its own writes from the source, show pending state or tolerate the delay. We document the expected lag for each projection and alert when consumer lag exceeds it.

A cache is a second copy of data with its own consistency problem. Cache-aside with a time to live is the simplest correct pattern: read from Redis, fall back to the database on a miss, write the result with an expiry, and delete the key when the underlying row changes. Deleting on write is safer than updating, because concurrent writers can otherwise leave the older value in place.

The failure modes are predictable and need explicit handling. A stampede occurs when a popular key expires and many requests recompute it at once, addressed with request coalescing, probabilistic early refresh or a short lock. Hot keys concentrate load on one shard, and an unbounded key space exhausts memory unless eviction policy and key cardinality are planned. The service must also remain correct, if slower, when the cache is unavailable.