Building scalable event-driven architectures

In an event-driven architecture, a service records that something happened and publishes that fact. Other services react in their own time. The producer does not know who is listening and does not wait for them. That decoupling is the source of the scalability: consumers can be added, scaled and taken offline without changing the producer, and a burst of load is absorbed by the log instead of cascading through a chain of synchronous calls.

The cost is that correctness no longer comes from a single transaction. It has to be built from delivery guarantees, idempotency and explicit handling of ordering. Our view is that event-driven design is worth it where those properties are engineered deliberately, and a liability where a broker is added to a system that still assumes immediate consistency.

An event is a statement in the past tense about something the producer owns: OrderPlaced, PaymentCaptured. A command is a request for another service to act. Mixing the two produces events such as SendWelcomeEmail, which couple the producer to a specific consumer and remove the benefit of publishing.

Payload design is a trade-off. A thin notification carrying only an identifier forces every consumer to call back for details, which recreates synchronous coupling and load. Event-carried state transfer includes the relevant data so consumers can build their own read models, at the price of larger messages and a wider contract to maintain. We default to carrying the state a consumer needs to act, and never the producer's internal table structure.

Once an event is published, unknown consumers depend on it and old events remain in the log, so the schema is a public contract. A schema registry with compatibility checks in the build pipeline, using Avro, Protobuf or JSON Schema, prevents a producer from shipping a change that breaks readers. The safe changes are additive: new optional fields with defaults. Renaming or removing a field, or changing its meaning, calls for a new event type and a migration period in which both are published.

A service that commits to its database and then publishes to a broker performs two writes with no shared transaction. If the process fails between them, the state changes and the event is never sent, or the event is sent for a change that rolled back. This is the most common source of silent inconsistency in event-driven systems.

The transactional outbox pattern removes it. The service writes the business change and an outbox row in the same database transaction. A separate relay publishes outbox rows to the broker, either by polling or by change data capture from the database log with a tool such as Debezium. The event is then published if and only if the change committed, at least once.

Brokers, relays and consumers all retry, so duplicates are normal. Exactly-once semantics in Kafka apply to reads and writes within Kafka transactions; they do not extend to a consumer that calls an external API or writes to another database. The dependable approach is to make consumers idempotent.

A partitioned log such as Kafka guarantees order only within a partition. Events for the same entity must share a partition key, usually the aggregate identifier, or a consumer may see OrderCancelled before OrderPlaced. Global ordering would require a single partition, which caps throughput at one consumer.

Partition keys also determine how load spreads. A key with a skewed distribution, such as one very large tenant, creates a hot partition that lags while the others sit idle. Retries complicate ordering further: if a failed message is set aside and later messages for the same key continue, the consumer has reordered them. Decide per topic whether to block the partition on failure or to accept reordering and rely on version checks.

A consumer needs a policy for messages it cannot process, decided before the first one arrives.

The key health signal is consumer lag: how far each consumer group is behind the head of the log, measured in messages and in time. Lag that keeps growing means consumers are under-provisioned or stuck. Scale consumers on lag, bounded by the partition count, and trace events end to end by propagating a correlation identifier in headers.

A business process that spans services cannot use a distributed transaction without giving up the independence that motivated the design. The saga pattern replaces it with a sequence of local transactions, each with a compensating action that reverses it on failure. Choreography, where services react to each other's events, suits short flows. Beyond three or four steps the process becomes hard to see, and an orchestrator that holds the state machine explicitly is easier to reason about, test and operate.

Events are the wrong tool when the caller needs an answer before it can continue, such as validating a card or checking stock at the moment of purchase. They are also poor value for a small system with one team and one database, where a transaction provides stronger guarantees with far less machinery.

Eventual consistency is a product decision as well as a technical one. If users cannot tolerate reading stale data immediately after a write, either keep that path synchronous or design the interface to reflect pending state. Most well-built systems are mixed: synchronous at the edge where a user is waiting, asynchronous behind it where work can be deferred.