Kafka and event-driven architecture

Kafka is often introduced as a message queue, which leads teams to expect queue behaviour and then to be surprised. It is closer to a distributed commit log: producers append records to partitions, records stay there for a configured period whether or not anyone has read them, and each consumer tracks its own position.

We use Kafka where that model is what the system needs: several independent consumers of the same stream, the ability to replay history, and ordered processing per entity at high volume. Where those needs are absent, we argue against it. The sections below describe the mechanisms that decide whether an event-driven design holds up in production.

Kafka guarantees order within a partition and nowhere else. The record key decides the partition, so choosing the key is choosing the unit of ordering. Keying by account identifier means all events for one account are consumed in the order they were produced, while events for different accounts are processed in parallel.

Two consequences follow. A skewed key, such as one very large tenant, creates a hot partition that a single consumer must work through alone. And increasing the partition count later changes which partition a key maps to, so ordering for existing keys is broken across the change. We size partitions with growth in mind and treat an increase as a migration, not a setting.

Consumers in the same group divide a topic's partitions between them, and each partition is owned by one member at a time. Parallelism is therefore capped by the partition count; extra consumers sit idle. Different groups read the same topic independently, which is what allows a new service to consume an existing stream without touching the producers.

When a member joins, leaves or stops polling, the group rebalances and partitions move. A consumer that takes longer than max.poll.interval.ms to process a batch is considered dead and removed, its partitions are reassigned, and its uncommitted work is processed again elsewhere. Slow handlers therefore cause rebalance loops that look like an outage. Cooperative incremental rebalancing and static group membership reduce the disruption during deployments, and small batches with bounded processing time prevent the loop in the first place.

The practical default is at-least-once: a consumer processes a record and then commits its offset. A crash between the two means the record is processed again after restart. Committing first gives at-most-once, which loses records on a crash and is rarely acceptable for business events.

Kafka's idempotent producer removes duplicates caused by producer retries, and transactions make a consume, transform and produce cycle atomic within Kafka. Neither extends to side effects outside Kafka. Once a handler writes to a database, calls an API or sends an email, duplicates are possible and the consumer has to be idempotent.

A service that updates its database and then publishes an event performs two writes with no shared transaction. If the process dies between them, the state has changed and the event is never sent, or the event is sent for a change that rolled back. This dual-write problem is the most common source of silent inconsistency in event-driven systems.

The outbox pattern removes it. The service inserts the event into an outbox table in the same local transaction as the business change. A separate relay then publishes outbox rows to Kafka, either by polling or by change data capture from the database's write-ahead log using a tool such as Debezium. Delivery from the outbox is at-least-once, which is acceptable because consumers are already idempotent. The cost is an extra table, a relay to operate and a small publication delay.

An event is a public contract with consumers the producer may not know about, and it will outlive the code that first wrote it. We define events in a schema language such as Avro or Protobuf, register them in a schema registry, and enforce compatibility in CI so that an incompatible change fails the build instead of a consumer at runtime.

The working rules are simple: add fields as optional with defaults, never repurpose a field or change its type, and do not remove a field while any consumer reads it. A change that cannot be made compatibly gets a new topic or a new event type, with both published during the transition.

Retention is configured per topic by time or size, and it determines how far back a consumer can replay. A consumer that falls further behind than the retention window loses data without an error, so consumer lag against retention is one of the alerts we consider mandatory.

Log compaction is a different policy. Kafka keeps at least the latest record for each key and discards older ones, with a null-value tombstone marking a deletion. Compacted topics suit current-state data such as configuration or the latest profile for each customer, and allow a new consumer to rebuild state without replaying all history. They do not suit event streams where every occurrence matters.

Kafka adds a cluster to operate or a managed service to pay for, plus a set of concepts every engineer on the team must understand. That is justified by multiple consumers, replay and sustained high volume. It is not justified by a single service that needs to run background jobs.