Observability for distributed systems

Monitoring tells you that a known failure has happened. Observability is the ability to explain a failure nobody predicted, using only the telemetry the system already emits. In a distributed system that distinction matters, because the interesting failures are interactions between services that each look healthy alone.

Our position is that observability should be designed from the user's experience backwards. Start with service level objectives, instrument the request path so that one request can be followed end to end, and collect everything else only when it answers a question someone will ask. Collecting all possible telemetry and hoping it is useful later produces a large bill and a slow investigation.

Metrics, logs and traces are not interchangeable. Each is cheap for one kind of question and expensive or useless for the others.

A metric is stored as one time series per unique combination of label values. A request counter labelled by service, endpoint and status code is a few hundred series. Add a user ID label and it becomes one series per user per endpoint per status, and a Prometheus-style store will run out of memory long before it becomes useful.

The rule we apply: metric labels must be bounded and small. Route templates, not raw URLs. Status classes, not error messages. High-cardinality identifiers such as user, tenant, order or request ID belong in traces and structured logs, where each event is stored individually and the cost is proportional to volume instead of to the product of dimensions. Exemplars bridge the two by attaching a sample trace ID to a metric bucket, so a spike on a latency graph links directly to a request that experienced it.

Two small frameworks cover most dashboards. RED applies to anything that serves requests: rate, errors and duration. USE applies to anything that can be exhausted: utilisation, saturation and errors, for CPUs, disks, connection pools, thread pools and queues.

Duration must be recorded as a histogram, not an average. An average hides the slow requests that users complain about, and percentiles computed per instance cannot be combined afterwards. Histograms with sensible bucket boundaries can be aggregated across instances and then queried for p95 or p99. Saturation is the signal most often missing: a pool at full utilisation with nothing waiting is fine, and the same pool with a growing wait queue is an incident about to happen.

A service level indicator is a measurement of user-visible behaviour, such as the proportion of checkout requests that succeed within 500 ms. A service level objective is the target for it over a window, say 99.9 per cent across 30 days. The difference between the target and perfection is the error budget: the amount of failure the service is permitted.

The budget makes two decisions less political. If it is being spent quickly, reliability work takes priority over features. If it is barely touched, the team can ship faster or accept more risk. It also gives alerting a principled basis. We alert on burn rate: a fast burn over a short window pages someone immediately, and a slow burn over a long window opens a ticket. Pairing a short and a long window on each alert stops a brief blip from paging and stops a recovered incident from continuing to fire.

The trade-off is effort. Defining good indicators requires agreement on what users care about, and a poorly chosen indicator gives false comfort. For an internal batch system with no interactive users, a simpler freshness or completion check is usually enough.

A trace is only useful if it is complete. Each service must read the incoming trace context, create its own spans as children, and pass the context to everything it calls. The W3C Trace Context standard defines the traceparent header for this, and OpenTelemetry's instrumentation handles it for common HTTP and gRPC libraries.

The breaks happen at boundaries the automatic instrumentation does not cover: a message published to Kafka without context in its headers, a job handed to a thread pool, a scheduled task, a call through a proxy that strips unknown headers. Each one splits a trace into unrelated fragments. We treat propagation across asynchronous boundaries as explicit work, and we put the trace ID in every structured log line so logs and traces can be joined.

Sampling is the other decision. Head sampling chooses at the start of a request, is cheap and discards most of the errors along with the routine traffic. Tail sampling decides after the trace is complete, so it can keep every error and every slow request, at the cost of buffering spans in a collector. For systems where failures are rare and valuable to inspect, tail sampling is worth the extra component.

An on-call engineer who is paged for things that need no action learns to ignore pages. The cause is nearly always alerting on causes instead of symptoms: CPU above a threshold, a single pod restarting, a queue briefly longer than usual. These are worth graphing and rarely worth waking someone for.

Distributed tracing, a metrics pipeline and a log platform carry real operating cost. A modular monolith with one database does not need a tracing backend: structured logs with a request ID, RED metrics at the edge and database query timings will explain most problems. We add tracing when a request routinely crosses more than a few process boundaries, because that is the point at which logs alone stop being able to tell the story.