Building high-performance fintech systems

Performance work in most software is a negotiation with correctness: a little staleness here, an approximate count there. Financial systems remove that option. A balance that is briefly wrong, an order applied twice, or a ledger entry that cannot be explained six months later is a defect regardless of how quickly it was produced.

Our view is that high performance in this domain means predictable behaviour under load, not a headline throughput figure. The engineering effort goes into the tail of the latency distribution, into removing contention at the points where money changes state, and into keeping an audit trail that costs almost nothing on the hot path.

An end-to-end target is only actionable once it is divided. If a payment authorisation has, say, a 300 ms budget at the 99th percentile, each hop receives a share: edge and authentication, risk checks, the ledger write, the call to an external rail. A component that exceeds its share is identified by measurement rather than by argument.

Budgets are stated at high percentiles because averages hide the requests that matter. A user-facing operation that fans out to several services experiences the slowest of them, so tail latency compounds with every dependency. Load tests have to be open-loop, sending requests at a fixed rate regardless of responses. A closed-loop test slows down when the system does, which conceals exactly the queueing it was meant to reveal.

Financial workloads concentrate writes on a small number of rows. A merchant settlement account, a treasury wallet or a popular instrument is touched by a large share of all transactions, and row-level locks on it serialise the system no matter how many application instances exist. Adding hardware does not help when the constraint is one lock.

The designs we favour reduce what happens while a lock is held and reduce how often it is needed.

Optimistic concurrency, a version column checked at write time, performs well when conflicts are rare, because readers and writers never wait. On a hot account it degrades into repeated retries that waste work and lengthen the tail. Pessimistic locking with SELECT FOR UPDATE queues writers in order and gives steadier latency under high conflict, at the price of holding connections while they wait.

The choice follows the measured conflict rate per entity, and a system may reasonably use both: optimistic for customer accounts, a serialised single-writer path for the handful of accounts that everything touches.

Ordering is a correctness property here. Every state change should carry a monotonically increasing sequence within its scope, so that consumers can detect a missed message by the gap and request a replay instead of silently diverging. Wall-clock timestamps are not a substitute: clocks drift between hosts and two events can share a millisecond.

Database sequences are a common trap. They are deliberately non-transactional, so a rolled-back transaction consumes a value and leaves a hole, and values can be committed out of numerical order. That is acceptable for surrogate keys. It is not acceptable where numbering must be gapless, such as certain invoice or statement series, or where a reader polls by highest seen identifier and skips a row that commits late. Gapless numbering needs a serialised allocator inside the same transaction, which is slower by design and should be confined to where it is required.

A system without limits fails badly: queues grow, latency rises, clients time out and retry, and the retries add load to a system that is already behind. Every queue and connection pool should be bounded, and the behaviour at the bound should be a decision. For some operations the right response is to reject quickly with a retryable error; for others it is to shed low-priority work such as report generation to protect authorisations.

Deadlines should propagate with the request, so a downstream service does not spend effort on work whose caller has already given up. Retries need capped exponential backoff with jitter and a budget, and must carry an idempotency key, otherwise recovery from an incident produces duplicate money movement.

Audit is often added as a second write to a log table, which doubles the work and can still disagree with the primary record. We prefer a design where the audit trail is the primary record: an append-only, double-entry ledger in which every entry is immutable, references the command that caused it, and is corrected by a reversing entry rather than an update. Amounts are stored as integers in minor units or as fixed-precision decimals, never as binary floating point.

An append-only log is also fast. Sequential writes are the cheapest operation a storage engine offers, and the same log supports replay, reconciliation and investigation. The cost is that read models have to be derived and kept current, and that storage grows without bound unless snapshots and archival are planned.

Single-writer partitioning and in-memory state give the best latency and the hardest operational model: failover, replay time and partition rebalancing all need engineering. For a system with moderate volume, a well-indexed PostgreSQL database with short transactions and careful lock ordering is simpler, easier to reason about and usually sufficient. We reach for the specialised architecture when measurement shows the relational design has run out of room, not before.