Designing resilient payment systems

The successful path through a payment is short: validate, call a provider, record the result. Almost all of the engineering lies elsewhere, in the cases where a network call times out, a callback arrives twice or out of order, a process dies between two writes, or a provider's record disagrees with yours.

Our position is that a payment system should be designed around the assumption that every external call can end in one of three states: succeeded, failed, or unknown. Systems that model only the first two eventually charge someone twice or lose track of money. The practices below exist to make the third state safe.

Retries are unavoidable, so every operation that moves money must be safe to repeat. The client generates an idempotency key per logical operation and sends it with each attempt. The server stores the key under a unique constraint together with a fingerprint of the request and, once complete, the response. A repeat with the same key returns the stored response without executing again.

The details decide whether it works. The key must be inserted before the side effect, in the same transaction as the payment record, so that two concurrent attempts cannot both proceed. A request that reuses a key with a different payload should be rejected, not silently treated as the original. A repeat that arrives while the first attempt is still in flight should receive a conflict or wait, not start a second execution. The same key, or one derived from it, is then passed downstream to the provider, because idempotency that stops at your own API does not prevent a duplicate charge at the acquirer.

A mutable balance column answers one question, the current balance, and destroys the information needed to answer every other. We record money movement in an append-only, double-entry ledger: each transaction consists of entries whose debits and credits sum to zero, entries are never updated or deleted, and a mistake is corrected with a reversing transaction that leaves the history intact.

The zero-sum invariant is cheap to check and catches a wide class of defects, because money cannot appear or vanish without an unbalanced transaction. Amounts are integers in the currency's minor unit with the currency stored alongside, never floating point. Balances are derived from entries and may be cached, but the cached figure is always reproducible. The trade-off is more storage and more complex read queries, which we consider a fair price for a record that explains itself.

A payment passes through states such as created, authorised, captured, settled, failed and refunded. Encoding these as an explicit state machine, with a fixed table of allowed transitions, turns a large family of bugs into rejected operations. A capture on a payment that was never authorised, or a second refund beyond the captured amount, fails at the transition check.

When a call to a provider times out, the request may never have arrived, may have been declined, or may have succeeded with the response lost. Treating a timeout as a failure and letting the customer try again is how double charges happen. Treating it as success is how goods are released without payment.

The correct handling is to move the payment to an unknown state and resolve it. The system queries the provider's status endpoint using its own reference, or retries the original request with the same idempotency key, with backoff until a definite answer arrives. If none arrives within a defined window, the case is escalated to an operations queue with enough context for a person to resolve it.

Providers report asynchronous outcomes through webhooks, and webhooks are delivered at least once, in no guaranteed order, and occasionally not at all. A handler has to assume all three.

However careful the online path, the internal ledger and the provider's records will sometimes disagree. Reconciliation compares them systematically: each internal transaction is matched against the provider's settlement report, and the settled totals are matched against the bank statement. Every unmatched item on either side becomes a tracked break with an owner and an age.

It needs stable references shared with the provider, tolerance for timing differences across cut-off times, explicit handling of fees and currency conversion, and alerts when breaks exceed an expected level. A design that cannot be reconciled automatically is incomplete, because reconciliation is the control that detects the failures every other mechanism missed.

The safest way to protect card data is not to handle it. Hosted payment fields or a redirect to the provider keep the card number out of your browser code, servers, logs and backups, and the provider returns a token that stands in for the card. This greatly reduces the part of the estate that PCI DSS requirements apply to. Where card data must be handled directly, that function is isolated in a small, separately controlled service designed to support those requirements, and nothing else is allowed to see the data.

These practices carry cost, and proportion matters. A business that takes payments through a single provider's hosted checkout can reasonably rely on that provider's records and a daily reconciliation, without building its own ledger. A full double-entry ledger, state machine and reconciliation engine become necessary when the system holds balances, splits funds between parties, or uses several providers. The principle that does not scale down is idempotency: any code that initiates a payment needs it from the first release.