Web3 infrastructure beyond smart contracts

Discussion of blockchain engineering concentrates on smart contracts, which is understandable: they are immutable once deployed, publicly inspectable and unforgiving of mistakes. They deserve audits and formal care. They are also a small fraction of the code in a working product.

Our view is that the reliability of a Web3 system is decided mostly off-chain. A correct contract is of little use if the application reads stale state from a lagging node, credits a deposit that is later reorganised away, or cannot submit transactions because a nonce is stuck. This article describes the infrastructure that surrounds the contract and the decisions it forces.

Every read and write passes through a node's RPC interface, whether self-hosted or provided as a service. That interface fails in ordinary ways, with rate limits, timeouts and outages, and in ways specific to blockchains: a node can be healthy and several blocks behind, and two nodes behind the same load balancer can answer consecutive requests from different chain heads.

We design RPC access with the same care as any critical upstream. Requests go through an internal gateway that holds several providers, tracks each one's latest block height, removes laggards from rotation and retries idempotent reads elsewhere. Reads that must be mutually consistent are pinned to a specific block number or hash rather than to latest. Running your own nodes gives control and independence at the cost of storage, synchronisation time and upgrades at every network fork; archive nodes, needed for historical state queries, are considerably heavier.

A node answers narrow questions: the state at an address, the logs in a block range. It cannot answer product questions such as every transfer involving a given user, sorted by time and filtered by asset. Those need an indexer that consumes blocks and event logs, decodes them and writes them to a database built for queries.

An indexer is a data pipeline with one unusual property: its source can retract what it has already said. A sound design stores the block number and block hash with every derived row, checks each new block's parent hash against the stored one, and on a mismatch rewinds to the common ancestor and reprocesses. Handlers must be idempotent and deterministic so that a full re-index from genesis reproduces the same state.

A transaction included in a block is not yet permanent. On proof-of-work chains finality is probabilistic and grows with each subsequent block. Proof-of-stake networks such as Ethereum add explicit checkpoints, exposed through the safe and finalized block tags, after which reversal would require a large share of validators to be penalised. Layer 2 networks add a further distinction between sequencer confirmation and settlement on the underlying chain.

The engineering consequence is that confirmation policy is a risk decision per chain and per action, and should live in configuration rather than code.

A private key authorises anything its address is permitted to do, and a signed transaction cannot be recalled. Keys held in environment variables or application memory turn any server compromise into a loss of funds. We keep signing behind a dedicated service, so that application code requests a signature and never holds key material.

On account-based chains each transaction from an address carries a sequential nonce, and the network processes them strictly in order. Two workers that read the same pending count will both use the same nonce and one transaction will be dropped. A transaction that is underpriced for current fees sits in the mempool and blocks every later transaction from that address.

A production sender therefore treats nonce allocation as a serialised resource: one allocator per address, backed by a durable store, issuing nonces in order and recording each transaction's state through submission, inclusion and finality. A watcher detects transactions that have been pending too long and replaces them with the same nonce and a higher fee, and fills any gap left by a failed submission. The business operation behind each transaction needs its own idempotency key, so a retry after a crash does not send the same transfer twice.

Standard service metrics are necessary and insufficient here. The signals that predict incidents are specific: the gap between each RPC provider's head and the highest known block, indexer lag in blocks and in seconds, the age of the oldest pending transaction per sender, the native-token balance of every address that pays fees, the rate of reverted transactions, and reorganisation depth. Contract events with security meaning, such as ownership transfers, upgrades and pauses, should raise alerts directly.

All of this is centralised infrastructure wrapped around a decentralised system, and that tension should be acknowledged. An indexer and an RPC gateway are components users have to trust, and a product whose purpose is to remove trusted intermediaries may prefer light clients, direct wallet reads and on-chain verification, accepting slower and plainer interfaces as the price.

For most products that combine on-chain settlement with conventional accounts and user experience, the pragmatic position is to keep on-chain what must be verifiable, keep off-chain what must be fast or private, and engineer the boundary carefully. It is also worth asking whether a blockchain is required at all. Where a single organisation controls every participant, a database with an append-only audit log is simpler, cheaper and easier to operate.