Payment system
A payments API takes a merchant’s “charge this card 49.99” and turns it into money actually moving, through a card processor you don’t control. Throughput is modest; the hard part is correctness when every network call can fail halfway. The standard answer has four pieces: idempotency keys claimed before any side effect, a payment state machine where “outcome unknown” is a real state, reconciliation instead of blind retries, and an append-only double-entry ledger checked daily against the processor’s settlement file. Every request is O(1) work plus one external call; storage grows linearly and is kept for years.
The problem
Section titled “The problem”A merchant’s server calls your API to charge a customer. You call a card processor (an acquirer, which talks to Visa or Mastercard, which talks to the customer’s bank). Any hop can time out after it has already moved the money.
| Goal | What it means | Pulls toward |
|---|---|---|
| Exactly-once effect | A retried request never charges twice | Idempotency keys, durable state before side effects |
| Never lose money | Every cent that moves is recorded, and the records add up | Double-entry ledger, reconciliation, immutability |
| Availability | Checkout works during partial outages | Async paths, retries, queueing, degraded modes |
| Low latency | A card payment answers in about 1-2 s | Synchronous call to the processor on the hot path |
Correctness wins every conflict. A payment that takes 5 s is a nuisance; one charged twice is a support ticket, a refund, and possibly a regulator.
Requirements and scope
Section titled “Requirements and scope”| In scope (typical) | Ask about | |
|---|---|---|
| Functional | Create and confirm a payment, capture (now or later), cancel, refund (full or partial), query status, webhooks to the merchant. | Auth and capture separately (hotels, marketplaces) or one step? Partial refunds? Multiple currencies? |
| Scale | ~10M payments/day, peaks of 5-10× average (Black Friday). | Peak shape? One region or global? |
| Latency | p99 under 2 s for a card payment, dominated by the processor. | Is async (“we’ll tell you by webhook”) acceptable for some methods? |
| Consistency | No double charges, no lost payments, the ledger always balances. Strong consistency within a payment. | Which reads can be stale (dashboards, reports)? |
| Availability | 99.99% for the API; degrade by queueing, never by guessing. | What does the merchant do if we return “processing”? |
| Out of scope | Fraud scoring, card vaulting and PCI scope, payouts to bank accounts, tax. |
Back of the envelope
Section titled “Back of the envelope”What to say out loud: “Throughput is low, hundreds to low thousands per second, so I’ll use a relational database with real transactions and shard by merchant only if I must. Storage is dominated by an append-only ledger kept for years. The design effort goes into correctness under partial failure, not scale.”
API and data model
Section titled “API and data model”POST /payments Idempotency-Key: <client uuid> { amount: 4999, currency: "usd", payment_method, capture: "automatic" | "manual", metadata } -> 200 { id: "pi_7", status, amount, ... } -- same body on every retry with that keyPOST /payments/{id}/capture Idempotency-Key: ... { amount_to_capture? }POST /payments/{id}/cancel Idempotency-Key: ...POST /refunds Idempotency-Key: ... { payment: "pi_7", amount: 1000 }GET /payments/{id} -- safe to repeat, no key neededWebhook to merchant: { id: "evt_...", type: "payment.succeeded", data: {...} } -- at least once
Records: payment { id, merchant, amount (int, minor units), currency, status, version, processor_ref } attempt { id, payment, reference (unique), sent_at, result: approved | declined | unknown } idem_key { merchant, key, fingerprint, locked_at, recovery_point, response_code, response_body, created_at } journal_entry { id, payment?, created_at, kind } posting { entry, account, amount (signed int), currency } -- postings in an entry sum to 0The payment’s states, modelled on Stripe’s PaymentIntent (see State machine):
| State | Means | Leaves by |
|---|---|---|
requires_payment_method |
Created; no card yet, or the last attempt declined | attaching a method and confirming |
requires_action |
Customer must authenticate (3-D Secure) | the customer completes or abandons it |
processing |
Sent; outcome not yet known (bank debits, processor timeouts) | webhook or reconciliation |
requires_capture |
Authorized: funds held on the card, not yet taken | capture, cancel, or the auth expiring (about 7 days for cards) |
succeeded |
Money captured. Terminal for the payment; refunds are separate objects | refunds don’t change it |
canceled |
Abandoned before capture; any hold is released | terminal |
Two decisions hide in that sketch:
- Amounts are integers in minor units (cents), with the currency alongside. Floats can’t represent 0.10 exactly. Not every currency has 2 decimals: JPY has 0 and KWD has 3, so the exponent comes from a currency table, never from code. See Money arithmetic.
- Every attempt has your own unique reference, sent to the processor and stored before the call. It is the only handle you have for asking “did that go through?” later.
High-level design
Section titled “High-level design”| Component | Role | Why it’s there |
|---|---|---|
| Payments API (×N) | Validates, claims the idempotency key, drives the state machine, calls the processor. | Stateless, so any node can resume any request. |
| Idempotency keys | One row per (merchant, key): fingerprint, lock, recovery point, saved response. | Turns retries and duplicates into replays instead of new charges. |
| Payments DB | Payments, attempts, ledger and outbox, in one relational database. | One transaction can update state, write the ledger and enqueue the event together. |
| Card processor | External. Authorizes, captures, refunds; sends webhooks and nightly settlement files. | You don’t control its latency, availability or ordering. |
| Webhook ingest | Verifies signatures, dedupes event ids, applies legal transitions. | Async outcomes (bank debits, disputes) arrive here, late and repeated. |
| Reconciler | Resolves stuck processing payments; matches settlement files to the ledger. |
The safety net for every failure the hot path couldn’t resolve. |
| Outbox | Rows written in the business transaction, relayed to merchant webhooks. | No “state changed but event lost” gap. See Event-driven. |
Deep dive: idempotency keys
Section titled “Deep dive: idempotency keys”A client that times out can’t know whether its request ran. The only safe move is to retry, and retries are only safe if the server can recognise them. Stripe’s answer, described in Brandur Leach’s posts, is a client-chosen key on every mutating request. The server remembers the key, the request, and the response. Pick a scenario and a protection level below and step through it.
What the simulator shows:
| Protection | Lost response | Concurrent duplicate | Crash mid-request | Reused key, new params |
|---|---|---|---|---|
| None | charges twice | charges twice | charges twice | (two real charges) |
| Look up key, save response at the end | safe | charges twice (race) | charges twice (nothing saved) | wrong reply |
| Claim key first + fingerprint + recovery point | safe | safe (loser gets 409) | safe (resume) | safe (rejected) |
The production version, from Brandur’s “Implementing Stripe-like Idempotency Keys in Postgres”, splits a request into atomic phases between external calls, and records a recovery point after each one:
handle(merchant, key, params): txn: -- phase 0: claim row <- INSERT idem_key (merchant, key, fingerprint(params), locked_at = now, recovery_point = "started") ON CONFLICT: load existing row if row.fingerprint != fingerprint(params): return 400 key reused with different params if row.recovery_point == "finished": return row.response -- replay if row.locked_at is recent: return 409 in progress take over lock (locked_at = now) -- previous owner presumed dead
loop by row.recovery_point: "started": -- phase 1: local bookkeeping only txn: create payment + attempt(reference = derived from key); point <- "attempt_created" "attempt_created": -- phase 2: the one external call result <- processor.charge(attempt.reference, amount) -- processor dedupes on reference if timeout: txn: payment.status <- "processing"; point <- "finished", save 202; stop txn: apply result to payment, write ledger entry, write outbox row, save response, point <- "finished"Details that come up:
- Claim before you act. The unique index on
(merchant, key)is the lock that settles a race. Looking the key up and saving it at the end leaves a window where two requests both see “not found”. - Fingerprint the request. Hash method, path and body. A reused key with a different body is a client bug: reject it, don’t replay a response for a different request. Stripe compares parameters and errors on mismatch.
- Pass the key downstream. Derive the processor reference from the key so a resumed request repeats the same external call, which the processor dedupes. If a provider has no idempotency, query by reference before re-sending.
- What gets saved. Stripe (API v1) saves the first response once execution starts, including 500s, so a retry gets the same answer. Validation failures and 409s are not saved; the client can simply retry.
- Retention. Keys are pruned after about 24 hours (Stripe’s documented minimum). A reuse after pruning is a new request. Keys are up to 255 characters; V4 UUIDs are the usual choice.
The Idempotency concept page covers keys, fingerprints and natural idempotency in general.
Deep dive: unknown outcomes and the state machine
Section titled “Deep dive: unknown outcomes and the state machine”The single most important rule: a timeout is not a failure. When the processor doesn’t answer, the card may or may not have been charged. You have three options. The simulator runs 48 payments through a processor that times out, declines and sends messy webhooks.
| On timeout | Double charges | Wrong final state | Latency to resolve | Needs |
|---|---|---|---|---|
| Retry with a new reference | Yes, whenever attempt 1 secretly succeeded | Yes, when attempt 1 succeeded and attempt 2 declined | Fast | Nothing (that’s the trap) |
| Retry with the same idempotency key | No | No | Fast | Processor-side idempotency; a retry cap |
Mark processing, reconcile |
No | No | Seconds to minutes | A status query or webhook; merchants that handle “pending” |
Real systems combine the last two: retry a few times with the same key and backoff, then park the payment in processing and let the reconciler and webhooks finish it.
Webhooks from the processor are at least once and unordered. A handler that applies whatever arrives will move a succeeded payment back to processing, and post the ledger twice on a duplicate. The guarded handler does three things:
on_processor_event(evt): verify signature(evt) or reject -- anyone can POST to your URL txn: if INSERT processed_event(evt.id) conflicts: return 200 -- duplicate delivery p <- SELECT payment WHERE processor_ref = evt.ref FOR UPDATE if (p.status, evt.type) not in ALLOWED: record anomaly; return 200 -- stale or impossible p.status <- next(p.status, evt.type); p.version += 1 write ledger entry for this transition -- unique (payment, transition): posts once write outbox row for the merchant return 200 quickly; slow work goes to a queueA transition seen on a terminal state, such as “succeeded” for a payment you recorded as declined, is not silently applied or ignored. It goes to a review queue: that is real money you don’t account for.
Deep dive: the ledger and reconciliation
Section titled “Deep dive: the ledger and reconciliation”Double-entry means every movement of money is a journal entry of two or more postings that sum to zero: money leaves one account and arrives in another. You never store a balance you can’t recompute; a balance is the sum of an account’s postings. Stripe’s internal Ledger and Modern Treasury’s “Accounting for developers” both describe this model. The Ledger page covers the mechanics.
A 49.99 card payment with a 1.75 platform fee, in cents. After settlement the receivable is back to zero and the merchant is owed 4824:
| Entry | Account | Amount |
|---|---|---|
| capture | processor receivable | +4999 |
| capture | merchant balance (payable) | −4999 |
| fee | merchant balance | +175 |
| fee | fee revenue | −175 |
| settlement (next day) | cash at bank | +4999 |
| settlement | processor receivable | −4999 |
post(kind, payment, legs): -- legs: [(account, signed amount, currency)] require for each currency: sum(legs.amount) == 0 -- reject unbalanced entries, always require all amounts are integers in minor units txn: INSERT journal_entry; INSERT postings -- append only; never UPDATE or DELETE corrections are new reversing entries, never editsReconciliation compares your records with someone else’s. Every night the processor sends a settlement file listing what actually settled, with references, amounts and fees. The reconciler:
- Parses the file into lines keyed by processor reference (see Parsing).
- Matches each line to an attempt and its ledger postings. Compare amount and currency, not just existence.
- Classifies the rest: missing on our side (a timeout we never resolved), missing on theirs (a capture that didn’t go through), amount differs (fees, partial captures, FX), timing (crossed midnight; usually clears the next day).
- Posts the settlement and fee entries for matched lines, and sends breaks to an exceptions queue with both sides attached.
The Reconciliation page goes deeper on matching strategies.
Deep dive: scaling and the outbox
Section titled “Deep dive: scaling and the outbox”- Shard by merchant. All of a payment’s rows (payment, attempts, idempotency keys, its ledger entries) live with the merchant, so every write stays a single-shard transaction. Platform-wide accounts (fee revenue, the processor receivable) become hot rows; post to them in batches or split them into sub-accounts that are summed on read.
- Transactional outbox. Don’t call the merchant’s webhook (or Kafka) inside the request. Write an outbox row in the same transaction as the state change; a relay publishes it and marks it sent. Delivery is at least once, so merchant events carry ids the merchant dedupes on. See Webhook delivery.
- Optimistic concurrency on the payment. A
versioncolumn andUPDATE ... WHERE id = ? AND version = ?keep a capture and a webhook from both moving the same payment. See Versioning. - Multiple processors. Route by cost, method and health. Never fail over a timed-out attempt to a second processor: the first may have succeeded.
- Multi-region. Keep a merchant’s writes in a home region. Active-active writes to one payment would need conflict resolution on money, which nobody wants.
Failure modes
Section titled “Failure modes”| What breaks | What you see | Design response |
|---|---|---|
| Client times out | Retry arrives with the same key | Replay the saved response; 409 if still in flight |
| API node dies mid-request | Key locked at an intermediate recovery point | Lock timeout, then resume from the recovery point with the same downstream reference |
| Processor timeout | No answer to an authorize call | Mark processing; retry with the same key a few times; reconcile; never a fresh reference |
| Processor down | Errors on every call | Circuit breaker; fail fast with a clear error, or route to a second processor for new payments only |
| Duplicate or out-of-order webhooks | Repeated events, stale states | Dedupe on event id; transition table; ledger posting unique per transition |
| Settlement file mismatch | Breaks in the nightly report | Exceptions queue, aging alerts, correcting entries |
| DB primary failover | Seconds of write errors | Synchronous replica; clients retry with keys, so retries are safe |
| Bug posts wrong amounts | Ledger doesn’t match processor | Immutable ledger: reversing entries, then re-post; never edit history |
What it costs
Section titled “What it costs”| Operation | Time | Space |
|---|---|---|
| Create + confirm a payment | O(1) DB work: ~3 short transactions + 1 processor call (100 ms to 2 s) | ≈1-2 KB (payment, attempt, key row) |
| Replay on retry | O(1): one indexed lookup | none new |
| Post a journal entry | O(legs), usually 2-6 rows | ≈250 B per posting with indexes |
| Account balance | O(postings) naively; O(1) with a running balance updated in the same txn | one row per account |
| Webhook in | O(1): dedupe insert + row lock | ≈100 B per processed event id |
| Nightly reconciliation | O(n) to hash-join n lines with n attempts | O(n) for the day’s index |
| Storage growth | — | ≈ payments × entries × row size, kept 7+ years; idempotency rows pruned after ~24 h |
In the interview
Section titled “In the interview”- Clarify (5 min). Auth/capture split, refunds, currencies, async methods, can the API say “processing”, scale and peak.
- API and money (5 min). Integer minor units, currency codes, idempotency key header, the state table. Say “timeouts are unknown, not failed” early.
- Happy path (10 min). Architecture: stateless API, relational DB, processor, webhooks. Walk the card payment: claim key, record attempt, call, commit state + ledger + outbox together.
- Failure paths (15 min). Client retry, concurrent duplicate, crash mid-request (recovery points), processor timeout (reconcile), webhook duplicates and ordering.
- Ledger and reconciliation (5-10 min). Double-entry, immutability, settlement file matching, exceptions queue. Then sharding by merchant if time allows.
Follow-ups to expect: “The processor has no idempotency support.” Query by your reference before re-sending, and reconcile from the settlement file. “How do you refund half a payment twice, safely?” Each refund is its own object with its own key; sum of refunds is capped by the captured amount, checked under a row lock. “Show me the balance is right.” It’s derived from postings that each sum to zero, and checked daily against an external source. “Multi-currency?” Store original currency and amount; convert at a recorded rate in a separate FX entry; never mix currencies in one sum.
Check yourself
Section titled “Check yourself”Go deeper
Section titled “Go deeper”- Brandur Leach: Implementing Stripe-like Idempotency Keys in Postgres: atomic phases, recovery points, locking and the
idempotency_keysschema. - Stripe: Designing robust and predictable APIs with idempotency and Stripe API docs: Idempotent requests: keys, retries with backoff and jitter, what gets saved, 24-hour pruning.
- Stripe docs: How PaymentIntents work: the real state machine, including
processingandrequires_capture. - Stripe: Ledger, Stripe’s system for tracking and validating money movement and Modern Treasury: Accounting for developers: double-entry and immutable ledgers for engineers.
Concept pages to drill: Idempotency · Ledger · Reconciliation · Money arithmetic · State machine · Concurrency · Event-driven