Event-driven order pipeline

advancedCommon atStripeAnthropicAmazonUberConfluentRead firstEvent-drivenIdempotencyDistributed thinkingPractise in the Sandbox →

An order touches payments, inventory, shipping and email, and you don’t want the checkout request to wait on all of them. So the order service records a fact (“OrderPlaced”) and the other services react to it from a durable, partitioned event log (Kafka-style). The design rests on four decisions: key events by order id so each order’s events stay in order; publish through a transactional outbox so the order and its event can never disagree; make every consumer idempotent, because delivery is at least once; and handle failures with sagas (compensating steps) and dead-letter queues. An append costs O(1), one partition is read by one consumer per group, and partition count caps parallelism.

The problem

Picture checkout at a large retailer. One click must reserve stock, charge a card, create a shipment and send a confirmation email. Each of those is owned by a different team and a different service with its own database.

If the order service calls each one synchronously, checkout is only as available as the least available service, and only as fast as the slowest. A 2-second email provider timeout becomes a 2-second checkout. The event-driven answer is to decouple: write the order, emit an event, return. Downstream services consume the event on their own schedule.

That trade buys availability and team independence, and costs you three things you must now design for:

Goal	What it means	Pulls toward
Decoupling	Producers don’t know or wait for consumers; new consumers attach without changes upstream	Async events, a durable log, schemas as the contract
Correctness	Every order is charged exactly once, never for a ghost order, in the right sequence	Ordering per key, outbox, idempotent consumers, sagas
Operability	You can see where an order is stuck, recover from bad events, and replay history	DLQs, lag metrics, tracing, retention

Synchronous calls give you correctness for free (one call, one answer) and lose decoupling. Events flip that.

Requirements and scope

	In scope (typical)	Ask about
Functional	Place an order; downstream services reserve stock, take payment, ship, notify. Order moves PENDING → CONFIRMED or → CANCELLED. Events can be replayed.	Must checkout return a final answer (paid or not), or is “pending” fine? Which steps can be undone?
Ordering	Events for one order are processed in order. No global order across orders.	Is any cross-entity order needed (e.g. per customer, per SKU)?
Delivery	At least once, with idempotent effects. No lost events, no ghost events.	Is a duplicate email acceptable? A duplicate charge clearly isn’t.
Scale	~2M orders/day, ~6 events each, 5-10× peaks on sale days.	Peak or average? How many consumer services?
Latency	Checkout p99 under 300 ms; order confirmed within seconds (p99 under 10 s).	Is a confirmation within a minute acceptable during peaks?
Availability	Checkout keeps working if payments, email or the broker is degraded.	What’s the tolerance for “accepted but not yet charged”?
Out of scope	The payment processor internals, fraud scoring, stream analytics.

The clarifying question that changes the most

“Does the user need to know, inside the checkout request, whether payment succeeded?” If yes, payment stays a synchronous call (with an idempotency key) and only the rest goes async. If “pending, then a confirmation” is fine, the whole pipeline can be event-driven and the API returns 201 with status: PENDING. Most real systems are this hybrid: authorize the card synchronously, do everything else from events.

Back of the envelope

Event logs are rarely bounded by storage. The numbers that drive the design are peak events per second divided by what one consumer can handle, which sets the partition count, and the backlog a short outage creates, which sets how much spare consumer capacity you need.

Back of the envelope

drag the assumptions

assumptions

orders per day2Mevents per order (placed, paid, reserved, shipped...)6peak-to-average ratio5.00×event size (bytes)1 KBretention (days)7events/s one consumer handles500/s

so you need

peak events/s on the topic694/s

peak ingress694 KB/sbefore replication; brokers write it 3×

disk for retention (replication 3)252 GB

partitions needed for the slowest group2one consumer per partition is the parallelism ceiling

backlog after a 10-minute consumer outage417Kwhat you must drain with spare capacity afterwards

2M orders/day × 6 events is only ~140 events/s on average, ~700/s at a 5× peak, a few MB/s. One broker could hold it. Partition count is driven by the slowest consumer: if a handler calls a third-party API at 50 events/s, you need 14+ partitions for that group alone. Partitions are hard to add later without moving keys, so overprovision.

What to say out loud: “Throughput is modest; a few hundred to a few thousand events per second. Partitions are the unit of parallelism and of ordering, so I’ll size them for the slowest consumer at peak with headroom, say 3×, and pick a count I won’t need to change, like 24 or 48.”

API and data model

The external API stays small. The real contract is the event schema: other teams build against it, so it is versioned like a public API.

pseudocode: the contract

POST /orders  (Idempotency-Key: k-91f)  { customer_id, items[], payment_method }
    -> 201 { order_id, status: "PENDING" }
GET  /orders/{id}  -> { status: PENDING | CONFIRMED | CANCELLED, steps: {...} }

Event envelope (every topic):
    event_id      uuid               -- dedupe key for consumers
    type          "OrderPlaced"      -- past tense: a fact, not a command
    key           order_id           -- partition key, gives per-order ordering
    occurred_at   timestamp
    schema_ver    3                  -- registered in the schema registry
    trace_id      for following one order across services
    payload       { order_id, customer_id, items[], total_minor, currency }

Tables:
    orders(order_id PK, status, ..., version)
    outbox(seq BIGSERIAL PK, event_id UNIQUE, key, type, payload, created_at, sent_at NULL)
    processed_events(consumer, event_id, PRIMARY KEY (consumer, event_id))  -- in each consumer's DB

Two decisions hide in that sketch:

• The partition key. order_id keeps one order’s events in order and spreads load evenly. customer_id would also order a customer’s orders relative to each other, at the cost of hot partitions for big accounts. Choose the smallest entity whose events must be ordered.
• Events are facts, not commands. OrderPlaced says what happened; each consumer decides what to do about it. ChargeCard sent to one specific service is a command; that’s fine inside an orchestrated saga, but then it should go on a topic the payments team owns.

High-level design

Step through the flows. Note that the checkout request never touches the broker, and that every failure ends in either a retry, a compensation or a dead letter, never in silence.

Order pipeline on an event log

step through

flowPlace orderSaga: payment failsPoison eventConsumer crash

The request path only touches the local database. Everything else happens asynchronously off the log.

Idempotency-Keyk-91f

Checkout calls POST /orders with an idempotency key, so a client retry cannot create a second order.

↺◀▶ Play▶1 / 6

Component	Role	Why it’s there
Order service	Validates the order and writes it, plus an outbox row, in one transaction.	The only synchronous step. It depends on its own database and nothing else.
Outbox relay	Reads unsent outbox rows (poll or CDC) and publishes them.	Turns one local commit into a guaranteed publish, without a distributed transaction.
Event log	Partitioned, replicated, append-only. Retains events for days.	Durable buffer between teams. Consumers can fall behind, crash, or replay without losing anything.
Consumer groups	One group per service. Partitions are spread over the group’s instances.	Each service gets every event; within a service, work is split by partition.
Schema registry	Stores event schemas and enforces compatible changes.	Stops a producer deploy from breaking every consumer.
Dead-letter topic	Holds events that failed after retries, with the error.	One poison event doesn’t block its partition forever.

Deep dive: the dual-write problem and the outbox

The most common bug in event-driven systems is writing to the database and publishing to the broker as two separate steps. There is no transaction spanning Postgres and Kafka, so a crash between the two leaves them disagreeing. Try each strategy against each fault.

The dual-write problem vs the transactional outbox

step through

Order service

handles POST

Orders DB

Postgres

(empty)

Event log

orders topic

(empty)

Payments

consumer

(empty)

POST /orders arrives. The service must do two things: save order 42 and tell the rest of the system it happened.

↺◀▶ Play▶1 / 3

Strategy	Crash in the middle	Broker down	Verdict
Commit, then publish	Lost event: order exists, nobody charges it	Event lost or late	Unsafe
Publish, then commit	Ghost event: charge for an order that doesn’t exist	Orders fail: broker on the critical path	Unsafe
Two-phase commit (XA)	Safe	Orders block	Rarely supported by brokers; slow; avoid
Transactional outbox	Safe; relay may publish twice	Orders keep flowing; events delayed	Standard answer
Event sourcing	Safe: the event is the write	Same as outbox	Larger change: state is derived from events

The relay has two common forms:

pseudocode: polling relay (one relay owns the table, or one per shard)

loop:
    rows <- SELECT * FROM outbox WHERE sent_at IS NULL ORDER BY seq LIMIT 500
    if rows is empty: sleep 50 ms; continue
    for row in rows:                       -- in seq order, so per-key order is preserved
        broker.send(topic_for(row.type), key = row.key, value = row.payload,
                    headers = { event_id: row.event_id })
    broker.flush()                         -- wait for acks (acks = all)
    UPDATE outbox SET sent_at = now() WHERE seq IN rows.seq
    -- a crash between flush and UPDATE re-sends the batch: duplicates, never gaps
periodically: DELETE FROM outbox WHERE sent_at < now() - 1 day

The alternative is change data capture (CDC): a connector such as Debezium reads the database’s write-ahead log and publishes outbox inserts. No polling load, lower latency, and it tracks its position in the log rather than updating rows. Either way the relay is at least once, which is why the consumer side matters.

Deep dive: partitions, consumer groups and ordering

A topic is split into partitions. Each partition is an ordered, append-only log; each event in it has an offset. Kafka guarantees order within a partition only. A consumer group is a set of instances of one service that share the work: each partition is assigned to exactly one consumer in the group, and the group records how far it has read as a committed offset per partition.

Three consequences show up in every interview. Run them below:

• Ordering per key comes from the partitioner: same key, same partition, same consumer, read in offset order. Switch to round-robin and watch one order’s events get handled out of sequence.
• Parallelism is capped by partitions. Set consumers above partitions and the extras sit idle.
• At-least-once comes from committing after processing. Kill a consumer: the new owner restarts at the last committed offset, and everything processed since is delivered again.

Partitions, a consumer group, and lag

simulator

reshuffle

partitions P = 4consumers C = 2events in / tick = 3events / consumer / tick = 2commit every N ticks = 4

C1: P0 P2C2: P1 P3

P0 → C1

lag 0

P1 → C2

lag 0

P2 → C1

lag 0

P3 → C2

lag 0

lag: 0processed: 0out of order for a key: 0redelivered: 0idle consumers: 0

pending event (order + sequence)hot key / rebalance pausedead consumer

Keyed by order id, every event for one order sits in one partition and is read in offset order, so per-order ordering holds at any C.

↺◀▶ Run▶t = 0 / 40

Lag is the number of events between a partition’s newest offset and the group’s committed offset. It is the health metric for an event-driven system: growing lag means consumers are slower than producers. The responses, in order: add consumers (up to the partition count), make the handler faster (batch writes, more concurrency inside a partition for unrelated keys), then add partitions. Adding partitions changes hash(key) % P, so in-flight keys move to new partitions and lose ordering at the boundary. That’s why you overprovision up front.

Backpressure is built in: a pull-based consumer reads only as fast as it processes, and the log absorbs the difference up to its retention. The risk moves to the edge of retention: if lag exceeds 7 days of data, events are deleted before they are read. Alert on lag in time, not just in count.

Rebalancing. When a consumer joins, leaves or misses heartbeats past the session timeout, the group reassigns partitions. The old “eager” protocol stopped the whole group during the move; cooperative rebalancing (and Kafka 4.0’s new consumer group protocol) moves only the partitions that change owner. Either way, a rebalance is a short pause in that group, and it’s why consumers commit offsets for revoked partitions before giving them up.

Deep dive: idempotent consumers, sagas and dead letters

Idempotent consumers. “Exactly once” end to end is really at-least-once delivery plus idempotent effects. The building block is a dedupe record written in the same transaction as the effect:

pseudocode: handle one event, effect and dedupe commit together

handle(event):
    begin transaction (consumer's own database)
        inserted <- INSERT INTO processed_events (consumer, event_id)
                    VALUES ('inventory', event.event_id) ON CONFLICT DO NOTHING
        if not inserted: rollback; return              -- seen before: skip
        apply effect (reserve stock for event.payload.items)
        INSERT INTO outbox (... StockReserved ...)      -- chain the next event safely
    commit
    -- commit the Kafka offset afterwards; a crash here only causes a skipped duplicate

If the effect lives in another system (a card network), pass a deterministic idempotency key derived from the event, such as charge:{order_id}, so the external system dedupes. See Idempotency for key scoping and expiry. Kafka’s own transactions give exactly-once for Kafka-to-Kafka processing (read, transform, write, commit offsets atomically), but not for side effects in your database or a third party.

Sagas. A saga is a sequence of local transactions, each with a compensating action that semantically undoes it: release stock, refund a charge, cancel a shipment. There is no isolation, so other readers can see the in-between states (PENDING). Two ways to run one:

	Choreography	Orchestration
How	Each service reacts to events and emits its own	One orchestrator sends commands and tracks state
Where the flow lives	Spread across services	In one place, as a state machine
Good for	2-4 steps, independent teams	Many steps, branches, timeouts, visibility
Risk	Cyclic dependencies; hard to answer “where is order 42?”	The orchestrator is a central dependency to scale and keep durable

pseudocode: orchestrator transitions for one order

on OrderPlaced:          state = RESERVING;  send ReserveStock(order)
on StockReserved:        state = CHARGING;   send ChargeCard(order, key = "charge:" + order_id)
on PaymentSucceeded:     state = CONFIRMED;  send CreateShipment(order)
on PaymentFailed:        state = COMPENSATING; send ReleaseStock(order)
on StockReleased:        state = CANCELLED
on timeout(state, 10 min): treat as failure of the current step -> compensate
-- state is persisted per order with a version; every transition is idempotent

Order the steps so the hardest-to-undo action comes last (“pivot” transaction): reserve stock first (easy to release), charge second, ship last (hard to recall).

Dead letters. A poison event fails no matter how many times you retry. Because a partition is processed in order, it blocks everything behind it. Retry a few times with backoff (transient errors), then move it to a dead-letter topic with the error, original topic, partition and offset, commit past it, and alert on DLQ depth. For per-key ordering you may instead park all later events for that key until the poisoned one is fixed; that’s stricter and more work. Uber’s engineering blog describes a tiered version: retry topics with growing delays, then a DLQ.

Deep dive: replay and schema evolution

The log is retained (days by default, forever with compaction or tiered storage), so you can replay: reset a group’s offsets to a time and reprocess. That rebuilds a broken read model, backfills a new service, or recovers from a consumer bug. Replay is only safe because handlers are idempotent, and only useful if old events are still readable by new code.

Log compaction keeps the latest event per key instead of a time window. That suits “current state” topics (latest order status), not “history” topics (every order event).

Schema evolution is the long-term risk. Producers and consumers deploy independently, so every change must work in both directions for a while:

Change	Backward compatible? (new reader, old data)	Safe?
Add an optional field with a default	Yes	Yes
Remove an optional field	Yes, if readers tolerate its absence	Usually
Rename a field / change its type	No	No: add a new field, migrate, then remove
Change meaning of a field (cents to dollars)	Silently no	Never: new field or new event type

A schema registry (Confluent’s defaults to BACKWARD compatibility) rejects incompatible schemas at publish time, before they reach consumers. For a breaking change, publish a new event type or version on the side, migrate consumers, then retire the old one.

Failure modes

What breaks	What you see	Design response
Crash between DB write and publish	Orders that never progress, or charges for ghost orders	Transactional outbox or CDC; never dual-write
Broker unavailable	Publish errors	Outbox absorbs it; orders keep committing; relay retries with backoff
Consumer crash or redeploy	Redelivered events after rebalance	Commit after processing; idempotent handlers with a dedupe table
Poison event	One partition’s lag climbs, others are fine	Bounded retries, DLQ with metadata, alert, replay tool
Consumers too slow	Lag grows steadily	Scale consumers up to P, batch work, then repartition; alert on lag in seconds
Hot key (a flash-sale SKU keyed by SKU)	One partition lags, rest idle	Key by a finer entity; split work within the handler; dedicated topic
Payment fails mid-saga	Stock reserved for a dead order	Compensation event; saga timeout; reconciliation job
Incompatible schema deployed	Deserialization errors across consumers	Registry compatibility checks in CI and at publish
Lag exceeds retention	Events deleted before being read	Retention sized for worst-case outage; lag-in-time alert well below it
Consumer bug wrote bad data	Corrupt downstream state	Fix, reset offsets, replay (requires idempotency and old-schema reads)

What it costs

Operation	Time	Space
Append an event (producer)	O(1) amortized; one batched network write, acks from replicas	O(event size) × replication factor
Read next events (consumer)	O(1) per event, sequential disk/page-cache reads	Committed offset: one integer per (group, partition)
Outbox write	One extra row in the same transaction	O(unsent events); pruned after sending
Dedupe check	O(1) index lookup in the consumer’s DB	O(events in dedupe window) per consumer
Rebalance	Pause of seconds in that group	—
Storage	—	events/day × size × retention days × replication; ≈250 GB for the defaults (84 GB before replication)
Throughput	A partition handles thousands to tens of thousands of events/s; a consumer is usually the bottleneck	Partition count ≈ peak / per-consumer rate × headroom

In the interview

A good 45-minute arc:

1. Clarify (5 min). Does checkout need a final payment answer? Which steps can be undone? Ordering scope (per order?) and duplicate tolerance per effect.
2. Sketch the flow (8 min). Order service, event log, one consumer group per service. Name events as facts. State per-key ordering via the partition key.
3. Make publishing safe (8 min). Show the dual-write failure, then the outbox (or CDC). State that delivery is now at least once.
4. Make consuming safe (10 min). Idempotent handlers with a dedupe table in the same transaction. Offsets committed after processing. Saga with compensations, choreography or orchestration, and why.
5. Operate it (10 min). Lag as the core metric, DLQs for poison events, replay, schema evolution, partition sizing.
6. Stretch (5 min). Multi-region (replicate topics, keep a key’s writes in one region), exactly-once inside Kafka with transactions, an event-sourced order.

Follow-ups to expect: “How do you guarantee the email is sent exactly once?” You can’t across an SMTP provider; dedupe on your side with the event id and accept a rare duplicate. “How do you know where an order is stuck?” Orchestrator state, or a trace id on every event plus a per-order status projection. “Can you add partitions later?” Yes, but keys remap; drain or accept a brief ordering break. “What if two events for an order are produced by two different services?” Both key by order id, but ordering holds only within each topic; consumers that join topics must tolerate either arriving first (use versions or state checks).

Check yourself

Check yourself

pick one

An orders topic has 12 partitions keyed by order_id. The payments group runs 20 instances because lag is high. What happens?

AEach instance gets 0.6 partitions and lag drops proportionallyBTwelve instances each own one partition and eight sit idle; lag improves only up to the 12-consumer limitCKafka splits each partition between two consumers to keep orderingDThe group fails to start

Check yourself

pick one

The order service commits the order row and then calls producer.send(OrderPlaced). A deploy kills pods every few hours. What failure appears, and what's the fix?

ADuplicate events; make consumers idempotentBOccasional orders with no event, which never get charged; write the event to an outbox table in the same transactionCOut-of-order events; key by customer_idDGhost events; publish before committing

Check yourself

pick one

A consumer processes events, writes to its database, and commits Kafka offsets every 5 seconds. After a crash, some stock was reserved twice. Which change fixes it without slowing throughput much?

ACommit the offset before processing each eventBCommit offsets after every eventCRecord event_id in a processed_events table inside the same DB transaction as the reservation, and skip seen idsDLower the session timeout

Go deeper

• Apache Kafka documentation: Design: partitions, consumer groups, offsets, delivery semantics and log compaction, from the source.
• microservices.io: Transactional outbox and Saga: Chris Richardson’s pattern write-ups, choreography vs orchestration.
• Confluent: Exactly-once semantics in Apache Kafka: the idempotent producer, transactions, and what they do and don’t cover.
• Garcia-Molina & Salem, “Sagas” (1987): the original paper on long-lived transactions with compensations. Also: Debezium outbox event router for CDC-based outboxes.

Concept pages to drill: Event-driven · Idempotency · State machines · Streaming · Distributed thinking · Versioning

Related

Event-driven (concept)Pub/sub, queues and logs from first principles.

Job queueLeases, retries and backoff for background work.

Payment systemWhere the charge step of this saga goes deep.

Webhook deliveryThe same events, delivered to other companies.