Webhook delivery

intermediateCommon atStripeShopifyGitHubTwilioAnthropicRead firstEvent-drivenIdempotencySchedulingPractise in the Sandbox →

Webhooks are how a platform tells its customers’ servers that something happened: charge.succeeded, push, order.paid. Underneath, it’s a job queue where every job is an HTTP POST to a server you don’t control. The design follows from that: store each event once, fan out one delivery per subscribed endpoint, sign each payload with HMAC and a timestamp so receivers can verify it, retry with backoff for days, disable endpoints that stay dead, and give each endpoint its own queue and concurrency cap so one slow customer can’t delay everyone else. Delivery is at least once and unordered, so receivers must dedupe by event id. Each delivery costs O(1) work, and the real costs are open connections (attempts/s × response time) and egress.

The problem

A payments platform processes a charge. The merchant’s server needs to know so it can ship the order. Polling the API every few seconds for millions of merchants is wasteful, so the platform pushes: an HTTP POST to a URL the merchant registered.

What makes this hard is that the receiving side is out of your hands. Customer endpoints are slow, flaky, misconfigured, behind firewalls, deleted without notice, and occasionally malicious (pointing the URL at your internal network). Every requirement is about staying correct and fair in spite of them.

Goal	What it means	Pulls toward
Reliability	Every event reaches every subscribed endpoint eventually, through outages lasting hours or days	Durable events, long retry schedules, replay API
Isolation	One slow or dead endpoint doesn’t delay or starve anyone else	Per-endpoint queues, concurrency caps, timeouts, auto-disable
Security	Receivers can trust a webhook is genuine and fresh; senders can’t be turned against internal systems	HMAC signatures, timestamps, secret rotation, SSRF-safe egress
Debuggability	Customers can see what was sent, what came back, and resend	Attempt log, dashboard, manual retry

Requirements and scope

	In scope (typical)	Ask about
Functional	Register endpoints (URL, event types, secret). Deliver matching events as signed POSTs. Retry failures. Dashboard of attempts; manual resend; events API for catch-up.	Can endpoints filter by event type? Per-account or per-sub-account endpoints? Do customers need ordering?
Delivery	At least once, no ordering guarantee, signed.	Is “at most once” ever wanted? (Almost never.) How long must you keep retrying?
Scale	~500M events/day, 1-2 endpoints per event on average, some accounts with 10+.	What’s the biggest single account’s event rate?
Latency	p50 delivery under 1 s after the event when the endpoint is healthy.	Is there a premium tier with a tighter SLO?
Availability	Core product (charges) never blocks on webhooks.	Can event creation lag the business write by seconds?
Out of scope	Inbound webhooks, guaranteed exactly-once processing on the receiver.

The clarifying question that changes the most

“Do receivers need events in order?” If yes, you need per-endpoint (or per-object) serialization: one delivery in flight at a time, and a failing event blocks everything behind it for that endpoint, possibly for days. Stripe, GitHub and most platforms answer no: they deliver unordered, and tell receivers to fetch the current state from the API when order matters. That single choice is what lets retries, parallelism and isolation stay simple.

Back of the envelope

Back of the envelope

drag the assumptions

assumptions

events per day500Mendpoints per event (avg)1.50×first attempts that fail5.00%average endpoint response (s)0.4 spayload size (bytes)3 KBevent retention (days)30

so you need

deliveries/s (average)8.68K/s

attempts/s including retries10.4K/sa failing delivery averages ~4 more tries

open HTTP connections (Little's law)4.17Kattempts/s × seconds per attempt

egress bandwidth31.3 MB/s

event storage for retention45 TB

500M events/day × 1.5 endpoints is ~8,700 deliveries/s, ~10k attempts/s with retries. At 0.4 s per response that's ~4,000 open connections; if many endpoints slow to 10 s, it's 100,000. Connection count, not CPU, sizes the sender fleet, which is why senders use async I/O and hard timeouts.

What to say out loud: “Throughput is modest, about ten thousand attempts per second. The variable that matters is customer response time, because open connections equal attempts per second times latency. A slow endpoint multiplies my fleet size, so timeouts and per-endpoint concurrency caps are the first design decisions.”

API and data model

pseudocode: the contract

Customer-facing:
    POST /webhook_endpoints { url, enabled_events: ["charge.succeeded", ...] } -> { id, secret: "whsec_..." }
    POST /webhook_endpoints/{id}/rotate_secret                                 -- old secret valid 24 h
    GET  /events?created[gte]=T                                                -- catch-up / replay
    POST /events/{id}/resend?endpoint={id}

What a receiver gets:
    POST https://merchant.example/hooks
    webhook-id:        evt_1Q...                 -- dedupe key
    webhook-timestamp: 1763164800                -- seconds; reject if too old
    webhook-signature: v1,<base64 HMAC-SHA256(secret, id + "." + timestamp + "." + body)>
    body: { id, type: "charge.succeeded", created, data: { object: {...} } }

Stored:
    events(id, account_id, type, payload, created_at)                  -- written once, 30 days
    endpoints(id, account_id, url, secrets[], event_types, state: enabled | disabled, failing_since)
    deliveries(event_id, endpoint_id, state, attempts, next_attempt_at, last_status, last_error)

The header names are from the Standard Webhooks spec; Stripe’s equivalent is one Stripe-Signature: t=...,v1=... header. Two decisions hide in the sketch:

• Events are immutable, deliveries are mutable. An event is written once, then shared by every endpoint and every retry. Delivery state (attempts, next try) is per (event, endpoint). That keeps fan-out cheap and lets the events API serve replays.
• Thin or fat payloads. A fat payload carries the full object, which is convenient but stale by the time a retry lands. A thin payload ({ id, type }) makes the receiver fetch fresh state from the API, which also solves ordering. Many platforms send fat payloads and recommend refetching when it matters.

High-level design

Outbound webhooks: event to customer server

step through

flowDeliverRetryEndpoint diesSlow endpoint

An event is stored once, fanned out to every matching endpoint, signed and POSTed.

eventevt_1Q · charge.succeeded

accountacct_9

The charge and its event commit in one transaction (an outbox), so no charge exists without its event.

↺◀▶ Play▶1 / 6

Component	Role	Why it’s there
Events DB + outbox	Event written in the same transaction as the business change.	No charge without its event (see the outbox).
Fan-out	Matches an event to subscribed endpoints, creates one delivery each.	Decouples “something happened” from “who wants to know”.
Delivery queues	Logically one queue per endpoint, with its own concurrency cap and retry timers.	Isolation: failures and slowness stay with their endpoint.
Senders (×N)	Async HTTP clients: sign, POST with a hard timeout, record the attempt.	Holding many connections cheaply matters more than CPU.
Egress proxy	Resolves DNS and blocks private, loopback and metadata IPs.	Customer-chosen URLs are an SSRF vector into your network.
Endpoints DB	URLs, secrets, subscriptions, health state.	Read on every fan-out; cached heavily.

Deep dive: retries over days, and why order is lost

A customer’s server going down for a deploy, a weekend, or until someone notices is normal. So retries must cover days, with few enough attempts that a dead endpoint doesn’t cost much. Stripe retries for up to three days with exponential backoff in live mode; Svix uses a fixed ladder (immediately, 5 s, 5 min, 30 min, 2 h, 5 h, 10 h, 10 h). Try each against a short outage, a 30-hour one and a 4-day one.

Retrying a flaky endpoint over days

simulator

reshuffle

endpoint down for = 3 hthen succeeds = 70% of tries

delivered: 6 / 6attempts: 35worst delay after recovery: 14.6harrived out of order: 10 pairs

arrival order: evt_5 → evt_6 → evt_1 → evt_3 → evt_4 → evt_2

endpoint down / failed attemptdelivered (2xx)

A fixed ladder front-loads quick retries (most blips last seconds) and then spaces out to hours, so 8 attempts cover about 28 hours. Each event follows its own ladder from its own creation time, which is why they recover out of order.

What to notice:

• Each event retries on its own clock, from its own creation time. After an outage they arrive in an order that has nothing to do with when they were created. Ordering is lost by retries even if the first attempts were in order.
• Long gaps after recovery. With exponential backoff, an endpoint that came back at hour 3 may not see its next attempt until hour 5. Platforms offset this with a “resend” button, an events API for catch-up, and sometimes a fast-track: once an endpoint succeeds again, pull its pending retries forward.
• Give up, then disable. A schedule ends. An endpoint failing for days gets disabled with an email to the owner, so the platform stops spending senders on it.

The receiver’s side of the contract:

pseudocode: receiving a webhook safely

on POST /hooks:
    raw <- request body bytes                         -- verify the bytes, not re-serialized JSON
    if abs(now - header.timestamp) > 5 min: return 400   -- stops replay of a captured request
    expected <- HMAC_SHA256(secret, header.id + "." + header.timestamp + "." + raw)
    if not any(constant_time_equal(expected, sig) for sig in header.signatures): return 400
    if INSERT INTO seen_events(id) ON CONFLICT DO NOTHING inserted nothing: return 200   -- duplicate
    enqueue local job(raw)                            -- do the work async
    return 200 within a few seconds                   -- slow handlers cause timeouts and retries

Why each line matters: the timestamp is inside the signed string, so an attacker can’t replay an old request with a fresh timestamp. Constant-time comparison avoids leaking the signature byte by byte through timing. Multiple signatures in the header let the platform rotate secrets: during rotation it signs with both old and new. Dedupe by event id handles at-least-once delivery (see Idempotency). Ack fast, process later: the sender’s timeout is seconds, and a handler doing real work inline turns into retries and duplicates.

Deep dive: isolating slow endpoints

Most outages of a webhook system are caused by one customer’s endpoint. It doesn’t fail cleanly; it gets slow, and every delivery to it holds a sender connection for the full timeout. In a shared FIFO queue those deliveries crowd out everyone else. Watch the healthy endpoints’ delay in each mode.

One slow endpoint: shared queue vs bulkheads

simulator

delivery workers = 4endpoint A responds in = 10 ticksevents / tick (all endpoints) = 2

workers

worker 1

A

worker 2

B

worker 3

worker 4

endpoint	waiting	delivered	avg delay
A (slow)	0	0	–
B	0	0	–
C	0	0	–
D	0	0	–
E	0	0	–

healthy endpoints B-E, avg delay: – ticksworkers stuck on A: 1 / 4

fast deliverywaiting on slow endpoint Atimed out / backlog

A gets a fifth of the events but each one holds a worker for 10 ticks. Soon every worker is waiting on A, and B to E queue behind A's events in the same FIFO: head-of-line blocking. Healthy customers see delays because of someone else's server.

↺◀▶ Run▶t = 0 / 48

Technique	What it stops	Cost
Hard timeout (e.g. 10-30 s total, a few seconds to connect)	One request holding a connection forever	Slow but working endpoints see timeouts and retries
Per-endpoint queue + concurrency cap	A slow endpoint taking more than its share of senders	Per-endpoint scheduling state; many mostly-empty queues
Per-endpoint rate limit	Overwhelming a customer’s small server (you are the traffic spike)	Backlog builds at the platform instead
Adaptive cap (shrink on timeouts and 5xx, grow on success, like TCP’s AIMD)	Static caps being wrong for both tiny and huge customers	More moving parts
Separate pools by tier or health	Known-bad endpoints sharing a pool with healthy ones	Routing logic; pool sizing
Circuit breaker per endpoint	Spending attempts on an endpoint that fails everything	Delays the first success after recovery

“Queue per endpoint” rarely means one physical queue per endpoint. It means scheduling by endpoint: deliveries are stored with (endpoint_id, next_attempt_at), and a scheduler hands senders work from endpoints that are under their in-flight cap, round-robin. That is weighted fair queuing again, the same idea as tenant fairness in a job queue.

pseudocode: choose the next delivery for a free sender

for endpoint in ready_endpoints.round_robin():           -- endpoints with due deliveries
    if inflight[endpoint] >= cap[endpoint]: continue      -- bulkhead
    if breaker[endpoint] is open: continue                -- probe occasionally instead
    d <- oldest due delivery for endpoint
    inflight[endpoint] += 1
    return d
on attempt done(endpoint, ok, latency):
    inflight[endpoint] -= 1
    cap[endpoint] <- ok ? min(max_cap, cap + 1) : max(1, cap / 2)   -- AIMD

Deep dive: security and fan-out at scale

SSRF. A customer can register http://169.254.169.254/latest/meta-data/ (a cloud metadata service) or http://10.0.0.5/admin. If senders follow it, your platform attacks itself. Validate at registration and at send time (DNS can change to a private IP after validation; that’s DNS rebinding): resolve once, check the IP against deny-lists, and connect to that exact IP. Stripe open-sourced Smokescreen, an egress proxy that does this. Don’t follow redirects, or re-check each hop.

Secrets. Each endpoint has its own secret, shown once. Rotation keeps the old secret valid for a window and signs with both. Some platforms also offer asymmetric signatures (Ed25519) so receivers hold only a public key.

Fan-out. Most events match 0-2 endpoints, but platforms like Shopify see apps subscribed across hundreds of thousands of shops, and a single account can have dozens of endpoints. Fan-out work is O(matching endpoints) per event; index subscriptions by (account, event type) and cache them. A bulk operation (a merchant importing 1M products) produces a burst for one account: the per-endpoint cap absorbs it as backlog rather than letting it take the fleet.

Failure modes

What breaks	What you see	Design response
Endpoint down for hours	Retries accumulating	Backoff schedule over days; events API for catch-up; fast-track after recovery
Endpoint permanently gone	Endless failures	Auto-disable after N days of 100% failure; email owner
Endpoint slow	Senders tied up; others delayed	Hard timeouts; per-endpoint concurrency caps; separate pool for unhealthy endpoints
Receiver processes slowly inline	Timeouts, then duplicates	Docs + SDKs: verify, store, return 2xx, process async
Duplicate delivery	Receiver double-processes	Stable event id in a header; receivers dedupe
Out-of-order delivery	subscription.updated before .created	No ordering promise; include created and object version; recommend refetching state
Replayed captured request	Fake action on receiver	Timestamp in the signed string; tolerance window (Stripe’s libraries default to 5 min)
SSRF via customer URL	Requests to internal hosts	Egress proxy with IP deny-list, resolve-then-connect, no redirects
Event created but not delivered	Missing webhooks after a crash	Outbox for event creation; deliveries are durable rows
Massive burst from one account	Shared backlog	Per-account/endpoint caps; burst becomes local backlog

What it costs

Operation	Time	Space
Create event	One insert in the business transaction	O(payload), kept ~30 days
Fan-out	O(matching endpoints), via an index on (account, type)	One delivery row per (event, endpoint)
One attempt	O(payload) to sign (HMAC is linear in bytes) + one HTTP round trip	Attempt log row (status, latency, snippet)
Choose next delivery	O(1) amortized with round-robin over ready endpoints	O(endpoints with pending work) scheduler state
Connections	attempts/s × response time (≈4,000 at defaults)	Memory per connection in senders
Storage	—	events/day × size × days ≈ 45 TB at 500M/day × 3 KB × 30 days

In the interview

A good 45-minute arc:

1. Clarify (5 min). Ordering? Retry duration? Scale and the biggest account? Payload style? Security expectations?
2. Core path (10 min). Event stored with the business write, fan-out to deliveries, sender POSTs, 2xx means done. State at-least-once and no ordering, and what receivers must do.
3. Retries and health (10 min). Backoff schedule over days, jitter, give-up, auto-disable, manual resend, events API.
4. Isolation (10 min). Timeouts, per-endpoint queues and caps, adaptive concurrency, separate pools. Connection math.
5. Security (5-10 min). HMAC over id + timestamp + body, tolerance window, constant-time compare, secret rotation, SSRF.

Follow-ups to expect: “A customer insists on ordering.” Offer per-object versions or a thin payload plus refetch; true ordering means one in flight per endpoint and blocking on failures. “How do you test this?” A fake endpoint fleet with programmable latency and error rates; chaos tests where 5% of endpoints hang. “Can a receiver tell it missed a webhook?” Only by reconciling against the events API; recommend a periodic sync. “Why not let customers poll?” Offer both: webhooks for latency, the events list for completeness.

Check yourself

Check yourself

pick one

A webhook sender uses one shared FIFO queue and 200 senders with a 30 s timeout. One large customer's endpoint starts timing out on every request and receives 10% of all deliveries. What happens to other customers?

ANothing; timeouts cap the damage at 30 s per requestBTheir deliveries slow down as most senders end up waiting 30 s on the one bad endpointCTheir deliveries fail with 503DThe bad endpoint is disabled immediately

Check yourself

pick one

Why does a webhook signature cover the timestamp as well as the body?

ATo let the receiver measure latencyBSo an attacker who captures a valid request cannot replay it later with a fresh timestamp; the receiver rejects stale timestampsCBecause HMAC requires at least two inputsDTo guarantee ordering

Check yourself

pick one

A merchant's server was down for 6 hours. After it recovers, it receives invoice.paid for an invoice before invoice.created. What's the right design response from the platform?

AGuarantee ordering by retrying events strictly in sequence per endpointBDocument that order isn't guaranteed, include created timestamps and ids, and let receivers fetch current state from the APICDrop older events once newer ones succeedDStop retrying after the first failure

Go deeper

• Stripe docs: Receive webhook events: retries for up to three days, no ordering guarantee, duplicates, and quick 2xx responses. Plus signature verification and its 5-minute tolerance.
• Standard Webhooks specification: the webhook-id / webhook-timestamp / webhook-signature headers, signing scheme and key rotation.
• Svix docs: Retries: a production retry ladder and endpoint disabling.
• Stripe Smokescreen: an egress proxy that keeps customer-supplied URLs from reaching internal hosts. Also GitHub: Best practices for using webhooks.

Concept pages to drill: Event-driven · Idempotency · Scheduling · Rate limiting · Time handling · API design

Related

Job queueLeases, backoff with jitter and fairness: the engine under webhooks.

Event-driven order pipelineWhere the events come from: outbox, logs and consumers.

Payment systemThe charges these webhooks announce.

Idempotency (concept)Deduping at-least-once deliveries on the receiver.