Notification system
A notification system turns “something happened to user 7” into the right message, on the right channel, at the right time, or deliberately into nothing. Product services call one API with a type and a user. A router applies preferences and opt-outs, quiet hours in the user’s time zone, and templates. It puts one message per channel on priority queues per channel, so an OTP never waits behind a marketing blast. Senders enforce per-user and per-provider rate limits, call providers (APNs, FCM, email, SMS) with dedupe keys, and fail over between providers. Every message has a tracked status. Each send is O(1) work; the hard parts are policy (don’t annoy people) and isolation (don’t let bulk delay urgent).
The problem
Section titled “The problem”Every consumer app sends notifications: “your order shipped”, “your login code is 482913”, “Ana liked your photo”, “spring sale starts tomorrow”. Without a central system, each team calls APNs and an email API directly. Then users get four pushes in a minute from four teams, unsubscribes are ignored by one team, and an SMS provider outage breaks login.
A notification system centralizes the policy. The design is a pipeline with a few competing goals:
| Goal | What it means | Pulls toward |
|---|---|---|
| Timely when it matters | OTPs and security alerts arrive in seconds, always | Priority queues, provider failover, bypass rules |
| Respectful | Users control what they get; no floods, no 3 AM buzzes | Preferences, frequency caps, digests, quiet hours |
| Reliable and exactly-once-ish | Nothing important is lost; nothing is sent twice | Durable queues, dedupe keys, idempotent provider calls |
| Cheap and compliant | SMS costs money; email reputation and unsubscribe rules are legal and deliverability issues | Rate limits, suppression lists, one-click unsubscribe |
Requirements and scope
Section titled “Requirements and scope”| In scope (typical) | Ask about | |
|---|---|---|
| Functional | notify(user, type, data) from any service; channels push (iOS/Android/web), email, SMS, in-app inbox. Preferences per category and channel. Templates with locales. Campaigns to segments. Status tracking. |
Which categories exist (transactional, security, social, marketing)? Can users choose channels per category? |
| Delivery | At least once internally; deduped so the user sees one. Some messages expire (an OTP after 10 min). | What happens to a message that’s late? Send anyway, or drop? |
| Scale | 50M DAU, ~5 notifications/user/day, campaigns 10× peaks. | Largest campaign size and how fast it must go out? |
| Latency | OTP / security under 5 s end to end. Transactional under 30 s. Marketing: minutes are fine. | Per-category SLOs? |
| Availability | Login (OTP) never fails because one provider is down. | Multiple providers per channel? |
| Out of scope | Building APNs/FCM/SMTP, recommendation of what to notify, the email editor. |
Back of the envelope
Section titled “Back of the envelope”What to say out loud: “Sends per second are moderate; campaigns are the peak, so they go through a rate-controlled batch path on low-priority queues. SMS is the expensive channel and an abuse target, so I’ll put strict per-user and per-destination limits on it and prefer push where possible.”
API and data model
Section titled “API and data model”POST /notifications -- called by product services { user_id, type: "order_shipped", data: { order_id, eta }, dedupe_key: "ship:42", priority?: override, expires_at?: time } -> 202 { notification_id }
POST /campaigns { segment_query, type, data, send_at, rate_per_s }
Types (config, owned by the platform): order_shipped = { category: transactional, priority: high, channels: [push, email], template: "order_shipped@v4", ttl: 24 h, quiet_hours: respect } login_otp = { category: security, priority: critical, channels: [sms], fallback: [voice], ttl: 10 min, quiet_hours: bypass }
Stored: preferences(user_id, category, channel, enabled) -- defaults per category users(user_id, tz: "Europe/Berlin", locale, email, phone) devices(user_id, platform, token, last_seen) -- many per user; prune on 410 / unregistered messages(id, notification_id, user_id, channel, provider, status, attempts, timestamps) status: queued -> sent -> delivered -> opened | failed | suppressed(reason) | expiredTwo decisions hide in the sketch:
- Callers send events, not messages. A product team says “order shipped for user 7”, not “push this text to this token”. Channel choice, wording, timing and caps live in the platform, so policy is consistent and users’ choices are honored everywhere.
- A notification fans out to messages. One notification becomes one message per (channel, device). Status is tracked per message; the notification’s status is a rollup. “Suppressed” is a status, not silence, so support can answer “why didn’t I get it?”.
High-level design
Section titled “High-level design”| Component | Role | Why it’s there |
|---|---|---|
| Notification API | Validates, dedupes on the caller’s key, returns 202. | Product teams get one fast, idempotent call. |
| Preferences store | Per-user, per-category, per-channel choices; time zone; locale; devices. Cached. | Read on every notification; the source of legal opt-outs. |
| Router | Applies preferences, quiet hours, frequency caps; renders templates; picks channels. | All policy in one place. |
| Delay / digest queue | Holds messages until a release time (quiet hours end, digest window closes). | Waiting is a first-class outcome, not a dropped message. |
| Channel queues | One per channel × priority (sms.critical, push.high, email.low). |
Isolation: channels fail and scale independently; bulk never delays urgent. |
| Channel senders | Per-provider clients with rate limits, breakers, retries. | Each provider has its own protocol, limits and failure modes. |
| Delivery log | Status per message, updated by provider receipts and app pings. | Debugging, analytics, deliverability, compliance. |
| In-app inbox | Durable list per user, shown in the app. | The record of truth; push is just a doorbell. |
Deep dive: the pipeline, one decision per stage
Section titled “Deep dive: the pipeline, one decision per stage”Pick a scenario and step through it. Each stage either passes the message on, holds it, or stops it with a recorded reason.
The decisions that come up:
- Dedupe at the edge.
SET dedupe_key NX EX 86400on a fast store. A hit returns the original notification id. That turns upstream retries (very common: callers time out and retry) into no-ops. See Idempotency. - Preferences are checked twice for delayed messages: at enqueue and again at release. A user who unsubscribes at midnight shouldn’t get the 07:00 promo.
- Expiry. Every message carries
expires_at. An OTP that’s 10 minutes late is worse than none; a “your driver is arriving” push from 20 minutes ago is noise. Senders drop expired messages and mark themexpired. - Channel fallback is different from provider failover. Failover tries another provider for the same channel (Twilio → another SMS vendor). Fallback tries another channel (push failed because the user has no device token → email). Both are configured per type.
send(message): if now > message.expires_at: mark expired; return for provider in routing_order(message.channel, message.country): -- by weight, health, cost if breaker[provider].open: continue result <- provider.send(message, idempotency_key = message.id, timeout = budget_for(priority)) if result.ok: mark sent(provider, result.provider_id); return breaker[provider].record_failure() if result.permanent: -- invalid token, unsubscribed number, bad address handle permanent (prune token, add to suppression list); mark failed; return requeue with backoff + jitter, or fall back to the next channel if configuredProvider specifics worth knowing: APNs is HTTP/2 with JWT auth; apns-priority: 10 means deliver now, 5 means power-friendly; apns-collapse-id replaces an older undelivered notification with a newer one; a 410 Unregistered response means the device token is dead and must be removed. FCM’s HTTP v1 API has collapse_key, high vs normal priority, and a TTL. Both cap payloads at about 4 KB, which is why pushes carry ids, not documents.
Deep dive: don’t annoy people (caps, digests, quiet hours)
Section titled “Deep dive: don’t annoy people (caps, digests, quiet hours)”The fastest way to lose a user’s notification permission is to send too much. On iOS, a user who turns off notifications for your app takes away the whole push channel. So the system decides what not to send. Compare the three policies over a day of bursty social events.
| Technique | How | Trade-off |
|---|---|---|
| Per-user frequency cap | Sliding-window count per (user, category); over the cap, skip the push, keep the inbox entry | Bounded interruptions; arbitrary which events survive |
| Digest / batching | First event opens a window; later events join; one summary at the end (“Ana and 11 others…”) | Nothing lost, fewer pushes, adds up to one window of latency |
| Collapse / replace | Newer push replaces an undelivered older one (apns-collapse-id, FCM collapse_key) |
Free on the device side; only for “latest state” messages (score, ETA) |
| Quiet hours | Hold non-urgent messages until the user’s morning, in their time zone | Needs reliable time zones; DST; per-region legal windows (e.g. US marketing SMS rules) |
| Ranking / budget | Daily budget per user; send the highest-value events | Needs a model of value; best at large scale |
Quiet hours need the user’s IANA time zone (Europe/Berlin, not “UTC+1”), because offsets change with daylight saving. “07:00 local” is computed per user with a time zone library; see Time handling. A held message goes into a delay queue keyed by release time, the same mechanism as delayed jobs in a job queue. Releasing 10M messages at exactly 07:00 per zone creates a spike each hour, so add a few minutes of jitter.
on event(user, category, item): key <- "digest:" + user + ":" + category items_count <- RPUSH key item -- append, returns the new length if items_count == 1: -- first item opens the window schedule release(user, category) at now + window(category)on release(user, category): items <- read and delete key atomically if items is empty: return send one message rendered from items ("{first} and {n-1} others liked your post")Deep dive: priorities, campaigns and scale
Section titled “Deep dive: priorities, campaigns and scale”Priority isolation. Separate queues and sender pools per priority (critical, high, low) per channel. A 50M-user campaign then fills push.low and its lag doesn’t touch push.high. Within one priority, partition by user id so per-user state (caps, digests) stays on one consumer.
Campaigns are batch jobs. Don’t accept 50M recipients in one call. Store the segment query, resolve it in pages by a batch job, and feed the pipeline at a configured rate (rate_per_s), so providers’ throughput limits and your own fleet are respected. Throttle per provider too: email providers enforce sending rates, and ramping a new sending IP or domain too fast hurts deliverability.
Email deliverability is a system constraint. Gmail and Yahoo require bulk senders to authenticate (SPF, DKIM, DMARC), offer one-click unsubscribe (RFC 8058), and keep spam complaint rates low (Google’s threshold is 0.3%). So: maintain a suppression list (hard bounces, complaints, unsubscribes) checked before every send; separate transactional and marketing sending domains, so a bad campaign doesn’t land receipts in spam.
Delivery tracking. Providers report asynchronously: APNs only tells you it accepted the push; email providers send bounce and complaint webhooks; SMS carriers send delivery receipts (DLRs). Write one status row per message and update it from these events. Opens come from the app (push opened) or tracking pixels (email, increasingly unreliable). This log, keyed by message id, is large but append-mostly: keep 30-90 days hot.
Failure modes
Section titled “Failure modes”| What breaks | What you see | Design response |
|---|---|---|
| SMS / email provider outage | Timeouts, 5xx, OTPs not arriving | Per-provider circuit breakers; weighted multi-provider routing; failover with the same idempotency key |
| Caller retries | Duplicate notifications | Dedupe key at the API (24 h); idempotency keys to providers |
| Campaign floods the system | Transactional latency rises | Separate priority queues and pools; rate-controlled campaign feeder |
| Stale device tokens | High push failure rate | Prune on 410 Unregistered / FCM unregistered errors; refresh tokens on app start |
| Notification storm (bug loops events) | One user gets hundreds | Per-user caps as a safety net even for transactional; anomaly alerts |
| Wrong time zone / DST | 3 AM pushes | Store IANA zone; compute release times with a tz library; default conservatively |
| OTP abuse (SMS pumping) | Huge SMS bill to premium numbers | Per-number and per-IP limits, country allow-lists, CAPTCHA before sending |
| Unsubscribe not honored | Complaints, legal exposure | Single preferences service checked at send time; suppression list; recheck on release |
| Late delivery | Old “driver arriving” push | expires_at per message; drop and mark expired |
What it costs
Section titled “What it costs”| Operation | Time | Space |
|---|---|---|
| API call with dedupe | O(1): one set-if-absent on a cache | O(keys in 24 h window) |
| Routing | O(channels × devices) per notification; cached prefs lookup | Prefs ≈ users × categories × channels (small rows) |
| Frequency cap check | O(1) with a counter per window (rate limiting) | O(active users) |
| Digest | O(1) append; one send per window | O(items held) until release |
| Provider call | One HTTP round trip (APNs over a persistent HTTP/2 connection) | — |
| Tracking | One write per status change (≈3 per message) | ≈ 2 TB/month at 250M/day × 300 B |
| Throughput | ≈ 29k sends/s peak at defaults, sharded queues | — |
In the interview
Section titled “In the interview”A good 45-minute arc:
- Clarify (5 min). Channels, categories and their urgency, preferences granularity, scale, campaign size, multi-provider.
- API and flow (10 min).
notify(user, type, data, dedupe_key); router with preferences and templates; per-channel queues; senders; tracking. Callers send events, not text. - Reliability (10 min). Dedupe at the edge, idempotent provider calls, retries with backoff, circuit breakers and provider failover, expiry.
- User experience (10 min). Frequency caps, digests, collapse ids, quiet hours in local time, recheck preferences on release.
- Scale and cost (5-10 min). Priority isolation, campaign feeder, SMS cost and abuse, email deliverability rules.
Follow-ups to expect: “How do you guarantee the user sees it?” You can’t; push is best effort. The in-app inbox is the record, and critical messages fall back across channels. “A user has five devices.” Send to all active tokens; collapse on the others when one is opened (or let the OS do it). “How would you A/B test copy?” Template versions assigned by user hash; measure opens from the tracking log. “Real-time in-app?” Write to the inbox, then push over a websocket if the user is connected (see chat system).
Check yourself
Section titled “Check yourself”Go deeper
Section titled “Go deeper”- Apple: Sending notification requests to APNs: HTTP/2 requests, priority, collapse ids, expiration and error codes such as 410.
- Firebase Cloud Messaging documentation: the HTTP v1 API, message priority, TTL and collapsible messages.
- Google: Email sender guidelines: authentication, one-click unsubscribe and spam-rate thresholds for bulk senders, with RFC 8058 for the unsubscribe header.
- Amazon SES: Deliverability concepts: bounces, complaints, suppression lists and reputation.
Concept pages to drill: Event-driven · Rate limiting · Time handling · Idempotency · Scheduling · TTL and expiry