Metrics pipeline
A metrics system stores time series: a name plus labels (http_requests_total{route="/pay",code="500"}) and a stream of (timestamp, float) samples. Writes are huge and uniform, reads are range scans over recent data. The core decisions: cost scales with the number of series (cardinality), not samples, so you police labels; samples are compressed to ≈1.4 bytes (delta-of-delta timestamps, XOR values, from Facebook’s Gorilla) in 2-hour blocks; old data is rolled up into coarser tiers; and percentiles come from mergeable histograms, never from averaging per-host p99s. 10M series at 15 s is ≈670k samples/s and ≈1.3 TB for 15 days.
The problem
Section titled “The problem”Every service exports numbers: request counts, error counts, latency, queue depth, CPU. Engineers need to graph them, compare against last week, and get paged when something breaks. Prometheus, Datadog, Monarch (Google), Gorilla (Facebook) and M3 (Uber) all solve this.
The goals pull against each other:
| Goal | What it means | Pulls toward |
|---|---|---|
| Ingest everything | Millions of samples/s, bursty, never block the app | Async, batched, lossy under overload |
| Fast recent queries | Dashboards and alerts read the last hours in milliseconds | Recent data in memory |
| Long history, cheaply | A year of trends for capacity planning | Compression, rollups, object storage |
| Correct aggregates | p99 across 500 hosts must be the real p99 | Mergeable data structures |
| Alert when it matters | Page within a minute, but not on noise | Frequent evaluation, for durations, dedupe |
Unlike a database of records, a metrics system is allowed to be slightly lossy: a dropped sample barely changes a graph. That relaxation is what makes the scale affordable.
Requirements and scope
Section titled “Requirements and scope”| In scope (typical) | Ask about | |
|---|---|---|
| Functional | Ingest counters, gauges and histograms with labels; query by label match over a time range; aggregate (sum, rate, quantile); alert rules; dashboards. | Push or pull? Logs and traces too, or metrics only? Multi-tenant? |
| Scale | 10k hosts × 1k series = 10M active series, 15 s resolution, ≈670k samples/s. | Active series, and how fast they churn (pods, deploys)? |
| Latency | Alert evaluation every 30-60 s; dashboard queries over 24 h in under 1 s. | Freshness: how soon must a sample be queryable? |
| Retention | Raw 15 days; 5-minute rollups for 90 days; hourly for 13 months. | Is raw resolution ever needed for old data? |
| Consistency | Eventual; a few seconds of lag; occasional dropped samples are fine. | Is this also billing data? (Then it is not a metrics system.) |
| Out of scope | Logs, distributed tracing, anomaly detection ML, the UI. |
Back of the envelope
Section titled “Back of the envelope”Drag the assumptions. The two numbers that surprise people: ingester memory scales with series (not samples), and compression is what makes retention affordable. Set bytes per sample to 16 and watch storage grow 10×.
What to say out loud: “About 670k samples a second, so a sharded write path with a queue in front. Storage is small once compressed, around a terabyte for two weeks. Memory for active series is the constraint, so I’ll shard ingesters by series hash and put hard limits on series per tenant.”
API and data model
Section titled “API and data model”-- write: batched, from agentspush(tenant, [ { labels: {__name__, k1: v1, ...}, samples: [(ts_ms, float64), ...] } ])
-- read: a selector, a range, a step, an aggregationquery(tenant, expr, start, end, step) -> [ { labels, points: [(ts, value)] } ] expr e.g. histogram_quantile(0.99, sum by (le, route) (rate(http_latency_bucket[5m])))
-- alertingRule { expr, for: duration, labels, annotations } evaluated every 30 s
Series id = hash(sorted label pairs) -- identity of a seriesChunk = { series_id, t_start, t_end, compressed bytes } -- ~120 samples or 2 hIndex = label pair -> sorted list of series ids ("postings")Two decisions hide in that sketch:
- A series is identified by its full label set. Change one label value and it is a different series with its own memory and index entries. That is why cardinality, not volume, is the cost.
- Metric types decide what can be aggregated. A counter only goes up (so you query its
rate, which survives restarts); a gauge is a level; a histogram is a set of counters, one per latency bucket, which is what makes percentiles mergeable. Choosing types well up front is most of the design.
High-level design
Section titled “High-level design”Step through each flow. The write path is long and asynchronous; the read path is a fan-out over two places: ingesters (last ~2 hours) and object storage (everything older).
| Component | Role | Why it’s there |
|---|---|---|
| Agent | Scrapes targets (pull) or receives pushes; batches and compresses. | One connection per host instead of per service; buffering during outages. |
| Ingest API | AuthN, per-tenant rate and series limits. | The firewall against one tenant’s cardinality bug. |
| Log (Kafka) | Durable buffer, partitioned by series hash. | Absorbs bursts; lets ingesters restart and replay. |
| Ingesters (×N, RF 3) | Own a hash range of series; hold the head block in memory with a WAL. | Recent data must be queryable within seconds and compressible as it arrives. |
| Block store | Immutable 2-hour blocks (chunks + index) in object storage. | Cheap, durable, effectively infinite. |
| Compactor | Merges blocks, downsamples, applies retention. | Fewer, bigger files; coarse tiers for long ranges. |
| Query | Resolves label matchers via the index, fetches chunks, aggregates. | Pushes aggregation down to keep fan-out data small. |
| Rule evaluator + Alertmanager | Runs alert queries on a timer; groups, dedupes, routes pages. | Alerting is just a scheduled query plus a state machine. |
Deep dive: the data model and cardinality
Section titled “Deep dive: the data model and cardinality”An ingester keeps, per active series, a small buffer of recent samples and its entries in the inverted index (label pair → posting list of series ids, the same idea as a search engine’s index). A query like {service="pay", code=~"5.."} intersects posting lists, then reads only those series’ chunks.
So the cost of a metric is the product of its labels’ distinct values. Toggle labels below:
| Label kind | Example | Verdict |
|---|---|---|
| Bounded, small | method, status, region |
Fine |
| Bounded, templated | route="/users/:id" |
Fine; never the raw path |
| Bounded but churning | pod, container_id |
OK, but every deploy creates new series; budget for churn |
| Unbounded | user_id, email, request_id, raw URL |
Never a label. Put it in logs or traces |
The Prometheus instrumentation guide puts it bluntly: try to keep the cardinality of your metrics below 10, and if a metric is over 100 or could grow that large, look for another approach. Production systems enforce this with per-tenant active-series limits at the ingest API, and a “top series by metric” report so teams can find their own bug.
Deep dive: compression, blocks and rollups
Section titled “Deep dive: compression, blocks and rollups”A raw sample is 16 bytes: an 8-byte timestamp and an 8-byte float. Facebook’s Gorilla (VLDB 2015) observed that both are very predictable, and got the average to 1.37 bytes per sample, about 12× smaller. Prometheus uses the same scheme and quotes 1-2 bytes per sample.
Timestamps: delta of delta. Scrapes arrive every 15 s, so the gap between gaps is usually 0.
first sample: write t0 in full (64 bits); write delta d1 = t1 - t0 (14 bits)each next sample: dod <- (t_n - t_{n-1}) - (t_{n-1} - t_{n-2}) if dod == 0: write '0' -- 1 bit, ~96% of samples elif dod in [-63, 64]: write '10' + 7 bits elif dod in [-255, 256]: write '110' + 9 bits elif dod in [-2047, 2048]: write '1110' + 12 bits else: write '1111' + 32 bitsValues: XOR with the previous value. A gauge that didn’t change XORs to zero; one that changed slightly shares sign, exponent and leading mantissa bits.
x <- bits(v_n) XOR bits(v_{n-1})if x == 0: write '0' -- ~51% of sampleselse: write '1' if x's meaningful bits fit inside the previous leading/trailing-zero window: write '0' + the meaningful bits only else: write '1' + 5 bits (leading zeros) + 6 bits (length) + meaningful bitsChunks are append-only bit streams, decoded sequentially, which suits range scans. They are cut into 2-hour blocks: the head block lives in memory, protected by a write-ahead log, and is flushed as an immutable directory of chunks plus its own index. Immutability makes compaction, caching and deletion simple: retention is “delete blocks older than X”.
Rollups (downsampling). A 1-year graph at 15 s resolution would be 2.1M points per line; no screen needs that. The compactor writes coarser tiers:
| Tier | Resolution | Typical retention | Points per series per day |
|---|---|---|---|
| Raw | 15 s | 15 days | 5,760 |
| 5-minute | 5 min | 90 days | 288 |
| 1-hour | 1 h | 13 months | 24 |
Each rollup point stores min, max, sum, count (and last): sum / count gives the exact average, and min/max keep spikes visible that an average would hide. For counters, roll up the increase, not the value. For histograms, add bucket counts. The query engine picks the tier from the range and step. See data aggregation for why sum and count compose and averages don’t.
Deep dive: percentiles and alerts
Section titled “Deep dive: percentiles and alerts”Latency is the metric people care about most, and the one most often computed wrong. You cannot average percentiles: a p99 is a position in a sorted list, and the per-host lists have different lengths. Try it:
The fix is to ship something mergeable from every host and compute the quantile once, at query time:
| Approach | Merge | Error | Notes |
|---|---|---|---|
| Per-host p99 (Prometheus “summary”) | Not mergeable | Unbounded when combined | OK for one process; useless across a fleet |
| Fixed-bucket histogram | Add bucket counts | Up to a bucket’s width | Prometheus classic; choose buckets near your SLO |
| Exponential / native histogram | Add bucket counts | Relative, a few % | Prometheus native histograms, OpenTelemetry exponential histograms |
| DDSketch | Add bucket counts | Guaranteed relative error (e.g. 1%) | Datadog; log-spaced buckets, bounded memory |
| t-digest | Merge centroids | Small at the tails, no hard bound | Good accuracy at p99.9 with few centroids |
merged[b] <- sum over hosts of count[host][b] -- for every bucket upper bound brank <- 0.99 * total(merged)walk buckets in order, accumulating counts, until cumulative >= rankestimate <- lower(b) + (upper(b) - lower(b)) * (rank - cum_before) / merged[b]The same rule applies over time: a 5-minute rollup of p99s can’t produce a daily p99; a rollup of bucket counts can.
Alerting is a query on a timer feeding a small state machine: inactive → pending when the expression first returns a result, pending → firing once it has held for the for duration, back to inactive when it stops. Good rules:
- Alert on symptoms (error rate, latency SLO burn) more than causes (CPU).
- Use
for:to absorb a single bad scrape, and multi-window burn-rate alerts for SLOs (fast window to page, slow window to confirm). - Alert on absence: a dead service has no error rate. With pull,
up == 0is free; with push, you needabsent()or heartbeats. - Run evaluators as an HA pair; Alertmanager deduplicates, groups (“50 hosts down” is one page) and supports silences.
Deep dive: ingestion at scale, push vs pull
Section titled “Deep dive: ingestion at scale, push vs pull”| Pull (Prometheus) | Push (Datadog agent, StatsD, OTLP) | |
|---|---|---|
| Who knows the targets | The server, via service discovery | Nobody, until data arrives |
| Detects a dead target | Yes, up == 0 |
Only by absence / heartbeat |
| Short-lived jobs, serverless | Awkward (needs a push gateway) | Natural |
| Behind NAT / firewalls | Hard | Easy |
| Back-pressure | Server controls scrape rate | Clients can flood you; need limits |
Large systems mix both: an agent per host scrapes local targets (pull, so up works) and pushes batches to a central, multi-tenant ingest tier (Cortex, Mimir, Thanos receive, Datadog).
Sharding. Hash the series id onto a ring of ingesters with replication factor 3; a write succeeds on a quorum of 2. Queries for recent data ask all replicas holding the range and dedupe. Sharding by series (not by tenant) spreads a big tenant’s load; shuffle sharding gives each tenant a subset of ingesters so one tenant’s spike can’t hit every machine. For HA Prometheus pairs scraping the same targets, the ingest tier keeps samples from one replica per pair (elected by label) to avoid double counting.
Query fan-out. A query over 7 days touches ingesters plus dozens of blocks. Split it by time (day by day, in parallel), cache step-aligned results, and push sum by aggregation down to each shard so only small results travel.
Failure modes
Section titled “Failure modes”| What breaks | What you see | Design response |
|---|---|---|
| Cardinality explosion | Ingester memory climbs, OOM, ingestion stalls for all | Per-tenant series limits at the edge, top-series reports, reject new series not samples |
| Ingester crash | Recent data missing from queries | WAL replay on restart; RF 3 so the other replicas serve |
| Ingest backlog | Graphs lag, alerts evaluate stale data | Kafka buffers; alert on ingestion lag; drop oldest under extreme pressure |
| Monitoring is down during an outage | No alerts exactly when needed | Run the monitoring stack independently; a meta-monitor (“dead man’s switch” alert that always fires) |
| Expensive query | Query nodes OOM, dashboards time out | Limits on series and samples per query, time-splitting, result caching |
| Clock skew on clients | Out-of-order samples rejected | Server-side timestamps for pull; a small out-of-order window for push |
| Counter resets on restart | Negative rates | rate() detects decreases and treats them as resets |
| Averaged percentiles | Dashboards disagree with users’ experience | Histograms or sketches end to end, merge then quantile |
What it costs
Section titled “What it costs”| Operation | Time | Space |
|---|---|---|
| Append a sample | O(1), a few bit writes | ≈1.4 B per sample (vs 16 raw) |
| Active series | — | ≈ a few KB each in memory (index + head chunk); 10M ≈ 40 GB per replica |
| Label match | O(posting list sizes), intersection of sorted lists | Index ≈ O(series × labels) |
| Range query | O(series matched × points in range), decode sequentially | Bounded by query limits |
| Merge histograms | O(hosts × buckets) | O(buckets) per series |
| Rollup a block | O(samples) once | 5 values per coarse point |
| Throughput | 10M series / 15 s ≈ 670k samples/s | ≈1.3 TB per 15 days raw |
In the interview
Section titled “In the interview”A good 45-minute arc:
- Clarify (5 min). Series count and churn, resolution, retention, push or pull, multi-tenant, alert latency.
- Estimate (5 min). Samples/s, memory for active series, storage with and without compression.
- Data model (5 min). Series = name + labels; samples; counter/gauge/histogram; the inverted index. Say “cardinality” early.
- Write path (10 min). Agent → ingest API with limits → log → sharded, replicated ingesters → 2-hour blocks in object storage. Explain Gorilla compression in two sentences.
- Read path and rollups (10 min). Query fan-out to ingesters and blocks, tier selection, step-aligned caching, compactor.
- Percentiles and alerting (5-10 min). Why averaging p99 is wrong, histograms or DDSketch, alert state machine, absence alerts, Alertmanager dedupe.
Follow-ups to expect: “A team adds user_id as a label.” Per-tenant series limit rejects new series; tell them to use logs or exemplars. “Show p99 for the whole fleet over 30 days.” Sum histogram buckets across hosts and over time, then quantile. “How do you know the monitoring system itself is healthy?” An always-firing watchdog alert routed to an external dead man’s switch. “Make it multi-region.” Ingest and store per region; global queries fan out across regions and merge, the same way they merge across shards.
Check yourself
Section titled “Check yourself”Go deeper
Section titled “Go deeper”- Pelkonen et al.: Gorilla, a fast, scalable, in-memory time series database (VLDB 2015): delta-of-delta and XOR compression, 1.37 bytes per sample.
- Prometheus docs: Storage: head block, WAL, 2-hour blocks, compaction and the disk sizing formula.
- Prometheus docs: Histograms and summaries: why summaries can’t be aggregated, and bucket choice errors.
- Masson, Rim and Lee: DDSketch, a fast and fully-mergeable quantile sketch (VLDB 2019): relative-error quantiles at Datadog.
Concept pages to drill: Data aggregation · Streaming · Indexing · Memory efficiency · Bit manipulation · State machine · Time handling