Time-Series Databases

This data has been chasing us for three lessons. In Document Databases, the courier's GPS trail was the unbounded array that marched a document toward the 16 MB wall. In Wide-Column Databases, we stored it properly — ((courier_id, day), ping_time) — and could finally read the pings back, in order, forever.

Then the ops team asks their actual question: "average courier speed per 5-minute window, for the last 6 hours, per city — and keep 90 days of history without bankrupting us."

And the wide-column table goes quiet. It can hand you the raw pings, beautifully ordered — but the question wasn't "give me the points." It was "give me statistics over windows of the points." Computing that means dragging every raw ping across the wire and aggregating in your app, every dashboard refresh, for every city. And those 90 days? At 5 pings a second per courier, you're buying disks for data nobody will ever read at full resolution.

There's a family built around exactly this gap, and it runs on three assumptions so specific they're almost a personality: writes only append (the past never changes), reads are windows (nobody asks for one ping; everyone asks for aggregates over ranges), and data ages (this hour matters at full resolution; last month matters as a trend line; last year is a lawsuit-retention question).

In this lesson: the workload triangle, then the family's real secret — a compression scheme so effective it's not an optimization but the architecture — then the failure mode that has its own name in every ops team's vocabulary, and the honest boundaries with the families you already know.

Scope: what to measure and how to label it — observability practice — lives in Architecting Production Systems (The Three Pillars, Metrics & Cardinality); computing windows on data in flight is stream processing (Streaming Engines, Windowing & Exactly-Once). This lesson is the storage family.

Watch what each assumption buys the engine, because generic databases can exploit none of them.

Writes only append. No updates means no locking the past, no rewriting pages, no MVCC gymnastics — ingestion becomes a pure sequential write path that comfortably absorbs millions of points per second. (A relational table can hold the pings; it just insists on being ready for an UPDATE that will never come, and pays index maintenance on every single point for the privilege.)

Reads are windows. Nobody ever asks "what was CPU at 14:32:17?" They ask "p95 over 5-minute buckets, last 6 hours." So the engine indexes series, not points — one index entry per metric-stream, pointing at time-ordered blocks — and answers windows by scanning compressed ranges sequentially. Compare the generic approach: a B-tree entry per point, billions of entries, for lookups nobody performs.

Data ages. In this family, retention isn't a cron job you write — it's schema. You declare the staircase: raw for 2 days, 5-minute rollups for 30, hourly for 2 years, then gone. The engine downsamples and deletes continuously. (Remember wide-column's tombstone agony — deletes as expensive writes? Append-only, time-ordered storage deletes by dropping whole expired blocks. Aging isn't a wound here; it's a feature, because the layout matched the lifecycle.)

Each assumption narrows the family's world — and everything it gives up comes back as capability. You've seen this trade on every stop of the map. But this family's biggest payoff comes from somewhere subtler: what time-ordering does to the bytes themselves.

In 2015, Facebook published the paper that quietly defines this family: Gorilla (VLDB 2015). Their monitoring system was drowning — billions of points arriving, dashboards needing them fast, RAM being the only place fast enough. The solution wasn't a bigger cluster. It was noticing that time-series data is spectacularly, exploitably predictable.

A raw point is 16 bytes: an 8-byte timestamp and an 8-byte float. Gorilla attacks both halves:

Timestamps: delta-of-delta. Scrapes arrive on a schedule — every 60 seconds, say. So the deltas are 60, 60, 60… and the delta of the deltas is 0, 0, 0. Gorilla stores that zero in one bit. In Facebook's production data, 96% of all timestamps compressed to a single bit — the schedule IS the data, so you barely need to store it. Jitter falls into small variable-width buckets; only a genuinely irregular gap pays full price.

Values: XOR the bits. A courier's speed doesn't teleport; CPU doesn't jump from 42% to 97% between scrapes. Neighboring float64 values share sign, exponent, and most of the mantissa — so bits(v) XOR bits(prev) is mostly zeros. Identical value? XOR is all-zero: one bit (51% of Facebook's values). Small change? Store only the narrow window of bits that actually differ, plus a tiny header.

Net result across their production mix: 1.37 bytes per point — a 12× reduction. And here is the sentence that upgrades this from a storage trick to an architecture, connecting straight back to Sequential vs Random Access: the redundancy lives between neighbors — between this point and the previous one. Store points scattered across a generic table's pages and the redundancy is invisible. Store them time-ordered, and it's free money. Compression this good is a dividend of the layout — the whole family exists to collect it.

The compounding consequences: at 12×, weeks of data fit in RAM (Gorilla held 26 hours in memory; Netflix's Atlas holds its entire hot set — 1.2 billion live series — the same way). Window queries become sequential scans of compressed blocks. Long retention stops being a hardware budget line. Prometheus, InfluxDB, M3, VictoriaMetrics — every serious TSDB since ships a variant of this encoding. You're about to compute it yourself.

The widget below performs the actual Gorilla bit-math — real float64 XOR, real delta-of-delta buckets — on a series you shape.

Part A: start with steady CPU % @60s and read the chips: most points cost 2 bits where raw storage spends 128. Switch to spiky request rate — predictability gone, watch the red chips. Then irregular scrape — now even the timestamps pay. Part B: arm the label panel. Region ×4, instance ×100, status ×5 — comfortable. Now press user_id ×1M, watch the series counter and the memory meter, and meet the failure mode the next section names.

You just detonated it; now let's name it precisely, because this is the failure mode that pages TSDB operators at 3 a.m.

A "metric" isn't one series. http_requests{region, instance, status} is one series per unique label combination — the engine must track, index, and hold the recent window of each in memory (Prometheus keeps the last two hours of every active series in its RAM "head block", plus an index entry for every label value). Series count is therefore the product of label cardinalities: 4 regions × 100 instances × 5 statuses = 2,000 series. Harmless.

Then someone instruments user_id. One line of code. 4 × 100 × 5 × 1,000,000 = 2 billion series — and your monitoring stack, whose day job is watching other systems die, dies first. The crisis shape is well documented: memory climbs, queries crawl, and you're choosing between dropping data and restarting into the same explosion. The field test that prevents it: could this label plausibly have 10,000 unique values in production? Then it belongs in a trace, not a metric label. (The practice side — what to label, what goes to traces — is C4's Metrics & Cardinality; here you needed the mechanics.)

And notice something taxonomically satisfying. The hot key (KV), the hot partition (wide-column), the supernode (graph) — those were concentration: too much load on one thing. The cardinality bomb is different: nothing is hot, everything is numerous — destruction by multiplication. Two demons, both summoned by one innocent-looking line, and now you know both by name. Between them they explain most self-inflicted database incidents you'll ever see.

The third corner of the triangle deserves its own beat, because it's where teams quietly decide — often without noticing — which future questions they'll be able to answer.

The standard staircase: raw resolution for days (debugging needs every point), 5-minute rollups for weeks (capacity trends), hourly for years (planning, compliance), then deletion. Declared as policy, executed continuously by the engine, paid for by the 12× compression underneath.

The honest caveat: downsampling is a one-way door. A 1-hour average serenely erases the 30-second spike that took you down; once the raw window expires, no query can resurrect it. So retention design is really a bet about future questions: keep raw long enough to debug your slowest-discovered incidents, keep rollups long enough to see seasonality, and write the policy while calm — because you'll only regret it during a post-mortem, which is precisely when it can't be changed.

Three boundary calls, review-room sharp.

vs a plain relational table. Low volume, short retention, a handful of series? A readings table with a timestamp index is fine — don't buy a runbook for a workload Postgres shrugs at. And when it grows, note the convergence bell ringing yet again: Timescale is the time-series model as a Postgres extension — hypertables, continuous aggregates, compression — the family's capabilities without leaving the default engine. The dedicated TSDB earns its keep at high ingest × long retention × window-heavy reads (and at metrics-infrastructure scale, it's not optional: Prometheus is the second project CNCF ever adopted, after Kubernetes, because everyone has this workload).

vs wide-column (the boundary, in one sentence). Wide-column gives you ordered retrieval of the points (Discord: "the messages, in order"); a TSDB gives you statistics over windows of the points, plus the compression and retention machinery that make it affordable. If dashboards aggregate it, it's this family; if users scroll it, it's the last one.

vs stream processing. A TSDB computes windows over data at rest and arriving; a stream processor computes windows over data in flight, before storage, for reactions in milliseconds. They're teammates, not rivals — and the in-flight story is C4's Streaming Engines.

One breadth note so the family isn't typecast: metrics made TSDBs famous, but the triangle — append, window, age — belongs equally to IoT sensor fleets, market ticks, energy meters, and yes, courier GPS pings. Our fleet was never a DevOps team; it was time-series-shaped from the day Document Databases flagged it.

Four calls from the marketplace's data-platform review. For each: TSDB, or something you already have — one sentence why. Commit first.

1. Courier GPS pings: 2,000 couriers × 5/s; dashboards want speed/distance per 5-min window; 90-day retention.
2. Order status history (placed → cooking → delivering → delivered): support agents scroll a specific order's timeline.
3. Per-service request metrics for the new SRE team: p95 latency, error rates, per region/service/instance — engineers keep asking to add a customer_id label "for debugging."
4. "Payments per hour, last 30 days" for the finance dashboard — computed from the orders table, which already exists, updated hourly.

Answers. (1) TSDB — the founding workload: brutal append rate, window reads, aging data; the 12× compression is the difference between 90-day retention and a hardware negotiation. (2) Not a TSDB — ~6 events per order, read as "this order's timeline" (ordered retrieval of points, low volume): a plain table, or the wide-column shape if volume explodes. Having timestamps doesn't make it time-series. (3) TSDB, and block the label — region × service × instance is healthy cardinality; customer_id is the bomb from the widget. Offer them traces (that's exactly what traces are for — Distributed Tracing, C4). (4) Neither — an hourly GROUP BY on the relational store you already run; a materialized view if it gets slow (Materialized Views, next section). The lesson of 1-vs-2-vs-4 in one line: the triangle decides, never the presence of a timestamp column.

• "Any table with a timestamp is time-series data." The family is defined by the workload triangle — append-only, window reads, aging — not by the column. Orders have timestamps; you update them, join them, fetch them by ID: relational.
• "Wide-column already does this." It orders the raw points (that's the wide-column courier table). It has no windowed-aggregation primitive, no Gorilla compression, no retention DDL. Retrieval of points ≠ statistics over windows.
• "Compression is a storage nicety." It's the load-bearing wall: 12× is why weeks fit in RAM, why windows scan sequentially, why retention is affordable. You computed it — 2 bits where raw spends 128. Remove it and the family collapses into a slow generic table.
• "More labels = more observability." Series = the product of cardinalities. One user_id label turned 2,000 series into 2 billion in your hands. The 10,000-values test; high-cardinality identities belong in traces.
• "Downsample everything, keep it forever." The hourly average erases the 30-second spike — downsampling is a one-way door. Retention is a bet on future questions; place it deliberately.
• "TSDBs are a DevOps tool." Metrics made them famous; the triangle owns IoT, finance ticks, energy, and GPS fleets. Sensor-shaped data is everywhere — so is this family.

• The family = the workload triangle: writes APPEND (the past is immutable), reads are WINDOWS (aggregates over ranges, never point lookups), data AGES (retention and downsampling as first-class schema).
• The secret is compression as architecture (Gorilla, VLDB 2015): delta-of-delta timestamps (96% → 1 bit) + XOR'd floats (51% → 1 bit) = 16 B → 1.37 B, 12× — possible only because time-ordering puts the redundancy between neighbors (Sequential vs Random Access, cashed as an information dividend). Consequence: weeks in RAM (Atlas: 1.2 B live series), sequential window scans, affordable retention.
• The signature failure is the cardinality bomb: series = the PRODUCT of label cardinalities; one user_id label = 2 billion series in head-block RAM. The taxonomy worth keeping: hot keys/partitions/supernodes concentrate; cardinality multiplies.
• Boundaries: plain table until high-ingest × long-retention × window-reads (Timescale = the model inside Postgres — convergence again); wide-column retrieves points in order, TSDB computes statistics over them; stream processors do windows in flight (⟶ C4).
• Retention is a bet on future questions — downsampling is a one-way door; write the policy while calm.
• Next: we've organized data by key, aggregate, sort order, relationships, and time. One question shape remains before the map is whole: which documents are RELEVANT to these words? — Full-Text Search Databases.