Batch vs Stream

Every data team eventually has the argument. One side wants to build everything on a streaming platform because batch feels like the past; the other wants to keep the trusted nightly jobs because streaming feels like a way to page yourself at 3 a.m. Both are arguing about tools. The argument is not about tools.

Strip the logos away and every batch-versus-stream decision is really two plain questions about your problem:

1. What shape is the input? Do you already have all the data, a finite pile with an end, or does it keep arriving with no end in sight?
2. How fresh must the answer be? If the answer is an hour old, who is hurt? What does staleness actually cost you: a cent, a customer, a life?

Answer those two and the technology mostly picks itself. Get them backwards and no tool will save you: you will either pay for a real-time pipeline to power a report nobody reads before lunch, or run a nightly job to catch fraud that cleared out your accounts by breakfast.

This section is the whole story of processing data at scale — how it is stored, moved, batched, streamed, combined, and counted. This first lesson is the frame the rest hangs on, and it fits in one sentence you will spend the lesson earning: a batch is a stream you chose to wait for, and a stream is a batch you refused to wait for.

Start with the first question — the shape — because it is the one people skip, and it decides more than they think.

Bounded data is data at rest. It is a finite set with a start and an end: all of yesterday's orders, the last hour's logs, the export you were handed this morning. Because every record is already there, you can do something a streaming system can never do — see the whole thing at once. You can sort it, count it exactly, join it against everything else, run the job, get one answer, and stop. The processor has a finish line.

Unbounded data is data in motion. It is an open pipe: clicks, payments, sensor readings, log lines, records that keep coming with no natural end. You can never 'see the whole thing,' because there is no whole thing; there is only what has arrived so far. So the processor never stops. It runs forever, holds what it needs to remember, and emits answers as it goes, each one true as of this moment and already going stale.

Here is the trick that makes the two collapse into one: it is often the same data, seen at a different time. Yesterday's clickstream is a bounded box you can process once. Today's clickstream is an unbounded pipe still filling. Same clicks; the only difference is whether you waited for the day to end before you looked. Hold onto that; the whole lesson turns on it.

Because bounded is really just unbounded-with-an-end, 'batch or stream' is not a switch with two positions. It is a dial, and the thing the dial sets is how long you are willing to wait before you answer.

• Batch sits at one end: wait a long time, hours to days, gather a big pile, process it all at once. It is the cheapest and simplest thing you can build (a scheduled script), it gets enormous throughput because it sees everything, and its answers are as stale as the wait. The nightly billing run lives here.
• Streaming sits at the other end: wait almost no time, milliseconds to a few seconds, and emit on essentially every event. It is the freshest thing you can build and the most expensive and involved. The card-fraud check, which has maybe 50 ms to decide inside a 200 ms payment window, lives here.
• Micro-batch is the wide, underrated middle: wait a little, seconds to a few minutes, and process small piles on a tight loop. It feels almost live and costs a fraction of true streaming.

That middle is where most real systems belong, and it is the thing teams miss when they get swept up in real-time fever. For the majority of analytics, one-to-five minutes of freshness is completely fine, and buying it with a micro-batch instead of an always-hot streaming stack can be dramatically cheaper: one well-known setup keeps only hours of hot data live and tiers the rest to cheap storage at roughly 38× less cost. Even hard-real-time shops split the work: a card network scores obvious cases in its 50 ms budget but sends the subtle ones to a 30-to-60 second review, because most fraud decisions do not actually need the millisecond.

So the rule is not 'go real-time.' The rule is: pick the cheapest point on the dial that still meets the freshness the answer truly needs, no fresher.

The reason batch and stream keep collapsing into one dial is that, underneath, they are two views of the same thing. This is the idea that took the industry a decade to say clearly, and once you see it you cannot unsee it.

Picture a shopping cart. One way to store it is a table: a single row that always holds the cart's current contents. Another way is a stream: an append-only log of every change — added milk, added bread, removed milk — each stamped with a time. These are not two different pieces of data. They are the same fact told two ways, and you can turn either into the other:

• Replay the stream and you rebuild the table. Start empty, apply every change in order, and you arrive at exactly the current contents. The table is the stream folded up — the latest value per key.
• Watch the table change and you get the stream. Every update to the row is one more entry in the log. The stream is the table's changelog unrolled.

That is the stream-table duality, and it is why the batch-versus-stream war was always a little fake. A table is a stream frozen at a moment; a stream is a table left running. A batch job reads a table — a stream someone already froze for you. A streaming job reads the log — a table still being written. Both are computing a result derived from the same underlying facts; the only real difference is whether they read the facts frozen or flowing.

This is also why 'reprocessing' works the way it does: because the log is the truth and the table is just a cached view of it, you can throw away a wrong answer and replay the log to rebuild it correctly — a stream you can replay is a batch you can re-run. Hold this duality; the next control lets you feel it.

If a stream is just a table left running, why not run everything as a stream and always have fresh answers? Because the moment you stop waiting, four bills come due — and they are the reason 'just make it real-time' is the most expensive sentence in data engineering.

1. Data arrives out of order. When you have the whole bounded pile, order is free: you sort it. In motion, events that left their sources as 1-2-3 show up as 2-1-3 because networks reorder, mobile clients buffer, and partitions race. Your answer has to be right anyway.

2. Data arrives late. A phone goes into a tunnel and its events land ten minutes after you already emitted 'the total for 3:00 p.m.' Do you correct the answer you already sent? Batch never faces this: it waited until everything was in.

3. There are two clocks. Event time is when something actually happened at the source; processing time is when your system got around to it. In batch they might as well be the same. In a stream they drift apart constantly, and almost every hard question, like 'how many in the last minute?', has to say which clock it means. (Taming those two clocks is its own craft, watermarks and windows, and it gets a whole lesson later in this section.)

4. The job never sleeps. A batch job starts, runs, and ends; if it fails you re-run it and it is easy to see which run broke. A streaming job runs forever, so it must survive machine restarts without losing its place, keep its running state safe, and be replayable from the log when something goes wrong, all while never stopping. That is a permanently harder operational contract.

None of this makes streaming wrong. It makes streaming a cost, one you should pay only when the freshness is worth it.

Put the two questions back together and the decision becomes almost mechanical. Forget which tool is trendy and ask one thing: what does a stale answer cost?

When staleness is expensive, buy freshness — stream. If a late answer loses money, misses fraud, breaks a user-facing feature, or fails to react to the world in time, the low latency is worth its complexity tax. Fraud scoring, live personalisation, alerting, an ETA that must render in under half a second — these earn the stream.

When staleness is free, keep it simple — batch. If the consumer is a report read at 9 a.m., a model trained on last month's data, or a historical rollup, then freshness buys nothing and simplicity buys a lot: batch is cheaper to run, easier to reason about, trivial to reprocess in bulk, and when it breaks you can point at the exact run that failed. Reprocessing a terabyte of history or building a training set is cheaper and saner in one big bounded pass.

When you're not sure, you're probably in the middle — micro-batch. Most of the world lives here and should admit it.

Two traps sit on either side of this, and good engineers fall into both:

• Streaming theatre: building an always-on, exactly-once, event-time pipeline to feed a dashboard three people glance at once a day. You paid the whole freshness bill for freshness nobody wanted.
• The nightly-batch blind spot: running yesterday's-data jobs while the business quietly needed to act in seconds — the fraud that cleared, the outage you saw at midnight from 6 p.m. logs. You saved on complexity and lost on the thing that mattered.

The answer to both is the same discipline: price the staleness first, then pick the cheapest point on the dial that pays that price — and no point fresher.

You now have the frame; the rest of this section builds the machinery under it, and every lesson is one layer of the same picture:

• How batch actually runs at scale: the pattern that split one huge bounded job across a thousand machines, and everything that grew from it.
• How data gets from where it's born to where it's used: the pipelines that move and reshape it, and the fight over whether to transform it before or after you load it.
• Where all that data lives: the lake, the warehouse, and the lakehouse that tried to be both.
• How teams combine batch and stream on purpose: the two named architectures that either run both side by side or dare to run everything as one stream.
• The engines that do the streaming, and what actually separates them.
• The hard parts up close: cutting an endless stream into meaningful windows, and the two clocks and delivery guarantees that decide whether your counts are even true.
• Counting the uncountable: answering 'how many' and 'top ten' over a firehose too big to store.

Every one of them is a variation on the two questions you just learned to ask: what shape is the data, and how fresh must the answer be?

• It's not a tools fight. Batch versus stream is two questions about your problem: what shape is the input (bounded, at rest — or unbounded, in motion), and how fresh must the answer be (what does staleness cost)?
• A batch is a stream you waited for. Bounded data is just unbounded data with an end, so 'batch or stream' is a dial — how long you wait before answering — with batch, micro-batch, and streaming as points on it, not a switch.
• A table is a stream frozen; a stream is a table running. The stream-table duality: replay a log to rebuild a table, watch a table change to get the log. Both compute derived data from the same facts, read frozen or flowing.
• Freshness is a cost, not a virtue. Stop waiting and you take on out-of-order data, late data, two clocks, and a job that never sleeps. Pay it only when the answer's freshness is worth it.
• Price the staleness, then pick the cheapest point that pays it. Stream when stale answers hurt; batch when they don't; micro-batch far more often than teams admit — most analytics are fine at 1–5 minutes and far cheaper for it.

Keep the dial in your head. The next lesson turns to the batch end of it and asks a question that once looked impossible: how do you run a single job over more data than any one machine can hold?