The 10K Connections Problem

The last lesson ended on a cliffhanger. A thread-per-connection server needs 10,000 threads to serve 10,000 users; a process-per-connection one needs 10,000 processes. So how does a single nginx worker — one process, sometimes one thread — hold tens of thousands of open connections at once? It's not a trick of tuning. It's a different idea about what a server is even doing.

This puzzle has a name and a birth year. In 1999, Dan Kegel wrote it down as the C10K problem: can one server handle ten thousand simultaneous connections? At the time the answer was "not really" — and the reason why, plus the fix that emerged, reshaped how essentially every high-traffic system on the internet is built. Chat apps, live dashboards, multiplayer games, API gateways, streaming — all of them are holding piles of connections open, and all of them inherited the answer to this one question.

Here's the whole lesson in one reframe, which we'll spend the rest of it earning: the hard part was never doing 10,000 things at once — it's waiting on 10,000 things at once without paying for a worker to sit and wait on each one. Get that, and the difference between a server that dies at 5,000 connections and one that shrugs at 2,000,000 stops being magic and becomes obvious.

Scope: this lesson is about how one worker holds many connections. Reusing open connections instead of re-opening them is the very next lesson (Connection Pools); the deep definitions of concurrency versus parallelism come a little later.

Start with the design every one of us writes first, because it's the simplest thing that works. A connection arrives; you read from it, do some work, write back. In code, reading looks like one innocent line:

data = conn.recv(4096)   # <- the whole story hides here

That recv is a blocking call. If the client hasn't sent anything yet, the line does not return — the entire thread parks, frozen, until bytes actually arrive. That's usually what you want for one connection: the thread sleeps cheaply, the OS wakes it when data comes.

But now serve two clients. Thread A is parked on client 1's recv; it cannot also watch client 2, because it's stuck on that line. So you need a second thread for client 2. And a third for client 3. The design forces a simple, fatal rule: one blocked thread per connection, because a blocked thread can only wait on one thing.

Ten thousand connections, then, means ten thousand threads. And here's the part that makes it absurd: on a typical server, the vast majority of those connections are idle at any given instant — a chat socket between messages, a browser keeping a connection warm, a dashboard waiting for the next tick. You've hired ten thousand workers, and at any moment nine thousand nine hundred of them are standing perfectly still, paid to wait.

Let's make the cost concrete, because "ten thousand threads" sounds survivable until you price it. Thread-per-connection slams into two walls at once:

The memory wall. Every thread needs its own stack. On Linux that's up to 8 MB of virtual address space reserved (only the touched pages become real memory, but real they become as the thread does work), plus kernel bookkeeping. In practice, a thread-per-connection server like Apache's classic modes spends on the order of 1 MB of resident memory per connection — so 10,000 connections is 5–10 GB of RAM, and it grows linearly: every new connection is another slice off the same loaf. Compare that to nginx holding the same 10,000 connections in an event loop for 50–100 MB total — roughly a hundred times less, and flat no matter how many connections pile on.

The scheduler wall. Even if you had the RAM, the kernel now has to schedule thousands of threads onto a handful of CPU cores. Each time it switches from one thread to another it saves and restores registers, and — the expensive part — pollutes the CPU caches and TLB, so the next thread starts cold. With thousands of mostly-sleeping threads waking, running for a microsecond, and sleeping again, the machine starts spending more of its time switching between workers than doing their work. This is called thrashing, and it's why adding threads eventually makes a server slower, not faster.

The design's flaw in one line: it dedicates a whole worker to each connection, when almost every connection is doing nothing but waiting. We're paying, in memory and scheduling, for waiting.

Sit with that last sentence, because the fix falls straight out of it. If the problem is paying a worker to wait, then the answer isn't a faster worker or more workers — it's to stop assigning a worker to wait at all.

Reframe the whole task. You are not trying to do 10,000 things at once — at any instant, only a handful of connections actually have data ready to process. You are trying to wait on 10,000 things at once, cheaply, and spring into action only for the few that are ready right now.

Picture a restaurant. The thread-per-connection design hires one waiter per table: a table takes twenty minutes to decide, and its waiter just stands there, useless, for twenty minutes. Ten thousand tables, ten thousand waiters, almost all idle. The event-loop design hires one super-waiter who never waits: they take an order, drop it at the kitchen, and immediately move to the next table — returning to a table only when the kitchen rings the bell that its food is ready. One waiter serves a whole floor, because the waiting happens in the kitchen, not at the table.

That bell — "this one is ready now" — is the entire secret. If something else can watch all the connections and tell you the moment one is ready, you never have to park a thread on any single one. So who rings the bell? The one component that already sees every socket: the kernel.

The fix has two moving parts, and together they are the event loop.

Part one: non-blocking sockets. You tell the OS to make every socket non-blocking. Now recv on a socket with no data doesn't park the thread — it returns immediately with "nothing yet." The thread is never frozen against its will.

Part two: readiness notification. You hand the kernel your whole pile of sockets and ask a single question — "which of these has something ready right now?" The kernel, which is already watching every socket, hands back just the ready ones. Your one thread then loops forever:

loop:
  ready = ask_kernel_which_sockets_are_ready(all_sockets)   # the bell
  for sock in ready:
    handle(sock)   # read + work + write — never blocks, data is already there

That's it. That's how one thread holds tens of thousands of connections. It never waits on an idle socket — idle sockets just sit in the kernel's set costing a few kilobytes of buffer, not a whole thread. The single thread only ever does real work, hopping from one ready socket to the next, exactly like the super-waiter working the ready tables.

And this is why the event loop's memory is flat: it replaced N threads (N stacks, N scheduler entries, N context switches) with one thread plus one kernel set. Ten connections or ten thousand, the worker cost barely moves. The connection count stopped being a payroll problem.

Numbers on a page are one thing; watching a server die is another. Below, three servers run side by side — thread-per-connection, event loop, and virtual threads — and you turn up the pressure. Drag the connection count from a handful toward 20,000 and set how many are idle, then watch each lane's live meters: threads spawned, RAM, scheduler load, and health.

Things worth trying: ramp the thread lane up slowly and find the exact point its RAM and scheduler meters turn red and it dies — that's the wall Apache hit. Then notice the event-loop lane holding the same load with a flat memory bar. Finally, hit "fire a CPU-heavy request" on the event loop and watch one slow task freeze every connection at once — the one trap this whole design has, which the next section explains.

"Ask the kernel who's ready" sounds simple, but how you ask decided whether C10K was solvable — and it's a beautiful little lesson in algorithmic complexity showing up in the real world.

The first tools for this, select() and poll(), had a fatal shape: every time through the loop, you handed the kernel your entire list of sockets, the kernel scanned all of them to see which were ready, and copied the whole list back. That's O(n) — linear in the number of connections. At 10,000 sockets, a single poll() call takes on the order of 1 millisecond just to scan — and you do it on every single loop iteration. The bookkeeping becomes the bottleneck; the server spends its life re-scanning mostly-idle sockets.

epoll (Linux, 2002) and its cousins kqueue (BSD/macOS) and IOCP (Windows) fixed the shape. You register your sockets with the kernel once; it keeps them in an efficient structure (a red-black tree) and maintains a ready list as events actually happen. When you ask, it hands back only the sockets that are ready — no scan of the idle ones. That's effectively O(1) in the number of idle connections: at 10,000 sockets, an epoll_wait costs well under a microsecond. You pay only for sockets that have events, never for the ones sitting quiet.

That single change — from "re-scan everything every time" to "tell me only what changed" — is the difference between a server that buckles at a few thousand connections and one that holds millions. Same idea both times ("who's ready?"); the data structure behind the question is what scaled.

This isn't academic — it's the load-bearing decision inside the software you use every second.

• nginx runs a small number of worker processes (roughly one per CPU core), each an event loop on epoll. That's how it holds tens of thousands of connections per worker in tens of megabytes, and why it beat Apache's process-per-connection model so decisively that it took over the web.
• Node.js is an event loop (via a library called libuv) as its whole personality — one thread handling all your connections, with a tiny background thread pool reserved only for work the OS can't do asynchronously (file reads, DNS, crypto). "Node is single-threaded" means the event loop is single-threaded.
• Redis serves its enormous throughput from a single main thread on an event loop — no per-connection threads, no locks on the hot path. Simplicity as a performance strategy.
• The ceiling keeps rising. Push the same idea hard and the numbers get silly: WhatsApp held over 2 million concurrent connections on a single server (with Erlang), and specialized messaging systems have demonstrated 10+ million on one box. The industry even has a name for the sequel: the C10M problem. None of it is possible with a thread per connection.

Notice the pattern: the winners don't hold connections with workers. They hold them with a set of file descriptors and a kernel that rings a bell.

The event loop's superpower — one thread handles everything — is also its one knife, and every engineer who adopts it eventually cuts themselves on it. Say it out loud: there is only one thread.

So if any single piece of work on that thread takes a long time — resizing an image, hashing a password, parsing a huge JSON blob, running a tight computational loop — the loop cannot move on. It's stuck in your callback. And while it's stuck, every other connection waits, because there's no other thread to serve them. One slow, CPU-heavy request can make a server that was happily juggling 20,000 connections go completely unresponsive. The rule has a name that gets tattooed on Node developers: never block the event loop.

This draws the honest boundary of the whole model. The event loop is a genius at I/O-bound work — work that is mostly waiting (on the network, on disk, on another service). It is terrible at CPU-bound work — actual heavy computation — because that needs real parallelism across cores, which one thread cannot provide. The fix is to hand CPU-heavy jobs to a pool of worker threads or separate processes, and keep the loop itself doing nothing but shuffling I/O. (It's the same split from the last lesson: concurrency — juggling many waits — is not parallelism — doing many computations at once.)

And the modern happy ending: goroutines (Go) and virtual threads (Java 21+, from Project Loom) dissolve the trade-off. You write plain, simple blocking code — read(), write(), no callbacks — but the runtime gives each connection a featherweight thread and quietly runs an event loop underneath, parking a connection off its real OS thread the instant it blocks on I/O. You get thread-per-connection's simplicity with the event loop's scale. It's the "third answer" the last lesson hinted at, now revealed as the resolution to C10K itself.

"The event loop is just faster." Not per request — a thread serves a single request just as fast. The event loop's win is memory: holding thousands of mostly-idle connections for almost nothing. It's about concurrency at rest, not raw speed.

"10,000 connections means 10,000 requests per second." Connections aren't requests. Ten thousand open sockets (websockets, keep-alives) can be almost entirely idle. C10K is about holding idle connections cheaply, not throughput.

"epoll makes I/O faster." It speeds up no individual read or write. It makes the act of waiting on thousands of sockets O(1), so one thread can watch them all.

"Async means parallel." A single event loop does one thing at a time — it's concurrent, not parallel. It just never blocks while waiting. Real parallelism still needs multiple cores (multiple processes or threads).

"Node is single-threaded, so it wastes my 8 cores." For I/O you run one event loop per core (nginx does exactly this); for CPU work you offload to worker threads. The single thread is the loop, not the whole program.

"The event loop always beats threads." Only for I/O-bound, high-idle concurrency. CPU-bound work still wants threads or processes — and virtual threads / goroutines now give you both at once.

Ten minutes turns this from something you read into something you watched happen. Write a prediction first: if you open a live dashboard or a chat app and count its connections, how many will be ESTABLISHED but silent?

# How many connections is a process holding right now?
# (grab a PID from your browser, a local nginx, or a node app)
ss -tnp | grep ESTAB | wc -l          # Linux: count established TCP connections
#   or per-process:
lsof -p <PID> -a -i TCP | grep ESTABLISHED | wc -l

# Watch nginx hold connections flat while you hammer it (if you have it):
#   in one terminal:  while true; do ss -s | grep estab; sleep 1; done
#   in another:       ab -n 100000 -c 5000 http://localhost/   # 5000 concurrent

Two things to notice: (1) a busy server is holding far more open connections than it has threads — that gap is the event loop; and (2) as you crank the concurrency with ab, nginx's memory barely moves. If you can find a thread-per-connection server to compare, watch its memory climb with every connection instead. The gap between your prediction and the count is where the lesson lands.

The whole lesson, compressed:

• C10K asked: can one server hold 10,000 connections? The naive answer — a thread per connection — hits two walls: memory (~1 MB/connection → 5–10 GB) and the scheduler (thousands of threads thrash the CPU). It pays a whole worker to wait on each mostly-idle socket.
• The reframe: you're not doing 10,000 things at once, you're waiting on them. So don't assign a worker to wait — make sockets non-blocking and let the kernel tell you which are ready (the event loop). One thread, tens of thousands of connections, flat memory (~50–100 MB).
• The API decided it: select/poll re-scan everything (O(n)); epoll/kqueue return only what's ready (O(1)).
• The trap: one thread means never block the loop — offload CPU-bound work to threads. Event loops are for I/O-bound concurrency; goroutines / virtual threads give simple blocking code and event-loop scale.

You now know how one worker holds thousands of connections. But there's a cost we quietly skipped: opening each connection in the first place isn't free — a TCP (and often TLS) handshake is a real round-trip tax, paid before a single byte of your data moves. When a server talks to a database thousands of times a second, re-opening a connection every time is its own kind of C10K disaster. The fix is to open a few and keep them. Next up: Connection Pools — Reuse Beats Re-open.