Connection Pools: Reuse Beats Re-open

The last lesson taught you how one server can hold tens of thousands of open connections cheaply. This one is about the cost we quietly skipped: opening each of those connections in the first place is anything but cheap.

Here's the trap it sets. Your app needs to talk to its database, so — reasonably — for each incoming request it opens a connection to the database, runs the query, and closes it. Clean, simple, correct. And at ten requests a second it's fine. At ten thousand requests a second it's a catastrophe: your app is now paying a heavy setup tax ten thousand times a second, and — worse — trying to open ten thousand connections to a database that falls over at a few hundred. The connection-holding lesson you just learned has an evil twin: connection opening.

The fix is one of the most reused ideas in all of infrastructure, and its whole spirit fits in the lesson title: reuse beats re-open. Don't build a new connection for every request — open a small handful once, keep them warm, and share them. That shared handful is a connection pool, and by the end of this lesson you'll know why the smart move is often to have fewer connections than you'd guess, not more.

Scope: we cover the pattern generically — it's the same move for database connections, HTTP connections, and worker threads. The exact bytes of a TCP/TLS handshake come later (Networking), and database-specific pool tuning gets its own treatment in the data-systems course. Today: the idea, and how to size it.

"Open a connection" sounds instant. It isn't — and the reason is round trips. Before a single byte of your actual request can travel, the two machines have to negotiate, and each step of that negotiation is a full round trip across the network:

• The TCP handshake (SYN → SYN-ACK → ACK) establishes the connection: 1 round trip before any data.
• The TLS handshake on top of it — exchanging certificates and deriving encryption keys — adds 1 more round trip (TLS 1.3) or 2 (the older TLS 1.2).

So an encrypted connection costs roughly 2–3 round trips of pure waiting before your first useful byte moves. And here's the sting: a round trip's duration is set by distance, not by how fast your servers are. On a phone network at 80 ms per round trip, that's ~240 ms of handshake. From Asia to a US server at ~250 ms per round trip, it's 600–750 ms — most of a second — spent just opening the line, before the server has done a lick of work.

For a database it's worse still, because on top of TCP you pay an authentication handshake and the server-side cost of allocating the connection (remember from two lessons ago: PostgreSQL forks a whole process per connection). Opening a fresh database connection routinely costs tens to hundreds of milliseconds.

The shape of the cost is the key: it's a fixed, one-time setup tax per connection. Pay it once and you can send a million requests down that line. Pay it per request and it's the most expensive thing you do all day.

Once you see the cost as a one-time setup tax, the fix writes itself: pay it once, then reuse.

Picture a city. The open-a-connection-per-request design is a world where every passenger who wants a ride builds a car from scratch, drives it to their destination, and scraps it on arrival. Every single trip: build a car, use it once, junk it. It's absurd — and it's exactly what reconnecting per request does.

A connection pool is the obvious sane alternative: a taxi fleet. A dispatcher keeps a small number of cars warm and circulating. A rider walks up, takes a free taxi (no building a car — it's already running), rides to their destination, and the taxi returns to the stand for the next rider. If every taxi is out, new riders wait in a short line until one comes back. A handful of cars moves an entire city, because each car is reused, not rebuilt.

In real terms, the pool does exactly three things on repeat:

1. Borrow (checkout): a request asks the pool for a connection. Idle one available? handed over instantly — no handshake. All busy? the request waits in a queue until one frees or a timeout fires.
2. Use: the request runs its query on the warm connection.
3. Return (release): the connection is reset to a clean state (any open transaction rolled back, temporary settings cleared) and put back in the pool — not closed. It's immediately ready for the next request.

That's the whole engine. The handshake tax from the last section is now paid once per connection at startup and amortized across every request that borrows it. A connection opened once can serve millions of queries over its life. Reuse beats re-open — not by a little, by orders of magnitude.

This is a system with a sweet spot, and the only way to feel a sweet spot is to overshoot it in both directions. Below is a live pool. You set how many connections it holds and how fast requests arrive; watch each request borrow a warm connection, run, and return it — or pile up in the queue and time out when they're all busy.

Two experiments worth running. First, start at a pool size of 1 and drag it up: throughput climbs fast. Keep going — past a point, throughput stops rising and starts falling, because now the database itself is thrashing under too many connections. That peak is the sweet spot the rest of this lesson explains. Second, hit "leak a connection" a few times and watch the pool quietly bleed capacity until requests hang forever — the failure mode that pages engineers at 3 a.m.

Here's the result that trips up almost everyone. Faced with 3,000 concurrent users, the instinct is "give me a big pool — 200 connections, 500, more is safer." The opposite is true. The pool that serves those 3,000 users fastest is often around ten connections.

The reason is the database on the other end. A database server has a fixed number of CPU cores and disks; it can only truly do a handful of things at once. Hand it 200 connections and it doesn't do 200 queries in parallel — it does a few, and spends the rest of its energy context-switching between them, contending for locks and buffers, doing the paperwork of juggling instead of the work of querying. Throughput doesn't rise with connection count; it rises, peaks, and then falls. More connections past the peak make the database slower. (It's the same wall as the last lesson: more workers than the machine can run means the machine spends its time switching, not working.)

There's even a rule of thumb from HikariCP, a popular pool: the sweet spot is roughly (CPU cores × 2) + number of disks. A humble 4-core database wants a pool of about 10 — and that pool can serve thousands of front-end users at thousands of queries per second, because at any given instant most of those users aren't actually running a query; they're thinking, reading, waiting on the network. A tool called pgbouncer leans all the way into this: it lets roughly 20 real database connections serve 300 application clients, handing a connection to a client only for the length of a single transaction.

The mental correction: size the pool to the downstream's capacity, not to your number of users. The pool isn't there to give every request its own connection — it's there to feed the database exactly as much work as it can actually do, and make everyone else wait their (very short) turn.

That "make everyone else wait their turn" is not a bug — it's the pool's second, underrated superpower, and it directly rescues the disaster we opened with.

Because a pool has a maximum size, it's also a cap. When all connections are checked out, the next request doesn't get to pile more load onto the database — it waits in the queue (or fails fast after a timeout). That queue is backpressure: a controlled pushback that keeps an overwhelmed system from being pushed past the edge.

Think back to the 10,000-connections lesson. Your event-loop server can happily hold 10,000 client connections. But if all 10,000 tried to hit the database at once with their own connection, the database — capped at a few hundred — would instantly fall over, and take your whole system with it. The pool stands between them like a turnstile: 10,000 requests can arrive, but only, say, 10 are ever touching the database at once; the other 9,990 briefly queue. The system slows gracefully under overload instead of collapsing. A pool converts a stampede into an orderly line.

This is why a connection pool is a reliability feature as much as a performance one. It's the difference between "we got slow for a minute during the traffic spike" and "the database crashed and we were down for an hour."

A pool is a shared, finite resource, and that shape hands you a set of failure modes the naive open-per-request design never had. This is the honest bill: a pool buys enormous speed and backpressure, and charges you in operational discipline.

• Leaks — the silent killer. If a request borrows a connection and forgets to return it (a missing release, an exception thrown before the return), that connection is gone from the fleet forever. One leak shrinks the pool by one; a slow leak is invisible until the pool hits zero — and then every request hangs forever, waiting on a connection that will never come back. Even a ~1% leak rate can take a service down under load. The fix is a habit: always return in a finally/using/context-manager block, and turn on leak detection.
• The death spiral. Pool empty → requests hit their acquire timeout → the client retries → the retry demands another connection from the still-empty pool → things get worse, faster. Under load, retries on an exhausted pool are gasoline on a fire.
• Stale connections. A pooled connection can die quietly — the database restarted, a firewall dropped an idle socket. Borrow a dead one and your query fails. The fix: validate a connection before handing it out (a cheap ping) and cap its max lifetime so connections are recycled before they rot.
• Mis-sizing, both ways. Too small and requests queue and time out while the database sits half idle — you starved yourself. Too big and you swamp the database (or blow past its connection limit entirely). There's a sweet spot, and "more" is not the safe direction.

None of this makes pooling optional — at any real scale it's mandatory. It just means a pool is a thing you operate, not a thing you set and forget.

Once you have the shape — a small set of expensive-to-create things, kept warm and shared — you start seeing it everywhere, because it's one of the most reused patterns in systems:

• Database connection pools are the classic case: HikariCP (Java), the pool built into nearly every database driver, and external poolers like pgbouncer and pgpool that sit in front of PostgreSQL.
• HTTP keep-alive is the exact same move for the web: instead of a fresh TCP+TLS handshake for every request, a browser or service reuses one open connection for many requests. It's the default in HTTP/1.1 and the whole point of it.
• Thread pools (from the last lessons) are pooling applied to workers rather than connections: creating a thread is expensive, so keep a fixed set of them warm and hand out tasks.

Different resources, identical idea: creating the thing is expensive, so create a few and reuse them under a cap. When you meet database internals later in the course, connection pooling will already be an old friend — you'll just be learning the specific dials.

"A bigger pool means more throughput." Usually false. Past the downstream's real capacity, extra connections add contention and context-switching and throughput drops. Size to the database's cores, not your user count.

"The pool makes each query faster." No — the query runs at the same speed. The pool removes the per-request opening cost (handshake + auth + TLS), which is often bigger than the query itself.

"Just open a connection when you need it — it's simplest." Fine at toy scale; fatal at real scale, where the handshake tax and the connection explosion (10,000 requests → 10,000 database connections → a dead database) sink you.

"A returned connection is closed." The opposite — return means recycled, reset and kept warm for the next borrower. Reuse is the entire point.

"With a pool I can never run out of connections." You absolutely can — through leaks or an undersized pool. Return-discipline, sizing, and timeouts are the job.

"The pool should have one connection per app thread or user." No — many requests share a few connections, because at any instant most requests aren't actually running a query. Size to the downstream, not the crowd.

Ten minutes turns this from a diagram into a dial you've turned. Predict first: if you point a load test at a small pool and then a huge one, which gives higher throughput?

# 1) See the handshake tax with your own eyes (reconnect every request vs reuse):
#    -H '' forces a new connection each time; keep-alive reuses one.
ab -n 2000 -c 50 https://your-service/health          # reuses connections (keep-alive)
ab -n 2000 -c 50 -H 'Connection: close' https://your-service/health   # new conn each time
#    Compare the 'Time per request' — the gap is the setup tax.

# 2) If you run Postgres, watch live connections while you load-test your app:
watch -n1 "psql -c 'SELECT count(*) FROM pg_stat_activity;'"   # should plateau at pool size

Two things to notice: the keep-alive run posts dramatically lower per-request times (that gap is the handshake you stopped paying), and your database's live connection count plateaus at your pool size no matter how many requests you throw — that plateau is the backpressure cap doing its job. Then, in your app's pool config, try setting the size absurdly high and re-running the load test; on a busy database you'll often watch throughput get worse.

The whole lesson, compressed:

• Opening a connection is expensive — a fixed setup tax of 2–3 network round trips (TCP + TLS), hundreds of milliseconds on a real link, plus auth for a database. Paying it per request is the most wasteful thing you can do.
• A connection pool pays that tax once per connection and reuses warm connections: borrow → use → return (recycled, not closed), with a queue when all are busy. Reuse beats re-open by orders of magnitude.
• Size to the downstream, not the crowd. A small pool (~cores × 2) often beats a big one; more connections than the database can run makes it slower. Around 10 connections can serve thousands of users.
• The pool is also a backpressure cap — it turns a stampede into an orderly queue and protects the database from collapse. The price is operational discipline: leaks, the death spiral, stale connections, and mis-sizing are all yours to manage.

Notice a thread running through the last three lessons: threads, event loops, and pools are all about how many things happen at once, and who's allowed to. We've been using the words "concurrent" and "parallel" almost interchangeably — but they are not the same thing, and conflating them is behind a huge share of performance confusion (including why more threads or a bigger pool can make things slower). It's time to pull them apart cleanly. Next: Concurrency vs Parallelism.