AWS Kinesis, 2020: The Thread-Limit Cascade

It was the Wednesday before Thanksgiving, 2020, and someone at Amazon did the most routine thing an operator can do: they added a little capacity to a busy service. A few more servers in a fleet that already had thousands. The kind of change that happens hundreds of times a day across a cloud provider and gets noticed by no one.

By the end of the day, a huge slice of the internet had been broken for about 17 hours. Not just Amazon's own Kinesis service, but CloudWatch, Cognito, Lambda, and a long tail of companies whose apps quietly ran on top of them. And here is the twist that makes this outage worth a whole lesson: the problem wasn't too little capacity. It was that adding capacity is what broke it.

Sit with how backwards that is. When a system is struggling, the reflex baked into every on-call engineer is add more servers. More capacity, more headroom, more safety. This is the story of a system where more servers meant more load on every existing server, until they all crossed an invisible line at the same instant and fell over together. It's a lesson about costs that grow faster than the thing you're scaling, and the hidden ceilings they hide behind.

In this lesson: the boring change that triggered it; how the front-end fleet's own chattiness became a trap; the operating-system limit nobody was watching; why the failure spread to services that had nothing to do with the change; and the one question about your own system this should leave you asking.

Scope: the general shape these failures share is drawn out in Anatomy of a Cascading Failure; here we stay with this one event and what it teaches.

To see what happened you need to know one word: Kinesis. It's Amazon's service for moving streams of data in real time. You fire records at it, it buffers and routes them, and a small army of other systems reads from it. What matters here is the piece that takes your incoming requests and figures out where they should go: the front-end fleet, thousands of servers sitting in front of the actual storage.

At 2:44 in the morning, Pacific time, Amazon began adding a relatively small amount of capacity to that front-end fleet. A few servers. It finished at 3:47 a.m. Nothing alarmed, nothing paged, nobody watching thought twice. The change was so ordinary that its own authors would have struggled to describe it as anything but housekeeping.

Then, a little after 5 a.m., the alarms started. Kinesis began returning errors. And the engineers who came online to look had a genuine mystery on their hands, because the change that caused it looked completely innocent, and the thing it broke was buried in a corner of the system nobody thinks about until it kills you: the way the servers keep track of each other.

Here's the piece of design that turned a small change into a large disaster. Every server in the front-end fleet needs to know about every other server, so it can build what Amazon calls a shard-map: a cache of which back-end machine owns which slice of the data, so an incoming request can be sent to the right place. It's the same idea you met in Consistent Hashing and Heartbeats & Failure Detection, a membership picture each node keeps of the cluster it lives in.

The trouble is how they built it. Each front-end server opened a connection to every other front-end server to learn the map, and each of those connections got its own operating-system thread. Read that again, because the whole outage is hiding in it. If there are a thousand servers, each server is running roughly a thousand of these threads, one per peer. If there are two thousand servers, each server runs two thousand threads. The number of threads on every single server grows with the size of the whole fleet.

This is a full mesh: everyone connected to everyone, a thread for each. And it has a property that's easy to miss when the fleet is small and everything works. Adding one server doesn't just add one server's worth of load. It adds one more thread to every server that already exists. The cost of the fleet's own bookkeeping doesn't grow with the fleet, it grows with the fleet squared — and that's the kind of cost that stays invisible right up until the moment it isn't.

Every operating system caps how many threads a single process is allowed to create. It's a sane, boring safety limit, set in a config file somewhere, the sort of number you set once and forget for years. On the Kinesis front-end servers, that number was about to matter more than anything else in the building.

When the new servers joined the fleet, every existing server dutifully opened a new thread to each of them. That nudged the thread count on every server upward, and because the fleet was already large, that nudge was enough to push them over the operating system's maximum. Once a server hits that wall, it can't create the threads it needs, so it can't finish building its shard-map. It ends up holding a useless map, doesn't know where anything lives, and starts failing the requests it was supposed to route. Not one server. Every server in the fleet, at essentially the same time, for the same reason.

This is the counterintuitive heart of the whole event, and it's worth stating as a rule you can carry to any system you build: any cost that grows faster than the thing you're scaling will fail catastrophically after a modest amount of growth, and it will do it without warning. A linear cost gives you a gentle, readable slope: twice the servers, twice the load, you see it coming. A cost that grows with the square hides flat and quiet until it suddenly goes vertical and slams into a limit you forgot you had. Nobody was graphing threads per server. Why would they? It had never been a problem. The console below lets you feel this for yourself: grow the fleet, watch the threads-per-server climb toward the ceiling, and try to fix an overloaded fleet the way a real operator would, by adding servers.

A broken Kinesis is bad. What made this a headline is everything that broke because of it, and none of those things had anything to do with the capacity change. This is the shape you studied in Cascading Failures, playing out at the scale of a public cloud.

Kinesis wasn't just a product Amazon sold to customers. It was plumbing that Amazon's own services ran on, and once it started failing, the failure walked straight up the dependency chain. CloudWatch, the monitoring service, used Kinesis to move metrics and logs, so its APIs began erroring, which meant a lot of teams lost visibility right when they needed it most. Cognito, which handles sign-in, used Kinesis for some of its data, and a latent bug meant its web servers blocked on the backlogged Kinesis buffers instead of shrugging the errors off, so people simply couldn't log in to apps that had nothing to do with streaming data. Lambda published metrics through the failing path, and the buffered metric data grew until it caused memory pressure on the hosts. EventBridge delayed events, which delayed ECS and EKS from launching containers. One boring change, radiating outward through the plumbing.

The lesson underneath the wreckage is about shared internal dependencies. Cognito failing because Kinesis failed is not a coincidence of bad luck. It's the predictable result of a single service sitting quietly beneath a dozen others, so that its worst day automatically becomes their worst day too. The more central and invisible a dependency is, the wider its blast radius when it goes.

There's a detail in Amazon's own writeup that lands like a punchline, and it will feel familiar if you just read Facebook, 2021: The BGP Self-Lockout. As the outage spread, Amazon tried to tell customers what was happening by updating its public Service Health Dashboard — the status page. They couldn't. The tool they use to post updates to the status page itself depended on Cognito, and Cognito was one of the services that was down.

So the outage had, in effect, disabled the outage's own alarm bell. It's the same species of problem as Facebook's engineers being locked out by their own badge readers: a path you depend on in an emergency, quietly wired through the very system that's on fire. Amazon had a backup way to post updates that leaned on fewer dependencies, but it was clumsier and less familiar, and the people reaching for it in the moment weren't practiced with it, so the early updates came slowly.

That's worth a rule of its own, because it's a trap teams fall into constantly: a fallback you never use is a fallback that doesn't work. The backup tool existed. On paper the company was covered. But a recovery path that only gets exercised during a real disaster will have quietly rusted shut by the time you need it, and your status page, of all things, should not share fate with the systems it's meant to report on.

Knowing the cause didn't make the fix fast, and the reasons it stayed broken for most of a day are as instructive as the break itself.

The obvious move once you understand the thread limit is to just raise it. Amazon deliberately didn't. Changing an operating-system limit across a huge fleet, in the middle of an active incident, without time to test what else it might destabilize, is exactly the kind of heroic fix that turns one outage into two. Instead they did the humbler thing: they removed the capacity they'd added, to get the thread counts back under the ceiling. As the saying among the engineers who study these events goes, the most common cause of an outage is a change, and the most reliable fix is undoing it.

But bringing the fleet back was its own slow grind, for a reason that rhymes with Facebook's careful restart. When a front-end server starts up, building its shard-map is expensive — it competes for the very resources the server needs to answer live requests. Restart too many servers at once and they all sit there building maps, starving the actual traffic, and you never climb out of the hole. So recovery had to be throttled, adding servers back at a rate of only a few hundred an hour, each one slowly learning the map before the next batch joined. The first servers took traffic again around 10 a.m.; Kinesis wasn't fully healthy until 10:23 that night. Recovery, once more, was not a switch you flip. It was a controlled climb.

You're almost certainly not running a fleet of thousands. But the trap that caught Kinesis is small and general, and it's probably already in your system somewhere, waiting for a growth spurt to find it.

Find the costs that grow faster than your load. Walk your system and ask where the work grows faster than the thing driving it. Anything where every node talks to every other node is the classic one: full-mesh service discovery, every worker subscribing to every other worker, a cache that every instance broadcasts invalidations to. Those are the square-shaped costs. They're fine at ten nodes and fatal at ten thousand, and the danger is that they look identical until the day they don't.

Know where your ceilings are, and alarm before them. Threads per process, open file descriptors, ports, connections in a pool, entries in a table with a hard cap — every system has limits like these, usually invisible until you hit them. The fix is boring and it works: estimate roughly where each limit sits, then put an alert at, say, 70% of it, so you get a warning weeks before the wall instead of an outage at it.

Restructure the cost so it grows linearly. When you find a super-linear cost, the answer isn't to nudge the limit up and hope. It's to change the shape. Use bigger, fewer machines so there are fewer peers to track. Split the fleet into cells, the isolation idea from Cell-Based Architecture & Deployment Stamps, so each server only coordinates inside a small group and the per-server cost stops growing with the whole system. Or replace all-to-all chatter with gossip, from Gossip Protocols, where each node talks to only a few random peers and the news still spreads, but the cost per node stays flat. Amazon's own fixes were exactly these: larger servers, and cellularizing the front-end fleet to keep it inside a range they'd actually tested.

And let dependents degrade, not collapse. Cognito didn't have to fall over when Kinesis did; it fell over because it blocked on a backed-up buffer instead of shedding the load. The defenses from Bulkheads, Backpressure, and Load Shedding & Graceful Degradation exist precisely so that a dependency's bad day degrades you instead of taking you down with it.

Reason through this before you read the answer. You run a fleet of 50 stream-processing workers. For coordination, each worker keeps a live connection open to every other worker so they can rebalance work between them, and each connection costs one thread plus a little memory. Traffic doubles for a big launch, so you double the fleet to 100 workers to keep up. What happens to the per-worker thread count, and what's the real fix?

Work it out, then check yourself:

• What happens: going from 50 to 100 workers doesn't just add load, it roughly doubles the threads and connections on every existing worker, because each one now tracks 99 peers instead of 49. The total coordination cost across the fleet roughly quadruples (it grows with the square). You added capacity to handle more traffic and may have moved closer to a hidden per-worker limit, not further from it — the Kinesis trap exactly.
• The real fix: don't scale the mesh. Break the workers into cells of, say, 10 that only coordinate within their group, so per-worker connections stay near 9 no matter how large the fleet grows; or move coordination to a small dedicated service (or a gossip scheme) so no worker has to know about all the others. And put an alarm on per-worker connections at 70% of the OS limit, so the next doubling warns you in advance instead of paging you at 3 a.m.
• The tell to remember: whenever the honest answer to what does adding a node cost? is a little more on every other node, you have a square-shaped cost, and it needs restructuring before it needs scaling.

This outage gets miscompressed in a few predictable ways, and each wrong version hides a wrong lesson.

• Myth: it was a memory leak / the servers ran out of RAM. No. The wall they hit was the operating system's limit on the number of threads a process may create. It was thread exhaustion, not memory. (Lambda hit memory pressure later, but that was a downstream effect, not the cause.)
• Myth: adding capacity should always help. This is the whole point. Because each server's thread count grows with the fleet size, adding servers raised the load on every server and pushed them past the limit. Scaling out was the trigger, not the cure.
• Myth: they should have just raised the thread limit. Tempting, and exactly what Amazon refused to do mid-incident. Bumping a low-level OS limit across a giant fleet with no time to test what else it affects is how you cause a second, worse outage. Removing the added capacity was the safer undo.
• Myth: once they knew the cause, recovery was quick. No. Cold-starting a front-end server means building the shard-map, which competes with serving traffic, so the fleet had to be brought back a few hundred servers an hour. Knowing the cause at 9:39 a.m. still meant a recovery that ran until after 10 p.m.
• Myth: it was a Kinesis problem, so only Kinesis users were hit. No. Kinesis was internal plumbing under CloudWatch, Cognito, Lambda, and more, so the blast radius reached apps and companies that had never knowingly touched Kinesis in their lives — and even Amazon's own status page.

• The trigger was a routine capacity addition; the root cause was a hidden ceiling. A small addition to the Kinesis front-end fleet pushed every server past the operating system's maximum thread count, so none of them could finish building their shard-maps.
• The fleet's coordination cost grew with the square of its size. Every front-end server kept one thread open to every other server, so adding servers raised the thread count on all of them at once. A cost that grows faster than your load hides flat, then fails catastrophically without warning.
• Adding capacity caused the outage. The reflex to scale out made it worse, because the bottleneck wasn't capacity, it was a per-server limit that scaling out pushed everyone into simultaneously.
• A shared internal dependency spread one failure everywhere. Kinesis sat under CloudWatch, Cognito, Lambda, EventBridge, and ECS/EKS, so its failure became theirs — and the status page couldn't even be updated, because its own tool depended on a service that was down.
• Recovery was slow on purpose. Amazon removed the capacity rather than risk raising the OS limit mid-incident, then brought the fleet back a few hundred servers an hour because cold-start map-building competes with serving traffic.
• The rule to carry: find the costs in your system that grow faster than your load, put an alarm below the ceiling they're heading for, and restructure them to grow linearly with bigger boxes, cells, or gossip before you try to scale through them.

One outage came from a system that locked itself out; this one came from a system that scaled itself over a cliff. Both happened in the same place: Amazon's us-east-1 region, the one so much of the internet quietly leans on. That's not a coincidence, and it's where we go next — us-east-1: The Region the Internet Leans On.