Codelab: Run a Chaos Experiment

Goal: Your First Real Experiment

Nine lessons of resilience end the only way resilience can be proven: by breaking something on purpose and watching what happens. In the next fifteen minutes you will define what normal means for a real service, write a falsifiable hypothesis, inject a real failure into a real dependency, and discover a genuine weakness that no amount of reading could have told you was there. Then you'll fix it and prove the fix holds under the exact same failure.

The experiment is deliberately small, because Chaos Engineering was clear that small is the whole point. A tiny checkout service reads one value from Redis. Between them sits Toxiproxy, a proxy built for exactly this: it forwards traffic normally until you tell it, at runtime, to add latency or to stop answering, and it stops the moment you say so. That is your valve, and the valve is what makes this safe: a blast radius of one dependency and an abort that is one command away. Nothing here is simulated. Every command and every number below was run exactly as printed (Toxiproxy 2.12.0, hey 0.1.5, Redis 8.8.0). You already know every idea you're about to type. The point is to feel it.

One laptop running a whole chaos experiment. A checkout service on port 4600 on the left calls Redis on port 6399 on the right, but only through a Toxiproxy valve in the middle, drawn with a bowtie valve body and a turn-handle; a red toxic burst sits on the wire from the valve to Redis. Across the top runs the steady-state success rate: it holds flat along a 100 percent line, falls off a cliff to the 0 percent floor the moment the valve is shut and the dependency stops answering (labeled valve shut, hang), then climbs back to 100 percent when the valve reopens. Law band: define normal, then break it on purpose; a blast radius of one, an abort one command away. Foot: you find the missing timeout in a lab, not at 3 a.m.

The Setup: A Service, a Dependency, and a Valve

Three installs, all standard. On macOS with Homebrew (Linux: your package manager has all three, or build Toxiproxy with go install):

$ brew install toxiproxy hey
$ redis-server --version
Redis server v=8.8.0 (00000000)
$ toxiproxy-server --version
toxiproxy-server version 2.12.0

Now the service. It is about fifteen lines and needs no framework and no npm install: it opens a raw socket to Redis, asks for one key, and returns it. Save it as app.js. Read the one comment near the bottom twice, because that missing line is the entire lesson:

// app.js — a checkout service that reads one key from Redis. ~15 lines, no framework.
const http = require('http'), net = require('net')
const REDIS = Number(process.env.REDIS_PORT || 26379)   // point at the PROXY, not at Redis

function getColor() {                                    // read one key over a raw socket
  return new Promise((resolve, reject) => {
    const sock = net.connect(REDIS, '127.0.0.1', () => sock.write('GET color\r\n'))
    let buf = ''
    sock.on('data', d => { buf += d
      if (buf.startsWith('$')) { sock.end(); resolve(buf.split('\r\n')[1]) } })
    sock.on('error', reject)
    // NOTE: there is no timeout here. Hold that thought.
  })
}

http.createServer(async (req, res) => {
  try { const c = await getColor(); res.writeHead(200); res.end(c + '\n') }
  catch (e) { res.writeHead(502); res.end('error\n') }
}).listen(4600, () => console.log('checkout-svc on :4600 -> redis :' + REDIS))

The wiring is the important part, and it is the reason this is safe. The service will not talk to Redis directly. It will talk to Toxiproxy, and Toxiproxy talks to Redis. Start a real Redis on its own port, put the proxy in front of it, then point the service at the proxy:

# 1. a real dependency, on its own port (so we never touch anything else)
$ redis-server --port 6399 --save '' --daemonize yes
$ redis-cli -p 6399 set color blue
OK

# 2. the valve: start Toxiproxy, then create a proxy in front of Redis
$ toxiproxy-server &
...  "Starting Toxiproxy"  ...  api listening on localhost:8474
$ toxiproxy-cli create -l 127.0.0.1:26379 -u 127.0.0.1:6399 redis
Created new proxy redis
$ toxiproxy-cli list
redis	127.0.0.1:26379	127.0.0.1:6399	enabled	0

# 3. point the service at the PROXY (:26379), not at Redis (:6399)
$ REDIS_PORT=26379 node app.js
checkout-svc on :4600 -> redis 127.0.0.1:26379

The lab topology, wired in a line. On the left, the checkout service on port 4600. It does not talk to Redis directly; its arrow points at a Toxiproxy valve on port 26379 in the middle, drawn with a bowtie valve body and a turn-handle, and only the valve reaches Redis on port 6399 on the right, drawn as a cylinder. A dashed green blast-radius ring wraps the valve, the wire, and Redis. A red toxic burst sits on the wire from the valve to Redis, labeled toxic here. Law band: one dependency on a leash; every failure reversible with one command.

That middle hop is the whole trick. Every byte between the service and its dependency now flows through a valve you control, and everything you do to that valve you can undo instantly. This is the blast radius made literal: one dependency, one proxy, one command to heal. It is Latency Monkey from the last lesson, running on your laptop, with a hand on the handle.

Step 1: Define Steady State

You cannot tell that a system broke unless you first agreed on what working looks like, in a number. That number is the steady state, and the rule from Chaos Engineering is that it should be user-facing, not machine-facing: measure what a customer feels, not CPU. For this service, the customer feels two things — does the request succeed, and how long did it take. So put real load on it and write down normal. hey fires a burst of requests and reports the percentiles you met in Tail Latency: Living at p99:

$ curl -s http://127.0.0.1:4600/
blue
$ hey -n 2000 -c 50 http://127.0.0.1:4600/
Summary:
  Requests/sec:	9662.6
  Average:	0.0050 secs
Latency distribution:
  50% in 0.0049 secs
  99% in 0.0206 secs
Status code distribution:
  [200]	2000 responses

There is your normal, in three numbers you will hold the rest of the experiment against: 100 percent of requests succeed (all two thousand came back 200), p50 about 5 milliseconds, p99 about 20 milliseconds. Write them on a sticky note. Everything that follows is measured as a distance from this line.

Step 2: Write the Hypothesis (and the Stop Condition)

Now the sentence that separates an experiment from vandalism. A hypothesis is a falsifiable prediction about the steady state under a specific failure, written before you inject it:

If Redis slows down by 300 milliseconds, checkout still succeeds for 100 percent of requests, because a little dependency latency should only make us a little slower.

Say it out loud, because you are about to try to prove yourself wrong. That is the job: not to confirm the system works, but to try to disprove the hypothesis and see what you learn when it survives, or when it doesn't. And before you touch anything, decide your stop condition and your abort, in advance, the way the last lesson insisted: the blast radius is this one proxy, and the abort is a single command, toxiproxy-cli toxic remove, which you will keep in a terminal ready to run. If anything looks worse than you can stomach, you pull that trigger and the failure is gone. Decided now, in cold blood, so you never have to decide it in a panic.

Step 3: Inject a Little (and Watch It Hold)

Turn the valve a quarter turn. Add 300 milliseconds of latency to every byte coming back from Redis, then measure against your sticky note:

$ toxiproxy-cli toxic add -t latency -a latency=300 redis
Added downstream latency toxic 'latency_downstream' on proxy 'redis'
$ hey -n 200 -c 20 http://127.0.0.1:4600/
  Average:	0.3063 secs
  50% in 0.3059 secs
  99% in 0.3163 secs
Status code distribution:
  [200]	200 responses

Read it carefully, because this is the boring result that matters. Latency went exactly where you pushed it — p50 and p99 both parked around 305 milliseconds, three hundred slower than baseline, precisely the toxic you added. But success stayed at 100 percent. Every request came back 200. Your hypothesis survived its first contact with reality: a slow dependency made the service slower, and nothing else. This is the Tail Latency lesson in one line of output, and if you stopped here you would walk away with false confidence. A real experiment does not stop at the first result that flatters it. It escalates until something gives.

Break It: Escalate Until Something Gives

Slow is the gentle failure. The cruel one, the one that actually takes services down, is the dependency that does not slow or die but simply stops answering — the gray failure from the start of this section, the half-open socket that will never send another byte. Toxiproxy has a toxic for exactly that. Swap the latency for a timeout toxic with timeout=0, which means the connection stays open and the data never comes:

$ toxiproxy-cli toxic remove -n latency_downstream redis
Removed toxic 'latency_downstream' on proxy 'redis'
$ toxiproxy-cli toxic add -t timeout -a timeout=0 redis
Added downstream timeout toxic 'timeout_downstream' on proxy 'redis'

# one request, by hand, with a 4-second patience limit:
$ curl -s -m 4 http://127.0.0.1:4600/
curl: (28) Operation timed out after 4001 milliseconds

The request never came back. curl gave up after four seconds; the service itself would have waited forever, because look again at app.js — there is no timeout on that socket. Now measure it under load and watch the steady state fall through the floor:

$ hey -z 8s -c 20 -t 3 http://127.0.0.1:4600/
  Requests/sec:	6.66
Status code distribution:
  [no successful responses]
Error distribution:
  [60]	Get "http://127.0.0.1:4600/": context deadline exceeded
        (Client.Timeout exceeded while awaiting headers)

The experiment as one timeline of the steady-state success rate, left to right, in four acts. Act 1, baseline: the line rides flat along the top at 100 percent success, about 5 milliseconds, healthy. Act 2, plus 300ms latency: the line stays at 100 percent success but the latency band swells to about 300 milliseconds — slow, but still up. Act 3, the timeout toxic: the dependency stops answering and, with no client timeout, the line falls off a cliff to zero percent success, every request hanging until it times out. Act 4, abort: one command removes the toxic and the line snaps back to the top. A callout with an arrow to the cliff reads one slow dependency equals whole service down. Law band: the same line, holding then falling; slow is survivable, silent is the killer.

Sit with this, because it is the most important minute in the section. Redis is not down. One connection is merely slow to the point of silence — and your service went from 100 percent success to zero. Not slow. Not degraded. Completely dark, every request hanging on a socket that will never answer, the service's own capacity filling with frozen requests that each hold their resources and wait. That is the whole of Cascading Failures reproduced on your laptop in one command: a single slow dependency, with no timeout to stop it, becoming a total outage. Your hypothesis is disproven, and this is the good outcome — you found the missing timeout in a lab on a Tuesday, not in production at 3 a.m. The finding is not "the system is broken." The finding is a specific, fixable gap: this service has no client timeout.

The Stop Condition: One Command Back to Safe

Before you fix anything, prove to yourself that the abort works, because that is the promise that let you run this without fear. Pull the trigger you loaded in Step 2:

$ toxiproxy-cli toxic remove -n timeout_downstream redis
Removed toxic 'timeout_downstream' on proxy 'redis'
$ curl -s http://127.0.0.1:4600/
blue
$ hey -n 1000 -c 50 http://127.0.0.1:4600/
  Average:	0.0053 secs
  [200]	1000 responses

Instant. The moment the toxic was gone the service was back to its 5-millisecond, 100-percent self, as if nothing had happened, because from Redis's point of view nothing did. This is what the blast radius bought you: a failure you could turn on to learn from, and turn off the instant you were done. A controlled burn, put out with one bucket. Now go close the gap you found.

Close the Gap: and Prove It

The weakness was a socket with no deadline, so give it one. Two lines change in app.js: a 200-millisecond timeout on the Redis call, and, instead of erroring when it fires, a fallback to the last value we successfully read. This is Timeout Hierarchies and the graceful degradation of Load Shedding & Graceful Degradation, together, in the smallest possible form:

// app.js — two lines change. A client timeout, and a fallback instead of an error.
let lastGood = 'blue'                                    // remember the last value we read

    const sock = net.connect(REDIS, '127.0.0.1', () => sock.write('GET color\r\n'))
    sock.setTimeout(200, () => { sock.destroy(); reject(new Error('redis timeout')) })  // <- ADD

  try { const c = await getColor(); lastGood = c; res.writeHead(200); res.end(c + '\n') }
  catch (e) { res.writeHead(200, {'X-Degraded': 'true'}); res.end(lastGood + ' (cached)\n') }  // <- was 502

Restart the service, warm it once so it has a value to fall back to, and then run the exact same experiment that just took it down — same toxic, same load. This is the step most people skip and the only one that proves anything:

$ toxiproxy-cli toxic add -t timeout -a timeout=0 redis
Added downstream timeout toxic 'timeout_downstream' on proxy 'redis'
$ curl -s -D - http://127.0.0.1:4600/ | grep -iE 'HTTP|Degraded|cached'
HTTP/1.1 200 OK
X-Degraded: true
blue (cached)
$ hey -n 300 -c 20 -t 3 http://127.0.0.1:4600/
  Average:	0.2052 secs
  99% in 0.2086 secs
Status code distribution:
  [200]	300 responses

The same dead dependency, before and after the fix, side by side. Left, before, no timeout: the service calls a Redis cylinder marked with a red cross (dead), and with no timeout the requests pile up forever, drawn as three stacked red bars; the verdict is 0 of 300 ok. Right, after, a 200ms timeout: the same dead Redis, but a clock on the call makes the service give up at 200 milliseconds and serve a cached value instead, fast and a little stale; the verdict is 300 of 300 ok. Foot: a timeout turns an outage into a blip.

The identical failure that produced zero successful responses now produces three hundred out of three hundred. The dependency is just as dead as before; the difference is that the service no longer waits on it forever. It waits 200 milliseconds, gives up, and serves the last-known answer with a header that admits it is stale. Latency is capped at the timeout, success is back to 100 percent, and a customer gets a slightly old color instead of a spinner that never ends. That is the entire argument of this section, earned rather than asserted: you injected a real failure, watched a real defense hold, and now you know — not hope — that a slow Redis cannot take checkout down. A hypothesis, disproven, fixed, and re-proven, in fifteen minutes.

Clean Up

$ toxiproxy-cli toxic remove -n timeout_downstream redis   # heal the valve
$ redis-cli -p 6399 shutdown nosave                        # stop Redis
$ kill %1                                                  # stop toxiproxy-server
$ pkill -f 'node app.js'                                   # stop the service

What Just Happened: Every Toxic Was a Lesson

Scroll back through your terminal. You did not run a demo; you ran the whole section, wearing work clothes.

The steady state, in a percentile — success rate and p99, measured under load before you touched anything: the user-facing metric from Chaos Engineering, and the percentiles from Tail Latency: Living at p99.
The proxy as a blast radius — one dependency on a leash, every failure reversible with one command: the small, contained, abortable experiment the whole discipline is built on.
The latency toxic that held — 300 milliseconds of slowness that only made you slower: a dependency behaving badly within tolerance, and a hypothesis surviving.
The timeout toxic that didn't — a dependency that stopped answering, with no client timeout to stop it, turning one slow socket into a dark service: Cascading Failures, live, and the exact gap Timeout Hierarchies exists to close.
The one-command abort — the stop condition you loaded before you fired, proving the burn was always controlled.
The fix that held — a timeout plus a cached fallback, converting an outage into a barely-noticed blip: Timeout Hierarchies and Load Shedding & Graceful Degradation doing their jobs under a failure you could see.

Change one thing and run it again, because that is what the practice actually is. Add a retries loop with no budget and watch a Retries, Budgets & Retry Storms storm build. Wrap the call in a breaker and watch Circuit Breakers trip. Give the fallback its own bounded pool and you've drawn a Bulkhead. The lab is the same; only the toxic changes.

Key Takeaways

An experiment needs a number, a sentence, and an escape hatch. A steady state you can measure, a falsifiable hypothesis written before you inject, and a stop condition decided in advance. Skip any one and you have a demo, or an incident.
Slow is not the failure that kills you. Silent is. The dependency that stops answering, not the one that dies, is what fills your service with frozen requests — and the only thing standing between one silent socket and a total outage is a timeout you set on purpose.
A defense is a hypothesis until you inject the failure. You did not read that the fix worked; you watched zero-out-of-many become many-out-of-many under the identical toxic. That is the difference chaos engineering exists to create.
The blast radius is what makes it safe, not brave. A proxy you can heal with one command let you break production-shaped things without breaking production. Small, reversible, on purpose.

That is the section, cashed at a keyboard: every defense from Timeout Hierarchies through Chaos Engineering, proven by breaking something and watching it hold. What remains is to prove you own it — ten decisions drawn from the whole of resilience and overload control, in Checkpoint: Resilience & Overload Control.