Webhooks: We'll Call You

Every tool in this section so far has had one thing in common, and you might not have noticed it: you started the conversation. Polling, WebSockets, Server-Sent Events — in all of them, a client you control reaches out and opens a connection to a server you control, and then data flows across that held-open line. The whole of §7 has been about keeping your own client and your own server talking.

But look at where a huge amount of real-world "something happened" actually originates: inside another company. Stripe is the one who knows a customer's card finally cleared. GitHub is the one who knows someone pushed code. Twilio knows the SMS was delivered; Shopify knows the order shipped. That news is born on their servers, and your backend desperately needs it — to fulfill the order, to run the build, to update the dashboard.

You could poll them ("any new payments? any new payments?"), and for a while people did — but it's the freight-truck-for-a-letter problem at planetary scale: millions of integrators hammering Stripe every few seconds for events that rarely happen. And you obviously can't hold a WebSocket open from your server to every third-party service on Earth. So the arrow does the one thing it hasn't done all section: it reverses. Instead of you calling them, they call you. You hand a provider a URL — https://yourapp.com/webhooks/stripe — and when the event happens, they make a plain HTTP POST to it. That's a webhook: a reverse API, "don't call us, we'll call you."

And here's the thing that makes this a real lesson and not a footnote: the POST is trivial; receiving it safely is not. You're now accepting an unsolicited request, from another company, over the open internet, that you didn't initiate and can't retry. That flips every assumption, and it comes down to three guards. Skip any one and it's a genuine incident: fraud, a double charge, or a retry pileup. Let's build them.

Start with the guard people skip most and regret most. Your webhook endpoint is a public URL. It has to be — the provider's servers, out on the internet, need to reach it. But that means anyone else can reach it too. Nothing stops an attacker from sending your /webhooks/stripe endpoint a POST that says:

{ "id": "evt_fake", "type": "payment.succeeded", "amount": 4900 }

If your code trusts that at face value, you just shipped a $49 order to a fraudster who never paid a cent. And no, HTTPS does not save you — TLS encrypts the pipe and proves you're talking to your own server; it says nothing about whether the sender is really Stripe. This is the single most common webhook vulnerability, and it's why every serious provider signs its deliveries.

Here's how the signature works, and it's the JWT idea from §6 pointed the other way. When you register the endpoint, the provider gives you a shared secret. On every delivery, it computes an HMAC — a hash of the message keyed by that secret — and sends it in a header. Stripe's looks like:

Stripe-Signature: t=1700000000,v1=1baa87d9189ebcd0…

where v1 = HMAC-SHA256(secret, timestamp + "." + raw_request_body). To verify, your server does the same computation with the same secret and checks the results match. Only someone holding the secret can produce a matching signature, so a forged POST — which can't — is rejected. I ran exactly this: a genuine event verified; the same event with the amount tampered to 999999 failed with signature mismatch (the attacker changed the payload but couldn't re-sign it); a forged signature failed; and a replayed old delivery was rejected on its stale timestamp.

Two details separate people who get this right from people who get paged:

• Hash the RAW body, not the parsed JSON. You must HMAC the exact bytes that arrived. If you JSON.parse the body and re-serialize it to hash, the key order, whitespace, or number formatting can shift — different bytes, different hash, verification fails on perfectly valid events. Read the raw body, verify, then parse. This is the number-one signature bug.
• Compare in constant time, and check the timestamp. Use a constant-time comparison (timingSafeEqual) so an attacker can't tease out the secret by timing your responses, and reject deliveries whose timestamp is too old (~5 minutes) so a captured-and-replayed webhook can't be fired at you later.

Now the guard that's less obvious and even more load-bearing. Ask a simple question: how does the provider know you received the event? It waits for your server to return a 2xx response. No 2xx — because your server was briefly down, or slow, or the network dropped the reply — and, by design, it retries.

Follow the nightmare case carefully, because it's not a bug, it's physics. Your server receives payment.succeeded, does the work (ships the order), and sends back 200 OK — but that 200 is lost on the way back to Stripe. Stripe never heard success, so it retries. Your server receives the identical event again and, if it's naive, ships the order a second time. Nothing was broken. The event genuinely arrived twice.

This is not an edge case you can engineer away. Every major webhook provider guarantees at-least-once delivery, never exactly-once — because "exactly once" over an unreliable network is provably impossible, and honest systems don't pretend otherwise. Retries are the price of reliable delivery, and duplicates are the change you get back. So your receiver must be built to process the same event twice with no double effect.

If that sounds familiar, it should — it's the idempotency lesson from §6 (L40, Safe to Retry) coming back to collect a debt, and this is where it stops being a nice-to-have and becomes the load-bearing wall of the whole integration. The mechanism is the one you already know: every webhook carries a unique event_id. Before doing any work, check whether you've already processed that id; if you have, skip the work and just return 200. If you haven't, do the work and record the id. Keep those seen ids in a store with a TTL of around 72 hours — comfortably longer than any provider's retry window. That's the entire fix: event_id is the idempotency key, and "safe to retry" (§6) has become "safe to receive twice."

The third guard is the one that turns a working integration into a self-inflicted outage under load, and it follows directly from Guard 2. The provider doesn't wait forever for your 2xx — it has a timeout (seconds, not minutes). If your response doesn't come back in time, it does the same thing a lost 200 does: assumes you failed, and retries.

Now imagine your webhook handler does the real work inline — verify, then charge, call three internal services, render a PDF, send an email — and that takes 8 seconds. The provider's timeout fires at, say, 5. From its side, the delivery failed, so it retries — while your first handler is still running and about to succeed. You now do the entire job twice (idempotency saves your data, but you've still burned the work), and under a burst of events every slow handler spawns retries that pile onto an already-struggling server. That's how a webhook endpoint melts down.

The fix is a pattern you'll use everywhere real-time meets slow work: acknowledge immediately, process asynchronously. The moment a webhook arrives, do only the cheap, fast things — verify the signature, check the dedupe store, and drop the event onto a queue — then return 200 in a few milliseconds. A separate worker pulls from the queue and does the heavy lifting on its own time, retrying internally if it fails, completely decoupled from the provider's timeout. The provider gets its fast 200 and stops retrying; your slow work happens safely off to the side.

Notice what just happened: to receive a webhook well, you reached for a queue — a producer (the webhook handler) handing work to a consumer (the worker) so they can run at their own paces. That's not a coincidence; it's the exact shape of the next few lessons, and the arrow out of this one points straight at them.

Three guards are easy to nod along to and hard to feel until you've watched each one fail. So below is a live provider firing signed events across a company boundary at your endpoint — and your endpoint starts naive, all three guards off, so you can break it on purpose.

Fire a genuine payment.succeeded and it sails through. Now have the attacker lane forge the same event with no secret, and watch your naive endpoint cheerfully ship a free order — then flip verify signature on and watch the forgery bounce with a 401, because the HMAC (recomputed live, with real crypto, over the raw body) doesn't match. Type your own secret, or tamper the amount, and watch the hash break in real time.

Then drop your own 200 ACK and watch the provider — which only stops when it hears success — resend the identical event and charge your customer twice; flip dedupe on and the resend is answered from the seen-store with no double effect. Finally, drag your processing time past the provider's timeout and watch it decide you failed and retry a job that actually worked; flip ack-fast, work-async on and the real job slides onto a queue while the POST gets its instant 200. Every failed delivery schedules a retry on exponential backoff — 1s, 2s, 4s, 8s — and after enough failures your endpoint is disabled and the event is dead-lettered, exactly as Stripe does over three days. The scoreboard tallies what your choices cost: orders shipped to fraudsters, double charges, retries, dead letters.

Zoom out to the provider's side, because its reliability machinery is the other half of the story and it explains behavior you'll otherwise find baffling.

When a delivery fails — your endpoint returns a non-2xx, times out, or is unreachable — the provider doesn't give up and it doesn't hammer you. It retries on an exponential backoff with jitter: roughly 1s, then 2s, 4s, 8s, 16s, doubling each time, with a little randomness added so that a thousand integrators recovering from the same blip don't all retry in the exact same instant (the thundering-herd jitter from L1, back again). Stripe keeps this up for up to three days in live mode. If your endpoint is still failing after all that, it disables the endpoint and emails you — a failing webhook is treated as a broken integration, not an infinite backlog.

Two more provider-side realities to design around:

• Ordering is not guaranteed. Events can arrive out of order — charge.succeeded might land before the customer.created it belongs to, because retries and parallel delivery scramble the sequence. Never assume order. If you truly need it, sort by the event's own timestamp or sequence number and hold events that arrive too early. Mostly, the better answer is to make each handler not care about order.
• Missed webhooks are recoverable. Because the provider retries for days and keeps an event log, a webhook you dropped during an outage isn't lost — providers offer manual replay / redelivery from their dashboard, and an API to list past events. But replay is only safe if your endpoint is idempotent (Guard 2); otherwise re-sending a day of events double-charges everyone. The guards aren't independent — they hold each other up.

• "A webhook is just another API endpoint." It's an endpoint someone else calls, unprompted, from another company, over the open internet. You didn't initiate it, you can't retry it, anyone can forge it, and you'll get duplicates — every trust and reliability assumption flips.
• "It's HTTPS, so it's secure — I don't need signatures." HTTPS encrypts the connection; it does not prove the sender is the real provider. Your URL is public. Without HMAC verification, an attacker POSTs payment.succeeded and you ship free goods. Signatures are mandatory.
• "Verify the signature against the JSON object I parsed." Verify against the raw body bytes. Parsing and re-serializing changes the bytes and breaks the hash on valid events. Read raw → verify → then parse.
• "Providers deliver each event exactly once." Never — exactly-once is impossible over an unreliable network. Every major provider is at-least-once; duplicates are guaranteed, and idempotency (dedupe by event_id) is the load-bearing wall.
• "Just do the work in the handler and return when it's done." Slow inline work blows the provider's timeout → it thinks you failed → retries → you do the work again. Ack 2xx in milliseconds, process async on a queue.
• "If I miss a webhook, that data is gone." Providers retry for days and offer manual replay plus an event log — a dropped webhook is recoverable, if your endpoint is idempotent so the replay is safe.

The whole security guard fits in a dozen lines of dependency-free Node, and the most convincing way to believe in signatures is to forge a webhook and watch it get rejected. Save as webhook.js:

// webhook.js — sign like a provider, verify like a receiver. No dependencies.
const crypto = require('crypto');
const SECRET = 'whsec_test_123';                 // shared with the provider at setup

// PROVIDER side: sign  v1 = HMAC-SHA256(secret, `${t}.${rawBody}`)
function sign(rawBody, t) {
  const v1 = crypto.createHmac('sha256', SECRET).update(`${t}.${rawBody}`).digest('hex');
  return `t=${t},v1=${v1}`;
}

// RECEIVER side: recompute over the RAW body, constant-time compare, reject stale timestamps
function verify(rawBody, header, now, toleranceS = 300) {
  const p = Object.fromEntries(header.split(',').map(kv => kv.split('=')));
  if (Math.abs(now - Number(p.t)) > toleranceS) return 'REJECTED: stale timestamp (replay)';
  const expected = crypto.createHmac('sha256', SECRET).update(`${p.t}.${rawBody}`).digest('hex');
  const a = Buffer.from(expected), b = Buffer.from(p.v1 || '');
  const ok = a.length === b.length && crypto.timingSafeEqual(a, b);
  return ok ? 'OK: verified' : 'REJECTED: signature mismatch (forged/tampered)';
}

const t = Math.floor(Date.now() / 1000);
const body = JSON.stringify({ id: 'evt_1', type: 'payment.succeeded', amount: 4900 });
const header = sign(body, t);

console.log('genuine  :', verify(body, header, t + 10));
console.log('tampered :', verify(JSON.stringify({ id: 'evt_1', type: 'payment.succeeded', amount: 999999 }), header, t + 10));
console.log('forged   :', verify(body, `t=${t},v1=${'deadbeef'.repeat(8)}`, t + 10));
console.log('replayed :', verify(body, header, t + 9999));

Run node webhook.js. You should see, exactly:

genuine  : OK: verified
tampered : REJECTED: signature mismatch (forged/tampered)
forged   : REJECTED: signature mismatch (forged/tampered)
replayed : REJECTED: stale timestamp (replay)

Sit with tampered: the attacker took a genuine, correctly-signed event and changed the amount from 4900 to 999999 — and the signature instantly fails, because they'd need the secret to re-sign the new bytes. That's the entire security model of webhooks, running in your terminal. Then, for the real thing: register a test webhook on any Stripe/GitHub account, use their CLI (stripe listen --forward-to localhost) to forward live events to your machine, and watch real signed deliveries — including the retries when you make your endpoint return a 500 on purpose.

• A webhook reverses the arrow: you register a URL, and when an event happens inside another company, they make a plain HTTP POST to you. A reverse API — "we'll call you" — for news that's born on someone else's servers.
• The POST is trivial; receiving it safely is the lesson, and it's three guards. ① Verify it's them — HMAC-SHA256 over the raw body with a shared secret, constant-time compare, reject stale timestamps (HTTPS is not enough; your URL is public). ② Survive the duplicate — delivery is at-least-once, never exactly-once, so dedupe by event_id (this is §6 idempotency as the load-bearing wall). ③ Answer fast — ack 2xx in milliseconds and push the real work to a queue, or slow handlers blow the timeout and trigger retries.
• The provider's machinery: retries on exponential backoff + jitter (1s, 2s, 4s…) for up to days, then disables the endpoint and dead-letters; ordering is not guaranteed; missed events are recoverable via replay — if you're idempotent.
• The guards hold each other up: replay is only safe because you dedupe; fast-ack only works because a queue absorbs the work.

That last point is the doorway to the rest of the section. To receive a webhook well, you had to reach for a queue — a place to drop work so a worker can handle it later, at its own pace, decoupled from whoever produced it. That decoupling — do it now, or trust it'll happen later — is the biggest idea left in §7, and it's next: Sync vs Async: The Decoupling Decision. After that, the queues themselves. The synchronous world you've lived in since §6 is about to come apart on purpose.