webhook

Starting point: The most straightforward approach is a single service that receives the HTTP request, processes the event, writes to the database, and returns 200 OK.

The request handler does everything: validate the request, execute business logic (e.g., update payment status), write to the database, and respond.

The critical flaw: The request handler has too many responsibilities — HTTP handling, business logic, database writes. If it crashes after processing but before the database write succeeds, the event is lost. Worse, if the handler is slow (heavy business logic), it blocks the HTTP response. External providers typically have aggressive timeouts (5-30 seconds). If we don't respond fast enough, the provider considers the delivery failed and retries — even though we might have already processed the event.

We need to separate concerns: accept the event quickly, then process it asynchronously.

The solution is a classic producer-consumer pattern with a message queue:

• Request Handler (producer): Receives HTTP, validates, enqueues event, returns 200 OK
• Message Queue (buffer): Durably stores events until consumed
• Queue Consumer (consumer): Pulls events, processes them, writes results to database

This separation gives us three critical properties:

Request Flow — Step by Step

Step 1 — Event arrives. Stripe sends POST /webhook with event payload and HMAC signature in headers.

Step 2 — Signature verification. The handler computes HMAC-SHA256 over the request body using the shared secret key. If the computed hash doesn't match the signature in the header, return 401 Unauthorized.

Step 3 — Idempotency check. Query the database: does an event with this event_id already exist? If yes, return 200 OK with status: "already_processed". No further action needed.

Step 4 — Persist event record. Insert a new row in webhook_events with status: 'pending'. This creates the audit trail immediately.

Step 5 — Enqueue for processing. Push the event to the message queue. The queue durably stores the message.

Step 6 — Acknowledge to provider. Return 200 OK. Total time: ~10-50 ms. The provider knows we received the event.

Step 7 — Consumer processes event. A queue consumer pulls the message, executes the business logic (e.g., update payment status, trigger email), and writes the result to the database.

Step 8 — Acknowledge to queue. Only after the database write succeeds does the consumer ACK the message. If the consumer crashes before ACK, the message becomes visible again after a visibility timeout, and another consumer retries it.

This is the key reliability guarantee: The message stays in the queue until we prove (via database write + ACK) that processing succeeded. No event loss is possible after step 6.

Each component in the pipeline can fail. The architecture must handle every failure mode without losing events.

Request Handler Failures

Before enqueue: If the handler crashes before enqueuing and returning 200 OK, the external provider never receives acknowledgment. The provider retries the delivery (typically 3-5 times with exponential backoff). Since we never saved the event, the retry is a fresh delivery — no data loss.

After enqueue, before response: The event is safely in the queue, but the provider didn't receive 200 OK. The provider retries and sends the event again. Our idempotency check catches the duplicate — the event_id already exists in the database — and we return 200 OK without reprocessing.

Message Queue Failures

Durable queues persist messages to disk, surviving process crashes. Multi-node replication (Kafka's replication.factor=3, or SQS's built-in multi-AZ) ensures that even if an entire server dies, messages are preserved on other nodes.

Queue Consumer Failures

The message queue uses a visibility timeout mechanism. When a consumer pulls a message, the message becomes invisible to other consumers for a configured period (e.g., 30 seconds). If the consumer successfully processes the event and ACKs within this window, the message is permanently deleted. If the consumer crashes, the visibility timeout expires and the message reappears — allowing another consumer to pick it up.

This is why consumer-side idempotency is critical. The same event may be delivered to multiple consumers (if the first consumer was slow or crashed). Each consumer must check the event_id before executing business logic to avoid duplicate processing.

Database Failures

Database failures are handled with standard resilience patterns:

• Write retries with exponential backoff — If the first write fails, retry after 100 ms, then 200 ms, 400 ms, etc. Most transient failures (connection timeout, deadlock) resolve within a few retries.
• Database replication with automatic failover — A standby replica promotes to primary if the primary fails. The application reconnects to the new primary within seconds.
• Consumer waits for DB recovery — If the database is down for an extended period, the consumer stops ACKing messages. Messages accumulate in the queue (which has much higher capacity than the DB). When the database recovers, consumers drain the backlog.

Component Ownership & Scaling

Dead Letter Queue (DLQ)

After a configurable number of retry attempts (e.g., 5), a message moves to the dead letter queue. This prevents a single malformed event from blocking the entire pipeline. Common DLQ scenarios:

• Malformed payload — Provider sent invalid JSON that can't be parsed
• Missing handler — Event type has no registered processor
• Persistent downstream failure — External API that the processor calls is permanently down
• Bug in consumer code — Logic error that crashes on specific event patterns

A monitoring alert fires when the DLQ receives messages. Engineers investigate, fix the root cause, and replay the events from the DLQ back into the main queue.

How do we secure the webhook endpoint against forged requests?

Our webhook endpoint is a publicly accessible URL. Anyone who discovers it can send fake events — spoofing payment confirmations, fabricating order updates, or flooding us with garbage data. We need multiple layers of defense.

Layer 1: HMAC Signature Verification

The webhook provider (e.g., Stripe) and our service share a secret key (configured during webhook registration). When the provider sends an event:

1. The provider computes HMAC-SHA256(secret_key, request_body) and includes the hash in the X-Webhook-Signature header
2. Our handler computes the same HMAC using the shared secret and the received body
3. If the hashes match, the request is authentic — only someone with the secret key could produce that signature

Why hmac.compare_digest instead of ==? Regular string comparison short-circuits on the first differing character. An attacker could measure response times to deduce the expected signature one character at a time (timing attack). compare_digest takes constant time regardless of where strings differ.

Why include the timestamp? Without it, an attacker who intercepts a valid request could replay it later. By requiring the timestamp to be within a window (e.g., ±5 minutes), we reject stale requests.

Layer 2: IP Allowlisting

Configure the load balancer or firewall to accept webhook requests only from known provider IP ranges. Stripe publishes their webhook IP addresses; so does GitHub and Shopify.

Limitation: Provider IPs can change. IP allowlisting is a defense-in-depth measure — not a primary authentication mechanism. Always use HMAC verification as the primary check.

Layer 3: Rate Limiting

Set rate limits per IP or per API key to prevent denial-of-service attacks:

• Normal provider traffic: ~12 events/sec average, ~70/sec peak → set limit at 200/sec per provider IP
• Abuse traffic: If any source exceeds 200/sec, return 429 Too Many Requests

Rate limiting protects against both malicious flooding and buggy providers that accidentally send duplicate events in loops.

Defense Summary

How do we handle duplicate webhook deliveries?

Duplicate events are inevitable, not exceptional. They occur from:

• Provider retries — Provider didn't receive 200 OK (network issue) and resends
• Consumer retries — Consumer crashed mid-processing; message redelivered after visibility timeout
• Intentional replay — Provider resends events after an outage recovery

Without idempotency, processing the same payment_intent.succeeded event twice charges the customer twice. Processing the same order.created event twice creates two orders.

Idempotency Key Strategy

Every webhook event has a unique identifier assigned by the provider (e.g., Stripe's evt_1MqLSbKJFk9d2k). We use this as an idempotency key:

Why check-then-insert instead of just INSERT with ON CONFLICT? The SELECT first is cheaper than catching exceptions on every request. 99% of events are new — the SELECT returns "not found" and we proceed. The UNIQUE constraint is a safety net for the rare race condition where two handlers receive the same event simultaneously.

Consumer-Side Idempotency

The handler-side check prevents duplicate enqueueing. But messages can still be delivered twice to consumers (visibility timeout expiry, queue retry). Consumers must also be idempotent:

The key line is WHERE event_id = $1 AND status = 'pending'. This is an atomic compare-and-swap. If two consumers try to process the same event simultaneously, only one succeeds in changing the status from 'pending' to 'processing'. The other gets rows_updated = 0 and exits immediately.

Queue-Level Deduplication

Some message queues offer built-in deduplication:

Queue deduplication is a defense-in-depth addition — it reduces duplicates but doesn't eliminate them (e.g., messages delivered across the deduplication window). Application-level idempotency is still required.

How do we handle events that arrive out of order?

Webhook providers send events independently. Network latency, retry timing, and provider-side batching can cause events to arrive in a different order than they occurred. For example:

• Stripe sends invoice.created at 10:00:00
• Stripe sends invoice.paid at 10:00:05
• Due to a network retry, invoice.paid arrives at our service at 10:00:06
• invoice.created arrives at 10:00:08

If our processor blindly processes events in arrival order, it would try to mark an invoice as "paid" before it exists in our database.

Strategy 1: Fetch Latest State from Source of Truth

Instead of relying on event data to update local state, fetch the current state from the provider's API when processing each event:

The WHERE invoices.updated_at < EXCLUDED.updated_at clause ensures we never overwrite newer data with older data. If invoice.paid (newer timestamp) was processed first, and invoice.created (older timestamp) arrives later, the UPDATE silently does nothing because the existing updated_at is already newer.

Strategy 2: Timestamp-Based Conflict Resolution

When you can't call the provider's API (rate limits, latency concerns), use the event's timestamp to determine ordering:

Key Takeaways

1. Never assume event order. Design processing logic that produces correct results regardless of arrival sequence.
2. Use the provider's API as the source of truth. Event payloads are notifications, not authoritative state updates.
3. Timestamp-based conflict resolution works when provider API calls are impractical. The WHERE updated_at < new_timestamp pattern prevents stale overwrites.
4. Log skipped events. When an out-of-order event is skipped, mark it as skipped_stale in the database for debugging — don't silently drop it.

How do we design a robust retry strategy with exponential backoff?

When event processing fails, we need to retry — but naively retrying immediately can overwhelm a struggling dependency. If the database is temporarily overloaded and 1,000 events fail simultaneously, immediately retrying all 1,000 creates a thundering herd that makes the situation worse.

Exponential Backoff with Jitter

The standard approach: increase the delay between retries exponentially, and add random jitter to prevent synchronized retries.

delay = min(base_delay × 2^attempt + random_jitter, max_delay)

Why Full Jitter Over Equal Jitter?

AWS's analysis shows full jitter provides the best overall throughput when many clients retry against a shared resource.

Dead Letter Queue (DLQ) Policy

After MAX_RETRIES failed attempts, the event moves to the dead letter queue. The consumer must not keep retrying — the event is likely a poison message (malformed data, unhandled event type, persistent downstream failure). Retrying forever would waste resources and potentially block the queue.

Total time before DLQ: ~15-30 seconds. Fast enough to catch transient failures (network blip, DB failover) but not so aggressive that it overwhelms recovering systems.

{
  "maxReceiveCount": 5,
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789:webhook-dlq"
}

What observability do we need for a webhook processing pipeline?

A webhook pipeline processes events from external systems — systems we don't control. When something goes wrong, we need to identify whether the issue is on our side (handler bug, DB outage) or the provider's side (malformed payload, changed API). Observability is critical because webhook failures are often silent — no user is clicking a button and seeing an error page.

Key Metrics to Track

Structured Logging for Every Event

Each event should produce a log trail that allows full reconstruction of its lifecycle:

Dashboard Layout

A webhook monitoring dashboard should answer three questions at a glance:

1. Is the system healthy? — Ingestion rate, processing latency, error rate. Green/yellow/red indicators.
2. Is there a backlog? — Queue depth, consumer lag. If the queue is growing, consumers need scaling.
3. Are there poison messages? — DLQ count, top error categories. Which event types are failing and why?

Alerting Strategy

The following topics contain open-ended architectural questions designed for staff+ conversations where you demonstrate systems thinking, trade-off analysis, and cross-cutting architectural decisions.