Webhooks

The Problem

When events happen in your system (payment completed, order shipped, user signed up), external systems need to know. Webhooks push HTTP POST requests to customer-configured URLs when events occur. The challenge: the internet is unreliable. Customer endpoints go down, respond slowly, or return errors. You must ensure delivery with retries, prevent duplicate processing, secure payloads with signatures, and manage thousands of webhook endpoints with wildly varying reliability — all without letting slow endpoints back up your event pipeline.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

HMAC Signature Verification

Webhook payloads must be verified by the receiver to ensure authenticity and integrity.

Signing process (your side):

1. Generate a unique secret per endpoint (32+ random bytes, base64 encoded)
2. For each delivery, compute: signature = HMAC-SHA256(secret, timestamp + "." + payload_body)
3. Send headers: X-Webhook-Signature: t=1707100000,v1=abc123... and X-Webhook-Timestamp: 1707100000

Verification process (customer side):

1. Extract the timestamp and signature from the header
2. Reject if timestamp is older than 5 minutes (replay protection)
3. Compute expected signature: HMAC-SHA256(secret, timestamp + "." + raw_body)
4. Compare with constant-time comparison to prevent timing attacks
5. If match, process the webhook; if not, return 401

Including the timestamp in the signature prevents replay attacks — an attacker can’t re-send a captured webhook because the timestamp check will fail.

Secret rotation: Support multiple active secrets per endpoint. When rotating, sign with the new secret and include signatures for both old and new secrets in the header. Customers verify against both during the rotation window.

Retry Strategy and Circuit Breaking

Not all failures are equal. The retry strategy should adapt to the failure type.

Retryable failures:

• Network timeout → retry with exponential backoff
• 5xx server error → retry (server is temporarily down)
• Connection refused → retry (server restarting)

Non-retryable failures:

• 4xx client error (except 429) → don’t retry, the endpoint is misconfigured
• Invalid URL → don’t retry, notify the customer
• TLS certificate error → don’t retry, security issue
• 429 Too Many Requests → retry, but respect the Retry-After header

Retry schedule: 1min → 5min → 30min → 2hr → 8hr → 24hr (6 attempts over ~34 hours). After final failure, move to DLQ.

Circuit breaker per endpoint: If an endpoint fails 10 consecutive deliveries, “open” the circuit — pause all deliveries to that endpoint. Periodically send a “health check” delivery. If it succeeds, close the circuit and resume. This prevents wasting resources on consistently failing endpoints.

Auto-disable: After 72 hours of continuous failures, automatically disable the endpoint and notify the customer via email. Require them to re-enable manually after fixing the issue.

Ordering and Idempotency

Delivery ordering: Webhooks may arrive out of order due to retries and parallel dispatch. If order matters (e.g., order.created must arrive before order.shipped), options:

• Include a sequence number in the payload. Receivers buffer and reorder.
• Use per-endpoint FIFO queues (SQS FIFO, or single-worker-per-endpoint). Slower but ordered.
• Accept out-of-order delivery and include enough state in each event for the receiver to reconcile.

Idempotency: Receivers must handle duplicate deliveries. Include a unique event_id in every webhook. Receivers store processed event IDs and skip duplicates. Your documentation should clearly state that webhooks have at-least-once delivery semantics.

Event payload design:

• Include the full resource state (“fat events”) so receivers don’t need to call back to your API
• OR include only the event type and resource ID (“thin events”) and let receivers fetch the current state. Thin events are simpler but require an API call per webhook.
• Best practice: include the changed fields and the event type. Receivers can process directly or fetch if they need more context.

Versioning: Include an API version in the webhook payload or URL path. When you change the payload schema, maintain backward compatibility or introduce a new version. Customers choose which version they receive.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Webhooks

You are an expert software architect and teacher. I want to understand webhooks as a practical system design capability. (Reliable event delivery to external systems with retry, HMAC verification, and monitoring).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

When events happen in your system (payment completed, order shipped, user signed up), external systems need to know. Webhooks push HTTP POST requests to customer-configured URLs when events occur. The challenge: the internet is unreliable. Customer endpoints go down, respond slowly, or return errors. You must ensure delivery with retries, prevent duplicate processing, secure payloads with signatures, and manage thousands of webhook endpoints with wildly varying reliability — all without letting slow endpoints back up your event pipeline.

## Architectural Approaches

<ApproachList>
<Approach
  name="Synchronous Dispatch with Retry Queue"
  pros={[
    'Lowest latency for healthy endpoints — near-instant delivery',
    'Simple architecture — direct HTTP call + retry queue',
    'Easy to reason about delivery order',
  ]}
  cons={[
    'Slow endpoints block the dispatch thread',
    'Must handle timeouts carefully (5-10 second max)',
    'Synchronous dispatch can overwhelm the event producer under load',
    'Retry queue can grow unbounded if many endpoints are down',
  ]}
>
Attempt delivery immediately when the event occurs. If the endpoint is down or slow, enqueue the webhook for retry. A background worker processes the retry queue with exponential backoff. Simple and low-latency for healthy endpoints.
</Approach>
<Approach
  name="Fully Async Queue-Based Dispatch"
  pros={[
    'Event producer is completely decoupled from deliv...

[Full prompt copied to clipboard]