---
title: "Webhooks Explained: How They Work, How to Secure Them, How to Scale Them"
description: "What a webhook is, how it compares with API polling and WebSockets, how Stripe, GitHub, Shopify and Slack sign and retry their deliveries, how to verify HMAC signatures without the classic raw-body bug, and how to receive and send webhooks reliably at scale."
author: Aleksei Aleinikov
date: 2026-10-13
lang: en
tags: [webhooks, webhook-security, hmac, idempotency, event-driven-architecture, api-design, standard-webhooks]
canonical: https://www.alekseialeinikov.com/en/blog/topics/architecture/webhooks-explained-signatures-retries-idempotency
source: alekseialeinikov.com
---

# Webhooks Explained: How They Work, How to Secure Them, How to Scale Them

Every integration you have ever built eventually asks the same question: *how do I find out that something changed on the other side?* You can ask every minute — or you can give the other side a URL and let it tell you. That second option is a webhook, and it is how Stripe tells you a payment succeeded, GitHub tells your CI that someone pushed, and Shopify tells your warehouse that an order exists.

Webhooks look trivial: an HTTP POST with some JSON. That is exactly why so many implementations break in production. The request can be forged, it can arrive twice, it can arrive out of order, it can arrive three days late, and if your handler takes four seconds to answer, some providers will send it again while you are still processing the first one. This guide covers the whole picture — what webhooks are, how the big providers actually deliver them, and what it takes to receive *and* send them without losing events or trusting the wrong caller.

![Webhooks explained: a signed HTTP POST, retries with growing gaps and an ID that makes duplicates harmless.](https://www.alekseialeinikov.com/blog/webhooks-explained.webp)

## What Is a Webhook?

A **webhook** is an HTTP request that a service sends to a URL you registered, when an event you subscribed to happens. Nothing more. The Standard Webhooks specification calls them *"a common name for HTTP callbacks"* and describes them as a sort of **reverse API**: with an API you call the service; with a webhook the service calls you.

A typical delivery looks like this:

```http
POST /webhooks/billing HTTP/1.1
Host: api.example.com
Content-Type: application/json
webhook-id: msg_2KWPBgLlAfxdpx2AI54pPJ85f4W
webhook-timestamp: 1674087231
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=

{"type":"invoice.paid","timestamp":"2022-11-03T20:26:10.344522Z","data":{"id":"inv_123"}}
```

Three parts matter:

- **The webhook URL** — the endpoint you registered with the provider, for example `https://api.example.com/webhooks/billing`. In production it must be public HTTPS.
- **The payload** — a JSON body with the event type (`invoice.paid`), when it happened, and the data.
- **The headers** — a unique ID for this message, a timestamp, and a signature that proves the provider sent it.

Your endpoint answers with a `2xx` status code to say *received*. Anything else — a `500`, a timeout, even a redirect — counts as a failed delivery, and the provider will usually try again.

One naming trap before we go further: Kubernetes also has **admission webhooks**, which the API server calls to validate or mutate objects before they are stored. They use the same HTTP-callback idea but solve a different problem — see [OPA vs Kyverno](https://www.alekseialeinikov.com/en/blog/topics/security/opa-vs-kyverno-2026-kubernetes-policy-engine) for those. This article is about event webhooks between services.

## How to Set Up a Webhook in Four Steps

Whatever the provider, receiving your first webhook follows the same four steps:

1. **Create an endpoint** in your application that accepts `POST` requests and answers `2xx` — for example `/webhooks/billing`. Locally, expose it through a tunnel (see [testing](#testing-webhooks-locally) below).
2. **Register the URL with the provider** and pick only the event types you need. Stripe and GitHub both advise against subscribing to everything: every extra event is load on your endpoint.
3. **Store the signing secret** the provider shows you — Stripe's starts with `whsec_` — in a secret manager, not in code.
4. **Send a test event** and check both sides: GitHub sends a `ping` event and can redeliver past deliveries, Stripe has `stripe trigger`, and every provider shows a delivery log with the status code your endpoint returned.

That is enough for a demo. The rest of this guide is about what makes it survive production.

## Webhook vs API Polling vs WebSockets

Webhooks exist because the alternative — polling — wastes almost every request. If an order changes twice a day and you poll every minute, 1,438 of your 1,440 daily requests return "nothing new", and you still learn about each change up to a minute late.

![Polling vs webhooks vs WebSockets: polling asks repeatedly and mostly hears nothing new, a webhook sends one POST per event, a WebSocket keeps a two-way connection open.](https://www.alekseialeinikov.com/blog/webhooks-explained-polling-vs-webhook.webp "Polling pays for every question. A webhook pays only for the answers.")

| | API polling | Webhook | WebSocket / SSE |
|---|---|---|---|
| Who starts the request | You, on a schedule | The provider, per event | Client opens once, server pushes |
| Latency | Up to one polling interval | Seconds | Milliseconds |
| Wasted requests | Most of them | None | None, but a connection per client |
| What you must run | A scheduler | A public HTTPS endpoint | A stateful connection server |
| Missed events | You catch up on the next poll | Lost unless retried or reconciled | Lost while disconnected |
| Typical use | Small volumes, no webhook offered | Backend-to-backend integration | Live UIs: chat, dashboards, presence |

The honest summary: **webhooks are the default for server-to-server integration**, WebSockets or server-sent events are for pushing updates to *clients that are online right now*, and polling is still the right tool when the provider offers nothing else — or as a **reconciliation job** that catches whatever webhooks missed. Most robust integrations combine webhooks with a periodic reconciliation, and Shopify's own documentation recommends exactly that.

## How a Webhook Delivery Actually Works

### Payloads: thin or full

Providers make a design choice about what to put in the body:

- A **full** (or *snapshot*) payload carries the whole object as it was when the event happened.
- A **thin** payload carries only identifiers — `{"id": "inv_123"}` — and you call the API to fetch the current state.

Thin payloads are smaller, never stale, and make data access auditable, because the receiver has to ask for what it needs. Full payloads save an API call. Stripe now offers both and recommends thin events for new integrations; the Standard Webhooks spec discusses both and recommends keeping payloads small, usually under 20 KB, and linking to large data instead of embedding it.

### Status codes and timeouts

A `2xx` means *delivered*. Everything else is a failure, and the Standard Webhooks spec suggests how senders should read it:

- `3xx` — failure. Don't follow redirects; the consumer should update the URL.
- `410 Gone` — the endpoint no longer wants these webhooks; disable it.
- `429`, `502`, `504` — the receiver is overloaded; slow down, and honour `Retry-After` if present.

Timeouts are where receivers get hurt, because they are short and they differ:

| Provider | Your endpoint must answer within |
|---|---|
| Slack (Events API) | 3 seconds |
| Shopify | 5 seconds (and connect within 1 second) |
| GitHub | 10 seconds |
| Stripe | "quickly", before any complex logic |
| Standard Webhooks (recommendation for senders) | a timeout of 15–30 seconds |

### At-least-once, not exactly-once

Every provider in this article delivers **at least once**. A retry happens whenever the provider doesn't see your `2xx` — including when your handler did the work but the response was lost on the way back. So the same event *will* reach you twice sometimes. Stripe also states plainly that it doesn't guarantee events arrive in the order they were generated. That is why the receiving side of this article is mostly about **idempotency** — the same idea as for [API retries with idempotency keys](https://www.alekseialeinikov.com/en/blog/topics/architecture/idempotency-in-practice-api-retries-2026), applied to inbound events.

## How Stripe, GitHub, Shopify and Slack Do It

There is no single webhook protocol, which is why every provider's docs look slightly different. Here is how the four most-integrated providers and the Standard Webhooks spec compare:

| | Stripe | GitHub | Shopify | Slack (Events API) | Standard Webhooks |
|---|---|---|---|---|---|
| Signature header | `Stripe-Signature: t=…,v1=…` | `X-Hub-Signature-256: sha256=…` | `X-Shopify-Hmac-SHA256` | `X-Slack-Signature: v0=…` | `webhook-signature: v1,…` |
| Algorithm, encoding | HMAC-SHA256, hex | HMAC-SHA256, hex | HMAC-SHA256, base64 | HMAC-SHA256, hex | HMAC-SHA256 base64, or Ed25519 (`v1a`) |
| What is signed | `timestamp.body` | body | body | `v0:timestamp:body` | `id.timestamp.body` |
| Replay protection | Timestamp; libraries default to 5 min | Not in the signature; track `X-GitHub-Delivery` | Not in the signature; track the webhook ID | Timestamp; reject after 5 min | Timestamp and message ID |
| Automatic retries | Up to 3 days, exponential backoff (live mode) | **None** — redeliver manually or via the REST API | 8 times over 4 hours, then the subscription is removed* | 3 times: immediately, after 1 min, after 5 min | Example schedule spanning about 3 days |
| Deduplication key | Event `id` | `X-GitHub-Delivery` | `X-Shopify-Webhook-Id` | `event_id` | `webhook-id` |

\* Shopify removes subscriptions created through the Admin API after eight consecutive failures and emails the app's emergency contact. Slack temporarily disables event subscriptions when more than 95% of deliveries fail within 60 minutes (apps receiving fewer than 1,000 events an hour are exempt) and offers an opt-in *Delayed Events* mode that keeps retrying hourly for 24 hours.

What this table means in practice:

- **GitHub will not save you.** If your endpoint is down, those deliveries stay failed until you redeliver them. GitHub's own docs suggest a scheduled script that lists failed deliveries through the REST API and redelivers them.
- **Signing the timestamp is not universal.** GitHub and Shopify sign only the body, so their signatures alone don't stop a replay. You need to track the delivery ID.
- **Encodings differ.** Shopify's signature is base64, Stripe's, GitHub's and Slack's are hex. Comparing a base64 digest with a hex one is a classic "the signature never matches" afternoon.

## Webhook Security: Verifying Signatures

Your webhook URL is public. Anyone who finds it can POST to it. Stripe's docs put the risk plainly: without verification, an attacker could send fake events to trigger actions like fulfilling orders or granting account access. The defence is a **shared-secret HMAC signature**, and it only works if every step is done right.

![Webhook signature verification steps: read the raw body, rebuild the signed content from id, timestamp and body, compute HMAC-SHA256 with the secret, compare in constant time, check the timestamp window, then deduplicate by ID.](https://www.alekseialeinikov.com/blog/webhooks-explained-signature.webp "Six checks, in this order. Skipping any one of them reopens a hole the others can't close.")

1. **Read the raw body.** Verify the exact bytes the provider sent, before any JSON parsing.
2. **Rebuild the signed content** the way the provider defines it — body only (GitHub, Shopify), `timestamp.body` (Stripe), `v0:timestamp:body` (Slack), `id.timestamp.body` (Standard Webhooks).
3. **Compute HMAC-SHA256** with the endpoint's secret.
4. **Compare in constant time** — `hmac.compare_digest` in Python, `crypto.timingSafeEqual` in Node.js. GitHub's docs say it directly: never use a plain `==`.
5. **Check the timestamp** against your clock. Five minutes is the common tolerance; Stripe warns that a tolerance of `0` disables the check entirely rather than making it strict.
6. **Deduplicate by ID** — covered in the next section.

### The raw-body bug

The single most common reason verification fails is that the handler verifies a body that has already been **parsed and re-serialized**. Express's `express.json()`, Flask's `request.json` and most API frameworks turn the body into an object before your code runs. Serializing it back changes whitespace or key order, and a cryptographic signature changes with a single byte. Both Stripe and Shopify call this out in their troubleshooting guides; Stripe's fix for Express is to register the webhook route *before* the JSON middleware.

### A verification function you can read

Here is a complete verifier for Standard Webhooks signatures in Python — no library, so every step is visible:

```python title="verify.py"
import base64
import hashlib
import hmac
import time

TOLERANCE_SECONDS = 5 * 60


def verify(secret: str, headers, body: bytes) -> None:
    """Raise ValueError unless this is a fresh, correctly signed delivery."""
    msg_id = headers["webhook-id"]
    timestamp = headers["webhook-timestamp"]

    key = base64.b64decode(secret.removeprefix("whsec_"))
    signed_content = f"{msg_id}.{timestamp}.".encode() + body
    expected = base64.b64encode(
        hmac.new(key, signed_content, hashlib.sha256).digest()
    ).decode()

    # Several signatures are sent while a secret is being rotated.
    for signature in headers["webhook-signature"].split(" "):
        version, _, value = signature.partition(",")
        if version == "v1" and hmac.compare_digest(value, expected):
            break
    else:
        raise ValueError("no matching signature")

    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        raise ValueError("timestamp outside tolerance")
```

Two details are easy to miss. The signature header is a **list**: during secret rotation the sender signs with both the old and the new key, so verification passes for whichever one you have. And the secret is base64 after the `whsec_` prefix — you decode it to get the key bytes.

In production, prefer the provider's official SDK or the Standard Webhooks reference library for your language — they exist for Python, JavaScript/TypeScript, Java/Kotlin, Go, Rust, Ruby, PHP, C# and Elixir. Writing it yourself once is still the best way to understand what the library protects you from.

> [!TIP]
> GitHub publishes a test vector you can put straight into a unit test. With the secret `It's a Secret to Everybody` and the payload `Hello, World!`, the HMAC-SHA256 hex digest must be `757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17`.

### Beyond the signature

- **IP allowlists.** Stripe publishes its webhook IP addresses and GitHub exposes its ranges through the `GET /meta` API endpoint. Useful as an extra layer; they change over time, so automate the update.
- **Mutual TLS.** Slack can authenticate itself with a client certificate on top of signing; your TLS terminator checks it.
- **Secret rotation.** Stripe lets you keep the previous secret valid for up to 24 hours while you deploy the new one, and sends one signature per active secret. Plan rotation before a leak forces it.
- **Exempt the route from CSRF protection**, not from authentication. Frameworks like Rails and Django reject POSTs without a CSRF token; the signature is your authentication instead.
- **Never put credentials in the webhook URL.** A `?token=` in the payload URL ends up in proxy and access logs. GitHub's docs warn against including API keys or other credentials in the URL and point to the webhook secret instead.

## Receiving Webhooks Reliably: Acknowledge Fast, Process Later

A handler that verifies the request, updates three tables, calls two APIs and then returns `200` works fine in development. In production, the first slow database query pushes it past Slack's 3 seconds or Shopify's 5, the provider retries, and now two copies of the same event are running at the same time.

The pattern every provider recommends is the same: **verify, enqueue, return 2xx, and process in a worker.**

```python title="app.py"
from fastapi import FastAPI, Request, Response

app = FastAPI()


@app.post("/webhooks/billing")
async def billing_webhook(request: Request) -> Response:
    body = await request.body()  # raw bytes, before any JSON parsing
    try:
        verify(WEBHOOK_SECRET, request.headers, body)
    except (KeyError, ValueError):
        return Response(status_code=401)

    await queue.publish(body, webhook_id=request.headers["webhook-id"])
    return Response(status_code=204)
```

The handler now does almost nothing that can be slow, so it answers in milliseconds even during a spike — and spikes are real: Stripe's docs point out that the start of the month, when all subscriptions renew, can overwhelm endpoints that process synchronously.

### Make the worker idempotent

The worker has to assume every event may arrive twice. Store the IDs you have processed with a uniqueness constraint, and do it in the **same transaction** as the side effect:

```sql
INSERT INTO processed_webhooks (webhook_id, processed_at)
VALUES ($1, now())
ON CONFLICT (webhook_id) DO NOTHING
RETURNING webhook_id;
-- No row returned: a duplicate. Commit nothing else and acknowledge.
```

If the insert and the business update commit together, a crash in between can't leave you with "processed but not recorded" or the reverse. Use the provider's stable ID — Stripe's event `id`, `X-GitHub-Delivery`, `X-Shopify-Webhook-Id`, Slack's `event_id` or `webhook-id`. Watch one subtlety: Shopify sends a separate delivery per subscription, each with its own `X-Shopify-Webhook-Id` but the same `X-Shopify-Event-Id`, and Stripe notes that in some cases two separate Event objects are generated for the same change, which you can identify by the object ID plus the event type.

> [!KEY]
> A webhook handler is correct when receiving the same event twice, late or out of order produces the same result as receiving it once, on time.

### Don't trust the order

Because events can arrive out of order — `invoice.paid` before `invoice.created` — don't let an older event overwrite newer state. Either compare a version or `updated` field before writing, or treat the webhook as a signal and **fetch the current object from the API**. Stripe recommends the second approach and warns against using the `created` timestamp to order events, since distinct events can share the same second.

### Plan for the outage you will have

Your endpoint *will* be down at some point. What happens next depends on the provider: Stripe keeps retrying for three days, Shopify gives up after four hours, and GitHub never retried in the first place. So build two things before you need them:

- **A reconciliation job** that periodically fetches recent changes from the provider's API and feeds them into the same idempotent worker.
- **A recovery runbook**: Stripe's List Events API can return events that failed delivery (`delivery_success=false`) for the last 30 days; GitHub's REST API lists and redelivers failed deliveries.

### Watch the numbers that warn you first

Most webhook incidents are visible in the metrics before anyone files a ticket:

- **Failure rate** of your responses. Shopify's own guidance treats a failed delivery rate above 0.5% as higher than average.
- **Response time**, as a high percentile rather than an average. Shopify charts the 90th percentile and flags responses between four and five seconds — one slow query away from its timeout.
- **Signature failures.** A sudden spike means either an attack or a secret that was rotated on one side only. Both deserve a page.
- **Queue depth and dead-letter count** on the worker side. A growing backlog is an outage that hasn't happened yet.

## Sending Webhooks: What Providers Owe Their Consumers

If your product *sends* webhooks, you are on the hard side. The receiver needs a few good habits; the sender needs infrastructure.

![Webhook delivery pipeline: the event is written to an outbox in the same transaction, a dispatcher signs it and sends it through an egress proxy, failures retry with exponential backoff and jitter, and exhausted messages go to a dead-letter queue while the receiver verifies, acknowledges and queues.](https://www.alekseialeinikov.com/blog/webhooks-explained-delivery-pipeline.webp "Reliable delivery is a pipeline on both sides: an outbox and retries on the sender, a queue and deduplication on the receiver.")

### 1. Never lose the event: transactional outbox

If you commit the order and then call the webhook dispatcher, a crash between the two loses the event. Write the event to an **outbox table in the same transaction** as the business change, and let a separate process publish it. It is the same pattern that makes [sending a million notifications](https://www.alekseialeinikov.com/en/blog/topics/architecture/send-one-million-notifications-without-falling-over-2026) survivable, and it turns "we might have sent it" into "it's in the table until delivered".

### 2. Retry with exponential backoff and jitter

Failures are normal. The Standard Webhooks spec recommends a schedule spanning several days with exponential backoff and **random jitter**, so that thousands of retries don't hit a recovering endpoint at the same instant. Its example schedule is: immediately, then after 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, 14 hours, 20 hours and 24 hours. The classic implementation is "full jitter", described by Marc Brooker on the AWS Architecture Blog:

```python
delay = min(max_delay, base_delay * 2**attempt)
sleep_for = random.uniform(0, delay)
```

Generate a **fresh timestamp and signature for every attempt** — Stripe does this — so a legitimate retry never fails the receiver's replay window.

### 3. Dead-letter, disable, and tell a human

After the last retry, move the message to a **dead-letter queue** rather than dropping it, and give the customer a way to replay it. If an endpoint keeps failing for a long time, disable it and notify the owner through another channel — Shopify emails the app's emergency developer contact, and Slack emails the app owner. Treat `410 Gone` as an explicit request to stop.

### 4. Protect your platform from your customers' endpoints

Every endpoint you call is a URL a customer typed in. That makes your dispatcher a perfect **server-side request forgery (SSRF)** tool unless you lock it down — an attacker registers `http://169.254.169.254/` or an internal service address and your system calls it for them. The Standard Webhooks spec recommends two controls together: send all webhook traffic through an **egress proxy that filters internal IP addresses** (Stripe's open-source Smokescreen is one), and run the dispatcher in a **subnet that can't reach internal services**. Resolve DNS and check the resulting IP at connection time, not only when the URL is saved, or a DNS record changed after validation gets around the check.

Also limit each endpoint: a timeout (15–30 seconds is the spec's recommendation), a per-endpoint concurrency cap, and back-off on `429` and `5xx` — the same principles as [rate limiting your own API](https://www.alekseialeinikov.com/en/blog/topics/architecture/rate-limiting-in-practice-algorithms-headers-2026), turned outward. One slow customer endpoint must not stall deliveries to everyone else.

### 5. Give consumers what they need to trust you

- **One secret per endpoint**, never shared across customers, with zero-downtime rotation by signing with both keys for a while.
- **Event type filtering**, so consumers only receive what they subscribed to.
- **Static source IPs** if your customers sit behind corporate firewalls.
- **Delivery logs and manual replay** — the spec calls visibility into failures and replay "immensely important".
- **Versioned payloads**: a given event type should always have the same schema.

## Standard Webhooks: One Format Instead of Fifty

Every row of the provider table above is a little different, and every consumer has to relearn verification for each one. **Standard Webhooks** is the industry's attempt to fix that: an open specification (version 1.0.0, Apache 2.0) written by a steering committee with engineers from Svix, Zapier, Twilio, Kong, ngrok, Supabase, Mux and Lob.

It standardises what this article has been describing:

- Three headers: `webhook-id`, `webhook-timestamp` (Unix seconds) and `webhook-signature`.
- A signature over `id.timestamp.body` — HMAC-SHA256 (`v1`) with a `whsec_`-prefixed secret, or Ed25519 (`v1a`) for asymmetric signing, where consumers verify with a public key and the signing key never leaves the sender. The spec prefers asymmetric signatures where you can afford them.
- A payload shape with `type`, `timestamp` and `data`.
- Operational rules: retries with backoff and jitter, status-code handling, a 15–30 second timeout, HTTPS, and SSRF protection.

OpenAI, Anthropic, Google Gemini, Twilio, Kong, PagerDuty, Clerk and Svix are among the implementations listed as compatible. If you are designing a new webhook system, there is little reason not to adopt it — it can be added next to an existing scheme by sending the new headers alongside the old ones, and your consumers get reference libraries in nine languages for free.

### What about CloudEvents?

**CloudEvents** is often mentioned in the same breath, but it answers a different question. It is a CNCF specification — graduated in January 2024 — for *describing* an event in a common format, with bindings for HTTP, Kafka, AMQP, MQTT and NATS and its own web hook specification. Google Cloud's Eventarc delivers events in the CloudEvents format. Standard Webhooks, by contrast, is about *delivering* events over HTTP safely: signatures, replay protection, retries and SSRF. Use CloudEvents when the same events travel across several transports and platforms; use Standard Webhooks when you send HTTP callbacks to other companies' endpoints. They don't exclude each other.

## Webhooks at Scale on Google Cloud

The patterns above map directly onto managed services. On Google Cloud, the setup I would build for both directions looks like this:

**Receiving:**

- A **Cloud Run** service that verifies the signature, publishes the raw event to **Pub/Sub**, and returns `2xx`. Cloud Run scales to the burst; Pub/Sub absorbs it.
- Workers on a Pub/Sub subscription with a **dead-letter topic** (configurable after 5 to 100 delivery attempts) and the deduplication table in Cloud SQL, AlloyDB or Spanner.
- Signing secrets in **Secret Manager**, and **Cloud Armor** in front of the endpoint if the provider publishes source IPs.
- Some providers skip HTTP entirely: Shopify can deliver webhooks straight to a Pub/Sub topic, and for Google Cloud's own events, [Eventarc](https://www.alekseialeinikov.com/en/blog/topics/architecture/pubsub-vs-eventarc-2026-event-driven-gcp-without-spaghetti) routes them without any webhook endpoint at all.

**Sending:**

- The **outbox** in your database, relayed into Pub/Sub.
- **Cloud Tasks** for the actual HTTP calls: each queue has its own dispatch rate, concurrency limit and retry policy with exponential backoff, which is most of a webhook dispatcher out of the box.
- The dispatcher in a dedicated VPC with **Cloud NAT on reserved static IPs** — that gives consumers fixed source addresses to allowlist — and egress firewall rules that deny private ranges and the metadata server.

## Testing Webhooks Locally

Providers can't reach `localhost`, which makes webhooks awkward to develop. The usual tools, from quickest to most realistic:

- **webhook.site** — a throwaway public URL that shows every request it receives. Perfect for seeing exactly what a provider sends before you write any code.
- **A tunnel** — **ngrok** or **cloudflared** gives your local port a public HTTPS URL. Stripe's docs suggest ngrok for exactly this.
- **Provider CLIs** — `stripe listen --forward-to localhost:4242/webhook` forwards real test events and prints a signing secret, and `stripe trigger payment_intent.succeeded` fires one on demand.
- **smee.io** — the relay GitHub's own testing guide uses to forward webhook deliveries to a local server.
- **Standard Webhooks tools** — the project offers web tools to sign, verify and simulate messages.

Then turn it into tests: save real payloads as fixtures and replay them against your handler, including **a wrong signature, an expired timestamp, a duplicate delivery and two events in reverse order**. If those four tests pass, your handler survives most of what production will do to it.

## Build or Buy?

You don't have to build all of this. Managed and open-source options cover both directions:

- **Svix** — webhook *sending* as a service, with an MIT-licensed server you can self-host that blocks dispatches to internal IP addresses by default; its CEO is on the Standard Webhooks committee.
- **Hookdeck** — an inbound webhook gateway that queues and retries deliveries in front of your endpoint; GitHub's docs mention it as one way to respond on time.
- **Convoy** — a webhooks gateway for sending and receiving, with retries, rate limiting, static IPs and circuit breaking. Its source is public under the Elastic License 2.0, which is source-available rather than an OSI open-source license — check the terms before you build a product on it.

Build it yourself when webhooks are a few event types on infrastructure you already run — Pub/Sub plus Cloud Tasks gets you far. Buy when webhooks *are* a product feature for your customers: endpoint management UIs, per-customer logs, replay and analytics are a lot of work that isn't your core business.

## Production Checklist

**Receiving webhooks:**

- HTTPS endpoint, CSRF exemption for the route only
- Signature verified over the **raw** body, constant-time comparison, correct encoding
- Timestamp tolerance of a few minutes, server clock synced with NTP
- Verify, enqueue, return `2xx` — well inside the provider's timeout
- Idempotent worker: unique constraint on the event ID, in the same transaction as the side effect
- Out-of-order safe: version checks or fetch-current-state
- Reconciliation job and a documented recovery runbook
- Secrets in a secret manager, rotation tested

**Sending webhooks:**

- Transactional outbox, so no event is lost
- Retries with exponential backoff and jitter over days, a fresh signature per attempt
- Dead-letter queue, endpoint auto-disable and owner notification; `410` stops delivery
- SSRF protection: egress proxy, isolated network, IP check at connect time
- Per-endpoint timeout, concurrency cap and back-off on `429`/`5xx`
- One secret per endpoint, zero-downtime rotation, static source IPs if customers need them
- Delivery logs, manual replay, event type filtering, versioned schemas
- Standard Webhooks headers and signature scheme

## The Bottom Line

A webhook is the simplest integration primitive there is — one HTTP POST per event — and that simplicity is precisely what hides its failure modes. The request might not be from who it says, might come twice, might come late, and might come before the event it depends on.

Every rule in this guide follows from one idea: **delivery is at-least-once and the network is hostile, so make the receiver verify everything and tolerate repetition, and make the sender never forget and never trust a URL blindly.** Get those two halves right and webhooks stop being the flakiest part of your integration and become the most boring one — which is exactly what you want.

> Retries on the other side of a webhook are the same problem as retries on your own API — [idempotency keys](https://www.alekseialeinikov.com/en/blog/topics/architecture/idempotency-in-practice-api-retries-2026) explain the storage side in depth.