Webhook reliability is one of those problems that looks small until it starts costing real money. If your billing, fulfillment, CRM sync, or internal automation depends on third-party callbacks, the weak link is rarely the provider — it is your own handling of retries, duplicates, and partial failures.

For teams running revenue-critical workflows, webhook reliability is not a nice-to-have. It is the difference between a customer getting access immediately, a payment being recorded twice, or a downstream job quietly never running.

Why webhook reliability fails in real systems

The first mistake is assuming a webhook is a single event. It is not. It is a delivery attempt, and delivery attempts come with retries, duplicates, out-of-order arrival, and occasional provider bugs. A payment provider may send the same event three times. A CRM may retry after a 500 even though your handler already wrote the row. A queue worker may crash after committing to the database but before returning 200.

That is why webhook reliability starts with a mindset shift: the provider only promises at-least-once delivery. You own exactly-once side effects. Those are different problems. Treating them as the same thing is how teams end up with double invoices, duplicate entitlements, and support tickets that take hours to unwind.

The other failure mode is hidden latency. A handler that “usually” finishes in 300 ms can spike to 8 seconds when a downstream dependency slows down. If the provider times out at 5 seconds, you get a retry even though the work eventually completed. Now you have two overlapping attempts racing through the same code path. This is where a lot of webhook bugs become intermittent and expensive.

In practice, the safest design is usually:

  1. verify the signature fast,
  2. persist the raw event,
  3. acknowledge quickly,
  4. process asynchronously,
  5. make the handler idempotent.

That sequence is dull. Good. Dull is what you want when revenue is on the line. It also plays well with the rest of your backend: a transaction boundary in the database, a queue for work, and clear retry semantics. If you want a related deep dive on the queue side, our Transactional Outbox Implementation: Reliable Message Queues post is the right companion piece.

One more thing: teams often over-index on the provider’s documentation and under-index on their own failure budget. If a missed webhook costs you $500 in support and churn recovery, then the engineering time to harden the path is cheap. If it gates access to a paid product, it is not an integration detail. It is core infrastructure.

The webhook reliability contract you should enforce

Every webhook handler should define a contract. Not a vague “we accept events.” A real contract. What event types are accepted? Which ones are ignored? What payload fields are required? What is the source of truth for event identity? What is the maximum retry window you support before an event is considered stale?

This contract matters because providers evolve. Fields get renamed. New event types appear. Old ones are deprecated but not removed. If your handler is permissive in the wrong place, you will accept malformed data and ship garbage into your system. If it is too strict, you will reject legitimate events after a provider rollout. The sweet spot is strict validation at the boundary, tolerant parsing where the provider is known to add optional fields, and explicit versioning in your internal event model.

A practical pattern looks like this:

type IncomingWebhook = {
  id: string;
  type: string;
  created_at: string;
  data: Record<string, unknown>;
};

function validateWebhook(input: unknown): IncomingWebhook {
  // Parse, verify required fields, reject unknown top-level shape if needed.
  // Keep the boundary narrow.
  return input as IncomingWebhook;
}

That snippet is intentionally plain. The point is not the syntax. The point is that your internal code should never wander around with raw provider payloads. Map once at the edge. Then work with your own domain objects. That makes auditing easier, tests cleaner, and downstream refactors less dangerous.

For teams using Stripe, GitHub, Shopify, or similar providers, the event ID and delivery attempt ID are not interchangeable. Store both if they are available. The event ID is what helps you dedupe. The delivery ID can help you debug repeated attempts. When a support case comes in, the difference between those two fields can save an hour.

This is also where architecture reviews pay for themselves. Kevin has been engineering software since 1998, and the same mistake shows up in different clothes: teams let external payloads leak deep into the app. That works until the provider changes a field name or a nullable attribute starts arriving as an empty object. Then the blast radius is larger than it needed to be. If you want to see how we think about senior-level architecture judgment, start with Kevin's 28 years of senior engineering and the kind of work we ship in our Sprint, Build, or Fractional engagements.

Idempotency and deduplication for webhook reliability

If you remember one rule, make it this: every side effect must be safe to repeat. That means database writes, entitlement changes, email sends, provisioning jobs, and billing updates all need a dedupe strategy. Idempotency is not a feature you add after the handler works. It is the design constraint that makes the handler safe at all.

The simplest dedupe table is often enough:

create table webhook_events (
  provider text not null,
  event_id text not null,
  received_at timestamptz not null default now(),
  processed_at timestamptz null,
  status text not null default 'received',
  payload jsonb not null,
  primary key (provider, event_id)
);

That primary key does the heavy lifting. Insert first. If the insert fails, you already saw the event. If it succeeds, you own processing. This is better than checking first and inserting later, because race conditions live in the gap between those two steps. In a busy system, two workers can both see “not found” and both proceed. The database constraint closes that gap.

But dedupe alone is not enough. You also need idempotent business logic. For example, if a payment webhook grants access, do not “add 30 days” every time the event arrives. Instead, set the subscription state based on the canonical provider record. If the provider says the subscription is active until March 31, write that exact date. That way repeated handling converges on the same result.

There is a subtle trap here. Some side effects are naturally idempotent, some are not. Sending a welcome email is not idempotent unless you store a sent marker. Creating a support ticket is not idempotent unless the external system supports an idempotency key. Triggering a downstream job may be idempotent only if the worker checks for an existing record before acting. You need to classify each action explicitly.

When we audit this kind of flow, we usually map the path into three buckets:

  • safe to repeat: writes that converge to one state
  • safe with a key: external calls that accept an idempotency token
  • unsafe without a guard: emails, tickets, provisioning, fulfillment

If you are building around Stripe, their webhook model and event semantics are worth studying alongside our own post on Stripe Webhook Reliability: Handling Failures at Scale. The pattern generalizes, but the edge cases are different enough that the details matter.

One more practical point: do not use Redis as your primary dedupe store unless you have a very specific reason. Ephemeral caches are fine for temporary backoff state. They are a poor substitute for durable uniqueness constraints. If the event matters enough to protect, put the protection in the database.

Retries, queues, and dead letters in webhook reliability

Fast acknowledgment is not the same as fast processing. A good webhook endpoint should do as little work as possible before returning 200. Signature verification, schema validation, and durable enqueueing are usually enough. The real processing belongs in a worker that can retry independently.

This is where queue choice matters. For many teams, SQS, RabbitMQ, Sidekiq, BullMQ, or Kafka can all work. The right answer depends on the shape of your work. If you need simple retries and visibility timeouts, SQS is fine. If you need ordered streams and high event volume, Kafka may be a better fit. If you need in-process job ergonomics in Node.js, BullMQ can be a clean choice. The important thing is not the brand. It is that your queue gives you retry control and a place to isolate failures.

A worker should have bounded retries with backoff and then fail into a dead-letter queue. Not every webhook can be processed forever. If a payload is malformed, if a downstream dependency is down for an hour, or if a business rule no longer exists, endless retries only create noise. A dead-letter queue gives you a controlled place to inspect failures, replay after a fix, and measure what is actually breaking.

Here is the operational rule we use: retry transient failures, dead-letter permanent ones, and page only when the dead-letter rate crosses a threshold that threatens backlog growth. A queue with 20 dead letters in a day is a debugging problem. A queue with 20,000 is a revenue problem.

Backoff matters too. Fixed retry intervals create thundering herds. Exponential backoff with jitter smooths the load. If your webhook source retries aggressively as well, you can easily create synchronized bursts against your own database. This is especially visible when a provider outage resolves and all missed events come back at once. If you want to understand why downstream systems need to be resilient under bursty recovery, our Bulk Data Imports Without Breaking Production piece covers the same operational shape from a different angle.

One practical architecture description: webhook endpoint at the edge, database table for raw events, queue for processing jobs, worker pool for domain actions, dead-letter queue for unresolved failures, and a replay tool that reads from the raw event table. That replay tool is the part teams forget. Without it, every bad event becomes a manual production fix.

Operational controls that keep webhook reliability boring

Once the code is in place, the hard part becomes keeping it healthy. The best webhook systems are boring because the controls are visible. You should know how many events arrived, how many were deduped, how many were processed, how many failed, and how long each stage took. If those numbers are not on a dashboard, the first sign of trouble will be a customer complaint.

At minimum, track these metrics:

  • events received per provider and event type
  • duplicate rate by event type
  • processing latency from receipt to completion
  • retry count and dead-letter volume
  • signature verification failures

These numbers tell you where the system is unhealthy before customers do. A rising duplicate rate can mean a provider is retrying because your handler is slow. A spike in signature failures can mean a secret rotation went wrong. Growing latency with flat volume often means a downstream dependency is degrading silently.

Alerting should be selective. Do not page on every failed event. Page when the system crosses a business threshold: backlog age above X minutes, dead-letter rate above Y percent, or a specific provider failing for more than Z minutes. Teams that page on every exception learn to ignore the pager. Teams that page on business impact stay sharp.

Security matters here too. Verify signatures before parsing untrusted content deeply. Reject replayed requests outside your accepted window if the provider supports timestamps. Store webhook secrets in a proper secret manager, not environment variables scattered across deploy scripts. If you are hardening the rest of the pipeline as well, our CI/CD Security Hardening: Protecting Your Pipeline article is a useful companion.

For testing, build three classes of cases: happy path, duplicate delivery, and malformed payload. Then add one more: delayed retry after partial success. That last one is the bug people miss. It is the scenario where the handler wrote to the database, crashed before acking, and the provider sent the event again. If your test suite covers that case, you are ahead of most teams already.

Webhook systems fail in the gaps between assumptions. The fix is not heroic code. It is a narrow contract, durable dedupe, bounded retries, and metrics that tell the truth. If that is the kind of problem you need to remove from the business, you can apply for an engagement; the application takes ten minutes, and we take three engagements a quarter. For a single focused outcome, a Sprint engagement is often the right shape.