Stripe Webhook Reliability: Handling Failures at Scale

Stripe webhooks are how your billing system learns that a customer’s card was charged, a subscription renewed, or a dispute was filed. They’re also one of the most fragile integration points in a SaaS platform.

Most teams treat webhooks like a fire-and-forget system: Stripe sends an event, your endpoint processes it, done. That works until it doesn’t. A brief network hiccup, a deploy that takes 90 seconds, a typo in error handling—and suddenly you’re missing refunds, double-charging customers, or worse, not updating subscription status when a payment fails.

The difference between a webhook system that holds up under load and one that bleeds money is idempotency, retry logic, and event deduplication. This post covers how to engineer that.

Why Stripe Webhooks Fail (And Why You’ll Miss It)

Stripe retries failed webhooks for up to three days, with exponential backoff: 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 24 hours. That’s generous by API standards, but it assumes your endpoint is reliable. Most aren’t.

Here’s what actually happens in production: Your webhook endpoint processes a payment.succeeded event, charges the customer in your database, and returns a 200. Stripe marks the event as delivered. But what if your response never reaches Stripe? Maybe a load balancer times out. Maybe your app server crashes after the response is sent but before the client receives the ACK. Stripe never got the 200, so it retries. Your endpoint processes the same event again, charges the customer again, and now they’re angry.

Or you deploy code that adds a new step to webhook processing—say, sending an email to the customer. The email service is down. Your endpoint throws an exception and returns a 500. Stripe retries. But your code is still broken, so it fails again. Stripe keeps retrying, and your error logs fill with noise while the customer’s payment sits unprocessed for hours.

The root cause: your webhook handler isn’t idempotent. Processing the same event twice shouldn’t create two charges or two emails. It should be safe, predictable, and observable.

Idempotency Keys: The Foundation

Stripe gives you a built-in tool for this: the event ID. Every webhook event has a unique id field that never changes, even if Stripe retries the delivery. Your job is to store that ID and check for it before processing the event.

The pattern is simple: before you mutate any state (charge a customer, update a subscription, send an email), write a record to your database with the event ID and a lock. Use a database constraint to make that insert fail if the ID already exists. If the insert succeeds, process the event. If it fails, you’ve seen this event before—skip processing and return 200 (the event is already handled).

Here’s what that looks like in Node.js with PostgreSQL:

const express = require('express');
const { Pool } = require('pg');

const pool = new Pool();

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Attempt to claim this event for processing
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    
    // This insert will fail if the event_id already exists (unique constraint)
    const insertResult = await client.query(
      `INSERT INTO webhook_events (event_id, event_type, payload, processed_at)
       VALUES ($1, $2, $3, NOW())
       ON CONFLICT (event_id) DO NOTHING
       RETURNING id`,
      [event.id, event.type, JSON.stringify(event.data)]
    );

    // If the insert returned no rows, this event was already processed
    if (insertResult.rows.length === 0) {
      console.log(`Event ${event.id} already processed, skipping`);
      await client.query('COMMIT');
      return res.sendStatus(200);
    }

    // Process the event
    await handleStripeEvent(event, client);
    await client.query('COMMIT');
    res.sendStatus(200);
  } catch (err) {
    await client.query('ROLLBACK');
    console.error(`Webhook processing failed for ${event.id}:`, err);
    // Return 500 so Stripe retries
    res.status(500).send('Internal error');
  } finally {
    client.release();
  }
});

async function handleStripeEvent(event, client) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      const paymentIntent = event.data.object;
      await client.query(
        `UPDATE subscriptions SET status = $1, last_paid_at = NOW() WHERE stripe_id = $2`,
        ['active', paymentIntent.customer]
      );
      break;
    case 'customer.subscription.deleted':
      const subscription = event.data.object;
      await client.query(
        `UPDATE subscriptions SET status = $1, cancelled_at = NOW() WHERE stripe_id = $2`,
        ['cancelled', subscription.id]
      );
      break;
  }
}

The key moves here: Use PostgreSQL’s ON CONFLICT DO NOTHING to make the insert atomic. If the event already exists, the insert silently fails and returns no rows. Check the result—if it’s empty, you’ve already processed this event, so return 200 immediately. If it succeeds, process the event inside the same transaction. If anything throws, rollback the whole thing and return 500 so Stripe retries.

This pattern ensures that no matter how many times Stripe delivers the same event, your system will only mutate state once. The customer gets charged once. The subscription status updates once. The email sends once (if you handle email idempotently too, which we’ll cover).

Handling Downstream Failures Without Losing Events

Idempotency keys solve the “same event, multiple times” problem. But they don’t solve the case where processing the event itself fails partway through.

Say your webhook handler updates the subscription status in your database, then tries to send a welcome email. The email service times out. Your endpoint returns a 500. Stripe retries the event. The second time, the subscription update succeeds (idempotent), but the email still fails. On the third retry, the email service is back up, so it sends—but now the customer got the email 30 minutes after the payment, which is jarring.

The solution: defer side effects to a separate queue. Your webhook handler should only update the critical state—subscription status, payment records, balance updates. Everything else—emails, webhook deliveries to third parties, analytics events—goes into a Redis queue or SQS, where a separate worker processes it with its own retry logic.

Here’s the revised pattern:

async function handleStripeEvent(event, client) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      const paymentIntent = event.data.object;
      
      // Update critical state in the same transaction
      await client.query(
        `UPDATE subscriptions SET status = $1, last_paid_at = NOW() WHERE stripe_id = $2`,
        ['active', paymentIntent.customer]
      );

      // Queue the side effect for async processing
      // This job will be retried independently if it fails
      await redis.lpush('email_queue', JSON.stringify({
        type: 'payment_received',
        customer_id: paymentIntent.customer,
        amount: paymentIntent.amount,
        timestamp: Date.now()
      }));
      
      break;
  }
}

Now the webhook handler returns 200 as soon as the critical state is updated. If the email job fails, it retries independently without causing the webhook to be marked as failed. This decouples your webhook reliability from the reliability of every downstream service.

Use Redis lists with BLPOP for simple FIFO queues, or Bull (a Redis-backed job queue library) if you need retries, dead-letter queues, and job scheduling. For high volume, AWS SQS or Kafka are more robust, but Redis is often enough for mid-scale SaaS.

Detecting and Recovering From Silent Failures

Even with idempotent processing, events can slip through the cracks. A database connection dies mid-transaction. A deployment happens and your endpoint is down for 10 seconds. A bug in your code causes a silent failure that doesn’t log an error.

You need observability. Log every webhook, including the event ID, type, and outcome. Then, at least once a day, query Stripe’s API to fetch all events from the past 24 hours and check which ones you’ve processed. If you find a gap, replay those events.

async function reconcileWebhookEvents() {
  const yesterday = new Date(Date.now() - 24 * 60 * 60 * 1000);
  
  // Fetch all events from Stripe from the past 24 hours
  const stripeEvents = await stripe.events.list({
    created: { gte: Math.floor(yesterday.getTime() / 1000) },
    limit: 100
  });

  for (const event of stripeEvents.data) {
    // Check if we've processed this event
    const result = await pool.query(
      'SELECT id FROM webhook_events WHERE event_id = $1',
      [event.id]
    );

    if (result.rows.length === 0) {
      console.warn(`Missing event ${event.id} (${event.type}), replaying`);
      // Replay the event
      await replayWebhookEvent(event);
    }
  }
}

Stripe’s API is paginated, so you’ll need to loop through batches. This is a background job that runs daily—it’s not fast, but it catches the rare edge case where a webhook was never delivered and never retried (which can happen if Stripe’s system itself has a bug).

Store the webhook events you process in a table with an index on event_id. This is cheap—you’re only storing the ID and type. After 30 days, you can archive or delete them. But keep them long enough to catch gaps.

Monitoring and Alerting on Webhook Health

The most common reason webhooks fail in production is that nobody knows they’re failing. By the time a customer notices their subscription wasn’t charged, you’ve lost days of revenue.

Set up monitoring on three things:

1. Webhook endpoint response time. If your endpoint is taking more than 5 seconds to respond, Stripe might timeout. Track p50, p95, and p99 latency. Alert if p95 exceeds 3 seconds.

2. Webhook processing error rate. Log every webhook outcome (success, already processed, failed). Calculate the error rate every 5 minutes. Alert if it exceeds 1% for more than 10 minutes.

3. Age of unprocessed queue items. If you’re using a queue for side effects, monitor how long items sit in the queue before they’re processed. Alert if the max age exceeds 30 minutes.

Use Prometheus or a similar system to scrape these metrics. Grafana to visualize them. PagerDuty to alert on-call engineers. Here’s a minimal Prometheus setup:

const promClient = require('prom-client');

const webhookDuration = new promClient.Histogram({
  name: 'stripe_webhook_duration_seconds',
  help: 'Time to process a Stripe webhook',
  labelNames: ['event_type', 'status'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

const webhookErrors = new promClient.Counter({
  name: 'stripe_webhook_errors_total',
  help: 'Number of failed webhook deliveries',
  labelNames: ['event_type', 'error_code']
});

app.post('/webhooks/stripe', async (req, res) => {
  const start = Date.now();
  try {
    // ... webhook processing
    webhookDuration.labels(event.type, 'success').observe((Date.now() - start) / 1000);
    res.sendStatus(200);
  } catch (err) {
    webhookErrors.labels(event.type, err.code).inc();
    webhookDuration.labels(event.type, 'error').observe((Date.now() - start) / 1000);
    res.status(500).send('Failed');
  }
});

app.get('/metrics', (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(promClient.register.metrics());
});

This gives you a clear signal when something is wrong. And when something goes wrong, you’ll have logs and metrics to debug it fast.

Testing Webhook Reliability in CI/CD

Your webhook handler is critical billing code. It needs test coverage.

At minimum, test these scenarios:

1. First delivery of an event succeeds. The event is processed, the database is updated, a 200 is returned.

2. Duplicate delivery of the same event. The event is processed once, subsequent deliveries are idempotent (no duplicate charges, no duplicate state changes).

3. Downstream failure (email service down). The critical state is updated, the side effect is queued, the endpoint returns 200 (not 500).

4. Database error during processing. The endpoint returns 500 so Stripe retries. No partial state changes.

describe('Stripe Webhooks', () => {
  it('processes a payment_intent.succeeded event', async () => {
    const event = {
      id: 'evt_test_1',
      type: 'payment_intent.succeeded',
      data: { object: { customer: 'cus_123', amount: 10000 } }
    };

    const response = await request(app)
      .post('/webhooks/stripe')
      .set('stripe-signature', createSignature(event))
      .send(event);

    expect(response.status).toBe(200);
    const sub = await pool.query('SELECT status FROM subscriptions WHERE stripe_id = $1', ['cus_123']);
    expect(sub.rows[0].status).toBe('active');
  });

  it('is idempotent: duplicate events do not double-charge', async () => {
    const event = {
      id: 'evt_test_2',
      type: 'payment_intent.succeeded',
      data: { object: { customer: 'cus_456', amount: 5000 } }
    };

    // First delivery
    await request(app)
      .post('/webhooks/stripe')
      .set('stripe-signature', createSignature(event))
      .send(event);

    // Second delivery (Stripe retry)
    const response = await request(app)
      .post('/webhooks/stripe')
      .set('stripe-signature', createSignature(event))
      .send(event);

    expect(response.status).toBe(200);
    const charges = await pool.query(
      'SELECT COUNT(*) FROM charges WHERE event_id = $1',
      ['evt_test_2']
    );
    expect(parseInt(charges.rows[0].count)).toBe(1); // Only one charge, not two
  });

  it('queues side effects even if email service is down', async () => {
    jest.spyOn(emailService, 'send').mockRejectedValue(new Error('Service down'));

    const event = {
      id: 'evt_test_3',
      type: 'payment_intent.succeeded',
      data: { object: { customer: 'cus_789', amount: 15000 } }
    };

    const response = await request(app)
      .post('/webhooks/stripe')
      .set('stripe-signature', createSignature(event))
      .send(event);

    expect(response.status).toBe(200); // Still returns 200, not 500
    const queuedJob = await redis.lpop('email_queue');
    expect(queuedJob).toBeDefined(); // Email is queued for retry
  });
});

Run these tests on every deploy. If a test fails, the deploy blocks. This prevents shipping code that breaks billing.

Common Mistakes That Break Webhooks at Scale

1. Not validating the signature. Stripe signs every webhook with a secret. If you don’t verify the signature, an attacker can forge events and charge your customers. Always call stripe.webhooks.constructEvent().

2. Logging sensitive data. Don’t log customer IDs, card tokens, or amounts to stdout. Log to a structured logging service with access controls. PII in logs is a compliance risk.

3. Processing webhooks synchronously in your HTTP request. If processing takes more than a few seconds, the HTTP request times out. Use a queue or async worker, even if you think processing is fast. Speed requirements change.

4. Returning 200 before processing is complete. If you return 200 and then the process crashes, Stripe won’t retry and the event is lost. Process the event, then return 200. Or queue the event and have a separate worker process it, but don’t return 200 until it’s queued.

5. Handling only the events you care about. Your code might handle payment_intent.succeeded, but what about payment_intent.canceled? customer.subscription.deleted? If you ignore them, your subscription records will get out of sync with Stripe. List all the events you care about and handle each one explicitly.

When to Use Webhooks vs. Polling

Webhooks are great for real-time events, but they’re not the only way to keep your system in sync with Stripe. Some teams supplement webhooks with polling: once an hour, query Stripe’s API for recent changes and reconcile your database.

Polling is slower (your data is at most an hour out of date) but it’s simpler and more reliable. You don’t have to worry about webhook delivery failures or signature validation. You just query the API and update your database.

The best approach: use webhooks for immediate notification, use polling as a backup safety net. Webhooks handle 99% of updates in real-time. Polling catches the 1% that slip through the cracks. This is what mature SaaS companies do.

A webhook-first system with daily polling reconciliation is the gold standard. It handles failures gracefully and keeps your data accurate even if Stripe’s infrastructure has a blip.

Building reliable billing infrastructure is non-trivial—it’s one of those systems where a small bug costs real money, and the bugs are often subtle (a race condition, a network timeout, an off-by-one in retry logic). If you’re scaling to the point where webhook reliability is a blocker, it’s worth having a senior engineer review the architecture. We take engagements like this—webhook systems, billing integrations, event-driven architectures that hold up under load. A Sprint engagement often uncovers the gaps and ships a reliable foundation in 2-4 weeks.