Webhooks look simple until the first duplicate delivery charges a customer twice or the first outage drops an event on the floor. We have seen both, and our current design exists because of them.
Here is the whole pipeline: acknowledge fast, persist first, process from a queue, and make every handler safe to run twice.
#Acknowledge first, work later
The HTTP handler verifies the signature, writes the raw event to a table keyed by the Stripe event ID, and returns 200. Nothing else happens in the request.
A unique constraint on that event ID is our first line of idempotency: a replay of the same event is a no-op insert, not a second charge.
- Verify the signature on the raw body before parsing
- Persist the event, then enqueue its ID, never the payload
- Alert on dead-letter depth, not on individual failures
#Retries and the dead-letter queue
Workers retry with exponential backoff for up to 24 hours. After the final attempt the job lands in a dead-letter queue and pages a human, with the event ID and last error attached.
We deliberately do not alert on a single failure. Transient errors are normal; a growing dead-letter queue is not.
export const webhookQueue = new Queue('stripe-webhooks', {
connection,
defaultJobOptions: {
attempts: 12,
backoff: { type: 'exponential', delay: 5_000 },
removeOnComplete: { age: 60 * 60 * 24 },
removeOnFail: false, // failed jobs stay for the dead-letter review
},
})
await db.insert(stripeEvents)
.values({ id: event.id, payload: raw })
.onConflictDoNothing()
await webhookQueue.add('process', { eventId: event.id }, { jobId: event.id })
