Designing a Webhook Processing System That Survives Failure

by Souy Soeng /October 03, 2026

Diagram of a webhook pipeline: sender, receiver verifying and replying, queue, background workers, and a database

A webhook processing system is the part of your app that receives event messages from other services, such as a payment provider, and handles them quickly, safely and exactly once. A customer might pay on your website, but the payment happens on someone else's servers. Webhooks are how your app finds out within seconds.

This guide follows a single event from the moment it is sent to the moment your app has finished with it. Along the way it covers five ideas: checking who sent a message, replying quickly, using a queue, avoiding double work, and handling failure calmly. Each step builds on the last.

In short

A webhook is an HTTP request another system sends to your app when an event happens.
Verify the signature first, then reply with a success code right away.
Put events on a queue and let background workers do the slow work.
Use event IDs so duplicate deliveries change nothing.
Retry failures with growing delays, then park stubborn events in a dead-letter queue.

What is a webhook, and why does it matter?

A webhook is an HTTP request that one system sends to another when something happens. You give the other service a web address that belongs to your app, called an endpoint. When an event happens on their side, such as a successful payment, they pack the details, like the amount and order number, into a message (usually JSON) and send it as an HTTP POST to that address.

Think of it as a doorbell. Without one, you would walk to the front door every few minutes to check for visitors. With a doorbell, you get on with your day until it rings. Webhooks also flip the usual direction: normally your app calls another service's API, but with a webhook, their system calls yours.

Getting this right matters. Payments, deliveries and sign-ups often happen on outside services. If your app misses a payment event, a customer who paid gets no confirmation and their order never ships. If it handles the same event twice, you might ship two parcels or add credit twice. Both mistakes cost money and trust.

Missed events: orders sit unpaid and customers get silence.
Doubled events: duplicate shipments or double credit.
Done well, webhook handling is invisible to everyone.

Polling vs webhooks: what's the difference?

The alternative to webhooks is polling: your app keeps asking the other service whether anything has changed. Both approaches eventually get the same information into your app. The difference is who starts the conversation, how often it happens, and how quickly you learn about the event.

With polling, most answers are 'nothing new', and even real events are only noticed at the next check. It wastes requests on both sides. Polling is a child in the back seat asking 'are we there yet?' every minute. A webhook is the driver announcing 'we're here' once, at the right moment.

Webhooks are not free, though. Your app needs a public endpoint that is ready to receive requests at any moment. Those requests can arrive in bursts or fail partway through, which is exactly why the rest of the design matters.

Polling: you ask repeatedly, updates arrive late, many wasted requests.
Webhooks: the sender tells you once, right when it happens.
Trade-off: you must run an always-available public endpoint.

The parts of a webhook processing system

A solid webhook processing system is not one big function. It is a short pipeline where each part has one clear job. The sender fires an event. A small receiver checks it and replies straight away. The event waits in a queue, a worker picks it up, and the result lands in your database.

The receiver works like a front desk: it checks the visitor's ID, says thanks, and hands them along. It deliberately avoids anything slow, so it can always answer quickly. Behind the scenes, workers do the heavy lifting, such as sending receipts and updating stock. Separating fast work from slow work is the single most important idea in the whole design.

Receiver: verify and acknowledge.
Queue: hold events safely.
Workers: do the real processing in the background.
Database: where results are recorded.

How do you verify a webhook signature?

Your endpoint is public, so anyone on the internet could send it a fake 'payment succeeded' message. The first thing the receiver does is check that the request is real. Verification answers two questions at once: did this really come from the service you trust, and is the content exactly what they sent, with no amount or order number quietly changed?

The usual approach relies on a shared secret key. The provider uses it to compute a signature from the message body and puts that signature in a request header. Your receiver computes the same signature itself and compares the two. Without the secret, an attacker cannot produce a matching signature, much like a forger cannot copy a wax seal they don't own.

If the signatures don't match, stop right there. Don't save the event, don't queue it, just reply with an error and do nothing else with that request.

# shared secret only you and sender know
const sig = req.headers['x-signature'];
const expected = hmac(secret, rawBody);
if (!safeEqual(sig, expected)) {
  return res.status(401).end(); }

Why reply fast and put events in a queue?

Once a request is verified, the sender is waiting for your answer, and it won't wait forever. Providers only allow a limited time for a reply. If your handler is busy sending emails or updating inventory, the sender may give up, decide delivery failed, and send the event again later, creating duplicates you never needed.

So the rule is simple: verify the event, place it somewhere safe, then immediately answer with a 2xx success code such as 200. It's like signing for a parcel at the door and opening it later at your desk. The sender only needs to know you received the event safely.

That safe place is a queue, a waiting line that holds events until something is ready to handle them. The receiver only adds events, and workers only take them, so neither needs to know how busy the other is. It works like a restaurant ticket rail: the waiter clips orders up, and cooks pull them off when ready.

Queues really pay off during busy moments. If a big sale sends a flood of payments at once, the queue simply gets longer for a while. Nothing is dropped, and workers steadily catch up. If the backlog grows, you can add more workers to share the load.

What do workers do, and how do you stay idempotent?

A worker is a small program running on its own that follows the same loop over and over. First it pulls the next event from the queue; most queues hide that event from other workers while it is being handled. Then it does the business logic, such as marking the order paid, emailing a receipt and lowering the stock count. Finally it tells the queue it is finished.

The order of those steps matters. The event is only removed after the work succeeds. If the worker crashes before confirming, the event reappears for another worker, so nothing quietly slips through the cracks.

Senders may also deliver the same event more than once, because of a slow reply, a network blip, or their own retry logic. That is normal, not a sign that something is broken. Your handling should be idempotent: doing it twice has the same result as doing it once, just as pressing a light switch 'on' twice leaves the light on.

The tool for this is the event ID included with each webhook. Before doing any work, the worker checks whether it has already seen that ID. If it has, it stops. If not, it does the work and records the ID. Ideally, save the ID together with the work itself, so a crash can't leave you halfway between.

# every event carries a unique id
if (await seen(event.id)) return;
await markOrderPaid(event.orderId);
await rememberEvent(event.id);

Retries with backoff and the dead-letter queue

Real systems fail. The database is busy, or the email service is down for a minute. Most of these failures are temporary, so giving up on the first error would throw away events that would have worked moments later. Instead, the worker tries, fails, waits, and tries again, with the wait growing each round, perhaps doubling. That growing delay is called backoff.

Backoff exists because hammering a struggling service right away only makes it struggle more. It's like calling a friend whose line is busy: redialing every second won't help, but waiting a little longer each time gives them room to finish.

Retries can't go on forever, though. Some events will never succeed, perhaps because of a bug or bad data, such as an order number your database has never heard of. After a set number of attempts, the event moves to a dead-letter queue: a separate holding area for events that keep failing, named after the post office's pile of undeliverable letters.

Parking the event there keeps healthy events flowing. Set an alert so your team knows when something lands in it. A developer can then find the cause, fix it, and send the event back into the main queue. Nothing was lost, only delayed.

Retry temporary failures with growing delays.
Cap the number of attempts.
Move stubborn events to a dead-letter queue and alert on it.
Fix the cause, then replay the event.

Example: one payment, end to end

Picture a customer named Maya buying headphones in your shop. Her payment provider sends a payment.succeeded webhook for order A17. The signature checks out. The receiver replies 200 OK and puts the event on the queue. A worker marks order A17 as paid. Later the same event arrives again, and it is skipped because its ID has already been seen.

Now suppose the receipt email fails because the email service is briefly down. The worker retries with backoff, while the order is already safely marked paid, so shipping isn't held up. From Maya's side, it all just works: a confirmation within seconds and exactly one receipt, even though the event arrived twice.

Common webhook mistakes and how to avoid them

The most common mistake is doing all the work inside the request: emails, database updates, everything. It works in testing, then times out under real load, triggering retries and duplicates. Keep the receiver tiny, reply first, and push work to the queue.

The second is skipping signature verification because it feels like extra work during a quick prototype. That leaves a door wide open for anyone to fake a payment. The third is assuming each event arrives exactly once and always succeeds. Build event ID checks in from day one, and actually look at your dead-letter queue rather than letting it fill up quietly.

Doing the work before replying → reply first, then queue.
Skipping the signature → always verify before anything else.
Assuming one delivery → use event IDs and watch your DLQ.

Key takeaways

Verify the signature before trusting anything in the request.
Acknowledge fast with a 2xx success code once the event is safely queued.
Let background workers do slow work, and only confirm events after success.
Use event IDs so repeated deliveries do no harm.
Retry with backoff, cap attempts, and park failures in a dead-letter queue.

Frequently asked questions

What is the difference between a webhook and an API?

With a regular API, your app calls another service to ask for data. With a webhook, the other service calls your app's endpoint when an event happens. It is often described as a reverse API call.

Why do I receive the same webhook more than once?

Duplicates can come from a slow reply, a network problem, or the sender's own retry logic. This is normal behaviour, so design your handling to be idempotent by checking event IDs before doing any work.

Should I process a webhook before returning 200 OK?

No. Verify the request, place the event on a queue, and reply with a success code right away. Doing slow work first risks a timeout, which makes the sender retry and creates duplicates.

Why do webhooks need signature verification?

Your endpoint is public, so anyone could send it a fake event. Checking the signature with a shared secret proves the message came from your provider and wasn't changed on the way.

What is a dead-letter queue used for?

It holds events that have failed all their retries, so they stop blocking healthy events but are not lost. A developer can fix the cause and send them back to the main queue.

Watch the full video on YouTube →

Tag

Ad