Skip to content
StormedoStart free
Menu

Engineering guide

Reliable webhook delivery without duplicate work

A webhook can time out after the receiver has already done the work. Here's how to retry failed deliveries without processing the same event twice.

A customer places an order in your store. Your app sends an order.created webhook to the warehouse so it can prepare a shipment.

When the warehouse responds with 200 OK, delivery looks straightforward. But what should your app do if the request times out? Sending it again might create a second shipment. Giving up might leave the order unshipped.

Let’s follow that order through the failure and work out what each side needs to do. Your app sends the event, a delivery service such as Stormedo handles the HTTP attempts, and the warehouse receives it.

A timeout leaves the result unknown

Consider this sequence:

  1. Your app sends the order webhook.
  2. The warehouse records the shipment.
  3. The connection breaks before the success response reaches the sender.
  4. The sender times out and sends the same event again.

The second delivery is valid. The warehouse has already done the work.

A timeout tells you that the sender did not receive a response. It cannot tell you whether the receiver acted. Increasing the timeout gives a slow receiver more time, but it cannot recover a response lost to a broken connection.

The warehouse needs to recognize the repeated event and acknowledge it without creating another shipment. That behavior is called idempotency: repeating an operation has the same effect as doing it once.

Delivery simulationOne order. Two shipments?See what happens when the warehouse's reply gets lost.
StormedoSender
Order #1042
WarehouseReceiver
1 Send orderCreate a shipment
2 Response lost200 OK never reaches Stormedo
3 Retry same orderSame request ID, another shipment
At the warehouse2 shipments for 1 order

Store the request before promising delivery

There is another way to lose this order. Your app could start sending the webhook in the background, then restart before the request finishes. If the pending request existed only in memory, nothing is left to retry.

Store pending delivery work somewhere that survives a restart. You can own that storage and the worker that sends requests, or hand the request to a delivery service.

Stormedo stores a request before returning 202 Accepted. It then attempts delivery independently of your app’s connection. Keep the returned data.id so you can look up the result later.

The 202 response means Stormedo has accepted the work. The warehouse may not have received it yet. HTTP’s definition of 202 does not itself promise storage or completed processing, so this distinction depends on the service’s contract. RFC 9110 describes that distinction.

If your app can restart between saving the order and submitting its webhook, that gap also needs a persistent record of the event to send. Stormedo can take responsibility only after it accepts the request.

Retry failures that may clear on their own

A warehouse endpoint that is briefly unavailable may work a few seconds later. An endpoint rejecting the wrong credentials will keep rejecting them until someone fixes the configuration.

Stormedo handles delivery outcomes as follows:

Outcome What Stormedo does
Connection or DNS failure, timeout, 408, or 5xx Retries while attempts remain. The receiver may already have acted.
429 Too Many Requests Waits before retrying while attempts remain. A valid Retry-After can extend the wait.
Other 4xx, such as 400 or 401 Stops. Inspect the request or receiver before sending again.
3xx redirect Stops without following the redirect. Use the final destination URL.
Any 2xx response Marks delivery successful.

Retries need a delay and a stopping point. Waiting longer after repeated failures is called backoff. If you build your own sender, add some randomness to the wait, called jitter, so failed requests do not all return at once. AWS recommends backoff, jitter, and a retry limit.

Stormedo currently uses increasing waits without jitter. Its quick-send API allows you to choose the retry count; retry=3 permits one initial attempt and up to three retries. The default timeout is 50 seconds per attempt. See the retry reference for exact timing and Retry-After handling.

Make repeated delivery safe at both ends

There are two calls to protect: your app’s submission to Stormedo, and Stormedo’s delivery to the warehouse.

When your app calls Stormedo

Give each event a stable Idempotency-Key, such as order-123-created. If that submission times out, loses its response, or returns 500, 502, or 503, acceptance may be unknown. Retry with the same project, key, and input.

For 24 hours, Stormedo returns the original request when the same project submits the same key and input. Changing the input with that key returns 409. Creating a new key to recover a failed call can create a second request.

Use a new key for a new event. See submission recovery for what to do when the original call had no key or the window has expired.

When the warehouse receives the event

Stormedo uses at-least-once delivery, so a request can arrive more than once. Automatic retries carry the same Stormedo-Request-Id. After verifying the request, the warehouse can use that ID to recognize completed work.

For the shipment example, save the request ID and create the shipment record in the same database transaction. Give the request ID a unique constraint so concurrent deliveries cannot both create a shipment. If the transaction fails, roll back both changes. A repeated delivery can return success once the original transaction has committed.

That is the production step. You do not need a database to try receiving and verifying an event. The idempotency guide explains how to add duplicate handling when you introduce business actions.

Manual replay creates a new Stormedo request ID. If replay must not create another shipment either, include a stable event ID in the payload and check that too. For calls to another API, use that API’s idempotency support. A local database transaction cannot make an external call happen exactly once.

Verify the event before acting on it

A request ID alone does not prove who sent the event. Verify the signature and the original body bytes before trusting the payload or checking for duplicates. The Stormedo SDK checks the expected project, destination URL, and other signed request details for you.

Keep the raw body intact until verification finishes. Parsing JSON and turning it back into a string can change the bytes and break verification. The Express and FastAPI examples show the required order.

Return 2xx after the work has committed, or after you have stored it in a queue that survives restarts. Returning success and then starting an in-memory background task can lose work after Stormedo stops retrying. Once you acknowledge a queued event, your queue and workers are responsible for finishing it.

Stormedo treats every 2xx as success. A response body containing {"success":false} will not trigger a retry if the HTTP status is 200.

Check what happened before replaying

When an order webhook fails, keep its request ID and open its attempt history. Check the response code or connection error, how many attempts ran, and whether another attempt is scheduled. For a timeout, check the warehouse’s records too. The shipment may already exist.

Stormedo shows the request and completed attempts in its dashboard and API. Use delivery troubleshooting to investigate, then review the replay behavior before sending it again.

One useful integration test is to let the warehouse save a shipment, then drop the response. Send the event again and check that there is still only one shipment. This exercises the failure that a successful first delivery misses.

Try a delivery with Stormedo

Start with Stormedo’s test destination. It accepts the request and returns 200 OK, so you can inspect a delivery before building a receiver.

Create a project API key in the dashboard, then set STORMEDO_TOKEN in your terminal. The quickstart walks through setup.

Bash
curl --request POST \
'https://api.stormedo.com/send?url=https://api.stormedo.com/ping&retry=3' \
--header "Authorization: Bearer $STORMEDO_TOKEN" \
--header 'Idempotency-Key: guide-order-123-created' \
--header 'Content-Type: application/json' \
--data '{"type":"order.created","order_id":"ord_123"}'

The API returns 202. Copy data.id from the response and find it in Requests. The request should reach succeeded, with a completed attempt showing 200. The test destination does not create an order or shipment.

When you are ready to send to your own app, follow Receive deliveries and use your receiver’s public HTTPS URL as the destination.