Meta Description: Build reliable webhook systems with proper delivery guarantees, retry logic, security verification, and idempotency. Learn from real-world examples.
Keywords: webhooks, event-driven architecture, api callbacks, webhook security, webhook retry, webhook best practices
Word Count: ~2,400 words
Your API needs to notify external systems when events happen. You could make them poll your API every minute, but that's wasteful.
Webhooks solve this. When an event occurs, your API sends an HTTP request to the client's URL. They get notified instantly without polling.
But webhooks are harder than they look. Messages get lost. Clients go offline. Requests fail. You need proper delivery guarantees, retry logic, and security.
Here's how to build webhooks that work.
Webhooks are "reverse APIs." Instead of clients calling your API, your API calls theirs.
Traditional API (client pulls):
Client → GET /orders/123/status → API
Client → GET /orders/123/status → API (poll every minute)
Client → GET /orders/123/status → API
Webhook (server pushes):
API → POST https://client.com/webhooks → Client
(only when status changes)
Webhooks are event-driven. Clients register a URL, and you POST events to it.
Clients register webhook URLs through your API:
POST /v1/webhooks
{
"url": "https://client.com/webhooks/petstore",
"events": ["pet.adopted", "pet.status_changed"],
"secret": "whsec_abc123..."
}
Response: 201 Created
{
"id": "wh_019b4132-70aa-764f-b315-e2803d882a24",
"url": "https://client.com/webhooks/petstore",
"events": ["pet.adopted", "pet.status_changed"],
"status": "active",
"createdAt": "2026-03-13T10:30:00Z"
}
Store this configuration in your database:
CREATE TABLE webhooks (
id UUID PRIMARY KEY,
user_id UUID NOT NULL,
url TEXT NOT NULL,
events TEXT[] NOT NULL,
secret TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active',
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
Use a consistent payload structure:
{
"id": "evt_019b4145-8f3a-7c2d-a1b5-f4e8d9c3a7b2",
"type": "pet.adopted",
"createdAt": "2026-03-13T10:35:00Z",
"data": {
"pet": {
"id": "019b4132-70aa-764f-b315-e2803d882a24",
"name": "Max",
"species": "DOG",
"status": "ADOPTED"
},
"adopter": {
"id": "019b4137-6d8e-5c2b-e9f4-a3b5c8d7e2f1",
"name": "John Doe",
"email": "john@example.com"
},
"adoptedAt": "2026-03-13T10:35:00Z"
}
}
Key fields: - id: Unique event ID (for idempotency) - type: Event type (for routing) - createdAt: When the event occurred - data: Event-specific payload
Webhooks are public endpoints. Anyone can POST to them. You need to verify requests come from your API.
Sign webhook payloads with HMAC:
const crypto = require('crypto');
function signPayload(payload, secret) {
const hmac = crypto.createHmac('sha256', secret);
hmac.update(JSON.stringify(payload));
return hmac.digest('hex');
}
// When sending webhook
const payload = {
id: 'evt_123',
type: 'pet.adopted',
data: { ... }
};
const signature = signPayload(payload, webhook.secret);
await fetch(webhook.url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': signature,
'X-Webhook-ID': payload.id,
'X-Webhook-Timestamp': Date.now().toString()
},
body: JSON.stringify(payload)
});
Clients verify the signature:
function verifySignature(payload, signature, secret) {
const expectedSignature = signPayload(payload, secret);
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
// In webhook handler
app.post('/webhooks/petstore', (req, res) => {
const signature = req.headers['x-webhook-signature'];
const payload = req.body;
if (!verifySignature(payload, signature, WEBHOOK_SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Process webhook
processWebhook(payload);
res.status(200).json({ received: true });
});
Prevent replay attacks by checking timestamps:
function verifyTimestamp(timestamp, maxAge = 300000) { // 5 minutes
const now = Date.now();
const age = now - parseInt(timestamp);
return age >= 0 && age <= maxAge;
}
app.post('/webhooks/petstore', (req, res) => {
const timestamp = req.headers['x-webhook-timestamp'];
if (!verifyTimestamp(timestamp)) {
return res.status(401).json({ error: 'Timestamp too old' });
}
// Verify signature and process
});
Publish your webhook source IPs. Clients can allowlist them:
Webhook requests come from:
- 203.0.113.10
- 203.0.113.11
- 203.0.113.12
This adds defense in depth but shouldn't replace signature verification.
Webhooks can fail. Networks are unreliable. Clients go offline. You need retry logic.
Use exponential backoff with jitter:
const RETRY_DELAYS = [
1000, // 1 second
5000, // 5 seconds
30000, // 30 seconds
300000, // 5 minutes
1800000, // 30 minutes
3600000 // 1 hour
];
async function deliverWebhook(webhook, payload, attempt = 0) {
try {
const response = await fetch(webhook.url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': signPayload(payload, webhook.secret),
'X-Webhook-ID': payload.id,
'X-Webhook-Timestamp': Date.now().toString(),
'X-Webhook-Attempt': (attempt + 1).toString()
},
body: JSON.stringify(payload),
timeout: 10000 // 10 second timeout
});
if (response.status >= 200 && response.status < 300) {
// Success
await markWebhookDelivered(payload.id, webhook.id);
return { success: true };
}
// Failed, retry
if (attempt < RETRY_DELAYS.length) {
const delay = RETRY_DELAYS[attempt] + Math.random() * 1000; // Add jitter
await scheduleRetry(webhook, payload, attempt + 1, delay);
return { success: false, retry: true };
}
// Max retries exceeded
await markWebhookFailed(payload.id, webhook.id);
await disableWebhook(webhook.id); // Disable after repeated failures
return { success: false, retry: false };
} catch (error) {
// Network error, retry
if (attempt < RETRY_DELAYS.length) {
const delay = RETRY_DELAYS[attempt] + Math.random() * 1000;
await scheduleRetry(webhook, payload, attempt + 1, delay);
return { success: false, retry: true };
}
await markWebhookFailed(payload.id, webhook.id);
await disableWebhook(webhook.id);
return { success: false, retry: false };
}
}
Store delivery attempts:
CREATE TABLE webhook_deliveries (
id UUID PRIMARY KEY,
webhook_id UUID NOT NULL REFERENCES webhooks(id),
event_id TEXT NOT NULL,
attempt INT NOT NULL DEFAULT 1,
status TEXT NOT NULL, -- 'pending', 'delivered', 'failed'
response_status INT,
response_body TEXT,
error TEXT,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
delivered_at TIMESTAMP
);
This lets you: - Show delivery status to users - Debug failed webhooks - Retry manually - Generate delivery reports
Webhooks may be delivered multiple times (due to retries). Clients must handle duplicates.
Include a unique event ID:
{
"id": "evt_019b4145-8f3a-7c2d-a1b5-f4e8d9c3a7b2",
"type": "pet.adopted",
"data": { ... }
}
Clients store processed event IDs:
const processedEvents = new Set();
app.post('/webhooks/petstore', async (req, res) => {
const event = req.body;
// Check if already processed
if (processedEvents.has(event.id)) {
return res.status(200).json({ received: true, duplicate: true });
}
// Process event
await processEvent(event);
// Mark as processed
processedEvents.add(event.id);
await db.query('INSERT INTO processed_events (event_id) VALUES ($1)', [event.id]);
res.status(200).json({ received: true });
});
Design operations to be idempotent:
Bad (not idempotent):
// Increment counter
await db.query('UPDATE stats SET adoption_count = adoption_count + 1');
If the webhook is delivered twice, the counter increments twice.
Good (idempotent):
// Set specific value
await db.query('UPDATE pets SET status = $1 WHERE id = $2', ['ADOPTED', petId]);
Setting a value is idempotent. Doing it twice has the same effect as once.
Provide a test endpoint:
POST /v1/webhooks/{id}/test
{
"eventType": "pet.adopted"
}
This sends a test event to the webhook URL. Clients can verify their endpoint works before going live.
Show delivery logs to users:
GET /v1/webhooks/{id}/deliveries
Response:
{
"data": [
{
"id": "del_123",
"eventId": "evt_456",
"eventType": "pet.adopted",
"attempt": 1,
"status": "delivered",
"responseStatus": 200,
"createdAt": "2026-03-13T10:35:00Z",
"deliveredAt": "2026-03-13T10:35:01Z"
},
{
"id": "del_124",
"eventId": "evt_457",
"eventType": "pet.status_changed",
"attempt": 3,
"status": "failed",
"responseStatus": 500,
"error": "Internal Server Error",
"createdAt": "2026-03-13T10:40:00Z"
}
]
}
This helps users debug webhook issues.
Let users retry failed webhooks:
POST /v1/webhooks/{id}/deliveries/{deliveryId}/retry
This re-sends the event immediately.
Automatically disable webhooks after repeated failures:
async function checkWebhookHealth(webhookId) {
const recentFailures = await db.query(`
SELECT COUNT(*) as failures
FROM webhook_deliveries
WHERE webhook_id = $1
AND status = 'failed'
AND created_at > NOW() - INTERVAL '1 hour'
`, [webhookId]);
if (recentFailures.rows[0].failures >= 10) {
await db.query(`
UPDATE webhooks
SET status = 'disabled',
disabled_reason = 'Too many failures'
WHERE id = $1
`, [webhookId]);
// Notify user
await sendEmail(webhook.userId, {
subject: 'Webhook Disabled',
body: 'Your webhook has been disabled due to repeated failures.'
});
}
}
Process webhooks asynchronously:
app.post('/webhooks/petstore', async (req, res) => {
const event = req.body;
// Verify signature
if (!verifySignature(event, req.headers['x-webhook-signature'], SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Queue for processing
await queue.add('process-webhook', event);
// Return immediately
res.status(200).json({ received: true });
});
Don't make the webhook sender wait for your processing.
Clearly document all event types:
## Event Types
### pet.adopted
Triggered when a pet is adopted.
Payload:
- pet: Pet object
- adopter: User object
- adoptedAt: ISO 8601 timestamp
### pet.status_changed
Triggered when a pet's status changes.
Payload:
- pet: Pet object
- oldStatus: Previous status
- newStatus: New status
- changedAt: ISO 8601 timestamp
Include API version in webhook payloads:
{
"id": "evt_123",
"type": "pet.adopted",
"apiVersion": "2024-03-01",
"data": { ... }
}
This lets you evolve webhook payloads without breaking clients.
Build a webhook testing tool where users can: - See example payloads - Test their endpoints - View delivery logs - Retry failed deliveries
This improves developer experience significantly.
Webhooks are powerful but require careful implementation:
The Modern PetStore API implements all these patterns. Check the webhook documentation for complete examples.
Build webhooks that work reliably, and your users will thank you.
Related Articles: - WebSocket vs Server-Sent Events: Real-Time APIs Explained - Building Event-Driven APIs: From Webhooks to Message Queues - API Security Best Practices: Authentication and Authorization
