You're building an integration between two systems. One system needs to know when something happens in the other. Maybe it's when a new order is placed, when a payment completes, or when a user signs up.
You have two main options: polling or webhooks. Polling means repeatedly asking "did anything happen yet?" Webhooks mean "I'll tell you when something happens."
Both approaches work, but they have very different trade-offs. In this guide, we'll explore when to use each one, how to implement them properly, and how to handle the tricky parts like security and reliability.
With polling, the client repeatedly requests data from the server to check for changes:
// Client polls every 30 seconds
setInterval(async () => {
const response = await fetch('https://api.example.com/orders?status=new');
const orders = await response.json();
orders.forEach(order => {
processNewOrder(order);
});
}, 30000);
With webhooks, the server sends an HTTP request to the client when something happens:
// Server sends webhook when order is created
app.post('/orders', async (req, res) => {
const order = await createOrder(req.body);
// Send webhook notification
await fetch('https://client.example.com/webhooks/order-created', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
event: 'order.created',
data: order
})
});
res.status(201).json(order);
});
Polling makes sense when:
1. You need to pull data on your schedule
Maybe you only process orders once per hour, or you want to batch updates. Polling lets you control the timing:
// Process orders every hour
cron.schedule('0 * * * *', async () => {
const orders = await fetchNewOrders();
await batchProcessOrders(orders);
});
2. The data source doesn't support webhooks
Not all APIs offer webhooks. If you're integrating with a legacy system or a third-party API without webhook support, polling is your only option.
3. You're behind a firewall
Webhooks require an publicly accessible endpoint. If your system is behind a corporate firewall or NAT, receiving webhooks can be complicated. Polling works fine because you're making outbound requests.
4. Events are frequent and you need them all
If events happen constantly and you need to process them in batches, polling can be more efficient than handling thousands of individual webhook calls.
5. You want simplicity
Polling is conceptually simpler. You don't need to worry about webhook security, retry logic, or managing webhook endpoints.
Webhooks are better when:
1. You need real-time notifications
If you need to react immediately when something happens, webhooks are the way to go. Polling has inherent latency—you might not see an event for 30 seconds or more.
// Webhook delivers notification instantly
app.post('/webhooks/payment-completed', async (req, res) => {
const payment = req.body.data;
// Immediately fulfill the order
await fulfillOrder(payment.orderId);
// Send confirmation email
await sendEmail(payment.customerEmail, 'Order confirmed!');
res.status(200).send('OK');
});
2. You want to reduce server load
Polling creates constant traffic, even when nothing is happening. If you have 1000 clients polling every 30 seconds, that's 2.88 million requests per day—most of which return "no new data."
Webhooks only send requests when something actually happens.
3. You want to scale efficiently
As you add more clients, polling load increases linearly. With webhooks, the server only does work when events occur, regardless of how many clients are listening.
4. The API provider recommends it
Many modern APIs (Stripe, GitHub, Shopify) strongly encourage webhooks over polling. They often rate-limit polling endpoints aggressively.
Let's build a complete webhook system, starting with the server side.
const crypto = require('crypto');
class WebhookService {
constructor() {
this.subscribers = new Map(); // In production, use a database
}
// Register a webhook endpoint
subscribe(url, events, secret) {
const id = crypto.randomBytes(16).toString('hex');
this.subscribers.set(id, {
id,
url,
events, // ['order.created', 'order.updated']
secret,
createdAt: new Date()
});
return id;
}
// Remove a webhook endpoint
unsubscribe(id) {
return this.subscribers.delete(id);
}
// Send webhook to all subscribers
async dispatch(event, data) {
const subscribers = Array.from(this.subscribers.values())
.filter(sub => sub.events.includes(event));
const results = await Promise.allSettled(
subscribers.map(sub => this.sendWebhook(sub, event, data))
);
return results;
}
// Send individual webhook
async sendWebhook(subscriber, event, data, attempt = 1) {
const payload = {
id: crypto.randomBytes(16).toString('hex'),
event,
data,
timestamp: new Date().toISOString()
};
const signature = this.generateSignature(
JSON.stringify(payload),
subscriber.secret
);
try {
const response = await fetch(subscriber.url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': signature,
'X-Webhook-Event': event,
'X-Webhook-Delivery': payload.id
},
body: JSON.stringify(payload),
timeout: 5000 // 5 second timeout
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
console.log(`Webhook delivered to ${subscriber.url}`);
return { success: true, attempt };
} catch (error) {
console.error(`Webhook delivery failed (attempt ${attempt}):`, error);
// Retry with exponential backoff
if (attempt < 3) {
const delay = Math.pow(2, attempt) * 1000; // 2s, 4s, 8s
await new Promise(resolve => setTimeout(resolve, delay));
return this.sendWebhook(subscriber, event, data, attempt + 1);
}
return { success: false, attempt, error: error.message };
}
}
// Generate HMAC signature
generateSignature(payload, secret) {
return crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
}
}
// Usage
const webhooks = new WebhookService();
// API endpoint to register webhooks
app.post('/webhooks/subscribe', (req, res) => {
const { url, events, secret } = req.body;
const id = webhooks.subscribe(url, events, secret);
res.status(201).json({
id,
url,
events,
message: 'Webhook registered successfully'
});
});
// Trigger webhook when order is created
app.post('/orders', async (req, res) => {
const order = await createOrder(req.body);
// Dispatch webhook asynchronously
webhooks.dispatch('order.created', order).catch(err => {
console.error('Webhook dispatch error:', err);
});
res.status(201).json(order);
});
const express = require('express');
const crypto = require('crypto');
const app = express();
app.use(express.json());
// Webhook endpoint
app.post('/webhooks/orders', (req, res) => {
// 1. Verify signature
const signature = req.headers['x-webhook-signature'];
const payload = JSON.stringify(req.body);
const secret = process.env.WEBHOOK_SECRET;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
if (signature !== expectedSignature) {
console.error('Invalid webhook signature');
return res.status(401).json({ error: 'Invalid signature' });
}
// 2. Check for duplicate delivery
const deliveryId = req.headers['x-webhook-delivery'];
if (isProcessed(deliveryId)) {
console.log('Duplicate webhook, ignoring');
return res.status(200).send('OK');
}
// 3. Process webhook
const { event, data } = req.body;
try {
switch (event) {
case 'order.created':
handleOrderCreated(data);
break;
case 'order.updated':
handleOrderUpdated(data);
break;
case 'order.cancelled':
handleOrderCancelled(data);
break;
default:
console.warn('Unknown event type:', event);
}
// 4. Mark as processed
markProcessed(deliveryId);
// 5. Respond quickly
res.status(200).send('OK');
} catch (error) {
console.error('Error processing webhook:', error);
// Return 500 to trigger retry
res.status(500).json({ error: 'Processing failed' });
}
});
// Simple in-memory duplicate detection (use Redis in production)
const processedWebhooks = new Set();
function isProcessed(deliveryId) {
return processedWebhooks.has(deliveryId);
}
function markProcessed(deliveryId) {
processedWebhooks.add(deliveryId);
// Clean up old entries after 24 hours
setTimeout(() => processedWebhooks.delete(deliveryId), 24 * 60 * 60 * 1000);
}
app.listen(3000, () => {
console.log('Webhook receiver listening on port 3000');
});
The most important part of webhook security is verifying that the webhook actually came from the expected source. HMAC signatures solve this problem.
Here's a complete implementation:
// Server: Generate signature
function signWebhook(payload, secret) {
const hmac = crypto.createHmac('sha256', secret);
hmac.update(payload);
return hmac.digest('hex');
}
// Client: Verify signature
function verifyWebhook(payload, signature, secret) {
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
// Use timing-safe comparison to prevent timing attacks
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
// Middleware for webhook verification
function verifyWebhookMiddleware(req, res, next) {
const signature = req.headers['x-webhook-signature'];
const secret = process.env.WEBHOOK_SECRET;
if (!signature) {
return res.status(401).json({ error: 'Missing signature' });
}
// Get raw body (important: must be the exact bytes that were signed)
const payload = JSON.stringify(req.body);
if (!verifyWebhook(payload, signature, secret)) {
return res.status(401).json({ error: 'Invalid signature' });
}
next();
}
// Use the middleware
app.post('/webhooks/orders', verifyWebhookMiddleware, (req, res) => {
// Webhook is verified, safe to process
processWebhook(req.body);
res.status(200).send('OK');
});
// 1. Validate webhook source IP
const ALLOWED_IPS = ['192.0.2.1', '198.51.100.1'];
function validateIP(req, res, next) {
const ip = req.ip || req.connection.remoteAddress;
if (!ALLOWED_IPS.includes(ip)) {
return res.status(403).json({ error: 'Forbidden' });
}
next();
}
// 2. Check timestamp to prevent replay attacks
function validateTimestamp(req, res, next) {
const timestamp = req.headers['x-webhook-timestamp'];
const now = Date.now();
const webhookTime = new Date(timestamp).getTime();
// Reject webhooks older than 5 minutes
if (now - webhookTime > 5 * 60 * 1000) {
return res.status(401).json({ error: 'Webhook too old' });
}
next();
}
// 3. Rate limiting
const rateLimit = require('express-rate-limit');
const webhookLimiter = rateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes
max: 100, // Limit each IP to 100 requests per window
message: 'Too many webhook requests'
});
app.post('/webhooks/orders',
webhookLimiter,
validateIP,
validateTimestamp,
verifyWebhookMiddleware,
handleWebhook
);
Webhooks can fail for many reasons: network issues, server downtime, bugs. Good retry logic is essential.
class WebhookQueue {
constructor() {
this.queue = [];
this.processing = false;
}
async enqueue(webhook) {
this.queue.push({
...webhook,
attempts: 0,
nextRetry: Date.now()
});
if (!this.processing) {
this.process();
}
}
async process() {
this.processing = true;
while (this.queue.length > 0) {
const webhook = this.queue[0];
// Wait until it's time to retry
const now = Date.now();
if (webhook.nextRetry > now) {
await new Promise(resolve =>
setTimeout(resolve, webhook.nextRetry - now)
);
}
try {
await this.sendWebhook(webhook);
// Success: remove from queue
this.queue.shift();
} catch (error) {
webhook.attempts++;
if (webhook.attempts >= 5) {
// Max retries reached: move to dead letter queue
console.error('Webhook failed after 5 attempts:', webhook);
await this.moveToDeadLetterQueue(webhook);
this.queue.shift();
} else {
// Schedule retry with exponential backoff
const delay = Math.pow(2, webhook.attempts) * 1000;
webhook.nextRetry = Date.now() + delay;
console.log(`Webhook retry scheduled in ${delay}ms`);
}
}
}
this.processing = false;
}
async sendWebhook(webhook) {
const response = await fetch(webhook.url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': webhook.signature
},
body: JSON.stringify(webhook.payload),
timeout: 5000
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
}
async moveToDeadLetterQueue(webhook) {
// Store failed webhooks for manual review
await db.deadLetterQueue.insert({
...webhook,
failedAt: new Date()
});
}
}
Make sure webhook handlers are idempotent—they can be called multiple times with the same data without causing problems:
app.post('/webhooks/payment', async (req, res) => {
const { id, orderId, amount } = req.body.data;
// Use webhook ID as idempotency key
const existing = await db.payments.findOne({ webhookId: id });
if (existing) {
// Already processed this webhook
console.log('Duplicate webhook, skipping');
return res.status(200).send('OK');
}
// Process payment
await db.payments.insert({
webhookId: id,
orderId,
amount,
processedAt: new Date()
});
await fulfillOrder(orderId);
res.status(200).send('OK');
});
Webhooks can be tricky to debug because they involve two systems. Here are some tools and techniques:
class WebhookLogger {
async log(webhook, response, error = null) {
await db.webhookLogs.insert({
webhookId: webhook.id,
url: webhook.url,
event: webhook.event,
payload: webhook.payload,
response: {
status: response?.status,
body: response?.body
},
error: error?.message,
timestamp: new Date()
});
}
}
// Log all webhook attempts
const logger = new WebhookLogger();
async function sendWebhook(webhook) {
try {
const response = await fetch(webhook.url, {
method: 'POST',
body: JSON.stringify(webhook.payload)
});
await logger.log(webhook, response);
return response;
} catch (error) {
await logger.log(webhook, null, error);
throw error;
}
}
Use tools like ngrok to expose your local server for webhook testing:
# Install ngrok
npm install -g ngrok
# Expose local port 3000
ngrok http 3000
# Use the ngrok URL for webhook registration
# https://abc123.ngrok.io/webhooks/orders
Build a replay feature for debugging:
app.post('/admin/webhooks/:id/replay', async (req, res) => {
const log = await db.webhookLogs.findById(req.params.id);
if (!log) {
return res.status(404).json({ error: 'Webhook log not found' });
}
// Resend the webhook
const result = await sendWebhook({
url: log.url,
event: log.event,
payload: log.payload
});
res.json({
message: 'Webhook replayed',
result
});
});
Sometimes the best solution is both:
// Use webhooks for real-time notifications
app.post('/webhooks/order-created', async (req, res) => {
await processOrder(req.body.data);
res.status(200).send('OK');
});
// Poll periodically to catch any missed webhooks
cron.schedule('*/5 * * * *', async () => {
const lastCheck = await getLastPollTime();
const missedOrders = await fetchOrdersSince(lastCheck);
for (const order of missedOrders) {
await processOrder(order);
}
await updateLastPollTime();
});
This gives you the best of both worlds: real-time updates via webhooks, with polling as a safety net.
Choosing between webhooks and polling depends on your specific needs:
Use polling when:- You need to control timing - You're behind a firewall - The API doesn't support webhooks - Simplicity is more important than real-time updates
Use webhooks when:- You need real-time notifications - You want to reduce server load - You're building at scale - The API provider recommends it
And remember: - Always verify webhook signatures - Implement retry logic with exponential backoff - Make handlers idempotent - Log everything for debugging - Consider a hybrid approach for critical systems
Whether you choose webhooks, polling, or both, the key is to implement them thoughtfully with proper error handling, security, and monitoring.
