Webhook Implementation and Reliability — Delivery, Retries, and Security
In this tutorial, you'll learn about Webhook Implementation and Reliability. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Webhooks are HTTP callbacks triggered by events in one system that send real-time notifications to another system, enabling event-driven communication between services without polling.
What You'll Learn
By the end of this tutorial, you will implement a reliable Webhook delivery system with retry logic, payload signing for security, idempotency keys, dead-letter queues, and delivery monitoring dashboards.
Why It Matters
Webhooks are the backbone of event-driven integrations. A single failed Webhook can mean a missed payment notification, an undelivered order confirmation, or a security alert that never arrives. Doda Browser uses Webhooks to notify extensions about browser events, and the Webhook system must deliver every notification with 99.99% reliability.
Real-World Use
When a customer completes a payment on Stripe, Stripe sends a payment_intent.succeeded Webhook event to the merchant's endpoint. The merchant's system processes the order, updates the database, and sends a confirmation email. If the Webhook is lost, the order is never fulfilled.
Delivery Architecture
sequenceDiagram
participant Producer as Event Source
participant Queue as Delivery Queue
participant Dispatcher
participant Endpoint as Consumer
participant DLQ as Dead Letter Queue
Producer->>Queue: Enqueue Webhook event
Queue->>Dispatcher: Dequeue for delivery
Dispatcher->>Endpoint: POST /Webhook (with signature)
alt Success 200
Endpoint-->>Dispatcher: 200 OK + idempotency
Dispatcher->>Queue: Acknowledge delivery
else Failure 5xx / Timeout
Dispatcher->>Queue: Schedule retry (exponential backoff)
Queue-->>DLQ: After max retries exhausted
end
Webhook events are queued for delivery, dispatched with retry logic, and moved to a dead-letter Queue after exhausting retries. Each delivery attempt includes a signature header for verification.
Python: Webhook Dispatcher
# Webhook_dispatcher.py
# Reliable Webhook delivery system
import hashlib
import hmac
import JSON
import time
import requests
from datetime import datetime, timedelta
from dataclasses import dataclass
from typing import Optional
@dataclass
class WebhookEvent:
"""Represents a Webhook event to be delivered."""
event_id: str
event_type: str
payload: dict
destination_URL: str
secret: str
created_at: datetime
class WebhookDispatcher:
"""Dispatches Webhooks with retry and delivery guarantees."""
def __init__(self, max_retries=5, BASE_delay=60):
self.max_retries = max_retries
self.BASE_delay = BASE_delay
def sign_payload(self, payload: dict, secret: str) -> str:
"""Create HMAC-SHA256 signature for payload verification."""
payload_bytes = JSON.dumps(payload, separators=(',', ':')).encode()
signature = hmac.new(
secret.encode(),
payload_bytes,
hashlib.sha256
).hexdigest()
return signature
def deliver(self, event: WebhookEvent) -> bool:
"""Attempt to deliver a Webhook with retry logic."""
signature = self.sign_payload(event.payload, event.secret)
headers = {
'Content-Type': 'application/JSON',
'X-Webhook-ID': event.event_id,
'X-Webhook-Signature': f'sha256={signature}',
'X-Webhook-Timestamp': str(int(event.created_at.timestamp())),
'User-Agent': 'DodaTech-Webhook/1.0',
}
for attempt in range(1, self.max_retries + 1):
try:
response = requests.post(
event.destination_URL,
JSON=event.payload,
headers=headers,
timeout=30,
)
if response.status_code == 200:
print(f"Webhook {event.event_id} delivered on attempt {attempt}")
return True
if response.status_code == 410:
print(f"Webhook {event.event_id}: endpoint gone (410)")
return False
print(f"Webhook {event.event_id}: got {response.status_code}, retrying")
except requests.Timeout:
print(f"Webhook {event.event_id}: timeout on attempt {attempt}")
except requests.ConnectionError:
print(f"Webhook {event.event_id}: connection error on attempt {attempt}")
if attempt < self.max_retries:
delay = self.BASE_delay * (2 ** (attempt - 1)) # Exponential backoff
print(f"Retrying in {delay}s (attempt {attempt + 1}/{self.max_retries})")
time.sleep(delay)
print(f"Webhook {event.event_id}: all retries exhausted, moving to DLQ")
return False
# Usage
dispatcher = WebhookDispatcher(max_retries=5, BASE_delay=60)
event = WebhookEvent(
event_id='evt_abc123',
event_type='order.created',
payload={'order_id': 'ORD-456', 'amount': 2999},
destination_URL='HTTPS://API.example.com/Webhooks/orders',
secret='whsec_yoursecret',
created_at=datetime.utcnow(),
)
dispatcher.deliver(event)
Node.js: Webhook Receiver
// Webhook-receiver.js
// Secure Webhook endpoint with signature verification
const Express = require('Express');
const crypto = require('crypto');
const app = Express();
app.use(Express.JSON({
verify: (req, res, buf) => {
req.rawBody = buf.toString();
},
}));
const Webhook_SECRETS = {
Stripe: 'whsec_Stripe_secret',
github: 'whsec_github_secret',
dodatech: 'whsec_dodatech_secret',
};
function verifySignature(req, res, next) {
const signature = req.headers['x-Webhook-signature'];
const eventId = req.headers['x-Webhook-id'];
const timestamp = req.headers['x-Webhook-timestamp'];
const source = req.headers['x-Webhook-source'];
if (!signature || !eventId || !timestamp || !source) {
return res.status(401).JSON({ error: 'Missing Webhook headers' });
}
const secret = Webhook_SECRETS[source];
if (!secret) {
return res.status(403).JSON({ error: 'Unknown Webhook source' });
}
// Reject Webhooks older than 5 minutes
const eventTime = parseInt(timestamp, 10);
if (Date.now() / 1000 - eventTime > 300) {
return res.status(400).JSON({ error: 'Stale Webhook event' });
}
// Verify HMAC signature
const expectedSig = crypto
.createHmac('sha256', secret)
.update(req.rawBody)
.digest('hex');
const receivedSig = signature.replace('sha256=', '');
if (!crypto.timingSafeEqual(Buffer.from(expectedSig), Buffer.from(receivedSig))) {
return res.status(401).JSON({ error: 'Invalid signature' });
}
req.webhookEvent = { id: eventId, source };
next();
}
// Webhook processing middleware
async function processWebhook(req, res) {
const { id: eventId, source } = req.webhookEvent;
const event = req.body;
console.log(`Processing Webhook ${eventId} from ${source}: ${event.type}`);
// Acknowledge immediately
res.status(200).JSON({ received: true, eventId });
// Process asynchronously (do not block the response)
setImmediate(async () => {
try {
await handleWebhookEvent(source, event);
console.log(`Webhook ${eventId} processed successfully`);
} catch (error) {
console.error(`Webhook ${eventId} processing failed:`, error);
// Store failed event for manual inspection
await storeFailedWebhook(eventId, source, event, error);
}
});
}
function handleWebhookEvent(source, event) {
switch (event.type) {
case 'order.created':
return fulfillOrder(event.data);
case 'payment.succeeded':
return updatePaymentStatus(event.data);
default:
console.log(`Unknown event type: ${event.type}`);
}
}
app.post('/Webhooks/orders', verifySignature, processWebhook);
app.listen(3000);
Idempotency Handling
# idempotency.py
# Webhook idempotency using Redis deduplication
import Redis
from datetime import timedelta
class IdempotencyGuard:
"""Prevent duplicate Webhook processing using event IDs."""
def __init__(self, Redis_client, ttl_hours=24):
self.Redis = Redis_client
self.ttl = timedelta(hours=ttl_hours)
def is_duplicate(self, event_id: str) -> bool:
"""Check if event was already processed."""
key = f"Webhook:processed:{event_id}"
# SET NX returns True if key was set (first time)
return not self.Redis.setnx(key, '1')
def mark_processed(self, event_id: str):
"""Mark event as processed with TTL."""
key = f"Webhook:processed:{event_id}"
self.Redis.expire(key, int(self.ttl.total_seconds()))
def Process_event(self, event_id: str, event_type: str, payload: dict):
"""Safely Process event with idempotency check."""
if self.is_duplicate(event_id):
print(f"Skipping duplicate Webhook: {event_id}")
return {'status': 'skipped', 'reason': 'duplicate'}
try:
# Process the event
result = self._handle_event(event_type, payload)
self.mark_processed(event_id)
return {'status': 'processed', 'result': result}
except Exception as e:
print(f"Error processing {event_id}: {e}")
raise
def _handle_event(self, event_type: str, payload: dict):
"""Actual event processing logic."""
handlers = {
'order.created': self._handle_order_created,
'payment.succeeded': self._handle_payment_succeeded,
'subscription.updated': self._handle_subscription_updated,
}
handler = handlers.get(event_type)
if not handler:
print(f"No handler for event type: {event_type}")
return None
return handler(payload)
R = Redis.Redis(host='localhost', port=6379, db=0)
guard = IdempotencyGuard(R)
Common Errors
1. Not Verifying Webhook Signatures
Accepting any POST to your Webhook endpoint allows attackers to send fake events. Always verify the HMAC signature using a shared secret. Use crypto.timingSafeEqual in Node.js or hmac.compare_digest in Python to prevent timing attacks.
2. Blocking the Webhook Response
Performing database writes, external API calls, or email sending inside the Webhook handler slows down the response. Return HTTP 200 immediately and Process the event asynchronously. Slow responses cause the sender to timeout and retry.
3. Not Handling Idempotency
Webhook senders may deliver the same event multiple times (at-least-once delivery). Without idempotency checks, duplicate Webhooks cause duplicate orders, duplicate charges, or duplicate emails. Use event IDs to deduplicate.
4. Ignoring Webhook Delivery Failures
A single failed delivery due to a network blip can lose a critical event. Implement retry with exponential backoff (1min, 2min, 4min, 8min, 16min), then move failed deliveries to a dead-letter Queue for manual inspection.
5. Using a Single Secret for All Sources
If a third-party service is compromised and your shared Webhook secret is exposed, an attacker can impersonate any Webhook source. Use unique secrets per Webhook source and rotate them regularly.
6. Not Logging Webhook Payloads
When a Webhook fails, you need to inspect the payload to understand what went wrong. Log Webhook requests (with sensitive data masked) to a searchable store like Elasticsearch for debugging.
Practice Questions
1. How does Webhook signature verification work?
The sender creates an HMAC-SHA256 hash of the request body using a shared secret and includes it in a header (X-<a href="/backend/webhooks/">Webhook</a>-Signature). The receiver recomputes the hash from the raw request body and compares it using a constant-time comparison function.
2. What is the difference between Webhooks and polling?
Webhooks push data immediately when an event occurs (event-driven). Polling checks for new data at regular intervals (time-driven). Webhooks are more efficient with lower latency, but require the receiver to have a publicly accessible endpoint.
3. How do you handle Webhooks when the receiver is down?
The sender should Queue undelivered Webhooks and retry with exponential backoff (Stripe retries for 3 days). The receiver should use a dead-letter Queue for events that fail after all retries, and alert the operations team.
4. What is a Webhook idempotency key?
An idempotency key is a unique identifier for each Webhook event. The receiver checks if it has already processed an event with the same key before processing it again. This prevents duplicate processing while allowing reliable at-least-once delivery.
5. Challenge: Build a Webhook delivery system that handles 10,000 events per minute with: HMAC-SHA256 signing, exponential backoff retry (3 attempts), dead-letter Queue for failed events, idempotency deduplication using Redis, a delivery monitoring dashboard showing success/failure rates and latency percentiles, and Webhook secret rotation without downtime.
Mini Project: Webhook Delivery Monitor
# Webhook_monitor.py
# Webhook delivery monitoring dashboard
from datetime import datetime, timedelta
from collections import defaultdict
class WebhookMonitor:
"""Track Webhook delivery metrics for monitoring."""
def __init__(self):
self.events = []
self.delivery_times = []
self.failures = defaultdict(list)
def record_delivery(self, event_id, destination, status_code,
latency_ms, attempt):
"""Record a Webhook delivery attempt."""
record = {
'event_id': event_id,
'destination': destination,
'status_code': status_code,
'latency_ms': latency_ms,
'attempt': attempt,
'timestamp': datetime.utcnow(),
}
self.events.append(record)
self.delivery_times.append(latency_ms)
if status_code >= 400:
self.failures[destination].append(record)
def summary(self):
"""Generate delivery statistics."""
if not self.events:
return {'status': 'no_data'}
total = len(self.events)
successes = sum(1 for e in self.events if e['status_code'] == 200)
failures_count = total - successes
success_rate = (successes / total) * 100
latencies = sorted(self.delivery_times)
p50 = latencies[len(latencies) // 2]
p95 = latencies[int(len(latencies) * 0.95)]
p99 = latencies[int(len(latencies) * 0.99)]
return {
'period': f"{self.events[0]['timestamp']} to {self.events[-1]['timestamp']}",
'total_deliveries': total,
'success_rate': f"{success_rate:.2f}%",
'failures': failures_count,
'latency_ms': {
'p50': p50,
'p95': p95,
'p99': p99,
},
'failed_endpoints': {
dest: len(fails)
for dest, fails in self.failures.items()
},
}
def generate_report(self):
"""Print a formatted delivery report."""
stats = self.summary()
print("=== Webhook Delivery Report ===")
print(f"Period: {stats['period']}")
print(f"Total: {stats['total_deliveries']}")
print(f"Success Rate: {stats['success_rate']}")
print(f"Failures: {stats['failures']}")
print(f"Latency P50: {stats['latency_ms']['p50']}ms")
print(f"Latency P95: {stats['latency_ms']['p95']}ms")
print(f"Latency P99: {stats['latency_ms']['p99']}ms")
if stats['failed_endpoints']:
print("\nFailed Endpoints:")
for dest, count in stats['failed_endpoints'].items():
print(f" {dest}: {count} failures")
monitor = WebhookMonitor()
monitor.record_delivery('evt_001', 'HTTPS://API.example.com/Webhooks', 200, 45, 1)
monitor.record_delivery('evt_002', 'HTTPS://API.example.com/Webhooks', 502, 30000, 3)
monitor.generate_report()
FAQ
Related Concepts
What's Next
You now understand Webhook implementation and reliability. Next, learn about webhook architecture for designing event-driven systems, then explore message Queue patterns for reliable event propagation.
- Practice daily — Add signature verification to an existing Webhook endpoint
- Build a project — Build a Webhook relay service that receives events, signs them, and forwards to multiple subscribers with retry guarantees
- Explore related topics — Check out Webhook event schemas and OpenAPI specifications for Webhook APIs
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro