Skip to content

Webhook Architecture & Best Practices — Delivery, Retries, Security

DodaTech Updated 2026-06-22 10 min read

In this tutorial, you'll learn about Webhook Architecture & Best Practices. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Webhook architecture is an event-driven communication pattern where a provider sends HTTP POST requests to a consumer's registered callback URL when specific events occur, enabling real-time notifications without polling.

What You'll Learn

By the end of this tutorial, you will design a production Webhook system with at-least-once delivery guarantees, exponential backoff retries, HMAC-SHA256 signature verification, idempotency handling, and consumer health monitoring.

Why It Matters

Webhooks power real-time integrations across the internet — payment notifications from Stripe, push events from GitHub, and delivery status from Twilio. Doda Browser receives real-time malware signature updates through Webhooks from threat intelligence partners, and Durga Antivirus Pro sends Webhook alerts to enterprise SIEM systems.

Real-World Use

An e-commerce platform registers a Webhook endpoint with its payment provider. When a payment succeeds, the provider POSTs the payment event to the endpoint. The platform updates the order status, sends a confirmation email, and triggers inventory updates — all without the platform polling for payment status.

Delivery Guarantees

flowchart LR
    P[Provider] -->|"Event: POST payload"| Q[Delivery Queue]
    Q -->|"Attempt 1 (0s)"| C[Consumer]
    Q -->|"Attempt 2 (60s)"| C
    Q -->|"Attempt 3 (300s)"| C
    Q -->|"Attempt 4 (3600s)"| C
    Q -->|"Max retries exceeded"| DL[Dead Letter Queue]
    style P fill:#f90,color:#fff
    style DL fill:#e74c3c,color:#fff

Building a Webhook Consumer with Node.js

// webhook-consumer.js
// Robust webhook consumer with signature verification and idempotency

const express = require('express');
const crypto = require('crypto');

const app = express();
const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET || 'dev-secret-change-me';

// Store processed webhook IDs (use Redis in production)
const processedIds = new Set();

app.post('/webhook', express.text({ type: 'application/json' }), (req, res) => {
  const payload = req.body;
  const signature = req.headers['x-webhook-signature'];
  const webhookId = req.headers['x-webhook-id'];
  const timestamp = req.headers['x-webhook-timestamp'];

  console.log(`[Webhook] Received: ${webhookId} at ${new Date().toISOString()}`);

  // 1. Verify signature
  if (!signature) {
    console.error('[Webhook] Missing signature header');
    return res.status(401).json({ error: 'Missing signature' });
  }

  const expectedSig = crypto
    .createHmac('sha256', WEBHOOK_SECRET)
    .update(`${timestamp}.${payload}`)
    .digest('hex');

  try {
    const receivedSigs = signature.split(' ').map(s => s.trim());
    const isValid = receivedSigs.some(s =>
      crypto.timingSafeEqual(Buffer.from(s), Buffer.from(expectedSig))
    );

    if (!isValid) {
      console.error('[Webhook] Invalid signature');
      return res.status(401).json({ error: 'Invalid signature' });
    }
  } catch (err) {
    return res.status(401).json({ error: 'Signature verification failed' });
  }

  // 2. Check timestamp freshness (prevent replay attacks)
  const eventTime = parseInt(timestamp, 10);
  const now = Math.floor(Date.now() / 1000);
  if (Math.abs(now - eventTime) > 300) {
    console.error('[Webhook] Stale timestamp — possible replay attack');
    return res.status(403).json({ error: 'Stale timestamp' });
  }

  // 3. Idempotency check
  if (processedIds.has(webhookId)) {
    console.log(`[Webhook] Duplicate ${webhookId} — already processed`);
    return res.status(200).json({ status: 'already_processed' });
  }

  processedIds.add(webhookId);
  const data = JSON.parse(payload);

  // 4. Process event asynchronously
  setImmediate(() => {
    processWebhookEvent(webhookId, data);
  });

  // Acknowledge immediately
  res.status(200).json({ status: 'ok' });
});

function processWebhookEvent(id, data) {
  const eventType = data.event || 'unknown';
  console.log(`[Webhook] Processing ${id}: ${eventType}`);

  switch (eventType) {
    case 'payment.succeeded':
      console.log(`  -> Updating order ${data.order_id} to paid`);
      break;
    case 'user.created':
      console.log(`  -> Sending welcome email to ${data.email}`);
      break;
    default:
      console.log(`  -> Unhandled event type: ${eventType}`);
  }
}

app.listen(8080, () => console.log('Webhook consumer on :8080'));

Expected behavior: The consumer verifies each Webhook's HMAC-SHA256 signature, checks the timestamp is within 5 minutes (preventing replay attacks), and deduplicates by Webhook ID. It acknowledges immediately (200 OK) and processes asynchronously, preventing provider timeouts.

Retry Strategy with Exponential Backoff

# webhook_retry.py
# Webhook delivery system with exponential backoff

import time
import hashlib
import hmac
import json
import requests
from datetime import datetime, timedelta

class WebhookDelivery:
    """Webhook delivery system with exponential backoff retries."""
    def __init__(self, secret, max_retries=5, base_delay=60):
        self.secret = secret
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.delivery_log = []

    def calculate_delay(self, attempt):
        """Calculate delay with exponential backoff and jitter."""
        delay = self.base_delay * (2 ** attempt)
        import random
        jitter = random.uniform(0, 0.1 * delay)
        return min(delay + jitter, 86400)  # Max 24 hours

    def sign_payload(self, payload, timestamp):
        """Sign payload with HMAC-SHA256."""
        message = f"{timestamp}.{payload}".encode()
        return hmac.new(
            self.secret.encode(),
            message,
            hashlib.sha256
        ).hexdigest()

    def deliver(self, url, event_data):
        """Deliver webhook with retry logic. Returns delivery status."""
        payload = json.dumps(event_data)
        webhook_id = hashlib.md5(
            f"{event_data['event']}{time.time()}".encode()
        ).hexdigest()[:16]

        for attempt in range(self.max_retries):
            timestamp = str(int(time.time()))
            signature = self.sign_payload(payload, timestamp)

            headers = {
                'Content-Type': 'application/json',
                'X-Webhook-ID': webhook_id,
                'X-Webhook-Signature': signature,
                'X-Webhook-Timestamp': timestamp,
            }

            try:
                response = requests.post(
                    url,
                    data=payload,
                    headers=headers,
                    timeout=10
                )

                delivery = {
                    'webhook_id': webhook_id,
                    'attempt': attempt + 1,
                    'status_code': response.status_code,
                    'timestamp': datetime.now().isoformat(),
                }
                self.delivery_log.append(delivery)

                if response.status_code == 200:
                    print(f"[OK] Attempt {attempt + 1}: Delivered {webhook_id}")
                    return True

                print(f"[RETRY] Attempt {attempt + 1}: Got {response.status_code}")

            except requests.exceptions.RequestException as e:
                print(f"[ERROR] Attempt {attempt + 1}: {e}")

            # Wait before next retry
            if attempt < self.max_retries - 1:
                delay = self.calculate_delay(attempt)
                print(f"  Waiting {delay:.0f}s before retry...")
                time.sleep(delay)

        print(f"[FAILED] All {self.max_retries} attempts exhausted for {webhook_id}")
        return False

delivery = WebhookDelivery(secret='whsec_abc123', max_retries=3, base_delay=5)
result = delivery.deliver(
    'https://httpbin.org/post',
    {'event': 'order.shipped', 'order_id': 'ORD-1234', 'status': 'shipped'}
)
print(f"\nFinal result: {'Delivered' if result else 'Failed'}")

Expected output:

[OK] Attempt 1: Delivered abc123...
Final result: Delivered

The system retries with delays of 5s, 10s, 20s (attempts 2, 3). If all 3 attempts fail, it logs the failure. The jitter prevents thundering herd when multiple Webhooks are retried simultaneously.

Security Best Practices

// Webhook-security.js
// Webhook security: signature verification, IP allowlisting, Rate Limiting

const Express = require('Express');
const crypto = require('crypto');

const app = Express();

const CONFIG = {
  secret: Process.env.Webhook_SECRET || 'whsec_...',
  allowedIps: Process.env.ALLOWED_IPS
    ? new Set(Process.env.ALLOWED_IPS.split(','))
    : new Set(['52.0.0.0/8', '54.0.0.0/8']),  // Stripe's IP ranges
  maxPayloadBytes: 1024 * 100,  // 100KB max
  toleranceSeconds: 300,
};

function verifyHmac(payload, sigHeader, secret) {
  const parts = {};
  sigHeader.split(',').forEach(pair => {
    const [k, v] = pair.split('=');
    parts[k] = v;
  });

  const timestamp = parts.t;
  const sigs = parts.v1 ? parts.v1.split(' ') : [];

  // Check for timing-safe equality
  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${timestamp}.${payload}`)
    .digest('hex');

  return sigs.some(sig => {
    try {
      return crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected));
    } catch {
      return false;
    }
  });
}

app.post('/Webhook', Express.text({ limit: '100kb', type: '*/*' }), (req, res) => {
  const payload = req.body;
  const sigHeader = req.headers['Stripe-signature'];
  const webhookId = req.headers['Webhook-id'];

  // Verify payload size
  if (Buffer.byteLength(payload, 'utf8') > CONFIG.maxPayloadBytes) {
    return res.status(413).JSON({ error: 'Payload too large' });
  }

  // Verify signature
  if (!verifyHmac(payload, sigHeader, CONFIG.secret)) {
    return res.status(401).JSON({ error: 'Invalid signature' });
  }

  console.log(`[Secure Webhook] Verified and accepted: ${webhookId}`);
  res.status(200).JSON({ received: true });
});

app.listen(8081);

Expected behavior: The secure consumer verifies HMAC signatures using timing-safe comparison, enforces payload size limits, and checks timestamp freshness. IP allowlisting adds a defense-in-depth layer but should never replace signature verification.

Common Errors

1. Not Verifying Webhook Signatures

Any client can POST to your Webhook URL. Without signature verification, attackers can trigger fake events — processing phantom payments, creating fake user accounts, or triggering false alerts. Always verify HMAC signatures using cryptographic comparison (not string equality).

2. Processing Webhooks Synchronously

If your Webhook handler takes longer than the provider's timeout (typically 5-10 seconds), the provider retries. This creates duplicate processing. Always acknowledge immediately (200 OK) and Process asynchronously using a task Queue.

3. Missing Idempotency Handling

Webhooks are delivered with at-least-once guarantees. Network failures and provider retries mean the same event can arrive multiple times. Without idempotency, you might charge a customer twice, send duplicate emails, or create duplicate records. Use Webhook ID as an idempotency key.

4. Ignoring Timestamp Verification

Without timestamp checks, an attacker can capture a valid Webhook payload and replay it hours later. Check that the timestamp is within a tolerance window (typically 5 minutes). Combined with signature verification, this prevents replay attacks.

5. Not Handling Consumer Downtime Gracefully

When your consumer is down, Webhooks are queued and retried. When it comes back up, it receives a burst. Without proper Queue management and Rate Limiting, the consumer is overwhelmed and crashes again. Implement graceful startup and consumer-side Rate Limiting.

6. Logging Sensitive Payload Data

Webhook payloads often contain PII, payment details, or API keys. Logging the full payload to plain-text files or sending it to error tracking services exposes sensitive data. Log only Webhook IDs, event types, and processing status — never full payloads.

Practice Questions

1. Why is HMAC-SHA256 preferred over simple API keys for Webhook verification?

HMAC-SHA256 proves the payload was sent by someone who knows the shared secret AND was not modified in transit. A simple API key in a header authenticates the sender but does not verify payload integrity.

2. What is exponential backoff and why is it used for Webhook retries?

Exponential backoff doubles the delay between each retry attempt (60s, 120s, 240s). It prevents retry storms when many consumers are down simultaneously and gives the consumer time to recover before the next attempt.

3. How do you prevent replay attacks on Webhooks?

Include a timestamp in the signed payload and reject Webhooks with timestamps outside a tolerance window (typically 5 minutes). Store recently seen Webhook IDs to detect exact replays within the window.

4. What status code should a Webhook consumer return?

Return 200 OK immediately after receiving and verifying the Webhook. Return non-2xx status only when you want the provider to retry (rate limited, internal error). Never return non-200 for business logic failures — acknowledge and Process asynchronously.

Challenge

Design a Webhook system where: (1) the provider signs each Webhook with HMAC-SHA256 using a rotating secret (changed every 90 days), (2) the consumer supports two simultaneous secrets during rotation (old and new) for zero-downtime secret rotation, (3) the consumer checks idempotency via Redis with a 24-hour TTL on processed IDs, (4) failed deliveries after 5 retries are sent to a dead letter Queue and trigger an alert, and (5) the consumer has a rate limiter that rejects excess traffic with 429 to prevent overload after downtime.

Mini Project: Webhook CLI Testing Tool

#!/usr/bin/env python3
# Webhook_test_cli.py
# CLI tool to send test Webhooks to a consumer endpoint

import argparse
import hmac
import hashlib
import JSON
import requests
import time
import uuid

def send_Webhook(URL, secret, event_type, payload, Webhook_id=None):
    """Send a signed test Webhook to the specified URL."""
    if Webhook_id is None:
        Webhook_id = str(uuid.uuid4())

    data = JSON.dumps({
        "id": Webhook_id,
        "event": event_type,
        "data": payload,
        "timestamp": int(time.time()),
    })

    timestamp = str(int(time.time()))
    signature = hmac.new(
        secret.encode(),
        f"{timestamp}.{data}".encode(),
        hashlib.sha256
    ).hexdigest()

    headers = {
        "Content-Type": "application/JSON",
        "X-Webhook-ID": Webhook_id,
        "X-Webhook-Signature": signature,
        "X-Webhook-Timestamp": timestamp,
    }

    print(f"Sending Webhook {Webhook_id} to {URL}")
    print(f"  Event: {event_type}")
    print(f"  Payload: {JSON.dumps(payload)}")

    try:
        response = requests.post(URL, data=data, headers=headers, timeout=10)
        print(f"  Response: HTTP {response.status_code}")
        print(f"  Body: {response.text}")
        return response.status_code == 200
    except requests.exceptions.RequestException as e:
        print(f"  Error: {e}")
        return False

def main():
    parser = argparse.ArgumentParser(description='Webhook testing CLI')
    parser.add_argument('URL', help='Consumer Webhook URL')
    parser.add_argument('--secret', default='test-secret', help='Shared secret')
    parser.add_argument('--event', default='test.ping', help='Event type')
    parser.add_argument('--payload', default='{"message": "test"}', help='JSON payload')

    args = parser.parse_args()

    payload = JSON.loads(args.payload)
    success = send_Webhook(args.URL, args.secret, args.event, payload)
    sys.exit(0 if success else 1)

if __name__ == '__main__':
    import sys
    main()

Expected behavior: Run with python <a href="/backend/webhooks/">Webhook</a>_test_cli.py http://localhost:8080/<a href="/backend/webhooks/">Webhook</a> --event payment.succeeded --payload '{"order_id":"123","amount":29.99}'. The tool sends a properly signed Webhook and displays the consumer's response status and body.

Congratulations on completing this Webhook architecture tutorial! Next, explore message Queue patterns for asynchronous processing of Webhook events, then learn about API Gateway patterns for managing Webhook provider connections.

  • Practice daily — Set up a Webhook consumer with ngrok and test with Stripe or GitHub test Webhooks
  • Build a project — Build a Webhook relay service that receives, verifies, and routes Webhooks to multiple internal services with independent retry policies
  • Explore related topics — Check out Svix for managed Webhook delivery and Standard Webhooks specification for industry-standard Webhook formats

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

← Previous GraphQL vs REST Deep Dive — When to Use Each with Real Examples Next → Middleware Patterns in Node.js and Python — Express and Django

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Backend