Graceful Shutdown and Signal Handling — Zero-Downtime Application Stops
In this tutorial, you'll learn about Graceful Shutdown and Signal Handling. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Graceful shutdown is the Process of stopping an application in an orderly manner by completing in-flight requests, closing open connections, releasing resources, and notifying load balancers before the Process exits.
What You'll Learn
By the end of this tutorial, you will implement graceful shutdown in Python (uvicorn, gunicorn) and Node.js (Express, HTTP server), handle OS signals (SIGTERM, SIGINT), drain connection pools, and configure Kubernetes preStop hooks for zero-downtime deployments.
Why It Matters
Killing a server Process without graceful shutdown drops active HTTP requests, corrupts database transactions, leaves file handles open, and causes Connection Pool exhaustion on clients. Doda Browser's sync service must complete in-progress file transfers before shutting down, or users lose partially synced data.
Real-World Use
A Kubernetes pod is terminated during a rolling update. Without graceful shutdown, the pod stops accepting requests immediately but the load balancer still routes traffic to it, causing 502 errors. With graceful shutdown, the pod deregisters from the load balancer, completes active requests within a timeout, then exits.
Shutdown Sequence
sequenceDiagram
participant K8s as Kubernetes / OS
participant App as Application
participant LB as Load Balancer
participant DB as Database
participant Queue as Message Queue
K8s->>App: SIGTERM
App->>App: Start shutdown timer (30s)
App->>LB: Deregister / stop health check OK
App->>App: Stop accepting new requests
App->>App: Drain in-flight requests (max 10s)
App->>DB: Close Connection Pool
App->>Queue: Acknowledge pending messages
App->>App: Clean up resources (files, temp)
App-->>K8s: Process exits (0)
The application receives SIGTERM, enters shutdown mode, stops accepting new traffic, completes active work, closes dependencies, and exits within the timeout window.
Python: FastAPI Graceful Shutdown
# graceful_shutdown.py
# Graceful shutdown with FastAPI and uvicorn
import asyncio
import signal
import sys
from contextlib import asynccontextmanager
from FastAPI import FastAPI
from typing import Optional
class ShutdownManager:
"""Manages graceful shutdown sequence."""
def __init__(self, timeout_seconds=30):
self.timeout = timeout_seconds
self._shutdown_event = asyncio.Event()
self._active_requests = 0
self._is_shutting_down = False
async def request_started(self):
"""Track a new incoming request."""
self._active_requests += 1
async def request_finished(self):
"""Track a completed request."""
self._active_requests -= 1
if self._is_shutting_down and self._active_requests == 0:
self._shutdown_event.set()
async def wait_for_drain(self):
"""Wait for active requests to complete or timeout."""
if self._active_requests > 0:
print(f"Draining {self._active_requests} active requests...")
try:
await asyncio.wait_for(
self._shutdown_event.wait(),
timeout=self.timeout
)
except asyncio.TimeoutError:
print(f"Drain timeout after {self.timeout}s, forcing shutdown")
else:
print("No active requests to drain")
async def shutdown_sequence(self):
"""Execute the full shutdown sequence."""
print("Starting graceful shutdown...")
self._is_shutting_down = True
# Wait for active requests to complete
await self.wait_for_drain()
# Close database connections
print("Closing database connections...")
await self.close_db()
# Close message Queue connections
print("Closing message Queue connections...")
await self.close_Queue()
# Release other resources
print("Cleaning up resources...")
await self.cleanup_resources()
print("Shutdown complete")
self._shutdown_event.set()
async def close_db(self):
"""Close database Connection Pool gracefully."""
# Simulated database close
await asyncio.sleep(0.1)
print(" Database pool closed")
async def close_Queue(self):
"""Close message Queue connection."""
# Simulated Queue close
await asyncio.sleep(0.1)
print(" Message Queue disconnected")
async def cleanup_resources(self):
"""Clean up temporary files and resources."""
# Simulated cleanup
print(" Temporary files removed")
shutdown_manager = ShutdownManager(timeout_seconds=30)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan with signal handling."""
print("Application starting...")
yield
print("Application shutting down...")
await shutdown_manager.shutdown_sequence()
app = FastAPI(lifespan=lifespan)
@app.get("/health")
async def health_check():
"""Health endpoint that reports shutdown status."""
if shutdown_manager._is_shutting_down:
return {"status": "shutting_down", "active_requests": shutdown_manager._active_requests}
return {"status": "healthy"}
@app.post("/Process")
async def Process_data(data: dict):
"""Simulate a long-running request."""
await shutdown_manager.request_started()
try:
# Simulate processing
await asyncio.sleep(2)
return {"status": "processed", "data": data}
finally:
await shutdown_manager.request_finished()
# Handle OS signals
def handle_signal(sig, frame):
"""Handle SIGTERM/SIGINT by setting shutdown event."""
print(f"Received signal {sig}, initiating shutdown...")
shutdown_manager._is_shutting_down = True
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
Node.js: Express Graceful Shutdown
// graceful-shutdown.js
// Express server with graceful shutdown handling
const express = require('express');
const http = require('http');
const app = express();
const server = http.createServer(app);
class GracefulShutdown {
constructor(server, options = {}) {
this.server = server;
this.timeout = options.timeout || 30000;
this.isShuttingDown = false;
this.activeConnections = new Set();
server.on('connection', (conn) => {
this.activeConnections.add(conn);
conn.on('close', () => {
this.activeConnections.delete(conn);
});
});
}
async shutdown(signal) {
if (this.isShuttingDown) return;
this.isShuttingDown = true;
console.log(`Received ${signal}. Starting graceful shutdown...`);
// 1. Stop accepting new connections
server.close(() => {
console.log('HTTP server closed, no new connections');
});
// 2. Set a forced shutdown timer
const forceTimeout = setTimeout(() => {
console.error(`Forced shutdown after ${this.timeout}ms`);
process.exit(1);
}, this.timeout);
// 3. Drain active connections
console.log(`Draining ${this.activeConnections.size} active connections...`);
for (const conn of this.activeConnections) {
conn.end();
}
// 4. Close database connection pool
try {
await this.closeDatabase();
} catch (err) {
console.error('Database close error:', err);
}
// 5. Close Redis connections
try {
await this.closeRedis();
} catch (err) {
console.error('Redis close error:', err);
}
clearTimeout(forceTimeout);
console.log('Graceful shutdown complete');
process.exit(0);
}
async closeDatabase() {
// Database connection pool draining
// In production, call: await prisma.$disconnect()
console.log('Closing database connections...');
return new Promise((resolve) => setTimeout(resolve, 500));
}
async closeRedis() {
// Redis client disconnect
console.log('Closing Redis connections...');
return new Promise((resolve) => setTimeout(resolve, 200));
}
}
const shutdown = new GracefulShutdown(server, { timeout: 30000 });
// Middleware to reject requests during shutdown
app.use((req, res, next) => {
if (shutdown.isShuttingDown) {
res.set('Connection', 'close');
return res.status(503).json({
error: 'Server is shutting down',
retryAfter: 5,
});
}
next();
});
app.get('/health', (req, res) => {
res.json({
status: shutdown.isShuttingDown ? 'shutting_down' : 'healthy',
uptime: process.uptime(),
});
});
app.get('/process', async (req, res) => {
// Simulate processing
await new Promise((resolve) => setTimeout(resolve, 2000));
res.json({ processed: true });
});
// Handle OS signals
process.on('SIGTERM', () => shutdown.shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown.shutdown('SIGINT'));
server.listen(3000, () => {
console.log('Server listening on port 3000');
});
Kubernetes Integration
# deployment.yaml
# Kubernetes deployment with preStop hook for graceful shutdown
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-backend
spec:
replicas: 3
selector:
matchLabels:
app: myapp-backend
template:
metadata:
labels:
app: myapp-backend
spec:
containers:
- name: app
image: myapp:latest
ports:
- containerPort: 3000
env:
- name: SHUTDOWN_TIMEOUT
value: "30"
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- |
echo "preStop hook: deregistering from load balancer"
# Wait briefly for the endpoint controller to remove this pod
sleep 5
echo "preStop hook: sending SIGTERM to application"
kill -TERM 1
readinessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 15
periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
name: myapp-backend
spec:
selector:
app: myapp-backend
ports:
- port: 80
targetPort: 3000
Common Errors
1. Zero-Second Termination Grace Period
If terminationGracePeriodSeconds is too short (default 30s), the pod is killed before the application finishes draining. Set the grace period to at least 2x the expected max request duration. Monitor actual shutdown times.
2. Not Deregistering from Load Balancer
Without a preStop hook, the load balancer continues routing traffic to a terminating pod until the endpoint controller removes it (up to 30s). Add a 5-second sleep in the preStop hook before sending SIGTERM.
3. Hardcoding Shutdown Timeout
Different environments (dev, staging, production) have different request durations. Make the shutdown timeout configurable via environment variables. Production typically needs 30-60 seconds; dev environments can use 5-10 seconds.
4. Ignoring Readiness Probe Failures
If readiness probes fail before shutdown, the pod is removed from the service immediately. Ensure the application continues returning healthy readiness checks during the drain phase, and only fail the probe after the drain timeout expires.
5. Not Handling Database Pool Draining
Killing a Process with open database connections causes the database to keep connections in CLOSE_WAIT State until the TCP timeout. Close the Connection Pool gracefully using pool.close() or prisma.$disconnect() during shutdown.
6. Losing Message Queue Acknowledgements
If a worker processes a message from RabbitMQ or SQS and is killed before acknowledging it, the message goes back to the Queue. Implement shutdown hooks that finish processing the current message and acknowledge it before exiting.
Practice Questions
1. What is the difference between SIGTERM and SIGKILL?
SIGTERM (signal 15) is a polite termination request that the application can catch and handle. SIGKILL (signal 9) terminates the Process immediately and cannot be caught. Kubernetes sends SIGTERM first, then SIGKILL after the termination grace period expires.
2. How does the preStop hook work in Kubernetes?
The preStop hook runs before the container receives SIGTERM. It gives the container a chance to deregister from load balancers, notify monitoring systems, or complete a critical task. The hook must complete within terminationGracePeriodSeconds.
3. What is connection draining and why is it important?
Connection draining allows in-flight requests to complete before the server stops. The server stops accepting new requests but continues processing active ones. Once all requests complete (or the timeout expires), the server shuts down.
4. How do you handle WebSocket connections during shutdown?
Close WebSocket connections gracefully by sending a close frame with a reason code. The client receives the close event and can reconnect to another server instance. Set a shorter drain timeout for WebSocket connections than HTTP requests.
5. Challenge: Design a graceful shutdown system for a microservice that: (1) handles multiple connection types (HTTP, WebSocket, gRPC) with different drain priorities (2) deregisters from service discovery (Consul/Kubernetes) before draining (3) drains database Connection Pool (4) flushes pending logs to the logging pipeline (5) acknowledges in-flight message Queue messages (6) completes within a 45-second Kubernetes termination grace period. Include fallback behavior for connections that do not complete in time.
Mini Project: Shutdown-Aware HTTP Client
# shutdown_aware_client.py
# HTTP client that respects server shutdown signals
import asyncio
import aiohttp
from datetime import datetime
class ShutdownAwareClient:
"""HTTP client that handles server shutdown gracefully."""
def __init__(self, BASE_URL: str):
self.BASE_URL = BASE_URL
self.session: aiohttp.ClientSession = None
self.retry_delay = 1
async def __aenter__(self):
self.session = aiohttp.ClientSession()
return self
async def __aexit__(self, *args):
await self.session.close()
async def send_request(self, endpoint: str, data: dict, max_retries=3):
"""Send request with retry for shutdown scenarios."""
for attempt in range(max_retries):
try:
async with self.session.post(
f"{self.BASE_URL}{endpoint}",
JSON=data,
timeout=aiohttp.ClientTimeout(total=30),
) as response:
if response.status == 503:
# Server is shutting down
retry_after = response.headers.get('Retry-After', 5)
print(f"Server shutting down, retrying in {retry_after}s")
await asyncio.sleep(int(retry_after))
continue
response.raise_for_status()
return await response.JSON()
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
print(f"Attempt {attempt + 1} failed: {e}")
if attempt < max_retries - 1:
await asyncio.sleep(self.retry_delay * (2 ** attempt))
else:
raise
raise RuntimeError("Max retries exceeded")
async def main():
"""Simulate client requests during server shutdown."""
async with ShutdownAwareClient("HTTP://localhost:3000") as client:
tasks = [
client.send_request("/Process", {"task": f"task-{i}"})
for i in range(5)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
for i, result in enumerate(results):
if isinstance(result, Exception):
print(f"Task {i} failed: {result}")
else:
print(f"Task {i} succeeded: {result}")
asyncio.run(main())
FAQ
Related Concepts
What's Next
You now understand graceful shutdown and signal handling. Next, learn about health check endpoints for readiness and liveness probes, then explore environment configuration for managing shutdown timeouts per environment.
- Practice daily — Add SIGTERM handling to your current backend server
- Build a project — Build a graceful shutdown test harness that sends SIGTERM to a server while running load tests
- Explore related topics — Check out Kubernetes pod lifecycle and container lifecycle hooks
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro