Health Check Endpoints and Readiness Probes — Complete Implementation Guide
In this tutorial, you'll learn about Health Check Endpoints and Readiness Probes. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Health check endpoints are HTTP endpoints that report whether a service is running correctly, enabling Orchestration systems (Kubernetes) and load balancers to make automated decisions about routing traffic and restarting unhealthy instances.
What You'll Learn
By the end of this tutorial, you will implement liveness, readiness, and startup probes in Python (FastAPI) and Node.js (Express), check dependencies (database, Redis, external APIs), configure Kubernetes probes, and build a health monitoring dashboard.
Why It Matters
Kubernetes kills and restarts containers that fail liveness probes. Load balancers stop routing to instances that fail readiness probes. Without proper health checks, failures Go unnoticed, traffic is sent to dead instances, and downtime extends. Doda Browser uses granular health checks for each microservice.
Real-World Use
A database Connection Pool exhausts all connections. The readiness probe detects this and marks the pod as not ready. The load balancer stops sending traffic. The pod's connections drain, and the Connection Pool recovers. The readiness probe then passes, and traffic resumes. Users never saw a 502 error.
Health Check Architecture
graph TD
subgraph "Kubernetes"
K[Kubelet] --> L[/healthz/liveness]
K --> R[/healthz/readiness]
K --> S[/healthz/startup]
end
subgraph "Application"
L --> A[Is Process Alive?]
R --> B[Can Accept Traffic?]
S --> C[Is Startup Complete?]
B --> D[(Database)]
B --> E[(Redis)]
B --> F[External API]
end
subgraph "External"
LB[Load Balancer] --> R
end
style L fill:#4CAF50,color:#fff
style R fill:#f90,color:#fff
style S fill:#2196F3,color:#fff
Three probe types serve different purposes: liveness proves the Process is alive, readiness proves the service can handle traffic, and startup proves initialization is complete.
Python: FastAPI Health Checks
# health_check.py
# Comprehensive health check endpoints with FastAPI
import time
from datetime import datetime
from fastapi import FastAPI, APIRouter
from pydantic import BaseModel
from typing import Dict, Any
import asyncpg
import redis.asyncio as aioredis
app = FastAPI()
health_router = APIRouter()
# Application start time for uptime calculation
APP_START_TIME = time.time()
class HealthStatus(BaseModel):
status: str
version: str
uptime_seconds: float
timestamp: str
checks: Dict[str, Any]
class DependencyCheck:
"""Base class for dependency health checks."""
def __init__(self, name: str, critical: bool = True):
self.name = name
self.critical = critical
async def check(self) -> dict:
"""Run the health check. Override in subclasses."""
raise NotImplementedError
class DatabaseCheck(DependencyCheck):
"""Check database connectivity."""
def __init__(self, dsn: str):
super().__init__("database", critical=True)
self.dsn = dsn
async def check(self) -> dict:
start = time.time()
try:
conn = await asyncpg.connect(self.dsn)
version = await conn.fetchval("SELECT version()")
await conn.close()
return {
"status": "healthy",
"latency_ms": round((time.time() - start) * 1000, 2),
"version": version.split(",")[0],
}
except Exception as e:
return {
"status": "unhealthy",
"latency_ms": round((time.time() - start) * 1000, 2),
"error": str(e),
}
class RedisCheck(DependencyCheck):
"""Check Redis connectivity."""
def __init__(self, url: str):
super().__init__("redis", critical=False)
self.url = url
async def check(self) -> dict:
start = time.time()
try:
conn = aioredis.from_url(self.url)
pong = await conn.ping()
await conn.close()
return {
"status": "healthy" if pong else "unhealthy",
"latency_ms": round((time.time() - start) * 1000, 2),
}
except Exception as e:
return {
"status": "unhealthy",
"latency_ms": round((time.time() - start) * 1000, 2),
"error": str(e),
}
class ExternalAPICheck(DependencyCheck):
"""Check external API availability."""
def __init__(self, name: str, url: str):
super().__init__(name, critical=False)
self.url = url
async def check(self) -> dict:
import httpx
start = time.time()
try:
async with httpx.AsyncClient(timeout=5.0) as client:
response = await client.get(self.url)
return {
"status": "healthy" if response.is_success else "degraded",
"status_code": response.status_code,
"latency_ms": round((time.time() - start) * 1000, 2),
}
except Exception as e:
return {
"status": "unhealthy",
"latency_ms": round((time.time() - start) * 1000, 2),
"error": str(e),
}
# Dependency registry
health_checks = []
def register_check(check: DependencyCheck):
"""Register a health check."""
health_checks.append(check)
return check
# Register default checks
register_check(DatabaseCheck("postgresql://user:pass@localhost:5432/myapp"))
register_check(RedisCheck("redis://localhost:6379/0"))
register_check(ExternalAPICheck("payment-gateway", "https://api.payment.com/health"))
@health_router.get("/healthz/liveness", status_code=200)
async def liveness():
"""Liveness probe: is the process running? Always returns 200."""
return {"status": "alive", "uptime": int(time.time() - APP_START_TIME)}
@health_router.get("/healthz/readiness")
async def readiness():
"""Readiness probe: can the service handle traffic?"""
all_healthy = True
results = {}
for check in health_checks:
result = await check.check()
results[check.name] = result
if check.critical and result["status"] != "healthy":
all_healthy = False
status = "healthy" if all_healthy else "unhealthy"
return HealthStatus(
status=status,
version="2.0.0",
uptime_seconds=time.time() - APP_START_TIME,
timestamp=datetime.utcnow().isoformat(),
checks=results,
), 200 if all_healthy else 503
@health_router.get("/healthz/startup")
async def startup():
"""Startup probe: has initialization completed?"""
# Check if critical services are available
return await readiness()
app.include_router(health_router)
Node.js: Express Health Checks
// health-check.js
// Express health check endpoints
const express = require('express');
const router = express.Router();
const APP_START = Date.now();
class HealthRegistry {
constructor() {
this.checks = [];
}
register(name, checkFn, critical = true) {
this.checks.push({ name, checkFn, critical });
}
async runChecks() {
const results = {};
let allHealthy = true;
for (const { name, checkFn, critical } of this.checks) {
try {
const result = await checkFn();
results[name] = result;
if (critical && result.status !== 'healthy') {
allHealthy = false;
}
} catch (error) {
results[name] = { status: 'unhealthy', error: error.message };
if (critical) allHealthy = false;
}
}
return { allHealthy, results };
}
}
const healthRegistry = new HealthRegistry();
// Database check
healthRegistry.register('database', async () => {
const start = Date.now();
const { Pool } = require('pg');
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
try {
const client = await pool.connect();
const result = await client.query('SELECT version()');
client.release();
return {
status: 'healthy',
latencyMs: Date.now() - start,
version: result.rows[0].version.split(' ')[0],
};
} finally {
await pool.end();
}
});
// Redis check
healthRegistry.register('redis', async () => {
const start = Date.now();
const Redis = require('ioredis');
const redis = new Redis(process.env.REDIS_URL);
try {
const pong = await redis.ping();
return {
status: pong === 'PONG' ? 'healthy' : 'unhealthy',
latencyMs: Date.now() - start,
};
} finally {
redis.disconnect();
}
}, false); // Redis is non-critical
// External API check
healthRegistry.register('payment-gateway', async () => {
const start = Date.now();
const response = await fetch('https://api.payment.com/health', {
timeout: 5000,
});
return {
status: response.ok ? 'healthy' : 'degraded',
statusCode: response.status,
latencyMs: Date.now() - start,
};
}, false);
// Liveness probe
router.get('/healthz/liveness', (req, res) => {
res.json({ status: 'alive', uptime: Math.floor((Date.now() - APP_START) / 1000) });
});
// Readiness probe
router.get('/healthz/readiness', async (req, res) => {
const { allHealthy, results } = await healthRegistry.runChecks();
res.status(allHealthy ? 200 : 503).json({
status: allHealthy ? 'healthy' : 'unhealthy',
version: process.env.APP_VERSION || '1.0.0',
uptimeSeconds: Math.floor((Date.now() - APP_START) / 1000),
timestamp: new Date().toISOString(),
checks: results,
});
});
// Startup probe
router.get('/healthz/startup', async (req, res) => {
const { allHealthy, results } = await healthRegistry.runChecks();
res.status(allHealthy ? 200 : 503).json({ status: allHealthy ? 'ready' : 'starting', checks: results });
});
module.exports = { healthRouter: router, healthRegistry };
Kubernetes Probe Configuration
# kubernetes-probes.yaml
# Kubernetes deployment with all three probe types
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-backend
spec:
replicas: 3
selector:
matchLabels:
app: myapp-backend
template:
metadata:
labels:
app: myapp-backend
spec:
containers:
- name: app
image: myapp:latest
ports:
- containerPort: 8000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-secret
key: url
# Liveness probe: restart if app crashes or deadlocks
livenessProbe:
httpGet:
path: /healthz/liveness
port: 8000
initialDelaySeconds: 10
periodSeconds: 15
failureThreshold: 3
timeoutSeconds: 5
# Readiness probe: stop traffic if dependencies fail
readinessProbe:
httpGet:
path: /healthz/readiness
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 2
timeoutSeconds: 5
# Startup probe: delay liveness until initialization complete
startupProbe:
httpGet:
path: /healthz/startup
port: 8000
initialDelaySeconds: 3
periodSeconds: 5
failureThreshold: 30
timeoutSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
name: myapp-backend
spec:
selector:
app: myapp-backend
ports:
- port: 80
targetPort: 8000
Common Errors
1. Returning the Same Response for All Probes
Liveness, readiness, and startup probes serve different purposes and should return different information. Liveness should never depend on external services (it could cause cascading restarts). Readiness should check all critical dependencies. Startup should verify initialization.
2. Making Liveness Probes Dependent on External Services
If the database is down and the liveness probe fails, Kubernetes restarts the pod. But restarting wont fix the database. The pod enters a crash loop. Make liveness probes Process-only. Database failures should only affect readiness probes.
3. Not Setting Appropriate Timeouts
A health check that waits 30 seconds for a response blocks the kubelet from checking other pods. Set timeouts (3-5 seconds) for health checks. If a dependency does not respond within the timeout, treat it as unhealthy.
4. Checking Too Many Dependencies
Checking every downstream service, cache, Queue, and storage bucket in a single readiness probe makes the endpoint slow and fragile. Check critical dependencies only. LESS important services can have separate, non-critical checks.
5. No Startup Probe for Slow-Starting Applications
Applications that take 30+ seconds to start (loading models, warming caches) will be killed by the liveness probe before they finish starting. Use a startup probe with a high failureThreshold to give initialization time before liveness checks begin.
6. Not Excluding Health Check Endpoints from Metrics
Health check requests that fail due to dependency issues inflate error rate metrics and trigger false alarms. Exclude /healthz/* paths from error rate calculations in your monitoring system.
Practice Questions
1. What is the difference between liveness and readiness probes?
Liveness probes determine if the pod should be restarted (Process level). Readiness probes determine if the pod should receive traffic (dependency level). A pod can be alive but not ready (e.g., database connection lost). A dead pod is never ready.
2. Why should liveness probes not depend on external services?
If a database outage causes liveness probe failures, Kubernetes restarts all pods simultaneously. This makes the problem worse: all pods restart, but the database is still down. The system enters a crash loop. Liveness should be Process-only.
3. How does the startup probe prevent premature restarts?
The startup probe runs before liveness probes begin. It gives the application time to initialize (connect to databases, load models, warm caches). Once the startup probe passes, liveness probes take over with their shorter intervals.
4. What HTTP status code should a healthy readiness probe return?
200 (OK) for healthy, 503 (Service Unavailable) for unhealthy. Some implementations use 200 for everything and include the status in the body, but using the HTTP status code is more standard and allows infrastructure tools (load balancers, kubelet) to check without parsing the body.
5. Challenge: Design a health check system for a Microservices Architecture with: (1) a payment service that depends on database + Redis + external fraud API (2) an order service that depends on database + payment service (3) a notification service that depends on message Queue + email provider. Implement liveness, readiness, and startup probes for each service. The payment service should be "ready" only if database and Redis are healthy (fraud API failure reduces to degraded). The order service readiness checks its database and pings the payment service health endpoint. All probes must respond within 3 seconds. Include a health dashboard that aggregates all service statuses.
Mini Project: Health Dashboard
# health_dashboard.py
# Aggregate health check dashboard
import asyncio
import aiohttp
from datetime import datetime
class HealthDashboard:
"""Aggregate health status from multiple services."""
def __init__(self, services: dict):
self.services = services # {name: health_URL}
async def check_service(self, session: aiohttp.ClientSession, name: str, URL: str) -> dict:
"""Check a single service health endpoint."""
start = datetime.utcnow()
try:
async with session.get(URL, timeout=aiohttp.ClientTimeout(total=5)) as response:
data = await response.JSON()
return {
'service': name,
'status': data.get('status', 'unknown'),
'HTTP_status': response.status,
'checks': data.get('checks', {}),
'latency_ms': (datetime.utcnow() - start).total_seconds() * 1000,
'timestamp': start.isoformat(),
}
except asyncio.TimeoutError:
return {'service': name, 'status': 'timeout', 'error': '5s timeout', 'timestamp': start.isoformat()}
except Exception as e:
return {'service': name, 'status': 'error', 'error': str(e), 'timestamp': start.isoformat()}
async def check_all(self) -> list:
"""Check all services concurrently."""
async with aiohttp.ClientSession() as session:
tasks = [
self.check_service(session, name, URL)
for name, URL in self.services.items()
]
return await asyncio.gather(*tasks)
async def generate_report(self):
"""Generate a health report for all services."""
results = await self.check_all()
healthy = sum(1 for R in results if R.get('status') == 'healthy')
degraded = sum(1 for R in results if R.get('status') == 'degraded')
unhealthy = sum(1 for R in results if R.get('status') not in ('healthy', 'degraded'))
print("=== Health Dashboard ===")
print(f"Time: {datetime.utcnow().isoformat()}")
print(f"Healthy: {healthy}/{len(results)}")
print(f"Degraded: {degraded}/{len(results)}")
print(f"Unhealthy: {unhealthy}/{len(results)}")
print()
for R in results:
icon = {'healthy': '[OK]', 'degraded': '[!!]', 'timeout': '[!!]', 'error': '[!!]'}.get(R.get('status'), '[??]')
latency = R.get('latency_ms', 0)
print(f" {icon} {R['service']}: {R.get('status', 'unknown')} ({latency:.0f}ms)")
dashboard = HealthDashboard({
'payment-API': 'HTTP://payment-service:8000/healthz/readiness',
'order-API': 'HTTP://order-service:8000/healthz/readiness',
'notification-API': 'HTTP://notification-service:8000/healthz/readiness',
})
asyncio.run(dashboard.generate_report())
FAQ
Related Concepts
What's Next
You now understand health check endpoints and probes. Next, learn about graceful shutdown for managing pod termination, then explore backend logging patterns for structured log aggregation.
- Practice daily -- Add liveness, readiness, and startup probes to your backend application
- Build a project -- Build a health check aggregator dashboard that monitors multiple Microservices and alerts on status changes
- Explore related topics -- Check out Kubernetes pod lifecycle and Prometheus health check metrics
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro