Incident Response: Handling Production Outages

# incident_commander.py
class Incident:
    def __init__(self, severity, service, description):
        self.severity = severity
        self.service = service
        self.description = description
        self.timeline = []
        self.ic = None
        self.responders = []
        self.status = "detected"

    def assign_ic(self, name):
        self.ic = name
        self.timeline.append(f"IC assigned: {name}")

    def add_responder(self, name, role):
        self.responders.append({"name": name, "role": role})
        self.timeline.append(f"Responder added: {name} ({role})")

    def update_status(self, status, message):
        self.status = status
        self.timeline.append(f"[{status.upper()}] {message}")

    def summary(self):
        print(f"=== Incident Summary ===")
        print(f"Severity: {self.severity}")
        print(f"Service: {self.service}")
        print(f"Description: {self.description}")
        print(f"IC: {self.ic}")
        print(f"Status: {self.status}")
        for entry in self.timeline:
            print(f"  {entry}")

incident = Incident("SEV-1", "api-gateway", "All requests returning 502")
incident.assign_ic("Alice")
incident.add_responder("Bob", "Networking")
incident.add_responder("Carol", "Backend")
incident.update_status("mitigating", "Rolling back last deployment")
incident.update_status("resolved", "Rollback complete, traffic restored")
incident.summary()

# INCIDENT NOTIFICATION
# --------------------
# Severity: SEV-1
# Service: api-gateway
# Start Time: 2024-06-20 10:30 UTC
# Summary: All requests returning 502
# Impact: 100% of users affected
# IC: @alice
# Responders: @bob (networking), @carol (backend)
# Status: Investigation ongoing
# Updates: Every 15 minutes in #incident-api-gateway

# RESOLUTION POSTMORTEM
# --------------------
# What happened: v2.3.1 deployment included a misconfigured upstream
# Root cause: Config change in nginx.conf removed proxy_pass directive
# Detection: Datadog alert "Gateway Error Rate > 1%" fired
# Resolution: Rolled back to v2.3.0
# Time to detection: 2 minutes
# Time to resolution: 12 minutes

# PagerDuty best practices:
# 1. Create schedules with follow-the-sun rotation
#    - US team: 9am-5pm EST
#    - EU team: 9am-5pm CET
#    - APAC team: 9am-5pm JST
# 2. Set up escalation policies:
#    - Level 1: Primary on-call (15 min)
#    - Level 2: Secondary on-call (15 min)
#    - Level 3: Engineering manager
# 3. Configure service integrations:
#    - Prometheus Alertmanager → PagerDuty webhook
#    - Datadog → PagerDuty direct integration

# Alertmanager config for PagerDuty
route:
  receiver: pagerduty-prod
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: pagerduty-prod
    pagerduty_configs:
      - routing_key: "<your-pagerduty-integration-key>"
        severity: "{{ .CommonLabels.severity }}"

# Runbook: api-gateway

## Symptoms
- 502 Bad Gateway
- Latency > 5s
- Connection refused

## Initial Checks
1. Check Datadog dashboard: "API Gateway Overview"
2. Check upstream service health
3. Check recent deployments (last 1 hour)

## Common Fixes
### Issue: All 502 errors
1. Identify last deployment: `kubectl rollout history deployment/api-gateway`
2. Rollback: `kubectl rollout undo deployment/api-gateway`
3. Verify: `curl -I https://api.example.com/health`

### Issue: High latency
1. Check upstream response times
2. Scale replicas: `kubectl scale deployment/api-gateway --replicas=10`
3. Check database connection pool

## Escalation
- Private: Slack #incident-api-gateway
- PagerDuty API on-call phone number

# Postmortem Template

## Incident Summary
- Date: 2024-06-20
- Duration: 12 minutes
- Severity: SEV-1
- Services affected: api-gateway

## Timeline
- 10:30 UTC - Alert fired: Gateway error rate > 5%
- 10:31 UTC - IC assigned: @alice
- 10:32 UTC - Identified as deployment rollback candidate
- 10:38 UTC - Rollback initiated
- 10:42 UTC - Traffic restored, all healthy

## Root Cause
A config change in nginx.conf (deployed in v2.3.1) removed the proxy_pass
directive. All requests hit NGINX but had no upstream to forward to.

## Action Items
| Action | Owner | Severity |
|--------|-------|----------|
| Add automated nginx config validation to CI pipeline | @bob | P0 |
| Add "proxy_pass present" check to deployment health check | @carol | P0 |
| Runbook: add step for verifying upstream after deploy | @alice | P1 |

## What Went Well
- Alert fired within 2 minutes of deployment
- Rollback completed in 6 minutes
- Communication stayed in the designated channel

## What Went Wrong
- No validation step in CI for nginx configs
- Deployment health check didn't verify upstream connectivity
- On-call had to manually identify the problematic deployment