Open-source, production-ready incident runbooks for cloud engineering teams.
Browse, adopt, and contribute runbooks that help on-call engineers resolve incidents faster.
Runbook Hub is a community-maintained collection of structured incident runbooks for the most common infrastructure and application failures. Each runbook provides a step-by-step path from alert to resolution — covering triage, mitigation, verification, and escalation.
Every runbook is:
- Copy-pasteable — all commands use standard tools (
kubectl,psql,aws,redis-cli) - Environment-agnostic — generic placeholders so any team can adopt them
- Battle-tested — follows a consistent 11-section structure designed for 3 AM incidents
- AI-ready — structured for both human engineers and AI-powered incident response agents like CloudThinker
| Runbook | Severity | Difficulty |
|---|---|---|
| 5xx Error Rate Spike | Critical | Medium |
| API P99 Latency SLA Breach | Critical | Medium |
| Deployment Rollback | High | Medium |
| Celery Worker Queue Backlog | High | Medium |
| SQS Dead Letter Queue Growing | High | Medium |
| Application Memory Leak | High | Hard |
| Runbook | Severity | Difficulty |
|---|---|---|
| PostgreSQL High CPU Usage | High | Medium |
| PostgreSQL Long-Running Queries | High | Medium |
| Redis Memory Pressure | High | Medium |
| PostgreSQL Replication Lag | Critical | Hard |
| PostgreSQL Connection Pool Exhaustion | Critical | Hard |
| Runbook | Severity | Difficulty |
|---|---|---|
| Pod CrashLoopBackOff | Medium | Easy |
| HPA Max Replicas Reached | Medium | Easy |
| PVC Pending / Storage Full | High | Medium |
| OOMKilled Pod Recovery | High | Medium |
| High CPU Usage (EKS) | High | Medium |
| Node Not Ready | Critical | Hard |
| Runbook | Severity | Difficulty |
|---|---|---|
| SSL/TLS Certificate Expiry | High | Easy |
| DNS Resolution Failure | Critical | Medium |
| ALB 502/504 Errors | Critical | Medium |
| Runbook | Severity | Difficulty |
|---|---|---|
| Credential / Secret Leak Response | Critical | Hard |
| Suspicious API Activity | High | Hard |
| Runbook | Severity | Difficulty |
|---|---|---|
| Idle Resource Cleanup | Low | Easy |
| Unexpected Cloud Spend Spike | Medium | Medium |
| Right-sizing Over-provisioned Instances | Low | Hard |
1. Fork or clone this repository into your organization.
2. Find & replace the generic placeholders with your environment values:
| Placeholder | Replace With |
|---|---|
<your-grafana-url> |
Your Grafana instance URL |
<your-monitoring-tool> |
Your monitoring tool (Datadog, New Relic, Prometheus, etc.) |
@your-team |
Your team's handle or on-call alias |
@your-escalation-contact |
Your escalation contact or manager |
your-domain.com |
Your organization's domain |
your-cluster-prod |
Your production cluster name |
${DB_NAME} |
Your database name |
service-a, service-b |
Your actual service names |
3. Link to your alerting system using the alert reference table below.
Map these alert names to runbooks in your monitoring tool for one-click access during incidents.
Every runbook follows a consistent 11-section format. Use the template when writing new ones.
| Section | Purpose |
|---|---|
| Metadata | Service, owner, severity, last updated |
| Summary | One-line description of the incident type |
| Impact | What breaks when this happens |
| Prerequisites | Tools, access, and permissions needed |
| Triage & Diagnosis | How to confirm and scope the problem |
| Mitigation Steps | Step-by-step resolution procedures |
| Verification | How to confirm the fix worked |
| Rollback | How to undo changes if mitigation fails |
| Escalation | Who to contact and when |
| Related Runbooks | Links to related incident types |
| Changelog | History of updates to the runbook |
We welcome runbooks that encode real-world operational expertise.
- Fork this repository
- Copy
templates/runbook-template.mdinto the appropriate domain directory - Write your runbook following the 11-section structure
- Open a pull request with the
runbooklabel
Read CONTRIBUTING.md for the quality checklist and review process.
- MongoDB replication issues
- Kafka consumer lag
- Elasticsearch cluster health
- GitHub Actions workflow failures
- Terraform state lock issues
- Docker registry storage full
- gRPC deadline exceeded errors
- AWS Lambda cold start optimization
MIT — see LICENSE
Built with care by CloudThinker