Incident Response Guide

  # Check application health
curl https://api.yourapp.com/health

# Check recent deployments
# Railway dashboard → Deployments

# Check error logs
railway logs --environment production | grep ERROR

# Check metrics
# CloudWatch → Dashboards → API Metrics

# Check database
# RDS → Monitoring → CPU, Connections
  

  Update: Identified issue with [component]
Started: ~10 minutes ago
Affected: [X users / all users / specific feature]
Investigating: [what you're checking]
  

  # If caused by recent deployment → ROLLBACK
railway rollback

# If database issue → Increase resources temporarily
# AWS RDS → Modify → Instance class

# If rate limiting → Increase limits temporarily

# If memory leak → Restart containers
railway restart

# If external API down → Enable fallback/cache
  

  # Check logs around incident start time
railway logs --since 30m | grep ERROR

# Check database for locks
SELECT * FROM pg_stat_activity WHERE state = 'active';

# Check memory/CPU usage trends
# CloudWatch → Metrics

# Check recent code changes
git log --since="2 hours ago" --oneline

# Test hypothesis locally
# Reproduce the issue if possible
  

  # Create hotfix branch
git checkout main
git pull
git checkout -b hotfix/fix-description

# Make fix
# ... edit code ...

# Test locally
pytest
npm test

# Commit and push
git add .
git commit -m "fix: [description]"
git push origin hotfix/fix-description

# Create PR with "HOTFIX" label
# Get 1 approval (instead of usual 2)
# Merge and deploy
  

  # Implement temporary workaround
# Document in code that it's temporary
# Create issue for proper fix
  

  # Add feature flag
if settings.ENABLE_NEW_FEATURE:
    return new_feature()
else:
    return old_feature()

# Set in environment
ENABLE_NEW_FEATURE=false
  

  # Check health endpoint
curl https://api.yourapp.com/health

# Check error rate
# Should return to normal

# Check metrics
# Response times, error rates

# Test affected functionality
# Manual testing or automated tests

# Monitor for 30 minutes
# Ensure no regressions
  

  ✅ RESOLVED: [Brief description]
Fix deployed: [deployment link]
Monitoring: Will monitor for 30 minutes
Root cause: [brief explanation]
  

  Initial:
🚨 INCIDENT: Login endpoint returning 500
Severity: 2
Time detected: 14:30 UTC
Status: Investigating

Update 1 (15 min):
Identified: Database connection pool exhausted
Action: Increasing pool size
ETA: 10 minutes

Update 2 (30 min):
Deployed: Pool size increased
Status: Monitoring
Error rate: Decreasing

Resolution (45 min):
✅ RESOLVED: Login endpoint healthy
Duration: 45 minutes
Root cause: Traffic spike + insufficient pool size
Next steps: Post-mortem scheduled
  

  Subject: Service Disruption - Resolved

Hi [stakeholders],

We experienced a brief service disruption today from 14:30-15:15 UTC affecting login functionality.

Issue: Some users unable to log in
Impact: ~10% of login attempts failed
Resolution: Database configuration updated
Status: Fully resolved, monitoring ongoing

We apologise for any inconvenience and have implemented measures to prevent recurrence.

If you have questions, please contact support@yourapp.com.
  

  🚨 INCIDENT

Description: [Brief description]
Severity: [1-4]
Started: [Time/duration]
Impact: [Who/what is affected]
Status: Investigating

Lead: @[your-name]
Updates: Every [15/30/60] minutes
  

  📊 UPDATE - [Time since start]

Status: [Investigating / Fixing / Testing / Resolved]
Findings: [What you've learned]
Actions: [What you're doing]
ETA: [When you expect resolution]
  

  ✅ RESOLVED - [Total duration]

Issue: [What was wrong]
Fix: [What was done]
Impact: [Summary of impact]
Prevention: [How we'll prevent this]

Post-mortem: [Link / date scheduled]
  

  # Post-Mortem: [Incident Title]

**Date**: [YYYY-MM-DD]
**Duration**: [Start time] - [End time] ([Duration])
**Severity**: [1-4]
**Impact**: [Summary of user impact]

## Timeline

- 14:30 - Issue started (database pool exhausted)
- 14:35 - Alert triggered, investigation began
- 14:45 - Root cause identified
- 14:50 - Fix deployed (increased pool size)
- 15:00 - Monitoring, errors decreasing
- 15:15 - Fully resolved

## Root Cause

Database connection pool size (10 connections) insufficient for traffic spike from marketing campaign. Pool exhausted, new requests failed.

## Impact

- **Users affected**: ~1,000 users (10% of login attempts)
- **Duration**: 45 minutes
- **Revenue impact**: Minimal (no orders lost)
- **Customer complaints**: 3 support tickets

## What Went Well

- Quick detection (5 minutes via automated alert)
- Fast rollback capability available
- Clear communication in Slack
- Fix deployed within 20 minutes

## What Went Wrong

- Insufficient load testing before campaign
- No alerts on connection pool utilization
- Manual intervention required (should be auto-scaling)

## Action Items

| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Add connection pool monitoring | @dev1 | 2025-01-20 | ✅ Done |
| Auto-scale pool based on traffic | @dev2 | 2025-01-25 | 🔄 In Progress |
| Load test before major campaigns | @dev1 | Ongoing | 📋 Process |
| Document runbook for pool issues | @dev3 | 2025-01-18 | ✅ Done |

## Lessons Learned

1. Monitor resource utilization proactively
2. Load test matches real-world patterns
3. Automated scaling reduces manual intervention
4. Clear communication helped quick resolution
  

  # Production logs
railway logs --environment production

# Database logs
# AWS RDS → Logs