The Incident MTTR Playbook: From Alert to Resolution

Understanding MTTR

Mean Time to Restore (MTTR) is the average time it takes to recover from a production incident. Lower MTTR means faster recovery, less customer impact, and better reliability.

The Incident Lifecycle

1. Detection (Target: < 5 minutes)

How quickly do you know something is wrong?

• Automated monitoring and alerting
• Customer reports
• Internal discovery

2. Response (Target: < 15 minutes)

How quickly does someone start investigating?

• On-call engineer notified
• Incident acknowledged
• Initial assessment begun

3. Diagnosis (Target: < 30 minutes)

How quickly can you identify the root cause?

• Log analysis
• Metric correlation
• System health checks

4. Resolution (Target: < 2 hours)

How quickly can you restore service?

• Apply fix or workaround
• Verify restoration
• Monitor for recurrence

5. Recovery (Target: < 4 hours)

How quickly can you return to normal operations?

• Service fully restored
• Customers notified
• Monitoring confirmed stable

The Incident Response Framework

Severity Levels

Sev-1: Critical

• Complete service outage
• Data loss or corruption
• Security breach
• Response time: Immediate
• Target MTTR: < 1 hour

Sev-2: High

• Partial service degradation
• Critical feature unavailable
• Affecting multiple customers
• Response time: < 30 minutes
• Target MTTR: < 4 hours

Sev-3: Medium

• Minor feature issues
• Workaround available
• Affecting few customers
• Response time: < 2 hours
• Target MTTR: < 24 hours

Incident Roles

Incident Commander

Responsibilities:

• Coordinate response efforts
• Make key decisions
• Communicate with stakeholders
• Ensure documentation
• Call in additional resources

Technical Lead

Responsibilities:

• Lead technical investigation
• Coordinate with engineering teams
• Implement fixes
• Verify resolution

Communications Lead

Responsibilities:

• Update status page
• Notify customers
• Brief executives
• Handle external communications

The Runbook Template

Service Overview

• What does this service do?
• What are its dependencies?
• What services depend on it?

Common Issues

For each common issue:

• Symptoms: How to recognize it
• Diagnosis: Where to look
• Resolution: Step-by-step fix
• Prevention: How to avoid it

Monitoring & Alerts

• Key metrics to watch
• Dashboard links
• Log locations
• Alert thresholds

Emergency Procedures

• How to restart the service
• How to rollback deployment
• How to scale resources
• How to enable maintenance mode

Escalation Paths

Level 1: On-Call Engineer

• First responder
• Follows runbooks
• Escalates if needed

Level 2: Senior Engineer

• Deep system knowledge
• Complex diagnosis
• Escalates to team lead

Level 3: Engineering Lead

• Architecture decisions
• Cross-team coordination
• Escalates to VP if needed

Communication Templates

Initial Alert

Subject: [Sev-X] [Service Name] Incident

We are investigating reports of [issue description].
Our team is actively working on resolution.

Status: Investigating
Impact: [describe customer impact]
Next update: [time]

Update

Subject: [Sev-X] [Service Name] Update

We have identified [root cause].
We are implementing [solution].

Status: Resolving
Impact: [current status]
ETA: [estimated resolution time]
Next update: [time]

Resolution

Subject: [Resolved] [Service Name] Incident

The incident has been resolved.
Service is fully operational.

Root cause: [brief explanation]
Resolution: [what was done]
Prevention: [what we're doing to prevent recurrence]

Post-mortem: [link] (available within 48 hours)

Blameless Post-Mortems

Post-Mortem Template

Incident Summary

• Date and time
• Duration
• Severity
• Services affected
• Customer impact

Timeline

• When was it detected?
• When did investigation start?
• When was root cause identified?
• When was it resolved?
• Key actions taken

Root Cause Analysis

• What happened?
• Why did it happen?
• Why wasn't it caught earlier?
• What was the blast radius?

Action Items

For each action item:

• Description
• Owner
• Due date
• Priority
• Type (fix, monitoring, documentation, etc.)

What Went Well

• What helped us respond quickly?
• What processes worked?
• What tools were helpful?

What Can Improve

• What slowed us down?
• What gaps did we discover?
• What would we do differently?

Reducing MTTR: Strategies

1. Better Observability

• Comprehensive logging
• Distributed tracing
• Real-user monitoring
• Application performance monitoring

2. Faster Detection

• Proactive monitoring
• Synthetic checks
• Anomaly detection
• Customer impact alerts

3. Rapid Diagnosis

• Centralized logging
• Pre-built dashboards
• Correlation between metrics
• Historical comparison tools

4. Quick Resolution

• Automated rollback
• Feature flags
• Chaos engineering
• Regular game days

Automation Opportunities

Auto-Remediation

• Automatic service restarts
• Auto-scaling on load
• Automatic traffic rerouting
• Self-healing systems

ChatOps

• Incident creation from Slack
• Status updates in team channels
• Deploy commands from chat
• Query logs and metrics from chat

MTTR Metrics Dashboard

Track these metrics:

• Average MTTR (by severity)
• Time to detect
• Time to acknowledge
• Time to resolve
• Number of incidents (by severity)
• Repeat incidents
• Post-mortem action items completed

Key Takeaways

• MTTR is a key reliability metric
• Clear roles and responsibilities accelerate response
• Runbooks are essential for consistent resolution
• Blameless post-mortems drive continuous improvement
• Automation and observability reduce MTTR
• Regular practice (game days) builds muscle memory

Remember: The goal isn't zero incidents—it's fast recovery and continuous learning. Every incident is an opportunity to improve your systems and processes.