Incident Management

Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.

When to Use

• Establishing incident management processes for production systems
• Defining severity levels and escalation procedures
• Running war rooms and coordinating incident response
• Conducting blameless post-incident reviews
• Building on-call schedules and notification workflows
• Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)

Severity Levels

severity_definitions:
  SEV1_critical:
    impact: "Complete service outage or data breach affecting all/most customers"
    examples:
      - Production site completely down
      - Data breach confirmed or suspected
      - Complete loss of a critical business function
      - Security incident with active exploitation
    response_time: "Immediate (within 5 minutes)"
    update_frequency: "Every 15-30 minutes"
    who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
    communication: "Status page update, customer email, executive notification"
    resolution_target: "< 1 hour to mitigate"

  SEV2_major:
    impact: "Major feature broken or severe degradation affecting many customers"
    examples:
      - Key feature completely non-functional
      - Significant performance degradation (>5x latency)
      - Data processing pipeline completely stalled
      - Partial outage affecting a region or segment
    response_time: "Within 15 minutes"
    update_frequency: "Every 30-60 minutes"
    who_is_paged: "On-call engineer, engineering manager"
    communication: "Status page update if customer-facing"
    resolution_target: "< 4 hours to mitigate"