Alerting & On-Call

Configure effective alerting and on-call management for production systems.

When to Use This Skill

Use this skill when:

• Setting up alerting rules and thresholds
• Configuring on-call rotations and schedules
• Implementing alert routing and escalation
• Reducing alert fatigue
• Managing incident response workflows

Prerequisites

• Monitoring system (Prometheus, Datadog, etc.)
• On-call platform (PagerDuty, Opsgenie, Grafana OnCall)
• Communication channels (Slack, email)

Alerting Best Practices

Alert Categories

# Severity levels
critical:
  - Service completely down
  - Data loss imminent
  - Security breach
  response: Immediate page, wake people up

high:
  - Service degraded significantly
  - Error rate above SLO
  - Capacity near limit
  response: Page during business hours, notify after hours

medium:
  - Performance degradation
  - Non-critical component failure
  - Warning thresholds exceeded
  response: Notify via Slack, review next business day

low:
  - Informational alerts
  - Capacity planning triggers
  - Routine maintenance needed
  response: Email notification, weekly review

Alert Design Principles

# Good alert characteristics
alerts:
  actionable:
    - Every alert should require human action
    - Include runbook links
    - Clear remediation steps

  relevant:
    - Alert on symptoms, not causes
    - Focus on user impact
    - Avoid alerting on expected behavior

  timely:
    - Appropriate thresholds
    - Suitable evaluation windows
    - Account for normal variance

  unique:
    - No duplicate alerts
    - Proper alert grouping
    - Clear ownership

Prometheus Alerting

Alert Rules