AI SRE Incident Response

Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.

When to Use This Skill

• An LLM endpoint begins returning degraded or hallucinated answers
• Token spend spikes beyond budget thresholds
• A model provider goes down and traffic must fail over
• Safety guardrails fire at abnormal rates
• A new model deployment causes latency or accuracy regression

Prerequisites

• Prometheus and Alertmanager deployed with scrape targets for AI services
• Grafana dashboards for golden signals (latency, error rate, cost, quality)
• On-call rotation configured in PagerDuty, Opsgenie, or equivalent
• Runbook repository accessible to responders
• Rollback mechanism for model and prompt versions (GitOps or feature flags)

AI Incident Classes

• Availability incident: model/provider unavailable, timeout storm.
• Quality incident: answer accuracy or tool success drops below SLO.
• Safety incident: harmful or policy-violating outputs increase.
• Cost incident: unexpected token or provider spend spike.

Severity Framework

Severity	Criteria	Response Time	Notification
SEV1	User-facing outage, compliance risk, data leak	5 min	Page on-call + incident commander
SEV2	Major degradation in key flows	15 min	Page on-call
SEV3	Limited impact or internal-only issue	1 hour	Slack alert
SEV4	Cosmetic or low-priority regression	Next business day	Ticket

Golden Signals for AI Services

• Request success rate
• Latency (queue + generation + tool execution)
• Hallucination/groundedness proxy metrics
• Cost per minute and per tenant
• Guardrail violation rate

Prometheus Alert Rules