# AI Sre Incident Response

Build AI-focused SRE incident response practices for LLM outages, degraded quality...

- Repository: https://github.com/BagelHole/DevOps-Security-Agent-Skills (1187 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-05-22
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response -g -a claude-code -y`
Codex: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response -g -a codex -y`
Cursor: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response -g -a cursor -y`
Gemini CLI: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response -g -a gemini-cli -y`

## Description

Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# AI SRE Incident Response

Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.

## When to Use This Skill

- An LLM endpoint begins returning degraded or hallucinated answers
- Token spend spikes beyond budget thresholds
- A model provider goes down and traffic must fail over
- Safety guardrails fire at abnormal rates
- A new model deployment causes latency or accuracy regression

## Prerequisites

- Prometheus and Alertmanager deployed with scrape targets for AI services
- Grafana dashboards for golden signals (latency, error rate, cost, quality)
- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
- Runbook repository accessible to responders
- Rollback mechanism for model and prompt versions (GitOps or feature flags)

## AI Incident Classes

- **Availability incident**: model/provider unavailable, timeout storm.
- **Quality incident**: answer accuracy or tool success drops below SLO.
- **Safety incident**: harmful or policy-violating outputs increase.
- **Cost incident**: unexpected token or provider spend spike.

## Severity Framework

| Severity | Criteria | Response Time | Notification |
|----------|----------|---------------|--------------|
| SEV1 | User-facing outage, compliance risk, data leak | 5 min | Page on-call + incident commander |
| SEV2 | Major degradation in key flows | 15 min | Page on-call |
| SEV3 | Limited impact or internal-only issue | 1 hour | Slack alert |
| SEV4 | Cosmetic or low-priority regression | Next business day | Ticket |

## Golden Signals for AI Services

- Request success rate
- Latency (queue + generation + tool execution)
- Hallucination/groundedness proxy metrics
- Cost per minute and per tenant
- Guardrail violation rate

## Prometheus Alert Rules

The rest: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/HEAD/devops/ai/ai-sre-incident-response/SKILL.md