# Agent Observability

Instrument AI agents with tracing, token metrics, latency, and cost visibility.

- Repository: https://github.com/BagelHole/DevOps-Security-Agent-Skills (1187 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-05-22
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -g -a claude-code -y`
Codex: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -g -a codex -y`
Cursor: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -g -a cursor -y`
Gemini CLI: `npx skills add BagelHole/DevOps-Security-Agent-Skills --skill agent-observability -g -a gemini-cli -y`

## Description

Instrument AI agents with tracing, token metrics, latency, and cost visibility. Use for reliability and debugging.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Agent Observability

Monitor AI agent behavior with logs, traces, metrics, and cost telemetry. This skill covers the full observability stack for LLM-powered applications: from raw Prometheus counters to Grafana dashboards, OpenTelemetry tracing, structured logging, cost tracking, SLO definition, and PII redaction.

---

## When to Use

Apply this skill whenever you operate:

- **Autonomous AI agents** that make multi-step tool calls (e.g., coding agents, support agents, data-pipeline agents).
- **LLM-backed APIs** serving chat completions, summarisation, or classification behind a REST or gRPC gateway.
- **RAG pipelines** where a retriever fetches context from a vector store before prompting a model.
- **Multi-agent orchestrations** (crew-style or graph-based) where several agents collaborate on a single task.
- **Batch inference jobs** that process thousands of prompts against a model endpoint.

Key signals that you need this skill:

1. You cannot answer "what is p95 latency for agent responses this week?"
2. You have no per-request cost attribution.
3. Debugging a bad agent response requires grepping raw application logs.
4. You have no alerting on token-usage spikes or elevated error rates.

---

## Core Metrics

Define these metrics at the application layer. All examples use the Prometheus client library naming conventions.

### Latency

```python
from prometheus_client import Histogram

# Total end-to-end latency for a full agent turn (user prompt -> final response)
AGENT_LATENCY = Histogram(
    "agent_request_duration_seconds",
    "End-to-end latency of an agent request",
    labelnames=["agent_name", "model", "status"],
    buckets=(0.25, 0.5, 1, 2, 5, 10, 30, 60, 120),
)

# Latency of a single LLM API call (one completion request)
LLM_CALL_LATENCY = Histogram(
    "llm_call_duration_seconds",
    "Latency of an individual LLM API call",
    labelnames=["model", "provider", "stream"],
    buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
)

The rest: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/HEAD/devops/ai/agent-observability/SKILL.md