Agent Benchmark
Framework for measuring and tracking agent response quality over time.
Install
npx skills add vibeeval/vibecosystem --skill agent-benchmark -g -a claude-code -y- Repository
- vibeeval/vibecosystem541
- Category
- AI and agents
- Updated
- 3mo ago
- License
- MIT
- Context cost
- Mediumabout 2,818 tokens
- Runs scripts
- No, instructions only
What it does
Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production. Use when evaluating agent changes, auditing quality, or establishing performance baselines.
Agent Benchmark Framework
Without benchmarks, we cannot know whether agent changes improve or degrade quality. This skill defines how to measure, track, and protect agent performance.
When to Activate
- Before and after modifying any agent definition file
- When adding a new skill that an agent depends on
- Periodic quality audits (weekly/monthly)
- When a user reports degraded agent output
- Before promoting an agent from experimental to production
Core Concepts
Why Benchmarks Matter
Agent quality degrades silently. A prompt tweak that improves one response can break ten others. Without a baseline to compare against, every change is a guess. Benchmarks make quality visible and regressions detectable.
Benchmark Types
| Type | Scope | Cost | Frequency |
|---|---|---|---|
| Prompt Benchmark | Single agent, single task | Low | Every agent change |
| Task Benchmark | End-to-end scenario | Medium | Feature changes |
| Regression Suite | All critical agents | High | Weekly / before release |
