Creating MCP Agent Evals
Use when adding Langfuse MCP agent evals for a tool family of the Apify MCP server...
Install
npx skills add apify/apify-mcp-server --skill creating-mcp-agent-evals -g -a claude-code -y- Repository
- apify/apify-mcp-server14.2k
- Category
- AI and agents
- Updated
- 4d ago
- License
- MIT
- Context cost
- Mediumabout 2,469 tokens
- Runs scripts
- No, instructions only
What it does
Use when adding Langfuse MCP agent evals for a tool family of the Apify MCP server ("create evals for the storage tools"), when eval cases fail and you must decide whether the case, the tool, or its description is at fault, or when eval runs show tool errors in Langfuse traces.
Creating MCP agent evals for an MCP tool family
Overview
Build a small, calibrated Langfuse eval suite for one tool family (tasks, storage, runs, …), then use its failures to fix the tools. Core principle: evals are designed from user intent, never from tool descriptions, the eval defines what should work; descriptions get fixed afterward to make naive agents pass it.
Commands, item shapes, probe patterns, and sweep queries: reference.md.
