# Search Benchmark

Generate a search quality benchmark for the AI Registry.

- Repository: https://github.com/agentic-community/mcp-gateway-registry (1032 stars)
- Category: AI and agents
- License: Apache-2.0
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -g -a claude-code -y`
Codex: `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -g -a codex -y`
Cursor: `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -g -a cursor -y`
Gemini CLI: `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -g -a gemini-cli -y`

## Description

Generate a search quality benchmark for the AI Registry. Generates ground truth from the registry's assets, runs 100+ queries against the semantic search API, evaluates results using NDCG@10/MRR/Recall, and produces a markdown report. Use when you want to measure search quality after changes to the scoring algorithm, embedding model, or indexed content.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Search Benchmark Skill

Measure semantic search quality against a deployed AI Registry. Generates a ground truth dataset from the registry's own assets, runs queries against the live API, evaluates results using standard information retrieval metrics (NDCG@10, MRR, Recall@10), and produces a markdown report.

## Prerequisites

1. **Registry URL** - The base URL of the deployed registry (e.g., `https://d2xl2zfuhgc4l0.cloudfront.net`)
2. **JWT Token** - A valid admin token in `.token` file (get from "Get JWT Token" button in registry UI)
3. **Registry must have assets indexed** - At least some servers, agents, or skills registered

The `.token` file supports both raw JWT format and the full JSON response from the registry UI.

## Input

```
/search-benchmark [REGISTRY_URL] [TOKEN_FILE]
```

- **REGISTRY_URL** - Base URL of the registry to benchmark (default: reads from user or uses `http://localhost`)
- **TOKEN_FILE** - Path to the token file (default: `.token`)

## Workflow

### Step 1: Check for Existing Ground Truth

Check if a ground truth dataset already exists:

```bash
ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/null
```

If the file exists, report how many queries it contains and ask the user: "A ground truth dataset already exists (N queries). Do you want to use it or generate a new one from this registry?"

- If **use existing**: skip to Step 2
- If **generate new**: proceed to Step 1b

### Step 1b: Generate Expert Ground Truth

This is NOT a simple programmatic generation. You must deeply analyze the registry's assets and craft queries like a search expert. Follow this process:

**1b.1: Fetch all assets as JSON**

```bash
TOKEN=$(python3 -c "
import json
with open('{TOKEN_FILE}') as f:
    raw = f.read().strip()
if raw.startswith('{'):
    data = json.loads(raw)
    print(data.get('tokens',{}).get('access_token') or data.get('access_token',''))
else:
    print(raw.replace('Bearer ',''))
")

The rest: https://github.com/agentic-community/mcp-gateway-registry/blob/HEAD/.claude/skills/search-benchmark/SKILL.md