Model Evaluation CLI
Creates and runs eval suites, compares model scores and adds eval runs to CI.
Install
npx skills add cheaper-inference/OmniRoute --skill cli-eval -g -a claude-code -y- Repository
- cheaper-inference/OmniRoute83.6k
- Category
- Testing
- Updated
- 4d ago
- License
- MIT
- Context cost
- Lightabout 1,270 tokens
- Runs scripts
- No, instructions only
What it does
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->
Overview
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
Quick install
npm install -g omniroute # or: npx omniroute
omniroute --version
Subcommands
eval
Example:
omniroute eval
eval suites
Example:
omniroute eval suites
eval list
Example:
omniroute eval list
eval get <suiteId>
Example:
omniroute eval get <suiteId>
eval create
Flags:
--file <path>
Example:
omniroute eval create
eval run <suiteId>
Flags:
-m, --model <id>--combo <name>--concurrency <n>--tag <tag>--watch
Example:
omniroute eval run <suiteId>
eval list
Flags:
--suite <id>--status <s>--since <ts>--limit <n>
Example:
omniroute eval list
eval get <runId>
Example:
omniroute eval get <runId>
eval results <runId>
Flags:
--failed
Example:
omniroute eval results <runId>
eval cancel <runId>
Flags:
--yes
Example:
omniroute eval cancel <runId>
eval scorecard <runId>
Example:
omniroute eval scorecard <runId>
simulate [prompt]
Flags:
--file <path>-m, --model <id>--combo <name>--reasoning-effort <level>--thinking-budget <n>--explain
Example:
omniroute simulate [prompt]
