Research Observability
Add file-based observability, tracing, and run analysis to LLM/agent experiments, ...
Install
npx skills add Leanmcp/gateway-skills --skill research-observability -g -a claude-code -y- Repository
- Leanmcp/gateway-skills2.1k
- Category
- Testing
- Updated
- 28d ago
- License
- MIT
- Context cost
- Mediumabout 1,634 tokens
- Runs scripts
- Yes, review before use
What it does
Add file-based observability, tracing, and run analysis to LLM/agent experiments, per-turn transcripts, tool calls and results, reasoning traces, token/cost accounting, per-episode artifacts, plus terminal viewers and a run-comparison report. Platform-agnostic: one small adapter covers OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, vLLM, or any custom inference stack, and the same tooling then works unchanged. Use this skill whenever the user wants observability, tracing, logging, or run tracking for model experiments; asks why an agent failed a task or wants to replay/debug/diff an episode; wants to compare models, arms, or ablations; wants to instrument a new folder, repo, or platform ("add observability here", "do the same for Tinker/Gemini/OpenAI"); wants to import or tie in existing experiment results so old runs show up alongside new ones; or wants eval runner scripts where many models share one protocol. Reach for this even when they only say "log what the model is doing", "I can't tell what happened in this run", or "set up traces for this experiment".
Research observability
Experiments you cannot replay are experiments you have to rerun. This skill installs a small, file-based observability layer over any LLM/agent experiment and the tools to read it: every generation, every tool call and result, every reasoning trace, every reward, on disk as plain JSONL, greppable, diffable, and re-gradable months later.
The design bet: the recording format is fixed and platform-independent, so adding a platform is writing one ~15-line normalizer, not re-plumbing observability. LiteLLM, OpenAI, Gemini, Anthropic, Tinker and a bespoke inference server all produce identical files, and every viewer, report and diff downstream works with no further changes.
Start here
Adding observability to a project. Follow references/integration.md end to end, it is a seven-step procedure, starting with a survey of the repo (which platform, where the generations happen, where results go today) and ending with the existing runs imported so old and new sit in the same viewer. Do the survey before writing code; the answers determine every later choice.
A platform not yet wired. references/platforms.md has working recipes for OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, and vLLM/self-hosted, plus the 15-line template for anything else and the framework-level patterns (subclass the model class, or use the callback API).
Understanding or extending the format. references/data-model.md is the contract: the directory layout, every field, both sequence counters and why there are two, and the concurrency rules.
Reading and comparing runs. references/viewers.md covers the three tools, the diffing recipe, plotting, and how to add domain-specific panels without changing the trace format.
