# Research Observability

Add file-based observability, tracing, and run analysis to LLM/agent experiments, ...

- Repository: https://github.com/Leanmcp/gateway-skills (2122 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-07
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add Leanmcp/gateway-skills --skill research-observability -g -a claude-code -y`
Codex: `npx skills add Leanmcp/gateway-skills --skill research-observability -g -a codex -y`
Cursor: `npx skills add Leanmcp/gateway-skills --skill research-observability -g -a cursor -y`
Gemini CLI: `npx skills add Leanmcp/gateway-skills --skill research-observability -g -a gemini-cli -y`

## Description

Add file-based observability, tracing, and run analysis to LLM/agent experiments, per-turn transcripts, tool calls and results, reasoning traces, token/cost accounting, per-episode artifacts, plus terminal viewers and a run-comparison report. Platform-agnostic: one small adapter covers OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, vLLM, or any custom inference stack, and the same tooling then works unchanged. Use this skill whenever the user wants observability, tracing, logging, or run tracking for model experiments; asks why an agent failed a task or wants to replay/debug/diff an episode; wants to compare models, arms, or ablations; wants to instrument a new folder, repo, or platform ("add observability here", "do the same for Tinker/Gemini/OpenAI"); wants to import or tie in existing experiment results so old runs show up alongside new ones; or wants eval runner scripts where many models share one protocol. Reach for this even when they only say "log what the model is doing", "I can't tell what happened in this run", or "set up traces for this experiment".

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Research observability

Experiments you cannot replay are experiments you have to rerun. This skill
installs a small, file-based observability layer over any LLM/agent experiment
and the tools to read it: every generation, every tool call and result, every
reasoning trace, every reward, on disk as plain JSONL, greppable, diffable, and
re-gradable months later.

The design bet: **the recording format is fixed and platform-independent, so
adding a platform is writing one ~15-line normalizer, not re-plumbing
observability.** LiteLLM, OpenAI, Gemini, Anthropic, Tinker and a bespoke
inference server all produce identical files, and every viewer, report and diff
downstream works with no further changes.

## Start here

**Adding observability to a project.** Follow
[references/integration.md](references/integration.md) end to end, it is a
seven-step procedure, starting with a survey of the repo (which platform, where
the generations happen, where results go today) and ending with the existing
runs imported so old and new sit in the same viewer. Do the survey before
writing code; the answers determine every later choice.

**A platform not yet wired.** [references/platforms.md](references/platforms.md)
has working recipes for OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, and
vLLM/self-hosted, plus the 15-line template for anything else and the
framework-level patterns (subclass the model class, or use the callback API).

**Understanding or extending the format.**
[references/data-model.md](references/data-model.md) is the contract: the
directory layout, every field, both sequence counters and why there are two, and
the concurrency rules.

**Reading and comparing runs.** [references/viewers.md](references/viewers.md)
covers the three tools, the diffing recipe, plotting, and how to add
domain-specific panels without changing the trace format.

The rest: https://github.com/Leanmcp/gateway-skills/blob/HEAD/skills/research-observability/SKILL.md