# Kermt Monitor

Check progress for a detached KERMT run (pretrain, finetune, or any kermt_run_detached...

- Repository: https://github.com/NVIDIA/skills (3812 stars)
- Category: DevOps
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add NVIDIA/skills --skill kermt-monitor -g -a claude-code -y`
Codex: `npx skills add NVIDIA/skills --skill kermt-monitor -g -a codex -y`
Cursor: `npx skills add NVIDIA/skills --skill kermt-monitor -g -a cursor -y`
Gemini CLI: `npx skills add NVIDIA/skills --skill kermt-monitor -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add NVIDIA/skills`

## Description

Check progress for a detached KERMT run (pretrain, finetune, or any kermt_run_detached invocation). Reads run.json, queries docker for container state, tails the pretrain/finetune log, and parses progress lines (epoch, step, val loss).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# kermt-monitor

Companion skill for any KERMT workflow that runs detached: the three pretrain
skills (`kermt-continue-pretrain`, `kermt-pretrain-scratch`,
`kermt-add-cmim-pretrain`) plus `kermt-finetune`. `kermt-infer` and
`kermt-embed` run blocking by default and don't need this skill, but if a
user launches them detached on purpose the monitor still works (the
workflow-dispatch in step 4 handles unknown workflows by tailing the
most-recent log file in the run dir). Reads the run directory's `run.json`,
queries docker for the container's state, surfaces the latest progress,
and either tails or follows the log.

## Hardware requirements

None. This skill only reads disk + queries docker; no GPU compute.

## Inputs

One of:

- `<run-dir>`, a positional argument pointing at the directory containing
  `run.json` (e.g. `runs/continue-pretrain_2026-05-17T10-23Z`). Preferred.
- `--container <name-or-id>`, direct container reference; the skill still
  reads `run.json` from the run dir referenced inside the container's
  inspect output if available, but works degraded-mode without it.

Optional:

- `--lines N`, number of trailing log lines to print (default 50).
- `--follow`, stream `docker logs -f` until ^C. Useful for "watch the
  loss". Without it, the skill is one-shot and exits.
- `--json`, emit a structured status report instead of human-readable text.
  Useful when the parent agent wants to take downstream action.

## Workflow

Let `RUN_DIR=$1` (or whatever path the user supplies).

1. **Locate the manifest.**
   ```
   MANIFEST=$RUN_DIR/run.json
   ```
   Refuse to proceed if it doesn't exist; surface a helpful message
   pointing the user at the run-dir convention (`runs/<workflow>_<ts>/`).

The rest: https://github.com/NVIDIA/skills/blob/HEAD/skills/bionemo-kermt-monitor/SKILL.md