Skills

Kermt Monitor

Check progress for a detached KERMT run (pretrain, finetune, or any kermt_run_detached...

Install

npx skills add NVIDIA/skills --skill kermt-monitor -g -a claude-code -y
Repository
NVIDIA/skills3.8k
Category
DevOps
Updated
5d ago
License
Apache-2.0
Context cost
Mediumabout 1,623 tokens
Runs scripts
No, instructions only

What it does

Check progress for a detached KERMT run (pretrain, finetune, or any kermt_run_detached invocation). Reads run.json, queries docker for container state, tails the pretrain/finetune log, and parses progress lines (epoch, step, val loss).

kermt-monitor

Companion skill for any KERMT workflow that runs detached: the three pretrain skills (kermt-continue-pretrain, kermt-pretrain-scratch, kermt-add-cmim-pretrain) plus kermt-finetune. kermt-infer and kermt-embed run blocking by default and don't need this skill, but if a user launches them detached on purpose the monitor still works (the workflow-dispatch in step 4 handles unknown workflows by tailing the most-recent log file in the run dir). Reads the run directory's run.json, queries docker for the container's state, surfaces the latest progress, and either tails or follows the log.

Hardware requirements

None. This skill only reads disk + queries docker; no GPU compute.

Inputs

One of:

  • <run-dir>, a positional argument pointing at the directory containing run.json (e.g. runs/continue-pretrain_2026-05-17T10-23Z). Preferred.
  • --container <name-or-id>, direct container reference; the skill still reads run.json from the run dir referenced inside the container's inspect output if available, but works degraded-mode without it.

Optional:

  • --lines N, number of trailing log lines to print (default 50).
  • --follow, stream docker logs -f until ^C. Useful for "watch the loss". Without it, the skill is one-shot and exits.
  • --json, emit a structured status report instead of human-readable text. Useful when the parent agent wants to take downstream action.

Workflow

Let RUN_DIR=$1 (or whatever path the user supplies).

  1. Locate the manifest.
    MANIFEST=$RUN_DIR/run.json
    
    Refuse to proceed if it doesn't exist; surface a helpful message pointing the user at the run-dir convention (runs/<workflow>_<ts>/).
Read the whole SKILL.md on GitHub

More in DevOps