# Kermt Pretrain Scratch

Pretrain a fresh KERMT model from scratch on a user-provided corpus.

- Repository: https://github.com/NVIDIA/skills (3812 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add NVIDIA/skills --skill kermt-pretrain-scratch -g -a claude-code -y`
Codex: `npx skills add NVIDIA/skills --skill kermt-pretrain-scratch -g -a codex -y`
Cursor: `npx skills add NVIDIA/skills --skill kermt-pretrain-scratch -g -a cursor -y`
Gemini CLI: `npx skills add NVIDIA/skills --skill kermt-pretrain-scratch -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add NVIDIA/skills`

## Description

Pretrain a fresh KERMT model from scratch on a user-provided corpus. Builds a new vocabulary from the corpus, instantiates the model architecture from defaults, and launches pretrain_ddp.py inside the kermt container (detached for long runs). Unlike kermt-continue-pretrain, no starting checkpoint is loaded, the model is randomly initialized.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# kermt-pretrain-scratch

Pretrain a brand-new KERMT model from scratch on a user-provided corpus. Useful
when you want to retrain a model on a custom chemistry domain rather than
extending one of the released checkpoints. **Significantly more expensive than
`kermt-continue-pretrain`**, no warm start, so the loss curves need to descend
from scratch over many epochs.

## Skill and runtime paths

Set `SKILL_DIR` to the absolute path of this installed skill directory. Export
`KERMT_REPO` as the absolute path to the KERMT checkout used for model
execution. The bundled container helper mounts that checkout at
`/workspace` and this skill at `/skill` (read-only). Commands inside
the container use `/skill/scripts/`; defaults are bundled in `config/`.

## Hardware requirements

Same as `kermt-continue-pretrain`:

- **GPUs**: 1, N CUDA-capable. The runner auto-detects via
  `torch.cuda.device_count()`; `--gpus 0,2` overrides. Single-GPU fallback:
  `--batch_size 32 --save_interval 500`. Multi-GPU keeps defaults
  (`--batch_size 256` etc.). Note: `--gpus N` uses **torch.cuda** indexing,
  which can differ from `nvidia-smi`'s display order on multi-GPU hosts
  (PCI bus vs. CUDA enumeration). To target a specific physical GPU, set
  `CUDA_VISIBLE_DEVICES` before invoking, or run
  `python -c "import torch; print([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])"`
  to confirm which device you're picking.
- **VRAM**: the default `--batch-size 256` is sized for A100-class hardware
  (80 GB VRAM). On smaller GPUs, downscale to avoid OOM:

  | GPU class                | VRAM       | Suggested `--batch-size` |
  |--------------------------|------------|--------------------------|
  | L4, T4, V100 16 GB       | 16, 24 GB   | 32, 64                    |
  | A100 40 GB, L40, A40     | 40, 48 GB   | 128                      |
  | A100 80 GB, H100, H200   | 80 GB      | 256 (default)            |

The rest: https://github.com/NVIDIA/skills/blob/HEAD/skills/bionemo-kermt-pretrain-scratch/SKILL.md