Trl Training
Post-train LLMs with TRL (Transformers Reinforcement Learning), SFT, DPO, GRPO, KTO...
Install
npx skills add huggingface/skills --skill trl-training -g -a claude-code -y- Repository
- huggingface/skills11.2k
- Category
- Testing
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Lightabout 993 tokens
- Runs scripts
- No, instructions only
What it does
Post-train LLMs with TRL (Transformers Reinforcement Learning), SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
TRL
Each method pairs a *Trainer class with a *Config dataclass. Configs extend transformers.TrainingArguments, so all of its arguments work in any trainer config.
| Trainer | Dataset type |
|---|---|
SFTTrainer | language modeling or prompt-completion |
DPOTrainer | preference (chosen/rejected pairs) |
GRPOTrainer | prompt-only + reward function(s) |
DistillationTrainer | prompt-only + a teacher model (on-policy distillation) |
KTOTrainer | unpaired preference (per-sample bool label) |
RewardTrainer | preference (chosen/rejected pairs); trains a scalar reward model, not a policy |
Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in trl.experimental with unstable APIs: https://huggingface.co/docs/trl/experimental_overview
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B", # model ID or a PreTrainedModel instance
args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()
Pass model as a string and route loading kwargs through model_init_kwargs (e.g. {"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}) instead of calling from_pretrained yourself. The tokenizer/processor is inferred from the model; pass processing_class only when it differs. For LoRA, pass peft_config=LoraConfig(...).
Dataset formats
Conversational: {"messages": [{"role": ..., "content": ...}]} (language modeling) or {"prompt": [...], "completion": [...]}. The chat template is applied automatically, never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats
