# Trl Training

Post-train LLMs with TRL (Transformers Reinforcement Learning), SFT, DPO, GRPO, KTO...

- Repository: https://github.com/huggingface/skills (11249 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add huggingface/skills --skill trl-training -g -a claude-code -y`
Codex: `npx skills add huggingface/skills --skill trl-training -g -a codex -y`
Cursor: `npx skills add huggingface/skills --skill trl-training -g -a cursor -y`
Gemini CLI: `npx skills add huggingface/skills --skill trl-training -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add huggingface/skills`

## Description

Post-train LLMs with TRL (Transformers Reinforcement Learning), SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# TRL

Each method pairs a `*Trainer` class with a `*Config` dataclass. Configs extend `transformers.TrainingArguments`, so all of its arguments work in any trainer config.

| Trainer | Dataset type |
|---|---|
| `SFTTrainer` | language modeling or prompt-completion |
| `DPOTrainer` | preference (chosen/rejected pairs) |
| `GRPOTrainer` | prompt-only + reward function(s) |
| `DistillationTrainer` | prompt-only + a teacher model (on-policy distillation) |
| `KTOTrainer` | unpaired preference (per-sample bool label) |
| `RewardTrainer` | preference (chosen/rejected pairs); trains a scalar reward model, not a policy |

Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in `trl.experimental` with unstable APIs: https://huggingface.co/docs/trl/experimental_overview

```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",  # model ID or a PreTrainedModel instance
    args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()
```

Pass `model` as a string and route loading kwargs through `model_init_kwargs` (e.g. `{"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}`) instead of calling `from_pretrained` yourself. The tokenizer/processor is inferred from the model; pass `processing_class` only when it differs. For LoRA, pass `peft_config=LoraConfig(...)`.

## Dataset formats

Conversational: `{"messages": [{"role": ..., "content": ...}]}` (language modeling) or `{"prompt": [...], "completion": [...]}`. The chat template is applied automatically, never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats

## SFT: the fields that matter

The rest: https://github.com/huggingface/skills/blob/HEAD/skills/trl-training/SKILL.md