# Verl Rl Training

Provides guidance for training LLMs with reinforcement learning using verl (Volcano...

- Repository: https://github.com/Orchestra-Research/AI-Research-SKILLs (14176 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-10-11
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training -g -a claude-code -y`
Codex: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training -g -a codex -y`
Cursor: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training -g -a cursor -y`
Gemini CLI: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill verl-rl-training -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`

## Description

Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# verl: Volcano Engine Reinforcement Learning for LLMs

verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.

## When to Use verl

**Choose verl when you need:**
- Production-ready RL training at scale (tested up to 671B parameters)
- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)
- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)
- Multi-turn rollout with tool calling for agentic workflows
- Vision-language model RL training

**Consider alternatives when:**
- You need Megatron-native training → use **slime** or **miles**
- You want PyTorch-native abstractions with Monarch → use **torchforge**
- You only need simple SFT/DPO → use **TRL** or **Axolotl**

## Key Features

- **Training backends**: FSDP, FSDP2, Megatron-LM
- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers
- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO
- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)
- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools

## Installation

```bash
# Option 1: pip install
pip install verl[vllm]  # or verl[sglang] for SGLang backend

# Option 2: Docker (recommended for production)
docker pull verlai/verl:vllm011.latest

# Option 3: From source
git clone https://github.com/volcengine/verl.git
cd verl && pip install -e .[vllm,math]
```

## Quick Start: GRPO Training

```bash
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=~/data/gsm8k/train.parquet \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    trainer.n_gpus_per_node=8
```

## Core Architecture

The rest: https://github.com/Orchestra-Research/AI-Research-SKILLs/blob/HEAD/06-post-training/verl/SKILL.md