miles: Enterprise-Grade RL for Large-Scale Model Training

miles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.

When to Use miles

Choose miles when you need:

• Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)
• FP8 or INT4 quantization-aware training
• Bit-wise identical train-inference alignment
• Speculative RL for maximum throughput
• Production stability with enterprise support

Consider alternatives when:

• You want the research-grade original → use slime
• You need flexible backend swapping → use verl
• You want PyTorch-native abstractions → use torchforge

Key Features

Low-Precision Training

• Unified FP8: End-to-end FP8 for both inference and training
• INT4 QAT: 1TB models on single-machine VRAM (H200)
• Rollout Routing Replay (R3): Bit-wise expert alignment for MoE

Performance Optimizations

• Speculative RL: 25%+ rollout speedup with online SFT draft models
• Zero-Copy Weight Sync: CUDA IPC zero-copy mapping
• Partial Rollout: Recycle half-finished trajectories

Train-Inference Alignment

• TIS/MIS: Truncated/Masked Importance Sampling for off-policy correction
• Kernel-level optimization: FlashAttention-3, DeepGEMM integration

Installation

# Recommended: Docker
docker pull radixark/miles:latest
docker run --rm --gpus all --ipc=host --shm-size=16g \
  -it radixark/miles:latest /bin/bash

# From source
git clone https://github.com/radixark/miles.git
cd miles
pip install -r requirements.txt
pip install -e .

Quick Start

miles inherits slime's configuration system. Basic training: