Fine-tune MiniCPM5-1B and MiniCPM5-2B with ms-swift

Select the version and template that match the model:

Model	ms-swift version	Arguments
MiniCPM5-1B	PyPI 4.5.3	--model_type llama --template minicpm5
MiniCPM5-2B	>=4.6.0.dev0	--model_type llama --template minicpm5_2b

Both models require transformers>=5.6.

ms-swift 4.x renamed --train_type to --tuner_type. Use --tuner_type lora for LoRA training.

Required input

Var	Example	Default
BASE_MODEL	openbmb/MiniCPM5-2B	required; openbmb/MiniCPM5-1B is also supported
TEMPLATE	minicpm5_2b	use minicpm5 for MiniCPM5-1B
DATA	path to messages-format jsonl	required
OUTPUT_DIR	./runs/minicpm5_swift	required
GPU_ID	0	0

Each line of DATA: {"messages": [{"role":"...","content":"..."}, ...]}.

Steps

1. Install (once)

For MiniCPM5-1B:

pip install "ms-swift==4.5.3" "transformers>=5.6"

For MiniCPM5-2B:

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
git checkout 654e24f17b5d9f40ed4b9ee4c56a723320244db5
pip install -e .
pip install "transformers>=5.6,<5.17"

For Megatron-SWIFT:

pip install mcore-bridge==1.6.4

2. Train (LoRA SFT)

CUDA_VISIBLE_DEVICES=${GPU_ID} swift sft \
    --model "${BASE_MODEL}" \
    --model_type llama \
    --template "${TEMPLATE}" \
    --tuner_type lora \
    --dataset "${DATA}" \
    --output_dir "${OUTPUT_DIR}" \
    --num_train_epochs 2 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --learning_rate 2e-4 \
    --lora_rank 16 --lora_alpha 32 --lora_dropout 0.05 \
    --target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj \
    --max_length 4096 \
    --warmup_ratio 0.03 \
    --bf16 true \
    --logging_steps 10 \
    --save_steps 200