Deploy MiniCPM5-1B and MiniCPM5-2B with MLX (Apple Silicon)

Apple's on-device tensor framework. Highest throughput on M-series. Stays inside one Python process, no separate server, no llama.cpp build chain.

Required input

Var	Example	Default
MLX_REPO	openbmb/MiniCPM5-2B-MLX (pre-converted 4-bit affine)	required; openbmb/MiniCPM5-1B-MLX also works
OR HF_REPO + QUANT	openbmb/MiniCPM5-2B, 4bit or bf16	for local conversion
MAX_TOKENS	200	200

Steps

1. Install (once)

pip install "mlx-lm>=0.31" "gguf"

2A. Use a pre-converted MLX repo (recommended when available)

mlx_lm.generate --model "${MLX_REPO}" \
    --prompt "<|im_start|>user
1+1=?<|im_end|>
<|im_start|>assistant
" \
    --max-tokens ${MAX_TOKENS} --temp 1.0 --top-p 0.95

2B. Convert from a HF checkpoint locally (advanced)

Use mlx_lm.convert only if you have a self-trained HF fp16 checkpoint:

HF=/path/to/your-fp16-hf

# Convert: bf16
mlx_lm.convert --hf-path "$HF" --mlx-path ./minicpm5-mlx-bf16

# Convert: 4-bit (smaller / faster)
mlx_lm.convert --hf-path "$HF" --mlx-path ./minicpm5-mlx-q4 -q --q-bits 4

Then run as in 2A.

3. Validate

The reply should contain "2" for 1+1=?.

OpenAI-compatible server (mlx-lm)

mlx_lm.server --model "${MLX_REPO}" --host 127.0.0.1 --port 8000

curl http://127.0.0.1:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "default",
        "messages": [{"role":"user","content":"1+1=?"}],
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 64
    }'

Common pitfalls