# Benchmarking

Use this skill when the user wants to benchmark an MLX-VLM change and present the...

- Repository: https://github.com/Blaizzy/mlx-vlm (5707 stars)
- Category: Git and review
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Blaizzy/mlx-vlm --skill benchmarking -g -a claude-code -y`
Codex: `npx skills add Blaizzy/mlx-vlm --skill benchmarking -g -a codex -y`
Cursor: `npx skills add Blaizzy/mlx-vlm --skill benchmarking -g -a cursor -y`
Gemini CLI: `npx skills add Blaizzy/mlx-vlm --skill benchmarking -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Blaizzy/mlx-vlm`

## Description

Use this skill when the user wants to benchmark an MLX-VLM change and present the numbers in a PR, fork-vs-main A/B comparisons, isolated-module micro-benchmarks, median-of-N timing with warmup, peak-memory reporting, correctness checks, parameter sweeps, and self-contained reproducible bench scripts to paste into a PR description.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Benchmarking & PR A/B Testing

Use this workflow to produce **credible, reproducible** performance numbers for a PR.

## Rules for a trustworthy benchmark

- **Median of N** (not mean), with a few warmup iterations discarded.
- Call `mx.eval(...)` / `mx.synchronize()` before stopping the timer (MLX is lazy, untimed work hides otherwise).
- Report **peak memory** via `mx.get_peak_memory()/1e9` (GB); reset with `mx.reset_peak_memory()` if available.
- Include an **inline correctness check**, a benchmark that computes the wrong thing is meaningless. Assert the new path matches the baseline (exactly in the degenerate limit, or within tolerance).
- **Sweep one parameter** (sequence length, batch, tokens, layers) and print a **markdown table** with `flush=True`.
- Prefer a **random-init synthetic model** (tiny config) so the script needs no checkpoint download and runs on a fresh checkout.
- State the hardware (Apple-Silicon chip + RAM). A model must fit in RAM; isolate the module under test with random weights if the full model won't fit, don't run large models on an 8 GB machine.

## Fork-vs-main A/B (paste into the PR body)

Clones upstream `main` and your fork, runs the *identical* bench on both, and joins into a speedup table:

```bash
#!/usr/bin/env bash
# <feature> perf: upstream main vs fork, random-init tiny <model>, swept over <param>.
set -euo pipefail
W=$(mktemp -d)
git clone -q --depth 1 https://github.com/Blaizzy/mlx-vlm "$W/main"
git clone -q --depth 1 https://github.com/<you>/mlx-vlm   "$W/fork"

The rest: https://github.com/Blaizzy/mlx-vlm/blob/HEAD/skills/skills/benchmarking/SKILL.md