Benchmarking
Use this skill when the user wants to benchmark an MLX-VLM change and present the...
Install
npx skills add Blaizzy/mlx-vlm --skill benchmarking -g -a claude-code -y- Repository
- Blaizzy/mlx-vlm5.7k
- Category
- Git and review
- Updated
- 3d ago
- License
- MIT
- Context cost
- Lightabout 936 tokens
- Runs scripts
- No, instructions only
What it does
Use this skill when the user wants to benchmark an MLX-VLM change and present the numbers in a PR, fork-vs-main A/B comparisons, isolated-module micro-benchmarks, median-of-N timing with warmup, peak-memory reporting, correctness checks, parameter sweeps, and self-contained reproducible bench scripts to paste into a PR description.
Benchmarking & PR A/B Testing
Use this workflow to produce credible, reproducible performance numbers for a PR.
Rules for a trustworthy benchmark
- Median of N (not mean), with a few warmup iterations discarded.
- Call
mx.eval(...)/mx.synchronize()before stopping the timer (MLX is lazy, untimed work hides otherwise). - Report peak memory via
mx.get_peak_memory()/1e9(GB); reset withmx.reset_peak_memory()if available. - Include an inline correctness check, a benchmark that computes the wrong thing is meaningless. Assert the new path matches the baseline (exactly in the degenerate limit, or within tolerance).
- Sweep one parameter (sequence length, batch, tokens, layers) and print a markdown table with
flush=True. - Prefer a random-init synthetic model (tiny config) so the script needs no checkpoint download and runs on a fresh checkout.
- State the hardware (Apple-Silicon chip + RAM). A model must fit in RAM; isolate the module under test with random weights if the full model won't fit, don't run large models on an 8 GB machine.
Fork-vs-main A/B (paste into the PR body)
Clones upstream main and your fork, runs the identical bench on both, and joins into a speedup table:
#!/usr/bin/env bash
# <feature> perf: upstream main vs fork, random-init tiny <model>, swept over <param>.
set -euo pipefail
W=$(mktemp -d)
git clone -q --depth 1 https://github.com/Blaizzy/mlx-vlm "$W/main"
git clone -q --depth 1 https://github.com/<you>/mlx-vlm "$W/fork"
