# Server Inference

Use this skill when the user wants to run or debug MLX-VLM server inference, including...

- Repository: https://github.com/Blaizzy/mlx-vlm (5707 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Blaizzy/mlx-vlm --skill server-inference -g -a claude-code -y`
Codex: `npx skills add Blaizzy/mlx-vlm --skill server-inference -g -a codex -y`
Cursor: `npx skills add Blaizzy/mlx-vlm --skill server-inference -g -a cursor -y`
Gemini CLI: `npx skills add Blaizzy/mlx-vlm --skill server-inference -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Blaizzy/mlx-vlm`

## Description

Use this skill when the user wants to run or debug MLX-VLM server inference, including uv run mlx_vlm.server, /v1/models, /v1/chat/completions, /v1/responses, streaming, OpenAI-compatible clients, health checks, metrics, model unload/reload, adapters, trust-remote-code, and server request/response failures.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Server Inference

Use this workflow for the FastAPI server and API-compatible inference.

## First Checks

1. Identify the server command, model, port, request endpoint, request body, and expected response.
2. Start with health/model-list checks before debugging generation.
3. Separate server startup failures from request-handling failures.
4. Keep streaming and non-streaming repros separate.

## Startup

```bash
uv run mlx_vlm.server \
  --model <model-or-path> \
  --port 8080
```

Useful startup flags include `--adapter-path`, `--trust-remote-code`, `--log-level`, `--enable-thinking`, `--thinking-budget`, `--draft-model`, `--draft-kind`, `--kv-bits`, `--kv-quant-scheme`, `--max-kv-size`, and `--vision-cache-size`.

## Minimal Checks

```bash
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8080/metrics
```

Minimal chat request:

```bash
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-or-path>",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 32,
    "stream": false
  }'
```

Minimal Responses API request:

```bash
curl -s http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-or-path>",
    "input": "Say hello.",
    "max_output_tokens": 32
  }'
```

## Endpoint Surface (beyond chat/responses)

The server exposes more than chat, cover or route to these as needed:

The rest: https://github.com/Blaizzy/mlx-vlm/blob/HEAD/skills/skills/server-inference/SKILL.md