# Vss Benchmark Vlm QA

Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss...

- Repository: https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization (1959 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-11-01
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-benchmark-vlm-qa -g -a claude-code -y`
Codex: `npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-benchmark-vlm-qa -g -a codex -y`
Cursor: `npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-benchmark-vlm-qa -g -a cursor -y`
Gemini CLI: `npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-benchmark-vlm-qa -g -a gemini-cli -y`

## Description

Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss vlm run, using questions and videos from the DSS vss-devx-base dataset. Replaces the deprecated nat eval / vss-agent QA path. Not for tool-calling or trajectory evaluation, and not for LVS summarization throughput.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Benchmark video Q&A via `vss vlm`

Measure **accuracy** (LLM-as-judge vs ground truth) and **latency** of end-to-end
video question answering by calling **`vss vlm run`** against a deployed Cosmos
Reason 3 RT-VLM. Questions and clips come from DSS dataset **`vss-devx-base`**
(`nvdataset`).

This replaces `docker exec vss-agent nat eval` for the QA slice. It does **not**
score tool-calling or trajectories.

## When to use

- The user asks to benchmark / evaluate VLM video Q&A after vss-agent / NAT eval
  was removed.
- The user wants latency and answer accuracy on `vss-devx-base`.

## When not to use

- Tool-calling or trajectory evaluation, out of scope.
- LVS summarization throughput, use `vss-benchmark-video-summarization`.
- Ad-hoc single questions, use `/vss-ask-video`.

## Prerequisites

- A VSS stack with RT-VLM serving Cosmos Reason 3, and `vss configure` already run
  so `vss configure check` lists `rt_vlm` as `ok` and `vst` as `ok`.

The rest: https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/HEAD/skills/benchmarking/vss-benchmark-vlm-qa/SKILL.md