Vss Benchmark Vlm QA
Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss...
Install
npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-benchmark-vlm-qa -g -a claude-code -y- Category
- Testing
- Updated
- 4d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,260 tokens
- Runs scripts
- Yes, review before use
What it does
Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss vlm run, using questions and videos from the DSS vss-devx-base dataset. Replaces the deprecated nat eval / vss-agent QA path. Not for tool-calling or trajectory evaluation, and not for LVS summarization throughput.
Benchmark video Q&A via vss vlm
Measure accuracy (LLM-as-judge vs ground truth) and latency of end-to-end
video question answering by calling vss vlm run against a deployed Cosmos
Reason 3 RT-VLM. Questions and clips come from DSS dataset vss-devx-base
(nvdataset).
This replaces docker exec vss-agent nat eval for the QA slice. It does not
score tool-calling or trajectories.
When to use
- The user asks to benchmark / evaluate VLM video Q&A after vss-agent / NAT eval was removed.
- The user wants latency and answer accuracy on
vss-devx-base.
When not to use
- Tool-calling or trajectory evaluation, out of scope.
- LVS summarization throughput, use
vss-benchmark-video-summarization. - Ad-hoc single questions, use
/vss-ask-video.
Prerequisites
- A VSS stack with RT-VLM serving Cosmos Reason 3, and
vss configurealready run sovss configure checklistsrt_vlmasokandvstasok.
