Vss Evaluate Caption Accuracy
Measure whether an RT-VLM configuration change altered caption quality, capture paired...
Install
npx skills add NVIDIA-AI-Blueprints/video-search-and-summarization --skill vss-evaluate-caption-accuracy -g -a claude-code -y- Category
- AI and agents
- Updated
- 4d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 1,844 tokens
- Runs scripts
- Yes, review before use
What it does
Measure whether an RT-VLM configuration change altered caption quality, capture paired baseline and candidate captions for a set of videos, score both against a ground truth with an LLM judge, and emit an accuracy and processing-time table. Use when changing frame selection, decode, or model settings and you need evidence there is no accuracy regression.
Purpose
Answer one question with evidence: did this RT-VLM change make captions worse?
A configuration change that saves processing time is only useful if caption quality holds. This skill captures captions twice over the same videos, once with the change (HYP) and once without (REF), scores both against a ground truth using an LLM judge, and reports accuracy delta alongside time saved.
When to Use
- Before/after changing RT-VLM frame selection, decode, or model settings, to prove there is no caption-quality regression
- Quantifying accuracy delta and processing-time saved for a candidate config change
- Verifying a change was behavior-neutral via frame-level provenance, not just matching totals
Not for deploying RT-VLM, general LVS/RTVI benchmarking (see vss-benchmark-video-summarization), or one-off caption spot checks with no paired baseline.
Prerequisites
Everything except the judge runs inside the RT-VLM container. The container
needs a GPU, the model on disk, and the nvdsframeselector DeepStream plugin
(shipped with DeepStream in the RT-VLM image).
