Gemini Video Understanding Skill

Purpose

This skill enables video analysis and object counting using the Google Gemini API, with a focus on counting pedestrians, detecting objects, tracking movement, and analyzing surveillance footage. It supports precise prompting for differentiated counting (e.g., pedestrians vs cyclists vs vehicles).

When to Use

• Counting pedestrians, vehicles, or other objects in surveillance videos
• Distinguishing between different types of objects (walkers vs cyclists, cars vs trucks)
• Analyzing traffic patterns and movement through a scene
• Processing multiple videos for batch object counting
• Extracting structured count data from video footage

Required Libraries

The following Python libraries are required:

from google import genai
from google.genai import types
import os
import time

Input Requirements

• File formats: MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP
• Size constraints:
  • Use inline bytes for small files (rule of thumb: <20MB).
  • Use the File API upload flow for larger videos (most surveillance footage).
  • Always wait for processing to complete before analysis.
• Video quality: Higher resolution provides better counting accuracy for distant objects
• Duration: Longer videos may require longer processing times; consider the full video length for accurate counting

Output Schema

For object counting tasks, structure results as JSON:

{
  "success": true,
  "video_file": "surveillance_001.mp4",
  "model": "gemini-2.0-flash-exp",
  "counts": {
    "pedestrians": 12,
    "cyclists": 3,
    "vehicles": 5
  },
  "notes": "Optional observations about the counting process or edge cases"
}