Localize over MCP

localize answers from memory: the colour frames the module has embedded, lifted to 3D through depth or lidar. It does not look at the current frame. Every answer is a verified 3D object with a score and a view count, and an empty answer is an answer.

Before the first call

nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv,noheader
uv run dimos mcp call state

• Probe the GPU first. How many labels a call can carry depends on it, see "Labels and GPU".
• state() returns ready: N frames embedded, localize will answer or not ready: <stage>. Until it reads ready, localize returns its stage instead of an answer.
• Error: no running MCP server (is DimOS running?) means the stack is down. Check uv run dimos status and report it. Do not stop or restart the stack.

Calling localize

uv run dimos mcp call localize -j '{"objects": "chair,door", "start": -30, "duration": 30}' --timeout 300

• localize(objects, start=-10, duration=10, params="") returns once the answer is built. Send the next call after it returns.
• objects is one label, or a few separated by commas. The labels of one call share one detector pass per frame.
• The client waits 30 s. A window with many unread frames takes longer, so pass --timeout 300 or set MCP_TIMEOUT=300.

Building the window

start and duration are seconds over the embedded frames the module holds.

arguments	window
start=-10, duration=10 (default)	the last 10 s
start=-60, duration=60	the last minute
start=-60, duration=20	from 60 s ago to 40 s ago
start=30, duration=20	20 s, from 30 s after the oldest frame held