WeaveBench CUA-Harness Reproduction

Use this skill to guide an agent through a complete, reproducible WeaveBench CUA-Harness run. The expected repository layout is:

WeaveBench-harness/
  WeaveBench/
  cua-harness/
  skills/weavebench-cua-reproduce/

Do not invent alternate launch commands. Prefer the bundled helper script and the project scripts under WeaveBench/scripts/.

Supported Scope

Supported automation:

• Local Docker/KVM WeaveBench evaluation.
• Qwen 3.7-Plus through an Anthropic-compatible endpoint.
• Claude Code execution backend through cua_harness_claudecode.
• OpenClaw judge setup and verification.
• Official WeaveBench assets downloaded from the HuggingFace dataset.
• A 120G copy of the official Ubuntu qcow2 plus per-task rootfs growth.

Not automated:

• Cloud VM provisioning.
• Public image-hosting service deployment.
• New model-provider adapters beyond the existing Anthropic-compatible path.
• Non-Claude-Code execution backend reproduction.
• Deleting or pruning existing results.

If the user requests unsupported automation, explain the boundary and provide the closest supported local Docker/KVM path.

Start With Intake

Before doing setup work on a new machine, run or mentally follow:

./skills/weavebench-cua-reproduce/scripts/reproduce.sh intake

Confirm:

• Linux Docker/KVM is available or installable.
• The user accepts large downloads and creation of Ubuntu_120G.qcow2.
• The model endpoint and API key are available.
• Image proxy should be enabled with public upload/show URLs, or disabled because the endpoint accepts large base64 screenshots.
• The desired scope is doctor, smoke, subset, or full 114-task evaluation.
• Long-running/full evaluation is allowed, especially if other experiments may already be active.

Workflow