clawpathy-autoresearch

Eval-driven skill development. The system iteratively rewrites a SKILL.md so a downstream executor agent performs better at a task class, as judged by an LLM against a paper/task-specific rubric.

Core idea

  propose (sonnet)  →  execute (sonnet, shell)  →  judge (opus, rubric)
       ↑                                                       │
       └──────── feedback: verdict + recommended edits ────────┘

• Proposer rewrites SKILL.md based on the last judge verdict.
• Executor runs the new SKILL.md end-to-end inside a workspace.
• Judge scores methodology (primary) and outputs (secondary) against a per-task rubric. Lower is better; 0 = perfect.
• Keep the new SKILL.md only if it strictly beats the best score; else revert. Stop on target_score or on early_stop_n consecutive regressions.

You are the orchestrator

You (the agent reading this) don't run the loop yourself. You dispatch subagents to build the workspace, then hand off to the Python loop.

Phase 1, Scout

Dispatch a subagent with prompts/scout.md to research the paper/task. Report key findings to the user in a few lines.

Phase 2, Scope (you + user)

Have a conversation. Ask ONE question at a time, multiple-choice where helpful. Agree on:

• what to reproduce / what success looks like
• which data sources are in-bounds
• what methodology expectations belong in the rubric
• iteration budget and target_score (if any)

Present a summary and get approval.

Phase 3, Build

Dispatch a builder subagent with prompts/builder.md and the agreed scope. It writes:

• task.json
• rubric.md, the authoritative scoring rubric for the LLM judge
• reference/ (optional; judge-only)
• skill/SKILL.md, seed

Validate:

from skills.clawpathy_autoresearch import validate_workspace
print(validate_workspace(Path("WORKSPACE")))  # [] means valid

Phase 4, Loop