Content Refinement Agent
Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018).
Install
npx skills add Ar9av/PaperOrchestra --skill content-refinement-agent -g -a claude-code -y- Repository
- Ar9av/PaperOrchestra694
- Category
- Engineering
- Updated
- 24d ago
- License
- See repository
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- Yes, review before use
What it does
Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
Content Refinement Agent (Step 5)
Faithful implementation of the Content Refinement Agent from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49, 51).
Cost: ~5, 7 LLM calls (App. B), typically ~3 refinement iterations, each consisting of one reviewer call and one revision call.
The paper highlights this step as one of the largest contributors to overall quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute acceptance-rate improvement (Fig. 4). Get this step right.
Inputs
workspace/drafts/paper.tex, output of Step 4workspace/inputs/conference_guidelines.mdworkspace/inputs/experimental_log.md, used as ground truth for the hallucination checkworkspace/citation_pool.json/workspace/refs.bib, the allowed bibliography
Outputs
workspace/refinement/iter1/,iter2/,iter3/, per-iteration snapshots containingpaper.tex,paper.pdf,review.json,score.jsonworkspace/refinement/worklog.json, append-only history of decisionsworkspace/final/paper.texandworkspace/final/paper.pdf, copy of the best accepted snapshot
The refinement loop
prev_score = score(paper.tex) # baseline from initial draft
snapshot iter0/
for iter in 1..ITER_CAP (default 3):
1. simulate_review(paper.tex) → review.json
(uses `references/reviewer-rubric.md` rubric)
2. apply_revision(paper.tex, review.json) → new_paper.tex
(uses verbatim Refinement Agent prompt at `references/prompt.md`)
3. snapshot iter<N>/ with new_paper.tex, review.json
latexmk -pdf new_paper.tex → iter<N>/paper.pdf
4. score(new_paper.tex) → curr_score
5. decide via score_delta.py:
- if curr.overall > prev.overall: ACCEPT
- elif curr.overall == prev.overall and net_subaxis ≥0: ACCEPT
- else: REVERT
6. apply_worklog.py to append the decision
