Acl Reproducibility
Use when strengthening reproducibility evidence for an ACL paper reviewed through ACL...
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-reproducibility -g -a claude-code -y- Repository
- brycewang-stanford/Awesome-Journal-Skills1.3k
- Category
- AI and agents
- Updated
- 1mo ago
- License
- MIT
- Context cost
- Lightabout 1,355 tokens
- Runs scripts
- No, instructions only
What it does
Use when strengthening reproducibility evidence for an ACL paper reviewed through ACL Rolling Review, covering the Responsible NLP checklist end to end, hyperparameter and compute reporting, prompt and decoding disclosure for LLM experiments, data contamination auditing, variance across runs, and checklist-to-paper consistency.
ACL Reproducibility
Use this before an ARR deadline and again at camera-ready. At ACL the reproducibility instrument is the Responsible NLP checklist: it is mandatory, reviewers read it alongside the paper, and ARR policy makes incorrect or misleading checklist content a desk-rejection ground. Treat it as a claims audit, not paperwork.
The checklist as a claims audit
- Section A: a real Limitations discussion and a risks discussion, reviewers are told honest limitations must not be penalized, so under-disclosing is strictly worse than disclosing.
- Section B: every dataset and model you used needs citation, version,
license, and intended-use consistency (see
acl-artifact-evaluation). - Section C: computational experiments, parameters, budget, infrastructure, hyperparameter search, and descriptive statistics with error bars.
- Section D: human annotators/participants, instructions, pay, consent, ethics-board status, demographics where relevant.
- Section E: AI assistants used in research, coding, or writing.
Every "yes" answer should carry a section/appendix pointer; every "N/A" should survive a hostile reading of the paper.
Reporting floor for the modern NLP paper
| Experiment type | Minimum disclosure that survives ACL review |
|---|---|
| Fine-tuned models | Model + version, seeds, LR/schedule, epochs, selection criterion, dev-set use, runs count |
| Prompted LLMs | Exact prompts, decoding params (temperature, top-p, max tokens), model snapshot date/version, n samples |
| API-based closed models | Access dates, version string, cost/queries, caching strategy, note on irreproducibility risk |
| Human evaluation | Instructions, item counts, raters per item, agreement statistic, pay |
| New metrics | Implementation source, correlation evidence, code in supplement |
