# Acl Reproducibility

Use when strengthening reproducibility evidence for an ACL paper reviewed through ACL...

- Repository: https://github.com/brycewang-stanford/Awesome-Journal-Skills (1346 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-09-27
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-reproducibility -g -a claude-code -y`
Codex: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-reproducibility -g -a codex -y`
Cursor: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-reproducibility -g -a cursor -y`
Gemini CLI: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-reproducibility -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add brycewang-stanford/Awesome-Journal-Skills`

## Description

Use when strengthening reproducibility evidence for an ACL paper reviewed through ACL Rolling Review, covering the Responsible NLP checklist end to end, hyperparameter and compute reporting, prompt and decoding disclosure for LLM experiments, data contamination auditing, variance across runs, and checklist-to-paper consistency.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# ACL Reproducibility

Use this before an ARR deadline and again at camera-ready. At ACL the
reproducibility instrument is the **Responsible NLP checklist**: it is
mandatory, reviewers read it alongside the paper, and ARR policy makes
incorrect or misleading checklist content a desk-rejection ground. Treat it as
a claims audit, not paperwork.

## The checklist as a claims audit

- Section A: a real Limitations discussion and a risks discussion, reviewers
  are told honest limitations must not be penalized, so under-disclosing is
  strictly worse than disclosing.
- Section B: every dataset and model you *used* needs citation, version,
  license, and intended-use consistency (see `acl-artifact-evaluation`).
- Section C: computational experiments, parameters, budget, infrastructure,
  hyperparameter search, and descriptive statistics with error bars.
- Section D: human annotators/participants, instructions, pay, consent,
  ethics-board status, demographics where relevant.
- Section E: AI assistants used in research, coding, or writing.

Every "yes" answer should carry a section/appendix pointer; every "N/A" should
survive a hostile reading of the paper.

## Reporting floor for the modern NLP paper

| Experiment type | Minimum disclosure that survives ACL review |
|---|---|
| Fine-tuned models | Model + version, seeds, LR/schedule, epochs, selection criterion, dev-set use, runs count |
| Prompted LLMs | Exact prompts, decoding params (temperature, top-p, max tokens), model snapshot date/version, n samples |
| API-based closed models | Access dates, version string, cost/queries, caching strategy, note on irreproducibility risk |
| Human evaluation | Instructions, item counts, raters per item, agreement statistic, pay |
| New metrics | Implementation source, correlation evidence, code in supplement |

## Contamination and leakage auditing

The rest: https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/ACL-Skills/skills/acl-reproducibility/SKILL.md