# Aaai Reproducibility

Use when strengthening an AAAI paper's reproducibility checklist (placed after...

- Repository: https://github.com/brycewang-stanford/Awesome-Journal-Skills (1346 stars)
- Category: Data and ML
- License: MIT
- Last updated: 2026-09-27
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill aaai-reproducibility -g -a claude-code -y`
Codex: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill aaai-reproducibility -g -a codex -y`
Cursor: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill aaai-reproducibility -g -a cursor -y`
Gemini CLI: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill aaai-reproducibility -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add brycewang-stanford/Awesome-Journal-Skills`

## Description

Use when strengthening an AAAI paper's reproducibility checklist (placed after references), experimental traceability, seed and hyperparameter reporting, compute and cost disclosure, dataset access and licensing, code/data ZIP readiness, and the claim-to-evidence map that Phase-1 reviewers use to judge rigor across AAAI's broad AI scope.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# AAAI Reproducibility

Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the
reproducibility checklist to be **uploaded separately** from the main PDF, in its own field on the
submission form (AAAI-26 carried it inside the PDF after the references), so it is a document a
reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an
afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually
submitted, and that material promised "after acceptance or publication" is not evidence it exists.

## Reproducibility audit

- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item,
  checklist answer, or code/data artifact.
- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt
  selection, and hardware.
- Report variance or uncertainty when stochasticity affects conclusions.
- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and
  annotation procedures.
- Separate training compute, inference compute, and experiment search cost.
- Check the reproducibility checklist for contradictions with the main text and supplement.

## Common AAAI weaknesses

- Checklist says code/data are available but supplement lacks runnable commands.
- Main results rely on one seed, one benchmark, or one prompt family.
- Baselines are weaker than current open-source or widely cited systems.
- Evaluation uses closed data or APIs with no reproducibility substitute.
- Human evaluation omits annotator instructions or quality control.

## Checklist-to-evidence consistency grid

AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes"
against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no",
because it signals the checklist was filled in carelessly.

The rest: https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/AAAI-Skills/skills/aaai-reproducibility/SKILL.md