# Acl Experiments

Use when designing or auditing experiments for an ACL paper, covering tuned LLM...

- Repository: https://github.com/brycewang-stanford/Awesome-Journal-Skills (1346 stars)
- Category: Design
- License: MIT
- Last updated: 2026-09-27
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-experiments -g -a claude-code -y`
Codex: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-experiments -g -a codex -y`
Cursor: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-experiments -g -a cursor -y`
Gemini CLI: `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill acl-experiments -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add brycewang-stanford/Awesome-Journal-Skills`

## Description

Use when designing or auditing experiments for an ACL paper, covering tuned LLM baselines, multi-dataset and multilingual evaluation, statistical significance and variance, human evaluation with agreement reporting, contamination and prompt-sensitivity controls, ablations, and error-analysis expectations in NLP reviewing.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# ACL Experiments

Use this while the experimental story can still change. The ACL evidence bar
is not "beats the baseline once": it is a defensible measurement of a language
capability, with the failure modes examined.

## Baseline honesty

- Include the strongest cheap baseline: a well-prompted current LLM has become
  mandatory context for most tasks, a method beating only pre-LLM systems
  invites the "does this matter now?" review.
- Tune baselines with the same care as your method (same search budget, same
  data); reviewers explicitly probe for asymmetric tuning.
- Report the trivial baselines (majority class, copy input, retrieval-only)
  when they contextualize how hard the task actually is.

## Evaluation design

- Breadth must match the claim: a "general" claim needs multiple datasets;
  a cross-lingual claim needs typologically distinct languages, not three
  Romance neighbors.
- Automatic metrics need justification for generation tasks, pair n-gram or
  embedding metrics with human or LLM-judge evaluation, and validate any
  LLM-judge against human labels before leaning on it.
- Fix the evaluation protocol before final runs: dev-set peeking on the test
  set via repeated submissions is unreportable and unrepairable.

## Statistical floor

| Result flavor | Required rigor at ACL |
|---|---|
| Small deltas between systems | Significance test (bootstrap/permutation) or overlapping-interval honesty |
| Fine-tuning results | Multiple seeds; mean and deviation in the table, defined in the caption |
| Prompted-LLM results | Multiple prompt paraphrases and/or samples; sensitivity range reported |
| Human evaluation | Raters per item, agreement statistic (e.g., Krippendorff's alpha), pay disclosed |
| Correlation claims (metrics) | Confidence intervals and comparison against existing metric correlations |

The rest: https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/ACL-Skills/skills/acl-experiments/SKILL.md