ACL Experiments

Use this while the experimental story can still change. The ACL evidence bar is not "beats the baseline once": it is a defensible measurement of a language capability, with the failure modes examined.

Baseline honesty

• Include the strongest cheap baseline: a well-prompted current LLM has become mandatory context for most tasks, a method beating only pre-LLM systems invites the "does this matter now?" review.
• Tune baselines with the same care as your method (same search budget, same data); reviewers explicitly probe for asymmetric tuning.
• Report the trivial baselines (majority class, copy input, retrieval-only) when they contextualize how hard the task actually is.

Evaluation design

• Breadth must match the claim: a "general" claim needs multiple datasets; a cross-lingual claim needs typologically distinct languages, not three Romance neighbors.
• Automatic metrics need justification for generation tasks, pair n-gram or embedding metrics with human or LLM-judge evaluation, and validate any LLM-judge against human labels before leaning on it.
• Fix the evaluation protocol before final runs: dev-set peeking on the test set via repeated submissions is unreportable and unrepairable.

Statistical floor

Result flavor	Required rigor at ACL
Small deltas between systems	Significance test (bootstrap/permutation) or overlapping-interval honesty
Fine-tuning results	Multiple seeds; mean and deviation in the table, defined in the caption
Prompted-LLM results	Multiple prompt paraphrases and/or samples; sensitivity range reported
Human evaluation	Raters per item, agreement statistic (e.g., Krippendorff's alpha), pay disclosed
Correlation claims (metrics)	Confidence intervals and comparison against existing metric correlations