Evaluating Code Models
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks...
Install
npx skills add Orchestra-Research/AI-Research-SKILLs --skill evaluating-code-models -g -a claude-code -y- Repository
- Orchestra-Research/AI-Research-SKILLs14.2k
- Category
- Testing
- Updated
- 25d ago
- License
- MIT
- Context cost
- Mediumabout 2,765 tokens
- Runs scripts
- No, instructions only
What it does
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
BigCode Evaluation Harness - Code Model Benchmarking
Quick Start
BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).
Installation:
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config
Evaluate on HumanEval:
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--n_samples 20 \
--batch_size 10 \
--allow_code_execution \
--save_generations
View available tasks:
python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"
Common Workflows
Workflow 1: Standard Code Benchmark Evaluation
Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).
Checklist:
Code Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model and generation
- [ ] Step 3: Run evaluation with code execution
- [ ] Step 4: Analyze pass@k results
Step 1: Choose benchmark suite
Python code generation (most common):
- HumanEval: 164 handwritten problems, function completion
- HumanEval+: Same 164 problems with 80× more tests (stricter)
- MBPP: 500 crowd-sourced problems, entry-level difficulty
- MBPP+: 399 curated problems with 35× more tests
Multi-language (18 languages):
- MultiPL-E: HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc.
Advanced:
- APPS: 10,000 problems (introductory/interview/competition)
- DS-1000: 1,000 data science problems across 7 libraries
Step 2: Configure model and generation
