# Advanced Evaluation

Design LLM-as-judge rubrics, direct scores and pairwise comparisons.

- Repository: https://github.com/sickn33/agentic-awesome-skills (48984 stars)
- Category: Design
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add sickn33/agentic-awesome-skills --skill advanced-evaluation -g -a claude-code -y`
Codex: `npx skills add sickn33/agentic-awesome-skills --skill advanced-evaluation -g -a codex -y`
Cursor: `npx skills add sickn33/agentic-awesome-skills --skill advanced-evaluation -g -a cursor -y`
Gemini CLI: `npx skills add sickn33/agentic-awesome-skills --skill advanced-evaluation -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add sickn33/agentic-awesome-skills`

## Description

Design LLM-as-judge rubrics, direct scores and pairwise comparisons. Use for judge models, evals, position bias and calibration against human labels.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Advanced Evaluation

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.

**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

## When to Use
Activate this skill when:

- Building automated evaluation pipelines for LLM outputs
- Comparing multiple model responses to select the best one
- Establishing consistent quality standards across evaluation teams
- Debugging evaluation systems that show inconsistent results
- Designing A/B tests for prompt or model changes
- Creating rubrics for human or automated evaluation
- Analyzing correlation between automated and human judgments

## Core Concepts

### The Evaluation Taxonomy

Evaluation approaches fall into two primary categories with distinct reliability profiles:

**Direct Scoring**: A single LLM rates one response on a defined scale.
- Best for: Objective criteria (factual accuracy, instruction following, toxicity)
- Reliability: Moderate to high for well-defined criteria
- Failure mode: Score calibration drift, inconsistent scale interpretation

**Pairwise Comparison**: An LLM compares two responses and selects the better one.
- Best for: Subjective preferences (tone, style, persuasiveness)
- Reliability: Higher than direct scoring for preferences
- Failure mode: Position bias, length bias

Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.

### The Bias Landscape

The rest: https://github.com/sickn33/agentic-awesome-skills/blob/HEAD/plugins/agentic-awesome-skills-claude/skills/advanced-evaluation/SKILL.md