# LLM Judge Evaluation

Designs LLM-as-judge systems, pairwise comparisons, rubrics, and bias controls.

- Repository: https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering (18272 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-12
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation -g -a claude-code -y`
Codex: `npx skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation -g -a codex -y`
Cursor: `npx skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation -g -a cursor -y`
Gemini CLI: `npx skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering`

## Description

This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Advanced Evaluation

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.

**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

## When to Activate

Activate this skill when:

- Building LLM-as-judge systems for LLM outputs
- Comparing multiple model responses to select the best one
- Establishing consistent quality standards across evaluation teams
- Debugging evaluation systems that show inconsistent results
- Designing A/B tests for prompt or model changes
- Creating rubrics specifically for LLM or human/LLM hybrid judges
- Analyzing correlation between automated and human judgments

Do not activate this skill for adjacent work owned by other skills:
- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.
- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.
- Tool API contracts for evaluation tools: `tool-design`.

## Core Concepts

### The Evaluation Taxonomy

Select between two primary approaches based on whether ground truth exists:

**Direct Scoring**, Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.

The rest: https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering/blob/HEAD/skills/advanced-evaluation/SKILL.md