# Write Judge Prompt

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot...

- Repository: https://github.com/ai-evals-course/evals-skills (1700 stars)
- Category: Design
- License: Apache-2.0
- Last updated: 2026-10-24
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai-evals-course/evals-skills --skill write-judge-prompt -g -a claude-code -y`
Codex: `npx skills add ai-evals-course/evals-skills --skill write-judge-prompt -g -a codex -y`
Cursor: `npx skills add ai-evals-course/evals-skills --skill write-judge-prompt -g -a cursor -y`
Gemini CLI: `npx skills add ai-evals-course/evals-skills --skill write-judge-prompt -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ai-evals-course/evals-skills`

## Description

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Write LLM-as-Judge Prompt

Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.

## Prerequisites

- Error analysis is complete. The failure mode is identified.
- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge, many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.

## The Four Components

Every judge prompt requires exactly four components:

### 1. Task and Evaluation Criterion

State what the judge evaluates. One failure mode per judge.

```
You are an evaluator assessing whether a real estate assistant's email
uses the appropriate tone for the client's persona.
```

Not: "Evaluate whether the email is good" or "Rate the email quality from 1-5."

### 2. Pass/Fail Definitions

Outcomes are strictly binary: Pass or Fail. No Likert scales, no letter grades, no partial credit. Define exactly what constitutes Pass and Fail. These definitions come from your error analysis failure mode descriptions.

```
## Definitions

PASS: The email matches the expected communication style for the client persona:
- Luxury Buyers: formal language, emphasis on exclusive features, premium
  market positioning, no casual slang
- First-Time Homebuyers: warm and encouraging tone, educational explanations,
  avoids jargon, patient and supportive
- Investors: data-driven language, ROI-focused, market analytics, concise
  and professional

The rest: https://github.com/ai-evals-course/evals-skills/blob/HEAD/skills/write-judge-prompt/SKILL.md