# Redteam

Use when the user wants to test their LLM/agent application for safety and security...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill redteam -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill redteam -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill redteam -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill redteam -g -a gemini-cli -y`

## Description

Use when the user wants to test their LLM/agent application for safety and security vulnerabilities, jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<HARD-GATE>
NO ASR report WITHOUT paired over-refusal rate measurement.
NO attack vector distribution WITHOUT reading a policy document first.
NO regulated-stakes redteam WITHOUT a sign-off block in the audit document.
</HARD-GATE>

# Redteam

Test your application's safety boundaries systematically. This skill generates
attack prompts from a policy document, measures what gets through, and pairs
the Attack Success Rate (ASR) with the Over-Refusal Rate so you don't reward
models that simply refuse everything.

## When to Activate

- Pre-deployment safety audit
- Regulatory compliance check
- After major model or prompt changes that could affect safety
- User reports a jailbreak or injection vulnerability
- Periodic (quarterly) safety re-evaluation

## Checklist

You MUST create a task for each item and complete them in order:

1. **Establish policy**, what's prohibited, what's allowed, what's gray
2. **Derive attack vectors from policy**, not a fixed template
3. **Generate attack set + over-refusal set**, paired measurement
4. **Run attacks and evaluate**, ASR per policy category
5. **Compute over-refusal rate**, paired with ASR
6. **Output safety scorecard + audit document**

## Fast path: run the bundled script

Once attacks and benign prompts have been run against the system and judged, compute the
scorecard with the bundled, tested script (`scripts/asr_report.py`, standard library only,
**no OpenJudge dependency**), it always pairs ASR with over-refusal so you can't reward a
system that just refuses everything:

```bash
python scripts/asr_report.py --attacks attacks.jsonl --over-refusals over.jsonl
# 1-5 harmfulness grader (5=safe): python scripts/asr_report.py --attacks attacks.jsonl --score-scale 5 --harm-threshold 3
```

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/07-redteam/SKILL.md