# Model Evaluation

Generates python code that evaluates SageMaker models.

- Repository: https://github.com/awslabs/agent-plugins (933 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-15
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add awslabs/agent-plugins --skill model-evaluation -g -a claude-code -y`
Codex: `npx skills add awslabs/agent-plugins --skill model-evaluation -g -a codex -y`
Cursor: `npx skills add awslabs/agent-plugins --skill model-evaluation -g -a cursor -y`
Gemini CLI: `npx skills add awslabs/agent-plugins --skill model-evaluation -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add awslabs/agent-plugins`

## Description

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Model Evaluation

Generate code that evaluates a SageMaker model.

## Prerequisites

- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the `sdk-getting-started` skill first.

## Principles

1. **One thing at a time.** Each response advances exactly one decision. Never combine multiple questions in a single turn.
2. **Confirm before proceeding.** Wait for the user to agree before moving to the next step.
3. **Don't read files until you need them.** Only read reference files when you've reached the step that requires them.
4. **Don't ask what you already know.** If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read, use it. Confirm if unsure, but don't re-ask.
5. **No narration.** Share outcomes and ask questions. Keep responses short.
6. **No repetition.** If you said something before a tool call, don't repeat it after.

## Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization, both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

> "I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

## Evaluation Types

There are two evaluation types:

- **LLM-as-Judge**, an LLM grades your model's responses. (OSS models only, not supported for Nova.)
- **Custom Scorer**, programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

## Workflow

### Step 1: Determine evaluation type

**Do you already know which evaluation type to use?**

The rest: https://github.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker-ai/skills/model-evaluation/SKILL.md