Model Evaluation
Generates python code that evaluates SageMaker models.
Install
npx skills add awslabs/agent-plugins --skill model-evaluation -g -a claude-code -y- Repository
- awslabs/agent-plugins933
- Category
- Testing
- Updated
- 20d ago
- License
- Apache-2.0
- Context cost
- Lightabout 1,236 tokens
- Runs scripts
- Yes, review before use
What it does
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
Model Evaluation
Generate code that evaluates a SageMaker model.
Prerequisites
- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the
sdk-getting-startedskill first.
Principles
- One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
- Confirm before proceeding. Wait for the user to agree before moving to the next step.
- Don't read files until you need them. Only read reference files when you've reached the step that requires them.
- Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read, use it. Confirm if unsure, but don't re-ask.
- No narration. Share outcomes and ask questions. Keep responses short.
- No repetition. If you said something before a tool call, don't repeat it after.
Scope
This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization, both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.
Tell the user when the skill is activated:
"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."
If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.
Evaluation Types
There are two evaluation types:
- LLM-as-Judge, an LLM grades your model's responses. (OSS models only, not supported for Nova.)
- Custom Scorer, programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.
Workflow
Step 1: Determine evaluation type
Do you already know which evaluation type to use?
