# Dataset Evaluation

Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or...

- Repository: https://github.com/awslabs/agent-plugins (933 stars)
- Category: Data and ML
- License: Apache-2.0
- Last updated: 2026-10-15
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add awslabs/agent-plugins --skill dataset-evaluation -g -a claude-code -y`
Codex: `npx skills add awslabs/agent-plugins --skill dataset-evaluation -g -a codex -y`
Cursor: `npx skills add awslabs/agent-plugins --skill dataset-evaluation -g -a cursor -y`
Gemini CLI: `npx skills add awslabs/agent-plugins --skill dataset-evaluation -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add awslabs/agent-plugins`

## Description

Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR). Use when the user says "is my dataset okay", "evaluate my data", "check my training data", "I have my own data", or before starting any fine-tuning job. Detects file format, checks schema compliance against the selected model and technique, and reports whether the data is ready for training or evaluation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Workflow Instruction

Follow the workflow shown below. Locate the dataset, check the file type, and resolve any issues with missing files or wrong file types. Determine the fine-tuning model and fine-tuning strategy. Run the appropriate validation based on the model family. Summarize the results: is the dataset ready for fine-tuning?

## Prerequisites

- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the `sdk-getting-started` skill first.

---

## Workflow

1. **Locate Dataset**:
   - The full path may be a local file path, or an S3 URI
   - Resolve the full path to the dataset file, make sure read permissions are available, and help the user if the file is not found

2. **Determine strategy and model**:
   - File formatting depends on the currently selected fine-tuning strategy and fine-tuning base model.
   - If the strategy and model are already known from the conversation context (e.g., selected via the model-selection and finetuning-technique skills), use them.
   - If not available in context, activate the model-selection and/or finetuning-technique skills to determine them before proceeding.
   - **Exception:** If the user is validating an evaluation dataset (not a training dataset), neither model nor technique is required, the format detector can validate eval format (query/response structure) independently. Do not block on model-selection or finetuning-technique for eval dataset validation.

3. **Check File Formatting**: Run the tool format_detector.py to make sure the file conforms to formatting requirements.
   - Send the full path directly to the format_detector script as an argument
   - Do not send the model and strategy as arguments
   - Do not download data from S3
   - Do not make local copies of data

The rest: https://github.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker-ai/skills/dataset-evaluation/SKILL.md