
Scoring Checks
Add a new deterministic scoring check in src/scoring/checks/ that evaluates config...
Install
npx skills add caliber-ai-org/ai-setup --skill scoring-checks -g -a claude-code -y- Repository
- caliber-ai-org/ai-setup1.3k
- Category
- Engineering
- Updated
- 1mo ago
- License
- MIT
- Context cost
- Mediumabout 2,877 tokens
- Runs scripts
- No, instructions only
What it does
Add a new deterministic scoring check in src/scoring/checks/ that evaluates config quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes or refactoring scoring logic.
Adding a Scoring Check
Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.
Critical
- Check must be deterministic: Same filesystem state → same result every time. No randomness, no external APIs.
- Point values come from constants.ts: Every
earnedPointsandmaxPointsmust referencePOINTS_*fromsrc/scoring/constants.ts. Do NOT hardcode numbers. - Always return
Check[]array: Export a functioncheck<Category>(dir: string): Check[]where category is one of:existence,quality,grounding,accuracy,freshness,bonus. - Every check must have:
id(kebab-case, unique),name,category,maxPoints,earnedPoints,passed,detail, and optionalsuggestion/fix. - Fix object fields:
action(string describing what to do),data(context for the fix),instruction(user-facing guidance). - Register in src/scoring/index.ts: Add the import and spread the result into the
allChecksarray incomputeLocalScore(). - Target filtering: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate
*_ONLY_CHECKSset inconstants.ts.
Instructions
Step 1: Define point constants in src/scoring/constants.ts
Verify before proceeding: Is your check measurable with a numeric point value?
Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):
// In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical
// If threshold-based, add a companion array:
export const YOUR_THRESHOLD_ARRAY = [
{ minValue: 10, points: 4 },
{ minValue: 5, points: 2 },
] as const;
