Skills

Scoring Checks

Add a new deterministic scoring check in src/scoring/checks/ that evaluates config...

Install

npx skills add caliber-ai-org/ai-setup --skill scoring-checks -g -a claude-code -y
Category
Engineering
Updated
1mo ago
License
MIT
Context cost
Mediumabout 2,877 tokens
Runs scripts
No, instructions only

What it does

Add a new deterministic scoring check in src/scoring/checks/ that evaluates config quality. Follows the Check[] return pattern, uses point constants from src/scoring/constants.ts, and integrates via filterChecksForTarget() in src/scoring/index.ts. Use when user says 'add scoring check', 'new check', 'modify scoring criteria', or works in src/scoring/checks/. Do NOT use for display changes or refactoring scoring logic.

Adding a Scoring Check

Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.

Critical

  • Check must be deterministic: Same filesystem state → same result every time. No randomness, no external APIs.
  • Point values come from constants.ts: Every earnedPoints and maxPoints must reference POINTS_* from src/scoring/constants.ts. Do NOT hardcode numbers.
  • Always return Check[] array: Export a function check<Category>(dir: string): Check[] where category is one of: existence, quality, grounding, accuracy, freshness, bonus.
  • Every check must have: id (kebab-case, unique), name, category, maxPoints, earnedPoints, passed, detail, and optional suggestion/fix.
  • Fix object fields: action (string describing what to do), data (context for the fix), instruction (user-facing guidance).
  • Register in src/scoring/index.ts: Add the import and spread the result into the allChecks array in computeLocalScore().
  • Target filtering: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate *_ONLY_CHECKS set in constants.ts.

Instructions

Step 1: Define point constants in src/scoring/constants.ts

Verify before proceeding: Is your check measurable with a numeric point value?

Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):

// In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical

// If threshold-based, add a companion array:
export const YOUR_THRESHOLD_ARRAY = [
  { minValue: 10, points: 4 },
  { minValue: 5, points: 2 },
] as const;
Read the whole SKILL.md on GitHub

More in Engineering