# Experiment Queue

SSH job queue for multi-seed/multi-config ML experiments with OOM-aware retry...

- Repository: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep (18443 stars)
- Category: Data and ML
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-queue -g -a claude-code -y`
Codex: `npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-queue -g -a codex -y`
Cursor: `npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-queue -g -a cursor -y`
Gemini CLI: `npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-queue -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add wanshuiyin/Auto-claude-code-research-in-sleep`

## Description

SSH job queue for multi-seed/multi-config ML experiments with OOM-aware retry, stale-screen cleanup, and wave-transition race prevention. Use when user says "batch experiments", "队列实验", "run grid", "multi-seed sweep", "auto-chain experiments", or when /run-experiment is insufficient for 10+ jobs that need orchestration.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Experiment Queue

> ⏱ **External cadence: visibility only.** This skill already runs its own
> detached server-side scheduler (60s poll + `depends_on` + wave transitions).
> Use its status output for overnight visibility (N done / N running / N
> pending); do **not** wrap it in a second `/loop` / `CronCreate` poll, that
> duplicates the scheduler on an uncoordinated clock and races the
> wave-transition logic it was built to prevent. See
> [`shared-references/external-cadence.md`](../shared-references/external-cadence.md)
> ("don't duplicate an existing scheduler").

Orchestrate large batches of ML experiments on SSH remote GPU servers with proper state tracking, OOM retry, stale cleanup, and wave transitions.

## When to Use This Skill

Use when `/run-experiment` is insufficient:
- **≥10 jobs** that need batching across GPUs
- **Multi-seed sweeps** (e.g., 21 seeds × 12 cells)
- **Wave transitions** (run wave 1, wait, run wave 2, wait, run wave 3...)
- **Teacher+student chains** (train teacher then distill; auto-trigger student after teacher done)
- **OOM-prone configs** where you need to retry with different GPU or wait
- **Mixed seed grids** where failed cells need re-running

Do NOT use for:
- Single ad-hoc experiment (use `/run-experiment`)
- Modal/Vast.ai deployments (those have their own orchestration)
- Experiments that need manual inspection between runs

## Why This Exists

Based on session audit (2026-04-16), the major wall-clock sinks in multi-seed grid experiments are:

1. **Stale screens**, python finishes, wandb uploads, screen hangs, next wave blocked
2. **OOM on shared GPU**, previous job's memory not yet released
3. **Wave race**, new wave launches before previous wave fully settles
4. **Missing checkpoints**, student launches before teacher saved
5. **Parser duplication**, rewriting multi-seed analysis python every batch

All of these are pure engineering friction that can be orchestrated.

## Core Concepts

The rest: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/blob/HEAD/skills/experiment-queue/SKILL.md