Data Pipelines

This is an OMH data-pipelines workflow skill, projected for Agent Skills hosts (Claude Code, Codex, Cursor, opencode, OpenClaw, pi).

Why This Exists

data-pipelines exists because pipeline work had no owner: backend owns a service's schema migration, data-analysis analyzes data it is handed, and relational-db owns a database's locks and indexes, while a backfill that duplicated events or a schema change with unknown readers reached memory and event lanes with no idempotency contract or replay bound at all.

First Steps

• Ask what makes a row unique at the sink before planning any rerun.
• Bound the window and the targets before ordering any replay or backfill step.

Do Not Use When

• The ask is a service's own database migration, API, or queue design; use omh-backend.
• The ask is analyzing, charting, or summarizing a dataset that was handed over; use omh-data-analysis.
• The ask is a slow query, an index, or DDL locking a live table; use omh-relational-db.
• The ask is remembering or syncing what the assistant knows about the user.

Examples

Good example:

• Prompt: our airflow etl backfill is producing duplicate events
• Expected behavior: Find the sink's unique key, name the append that duplicated rows, write the idempotency contract (event-id dedupe or partition overwrite), then bound the backfill window and gate it on key uniqueness and row count against the prior window.
• Why: Rerunning an appending backfill doubles the duplicates it was meant to fix.

Bad example:

• Prompt: just delete the duplicates and rerun the whole history
• Expected behavior: Refuse the unbounded rerun: fix the write to be idempotent first, then backfill a bounded window behind a quality gate.
• Why: Deleting duplicates without fixing the write guarantees the next rerun duplicates again.

Completion Checklist