# Write Pipeline

MUST use when creating or modifying a data pipeline, a set of scripts marked `pipeline`...

- Repository: https://github.com/windmill-labs/windmill (18352 stars)
- Category: Engineering
- License: see repository
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add windmill-labs/windmill --skill write-pipeline -g -a claude-code -y`
Codex: `npx skills add windmill-labs/windmill --skill write-pipeline -g -a codex -y`
Cursor: `npx skills add windmill-labs/windmill --skill write-pipeline -g -a cursor -y`
Gemini CLI: `npx skills add windmill-labs/windmill --skill write-pipeline -g -a gemini-cli -y`

## Description

MUST use when creating or modifying a data pipeline, a set of scripts marked `pipeline` and wired together by `on` / `materialize` annotations.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data pipeline authoring

A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of **independent scripts**, each deployed on its own, that form a DAG by reading and writing shared **storage assets** (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution **triggers**. The pipeline is visualized and edited at `/pipeline/<folder>`; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a "data pipeline" (or to "ingest / transform / materialize" data across steps), build pipeline-annotated scripts, do NOT build a flow.

## Default to DuckDB + DuckLake

A pipeline node that produces a table should almost always be a **`duckdb`** node that materializes its output into a **DuckLake** table with `-- materialize ducklake://<name>/<table>` (in a DuckDB node the annotation uses SQL `--` comment syntax; write the body as a bare `SELECT` and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else:

- `postgresql` / data tables, only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live).
- `bun` / `python3`, only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it **straight into DuckLake** (or a data table) from the same script, see "Ingesting from Python / TypeScript" below, rather than inventing a parallel store.

Do not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal.

## Ingesting from Python / TypeScript

The rest: https://github.com/windmill-labs/windmill/blob/HEAD/system_prompts/auto-generated/skills/write-pipeline/SKILL.md