# Data Pipeline

Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

- Repository: https://github.com/RightNow-AI/openfang (18244 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-07-02
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add RightNow-AI/openfang --skill data-pipeline -g -a claude-code -y`
Codex: `npx skills add RightNow-AI/openfang --skill data-pipeline -g -a codex -y`
Cursor: `npx skills add RightNow-AI/openfang --skill data-pipeline -g -a cursor -y`
Gemini CLI: `npx skills add RightNow-AI/openfang --skill data-pipeline -g -a gemini-cli -y`

## Description

Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data Pipeline Expert

A data engineering specialist with extensive experience designing and operating production ETL/ELT pipelines, orchestration frameworks, and data quality systems. This skill provides guidance for building reliable, observable, and scalable data pipelines using industry-standard tools like Apache Airflow, Spark, and dbt across batch and streaming architectures.

## Key Principles

- Prefer ELT over ETL when your target warehouse can handle transformations; load raw data first, then transform in place for reproducibility and auditability
- Design every pipeline step to be idempotent; re-running a task with the same inputs must produce the same outputs without side effects or duplicates
- Partition data by time or logical keys at every stage; partitioning enables incremental processing, efficient pruning, and manageable backfill operations
- Instrument pipelines with data quality checks between stages; catching bad data early prevents cascading corruption through downstream tables
- Separate orchestration (when and what order) from computation (how); the scheduler should not perform heavy data processing itself

## Techniques

The rest: https://github.com/RightNow-AI/openfang/blob/HEAD/crates/openfang-skills/bundled/data-pipeline/SKILL.md