# Data Engineering

Builds and operates data pipelines, ingestion, transformation, orchestration, quality...

- Repository: https://github.com/cbrock84/headcount (2554 stars)
- Category: Design
- License: MIT
- Last updated: 2026-10-18
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add cbrock84/headcount --skill data-engineering -g -a claude-code -y`
Codex: `npx skills add cbrock84/headcount --skill data-engineering -g -a codex -y`
Cursor: `npx skills add cbrock84/headcount --skill data-engineering -g -a cursor -y`
Gemini CLI: `npx skills add cbrock84/headcount --skill data-engineering -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add cbrock84/headcount`

## Description

Builds and operates data pipelines, ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data engineering

Pipelines are production systems whose failures are quiet. A broken service pages someone; a broken
pipeline produces plausible numbers that people act on for a week.

This is movement and transformation. Schema and semantics belong to `data-analytics:data-modeling`,
policy and stewardship to `data-analytics:data-governance`.

## Land raw, transform downstream

Keep an immutable copy of source data exactly as received. Transformation logic will be wrong at some
point, and raw data is what lets you reprocess rather than re-request from a source that may no
longer have it.

Business logic belongs downstream where it is visible and testable, not buried in ingestion. The
exception is transformation required for privacy, minimization, pseudonymization, dropping fields
you have no basis to hold, which belongs at ingest precisely because raw storage is what the
obligation attaches to. See `legal-risk:privacy-and-data-protection`.

## Idempotence is the property that matters

Every pipeline will be re-run: after a failure, after a fix, after a late-arriving correction. A
re-run that double-counts is worse than a failure, because it produces a wrong answer silently.

Design for exactly-once effect at the destination, deterministic keys, merges rather than blind
appends, partitioned overwrites. Then re-running is safe and recovery stops being frightening.

## Late, duplicate and out-of-order data

Real sources deliver all three. Decide explicitly, per pipeline: how late is an event still accepted,
what happens to one arriving after its window closed, and how duplicates are identified.

Distinguish **event time** from **processing time** and partition on event time. Aggregations built
on arrival time silently reassign yesterday's activity to today whenever a delivery is delayed.

## Test data, not just code

The rest: https://github.com/cbrock84/headcount/blob/HEAD/plugins/data-analytics/skills/data-engineering/SKILL.md