# Transforming Data

Transform raw data into analytical assets using ETL/ELT patterns, SQL (dbt), Python...

- Repository: https://github.com/ancoleman/ai-design-components (526 stars)
- Category: Testing
- License: MIT
- Last updated: 2025-12-11
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add ancoleman/ai-design-components --skill transforming-data -g -a claude-code -y`
Codex: `npx skills add ancoleman/ai-design-components --skill transforming-data -g -a codex -y`
Cursor: `npx skills add ancoleman/ai-design-components --skill transforming-data -g -a cursor -y`
Gemini CLI: `npx skills add ancoleman/ai-design-components --skill transforming-data -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ancoleman/ai-design-components`

## Description

Transform raw data into analytical assets using ETL/ELT patterns, SQL (dbt), Python (pandas/polars/PySpark), and orchestration (Airflow). Use when building data pipelines, implementing incremental models, migrating from pandas to polars, or orchestrating multi-step transformations with testing and quality checks.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data Transformation

Transform raw data into analytical assets using modern transformation patterns, frameworks, and orchestration tools.

## Purpose

Select and implement data transformation patterns across the modern data stack. Transform raw data into clean, tested, and documented analytical datasets using SQL (dbt), Python DataFrames (pandas, polars, PySpark), and pipeline orchestration (Airflow, Dagster, Prefect).

## When to Use

Invoke this skill when:

- Choosing between ETL and ELT transformation patterns
- Building dbt models (staging, intermediate, marts)
- Implementing incremental data loads and merge strategies
- Migrating pandas code to polars for performance improvements
- Orchestrating data pipelines with dependencies and retries
- Adding data quality tests and validation
- Processing large datasets with PySpark
- Creating production-ready transformation workflows

## Quick Start: Common Patterns

### dbt Incremental Model

```sql
{{
  config(
    materialized='incremental',
    unique_key='order_id'
  )
}}

select order_id, customer_id, order_created_at, sum(revenue) as total_revenue
from {{ ref('int_order_items_joined') }}
group by 1, 2, 3

{% if is_incremental() %}
    where order_created_at > (select max(order_created_at) from {{ this }})
{% endif %}
```

### polars High-Performance Transformation

```python
import polars as pl

result = (
    pl.scan_csv('large_dataset.csv')
    .filter(pl.col('year') == 2024)
    .with_columns([(pl.col('quantity') * pl.col('price')).alias('revenue')])
    .group_by('region')
    .agg(pl.col('revenue').sum())
    .collect()  # Execute lazy query
)
```

### Airflow Data Pipeline

```python
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

The rest: https://github.com/ancoleman/ai-design-components/blob/HEAD/skills/transforming-data/SKILL.md