Pandas Pro

Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns.

Core Workflow

1. Assess data structure, Examine dtypes, memory usage, missing values, data quality:

   print(df.dtypes)
   print(df.memory_usage(deep=True).sum() / 1e6, "MB")
   print(df.isna().sum())
   print(df.describe(include="all"))
   

2. Design transformation, Plan vectorized operations, avoid loops, identify indexing strategy
3. Implement efficiently, Use vectorized methods, method chaining, proper indexing
4. Validate results, Check dtypes, shapes, null counts, and row counts:

   assert result.shape[0] == expected_rows, f"Row count mismatch: {result.shape[0]}"
   assert result.isna().sum().sum() == 0, "Unexpected nulls after transform"
   assert set(result.columns) == expected_cols
   

5. Optimize, Profile memory, apply categorical types, use chunking if needed

Reference Guide

Load detailed guidance based on context:

Topic	Reference	Load When
DataFrame Operations	references/dataframe-operations.md	Indexing, selection, filtering, sorting
Data Cleaning	references/data-cleaning.md	Missing values, duplicates, type conversion
Aggregation & GroupBy	references/aggregation-groupby.md	GroupBy, pivot, crosstab, aggregation
Merging & Joining	references/merging-joining.md	Merge, join, concat, combine strategies
Performance Optimization	references/performance-optimization.md	Memory usage, vectorization, chunking

Code Patterns

Vectorized Operations (before/after)

# ❌ AVOID: row-by-row iteration
for i, row in df.iterrows():
    df.at[i, 'tax'] = row['price'] * 0.2

# ✅ USE: vectorized assignment
df['tax'] = df['price'] * 0.2

Safe Subsetting with .copy()