Spark Engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster...
Install
npx skills add Jeffallan/claude-skills --skill spark-engineer -g -a claude-code -y- Repository
- Jeffallan/claude-skills12.2k
- Category
- Testing
- Updated
- 5d ago
- License
- MIT
- Context cost
- Lightabout 1,443 tokens
- Runs scripts
- No, instructions only
What it does
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
Spark Engineer
Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
Core Workflow
- Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
- Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
- Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
- Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
- Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with
df.rdd.getNumPartitions(); if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
Reference Guide
Load detailed guidance based on context:
| Topic | Reference | Load When |
|---|---|---|
| Spark SQL & DataFrames | references/spark-sql-dataframes.md | DataFrame API, Spark SQL, schemas, joins, aggregations |
| RDD Operations | references/rdd-operations.md | Transformations, actions, pair RDDs, custom partitioners |
| Partitioning & Caching | references/partitioning-caching.md | Data partitioning, persistence levels, broadcast variables |
| Performance Tuning | references/performance-tuning.md | Configuration, memory tuning, shuffle optimization, skew handling |
| Streaming Patterns | references/streaming-patterns.md | Structured Streaming, watermarks, stateful operations, sinks |
Code Examples
Quick-Start Mini-Pipeline (PySpark)
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
