# Executing Spark

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a...

- Repository: https://github.com/data-goblin/power-bi-agentic-development (1079 stars)
- Category: Engineering
- License: GPL-3.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -g -a claude-code -y`
Codex: `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -g -a codex -y`
Cursor: `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -g -a cursor -y`
Gemini CLI: `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add data-goblin/power-bi-agentic-development`

## Description

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Executing Spark Code in Fabric (No Notebook)

Run arbitrary PySpark or Python code on Fabric Spark compute via the Livy API. No notebook artifact is created or persisted; sessions are ephemeral. Full read/write access to lakehouse Delta tables via Spark SQL.

## Prerequisites

- Azure CLI authenticated (`az login`)
- A lakehouse in the target workspace (the Livy session runs against it)
- Fabric capacity (F or trial)

## Critical: Authentication

The Livy API requires a token from `az account get-access-token --resource https://api.fabric.microsoft.com`. Tokens from `fab auth` do **not** work for OneLake storage access inside the Spark session.

```python
import subprocess, json

result = subprocess.run(
    ["az", "account", "get-access-token", "--resource", "https://api.fabric.microsoft.com"],
    capture_output=True, text=True
)
token = json.loads(result.stdout)["accessToken"]
```

Do not output or log the token. Pass it directly to the API call.

## Lifecycle

```
1. Create session   POST .../sessions              {"kind": "pyspark"}
2. Wait for idle    GET  .../sessions/{id}          poll until state: "idle" (~30-90s)
3. Submit code      POST .../sessions/{id}/statements   {"code": "...", "kind": "pyspark"}
4. Get result       GET  .../sessions/{id}/statements/{n}   poll until state: "available"
5. Delete session   DELETE .../sessions/{id}        ALWAYS do this
```

Base URL: `https://api.fabric.microsoft.com/v1/workspaces/{wsId}/lakehouses/{lhId}/livyapi/versions/2023-12-01`

**CRITICAL: Always delete sessions when done.** Idle sessions consume Fabric capacity units (CUs). A forgotten session burns compute until it times out (default: 20 minutes). In automation, wrap cleanup in a `finally` block.

## Getting IDs

```bash
WS_ID=$(fab get "Workspace.Workspace" -q "id" | tr -d '"')
LH_ID=$(fab get "Workspace.Workspace/Lakehouse.Lakehouse" -q "id" | tr -d '"')
```

## Submitting Code

The rest: https://github.com/data-goblin/power-bi-agentic-development/blob/HEAD/plugins/etl/skills/executing-spark/SKILL.md