# Hdinsight Migration

Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and...

- Repository: https://github.com/microsoft/skills-for-fabric (1306 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-10-29
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add microsoft/skills-for-fabric --skill hdinsight-migration -g -a claude-code -y`
Codex: `npx skills add microsoft/skills-for-fabric --skill hdinsight-migration -g -a codex -y`
Cursor: `npx skills add microsoft/skills-for-fabric --skill hdinsight-migration -g -a cursor -y`
Gemini CLI: `npx skills add microsoft/skills-for-fabric --skill hdinsight-migration -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add microsoft/skills-for-fabric`

## Description

Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

> **Telemetry, MANDATORY.** Every `api.fabric.microsoft.com` call must carry
> `x-ms-fabric-skill: hdinsight-migration` (`az rest`: `--headers "x-ms-fabric-skill=hdinsight-migration"`),
> including every LRO poll, `fabric_lro` and retry. Snippets omit it, add it anyway.

> **CRITICAL NOTES**
> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
> 3. HDInsight has no `mssparkutils` or `dbutils` equivalent, `notebookutils` is net-new capability being introduced
> 4. `HiveContext` and `SQLContext` are legacy Spark 1.x/2.x APIs, Fabric uses Spark 3.x `SparkSession` exclusively
> 5. `wasb://` paths are deprecated and require a Storage Account key or SAS, replace with OneLake shortcuts
> 6. Fabric cannot read or shortcut `hdfs://` directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.

# HDInsight → Microsoft Fabric Migration

## Prerequisite Knowledge

Read these companion documents before executing migration tasks:

- [COMMON-CORE.md](../../common/COMMON-CORE.md), Fabric REST API patterns, authentication, token audiences, item discovery
- [COMMON-CLI.md](../../common/COMMON-CLI.md), `az rest`, `az login`, token acquisition, Fabric REST via CLI
- [SPARK-AUTHORING-CORE.md](../../common/SPARK-AUTHORING-CORE.md), Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see [spark-cli](../spark-cli/SKILL.md).
For Fabric Warehouse DDL/DML authoring, see [sqldw-cli](../sqldw-cli/SKILL.md).

---

## Table of Contents

The rest: https://github.com/microsoft/skills-for-fabric/blob/HEAD/plugins/fabric-skills/skills/hdinsight-migration/SKILL.md