# Sn Da Large File Analysis

万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet...

- Repository: https://github.com/OpenSenseNova/SenseNova-Skills (6002 stars)
- Category: Documents
- License: MIT
- Last updated: 2026-10-29
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis -g -a claude-code -y`
Codex: `npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis -g -a codex -y`
Cursor: `npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis -g -a cursor -y`
Gemini CLI: `npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis -g -a gemini-cli -y`

## Description

万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet 转换加速、内存优化、分块处理和大文件写入模式。**遇到以下任一情况就主动使用本 skill**：①数据行数 ≥ 10k（由 sn-da-excel-workflow 的行数评估步骤触发）；②用户出现触发词：大文件 / 大数据量 / 性能优化 / 内存不足 / OOM / 百万行 / 十万行 / 流式读取 / Parquet / 分块处理 / large file / big data / streaming read / chunked processing；③直接使用 pd.read_excel() 导致超时或内存溢出；④用户明确要求对大规模数据集进行高性能处理。仅不用于：小于 10k 行的常规 Excel 分析（使用 sn-da-excel-workflow 即可）。

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Large Scale Excel Analysis Skill

## Mandatory Rules

> **When total rows >= 10,000, you MUST use the methods in this skill.**

| Data Scale | Read Strategy | Reason |
|-----------|---------------|--------|
| < 10k rows | `pd.read_excel()` directly | No memory pressure |
| 10k, 100k rows | `pd.read_excel()` → convert to Parquet → `pd.read_parquet()` for analysis | Avoid repeated slow reads |
| 100k, 1M rows | **openpyxl `read_only` + `iter_rows` streaming** → Parquet | `pd.read_excel()` will OOM or timeout |
| > 1M rows | Streaming read + **multi-sheet split** (Excel max 1,048,576 rows per sheet) | Must chunk |

**Prohibited:**
- Do NOT use `pd.read_excel()` to fully load 100k+ row files
- Do NOT search for fonts with `fc-list`, `find ... fonts`, or install packages with `pip install`
- Do NOT use `df.iterrows()` on large DataFrames (use `itertuples()` or vectorized ops)
- Do NOT use `df.apply(lambda...)` for operations that can be vectorized

---

## Environment Setup

```python
import pandas as pd
import numpy as np
import os
import gc

pd.options.mode.copy_on_write = True

# CJK font setup (fixed paths, do NOT search for fonts)
# ⚠️ Copy this block as-is. Do NOT use fc-list, find, subprocess, or glob to locate fonts.
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.font_manager as fm

_FONT_PATHS = [
    '/mnt/afs_agents/SimHei.ttf',
    '/mnt/afs_agents/mnt/data/SimHei.ttf',
    os.path.expanduser('~/.fonts/SimHei.ttf'),
    '/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc',
    '/usr/share/fonts/SimHei.ttf',
]
for _p in _FONT_PATHS:
    if os.path.exists(_p):
        fm.fontManager.addfont(_p)
        matplotlib.rcParams['font.family'] = fm.FontProperties(fname=_p).get_name()
        break
matplotlib.rcParams['axes.unicode_minus'] = False
```

---

## Core Method 1: Inspect File Structure (Without Loading Data)

Before any operation on a large file, inspect sheets and row counts **without loading data into memory**:

```python
import openpyxl

The rest: https://github.com/OpenSenseNova/SenseNova-Skills/blob/HEAD/skills/sn-da-large-file-analysis/SKILL.md