Sn Da Large File Analysis
万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet...
Install
npx skills add OpenSenseNova/SenseNova-Skills --skill sn-da-large-file-analysis -g -a claude-code -y- Repository
- OpenSenseNova/SenseNova-Skills6k
- Category
- Documents
- Updated
- 6d ago
- License
- MIT
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
万行以上 Excel 数据集的高性能分析引擎。提供 openpyxl read_only 流式读取（iter_rows 支持 10 万行以上）、Parquet 转换加速、内存优化、分块处理和大文件写入模式。**遇到以下任一情况就主动使用本 skill**：①数据行数 ≥ 10k（由 sn-da-excel-workflow 的行数评估步骤触发）；②用户出现触发词：大文件 / 大数据量 / 性能优化 / 内存不足 / OOM / 百万行 / 十万行 / 流式读取 / Parquet / 分块处理 / large file / big data / streaming read / chunked processing；③直接使用 pd.read_excel() 导致超时或内存溢出；④用户明确要求对大规模数据集进行高性能处理。仅不用于：小于 10k 行的常规 Excel 分析（使用 sn-da-excel-workflow 即可）。
Large Scale Excel Analysis Skill
Mandatory Rules
When total rows >= 10,000, you MUST use the methods in this skill.
| Data Scale | Read Strategy | Reason |
|---|---|---|
| < 10k rows | pd.read_excel() directly | No memory pressure |
| 10k, 100k rows | pd.read_excel() → convert to Parquet → pd.read_parquet() for analysis | Avoid repeated slow reads |
| 100k, 1M rows | openpyxl read_only + iter_rows streaming → Parquet | pd.read_excel() will OOM or timeout |
| > 1M rows | Streaming read + multi-sheet split (Excel max 1,048,576 rows per sheet) | Must chunk |
Prohibited:
- Do NOT use
pd.read_excel()to fully load 100k+ row files - Do NOT search for fonts with
fc-list,find ... fonts, or install packages withpip install - Do NOT use
df.iterrows()on large DataFrames (useitertuples()or vectorized ops) - Do NOT use
df.apply(lambda...)for operations that can be vectorized
Environment Setup
import pandas as pd
import numpy as np
import os
import gc
pd.options.mode.copy_on_write = True
# CJK font setup (fixed paths, do NOT search for fonts)
# ⚠️ Copy this block as-is. Do NOT use fc-list, find, subprocess, or glob to locate fonts.
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.font_manager as fm
_FONT_PATHS = [
'/mnt/afs_agents/SimHei.ttf',
'/mnt/afs_agents/mnt/data/SimHei.ttf',
os.path.expanduser('~/.fonts/SimHei.ttf'),
'/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc',
'/usr/share/fonts/SimHei.ttf',
]
for _p in _FONT_PATHS:
if os.path.exists(_p):
fm.fontManager.addfont(_p)
matplotlib.rcParams['font.family'] = fm.FontProperties(fname=_p).get_name()
break
matplotlib.rcParams['axes.unicode_minus'] = False
Core Method 1: Inspect File Structure (Without Loading Data)
Before any operation on a large file, inspect sheets and row counts without loading data into memory:
import openpyxl
