# Large File

Use BEFORE reading any data file that could be large (CSV/TSV, Parquet, HDF5, FITS...

- Repository: https://github.com/ai4s-research/open-science (2069 stars)
- Category: Engineering
- License: see repository
- Last updated: 2026-11-01
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add ai4s-research/open-science --skill large-file -g -a claude-code -y`
Codex: `npx skills add ai4s-research/open-science --skill large-file -g -a codex -y`
Cursor: `npx skills add ai4s-research/open-science --skill large-file -g -a cursor -y`
Gemini CLI: `npx skills add ai4s-research/open-science --skill large-file -g -a gemini-cli -y`

## Description

Use BEFORE reading any data file that could be large (CSV/TSV, Parquet, HDF5, FITS, NetCDF, NDJSON, genomics FASTQ/FASTA/VCF/BAM, GRIB, ROOT, or big text/simulation logs like VASP OUTCAR). Returns a compact memory pointer, header/schema/shape/sample/key numbers, by introspection and sampling in bounded memory, so you never load a file bigger than the context window into the model. Reference data via the pointer; read specific ranges deterministically.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Large files: reference, don't load

Scientific files routinely dwarf any context window (90 GB FASTQ, multi-GB
HDF5/FITS snapshots, 20 GB+ NetCDF rasters, huge VASP logs). Reading them raw
both OOMs and hallucinates, the materials case that consumed 20M+ tokens and
**failed** succeeded in ~1200 tokens with a memory-pointer approach.

**Rule: never `cat`/read a whole data file into your context.** Probe it first,
work from the returned pointer (schema + sample + key numbers), then read only
the specific rows/columns/ranges you need with the real library.

## Probe a file

The probe ships beside this SKILL.md. Run it on any data file **before** opening
it:

```bash
python "$XDG_CONFIG_HOME/opencode/skills/large-file/large_file_probe.py" DATA_FILE [--sample N]
```

It prints one compact JSON pointer on stdout, always tiny, regardless of file
size (a 13 MB CSV → ~800 bytes; a 16 MB HDF5 → ~450 bytes).

## What you get back

The rest: https://github.com/ai4s-research/open-science/blob/HEAD/runtime/skills/core/large-file/SKILL.md