Large File
Use BEFORE reading any data file that could be large (CSV/TSV, Parquet, HDF5, FITS...
Install
npx skills add ai4s-research/open-science --skill large-file -g -a claude-code -y- Repository
- ai4s-research/open-science2.1k
- Category
- Engineering
- Updated
- 3d ago
- License
- See repository
- Context cost
- Lightabout 836 tokens
- Runs scripts
- Yes, review before use
What it does
Use BEFORE reading any data file that could be large (CSV/TSV, Parquet, HDF5, FITS, NetCDF, NDJSON, genomics FASTQ/FASTA/VCF/BAM, GRIB, ROOT, or big text/simulation logs like VASP OUTCAR). Returns a compact memory pointer, header/schema/shape/sample/key numbers, by introspection and sampling in bounded memory, so you never load a file bigger than the context window into the model. Reference data via the pointer; read specific ranges deterministically.
Large files: reference, don't load
Scientific files routinely dwarf any context window (90 GB FASTQ, multi-GB HDF5/FITS snapshots, 20 GB+ NetCDF rasters, huge VASP logs). Reading them raw both OOMs and hallucinates, the materials case that consumed 20M+ tokens and failed succeeded in ~1200 tokens with a memory-pointer approach.
Rule: never cat/read a whole data file into your context. Probe it first,
work from the returned pointer (schema + sample + key numbers), then read only
the specific rows/columns/ranges you need with the real library.
Probe a file
The probe ships beside this SKILL.md. Run it on any data file before opening it:
python "$XDG_CONFIG_HOME/opencode/skills/large-file/large_file_probe.py" DATA_FILE [--sample N]
It prints one compact JSON pointer on stdout, always tiny, regardless of file size (a 13 MB CSV → ~800 bytes; a 16 MB HDF5 → ~450 bytes).
