Rerun Chunk Processing
Core mechanics of the Rerun Chunk Processing API (rerun.chunk), LazyChunkStream...
Install
npx skills add rerun-io/rerun --skill rerun-chunk-processing -g -a claude-code -y- Repository
- rerun-io/rerun11.7k
- Category
- Data and ML
- Updated
- 3d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
Core mechanics of the Rerun Chunk Processing API (rerun.chunk), LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks, no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.
Rerun chunk processing
The pipeline layer between raw data and an RRD: readers produce Chunks,
streams transform them, terminal calls execute. This skill is the generic
mechanics only. Decide the data model first (rerun-data-model), then pick the
importer skill for each source:
| Source | Reader | Skill |
|---|---|---|
| MCAP file (ROS2, protobuf, Foxglove) | McapReader(path).stream() | rerun-mcap |
| URDF robot model (+ joint states → FK) | UrdfTree.from_file_path(...).stream() | rerun-urdf |
| Parquet table (trajectories, sensor logs) | ParquetReader(path).stream() | rerun-parquet |
| mp4 camera video | Mp4Reader(path).stream() | rerun-mp4 |
| HDF5 file (groups → entities) | Hdf5Reader(path).stream() | none yet |
| LeRobot dataset directory | built-in importer, then RrdReader | rerun-lerobot |
| Existing RRD | RrdReader(path) | here, below |
| Sidecar files (JSON calib, metadata) | Chunk.from_columns + from_iter | here, below |
The API is rerun.chunk; when
behavior matters, check the installed surface:
python -c "from rerun.chunk import LazyChunkStream; help(LazyChunkStream)".
Decision rule: where does each component come from?
Default: a reader produces the chunks; lenses shape them. Walk this before writing any conversion code, most "build it by hand" instincts are wrong here:
