Rerun Catalog Queries
Performance patterns and gotchas for querying a Rerun catalog from Python.
Install
npx skills add rerun-io/rerun --skill rerun-catalog-queries -g -a claude-code -y- Repository
- rerun-io/rerun11.7k
- Category
- Data and ML
- Updated
- 3d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.
Rerun catalog queries
Practical performance patterns for querying a Rerun catalog
from Python (rerun.catalog.CatalogClient →
dataset.reader(...) → DataFusion DataFrame). The DataFusion side of
the stack is covered by the datafusion-python skill, load that
for DataFrame / SessionContext / expression-API references. This
skill focuses on catalog-specific behaviors and the round-trip costs
that catch teams off guard.
Connect with a rerun:// URL
from rerun.catalog import CatalogClient
client = CatalogClient("rerun+https://api.example.cloud.rerun.io:443")
dataset = client.get_dataset(id=dataset_id)
CatalogClient takes a rerun:// or rerun+https:// URL, or
rerun+http:// for a local server. Do not pass a plain https:// URL.
A segment URL such as
rerun://host:443/dataset/<dataset-id>?segment_id=<segment-id> gives
the server, the dataset id and the segment id.
The client reads the token from REDAP_TOKEN, or from the stored
credentials of rerun auth login, so token= is usually not needed.
The query cost model in one sentence
Every materialization of a dataset.reader(...) DataFrame is one
cloud round-trip, with most of the cost being the network/decode
pair, not the compute. Plan for round-trip count and payload bytes, in
that order.
A typical catalog round-trip is a few seconds even for a tiny result. So:
- 30 segments × 1 query each ≈ 90s. (Naive per-segment loop.)
- 30 segments × 4 queries each ≈ 6 minutes. (A splitter that runs
count+collect_columnfor both starts and stops.) - 1 query covering all 30 segments ≈ 3s.
The same fan-out happens along the entity axis: 10 entities ×
1 filter_contents([one_entity]).reader() each ≈ 30s, vs one
filter_contents([all_entities]).reader() ≈ 3s. Push as much as you
can into one round-trip, across segments and across entities.
