Dask
Scales pandas, NumPy, and custom Python research workflows beyond memory or across...
Install
npx skills add K-Dense-AI/scientific-agent-skills --skill dask -g -a claude-code -y- Repository
- K-Dense-AI/scientific-agent-skills55.7k
- Category
- Data and ML
- Updated
- 9d ago
- License
- BSD-3-Clause license
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
Scales pandas, NumPy, and custom Python research workflows beyond memory or across clusters with Dask. Covers DataFrames, Arrays, Bags, Futures, chunking, schedulers, and distributed diagnostics. Use for partitioned file processing, scientific array computation, or parallel tasks whose memory and dependency structure require Dask.
Dask
Overview
Dask is a Python library for parallel and distributed computing that enables three critical capabilities:
- Larger-than-memory execution on single machines for data exceeding available RAM
- Parallel processing for improved computational speed across multiple cores
- Distributed computation supporting terabyte-scale datasets across multiple machines
Capacity depends on partition sizes, intermediate data, concurrency, storage, and available memory.
Reviewed release: dask/distributed 2026.8.0 (September 30, 2026 review). Docs: docs.dask.org. Since 2025.1.0, the expression-based DataFrame API with query planning is the only implementation, do not install dask-expr separately or set dataframe.query-planning: False.
Quick Start
Installation
uv pip install "dask[array,dataframe]==2026.8.0"
For a typical pandas/NumPy workflow with the distributed scheduler and dashboard:
uv pip install "dask[complete]==2026.8.0"
Remote object storage (requires provider credentials for private data):
uv pip install s3fs # s3:// paths
uv pip install gcsfs # gs:// paths
Requires Python 3.10+, pandas 2+, PyArrow 16+. Zarr, HDF5, SciPy, Xarray, Dask-ML, and cluster deployment packages are separate optional dependencies. See review and validation for tested versions, source links, and limits. File paths and undefined application functions below are illustrative. Put process scheduler/Client() execution in a main() guarded by if __name__ == "__main__": in scripts; close clients and clusters afterward.
