# Xberg Chunking

Use when splitting extracted text into chunks for LLM context windows or RAG ingestion.

- Repository: https://github.com/xberg-io/xberg (9506 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add xberg-io/xberg --skill xberg-chunking -g -a claude-code -y`
Codex: `npx skills add xberg-io/xberg --skill xberg-chunking -g -a codex -y`
Cursor: `npx skills add xberg-io/xberg --skill xberg-chunking -g -a cursor -y`
Gemini CLI: `npx skills add xberg-io/xberg --skill xberg-chunking -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add xberg-io/xberg`

## Description

Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<!--
AI-RULEZ :: GENERATED FILE, DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:4a90223a333b0f963ae8cdec4f38db88b769132c5dd1b7f39a7fe8e84f57f83e
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

## Inline during extraction

Turn on chunking with `--chunk` and the chunks appear on the structured
result under `chunks`:

```bash
# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
xberg extract report.pdf --chunk --format json | jq '.result.chunks | length'

# Explicit size + overlap
xberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json
```

Overlap must be smaller than chunk size, the CLI rejects
`--chunk-overlap >= --chunk-size`. When you set only `--chunk-overlap`
against an existing config, an overlap that exceeds the size is clamped to
`chunk_size / 4`.

## Standalone `chunk` command

Chunk text you already have, from `--text` or stdin. Output defaults to
JSON:

```bash
# From a flag
xberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100

# From stdin (pipe extracted content straight in)
xberg extract notes.md | xberg chunk --chunk-size 500 --format json
```

JSON output carries `chunks` (array of strings), `chunk_count`, the
resolved `config` (`max_characters`, `overlap`, `chunker_type`), and
`input_size_bytes`. Use `--format text` for a human-readable dump with
`--- chunk N ---` separators.

The rest: https://github.com/xberg-io/xberg/blob/HEAD/plugin/.cursor-plugin/skills/xberg-chunking/SKILL.md