Xberg Chunking
Use when splitting extracted text into chunks for LLM context windows or RAG ingestion.
Install
npx skills add xberg-io/xberg --skill xberg-chunking -g -a claude-code -y- Repository
- xberg-io/xberg9.5k
- Category
- AI and agents
- Updated
- 3d ago
- License
- MIT
- Context cost
- Mediumabout 1,511 tokens
- Runs scripts
- No, instructions only
What it does
Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.
Chunking
Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's chunks field), or standalone via the chunk command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.
Inline during extraction
Turn on chunking with --chunk and the chunks appear on the structured
result under chunks:
# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
xberg extract report.pdf --chunk --format json | jq '.result.chunks | length'
# Explicit size + overlap
xberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json
Overlap must be smaller than chunk size, the CLI rejects
--chunk-overlap >= --chunk-size. When you set only --chunk-overlap
against an existing config, an overlap that exceeds the size is clamped to
chunk_size / 4.
Standalone chunk command
Chunk text you already have, from --text or stdin. Output defaults to
JSON:
# From a flag
xberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100
# From stdin (pipe extracted content straight in)
xberg extract notes.md | xberg chunk --chunk-size 500 --format json
JSON output carries chunks (array of strings), chunk_count, the
resolved config (max_characters, overlap, chunker_type), and
input_size_bytes. Use --format text for a human-readable dump with
--- chunk N --- separators.
