<!-- AI-RULEZ :: GENERATED FILE, DO NOT EDIT Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520 Source-Hash: blake3:4a90223a333b0f963ae8cdec4f38db88b769132c5dd1b7f39a7fe8e84f57f83e Schema-Version: v1 -->

Batch extraction

Use this when processing a directory or glob of documents in one pass. xberg batch shares one extraction config across every file, runs extractions concurrently, and returns a JSON envelope, failures on individual files do not abort the run.

Basic usage

# Glob expands to many paths; results come back in a JSON envelope (default)
xberg batch *.pdf

# Mixed formats, markdown content for LLM ingestion
xberg batch docs/*.docx --content-format markdown

# Recurse with the shell, then extract
xberg batch $(find ./corpus -name '*.pdf')

batch defaults to --format json (vs --format text for single extract). The envelope contains results, total_ms, per_file_ms and, when failures occur, errors. Each results entry is a full extracted document. One input may produce multiple documents, so do not pair results with inputs by position. per_file_ms is aligned with inputs; inspect errors for per-input failures.

xberg batch reports/*.pdf \
  | jq '.results[] | {chars: (.content | length), mime: .mime_type}'

Parallelism

--max-concurrent caps how many files extract at once. When omitted, the scheduler derives document concurrency from the total thread budget. Lower it on memory-constrained hosts or when OCR/ML models are active, since each in-flight extraction holds its own buffers. Layout-heavy batches are further limited (1 concurrent extraction for all-PDF-layout batches, 2 for mixed layout):

# Cap at 4 concurrent extractions
xberg batch scans/*.pdf --ocr true --max-concurrent 4

--max-threads additionally caps total internal threads (Rayon, ONNX intra-op, the batch semaphore) for tightly constrained environments: