Sigma Ingest

Sigma ingest turns staged data model specs and workbook summaries into durable ktx wiki knowledge. The deterministic project() step has already written semantic-layer YAML for all warehouse-table data model elements before this skill runs, do not re-write those SL sources.

Work unit structure

Sigma produces at minimum two work units per ingest run:

• sigma-data-models or sigma-data-models-N
  • rawFiles: data-models/<id>.json files (one per data model in this batch)
  • peerFileIndex: workbooks/<id>.json files + sigma-manifest.json + sigma-projection-config.json
  • When the workspace has more than 50 data models, split into batches: sigma-data-models-0, sigma-data-models-1, … with displayLabel like "Sigma: data models (1/8)". When ≤50 data models, the unitKey is simply sigma-data-models with no suffix.
• sigma-workbooks or sigma-workbooks-N
  • rawFiles: workbooks/<id>.json files (one per workbook in this batch)
  • peerFileIndex: data-models/<id>.json files + sigma-manifest.json + sigma-projection-config.json
  • When the workspace has more than 2000 workbooks, split into batches: sigma-workbooks-0, sigma-workbooks-1, … with displayLabel like "Sigma: workbooks (1/4)". When ≤2000 workbooks, the unitKey is simply sigma-workbooks with no suffix.

sigma-manifest.json and sigma-projection-config.json are never in rawFiles. They live at the staged dir root and always appear in peerFileIndex.

Staged file shapes