Data Catalog Workflow

Goal

Produce a customer-readable Markdown catalog whose YAML frontmatter is a valid DS_CATALOG_V1 machine contract. Preserve uncertainty explicitly so inferred or assumed relationships never appear confirmed.

Flow

1. Confirm the engagement name and the caller-approved durable output path.
2. Inventory entities at business grain. Record source access, tier, volume, profile pointer, classification, lineage, and open questions without copying column-level profile data.
3. Assign every relationship a stable rel-* identifier. Record endpoints, maximum cardinality, both endpoint minimums, one or more paired join-key fields, confidence, and evidence basis.
4. Reconcile coverage counts with the entity and relationship records.
5. Render the human-readable sections from the YAML facts, ending with the canonical Data Science and Engineering Coaching disclaimer footer. Narrative can explain facts but cannot redefine them.
6. Validate the artifact with scripts/validate_catalog.py before treating it as ready for review.

Inputs

• Engagement context and a caller-approved output path
• Data source inventory and access status
• Business entity names, grain, and declared relationships
• Existing per-dataset profile paths, when available
• Privacy classifications or standards citations produced by the owning privacy workflow

Success criteria