Data Research

Structured research pipeline: search sources, extract structured data, archive raw, deduplicate, update canonical trackers, backlink entities.

Contract

One skill for any email-to-structured-data pipeline. The only differences between tracking investor updates, expenses, and company metrics are the search queries, extraction schemas, and tracker page format. All three use the same 7-phase pipeline with parameterized recipes.

When to Use

• User wants to track structured data from email, web, or API sources
• User says "research", "track", "extract from email", "build a tracker"
• User mentions investor updates, donations, company metrics, filings
• User wants to set up recurring data collection (with cron recipe)

Phases

Phase 1: Define Research Recipe

Infer the research target from conversation context, recent brain activity, active tasks (ops/tasks.md), and memory files. If the request is ambiguous, present the most likely interpretation based on what the user has been working on. Only ask for clarification if context is genuinely insufficient. Options:

• Pick a built-in recipe: investor-updates, expense-tracker, company-updates
• Define a custom recipe with: source queries, classification rules, extraction schema, tracker page path, tracker format

Recipes are YAML files at ~/.gbrain/recipes/{name}.yaml. Scaffold a new one by copying a built-in recipe file and editing its fields.

Phase 2: Search Sources

Brain first (maybe we already have this data). Then:

• Email via credential gateway: windowed queries (quarterly, monthly if truncated)
• Web via search: public filings, press releases, regulatory data
• APIs: any structured data source the recipe defines
• Attachments: PDF extraction, HTML stripping

Phase 3: Classify