AI Pipeline Orchestration

Build reliable, observable AI workflows, from document ingestion to batch inference to model training pipelines.

When to Use This Skill

Use this skill when:

• Scheduling recurring RAG document ingestion and re-indexing
• Orchestrating multi-step batch LLM processing workflows
• Running nightly model evaluation and fine-tuning jobs
• Building ETL pipelines that feed into AI models
• Managing dependencies between data preparation and model serving

Tool Selection

Tool	Best For	Complexity	GPU Jobs
Prefect	Modern Python-first; easy to adopt	Low	Good
Airflow	Complex DAGs; large teams; existing usage	High	Good
Dagster	Asset-centric; strong data lineage	Medium	Excellent
Temporal	Long-running workflows; reliability-first	Medium	Good

Prefect, Quick Start

pip install prefect prefect-kubernetes

# Start Prefect server (or use Prefect Cloud)
prefect server start

# In another terminal
prefect worker start --pool default-agent-pool

Prefect: RAG Ingestion Pipeline

from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash
from datetime import timedelta
import hashlib

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))
def fetch_documents(source_url: str) -> list[dict]:
    """Fetch documents from source; cached to avoid re-fetching."""
    logger = get_run_logger()
    logger.info(f"Fetching from {source_url}")
    # ... fetch logic
    return documents