Upstash Workflow Implementation Guide

Standard patterns for implementing Upstash Workflow + QStash async workflows in the LobeHub codebase.

🎯 The Three Core Patterns

Every workflow in LobeHub combines these three patterns. They exist because the platform constrains you in three ways: rate limits make blind fan-out dangerous, step limits cap a single workflow's size, and idempotency demands that retries don't double-process.

1. 🔍 Dry-Run Mode, get statistics without triggering actual execution
2. 🌟 Fan-Out Pattern, split large batches into smaller chunks for parallel processing
3. 🎯 Single Task Execution, each workflow execution processes exactly ONE item

---

Architecture Overview

All workflows follow the same 3-layer architecture:

Layer 1: Entry Point (process-*)
  ├─ Validates prerequisites
  ├─ Calculates total items to process
  ├─ Filters existing items
  ├─ Supports dry-run mode (statistics only)
  └─ Triggers Layer 2 if work is needed

Layer 2: Pagination (paginate-*)
  ├─ Handles cursor-based pagination
  ├─ Implements fan-out for large batches
  ├─ Recursively processes all pages
  └─ Triggers Layer 3 for each item

Layer 3: Single Task Execution (execute-* / generate-*)
  └─ Performs actual business logic for ONE item

Real examples in this codebase: topicAutoSummary, agentEvalRun, see references/examples.md.

---

The Three Patterns in 60 Seconds

1. Dry-Run Mode

Short-circuit Layer 1 before any side effects so callers can preview what would happen:

if (dryRun) {
  return {
    ...result,
    dryRun: true,
    message: `[DryRun] Would process ${itemsNeedingProcessing.length} items`,
  };
}

Use case: check how many items will be processed before committing.

2. Fan-Out Pattern