# Wigolo Crawl

Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies...

- Repository: https://github.com/KnockOutEZ/wigolo (5517 stars)
- Category: Writing
- License: AGPL-3.0-only
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add KnockOutEZ/wigolo --skill wigolo-crawl -g -a claude-code -y`
Codex: `npx skills add KnockOutEZ/wigolo --skill wigolo-crawl -g -a codex -y`
Cursor: `npx skills add KnockOutEZ/wigolo --skill wigolo-crawl -g -a cursor -y`
Gemini CLI: `npx skills add KnockOutEZ/wigolo --skill wigolo-crawl -g -a gemini-cli -y`

## Description

Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later `cache` / `find_similar` queries.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# wigolo crawl

Crawl sites with configurable strategy, depth, and rate limiting. All pages enter the local cache with embeddings.

## Quick Reference

```json
// Crawl docs via sitemap (fastest, recommended for doc sites)
{ "url": "https://docs.example.com", "strategy": "sitemap", "max_pages": 30 }

// BFS crawl with scope filter
{ "url": "https://example.com", "strategy": "bfs", "max_depth": 3, "max_pages": 50, "include_patterns": ["^https://example\\.com/docs"] }

// URL discovery only (no content fetched, fastest for scoping)
{ "url": "https://example.com", "strategy": "map" }

// Authenticated crawl
{ "url": "https://app.example.com/docs", "strategy": "bfs", "use_auth": true, "max_pages": 20 }
```

## Parameters

| Parameter | Type | Default | When to use |
|-----------|------|---------|-------------|
| `url` | string | required | Seed URL |
| `strategy` | string | "bfs" | "sitemap" for doc sites, "map" for URL discovery only |
| `max_depth` | number | 2 | How many link levels to follow |
| `max_pages` | number | 20 | Hard cap on pages fetched |
| `include_patterns` | string[] | none | Regex whitelist, ALWAYS add to stay in scope |
| `exclude_patterns` | string[] | none | Regex blacklist |
| `use_auth` | boolean | false | For authenticated sites |
| `extract_links` | boolean | false | Return inter-page link graph |
| `max_total_chars` | number | 100000 | Total char budget |
| `max_tokens_out` | number | none | Token-budget cap (cl100k-base) |
| `include_full_markdown` | boolean | false | Pages return evidence-only by default |
| `citation_format` | string | "numbered" | "numbered" / "json" / "anthropic_tags" |

Pages are dedup'd by canonical URL (anchor-fragment aware: `/intro` and `/intro#install` collapse to one entry).

## After Crawling

All crawled pages enter the local cache with embeddings. This means:

The rest: https://github.com/KnockOutEZ/wigolo/blob/HEAD/skills/wigolo-crawl/SKILL.md