Wigolo Crawl
Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies...
Install
npx skills add KnockOutEZ/wigolo --skill wigolo-crawl -g -a claude-code -y- Repository
- KnockOutEZ/wigolo5.5k
- Category
- Writing
- Updated
- 3d ago
- License
- AGPL-3.0-only
- Context cost
- Lightabout 696 tokens
- Runs scripts
- No, instructions only
What it does
Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later `cache` / `find_similar` queries.
wigolo crawl
Crawl sites with configurable strategy, depth, and rate limiting. All pages enter the local cache with embeddings.
Quick Reference
// Crawl docs via sitemap (fastest, recommended for doc sites)
{ "url": "https://docs.example.com", "strategy": "sitemap", "max_pages": 30 }
// BFS crawl with scope filter
{ "url": "https://example.com", "strategy": "bfs", "max_depth": 3, "max_pages": 50, "include_patterns": ["^https://example\\.com/docs"] }
// URL discovery only (no content fetched, fastest for scoping)
{ "url": "https://example.com", "strategy": "map" }
// Authenticated crawl
{ "url": "https://app.example.com/docs", "strategy": "bfs", "use_auth": true, "max_pages": 20 }
Parameters
| Parameter | Type | Default | When to use |
|---|---|---|---|
url | string | required | Seed URL |
strategy | string | "bfs" | "sitemap" for doc sites, "map" for URL discovery only |
max_depth | number | 2 | How many link levels to follow |
max_pages | number | 20 | Hard cap on pages fetched |
include_patterns | string[] | none | Regex whitelist, ALWAYS add to stay in scope |
exclude_patterns | string[] | none | Regex blacklist |
use_auth | boolean | false | For authenticated sites |
extract_links | boolean | false | Return inter-page link graph |
max_total_chars | number | 100000 | Total char budget |
max_tokens_out | number | none | Token-budget cap (cl100k-base) |
include_full_markdown | boolean | false | Pages return evidence-only by default |
citation_format | string | "numbered" | "numbered" / "json" / "anthropic_tags" |
Pages are dedup'd by canonical URL (anchor-fragment aware: /intro and /intro#install collapse to one entry).
After Crawling
All crawled pages enter the local cache with embeddings. This means:
