BrightData
4-tier page fetching that moves from plain curl to curl with declared headers to a real...
Install
npx skills add danielmiessler/LifeOS --skill BrightData -g -a claude-code -y- Repository
- danielmiessler/LifeOS19.7k
- Category
- Writing
- Updated
- 8d ago
- License
- MIT
- Context cost
- Lightabout 1,205 tokens
- Runs scripts
- Yes, review before use
What it does
4-tier page fetching that moves from plain curl to curl with declared headers to a real browser through Interceptor to Bright Data's managed fetch product, single URL or multi-page crawl, output as markdown, within each target site's terms. USE WHEN Bright Data, scrape URL, web scraping, crawl site, extract page content, scrape whole site, spider domain, convert URL to markdown, page needs rendering. NOT FOR simple public content (WebFetch), social data with named actors, logged-in browser sessions (Interceptor), or multi-source investigation (Research).
BrightData
Fetches a single URL (FourTierScrape) or crawls a whole site (Crawl), moving through four tiers only as far as each page needs, always within the target site's terms and robots.txt. A site that refuses automated access is reported, not worked around. Output is always markdown. A Cloudflare Accept: text/markdown pre-check runs before Tier 1 (recipe in FourTierScrape.md).
Ideal State
- Every page comes back as clean markdown that matches its URL, labeled with the tier that actually retrieved it.
- Cost tracks need: free tiers are tried first, and paid Bright Data calls (Tier 4, whether for one URL or for the pages of a Light Crawl that need it, and Full Crawl) happen only when cheaper ones fail or the user confirms.
- A crawl returns a site map, the page contents and its stats, with failed pages named rather than silently missing.
Intent Routing
| Intent | Trigger | File |
|---|---|---|
| FourTierScrape | "scrape/fetch/pull/get/retrieve [URL]", "fetch this page as markdown", "this page needs rendering", "use Bright Data to fetch" | Intents/FourTierScrape.md |
| Crawl | "crawl this site", "spider this domain", "map this website", "get all pages from", "scrape the whole site", "crawl all pages under /docs" | Intents/Crawl.md |
References
| File | Read when |
|---|---|
Setup.md | first-time setup: the free tiers, the paid Bright Data pieces (MCP server, API key, crawl dataset id), and a proof |
