Web Scraping
Web scraping: scrape_page / scrape_pages MCP tools for fetching pages as markdown...
Install
npx skills add ginlix-ai/LangAlpha --skill web-scraping -g -a claude-code -y- Repository
- ginlix-ai/LangAlpha2.2k
- Category
- AI and agents
- Updated
- 3d ago
- License
- MIT
- Context cost
- Mediumabout 2,050 tokens
- Runs scripts
- No, instructions only
What it does
Web scraping: scrape_page / scrape_pages MCP tools for fetching pages as markdown, HTML, or text (fast HTTP, browser rendering, anti-bot stealth), plus the direct Scrapling Python API for selectors, sessions, and spiders
Web Scraping
Overview
Two ways to scrape in the sandbox:
- MCP tools (
scrape_page,scrape_pages), recommended for straight "give me this page's content". Synchronous, return dicts. - Direct Scrapling Python API, for CSS/XPath selectors, sessions, logins, and multi-page spiders. Async, returns Page objects with
.css()/.xpath().
Quick fetches can run inline via ExecuteCode. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to <task_name>/scraper.py and run it via Bash, edit-and-rerun beats resubmitting code.
MCP Tools
Import from tools.scrape. Synchronous, no await.
from tools.scrape import scrape_page, scrape_pages
Signatures
scrape_page(url: str, mode: str = "fast", extraction: str = "markdown",
timeout: float = 30.0, solve_cloudflare: bool = False) -> dict
scrape_pages(urls: list[str], mode: str = "fast", extraction: str = "markdown",
timeout: float = 30.0, solve_cloudflare: bool = False) -> dict
Parameters
| Param | Default | Notes |
|---|---|---|
mode | "fast" | "fast" plain HTTP · "browser" JS rendering · "stealth" bot-protected sites |
extraction | "markdown" | "markdown" (article text, cleaned) · "html" (raw) · "text" (plain) |
timeout | 30.0 | Per-fetch seconds, 1, 60, seconds in every mode, not ms |
solve_cloudflare | False | Only meaningful with mode="stealth" |
urls | , | scrape_pages only; max 10 per call |
Escalate modes only as needed: start fast, go to browser when the page needs JavaScript, stealth when you're getting blocked, and add solve_cloudflare=True only if stealth still returns a challenge page.
Return shape
scrape_page returns a flat dict:
