# Web Scraping

Web scraping: scrape_page / scrape_pages MCP tools for fetching pages as markdown...

- Repository: https://github.com/ginlix-ai/LangAlpha (2232 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ginlix-ai/LangAlpha --skill web-scraping -g -a claude-code -y`
Codex: `npx skills add ginlix-ai/LangAlpha --skill web-scraping -g -a codex -y`
Cursor: `npx skills add ginlix-ai/LangAlpha --skill web-scraping -g -a cursor -y`
Gemini CLI: `npx skills add ginlix-ai/LangAlpha --skill web-scraping -g -a gemini-cli -y`

## Description

Web scraping: scrape_page / scrape_pages MCP tools for fetching pages as markdown, HTML, or text (fast HTTP, browser rendering, anti-bot stealth), plus the direct Scrapling Python API for selectors, sessions, and spiders

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Web Scraping

## Overview

Two ways to scrape in the sandbox:

1. **MCP tools** (`scrape_page`, `scrape_pages`), recommended for straight "give me this page's content". Synchronous, return dicts.
2. **Direct Scrapling Python API**, for CSS/XPath selectors, sessions, logins, and multi-page spiders. Async, returns Page objects with `.css()` / `.xpath()`.

Quick fetches can run inline via `ExecuteCode`. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to `<task_name>/scraper.py` and run it via `Bash`, edit-and-rerun beats resubmitting code.

## MCP Tools

Import from `tools.scrape`. **Synchronous**, no `await`.

```python
from tools.scrape import scrape_page, scrape_pages
```

### Signatures

```python
scrape_page(url: str, mode: str = "fast", extraction: str = "markdown",
            timeout: float = 30.0, solve_cloudflare: bool = False) -> dict

scrape_pages(urls: list[str], mode: str = "fast", extraction: str = "markdown",
             timeout: float = 30.0, solve_cloudflare: bool = False) -> dict
```

### Parameters

| Param | Default | Notes |
|---|---|---|
| `mode` | `"fast"` | `"fast"` plain HTTP · `"browser"` JS rendering · `"stealth"` bot-protected sites |
| `extraction` | `"markdown"` | `"markdown"` (article text, cleaned) · `"html"` (raw) · `"text"` (plain) |
| `timeout` | `30.0` | Per-fetch **seconds**, 1, 60, seconds in every mode, not ms |
| `solve_cloudflare` | `False` | Only meaningful with `mode="stealth"` |
| `urls` |, | `scrape_pages` only; **max 10** per call |

Escalate modes only as needed: start `fast`, go to `browser` when the page needs JavaScript, `stealth` when you're getting blocked, and add `solve_cloudflare=True` only if `stealth` still returns a challenge page.

### Return shape

`scrape_page` returns a flat dict:

The rest: https://github.com/ginlix-ai/LangAlpha/blob/HEAD/plugins/alternative_data/skills/web-scraping/SKILL.md