网页抓取MCP服务

fast-webfetch-mcp

A web-fetching MCP server that renders pages with a real browser running on your own machine, then hands the model clean Markdown instead of a token bill for 2MB of JavaScript.

Fetching costs nothing here: no API keys, no per-page pricing. And when a site claims to block you, it is often just a headless-detection script that a real Chromium walks straight through.

How it works

Two small programs, one job:

MCP client (agent)
   │  JSON-RPC over stdio
   ▼
src/index.ts          Bun + MCP SDK — tool schemas, budgets, rendering
   │  spawns per request
   ▼
crawl4ai_worker.py    Python + Crawl4AI — headless Chromium, markdown extraction
   │
   ▼
The actual internet

The TypeScript server owns policy: timeouts, concurrency permits, output truncation, optional Gemini-powered summarization. The Python worker owns mechanics: browser lifecycle, rendering, retries. Neither knows the other's job, and the contract between them is one JSON document on stdout.

Install

Requirements: Bun, Python 3.12+, and a GPU-agnostic amount of patience for the first browser download.

git clone <this repo> ~/.local/share/mcp/fast-webfetch-mcp   # or anywhere
cd ~/.local/share/mcp/fast-webfetch-mcp
bun install
uv venv .venv --python 3.12 && uv pip install --python .venv/bin/python crawl4ai

Verify the browser side works:

FAST_WEBFETCH_SMOKE_URL=https://example.com bun run src/index.ts

You should see Markdown for the example domain. If instead you see a stack trace about Playwright browsers, run crawl4ai-setup from the venv and try again — it installs the right Chromium build for you.

Wire it into your agent

{
  "mcpServers": {
    "fast-webfetch": {
      "type": "stdio",
      "command": "/usr/bin/mullvad-exclude",
      "args": ["/home/you/.bun/bin/bun", "run", "/path/to/fast-webfetch-mcp/src/index.ts"],
      "env": { "PATH": "/home/you/.bun/bin:/usr/bin:/bin" }
    }
  }
}

The mullvad-exclude wrapper is optional; it just lets fetch traffic bypass a VPN so sites see your real IP. Plain "command": "bun" works identically.

Tools

fast_fetch — one URL to Markdown

fast_fetch_raw — one URL to raw HTML

Same arguments minus prompt. Use it when Markdown loses the thing you need: tables, data- attributes, meta tags, exact markup.

The options object

All three tools accept an optional options object for sites that need more than a plain load-and-grab:

{
  "url": "https://example.com/feed",
  "options": { "full_page": true, "wait_seconds": 1.5, "drop_overlays": true }
}

Mistyped keys are dropped silently rather than passed through — the options payload becomes CrawlerRunConfig flags on the Python side.

fast_fetch_multiple — up to 15 URLs in one batch

Each URL comes back as its own section with a metadata header (url, status, elapsed_ms, truncated). One slow site cannot starve the others — it just gets a per-item timeout error while its batch-mates succeed.

Configuration

All optional, all environment variables.

Server knobs

Browser knobs

Behavior worth knowing

JS-heavy sites often return an empty shell under the default wait strategy. When a 200 comes back with no content, the worker retries once with networkidle, inside the same timeout budget — worst case you wait once, not twice. Connection failures and HTTP errors, meanwhile, stay single-attempt: retrying a dead host is just a slower way to fail.

fast_fetch and fast_fetch_multiple draw from the same pool of 12 browser slots, so a big batch cannot starve a concurrent single fetch. Demand beyond the pool queues; a batch asking for more slots than exist gets what is free.

Finally, max_length exists because your agent's context window is a budget, not a landfill.

When things break, you get a log path

Failures are sorted into two piles: the internet being flaky (timeouts, slow sites, per-item batch deadline errors) and the tool actually breaking (worker won't spawn, worker crashed, stdout overflow, unparseable output). Only the second pile writes a log — the first just gets an honest error tag.

A tool-side failure ends with:

log: /path/to/fast-webfetch-mcp/logs/2026-08-22T09-02-06-780Z-fast_fetch.json

Inside: the tool, exact arguments, worker input, and the full failure record (stage, exit code, signal, stderr tail) — enough to replay the request verbatim:

FAST_WEBFETCH_INPUT='{"url":"https://example.com","max_length":40000}' \
  .venv/bin/python crawl4ai_worker.py

Redirect with FAST_WEBFETCH_LOGS_DIR.

Staying under 30 seconds

Agent harnesses tend to kill MCP calls around the 30-second mark, so this server treats 28s as the hard wall. Every tool answers before it: finished results if the fetch made it, otherwise structured per-item errors saying which stage ate the time. A batch that runs out of budget still returns the items that finished.

Troubleshooting

Development

bun install
bun test            # 18 tests, no network needed
bun run typecheck   # tsc --noEmit, strict + noUncheckedIndexedAccess

The Python worker can be driven directly, which is the fastest way to debug fetch behavior without the MCP layer:

FAST_WEBFETCH_INPUT='{"url":"https://example.com","max_length":500}' \
  .venv/bin/python crawl4ai_worker.py

See also

• ddg-search — the natural front end: searches, then hands result URLs to this server
• Crawl4AI — the crawling engine underneath
• Model Context Protocol — the wire protocol

License

MIT.