网页抓取与搜索MCP服务器

fetchaller-mcp

Fetch websites in Claude Code without per-domain permission prompts. Includes web search, complete logged-out New Reddit support, challenge detection, and automatic solving for the supported challenge types below.

Why fetchaller?

Claude Code's built-in WebFetch asks permission for every new domain and blocks Reddit entirely. fetchaller fixes both:

• fetch: Read any URL — solves supported passive challenges and reports unresolved blocks as errors
• search: Google by default, with automatic fallback; choose google, ddg, brave, or all. Every result names its source.
• browse_reddit: Browse subreddit listings (hot/new/top/rising)
• search_reddit: Search Reddit posts globally or within a subreddit
• search_marketplace: Search Kijiji, Craigslist, and Facebook Marketplace simultaneously with human-readable params (city name, category, price range)
• search_realtor: Search Canadian homes on realtor.ca for sale or rent with full filters (location, price, beds, baths, property/building type, ownership)
• search_linkedin_jobs / get_linkedin_job: Search LinkedIn's public logged-out job board (keywords, location, date posted, remote/hybrid/on-site, experience, job type, salary) and read full postings — no account needed
• search_gojobs / get_gojobs_job: Search Ontario Public Service jobs and read full posting and competition status details
• search_indeed / get_indeed_job: Search Indeed Canada and read full public posting data, including salary bands and expiry dates
• search_jobbank: Search Canada's federal Job Bank by title, city, and radius with local result rechecking
• search_gcjobs / get_gcjobs_job: Search GC Jobs, the federal public service board, and read full postings with the "Who can apply" eligibility line first
• get_unifi_manual: Download a Ubiquiti UniFi installation guide as a PDF or per-page PNGs. Ubiquiti publishes no PDF and draws these guides as vector artwork, so they carry no readable text — this rebuilds the pages into a real document
• get_aliexpress_product: AliExpress product details (price, specs, ratings, reviews)
• search_aliexpress: Search AliExpress products with price filters and sorting
• get_alibaba_product: Alibaba.com B2B product details (tiered pricing, MOQ, lead times, supplier info)
• search_alibaba: Search Alibaba.com B2B products

Quick Start

Local Installation (stdio mode)

# Clone and install
git clone https://github.com/Averyy/fetchaller-mcp.git
cd fetchaller-mcp
uv sync

# Point this at branded Google Chrome. The pinned Linux reference build in
# Dockerfile gives the closest browser/transport identity match.
export BROWSER_EXECUTABLE_PATH=/path/to/Chrome-for-Testing

# Add to Claude Code
claude mcp add fetchaller -- uv --directory "$(pwd)" run fetchaller-mcp

Add permissions to ~/.claude/settings.json:

{
  "permissions": {
    "allow": [
      "mcp__fetchaller__fetch",
      "mcp__fetchaller__search",
      "mcp__fetchaller__browse_reddit",
      "mcp__fetchaller__search_reddit",
      "mcp__fetchaller__search_marketplace",
      "mcp__fetchaller__search_realtor",
      "mcp__fetchaller__search_linkedin_jobs",
      "mcp__fetchaller__get_linkedin_job",
      "mcp__fetchaller__search_gojobs",
      "mcp__fetchaller__get_gojobs_job",
      "mcp__fetchaller__search_indeed",
      "mcp__fetchaller__get_indeed_job",
      "mcp__fetchaller__search_jobbank",
      "mcp__fetchaller__search_gcjobs",
      "mcp__fetchaller__get_gcjobs_job",
      "mcp__fetchaller__get_unifi_manual",
      "mcp__fetchaller__get_aliexpress_product",
      "mcp__fetchaller__search_aliexpress",
      "mcp__fetchaller__get_alibaba_product",
      "mcp__fetchaller__search_alibaba"
    ]
  }
}

Restart Claude Code.

Recommended CLAUDE.md Addition

Add this to your project's CLAUDE.md (or global ~/.claude/CLAUDE.md) to instruct Claude to prefer fetchaller:

## Web Fetching & Search

**ALWAYS use fetchaller tools instead of WebFetch and WebSearch.** fetchaller has no domain restrictions and produces cleaner output.

- `mcp__fetchaller__fetch(url, maxTokens?, timeout?)` — Fetch any URL → clean markdown
- `mcp__fetchaller__search(query, provider, page?)` — Web search; required provider: `google` (preferred), `ddg`, `brave`, or `all`
- `mcp__fetchaller__browse_reddit(subreddit, sort?, time?, limit?)` — Browse subreddit listings
- `mcp__fetchaller__search_reddit(query, subreddit?, sort?, time?, limit?)` — Search Reddit posts
- `mcp__fetchaller__search_marketplace(query, location, platforms?, category?, sort?, condition?, min_price?, max_price?)` — Search Kijiji + Craigslist + Facebook Marketplace
- `mcp__fetchaller__search_realtor(location, transaction?, property_type?, building_type?, min_price?, max_price?, min_beds?, min_baths?, ownership?, sort?, page?)` — Search realtor.ca homes
- `mcp__fetchaller__search_linkedin_jobs(keywords, location?, strict_location?, date_posted?, workplace?, experience?, job_type?, min_salary?, easy_apply?, under_10_applicants?, sort?, start?, limit?)` — Search LinkedIn public jobs; `strict_location` defaults to `true` and re-checks each posting's own location
- `mcp__fetchaller__get_linkedin_job(job_id)` — Full public detail for one LinkedIn posting
- `mcp__fetchaller__search_gojobs(title?, location?, job_category?, career_level?, min_salary?, strict_title?, limit?)` — Search Ontario Public Service jobs
- `mcp__fetchaller__get_gojobs_job(job_id)` — Full OPS posting and competition status
- `mcp__fetchaller__search_indeed(title?, location?, radius_km?, strict_title?, strict_location?, limit?)` — Search Indeed Canada
- `mcp__fetchaller__get_indeed_job(job_key)` — Full Indeed posting, salary band, and expiry
- `mcp__fetchaller__search_jobbank(title?, location?, radius_km?, strict_title?, strict_location?, limit?)` — Search Canada's federal Job Bank
- `mcp__fetchaller__search_gcjobs(title?, location?, organization?, min_salary?, language?, exclude_various_locations?, strict_title?, limit?)` — Search GC Jobs, the federal public service board
- `mcp__fetchaller__get_gcjobs_job(job_id)` — Full GC Jobs posting: eligibility, closing date, salary, classification
- `mcp__fetchaller__get_unifi_manual(url, format?)` — Rebuild a UniFi installation guide as `pdf`/`png`/`svg`. Writes into the server's `DATA_DIR`, so on a containerised deployment the files land inside the container
- `mcp__fetchaller__get_aliexpress_product(product_id, timeout?)` — AliExpress product details
- `mcp__fetchaller__search_aliexpress(query, page?, sort?, min_price?, max_price?, timeout?)` — Search AliExpress
- `mcp__fetchaller__get_alibaba_product(product_id, timeout?)` — Alibaba.com product details
- `mcp__fetchaller__search_alibaba(query, page?, sort?, min_price?, max_price?, timeout?)` — Search Alibaba.com

Usage

The mcp__fetchaller__fetch tool is now available:

# Fetch a URL
fetch https://example.com

# Fetch with token limit
fetch https://example.com maxTokens=10000

# Fetch slow site with longer timeout
fetch https://slow-site.com maxTokens=25000 timeout=60

Web Search

# Search the web
search "python asyncio tutorial"

# Page 2 of results
search "python asyncio tutorial" page=2

Web Research Pattern

1. Use search to find URLs
2. Use fetch to read them

Tool Reference

fetch(url, maxTokens?, timeout?, raw?, method?, headers?, body?)

POST

Many search and listing APIs answer only to POST — Getro (which powers a large share of VC portfolio job boards), Algolia, GraphQL gateways. Their pages look static and fetch fine, so a GET-only fetcher reports an empty board on a site that appears to work.

{
  "url": "https://api.getro.com/api/v2/collections/246/search/jobs",
  "method": "POST",
  "headers": { "Accept": "application/json" },
  "body": "{\"hitsPerPage\":100,\"page\":0}"
}

Content-Type defaults to application/json when the body is valid JSON. A POST skips site-specific handling and the response cache — every site interceptor answers a URL pattern by issuing its own request, which would discard the caller's body, and the cache is keyed by URL alone.

Only GET and POST are accepted. PUT/PATCH/DELETE have no retrieval use; connection and body-framing headers (Host, Content-Length, Transfer-Encoding, …) cannot be set. Credentials (Authorization, Cookie, …) are dropped if a redirect crosses to another host, and a 301/302/303 downgrades a POST to a bodyless GET as browsers do.

Returns

Clean markdown with:

• Page title as H1
• Scripts, styles, nav, footer, iframes removed (careers/jobs links in the site chrome are preserved — they are almost never in the body copy)
• HTML converted to markdown
• Redirects noted
• Content truncated at token limit
• A possibly_js_rendered warning when the page's content is assembled by JavaScript, so an un-hydrated shell is not mistaken for an empty site — with whatever the shell still declares (description, canonical URL, company profiles from JSON-LD) recovered alongside it, since on such a page that is the entire available answer

Edge Cases

search(query, provider, page?)

search({query: "Python asyncio documentation", provider: "google"})
search({query: "Python asyncio documentation", provider: "all", page: 2})

Google is the default choice, but provider must be included in every MCP call. A Google failure automatically falls back to DuckDuckGo, then Brave if needed. The output states why Google failed and which provider answered, and every result carries its source. An honest zero-result response does not trigger fallback. Explicit ddg and brave calls use only that provider. all queries all three concurrently and deduplicates URLs, keeping Google ranking first. Beyond page 10, DDG is explicitly skipped in all and Google fallback.

Google returns ten requested results per page through the Google Docs research-search JSON endpoint, using eligible public document IDs as request context; it searches the web, not those documents. No API key, login, browser or Opera Mini profile is used. All verified IDs are indexed in src/fetchaller/search/google_documents.json; each uncached Google request starts with a random healthy ID, avoiding consecutive reuse when alternatives exist. Document-specific failures are logged, cooled down for 15 minutes and skipped within a three-attempt total budget. HTTP 429 respects Retry-After and Google-wide backoff without rotating IDs. Requests across the entire Google pool are spaced at least six seconds apart. Pagination prefers server cursors and otherwise encodes the requested offset as a protobuf varint.

Override the verified pool with FETCHALLER_GOOGLE_DOCUMENT_IDS, a JSON array of eligible IDs. Do not add IDs without validating the search endpoint. See the handoff and verified pool. This legacy endpoint has no established parity with the ordinary Google SERP for ranking, freshness, SafeSearch, geography or time filters.

DuckDuckGo pages by walking its previous Next form, up to page 10; Brave uses a zero-based page offset and returns about twenty results per page. All three engines are paced at least six seconds apart per provider, matching Reddit’s default budget of ten requests per minute. DuckDuckGo and Brave add jitter, serialize HTTP requests, and also enable wafer’s six-second limiter; hidden transport retries are disabled. Their shared provider cooldowns cover explicit calls, all, and fallback: 60 seconds after HTTP 429 and 300 seconds after HTTP 403 (also DDG’s blocking 202), extended by a longer Retry-After. Google retains its escalating global backoff. Complete nonempty successes are cached for five minutes per query, page and provider; fallback and partial failures are retried on the next call. Provider errors always remain visible.

Brave delegates its verification challenge to wafer through the server’s existing guarded browser solver. Recovery uses the published wafer 0.7.9 release, required by fetchaller 3.8.6. The solver allows up to 120 seconds, the complete HTTP call 150 seconds, and the Brave provider 180 seconds including pacing and parsing. A challenge that remains unresolved is reported as a provider failure and starts a cooldown, including a challenge returned with HTTP 200. No challenge solving or cookie manipulation lives in fetchaller. Wafer 0.7.9 includes commit d01c811, which fixes the earlier reload stall and fast-recovery classification race. Real challenged MCP searches returned 20 Brave results on macOS and Linux, and native headed/headless checks verified fresh PoW, spare-token reuse and browser-free restart. Brave consumes token cookies after HTTP reads; wafer can use session-local spare tokens for later gates, with fresh work potentially needed after restart. See the verification record. Google and DuckDuckGo remain browser-free. Allow up to 360 seconds in MCP clients for the complete Google → DDG → Brave fallback chain. See local Brave verification.

Reddit Tools

Two tools for Reddit research:

browse_reddit - Browse Subreddit Listings

browse_reddit({
  subreddit: "LocalLLaMA",   // without r/ prefix
  sort: "hot",               // hot, new, top, rising
  time: "day",               // hour, day, week, month, year, all (for "top" only)
  limit: 10                  // 1-25
})

Returns post titles, scores, comment counts, and URLs. Use fetch to read full posts.

search_reddit - Search Posts

search_reddit({
  query: "best mass spectrometry software",
  subreddit: "labrats",      // optional - limit to subreddit
  sort: "relevance",         // relevance, hot, top, new, comments
  time: "year",              // hour, day, week, month, year, all
  limit: 10                  // 1-25
})

Returns matching posts with metadata. Use fetch to read full discussions.

Compact New Reddit Fetching

Normal Reddit URLs and emitted links are canonicalized to www.reddit.com. Structured JSON reads use the logged-out api.reddit.com origin and are rendered as compact Markdown; HTML-only legs remain on the fixed New Reddit www origin. /r/{subreddit}/wiki/pages/ first reads the canonical New Reddit SSR page tree, then falls back to New Reddit's own logged-out WikiPageRevisionsV2 page tree on the fixed /svc/shreddit/graphql route when a community does not server-render that tree. Both paths are anonymous: the page index needs no OAuth scope and no browser. For a genuine historical collection URL, fetchaller validates exact collection metadata from its archived New Reddit Redux snapshot and hydrates every archived post ID from Reddit's current /api/info endpoint. Archive shells, mismatched identities, empty collections, and missing current posts are errors rather than successful empty or fabricated output. Thread output includes the public vote score and upvote ratio returned by Reddit, comment count and nested comments, outbound/discussion links, and structured gallery, video, crosspost, poll, NSFW/spoiler/locked/archived, and deleted-content state. Comment permalink URLs preserve the selected comment and its parent context.

old.reddit.com input URLs are canonicalized to New Reddit; no normal request depends on Old Reddit. Explicit .json URLs still return raw JSON. raw=true on a normal Reddit URL returns the canonical New Reddit HTML representation. Embedded author-written Old Reddit links are rewritten to their equivalent www.reddit.com target.

Explicit JSON always remains parseable. When it exceeds maxTokens, fetchaller retains only whole source scalars in a structural prefix and adds the top-level JSON marker "_fetchaller_truncated": true (or a marker object at the end of a top-level array). It never slices a string, substitutes a value, or reports invalid JSON as success; a budget too small for both useful content and the marker returns an explicit error.

fetchaller reads Reddit anonymously only and holds no credential: there is no account, login, OAuth user token, or client ID/secret in fetchaller. Every read is logged out. Its transport, wafer (>=0.7.0), serves JSON reads through Reddit's Android app API on an anonymous install token it mints and caches itself, and HTML through New Reddit's logged-out web path; neither needs a browser. The wiki page index is served by the SSR tree or New Reddit's logged-out page-tree route. Routes Reddit serves only to a logged-in account — exact moderator rosters, and account-private upvoted/downvoted activity — return an explicit error saying so; no names or counts are ever guessed or reconstructed.

Vote data is not reconstructed: score is Reddit's public, fuzzed vote score, and upvote_ratio is displayed only when Reddit returns it. Separate true upvote/downvote counts are not available anonymously and are never invented.

Bounded Context Use

The compact renderer requests limits and nesting depth from the caller's token budget, emits complete sections, and names every omitted section. Compactness is never allowed to remove public content silently. Any future size benchmark must ship its corpus, raw outputs, command, and timestamp with the result.

Reproducible Live Parity Evidence

baselines/reddit-legacy-contract-v1.json is an independent, versioned inventory derived from the last generic Old Reddit implementation. The offline gate compares that contract with both the production router and baselines/reddit-parity-corpus.json, then exercises route/schema/renderer/MCP fixtures:

uv run pytest -q \
  tests/test_reddit_legacy_contract.py \
  tests/test_reddit_parity_corpus.py \
  tests/test_reddit.py

The separate live gate runs the corpus through a real stdio MCP server with its six-second default pacing (five-second minimum):

SMOKE_STDIO_COMMAND='["docker","run","--rm","-i","fetchaller-mcp:candidate","python","-m","fetchaller.main"]' \
  uv run python scripts/reddit_parity.py \
    --strict \
    --include-unstable \
    --output /tmp/fetchaller-reddit-parity

The runner starts with an empty dedicated wafer-cache directory, keeps it for the warm and recreated-process stages, and records every body (including a failed/challenge body), hash, command, timestamp, and outcome. For an exact docker run command it mounts beneath entrypoint-owned /app/data, aligns the runtime UID/GID with the host, validates a real owner-only Reddit cookie file after warm and recreated stages, records non-secret Reddit cache hydration and pure-HTTP verification counts, and rejects either recreated verification or guarded-browser egress. A conflicting cache/ownership environment or --env-file fails closed. When real OAuth credentials make the two OAuth corpus entries eligible, Docker receives only --env NAME forwarding (never credential values in the command or report); anonymous SSR output never counts as OAuth evidence. Default runs require stable anonymous targets in --strict mode. The complete release gate adds --include-unstable: the runner discovers current real post/comment, wiki-revision, public-multireddit, and live-update IDs through separately recorded MCP calls, then requires every public route cold, warm, and recreated. It never substitutes a fake opaque ID. Fixture-only targets always remain offline and can never become live evidence; they are limited to inherently non-public access states such as private, quarantined, banned, gated, forbidden, and not-found, and to surfaces Reddit itself retired (gilded listings, post collections, the gold-only directory), which are reported exactly as Reddit answers them and never rebuilt from an archive. Every route in the corpus is an anonymous public read and needs no configuration. The live Reddit gates are currently waived on GitHub's hosted runners only: Reddit answers the anonymous solve origin with 403 there, to a real headful Chrome as well as to the HTTP client, which is consistent with datacenter-IP blocking rather than anything request-side. Those runs carry an explicit warning annotation so a green build is never mistaken for validated Reddit, and the offline Reddit contract and fixture gate still runs on every build. Every offline entry must carry a mandatory reason and complete fixture evidence.

Rate Limits

browse_reddit and search_reddit each use one JSON call. A normal fetch uses one bounded JSON call for most URLs; a profile root uses five independent public sources for metadata, activity, trophies, multireddits, and moderated communities. Mapped fetch calls and both dedicated tools share the server's request queue. Anonymous reads share a durable wafer session; configured API reads share a separate cookie-isolated application session. Caller-selected raw representations use the generic fetch path and its Reddit domain limiter. Token refreshes and every API request use the same bounded request budget.

AliExpress & Alibaba Tools

get_aliexpress_product(product_id, timeout?) - Product Details

Accepts a numeric product ID (e.g., 1005006027485365) or full URL. Returns price, specifications, ratings, and recent reviews via AliExpress's MTop API. If MTop is unavailable after a live search, an exact matching validated search listing can supply a clearly labeled, narrower title/price/rating snapshot. The optional timeout is an end-to-end budget from 1 to 180 seconds (default 180).

search_aliexpress(query, page?, sort?, min_price?, max_price?, timeout?) - Search Products

get_alibaba_product(product_id, timeout?) - B2B Product Details

Accepts a numeric product ID or full URL. Returns tiered pricing, MOQ, lead times, supplier info, and specifications. The optional timeout is an end-to-end budget from 1 to 300 seconds (default 180).

search_alibaba(query, page?, sort?, min_price?, max_price?, timeout?) - Search B2B Products

Marketplace Search

search_marketplace — Search Kijiji, Craigslist, and Facebook Marketplace

Searches all three platforms concurrently with human-readable parameters and returns grouped results.

Kijiji is Canada-only and automatically skipped for US locations. Location matching supports exact names, common aliases (e.g. "niagara" → Hamilton CL area), and fuzzy matching for typos.

Real Estate Search

search_realtor — Search realtor.ca homes

Searches Canadian homes for sale or rent via realtor.ca's api2, with the full filter set so an assistant can narrow a home search. Returns listings with price, address, beds/baths, size, agent, and a realtor.ca URL.

Call fetch(url) on any listing URL for the full description, every property detail, and similar nearby homes. fetch also handles realtor.ca search/SEO/map pages (/{prov}/{city}/real-estate, /map) and all wellfound.com pages (startup job search, job detail, company profiles).

How It Works

1. Validates URL (http/https only)
2. Blocks private/internal IPs (SSRF protection with DNS rebinding prevention)
3. Fetches with browser-like TLS fingerprints via wafer (Rust/BoringSSL) — rotates Chrome versions automatically
4. If bot challenge detected: solves automatically (see Bot Challenge Bypass below)
5. Detects content type
6. For HTML: removes junk elements (nav, footer, ads, cookie banners), applies site-specific cleanup (25+ sites including GitHub, Reddit, HN, Wikipedia, Medium, Stack Overflow, Amazon, eBay, AliExpress, Alibaba, DigiKey, Mouser, realtor.ca, wellfound.com, vacuumwars.com, The Home Depot (homedepot.com + homedepot.ca), plus Ashby/Greenhouse/Lever/Gem/Dayforce/Cornerstone/Workday/BambooHR/JazzHR/Work-at-a-Startup job boards with embed + white-label detection, and more), converts to markdown
7. For JSON/XML/CSV/text: returns raw
8. For PDF: extracts text
9. Truncates to token limit

Bot Challenge Bypass

fetchaller detects bot challenges and attempts the supported solver. A request is successful only after the resulting content is no longer a challenge or block page; a Playwright navigation that remains blocked is an error, never a successful fetch. First requests can be slower, while later requests can reuse durably cached cookies. The timeout parameter is one total wall-clock budget covering transport retries, queueing, and browser solving.

Supported Challenges

All challenge solving is handled by wafer's BrowserSolver (Patchright-based). Cookies are cached per-domain so subsequent requests skip the challenge.

Arkose/FunCaptcha and hCaptcha image escalation are detect-only and return an explicit challenge error. Vercel and generic-JS handling supports passive JavaScript checks only. An interactive CAPTCHA or any page that still contains the challenge is reported as unresolved.

Requirements

Docker: The browser-complete image is linux/amd64 because Google does not publish Chrome for Testing for Linux arm64. The build downloads one immutable Chrome-for-Testing archive, verifies its SHA-256 digest and exact version, and sets BROWSER_EXECUTABLE_PATH to that binary. BrowserSolver validates that the configured executable is branded Chrome and warns if its version differs from wafer/wreq's default emulation; browser-bound HTTP identity is aligned to the installed browser, while the pinned image also keeps the TLS shape matched. The local Compose file requests amd64 automatically on Apple Silicon, and the cookie-data volume persists solved cookies across restarts.

Local (stdio): Browser support is included by default, but protected-site solves require installed branded Google Chrome. Set BROWSER_EXECUTABLE_PATH to that executable. Matching wafer's current emulation is recommended so the browser and TLS identities stay aligned; the current Linux reference version and archive checksum are the CHROME_VERSION and CHROME_SHA256 arguments in Dockerfile.

Architecture

Content Processing

src/fetchaller/content/ handles HTML→markdown conversion. Each site module exports is_<site>(url), SELECTORS_LIST, and optionally strip_<site>_junk(soup) / postprocess_<site>(markdown):

• html.py — Generic pipeline + dispatch. Universal junk selectors, markdownify, whitespace cleanup. Generic JSON-LD Product fallback.
• amazon.py — All TLDs (.com, .ca, .co.uk, .de, etc.). CSS selectors, soup cleanup, regex post-processors.
• github.py — CSS selectors, URL transforms, file tree extraction, issue/PR/discussion extraction from embedded JSON.
• reddit.py — Strict Reddit host recognition, New Reddit canonicalization, public URL→structured routing (JSON plus the canonical SSR wiki page tree), bounded compact renderers for threads/listings/profiles/rules/wiki, rich media, and structured access states.
• hackernews.py — CSS selectors, table unwrapping, story block reformatter.
• medium.py — CSS selectors (data-testid), HTML-based detection for unknown custom domains.
• huggingface.py — data-target attribute selectors, filter tag/button cleanup.
• stackoverflow.py — All Stack Exchange sites. CSS selectors, soup cleanup, regex post-processors.
• redflagdeals.py — RFD-specific CSS selectors, soup cleanup, regex post-processors.
• forums.py — Generic forum support (XenForo, vBulletin, phpBB, Discourse). RSS/Atom feed autodiscovery.
• wikipedia.py — CSS selectors for edit buttons, navboxes, TOC, reference lists.
• vacuumwars.py — Vacuum Wars (vacuumwars.com, compare.vacuumwars.com, robotvacs.com). Reads the robot-vacuum comparison tool's inline dataset (window.vwProducts), which the client-side app otherwise renders as an empty board; routes comparison reads to the tool's own uncached host rather than the cached WordPress copy, disclosing the swap and falling back loudly if it fails; separates lab-tested models from listed-only ones, collapses colour variants only where every printed figure agrees, and de-duplicates the leaderboard card that prints each product twice.
• alibaba.py — Embedded JSON extraction (window.detailData, window.__page__data_sse10), soup cleanup.
• aliexpress.py — CSS selectors, soup cleanup, regex post-processors.
• craigslist.py — All city subdomains. CSS selectors, regex post-processors. Search URL detection for SAPI intercept.
• facebook_marketplace.py — URL detection only. GraphQL client in facebook_marketplace/ package.
• digikey.py — All TLDs. CSS selectors, soup cleanup. Behind Akamai (wafer handles). HTML fallback without API key.
• ebay.py — All TLDs. JSON-LD product extraction, search result DOM extraction (.s-item), regex post-processors.
• molex.py — JSON-LD Product extraction (additionalProperty specs). CSR site — specs only in structured data.
• mouser.py — All TLDs. CSS selectors, soup cleanup. Behind Akamai. HTML fallback without API key.
• soylent.py — Shopify store cleanup, inventory extraction from gsf_conversion_data.
• ti.py — Document viewer support for lazy-loaded datasheets.
• ashby.py / greenhouse.py / lever.py / gem.py / dayforce.py / cornerstone.py / workday.py / bamboohr.py / jazzhr.py / teamtailor.py / workatastartup.py — Job-board platforms. All preserve the source's own field names, enum values, and section titles. Each posting and (where supported) board listing is dispatched to the platform's API/JSON shell before the generic HTML pipeline. Six embed/white-label detectors run during the HTML phase so company career pages like synaptivemedical.com/job-openings (white-label Dayforce), skywatch.com/careers/ (Ashby <script src="…/embed">), avidbots.com/company/careers/ (BambooHR <div id="BambooHR">), earthdaily.com/job-openings (JazzHR multi-tenant), and careers.oatly.com/jobs (Teamtailor on a custom domain) are upgraded to structured ATS output instead of returning empty SPA shells. See docs/site-apis.md for endpoints and detection details.

Search

src/fetchaller/search/ — Unified Google, DuckDuckGo and Brave search. Google uses the Google Docs JSON endpoint with a randomized healthy document pool, cursor pagination and global throttling backoff. Provider selection, automatic Google fallback, source labels, URL deduplication and a 5-minute cache use separate wafer sessions; Brave alone receives the server’s guarded browser solver for challenge recovery.

Site-Specific API Intercepts

CSR sites where HTML scraping produces garbage are intercepted in fetch_url() and routed to structured APIs:

• Craigslist (src/fetchaller/craigslist/) — SAPI v8 client (sapi.craigslist.org). Up to 120 items/request with total count. Area IDs from page HTML, cached per hostname. Listing pages stay in HTML pipeline.
• Kijiji (src/fetchaller/kijiji/) — Unauthenticated Apollo GraphQL (kijiji.ca/anvil/api). Search + listing detail. Prices in cents.
• Facebook Marketplace (src/fetchaller/facebook_marketplace/) — GraphQL (facebook.com/api/graphql/) sent in the logged-out Comet form (__a, __comet_req; without them search silently serves nothing). A search or city URL is read from the page's own preloaded query and results, so location and filters match facebook.com; listing detail with photos.
• AliExpress (src/fetchaller/aliexpress/) — MTop API for products (token bootstrap + MD5 signing). SSR HTML for search. Reviews from feedback.aliexpress.com.
• Alibaba.com (src/fetchaller/alibaba/) — SSR HTML with embedded JSON. No MTop API for international site.
• eBay — SSR search results extracted from .s-item DOM elements, formatted as numbered list.
• Mouser (src/fetchaller/mouser/) — Search API client. Requires MOUSER_API_KEY.
• DigiKey (src/fetchaller/digikey/) — OAuth2 client_credentials API. Requires DIGIKEY_CLIENT_ID + DIGIKEY_CLIENT_SECRET.
• Ubiquiti / UniFi (src/fetchaller/ubiquiti/) — *.store.ui.com, techspecs.ui.com, and ui.com/qig/<slug> installation guides. Store pages carry the price in their HTML but none of the technical specifications, which render client-side from __NEXT_DATA__; dispatch is by Next.js route, not URL shape. Guides are JS page assets of outlined vector art with no readable text, rebuilt into a PDF/PNG by get_unifi_manual.
• Costco (src/fetchaller/costco/) — costco.com and costco.ca search and category pages from the catalogue search the sites themselves use (Google Retail Search behind gdx-api.costco.com), configured from each page's own published service config: prices, sale prices, promotions, delivery and warehouse stock for the site's default warehouse (named), ratings, item numbers, paging, sort, brand refinement, and the site's own keyword redirects ("tv" → TVs) followed and disclosed.
• The Home Depot (src/fetchaller/homedepot/) — homedepot.com and homedepot.ca, neither needing a browser. homedepot.com's pages sit behind Akamai's sensor challenge, but their data comes from the site's own federation-gateway GraphQL, which answers plainly: products (specs, per-store price and pickup stock, shipping), search and category listings (paging, sort, price range, refinements, keyword redirects followed and disclosed), reviews (pooled across variants, each sibling's review marked) and store pages. homedepot.ca is read from its Angular page state, its search API and its price service (sale was-prices live only there), in English and French.
• Marketplace Search (src/fetchaller/marketplace/) — Unified orchestrator searching Kijiji, Craigslist, and Facebook Marketplace concurrently. Human-readable params mapped to platform-specific values. Auto-skips Kijiji for non-Canadian locations.
• Dayforce HCM (src/fetchaller/content/dayforce.py) — Posting detail from SSR'd __NEXT_DATA__. Board listing via CSRF-protected POST to /api/geo/{namespace}/jobposting/search (NextAuth /api/auth/csrf round-trip required). White-label deployments on company domains are detected via __NEXT_DATA__.runtimeConfig.BASE_URL and rewritten to the canonical jobs.dayforcehcm.com board URL.
• Cornerstone OnDemand (src/fetchaller/content/cornerstone.py) — SPA shell carries a JWT in csod.context. Posting from services/x/job-requisition/v2/requisitions/{reqid}/jobDetails; board listing POSTed to rec-job-search/external/jobs on the regional cloud host (us|eu|uk|au.api.csod.com).
• Workday (src/fetchaller/content/workday.py) — {tenant}.wd{1-103}.myworkdayjobs.com boards and postings. Posting GET /wday/cxs/{tenant}/{site}/job{externalPath}; board POST /wday/cxs/{tenant}/{site}/jobs paginated in batches of 20 (capped at 200).
• BambooHR (src/fetchaller/content/bamboohr.py) — {tenant}.bamboohr.com/careers. Board GET /careers/list; posting GET /careers/{id}/detail. Both return clean JSON unauthenticated. Widget embeds (<div id="BambooHR" data-domain="{tenant}.bamboohr.com">) on company sites are auto-detected and resolved to the tenant subdomain.
• Teamtailor (src/fetchaller/content/teamtailor.py) — {slug}.teamtailor.com/jobs or the company's own domain (careers.oatly.com/jobs, recognised from the page). The HTML list stops at 20 with no salaries, so the board is read from jobs.json (salary band, dates) joined to jobs.rss (department, location, remote status) on the posting UUID, walking both feeds' paging to the end — which works differently with a filter than without. Postings render from their own page.
• JazzHR (src/fetchaller/content/jazzhr.py) — {tenant}.applytojob.com/apply. Board SSR'd HTML (.list-group .list-group-item); posting reads schema.org JobPosting JSON-LD. Company sites that reference one or more JazzHR tenants via JS (e.g. earthdaily.com/job-openings) are auto-aggregated into a combined board.

HTTP Transport (Wafer)

All HTTP is handled by wafer (~/code/wafer). Fetchaller does NOT contain bot protection, challenge solving, or TLS fingerprinting code. If a site blocks requests, fix it in wafer.

Remote Deployment (HTTP Mode)

Deploy fetchaller as a remote MCP server for Claude.ai, Claude Desktop, or any MCP client.

Quick Start

# Local loopback HTTP with authentication and restart-stable OAuth
export MCP_API_KEY='your-secret-key'
export JWT_SECRET="$(openssl rand -hex 32)"
export MCP_SERVER_URL='http://localhost:6000'
uv run python -m fetchaller.main --http

# Production Compose also requires a public HTTPS origin. It binds the app
# itself to 127.0.0.1:6000 for a host TLS reverse proxy.
export MCP_SERVER_URL='https://mcp.example.com'
docker compose up -d

Production traffic must terminate TLS before reaching the loopback-bound app. Configure the host reverse proxy to forward to 127.0.0.1:6000. If the proxy sets X-Forwarded-For, set TRUSTED_PROXY_IPS to only that proxy's exact address or CIDR; untrusted forwarded headers are ignored.

Local Development

# Build and test locally
docker compose -f docker-compose.local.yml up --build

# Test endpoints
curl http://localhost:6000/health
curl -X POST http://localhost:6000/mcp \
  -H "Authorization: Bearer test-api-key-local" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

Claude Code/Desktop Config

{
  "mcpServers": {
    "fetchaller": {
      "type": "streamable-http",
      "url": "https://mcp.fetchaller.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Claude.ai Custom Connector (OAuth)

For Claude.ai web/mobile with cross-platform sync:

1. Go to Settings → Connectors → Add Custom Connector
2. Name: fetchaller
3. URL: https://mcp.fetchaller.com/mcp
4. Leave Client ID/Secret blank
5. Enter your API key when prompted

Environment Variables

JWT_SECRET (required for persistent deployments)

OAuth access tokens are stateless JWTs. When MCP_API_KEY is configured, HTTP startup fails if JWT_SECRET is unset. For local HTTP development only, ALLOW_EPHEMERAL_JWT=1 opts into a random per-process secret; every previously issued access token then becomes invalid when the process restarts. Registered clients and hashed refresh tokens persist under DATA_DIR, while short-lived authorization codes remain in memory.

Generate one value and keep it stable for the life of the deployment:

openssl rand -hex 32

Pass it to the container the same way as MCP_API_KEY (compose environment:, Unraid template variable, etc.). Verify it landed with printenv | grep JWT_SECRET inside the container — an empty value counts as unset and prevents authenticated HTTP startup.

Two caveats:

• Changing it invalidates every access token. Clients can normally recover with their persisted rotating refresh tokens, but rotating both the secret and OAuth state un-pairs every connector.
• Do not derive it from MCP_API_KEY. Token payloads carry api_key_hash in cleartext, so a derived secret is recoverable from any captured token, allowing forged tokens. Use an independent random value.

Bearer auth with a raw MCP_API_KEY is unaffected by any of this — it is validated against the environment variable directly and survives restarts.

Security

• SSRF Protection: Blocks localhost, private IPs, CGNAT/shared space, link-local addresses, and DNS rebinding services (nip.io, xip.io, etc.). Resolves hostnames, verifies every resolved address, and pins the connection to those addresses to close the DNS-rebinding window. Chromium challenge solving is covered too: every browser redirect and subresource is forced through a loopback-only SOCKS5 guard that re-applies the address policy and connects to an approved numeric IP. Public nonstandard ports remain supported; Chromium's own unsafe-port policy determines which ports a page may use. Transition addresses (NAT64, 6to4, Teredo, IPv4-mapped) are judged by the IPv4 they actually translate to, so 64:ff9b:: + a public host works on IPv6-only networks while 64:ff9b::7f00:1 (loopback) and 64:ff9b::a9fe:a9fe (cloud metadata) stay blocked. Unresolvable hosts fail closed but are reported as a DNS failure, not as a private-host block.
• OAuth 2.1: PKCE required for authorization-code exchanges. Refresh tokens are hashed at rest and rotate on use.
• Rate Limiting: Per-IP rate limiting with configurable limits.

Files

fetchaller-mcp/
├── pyproject.toml           # Python package config
├── src/fetchaller/          # Python source
│   ├── main.py              # Entry point
│   ├── server.py            # MCP server setup
│   ├── config.py            # Configuration
│   ├── http/                # HTTP server (FastAPI)
│   ├── tools/               # MCP tools (fetch, search, reddit, aliexpress, alibaba, marketplace)
│   ├── content/             # Content processing (HTML→markdown, site-specific cleanup)
│   ├── search/              # Web search (Google / DDG / Brave / all)
│   ├── aliexpress/          # AliExpress MTop API client, product, search, reviews
│   ├── alibaba/             # Alibaba.com product and search extraction
│   ├── mouser/              # Mouser Search API client
│   ├── craigslist/          # Craigslist SAPI client + location resolution
│   ├── kijiji/              # Kijiji GraphQL API client + location resolution
│   ├── facebook_marketplace/# Facebook Marketplace GraphQL client
│   ├── marketplace/         # Unified marketplace search orchestrator
│   ├── ubiquiti/            # UniFi store/techspecs specs + installation guides
│   ├── homedepot/           # homedepot.com gateway GraphQL + homedepot.ca page state
│   ├── digikey/             # DigiKey API client (OAuth2 + product/search)
│   ├── cache/               # Response caching
│   ├── queue/               # Reddit rate limiting
│   └── security/            # SSRF, crypto, XSS
├── docker-compose.yml       # Production deployment
├── docker-compose.local.yml # Local testing
├── Dockerfile               # Container build
├── docs/                    # Architecture & developer docs
├── CLAUDE.md                # Instructions for Claude
├── README.md                # This file
└── landing/                 # Static site (fetchaller.com)
    ├── index.html           # Landing page
    └── llms.txt             # LLM-readable project summary (llmstxt.org spec)

Dependencies

• wafer-py[browser] - HTTP transport with TLS fingerprinting, bot challenge bypass, and browser solver (Rust/BoringSSL + Patchright)
• mcp - MCP protocol SDK
• fastapi + uvicorn - HTTP server
• beautifulsoup4 + markdownify - HTML to markdown
• pymupdf4llm - PDF to markdown extraction
• pyjwt - OAuth tokens

Testing

# Run tests
uv sync --extra dev
uv run ruff check src/ tests/
uv run pytest tests/ -x -q

# Test in Docker
docker compose -f docker-compose.local.yml up --build
curl http://localhost:6000/health

License

MIT