学术文献检索与全文提取服务器

Academic MCP Server

中文版：README.zh-CN.md

A Model Context Protocol (MCP) server that exposes a uniform paper-discovery and full-text-extraction API on top of eleven academic / patent sources:

Every source is wrapped by a per-platform scraper that surfaces a consistent result schema, and every download/conversion is routed through a local MongoDB-backed library so repeated queries are served from cache, browser state is shared across concurrent MCP clients, and downloaded PDFs/Markdown are deduplicated on disk.

The whole thing is meant to be embedded as a tool provider in an LLM agent (Claude Desktop, Codex, etc.) via MCP, but the underlying tools are also useful as a CLI/library on their own.

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│  MCP client (Claude / Codex / ...)                                   │
└────────────────────────────┬─────────────────────────────────────────┘
                             │ MCP tool calls
                             ▼
┌──────────────────────────────────────────────────────────────────────┐
│  server.py  —  FastMCP entry, dispatches to per-platform scrapers    │
│  ┌──────────────────────────┐    ┌────────────────────────────────┐  │
│  │  Library (library.py)    │◄──►│  Scrapers (11 modules)         │  │
│  │  Mongo: papers,          │    │  - one Camoufox/Chromium       │  │
│  │         search_queries,  │    │    persistent context per      │  │
│  │         browser_state    │    │    platform (singleton)        │  │
│  │  Files: .repo/{platform} │    │  - pooled per-PID profile      │  │
│  │         /{native_id}/    │    │    when canonical is busy      │  │
│  └──────────────────────────┘    └────────────────────────────────┘  │
└────────────────────────────┬─────────────────────────────────────────┘
                             │ optional
                             ▼
┌──────────────────────────────────────────────────────────────────────┐
│  library_web/  —  FastAPI/Jinja2 management console (port 5577)      │
│  /  /papers  /papers/{p}/{id}  /searches  /browser-state  /file      │
└──────────────────────────────────────────────────────────────────────┘

Every MCP tool call passes through the Library first:

• Search: the (platform, query, filters) tuple is hashed; on cache hit the cached result is returned without launching a browser.
• Details / Download / Read: the paper is identified by (platform, native_id) extracted from the URL. If the metadata / PDF / Markdown is already on disk, it is mirrored to the caller's requested output_dir without re-fetching.
• Browser state: each Camoufox scraper pins a single fingerprint per platform and reuses verification cookies (cf_clearance / DataDome / …), so one manual CAPTCHA carries across reboots and across concurrent MCP clients.

Components

Requirements

• Python 3.10+ (tested on 3.11/3.12). 3.10 is the lower bound for the PEP 604 union syntax used in a few places.
• MongoDB 3.6+ locally reachable (tested against mongo:4.4 in Docker). Collections and their indexes are created automatically on first start; JSON-shaped fields (keywords, extra, results, filters, cookies, fingerprint) are stored as native BSON documents/arrays.
• Camoufox kernel (auto-fetched into %LOCALAPPDATA%\camoufox on Windows; under ~/.cache/camoufox on Linux).

Installation

python -m venv .venv
.\.venv\Scripts\activate          # Windows
# source .venv/bin/activate       # POSIX
pip install -r requirements.txt
python -m camoufox fetch          # downloads the patched Firefox kernel

Create .env from .env.example and point it at your MongoDB:

# Full connection URI (recommended — supports auth / replica sets / options)
MONGODB_URI=mongodb://127.0.0.1:27017
MONGODB_DATABASE=academic_mcp
# For an auth-enabled server, include the credentials + authSource, e.g.
# MONGODB_URI=mongodb://user:pass@127.0.0.1:27017/?authSource=admin

Alternatively, leave MONGODB_URI unset and provide discrete MONGODB_HOST / MONGODB_PORT / MONGODB_USER / MONGODB_PASSWORD parts.

The library creates the database, collections and indexes on first start. If MongoDB is unreachable the Library disables itself with a clear warning and the server falls back to passthrough mode (no caching, no shared browser state — but everything still works).

Configuration

mcp_runtime_config.json (committed) holds non-secret tunables:

Secrets live in .env (gitignored). Only the MongoDB connection so far.

MCP tools

All five tools are registered on the FastMCP server (server.py) and appear to the LLM with the docstrings preserved verbatim.

search_papers(query, platform="CNKI", ...)

Returns a JSON string with:

{
  "total_results": "1707",
  "papers": [
    {
      "id": "<8-hex hash of detail_link>",
      "title": "...",
      "author": "...",
      "source": "...",                // raw publisher line (Volume / Issue / etc.)
      "venue_name": "...",            // canonical journal / conference name when extractable
      "doi": "10.1145/...",           // ACM only; others fill in from get_paper_details
      "date": "March 2026",
      "db_type": "Research article",
      "detail_link": "https://..."
    }
  ],
  "_cache_hit": true                  // server-only flag; true when served from MongoDB cache
}

Cache key is (platform, normalized query, filters, pagination). Cache entries never expire by default (the user picked permanent caching during the initial design). To force a refresh, delete the row from the search_queries table or via the web console.

get_paper_details(url, platform="CNKI")

Fetches abstract / keywords / DOI for a specific paper. Re-uses cached data when the same paper has already been seen (abstract column in papers). Returns:

{
  "url": "...",
  "abstract": "...",
  "keywords": ["..."],
  "doi": "...",
  "_cache_hit": true
}

get_paper_details is wrapped in goto_with_retry — transient Firefox network aborts (NS_ERROR_ABORT, NS_ERROR_NET_INTERRUPT, ...) get up to two automatic retries with exponential backoff before the failure propagates.

As a side effect, every successful details fetch also stores an EndNote-importable RIS record on papers.ris_text (see RIS auto-store below).

download_paper(url, output_dir, platform="CNKI")

Downloads the PDF. If the file is already on disk under .repo/{PLATFORM}/{native_id}/...pdf it is just mirrored to output_dir. On a fresh download, it is written into the canonical location AND mirrored to output_dir. Returns the local filesystem path.

The current download stack is request-first. For protected platforms, the browser is used to establish a valid verification state and, when the site requires it, to reach the page state that mints a signed asset URL. The actual data path then uses direct HTTP, in-page fetch, response-body capture, or Playwright's download event instead of button-click automation.

Platform-specific behavior:

• ACM: attempts the DOI PDF endpoint with verified cookies. If both the detail page and /doi/pdf/... expose only abstract HTML, the item is classified as unavailable instead of a scraper failure.
• SD: search/details prefer direct HTTP. PDF retrieval navigates only far enough to trigger the signed ScienceDirect asset stream, then saves bytes through programmatic download/body capture. A solved PDF-host challenge can be reused by later requests until the publisher invalidates the token.
• AIAA: search, details, PDF, and RIS are request-first against Atypon endpoints (/action/doSearch, /doi/..., /doi/pdf/..., /action/downloadCitation). The browser is only used to refresh the Cloudflare verification state; direct requests reuse that state with the Firefox 135 network fingerprint required by the issued cf_clearance.
• MDPI: search/details/PDF/RIS are request-first. The scraper handles MDPI's Akamai interstitial in-process by parsing the bm-verify payload and posting to /_sec/verify?provider=interstitial, then reuses the resulting session cookie for HTML, /pdf, and /export requests.
• WOS: discovery-only. Search calls the captured /api/wosnx/core/runQuerySearch NDJSON endpoint and parses records, abstracts, citation counts, DOI, source, and document type metadata. search_field="ADVANCED" enables native Web of Science query syntax, for example TS=("aeroelastic flutter") AND PY=(2020-2026). Normal fielded search mirrors the WoS dropdown: ALL, TS, TI, AU, SO, PY, OG, FO, PUBL, DOP, AB, UT, AD, AI, AK, CF, DT, DO, ED, FG, GP, KP, LA, PMID, and WC, with English/Chinese aliases for the common labels. Boolean basic-search rows are supported either as JSON rows, for example [{"field":"TS","text":"flutter"},{"op":"OR","field":"TI","text":"aeroelastic"},{"op":"NOT","field":"AU","text":"Smith"}], or as multiline shorthand: TS=flutter, OR TI=aeroelastic, NOT AU=Smith. In the tested Tongji VPN route, plain Python/curl HTTP stacks fail during TLS while Chromium succeeds, so the scraper uses a verified Chromium profile to issue API-level fetch() requests without clicking through the UI. PDF download and full-text reading are intentionally unsupported; use DOI/publisher links from the result with the native publisher scraper.

read_paper_content(url, output_dir, platform="CNKI")

Three-level cache:

1. If Markdown is already in the library → mirror MD + images/ to output_dir and return immediately.
2. If only the PDF is cached → re-run pdf_utils.convert_pdf_to_markdown against the cached PDF.
3. Cache miss → full scrape, store PDF + MD + images, mirror to output_dir.

Returns the MD file path and a 1000-character preview.

convert_local_pdf(pdf_path, output_dir)

Standalone helper to convert a user-supplied PDF to Markdown using the same image-extraction pipeline. Does not touch the Library.

The Library (MongoDB + filesystem)

Collections

papers          unique index (platform, native_id); integer `id` (web UI)
                title, author, source, venue_name, pub_date, db_type, doi,
                detail_link, abstract, keywords (array),
                pdf_path, md_path, images_dir, extra (subdocument),
                ris_text (platform-native or synthesized RIS export),
                created_at, updated_at

search_queries  unique index (platform, md5(query + filters)); integer `id`
                query_text, filters (subdocument), total_results,
                results (array of subdocuments), fetched_at

browser_state   _id == platform
                fingerprint (subdocument: BrowserForge Fingerprint),
                cookies     (array: Playwright context.cookies()),
                user_agent, verified_at, updated_at, note

counters        _id == collection name; seq -> emulates AUTO_INCREMENT so the
                integer `id` the web UI relies on stays stable.

JSON-shaped fields are stored as native BSON documents/arrays (no more text round-tripping). The integer id on papers/search_queries is assigned from the counters collection only on genuine inserts, so the web UI's numeric paper-export / search-delete keys stay stable.

Filesystem layout

.repo/
  ARXIV/2605.16230/
    arxiv_2605_16230.pdf
    arxiv_2605_16230.md
    images/
      arxiv_2605_16230.pdf-0002-08.png
      ...
  IEEE/9266228/
    ieee_9266228.pdf
    ieee_9266228.md
    images/...
  CNKI/CJFD|...

native_id is normalized to a filesystem-safe slug. The canonical location is the single source of truth; every download mirrors to the caller's output_dir afterward.

RIS auto-store

Every get_paper_details call that produces a usable abstract also populates papers.ris_text with an EndNote-importable RIS record. The server prefers the platform's own RIS export endpoint (richer pagination, ISSN, conference proceedings name) and falls back to synthesizing one from the stored metadata (ris_utils.synthesize_ris) when the platform doesn't expose an endpoint or the network call fails. Either way the column is populated idempotently — re-running details on a cached row doesn't refetch.

A bundled export is available from the web console: filter /papers by state, tick the checkboxes, and POST to /papers/export-ris. Rows that are missing ris_text (e.g. a synthesizer was added after the row was first seen) are synthesized on demand during the export and the result is written back, so the next export hits cache.

Browser state pool (concurrency + verification reuse)

Firefox/Camoufox refuses to open the same profile directory from two processes simultaneously. Without coordination, running two MCP clients (e.g. one inside Claude Desktop and one inside Codex) would either hang on the "Firefox is already running" modal or force every client to clear its own CAPTCHA.

The browser-state pool solves both:

1. Pooled profile dirs. The first MCP server to claim a platform uses the canonical .<platform>_profile/. Concurrent servers transparently fall back to an ephemeral per-PID copy .<platform>_profile__p<pid>/ (auto-removed on close()). A PID-aware sentinel file (.mcp_owner) prevents stale locks from crashed processes from blocking new launches.

2. Pinned fingerprint. Cloudflare's cf_clearance and DataDome tokens are bound to the browser fingerprint. We generate ONE browserforge Fingerprint per platform with os="windows", store it pickled+base64 in browser_state.fingerprint, and pass it to every AsyncCamoufox(fingerprint=...) launch. All concurrent clients on a platform present the same identity.

3. Shared cookies. After a successful pass of CAPTCHA / Cloudflare / DataDome, the scraper captures context.cookies() into browser_state.cookies. The next launch (any client, any MCP server process) injects those cookies via context.add_cookies before the first navigation. One manual verification covers all clients until the platform invalidates the token.

A row in browser_state is marked verified_at = NOW() whenever the captured cookies contain a known clearance cookie (cf_clearance, datadome, __ddg, incap_ses, reese84).

Limitations

cf_clearance is bound to (IP, fingerprint, TLS JA3). We pin fingerprint; your IP is your local IP; but Camoufox's outgoing TLS fingerprint is not byte-identical across launches. In practice this means most launches reuse the verification, but a tightened Cloudflare policy can still occasionally force a re-verify. This is still strictly better than the unfixed state (every client, every restart needs manual interaction).

Web console pages

Running

MCP server (production)

Wire the server into your MCP client's configuration. For Claude Desktop's mcp.json (or Codex's equivalent):

{
  "academic-mcp": {
    "command": "E:\\Projects\\CNKI-MCP\\.venv\\Scripts\\python.exe",
    "args": ["E:\\Projects\\CNKI-MCP\\server.py"],
    "env": {
      "MCP_PROFILE_SUFFIX": "claude"
    }
  }
}

MCP_PROFILE_SUFFIX is optional. Leave it unset for the default behavior (one canonical profile, automatic per-PID fallback for the second client). Set it differently in each client only if you want each MCP server to have its own dedicated profile set (.ieee_profile_claude, .ieee_profile_codex, etc.). The DB-backed browser state is shared regardless.

MCP server (standalone, for debugging)

.\.venv\Scripts\python.exe server.py

The server speaks MCP over stdio, so a raw invocation will just hang waiting for JSON-RPC. Useful only when attached from an MCP client.

Management console

library_web_start.bat   :: starts uvicorn on the configured port, opens the browser
library_web_stop.bat    :: kills the listener on that port

PowerShell equivalents (library_web_start.ps1 / library_web_stop.ps1) are also provided. The console runs read-write (it can delete cache rows / clear browser state) but is bound to 127.0.0.1 by default — change library_web_host in mcp_runtime_config.json if you really want remote access.

Multi-client guidance

Validation and performance

Latest full-platform integration smoke run: scratch/full_platform_test.py, run id 20260614_full_final (2026-06-14). AIAA was added and tested separately with run id 20260628_aiaa_final_relevance (2026-06-28); MDPI was added and tested with run id 20260628_mdpi_retry (2026-06-28). Each downloadable platform was tested with up to three fresh sample downloads. WOS was added as a discovery-only platform and smoke-tested with run id 20260704_wos_fields_boolean (2026-07-04), covering native advanced-search payloads plus basic-field dropdown mappings and Boolean rows. These runs completed with 0 hard failures. unavailable means the publisher or the tool semantics did not expose a downloadable PDF for that sample; it is not counted as a transport or parser failure.

Operational notes:

• ACM, SD, and AIAA require valid verification state for some requests. Once the visible challenge has been solved, subsequent retrievals reuse the stored browser identity and cookies where the publisher accepts them.
• SD PDF success depends on the separate pdf.sciencedirectassets.com challenge as well as the ScienceDirect detail-page DataDome state.
• AIAA accepts direct programmatic requests only when the request fingerprint matches the Firefox profile that produced cf_clearance; the implemented path uses curl_cffi with firefox135.
• AIAA's latest-date book-chapter samples may return publisher purchase pages from /doi/pdf/...; these are classified as unavailable, not transport failures (20260628_aiaa_date_sort_classified had 0 hard failures).
• MDPI is open-access and uses a lightweight Akamai interstitial; the challenge is solved programmatically and does not require a browser warmup profile.
• WOS may require hCaptcha/passive verification. Run warmup_platform_auth(platforms="WOS") once when it returns Server.passiveVerificationRequired; after that, searches use the WoS NDJSON endpoint through the verified Chromium profile. WoS is not a PDF source in this MCP.
• Google Scholar is search-oriented in this server; PDF download/read is intentionally reported as not applicable unless a downstream source is added for a specific result.

Troubleshooting

Project conventions

• scratch/ is git-ignored. Debug scripts, dump HTML, screenshots, etc. belong there.
• Local handoff notes (HANDOFF.md, NEXT_SESSION_PROMPT.md) are ignored and should not be committed.
• .{platform}_profile/, suffixed profile dirs such as .sd_profile_codex/, .repo/, .venv/, .env are all git-ignored.
• The *_scraper.py modules export a scraper_instance singleton (server.py imports those directly) so we keep a long-lived Camoufox/Chromium context per platform per MCP server process.
• Per-platform native IDs (used as the library's primary identity):
  • ARXIV: 2605.16230 (the arXiv ID, version suffix kept if present)
  • IEEE: 9266228 (the arnumber from /document/<n>/)
  • ACM: 10.1145/3597503.3608128 (the DOI, parsed straight from /doi/...)
  • SD: S0263224125035523 (the PII)
  • AIAA: 10.2514/6.2022-2883 (the DOI, parsed from /doi/...)
  • MDPI: 2072-666X/17/7/785 (the article path; DOI is stored as metadata)
  • WOS: WOS:001064481000002 (the Web of Science accession number)
  • CNKI: DBCODE|FileName
  • GS: SHA1 of the result URL (external link → hashed)
  • PATYEE: the pn= value
  • DAWEI: the PNM (publication number)

License

See repository root.