分层网页检索与研究服务

IndexExpress

Privacy-first web research with tiered retrieval/report modes, source-cited outputs, MCP tools, and a simple user-facing terminal experience.

Feature Labels

• peek: pure link-discovery primitive returning ranked sources
• skim: citation-first distillation returning final_answer, claims, key_evidence
• analyze: comparative adjudication for explicit alternatives, with automatic fallback to research when comparison input is insufficient
• research: iterative planner-reviewer DAG returning final_synthesis, evidence_graph, and coverage status
• Citation contract: evidence links by source_id / evidence_id
• Output: optional markdown export for reports

Tier Matrix

Architecture

Client (MCP or TUI)
    |
    v
MCP Server (server.py)
    |- peek      -> PeekAgent (consensus-calibrated retrieval primitive)
    |- skim      -> SkimAgent (citation-first distillation pack)
    |- analyze   -> AnalyzeOrchestrator (adversarial claim adjudication)
    |- research  -> DeepSearchOrchestrator (iterative planner-reviewer DAG)
    |
    +-- SearXNGClient (retrieval)
    +-- LLMClient (OpenAI-compatible)
    +-- workflow_primitives.py (ranking/evidence/claims primitives)

Quick Start

1. Copy env template and set your model:

cp .env.example .env

2. Set LLM_MODEL_ID in .env to your loaded LM Studio model ID.

Recommended minimal .env for most users:

LLM_API_URL=http://localhost:1234/v1
LLM_MODEL_ID=<your-model-id>
LLM_MAX_PARALLEL=4
SEARXNG_URL=http://localhost:8888

Everything else in .env.example is optional tuning.

3. Build and start backend services:

docker compose up -d --build

4. Health checks:

curl http://localhost:8000/health
curl http://localhost:8000/ready
curl http://localhost:8000/metrics
curl http://localhost:8888/healthz

/ready is a stricter readiness gate for deploy checks, and /metrics returns request/tool counters with latency aggregates.

MCP Tools

Primary tools:

• peek
• skim
• analyze
• research
• fetch_url (raw page fetch helper for URL inspection)

Backward-compatible aliases:

• quicksearch -> skim
• deepresearch -> research

Example call:

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"skim","arguments":{"query":"fun facts about slugs","max_results":15}}}

Terminal UI (User-Facing)

Install the local CLI:

pip install -e .

Launch TUI:

expressindex --tui

Run direct command mode:

expressindex peek "what is searxng"
expressindex skim "fun facts about slugs" --save-md report.md
expressindex analyze "best python web framework" --save-md compare.md
expressindex analyze "python vs node.js for backend" --options "python,node.js"
expressindex research "vacation plan to aruba" --num-sub-queries 6 --save-md aruba.md
expressindex status
expressindex metrics --json

All terminal activations (all search modes + URL fetch helper):

# Search modes (CLI)
expressindex peek "slug facts"
expressindex skim "slug facts"
expressindex analyze "python vs node.js for backend" --options "python,node.js"
expressindex research "slug facts" --num-sub-queries 6

# URL fetch helper (MCP tool)
curl -sS http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "fetch_url",
      "arguments": {
        "url": "https://en.wikipedia.org/wiki/Slug",
        "max_chars": 4000
      }
    }
  }'

Direct mode output formats:

expressindex skim "fun facts" --output text
expressindex skim "fun facts" --output json
expressindex skim "fun facts" --output markdown

UI goals:

• clear progress prompts
• friendly error messages
• no MCP protocol noise in normal output
• simple command-loop UX with mock MCP-consumer summaries for multi-report modes

Report Contract

All query tools now return strict MCP-first JSON with shared primitives:

• sources[]: {source_id, url, title, domain, relevance_score, domain_trust_score, freshness_timestamp, intent_category}
• key_evidence[]: {evidence_id, source_id, exact_quote, relevance_score}
• claims[]: {claim_id, statement, support_evidence_ids, refute_evidence_ids, confidence_tier}
• _meta: {token_estimate, compute_ms, schema_version}

Mode-specific outputs:

• peek: query, sources, _meta
• skim: final_answer, claims, uncertainties
• analyze: requested_mode, executed_mode, route_reason, options_compared, recommended_option, confidence, why_not, recommended_position, decision_matrix
• research: final_synthesis, evidence_graph, coverage_report, open_questions, trace_log

Environment Variables

If the full table feels too configurable, use the 4-variable minimal config above and keep defaults for the rest.

Legacy alias still supported:

• LLM_MODEL -> LLM_MODEL_ID

LM Studio Parallelism Notes

• Increase model-side Max Concurrent Predictions in LM Studio.
• Keep Unified KV Cache enabled.
• Tune LLM_MAX_PARALLEL to match VRAM and model throughput.

Troubleshooting

• LLM unavailable:
  • verify LM Studio server is up on LLM_API_URL
  • verify LLM_MODEL_ID matches currently loaded model
• Slow responses:
  • lower LLM_MAX_PARALLEL
  • lower RESEARCH_MAX_SUB_QUERIES and/or MAX_CONCURRENT_AGENTS
• Citation quality issues:
  • report generator performs a citation-repair pass automatically

Todo

• Manual mode: fetch provider documentation on retrieving whole pages as Markdown and add an explicit manual workflow.

Project Map

• MCP server: mcp-server/server.py
• Tier agents: mcp-server/agents/quicksearch.py, mcp-server/agents/deepresearch.py
• Reporting engine: mcp-server/reporting.py
• Search/LLM clients: mcp-server/search/searxng_client.py, mcp-server/search/llm_client.py
• IndexExpress CLI/TUI: expressindex_cli/main.py
• Legacy helper script: mcp-server/tui.py
• Skill profile: .agents/skills/searxng-mcp-research/SKILL.md