LLM推理工具发现与执行

bunx mcp-discovery

MCP Discovery - LLM inference with local MCP tools

Executes LLM inference calls with automatic tool discovery from MCP servers.
Supports multiple providers and tool discovery strategies.

Requirements:
  - Bun runtime (https://bun.sh)
  - Environment variables for provider (e.g., GROQ_API_KEY)
  - MCP servers configured in mcp/servers.json

Usage:
  Single run:
    bunx mcp-discovery <strategy> -m <provider:model> [-s <servers>] -p "<prompt>"

  Benchmark mode:
    bunx mcp-discovery <strategy> -m <provider:model> [-s <servers>] -p "<prompt>" -n <runs> [-c <concurrency>]

Arguments:
  <strategy>           Tool discovery strategy (currently: 'all')
                       - all: Include all tools from all servers
                       - minimal: Use VCR cache to find minimal set of tools for task
                       - all-relevant: Use only tools from servers specified in prompts file
                       
                       Suffix with :strict to enable strict tool definition mode
                       Example: all:strict, minimal:strict

  -m <provider:model>  Provider and model specification
                       Format: provider:model
                       Examples: groq:llama-3.1-70b-versatile
                                 openai:gpt-4o-mini

  -s <servers>         [Optional] Comma-separated list of MCP server IDs
                       Available: ppt, playwright, word, chart, trends
                       If not provided, runs without tools

  -p <prompt>          User prompt to send to the LLM

  -n <runs>            [Optional] Number of times to run (default: 1)
                       If > 1, enters benchmark mode with statistics

  -c <concurrency>     [Optional] Number of concurrent workers (default: 1)
                       Each worker maintains its own MCP server pool

  -x <expectation>     [Optional] Expected text in <answer> block (multiple allowed)
                       When provided, pass/fail based on whether all expectations found
                       Text is normalized (case-insensitive, normalized punctuation)

  --help               Show this help message

Modes:
  Single run (n=1):    Outputs progress then complete JSON response

  Benchmark (n>1):     Runs multiple inferences concurrently
                       Shows progress, statistics, and latency metrics
                       Saves per-run logs to temp directory
                       Returns exit code 0 if all pass, 1 if any fail

Providers:
  Configure in providers.json. Currently supported:
  - groq (adapter: completions)
  - openai (adapter: completions, stub)
  - openrouter (adapter: completions, stub)
  - anthropic (adapter: messages, stub)
  - groq-responses (adapter: responses, stub)

Examples:
  # Single run - simple query
  bunx mcp-discovery all \
    -m groq:llama-3.3-70b-versatile \
    -p "What is 2+2?"

  # Single run - query bundled PowerPoint file (relative path)
  bunx mcp-discovery all \
    -m groq:llama-3.3-70b-versatile \
    -s ppt \
    -p "What is the title of data/ppt/build_effective_agents.pptx?"

  # Benchmark - test reliability with 50 runs
  bunx mcp-discovery all \
    -m groq:llama-3.3-70b-versatile \
    -s ppt \
    -p "What is the title of data/ppt/build_effective_agents.pptx?" \
    -n 50 -c 8

  # Benchmark - high concurrency stress test
  bunx mcp-discovery all \
    -m groq:llama-3.1-8b-instant \
    -p "What is 5+5?" \
    -n 100 -c 10

  # Benchmark with expectations (checks for specific text in answer)
  bunx mcp-discovery all \
    -m groq:llama-3.3-70b-versatile \
    -p "What is 2 + 2. Put only the answer between <answer></answer> tags." \
    -n 20 -c 4 \
    -x "4"

  # Run with "all-relevant" strategy (requires prompts file)
  bunx mcp-discovery all-relevant \
    -m groq:llama-3.3-70b-versatile \
    -i mcp/prompts.ts:0

  # Extract just the content from single run (filter out progress lines)
  bunx mcp-discovery all \
    -m groq:llama-3.3-70b-versatile \
    -p "Hello" \
    | tail -1 | jq -r '.choices[0].message.content'

src/
├── types/           Type definitions
├── config/          Configuration loaders
├── adapters/        LLM provider adapters
│   ├── completions/ OpenAI-style completions API
│   ├── responses/   [Stub] Responses API
│   └── messages/    [Stub] Messages API (Anthropic)
├── strategies/      Tool discovery strategies
│   └── discovery/
│       ├── all.ts   Include all tools
│       ├── minimal.ts Use VCR cache for minimal set
│       ├── all-relevant.ts Filter by relevant servers
│       ├── browse   [Stub] Interactive browsing
│       └── search   [Stub] Search-based discovery
├── mcp/            MCP client implementation
└── core/           Core tool execution loop

# Analyze a specific log directory
bun src/stats.ts /var/folders/xx/yyyy/T/mcp_discovery_run_id

# Clone the repository
git clone <repository-url>
cd mcp-discovery

# Install dependencies
bun install

# Run locally during development
bun src/mcp-discovery.ts all -m groq:llama-3.3-70b-versatile -p "Test"

# The CLI entrypoint is src/mcp-discovery.ts
# All other source code is in the src/ directory