MCP服务器测试库

pip install toolbench-mcp

# tests/test_myserver.py
from mcpbench import MCPBench, expect
from myapp.server import mcp  # your FastMCP server

bench = MCPBench(server=mcp, model="claude-sonnet-4-6")

@bench.scenario("user wants to search by name")
async def test_search():
    result = await bench.run("find all users named John", expected_tool="search_users")
    expect(result).called_tool("search_users")
    expect(result).tool_param("query", contains="John")
    return result

@bench.scenario("user wants a specific record")
async def test_get():
    result = await bench.run("get me user with id 42", expected_tool="get_user")
    expect(result).called_tool("get_user")
    expect(result).tool_param("user_id", equals=42)
    return result

mcpbench run tests/test_myserver.py --runs 5

Running 2 scenario(s) × 5 runs each...

✓ user wants to search by name     ████░  4/5  (80%)
✗ user wants a specific record     ███░░  3/5  (60%)  ← below threshold (80%)
    called list_users: 2x ✗
    called get_user:   3x ✓

Some scenarios below threshold.

mcpbench run tests/test_myserver.py --runs 5 --diagnose

✗ user wants a specific record     ███░░  3/5  (60%)
    called list_users: 2x ✗
    called get_user:   3x ✓

  Diagnosing confusion...

  Diagnosis: Both tools describe retrieving user data, and the prompt "get me
  user with id 42" could match a list operation filtered by ID. The distinction
  between fetching a single record versus listing with a filter isn't clear.

  Suggestion: Update get_user to say "Fetches a single user by their exact
  numeric ID. Use this when you have a specific ID, not for searches or lists."
  Update list_users to clarify it returns multiple records.

mcpbench run tests/test_myserver.py --runs 10 --save results/v1.json

mcpbench run tests/test_myserver.py --runs 10 --save results/v2.json

mcpbench diff results/v1.json results/v2.json

Comparing v1.json → v2.json
model: claude-sonnet-4-6  runs: 10

  scenario                          before  after   delta
  ─────────────────────────────────────────────────────────
✓ user wants to search by name       80%    100%    +20%
✗ user wants a specific record       60%     40%    -20%  ← regressed
  list open issues in a repo        100%    100%       —

1 scenario regressed.

mcpbench diff results/v1.json results/v2.json --regression-threshold 0.1

brew install github-mcp-server

# tests/github_server.py
import os
from fastmcp.client.transports import StdioTransport

token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GITHUB_PERSONAL_ACCESS_TOKEN")
transport = StdioTransport(
    command="github-mcp-server",
    args=["stdio"],
    env={**os.environ, "GITHUB_PERSONAL_ACCESS_TOKEN": token},
    keep_alive=False,
)

# tests/test_github.py
from mcpbench import MCPBench, expect
from tests.github_server import transport

bench = MCPBench(server=transport, model="claude-sonnet-4-6")

@bench.scenario("search for open issues in a repo")
async def test_search_issues():
    result = await bench.run(
        "search for open issues in the anthropics/anthropic-sdk-python repo",
        expected_tool="search_issues",
    )
    expect(result).called_tool("search_issues")
    return result

@bench.scenario("list open issues in a repo")
async def test_list_issues():
    result = await bench.run(
        "list all open issues in anthropics/anthropic-sdk-python",
        expected_tool="list_issues",
    )
    expect(result).called_tool("list_issues")
    return result

export GITHUB_TOKEN=ghp_your_token
export ANTHROPIC_API_KEY=sk-ant-your_key

mcpbench run tests/test_github.py --runs 5 --diagnose

Running 2 scenario(s) × 5 runs each...

✗ search for open issues in a repo    ░░░░░  0/5  (0%)
  ← below threshold (80%)
    called list_issues: 5x ✗

  Diagnosing confusion...

  Diagnosis: The LLM is confusing these tools because both descriptions mention
  finding issues in repositories, and the word 'search' in the prompt doesn't
  clearly distinguish between a filtered listing operation versus a query-based
  search. The descriptions don't emphasize that search_issues uses GitHub's
  search syntax for complex queries while list_issues simply retrieves all issues.

  Suggestion: Change search_issues description to clarify it uses GitHub's
  advanced search query syntax (e.g. 'label:bug author:username'). Change
  list_issues to say "Use this for browsing all issues, not for search queries."

✓ list open issues in a repo          █████  5/5  (100%)

Some scenarios below threshold.

git clone https://github.com/elnukk/mcpbench
cd mcp-bench
pip install -e .