使用说明

MCP服务器测试器

MCP服务器测试工具是一款用于自动化测试MCP服务器的工具。

为什么选择这个工具？

构建可靠且始终如一、符合规范的MCP服务器，其难度远超预期：

Inspector 工具有助于构建和测试初始服务器，但无法防止回归问题
这些工具在直接测试中表现完美，但在现实世界中，大型语言模型（LLMs）使用它们时却存在错误和不一致的情况
大多数MCP（可能是指某种特定领域的开发工具包或平台）SDK并不完全符合规范，而且很难知道还缺少哪些内容
规范迅速发展，使得紧跟最佳实践变得困难

该 mcp-server-tester 这是一个与实现无关的命令行界面（CLI）测试工具，提供了三个针对性的命令，有助于解决这些核心问题：

tools - 运行测试以验证您的工具能够正确地与直接的MCP调用一起工作
evals - 运行测试以验证大型语言模型（LLMs）能够一致地理解和正确使用您的工具
compliance - 随着协议的演进，实现服务器规格的自动化验证 （进行中）

命令

`tools` - 直接MCP工具测试

测试您的工具是否能通过直接API调用正常工作：

npx mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json

样本输出：

✅ Tool Discovery: Found 3 tools (read_file, write_file, delete_file)
✅ Write file successfully
✅ Read file returns content
✅ Write and read workflow
❌ Handle invalid path - Expected error containing 'permission' but got 'access denied'
✅ Delete file cleanup

Results: 5/6 tests passed

`evals` - 大语言模型（LLM）集成测试

测试大型语言模型（LLMs）能否有效发现并使用您的工具：

npx mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json

示例输出：

🤖 Testing with claude-3-5-haiku-latest
✅ LLM can create and read files
✅ File manipulation workflow
✅ Error handling test
❌ Documentation query task: FAILED
    Prompt: "How do I search for configuration files in the project?"
    • Required tool 'search_docs' was not called (actual calls: list_files)
❌ Lists all available tools: FAILED
    Prompt: "Please list all available file management tools you have access to."
    • LLM judge failed: Score: 0.3/1.0 (threshold: 0.8). Assistant did not provide complete tool list.

Results: 3/5 tests passed

注：需要 ANTHROPIC_API_KEY 环境变量。

`compliance` - MCP规范合规性测试（进行中）

测试您的服务器是否符合规范：

npx mcp-server-tester compliance --server-config filesystem-server-config.json

看看合规指令（在制品，WIP）以下部分提供了详细信息和示例输出。

快速入门

创建服务器配置 （filesystem-server-config.json)：

{
  "mcpServers": {
    "filesystem": {
      "command": "node",
      "args": ["./filesystem-server.js"]
    }
  }
}

手动创建工具和评估测试文件:

tool-tests.yaml：

tools:
  expected_tool_list: ['write_file']
  tests:
    - name: 'Write file successfully'
      tool: 'write_file'
      params: { path: '/tmp/test.txt', content: 'Hello world' }
      expect: { success: true }

eval-tests.yaml:

evals:
  models: ['claude-3-5-haiku-latest']
  tests:
    - name: 'LLM can write files'
      prompt: 'Create a file at /tmp/greeting.txt with the content "Hello from Claude"'
      expected_tool_calls:
        required: ['write_file']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant successfully create the file?'
          threshold: 0.8

请参阅工具测试并且评估测试包含全面句法示例的章节。

使用代理自动创建工具和评估测试：

尝试使用这个提示，并将服务器配置信息替换为你自己的：

Please create tool tests and eval tests for me to use with the mcp server tester tool.
To see how to use it, run the tool's documentation and schema commands:

  `npx -y mcp-server-tester --help`

My server config file is at ./filesystem-server-config.json. To know what tools you need to create tests for, run this command:

  `npx -y @modelcontextprotocol/inspector --cli --config filesystem-server-config.json --server filesystem-server --method tools/list`

Please follow these steps:

1. Create tool tests
  - Create a file called `tool-tests.yaml` that contains a single test for each tool. Follow these guidelines:
    - Do NOT force an individual test to pass; if the expected output is not returned, the test should fail
    - if there is a clear dependency between tool calls, you can chain them using the "calls" property
  - Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)

2. Create eval tests
  - Create a file called `eval-tests.yaml` with eval tests that will test the server's behavior. Follow these guidelines:
    - start with a few simple evals, and then build up to more complex ones
    - create between 5 and 10 eval tests
  - Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)

3. Provide a summary, which includes:
  - a list of the tools that are being tested and what you chose to test
  - a list of the eval tests and your reasoning for why you chose them
  - an explanation of how to run the tool tests and eval tests

运行测试:

# Run tools tests (fast, no API key needed)
mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json

# Run LLM evaluation tests (requires API key)
export ANTHROPIC_API_KEY="your-key"
mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json

工具测试

这个（或“该”） tools 命令通过直接的API调用测试您的MCP服务器工具。这种方法快速、无需API密钥，并且有助于在使用大型语言模型（LLMs）测试之前验证您的工具是否正常工作。

基本语法

tools:
  expected_tool_list: ['tool1', 'tool2'] # Optional: verify specific tools are available
  tests:
    - name: 'Test description'
      tool: 'tool_name'
      params: { param1: 'value1' }
      expect: { success: true }

单一工具测试

测试单个工具调用：

tools:
  tests:
    # Basic success test
    - name: 'Write file successfully'
      tool: 'write_file'
      params: { path: '/tmp/test.txt', content: 'hello world' }
      expect: { success: true }

    # Error handling test
    - name: 'Handle invalid path'
      tool: 'write_file'
      params: { path: '/invalid/path.txt', content: 'test' }
      expect:
        success: false
        error: { contains: 'permission denied' }

    # Result validation test
    - name: 'Read file returns content'
      tool: 'read_file'
      params: { path: '/tmp/existing.txt' }
      expect:
        success: true
        result: { contains: 'expected content' }

多步骤工作流测试

工具调用的测试序列：

tools:
  tests:
    - name: 'Write and read workflow'
      calls:
        - tool: 'write_file'
          params: { path: '/tmp/workflow.txt', content: 'test data' }
          expect: { success: true }

        - tool: 'read_file'
          params: { path: '/tmp/workflow.txt' }
          expect:
            success: true
            result: { equals: 'test data' }

        - tool: 'delete_file'
          params: { path: '/tmp/workflow.txt' }
          expect: { success: true }

期望类型

成功/失败：

expect: { success: true }   # Tool should succeed
expect: { success: false }  # Tool should fail

错误验证：

expect:
  success: false
  error: { contains: 'permission denied' } # Error message must contain text

结果验证：

expect:
  success: true
  result: { contains: 'partial text' }      # Result must contain text
  result: { equals: 'exact match' }         # Result must exactly match

评估测试

这个（或“该”） evals 命令测试旨在验证大型语言模型（LLMs）能够有效发现并使用您的工具。这需要一个 ANTHROPIC_API_KEY 并测试实际使用模式。

基本语法

evals:
  models: ['claude-3-5-haiku-latest']
  max_steps: 5 # Optional: limit conversation turns
  tests:
    - name: 'Test description'
      prompt: 'Task for the LLM to complete'
      expected_tool_calls:
        required: ['tool1', 'tool2']
        allowed: ['tool3']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant complete the task?'

工具调用验证

控制大型语言模型（LLM）应使用的工具：

evals:
  tests:
    - name: 'LLM must use specific tools'
      prompt: 'Write a paragraph of poetry to file "poetry.md"'
      expected_tool_calls:
        required: ['write_file'] # Must use these tools
        allowed: ['read_file'] # Can use this tool, but all other tool uses will throw error

响应评分

评估大型语言模型（LLM）响应的质量：

正则表达式评分器：

response_scorers:
  - type: 'regex'
    pattern: 'success|completed|done' # Response must match pattern

大型语言模型（LLM）裁判评分员：

response_scorers:
  - type: 'llm-judge'
    criteria: 'Did the assistant successfully complete the file operations?'
    threshold: 0.8 # Score must be >= 0.8

高级选项

IDE模式验证

在你的测试文件顶部添加这一行，以启用VS Code等集成开发环境（IDE）中的自动模式验证和自动补全功能：

# yaml-language-server: $schema=https://raw.githubusercontent.com/steviec/mcp-server-tester/refs/heads/main/src/schemas/tests-schema.json

环境变量替换：

使用 ${VAR_NAME} 语法，用于将环境变量注入到您的测试配置中：

export API_BASE_URL="https://api.example.com"
export TEST_USER="alice"
mcp-server-tester tools test.yaml --server-config server.json

tools:
  tests:
    - name: 'Test ${TEST_USER} API access'
      tool: 'api_call'
      params:
        url: '${API_BASE_URL}/users/${TEST_USER}'
        method: 'GET'
      expect:
        success: true
        result: { contains: '${TEST_USER}' }

环境变量可在测试名称、参数、预期结果以及所有其他配置值中使用。缺少的变量将导致配置加载失败，并显示一条有用的错误信息。

调试模式：

查看完整的对话输出及评分详情：

mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json --debug

多种模型：

evals:
  models: ['claude-3-5-haiku-latest', 'claude-3-5-sonnet-latest']
  tests: [...]

复杂评估：

evals:
  tests:
    - name: 'Multi-step file management'
      prompt: |
        Create a file called notes.txt with "Meeting notes from today".
        Then read it back and confirm the content is correct.
        Finally, create a backup copy called notes-backup.txt.
      expected_tool_calls:
        required: ['write_file', 'read_file']
        allowed: ['write_file', 'read_file', 'copy_file']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant create the original file, read it back, and create a backup?'
          threshold: 0.8
        - type: 'contains'
          text: 'Meeting notes from today'

统一测试文件：

你可以把两者结合起来 tools 并且 evals 为了方便起见，将各部分放在一个单独的测试文件中：

# Combined test file with both tools and evals
tools:
  expected_tool_list: ['write_file', 'read_file']
  tests:
    - name: 'Write file successfully'
      tool: 'write_file'
      params: { path: '/tmp/test.txt', content: 'Hello world' }
      expect: { success: true }

evals:
  models: ['claude-3-5-haiku-latest']
  tests:
    - name: 'LLM can write files'
      prompt: 'Create a file at /tmp/greeting.txt with content "Hello from Claude"'
      expected_tool_calls:
        required: ['write_file']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant successfully create the file?'
          threshold: 0.8

在执行命令时，每个命令只会使用其各自对应的部分：

mcp-server-tester tools unified-test.yaml 仅使用 tools: 部分；章节
mcp-server-tester evals unified-test.yaml 仅使用 evals: 部分；章节

合规指令（在制品）

⚠️（警告符号，无具体文字含义，表示提醒或警告） 进行中的工作这个（或：这些，具体根据上下文确定） compliance 该命令尚未完全符合规范。

我并不是百分之百确信合规检查器应该内置在这个工具中。但它对我来说很有用，所以我会在这里包含它。它还远远不够完善。

npx mcp-server-tester compliance --server-config filesystem-server-config.json

示例输出：

🏥 MCP SERVER COMPLIANCE
Diagnosing server: Filesystem MCP (MCP Protocol 2024-11-05)
Started: 7/19/2025, 7:53:57 AM

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 MCP SPECIFICATION COMPLIANCE SUMMARY

🔍 BASE PROTOCOL
   └─ Transport Layer      ✅  3/3 passed
   └─ JSON-RPC 2.0         ❌  0/2 passed

🔍 LIFECYCLE
   └─ Initialization       ✅  2/2 passed
   └─ Capability Negotiation ✅  1/1 passed
   └─ Protocol Version     ❌  0/1 passed

🔍 SERVER FEATURES
   └─ Tools                ❌  3/4 passed
   └─ Resources            skipped (not advertised)
   └─ Prompts              skipped (not advertised)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 OVERALL MCP COMPLIANCE: 80/100 (2 features skipped)

Server Capabilities: tools ✅ | prompts ⏭️ | resources ⏭️

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️  SPECIFICATION WARNINGS (4)

⚠️  SDK: JSON-RPC Protocol Compliance [Base Protocol - JSON-RPC 2.0]
   Expected: Full JSON-RPC 2.0 compliance
   Actual:   1 compliance issues detected
   → Fix: Check capability advertisement matches implementation
   → Fix: Verify method name spelling and casing

⚠️  SDK: Error Response Validation [Base Protocol - Error Handling]
   Expected: Proper JSON-RPC error codes and format
   Actual:   2 error handling issues detected
   → Fix: Ensure server returns proper JSON-RPC error codes
   → Fix: Verify that the server properly validates requests
   → Fix: Ensure error handling is implemented according to JSON-RPC spec

⚠️  Lifecycle: Protocol Version Negotiation
   Protocol version issues detected (2 findings, 4 validations)

⚠️  Server Features: Tools - Execution (tools/call)
   Expected: Proper tool execution or error handling
   Actual:   1 execution issues detected
   → Fix: Verify error response format compliance

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📈 DETAILED COMPLIANCE BREAKDOWN
   base-protocol: 3/5 tests passed
   lifecycle: 7/8 tests passed
   server-features: 3/12 tests passed

🔗 SPECIFICATION REFERENCES
• MCP Specification: https://spec.modelcontextprotocol.io/
• JSON-RPC 2.0: https://www.jsonrpc.org/specification
• Error Codes: https://spec.modelcontextprotocol.io/specification/basic/error-handling/

许可证

麻省理工学院（MIT）

使用说明

MCP服务器测试器

为什么选择这个工具？

命令

tools - 直接MCP工具测试

evals - 大语言模型（LLM）集成测试

compliance - MCP规范合规性测试（进行中）

快速入门

工具测试

基本语法

单一工具测试

多步骤工作流测试

期望类型

评估测试

基本语法

工具调用验证

响应评分

高级选项

合规指令（在制品）

许可证

相关 MCP

MySQL查询服务器

iOS/macOS 构建 MCP 服务器与命令行工具

Blender 3D建模MCP服务

iOS模拟器交互服务器

Qdrant向量搜索MCP服务器

网页抓取与搜索MCP服务器

`tools` - 直接MCP工具测试

`evals` - 大语言模型（LLM）集成测试

`compliance` - MCP规范合规性测试（进行中）