AI导哈

MCP测试框架

一个AI驱动的MCP服务器测试框架，通过真实AI代理进行对话和LLM评估结果，适用于自然语言交互的全面测试。

  • by golf-mcp
  • Python
  • 本地部署
  • 5 星标
  • 更新于
在 GitHub 查看 golf-mcp/golf-testing

使用说明

🧪 MCP Testing

AI-powered testing framework for MCP servers

Test your MCP servers with real AI agents conducting conversations and LLM judges evaluating results

为何选择MCP测试？

传统的测试方法并不适用于MCP服务器。你无法为自然语言交互编写单元测试。MCP测试通过以下方式解决了这一问题：

  • 🤖 真实的人工智能代理 - Claude和ChatGPT实际上使用的是您的MCP服务器
  • 👤 用户模拟 - AI模拟真实的多轮用户行为
  • ⚖️ 大语言模型（LLM）作为法官 - 智能评估，而非武断断言
  • 🎭 全面测试 - 安全、合规与性能集于一身的框架
  • 🔌 多种传输方式 - 支持HTTP和stdio服务器

快速入门

分三步进行测试：

  1. 安装与设置

    pip install mcp-testing
export ANTHROPIC_API_KEY="sk-ant-..."  # For AI agents
export OPENAI_API_KEY="sk-..."         # For LLM judge

  2. 互动式入职培训

    mcp-t quickstart  # Creates your first server & test suite

  3. 运行测试

    mcp-t run <suite-id> <server-id>
# Example: mcp-t run example_suite_001 hackernews_mcp_server

核心概念

测试流程

Your Test Case → AI Agent (Claude/GPT-4) → Your MCP Server
      ↓                    ↓                      ↓
 User Message         Tool Calls            Server Response
      ↓                    ↓                      ↓
User Simulator      Conversation Loop         More Tools
      ↓                    ↓                      ↓
   LLM Judge       Complete Transcript      Pass/Fail + Reasoning

配置文件

服务器配置 - HTTP (examples/server.json):

{
  "name": "linear_mcp_server",
  "transport": "http",
  "url": "https://mcp.linear.app/mcp"
}

服务器配置 - 标准输入输出（stdio） (examples/servers/time-server-stdio.json)：

{
  "name": "Time Server",
  "transport": "stdio",
  "command": "npx -y @modelcontextprotocol/server-time"
}

服务器配置 - 带环境的stdio（标准输入输出） (examples/servers/brave-search-stdio.json):

{
  "name": "Brave Search",
  "transport": "stdio",
  "command": "npx -y @modelcontextprotocol/server-brave-search",
  "env": {
    "BRAVE_API_KEY": "your-api-key-here"
  }
}

测试套件 （examples/suite.json)：

{
  "suite_id": "example_suite_001",
  "name": "Hacker News MCP Server Tests",
  "test_cases": [
    {
      "test_id": "hackernews_greeting",
      "user_message": "Hello! Can you help me browse Hacker News?",
      "success_criteria": "Agent should respond politely and explain Hacker News capabilities",
      "max_turns": 5
    }
  ]
}

测试类型

  • 💬 交谈式的 - 真实用户工作流程
  • 🔒 安全 - 认证与漏洞
  • ✅ 合规性 - MCP协议验证

命令

测试执行

mcp-t run <suite-id> <server-id>           # Run specific suite
mcp-t run example_suite_001 hackernews_mcp_server -v   # Verbose output

配置管理

mcp-t quickstart                 # Complete onboarding
mcp-t create server              # Interactive server setup
mcp-t create suite               # Create test suite
mcp-t create test-case           # Add test to suite
mcp-t list                       # Show all configs
mcp-t show suite example_suite_001   # View specific config

测试生成

运行向导，分析您的MCP服务器，并自动生成全面的测试用例

mcp-t generate

测试结果

理解评估

{
  "test_id": "hackernews_stories",
  "verdict": "PASS",
  "confidence_score": 0.89,
  "judge_reasoning": "The agent successfully fetched and displayed Hacker News stories. Good use of available tools and clear presentation of results.",
  "conversation_quality": 0.87,
  "tool_calls": [
    { "tool": "get_top_stories", "args": {} },
    { "tool": "get_story_details", "args": { "story_id": 123 } }
  ]
}

支持

Built with ❤️ for the MCP ecosystem

Made in San Francisco, CA

相关 MCP