MCP测试手册

MCP Testing Playbook

A comprehensive testing and validation playbook for MCP servers and MCP-exposed tools, with a focus on safety, correctness, and risk reduction in agentic AI systems.

Why Testing Is Uniquely Hard for Agentic AI

Traditional software testing assumes deterministic inputs, controlled execution paths, and predictable call patterns. Agentic AI breaks these assumptions:

• Non-deterministic callers: An LLM decides which tools to call, with what arguments, and in what order. Test inputs cannot be fully enumerated.
• Autonomous retries: Agents may retry failed calls, duplicate successful calls, or abandon mid-flow. The caller does not follow a fixed sequence.
• Context-dependent behavior: The same tool call may be valid or invalid depending on prior tool results. State accumulates across a session.
• Unbounded exploration: Agents can discover and invoke any exposed tool. There is no closed set of "supported workflows."
• Failure amplification: A single misbehaving tool can cascade into repeated harmful actions before human intervention.

These properties make exhaustive testing impossible and require a different discipline: testing the contract, failure modes, and safety boundaries rather than happy paths alone.

Why Traditional API Tests Are Insufficient for MCP

Traditional API testing focuses on:

• Request/response validation for known endpoints
• Authentication and authorization for human-initiated requests
• Rate limiting and error handling for predictable traffic patterns

MCP tools differ in critical ways:

• Tools are discovered dynamically: The agent receives a schema and decides whether and how to call. Schema errors propagate as bad tool usage, not HTTP 4xx.
• Tools are chained by an LLM: Order and repetition are emergent. A tool that works in isolation may fail when called repeatedly or out of sequence.
• Side effects matter more: APIs often assume idempotent reads. MCP tools frequently perform writes, deletions, or external actions. Duplicate calls have real consequences.
• The "user" is not human: Permission models designed for human users may be over-broad or under-specified when the caller is an autonomous agent.

Traditional API tests validate that a request returns the expected response. MCP testing must additionally validate that tools behave correctly under retries, misconfiguration, scope violations, and misuse by an autonomous caller.

How MCP Testing Differs from LLM Prompt Testing

LLM prompt testing optimizes for:

• Response quality and relevance
• Few-shot examples and instruction following
• Output format and style

MCP testing optimizes for:

• Tool contract correctness (inputs, outputs, side effects)
• Safety under retries and partial execution
• Permission and scope enforcement
• Deterministic replay and regression detection

Both matter. This playbook focuses on the latter: the infrastructure that tools must satisfy so that agents can use them safely. Prompt engineering determines whether an agent chooses the right tool; tool testing determines whether that tool behaves correctly when chosen.

What This Playbook Covers

• Tool contract testing: Schema validation, input/output contracts, documented side effects
• Idempotency: Retry-safe behavior for write tools
• Scope and permission enforcement: Missing, incorrect, and over-permissioned access
• Destructive action safety: Dry-run, confirmation, and fail-safe patterns
• Replay testing: Capturing and replaying tool calls for regression detection
• Failure mode testing: Intentional breakage (network, timeout, partial execution)
• Anti-patterns: Tool designs that resist reliable testing

The playbook uses principles, checklists, and examples. It does not prescribe a specific test framework or runtime.

What This Playbook Does NOT Cover

• Implementation of an MCP server
• LLM prompt design or evaluation
• Model fine-tuning or retrieval-augmented generation
• Specific MCP SDKs, frameworks, or vendors
• End-to-end agent behavior (only the tool boundary)
• Observability or logging (covered by separate playbooks)

How This Repo Fits With Other Resources

Use this playbook during development and CI. Use observability and safety playbooks in production.

Structure

mcp-testing-playbook/
├── tool-testing/           # Contract, idempotency
│   ├── contract-testing.md
│   └── idempotency.md
├── permission-testing/     # Scope enforcement
│   └── scope-enforcement.md
├── destructive-actions/    # Safety checks for destructive tools
│   └── safety-checks.md
├── agent-simulation/       # Replay testing
│   └── replay-testing.md
├── failure-modes/          # Intentional breakage
│   └── what-to-break.md
├── anti-patterns/          # Untestable designs
│   └── untestable-designs.md
└── examples/               # Framework-agnostic code examples
    ├── python.md
    └── typescript.md