代理视觉自动化服务器

Claude Code (Orchestrator)
    │
    │ run_task("Open Chrome and go to google.com")
    ▼
┌─────────────────────────────────────────────────────────────┐
│  CUA MCP Server (Agentic)                                   │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Internal Agent Loop                                  │  │
│  │  1. screenshot() → CUA sandbox                        │  │
│  │  2. screenshot → Claude API (computer_use tool)       │  │
│  │  3. Claude returns: click(x,y) / type("text") / done  │  │
│  │  4. Execute action on sandbox                         │  │
│  │  5. Loop until complete                               │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
{ success: true, summary: "Opened Chrome...", steps_taken: 5 }
(TEXT ONLY - no images)

api/mcp.ts                     # MCP protocol handler
lib/
├── agent/                     # Modular agent architecture
│   ├── index.ts               # Public exports
│   ├── types.ts               # Type definitions
│   ├── config.ts              # Model configurations
│   ├── validation.ts          # Coordinate validation helpers
│   ├── execute.ts             # Main agent loop
│   ├── describe.ts            # Screen description
│   ├── progress.ts            # Progress tracking
│   ├── utils.ts               # Utilities (sleep, generateTaskId)
│   └── actions/               # Action handler registry (16 handlers)
├── cua-client.ts              # CUA Cloud API client
└── tool-schemas.ts            # MCP tool definitions

{
  "mcpServers": {
    "cua": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://cua-mcp-server.vercel.app/mcp"]
    }
  }
}

You: "List my CUA sandboxes"
Claude: [Uses list_sandboxes tool]

You: "Start my-sandbox"
Claude: [Uses start_sandbox tool]

You: "Open Firefox and go to google.com on my-sandbox"
Claude: [Uses run_task with task="Open Firefox and navigate to google.com"]
→ Returns: { success: true, summary: "Opened Firefox, navigated to google.com", steps_taken: 4 }

You: "What's currently on the screen?"
Claude: [Uses describe_screen tool]
→ Returns: { description: "Firefox browser showing Google homepage with search box..." }

You: "On my-sandbox, open Chrome, go to github.com, and search for 'mcp server'"

Claude uses run_task:
- task: "Open Chrome browser, navigate to github.com, find the search box, type 'mcp server' and press Enter"
- Returns summary of what happened (no screenshots in your context)

You: "What's on the screen right now?"

Claude uses describe_screen:
- focus: "ui" (or "text" or "full")
- Returns text description of UI elements, buttons, text content

You: "Is there a login button visible?"

Claude uses describe_screen:
- question: "Is there a login button visible?"
- Returns: "Yes, there is a blue 'Sign In' button in the top right corner..."

# Clone the repository
git clone https://github.com/anthropics/cua-mcp-server.git
cd cua-mcp-server

# Install dependencies
npm install

# Deploy to Vercel
vercel --prod

{
  "mcpServers": {
    "cua": {
      "command": "npx",
      "args": [
        "-y", "mcp-remote",
        "https://your-deployment.vercel.app/mcp",
        "--header", "X-CUA-API-Key: sk_cua-api01_your-key-here"
      ]
    }
  }
}

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "id": 1,
  "params": {
    "name": "run_task",
    "arguments": {
      "sandbox_name": "s-linux-abc123",
      "task": "Open Firefox and navigate to google.com",
      "max_steps": 30,
      "timeout_seconds": 120
    }
  }
}

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [{
      "type": "text",
      "text": "{\"task_id\":\"task_123...\",\"success\":true,\"summary\":\"Opened Firefox, navigated to google.com\",\"steps_taken\":4,\"duration_ms\":8500}"
    }]
  }
}

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "id": 2,
  "params": {
    "name": "describe_screen",
    "arguments": {
      "sandbox_name": "s-linux-abc123",
      "focus": "ui",
      "question": "Is there a search box visible?"
    }
  }
}