MCP服务器爬虫

一个全面的MCP(模型上下文协议)服务器,用于爬取Dedalus Marketplace以发现并列出所有可用的MCP服务器及其元数据。

  • by BUZDOLAPCI
  • TypeScript
  • 本地部署
  • 1 星标
  • 更新于
在 GitHub 查看 BUZDOLAPCI/dedalus-marketplace-crawler-ts

使用说明

Dedalus市场爬虫MCP服务器

一个全面的MCP(模型上下文协议)服务器,用于抓取 Dedalus市场 以发现并列出所有可用的MCP服务器及其元数据。

特性

  • 基于API的发现:使用公共Dedalus Marketplace API(无需执行浏览器/JavaScript)
  • 综合元数据:提取名称、描述、GitHub URL、语言等
  • 双重运输:支持HTTP(流式HTTP)和STDIO传输
  • TypeScript:完全按照Dedalus准则实施
  • 健壮性测试:针对实时API的带有mocking+集成测试的单元测试

数据提取

对于市场上的每台服务器:

  • 姓名&蛞蝓:服务器标识符和完整slug(发布者/名称)
  • 出版商:组织或作者姓名
  • GitHub URL:链接到源存储库
  • 描述:服务器描述文本
  • 语言:编程语言(TypeScript、Python)
  • 热度得分:人气指标(0-100)
  • 赞成票计数:社区赞成票数量
  • 认证:必需的身份验证类型(api_key、oauth、none)
  • 工具 (可选):带说明的MCP工具列表

安装

先决条件

  • Node.js 18+
  • npm或纱线

设置

# Clone the repository
git clone https://github.com/dedalus-labs/dedalus-marketplace-crawler-ts.git
cd dedalus-marketplace-crawler-ts

# Install dependencies
npm install

# Build the project
npm run build

用法

HTTP传输(默认)

# Start the server on port 8080
npm start

# Or specify a custom port
npm start -- --port 3000

服务器将输出其MCP客户端配置:

{
  "mcpServers": {
    "dedalus-marketplace-crawler": {
      "url": "http://localhost:8080/mcp"
    }
  }
}

STDIO传输

npm run start:stdio

开发模式

# Run with tsx (no build required)
npm run dev

# Run STDIO mode
npm run dev:stdio

工具:扫描市场

服务器公开了一个名为 scan_marketplace.

参数

名称 类型 默认值 描述
include_tools 布尔值 false 如果为true,则获取每个服务器的工具信息(速度较慢但更完整)

输出示例

Found 41 servers in the Dedalus Marketplace

---

## tsion/brave-search-mcp
Name: brave-search-mcp
Publisher: tsion
Description: A Model Context Protocol server for web and local search using the Brave Search API
GitHub: https://github.com/dedalus-labs/brave-search-mcp
Marketplace: https://www.dedaluslabs.ai/marketplace/tsion/brave-search-mcp
Language: typescript
Heat Score: 99
Upvotes: 14
Auth Required: api_key

## windsor/ticketmaster-mcp
Name: ticketmaster-mcp
Publisher: windsor
Description: Search for events, venues, and attractions
GitHub: https://github.com/windsornguyen/ticketmaster-mcp
Marketplace: https://www.dedaluslabs.ai/marketplace/windsor/ticketmaster-mcp
Language: typescript
Heat Score: 88
Upvotes: 5
Auth Required: api_key

配置

环境变量

变量 默认值 描述
PORT 8080 HTTP服务器端口
NODE_ENV development 设置为 production 用于生产模式

客户端配置

克劳德桌面版

{
  "mcpServers": {
    "dedalus-marketplace-crawler": {
      "url": "http://localhost:8080/mcp"
    }
  }
}

VS Code

{
  "mcp": {
    "servers": {
      "dedalus-marketplace-crawler": {
        "url": "http://localhost:8080/mcp"
      }
    }
  }
}

测试

单元测试

# Run unit tests
npm test

集成测试

# Run integration tests against live API
INTEGRATION_TEST=true npm test

所有测试

# Run all tests
INTEGRATION_TEST=true npm test

API信息

爬虫使用公共Dedalus Marketplace API:

端点: https://www.dedaluslabs.ai/api/marketplace

响应结构:

{
  "repositories": [
    {
      "slug": "publisher/server-name",
      "git_slug": "github-org/repo-name",
      "description": "Server description",
      "language": "typescript",
      "heat_score": 99,
      "upvote_count": 14,
      "tags": {
        "auth": {"api_key": true, "oauth": false, "none": false},
        "language": "typescript"
      }
    }
  ]
}

项目结构

dedalus-marketplace-crawler-ts/
├── src/
│   ├── index.ts            # Main entry point
│   ├── cli.ts              # Command-line argument parsing
│   ├── config.ts           # Configuration management
│   ├── client.ts           # Marketplace API client
│   ├── server.ts           # MCP server instance
│   ├── types.ts            # TypeScript type definitions
│   ├── tools/
│   │   ├── index.ts        # Tool exports
│   │   └── crawler.ts      # Marketplace crawler tool
│   └── transport/
│       ├── index.ts        # Transport exports
│       ├── http.ts         # HTTP transport
│       └── stdio.ts        # STDIO transport
├── tests/
│   ├── client.test.ts      # Client unit tests
│   ├── tools.test.ts       # Tool unit tests
│   ├── cli.test.ts         # CLI unit tests
│   ├── config.test.ts      # Config unit tests
│   └── integration.test.ts # Live API tests
├── package.json
├── tsconfig.json
└── jest.config.js

许可证

该项目根据MIT许可证获得许可。

致谢