使用说明

MCP Scraper

Automated web scraper for collecting MCP (Model Context Protocol) server data from mcp.so.

Features

🚀 Automated Discovery - Scrapes latest MCP servers from mcp.so
🗄️ PostgreSQL Storage - Structured data with proper relationships
🔄 Smart Updates - Upserts existing records, inserts new ones
🛡️ Robust Error Handling - Retry logic with exponential backoff
📊 Detailed Logging - Track successes, failures, and statistics
⚡ Concurrent Scraping - Batched requests for efficiency
🏷️ Full Metadata - Extracts categories, tags, tools, config, and documentation

Architecture

├── src/
│   ├── index.js          # Main orchestrator
│   ├── scraper.js        # Playwright automation
│   ├── parser.js         # Data extraction & transformation
│   ├── database.js       # PostgreSQL operations
│   └── config.js         # Configuration management
├── migrations/
│   └── 001_init_schema.sql  # Database schema
└── logs/
    └── scrape_errors.log    # Error tracking

Prerequisites

Node.js 16+
PostgreSQL database (hosted on VPS: 195.26.250.232)
PowerTools environment profile with:
- POSTGRESQL_SERVER
- POSTGRESQL_USER
- POSTGRESQL_SCOTT_PW

Installation

1. Install Dependencies

npm install

2. Install Playwright Browsers

npx playwright install chromium

3. Create Database

# Option 1: Using npm script
npm run db:create

# Option 2: Manual
psql -h 195.26.250.232 -U your_username -c 'CREATE DATABASE mcp_scraper;'

4. Run Migrations

# Option 1: Using npm script (requires PowerTools profile active)
npm run migrate

# Option 2: Manual
psql -h 195.26.250.232 -U your_username -d mcp_scraper -f migrations/001_init_schema.sql

5. Configure Environment (Optional)

Copy .env.example to .env if you want to override defaults:

cp .env.example .env

Note: If you're using PowerTools environment profile, the scraper will automatically use POSTGRESQL_* variables. The .env file is only needed for additional configuration.

Usage

Run the Scraper

# Make sure PowerTools profile is active
npm start

Example Output

=== MCP Scraper Starting ===

✓ Database connected: 2026-01-21T06:00:00.000Z
✓ Browser initialized

--- Phase 1: Discovery ---
Navigating to: https://mcp.so/servers?tag=latest
✓ Found 187 servers

--- Phase 2: Detail Scraping ---

Batch 1/38 (5 servers):
  ✓ Playwright Mcp (12 tools)
  ✓ AlphaVantage (8 tools)
  ✓ Git MCP Server (6 tools)
  ✓ Filesystem MCP (15 tools)
  ✓ Brave Search (4 tools)

...

=== Scraping Summary ===
Total servers: 187
✓ Success: 185
✗ Failed: 2
Duration: 248.53s

=== Database Stats ===
Total servers: 185
Unique servers: 185
Total tools: 1342
Categories: 18
Tags: 94

--- Cleanup ---
✓ Browser closed
✓ Database pool closed
✓ All resources released

Database Schema

Tables

servers - Main MCP server information
tools - Tools/functions provided by each server
categories - Server categories (M:N with servers)
tags - Server tags/hashtags (M:N with servers)
server_categories - Junction table for server-category relationships
server_tags - Junction table for server-tag relationships

Query Examples

-- Find all servers with #automation tag
SELECT s.* FROM servers s
JOIN server_tags st ON s.id = st.server_id
JOIN tags t ON st.tag_id = t.id
WHERE t.name = 'automation';

-- Find all servers in 'Browser Automation' category
SELECT s.* FROM servers s
JOIN server_categories sc ON s.id = sc.server_id
JOIN categories c ON sc.category_id = c.id
WHERE c.name = 'Browser Automation';

-- Find all tools that mention 'screenshot'
SELECT * FROM tools
WHERE tool_description ILIKE '%screenshot%';

-- Get server with all its tools
SELECT s.name, t.tool_name, t.tool_description
FROM servers s
LEFT JOIN tools t ON s.id = t.server_id
WHERE s.slug = 'playwright-mcp';

Configuration

Environment Variables

The scraper uses the following configuration (in order of precedence):

PowerTools System Environment Variables
- POSTGRESQL_SERVER → Database host
- POSTGRESQL_USER → Database user
- POSTGRESQL_SCOTT_PW → Database password
Optional .env File (fallback/overrides)
- DB_NAME - Database name (default: mcp_scraper)
- DB_PORT - Database port (default: 5432)
- BASE_URL - Base URL for scraping (default: https://mcp.so)
- SCRAPE_DELAY - Delay between batches in ms (default: 1000)
- MAX_RETRIES - Retry attempts per page (default: 3)
- HEADLESS - Run browser headless (default: true)
- CONCURRENT_LIMIT - Concurrent requests (default: 5)

Error Handling

Retry Logic: 3 attempts per page with exponential backoff (2s → 4s → 8s)
Error Logging: Failed URLs logged to logs/scrape_errors.log
Graceful Degradation: One failed server doesn't stop the entire scrape
Transaction Safety: Database operations wrapped in transactions

Maintenance

Weekly Scraping

Run the scraper weekly to capture new servers:

npm start

The upsert logic ensures:

New servers are inserted
Existing servers are updated with latest data
No duplicates are created

Database Cleanup

To start fresh:

npm run db:drop
npm run db:create
npm run migrate

Development

Project Structure

mcp-scraper/
├── src/
│   ├── index.js       # Main entry point with orchestration
│   ├── scraper.js     # Playwright scraping logic
│   ├── parser.js      # Data parsing & transformation
│   ├── database.js    # PostgreSQL operations
│   └── config.js      # Configuration loader
├── migrations/        # Database schema migrations
├── logs/             # Error and activity logs
├── package.json      # Dependencies and scripts
└── README.md         # This file

Adding New Fields

Add column to migrations/001_init_schema.sql
Run migration
Update database.js upsert query
Update scraper.js extraction logic
Update parser.js if transformation needed

Troubleshooting

Database Connection Errors

Verify PowerTools profile is active
Check VPS is accessible: ping 195.26.250.232
Test connection: psql -h 195.26.250.232 -U your_user -d mcp_scraper

Scraping Timeouts

Increase timeout in scraper.js
Check internet connection
Verify mcp.so is accessible

Missing Data

Inspect HTML structure at mcp.so (sites change over time)
Update selectors in scraper.js
Add debugging: Set HEADLESS=false in .env

High Failure Rate

Check logs/scrape_errors.log for patterns
Increase SCRAPE_DELAY to be more polite
Decrease CONCURRENT_LIMIT to reduce load

License

MIT

Related Projects

Web Scraping Skill - Reusable patterns for building scrapers
Design Document - Full architectural design

Built with Playwright • PostgreSQL • Node.js