大数据管道监控与自动诊断系统

┌──────────────────────────────────────────────────────────────────────┐
│                    Simulated Big Data Pipeline                        │
│                                                                        │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐      │
│   │  Kafka Exporter │  │  Spark Exporter │  │  HDFS Exporter  │      │
│   │   (Python)      │  │   (Python)      │  │   (Python)      │      │
│   │   port :8001    │  │   port :8002    │  │   port :8003    │      │
│   └────────┬────────┘  └────────┬────────┘  └────────┬────────┘      │
└────────────┼────────────────────┼────────────────────┼───────────────┘
             │  /metrics          │  /metrics          │  /metrics
             ▼                    ▼                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     Prometheus  :9090                                 │
│   • Scrapes all exporters every 15 seconds                            │
│   • Evaluates 12 alert rules continuously                             │
│   • Stores time-series metric data                                    │
└──────────────────────────┬───────────────────────────────────────────┘
                           │  Alert fired (threshold breached)
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                    Alertmanager  :9093                                │
│   • Groups and deduplicates alerts                                    │
│   • Routes by severity (critical / warning)                           │
│   • Sends webhook POST to MCP Server                                  │
└──────────────────────────┬───────────────────────────────────────────┘
                           │  POST /webhook/alert
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     MCP Server  :8888                                 │
│   • FastAPI application (Model Context Protocol)                      │
│   • Receives and stores all firing/resolved alerts                    │
│   • Auto-triggers remediation engine per alert action label           │
│   • 11 YAML runbooks for structured diagnosis                         │
│   • Exposes 11 REST tools for agents and automation                   │
│   • Exposes its own /metrics endpoint (scraped by Prometheus)         │
└──────────────┬────────────────────────────┬─────────────────────────┘
               │                            │
               ▼                            ▼
┌──────────────────────┐      ┌─────────────────────────────────┐
│   Grafana  :3000     │      │    Remediation Agent CLI        │
│                      │      │                                 │
│  • 11-panel dashboard│      │  • Service health table         │
│  • Live time-series  │      │  • Active alerts view           │
│  • Color thresholds  │      │  • Remediation history          │
│  • Auto-provisioned  │      │  • Prometheus metric query      │
│                      │      │  • Manual trigger               │
│                      │      │  • Live monitor (10s refresh)   │
│                      │      │  • AI diagnosis (option 7)      │
│                      │      │  • AI chat mode (option 8)      │
└──────────────────────┘      └─────────────────────────────────┘

git clone https://github.com/AbdAllAh950/mcp-monitoring-project.git
cd mcp-monitoring-project

make setup

cd monitoring
docker compose up -d --build

# Open a new terminal tab — keep this running
cd mcp-server
python3 mcp_server.py

# Open another new terminal tab
cd remediation-agent
python3 agent.py

make demo

curl -X POST http://localhost:8888/webhook/alert \
  -H "Content-Type: application/json" \
  -d '{
    "receiver": "mcp-webhook",
    "status": "firing",
    "alerts": [{
      "status": "firing",
      "labels": {
        "alertname": "HDFSDataNodeDown",
        "severity": "critical",
        "service": "hdfs",
        "action": "restart_datanode"
      },
      "annotations": {
        "summary": "HDFS DataNode is DOWN",
        "description": "DataNode unreachable for 1 minute"
      }
    }],
    "groupLabels": {},
    "commonLabels": {},
    "commonAnnotations": {},
    "externalURL": ""
  }'

You: What alerts are firing right now?
AI:  2 active alerts:
     - KafkaBrokerDown [critical]: Broker unreachable for 30 seconds
     - KafkaConsumerLagHigh [warning]: Consumer group lagging 14,123 on topic transactions
     Data source: GET /tools/get_active_alerts

You: What should I do about the kafka broker?
AI:  1. Verify broker status via kafka_broker_up metric
     2. Review remediation history — restart was already attempted
     3. Trigger restart_broker via /tools/trigger_remediation
     4. Follow KafkaBrokerDown runbook for deep diagnostics
     Data source: GET /tools/get_active_alerts, get_service_health, get_remediation_history

You: Fix it
AI:  Triggered restart_broker for KafkaBrokerDown
     Status: executed
     Action taken: POST /tools/trigger_remediation

cp remediation-agent/.env.example remediation-agent/.env
# Add your OpenRouter API key — free at https://openrouter.ai

OPENAI_API_KEY=sk-or-v1-...
OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_MODEL=qwen/qwen3-8b

Prometheus detects: kafka_consumer_lag > 5000 for 1 minute
        ↓
Alertmanager fires: KafkaConsumerLagHigh (severity=warning, action=restart_consumer)
        ↓
MCP Server receives POST /webhook/alert
        ↓
Runbook lookup: KafkaConsumerLagHigh → symptom, diagnosis steps, safe actions
        ↓
Remediation Engine reads alert.labels.action = "restart_consumer"
        ↓
Runs playbook:
  Step 1: Detect affected consumer group via Prometheus query
  Step 2: Pause consumer group temporarily
  Step 3: Reset consumer offset to latest checkpoint
  Step 4: Restart consumer group
  Step 5: Verify lag is decreasing
        ↓
Result logged: { success: true, duration: 2.0s, steps: [...] }
        ↓
Alert auto-resolves when lag drops below 5000

# Start MCP server first, then:
python3 -m pytest tests/ -v

# Tab 1: Docker stack already running in background

# Tab 2: MCP Server
cd mcp-server && python3 mcp_server.py

# Tab 3: Agent
cd remediation-agent && python3 agent.py
# Press 6 for live auto-refresh monitor
# Press 7 for AI diagnosis
# Press 8 for AI chat mode

make demo
# Opens: Grafana, Prometheus /alerts, MCP API /docs, Alertmanager

curl -X POST http://localhost:8888/webhook/alert \
  -H "Content-Type: application/json" \
  -d '{"receiver":"mcp-webhook","status":"firing",
    "alerts":[{"status":"firing",
      "labels":{"alertname":"HDFSDataNodeDown","severity":"critical",
                "service":"hdfs","action":"restart_datanode"},
      "annotations":{"summary":"HDFS DataNode is DOWN",
                     "description":"DataNode unreachable for 1 minute"}}],
    "groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":""}'

mcp-monitoring-project/
│
├── Makefile                                 # Control panel: setup/start/stop/demo
├── README.md
├── .gitignore
│
├── monitoring/                              # Full Docker monitoring stack
│   ├── docker-compose.yml                   # 8 services in one file
│   ├── prometheus/
│   │   ├── prometheus.yml                   # Scrape configs for all targets
│   │   ├── alert_rules.yml                  # 12 alert rules (Kafka/Spark/HDFS/System)
│   │   └── alertmanager.yml                 # Webhook routing to MCP Server
│   └── grafana/
│       ├── provisioning/
│       │   ├── datasources/prometheus.yml   # Auto-connects Prometheus datasource
│       │   └── dashboards/dashboards.yml    # Auto-loads dashboard on startup
│       └── dashboards/
│           └── mcp-monitoring.json          # 11-panel live dashboard definition
│
├── exporters/                               # Metric simulators
│   ├── kafka_exporter.py                    # Kafka: lag, broker, messages, partitions
│   ├── Dockerfile.kafka
│   ├── spark_exporter.py                    # Spark: tasks, memory, jobs, executors
│   ├── Dockerfile.spark
│   ├── hdfs_exporter.py                     # HDFS: disk, datanodes, blocks, files
│   └── Dockerfile.hdfs
│
├── mcp-server/                              # MCP Server (the brain)
│   ├── mcp_server.py                        # FastAPI app: webhook + tools + remediation
│   ├── runbooks.yaml                        # 11 structured runbooks for every alert
│   └── requirements.txt
│
├── remediation-agent/                       # Interactive CLI agent
│   ├── agent.py                             # Rich terminal UI with AI chat (options 1-8)
│   ├── .env.example                         # Template for OpenRouter API key
│   └── requirements.txt
│
└── tests/                                   # 33 pytest tests
    ├── conftest.py
    ├── test_alert_rules.py                  # 12 tests: alert rule validation
    ├── test_mcp_endpoints.py                # 11 tests: REST API endpoints
    └── test_runbook_coverage.py             # 10 tests: runbook completeness

make setup          # Install all Python dependencies for MCP server and agent
make start          # Start Docker stack + MCP server
make stop           # Stop all services cleanly
make restart        # Stop then start everything
make logs           # Follow all Docker container logs live
make mcp-server     # Start only the MCP server
make agent          # Start only the Remediation Agent CLI
make status         # Check health of all services
make demo           # Open all 4 demo URLs in your browser
make clean          # Stop everything and delete all Docker volumes
make incident-kafka # Simulate Kafka broker failure
make incident-spark # Simulate Spark failure
make incident-hdfs  # Simulate HDFS failure
make incident-stop  # Recover all services