使用说明

🔍 OpenSearch集中式日志服务器

🚀 具有人工智能分析功能的生产就绪集中式测井平台

通过模型上下文协议（MCP）集成，利用实时安全监控、高级事件关联和智能自动化从Linux和Windows系统收集、处理和分析日志。

🚀 快速开始 • ✨ 特性 • 🔧 配置 • 📊 仪表盘 • 🛡️ 安全 • 🤝 贡献

✨ 特性

🔍 多平台日志收集

🐧 Linux系统：Syslog、身份验证日志、应用程序日志、systemd日志
🪟 Windows事件：事件日志、安全事件、通过Winlogbeat的应用程序日志
🌐 网络设备：防火墙、路由器、交换机和安全设备
☁️ 云服务：AWS CloudTrail、Azure日志、谷歌云日志

📊 高级分析引擎

🔍 实时搜索：数十亿日志的亚秒级查询性能
📈 时间序列分析：趋势检测和异常识别
🔗 事件关联：跨系统事件关系映射
🎯 自定义仪表板：基于角色的可视化和报告

🤖 人工智能驱动的智能

🧠 MCP服务器：用于人工智能辅助集成的HTTP API
📱 流媒体分析：高效处理海量数据集
🔮 预测分析：基于机器学习的威胁预测
💬 自然语言查询：通过REST API进行类似SQL的查询

🛡️ 企业安全

⚡ 实时威胁检测：MITRE ATT&CK框架集成
🚨 自动警报：Slack/团队/电子邮件通知
🔐 访问控制：基于角色的身份验证和授权
📋 合规报告：SOC 2、PCI-DSS、HIPAA就绪

🏗️ 建筑

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Linux Logs    │    │  Windows Events │    │  Network Logs   │
│   (Syslog/TCP)  │    │  (Winlogbeat)   │    │   (UDP/TCP)     │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌─────────────▼───────────────┐
                    │        Logstash             │
                    │   (Processing Pipeline)     │
                    └─────────────┬───────────────┘
                                  │
                    ┌─────────────▼───────────────┐
                    │      OpenSearch Cluster     │
                    │    (Node1 + Node2 + HA)    │
                    └─────────────┬───────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          │                       │                       │
┌─────────▼──────────┐  ┌─────────▼──────────┐  ┌─────────▼──────────┐
│   MCP HTTP Server  │  │ OpenSearch Dashboard│ │   Direct API       │
│  (AI Integration)  │  │   (Visualizations)  │ │    (REST/SQL)      │
└──────────|─────────┘  └────────────────────┘  └────────────────────┘
           | 
           |
┌─────────▼──────────┐
│   Claud Desktop    │  
│  (AI Integration)  │  
└────────────────────┘

🚀 快速开始

先决条件

码头工人 和 Docker Compose 安装
4GB+内存 推荐以获得最佳性能
可用端口: 5601, 8000, 9200, 9600, 514, 5044

1.克隆和配置

git clone https://github.com/cybernazmul/OpenLogAI.git
cd OpenLogAi
cp .env_example .env
# Edit .env with your settings

2.启动堆栈

# Start all services
docker-compose up -d

# Check service health
docker-compose ps
docker-compose logs -f

3.克劳德桌面集成

{
  "mcpServers": {
    "opensearch-logs": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://YOUR_SERVER_IP:8000/mcp"
      ]
    }
  }
}

4.访问服务

服务	URL	目的
🔍 OpenSearch API	https://localhost:9200	直接搜索和管理
📊 仪表板	http://localhost:5601	数据可视化
🤖 MCP服务器	http://localhost:8000	AI集成端点
📈 Logstash API	http://localhost:9600	管道监测

5.测试日志收集

# Test syslog ingestion
logger "Test message from $(hostname)"

# Verify in OpenSearch
curl -k -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD} \
  "https://localhost:9200/logstash-logs-*/_search?pretty&size=1"

🔧 配置

环境变量

所有配置都通过环境变量进行管理。复制 .env_example 向 .env 并自定义：

# OpenSearch Cluster
OPENSEARCH_INITIAL_ADMIN_PASSWORD=your_secure_password
OS_URL=https://opensearch-node1:9200
OS_USERNAME=admin
OS_PASSWORD=your_secure_password

# MCP Server Settings
MCP_HOST=0.0.0.0
MCP_PORT=8000
MCP_PATH=/mcp

# Optional: API Key authentication
OS_API_KEY=your_api_key_here

日志源配置

📧 Linux系统

# Add to /etc/rsyslog.conf or /etc/rsyslog.d/50-logstash.conf
*.* @@YOUR_SERVER_IP:514

# For specific facilities
auth.* @@YOUR_SERVER_IP:514
kern.* @@YOUR_SERVER_IP:514
mail.* @@YOUR_SERVER_IP:514

# Restart rsyslog
sudo systemctl restart rsyslog

🪟 Windows系统

# Install and configure Winlogbeat
# Download from: https://www.elastic.co/downloads/beats/winlogbeat

# winlogbeat.yml configuration:
winlogbeat.event_logs:
  - name: Application
    level: error, warning
  - name: Security
    event_id: 4624, 4625, 4648, 4656, 4719, 4720
  - name: System
  - name: Microsoft-Windows-Sysmon/Operational

output.logstash:
  hosts: ["YOUR_SERVER_IP:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

🐳 Docker容器

# Add to your docker-compose.yml services
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
    labels: "service,environment"

# Or use Filebeat for Docker logs

🌐 网络设备

# Cisco/Juniper/Fortigate syslog forwarding
logging YOUR_SERVER_IP:514

# pfSense/OPNsense
Configure System > Advanced > Logging
Remote Logging Options: YOUR_SERVER_IP:514

📊 仪表板和可视化

预构建仪表板类别

🔒 安全运营中心（SOC）：登录失败、权限升级、可疑活动
💻 系统健康：CPU、内存、磁盘使用率、服务状态
🌐 网络分析：流量模式、连接监控、防火墙事件
📱 应用程序日志：错误率、性能指标、用户活动
📋 合规：审计跟踪、访问日志、监管报告

示例查询

OpenSearch REST API

# Search recent security events
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/logstash-logs-*/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must": [
          {"range": {"@timestamp": {"gte": "now-1h"}}},
          {"match": {"syslog_program": "sshd"}}
        ]
      }
    },
    "size": 100
  }'

通过MCP进行SQL查询

-- Find failed SSH attempts
SELECT timestamp, source_ip, syslog_message 
FROM "logstash-logs-*" 
WHERE syslog_message LIKE '%Failed password%' 
AND timestamp > DATE_SUB(NOW(), INTERVAL 1 HOUR)
ORDER BY timestamp DESC;

-- Windows login analysis
SELECT user_name, logon_type, source_ip, COUNT(*) as login_count
FROM "windows-events-*"
WHERE event_id = 4624
  AND timestamp > DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY user_name, logon_type, source_ip
ORDER BY login_count DESC;

-- Top error producers
SELECT syslog_server, syslog_program, COUNT(*) as error_count
FROM "logstash-logs-*" 
WHERE syslog_severity <= 3
  AND timestamp > DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY syslog_server, syslog_program
ORDER BY error_count DESC
LIMIT 20;

🛡️ 安全功能

🎯 自动威胁检测

暴力检测：>5分钟内登录失败5次
横向移动：不寻常的跨系统访问模式
权限提升: sudo, su，管理员组更改
数据渗漏：大额出站转账或异常访问模式
恶意软件指示器：流程执行异常、文件修改

📊 安全监控规则

关键警报

# Failed Authentication Attempts
- Query: syslog_message:"Failed password" OR event_id:4625
- Threshold: >5 occurrences in 5 minutes
- Action: Immediate alert + IP blocking

# Administrative Account Usage
- Query: user_name:"admin" OR user_name:"root" 
- Time: Outside business hours (6PM-8AM)
- Action: Security team notification

# New Process Execution
- Query: event_id:4688 AND process_name:NOT_IN_WHITELIST
- Scope: Critical servers only
- Action: Investigation workflow trigger

警告警报

# Unusual Login Locations
- Geographic IP analysis for user accounts
- First-time login locations
- VPN vs direct connections

# File System Changes
- Sensitive directory access (/etc, /var/log, C:\Windows\System32)
- Configuration file modifications
- Log file tampering attempts

🔐 访问控制

# Create role-based users
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/_plugins/_security/api/internalusers/analyst" \
  -H 'Content-Type: application/json' \
  -d '{
    "password": "analyst_password",
    "roles": ["logs_reader", "dashboard_user"]
  }'

📈 性能和扩展

资源需求

部署	RAM	CPU	存储	每日日志量
发展	4GB	2核	20GB	<1GB
小型企业	8GB	4核	100GB	1-10GB
企业	32GB+	16+核	1TB+	100GB+
大规模	128GB+	32+核	10TB+	1TB+

扩展策略

# Horizontal scaling - add nodes
services:
  opensearch-node3:
    image: opensearchproject/opensearch:latest
    environment:
      - node.name=opensearch-node3
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2,opensearch-node3

# Load balancing
nginx:
  image: nginx:alpine
  ports:
    - "80:80"
    - "443:443"
  volumes:
    - ./nginx.conf:/etc/nginx/nginx.conf

性能优化

# Production tuning in .env
OPENSEARCH_JAVA_OPTS=-Xms8g -Xmx8g
LS_JAVA_OPTS=-Xmx4g -Xms4g

# Index lifecycle management
curl -X POST "localhost:9200/_plugins/_ism/policies/log_policy" \
  -H 'Content-Type: application/json' \
  -d '{
    "policy": {
      "states": [
        {
          "name": "hot",
          "actions": [],
          "transitions": [{
            "state_name": "warm",
            "conditions": {"min_index_age": "7d"}
          }]
        },
        {
          "name": "warm", 
          "actions": [{"warm": {}}],
          "transitions": [{
            "state_name": "delete",
            "conditions": {"min_index_age": "90d"}
          }]
        },
        {"name": "delete", "actions": [{"delete": {}}]}
      ]
    }
  }'

🤖 MCP服务器集成

API终点

# Health check
curl http://localhost:8000/health

# List all indices with metadata
curl -X POST http://localhost:8000/mcp/tools/list_indices \
  -H "Content-Type: application/json" \
  -d '{"pattern": "*", "include_system": false}'

# Execute SQL queries
curl -X POST http://localhost:8000/mcp/tools/execute_sql \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT * FROM \"logstash-logs-*\" WHERE syslog_severity <= 3 LIMIT 10"}'

# Search with complex filters
curl -X POST http://localhost:8000/mcp/tools/search_documents \
  -H "Content-Type: application/json" \
  -d '{
    "index": "logstash-logs-*",
    "query": {
      "bool": {
        "must": [
          {"range": {"@timestamp": {"gte": "now-1h"}}},
          {"match": {"syslog_program": "sshd"}}
        ]
      }
    },
    "size": 50
  }'

Claude桌面集成

{
  "mcpServers": {
    "opensearch-logs": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://YOUR_SERVER_IP:8000/mcp"
      ]
    }
  }
}

AI助手用例

🔍 智能日志分析：“显示最近一小时内所有失败的登录尝试”
📊 趋势分析：“本周十大错误源是什么？”
🚨 安全调查：“查找用户X的可疑活动模式”
📈 性能监控：“识别错误率不断上升的服务”

📊 仪表板示例

安全运营中心（SOC）

# Import pre-built dashboards
curl -X POST "localhost:5601/api/saved_objects/_import" \
  -H "kbn-xsrf: true" \
  -F file=@dashboards/soc-dashboard.ndjson

关键可视化

🔥 实时事件流：带有过滤功能的实时日志提要
🌍 地理攻击地图：按位置登录尝试次数
📊 安全事件时间表：历史攻击模式
⚠️ 警报摘要：当前的威胁及其严重性
💻 系统健康概述：所有受监控系统的基础设施状态

🛡️ 安全功能

🎯 内置检测规则

关键安全事件

# Brute Force Attacks
- Pattern: Multiple authentication failures
- Threshold: >5 failures in 5 minutes per source IP
- Response: Auto-block + SOC alert

# Privilege Escalation
- Pattern: sudo/su usage, admin group additions
- Scope: Non-service accounts
- Response: Immediate investigation trigger

# Suspicious Process Execution
- Pattern: PowerShell/cmd.exe with encoded commands
- Pattern: Unusual parent-child process relationships
- Response: Process analysis + containment

# Data Access Anomalies  
- Pattern: Large file transfers outside business hours
- Pattern: Access to sensitive directories
- Response: Data loss prevention workflow

Windows特定检测

# Event ID Monitoring
4624: Successful logon (track unusual locations/times)
4625: Failed logon (brute force detection)
4648: Explicit credential use (lateral movement)
4656: File/object access (data access monitoring)
4719: System audit policy changes (tampering detection)
4720: User account created (account management)

# PowerShell Security
4103: Module logging (malicious command detection)
4104: Script block logging (obfuscated script analysis)
4105: Script start (execution monitoring)
4106: Script stop (completion tracking)

🔐 身份验证和访问控制

# Create security roles
curl -k -u admin:${PASSWORD} -X PUT \
  "https://localhost:9200/_plugins/_security/api/roles/security_analyst" \
  -H 'Content-Type: application/json' \
  -d '{
    "cluster_permissions": ["cluster_composite_ops_ro"],
    "index_permissions": [{
      "index_patterns": ["logstash-logs-*", "security-events-*"],
      "allowed_actions": ["read", "search"]
    }]
  }'

# Create users with specific roles
curl -k -u admin:${PASSWORD} -X PUT \
  "https://localhost:9200/_plugins/_security/api/internalusers/analyst1" \
  -H 'Content-Type: application/json' \
  -d '{
    "password": "secure_password",
    "roles": ["security_analyst"],
    "attributes": {
      "department": "Security",
      "clearance_level": "L2"
    }
  }'

🔧 高级配置

高可用性设置

# Add load balancer
nginx:
  image: nginx:alpine
  ports:
    - "80:80"
    - "443:443"
  volumes:
    - ./nginx/nginx.conf:/etc/nginx/nginx.conf
    - ./nginx/ssl:/etc/nginx/ssl
  depends_on:
    - opensearch-node1
    - opensearch-node2

# Cross-cluster replication
opensearch-remote:
  image: opensearchproject/opensearch:latest
  environment:
    - cluster.name=opensearch-backup
    - replication.enabled=true

自定义日志处理管道

# Enhanced logstash pipeline for Windows events
filter {
  if [agent][type] == "winlogbeat" {
    # Windows Event ID classification
    if [winlog][event_id] in [4624, 4634] {
      mutate { add_tag => ["authentication"] }
    }
    if [winlog][event_id] in [4625, 4648] {
      mutate { add_tag => ["failed_auth", "security_alert"] }
    }
    if [winlog][event_id] in [4656, 4658, 4660, 4663] {
      mutate { add_tag => ["file_access", "audit"] }
    }
    
    # Extract user and computer information
    grok {
      match => {
        "[winlog][event_data][SubjectUserName]" => "(?<user_name>[^$]+)"
      }
      tag_on_failure => ["_grok_user_failure"]
    }
    
    # GeoIP enrichment for external IPs
    if [winlog][event_data][IpAddress] and [winlog][event_data][IpAddress] !~ /^(10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.)/ {
      geoip {
        source => "[winlog][event_data][IpAddress]"
        target => "geoip"
        database => "/usr/share/GeoIP/GeoLite2-City.mmdb"
      }
    }
  }
}

# Output to specialized indices
output {
  if "security_alert" in [tags] {
    opensearch {
      index => "security-events-%{+YYYY.MM.dd}"
      # ... connection details
    }
  } else if "authentication" in [tags] {
    opensearch {
      index => "auth-logs-%{+YYYY.MM.dd}"
      # ... connection details  
    }
  } else {
    opensearch {
      index => "logstash-logs-%{+YYYY.MM.dd}"
      # ... connection details
    }
  }
}

📊 监控和警报

健康监测

# Service health checks
docker-compose exec opensearch-node1 curl -k \
  "https://localhost:9200/_cluster/health?pretty"

# Logstash pipeline monitoring  
curl "http://localhost:9600/_node/stats/pipelines?pretty"

# MCP server status
curl "http://localhost:8000/health"

性能指标

# Index statistics
curl -k -u admin:${PASSWORD} \
  "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=store.size:desc"

# Search performance
curl -k -u admin:${PASSWORD} \
  "https://localhost:9200/_nodes/stats/indices/search?pretty"

# Memory usage
curl -k -u admin:${PASSWORD} \
  "https://localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu"

🔧 故障排除

常见问题

OpenSearch无法启动

# Check logs
docker-compose logs opensearch-node1

# Common fixes
sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

# Verify disk space
df -h

Logstash处理问题

# Check pipeline status
curl "http://localhost:9600/_node/stats/pipelines?pretty"

# Monitor processing rates
docker-compose logs logstash | grep -E "(ERROR|WARN)"

# Test configuration
docker-compose exec logstash logstash --config.test_and_exit

MCP服务器连接问题

# Test MCP connectivity
curl http://localhost:8000/health

# Check OpenSearch connectivity from MCP
docker-compose exec mcp-server python -c "
from mcp_tools import OpenSearchClient
client = OpenSearchClient('https://opensearch-node1:9200', 'admin', '${PASSWORD}')
print(client.health_check())
"

日志分析命令

# Find parsing failures
curl -k -u admin:${PASSWORD} \
  "https://localhost:9200/logstash-logs-*/_search?q=tags:_grokparsefailure"

# Monitor ingestion rates
curl -k -u admin:${PASSWORD} \
  "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=docs.count:desc"

# Check for duplicate events
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/logstash-logs-*/_search" \
  -d '{"aggs":{"duplicates":{"terms":{"field":"message.keyword","size":10}}}}'

🚀 生产部署

安全强化

# SSL/TLS Configuration
opensearch-node1:
  environment:
    - plugins.security.ssl.http.enabled=true
    - plugins.security.ssl.http.pemcert_filepath=certs/node1.pem
    - plugins.security.ssl.http.pemkey_filepath=certs/node1-key.pem
  volumes:
    - ./certs:/usr/share/opensearch/config/certs:ro

# Firewall rules (UFW example)
sudo ufw allow from TRUSTED_NETWORK to any port 9200
sudo ufw allow from TRUSTED_NETWORK to any port 5601
sudo ufw allow 514/tcp  # Syslog
sudo ufw allow 5044/tcp # Beats

备份和恢复

# Snapshot configuration
curl -k -u admin:${PASSWORD} -X PUT \
  "https://localhost:9200/_snapshot/backup_repository" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/usr/share/opensearch/backups",
      "compress": true
    }
  }'

# Automated daily snapshots
curl -k -u admin:${PASSWORD} -X PUT \
  "https://localhost:9200/_plugins/_ism/policies/snapshot_policy" \
  -H 'Content-Type: application/json' \
  -d '{
    "policy": {
      "description": "Daily snapshots",
      "default_state": "snapshot",
      "states": [{
        "name": "snapshot",
        "actions": [{"snapshot": {"snapshot": "daily-{now/d}"}}],
        "transitions": [{"state_name": "snapshot", "conditions": {"cron": {"cron": {"expression": "0 2 * * *"}}}}]
      }]
    }
  }'

📚 文档和资源

📖 其他文件

🔗 外部资源

🎓 学习资源

🤝 贡献

我们欢迎捐款！请阅读我们的贡献指南在提交PR之前。

开发设置

# Clone repository
git clone https://github.com/your-username/opensearch-centralized-logs.git
cd opensearch-centralized-logs

# Setup development environment
cp .env_example .env
docker-compose -f docker-compose.dev.yml up -d

# Run tests
make test-all

# Lint code
make lint

# Generate documentation
make docs

🐛 问题模板

错误报告：用于功能问题
功能请求：建议新功能
安保问题：私下报告安全漏洞
文档：改进文档和示例

📄 许可证

此项目根据MIT许可证获得许可-请参阅许可证文件以获取详细信息。

🌟 支持与社区

📧 问题: GitHub问题
💬 讨论: GitHub讨论
📖 维基: 维基工程
🆘 支持: 社区论坛

🏆 贡献者

感谢我们出色的贡献者！看贡献者.md 查看完整列表。

🎯 路线图

📅 2025年第一季度

[]增强的Windows事件解析
[]基于机器学习的异常检测
[]移动设备日志集成
[]高级合规报告

📅 2025年第二季度

[]Kubernetes日志收集
[]云原生部署选项
[]高级关联引擎
[]自定义插件市场

📅 2025年第3季度

[]多租户架构
[]企业SSO集成
[]高级威胁追踪工具
[]自动事件响应

⭐ 如果你觉得这个存储库有用，请为其加星！

🔗 在GitHub上查看 | 📖 文档 | 🚀 开始使用

内置于❤️ 面向网络安全和DevOps社区

使用说明

🔍 OpenSearch集中式日志服务器

✨ 特性

🔍 多平台日志收集

📊 高级分析引擎

🤖 人工智能驱动的智能

🛡️ 企业安全

🏗️ 建筑

🚀 快速开始

先决条件

1.克隆和配置

2.启动堆栈

3.克劳德桌面集成

4.访问服务

5.测试日志收集

🔧 配置

环境变量

日志源配置

📧 Linux系统

🪟 Windows系统

🐳 Docker容器

🌐 网络设备

📊 仪表板和可视化

预构建仪表板类别

示例查询

OpenSearch REST API

通过MCP进行SQL查询

🛡️ 安全功能

🎯 自动威胁检测

📊 安全监控规则

关键警报

警告警报

🔐 访问控制

📈 性能和扩展

资源需求

扩展策略

性能优化

🤖 MCP服务器集成

API终点

Claude桌面集成

AI助手用例

📊 仪表板示例

安全运营中心（SOC）

关键可视化

🛡️ 安全功能

🎯 内置检测规则

关键安全事件

Windows特定检测

🔐 身份验证和访问控制

🔧 高级配置

高可用性设置

自定义日志处理管道

📊 监控和警报

健康监测

性能指标

🔧 故障排除

常见问题

OpenSearch无法启动

Logstash处理问题

MCP服务器连接问题

日志分析命令

🚀 生产部署

安全强化

备份和恢复

📚 文档和资源

📖 其他文件

🔗 外部资源

🎓 学习资源

🤝 贡献

开发设置

🐛 问题模板

📄 许可证

🌟 支持与社区

🏆 贡献者

🎯 路线图

📅 2025年第一季度

📅 2025年第二季度

📅 2025年第3季度

相关 MCP

实时金融新闻分析服务