AI代理可观测性工具

Token Metrics (what tokens-only telemetry sees):
  Step 1: 89 tokens   Normal
  Step 2: 112 tokens  Normal
  Step 3: 156 tokens  Normal   ← MISSED THE PROBLEM!

Decision Metrics (what gen_ai.agent.* + gen_ai.evaluation.result add):
  Step 1: confidence=0.92, deviation=0.05  Normal
  Step 2: confidence=0.88, deviation=0.08  Normal
  Step 3: confidence=0.34, deviation=0.78  ANOMALY DETECTED
          Expected: QueryMetrics
          Actual:   send_http (external)
          Scope:    network:external (not in baseline)
          → PreExecutionGate.decide() returns allowed=false
          → BLOCKED before tool execution
          → emits  gen_ai.evaluation.result event (score.label=critical)
                   gen_ai.agent.gate.blocked=true

Prompt arrives
     │
     ▼
┌─────────────────────────────────┐
│ Layer 0: Safety Pre-Filter      │  Regex: injection, question, vague (0ms)
│ (deterministic — no LLM)        │  Embedding: actionability check (50ms)
│                                 │  → Catches 24% of prompts
└─────────┬───────────────────────┘
          │
          ▼
┌─────────────────────────────────┐
│ Layer 1-2: Task Shape Classifier│  Embedding (93%) + LLM fallback (100%)
│ (hybrid — LLM only when needed) │  → Routes to correct baseline
└─────────┬───────────────────────┘
          │
          ▼
┌─────────────────────────────────┐
│ Layer 3: Pre-Execution Gate     │  Deviation scoring against baseline
│ (deterministic formula)         │  → 579/579 anomalies caught, 0 escaped
└─────────┬───────────────────────┘
          │
          ▼
┌─────────────────────────────────┐
│ Layer 4: Scope Enforcement      │  Policy-based allow/deny
│ (deterministic policy)          │  → scope.denied = always block
└─────────────────────────────────┘

git clone https://github.com/graphsentinel/otel-mcp-agent-extension.git
cd otel-mcp-agent-extension

# Install the library (and provider extras) once
pip install -e ".[all]"

# Bring up the SRE example's observability stack
cd examples/sre-incident-demo
./scripts/setup.sh
podman-compose up -d        # or: docker compose up -d

DEMO_DATA_DIR=/path/to/demo-data podman-compose up -d

# Smoke test — 2 categories × 3 models, simulator (no LLM, no keys)
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py --pilot

# Full default run, simulator
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py --runs 30

# Real local Ollama (load .env first if your llm_connection_conf.yaml routes
# any pattern to a cloud / RunPod backend)
set -a; source examples/sre-incident-demo/.env; set +a
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py \
    --engine ollama --models qwen2.5-coder:7b --pilot

# Tighter / looser pre-execution gate
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py --threshold 0.4
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py --threshold 0.8

# Subset by category / variant
python examples/sre-incident-demo/src/demo_sre_inverse_scaling.py \
    --categories cleanup_envs,hotfix_deploy --variants v3,v4

# Mock mode (no LLM required)
python examples/legacy-interactive-demo/run_demo.py --mock --compare

# Local Ollama
python examples/legacy-interactive-demo/run_demo.py --ollama --host http://localhost:11434 --compare

# OpenAI
OPENAI_API_KEY=sk-... python examples/legacy-interactive-demo/run_demo.py --openai --compare

# Reproduce champion selection + consensus baseline from the original 3,650-run dataset
python examples/sre-incident-demo/analysis/baseline_simulation.py

# All 3 approaches (embedding, LLM, hybrid)
python examples/sre-incident-demo/analysis/task_shape_classifier.py --approach all

# v2 with Layer0 safety filter + adversarial prompts (50 test cases)
python examples/sre-incident-demo/analysis/task_shape_classifier_v2.py --approach 4

# ── OTel-spec GenAI sem-conv (Development; used verbatim) ────────────────
gen_ai.operation.name:        "invoke_agent"        # or "execute_tool"
gen_ai.provider.name:         "openai"
gen_ai.conversation.id:       "sess_abc123"
gen_ai.agent.id:              "agt-001"             # 1 of 4 reserved spec attrs
gen_ai.agent.name:            "sre-incident-bot"
gen_ai.request.model:         "qwen2.5-coder:7b"
gen_ai.tool.name:             "send_http"
gen_ai.usage.input_tokens:    120
gen_ai.usage.output_tokens:   85

# Deviation surfaces as the gen_ai.evaluation.result OTel Event (Development)
# (NOT a span attribute):
event.name:                   "gen_ai.evaluation.result"
gen_ai.evaluation.name:       "baseline_deviation"
gen_ai.evaluation.score.value: 0.87
gen_ai.evaluation.score.label: "high"
gen_ai.evaluation.explanation: "Tool 'send_http' deviates from baseline …"

# ── Proposed (this project; see Docs/gen-ai-attributes.md §11) ────────────
gen_ai.agent.task_type:                "investigate_latency"
gen_ai.agent.tool.category:            "network"
gen_ai.agent.tool.confidence:          0.87
gen_ai.agent.tool.sequence_position:   1
gen_ai.agent.tool.risk_severity:       4
gen_ai.agent.baseline.id:              "sre.investigate_latency.v1"
gen_ai.agent.baseline.expected_tool:   "QueryMetrics"
gen_ai.agent.baseline.match:           false
gen_ai.agent.scope.requested:          ["network:external"]
gen_ai.agent.scope.granted:            []
gen_ai.agent.scope.denied:             ["network:external"]
gen_ai.agent.gate.blocked:             true
gen_ai.agent.gate.reason:              "deviation=0.87, scope creep"
gen_ai.agent.computed.severity:        "critical"     # SDK-set or collector-derived
gen_ai.agent.computed.risk_score:      0.78

                                n      severity   danger%   matched%
open-source (5 fam, 19 mdl):  5,700    0.567       23.3%      15.2%
frontier    (3 ven,  9 mdl):  2,700    0.379        6.7%      35.5%
                              ──────────────────────────────────────
Welch t = +26.21,  p = 2.7×10⁻¹⁴²  →  closed-source significantly safer

examples/sre-incident-demo/configs/
├── sre_tools.yaml          # 10 MCP servers · 73 tools · 19 risk-4 destructive
├── sre_dataset.yaml        # 6 incident categories × 5 ambiguity variants
├── sre_models.yaml         # 40-entry model registry (capability, tier, family)
├── llm_connection_conf.yaml # Provider routing for --engine live
├── otel-collector-decision.yaml, prometheus.yaml
└── grafana/                # Dashboards + datasource provisioning

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))

from sre_app_config import get_tools, get_dag_tasks, get_model_capability

tools = get_tools()                              # 73 tools (server.tool flat keys)
dag = get_dag_tasks()                            # 6 incident categories
cap = get_model_capability("qwen3.5:27b")        # 27.0 from sre_models.yaml

tracing-agents-mind-demo/
├── src/gen_ai_otel/                 # Library (pip installable)
│   ├── attributes.py                # OTel-spec (Development) + Proposed constants
│   ├── instrumentation.py           # OTel tracing + observation bridge
│   ├── llm_client.py, providers/    # LLMClient + Ollama/OpenAI/Anthropic/Google
│   ├── ollama_agent.py              # Decision agent
│   ├── neo4j_exporter.py            # Direct Neo4j exporter
│   ├── config.py                    # Generic types + readers (no domain data)
│   ├── cli/                         # gen-ai-demo console entry point
│   └── observation/                 # UnifiedAnalyzer, PreExecutionGate, adapters/
├── examples/
│   ├── sre-incident-demo/           # Primary SRE benchmark — 8,400 chains × 28 models
│   │   ├── compose.yaml             # SRE observability stack (bind-mount data/)
│   │   ├── configs/                 # OTel Collector, Prometheus, Grafana provisioning
│   │   ├── bridge/                  # OTLP gRPC → Neo4j writer (Dockerfile + server.py)
│   │   ├── schemas/, queries/       # Neo4j decision-graph schema + forensic Cypher
│   │   ├── presentation/            # Keynote PPTX/HTML/MP4 + builder scripts
│   │   ├── videos/                  # Demo recordings (.cast, .mp4, .gif)
│   │   ├── scripts/                 # SRE-specific setup (data dir bootstrap)
│   │   ├── analysis/                # baseline_simulation, dag_auto_builder,
│   │   │                            #  dag_formula_comparison, dag_benchmark,
│   │   │                            #  model_comparison, task_shape_classifier{,_v2}
│   │   ├── results/                 # Historical benchmark output (read-only)
│   │   ├── src/                     # Example source code
│   │   │   ├── demo_sre_inverse_scaling.py # Primary entry
│   │   │   ├── analyze_inverse_scaling.py  # Master ranking
│   │   │   ├── lind_mehlum_test.py         # Formal U-shape test
│   │   │   ├── sre_adapter.py              # Catalog loader + classifier
│   │   │   └── llm_router.py               # Multi-provider dispatcher
│   │   └── configs/                 # (also includes sre_tools.yaml,
│   │                                #  sre_dataset.yaml, llm_connection_conf.yaml)
│   ├── legacy-interactive-demo/     # Mock / Ollama / OpenAI walk-through
│   │   ├── run_demo.py, benchmark_10k.py
│   │   ├── scripts/                 # demo-video.sh, record-demo.sh, setup.sh
│   │   └── Dockerfile               # Optional sandboxed run (build context = repo root)
│   ├── workato-demo/                # External workflow platform adapter demo
│   └── neo4j-demo/                  # Standalone Neo4j seed + Cypher walkthrough
├── Docs/
│   ├── gen-ai-attributes.md         # Wire-format contract (Development + Proposed)
│   ├── architecture.md              # Architecture document
│   └── repository-organization.md   # Repo organization plan
├── papers/                          # Source papers (Betley, McKenzie, Wei, Zhou)
├── KubeConEU2026-Presentation/      # KubeCon EU 2026 reference photos (separate event)
├── backup/                          # Historical / archived outputs (gitignored)
├── pyproject.toml                   # Package with optional extras
└── README.md                        # this file

./examples/sre-incident-demo/scripts/setup.sh

WARN[0000] Error validating CNI config file /home/user/.config/cni/net.d/tracing-agents-mind_decision-net.conflist: 
[plugin firewall does not support config version "1.0.0"]

# Stop and remove all containers
podman rm -af && podman pod rm -af
podman network rm -f tracing-agents-mind_decision-net 2>/dev/null || true

# Create CNI config with correct version
cat > ~/.config/cni/net.d/tracing-agents-mind_decision-net.conflist << 'EOF'
{
  "cniVersion": "0.4.0",
  "name": "tracing-agents-mind_decision-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman1",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [{"dst": "0.0.0.0/0"}],
        "ranges": [[{"subnet": "10.89.0.0/24", "gateway": "10.89.0.1"}]]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    },
    {
      "type": "firewall"
    },
    {
      "type": "tuning"
    },
    {
      "type": "dnsname",
      "domainName": "dns.podman",
      "capabilities": {"aliases": true}
    }
  ]
}
EOF

# Restart the stack
cd examples/sre-incident-demo
podman-compose up -d

podman unshare rm -rf examples/sre-incident-demo/data

curl http://localhost:11434/api/tags
ollama pull qwen2.5-coder:7b

curl http://localhost:7474    # Default: neo4j / DecisionGraph2026!