可观测性MCP服务器

otel-mcp-server

An MCP server that exposes your OpenTelemetry observability stack — traces, metrics, logs, and more — as tools for AI agents. Built on a Skill plugin architecture for easy extensibility.

Give any LLM agent the ability to query your Jaeger traces, run PromQL, search Loki logs, and investigate production issues — through a standard protocol.

┌─────────────────┐      MCP     ┌──────────────────┐──► Traces (Jaeger · Zipkin · Tempo · SkyWalking)
│  Claude Desktop │ ◄──────────► │                  │──► Prometheus · InfluxDB · OpenTSDB
│  GitHub Copilot │ (stdio/HTTP) │  otel-mcp-server │──► Loki · ClickHouse · Graylog (logs)
│  Custom Agent   │              │                  │──► Pinpoint · Elasticsearch · Alertmanager
└─────────────────┘              │   26 skills      │──► Grafana · Pyroscope · OPA
                                 │   114 tools      │──► Cilium · Kubernetes (eBPF/CRDs)
                                 │   authenticated  │──► Envoy · Consul · Kong · Traefik
                                 └──────────────────┘──► Fluent Bit · Beats · Vector · Alloy
                                                     └─► App API    (ZK/system)

Example

"What's running, what's healthy, and what needs attention?" — answered in seconds by an AI agent using this MCP server against a local Docker Compose stack:

"Tell me what happened to order ORD-1774382223417-7" — full distributed trace across 4 services, 40 spans, with ZK proof verification:

"What about the k8s cluster?"

Features

• 114 tools across 26 skills (+6 opt-in write tools) — a provider-agnostic traces layer (Jaeger/Zipkin/Tempo/SkyWalking via TRACES_PROVIDER), metrics (Prometheus/InfluxDB/OpenTSDB), logs (Loki/ClickHouse/Graylog), Pinpoint, Elasticsearch, Alertmanager, vmalert rule evaluation, Grafana, Cilium, Grafana Beyla (eBPF auto-instrumentation), Kubernetes, Pyroscope, OPA, service mesh (Envoy/Consul/Kong/Traefik), collection pipelines (Fluent Bit/Beats/Vector/Alloy), AgentRelay agent coordination, ZK proofs, system health, public exchange transparency
• Skill plugin architecture — each backend is a self-contained plugin; add new ones with a single file
• Two transports — stdio (Claude Desktop, Copilot) and HTTP (remote, multi-client)
• MCP 2026-07-28 ready — serves the stateless protocol core (no handshake, no session id) alongside ≤2025-11-25 clients on the same endpoint, validates the Mcp-Method/Mcp-Name routing headers, emits ttlMs/cacheScope cache hints, answers server/discover, and propagates W3C trace context from _meta into backend queries (see MCP 2026-07-28 support)
• Layered auth — backend credentials (Bearer/Basic/OAuth client-credentials/custom headers per backend) and client identity via static API keys, Just-in-Time scoped ephemeral tokens (OWASP MCP Top 10), or enterprise-managed authorization through your corporate IdP (MCP ext-auth ID-JAG). allowedTools scopes are enforced per session (see Client Authentication)
• Selective skills — enable only the skills you need (--tools traces,metrics,logs)
• Multi-version aware — a typed capability → product → protocol-adapter model tracks which versions and protocol features each backend supports; runtime detection surfaces live product/version on /health, and MCP_VERSION_GATING (off/warn/enforce) can guard version-sensitive features (unknown versions always pass optimistically)
• Multi-backend & failover — a single skill can address multiple named instances and fail over across replicas; tools accept an optional SSRF-safe target argument (see Multi-backend instances & failover)
• Self-metrics — GET /metrics endpoint with tool call counts, backend latencies, auth attempts, and JIT token issuance/rotation/revocation/denial counters
• Container-native — env-var config, K8s Secret mounting, multi-stage Dockerfile
• Zero dependencies beyond the MCP SDK and Zod

For role-based Studio workflows, see docs/studio-user-journeys.md.

Quick Start

New here? docs/getting-started.md walks the whole flow end to end — install, point at your backends, wire up Claude Desktop, and the first questions to ask — in about five minutes. The rest of this section is the condensed reference.

Install

Run directly with npx (no clone or build needed):

npx -y @moebiusx/otel-mcp-server

Or install globally:

npm install -g @moebiusx/otel-mcp-server
otel-mcp-server

git clone https://github.com/MoebiusX/otel-mcp-server.git
cd otel-mcp-server
npm install
npm run build

Run (stdio — for Claude Desktop / Copilot)

# Point at your backends
export JAEGER_URL=http://localhost:16686
export PROMETHEUS_URL=http://localhost:9090
export LOKI_URL=http://localhost:3100

node dist/index.js

Run (HTTP — for remote agents / containers)

node dist/index.js --http 3001
# ✓ otel-mcp-server v1.8.0 listening on http://0.0.0.0:3001
#   Health:  http://localhost:3001/health
#   Metrics: http://localhost:3001/metrics
#   Skills:  6/26 configured — 29 tools registered
#     ✓ traces         — Distributed Traces (5 tools) [Jaeger]
#     ✓ metrics        — Prometheus Metrics (6 tools) [Prometheus]
#     ✓ logs           — Structured Logs (4 tools) [Loki]
#     ✗ elasticsearch  — not configured
#     …
#     ✓ zk-proofs      — ZK Proofs (4 tools) [App API]
#     ✓ system         — System Health (5 tools) [App API, Jaeger]
#     ✓ public-exchange — Public Exchange (5 tools) [App API]

The same inventory is printed on stdio (to stderr, where MCP clients log it), so a missing backend is visible in either transport.

Docker

Published multi-arch images (amd64/arm64) — the entrypoint defaults to --http 3001:

docker run --rm -p 3001:3001 \
  -e PROMETHEUS_URL=http://host.docker.internal:9090 \
  moebiusx/otel-mcp-server:latest

localhost inside the container is the container — use host.docker.internal (Docker Desktop) or a compose/K8s service name to reach backends on the host or in the cluster.

docker build -t otel-mcp-server .
docker run -p 3001:3001 \
  -e JAEGER_URL=http://jaeger:16686 \
  -e PROMETHEUS_URL=http://prometheus:9090 \
  -e LOKI_URL=http://loki:3100 \
  -e ELASTICSEARCH_URL=http://elasticsearch:9200 \
  -e ALERTMANAGER_URL=http://alertmanager:9093 \
  -e GRAFANA_URL=http://grafana:3000 \
  -e GRAFANA_AUTH_TOKEN=glsa_xxx \
  -e MCP_AUTH_KEYS='{"keys":[{"id":"agent-1","key":"sk-my-secret-key"}]}' \
  otel-mcp-server

Configuration

All configuration is via environment variables. The commonly used backend, auth, and runtime variables are listed below.

Backend URLs

Backend Authentication

The MCP server authenticates to each backend independently. For each backend prefix (JAEGER_, TEMPO_, ZIPKIN_, SKYWALKING_, PROMETHEUS_, LOKI_, APP_API_, ELASTICSEARCH_, ALERTMANAGER_, GRAFANA_, CILIUM_, CLICKHOUSE_, PYROSCOPE_, OPA_, ENVOY_, CONSUL_, KONG_, TRAEFIK_, INFLUX_, OPENTSDB_, GRAYLOG_, PINPOINT_, FLUENTBIT_, BEATS_, VECTOR_, ALLOY_, AGENTRELAY_, VMALERT_), you can set:

Special:

Kubernetes uses its own credential scheme rather than the prefix above: it presents a ServiceAccount bearer token (auto-loaded from the in-cluster mount, or KUBERNETES_TOKEN / KUBERNETES_TOKEN_FILE) and validates TLS against the cluster CA (KUBERNETES_CA_FILE, or the in-cluster mount). See .env.example.

Example — Prometheus behind OAuth proxy + multi-tenant Loki:

PROMETHEUS_AUTH_TOKEN=eyJhbGci...
LOKI_AUTH_TOKEN=my-loki-token
LOKI_TENANT_ID=team-platform

OAuth 2.0 / OIDC (client-credentials)

When no static _AUTH_* var is set for a backend, the server can obtain a bearer token via the OAuth 2.0 client-credentials grant and refresh it transparently (cached in-memory, refreshed ~60s before expiry, concurrent requests de-duped). Client secrets are never logged or echoed in error messages. Use the same <PREFIX> as above with _AUTH_OAUTH_* suffixes:

# Generic OIDC (token endpoint auto-discovered from the issuer)
PROMETHEUS_AUTH_OAUTH_ISSUER=https://idp.example.com/realms/obs
PROMETHEUS_AUTH_OAUTH_CLIENT_ID=otel-mcp
PROMETHEUS_AUTH_OAUTH_CLIENT_SECRET=...
PROMETHEUS_AUTH_OAUTH_SCOPE=metrics:read

# Microsoft Entra ID (Azure AD) preset
TEMPO_AUTH_OAUTH_PROVIDER=entra
TEMPO_AUTH_OAUTH_TENANT=00000000-0000-0000-0000-000000000000
TEMPO_AUTH_OAUTH_CLIENT_ID=...
TEMPO_AUTH_OAUTH_CLIENT_SECRET=...
TEMPO_AUTH_OAUTH_AUDIENCE=api://obs-backend       # → scope api://obs-backend/.default

Static _AUTH_TOKEN / _AUTH_BASIC / _AUTH_HEADER always take precedence, so existing configs are unaffected.

Multi-backend instances & failover

A single skill can talk to multiple named backends and fail over across replicas. The single-URL config above keeps working unchanged — it simply becomes the default instance. Supported skills today: metrics (Prometheus), logs (Loki), and elasticsearch.

Named instances — add a __<NAME> suffix to the base URL var. Auth for a named instance uses the <PREFIX>__<NAME>_ prefix:

PROMETHEUS_URL=http://prom:9090                 # instance "default"
PROMETHEUS_URL__PROD=http://prom-prod:9090       # instance "PROD"
PROMETHEUS__PROD_AUTH_TOKEN=eyJhbGci...          # auth for "PROD"

Failover — any URL value may be a comma-separated list or JSON array. URLs are tried in order; the server fails over only on infrastructure errors (5xx / timeout / network) and never on a 4xx:

PROMETHEUS_URL=http://prom-a:9090,http://prom-b:9090

Rich form — MCP_BACKENDS (a JSON array) gives full control, including an explicit product (skips version auto-probe) and per-instance headers. It takes precedence over the env-var forms:

MCP_BACKENDS='[{"skill":"metrics","instance":"PROD",
  "urls":["http://mimir-a","http://mimir-b"],
  "authPrefix":"MIMIR_PROD","product":"Grafana Mimir",
  "extraHeaders":{"X-Scope-OrgID":"team-a"}}]'

Selecting a backend — tools on multi-backend skills accept an optional target argument naming the instance (e.g. "PROD"). Omit it to use the primary. target is validated against the configured instance names only, so a caller can never coerce the server into fetching an arbitrary URL (no SSRF).

Client Authentication (HTTP mode)

Clients connecting to the MCP server over HTTP must present an API key. Keys are loaded from (first match wins):

1. MCP_AUTH_KEYS env var — JSON string (best for containers / K8s Secrets)
2. MCP_AUTH_KEYS_FILE env var — path to a JSON file (K8s mounted Secret)
3. ./auth-keys.json — local file in cwd
4. ~/.otel-mcp/auth-keys.json — user home directory

If no keys are found, the server runs with open access (a warning is logged).

Key format:

{
  "keys": [
    {
      "id": "agent-1",
      "key": "sk-my-secret-key-here",
      "description": "Production RCA agent"
    },
    {
      "id": "ci-readonly",
      "key": "sk-ci-key",
      "description": "CI pipeline — restricted tools",
      "allowedTools": ["traces", "metrics"]
    }
  ]
}

Clients authenticate via either header:

• Authorization: Bearer sk-my-secret-key-here
• X-API-Key: sk-my-secret-key-here

The /health endpoint is always unauthenticated.

allowedTools is enforced at session creation: the MCP session only registers the listed skills, so out-of-scope tools do not exist for that session.

Just-in-Time (JIT) Privileged Identity

A static API key is standing privilege — whoever holds it holds it until someone rotates it. JIT identity treats the AI agent like a human employee: the static key becomes a role definition used only to sign in, and real access happens through short-lived, scoped, rotatable session tokens. Maps to the OWASP MCP Top 10 mitigations for MCP01 (token mismanagement), MCP02 (scope creep), MCP07 (authn/z), and MCP08 (audit & telemetry).

Enable with MCP_JIT_MODE:

Token lifecycle (the "badge" flow):

# 1. Exchange the static key for a scoped, ephemeral token (default TTL 15 min)
curl -X POST http://localhost:3001/auth/token \
  -H "Authorization: Bearer sk-my-static-key" \
  -d '{"scopes": ["traces", "metrics"], "ttlSeconds": 600}'
# → 201 { "token": "mcpj_<id>.<secret>", "token_id": "<id>", "scopes": [...],
#         "expires_at": "...", "not_after": "...", "generation": 0 }

# 2. Call MCP with the ephemeral token instead of the key
curl -X POST http://localhost:3001/mcp \
  -H "Authorization: Bearer mcpj_..." ...

# 3. Rotate before expiry — same scopes, same session, next generation.
#    The old token stays valid for a 30 s grace window.
curl -X POST http://localhost:3001/auth/token/refresh \
  -H "Authorization: Bearer mcpj_..."

# 4. Hand the badge back (self), or kill-switch by id with a static key
curl -X POST http://localhost:3001/auth/token/revoke -H "Authorization: Bearer mcpj_..."
curl -X POST http://localhost:3001/auth/token/revoke \
  -H "Authorization: Bearer sk-admin-key" -d '{"token_id": "<id>"}'

Guarantees:

• Least privilege — requested scopes must be a subset of the parent key's allowedTools (escalation → 403 scope_violation); omitting scopes inherits the parent's full grant. Tokens cannot mint further tokens, so every lineage is one auditable hop from a named key.
• Ephemeral — per-token TTL is clamped to 60 s–1 h; rotation never extends a lineage past MCP_JIT_MAX_LIFETIME_SECONDS (default 8 h — end of shift), after which the agent must re-authenticate with its static key.
• Session binding — an MCP session can only be reused by the credential lineage that created it (rotation preserves it; any other key or token gets 403). Tokens are held server-side as SHA-256 hashes only, and a restart invalidates everything outstanding.
• Auditable — every issuance, rotation, revocation, and denial is logged with the public token id (never the secret) and exported on /metrics: mcp_jit_tokens_issued_total{parent_key}, mcp_jit_rotations_total, mcp_jit_revocations_total{source}, mcp_jit_denials_total{reason}, mcp_jit_active_tokens.

Configuration:

Revocation has three forms: a token holder revokes itself by presenting the token; an operator revokes one generation with token_id; or an operator kills an entire rotation lineage with root_id (useful when a credential leaked and the current generation's id is unknown). A scope-restricted static key may only revoke tokens it minted — including after the lineage has rotated, where ownership is resolved from the lineage's live members rather than the (by then pruned) first-generation record. Unrestricted keys revoke anything.

Renewal re-checks the parent key, so access review cascades: remove a key and its live lineages can no longer refresh (they die at their current TTL); narrow a key's allowedTools and lineages holding wider scopes stop renewing.

JIT applies to the HTTP transport only — stdio is a local, single-user channel with no network credential to steal.

High availability (multiple replicas)

By default (MCP_JIT_STORE=memory) token state is in-process. That is the right failure mode for a single instance — a restart invalidates outstanding tokens and clients re-mint — but behind a load balancer it means a token minted on one replica is unknown on another, revocation only takes effect where it was processed, and an ID-JAG assertion could be redeemed once per replica.

For multi-replica deployments, point MCP_JIT_STORE at a module that exports createJitStore() returning { tokens, denylist } implementing the JitTokenStore and BoundedDenylist interfaces in src/jit-store.ts. Adapters are operator-supplied so no network client (Redis, a database driver) ever enters this package's dependencies. The interfaces are deliberately built from atomic compound primitives rather than get/set — insert performs the capacity check and the insert as one operation, rotate is a compare-and-set, and the denylist's addIfAbsent is a single check-and-record — so implement them with your backend's transactional facility (MULTI, a Lua script, a conditional write). A get/set implementation would reintroduce cross-replica races on the capacity cap, once-only rotation, and assertion single-use.

MCP sessions are a separate matter: they hold live transport objects and cannot be shared. Use sticky routing on Mcp-Session-Id, or the stateless transport mode, so any replica can serve any request. The principal binding is recomputed per request and survives either choice.

Enterprise-Managed Authorization (MCP extension)

Implements the server side of the MCP Enterprise-Managed Authorization extension (io.modelcontextprotocol/enterprise-managed-authorization): your identity provider (Okta, Entra ID, corporate SSO) decides who may use this MCP server, instead of locally provisioned API keys. The MCP client logs the employee in via SSO, obtains an ID-JAG (Identity Assertion JWT Authorization Grant) from the IdP, and exchanges it at this server's token endpoint for a scoped JIT session token — enterprise identities ride the same least-privilege, rotation, and audit rails as key-minted tokens. Onboarding, offboarding, and access policy live in the IdP admin console.

MCP_ENTERPRISE_AUTH_ISSUER=https://idp.example.com      # trusted IdP (required)
MCP_ENTERPRISE_AUTH_AUDIENCE=https://mcp.example.com    # this server's issuer id (required)
# optional:
MCP_ENTERPRISE_AUTH_RESOURCE=https://mcp.example.com    # resource id (default: audience)
MCP_ENTERPRISE_AUTH_JWKS_URL=https://idp.example.com/jwks  # default: OIDC discovery
MCP_ENTERPRISE_AUTH_DEFAULT_SCOPES=traces,metrics       # when the ID-JAG has no scope claim
MCP_ENTERPRISE_AUTH_MAX_REDEEMED_JTIS=50000             # replay-cache cap (fail-closed)

Configuring it auto-enables the JIT token infrastructure (and counts as client auth being "on" — an enterprise-only deployment is never open), and:

• Discovery — serves RFC 8414 metadata at /.well-known/oauth-authorization-server advertising urn:ietf:params:oauth:grant-profile:id-jag in authorization_grant_profiles_supported, plus RFC 9728 /.well-known/oauth-protected-resource; MCP 401s carry a WWW-Authenticate: Bearer resource_metadata="…" challenge.
• Token exchange — POST /auth/token with grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer and the ID-JAG as assertion (form-encoded or JSON) answers with an RFC 6749 token response whose access_token is a JIT session token scoped to the assertion's scope claim (intersected with enabled skills).
• Validation per the ID-JAG profile — typ: oauth-id-jag+jwt; RS256/384/512, ES256/384, or EdDSA (Ed25519) signature against the IdP JWKS (cached, kid-aware, stale-cache fallback); iss/aud/resource binding; exp/iat/nbf with 60 s skew; single-use jti (replay → invalid_grant; the replay cache is bounded by MCP_ENTERPRISE_AUTH_MAX_REDEEMED_JTIS, shared across replicas when MCP_JIT_STORE is configured, and surfaced as the mcp_jit_idjag_replay_cache_size gauge); client_id claim/request binding. Errors are RFC 6749 §5.2 token errors, counted in mcp_jit_denials_total{reason="idjag_*"}. If token issuance fails after the assertion verified, the jti is un-redeemed so the client can retry the same assertion rather than round-tripping the IdP.
• Account linking — the assertion sub is the audit identity (logged on issuance); tokens mint under parent_key="enterprise-idp" in metrics.

# Client-side exchange (the MCP client does this automatically)
curl -X POST https://mcp.example.com/auth/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer \
  -d assertion="$ID_JAG" -d client_id=my-mcp-client
# → { "access_token": "mcpj_…", "token_type": "Bearer", "expires_in": 900, "scope": "metrics traces" }

MCP 2026-07-28 support

The 2026-07-28 revision made MCP stateless: the initialize handshake and Mcp-Session-Id are gone, so any request can land on any server instance. This server implements that revision and keeps serving ≤2025-11-25 clients on the same endpoint — the revision is resolved per request, from the MCP-Protocol-Version header or (when absent) from the presence of the 2026 routing headers or _meta client info.

Trace propagation is the piece worth calling out: a tool call and the backend queries it triggers used to be unrelated spans. With a traceparent in the call's _meta, they now join up in any OpenTelemetry-compatible backend. If a proxy stamps HTTP trace headers instead, those are used as a fallback; _meta wins when both are present, and malformed values are dropped rather than forwarded.

GET /health reports the supported revisions, and /metrics carries mcp_spec_requests_total{version,mode}, mcp_trace_context_propagated_total{source}, and mcp_routing_header_rejections_total{header}.

On the SDK: @modelcontextprotocol/sdk v1 tops out at protocol 2025-11-25 and rejects a newer version header outright, so this support is implemented in this server's HTTP layer above the SDK: the body is buffered once (which is what makes the header/body cross-check possible), the version header is translated to one the SDK accepts, and 2026 requests are served from a fresh session-less transport per request. When an SDK with native support ships, that shim collapses; the behaviour clients see does not change.

Kubernetes Deployment

apiVersion: v1
kind: Secret
metadata:
  name: otel-mcp-auth
stringData:
  # Client keys
  auth-keys.json: |
    {"keys":[{"id":"rca-agent","key":"sk-prod-xxx"}]}
  # Backend tokens
  PROMETHEUS_AUTH_TOKEN: "my-prom-token"
  LOKI_AUTH_TOKEN: "my-loki-token"
  LOKI_TENANT_ID: "platform"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-mcp-server
spec:
  # Safe to scale past 1 once every client speaks MCP 2026-07-28 (stateless —
  # any replica serves any request). While pre-2026 clients remain, either keep
  # one replica or enable sticky routing on Mcp-Session-Id at the ingress; and
  # if JIT identity is enabled, set MCP_JIT_STORE to a shared adapter so tokens
  # and revocations are visible across replicas. See "High availability".
  replicas: 1
  template:
    spec:
      containers:
        - name: otel-mcp-server
          image: moebiusx/otel-mcp-server:latest
          ports:
            - containerPort: 3001
          env:
            - name: JAEGER_URL
              value: "http://jaeger-query.observability:16686"
            - name: PROMETHEUS_URL
              value: "http://prometheus.observability:9090"
            - name: LOKI_URL
              value: "http://loki.observability:3100"
            # Optional: uncomment to enable Elasticsearch / Alertmanager / Grafana skills
            # - name: ELASTICSEARCH_URL
            #   value: "http://elasticsearch.observability:9200"
            # - name: ALERTMANAGER_URL
            #   value: "http://alertmanager.observability:9093"
            # - name: GRAFANA_URL
            #   value: "http://grafana.observability:3000"
            # - name: GRAFANA_AUTH_TOKEN
            #   valueFrom:
            #     secretKeyRef:
            #       name: otel-mcp-auth
            #       key: GRAFANA_AUTH_TOKEN
            - name: PROMETHEUS_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: PROMETHEUS_AUTH_TOKEN
            - name: LOKI_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: LOKI_AUTH_TOKEN
            - name: LOKI_TENANT_ID
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: LOKI_TENANT_ID
            - name: MCP_AUTH_KEYS_FILE
              value: "/etc/otel-mcp/auth-keys.json"
          volumeMounts:
            - name: auth-keys
              mountPath: /etc/otel-mcp
              readOnly: true
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3001
            initialDelaySeconds: 5
          readinessProbe:
            httpGet:
              path: /health
              port: 3001
      volumes:
        - name: auth-keys
          secret:
            secretName: otel-mcp-auth
            items:
              - key: auth-keys.json
                path: auth-keys.json

Skills

Each telemetry backend is a skill — an independent plugin. The core OTel skills (traces, metrics, logs) and the app skills (system, zk-proofs, public-exchange) are always active, defaulting to localhost backends so the server works out of the box. Every other skill is opt-in: it activates only when its backend URL is set (e.g. ELASTICSEARCH_URL, CILIUM_URL, CLICKHOUSE_URL), and is silently skipped otherwise. Use --tools to restrict which skills load regardless of configuration.

Traces — traces — 5 tools

Provider-agnostic. Select backend with TRACES_PROVIDER (jaeger [default], tempo, zipkin, skywalking). The verb surface is stable across providers; capabilities the chosen backend doesn't support (e.g. traces_dependencies on Tempo) return a clear error.

Metrics (Prometheus) — metrics — 6 tools

Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Logs (Loki) — logs — 4 tools

Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Elasticsearch / OpenSearch — elasticsearch — 5 tools

Enabled when ELASTICSEARCH_URL is set. Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Alertmanager — alertmanager — 4 tools

Enabled when ALERTMANAGER_URL is set.

vmalert — vmalert — 4 tools

Enabled when VMALERT_URL is set (e.g. http://localhost:8880). vmalert is the rule-evaluation component in a VictoriaMetrics stack — VM single-node stores series but does not evaluate rules, so use this skill to query rules and active alerts from vmalert directly.

Grafana — grafana — 10 tools

Enabled when GRAFANA_URL is set. The 10 tools below are read-only and intended for verification/interrogation workflows. Three additional write tools are available when MCP_ENABLE_WRITES is set — see Write tools.

Write tools (opt-in)

Disabled by default. The server is read-only out of the box. Set MCP_ENABLE_WRITES=true (also accepts 1/yes/on) to advertise and enable the mutating tools below. The Grafana token must also carry the matching write scopes.

Write modes — each write tool takes an explicit mode so the caller controls overwrite behavior; the default is the safe one:

• create (default) — strict insert: fails with a clear conflict error (including the existing object's UID and version) if the target UID already exists. Use this for promotion workflows (e.g. staging → prod) where silently overwriting is dangerous.
• upsert — idempotent create-or-update by UID, for reconcile / infra-as-code loops.
• update (dashboards and alert rules) — strict update: fails if the target UID does not already exist.

Alert rules are Grafana-managed (the JSON provisioning API, no YAML dependency). A rule whose body includes a record object is treated as a recording rule; otherwise it is an alerting rule. Provisioning writes are sent with X-Disable-Provenance: true so the rules stay editable in the Grafana UI. (Mimir/Cortex ruler rules are YAML-based and remain a future follow-up.)

All write tools accept dry_run: true to validate and report the planned action without writing. Conflict detection for strict create/update uses a GET pre-check, and grafana_create_dashboard also sends Grafana's native overwrite=false as a second safety net. Returns the resulting UID and version on success. Existing read-only behavior is unchanged when writes are disabled.

Cilium (eBPF networking) — cilium — 6 tools

Enabled when CILIUM_URL is set. Targets the cilium-agent REST API. This is the agent control-plane surface; L3/L7 flow observability (Hubble) is gRPC and not yet wired.

Grafana Beyla (eBPF auto-instrumentation) — beyla — 4 tools

Enabled when BEYLA_PROMETHEUS_URL is set. Beyla has no query API of its own — it auto-instruments processes with eBPF (zero code changes) and exports OTel/Prometheus metrics, so point this at the Prometheus-compatible store that scrapes Beyla. The tools issue Beyla-aware PromQL so you don't need to know its metric/label schema.

Kubernetes (CRD reader) — kubernetes — 5 tools

Auto-enabled in-cluster (ServiceAccount mount), or set KUBERNETES_URL + token out-of-cluster. Read-only (GET only). The generic k8s_list/k8s_get work for any built-in resource or CRD, so the whole control-plane tier (Argo Rollouts, Flagger, Kyverno, Gatekeeper, KEDA, Chaos Mesh, Cilium policies, Inspektor Gadget, …) is queryable without a bespoke skill per product.

ClickHouse Logs — clickhouse — 5 tools

Enabled when CLICKHOUSE_URL is set. Uses ClickHouse's HTTP GET query path, which the engine forces to be read-only — writes are rejected by ClickHouse itself.

Pyroscope (continuous profiling) — pyroscope — 4 tools

Enabled when PYROSCOPE_URL is set. Works against OSS Pyroscope and Grafana Pyroscope.

Open Policy Agent — opa — 4 tools

Enabled when OPA_URL is set. Read-only (GET against the Data/Query APIs).

Envoy — envoy — 4 tools

Enabled when ENVOY_ADMIN_URL is set. Works for standalone Envoy and mesh sidecar proxies.

Consul — consul — 5 tools

Enabled when CONSUL_URL is set. Set CONSUL_AUTH_TOKEN for an ACL token.

Kong Gateway — kong — 4 tools

Enabled when KONG_ADMIN_URL is set.

Traefik — traefik — 4 tools

Enabled when TRAEFIK_URL is set.

InfluxDB — influx — 3 tools

Enabled when INFLUX_URL is set. Uses the InfluxQL /query endpoint (1.x and 2.x compatible).

OpenTSDB — opentsdb — 3 tools

Enabled when OPENTSDB_URL is set.

Graylog — graylog — 3 tools

Enabled when GRAYLOG_URL is set.

Grafana Tempo, Apache SkyWalking — see traces

Tempo and SkyWalking are now exposed through the provider-agnostic traces skill — set TRACES_PROVIDER=tempo or TRACES_PROVIDER=skywalking and point the matching URL var at your backend.

Pinpoint — pinpoint — 3 tools

Enabled when PINPOINT_URL is set. The API varies by Pinpoint version, so a read-only GET passthrough is provided for version-specific endpoints.

Collection Pipelines — pipeline — 4 tools

Enabled when any of FLUENTBIT_URL / BEATS_URL / VECTOR_URL / ALLOY_URL is set. Each tool errors clearly if its agent isn't configured.

ZK Proofs — zk-proofs — 4 tools

System — system — 5 tools

AgentRelay — agentrelay — 1 tool (+1 write tool via MCP_ENABLE_WRITES)

Enabled when AGENTRELAY_URL is set (e.g. https://api.agentrelay.tech). Set AGENTRELAY_AUTH_TOKEN for a bearer token, or AGENTRELAY_AUTH_HEADER to send a raw header value (e.g. an X-API-Key scheme).

Coordinate with other agents through the AgentRelay hosted REST API ("headless Slack for agents"). Read-only by default; the send tool is opt-in behind MCP_ENABLE_WRITES.

Public Exchange — public-exchange — 5 tools

Always available (only needs APP_API_URL, which defaults to http://localhost:5000). Read-only tools mirroring KrystalineX's /api/public/* transparency endpoints — designed for an unauthenticated public MCP deployment, since every endpoint already serves data that is public on the transparency website.

Selective Skills

Only load the skills you need:

# Core OTEL only (no ZK / system health)
node dist/index.js --tools traces,metrics,logs

# Traces + metrics + alertmanager
node dist/index.js --http 3001 --tools traces,metrics,alertmanager

Self-Metrics

In HTTP mode, GET /metrics exposes Prometheus-format metrics about the MCP server itself:

Protocol and request shape — use these to see which MCP revision clients speak, and whether callers are propagating trace context:

JIT privileged identity (see above) — only populated when MCP_JIT_MODE is set or enterprise auth is configured:

Scrape with Prometheus:

scrape_configs:
  - job_name: 'otel-mcp-server'
    static_configs:
      - targets: ['otel-mcp-server:3001']

Client Integration

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "otel": {
      "command": "npx",
      "args": ["-y", "@moebiusx/otel-mcp-server"],
      "env": {
        "JAEGER_URL": "http://localhost:16686",
        "PROMETHEUS_URL": "http://localhost:9090",
        "LOKI_URL": "http://localhost:3100"
      }
    }
  }
}

VS Code / GitHub Copilot

Add to .vscode/mcp.json:

{
  "servers": {
    "otel": {
      "command": "npx",
      "args": ["-y", "@moebiusx/otel-mcp-server"],
      "env": {
        "JAEGER_URL": "http://localhost:16686",
        "PROMETHEUS_URL": "http://localhost:9090",
        "LOKI_URL": "http://localhost:3100"
      }
    }
  }
}

HTTP Client (any agent)

# Health check — skills, backend versions, supported MCP revisions
curl -s http://localhost:3001/health

# List tools. MCP 2026-07-28 is stateless, so this is a single POST:
# no initialize handshake, no session id.
curl -s -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# Add the credential when MCP_AUTH_KEYS is set
#   -H "Authorization: Bearer sk-my-key"

Older clients that send initialize and carry an Mcp-Session-Id are served on the same endpoint unchanged — see MCP 2026-07-28 support. Omitting MCP-Protocol-Version puts the request on the pre-2026 path, where a bare tools/list is rejected with Server not initialized until the handshake completes.

Architecture

Each telemetry backend is a Skill — a self-contained plugin that declares its tools, self-configures from env vars, and registers MCP tools on the server.

src/
├── index.ts              # CLI entry point (stdio / HTTP transport)
├── server.ts             # MCP server factory (iterates skills)
├── skill.ts              # Skill interface + SkillHelpers factory
├── skills.ts             # Skill registry wrapper (+ version metadata)
├── skills.generated.ts   # GENERATED by `npm run gen:skills` — do not edit
├── config.ts             # env() helper
├── auth.ts               # Backend + client authentication
├── oauth.ts              # Backend OAuth 2.0 client-credentials
├── jit.ts                # JIT privileged identity — scoped ephemeral tokens
├── jit-store.ts          # Pluggable token/denylist store (MCP_JIT_STORE) — HA
├── enterprise-auth.ts    # Enterprise-managed authorization (MCP ext-auth ID-JAG)
├── mcp-spec.ts           # MCP spec revisions + per-revision feature flags
├── request-context.ts    # Per-request context (W3C trace propagation)
├── helpers.ts            # fetchJSON, createFetcher, utilities
├── metrics.ts            # Self-metrics (Prometheus format)
├── versions.ts           # Backend version-support model + feature gates
├── version-registry.ts   # Live backend version detection (TTL-cached)
├── protocols.ts          # Query-protocol catalog (PromQL, LogQL, …)
├── gating.ts             # MCP_VERSION_GATING enforcement
├── compat.ts             # Tool alias / argument-mapping facade
├── tools/
│   ├── traces.ts         # Traces layer — dispatches to a provider per TRACES_PROVIDER (5 tools)
│   ├── metrics.ts        # Prometheus metrics skill (6 tools)
│   ├── logs.ts           # Loki logs skill (4 tools)
│   ├── elasticsearch.ts  # ES/OpenSearch skill (5 tools)
│   ├── alertmanager.ts   # Alertmanager skill (4 tools)
│   ├── grafana.ts        # Grafana read-only skill (10 tools)
│   ├── cilium.ts         # Cilium eBPF networking skill (6 tools)
│   ├── beyla.ts          # Grafana Beyla eBPF auto-instrumentation skill (4 tools)
│   ├── kubernetes.ts     # Kubernetes CRD reader skill (5 tools)
│   ├── clickhouse.ts     # ClickHouse logs skill (5 tools)
│   ├── pyroscope.ts      # Pyroscope profiling skill (4 tools)
│   ├── opa.ts            # Open Policy Agent skill (4 tools)
│   ├── envoy.ts          # Envoy proxy admin skill (4 tools)
│   ├── consul.ts         # Consul skill (5 tools)
│   ├── kong.ts           # Kong Gateway skill (4 tools)
│   ├── traefik.ts        # Traefik skill (4 tools)
│   ├── influxdb.ts       # InfluxDB metrics skill (3 tools)
│   ├── opentsdb.ts       # OpenTSDB metrics skill (3 tools)
│   ├── graylog.ts        # Graylog logs skill (3 tools)
│   ├── pinpoint.ts       # Pinpoint skill (3 tools)
│   ├── pipeline.ts       # Collection pipelines skill (4 tools)
│   ├── zk-proofs.ts      # ZK proof skill (4 tools)
│   └── system.ts         # System health skill (4 tools)
├── providers/
│   └── traces/           # Trace provider implementations (Layer→Provider pattern)
│       ├── types.ts      # TracesProvider interface + factory type
│       ├── jaeger.ts     # Jaeger Query API provider (default)
│       ├── tempo.ts      # Grafana Tempo TraceQL provider
│       ├── zipkin.ts     # Zipkin v2 provider
│       └── skywalking.ts # SkyWalking OAP GraphQL provider
├── resources/
│   └── overview.ts       # MCP resource: auto-generated overview
└── transports/
    ├── session-store.ts  # HTTP session lifecycle + principal binding (pre-2026)
    ├── mcp-2026.ts       # MCP 2026-07-28 request handling (stateless, headers)
    └── jit-endpoints.ts  # /auth/token mint · refresh · revoke handlers

Adding a new skill

// 1. Create src/tools/tempo.ts
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import type { Skill, SkillHelpers } from '../skill.js';

function registerTools(server: McpServer, helpers: SkillHelpers): void {
  const tempoUrl = helpers.env('TEMPO_URL');
  const fetchJSON = helpers.createFetcher('TEMPO', 'tempo');

  server.tool('tempo_search', 'Search traces in Tempo', { ... }, async (params) => {
    // ...
  });
}

export const skill: Skill = {
  id: 'tempo',
  name: 'Grafana Tempo',
  description: 'Query traces via the Grafana Tempo API',
  tools: 1,
  backends: ['Tempo'],
  isAvailable: () => !!process.env.TEMPO_URL,
  register: registerTools,
};

// 2. Add to src/skills.ts
import { skill as tempo } from './tools/tempo.js';
export const allSkills: Skill[] = [...existingSkills, tempo];

Auth Flow

Client → [API Key] → MCP Server → [Backend Credentials] → Jaeger/Prometheus/Loki
                          │
                          ├── Authorization: Bearer <JAEGER_AUTH_TOKEN>  → Jaeger
                          ├── Authorization: Basic <PROMETHEUS_AUTH_BASIC> → Prometheus
                          └── Authorization: Bearer <LOKI_AUTH_TOKEN>    → Loki
                               X-Scope-OrgID: <LOKI_TENANT_ID>

With JIT identity enabled, the static key only signs in; tools are reached with a scoped, expiring token (see Just-in-Time (JIT) Privileged Identity):

Client → [API Key] → POST /auth/token ──► mcpj_… (scoped, TTL, rotatable)
Client → [mcpj_…]  → MCP Server (session bound to the token lineage)
                          └── refresh before expiry / revoke when done

Development

# Dev mode (tsx, no build step)
npm run dev             # stdio
npm run dev:http        # HTTP on port 3001

# Type check
npm run lint

# Build
npm run build

# Tests
npm test

# Tests with coverage (thresholds enforced on the auth surface)
npm run test:coverage

# Run a single test file
npx vitest run tests/auth.test.ts

# Docker live tests against local backend fixtures
npm run test:live
node scripts/live-test.mjs --skill metrics

For the Docker-backed one-skill-at-a-time workflow, fixture coverage, report viewer, and troubleshooting, see docs/live-testing.md.

Appendix: Live Cluster Analysis

The following analysis was generated entirely by an AI agent (GitHub Copilot CLI) using this MCP server to query a production KrystalineX cluster — 27 tool calls across 6 skills, zero manual commands. This is what "Proof of Observability" looks like in practice.

Cluster: KrystalineX crypto exchange · 3-node K8s (1 control-plane, 2 workers) · Helm-managed
MCP Server: v1.2.0 · 6/7 skills active (Elasticsearch disabled) · session-based HTTP transport
Date: 2026-03-24T19:30 UTC

Infrastructure

Prometheus Targets — 12/12 UP

All scrape targets healthy with zero errors:

krystalinex-server · payment-processor · Kong · Grafana · Jaeger · OTEL Collector · Prometheus · RabbitMQ · Redis · kube-state-metrics · node-exporter ×2

Application Services

Performance Snapshot

SLO Error Budgets

The latency SLO budget is being consumed faster than expected. A predict_linear rule forecasts exhaustion within 24 hours. Worth investigating tail latency in kx-exchange.

Active Alerts — 3

All critical rules — HighErrorRate, ServiceDown, ContainerCrashLooping, OOMKilled — are inactive.

Service Dependency Graph

kx-gateway ──(99 calls)──► kx-exchange ──(5 calls)──► kx-matcher
                                │
                                └──(94 calls)──► jaeger (OTEL export)

kx-wallet  ──(23 calls)──► kx-exchange
kx-wallet  ──(14 calls)──► kx-gateway

Logs & Traces

Tools Used

This analysis invoked 27 MCP tool calls:

Verdict

The cluster is healthy and stable with generous headroom on both active nodes. The main items to watch are the latency SLO budget trend and kube node disk usage at 62%. The offline worker2 node is a known hardware issue. No action required on the PodNotReady alerts — they are ephemeral artifacts of recent deployments.

Monorepo Integration (git subtree)

This directory is maintained as a git subtree of the standalone repo MoebiusX/otel-mcp-server. The standalone repo is the single source of truth.

Pull latest changes from upstream

cd /path/to/KrystalineX
git subtree pull --prefix=otel-mcp-server otel-upstream main --squash

Push monorepo changes back upstream

cd /path/to/KrystalineX
git subtree push --prefix=otel-mcp-server otel-upstream main

Initial setup (already done)

git remote add otel-upstream https://github.com/MoebiusX/otel-mcp-server.git
git subtree add --prefix=otel-mcp-server otel-upstream main --squash

Note: src/client.ts is currently monorepo-only. Push it upstream when ready.

License

Apache-2.0 — see LICENSE.