数据与研究

agent-observability-eval-bootstrap

datadog-labs/ag…t-skills

Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.

  • 561 次安装
  • 180 星标
  • 收录于

安装命令

npx skills add https://github.com/datadog-labs/agent-skills --skill agent-observability-eval-bootstrap

SKILL.md

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

  1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
  2. Check whether MCP tools are present in your active tool list. The canonical signal is whether mcp__datadog-llmo-mcp__list_llmobs_evals appears in your available tools.
  3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
  4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
  5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
  6. If neither is available → stop and tell the user:

    "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/v1/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.