AI导哈

测试与安全

error-discovery

ai-evals-course/evals-skills

Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

  • 1090 次安装
  • 1600 星标
  • 收录于

安装命令

npx skills add https://github.com/ai-evals-course/evals-skills --skill error-discovery

SKILL.md

Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.

This skill is meant for interactive sessions only.

This skill has two parts:

  • This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
  • review-loop.md covers phase 5. You run the interactive review session.

Read this file first to build everything. Once the app is running and the human starts reviewing, follow review-loop.md.

Progress updates

Each phase takes time, especially building the interface. Tell the user what you are doing at each step. Before starting a phase, say what you are about to do and why. When a step finishes, say what you did and what comes next. For example: "Reading 10 sample records to understand the data shape", "Building the HTML app with three views: article, map, and progress", "Clustering on 5 features to select diverse samples." Do not go silent for long stretches.

Phase 1: Understand the domain and data

Before building anything, study the dataset thoroughly.

相关 Skills

validate-evaluatorai-evals-course/evals-skillsCalibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).智能体工作流1110write-judge-promptai-evals-course/evals-skillsDesign LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.智能体工作流1105eval-auditai-evals-course/evals-skillsAudit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).测试与安全1088generate-synthetic-dataai-evals-course/evals-skillsCreate diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.数据与研究1027evaluate-ragai-evals-course/evals-skillsGuides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.测试与安全1011build-review-interfaceai-evals-course/evals-skillsBuild a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.测试与安全1010