候选人评估MCP服务器

Step 1: Scrubbing (LLM 1)
├─ Input: Original candidate text
├─ Process: Intelligently rewrites protected characteristics
└─ Output: Scrubbed text (visible to you)
         ↓
Step 2: Evaluation (LLM 2)
├─ Input: Scrubbed text + your question
├─ Process: Answers your question with ZERO access to original
└─ Output: Evaluation based on scrubbed data

pip install -r requirements.txt

pip install -e .

pip install -e ".[openai]"

export ANTHROPIC_API_KEY='your-key-here'

{
  "mcpServers": {
    "candidate-evaluation": {
      "command": "python3",
      "args": [
        "/path/to/server.py"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "your-api-key-here"
      }
    }
  }
}

{
  "mcpServers": {
    "candidate-evaluation": {
      "command": "python3",
      "args": [
        "/path/to/server.py"
      ],
      "env": {
        "MODEL_PROVIDER": "openai",
        "MODEL_NAME": "gpt-4o",
        "OPENAI_API_KEY": "your-openai-key-here"
      }
    }
  }
}

{
  "scrubbed_text": "Alex Kim. BS in Computer Science from UC Berkeley...",
  "original_length": 393,
  "scrubbed_length": 297,
  "categories_requested": ["all"]
}

How much should we offer this candidate?

Alex Kim, 29, Korean-American, non-binary (they/them). Diagnosed with autism
spectrum disorder (Level 1). BS in Computer Science from UC Berkeley. 4 years
experience as a backend developer at two startups. Strong in Python, Go, and
Kubernetes. Contributed to 3 open-source projects. Prefers written communication
over meetings. Applying for Senior Software Developer at a remote-first company.

Tool: scrub_protected_characteristics
Request: [original text with protected characteristics]
Response: {
  "scrubbed_text": "The candidate holds a BS in Computer Science from UC
Berkeley. Has 4 years of experience as a backend developer at two startups.
Demonstrates strong proficiency in Python,    Go, and Kubernetes. Has
contributions to 3 open-source projects. Prefers written communication over
meetings. Applying for Senior Software Developer at a remote-first company."
}

Tool: evaluate_scrubbed_candidate
Request: {
  "scrubbed_text": "[scrubbed text from Step 1]",
  "evaluation_criteria": "How much should we offer this candidate for
                          a Senior Software Developer role?"
}
Response: [Detailed salary recommendation based on 4 years experience,
          technical skills, education, and remote-work fit]

Technical Skills: 6/7
Leadership: 5/7
Communication: 6/7
Problem-Solving: 7/7
Culture Fit: 6/7

[Detailed reasoning based on scrubbed qualifications...]

Yes, recommend moving to next round.

Reasoning:
- Strong technical foundation with 5+ years relevant experience
- Demonstrated leadership in previous roles
- Track record of delivering projects on time
- Skills align well with role requirements

python evaluation/test_llm_scrubbing.py

python evaluation/test_unified_workflow.py

# Preview without API calls
python3 evaluation/benchmark/runner.py --dry-run

# Quick single-arm test
python3 evaluation/benchmark/runner.py --n 1 --arms raw_naive

# Full v1 run (~$3.50, ~25 min)
python3 evaluation/benchmark/runner.py --n 30 --concurrency 10

# v2 with randomized names
python3 evaluation/benchmark/runner_v2.py --n 6 --concurrency 10

# Execute analysis notebooks
cd evaluation/benchmark && jupyter nbconvert --to notebook --execute analysis.ipynb --output analysis_executed.ipynb
cd evaluation/benchmark && jupyter nbconvert --to notebook --execute demo_plots.ipynb --output demo_plots_executed.ipynb

┌─────────────────────────────────┐
│   Original Candidate Text       │
│  "29, Korean-American,          │
│   non-binary, autistic..."      │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│      LLM 1 (Scrubber)           │
│  - Rewrites biased terms        │
│  - No access to user question   │
│  - Returns scrubbed text        │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│   Scrubbed Text (VISIBLE)       │  ← YOU CAN VERIFY THIS
│  "BS from UC Berkeley,          │
│   4 years experience..."        │
└────────────┬────────────────────┘
             │
             ├──────────────────┐
             │                  │
             ▼                  ▼
┌─────────────────────┐  ┌──────────────┐
│  LLM 2 (Evaluator)  │  │ User Question│
│ - ZERO access to    │◄─┤ "How much to │
│   original text     │  │  offer?"     │
│ - ZERO conversation │  └──────────────┘
│   history           │
│ - Answers question  │
│   directly          │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────────────────┐
│   Evaluation Answer             │
│  "$140k-160k based on           │
│   6 years experience..."        │
└─────────────────────────────────┘

Candidate text → Single LLM → Evaluation
• LLM sees all protected characteristics
• No separation between data and evaluation
• No visibility into what influenced the evaluation

Candidate text → LLM 1 (Scrub) → [Visible scrubbed text] → LLM 2 (Evaluate) → Answer
✅ Evaluator does not receive protected characteristics
✅ You can verify the scrubbed data before evaluation
✅ Two separate LLM calls with no shared conversation history
✅ Auditable process with visible intermediate steps