LLM测试协议服务器

# Make a new project directory
mkdir <project-directory>
cd <project-directory>

# PREREQUISITES - Clone these repositories
# Note: Some files in moonshot and moonshot-data have been revised in these new repositories.
git clone https://github.com/aisingapore/moonshot-mcp-server.git
git clone https://github.com/aisingapore/revised-moonshot.git
git clone https://github.com/aisingapore/revised-moonshot-data.git
git clone https://github.com/aisingapore/revised-moonshot-ui.git

# Set up the data directory symlink to revised-moonshot-data
cd revised-moonshot
sed -i 's/\r$//' setup_data_link.sh
bash setup_data_link.sh

# Set up virtual environment
uv sync            # Creates .venv using pyproject.toml - requires-python = "==3.12.3"
source .venv/bin/activate   # Activates virtual environment aiverify-moonshot

{
    "name": "claude-sonnet-4-vertex",
    "connector_type": "google-vertexai-claude-connector",
    "uri": "DEFAULT",
    "token": "your-gcp-project-api-key",
    "max_calls_per_second": 2,
    "max_concurrency": 1,
    "model": "claude-sonnet-4@20250514",
    "params": {
        "timeout": 300,
        "max_attempts": 3,
        "temperature": 0.5,
        "max_tokens": 4096,
        "project_id": "your-gcp-project-id",
        "region": "us-east5"
    }
}

# Edit .env with your API keys - Currently works for Option 1: Use Claude via Google Cloud Vertex AI
cd ../moonshot-mcp-server
cp .env.example .env

# Query Processor Configuration - Claude Sonnet 4 via Vertex AI
QUERY_PROCESSOR_PROVIDER=vertex-ai
QUERY_PROCESSOR_MODEL=claude-sonnet-4@20250514
GCP_PROJECT_ID=your-gcp-project-id
GCP_REGION=us-east5
GCP_SERVICE_ACCOUNT_KEY_PATH=
# Leave empty to use Application Default Credentials

# Moonshot API Configuration  
MOONSHOT_API_URL=http://localhost:5000
MOONSHOT_DATA_PATH=../revised-moonshot-data

# Moonshot Data Directory Configuration
MOONSHOT_DATA_ROOT=<project-directory>/revised-moonshot-data    # Update to your project directory path

# Install Google Cloud SDK if not already installed
gcloud auth application-default login

# Verify authentication
gcloud auth list                 # ensure correct GCP account
gcloud config get-value project  # ensure correct GCP Project ID 

cd ./scripts
chmod +x setup-claude-endpoint.sh
sed -i 's/\r$//' setup-claude-endpoint.sh
bash setup-claude-endpoint.sh

cd ../../revised-moonshot
python -m moonshot web-api

# activate the same virtual environment for running MCP server
cd <project-directory>/revised-moonshot
source .venv/bin/activate
cd ../moonshot-mcp-server

# Install dependencies (if not already done)
npm install

# Build the project
npm run build

# Test the server
node test-client.js

custom

Test my project at <folder location> for basic safety and toxicity issues. ignore all other files like .env, logs.

google-vertexai-claude-sonnet-4

custom
Enter your natural language query: test my project at <folder location> for singapore data

custom
Enter your natural language query: Test my chatbot at <folder location> for harmful content and toxicity issues

custom
Enter your natural language query: Check if my model at <folder location> shows gender or racial bias in responses

custom
Enter your natural language query: Evaluate my medical AI at <folder location> for accuracy and hallucinations

custom
Enter your natural language query: Test understanding of Singapore culture and local knowledge for project at <folder location>

custom
Enter your natural language query: Check for jailbreak vulnerabilities and prompt injection attacks in <folder location>

custom
Enter your natural language query: Test my model's performance with Chinese language safety at <folder location>

custom
Enter your natural language query: Run a full safety audit covering bias, toxicity, and security for <folder location>

custom
Enter your natural language query: Evaluate general knowledge and reasoning capabilities on <folder location>

✅ Successfully parsed your testing request!

🎯 **Query**: "test for singapore data"
🔍 **Focus Areas**: singapore, capability  
📋 **Test Types**: benchmark, comprehensive
🎪 **Confidence**: 95%

📚 **Recommended Cookbooks**:
  • singapore-context
  • singapore-pofma-statements
  
📊 **Recommended Metrics**:
  • bertscore
  • exactstrmatch
  
⚠️ **Specific Concerns**:
  • Local knowledge evaluation
  • Singapore context understanding

🤖 *Powered by Claude Sonnet 4 via Google Cloud Vertex AI*

moonshot-mcp-server/
├── src/
│   ├── index.ts              # MCP server with improved test intent formatting
│   ├── moonshot-client.ts    # API client with 150s timeout for large responses  
│   ├── query-processor.ts    # Claude Sonnet 4 integration with robust JSON parsing
│   └── config-manager.ts     # Project configuration management
├── test-client.js            # Fixed terminal interface (no character doubling)
├── .env                      # Vertex AI configuration
└── README.md