检索增强生成系统

# Clone the repository
git clone https://github.com/yourusername/epsteindataset.git
cd epsteindataset

# Create virtual environment
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

# Install dependencies
pip install -r requirements.txt

# Download spaCy model for entity extraction
python -m spacy download en_core_web_sm

# Copy environment template
cp .env.example .env

# Option 1: Google Gemini API (Direct)
GEMINI_API_KEY=your_gemini_api_key

# Option 2: OpenRouter API
OPENROUTER_API_KEY=your_openrouter_api_key

import sqlite3

# Connect to the corpus
conn = sqlite3.connect('corpus.sqlite')
cursor = conn.cursor()

# Query example
cursor.execute("SELECT * FROM documents LIMIT 10")
for row in cursor.fetchall():
    print(row)

# Start the MCP server for LLM integration
python -m src.mcp_server.server

┌─────────────────────────────────────────────────────────────┐
│                     LLM Agent / User                        │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      MCP Server                             │
│  Tools: search, fetch_page, entity_lookup, evidence_pack    │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│  Hybrid Search  │ │  Entity Graph   │ │ Citation Builder│
│  (BM25+Vector)  │ │  (Co-occurrence)│ │  (Provenance)   │
└─────────────────┘ └─────────────────┘ └─────────────────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Storage Layer                            │
│  PostgreSQL/SQLite │ Meilisearch │ LanceDB/Qdrant          │
└─────────────────────────────────────────────────────────────┘

User: Who visited Little Saint James in 2005?

Agent: [calls search tool]
       Query: "Little Saint James visitors 2005"
       Filters: {date_start: "2005-01-01", date_end: "2005-12-31"}

Results: [
  {
    "text": "Flight log entry showing...",
    "citation": {
      "source": "DOJ Dataset 3",
      "page_number": 142,
      "bates_number": "DOJ-00045821"
    }
  }
]

epsteindataset/
├── src/
│   ├── embeddings/       # Gemini/OpenRouter embedding providers
│   ├── processing/       # PDF extraction, OCR, chunking
│   ├── search/           # Hybrid search, vector stores
│   ├── database/         # SQLAlchemy models, schema
│   └── mcp_server/       # MCP server implementation
├── scripts/              # Data processing scripts
├── config/               # Configuration files
├── tests/                # Test suite
├── corpus.sqlite         # Pre-processed document database
├── data_archive.zip      # Additional processed data
├── requirements.txt      # Python dependencies
├── .env.example          # Environment template
└── README.md