本地RAG检索增强生成系统

git clone https://github.com/hrayleung/ly_rag_mcp.git
cd ly_rag_mcp

conda create -n deep-learning python=3.10
conda activate deep-learning

# Core dependencies
pip install llama-index llama-index-embeddings-openai llama-index-vector-stores-chroma
pip install llama-index-postprocessor-cohere-rerank chromadb fastmcp

# Gemini support (optional)
pip install google-genai

# New features (Hybrid Search & Web Crawling)
pip install rank-bm25 llama-index-retrievers-bm25 firecrawl-py

export EMBEDDING_PROVIDER=openai
export EMBEDDING_MODEL=text-embedding-3-large
export OPENAI_API_KEY='your-openai-api-key'
export COHERE_API_KEY='your-cohere-api-key'     # Optional: For Reranking
export FIRECRAWL_API_KEY='your-firecrawl-key'   # Optional: For Web Crawling

export EMBEDDING_PROVIDER=gemini
export EMBEDDING_MODEL=text-embedding-004
export GEMINI_API_KEY='your-gemini-api-key'
export COHERE_API_KEY='your-cohere-api-key'     # Optional: For Reranking
export FIRECRAWL_API_KEY='your-firecrawl-key'   # Optional: For Web Crawling

# Incremental update (default - only processes new/modified files)
python build_index.py /path/to/your/documents

# Force full rebuild
python build_index.py /path/to/your/documents --rebuild

{
  "mcpServers": {
    "llamaindex-rag": {
      "command": "/path/to/conda/envs/deep-learning/bin/python",
      "args": ["/path/to/ly_rag_mcp/mcp_server.py"],
      "cwd": "/path/to/ly_rag_mcp",
      "env": {
        "EMBEDDING_PROVIDER": "openai",
        "EMBEDDING_MODEL": "text-embedding-3-large",
        "OPENAI_API_KEY": "your-openai-key",
        "COHERE_API_KEY": "your-cohere-key",
        "FIRECRAWL_API_KEY": "your-firecrawl-key"
      }
    }
  }
}

{
  "mcpServers": {
    "llamaindex-rag": {
      "command": "/path/to/conda/envs/deep-learning/bin/python",
      "args": ["/path/to/ly_rag_mcp/mcp_server.py"],
      "cwd": "/path/to/ly_rag_mcp",
      "env": {
        "EMBEDDING_PROVIDER": "gemini",
        "EMBEDDING_MODEL": "text-embedding-004",
        "GEMINI_API_KEY": "your-gemini-key",
        "COHERE_API_KEY": "your-cohere-key",
        "FIRECRAWL_API_KEY": "your-firecrawl-key"
      }
    }
  }
}

What are these documents about?
Use query_rag to search for "parallel computing"
Show me the index statistics

Query → Search Mode Detection (Adaptive)
    ↓
(Optional) HyDE Query Augmentation (if weak results)
    ↓
Parallel: Vector Search + BM25 Keyword Search
    ↓
RRF (Reciprocal Rank Fusion) Merge
    ↓
(Optional) Cohere Reranking
    ↓
Results (top_k)

Documents → DocumentLoader (20+ formats)
    ↓
DocumentProcessor (UTF-8 sanitize + context injection)
    ↓
DocumentChunker (AST for code, sentence for docs)
    ↓
IndexManager → LlamaIndex Storage
    ↓
Persist: ChromaDB + JSON manifests

storage/
└── {project_name}/
    ├── chroma_db/          # ChromaDB vector collection
    ├── project_metadata.json    # Project config, keywords
    ├── ingest_manifest.json     # File change tracking
    ├── indexed_files.json       # mtime/size tracking
    ├── docstore.json       # LlamaIndex document store
    ├── index_store.json    # LlamaIndex index metadata
    └── graph_store.json    # LlamaIndex graph store

1. Use iterative_search("Python async patterns", initial_top_k=3)
2. Review the 3 most relevant results
3. If more context needed: query_rag("Python async patterns", similarity_top_k=10)
4. Or refine: query_rag("asyncio event loop internals", similarity_top_k=5)

1. query_rag("machine learning", similarity_top_k=5) - understand scope
2. query_rag("neural network backpropagation", similarity_top_k=5) - focus on specific topic
3. query_rag("gradient descent optimization", similarity_top_k=3) - deep dive

# Full suite with coverage
pytest tests/ --cov=rag --cov-report=term -v

# Single file
pytest tests/test_query_tools.py --cov-report=term -v

# Single test
pytest tests/test_query_tools.py::test_specific_function --cov-report=term -v

"env": {
  "EMBEDDING_PROVIDER": "openai",
  "EMBEDDING_MODEL": "text-embedding-3-large"  // or "text-embedding-3-small"
}

"env": {
  "EMBEDDING_PROVIDER": "gemini",
  "EMBEDDING_MODEL": "text-embedding-004"  // or "embedding-001"
}

ly_rag_mcp/
├── mcp_server.py           # FastMCP server entry point (MCP stdio)
├── api_server.py           # Optional HTTP API + Vue UI
├── build_index.py          # CLI index builder (incremental)
├── verify_setup.py         # Setup verification
├── debug_rag.py            # Debug & profiling tool
├── rag/                    # Core package (modular architecture)
│   ├── __init__.py         # Lazy exports
│   ├── config.py           # RAGSettings, logging, constants
│   ├── models.py           # Data models (SearchMode, ProjectMetadata, etc.)
│   ├── embeddings.py       # Embedding factory (OpenAI/Gemini)
│   ├── storage/            # Storage layer
│   │   ├── chroma.py       # ChromaDB client manager (singleton)
│   │   └── index.py        # LlamaIndex storage manager (singleton)
│   ├── retrieval/          # Retrieval layer
│   │   ├── search.py       # Unified search engine (hybrid/semantic/keyword)
│   │   ├── reranker.py     # Cohere reranking manager
│   │   ├── bm25.py         # BM25 keyword search manager
│   │   └── hyde.py         # HyDE query augmentation
│   ├── ingestion/          # Ingestion layer
│   │   ├── loader.py       # Multi-format document loading
│   │   ├── processor.py    # Text cleaning, context injection
│   │   └── chunker.py      # Smart chunking (AST for code, sentence for docs)
│   ├── project/            # Multi-project isolation
│   │   ├── manager.py      # Project lifecycle, smart routing/selection
│   │   └── metadata.py     # Project metadata storage (atomic writes)
│   └── tools/              # MCP tool definitions
│       ├── __init__.py     # Tool registration facade
│       ├── query.py        # Search & retrieval tools
│       ├── ingest.py       # Document ingestion tools
│       └── admin.py        # Admin & management tools
├── tests/                  # Test suite (pytest)
├── frontend/               # Optional Vue 3 + Vite UI
│   ├── src/
│   │   ├── components/     # Vue components
│   │   ├── composables/    # API & polling composables
│   │   └── views/          # Page views
│   ├── index.html
│   ├── package.json
│   └── vite.config.ts
├── .env.example            # Environment template
│
└── storage/                # Generated indexes (git-ignored)
    └── {project}/          # Per-project isolation
        ├── chroma_db/      # ChromaDB vector database
        ├── project_metadata.json  # Project config
        ├── ingest_manifest.json   # Ingestion tracking
        ├── indexed_files.json     # File change tracking
        ├── docstore.json       # LlamaIndex document store
        ├── index_store.json    # LlamaIndex index metadata
        └── graph_store.json    # LlamaIndex graph store

# Run the comprehensive debug tool first!
python debug_rag.py

# This will check:
# - Environment variables (API keys)
# - Storage integrity
# - Index quality
# - Performance metrics
# - Edge cases

# Check if index has documents
python -c "from mcp_server import get_index_stats; print(get_index_stats())"

# If document_count is 0:
python build_index.py /path/to/your/documents

# Profile retrieval performance
python debug_rag.py --profile

# Check cache efficiency
python -c "from mcp_server import get_cache_stats; print(get_cache_stats())"

# If cache hit rate < 90% after multiple queries:
# - Check MCP client logs for server restarts
# - Server should stay running between queries

# Just re-run the same command - it will only process new/modified files
python build_index.py /path/to/your/documents

# Force full rebuild if needed
python build_index.py /path/to/your/documents --rebuild

"env": {
  "OPENAI_API_KEY": "...",
  "COHERE_API_KEY": "...",
  "RAG_LOG_LEVEL": "DEBUG"
}