RAG和长期记忆内核

rmcp-memex是一个定制的Rust MCP内核,通过LanceDB为AI代理提供RAG(检索增强生成)和长期记忆功能。

  • by vetcoders
  • Rust
  • 本地部署
  • 4 星标
  • 更新于
在 GitHub 查看 LibraxisAI/mcp_memex

使用说明

rmcp内存

rmcp-memex 是一个定制的Rust MCP内核,通过LanceDB为AI代理提供RAG和长期内存功能。

它揭示了来自单个规范曲面的两种显式传输模式:

  1. stdio (标准MCP):本地代理的本地MCP集成(例如Claude Desktop)。
  2. HTTP/SSE (多代理守护程序):一种中央守护进程模式,允许并发AI代理通过网络访问同一内存池,解决了LanceDB的独占锁约束。

二进制名称: rmcp-memex 是唯一支持的二进制名称。GitHub安装程序还创建 rmcp_memex 作为旧脚本的遗留兼容性符号链接。

MCP合同: 当前的MCP表面仅是有意使用的工具。 initialize 广告 tools,而 resources/* 尚未实施。

释放表面

  • 快速安装: curl -LsSf https://raw.githubusercontent.com/VetCoders/rmcp-memex/main/install.sh | sh
  • 预构建的二进制包: GitHub发布 从本地构建和签名的工件上传
  • 发布运行手册: docs/RELEASE.md
  • 配置指南: docs/02_configuration.md
  • HTTP/SSE引用: docs/HTTP_API.md文件
  • 静态启动页面来源: docs/index.html 由...出版 .github/workflows/pages.yml

快速开始

# Install from the latest GitHub Release
curl -LsSf https://raw.githubusercontent.com/VetCoders/rmcp-memex/main/install.sh | sh

# Start the MCP server
rmcp-memex serve

# Open the local dashboard in your browser
rmcp-memex dashboard

# Or run the multi-agent HTTP/SSE daemon
rmcp-memex sse

概述

作为MCP(模型上下文协议)服务器, rmcp-memex 提供:

  • RAG(检索增强生成) -文档索引与语义搜索
  • 混合搜索 -BM25关键字+语义向量搜索(基于Tantivy)
  • 向量内存 -文本块的语义存储与检索
  • 命名空间隔离 -命名空间中的数据隔离
  • 安全 -基于令牌的受保护命名空间访问控制
  • 洋葱片建筑 -层次嵌入(OUTER→中间→INNER→CORE)
  • 预处理 -对话导出中的自动噪声过滤(减少约36-40%)
  • 精确匹配重复数据删除 -基于SHA256的重复导出数据消除

建筑

┌─────────────────────────────────────────────────────────────┐
│                      rmcp-memex                              │
├─────────────────────────────────────────────────────────────┤
│  MCP Server (JSON-RPC over stdio)                           │
│  ├── handlers/mod.rs    - Request routing & validation      │
│  ├── security/mod.rs    - Namespace access control          │
│  └── rag/mod.rs         - RAG pipeline                      │
├─────────────────────────────────────────────────────────────┤
│  Storage Layer                                               │
│  ├── LanceDB           - Vector embeddings                  │
│  ├── Tantivy           - BM25 keyword index                 │
│  └── moka              - In-memory cache                    │
├─────────────────────────────────────────────────────────────┤
│  Embeddings (External Providers)                             │
│  ├── Ollama            - Local models (recommended)         │
│  ├── MLX Bridge        - Apple Silicon acceleration         │
│  └── OpenAI-compatible - Any compatible endpoint            │
└─────────────────────────────────────────────────────────────┘

特性

RAG工具

工具 说明
rag_index 从文件中索引文档
rag_index_text 索引原始文本
rag_search 语义搜索文档(支持 auto_route)

内存工具

工具 说明
memory_upsert 在命名空间中添加/更新块
memory_get 按ID获取块
memory_search 在命名空间中进行语义搜索(支持 auto_route)
memory_delete 删除块
memory_purge_namespace 删除命名空间中的所有块
dive 对所有洋葱层(外层/中层/内层/核心)进行深入探索

安全工具

工具 说明
namespace_create_token 为命名空间创建访问令牌
namespace_revoke_token 撤销令牌(命名空间变为公共)
namespace_list_protected 列出受保护的命名空间
namespace_security_status 安全系统状态

图书馆使用情况

rmcp-memex 可以用作Rust应用程序中的库。它提供了一个高层次 MemexEngine 用于矢量存储操作的API。

添加到Cargo.toml

# Full library with CLI
rmcp-memex = "0.5"

# Library only (no CLI dependencies)
rmcp-memex = { version = "0.5", default-features = false }

基本用法

use rmcp_memex::{MemexEngine, MemexConfig, MetaFilter, StoreItem};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Quick setup for any application
    let engine = MemexEngine::for_app("my-app", "documents").await?;

    // Store a document
    engine.store(
        "doc-1",
        "Patient presented with lethargy and decreased appetite",
        json!({"patient_id": "P-123", "visit_type": "checkup"})
    ).await?;

    // Search semantically
    let results = engine.search("lethargy symptoms", 10).await?;
    for r in &results {
        println!("{}: {} (score: {:.2})", r.id, r.text, r.score);
    }

    // Get by ID
    if let Some(doc) = engine.get("doc-1").await? {
        println!("Found: {}", doc.text);
    }

    // Delete
    engine.delete("doc-1").await?;

    Ok(())
}

Vista集成

对于Vista PIMS,请使用优化的构造函数:

use rmcp_memex::MemexEngine;

// Vista-optimized: 1024 dims, qwen3-embedding:0.6b model
let engine = MemexEngine::for_vista().await?;

// Store visit notes
engine.store(
    "visit-456",
    "SOAP note: Feline diabetes mellitus diagnosis...",
    json!({"patient_id": "P-789", "doc_type": "soap_note"})
).await?;

批量操作

use rmcp_memex::{MemexEngine, StoreItem};
use serde_json::json;

let engine = MemexEngine::for_app("my-app", "notes").await?;

let items = vec![
    StoreItem::new("doc-1", "First document").with_metadata(json!({"type": "note"})),
    StoreItem::new("doc-2", "Second document").with_metadata(json!({"type": "note"})),
    StoreItem::new("doc-3", "Third document").with_metadata(json!({"type": "note"})),
];

let result = engine.store_batch(items).await?;
println!("Stored {} documents", result.success_count);

GDPR合规删除

use rmcp_memex::{MemexEngine, MetaFilter};

let engine = MemexEngine::for_app("my-app", "patients").await?;

// Delete all documents for a specific patient
let filter = MetaFilter::for_patient("P-123");
let deleted = engine.delete_by_filter(filter).await?;
println!("Deleted {} documents", deleted);

混合搜索(BM25+矢量)

use rmcp_memex::{MemexEngine, SearchMode};

let engine = MemexEngine::for_app("my-app", "documents").await?;

// Hybrid search with BM25 + vector fusion (recommended)
let results = engine.search_hybrid("dragon mac studio", 10).await?;
for r in &results {
    println!("{}: {} (combined: {:.2}, vector: {:.2}, bm25: {:.2})",
        r.id, r.document, r.combined_score, r.vector_score, r.bm25_score);
}

// Explicit mode selection
let results = engine.search_with_mode("exact keyword", 10, SearchMode::Keyword).await?;
let results = engine.search_with_mode("semantic concept", 10, SearchMode::Vector).await?;
let results = engine.search_with_mode("best of both", 10, SearchMode::Hybrid).await?;

MCP合同和在制品助手

使用 tool_definitions() 当您需要由公开的确切MCP工具元数据时 stdio和HTTP/SSE传输。下面的辅助函数用于进程内 Rust调用者;它们是本地便利层,而不是第二个公共MCP 合同。它们的名称有意与MCP工具名称不同,因此两者 表面不会偶然漂移在一起。

use rmcp_memex::{MemexEngine, search_documents, store_document, tool_definitions};
use serde_json::json;

let engine = MemexEngine::for_app("agent", "memory").await?;

// Canonical MCP tool surface exposed by rmcp-memex transports
let tools = tool_definitions();
assert!(tools.iter().any(|tool| tool.name == "memory_upsert"));

// Local Rust helpers for in-process callers
let result = store_document(
    &engine,
    "mem-1".to_string(),
    "Important information to remember".to_string(),
    json!({"source": "conversation"}),
).await?;
assert!(result.success);

let results = search_documents(&engine, "important".to_string(), 5, None).await?;

功能标志

功能 描述 默认值
cli CLI二进制文件、TUI向导、进度条 是
provider-cascade Ollama/OpenAI兼容嵌入 是
# Build library only (no CLI)
cargo build --no-default-features

# Build with CLI
cargo build --features cli

配置指南

将rmcp-memex作为库集成到任何Rust项目中的完整指南。

先决条件

奥拉玛 (推荐)或任何与OpenAI兼容的嵌入API:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull an embedding model (choose based on your needs)
ollama pull qwen3-embedding:0.6b    # 1024 dims, ~600MB (fast, good quality)
ollama pull qwen3-embedding:8b      # 4096 dims, ~4GB (best quality)
ollama pull nomic-embed-text        # 768 dims, ~274MB (lightweight)

# Verify it's running
curl http://localhost:11434/api/tags

环境变量

通过配置 .env 或环境:

# =============================================================================
# EMBEDDING PROVIDER CONFIGURATION
# =============================================================================

# Ollama (default, recommended)
OLLAMA_BASE_URL=http://localhost:11434
EMBEDDING_MODEL=qwen3-embedding:0.6b
EMBEDDING_DIMENSION=1024

# Database storage (auto-created)
MEMEX_DB_PATH=~/.rmcp-servers/myapp/lancedb

# Optional: BM25 keyword search index
MEMEX_BM25_PATH=~/.rmcp-servers/myapp/bm25

# =============================================================================
# ADVANCED: Multiple providers (fallback cascade)
# =============================================================================

# Remote embedding server fallback
# DRAGON_BASE_URL=http://your-server.local
# DRAGON_EMBEDDER_PORT=12345

# MLX embedder for Apple Silicon
# EMBEDDER_PORT=12300
# MLX_MAX_BATCH_CHARS=32000
# MLX_MAX_BATCH_ITEMS=16
# DISABLE_MLX=1  # Set to disable MLX fallback

快速启动(自动配置)

use rmcp_memex::MemexEngine;
use serde_json::json;

// Auto-configures from defaults + environment
let engine = MemexEngine::for_app("my-app", "default").await?;

engine.store("doc-1", "Document content...", json!({"type": "note"})).await?;
let results = engine.search("content", 10).await?;

自定义配置

use rmcp_memex::{MemexConfig, MemexEngine};
use rmcp_memex::embeddings::{EmbeddingConfig, ProviderConfig};

// Read from your app's environment
let ollama_url = std::env::var("OLLAMA_BASE_URL")
    .unwrap_or_else(|_| "http://localhost:11434".to_string());
let model = std::env::var("EMBEDDING_MODEL")
    .unwrap_or_else(|_| "qwen3-embedding:0.6b".to_string());
let dimension: usize = std::env::var("EMBEDDING_DIMENSION")
    .unwrap_or_else(|_| "1024".to_string())
    .parse()
    .unwrap_or(1024);
let db_path = std::env::var("MEMEX_DB_PATH")
    .unwrap_or_else(|_| "~/.rmcp-servers/myapp/lancedb".to_string());

let config = MemexConfig {
    app_name: "my-app".to_string(),
    namespace: "default".to_string(),
    db_path: Some(db_path),
    dimension,
    embedding_config: EmbeddingConfig {
        required_dimension: dimension,
        providers: vec![ProviderConfig {
            name: "ollama".to_string(),
            base_url: ollama_url,
            model,
            priority: 1,
            endpoint: "/v1/embeddings".to_string(),
        }],
        ..EmbeddingConfig::default()
    },
    enable_bm25: false,
    bm25_config: None,
};

let engine = MemexEngine::new(config).await?;

提供者级联(多次回退)

配置多个提供程序-库按优先级顺序尝试它们:

use rmcp_memex::embeddings::{EmbeddingConfig, ProviderConfig};

let config = EmbeddingConfig {
    required_dimension: 1024,
    providers: vec![
        // Priority 1: Local Ollama (fastest)
        ProviderConfig {
            name: "ollama-local".to_string(),
            base_url: "http://localhost:11434".to_string(),
            model: "qwen3-embedding:0.6b".to_string(),
            priority: 1,
            endpoint: "/v1/embeddings".to_string(),
        },
        // Priority 2: Remote server fallback
        ProviderConfig {
            name: "remote-server".to_string(),
            base_url: "http://your-server:8080".to_string(),
            model: "text-embedding-3-small".to_string(),
            priority: 2,
            endpoint: "/v1/embeddings".to_string(),
        },
        // Priority 3: OpenAI API fallback
        ProviderConfig {
            name: "openai".to_string(),
            base_url: "https://api.openai.com".to_string(),
            model: "text-embedding-3-small".to_string(),
            priority: 3,
            endpoint: "/v1/embeddings".to_string(),
        },
    ],
    ..EmbeddingConfig::default()
};

命名空间策略

建议:每个应用程序一个命名空间,使用元数据进行筛选:

// ✅ CORRECT: Single namespace, filter by user_id/entity_id in metadata
let engine = MemexEngine::for_app("my-app", "default").await?;

// Store with entity IDs in metadata
engine.store("doc-1", "Document content...", json!({
    "user_id": "U-123",      // For multi-tenant filtering
    "project_id": "P-456",   // For project-level filtering
    "doc_type": "note",
    "created_at": "2024-12-28"
})).await?;

// Search within user context
let filter = MetaFilter::default().with_custom("user_id", "U-123");
let results = engine.search_filtered("query", filter, 10).await?;

// GDPR deletion: remove all user data
let deleted = engine.delete_by_filter(
    MetaFilter::default().with_custom("user_id", "U-123")
).await?;

嵌入模型参考

型号 尺寸 大小 用例
qwen3-embedding:0.6b 1024 ~600MB 速度快,质量好(推荐)
qwen3-embedding:8b 4096 ~4GB 质量最好,速度较慢
nomic-embed-text 768 ~274MB 轻便、快速
mxbai-embed-large 1024 ~670MB 良好的多语言能力
all-minilm 384 ~46MB 非常快,质量较低

故障排除

错误:“没有可用的嵌入提供程序”

# Check if Ollama is running
curl http://localhost:11434/api/tags

# Start Ollama
ollama serve

# Pull model if missing
ollama pull qwen3-embedding:0.6b

错误:“尺寸不匹配”

  • 创建后,每个表的LanceDB维度都是固定的
  • 使用不同 db_path 不同尺寸
  • 删除旧数据库以更改维度

错误:“连接被拒绝”

# Linux
systemctl status ollama
systemctl start ollama

# macOS
brew services info ollama
brew services start ollama

# Or run manually
ollama serve

性能调整:

# Larger batches (requires more VRAM)
MLX_MAX_BATCH_CHARS=64000
MLX_MAX_BATCH_ITEMS=32

快速开始

安装

快速安装(推荐):

curl -LsSf https://raw.githubusercontent.com/VetCoders/rmcp-memex/main/install.sh | sh

预构建的GitHub Release捆绑包是规范的安装路径,避免编译 目标机器上完整的LanceDB重Rust依赖图。

来源(仅限开发):

cargo install --path .

跑步

# Canonical MCP surface
rmcp-memex serve

# With security enabled
rmcp-memex serve --security-enabled

# With HTTP/SSE server for multi-agent access
rmcp-memex serve --http-port 8997

# HTTP-only daemon mode (no MCP stdio)
rmcp-memex serve --http-port 8997 --http-only

rmcp-memex 始终暴露一个规范的MCP工具表面。缩小运行时间 访问、使用 --allowed-paths、HTTP身份验证或命名空间安全,而不是 单独的“内存/满”模式开关。

HTTP/SSE服务器(多代理访问)

LanceDB使用独占文件锁——一次只能有一个进程访问数据库。 HTTP/SSE服务器通过为多个代理提供一个中央接入点来解决这个问题。

建筑

┌─────────────────────────────────────────────────────────────┐
│                     rmcp-memex daemon                        │
│  ┌─────────────────┐    ┌─────────────────┐                 │
│  │   MCP Server    │    │   HTTP/SSE      │                 │
│  │   (stdio)       │    │   (port 8997)   │                 │
│  └────────┬────────┘    └────────┬────────┘                 │
│           │                      │                          │
│           └──────────┬───────────┘                          │
│                      ▼                                      │
│              ┌─────────────┐                                │
│              │ RAGPipeline │ ← Single lock holder           │
│              └──────┬──────┘                                │
│                     ▼                                       │
│              ┌─────────────┐                                │
│              │   LanceDB   │                                │
│              └─────────────┘                                │
└─────────────────────────────────────────────────────────────┘
         ▲                    ▲
         │                    │
    Claude Desktop       HTTP Agents
    (MCP stdio)          (curl, fetch)

HTTP端点

端点 方法 描述
/health GET 健康检查(状态、db_path、embedding_provider)
/search POST 使用可选搜索 project, layer,以及 deep 过滤器(k 支持别名)
/sse/search GET SSE流媒体搜索,可选 project, layer,以及 deep 过滤器
/upsert POST 添加/更新文档
/index POST 使用洋葱片进行完整管道索引
/expand/{ns}/{id} GET 扩大洋葱片(带孩子)
/parent/{ns}/{id} GET 钻取父切片
/get/{ns}/{id} GET 按ID获取文档
/delete/{ns}/{id} POST 删除文档
/ns/{namespace} DELETE 清除整个命名空间

SSE端点上的MCP(克劳德代码兼容性)

端点 方法 描述
/sse/ GET SSE流-发送 endpoint 带有消息URL的事件
/messages/ POST JSON-RPC消息 ?session_id=xxx

在中配置 ~/.claude.json:

{
  "mcpServers": {
    "rmcp-memex": {
      "type": "sse",
      "url": "http://localhost:8997/sse/"
    }
  }
}

使用示例

# Open the local dashboard
rmcp-memex dashboard --db-path ~/.ai-memories/lancedb

# Start daemon
rmcp-memex sse --db-path ~/.ai-memories/lancedb &

# Health check
curl http://localhost:8997/health

# Store document
curl -X POST http://localhost:8997/upsert \
  -H "Content-Type: application/json" \
  -d '{"namespace": "agent1", "id": "mem1", "content": "Important context..."}'

# Search
curl -X POST http://localhost:8997/search \
  -H "Content-Type: application/json" \
  -d '{"query": "context", "namespace": "agent1", "k": 10, "project": "Vista", "deep": true}'

# SSE streaming search
curl -N "http://localhost:8997/sse/search?query=context&namespace=agent1&limit=5&project=Vista&layer=1"

多主机数据库路径

对于具有多台机器的设置(例如dragon、mgbook16),请使用每台主机的数据库路径:

# Per-host paths (each machine gets own database)
rmcp-memex serve --db-path ~/.ai-memories/lancedb.$(hostname -s)

# Or use the wizard for machine-agnostic configuration
rmcp-memex wizard

TUI向导自动检测主机名并提供:

  • 共享模式: ~/.ai-memories/lancedb (到处都是同一条路)
  • 每台主机模式: ~/.ai-memories/lancedb.dragon, ~/.ai-memories/lancedb.mgbook16等等。

配置(TOML)

# ~/.rmcp-servers/rmcp-memex/config.toml

db_path = "~/.rmcp-servers/rmcp-memex/lancedb"
cache_mb = 4096
log_level = "info"

# Whitelist of allowed paths
allowed_paths = [
    "~",
    "/Volumes/ExternalDrive/data"
]

# Security
security_enabled = true
token_store_path = "~/.rmcp-servers/rmcp-memex/tokens.json"

文档

洋葱片建筑

rmcp memex提供了分层的“洋葱片”,而不是传统的扁平分块:

┌─────────────────────────────────────────┐
│  OUTER (~100 chars)                     │  ← Minimum context, maximum navigation
│  Keywords + ultra-compression           │
├─────────────────────────────────────────┤
│  MIDDLE (~300 chars)                    │  ← Key sentences + context
├─────────────────────────────────────────┤
│  INNER (~600 chars)                     │  ← Expanded content
├─────────────────────────────────────────┤
│  CORE (full text)                       │  ← Complete document
└─────────────────────────────────────────┘

哲学: “最少信息→ 最大导航路径”

QueryRouter和自动路由

用于自动搜索模式选择的智能查询意图检测:

# Auto-detect query intent and select optimal mode
rmcp-memex search -n memories -q "when did we buy dragon" --auto-route
# Output: Query intent: temporal (confidence: 0.70)
#         Selects: hybrid mode with date boosting

# Structural queries suggest loctree
rmcp-memex search -n code -q "who imports main.rs" --auto-route
# Output: Query intent: structural (confidence: 0.80)
#         Consider: loctree query --kind who-imports --target main.rs

# Deep exploration with all onion layers
rmcp-memex dive -n memories -q "dragon" --verbose

意图类型:

意图 触发关键字 推荐模式
时间 时间,日期,昨天,前,2024 混合(日期增强)
结构化 导入、依赖、模块、谁使用 BM25+loctree建议
语义 相似、相关、解释 矢量
精确 “引号字符串” BM25
混合 (默认) 矢量+BM25融合

CLI命令

# Index with onion slicing (default)
rmcp-memex index -n memories /path/to/data/ --slice-mode onion

# Index with progress bar and ETA
rmcp-memex index -n memories /path/to/data/ --progress

# Index with flat chunking (backward compatible)
rmcp-memex index -n memories /path/to/data/ --slice-mode flat

# Search in namespace
rmcp-memex search -n memories -q "best moments" --limit 10

# Search only in specific layer
rmcp-memex search -n memories -q "query" --layer outer

# Drill down in hierarchy (expand children)
rmcp-memex expand -n memories -i "slice_id_here"

# Get chunk by ID
rmcp-memex get -n memories -i "chunk_abc123"

# RAG search (cross-namespace)
rmcp-memex rag-search -q "search term" --limit 5

# List namespaces with stats
rmcp-memex namespaces --stats

# Export namespace to JSON
rmcp-memex export -n memories -o backup.json --include-embeddings

预处理(噪声滤波)

自动消除对话输出中约36-40%的噪音:

  • MCP工具工件(<function_calls>, <invoke>等等)
  • CLI输出(git状态、货物构建、npm安装)
  • 元数据(UUID、时间戳→ 占位符)
  • 空/样板内容
# Index with preprocessing
rmcp-memex index -n memories /path/to/export.json --preprocess

精确匹配重复数据删除

基于SHA256的重叠导出数据消除(例如,包含6个月数据的季度导出):

# Dedup enabled (default)
rmcp-memex index -n memories /path/to/data/

# Disable dedup
rmcp-memex index -n memories /path/to/data/ --no-dedup

统计输出:

Indexing complete:
  New chunks:          234
  Files indexed:       67
  Skipped (duplicate): 33
  Deduplication:       enabled

代码结构

rmcp-memex/
├── src/
│   ├── lib.rs              # Public API & ServerConfig
│   ├── bin/
│   │   └── rmcp-memex.rs   # CLI binary (serve, index, search, get, expand, etc.)
│   ├── handlers/
│   │   └── mod.rs          # MCP request handlers
│   ├── security/
│   │   └── mod.rs          # Namespace access control
│   ├── rag/
│   │   └── mod.rs          # RAG pipeline + OnionSlice architecture
│   ├── preprocessing/
│   │   └── mod.rs          # Noise filtering for conversation exports
│   ├── storage/
│   │   └── mod.rs          # LanceDB + Tantivy (schema v3 with content_hash)
│   ├── embeddings/
│   │   └── mod.rs          # MLX/FastEmbed bridge
│   └── tui/
│       └── mod.rs          # Configuration wizard
└── Cargo.toml

Claude/MCP集成

添加 ~/.claude.json:

{
  "mcpServers": {
    "rmcp-memex": {
      "command": "rmcp-memex",
      "args": ["serve", "--security-enabled"]
    }
  }
}

VetCoders与人工智能代理共同创建Vibecraft(c)2025 LibraxisAI团队 合著者: 马修 & 克劳迪