Token Smithers：MCP令牌压缩代理

Claude Code  <-->  Token Smithers (proxy)  <-->  Backend MCP Server

                    ┌──→  MCP Server A  (filesystem — raw data)
                    │
Your AI Tool  ──────┼──→  MCP Server B  (GitHub — raw API responses)
                    │
                    ├──→  MCP Server C  (database — query results)
                    │
                    └──→  MCP Server D  (code intelligence — already optimized)

pip install "token-smithers[learning] @ git+https://github.com/shacharbard/token-smithers.git@stable"

pipx install "token-smithers[learning] @ git+https://github.com/shacharbard/token-smithers.git@stable"

pip install "token-smithers[learning] @ git+https://github.com/shacharbard/token-smithers.git@stable"

pip install "token-smithers[learning] @ git+https://github.com/shacharbard/token-smithers.git@stable"
token-smithers setup                      # Pick which MCP servers to compress
# ... use your AI coding tool normally ...
token-smithers stats                      # Check your savings

token-smithers setup

Found 2 MCP config files:

  Global (~/.claude.json): 3 servers
    1. github        npx -y @modelcontextprotocol/server-github
    2. slack         npx -y @anthropic/server-slack
    3. memory        npx -y @modelcontextprotocol/server-memory

  Project (.mcp.json): 2 servers
    4. filesystem    npx -y @modelcontextprotocol/server-filesystem .
    5. my-database   my-db-server --port 5432

Which servers should token-smithers compress? (comma-separated, or 'all')
> 1,4,5

token-smithers setup --undo

Unwrapping 3 servers:
  github        → restored to: npx -y @modelcontextprotocol/server-github
  filesystem    → restored to: npx -y @modelcontextprotocol/server-filesystem .
  my-database   → restored to: my-db-server --port 5432

Updated:
  ~/.claude.json — 1 server restored
  .mcp.json      — 2 servers restored

token-smithers stats

  "Excellent..."

  === Token Smithers — Session Stats ===
  Events:     142
  Original:   284,000 tokens
  Compressed: 156,200 tokens
  Saved:      127,800 tokens (45.0%)

  Smithers, we saved 127,800 tokens. Not a single one squandered.

  === Per-Strategy Breakdown ===
  Strategy                        Count   Original Compressed
  ------------------------------ ------ ---------- ----------
  whitespace_normalizer              142     284000     241400
  toon_compressor                     38     120000      54000
  yaml_transcoder                     67      98000      73500
  smart_truncation                    12      42000      28700

token-smithers status-line

Smithers: 1.2M (42%) | today: 52.0K (38%)     # green when >= 40%
Smithers: 340.5K (23%) | today: 12.1K (19%)   # blue when >= 20%
Smithers: 8.2K (6%) | today: 1.4K (5%)        # red when < 20%

{
  "statusLine": "token-smithers status-line"
}

# Show Smithers savings in your prompt
export PS1='$(token-smithers status-line) \$ '

set -g status-right '#(token-smithers status-line)'

# ~/.token-smithers/configs/filesystem.yaml
backend:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/dir"]

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/dir"]
    }
  }
}

{
  "mcpServers": {
    "filesystem": {
      "command": "token-smithers",
      "args": ["--config", "~/.token-smithers/configs/filesystem.yaml"]
    }
  }
}

{
  "mcpServers": {
    "filesystem": {
      "command": "token-smithers",
      "args": ["--config", "~/.token-smithers/configs/filesystem.yaml"]
    },
    "github": {
      "command": "token-smithers",
      "args": ["--config", "~/.token-smithers/configs/github.yaml"]
    }
  }
}

# Check what the original command was
cat ~/.token-smithers/configs/filesystem.yaml

backend:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/dir"]

# Backend MCP server to proxy
backend:
  command: "npx"                    # Command to start the backend server
  args: ["-y", "@modelcontextprotocol/server-filesystem", "."]
  env: {}                           # Extra environment variables

# Tool filtering (passthrough by default)
filter:
  mode: "passthrough"               # passthrough | allowlist | blocklist
  tools: []                         # Tool names to allow/block
  patterns: []                      # Regex patterns to match tool names

# Compression pipeline
compression:
  enabled: true
  size_gate_threshold: 2000         # Skip compression for results under this token count
  adapters:                         # Ordered list — first match wins
    - name: whitespace_normalizer   # Always-on cleanup
    - name: null_field_elider
    - name: path_prefix_deduplicator
    - name: timestamp_normalizer
    - name: log_level_filter        # Off by default
      enabled: false
    - name: error_stack_compressor
      enabled: false
    - name: code_comment_stripper
      enabled: false
    - name: sentence_scorer         # Requires 'prose' extra
      enabled: false
    - name: rle_encoder
    - name: toon_compressor         # JSON arrays -> columnar format
    - name: yaml_transcoder         # Non-tabular JSON -> YAML
    - name: file_redirect           # Oversized results -> temp file
      enabled: false
    - name: smart_truncation        # Safety net (always last)

# Schema virtualization (DietMCP-style)
schema_virtualization:
  enabled: false                    # Enable to compress tool schemas
  tier: 2                           # 1=lossless, 2=brief descriptions, 3=one-liner notation
  frequent_call_threshold: 3        # Tools called >= N times stay at Tier 1

# Semantic result caching
semantic_cache:
  enabled: false                    # Enable for similarity-based result caching
  similarity_threshold: 0.85        # 0.0-1.0, higher = stricter matching
  max_entries: 1000
  ttl_seconds: 86400                # Cache entry lifetime (null = no expiry)

# Cross-session learning
learning:
  enabled: true                     # SQLite persistence for usage stats + caching
  db_path: "~/.token-smithers/learning.db"

# Dashboard / metrics
dashboard:
  enabled: true
  metrics_file_path: "~/.token-smithers/metrics.json"

# System prompt optimization
system_prompt:
  enabled: true
  compress_instructions: true       # Compress backend server instructions at startup

# Statistical reranker
reranker:
  enabled: true                     # Reorder tools/list by usage frequency
  max_tools: 500
  recency_weight: 0.3

# Caching
cache:
  schema_cache_ttl: 3600            # Tools/list cache TTL (seconds)
  call_cache_max: 200               # Max exact-match cached results
  diff_store_max: 100               # Max semantic diff entries

# Observability
observability:
  metrics_to_stderr: true           # Emit [Token Smithers] log lines per compression event
  log_level: "INFO"

src/token_sieve/
  domain/          # Pure Python domain core (zero external deps)
    model.py       #   ContentEnvelope, CompressionEvent, value objects
    ports.py       #   CompressionStrategy, DeduplicationStrategy protocols
    pipeline.py    #   CompressionPipeline service
    ports_cache.py #   SemanticCachePort protocol
    ports_learning.py # LearningStore protocol
    ports_schema.py   # SchemaVirtualizerPort protocol
    metrics.py     #   InMemoryMetricsCollector
  adapters/        # Implementations (external deps allowed)
    compression/   #   17 compression strategy adapters
    cache/         #   Call cache, semantic cache, invalidation, schema cache
    backend/       #   MCP client transport + connector
    learning/      #   SQLite learning store
    schema/        #   Schema virtualizer (DietMCP notation)
    rerank/        #   Statistical reranker + persistence
    dedup/         #   Window-based deduplication
  server/          # MCP proxy server
    proxy.py       #   ProxyServer (MCP handlers, dependency wiring)
    metrics_sink.py #  Stderr metrics formatter
    metrics_writer.py # Periodic JSON metrics file writer
    tool_filter.py #   Allowlist/blocklist tool filtering
  config/          # YAML config loading + Pydantic validation
  cli/             # CLI entry point (proxy, pipe, stats modes)

# Token Smithers.yaml
backend:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/projects"]

backend:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "."]

compression:
  adapters:
    - name: whitespace_normalizer
    - name: null_field_elider
    - name: path_prefix_deduplicator
    - name: timestamp_normalizer
    - name: log_level_filter
      enabled: true                  # Enable log filtering
    - name: code_comment_stripper
      enabled: true                  # Strip code comments
    - name: sentence_scorer
      enabled: true                  # Requires 'prose' extra
    - name: rle_encoder
    - name: toon_compressor
    - name: yaml_transcoder
    - name: smart_truncation

schema_virtualization:
  enabled: true
  tier: 3                            # DietMCP one-liner notation

semantic_cache:
  enabled: true                      # Cache similar read results
  similarity_threshold: 0.90         # Strict matching

backend:
  command: "my-mcp-server"

filter:
  mode: "blocklist"
  tools: ["dangerous_tool"]
  patterns: ["^internal_.*"]          # Regex: block tools starting with "internal_"

# Compress a file directly (no MCP server needed)
cat large-output.json | token-smithers --pipe

# Compress from file
token-smithers --pipe input.txt

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov

# Run integration tests only
pytest -m integration

# Run benchmarks
pytest -m benchmark

# Run security checks yourself
pip install bandit pip-audit
bandit -r src/token_sieve/ -ll    # Static analysis
pip-audit                          # Dependency vulnerabilities