音乐分析与混音系统

🎧 Epistemic DJ

TL;DR: Most music tools are one of two things — an app that just plays you stuff (Spotify), or a full production studio you need years to learn (a DAW). Nothing in between for people who want to shape their listening without becoming a producer. epistemic-dj is that middle ground: it learns what you actually like — not by watching what you skip, but by asking you and by really listening to your tracks — and keeps a record of why, so "why did you play me this" has a real answer instead of a shrug. With that, it can find you more of what you'd genuinely want, and build actual mashups: match the tempo, line up the beats, even pull the vocals off one track and lay them over another track's instrumental. Every step is checked against the real audio, never a genre-tag guess standing in for actually listening. Full story: docs/human/overview.md.

Status: alpha, developers only. This is a working local MCP server you run from source and drive via Claude — not a packaged app, no installer, no stability guarantees between commits. Expect rough edges. See docs/human/setup.md for the real setup process.

Music taste profiling & AI-driven mashup generation system.

Analyze, curate, and create music grounded in learned epistemic profiles. epistemic-dj builds taste models from real music analysis and listener preferences, then uses a calibrated AI (predict → measure → resolve, Brier-scored — the same discipline Empirica uses on itself) to curate and render actual mashups: not a lookup table, and never a metadata/genre guess standing in for listening to the track.

Three core workflows:

1. Epistemic → Sound (original, JS side): Transform your cognitive state into music. High uncertainty? Dissonant, chaotic patterns. Deep focus? Clean, driving beats. Celebrating a win? Full-on euphoric build-ups.

2. Taste Profiling & Curation (Python side): Build epistemic profiles of musical taste from your real Bandcamp collection and YouTube library/playlists, then:

   • Analyze real audio (tempo, energy, valence — never metadata alone)
   • Curate tracks matching your taste profile, with the why attached
   • Calibrate every prediction against real measurement, Brier-scored, so confidence means something

3. Mashup Rendering (Python side, new): Beatmatch and overlay real tracks into an actual rendered mashup — tempo-matched via pitch-preserving time-stretch, alignment-scored via real audio cross-correlation (not a guess), auto-corrected against its own measurement. Offline, calibrated composition — not a real-time DJ-booth tool (see docs/human/overview.md for that distinction). Full-track overlay and selective stem overlay (vocals from one track over another's instrumental, via Demucs) both work today.

Features

Epistemic State → Sound (JS)

• MCP Tools for Claude Code integration
• Pattern Generator - 13 epistemic vectors → Strudel live coding patterns
• Mood Presets - focus, energize, reflect, debug, celebrate
• Web UI - Interactive sliders + embedded Strudel REPL

Music Taste Profiling & Curation (Python)

• Bandcamp + YouTube integration - real collection/library ingestion, cookie/header auth (no official personal-collection API exists for either)
• Real audio analysis - tempo, energy (kinetic_energy), mood (valence) fit via a DEAM-trained regression, and key/mode (Krumhansl-Schmuckler correlation over chroma) — never metadata guessing
• Calibration loop - every prediction (energy, tempo, tempo compatibility, key compatibility) is logged, resolved against real measurement, and Brier-scored — self-correcting confidence, not a static number
• Taste profiling - findings/patterns/anti-patterns as real Empirica- style artifacts, so "why did you play me this" has an actual answer

Mashup Rendering (Python, new)

• Beatmatching - pitch-preserving time-stretch (librosa phase vocoder) to a real measured target tempo, octave-aware (half/double-time) compatibility scoring
• Harmonic mixing - real key/mode detection (chroma + Krumhansl-Schmuckler correlation), Camelot-wheel compatibility scoring between two tracks, same predict/measure/resolve calibration loop as tempo
• Real renders - full-track overlay (render_mashup), fixed vocals-over-instrumental stem overlay (render_stem_mashup), and arbitrary independent stem combination from both tracks (render_multistem_mashup, e.g. drums+bass from one track under vocals+other from another) all write actual audio files (epistemic-dj/renders/)
• Alignment scoring - genuine cross-correlation of onset-strength envelopes measures how well two tracks' beats actually line up, not a guess — and the render auto-corrects using its own signal
• Beat-snapped render start (render_mashup only, not yet the stem-based renders) - the offset snaps to the nearest real detected beat instead of trusting an arbitrary fixed-second position (default on, snap_offset_to_beat=False to disable) — not full downbeat/phrase detection, that needs a heavier model
• Stem-separation quality, calibrated - real measured leakage score (pairwise onset-envelope correlation across a track's own separated stems) surfaced on every stem-based render, AND closed through the same predict → measure → resolve → Brier loop as tempo/key (cheap short-excerpt separation predicts the worst-leaking stem pair, verified this session to hold up on real audio unlike a metadata guess)
• EQ-aware overlay (render_mashup's highpass_b_hz, opt-in) - high-passes the overlaid track's sub-bass before summing (standard DJ "bass swap"), instead of a flat gain-sum — measured 56-60% reduction in bass-band spectral clash on real audio (mixing.render. spectral_band_overlap), reported before/after so the effect is checkable, not just asserted
• Robust tempo measurement - checkpoint spread beyond threshold triggers a denser re-measure rather than trusting a single-window octave guess (two signal-processing octave-correction heuristics were tried and both made things worse on real audio — more real measurements won, not cleverness)
• Next: YouTube upload pipeline; Bandcamp export (lowest priority, no confirmed public upload API)

Generative Composition (upcoming, JS)

• Strudel Integration - Algorithmic music composition via live-coding
• Profile-Driven Generation - Generate original music grounded in taste models
• Cross-Fade Patterns - Smooth transitions between epistemic states

Quick Start

# JS side: Epistemic State -> Sound
npm install
node src/generator/epistemic-to-strudel.js --mood celebrate   # pattern generator
node src/mcp/server.js                                        # MCP server
npx serve src/web                                              # web UI

# Python side: taste profiling, calibration, mashup rendering
cd python
uv sync
uv run epistemic-dj-mcp

# Optional: stem separation (render_stem_mashup) needs the extra --
# heavy, GPU-dependent deps kept out of the default install
uv sync --extra separation

See docs/human/setup.md for connecting your real Bandcamp/YouTube accounts (both need a one-time manual credential step — there's no OAuth flow for either).

MCP Tools (JS side)

generate_pattern

Generate a Strudel pattern from epistemic vectors.

{
  "vectors": {
    "know": 0.7,
    "uncertainty": 0.3,
    "engagement": 0.8,
    "clarity": 0.6,
    "coherence": 0.7,
    "signal": 0.5,
    "completion": 0.4
  },
  "mode": "full"
}

generate_mood

Generate a pattern for a mood preset.

{
  "mood": "focus"
}

explain_mapping

Understand how vectors map to music.

crossfade_pattern

Generate transition patterns between states.

MCP Tools (Python side)

The full tool list is large (Bandcamp/YouTube search+ingestion, taste findings/patterns/mixtapes, audio analysis, calibration, rendering) — see python/epistemic_dj/mcp_server.py for the authoritative, documented list. Highlights:

• bandcamp_get_collection / youtube_get_playlist_tracks — real source ingestion
• audio_analyze_track — real tempo/energy/valence from actual audio
• calibration_predict_tempo / calibration_resolve / calibration_brier — the predict → measure → resolve → score loop
• calibration_predict_tempo_compatibility / calibration_resolve_tempo_compatibility — pairwise mixability, audio-grounded on both ends
• audio_analyze_key — real key/mode/Camelot code from actual audio
• calibration_predict_key_compatibility / calibration_resolve_key_compatibility — pairwise harmonic mixability (Camelot wheel distance), same calibration discipline as tempo
• calibration_predict_stem_leakage / calibration_resolve_stem_leakage — worst pairwise stem-separation leakage, predicted from a cheap excerpt, resolved against a fuller one, Brier-scored
• render_mashup — real time-stretched, beat-aligned overlay render, writes actual .wav output; highpass_b_hz opts into EQ-aware overlay
• render_stem_mashup — Demucs-separated vocals overlaid on another track's instrumental, same beatmatch/alignment machinery as render_mashup (requires uv sync --extra separation)
• render_multistem_mashup — arbitrary stem selection from BOTH tracks (e.g. drums+bass from A, vocals+other from B), same machinery, independent overlay control instead of a fixed vocals/instrumental split (requires uv sync --extra separation)

Epistemic → Musical Mappings (JS side)

Claude Code Integration

Add to your Claude Code MCP config:

{
  "mcpServers": {
    "epistemic-dj": {
      "command": "node",
      "args": ["/path/to/epistemic-dj/src/mcp/server.js"]
    }
  }
}

Then in Claude:

Generate a pattern for my current epistemic state:
- know: 0.6 (decent understanding)
- uncertainty: 0.4 (some unknowns)
- engagement: 0.8 (highly focused)

Roadmap

Epistemic State → Sound (JS, stable)

• [x] Pattern generator (vectors → Strudel)
• [x] MCP server with tools
• [x] Web UI with sliders
• [ ] WebSocket bridge for real-time updates
• [ ] Empirica integration for automatic state tracking

Taste Profiling & Curation (Python, in progress)

• [x] Bandcamp + YouTube source integration (real ingestion, not mocked)
• [x] Real audio analysis (tempo/energy/valence from actual audio)
• [x] Calibration loop (predict → measure → resolve → Brier score), generalized beyond a single quantity
• [ ] Full onboarding-interview taste profile builder
• [ ] LLM curator matching profiles to sources at scale

Mashup Rendering (Python, in progress)

Full phase-by-phase detail: docs/dev/architecture.md.

• [x] Tempo prediction + pairwise compatibility, audio-grounded
• [x] Real time-stretched, beat-aligned overlay renders + auto-alignment
• [x] Stem separation (Demucs) for selective overlay — render_stem_mashup
• [x] Robust tempo measurement (adaptive checkpoint densification on instability, rather than a single-window octave-correction guess)
• [x] Independent multi-stem overlay (render_multistem_mashup) + stem-separation leakage diagnostic
• [x] Harmonic mixing: key/mode detection + Camelot-wheel compatibility scoring, calibrated (predict → measure → resolve)
• [x] Beat-snapped render offset (nearest real detected beat, not an arbitrary fixed second) — true downbeat/phrase detection remains a stretch goal (needs madmom or similar, not plain librosa)
• [x] Calibrate stem-separation leakage: worst pairwise leakage score, predict → measure → resolve → Brier, cheap-excerpt tolerance (0.15) set from real measured deltas, not guessed
• [x] EQ-aware overlay: high-pass the overlaid track's sub-bass before summing (render_mashup's highpass_b_hz, opt-in), measured 56-60% bass-band clash reduction on real audio
• [ ] YouTube upload pipeline
• [ ] Bandcamp export (lowest priority — no confirmed public upload API)

Generative Composition (planned)

• [ ] Profile-driven music generation (compose new tracks from taste models)
• [ ] Feedback loop (music influences cognition?)
• [ ] Cross-practice composition (mix epistemic state with taste profiles)
• [ ] Empirica artifact integration (log music as a form of thought)

Project structure

Two languages, on purpose — see docs/dev/architecture.md for why:

• src/ — existing JS/ESM MCP server: epistemic vectors → Strudel patterns (this doc, above)
• python/ — Python MCP server: Bandcamp + YouTube integration, real audio analysis, calibration loop, mashup rendering. Quickstart: cd python && uv sync && uv run epistemic-dj-mcp
• docs/human/ — product narrative and vision, for people
• docs/dev/ — technical architecture, for engineers and future Claude sessions

License

MIT

Credits

Built with:

• Strudel - Live coding music in the browser
• MCP SDK - Model Context Protocol
• Empirica - Epistemic self-assessment