说话人日志与语音转写服务器

Speaker Diarization Server

GPU-accelerated speaker diarization, transcription, and emotion recognition with a web interface, a REST API, WebSocket streaming, and an MCP server for AI agents. Combines pyannote.audio + faster-whisper or NVIDIA Parakeet + emotion2vec into one FastAPI service with persistent speaker identity across conversations.

Web interface

The server comes with a web interface at /ui/ (opening the server's address in a browser goes there). It is plain HTML, CSS and JavaScript served by the server itself, with no build step and nothing loaded from the internet, and it only uses the public API, so it doubles as an example client.

• Live — record from a microphone, or stream an audio file at real-time pace (or 2×/4×), and watch lines arrive with their speakers and emotions; voices start as Unknown, are renamed as the session recognises them (speaker_update), and can be named on the spot. Shows the delay from the end of speech to each line, and an event log of the WebSocket messages.
• Transcribe a file — upload a recording and see the result with the processing speed.
• Conversations — every upload and live recording: the recording with a timeline of who spoke when, talk time and emotions per speaker, and the transcript with words under 50% confidence underlined and the word being played highlighted. Name a voice, move a line to someone else, leave a line out of a voice profile, correct an emotion, recalculate emotions, reprocess, export (text, subtitles, JSON).
• Speakers — enrol someone from a sample or a recording made in the browser, rename, delete; play Unknown voices, merge one into the person it sounds like, clean up the ones heard only briefly; see what each person's emotion corrections have taught.
• Backups — save, restore, checkpoint, download and import profiles.
• Settings — the runtime settings (/api/v1/settings/voice) and server status.

Browsers only allow the microphone on HTTPS pages (or on localhost): to record from another machine, run ./generate_cert.sh once to create a self-signed certificate in certs/, restart the server, open https://<host>:8418/ and accept the certificate warning (see Remote access).

Screenshots

A demo with recordings from the AMI Meeting Corpus (CC BY 4.0): two people enrolled from short samples, a third named from an uploaded meeting, then a later meeting streamed live, in which all three are recognised by name and the voice nobody named stays the same Unknown.

An older, separate example client, the Next.js frontend, predates this interface and no longer matches the API.

Features

• Persistent speaker identity — enroll a voice once, recognized across every future recording.
• Retroactive updates — identify one segment and every past segment with that voice re-labels automatically.
• Transcription — NVIDIA Parakeet-TDT (the default for English: faster, lower WER on meetings) or faster-whisper (ASR_BACKEND=whisper: large-v3 or any Whisper variant, 99 languages), word-level timestamps.
• Emotion recognition — emotion2vec+ per segment, 9 categories, read for each speaker with their own corrections (personal calibration).
• Live streaming — WebSocket audio cut at pauses by a speech detector (Silero VAD); each voice is tracked through the session and named from everything it has said, so strangers stay one Unknown and known people are recognised within seconds.
• Batch processing — multi-GPU script for bulk ingest (run_batch.sh).
• Web interface at /ui/ — live recording, uploads, conversations, speakers, backups and settings in the browser (see Web interface).
• REST API at /api/v1/* with interactive Swagger docs at /docs.
• MCP server at /mcp (JSON-RPC 2.0 / HTTP) exposing 11 tools for Claude Desktop, Flowise, or any MCP client.
• Backup/restore — export speakers + segments + settings as JSON profiles and checkpoints.
• SQLite WAL — handles thousands of conversations with concurrent reads.

Tech Stack

Requirements

• GPU: NVIDIA with CUDA 12.x. Tested on RTX 3090 (24 GB) and RTX 5090. Runs on 6 GB+ cards if you pick a smaller Whisper model.
• VRAM ballpark: 2–3 GB diarization/embedding + 2 GB emotion2vec + Whisper (0.5–4 GB depending on model). large-v3 + emotion ≈ 8–9 GB total (turbo ≈ 6–7 GB).
• OS: Linux (tested on Ubuntu). macOS/Windows via Docker Desktop + WSL2.
• Other: ffmpeg, git. Python 3.11 or 3.12 if running outside Docker.
• HuggingFace token with pyannote terms accepted (see below).

Quick Start

1. HuggingFace access

Create an account + token at huggingface.co/settings/tokens, then accept the terms for:

• pyannote/speaker-diarization-community-1
• pyannote/embedding

2. Docker (recommended)

git clone https://github.com/snailbrainx/speaker-diarization-server.git
cd speaker-diarization-server
cp .env.example .env           # then edit .env and set HF_TOKEN
docker compose up --build      # first run downloads ~3-5 GB of models

3. Local development

sudo apt-get install -y ffmpeg git
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env           # edit HF_TOKEN
./run_local.sh

4. Verify

• Web interface: http://<host>:8418/ (https:// once a certificate is in certs/)
• Swagger UI: http://<host>:8418/docs
• Health: curl http://<host>:8418/api/v1/status
• MCP: http://<host>:8418/mcp

Remote access

No built-in auth. Use an SSH tunnel for remote hosts:

ssh -L 8418:localhost:8418 user@host

For network deployments put it behind a reverse proxy (nginx + HTTPS + auth). WebSocket microphone capture in browsers requires HTTPS/WSS — place cert.pem/key.pem in certs/ and the server starts in HTTPS mode automatically.

On a trusted local network a self-signed certificate is enough:

./generate_cert.sh                    # localhost, this machine's name and IP addresses
./generate_cert.sh 192.168.1.20 nas   # or add names and addresses of your own
docker compose restart

Browsers warn about a self-signed certificate once per address; accept it, or import certs/cert.pem as trusted on the devices you use. The key is readable only by its owner, and the container reads it as uid 1000.

Configuration

All configuration is environment variables (see .env.example for the curated list). Everything is optional except HF_TOKEN.

Required

Storage & networking

Live streaming limits

Voice & emotion tuning (also editable at runtime via /api/v1/settings/voice)

Models

GPU

Speed & speaker assignment

On by default. To get the original behaviour back, set WHISPER_BATCH_SIZE=0, WORD_LEVEL_SPEAKERS=false, PYANNOTE_ALLOW_TF32=false, SPEAKER_ASSIGNMENT=segment, SNAP_TO_TURNS=false and LIVE_SPEAKER_TRACKING=false.

Measured on public test sets (RTX 3090; AMI = 16 test meetings, VoxConverse = 40 dev clips):

¹ 6% less processing time than the speed settings alone, measured side by side on the same GPU. (Measured with WHISPER_BATCH_SIZE=16; batch 4, the default, is equally fast and gives the same transcripts.) ² Each AMI series has the same four people in four meetings: speakers were named in the first meeting with the identify endpoint, then the other three meetings were processed; a meeting from another series stood in for unenrolled speakers.

Speech-to-text: Parakeet or Whisper

For English the default is NVIDIA's Parakeet-TDT 0.6B v2 through NeMo (in the Docker image); ASR_BACKEND=whisper runs faster-whisper instead, and is the default when NeMo is not installed or WHISPER_LANGUAGE is not en. Both return the same segments (text, word timestamps, word confidences), so speaker assignment, emotion and storage are the same. Parakeet cuts long recordings at pauses into chunks of up to 60 s that cover the whole file, transcribed 8 at a time; sentences are joined into segments like Whisper's. v2 is English-only (PARAKEET_MODEL=nvidia/parakeet-tdt-0.6b-v3 covers 25 European languages); Whisper covers 99.

This release against the previous one, on the same GPU. Speed is server-side processing time, with each file sent to each version in turn; LibriSpeech = 40 test-clean clips, VoxConverse = 40 dev clips, AMI = the 16 test meetings (9.1 h).

Held-out sets, never used to choose a setting (previous release / this release with Parakeet): WER on 200 LibriSpeech test-clean clips 1.69% / 1.94%, on 200 test-other clips 3.24% / 2.76%, on the 18 AMI dev meetings 23.7% / 21.8%; DER on 40 other VoxConverse clips 12.5% / 8.7%, on AMI dev 30.6% / 24.1%.

Live, six 150 s VoxConverse clips with 3–5 speakers streamed in real time: median delay from speech to a labelled line 17.6 s → 4.5 s, worst 116 s → 15 s; DER 23.2% → 21.7%, speaker confusion 12.6% → 15.2%.

Speaker confusion rises a little because turn edges label ~8 points more of the speech, and those edges are where pyannote itself is least sure; DER, which counts confusion, falls 3–8 points. Parakeet trails Whisper on clean read speech (mostly names and compound words) and leads on harder speech and meetings.

³ Of all their speech, in the identification test above; 2.0% / 2.2% was given another person's name.

Diarization runtime: pyannote or speakrs

DIARIZATION_BACKEND=speakrs runs community-1's own models and clustering through speakrs, a Rust/CUDA implementation, instead of pyannote.audio. The speaker turns are the same: DER is within 0.05 points on every set below, held-out ones included. Diarization takes about a sixth of the time and a tenth of the CPU (a 17.5-minute meeting: 1.5 s instead of 9.3 s). The Docker image builds speakrs from a pinned commit, since it is not on PyPI; its models, converted from community-1, download from Hugging Face on first use. Measured as above, on the same GPU with each file sent to each server in turn:

Voice embeddings

Speaker identification compares voice embeddings: every segment's (from its words' span plus 0.15 s either side), a diarization speaker's (their duration-weighted mean), a stored person's (the mean over everything they were named in). The default model is WeSpeaker ResNet34 (pyannote/wespeaker-voxceleb-resnet34-LM, the speaker model inside pyannote's community-1 pipeline, 256-D); SPEAKER_EMBEDDING_MODEL=pyannote/embedding is the original (512-D). On live chunks of the AMI meetings, scored against voices enrolled in an earlier meeting:

ECAPA-TDNN, TitaNet-L and ReDimNet-B6 were measured the same way; none beat ResNet34 overall. In the file identification test above (speakers named in a first meeting, then three later meetings), named people's speech given their own name went from 92.6% to 97.5% of what was labelled (wrong name 2.3% both; Unknown 5.2% → 0.2%), and unenrolled visitors still got no one's name; transcription and diarization are unchanged. Stored voices made by another model cannot be compared with new ones, so when the model changes (including the first start after upgrading from pyannote/embedding), every named or Unknown speaker's voice is rebuilt from their recordings with the new model, once. A speaker whose recordings are gone keeps their name and segments and is recognised again after one identify. Backups record the model; restoring an older backup rebuilds its voices the same way, as part of the restore (if that fails, the restore fails and nothing is replaced).

Live speaker identification

A live chunk holds a few seconds of speech, too little to name someone reliably on its own. So a live session keeps one track per voice (LIVE_SPEAKER_TRACKING=true): each chunk's diarization speakers join the session's tracks by voice, a track's voice is the mean of everything it has said so far, and a track is named only when that voice clearly matches one stored speaker (similarity ≥ 0.62 and 0.05 above the next; one short line can be enough). Until then it is one Unknown of its own, the same for all its lines, and stored at once: a voice heard in one session is the same Unknown in the next. When a track gets a name, its earlier lines are renamed and the client receives a speaker_update message (segment_ids, speaker_id, speaker_name, is_known, and emotions: the lines' emotions read again with the speaker's corrections, if they have any). Identifying any one line of an Unknown names all of them, in every conversation.

Since every new voice is kept, stray ones (a word from the TV, a cough) collect as Unknowns too. The web interface's Speakers page lists them with how long they spoke, plays one of their lines, shows the enrolled person or other Unknown each one sounds like (GET /api/v1/speakers/unknown/matches) to merge it in one click, and deletes the ones heard for only a few seconds (DELETE /api/v1/speakers/unknown/all?max_seconds=5).

Speech is found by a speech detector (Silero VAD) rather than by loudness: quiet voices are kept, and fans, traffic, hum or keyboard noise never start a transcription (0% of such audio was taken for speech in a test, even when loud). The session clock is the audio itself: skipped silence still counts, so live timestamps are real times, and the saved recording keeps the silence so it lines up with them.

Measured the way the feature is used, on four groups of four people from the AMI meetings (each group meets four times), streamed in real time over the WebSocket, first 8 minutes of each meeting: nobody known in a first meeting, every Unknown then named after the person it mostly is (identify), then a later meeting of the same people, then a meeting of four people nobody named. Averages over the four groups:

¹ Three groups; in the fourth (a quiet recording) almost nothing was transcribed: its speech fell under the old loudness threshold. Most of the difference between the first two columns is the recorder (speech detection, timing), the voice model and the speech-to-text; tracking then takes strangers' wrong names to zero and halves the Unknowns to name. Known people are typically named in their first or second line (under 2 s of their speech).

People who talk in short phrases give each chunk a second or so of speech, and one such line's voice can be only about 0.5 similar to the same person's next line, and about 0.6 to their profile (in a test with a browser microphone). The thresholds above were set for that, on the same meetings replayed offline with every 0.8–2 s line as a chunk of its own: named people are named in 65% of their speech with 1.8% wrong, where the previous thresholds (0.70, and only after 2 s of a voice's speech) named 50% and a session of short replies could end without naming anyone. On the meetings as recorded the results are unchanged, and strangers are never given a name, even when only one person is enrolled.

Overlapping speech (optional)

One speech-to-text pass over mixed audio mostly keeps the louder voice where two people talk at once; on AMI meetings a quarter of the words are spoken over someone else, and half of those were missing. With OVERLAP_SEPARATION=true, every stretch where the diarization hears two people at once (plus 1 s either side) is separated into two voices (MossFormer2, Apache-2.0), both are transcribed, and each voice is compared with every speaker's voice: a voice that sounds like another speaker adds that speaker's words as new segments, and the main speaker's words the transcript missed join their segment. Measured as above (16 AMI test meetings; held out: the 18 AMI dev meetings and 40 other VoxConverse clips):

It is off by default: it roughly halves the speed on meetings, and about 0.5% more of the speech goes to the wrong person (the second voice is sometimes given to the wrong speaker), against more of what was said being transcribed. Segments added this way are marked as overlapping speech (is_overlap): their stretch of the recording holds two voices, so they never go into anyone's voice profile.

How it Works

upload / WS chunk (live: cut at pauses by Silero VAD)
      │
      ▼
┌─────────────────┐   ┌──────────────────┐
│ Parakeet/Whisper│   │ pyannote         │
│ transcription   │   │ diarization      │   run in parallel
│ (text + words)  │   │ (speaker turns)  │   (ThreadPoolExecutor)
└────────┬────────┘   └────────┬─────────┘
         └────────────┬────────┘
                      ▼
        segment alignment by timestamp
                      │
                      ▼
        voice embedding (WeSpeaker ResNet34) per segment
                      │
                      ▼
        cosine match vs stored speakers
        (files: once per diarization speaker, from its centroid;
         live: once per session track, from all its speech so far)
        │                              │
        ▼                              ▼
   known speaker                 Unknown_*  (one per new voice)
                      │
                      ▼
        optional: overlapping voices separated and transcribed
                      │
                      ▼
        emotion2vec (9 emotions), read with the speaker's corrections
                      │
                      ▼
             ConversationSegment  →  SQLite (WAL)

Key behaviors:

• Parallel speech-to-text + pyannote — ~2× faster than sequential.
• Decoded once — each file is decoded and resampled to 16 kHz once, and Parakeet, pyannote and the voice embeddings all work on that copy in memory. Given a file path, pyannote re-reads the file for every window it embeds: 2.3× slower on a 17-minute meeting, with identical results.
• Turn edges (SNAP_TO_TURNS) — a segment's start and end are widened to its speaker's diarization turn (word timestamps cover only the voiced core of the words). Voice embeddings still come from the words' own span, so this changes timing only, never who is identified.
• One-speaker mode — diarize=false (upload form field, or the live start message) skips pyannote when one person is talking; the speech is still matched against stored voices.
• Live turns — speech (Silero VAD, not loudness) is processed at each pause; a turn with no pause for STREAM_LONG_TURN_SECONDS (noisy rooms, crosstalk) is cut at its quietest point instead of being held until the speakers stop.
• Live tracks (LIVE_SPEAKER_TRACKING) — each voice in a live session is followed across chunks and named once its speech so far clearly matches a stored speaker; earlier lines are renamed (speaker_update).
• Context padding — 0.15 s of audio on each side of a segment makes the embedding robust to background noise.
• Auto-enrollment — an unidentified voice gets one Unknown_* profile (per diarization speaker in a file, per track in a live session) so it accumulates samples. Identify one of its segments and every segment with that Unknown_* is relabelled.
• Cluster mode (SPEAKER_ASSIGNMENT=cluster, the default for files) — instead of matching each segment's embedding on its own, each pyannote speaker is matched once from the centroid of all its segments, so one person gets one label per file (one Unknown_N if not enrolled) and a single identify names them everywhere.
• Misidentified-flag — segments flagged is_misidentified are excluded from embedding averaging, keeping profiles clean.
• Personal emotion calibration — a speaker's emotion corrections adjust how emotion2vec's scores are read for them (see Personal calibration).

Emotion Recognition

Categories (9)

emotion2vec+ large classifies each segment into one of these labels:

angry · disgusted · fearful · happy · neutral · other · sad · surprised · unknown

Personal calibration

emotion2vec+ reads each line on its own, and some voices sit where it hears the same emotion every time: an upbeat speaking voice is often called happy line after line. Correcting a few of a speaker's lines teaches the server how to read that speaker.

emotion2vec's nine scores are a softmax over a linear layer applied to the 1024-D utterance embedding stored with every segment, so each corrected line gives the model's raw scores and the emotion it really was, without its audio. From a speaker's corrections the server fits one offset per emotion, pulled towards zero so that a few corrections move only the doubtful lines, and reads each later line of that speaker with those offsets: an emotion the model calls too often for them is turned down, one it misses is turned up. Clear emotions stay what they are.

1. Three of Ann's ordinary lines come back happy; correct them: POST /api/v1/conversations/{id}/segments/{sid}/correct-emotion?corrected_emotion=neutral.
2. Ann's later lines (and "Recalculate emotions" on earlier conversations) read her ordinary lines as neutral; her angry, sad or surprised lines are unchanged.
3. learn=false fixes one line without learning from it; PATCH .../emotion-misidentified leaves an existing correction out of the learning; DELETE /api/v1/conversations/speakers/{id}/emotion-profiles forgets a speaker's corrections (the lines keep their emotions). GET .../emotion-profiles shows what was learned.

Measured on RAVDESS (24 actors, 8 acted emotions, 1,433 clips): each actor's corrections, then their other clips read by the server, against emotion2vec on its own and against the per-emotion profiles used before (an emotion profile and a voice profile per emotion, compared by embedding similarity):

One person's lines are alike whatever they feel, so the previous profiles matched nearly every line once three were corrected to neutral (angry clips recognised 96% → 6%). Acted clips are seldom misread the same way twice, so RAVDESS only shows that calibration costs nothing; its gain is on a voice the model misreads consistently. On conversational recordings of a speaker it called happy throughout, three ordinary lines corrected to neutral turned their other ordinary lines neutral and left their angry, disgusted, sad and surprised lines as they were.

Example — a line read with corrections

{
  "text": "Is everything okay?",
  "emotion_category": "neutral",
  "emotion_confidence": 0.71,
  "detector_breakdown": {
    "emotion2vec": { "emotion": "happy", "confidence": 0.67 },
    "personal_calibration": {
      "corrections": 3,
      "offsets": { "angry": -0.02, "disgusted": -0.01, "fearful": -0.01, "happy": -0.62, "neutral": 0.69,
                   "other": -0.01, "sad": -0.01, "surprised": -0.01, "<unk>": 0.0 }
    },
    "final_decision": { "emotion": "neutral", "confidence": 0.71,
                        "reason": "emotion2vec read with 3 correction(s) of this speaker" }
  }
}

Lines of a speaker without corrections carry emotion2vec's answer and no breakdown.

REST API

Full interactive docs at /docs (Swagger UI). The OpenAPI JSON is at /openapi.json. Endpoint groups:

• GET /api/v1/status — health + GPU.
• GET /api/v1/speakers, POST /api/v1/speakers/enroll, PATCH /api/v1/speakers/{id}/rename, DELETE /api/v1/speakers/{id}, DELETE /api/v1/speakers/unknown/all — list (with seconds of speech and when last heard), enroll, rename, delete (?max_seconds= deletes only Unknowns heard for at most that long).
• GET /api/v1/speakers/{id}/segments — a speaker's segments, newest (order=recent) or longest first; GET /api/v1/speakers/unknown/matches — each Unknown with the closest enrolled person and the closest other Unknown.
• GET/DELETE /api/v1/conversations/speakers/{id}/emotion-profiles — a speaker's emotion corrections and what was learned from them (calibration), or forget them. GET/PATCH .../emotion-threshold and PATCH .../emotion-profiles/{emotion}/{threshold,voice-threshold} are deprecated and have no effect.
• GET /api/v1/conversations, GET/PATCH/DELETE /api/v1/conversations/{id} — list, read, rename, delete conversations.
• POST /api/v1/conversations/{id}/reprocess, POST /api/v1/conversations/{id}/recalculate-emotions — re-run pipelines.
• POST /api/v1/conversations/{id}/segments/{sid}/{identify,correct-emotion}, PATCH .../{misidentified,emotion-misidentified} — segment corrections.
• GET /api/v1/conversations/segments/{sid}/audio — extract segment audio.
• GET /api/v1/conversations/{id}/audio — the whole recording (an upload as processed, a live recording once finished); answers range requests, so players can seek.
• POST /api/v1/process — upload and process a whole file (form field diarize=false skips speaker diarization when one person is talking: faster, one speaker, still matched against the stored voices).
• GET/POST /api/v1/settings/voice, POST /api/v1/settings/voice/reset — runtime-tunable voice settings.
• GET/POST /api/v1/profiles, checkpoints, download, import, restore — backup/restore.
• WS /api/v1/streaming/ws — live audio ingestion: a {"type": "start"} message, then binary PCM chunks (48 kHz float32 by default; "sample_rate": 16000, "format": "int16" in the start message is a sixth of the bandwidth; "diarize": false skips diarization per chunk), then {"type": "stop"}. Speech is processed at each pause, and a turn that runs 15 s without one is cut at its quietest point.

Example — upload + process

curl -X POST http://localhost:8418/api/v1/process \
  -F "audio_file=@meeting.wav"

Response (trimmed to one segment for readability):

{
  "id": 42,
  "title": "Uploaded: meeting.wav",
  "duration": 137.39,
  "status": "completed",
  "num_segments": 24,
  "num_speakers": 2,
  "transcript_segments": [
    {
      "id": 911,
      "conversation_id": 42,
      "speaker_id": 17,
      "speaker_name": "Unknown_1776798145477718",
      "text": "Call me Ishmael. Some years ago, never mind how long precisely...",
      "start_offset": 0.05,
      "end_offset": 6.71,
      "confidence": 1.0,
      "emotion_category": "neutral",
      "emotion_confidence": 1.0,
      "emotion_corrected": false,
      "emotion_misidentified": false,
      "detector_breakdown": null,
      "words": [
        { "word": " Call",    "start": 0.05, "end": 0.57, "probability": 0.91 },
        { "word": " me",      "start": 0.57, "end": 0.75, "probability": 0.99 },
        { "word": " Ishmael.","start": 0.75, "end": 1.37, "probability": 0.99 }
      ]
    }
  ]
}

Unidentified voices are auto-enrolled as Unknown_<ts> so they accumulate samples. Rename retroactively:

# Identify an unknown speaker (retroactively labels every past segment with that voice)
curl -X POST http://localhost:8418/api/v1/conversations/42/segments/911/identify \
  -H 'Content-Type: application/json' \
  -d '{"speaker_name": "Alice", "enroll": true}'

# Correct an emotion (the speaker's later lines are read with it)
curl -X POST "http://localhost:8418/api/v1/conversations/42/segments/911/correct-emotion?corrected_emotion=surprised"

MCP Server

JSON-RPC 2.0 over HTTP at /mcp. Compatible with Claude Desktop, Flowise, and any client that speaks the spec.

{
  "mcpServers": {
    "speaker-diarization": {
      "url": "http://localhost:8418/mcp",
      "transport": "http"
    }
  }
}

Eleven tools: list_conversations, get_conversation, get_latest_segments, list_speakers, rename_speaker, delete_speaker, delete_all_unknown_speakers, identify_speaker_in_segment, update_conversation_title, reprocess_conversation, search_conversations_by_speaker. All mutations (identify / rename) retroactively update past segments so the AI doesn't have to reconcile identities manually.

Data Persistence

speaker-diarization-server/
├── data/
│   ├── recordings/           # permanent audio (uploads + concatenated streams)
│   ├── stream_segments/      # per-conversation live-segment WAVs
│   └── temp/                 # transient
├── volumes/
│   ├── speakers.db           # SQLite (WAL)
│   └── huggingface_cache/    # pyannote + Whisper + emotion2vec
└── backups/
    ├── profile_<name>.json            # full speaker + segment + settings snapshot
    └── checkpoint_<name>_<ts>.json    # timestamped state capture

Docker bind-mounts ./volumes, ./data, ./backups, and ./certs (read-only) into the container. Nothing else is persisted between runs.

Troubleshooting

"HuggingFace token not found" — confirm HF_TOKEN in .env, no quotes/spaces, model terms accepted.

"Unable to load libcudnn_cnn.so.9" — run_local.sh sets LD_LIBRARY_PATH automatically. Manual: pip install nvidia-cudnn-cu12==9.* nvidia-cublas-cu12.

Docker permission errors on data/, volumes/, backups/ — the container runs as uid 1000. If your host uid differs, sudo chown -R 1000:1000 data/ volumes/ backups/ once.

Docker GPU not visible — verify docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi works; if not, reinstall the NVIDIA Container Toolkit.

Wrong GPU selected — set CUDA_VISIBLE_DEVICES to match the nvidia-smi index you want; CUDA_DEVICE_ORDER=PCI_BUS_ID is already pinned for you.

"CUDA out of memory" — lower CLEANUP_VRAM_THRESHOLD_GB so cleanup runs more often, or with Whisper switch WHISPER_MODEL to a smaller variant (small or base). Parakeet speech-to-text alone peaks at ~5.5 GB on long recordings.

DIARIZATION_BACKEND=speakrs needs the speakrs package — the Docker image builds it; it is not on PyPI, so outside Docker build it from source with a Rust toolchain: pip install maturin, then maturin build --release -m crates/speakrs-python/Cargo.toml in a clone of speakrs and pip install the wheel. Its models download from Hugging Face on first use.

ASR_BACKEND=parakeet needs NVIDIA NeMo — the Docker image includes NeMo; outside Docker, pip install 'nemo_toolkit[asr]==3.0.0' (it is in requirements.txt; without it the default is Whisper). Parakeet v2 transcribes English only: for other languages use PARAKEET_MODEL=nvidia/parakeet-tdt-0.6b-v3 (25 European languages) or ASR_BACKEND=whisper.

OVERLAP_SEPARATION=true outside Docker — the separation model comes in the clearvoice package, installed without its own dependencies (they pin old numpy/opencv versions it does not need for this model): pip install --no-deps clearvoice==0.1.2 (the model's own dependencies are in requirements.txt). Without it, files are processed without separation and a warning is logged.

First start after upgrading takes longer — voices stored by the previous voice model (pyannote/embedding) are rebuilt once from their recordings with the current one; the log says how many. Speakers whose recordings were deleted keep their names but need one identify before they are recognised again.

Speakers named before upgrading are recognised less often — profiles auto-enrolled by the old per-segment matching are often built from a single short segment, which may not reach CLUSTER_SPEAKER_THRESHOLD. Identify the speaker once in a recent conversation (the profile is then rebuilt from all of their segments), or set SPEAKER_ASSIGNMENT=segment to keep the old behaviour.

Speaker not recognized — enroll with 10–30 s of clean audio. In cluster mode the threshold is CLUSTER_SPEAKER_THRESHOLD (0.60), live it is the track threshold (0.62); speakers with only a few seconds of enrolled speech may stay Unknown until identified in more recordings. With per-segment matching (SPEAKER_ASSIGNMENT=segment), lower SPEAKER_THRESHOLD to ~0.20 for challenging audio (movies, music).

Browser microphone not working — browsers allow the microphone only on HTTPS pages or localhost. Run ./generate_cert.sh (or drop your own cert.pem/key.pem into certs/), restart, open https://<host>:8418/ and accept the certificate warning. The web interface can still stream an audio file without a microphone.

License

MIT — see LICENSE. All bundled dependencies (pyannote.audio, faster-whisper, FastAPI, PyTorch, SQLAlchemy, Pydantic) are under MIT or BSD; NVIDIA NeMo, speakrs and ClearerVoice-Studio (MossFormer2, code and weights) are Apache-2.0; Silero VAD (bundled with faster-whisper) is MIT. The pyannote models require HuggingFace account + token + terms acceptance; the software remains open-source. The WeSpeaker ResNet34 voice model (pyannote/wespeaker-voxceleb-resnet34-LM) and the Parakeet-TDT models are released under CC-BY-4.0.

Credits

• pyannote.audio — Hervé Bredin et al.
• faster-whisper — SYSTRAN
• Whisper — OpenAI
• NeMo / Parakeet-TDT — NVIDIA
• speakrs — a Rust/CUDA implementation of pyannote's community-1 pipeline
• WeSpeaker — the ResNet34 speaker-embedding model
• Silero VAD — speech detection for live streams
• ClearerVoice-Studio — MossFormer2 speech separation
• emotion2vec / FunASR — ACL 2024
• FastAPI — Sebastián Ramírez

Planned

• AI-generated conversation summaries and titles on finalize.
• Vector-search over transcripts for semantic conversation lookup.

Disclaimer

Provided as-is, no warranty. Speaker recognition is probabilistic — verify important identifications manually and obtain consent before recording others. Portions of this codebase were pair-programmed with an AI assistant; review before deploying in critical settings.