Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-17 05:40:42 -07:00
Code Issues Packages Projects Releases Wiki Activity

feat(mcp): local MCP server exposes voicebox.* tools to AI agents

Browse Source 
Mounts FastMCP at /mcp (Streamable HTTP) so Claude Code, Cursor,
Windsurf, and the VS Code MCP extensions can call voicebox.speak,
voicebox.transcribe, voicebox.list_captures, and voicebox.list_profiles
against the running Voicebox server.

Backend
- new backend/mcp_server package (tools, middleware, profile resolve,
  pub/sub events); named mcp_server to avoid shadowing the installed mcp
  PyPI package FastMCP imports internally
- app.py migrated from @app.on_event to lifespan= so FastMCP's session
  manager cohabits with Voicebox's startup/shutdown
- new MCPClientBinding table + /mcp/bindings CRUD; ClientIdMiddleware
  reads X-Voicebox-Client-Id into a ContextVar and stamps last_seen_at
- profile resolution precedence: explicit -> per-client binding ->
  capture_settings.default_playback_voice_id
- POST /speak REST wrapper for non-MCP callers (shell, ACP, A2A)
- GET /events/speak SSE broadcasts speak-start / speak-end so the pill
  surfaces agent-initiated speech
- backend/mcp_shim proxy (plain httpx) for stdio-only MCP clients
- PyInstaller spec updates + new --shim build target (~18 MB)

Frontend
- Settings -> MCP page with HTTP / stdio / claude-mcp-add copy snippets,
  default voice picker, per-client bindings table, connection status
- useMCPBindings, useSpeakEvents hooks
- CapturePill gains 'speaking' state; DictateWindow subscribes to SSE
  and emits dictate:show so the Rust side surfaces the pill window

Native
- tauri.conf.json externalBin now includes voicebox-mcp
- show_dictate_window helper + dictate:show listener in main.rs
- (also in this commit: InputMonitoringGate UX, hotkey_monitor tweaks,
  landing footer/navbar updates, new overview docs for captures /
  dictation / mcp-server / voice-personalities)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
James Pine
2026-04-22 22:05:30 -07:00
co-authored by Claude Opus 4.7
parent 87c582ad54
commit 0cef2c9fe1
52 changed files with 4094 additions and 336 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/docs/index.mdx
+25 -16
View File
@@ -1,23 +1,30 @@
---
title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is the **open-source, local-first AI voice studio** — a free
alternative to ElevenLabs and WisprFlow in one app. Clone voices, generate
speech across 7 TTS engines, dictate into any app with a global hotkey,
compose multi-voice projects, and let any MCP-aware agent speak in a voice
you own. Everything runs on your hardware.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
- **Native performance** -- built with Tauri (Rust), not Electron
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
- **Dictation** — hold a chord anywhere on your machine, speak, release; the transcript pastes into the focused field
- **Captures tab** — paired audio + transcript archive, retranscribe / refine / play-as-voice
- **Voice personalities** — per-profile compose / rewrite / respond, powered by a local LLM
- **Agents speak back** — any MCP-aware agent can call Voicebox to speak in one of your cloned voices
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, Kokoro
- **Cloning and preset voices** — zero-shot cloning or 50+ curated preset voices
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, filters
- **Expressive speech** — paralinguistic tags (`[laugh]`, `[sigh]`) and natural-language delivery control
- **Unlimited length** — auto-chunking with crossfade for long scripts
- **Stories editor** — multi-track timeline for conversations, podcasts, narratives
- **API-first** — REST + WebSocket API, MCP server for agent integrations
- **Complete privacy** — models, audio, transcripts, LLM output never leave your machine
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux (ROCm / CPU), Intel Arc, Docker
## Download
@@ -32,6 +39,8 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## Get Started
- [Installation](/overview/installation) -- download and install Voicebox
- [Quick Start](/overview/quick-start) -- get up and running in 5 minutes
- [API Reference](/api-reference) -- integrate voice synthesis into your apps
- [Installation](/overview/installation) — download and install Voicebox
- [Quick Start](/overview/quick-start) — get up and running in 5 minutes
- [Dictation](/overview/dictation) — start talking to your computer
- [Voice Personalities](/overview/voice-personalities) — compose, rewrite, respond in any profile
- [API Reference](/api-reference) — integrate voice synthesis into your apps
docs/content/docs/overview/captures.mdx
+194
View File
@@ -0,0 +1,194 @@
---
title: "Captures"
description: "The paired audio + transcript archive — every dictation, recording, and uploaded audio file shows up here, replayable and retranscribable."
---
## Overview
A **capture** is an audio clip paired with its transcript. The Captures tab
is where every dictation, manual recording, and uploaded audio file lands,
with the original audio kept alongside the text so you can replay, re-run
transcription with a different model, refine the transcript, or send the
content somewhere else — including generating it back as speech in any of
your voice profiles.
<Callout type="info">
The Captures tab shipped in **0.5.0**, alongside global dictation and the
per-profile personality modes. If you've used earlier versions, note that
the Audio tab moved into **Settings → Audio Channels** to make room for
this one.
</Callout>
## Where captures come from
| Source | How it shows up | Badge |
|---|---|---|
| **Dictation** | Triggered by the global hotkey (see [Dictation](/overview/dictation)). Auto-refined by default. | `dictation` |
| **In-app recording** | Recorded directly in the Captures tab using the built-in mic. | `recording` |
| **File upload** | Any audio file dropped into the Captures tab — `.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`. | `file` |
All three paths share the same backend pipeline, the same model picker, and
the same refinement flags. The source badge is there so you can visually
scan a long list.
## List view
The main Captures view is a chronological list. Each row shows:
- The transcript (raw or refined — the refined version wins if present)
- Duration + timestamp
- Source badge
- A play button for the original audio
- A meatballs menu with per-row actions
Filtering and search are a Tier-2 ask — ping if you need them.
## Detail view
Clicking into a capture opens the detail view:
- **Waveform player** for the original audio
- **Transcript editor** — click in and edit. Changes save on blur.
- **Refined vs. raw toggle** if refinement ran on this capture
- **Per-capture action bar** — retranscribe, refine, play as voice, delete
- **Settings snapshot** — STT model used, refinement flags at the time
this capture was processed, and the voice model if any was played
## Retranscribe
Runs the capture's original audio through a different Whisper model without
re-uploading or re-refining anything. Useful when:
- The default model mis-heard something and you want to try a larger model
- You used Base for a noisy clip and want to rerun with Turbo
- A non-English clip needs an explicit language hint
**Settings → Captures → Transcription** controls the default model and
language lock for new captures. Retranscribe uses those defaults unless you
override them per capture.
## Refine
Runs the raw transcript through the local LLM to produce a cleaned-up
version. The flags on the capture are snapshotted when refinement first
runs, so you can re-refine later with different flags without losing the raw
transcript:
| Flag | Effect |
|---|---|
| **Smart cleanup** | Remove fillers (`um`, `uh`, `like`), tidy punctuation and capitalization. |
| **Remove self-corrections** | Keep the final version when the speaker backtracks ("actually, no, on Tuesday"). |
| **Preserve technical terms** | Leave identifiers (`handleSubmit`, `npm install`) untouched. |
See the Refinement section of [Dictation](/overview/dictation#refinement) for
how Voicebox strips Whisper loop hallucinations *before* the LLM sees the
transcript — a capture can be re-refined any number of times without
re-introducing "thanks for watching thanks for watching" echoes.
The refinement model picker (three bundled Qwen3 sizes) lives in
**Settings → Captures → Refinement**.
## Play as voice
This is the capability no one else in the dictation category ships: take any
capture and play it back as speech in any of your voice profiles. One
dropdown over every profile, one click, and the capture's text runs through
`/generate` with the selected voice.
Use cases:
- Hear your own dictation back in a cloned voice of someone you like
- Send a message you dictated as an audio reply in a specific character
- Quickly prototype a line for a story without retyping
Playback uses whatever engine the selected profile is bound to — the same
rules as the Generate tab. There's no LLM in this path; the transcript goes
through unchanged. If you want the agent-style "transform the content before
speaking" flow, that's what the
[personality modes](/overview/voice-personalities) do — and the same
primitive is exposed to MCP-aware agents via the
[MCP Server](/overview/mcp-server) so Claude Code, Cursor, or Cline can speak
in one of your voices on their own.
<Callout type="info">
The default voice for the Captures tab's Play-as action is set in
**Settings → Captures → Playback → Default voice**. You can still override
it per capture.
</Callout>
## Send-to menu
Each capture has a Send-to menu for moving its content into other parts of
Voicebox:
- **Copy transcript** — to clipboard
- **Use as voice sample…** — promote this capture to a sample on a voice
profile of your choice. Opens a profile picker (with "+ New voice" for
cold starts) and a reference-text confirm dialog, because cloning needs
the `reference_text` to match the audio verbatim. Edit as needed and
save — the capture stays in the Captures tab untouched; the sample is a
copy, not a move.
## Storage and retention
**Settings → Captures → Storage** controls how long captures live on disk:
| Setting | Effect |
|---|---|
| **Retention: forever** | Never auto-delete. Default. |
| **Retention: 90 days / 30 days / 7 days** | Captures older than the window are pruned on app start. |
| **Clear all captures** | One-click nuke of every capture and its audio on disk. No undo. |
The original audio is always kept alongside the transcript — archival is on
by default. Every capture's audio file and metadata row can be re-processed
(retranscribe, refine, Play-as) as long as the audio file still exists.
## Short-recording guard
Audio clips under **300 ms** are short-circuited client-side and never
uploaded. This prevents a fumbled chord tap from landing an empty capture.
The threshold is tuned to filter accidents without cutting off intentional
short dictations.
## Keyboard shortcuts
Inside the Captures tab:
| Keys | Action |
|---|---|
| `Space` | Play / pause the selected capture |
| `↑` / `↓` | Previous / next capture in the list |
| `Enter` | Open the selected capture in detail view |
| `⌘ / Ctrl` + `C` (in detail view) | Copy the transcript |
## API surface
The Captures tab is backed by a small set of REST endpoints:
| Method | Endpoint | Use |
|---|---|---|
| `POST` | `/captures` | Upload audio + start the pipeline (STT, optional refinement, archival). |
| `GET` | `/captures` | List captures. |
| `GET` | `/captures/{id}` | Fetch one capture. |
| `POST` | `/captures/{id}/retranscribe` | Rerun STT with a chosen model. |
| `POST` | `/captures/{id}/refine` | Rerun refinement with chosen flags. |
| `POST` | `/profiles/{id}/samples/from-capture/{capture_id}` | Promote a capture to a voice profile sample. |
These endpoints are stable and usable from your own scripts — see
[Remote Mode](/overview/remote-mode) for running Voicebox as a server the rest
of your machine can talk to.
## Next steps
<Cards>
<Card title="Dictation" href="/overview/dictation">
The global hotkey flow that feeds most captures.
</Card>
<Card title="Voice Personalities" href="/overview/voice-personalities">
Per-profile compose / rewrite / respond modes for captures you want to
transform, not just transcribe.
</Card>
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
Promote a capture into a voice sample on a profile.
</Card>
</Cards>
docs/content/docs/overview/dictation.mdx
+208
View File
@@ -0,0 +1,208 @@
---
title: "Dictation"
description: "Hold a key anywhere on your machine, speak, release — the transcript lands in whatever text field you had focused."
---
## Overview
Dictation lets you turn speech into clean text anywhere on your computer. Hold
a chord, talk, release — Voicebox transcribes what you said with Whisper,
optionally cleans it up with a local LLM, and pastes the result into the text
field you had focused when you started.
Everything happens on your hardware. No cloud, no accounts, no audio leaving
the machine.
<Callout type="info">
Dictation was introduced in **0.5.0** alongside the Captures tab and the
per-profile personality modes. It's the "input" half of Voicebox's voice I/O
loop — cloning and TTS are still the "output" half.
</Callout>
## The flow
<Steps>
<Step title="Hold the chord">
Hold the push-to-talk chord anywhere on your machine. A small pill fades
in over your current app.
</Step>
<Step title="Speak">
The pill shows `Recording` with a live waveform and an elapsed-time
counter. Speak naturally — you don't have to wait for anything.
</Step>
<Step title="Release">
On release, the pill flips to `Transcribing`, then `Refining` if
auto-refine is on, then disappears.
</Step>
<Step title="Text lands in your app">
If auto-paste is enabled and Voicebox has Accessibility permission, the
transcript pastes into the text field you had focused when you started
talking — not wherever focus drifted while you were speaking.
</Step>
</Steps>
Either way, every capture also appears in the **Captures tab** with the
original audio and the transcript paired together. See
[Captures](/overview/captures) for what you can do with them after the fact.
## Push-to-talk and toggle modes
Voicebox ships two chord behaviors out of the box:
| Mode | Default (macOS) | Default (Windows) | Behavior |
|---|---|---|---|
| **Push-to-talk** | Right `⌘` + Right `⌥` | Right `Ctrl` + Right `Shift` | Recording stops when you release the chord. |
| **Toggle-to-talk** | Push-to-talk + `Space` | Push-to-talk + `Space` | Recording keeps going until you tap the chord again. |
**Holding PTT and tapping `Space` mid-hold upgrades a hold into a toggled
session** without a gap in the audio. This is the single most useful detail of
the chord system — short bursts feel fast, long-form narration feels
hands-free, and there's no decision up front about which mode you wanted.
## The on-screen pill
While you're dictating, a floating pill appears over the current app. It walks
through the states of the capture cycle and shows live signals for each:
| State | What it shows |
|---|---|
| `Recording` | Live waveform + elapsed time. |
| `Transcribing` | Thinking waveform while Whisper runs. |
| `Refining` | Same thinking waveform while the LLM cleans up the transcript (only if auto-refine is on). |
| Error | Red tint. Click the pill to copy the error to your clipboard. Auto-dismisses. |
The pill is transparent, always-on-top, and pre-created hidden at app start —
so it appears instantly when you hit the chord, with no window flash.
## Customizing the chord
Open **Settings → Captures → Dictation** to change either chord.
- **Left vs right modifier badges.** When you hold keys into the chord
picker, Voicebox records whether each modifier is the left or right variant.
That means you can bind to just the right `⌥` while leaving the left `⌥`
alone — useful if you want dictation on one hand and keep your
other-hand shortcuts intact.
- **Chord defaults are picked to stay out of your way.** On macOS, the
defaults deliberately avoid left-hand `Cmd+Option` chords so
`Cmd+Option+I` (devtools), `Cmd+Option+Esc` (force quit), and
`Cmd+Option+Space` (Spotlight) all remain yours. On Windows, the defaults
route around AltGr collisions on German / French / Spanish layouts where
`Ctrl+Alt` synthesizes AltGr.
- **Live reload.** Changing a chord in Settings takes effect immediately —
no restart, no tab reload.
## Auto-paste into the focused app
Once transcription finishes, Voicebox can synthesize a native paste into
whatever text field had focus when you started the chord. Your clipboard is
saved before and restored after, so nothing you had copied goes missing.
| Platform | Mechanism |
|---|---|
| macOS | `CGEventPost` at the HID tap with a full `⌘V` key sequence, preceded by reactivating the original app via `NSRunningApplication`. |
| Windows | `SendInput` with correct scan codes, plus a `SetForegroundWindow` + `AttachThreadInput` handshake to defeat foreground-lock when pasting into a window that wasn't frontmost at chord-start. |
**Focus is snapshotted at chord-start.** The paste targets the original field
even if focus drifts during transcribe / refine — that's the "pastes where you
were talking *from*, not where you're looking *now*" behavior.
<Callout type="info">
Auto-paste is optional. If Accessibility permission isn't granted (macOS),
or you prefer to keep synthetic input off, dictation still runs — transcripts
land in the Captures tab and you can copy them manually. The setting lives
inline next to the Accessibility prompt in Settings → Captures → Dictation,
not as a global banner.
</Callout>
## Refinement
If auto-refine is on, a local LLM cleans up the raw Whisper transcript
before it's pasted. The goal is to remove verbal clutter without rewriting
what you actually said.
What refinement typically fixes:
- Filler words (`um`, `uh`, `like` used as pauses, `you know`)
- Self-corrections — the LLM keeps the final version and drops earlier
attempts (`could you uh run the migration real quick, and then, yeah,
check the logs` → `Could you run the migration, then check the logs?`)
- Basic punctuation and capitalization
- Whisper loop hallucinations — Voicebox strips repeated tokens (six or
more identical tokens in a row, case-insensitive) *before* the LLM
sees the transcript, so a small refinement model can't echo them back
What refinement deliberately preserves:
- Technical terms and code identifiers (`npm install`, `handleSubmit`)
- Legitimate repetition (`no, no, no, no, no` has fewer than six identical
tokens, so it survives)
- Your intent — refinement is cleanup, not rewriting
Flags are snapshotted per capture, so you can re-refine the same raw
transcript later with different flags without losing the original. The
refinement model picker (**Settings → Captures → Refinement**) offers three
bundled Qwen3 sizes:
| Model | Size | Best for |
|---|---|---|
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual dictation. |
| Qwen3 1.7B | ~1.1 GB | Sweet spot when transcripts contain code identifiers. |
| Qwen3 4B | ~2.5 GB | Full quality, slowest. |
This is the same local LLM used by the per-profile personality modes — one
LLM in the app, not two. See [Voice Personalities](/overview/voice-personalities).
## Platform notes
### macOS
- **Accessibility permission** is required for auto-paste. The prompt lives
inline next to the toggle in **Settings → Captures → Dictation**, with a
deep link to **System Settings → Privacy & Security → Accessibility**.
- **TSM crash mitigation.** The global hotkey listener runs on a background
thread with `set_is_main_thread(false)` to sidestep a known
macOS 14+ crash in the `rdev` library. If you hit an unexpected dictation
failure on macOS, check the logs for TSM-related messages.
### Windows
- **UAC / UIPI caveat.** Synthetic paste into an *elevated* window from a
non-elevated Voicebox is blocked by Windows itself. Run Voicebox elevated
if you regularly dictate into elevated apps (e.g. an elevated terminal or
Task Manager).
- **Right-hand default chord** (`Ctrl+Shift`) avoids AltGr collisions on
keyboard layouts where `Ctrl+Alt` is the compose key (German, French,
Spanish, some others).
### Linux
- **Not yet in this release.** The Rust shim ships the macOS and Windows
paths in 0.5.0. Linux `uinput` / AT-SPI support and the Wayland paste
story are tracked in `docs/plans/VOICE_IO.md`.
## When auto-paste skips itself
A few cases where Voicebox deliberately does *not* synthesize a paste:
- **Focus was inside Voicebox** when the chord started. The transcript goes
to the Captures tab so a dictation-into-Voicebox round-trip doesn't
accidentally paste into the generate box.
- **No text focus detected.** The transcript still lands in the Captures
tab; copy it from there with one click.
- **Accessibility permission not granted** on macOS. Same — Captures tab
only.
## Next steps
<Cards>
<Card title="Captures" href="/overview/captures">
The paired audio + transcript archive every dictation lands in.
</Card>
<Card title="Voice Personalities" href="/overview/voice-personalities">
The same local LLM doubles as per-profile compose / rewrite / respond.
</Card>
<Card title="Transcription" href="/developer/transcription">
Developer-level details on Whisper, Whisper Turbo, and the STT backend.
</Card>
</Cards>
docs/content/docs/overview/introduction.mdx
+60 -20
View File
@@ -1,23 +1,48 @@
---
title: "Introduction"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
---
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is the **open-source, local-first AI voice studio**. It closes the
voice I/O loop in both directions on one machine, with no cloud and no
accounts:
- **Complete privacy** -- models and voice data stay on your machine
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
- **Native performance** -- built with Tauri (Rust), not Electron
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
- **Humans talk** — hold a chord anywhere on your machine and your
dictation lands as clean text in whatever text field you had focused
- **Agents talk back** — any MCP-aware agent can call Voicebox to speak in
one of your cloned voices
- **Voices speak for themselves** — voice profiles can carry a personality
that compose, rewrite, or respond to text before it's spoken
It's the free, local alternative to both ElevenLabs (voice cloning and TTS)
and WisprFlow (voice dictation for agents and power users) — covering both
sides of the same loop in one app, with a single model directory and LLM
shared between input and output.
## What's in the app
- **Dictation** — global hotkey, push-to-talk and toggle modes, auto-paste
into the focused field on macOS and Windows (see [Dictation](/overview/dictation))
- **Captures tab** — paired audio + transcript archive, retranscribe,
refine, play-as-voice, promote-to-sample (see [Captures](/overview/captures))
- **Voice cloning** — 5 cloning engines covering 23 languages. Zero-shot
cloning from a reference sample (see [Voice Cloning](/overview/voice-cloning))
- **Preset voices** — 50+ curated voices via Kokoro and Qwen CustomVoice
for when you don't want to clone (see [Preset Voices](/overview/preset-voices))
- **Voice personalities** — optional free-form personality on any profile
plus compose / rewrite / respond modes powered by a local LLM (see
[Voice Personalities](/overview/voice-personalities))
- **Post-processing effects** — pitch shift, reverb, delay, chorus,
compression, filters (Spotify's Pedalboard)
- **Expressive speech** — paralinguistic tags like `[laugh]` and `[sigh]`
via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** — auto-chunking with crossfade for long scripts
- **Stories editor** — multi-track timeline for conversations and podcasts
- **API-first** — REST + WebSocket API; MCP server for agent integrations
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux
(ROCm / CPU), Intel Arc, Docker
## TTS Engines
@@ -30,9 +55,21 @@ Seven engines with different strengths, switchable per-generation:
| **LuxTTS** | Cloned | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | Cloned | 23 | Broadest language coverage |
| **Chatterbox Turbo** | Cloned | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model -- 700s+ coherent audio |
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model — 700s+ coherent audio |
| **Kokoro** | Preset (50 voices) | 9 | 82M parameters, CPU realtime, lowest VRAM of any engine |
## STT and local LLM
Voicebox also runs a full speech recognition and local LLM stack, shared
between dictation, the Captures tab, and per-profile personality modes:
| Layer | Models |
|---|---|
| **STT** | Whisper Base / Small / Medium / Large / Turbo (PyTorch or MLX) |
| **LLM** | Qwen3 0.6B / 1.7B / 4B (refinement + per-profile compose / rewrite / respond) |
No cloud fallback, no bring-your-own-API-key. Local is the product.
## GPU Support
| Platform | Backend | Notes |
@@ -46,11 +83,13 @@ Seven engines with different strengths, switchable per-generation:
## Use Cases
- **Game development** -- generate dynamic dialogue for characters
- **Content creation** -- produce podcasts and video voiceovers
- **Accessibility** -- build text-to-speech tools for users who need them
- **Voice assistants** -- create custom voice interfaces
- **Production pipelines** -- automate voiceover workflows via the REST API
- **Dictation for humans and agents** — speak instead of type, in any app
- **Agent voice output** — any MCP-aware agent can speak in a cloned voice
- **Game development** — generate dynamic dialogue for characters
- **Content creation** — podcasts, video voiceovers, audiobooks
- **Accessibility** — speech-to-text for any field, TTS with a voice you own
- **Voice assistants** — custom voice interfaces without a cloud bill
- **Production pipelines** — automate voice workflows via the REST API
## Tech Stack
@@ -61,8 +100,9 @@ Seven engines with different strengths, switchable per-generation:
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
| STT | Whisper / Whisper Turbo (PyTorch or MLX) |
| Local LLM | Qwen3 0.6B / 1.7B / 4B (MLX or PyTorch) |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
| Database | SQLite |
| Audio | WaveSurfer.js, librosa |
docs/content/docs/overview/mcp-server.mdx
+300
View File
@@ -0,0 +1,300 @@
---
title: "MCP Server"
description: "Let Claude Code, Cursor, Cline, or any MCP-aware agent speak in one of your cloned voices — locally, with no cloud."
---
## Overview
Voicebox ships a built-in **Model Context Protocol** server so local AI
agents can call your Voicebox install directly: speak text in a voice
profile, transcribe audio, and list captures or profiles. The server runs
inside the same process as the rest of Voicebox and is mounted at `/mcp`
over Streamable HTTP.
Agent asks to speak → Voicebox plays audio on your speakers → an on-screen
pill surfaces the voice name for the whole duration so you always see what's
coming out of your machine.
<Callout type="info">
MCP shipped in **0.5.0** alongside [Dictation](/overview/dictation) and
[Voice Personalities](/overview/voice-personalities). The design goal is
"local voice layer for every agent on your machine" — the same app that
captures your voice can generate a response in any voice profile you've
cloned.
</Callout>
## Quick install
### Claude Code
```
claude mcp add voicebox \
--transport http \
--url http://127.0.0.1:17493/mcp \
--header "X-Voicebox-Client-Id: claude-code"
```
### Cursor / Windsurf / VS Code MCP / any HTTP MCP client
Drop this into the client's MCP config (usually `.mcp.json` or a Settings UI):
```json
{
"mcpServers": {
"voicebox": {
"url": "http://127.0.0.1:17493/mcp",
"headers": { "X-Voicebox-Client-Id": "cursor" }
}
}
}
```
Change `cursor` to whatever name you want the binding to show up as in
Voicebox → Settings → MCP. The value is just an identifier for the
per-client voice binding — not a secret, not a credential.
### Clients that only speak stdio
A stdio shim binary `voicebox-mcp` is bundled with the desktop app. Point
the client at that binary's absolute path:
<Tabs items={["macOS", "Windows", "Linux"]}>
<Tab value="macOS">
```json
{
"mcpServers": {
"voicebox": {
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
}
}
}
```
</Tab>
<Tab value="Windows">
```json
{
"mcpServers": {
"voicebox": {
"command": "C:\\Program Files\\Voicebox\\voicebox-mcp.exe",
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
}
}
}
```
</Tab>
<Tab value="Linux">
```json
{
"mcpServers": {
"voicebox": {
"command": "/opt/voicebox/voicebox-mcp",
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
}
}
}
```
</Tab>
</Tabs>
The shim waits up to 30 seconds for the Voicebox backend to come up, then
proxies JSON-RPC from stdio over Streamable HTTP. Voicebox must be running
for the shim to connect.
## Tools
| Tool | Use |
|---|---|
| `voicebox.speak` | Speak text in a voice profile. Returns a `generation_id` to poll. |
| `voicebox.transcribe` | Whisper transcription of base64 audio or an absolute local path. |
| `voicebox.list_captures` | Recent captures with transcripts, paginated. |
| `voicebox.list_profiles` | Available voice profiles (cloned + preset). |
### `voicebox.speak`
```ts
voicebox.speak({
text: "Deploy complete.",
profile?: "Morgan", // name or id; falls back to per-client binding, then default
engine?: "qwen", // qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro
intent?: "respond", // respond | rewrite | compose — only if the profile has a personality
language?: "en",
})
```
Returns:
```json
{
"generation_id": "…",
"status": "generating",
"profile": "Morgan",
"source": "mcp",
"poll_url": "/generate/<id>/status"
}
```
- **Plain TTS** — omit `intent`. Text is spoken as-is.
- **Persona mode** — pass `intent` and the profile must have a personality set.
The LLM transforms the text (respond to it, rewrite it in character, or
compose a fresh utterance) before TTS. See [Voice Personalities](/overview/voice-personalities).
### `voicebox.transcribe`
```ts
voicebox.transcribe({
audio_base64?: "<base64>", // exactly one of these two
audio_path?: "/absolute/path/to/file.wav",
language?: "en",
model?: "turbo", // base | small | medium | large | turbo
})
```
Returns `{ text, duration, language, model }`. 200 MB ceiling on either path.
### `voicebox.list_captures`
`{ limit?: 20, offset?: 0 }` → `{ captures: [...], total }`. `limit` is
clamped to `1..=200`.
### `voicebox.list_profiles`
No args → `{ profiles: [{ id, name, voice_type, language, has_personality }] }`.
## Voice resolution
Every call to `voicebox.speak` (and `POST /speak`) resolves the voice profile
in this order:
<Steps>
<Step title="Explicit `profile` arg">
Passed as a name (case-insensitive) or id. If the name/id doesn't match,
the call errors — the server doesn't silently fall back.
</Step>
<Step title="Per-client binding">
Looked up by the `X-Voicebox-Client-Id` header. Managed in
**Voicebox → Settings → MCP**. Lets you pin Claude Code to Morgan,
Cursor to Scarlett, etc.
</Step>
<Step title="Global default">
`capture_settings.default_playback_voice_id` — same default voice the
Captures tab's "Play as voice" action uses.
</Step>
</Steps>
If none of the three produce a profile the tool returns a helpful error
pointing at Settings.
## Per-client bindings
Voicebox → Settings → MCP shows one row per `client_id` Voicebox has heard
from, plus the config snippets you can copy into each agent. Each row
carries:
| Field | Purpose |
|---|---|
| `label` | Display name in the Settings UI (e.g. "Claude Code"). |
| `profile_id` | The voice this client uses when `profile` isn't passed. |
| `default_engine` | Override the TTS engine for this client. |
| `default_intent` | Default persona mode (`respond` / `rewrite` / `compose`). |
| `last_seen_at` | Last time the server saw a request from this client. |
`last_seen_at` is stamped automatically by middleware on every `/mcp/*`
request — useful when you're not sure whether your config took.
## The speaking pill
Every agent-initiated speak surfaces the floating pill the same way
[Dictation](/overview/dictation) does, in a new `Speaking` state showing the
profile name and an elapsed timer. The pill is intentionally unmissable —
silent background TTS is a trust hazard, so Voicebox always shows what's
being spoken and in what voice.
Behind the scenes, the backend broadcasts `speak-start` and `speak-end`
events on `GET /events/speak`, which `DictateWindow` subscribes to via SSE.
The pill overrides the capture session when both would render — you can't
hear two pills at once.
## Non-MCP REST surface
`POST /speak` is a thin wrapper on the same code path for callers that
don't speak MCP — shell scripts, ACP, A2A, GitHub Actions, whatever.
```bash
curl -X POST http://127.0.0.1:17493/speak \
-H 'Content-Type: application/json' \
-H 'X-Voicebox-Client-Id: ci' \
-d '{"text":"Build complete.","profile":"Morgan"}'
```
Body fields match the MCP tool: `text`, optional `profile`, `engine`,
`intent`, `language`. Returns a `GenerationResponse` — the same shape as
`POST /generate`.
## Debugging
Use the MCP Inspector to poke tools directly without plumbing through an
agent:
```
npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp
```
Start with `voicebox.list_profiles` to confirm wiring, then
`voicebox.speak` for end-to-end — you should hear audio and see the
generation land in the Captures tab.
<Callout type="info">
If an agent can't reach the server, the first thing to check is that
Voicebox is running — the backend only listens while the desktop app is
open. The stdio shim surfaces this as a JSON-RPC error on the client
side after its 30-second health-wait window elapses.
</Callout>
## Security
- **Localhost only.** The server binds to `127.0.0.1`. If you ever point
Voicebox at a non-loopback interface (e.g. remote-mode over a trusted
network), add a bearer token — it's on the roadmap but not in 0.5.0.
- **No auth today.** Any process that can connect to your loopback can
call MCP. That's the same trust boundary as the rest of Voicebox's REST
API and is appropriate for a single-user local tool.
- **`audio_path` reads are unrestricted** against the same trust
boundary. If you're scripting against a shared host, prefer
`audio_base64` so you don't have to think about path sandboxing.
- **Voice cloning consent applies.** See [Voice Cloning](/overview/voice-cloning#limitations)
— an agent being able to call `voicebox.speak` in someone's voice
doesn't change the ethics of whose voices you clone.
## Implementation notes
- **Transport:** Streamable HTTP (Nov-2025 MCP spec, post-SSE). Claude
Code, Cursor, Windsurf, and VS Code MCP extensions all support it.
- **Package naming:** the backend package is `backend/mcp_server/`, not
`mcp`, to avoid shadowing the PyPI `mcp` package FastMCP imports
internally.
- **Dependencies:** `fastmcp>=3.0,<4.0`, `sse-starlette>=2.0`.
- **Lifespan:** mounting FastMCP requires the `lifespan=` kwarg on
`FastAPI()` — the startup/shutdown event decorators are incompatible
with FastMCP's Streamable HTTP session manager. The Voicebox app.py
composes both into one async context manager.
For the full developer-facing tour of the code layout, see
`backend/mcp_server/README.md` in the repo.
## Next steps
<Cards>
<Card title="Voice Personalities" href="/overview/voice-personalities">
Persona mode (`intent=respond/rewrite/compose`) for agents that should
transform text in-character before speaking.
</Card>
<Card title="Dictation" href="/overview/dictation">
The pill that surfaces agent speech is the same one that surfaces
your dictations — one mental model for both directions of the loop.
</Card>
<Card title="Captures" href="/overview/captures">
Every agent-initiated speak lands in the Captures tab with its
generated audio — replay, download, repurpose.
</Card>
</Cards>
docs/content/docs/overview/meta.json
+4
View File
@@ -7,8 +7,12 @@
"docker",
"quick-start",
"gpu-acceleration",
"dictation",
"captures",
"voice-cloning",
"preset-voices",
"voice-personalities",
"mcp-server",
"stories-editor",
"recording-transcription",
"generation-history",
docs/content/docs/overview/recording-transcription.mdx
+88 -46
View File
@@ -1,64 +1,106 @@
---
title: "Recording & Transcription"
description: "Record audio and transcribe speech with Whisper"
description: "A map of the three places you can record and transcribe audio in Voicebox — dictation, captures, and voice-profile samples."
---
## Recording
## Overview
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
Voicebox records and transcribes audio in three different contexts, each
feeding a different surface in the app. This page is a map; follow the links
for the detail.
### Features
| Goal | Where | Docs |
|---|---|---|
| Speak and have your words land in another app | Global hotkey → Captures tab + auto-paste | [Dictation](/overview/dictation) |
| Record a thought, a meeting, or a voice memo inside Voicebox | Captures tab | [Captures](/overview/captures) |
| Record a clip to clone a voice from | Voices tab → profile samples | [Creating Voice Profiles](/overview/creating-voice-profiles) |
- **Microphone input** - Record from any audio input device
- **System audio capture** - Record desktop audio (macOS/Windows)
- **Waveform visualization** - See audio levels in real-time
- **Multiple formats** - Export as WAV, MP3, or M4A
All three paths share the same STT backend — it's the surrounding workflow
that differs.
### How to Record
## Dictation
<Steps>
<Step title="Select Input">
Choose your microphone or system audio
</Step>
<Step title="Start Recording">
Click the record button and speak clearly
</Step>
<Step title="Stop & Save">
Click stop when finished
</Step>
<Step title="Use or Export">
Use as voice sample or export to file
</Step>
</Steps>
The 0.5.0 headline feature. Hold a chord anywhere on your machine, speak,
release. The transcript lands in whatever text field you had focused,
cleaned up by a local LLM if auto-refine is on. Captures accumulate in the
Captures tab for later replay or re-transcription.
## Transcription
Covered end-to-end in [Dictation](/overview/dictation).
Automatic speech-to-text powered by OpenAI's Whisper model.
## Captures tab
### Features
When you don't need to paste into another app — you just want a clean
transcript of some audio — the Captures tab is the home. Record in-app,
drop in a file (`.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`), or dig
through dictations that already landed there. Every capture keeps its
original audio, can be retranscribed with a different model, and can be
played back through any voice profile you have.
- **High accuracy** - Industry-leading speech recognition
- **Multiple languages** - Supports 50+ languages
- **Automatic detection** - Language auto-detection
- **Timestamps** - Word-level timing information
Covered in [Captures](/overview/captures).
### How to Transcribe
## Voice profile samples
<Steps>
<Step title="Select Audio">
Choose a recording or upload an audio file
</Step>
<Step title="Choose Language">
Select language or use auto-detect
</Step>
<Step title="Transcribe">
Click transcribe and wait for processing
</Step>
<Step title="Review & Export">
Review text and export as needed
</Step>
</Steps>
A separate flow, in the Voices tab. When you're creating a profile from an
audio clip, the sample is what the cloning engine actually learns from —
the `reference_text` on a sample must match the audio *verbatim*, which is
why samples are a different data model from captures.
You can promote a capture to a sample from the Captures tab's Send-to menu
("Use as voice sample…"), which opens a reference-text confirm dialog so
you can correct the last ~10% of transcript accuracy before saving.
Covered in [Creating Voice Profiles](/overview/creating-voice-profiles).
## Transcription models
All three paths share the same Whisper models. Pick a default in
**Settings → Captures → Transcription**; override per capture if you need
to.
| Model | Size | When to pick it |
|---|---|---|
| Whisper Base | ~300 MB | Fast. Default. Good for clean speech. |
| Whisper Small | ~500 MB | Better quality, still fast. |
| Whisper Medium | ~1.5 GB | High quality. |
| Whisper Large | ~3 GB | Best quality, slow on CPU. |
| Whisper Turbo | ~1.5 GB | Large-tier quality, ~5× faster than Large. |
On Apple Silicon the model runs through **MLX-Whisper** (~8× faster than
PyTorch). Everywhere else it runs through PyTorch `transformers`. The
backend picks the right one — you don't configure it.
<Callout type="info">
Transcription is useful for creating voice samples from existing audio or generating subtitles.
For noisy clips, prefer **Turbo** or **Large**. Base can hallucinate on
hard inputs — most famously the "thanks for watching" loop. Voicebox
strips those loops deterministically before LLM refinement runs, so a
capture can be cleanly re-refined even if the raw transcript has them.
</Callout>
## Language
You can pass a language hint for short clips (under ~5 seconds) where
Whisper's auto-detect is unreliable. Set a default language lock in
**Settings → Captures → Transcription → Language**, or override per capture.
## Transcription API
Developer-level detail on the STT backend, model loading, preprocessing, and
the `/transcribe` endpoint lives in the
[Transcription developer guide](/developer/transcription). The Captures
pipeline also exposes `/captures` as a higher-level endpoint that wraps
STT + archival + optional refinement in one call — see
[Captures](/overview/captures#api-surface).
## Next steps
<Cards>
<Card title="Dictation" href="/overview/dictation">
Hold a chord anywhere on your machine, speak, release.
</Card>
<Card title="Captures" href="/overview/captures">
The paired audio + transcript archive.
</Card>
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
Record or upload samples for voice cloning.
</Card>
</Cards>
docs/content/docs/overview/voice-personalities.mdx
+193
View File
@@ -0,0 +1,193 @@
---
title: "Voice Personalities"
description: "Attach a personality to a voice profile and use Compose, Rewrite, or Respond to generate in-character speech — all powered by a local LLM."
---
## Overview
A **personality** is an optional free-form description attached to a voice
profile — who this voice is, how they speak, what they care about. Set one
and three new actions appear on the profile, each powered by a bundled
Qwen3 LLM running entirely locally:
- **Compose** — generate a fresh utterance in this character's voice
- **Rewrite** — restate your text in their voice while preserving every idea
- **Respond** — treat your text as a prompt and produce the character's reply
The LLM produces the text. The voice profile speaks it. No cloud round-trip,
no external API — the whole loop runs on your hardware.
<Callout type="info">
Personalities shipped in **0.5.0**. The same local LLM doubles as the
refinement model for [Dictation](/overview/dictation) — one LLM in the app,
not two, sharing one model cache and one GPU-memory footprint.
</Callout>
## Setting a personality
Open a voice profile's edit view. The **Personality** field is free-form text
up to **2,000 characters**. Describe the voice however helps you — past
lines they'd say, speech patterns, tone, boundaries.
Good descriptions tend to include:
- A one-line identity (who they are)
- Speech patterns (rhythm, vocabulary, what they avoid)
- Representative phrases — example lines show the LLM the target tone
better than adjectives
- What the character *wouldn't* do (they don't explain, they don't
apologize, they refuse to break character, etc.)
You can set a personality on any voice profile type — cloned or preset. The
three modes work identically regardless of engine.
## The three modes
Each mode is tuned for a specific job and the LLM temperature is adjusted
to match.
### Compose
Generate a fresh utterance in the character's voice, with no seed text.
Click again to get a different take.
- **When to use:** prototyping, sampling a character's voice, brainstorming
a line without typing one first
- **Temperature:** hot — variety is the point
- **Typical output:** a short, punchy line that fits the character's
register
### Rewrite
Take your input text and restate it in the character's voice while
preserving every idea. High-fidelity mode — the content doesn't change, only
the voice does.
- **When to use:** turning a dictated memo into in-character speech; lifting
a plain-English script into a specific voice without editing by hand
- **Temperature:** cold — faithfulness wins
- **Typical output:** same ideas, same order, different phrasing and cadence
### Respond
Treat your input as a prompt and produce the character's reply — as if
you'd said it *to* them.
- **When to use:** spoken-input agents; Q&A with a specific voice;
interactive character experiences
- **Temperature:** balanced — creative but grounded
- **Typical output:** a reply to your prompt, written in-character
## Speech-only framing
All three modes enforce **speech-only** output. The LLM is prompted to
produce things a person would actually say out loud — no narration, no
action tags (`*sighs*`, `[laughs]`), no meta-commentary, no markdown
formatting, no stage directions.
This is deliberate: the output is going straight into TTS, and anything that
isn't speakable ends up either ignored or read literally. The speech-only
framing also makes the output land cleanly inside dialogue, so you can drop
a Respond result straight into a Story.
## The local LLM
The bundled LLM is **Qwen3**, available in three sizes:
| Model | Download size | Best for |
|---|---|---|
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual use. |
| Qwen3 1.7B | ~1.1 GB | Sweet spot for character personalities with specific phrasing. |
| Qwen3 4B | ~2.5 GB | Full quality. Slowest. Useful for very particular tone. |
The model runs through the same backend split Voicebox already uses for TTS
— **MLX** (4-bit community quants) on Apple Silicon, **PyTorch** (transformers
`AutoModelForCausalLM`) everywhere else. Downloads go through the same cache
and model-management UI as TTS models.
Pick a size in **Settings → Captures → Refinement → Refinement model** — the
personality modes reuse it. If you switch models, both refinement and
personality output pick up the change on the next call.
## Using the modes
The three actions appear as buttons on the profile when a personality is
set. For each:
<Steps>
<Step title="Type or dictate your input">
Rewrite and Respond need input text. Compose doesn't.
</Step>
<Step title="Click the mode">
The LLM runs, then the result fills the generate box.
</Step>
<Step title="Hit generate">
The TTS engine speaks the LLM output in the profile's voice. The result
lands in generation history as a normal generation.
</Step>
</Steps>
Each button also has an inline regenerate affordance — click again to
resample. Compose will give you something totally different; Rewrite and
Respond will give you a variation on the same content.
## Use cases
- **Agents that speak in a voice you own.** Combine Respond with the
built-in [MCP Server](/overview/mcp-server) so Claude Code, Cursor,
Cline, or any MCP-aware agent can talk back through a profile with a
personality. The agent calls `voicebox.speak({ text, profile, intent:
"respond" })` and Voicebox produces in-character speech in your cloned
voice.
- **Interactive characters.** Games, narrative tools, accessibility
experiences. A character with a personality description plus a cloned
voice becomes a reusable prop.
- **Accessibility.** People who can't speak in their original voice can
keep a personality description of how they used to sound and use Rewrite
to turn typed input into in-character speech.
- **Creative drafting.** Write a plain outline, Rewrite line-by-line into
the character's voice, drop the audio into a Story.
## API surface
Personalities and the three modes are accessible via REST:
| Method | Endpoint | Body |
|---|---|---|
| `PUT` | `/profiles/{id}` | Include a `personality` field up to 2,000 chars to set it. |
| `POST` | `/profiles/{id}/speak` | Runs the LLM + TTS in one shot. Body includes `text`, `intent` (`compose`, `rewrite`, or `respond`), optional `engine`, `language`. |
The `/profiles/{id}/speak` endpoint is the same primitive MCP's
`voicebox.speak` tool calls when that ships. Scripts and agents can use it
directly today.
## Limits and gotchas
- **The personality is a prompt, not a fine-tune.** The LLM will sometimes
drift out of character, especially on Compose at high temperature. Click
again for another take.
- **Long personalities are not always better.** 2,000 chars is a ceiling,
not a goal. A sharp 300-char description with two example lines
typically outperforms a long one.
- **Speech-only framing is enforced, but not bulletproof.** Very large
prompts or unusual inputs can sneak an action tag through. If you see
`[laughs]` in TTS output, it's usually a personality-field hint the
model anchored onto — remove it from the description.
- **Rewrite is stricter than Respond.** If the output is changing your
meaning, you probably want Respond (or a wholesale Compose with context
in the input), not Rewrite.
## Next steps
<Cards>
<Card title="Dictation" href="/overview/dictation">
Dictate the input for Rewrite or Respond from anywhere on your machine.
</Card>
<Card title="Captures" href="/overview/captures">
Captures feed personalities naturally — dictate a memo, rewrite it in
a character voice, generate speech.
</Card>
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
Add a personality to an existing profile.
</Card>
</Cards>
docs/plans/MCP_SERVER.md
+344
View File
@@ -0,0 +1,344 @@
# MCP Server — Voicebox Speed Run
**Status:** v1 shipped — HTTP transport, all 4 tools, per-client bindings, `POST /speak`, stdio shim (binary built, bundled into Tauri sidecar), Settings UI, speak-pill via SSE with Rust-side `dictate:show` handler so agent-initiated speech surfaces the pill on screen. `cargo check` clean, `tsc` clean, full Inspector round-trip verified.
**Last reviewed:** 2026-04-23
## Status
### Shipped (backend)
- **`fastmcp` + `sse-starlette`** pinned in `backend/requirements.txt`.
- **`backend/mcp_server/`** package with `server.py`, `tools.py`, `context.py`, `resolve.py`, `events.py`, `README.md`. Named `mcp_server` (not `mcp`) to sidestep a shadowing conflict with the installed `mcp` PyPI package that FastMCP imports internally.
- **Streamable HTTP mount at `/mcp`** via FastMCP's `http_app(transport='http')`. Sub-app lifespan composed with Voicebox's own startup/shutdown through an `@asynccontextmanager lifespan=` in `backend/app.py` (migrated away from the deprecated `@app.on_event` handlers).
- **Four MCP tools**, dot-named to match the landing and ecosystem convention:
- `voicebox.speak(text, profile?, engine?, intent?, language?)`
- `voicebox.transcribe(audio_base64?, audio_path?, language?, model?)`
- `voicebox.list_captures(limit, offset)`
- `voicebox.list_profiles()`
- **`ClientIdMiddleware`** pulls `X-Voicebox-Client-Id` into a `ContextVar` on every `/mcp*` request; auto-stamps `MCPClientBinding.last_seen_at`, auto-creating the row if the client is new.
- **Profile resolution precedence** `explicit → per-client binding → capture_settings.default_playback_voice_id → error`. `services/profiles.get_profile_orm_by_name_or_id()` lets agents pass a voice by name ("Morgan") instead of UUID.
- **`MCPClientBinding` table** (new) via `Base.metadata.create_all` — no migration needed.
- **Bindings REST:** `GET|PUT /mcp/bindings`, `DELETE /mcp/bindings/{client_id}`.
- **`POST /speak`** REST wrapper for non-MCP callers (shell / ACP / A2A). Same `resolve_profile` precedence, same code path as the MCP tool.
- **Stdio shim** at `backend/mcp_shim/__main__.py` — ~200 lines of `httpx` proxy; reads env (`VOICEBOX_PORT`, `VOICEBOX_HOST`, `VOICEBOX_CLIENT_ID`), waits for `/health`, then streams JSON-RPC ↔ SSE. Rolled our own after the `mcp` SDK's session-management helpers mis-shook-hands. Smoke-tested: `initialize`, `tools/list`, and `tools/call` all round-trip cleanly.
- **Pill SSE:** `GET /events/speak` (`sse-starlette`) emits `speak-start` from the MCP tool and `POST /speak`, `speak-end` from `services/generation.run_generation`'s finally block.
- **PyInstaller:**
- `backend/build_binary.py` `--shim` flag builds a minimal `voicebox-mcp` binary (torch/transformers/mlx/etc. explicitly excluded, target <20 MB).
- The main server spec picks up `fastmcp`, `mcp`, `sse_starlette`, and `backend.mcp_server.*` via `--collect-all` / `--hidden-import`.
- **`backend/mcp_server/README.md`** quickstart (Inspector, `.mcp.json` snippets, tool reference).
### Shipped (frontend)
- **`Settings → MCP`** page (`app/src/components/ServerTab/MCPPage.tsx`):
- Three copy-paste snippets auto-filled with the detected `serverUrl`: HTTP (recommended), Claude Code CLI one-liner, stdio fallback.
- Default voice picker (bound to `capture_settings.default_playback_voice_id`, shared with Captures-tab "Play as voice").
- Per-client bindings table with inline profile picker, remove button, and a connection-status indicator that refreshes every 10 s.
- Add-binding form with client_id / label / profile dropdown.
- **`useMCPBindings`** TanStack hook (optimistic delete, invalidate on upsert).
- **`useSpeakEvents`** hook — auto-reconnecting `EventSource('/events/speak')`, tracks the active generation_id, exposes an elapsed-ms timer that ticks so the pill's clock advances.
- **`CapturePill`** has a new `'speaking'` state + "Speaking" label + playing-bars mode.
- **`DictateWindow`** subscribes to speak events and overrides `pillState` when an agent is speaking. Emits `dictate:show` on speak-start so the Rust side can surface the pill window.
- Router + `ServerTab` tab bar wired to `/settings/mcp`.
### Shipped (native shell)
- **`tauri.conf.json`** — `voicebox-mcp` added to `externalBin` (alongside `voicebox-server`).
- **`dictate:show` listener** in `tauri/src-tauri/src/main.rs` — invokes a new `show_dictate_window(app_handle)` helper that mirrors the hotkey-monitor's position+show logic (undo click-through, reposition to top-center of the current monitor, show). Agent-initiated speech now pops the pill visible on screen.
### Validated end-to-end (this session, via curl)
- `/mcp/` init → `tools/list` → `tools/call voicebox.speak` → actual audio plays (Jarvis, 1.68 s).
- `POST /speak` with `X-Voicebox-Client-Id: claude-code` resolves to the bound Jarvis profile without passing `profile`.
- `/events/speak` emits `ready`, `speak-start`, `speak-end` in order, generation_id threads through both.
- Stdio shim: `echo {…} | python -m backend.mcp_shim` returns valid JSON-RPC for all 4 methods.
- `last_seen_at` auto-stamps on first call; binding row auto-creates.
- Frontend `tsc --noEmit`: clean.
- `cargo check` on the Tauri crate: clean.
### Outstanding (must-do before release)
- **CI build for shim on Windows/Linux** — `python backend/build_binary.py --shim` is wired up and built cleanly for `aarch64-apple-darwin` (18 MB, installed at `tauri/src-tauri/binaries/voicebox-mcp-aarch64-apple-darwin`, Tauri `cargo check` green). The Windows and Linux triples (`x86_64-pc-windows-msvc`, `x86_64-unknown-linux-gnu`) need the same build in their respective CI runners and artifacts dropped alongside the macOS binary.
- **Windows/Linux paths in the stdio snippet** — the Settings page hardcodes the macOS path (`/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp`). Needs a per-OS switch (`%LOCALAPPDATA%\Programs\Voicebox\voicebox-mcp.exe`, Linux bundled-path), ideally with the Tauri shell resolving its own app path at runtime and injecting it into the snippet.
### Nice-to-have (follow-up passes)
- **One-click install buttons** — write/merge into `~/.claude/settings.json`, `~/.cursor/mcp.json`, etc. via a Tauri command. Copy-paste works today; this is pure ergonomics.
- **`.mcpb` desktop extension** for Claude Desktop (single file, double-click to install). Claude Desktop-only, so lower priority than the agent-harness crowd.
- **Refactor the hotkey_monitor.rs show-logic** to call `show_dictate_window()` instead of duplicating the position+show block. Skipped at ship to avoid regressing the well-tested chord path.
- **Source attribution on `Generation.source`** — currently `"manual" | "personality_speak"`; adding `"mcp"` / `"rest"` would let the Captures tab filter by MCP-originated rows.
## Context
Voicebox already ships the I/O surface (Captures, Generate, personality-driven `/profiles/{id}/speak`), but local AI agents can't reach any of it. This plan adds a Model Context Protocol server so Claude Code / Cursor / Cline can call `voicebox.speak`, `voicebox.transcribe`, `voicebox.list_captures`, and `voicebox.list_profiles` — turning Voicebox into the local voice layer for every agent on the user's machine (Phase 5 of `docs/plans/VOICE_IO.md`).
The shortest path to "Claude Code speaks in a cloned voice": mount **FastMCP** inside the existing FastAPI/uvicorn process at `/mcp` (Streamable HTTP), and users install it as a URL (`{"url": "http://127.0.0.1:17493/mcp"}`) — the ecosystem-idiomatic shape for a long-running local service. Per-client voice binding via a new `mcp_client_bindings` table + Settings UI, resolved from an `X-Voicebox-Client-Id` header. A **stdio shim binary** `voicebox-mcp` is bundled as a fallback sidecar for clients that can't speak HTTP MCP. A public `POST /speak` REST wrapper covers non-MCP callers (shell scripts, ACP, A2A). A `speaking` pill state gives agent-initiated audio visibility — trust-critical, non-negotiable.
## Architecture
```
Claude Code / Cursor / Windsurf / VS Code MCP
│
├─ HTTP (primary) ────────────────────┐
│ {"url": ".../mcp"} │
│ │
└─ stdio (fallback) ───────────────▶ [voicebox-mcp shim binary]
{"command": "/abs/path/voicebox-mcp"} (absolute path;
│ Settings page
│ copies it for you)
▼
uvicorn + FastAPI (port 17493)
├─ /mcp (FastMCP, Streamable HTTP)
└─ /speak (REST wrapper for non-MCP callers)
└─ tools call existing services
```
- **Transport:** Streamable HTTP as primary (Nov-2025 spec, post-SSE). Claude Code, Cursor, Windsurf, and the VS Code MCP extensions all support HTTP — it's the idiomatic shape for a long-running local service, which Voicebox already is.
- **Stdio fallback:** `voicebox-mcp` binary bundled inside the app for clients that can't speak HTTP MCP. The Settings page renders the exact snippet with the detected absolute path — user copies, pastes, done. No PATH manipulation, no custom CLI wrapper.
- **Identity:** HTTP clients set `X-Voicebox-Client-Id` header in their MCP config's `headers` block. Stdio clients set `VOICEBOX_CLIENT_ID` env var, which the shim forwards as the same HTTP header. Server reads it into a `ContextVar`.
- **Profile resolution precedence:** explicit tool arg → per-client `MCPClientBinding.profile_id` → `capture_settings.default_playback_voice_id` → error.
- **Port:** `17493`, matching `tauri/src-tauri/src/main.rs:63` (`SERVER_PORT` constant). Shim default with `VOICEBOX_PORT` env override.
- **Non-MCP access:** `POST /speak` is a thin REST wrapper around the same tool path — one endpoint for shell scripts, ACP, A2A, and anything that isn't MCP-native.
## Library choice
- **`fastmcp`** (PyPI — verify on install whether the canonical import is `fastmcp` standalone or `mcp.server.fastmcp` from the consolidated `mcp` package; the API is identical).
- **`sse-starlette`** for the `/events/speak` pill-state broadcast.
- **`httpx` + `anyio`** already present — used by the shim.
## Data model
New table, **one row per client_id** (not a singleton — scales to unknown clients, maps 1:1 to the Settings UI list):
```python
# backend/database/models.py
class MCPClientBinding(Base):
__tablename__ = "mcp_client_bindings"
client_id = Column(String, primary_key=True) # "claude-code", "cursor", ...
label = Column(String, nullable=True)
profile_id = Column(String, ForeignKey("profiles.id"), nullable=True)
default_engine = Column(String, nullable=True)
default_intent = Column(String, nullable=True) # "respond" | "rewrite" | "compose"
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
```
Global default stays in `capture_settings.default_playback_voice_id` — no duplication. Migration: new `_migrate_mcp_client_bindings()` in `backend/database/migrations.py` using `CREATE TABLE IF NOT EXISTS`, mirroring the existing idempotent-add-column pattern.
## File plan
### Backend — new
| File | Purpose |
|---|---|
| `backend/mcp/__init__.py` | Package marker |
| `backend/mcp/server.py` | `build_mcp_server()` + `mount_into(app)`; registers tools, middleware, mount at `/mcp` |
| `backend/mcp/tools.py` | The 4 `@mcp.tool()` functions — thin wrappers over existing services |
| `backend/mcp/context.py` | `current_client_id: ContextVar[str \| None]` + Starlette middleware |
| `backend/mcp/resolve.py` | `resolve_profile(explicit, client_id, db) -> VoiceProfile \| None` |
| `backend/mcp/events.py` | In-memory `asyncio.Queue` pub/sub for speak-start / speak-end |
| `backend/mcp/README.md` | MCP Inspector quickstart + `.mcp.json` snippets |
| `backend/mcp_shim/__init__.py`, `__main__.py` | Stdio ↔ Streamable HTTP proxy (~150 lines) |
| `backend/voicebox-mcp.spec` | PyInstaller spec for the shim (strips torch/transformers from `hiddenimports`) |
| `backend/routes/speak.py` | `POST /speak {text, profile?, engine?, intent?, language?}` — REST wrapper around `resolve_profile()` + `speak_in_character()` for non-MCP agents |
### Backend — modified
| File | Change |
|---|---|
| `backend/app.py` | Migrate `@app.on_event("startup"/"shutdown")` (lines 185, 268) to `lifespan=` kwarg on `FastAPI()` using `AsyncExitStack`; call `mount_into(application)` after `register_routers`. Register `ClientIdMiddleware`. |
| `backend/routes/profiles.py` | In `speak_in_character` (line 453): `events.publish("speak-start", {...})` on entry; completion hook publishes `speak-end`. Accept optional `source="mcp"` marker. |
| `backend/services/generation.py` | `run_generation` completion path publishes `speak-end`. |
| `backend/services/profiles.py` | New `async def get_profile_by_name_or_id(name_or_id, db)` — id lookup first, case-insensitive name fallback. |
| `backend/database/models.py` | Add `MCPClientBinding`. |
| `backend/database/migrations.py` | Add `_migrate_mcp_client_bindings`. |
| `backend/models.py` | Add `MCPClientBindingResponse`, `MCPClientBindingUpdate`. |
| `backend/routes/__init__.py` | Register `mcp_bindings_router`, `speak_router`, `events_router`. |
| `backend/routes/mcp_bindings.py` (new) | REST CRUD for bindings (list, upsert, delete). |
| `backend/routes/events.py` (new) | `GET /events/speak` — `EventSourceResponse` subscribed to the events queue. |
| `backend/requirements.txt` | `+ fastmcp` (or `mcp>=1.0`), `+ sse-starlette` |
| `backend/voicebox-server.spec` | `hiddenimports += ['mcp', 'mcp.server', 'fastmcp']` |
| `backend/build_binary.py` | Second PyInstaller invocation for `voicebox-mcp.spec`; copy to `tauri/src-tauri/binaries/` with target-triple suffix |
### Frontend — new
| File | Purpose |
|---|---|
| `app/src/components/ServerSettings/MCPBindings.tsx` | Settings section — default voice + per-client binding rows + `.mcp.json` copy-paste cheatsheet |
| `app/src/lib/hooks/useMCPBindings.ts` | TanStack Query mirror of `useCaptureSettings` |
| `app/src/lib/api/mcp.ts` | `listMCPBindings` / `upsertMCPBinding` / `deleteMCPBinding` |
### Frontend — modified
| File | Change |
|---|---|
| `app/src/components/DictateWindow/DictateWindow.tsx` | Open `EventSource('/events/speak')`; on `speak-start` set pill to `speaking` with profile name; dismiss on `speak-end`. |
| `app/src/components/CapturePill/CapturePill.tsx` | Add `speaking` branch — reuse the active waveform, swap status label to profile name. |
| `app/src/lib/hooks/useCaptureRecordingSession.ts` | Union a `speaking` injection into the derived pill state. |
| `app/src/lib/api/types.ts` | `MCPClientBinding`, `MCPClientBindingUpdate` types. |
| `app/src/components/ServerSettings/index.tsx` | Register the new MCP section in the tab aggregator. |
### Tauri
| File | Change |
|---|---|
| `tauri/src-tauri/tauri.conf.json` | `"externalBin": ["binaries/voicebox-server", "binaries/voicebox-mcp"]` |
| `tauri/src-tauri/binaries/voicebox-mcp-<triple>` | Build artifact from PyInstaller |
## Tool signatures
All tools read `current_client_id.get()` (from middleware). Return JSON-serializable dicts.
Tools are registered with **dotted names** (`voicebox.speak`, etc.) to match the landing page and the industry convention (`filesystem.read_file`, `github.create_issue`). Python function names stay snake_case; the dot goes in the `name=` kwarg.
```python
# backend/mcp/tools.py
@mcp.tool(name="voicebox.speak")
async def speak(text: str,
profile: str | None = None, # name OR id
engine: str | None = None,
intent: str = "respond", # "respond" | "rewrite" | "compose"
language: str | None = None) -> dict:
"""Speak text in a voice profile. Returns {generation_id, status, profile, poll}."""
# resolve profile via precedence, call speak_in_character (profiles.py:453)
# with persist=True so it lands in history.
@mcp.tool(name="voicebox.transcribe")
async def transcribe(audio_base64: str | None = None,
audio_path: str | None = None, # absolute local path
language: str | None = None,
model: str | None = None) -> dict:
"""Transcribe audio. Exactly one of audio_base64/audio_path. Returns {text, duration, language}."""
# validate path readable, size < 200 MB, then call services.transcribe.transcribe_bytes
@mcp.tool(name="voicebox.list_captures")
async def list_captures(limit: int = 20, offset: int = 0) -> dict:
"""Recent captures with transcripts. Returns {captures: [...]}"""
@mcp.tool(name="voicebox.list_profiles")
async def list_profiles() -> dict:
"""Available voice profiles. Returns {profiles: [{id, name, voice_type, has_personality}]}"""
```
### `POST /speak` (non-MCP REST wrapper)
```python
# backend/routes/speak.py
@router.post("/speak", response_model=GenerationResponse)
async def speak(data: SpeakRequest, request: Request, db: Session = Depends(get_db)):
"""Same behavior as the MCP tool — for shell scripts, ACP, A2A, or anything non-MCP."""
client_id = request.headers.get("X-Voicebox-Client-Id")
profile = resolve_profile(data.profile, client_id, db)
if profile is None: raise HTTPException(400, "No voice profile resolved.")
persist_req = PersonalitySpeakRequest(text=data.text, persist=True, language=data.language,
engine=data.engine, intent=data.intent or "respond")
return await speak_in_character(profile.id, persist_req, db)
```
`SpeakRequest`: `{ text: str, profile: str | None, engine: str | None, intent: str | None, language: str | None }`. Accepts name OR id for `profile` (via `resolve_profile`), and resolves via the same precedence as the MCP tool so the two surfaces behave identically.
## Mount point (`backend/app.py`)
```python
# After register_routers(application):
from .mcp.server import mount_into
mount_into(application)
```
`mount_into` installs `ClientIdMiddleware` and calls `app.mount("/mcp", mcp.streamable_http_app())`.
**Lifespan migration is load-bearing** — FastMCP's session manager requires the `lifespan=` kwarg, not `@app.on_event`. Wrap the existing startup/shutdown bodies in an `@asynccontextmanager` using `contextlib.AsyncExitStack` so both Voicebox's init and FastMCP's session manager run. Verify dev + packaged build after the migration.
## Stdio shim (`backend/mcp_shim/__main__.py`)
1. Port: `int(os.environ.get("VOICEBOX_PORT", "17493"))`.
2. Client id: `os.environ.get("VOICEBOX_CLIENT_ID", "unknown")`.
3. Health probe `GET /health` with 30 s tolerance (torch imports slowly). On failure, emit JSON-RPC error on stdout, exit 1.
4. Connect Streamable HTTP MCP client to `http://127.0.0.1:{port}/mcp` with `X-Voicebox-Client-Id: {client_id}` header.
5. Proxy JSON-RPC bidirectionally — stdin → HTTP, SSE → stdout. Use `mcp` SDK's built-in stdio↔HTTP bridge if available; otherwise ~40 lines of asyncio.
6. Stdout = JSON-RPC only. All logs to stderr.
PyInstaller spec keeps only `mcp`, `httpx`, `anyio`, `click` — target binary <20 MB.
## Pill `speaking` state
- `backend/mcp/events.py`: module-level `_subscribers: list[asyncio.Queue]` + `publish(kind, payload)` + `subscribe() -> Queue`.
- `speak_in_character` publishes `speak-start` with `{generation_id, profile_id, profile_name, source}` immediately after `task_manager.start_generation`; `run_generation`'s completion path publishes `speak-end`.
- `/events/speak` → `EventSourceResponse`.
- `DictateWindow` opens `EventSource` next to existing `dictate:*` listeners, maps `speak-start/end` → pill `speaking` mode with profile name.
- Optional filter: only show pill when `source === "mcp"` (avoids pill churn during manual speak flows). Settings toggle later.
## Settings UI (`MCPBindings.tsx`)
- **Global default voice** picker bound to `capture_settings.default_playback_voice_id` (reuses `useCaptureSettings`).
- **Per-client table** — add/edit/remove rows of `{client_id, label, profile_id, default_engine, default_intent}`. Uses `useMCPBindings`.
- **Connection cheatsheet** — two tabs, HTTP (default) and Stdio (fallback), with copy-to-clipboard snippets per known client:
HTTP form (primary):
```json
{"mcpServers": {"voicebox": {
"url": "http://127.0.0.1:17493/mcp",
"headers": {"X-Voicebox-Client-Id": "claude-code"}
}}}
```
Stdio form (fallback, absolute path auto-filled from detected app location):
```json
{"mcpServers": {"voicebox": {
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
"env": {"VOICEBOX_CLIENT_ID": "claude-code"}
}}}
```
Plus the Claude-Code-specific one-liner:
```
claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"
```
- **One-click install buttons** for known clients (v1: Claude Code via `claude mcp add` invocation, and a config-file writer for Cursor/Windsurf whose config locations are known). Each has a matching "Remove" button. Hide buttons for clients not detected on disk.
- **Connection status** — small indicator next to each binding showing the last time that `client_id` actually called the server (rolling timestamp recorded by middleware), so users can tell their install worked.
## Ordered task list (shortest path first)
1. `fastmcp` + `sse-starlette` → `backend/requirements.txt`; install.
2. Add `backend/mcp/{server,tools,context,resolve}.py` with the 4 tools registered as `voicebox.speak` etc. (no middleware yet — global default profile only).
3. Migrate `app.py` to `lifespan=`; mount FastMCP at `/mcp`.
4. **Milestone:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp` — call `voicebox.speak`, hear audio.
5. Add `get_profile_by_name_or_id`; wire the tool's `profile` arg.
6. `MCPClientBinding` model + migration; middleware; full `resolve_profile` precedence.
7. `backend/routes/speak.py` — `POST /speak` REST wrapper, reusing `resolve_profile` + `speak_in_character`.
8. `/mcp/bindings` REST + `MCPBindings.tsx` UI with HTTP and stdio copy-snippets, one-click install for detected clients, and connection-status indicators. **Users can install Voicebox as an MCP server after this step.**
9. `backend/mcp_shim/__main__.py` + PyInstaller spec + `build_binary.py` second pass; register `voicebox-mcp` as a Tauri sidecar. (Fallback path goes live.)
10. Events queue + `/events/speak` SSE + `DictateWindow` `speaking` pill state.
11. `backend/mcp/README.md` quickstart.
Claude Code can call `voicebox.speak` after step 4 (direct HTTP, manual config). Step 8 makes that a one-click experience. Step 9 adds the stdio fallback for clients that don't speak HTTP MCP.
## Verification
- **Step 4 smoke:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp`. Call `voicebox.list_profiles`, then `voicebox.speak(text="hello from mcp")`. Audio plays; generation appears in History with `source="personality_speak"` (or new `source="mcp"` if we add one).
- **REST wrapper:** `curl -X POST http://127.0.0.1:17493/speak -d '{"text":"hi","profile":"Morgan"}'` — same behavior, same pill surface.
- **Per-client:** open two Inspector sessions with different `X-Voicebox-Client-Id` headers, bind each to a different profile in Settings, verify distinct voices without `profile` arg.
- **Claude Code end-to-end (HTTP):** `claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"`, then ask Claude Code to speak. Pill shows `speaking: <profile>`, audio plays, capture appears in history.
- **Stdio fallback:** manually paste the stdio snippet from Settings into a client's config, verify same behavior. `VOICEBOX_CLIENT_ID=claude-code python -m backend.mcp_shim` while backend is up; pipe a tools/list JSON-RPC in, verify response over stdout.
- **Transcribe:** point at `/tmp/test.wav`; diff against `POST /transcribe` response.
- **Failure modes:** kill backend mid-speak — shim must surface a JSON-RPC error, not deadlock. When backend isn't running, HTTP clients should get a clear connection-refused surfaced by the client.
## Risks / open decisions
- **`fastmcp` vs `mcp` package name** — confirm on `pip install`; APIs are near-identical, adjust imports.
- **Lifespan migration** touches critical path (DB init, task queue, watchdog). Dev + packaged build both need a smoke after.
- **Shim binary size** — if `mcp` pulls in enough dep weight that PyInstaller output is awkward, fall back to a Rust shim (Tauri shell is already Rust; JSON-RPC framing is trivial).
- **Source attribution** — consider `source="mcp"` on the `Generation` model, or a dedicated `originator_client` column, if the Captures tab should filter MCP-originated generations.
- **`audio_path` in `voicebox_transcribe`** — local-only today, but if the server ever binds beyond 127.0.0.1 we need to restrict reads to `data_dir` + user-whitelist.
- **Auth** — none for now (127.0.0.1 only). If we bind outside, bearer token via `~/.voicebox/secret` + plumb through shim.
- **HTTP MCP client support** — the plan leads with direct HTTP. Claude Code, Cursor, Windsurf, and VS Code MCP extensions all support it as of 2026, but if we discover an important client is stdio-only we still have the shim fallback ready.
- **`.mcpb` desktop extension for Claude Desktop** (v2 polish) — Claude Desktop supports a double-clickable extension bundle format. Worth revisiting after v1 ships for an even cleaner install; skipped for now since Claude Desktop isn't the primary user (Claude Code + IDE users are).
## Critical files
- `backend/app.py`
- `backend/routes/profiles.py`
- `backend/routes/speak.py` (new)
- `backend/database/models.py`
- `backend/database/migrations.py`
- `backend/services/generation.py`
- `backend/build_binary.py`
- `tauri/src-tauri/tauri.conf.json`
- `tauri/src-tauri/src/main.rs` (port constant — no change, just reference)
- `app/src/components/DictateWindow/DictateWindow.tsx`
- `app/src/components/CapturePill/CapturePill.tsx`
- `app/src/components/ServerSettings/`