mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-16 13:20:39 -07:00
feat(mcp): local MCP server exposes voicebox.* tools to AI agents
Mounts FastMCP at /mcp (Streamable HTTP) so Claude Code, Cursor, Windsurf, and the VS Code MCP extensions can call voicebox.speak, voicebox.transcribe, voicebox.list_captures, and voicebox.list_profiles against the running Voicebox server. Backend - new backend/mcp_server package (tools, middleware, profile resolve, pub/sub events); named mcp_server to avoid shadowing the installed mcp PyPI package FastMCP imports internally - app.py migrated from @app.on_event to lifespan= so FastMCP's session manager cohabits with Voicebox's startup/shutdown - new MCPClientBinding table + /mcp/bindings CRUD; ClientIdMiddleware reads X-Voicebox-Client-Id into a ContextVar and stamps last_seen_at - profile resolution precedence: explicit -> per-client binding -> capture_settings.default_playback_voice_id - POST /speak REST wrapper for non-MCP callers (shell, ACP, A2A) - GET /events/speak SSE broadcasts speak-start / speak-end so the pill surfaces agent-initiated speech - backend/mcp_shim proxy (plain httpx) for stdio-only MCP clients - PyInstaller spec updates + new --shim build target (~18 MB) Frontend - Settings -> MCP page with HTTP / stdio / claude-mcp-add copy snippets, default voice picker, per-client bindings table, connection status - useMCPBindings, useSpeakEvents hooks - CapturePill gains 'speaking' state; DictateWindow subscribes to SSE and emits dictate:show so the Rust side surfaces the pill window Native - tauri.conf.json externalBin now includes voicebox-mcp - show_dictate_window helper + dictate:show listener in main.rs - (also in this commit: InputMonitoringGate UX, hotkey_monitor tweaks, landing footer/navbar updates, new overview docs for captures / dictation / mcp-server / voice-personalities) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
James Pine
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
87c582ad54
commit
0cef2c9fe1
+25
-16
@@ -1,23 +1,30 @@
|
||||
---
|
||||
title: "Voicebox Documentation"
|
||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
|
||||
---
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is the **open-source, local-first AI voice studio** — a free
|
||||
alternative to ElevenLabs and WisprFlow in one app. Clone voices, generate
|
||||
speech across 7 TTS engines, dictate into any app with a global hotkey,
|
||||
compose multi-voice projects, and let any MCP-aware agent speak in a voice
|
||||
you own. Everything runs on your hardware.
|
||||
|
||||
|
||||
|
||||
- **Complete privacy** -- models and voice data stay on your machine
|
||||
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
|
||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** -- REST API for integrating voice synthesis into your own projects
|
||||
- **Native performance** -- built with Tauri (Rust), not Electron
|
||||
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
|
||||
- **Dictation** — hold a chord anywhere on your machine, speak, release; the transcript pastes into the focused field
|
||||
- **Captures tab** — paired audio + transcript archive, retranscribe / refine / play-as-voice
|
||||
- **Voice personalities** — per-profile compose / rewrite / respond, powered by a local LLM
|
||||
- **Agents speak back** — any MCP-aware agent can call Voicebox to speak in one of your cloned voices
|
||||
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, Kokoro
|
||||
- **Cloning and preset voices** — zero-shot cloning or 50+ curated preset voices
|
||||
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, filters
|
||||
- **Expressive speech** — paralinguistic tags (`[laugh]`, `[sigh]`) and natural-language delivery control
|
||||
- **Unlimited length** — auto-chunking with crossfade for long scripts
|
||||
- **Stories editor** — multi-track timeline for conversations, podcasts, narratives
|
||||
- **API-first** — REST + WebSocket API, MCP server for agent integrations
|
||||
- **Complete privacy** — models, audio, transcripts, LLM output never leave your machine
|
||||
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux (ROCm / CPU), Intel Arc, Docker
|
||||
|
||||
## Download
|
||||
|
||||
@@ -32,6 +39,8 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
|
||||
|
||||
## Get Started
|
||||
|
||||
- [Installation](/overview/installation) -- download and install Voicebox
|
||||
- [Quick Start](/overview/quick-start) -- get up and running in 5 minutes
|
||||
- [API Reference](/api-reference) -- integrate voice synthesis into your apps
|
||||
- [Installation](/overview/installation) — download and install Voicebox
|
||||
- [Quick Start](/overview/quick-start) — get up and running in 5 minutes
|
||||
- [Dictation](/overview/dictation) — start talking to your computer
|
||||
- [Voice Personalities](/overview/voice-personalities) — compose, rewrite, respond in any profile
|
||||
- [API Reference](/api-reference) — integrate voice synthesis into your apps
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: "Captures"
|
||||
description: "The paired audio + transcript archive — every dictation, recording, and uploaded audio file shows up here, replayable and retranscribable."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A **capture** is an audio clip paired with its transcript. The Captures tab
|
||||
is where every dictation, manual recording, and uploaded audio file lands,
|
||||
with the original audio kept alongside the text so you can replay, re-run
|
||||
transcription with a different model, refine the transcript, or send the
|
||||
content somewhere else — including generating it back as speech in any of
|
||||
your voice profiles.
|
||||
|
||||
<Callout type="info">
|
||||
The Captures tab shipped in **0.5.0**, alongside global dictation and the
|
||||
per-profile personality modes. If you've used earlier versions, note that
|
||||
the Audio tab moved into **Settings → Audio Channels** to make room for
|
||||
this one.
|
||||
</Callout>
|
||||
|
||||
## Where captures come from
|
||||
|
||||
| Source | How it shows up | Badge |
|
||||
|---|---|---|
|
||||
| **Dictation** | Triggered by the global hotkey (see [Dictation](/overview/dictation)). Auto-refined by default. | `dictation` |
|
||||
| **In-app recording** | Recorded directly in the Captures tab using the built-in mic. | `recording` |
|
||||
| **File upload** | Any audio file dropped into the Captures tab — `.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`. | `file` |
|
||||
|
||||
All three paths share the same backend pipeline, the same model picker, and
|
||||
the same refinement flags. The source badge is there so you can visually
|
||||
scan a long list.
|
||||
|
||||
## List view
|
||||
|
||||
The main Captures view is a chronological list. Each row shows:
|
||||
|
||||
- The transcript (raw or refined — the refined version wins if present)
|
||||
- Duration + timestamp
|
||||
- Source badge
|
||||
- A play button for the original audio
|
||||
- A meatballs menu with per-row actions
|
||||
|
||||
Filtering and search are a Tier-2 ask — ping if you need them.
|
||||
|
||||
## Detail view
|
||||
|
||||
Clicking into a capture opens the detail view:
|
||||
|
||||
- **Waveform player** for the original audio
|
||||
- **Transcript editor** — click in and edit. Changes save on blur.
|
||||
- **Refined vs. raw toggle** if refinement ran on this capture
|
||||
- **Per-capture action bar** — retranscribe, refine, play as voice, delete
|
||||
- **Settings snapshot** — STT model used, refinement flags at the time
|
||||
this capture was processed, and the voice model if any was played
|
||||
|
||||
## Retranscribe
|
||||
|
||||
Runs the capture's original audio through a different Whisper model without
|
||||
re-uploading or re-refining anything. Useful when:
|
||||
|
||||
- The default model mis-heard something and you want to try a larger model
|
||||
- You used Base for a noisy clip and want to rerun with Turbo
|
||||
- A non-English clip needs an explicit language hint
|
||||
|
||||
**Settings → Captures → Transcription** controls the default model and
|
||||
language lock for new captures. Retranscribe uses those defaults unless you
|
||||
override them per capture.
|
||||
|
||||
## Refine
|
||||
|
||||
Runs the raw transcript through the local LLM to produce a cleaned-up
|
||||
version. The flags on the capture are snapshotted when refinement first
|
||||
runs, so you can re-refine later with different flags without losing the raw
|
||||
transcript:
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| **Smart cleanup** | Remove fillers (`um`, `uh`, `like`), tidy punctuation and capitalization. |
|
||||
| **Remove self-corrections** | Keep the final version when the speaker backtracks ("actually, no, on Tuesday"). |
|
||||
| **Preserve technical terms** | Leave identifiers (`handleSubmit`, `npm install`) untouched. |
|
||||
|
||||
See the Refinement section of [Dictation](/overview/dictation#refinement) for
|
||||
how Voicebox strips Whisper loop hallucinations *before* the LLM sees the
|
||||
transcript — a capture can be re-refined any number of times without
|
||||
re-introducing "thanks for watching thanks for watching" echoes.
|
||||
|
||||
The refinement model picker (three bundled Qwen3 sizes) lives in
|
||||
**Settings → Captures → Refinement**.
|
||||
|
||||
## Play as voice
|
||||
|
||||
This is the capability no one else in the dictation category ships: take any
|
||||
capture and play it back as speech in any of your voice profiles. One
|
||||
dropdown over every profile, one click, and the capture's text runs through
|
||||
`/generate` with the selected voice.
|
||||
|
||||
Use cases:
|
||||
|
||||
- Hear your own dictation back in a cloned voice of someone you like
|
||||
- Send a message you dictated as an audio reply in a specific character
|
||||
- Quickly prototype a line for a story without retyping
|
||||
|
||||
Playback uses whatever engine the selected profile is bound to — the same
|
||||
rules as the Generate tab. There's no LLM in this path; the transcript goes
|
||||
through unchanged. If you want the agent-style "transform the content before
|
||||
speaking" flow, that's what the
|
||||
[personality modes](/overview/voice-personalities) do — and the same
|
||||
primitive is exposed to MCP-aware agents via the
|
||||
[MCP Server](/overview/mcp-server) so Claude Code, Cursor, or Cline can speak
|
||||
in one of your voices on their own.
|
||||
|
||||
<Callout type="info">
|
||||
The default voice for the Captures tab's Play-as action is set in
|
||||
**Settings → Captures → Playback → Default voice**. You can still override
|
||||
it per capture.
|
||||
</Callout>
|
||||
|
||||
## Send-to menu
|
||||
|
||||
Each capture has a Send-to menu for moving its content into other parts of
|
||||
Voicebox:
|
||||
|
||||
- **Copy transcript** — to clipboard
|
||||
- **Use as voice sample…** — promote this capture to a sample on a voice
|
||||
profile of your choice. Opens a profile picker (with "+ New voice" for
|
||||
cold starts) and a reference-text confirm dialog, because cloning needs
|
||||
the `reference_text` to match the audio verbatim. Edit as needed and
|
||||
save — the capture stays in the Captures tab untouched; the sample is a
|
||||
copy, not a move.
|
||||
|
||||
## Storage and retention
|
||||
|
||||
**Settings → Captures → Storage** controls how long captures live on disk:
|
||||
|
||||
| Setting | Effect |
|
||||
|---|---|
|
||||
| **Retention: forever** | Never auto-delete. Default. |
|
||||
| **Retention: 90 days / 30 days / 7 days** | Captures older than the window are pruned on app start. |
|
||||
| **Clear all captures** | One-click nuke of every capture and its audio on disk. No undo. |
|
||||
|
||||
The original audio is always kept alongside the transcript — archival is on
|
||||
by default. Every capture's audio file and metadata row can be re-processed
|
||||
(retranscribe, refine, Play-as) as long as the audio file still exists.
|
||||
|
||||
## Short-recording guard
|
||||
|
||||
Audio clips under **300 ms** are short-circuited client-side and never
|
||||
uploaded. This prevents a fumbled chord tap from landing an empty capture.
|
||||
The threshold is tuned to filter accidents without cutting off intentional
|
||||
short dictations.
|
||||
|
||||
## Keyboard shortcuts
|
||||
|
||||
Inside the Captures tab:
|
||||
|
||||
| Keys | Action |
|
||||
|---|---|
|
||||
| `Space` | Play / pause the selected capture |
|
||||
| `↑` / `↓` | Previous / next capture in the list |
|
||||
| `Enter` | Open the selected capture in detail view |
|
||||
| `⌘ / Ctrl` + `C` (in detail view) | Copy the transcript |
|
||||
|
||||
## API surface
|
||||
|
||||
The Captures tab is backed by a small set of REST endpoints:
|
||||
|
||||
| Method | Endpoint | Use |
|
||||
|---|---|---|
|
||||
| `POST` | `/captures` | Upload audio + start the pipeline (STT, optional refinement, archival). |
|
||||
| `GET` | `/captures` | List captures. |
|
||||
| `GET` | `/captures/{id}` | Fetch one capture. |
|
||||
| `POST` | `/captures/{id}/retranscribe` | Rerun STT with a chosen model. |
|
||||
| `POST` | `/captures/{id}/refine` | Rerun refinement with chosen flags. |
|
||||
| `POST` | `/profiles/{id}/samples/from-capture/{capture_id}` | Promote a capture to a voice profile sample. |
|
||||
|
||||
These endpoints are stable and usable from your own scripts — see
|
||||
[Remote Mode](/overview/remote-mode) for running Voicebox as a server the rest
|
||||
of your machine can talk to.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
The global hotkey flow that feeds most captures.
|
||||
</Card>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
Per-profile compose / rewrite / respond modes for captures you want to
|
||||
transform, not just transcribe.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Promote a capture into a voice sample on a profile.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
title: "Dictation"
|
||||
description: "Hold a key anywhere on your machine, speak, release — the transcript lands in whatever text field you had focused."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Dictation lets you turn speech into clean text anywhere on your computer. Hold
|
||||
a chord, talk, release — Voicebox transcribes what you said with Whisper,
|
||||
optionally cleans it up with a local LLM, and pastes the result into the text
|
||||
field you had focused when you started.
|
||||
|
||||
Everything happens on your hardware. No cloud, no accounts, no audio leaving
|
||||
the machine.
|
||||
|
||||
<Callout type="info">
|
||||
Dictation was introduced in **0.5.0** alongside the Captures tab and the
|
||||
per-profile personality modes. It's the "input" half of Voicebox's voice I/O
|
||||
loop — cloning and TTS are still the "output" half.
|
||||
</Callout>
|
||||
|
||||
## The flow
|
||||
|
||||
<Steps>
|
||||
<Step title="Hold the chord">
|
||||
Hold the push-to-talk chord anywhere on your machine. A small pill fades
|
||||
in over your current app.
|
||||
</Step>
|
||||
<Step title="Speak">
|
||||
The pill shows `Recording` with a live waveform and an elapsed-time
|
||||
counter. Speak naturally — you don't have to wait for anything.
|
||||
</Step>
|
||||
<Step title="Release">
|
||||
On release, the pill flips to `Transcribing`, then `Refining` if
|
||||
auto-refine is on, then disappears.
|
||||
</Step>
|
||||
<Step title="Text lands in your app">
|
||||
If auto-paste is enabled and Voicebox has Accessibility permission, the
|
||||
transcript pastes into the text field you had focused when you started
|
||||
talking — not wherever focus drifted while you were speaking.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
Either way, every capture also appears in the **Captures tab** with the
|
||||
original audio and the transcript paired together. See
|
||||
[Captures](/overview/captures) for what you can do with them after the fact.
|
||||
|
||||
## Push-to-talk and toggle modes
|
||||
|
||||
Voicebox ships two chord behaviors out of the box:
|
||||
|
||||
| Mode | Default (macOS) | Default (Windows) | Behavior |
|
||||
|---|---|---|---|
|
||||
| **Push-to-talk** | Right `⌘` + Right `⌥` | Right `Ctrl` + Right `Shift` | Recording stops when you release the chord. |
|
||||
| **Toggle-to-talk** | Push-to-talk + `Space` | Push-to-talk + `Space` | Recording keeps going until you tap the chord again. |
|
||||
|
||||
**Holding PTT and tapping `Space` mid-hold upgrades a hold into a toggled
|
||||
session** without a gap in the audio. This is the single most useful detail of
|
||||
the chord system — short bursts feel fast, long-form narration feels
|
||||
hands-free, and there's no decision up front about which mode you wanted.
|
||||
|
||||
## The on-screen pill
|
||||
|
||||
While you're dictating, a floating pill appears over the current app. It walks
|
||||
through the states of the capture cycle and shows live signals for each:
|
||||
|
||||
| State | What it shows |
|
||||
|---|---|
|
||||
| `Recording` | Live waveform + elapsed time. |
|
||||
| `Transcribing` | Thinking waveform while Whisper runs. |
|
||||
| `Refining` | Same thinking waveform while the LLM cleans up the transcript (only if auto-refine is on). |
|
||||
| Error | Red tint. Click the pill to copy the error to your clipboard. Auto-dismisses. |
|
||||
|
||||
The pill is transparent, always-on-top, and pre-created hidden at app start —
|
||||
so it appears instantly when you hit the chord, with no window flash.
|
||||
|
||||
## Customizing the chord
|
||||
|
||||
Open **Settings → Captures → Dictation** to change either chord.
|
||||
|
||||
- **Left vs right modifier badges.** When you hold keys into the chord
|
||||
picker, Voicebox records whether each modifier is the left or right variant.
|
||||
That means you can bind to just the right `⌥` while leaving the left `⌥`
|
||||
alone — useful if you want dictation on one hand and keep your
|
||||
other-hand shortcuts intact.
|
||||
- **Chord defaults are picked to stay out of your way.** On macOS, the
|
||||
defaults deliberately avoid left-hand `Cmd+Option` chords so
|
||||
`Cmd+Option+I` (devtools), `Cmd+Option+Esc` (force quit), and
|
||||
`Cmd+Option+Space` (Spotlight) all remain yours. On Windows, the defaults
|
||||
route around AltGr collisions on German / French / Spanish layouts where
|
||||
`Ctrl+Alt` synthesizes AltGr.
|
||||
- **Live reload.** Changing a chord in Settings takes effect immediately —
|
||||
no restart, no tab reload.
|
||||
|
||||
## Auto-paste into the focused app
|
||||
|
||||
Once transcription finishes, Voicebox can synthesize a native paste into
|
||||
whatever text field had focus when you started the chord. Your clipboard is
|
||||
saved before and restored after, so nothing you had copied goes missing.
|
||||
|
||||
| Platform | Mechanism |
|
||||
|---|---|
|
||||
| macOS | `CGEventPost` at the HID tap with a full `⌘V` key sequence, preceded by reactivating the original app via `NSRunningApplication`. |
|
||||
| Windows | `SendInput` with correct scan codes, plus a `SetForegroundWindow` + `AttachThreadInput` handshake to defeat foreground-lock when pasting into a window that wasn't frontmost at chord-start. |
|
||||
|
||||
**Focus is snapshotted at chord-start.** The paste targets the original field
|
||||
even if focus drifts during transcribe / refine — that's the "pastes where you
|
||||
were talking *from*, not where you're looking *now*" behavior.
|
||||
|
||||
<Callout type="info">
|
||||
Auto-paste is optional. If Accessibility permission isn't granted (macOS),
|
||||
or you prefer to keep synthetic input off, dictation still runs — transcripts
|
||||
land in the Captures tab and you can copy them manually. The setting lives
|
||||
inline next to the Accessibility prompt in Settings → Captures → Dictation,
|
||||
not as a global banner.
|
||||
</Callout>
|
||||
|
||||
## Refinement
|
||||
|
||||
If auto-refine is on, a local LLM cleans up the raw Whisper transcript
|
||||
before it's pasted. The goal is to remove verbal clutter without rewriting
|
||||
what you actually said.
|
||||
|
||||
What refinement typically fixes:
|
||||
|
||||
- Filler words (`um`, `uh`, `like` used as pauses, `you know`)
|
||||
- Self-corrections — the LLM keeps the final version and drops earlier
|
||||
attempts (`could you uh run the migration real quick, and then, yeah,
|
||||
check the logs` → `Could you run the migration, then check the logs?`)
|
||||
- Basic punctuation and capitalization
|
||||
- Whisper loop hallucinations — Voicebox strips repeated tokens (six or
|
||||
more identical tokens in a row, case-insensitive) *before* the LLM
|
||||
sees the transcript, so a small refinement model can't echo them back
|
||||
|
||||
What refinement deliberately preserves:
|
||||
|
||||
- Technical terms and code identifiers (`npm install`, `handleSubmit`)
|
||||
- Legitimate repetition (`no, no, no, no, no` has fewer than six identical
|
||||
tokens, so it survives)
|
||||
- Your intent — refinement is cleanup, not rewriting
|
||||
|
||||
Flags are snapshotted per capture, so you can re-refine the same raw
|
||||
transcript later with different flags without losing the original. The
|
||||
refinement model picker (**Settings → Captures → Refinement**) offers three
|
||||
bundled Qwen3 sizes:
|
||||
|
||||
| Model | Size | Best for |
|
||||
|---|---|---|
|
||||
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual dictation. |
|
||||
| Qwen3 1.7B | ~1.1 GB | Sweet spot when transcripts contain code identifiers. |
|
||||
| Qwen3 4B | ~2.5 GB | Full quality, slowest. |
|
||||
|
||||
This is the same local LLM used by the per-profile personality modes — one
|
||||
LLM in the app, not two. See [Voice Personalities](/overview/voice-personalities).
|
||||
|
||||
## Platform notes
|
||||
|
||||
### macOS
|
||||
|
||||
- **Accessibility permission** is required for auto-paste. The prompt lives
|
||||
inline next to the toggle in **Settings → Captures → Dictation**, with a
|
||||
deep link to **System Settings → Privacy & Security → Accessibility**.
|
||||
- **TSM crash mitigation.** The global hotkey listener runs on a background
|
||||
thread with `set_is_main_thread(false)` to sidestep a known
|
||||
macOS 14+ crash in the `rdev` library. If you hit an unexpected dictation
|
||||
failure on macOS, check the logs for TSM-related messages.
|
||||
|
||||
### Windows
|
||||
|
||||
- **UAC / UIPI caveat.** Synthetic paste into an *elevated* window from a
|
||||
non-elevated Voicebox is blocked by Windows itself. Run Voicebox elevated
|
||||
if you regularly dictate into elevated apps (e.g. an elevated terminal or
|
||||
Task Manager).
|
||||
- **Right-hand default chord** (`Ctrl+Shift`) avoids AltGr collisions on
|
||||
keyboard layouts where `Ctrl+Alt` is the compose key (German, French,
|
||||
Spanish, some others).
|
||||
|
||||
### Linux
|
||||
|
||||
- **Not yet in this release.** The Rust shim ships the macOS and Windows
|
||||
paths in 0.5.0. Linux `uinput` / AT-SPI support and the Wayland paste
|
||||
story are tracked in `docs/plans/VOICE_IO.md`.
|
||||
|
||||
## When auto-paste skips itself
|
||||
|
||||
A few cases where Voicebox deliberately does *not* synthesize a paste:
|
||||
|
||||
- **Focus was inside Voicebox** when the chord started. The transcript goes
|
||||
to the Captures tab so a dictation-into-Voicebox round-trip doesn't
|
||||
accidentally paste into the generate box.
|
||||
- **No text focus detected.** The transcript still lands in the Captures
|
||||
tab; copy it from there with one click.
|
||||
- **Accessibility permission not granted** on macOS. Same — Captures tab
|
||||
only.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
The paired audio + transcript archive every dictation lands in.
|
||||
</Card>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
The same local LLM doubles as per-profile compose / rewrite / respond.
|
||||
</Card>
|
||||
<Card title="Transcription" href="/developer/transcription">
|
||||
Developer-level details on Whisper, Whisper Turbo, and the STT backend.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -1,23 +1,48 @@
|
||||
---
|
||||
title: "Introduction"
|
||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
|
||||
---
|
||||
|
||||
## What is Voicebox?
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is the **open-source, local-first AI voice studio**. It closes the
|
||||
voice I/O loop in both directions on one machine, with no cloud and no
|
||||
accounts:
|
||||
|
||||
- **Complete privacy** -- models and voice data stay on your machine
|
||||
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
|
||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** -- REST API for integrating voice synthesis into your own projects
|
||||
- **Native performance** -- built with Tauri (Rust), not Electron
|
||||
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
|
||||
- **Humans talk** — hold a chord anywhere on your machine and your
|
||||
dictation lands as clean text in whatever text field you had focused
|
||||
- **Agents talk back** — any MCP-aware agent can call Voicebox to speak in
|
||||
one of your cloned voices
|
||||
- **Voices speak for themselves** — voice profiles can carry a personality
|
||||
that compose, rewrite, or respond to text before it's spoken
|
||||
|
||||
It's the free, local alternative to both ElevenLabs (voice cloning and TTS)
|
||||
and WisprFlow (voice dictation for agents and power users) — covering both
|
||||
sides of the same loop in one app, with a single model directory and LLM
|
||||
shared between input and output.
|
||||
|
||||
## What's in the app
|
||||
|
||||
- **Dictation** — global hotkey, push-to-talk and toggle modes, auto-paste
|
||||
into the focused field on macOS and Windows (see [Dictation](/overview/dictation))
|
||||
- **Captures tab** — paired audio + transcript archive, retranscribe,
|
||||
refine, play-as-voice, promote-to-sample (see [Captures](/overview/captures))
|
||||
- **Voice cloning** — 5 cloning engines covering 23 languages. Zero-shot
|
||||
cloning from a reference sample (see [Voice Cloning](/overview/voice-cloning))
|
||||
- **Preset voices** — 50+ curated voices via Kokoro and Qwen CustomVoice
|
||||
for when you don't want to clone (see [Preset Voices](/overview/preset-voices))
|
||||
- **Voice personalities** — optional free-form personality on any profile
|
||||
plus compose / rewrite / respond modes powered by a local LLM (see
|
||||
[Voice Personalities](/overview/voice-personalities))
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus,
|
||||
compression, filters (Spotify's Pedalboard)
|
||||
- **Expressive speech** — paralinguistic tags like `[laugh]` and `[sigh]`
|
||||
via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** — auto-chunking with crossfade for long scripts
|
||||
- **Stories editor** — multi-track timeline for conversations and podcasts
|
||||
- **API-first** — REST + WebSocket API; MCP server for agent integrations
|
||||
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux
|
||||
(ROCm / CPU), Intel Arc, Docker
|
||||
|
||||
## TTS Engines
|
||||
|
||||
@@ -30,9 +55,21 @@ Seven engines with different strengths, switchable per-generation:
|
||||
| **LuxTTS** | Cloned | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||
| **Chatterbox Multilingual** | Cloned | 23 | Broadest language coverage |
|
||||
| **Chatterbox Turbo** | Cloned | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model -- 700s+ coherent audio |
|
||||
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model — 700s+ coherent audio |
|
||||
| **Kokoro** | Preset (50 voices) | 9 | 82M parameters, CPU realtime, lowest VRAM of any engine |
|
||||
|
||||
## STT and local LLM
|
||||
|
||||
Voicebox also runs a full speech recognition and local LLM stack, shared
|
||||
between dictation, the Captures tab, and per-profile personality modes:
|
||||
|
||||
| Layer | Models |
|
||||
|---|---|
|
||||
| **STT** | Whisper Base / Small / Medium / Large / Turbo (PyTorch or MLX) |
|
||||
| **LLM** | Qwen3 0.6B / 1.7B / 4B (refinement + per-profile compose / rewrite / respond) |
|
||||
|
||||
No cloud fallback, no bring-your-own-API-key. Local is the product.
|
||||
|
||||
## GPU Support
|
||||
|
||||
| Platform | Backend | Notes |
|
||||
@@ -46,11 +83,13 @@ Seven engines with different strengths, switchable per-generation:
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Game development** -- generate dynamic dialogue for characters
|
||||
- **Content creation** -- produce podcasts and video voiceovers
|
||||
- **Accessibility** -- build text-to-speech tools for users who need them
|
||||
- **Voice assistants** -- create custom voice interfaces
|
||||
- **Production pipelines** -- automate voiceover workflows via the REST API
|
||||
- **Dictation for humans and agents** — speak instead of type, in any app
|
||||
- **Agent voice output** — any MCP-aware agent can speak in a cloned voice
|
||||
- **Game development** — generate dynamic dialogue for characters
|
||||
- **Content creation** — podcasts, video voiceovers, audiobooks
|
||||
- **Accessibility** — speech-to-text for any field, TTS with a voice you own
|
||||
- **Voice assistants** — custom voice interfaces without a cloud bill
|
||||
- **Production pipelines** — automate voice workflows via the REST API
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -61,8 +100,9 @@ Seven engines with different strengths, switchable per-generation:
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
|
||||
| STT | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Local LLM | Qwen3 0.6B / 1.7B / 4B (MLX or PyTorch) |
|
||||
| Effects | Pedalboard (Spotify) |
|
||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||
| Database | SQLite |
|
||||
| Audio | WaveSurfer.js, librosa |
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
---
|
||||
title: "MCP Server"
|
||||
description: "Let Claude Code, Cursor, Cline, or any MCP-aware agent speak in one of your cloned voices — locally, with no cloud."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox ships a built-in **Model Context Protocol** server so local AI
|
||||
agents can call your Voicebox install directly: speak text in a voice
|
||||
profile, transcribe audio, and list captures or profiles. The server runs
|
||||
inside the same process as the rest of Voicebox and is mounted at `/mcp`
|
||||
over Streamable HTTP.
|
||||
|
||||
Agent asks to speak → Voicebox plays audio on your speakers → an on-screen
|
||||
pill surfaces the voice name for the whole duration so you always see what's
|
||||
coming out of your machine.
|
||||
|
||||
<Callout type="info">
|
||||
MCP shipped in **0.5.0** alongside [Dictation](/overview/dictation) and
|
||||
[Voice Personalities](/overview/voice-personalities). The design goal is
|
||||
"local voice layer for every agent on your machine" — the same app that
|
||||
captures your voice can generate a response in any voice profile you've
|
||||
cloned.
|
||||
</Callout>
|
||||
|
||||
## Quick install
|
||||
|
||||
### Claude Code
|
||||
|
||||
```
|
||||
claude mcp add voicebox \
|
||||
--transport http \
|
||||
--url http://127.0.0.1:17493/mcp \
|
||||
--header "X-Voicebox-Client-Id: claude-code"
|
||||
```
|
||||
|
||||
### Cursor / Windsurf / VS Code MCP / any HTTP MCP client
|
||||
|
||||
Drop this into the client's MCP config (usually `.mcp.json` or a Settings UI):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"url": "http://127.0.0.1:17493/mcp",
|
||||
"headers": { "X-Voicebox-Client-Id": "cursor" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change `cursor` to whatever name you want the binding to show up as in
|
||||
Voicebox → Settings → MCP. The value is just an identifier for the
|
||||
per-client voice binding — not a secret, not a credential.
|
||||
|
||||
### Clients that only speak stdio
|
||||
|
||||
A stdio shim binary `voicebox-mcp` is bundled with the desktop app. Point
|
||||
the client at that binary's absolute path:
|
||||
|
||||
<Tabs items={["macOS", "Windows", "Linux"]}>
|
||||
<Tab value="macOS">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab value="Windows">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "C:\\Program Files\\Voicebox\\voicebox-mcp.exe",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab value="Linux">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "/opt/voicebox/voicebox-mcp",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
The shim waits up to 30 seconds for the Voicebox backend to come up, then
|
||||
proxies JSON-RPC from stdio over Streamable HTTP. Voicebox must be running
|
||||
for the shim to connect.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Use |
|
||||
|---|---|
|
||||
| `voicebox.speak` | Speak text in a voice profile. Returns a `generation_id` to poll. |
|
||||
| `voicebox.transcribe` | Whisper transcription of base64 audio or an absolute local path. |
|
||||
| `voicebox.list_captures` | Recent captures with transcripts, paginated. |
|
||||
| `voicebox.list_profiles` | Available voice profiles (cloned + preset). |
|
||||
|
||||
### `voicebox.speak`
|
||||
|
||||
```ts
|
||||
voicebox.speak({
|
||||
text: "Deploy complete.",
|
||||
profile?: "Morgan", // name or id; falls back to per-client binding, then default
|
||||
engine?: "qwen", // qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro
|
||||
intent?: "respond", // respond | rewrite | compose — only if the profile has a personality
|
||||
language?: "en",
|
||||
})
|
||||
```
|
||||
|
||||
Returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"generation_id": "…",
|
||||
"status": "generating",
|
||||
"profile": "Morgan",
|
||||
"source": "mcp",
|
||||
"poll_url": "/generate/<id>/status"
|
||||
}
|
||||
```
|
||||
|
||||
- **Plain TTS** — omit `intent`. Text is spoken as-is.
|
||||
- **Persona mode** — pass `intent` and the profile must have a personality set.
|
||||
The LLM transforms the text (respond to it, rewrite it in character, or
|
||||
compose a fresh utterance) before TTS. See [Voice Personalities](/overview/voice-personalities).
|
||||
|
||||
### `voicebox.transcribe`
|
||||
|
||||
```ts
|
||||
voicebox.transcribe({
|
||||
audio_base64?: "<base64>", // exactly one of these two
|
||||
audio_path?: "/absolute/path/to/file.wav",
|
||||
language?: "en",
|
||||
model?: "turbo", // base | small | medium | large | turbo
|
||||
})
|
||||
```
|
||||
|
||||
Returns `{ text, duration, language, model }`. 200 MB ceiling on either path.
|
||||
|
||||
### `voicebox.list_captures`
|
||||
|
||||
`{ limit?: 20, offset?: 0 }` → `{ captures: [...], total }`. `limit` is
|
||||
clamped to `1..=200`.
|
||||
|
||||
### `voicebox.list_profiles`
|
||||
|
||||
No args → `{ profiles: [{ id, name, voice_type, language, has_personality }] }`.
|
||||
|
||||
## Voice resolution
|
||||
|
||||
Every call to `voicebox.speak` (and `POST /speak`) resolves the voice profile
|
||||
in this order:
|
||||
|
||||
<Steps>
|
||||
<Step title="Explicit `profile` arg">
|
||||
Passed as a name (case-insensitive) or id. If the name/id doesn't match,
|
||||
the call errors — the server doesn't silently fall back.
|
||||
</Step>
|
||||
<Step title="Per-client binding">
|
||||
Looked up by the `X-Voicebox-Client-Id` header. Managed in
|
||||
**Voicebox → Settings → MCP**. Lets you pin Claude Code to Morgan,
|
||||
Cursor to Scarlett, etc.
|
||||
</Step>
|
||||
<Step title="Global default">
|
||||
`capture_settings.default_playback_voice_id` — same default voice the
|
||||
Captures tab's "Play as voice" action uses.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
If none of the three produce a profile the tool returns a helpful error
|
||||
pointing at Settings.
|
||||
|
||||
## Per-client bindings
|
||||
|
||||
Voicebox → Settings → MCP shows one row per `client_id` Voicebox has heard
|
||||
from, plus the config snippets you can copy into each agent. Each row
|
||||
carries:
|
||||
|
||||
| Field | Purpose |
|
||||
|---|---|
|
||||
| `label` | Display name in the Settings UI (e.g. "Claude Code"). |
|
||||
| `profile_id` | The voice this client uses when `profile` isn't passed. |
|
||||
| `default_engine` | Override the TTS engine for this client. |
|
||||
| `default_intent` | Default persona mode (`respond` / `rewrite` / `compose`). |
|
||||
| `last_seen_at` | Last time the server saw a request from this client. |
|
||||
|
||||
`last_seen_at` is stamped automatically by middleware on every `/mcp/*`
|
||||
request — useful when you're not sure whether your config took.
|
||||
|
||||
## The speaking pill
|
||||
|
||||
Every agent-initiated speak surfaces the floating pill the same way
|
||||
[Dictation](/overview/dictation) does, in a new `Speaking` state showing the
|
||||
profile name and an elapsed timer. The pill is intentionally unmissable —
|
||||
silent background TTS is a trust hazard, so Voicebox always shows what's
|
||||
being spoken and in what voice.
|
||||
|
||||
Behind the scenes, the backend broadcasts `speak-start` and `speak-end`
|
||||
events on `GET /events/speak`, which `DictateWindow` subscribes to via SSE.
|
||||
The pill overrides the capture session when both would render — you can't
|
||||
hear two pills at once.
|
||||
|
||||
## Non-MCP REST surface
|
||||
|
||||
`POST /speak` is a thin wrapper on the same code path for callers that
|
||||
don't speak MCP — shell scripts, ACP, A2A, GitHub Actions, whatever.
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:17493/speak \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'X-Voicebox-Client-Id: ci' \
|
||||
-d '{"text":"Build complete.","profile":"Morgan"}'
|
||||
```
|
||||
|
||||
Body fields match the MCP tool: `text`, optional `profile`, `engine`,
|
||||
`intent`, `language`. Returns a `GenerationResponse` — the same shape as
|
||||
`POST /generate`.
|
||||
|
||||
## Debugging
|
||||
|
||||
Use the MCP Inspector to poke tools directly without plumbing through an
|
||||
agent:
|
||||
|
||||
```
|
||||
npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp
|
||||
```
|
||||
|
||||
Start with `voicebox.list_profiles` to confirm wiring, then
|
||||
`voicebox.speak` for end-to-end — you should hear audio and see the
|
||||
generation land in the Captures tab.
|
||||
|
||||
<Callout type="info">
|
||||
If an agent can't reach the server, the first thing to check is that
|
||||
Voicebox is running — the backend only listens while the desktop app is
|
||||
open. The stdio shim surfaces this as a JSON-RPC error on the client
|
||||
side after its 30-second health-wait window elapses.
|
||||
</Callout>
|
||||
|
||||
## Security
|
||||
|
||||
- **Localhost only.** The server binds to `127.0.0.1`. If you ever point
|
||||
Voicebox at a non-loopback interface (e.g. remote-mode over a trusted
|
||||
network), add a bearer token — it's on the roadmap but not in 0.5.0.
|
||||
- **No auth today.** Any process that can connect to your loopback can
|
||||
call MCP. That's the same trust boundary as the rest of Voicebox's REST
|
||||
API and is appropriate for a single-user local tool.
|
||||
- **`audio_path` reads are unrestricted** against the same trust
|
||||
boundary. If you're scripting against a shared host, prefer
|
||||
`audio_base64` so you don't have to think about path sandboxing.
|
||||
- **Voice cloning consent applies.** See [Voice Cloning](/overview/voice-cloning#limitations)
|
||||
— an agent being able to call `voicebox.speak` in someone's voice
|
||||
doesn't change the ethics of whose voices you clone.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- **Transport:** Streamable HTTP (Nov-2025 MCP spec, post-SSE). Claude
|
||||
Code, Cursor, Windsurf, and VS Code MCP extensions all support it.
|
||||
- **Package naming:** the backend package is `backend/mcp_server/`, not
|
||||
`mcp`, to avoid shadowing the PyPI `mcp` package FastMCP imports
|
||||
internally.
|
||||
- **Dependencies:** `fastmcp>=3.0,<4.0`, `sse-starlette>=2.0`.
|
||||
- **Lifespan:** mounting FastMCP requires the `lifespan=` kwarg on
|
||||
`FastAPI()` — the startup/shutdown event decorators are incompatible
|
||||
with FastMCP's Streamable HTTP session manager. The Voicebox app.py
|
||||
composes both into one async context manager.
|
||||
|
||||
For the full developer-facing tour of the code layout, see
|
||||
`backend/mcp_server/README.md` in the repo.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
Persona mode (`intent=respond/rewrite/compose`) for agents that should
|
||||
transform text in-character before speaking.
|
||||
</Card>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
The pill that surfaces agent speech is the same one that surfaces
|
||||
your dictations — one mental model for both directions of the loop.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
Every agent-initiated speak lands in the Captures tab with its
|
||||
generated audio — replay, download, repurpose.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -7,8 +7,12 @@
|
||||
"docker",
|
||||
"quick-start",
|
||||
"gpu-acceleration",
|
||||
"dictation",
|
||||
"captures",
|
||||
"voice-cloning",
|
||||
"preset-voices",
|
||||
"voice-personalities",
|
||||
"mcp-server",
|
||||
"stories-editor",
|
||||
"recording-transcription",
|
||||
"generation-history",
|
||||
|
||||
@@ -1,64 +1,106 @@
|
||||
---
|
||||
title: "Recording & Transcription"
|
||||
description: "Record audio and transcribe speech with Whisper"
|
||||
description: "A map of the three places you can record and transcribe audio in Voicebox — dictation, captures, and voice-profile samples."
|
||||
---
|
||||
|
||||
## Recording
|
||||
## Overview
|
||||
|
||||
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
|
||||
Voicebox records and transcribes audio in three different contexts, each
|
||||
feeding a different surface in the app. This page is a map; follow the links
|
||||
for the detail.
|
||||
|
||||
### Features
|
||||
| Goal | Where | Docs |
|
||||
|---|---|---|
|
||||
| Speak and have your words land in another app | Global hotkey → Captures tab + auto-paste | [Dictation](/overview/dictation) |
|
||||
| Record a thought, a meeting, or a voice memo inside Voicebox | Captures tab | [Captures](/overview/captures) |
|
||||
| Record a clip to clone a voice from | Voices tab → profile samples | [Creating Voice Profiles](/overview/creating-voice-profiles) |
|
||||
|
||||
- **Microphone input** - Record from any audio input device
|
||||
- **System audio capture** - Record desktop audio (macOS/Windows)
|
||||
- **Waveform visualization** - See audio levels in real-time
|
||||
- **Multiple formats** - Export as WAV, MP3, or M4A
|
||||
All three paths share the same STT backend — it's the surrounding workflow
|
||||
that differs.
|
||||
|
||||
### How to Record
|
||||
## Dictation
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Input">
|
||||
Choose your microphone or system audio
|
||||
</Step>
|
||||
<Step title="Start Recording">
|
||||
Click the record button and speak clearly
|
||||
</Step>
|
||||
<Step title="Stop & Save">
|
||||
Click stop when finished
|
||||
</Step>
|
||||
<Step title="Use or Export">
|
||||
Use as voice sample or export to file
|
||||
</Step>
|
||||
</Steps>
|
||||
The 0.5.0 headline feature. Hold a chord anywhere on your machine, speak,
|
||||
release. The transcript lands in whatever text field you had focused,
|
||||
cleaned up by a local LLM if auto-refine is on. Captures accumulate in the
|
||||
Captures tab for later replay or re-transcription.
|
||||
|
||||
## Transcription
|
||||
Covered end-to-end in [Dictation](/overview/dictation).
|
||||
|
||||
Automatic speech-to-text powered by OpenAI's Whisper model.
|
||||
## Captures tab
|
||||
|
||||
### Features
|
||||
When you don't need to paste into another app — you just want a clean
|
||||
transcript of some audio — the Captures tab is the home. Record in-app,
|
||||
drop in a file (`.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`), or dig
|
||||
through dictations that already landed there. Every capture keeps its
|
||||
original audio, can be retranscribed with a different model, and can be
|
||||
played back through any voice profile you have.
|
||||
|
||||
- **High accuracy** - Industry-leading speech recognition
|
||||
- **Multiple languages** - Supports 50+ languages
|
||||
- **Automatic detection** - Language auto-detection
|
||||
- **Timestamps** - Word-level timing information
|
||||
Covered in [Captures](/overview/captures).
|
||||
|
||||
### How to Transcribe
|
||||
## Voice profile samples
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Audio">
|
||||
Choose a recording or upload an audio file
|
||||
</Step>
|
||||
<Step title="Choose Language">
|
||||
Select language or use auto-detect
|
||||
</Step>
|
||||
<Step title="Transcribe">
|
||||
Click transcribe and wait for processing
|
||||
</Step>
|
||||
<Step title="Review & Export">
|
||||
Review text and export as needed
|
||||
</Step>
|
||||
</Steps>
|
||||
A separate flow, in the Voices tab. When you're creating a profile from an
|
||||
audio clip, the sample is what the cloning engine actually learns from —
|
||||
the `reference_text` on a sample must match the audio *verbatim*, which is
|
||||
why samples are a different data model from captures.
|
||||
|
||||
You can promote a capture to a sample from the Captures tab's Send-to menu
|
||||
("Use as voice sample…"), which opens a reference-text confirm dialog so
|
||||
you can correct the last ~10% of transcript accuracy before saving.
|
||||
|
||||
Covered in [Creating Voice Profiles](/overview/creating-voice-profiles).
|
||||
|
||||
## Transcription models
|
||||
|
||||
All three paths share the same Whisper models. Pick a default in
|
||||
**Settings → Captures → Transcription**; override per capture if you need
|
||||
to.
|
||||
|
||||
| Model | Size | When to pick it |
|
||||
|---|---|---|
|
||||
| Whisper Base | ~300 MB | Fast. Default. Good for clean speech. |
|
||||
| Whisper Small | ~500 MB | Better quality, still fast. |
|
||||
| Whisper Medium | ~1.5 GB | High quality. |
|
||||
| Whisper Large | ~3 GB | Best quality, slow on CPU. |
|
||||
| Whisper Turbo | ~1.5 GB | Large-tier quality, ~5× faster than Large. |
|
||||
|
||||
On Apple Silicon the model runs through **MLX-Whisper** (~8× faster than
|
||||
PyTorch). Everywhere else it runs through PyTorch `transformers`. The
|
||||
backend picks the right one — you don't configure it.
|
||||
|
||||
<Callout type="info">
|
||||
Transcription is useful for creating voice samples from existing audio or generating subtitles.
|
||||
For noisy clips, prefer **Turbo** or **Large**. Base can hallucinate on
|
||||
hard inputs — most famously the "thanks for watching" loop. Voicebox
|
||||
strips those loops deterministically before LLM refinement runs, so a
|
||||
capture can be cleanly re-refined even if the raw transcript has them.
|
||||
</Callout>
|
||||
|
||||
## Language
|
||||
|
||||
You can pass a language hint for short clips (under ~5 seconds) where
|
||||
Whisper's auto-detect is unreliable. Set a default language lock in
|
||||
**Settings → Captures → Transcription → Language**, or override per capture.
|
||||
|
||||
## Transcription API
|
||||
|
||||
Developer-level detail on the STT backend, model loading, preprocessing, and
|
||||
the `/transcribe` endpoint lives in the
|
||||
[Transcription developer guide](/developer/transcription). The Captures
|
||||
pipeline also exposes `/captures` as a higher-level endpoint that wraps
|
||||
STT + archival + optional refinement in one call — see
|
||||
[Captures](/overview/captures#api-surface).
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
Hold a chord anywhere on your machine, speak, release.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
The paired audio + transcript archive.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Record or upload samples for voice cloning.
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
---
|
||||
title: "Voice Personalities"
|
||||
description: "Attach a personality to a voice profile and use Compose, Rewrite, or Respond to generate in-character speech — all powered by a local LLM."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A **personality** is an optional free-form description attached to a voice
|
||||
profile — who this voice is, how they speak, what they care about. Set one
|
||||
and three new actions appear on the profile, each powered by a bundled
|
||||
Qwen3 LLM running entirely locally:
|
||||
|
||||
- **Compose** — generate a fresh utterance in this character's voice
|
||||
- **Rewrite** — restate your text in their voice while preserving every idea
|
||||
- **Respond** — treat your text as a prompt and produce the character's reply
|
||||
|
||||
The LLM produces the text. The voice profile speaks it. No cloud round-trip,
|
||||
no external API — the whole loop runs on your hardware.
|
||||
|
||||
<Callout type="info">
|
||||
Personalities shipped in **0.5.0**. The same local LLM doubles as the
|
||||
refinement model for [Dictation](/overview/dictation) — one LLM in the app,
|
||||
not two, sharing one model cache and one GPU-memory footprint.
|
||||
</Callout>
|
||||
|
||||
## Setting a personality
|
||||
|
||||
Open a voice profile's edit view. The **Personality** field is free-form text
|
||||
up to **2,000 characters**. Describe the voice however helps you — past
|
||||
lines they'd say, speech patterns, tone, boundaries.
|
||||
|
||||
Good descriptions tend to include:
|
||||
|
||||
- A one-line identity (who they are)
|
||||
- Speech patterns (rhythm, vocabulary, what they avoid)
|
||||
- Representative phrases — example lines show the LLM the target tone
|
||||
better than adjectives
|
||||
- What the character *wouldn't* do (they don't explain, they don't
|
||||
apologize, they refuse to break character, etc.)
|
||||
|
||||
You can set a personality on any voice profile type — cloned or preset. The
|
||||
three modes work identically regardless of engine.
|
||||
|
||||
## The three modes
|
||||
|
||||
Each mode is tuned for a specific job and the LLM temperature is adjusted
|
||||
to match.
|
||||
|
||||
### Compose
|
||||
|
||||
Generate a fresh utterance in the character's voice, with no seed text.
|
||||
Click again to get a different take.
|
||||
|
||||
- **When to use:** prototyping, sampling a character's voice, brainstorming
|
||||
a line without typing one first
|
||||
- **Temperature:** hot — variety is the point
|
||||
- **Typical output:** a short, punchy line that fits the character's
|
||||
register
|
||||
|
||||
### Rewrite
|
||||
|
||||
Take your input text and restate it in the character's voice while
|
||||
preserving every idea. High-fidelity mode — the content doesn't change, only
|
||||
the voice does.
|
||||
|
||||
- **When to use:** turning a dictated memo into in-character speech; lifting
|
||||
a plain-English script into a specific voice without editing by hand
|
||||
- **Temperature:** cold — faithfulness wins
|
||||
- **Typical output:** same ideas, same order, different phrasing and cadence
|
||||
|
||||
### Respond
|
||||
|
||||
Treat your input as a prompt and produce the character's reply — as if
|
||||
you'd said it *to* them.
|
||||
|
||||
- **When to use:** spoken-input agents; Q&A with a specific voice;
|
||||
interactive character experiences
|
||||
- **Temperature:** balanced — creative but grounded
|
||||
- **Typical output:** a reply to your prompt, written in-character
|
||||
|
||||
## Speech-only framing
|
||||
|
||||
All three modes enforce **speech-only** output. The LLM is prompted to
|
||||
produce things a person would actually say out loud — no narration, no
|
||||
action tags (`*sighs*`, `[laughs]`), no meta-commentary, no markdown
|
||||
formatting, no stage directions.
|
||||
|
||||
This is deliberate: the output is going straight into TTS, and anything that
|
||||
isn't speakable ends up either ignored or read literally. The speech-only
|
||||
framing also makes the output land cleanly inside dialogue, so you can drop
|
||||
a Respond result straight into a Story.
|
||||
|
||||
## The local LLM
|
||||
|
||||
The bundled LLM is **Qwen3**, available in three sizes:
|
||||
|
||||
| Model | Download size | Best for |
|
||||
|---|---|---|
|
||||
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual use. |
|
||||
| Qwen3 1.7B | ~1.1 GB | Sweet spot for character personalities with specific phrasing. |
|
||||
| Qwen3 4B | ~2.5 GB | Full quality. Slowest. Useful for very particular tone. |
|
||||
|
||||
The model runs through the same backend split Voicebox already uses for TTS
|
||||
— **MLX** (4-bit community quants) on Apple Silicon, **PyTorch** (transformers
|
||||
`AutoModelForCausalLM`) everywhere else. Downloads go through the same cache
|
||||
and model-management UI as TTS models.
|
||||
|
||||
Pick a size in **Settings → Captures → Refinement → Refinement model** — the
|
||||
personality modes reuse it. If you switch models, both refinement and
|
||||
personality output pick up the change on the next call.
|
||||
|
||||
## Using the modes
|
||||
|
||||
The three actions appear as buttons on the profile when a personality is
|
||||
set. For each:
|
||||
|
||||
<Steps>
|
||||
<Step title="Type or dictate your input">
|
||||
Rewrite and Respond need input text. Compose doesn't.
|
||||
</Step>
|
||||
<Step title="Click the mode">
|
||||
The LLM runs, then the result fills the generate box.
|
||||
</Step>
|
||||
<Step title="Hit generate">
|
||||
The TTS engine speaks the LLM output in the profile's voice. The result
|
||||
lands in generation history as a normal generation.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
Each button also has an inline regenerate affordance — click again to
|
||||
resample. Compose will give you something totally different; Rewrite and
|
||||
Respond will give you a variation on the same content.
|
||||
|
||||
## Use cases
|
||||
|
||||
- **Agents that speak in a voice you own.** Combine Respond with the
|
||||
built-in [MCP Server](/overview/mcp-server) so Claude Code, Cursor,
|
||||
Cline, or any MCP-aware agent can talk back through a profile with a
|
||||
personality. The agent calls `voicebox.speak({ text, profile, intent:
|
||||
"respond" })` and Voicebox produces in-character speech in your cloned
|
||||
voice.
|
||||
- **Interactive characters.** Games, narrative tools, accessibility
|
||||
experiences. A character with a personality description plus a cloned
|
||||
voice becomes a reusable prop.
|
||||
- **Accessibility.** People who can't speak in their original voice can
|
||||
keep a personality description of how they used to sound and use Rewrite
|
||||
to turn typed input into in-character speech.
|
||||
- **Creative drafting.** Write a plain outline, Rewrite line-by-line into
|
||||
the character's voice, drop the audio into a Story.
|
||||
|
||||
## API surface
|
||||
|
||||
Personalities and the three modes are accessible via REST:
|
||||
|
||||
| Method | Endpoint | Body |
|
||||
|---|---|---|
|
||||
| `PUT` | `/profiles/{id}` | Include a `personality` field up to 2,000 chars to set it. |
|
||||
| `POST` | `/profiles/{id}/speak` | Runs the LLM + TTS in one shot. Body includes `text`, `intent` (`compose`, `rewrite`, or `respond`), optional `engine`, `language`. |
|
||||
|
||||
The `/profiles/{id}/speak` endpoint is the same primitive MCP's
|
||||
`voicebox.speak` tool calls when that ships. Scripts and agents can use it
|
||||
directly today.
|
||||
|
||||
## Limits and gotchas
|
||||
|
||||
- **The personality is a prompt, not a fine-tune.** The LLM will sometimes
|
||||
drift out of character, especially on Compose at high temperature. Click
|
||||
again for another take.
|
||||
- **Long personalities are not always better.** 2,000 chars is a ceiling,
|
||||
not a goal. A sharp 300-char description with two example lines
|
||||
typically outperforms a long one.
|
||||
- **Speech-only framing is enforced, but not bulletproof.** Very large
|
||||
prompts or unusual inputs can sneak an action tag through. If you see
|
||||
`[laughs]` in TTS output, it's usually a personality-field hint the
|
||||
model anchored onto — remove it from the description.
|
||||
- **Rewrite is stricter than Respond.** If the output is changing your
|
||||
meaning, you probably want Respond (or a wholesale Compose with context
|
||||
in the input), not Rewrite.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
Dictate the input for Rewrite or Respond from anywhere on your machine.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
Captures feed personalities naturally — dictate a memo, rewrite it in
|
||||
a character voice, generate speech.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Add a personality to an existing profile.
|
||||
</Card>
|
||||
</Cards>
|
||||
Reference in New Issue
Block a user