mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-15 04:40:40 -07:00
feat: 0.5.0 Capture release — dictation, MCP, personalities (#544)
* feat(capture): dictation, personalities, 0.5.0 Ships the Capture release end to end. Global-hotkey dictation with synthetic paste into the focused app on macOS and Windows, an on-screen pill across recording / transcribing / refining, customizable push-to- talk and toggle chords, and an accessibility-permission prompt scoped to Settings → Captures with inline re-check feedback. Voice profiles gain optional personalities that power compose / rewrite / respond actions via a local Qwen3 LLM — shared with refinement, so there is one local LLM in the app, not two. Refinement hardened with deterministic Whisper-loop collapse before the LLM sees the transcript, per-capture flag snapshots for re-runs, and a ten-transcript evaluation harness across every bundled refinement size. Version bump 0.4.5 → 0.5.0. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(mcp): local MCP server exposes voicebox.* tools to AI agents Mounts FastMCP at /mcp (Streamable HTTP) so Claude Code, Cursor, Windsurf, and the VS Code MCP extensions can call voicebox.speak, voicebox.transcribe, voicebox.list_captures, and voicebox.list_profiles against the running Voicebox server. Backend - new backend/mcp_server package (tools, middleware, profile resolve, pub/sub events); named mcp_server to avoid shadowing the installed mcp PyPI package FastMCP imports internally - app.py migrated from @app.on_event to lifespan= so FastMCP's session manager cohabits with Voicebox's startup/shutdown - new MCPClientBinding table + /mcp/bindings CRUD; ClientIdMiddleware reads X-Voicebox-Client-Id into a ContextVar and stamps last_seen_at - profile resolution precedence: explicit -> per-client binding -> capture_settings.default_playback_voice_id - POST /speak REST wrapper for non-MCP callers (shell, ACP, A2A) - GET /events/speak SSE broadcasts speak-start / speak-end so the pill surfaces agent-initiated speech - backend/mcp_shim proxy (plain httpx) for stdio-only MCP clients - PyInstaller spec updates + new --shim build target (~18 MB) Frontend - Settings -> MCP page with HTTP / stdio / claude-mcp-add copy snippets, default voice picker, per-client bindings table, connection status - useMCPBindings, useSpeakEvents hooks - CapturePill gains 'speaking' state; DictateWindow subscribes to SSE and emits dictate:show so the Rust side surfaces the pill window Native - tauri.conf.json externalBin now includes voicebox-mcp - show_dictate_window helper + dictate:show listener in main.rs - (also in this commit: InputMonitoringGate UX, hotkey_monitor tweaks, landing footer/navbar updates, new overview docs for captures / dictation / mcp-server / voice-personalities) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(mcp): Rust-owned speaking pill with self-contained audio playback The pill window now surfaces for agent-initiated speech without main-window involvement. Rust subscribes to /events/speak via a tokio task + reqwest streaming body (speak_monitor.rs), shows the pill, and forwards events to the dictate webview over Tauri's event bus. The pill plays audio via a plain HTMLAudioElement and emits dictate:hide when playback ends. The pill stays hidden through the ~1 s generation wait and only surfaces when audio actually starts, with the counter armed at that moment. Fixes a shared-dict mutation in mcp_server/events.publish() that caused the second subscriber (Rust speak_monitor) to receive `event: message` instead of named speak-start/speak-end frames. Also teaches the speak_monitor parser to handle CRLF framing (sse-starlette default). Main-window AudioPlayer now skips autoplay for source in {mcp, rest} to avoid double-play when both windows are alive. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * readme and dev script * feat(capture): gate global hotkey on dictation readiness checklist Stops the "stuck pill" failure where pressing the chord with missing STT/LLM models triggers a recording that has nowhere to land. The hotkey now stays disarmed until every gate (models downloaded, Input Monitoring + Accessibility granted) is green; the empty-state checklist in CapturesTab surfaces each unmet gate with a one-click action and auto-arms the chord once everything turns green. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * color * model download status * progress * personality: bool API, i18n across the app - Collapse intent tri-state (respond/rewrite/compose) to `personality: bool` on /generate, /speak, and voicebox.speak. Drop respond entirely; keep compose as a standalone button via /profiles/{id}/compose. Remove /rewrite, /respond, and /speak profile endpoints. - FloatingGenerateBox: Wand2 persona toggle + Dices compose button appear when the selected profile has a personality. ProfileCard badges Wand2 alongside the effects Sparkles. - MCP bindings: default_intent column → default_personality: bool. Migration drops the legacy column. - i18n: en / ja / zh-CN / zh-TW translation files filled out and wired through the capture, server, and profile UI. ```ts voicebox.speak({ text: "Deploy complete.", profile: "Morgan", personality: true, // rewrite through the profile's personality LLM }); ``` * i18n: GenerationPage sidebar copy * fix: BOOL import for windows crate 0.62 BOOL moved from Win32::Foundation to windows::core in 0.62. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(capture): layout-aware V keycode for synthetic paste on macOS macOS apps match Cmd+V against the layout-translated character via NSMenu key equivalents, so posting kVK_ANSI_V (= 9, the QWERTY V position) on Dvorak produces Cmd+. and never triggers Paste. New keyboard_layout module resolves the active layout's V keycode via TISCopyCurrentKeyboardLayoutInputSource + UCKeyTranslate, caches it in an AtomicU16, and refreshes on kTISNotifySelectedKeyboardInputSourceChanged. All TIS calls run on the main thread (init from Tauri setup; observer callback delivered to the main runloop); synthetic_keys::send_paste reads the cached value once per paste. Falls back to kVK_ANSI_V when resolution fails or the active input source carries no Unicode key layout data. Windows is intentionally left on hardcoded VK_V — SendInput delivers WM_KEYDOWN with wParam = VK_V to the target regardless of the active layout, which is why `Send "^v"` works for AutoHotkey on Dvorak Windows. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(capture): cooperative app activation for synthetic paste on macOS 14+ macOS 14 deprecated NSRunningApplication.activateWithOptions: in favour of a cooperative-activation pattern: the caller first yields activation rights to the target, then the target activate()s against the tightened Sonoma foreground rules. Without the yield, activate() on 14+ sometimes silently fails or only bounces the dock icon — the exact "paste lands in the wrong app" symptom we were previously one API break away from. activate_pid now discovers the 14+ selector via respondsToSelector: and branches: on 14+ it yieldActivationToApplication:'s from NSRunningApplication.current then calls -activate on the target; on 11–13 it stays on -activateWithOptions: (still the only option). Both branches propagate the BOOL return — if activation is refused we error out before clobbering the clipboard instead of silently proceeding. The respondsToSelector: result is cached in a OnceLock so the probe isn't repeated on every paste. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(capture): conditional clipboard restore + always-attempt on paste failure Two bugs in paste_final_text' clipboard handling: 1. Restore was unconditional. If the user ⌘C'd in the target app during the 400 ms paste-consume window — or a clipboard history tool (Paste, Pastebot, Maccy) or Universal Clipboard sync snapshotted our staged text — the blind restore overwrote their newer content with the pre-paste snapshot, silently losing user data. 2. send_paste' errors were propagated with ? before the restore, so a CGEventPost / SendInput failure left the user's clipboard stuck on the transcript. Fix folds both into one pattern: capture the post-write change count, re-read it after paste-consume, restore only when they match (plus treat a change-count read failure as "unknown, don't overwrite"). Isolate send_paste's error so the restore runs regardless of paste success, then propagate the paste error after. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * chore(deps): pin rdev to jamiepine/rdev fork Upstream Narsil/rdev has shipped no release since 2023-06 (crates.io still serves 0.5.3), so the Sonoma main-thread fix we depend on — PR #147, applied at hotkey_monitor.rs:184 — is only reachable via a git pin. A pin to a third-party repo breaks the build whenever the remote force-pushes, renames, or is taken down, and Cargo does not durably cache git-dep archives the way it does crates.io tarballs. Forking to jamiepine/rdev at the same SHA removes that failure mode without changing crate behavior and gives us a place to cherry-pick future OS-compatibility fixes on our own timeline. The SHA was verified to exist on the fork before re-pinning. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(mcp): idle timeout + escalating backoff for speak-SSE monitor Two reliability gaps in the /events/speak subscriber: 1. resp.chunk().await had no idle timeout. A backend that accepts the TCP connection but stops producing frames (deadlocked SSE endpoint, zombie process) would block the task forever without reconnecting. The pill window would never surface for agent-initiated speech and there would be nothing to log. Backend emits a `:ping` heartbeat every 15 s, so 45 s without any data is now treated as a dead stream — the task errors out and the reconnect loop takes over. 2. Flat 2 s backoff escalates nowhere. Logs fill with reconnect lines when the backend is down for minutes, and a backend that accepts + immediately closes connections (no data) spins the loop tightly. Backoff now escalates 500 ms → 30 s on unproductive rounds and resets only when at least one frame arrives (the connection was genuinely productive, not just accepted). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(refinement): character-level loop collapse + pytest coverage The word-level pass catches single-word Whisper loops ("URL URL URL…") but misses two common hallucination patterns the PR had to claim as "edge cases": 1. Multi-word English loops — "thanks for watching thanks for watching…" × 6 sails through because no two consecutive tokens are identical after text.split(). 2. CJK loops — "謝謝觀看" × 7 sails through because text.split() returns a single unsplit token for the whole loop (no whitespace between characters). Add a character-level second pass: a non-greedy regex finds any 2–60 char substring that repeats min_run+ times immediately after itself and strips the run. The 2-char floor keeps emphasised single-letter runs ("wooooooow") intact. The 60-char ceiling covers every observed Whisper tail hallucination ("Please like and subscribe to my channel.", "Subtitles by the Amara.org community") while staying short enough that coincidental long-phrase repetition in legitimate speech doesn't hit the threshold. Whitespace normalisation only runs when the pass actually stripped something, so untouched transcripts keep their original spacing. New test_refinement_collapse.py gives the pre-processor its first deterministic unit-test coverage: 17 tests pinning the word-level legacy behaviour plus the new multi-word English / CJK / Japanese / emphasis-preservation cases. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(db): graceful fallback when SQLite < 3.35 on MCP bindings migration SQLite gained ALTER TABLE … DROP COLUMN in 3.35 (Mar 2021). Production PyInstaller builds bundle Python 3.12 which links to SQLite 3.40+ so that path is always safe, but a dev running the backend directly on Ubuntu 20.04 (3.31) or Debian 11 (3.34) would crash on first startup trying to drop the legacy default_intent column. Add _supports_drop_column(engine) — returns True on non-SQLite dialects (Postgres / MySQL have supported DROP COLUMN for decades) and gates on the runtime sqlite_version for SQLite. When unsupported, log a warning and leave the unused column in place: SQLAlchemy only maps declared columns, so a stray default_intent column does no reads or writes and can't interfere with runtime behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(mcp): correct lifespan shutdown order — drain MCP before unloading models The inline lifespan ran _run_shutdown inside the MCP context, so the TTS / Whisper / LLM models were unloaded *before* FastMCP's __aexit__ got a chance to cancel its in-flight session tasks. Any MCP request mid-generate at shutdown time would crash on "model unloaded" instead of receiving a clean session-cancelled error. Rewire via compose_lifespan (which was already defined in mcp_server.server for exactly this purpose but never used): AsyncExitStack enters factories in order and exits in LIFO, so MCP teardown fires first — cancelling sessions — and _run_shutdown runs after nothing is holding the models. Smoke test shows the log order flipped as expected: Ready StreamableHTTP session manager started ... running ... StreamableHTTP session manager shutting down ← was last, now first Voicebox server shutting down... ← was first, now last As a side benefit, _run_shutdown is now paired with _run_startup via try/finally inside voicebox_lifespan, so a partial startup (models half-loaded, MCP __aenter__ fails) still unloads whatever was loaded instead of leaking it to process exit. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(mcp): stamp last_seen_at on /speak too + tighten path predicate POST /speak is a REST wrapper around voicebox.speak for agents that don't talk MCP (shell scripts, ACP, A2A). It reads X-Voicebox-Client-Id and uses it for the same per-client profile resolution + default personality lookup the MCP tool does (speak.py:39-64), so its callers are first-class clients — but the ClientIdMiddleware only stamped last_seen_at on /mcp* paths. REST speak callers showed up as "never seen" in Settings → MCP despite actively acting on their bindings. Widen the stamp predicate to an explicit ("/mcp", "/speak") prefix list, and require a path boundary on match so future routes named /mcpfoo or /speakers don't silently inherit the stamp via the prefix. New test_client_id_middleware.py pins the scope with 17 parametrised cases (both the allowed set and the overlap cases that must not match). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(captures): scrubbable WaveSurfer player for capture detail view Replace the placeholder fake-waveform + play button in CapturesTab's audio card with a real CaptureInlinePlayer (wavesurfer.js). The player renders the actual waveform, lets users scrub through the clip, and shows a proper current/total timestamp pair in place of the duration-only label. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(ui): persist selectedProfileId across sessions Wrap useUIStore in zustand/middleware's persist under the key voicebox-ui. partialize only selectedProfileId so volatile UI state (dialog open flags, form drafts, engine/voice pickers, sidebar) stays in-memory as before — but reopening the app no longer loses whichever profile the user was last working with. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(captures): mirror readiness checklist into the settings sidebar The six-gate checklist only rendered in the CapturesTab empty state, so a user already on the settings page had no single surface showing which gate was red — the inline InputMonitoringNotice covered one, the model pickers covered another, and Accessibility was only hinted at by the auto-paste toggle. Mirror the same component into the right sidebar of the settings page so every gate (STT model, LLM model, Input Monitoring, Accessibility, plus the hotkey toggle in the main column) is always visible while the user configures dictation. New compact prop on DictationReadinessChecklist drops the centered header and empty-state max-width so it fits the 280 px sidebar next to the existing About / Differences blocks. Callers in compact mode own the heading — CapturesPage reuses the existing captures.readiness.title key (present in en / ja / zh-CN / zh-TW already) as an h3 matching the sibling sidebar sections. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(captures): move sidebar checklist below differences + hide when all green Two small follow-ups to the sidebar checklist placement. Move it below the What's different section so the sticky top of the sidebar stays the page's narrative context (About → differences) and the checklist reads as a status panel rather than preamble. Gate the whole block on !readiness.allReady so once every gate is green the sidebar drops back to just About + What's different — no value in real estate full of checkmarks. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(captures): refetch readiness immediately after STT/LLM model swap useCaptureSettings updated its own cache optimistically but never invalidated ['capture-readiness'], so for up to 5 s (the poll interval) after switching stt_model or llm_model the checklist kept showing the previous model's ready/missing state. The backend endpoint resolves the model live on each call — it was just the frontend cache that lagged. Invalidate in onSettled only when the patch touched a model field, so unrelated updates (chord keys, toggles) don't pay for a refetch. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(captures): hide macOS-only copy when running on Windows / Linux Two surfaces leaked macOS-specific copy onto other platforms: 1. The Input Monitoring + Accessibility rows in the readiness checklist rendered everywhere. On Windows/Linux the Rust permission stubs return true, so the rows showed as permanent green checkmarks with copy like "macOS allows Voicebox to detect your global shortcut." — nonsense when you're on Windows. Gate both rows on a userAgent-based isMacOS check so they only render where the underlying TCC permission actually exists. 2. The global-shortcut setting description ended with "macOS will ask for Input Monitoring permission the first time you turn this on." That sentence rendered on every platform. The readiness checklist already surfaces the TCC requirement at the right moment on macOS, so the description doesn't need the platform note — drop it from en / ja / zh-CN / zh-TW. Other macOS strings (AccessibilityNotice, InputMonitoringNotice, their "stillMissing" hints) are already gated behind the Rust permission booleans returning false, which never happens on Windows/Linux, so they stay inert without further changes. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(capture): swap the rdev fork for keytap 0.2, delete local chord state machine Dep swap: - Drop the git-pinned jamiepine/rdev fork we were carrying since the upstream crate is abandoned. - Depend on keytap 0.2 from crates.io — our own cross-platform global keyboard tap crate. Clean shutdown via Drop, Sonoma-safe by design (no TSMGetInputSourceProperty calls off the main thread, so `set_is_main_thread(false)` is gone), and properly versioned. Chord engine rewrite: - Delete hotkey_monitor.rs's internal Chord state machine (Match enum, KeyEvent enum, step()/classify() methods, associated unit tests). keytap's ChordMatcher subsumes it: Momentary chord for PTT, add_toggle() for Toggle-to-talk, longest-match resolution, sticky-end for Toggle. Net: -80 LOC in hotkey_monitor.rs; the remaining module is the dispatcher loop + Effect→Tauri translation. - Preserve the PTT→Toggle "RestartRecording" upgrade signal. keytap emits End(PTT)+Start(Toggle) atomically (same Instant) when the held set upgrades from a shorter chord to a longer superset. The dispatcher peeks at the matcher with a 5 ms recv_timeout after any End and coalesces the pair into Effect::RestartRecording so the frontend still gets the "discard the transition-moment audio" signal instead of an unrelated Stop+Start pair. - HotkeyMonitor::update_bindings now actually tears down the tap on empty bindings instead of leaving an idle CGEventTap around. New bindings rebuild the matcher and the dispatcher thread from scratch. key_codes.rs: - Rewrite the browser-code → Key table against keytap's cleaner Key variant names (`A`..`Z` not `KeyA`..`KeyZ`, `Digit0`..`Digit9` not `Num0`..`Num9`, `ArrowUp` not `UpArrow`, `AltLeft`/`AltRight` instead of `Alt`/`AltGr`, `Period` not `Dot`, …). On-disk chord string format (W3C `KeyboardEvent.code` identifiers) is unchanged, so capture_settings rows written before the swap round-trip identically. Legacy aliases (`Alt`, `AltGr`, `Num0`, `UpArrow`, `Dot`, …) kept for forward-compat on old rows. main.rs / input_monitoring.rs: - Update the few doc comments that referenced `rdev::listen` to describe keytap's Tap; no behavioural change. - build_chord_bindings now imports from keytap::Key. - enable_hotkey / disable_hotkey / update_chord_bindings reach into HotkeyMonitor via &mut since apply()/update_bindings() now mutate. Tests live in keytap now (22 chord-related tests in keytap 0.2, including the PTT→Toggle upgrade scenario that used to be tested in hotkey_monitor.rs). Voicebox's hotkey_monitor.rs is thin enough that local testing would be trivia. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * chore(deps): bump keytap 0.2 → 0.4 for macOS modifier-events fix 0.2 read CGEventFlags via CGEventGetIntegerValueField(event, 0x81), which is not a valid CGEventField id — macOS silently returned 0, so FlagsChanged events produced no KeyDown / KeyUp for any modifier key and the PTT / toggle chords never armed on macOS. 0.4 uses the documented CGEventGetFlags(event) API. 0.3 (tracing / serde / Fn / IntlBackslash) is picked up as a free consequence; no API surface we depend on changed. * perf(captures): stop polling readiness once both models are green useQuery was firing GET /capture/readiness every 5s forever, and also on every window focus. Once stt.ready and llm.ready are both true the answer can only change when the user swaps a model in settings, and useSettings already invalidates the query on that path — the polling was pure noise. Gate both refetchInterval and refetchOnWindowFocus on "not fully ready" so we fall silent once the checklist is green. * feat(ui): theme settings, stories polish, track editor restructure - add dark/light/system theme with persisted choice + OS change listener - restyle stories sidebar (search, item layout, border) to match captures - move floating generate box to right column of stories, add top fade mask - story track editor: sticky track labels aligned via flex rows, custom scrollbar with left/right zoom handles - capture pill light mode pass, fix inline waveform progress color - pull mcp_server hidden imports into the pyinstaller spec - notarization doc draft * fix mlx llm bundling * feat(ui): shared ListPane primitive + misc polish ListPane is a compound component (Header / TitleRow / Title / Actions / Search / Scroll) that owns the relative wrapper, faded right divider (50px top fade), top scroll mask, and absolute-positioned header used by every list-detail tab. Wires up CapturesTab, StoryList, and EffectsList. EffectsTab gets -mx-8 / pr-8 to match the edge-to-edge layout used elsewhere. Other changes: - MCPPage: native <select> → shadcn <Select> for default voice and per-binding voice pickers - Button outline variant: add hover:border-accent - Drop hover:text-destructive from trailing delete buttons (HistoryTable, GpuAcceleration, GpuPage, EffectsChainEditor, EffectsDetail) - HistoryTable empty state moved behind t('history.empty') - StoryContent scroll padding pt-14 → pt-16 - backend health reports the captures dir - landing CapturesMockup: "Send to" → "Export" with Download icon - CHANGELOG: drop [Unreleased] personality section * fix(captures): Play As autoplay + default voice + orphan recovery - Hand /generate ids to the global SSE watcher so playback fires on completion. The mutation onSuccess was checking audio_path on a queued row, which is always empty — autoplay never ran. - Bind the Play As voice selection to capture_settings.default_playback_voice_id, kept in sync with the Settings → Captures and Settings → MCP pickers. Picking from the split-button dropdown writes back to settings. - Extract AudioBars from HistoryTable into a shared component; use it for the Play As generating state in place of Loader2. - Stop the active-state hover from flashing white text when the button is in its lighter accent/10 fill. - Drop the gradient avatar swatches from the Settings → Captures voice dropdown. - Backend: when the gen worker exits without writing a terminal status (e.g. SQLite lock racing the failed-status write inside its own exception handler), the cancel endpoint now flips the row to failed instead of 409-ing. Worker also force-fails on its way out as a belt-and-suspenders. * fix(captures+chord): Stop button stops, ChordPicker accepts shorter chords Two unrelated correctness bugs caught in PR review: - The Play As "Stop" button was wired to handlePlayAs() unconditionally, so clicking it during playback kicked a fresh generation instead of halting. Now pauses the player when the click came from the main button while playbackState is 'playing'. Picking a different voice from the dropdown still kicks a new generation as before. - ChordPicker tracked the peak set of held keys but seeded the peak from initialKeys, so a user who opened the picker with a 3-key chord saved couldn't replace it with a 2-key chord — the candidate length never beat the seed. The peak now resets on the first press of a fresh sequence (when no keys were held immediately prior), then grows monotonically within that hold. * fix(settings): honor explicit null on nullable fields, ignore on the rest Routes were calling model_dump(exclude_none=True), which drops every client-sent null before it reaches the service. The service then layered on its own `if value is not None` guard. Net effect: setting a nullable column back to null was a no-op — the MCPPage default-voice picker sends null when the user picks "no default" and the row was silently keeping whatever was there before. Switched the routes to exclude_unset=True so absent fields stay absent but explicit nulls survive the dump, and centralised the per-field nullability check in the service. The check inspects the SQLAlchemy column metadata so non-nullable columns (stt_model, llm_model, the chord key lists) still drop nulls instead of crashing the request, while default_playback_voice_id can finally be cleared. * fix(captures): clean up audio files when create_capture fails The create flow wrote raw audio (and a transcoded .wav for non-wav sources) to data/captures before the DB row was committed, so any failure between the write and the commit — a webm that decoded to a 0-length array, a whisper model that errored mid-transcribe, a SQLite contention on the commit — left the audio on disk with nothing pointing at it. Over enough flaky uploads the directory grows without bound. Now every path written before the commit is tracked in a list, and the whole stretch from the first write to db.commit() runs inside a try/except that unlinks each tracked file on raise and re-raises. The transcode branch removes the raw file from the cleanup list only when the unlink actually succeeds, so an OSError on the raw-path delete still hands cleanup the original blob to retry. * fix(mcp): restrict voicebox.transcribe(audio_path=...) to loopback audio_path mode took any absolute filesystem path and returned its decoded contents as transcribed text with no caller verification beyond the existence/size checks. The X-Voicebox-Client-Id middleware records the header but never rejects an absent or fake one, so a Voicebox bound to 0.0.0.0 (the documented "remote access" mode) was effectively an unauthenticated arbitrary-local-file read primitive. The middleware now stashes the request's remote address in a ContextVar alongside the existing client_id, and audio_path mode refuses anything that doesn't parse as a loopback address (IPv4 127.0.0.0/8, IPv6 ::1). audio_base64 mode is unchanged — that path was always bounded to bytes the caller already has. Loopback callers (the Tauri webview, local CLI scripts, MCP clients on the same machine) keep working. Remote callers now have to send the audio over the wire if they want it transcribed. * fix: PR review nits — response shape, landing copy, form reset - /llm/generate's "model is downloading" branch was raising HTTPException(202, detail={...}), which wraps the payload in {"detail": ...} and forces clients to parse a success status as if it were an error. Switched to JSONResponse so the payload sits at the top level. - The landing page's "Language Models" card advertised "Qwen 3.5" with sizes 4B/2B/0.8B; we ship Qwen3 at 0.6B/1.7B/4B. Aligned to what's actually in the binary. - ProfileForm's discard-draft button reset the form without touching `personality` or `avatarFile`, so stale persona text and an attached avatar would survive the discard. The other three resets in the file already include both fields — this brings the discard path in line. * perf(mcp): move last_seen_at stamp off the request path ClientIdMiddleware was running the SQLAlchemy SELECT/INSERT/UPDATE/COMMIT inline on the event loop after every /mcp/* and /speak request. SQLite serialises writes, so concurrent MCP traffic queued behind the stamp write — the response sat waiting on a side-effect that the client never needs in band, and SSE streams would stall briefly per request. The middleware now hands the stamp to asyncio.to_thread via a fire-and- forget create_task so the response returns immediately and the write runs on the default executor. A module-level set keeps strong refs to in-flight tasks (per asyncio docs) so the GC can't collect them mid- write. The fallback path runs the stamp inline if no loop is available (tests/oddball callers) rather than silently dropping it. * fix(dictate): force-dismiss the speaking pill when SSE never comes back The pill subscribed to /generation/{id}/status to know when to start playback, but EventSource.onerror was a no-op — auto-reconnect was the intended recovery for transient drops. The gap: if the backend deletes the gen row mid-flight or the connection silently dies in a way the browser keeps retrying without ever getting a status event, the pill sits in 'speaking' forever and the user has no way to clear it. Added a 60-second hard cap that arms when the SSE opens and clears the moment any real status event lands. If it fires while the pill is still on the same id and audio never started, it force-dismisses. Same idea as the existing post-speak-end 15s grace, but covers the case where the backend never says anything at all. * fix: i18n cleanup + readiness checklist effect cadence + ChordPicker shadow - DictationReadinessChecklist was constructing downloadByModel as a fresh Map every render and listing it in the cleanup effect's deps. With the 1 s polling cadence and arbitrary parent rerenders the effect ran more often than it needed to. Memoised the Map on activeTasks; the effect now keys off the memo's identity. - zh-CN persona tooltipActive/ariaLabelActive matched their inactive twins byte-for-byte ("以人物设定朗读"). The other locales differentiate the active state with a -ing / -中 suffix; zh-CN now reads "正以人物设定朗读" when active. - personalityPlaceholder was a ~290-character paragraph that doubled as both the example text and the explanation, repeating most of what personalityHint already said. Trimmed to the example only and folded the explanation + leave-blank consequence into the hint, across all four locales. - Refinement model size keys were size06 / size17 / size4. Renamed the 4B variant to size40 so the decimal padding is consistent. - ChordPicker's open-effect bound a window.setTimeout id to a local `t`, shadowing the i18n `t` from useTranslation. Renamed to timeoutId. * perf(settings): persist generation sliders on release, not per pointer-move Both sliders on the generation settings page were calling update() — which is a React Query mutation that PATCHes /settings/generation — inside onValueChange. Dragging the chunk-limit slider from 800 to 3000 fired a request per pointer-move pixel, and a mid-drag failure plus optimistic rollback would leave persisted state visibly out of sync with the thumb position. Local state now mirrors each slider during a drag and the persist happens once on Radix's onValueCommit (pointer-up / keyboard-release). useEffects keep the local state in sync if the persisted value changes out-of-band — another window editing the same setting still updates the slider position cleanly. * chore(backend): Ruff lint pass — deprecated APIs, exception leaks, dead patterns Mechanical sweep of items called out in the PR review: - qwen_llm_backend: AutoModelForCausalLM.from_pretrained(torch_dtype=…) is deprecated in transformers ≥4.41 in favor of dtype=. Renamed. - routes/llm: try/except around backend.generate() raised HTTPException(500, detail=str(e)) which leaks stack traces / paths to clients and trips Ruff B904. Now logs the original exception server-side and hands the client a generic message; chained via `from e` to preserve traceback context. - mcp_bindings + mcp_server/context: datetime.utcnow() is deprecated since 3.12. Switched the two assignment sites to datetime.now(timezone.utc). The schema-level `default=datetime.utcnow` defaults in database/models.py are left for a later schema-aware pass. - routes/generations: `logger = …` sat between two import blocks (Ruff E402). Moved below imports. - mcp_server/server + tests/test_refinement_samples: typing.Callable / typing.Iterable have been preferred-via collections.abc since 3.9 (Ruff UP035). - routes/events: `except asyncio.TimeoutError` aliases plain `TimeoutError` since 3.11 (UP041). - services/captures: hoisted WHISPER_NATIVE_FORMATS to module scope (was a function-local UPPER_SNAKE that tripped N806) and replaced the raw_path.unlink try/except OSError-pass with contextlib.suppress (SIM105). Semantic equivalence preserved — written_files.remove(raw_path) still only runs when unlink succeeds because it sits inside the suppressed block after the unlink call. - database/migrations: hoisted the duplicate `import sqlite3` from inside two helper bodies to a single module-level import. * feat(stories): regenerate action on clips and the chat list dropdown The track editor's clip toolbar now has a regenerate icon next to Delete; clicking it kicks a fresh take of the selected clip's underlying generation through the same /generate/{id}/regenerate path the History table uses, and pushes the id into the global pending set so the SSE watcher picks it up. The chat list's per-item dropdown gets the same action between Play-from-here and Remove. Translation keys added under storyContent.itemActions / storyContent.toast across all four locales. * feat(stories): import external audio into the timeline (drag-drop + picker) You can now drop a music file onto the story content area or pick one through the new "Import audio" button in the add-clip popover. Both call POST /generate/import which writes the file to data/generations/<id>.<ext>, probes duration via librosa, and inserts a Generation row pointing at a singleton "Imported Audio" profile (created lazily on first import). The existing addStoryItem flow takes over from there — the timeline doesn't care that the row didn't come out of TTS. Engine field on the row is "import"; it's surfaced on StoryItemDetail so the chat list shows a music icon instead of the (missing) profile avatar and both the dropdown and the track-editor toolbar hide the Regenerate action — there's nothing to regenerate. Accepted formats: wav/mp3/flac/ogg/m4a/aac/webm, capped at 200 MB. Translation keys added across en/ja/zh-CN/zh-TW. * fix(audio): serve real Content-Type so imports decode in WaveSurfer /audio/{id} and /audio/version/{id} hardcoded media_type="audio/wav" on the FileResponse. That was a no-op when every generation came out of TTS (everything on disk was a .wav anyway), but imported audio keeps its source format — .mp3 / .m4a / .ogg — and the WaveSurfer MediaElement backend uses an <audio> tag that checks Content-Type before letting the clip play, so an MP3 announced as audio/wav silently failed to load. Both endpoints now derive the type via mimetypes.guess_type and fall back to audio/wav for unknown suffixes. Download filenames also keep the real extension instead of always saying ".wav". * feat(stories): zoom bar bounds tracked to project length, default 60s scope The track editor's zoom was clamped to a hardcoded [10, 200] pixels-per-second range, which had no relationship to the project — on a 4-minute story a "max zoom out" of 200 px/s still required scrolling, and on a 5-second story you could zoom all the way in to where every clip was a tiny sliver. Reframed the bounds in the unit the user actually thinks in: how many seconds of timeline are visible at once. Min scope is 10 s (most zoomed in), max scope is the entire project, and the default lands on a 60 s scope (or the full project, whichever is shorter) once the editor measures its visible track width on first mount. The pixels-per-second value still lives in component state (because every downstream calculation already uses it) but minPps/maxPps are computed from `containerWidth − LABEL_COL_WIDTH` and the project's effective duration, so the +/- buttons and the edge-drag handles on the scrollbar all clamp to bounds that move with the project. Re-clamping fires whenever those bounds shift — adding a long clip or resizing the window pulls the current zoom inside the new range instead of leaving the user parked outside it. * fix(stories): show the source filename on imported clips Imports were rendering as "Imported Audio" everywhere because every import points at the singleton voice profile. The filename was already being stored on the generation row (in the `text` field), so the chat item title and the timeline clip label now read from `text` when `engine === 'import'` and fall back to the profile name otherwise. The chat item also drops the language pill (always "en" on imports — not informative) and skips the transcript textarea since imports have no spoken text to show. * fix(stories): round split_time_ms before posting handleSplit was sending currentTimeMs - item.start_time_ms straight to the backend, which rejects it because StoryItemSplit.split_time_ms is typed as int and the playhead's currentTimeMs is a float (it's driven from HTMLAudioElement.currentTime, which carries sub-millisecond precision). Pydantic surfaced the mismatch as "Input should be a valid integer, got a number with a fractional part" and the toast read "Failed to split clip". Math.round at the call site, matching what the trim and move handlers already do. * feat(stories): per-clip volume control on the timeline Each story item now carries a volume column (linear gain, default 1.0, clamped 0.0–2.0 server-side). New PUT /stories/{}/items/{}/volume route + useUpdateStoryItemVolume hook + a Volume2 icon in the clip-edit toolbar that opens a popover with a 0–200% slider. Local slider state drives the visual during a drag; the persist fires once on onValueCommit, mirroring the generation-page slider pattern. Web Audio playback inserts a per-clip GainNode between source and master so volume changes apply live without re-decoding the buffer (source -> clipGain -> masterGain -> destination). Server-side mixdown in export multiplies the trimmed clip by its volume before summing into the timeline. Split + duplicate carry the volume forward to the new clips so trimming a faded section keeps the level you set. Migration adds the volume column with default 1.0 so existing rows read as full volume. * fix(stories): mute the clip waveform's media element so it can't bleed audio The clip waveforms drawn inside each timeline track use WaveSurfer with the default MediaElement backend, which creates an internal <audio> element to drive playback timing. Web Audio in useStoryPlayback is what actually produces sound, but WaveSurfer's element was happily preloading and — after the first user gesture unlocked browser autoplay — playing the source URL through the page output too. For TTS clips it was masked: they're short, both sources start at the same time, and stopping the BufferSourceNode at pause coincides with the natural end of the audio element. For long imports (a four-minute MP3) the BufferSourceNode stops on pause but WaveSurfer's element keeps going on its own track — which is exactly the "music keeps playing when I pause" symptom. Hand WaveSurfer a muted <audio> element via the `media` option so the visual still loads peaks but the element itself can never produce sound. preload="metadata" keeps the load lightweight. * fix(stories): hard-cut the audio graph on stop so long imports actually halt source.stop() was the only thing happening when a clip was halted, and on long imported buffers (multi-minute MP3s scheduled via source.start with a duration argument) it was silently failing to halt the buffer in some browsers — pause left the music playing and seek stacked another source on top of the original. The mute-the-WaveSurfer-element fix was a different bug along the same path; this is the one that actually addresses the duplicated audio. ActiveSource now carries the per-clip GainNode alongside the source, and stopSource detaches the onended handler before calling stop() (so the natural-end callback can't race with explicit teardown and re-delete a freshly rescheduled entry at the same id), then disconnects both nodes inside their own try/catch blocks. Even when stop() doesn't actually halt the buffer the graph is severed — no path from source to destination, no audio. * feat(stories): add empty tracks above/below the timeline Tiny + strips sit at the top of the topmost label cell and the bottom of the bottommost one, sticky-positioned in the label column so they follow horizontal scroll. Clicking either extends the visible track stack in that direction by one — above adds max(existing)+1, below adds min(existing)-1. Both compute against the full set (defaults + item-derived + previously-added) so successive clicks keep extending instead of fighting over the same number. Empty extras live in component state because a track only earns its keep once a clip lands on it. Once one does, item.track carries the number forward and the row keeps deriving from items naturally; if nothing lands there before reload, the empty row simply isn't there next time, which matches what the user expects of an unused affordance. * fix(mcp): bundle stdio shim sidecar * fix(captures): allow dictation without paste permission * fix(mcp): preserve speak engine defaults * fix(captures): use platform hotkey defaults * fix(mcp): preload speak pill window * fix(captures): hide unwired storage settings * feat(sponsors): add /sponsors page, homepage promo, and in-app strip * style(landing): drop pill chrome from /download maintainer kicker * changelog * better naming for sponsors * windows keybind note --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Jamie Pine
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
627d40b42d
commit
7df366d0c8
+25
-16
@@ -1,23 +1,30 @@
|
||||
---
|
||||
title: "Voicebox Documentation"
|
||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
|
||||
---
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is the **open-source, local-first AI voice studio** — a free
|
||||
alternative to ElevenLabs and WisprFlow in one app. Clone voices, generate
|
||||
speech across 7 TTS engines, dictate into any app with a global hotkey,
|
||||
compose multi-voice projects, and let any MCP-aware agent speak in a voice
|
||||
you own. Everything runs on your hardware.
|
||||
|
||||
|
||||
|
||||
- **Complete privacy** -- models and voice data stay on your machine
|
||||
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
|
||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** -- REST API for integrating voice synthesis into your own projects
|
||||
- **Native performance** -- built with Tauri (Rust), not Electron
|
||||
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
|
||||
- **Dictation** — hold a chord anywhere on your machine, speak, release; the transcript pastes into the focused field
|
||||
- **Captures tab** — paired audio + transcript archive, retranscribe / refine / play-as-voice
|
||||
- **Voice personalities** — per-profile compose button + persona-rewrite toggle, powered by a local LLM
|
||||
- **Agents speak back** — any MCP-aware agent can call Voicebox to speak in one of your cloned voices
|
||||
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, Kokoro
|
||||
- **Cloning and preset voices** — zero-shot cloning or 50+ curated preset voices
|
||||
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, filters
|
||||
- **Expressive speech** — paralinguistic tags (`[laugh]`, `[sigh]`) and natural-language delivery control
|
||||
- **Unlimited length** — auto-chunking with crossfade for long scripts
|
||||
- **Stories editor** — multi-track timeline for conversations, podcasts, narratives
|
||||
- **API-first** — REST + WebSocket API, MCP server for agent integrations
|
||||
- **Complete privacy** — models, audio, transcripts, LLM output never leave your machine
|
||||
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux (ROCm / CPU), Intel Arc, Docker
|
||||
|
||||
## Download
|
||||
|
||||
@@ -32,6 +39,8 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
|
||||
|
||||
## Get Started
|
||||
|
||||
- [Installation](/overview/installation) -- download and install Voicebox
|
||||
- [Quick Start](/overview/quick-start) -- get up and running in 5 minutes
|
||||
- [API Reference](/api-reference) -- integrate voice synthesis into your apps
|
||||
- [Installation](/overview/installation) — download and install Voicebox
|
||||
- [Quick Start](/overview/quick-start) — get up and running in 5 minutes
|
||||
- [Dictation](/overview/dictation) — start talking to your computer
|
||||
- [Voice Personalities](/overview/voice-personalities) — compose and rewrite in any profile
|
||||
- [API Reference](/api-reference) — integrate voice synthesis into your apps
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
---
|
||||
title: "Captures"
|
||||
description: "The paired audio + transcript archive — every dictation, recording, and uploaded audio file shows up here, replayable and retranscribable."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A **capture** is an audio clip paired with its transcript. The Captures tab
|
||||
is where every dictation, manual recording, and uploaded audio file lands,
|
||||
with the original audio kept alongside the text so you can replay, re-run
|
||||
transcription with a different model, refine the transcript, or send the
|
||||
content somewhere else — including generating it back as speech in any of
|
||||
your voice profiles.
|
||||
|
||||
<Callout type="info">
|
||||
The Captures tab shipped in **0.5.0**, alongside global dictation and the
|
||||
per-profile personality modes. If you've used earlier versions, note that
|
||||
the Audio tab moved into **Settings → Audio Channels** to make room for
|
||||
this one.
|
||||
</Callout>
|
||||
|
||||
## Where captures come from
|
||||
|
||||
| Source | How it shows up | Badge |
|
||||
|---|---|---|
|
||||
| **Dictation** | Triggered by the global hotkey (see [Dictation](/overview/dictation)). Auto-refined by default. | `dictation` |
|
||||
| **In-app recording** | Recorded directly in the Captures tab using the built-in mic. | `recording` |
|
||||
| **File upload** | Any audio file dropped into the Captures tab — `.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`. | `file` |
|
||||
|
||||
All three paths share the same backend pipeline, the same model picker, and
|
||||
the same refinement flags. The source badge is there so you can visually
|
||||
scan a long list.
|
||||
|
||||
## List view
|
||||
|
||||
The main Captures view is a chronological list. Each row shows:
|
||||
|
||||
- The transcript (raw or refined — the refined version wins if present)
|
||||
- Duration + timestamp
|
||||
- Source badge
|
||||
- A play button for the original audio
|
||||
- A meatballs menu with per-row actions
|
||||
|
||||
Filtering and search are a Tier-2 ask — ping if you need them.
|
||||
|
||||
## Detail view
|
||||
|
||||
Clicking into a capture opens the detail view:
|
||||
|
||||
- **Waveform player** for the original audio
|
||||
- **Transcript editor** — click in and edit. Changes save on blur.
|
||||
- **Refined vs. raw toggle** if refinement ran on this capture
|
||||
- **Per-capture action bar** — retranscribe, refine, play as voice, delete
|
||||
- **Settings snapshot** — STT model used, refinement flags at the time
|
||||
this capture was processed, and the voice model if any was played
|
||||
|
||||
## Retranscribe
|
||||
|
||||
Runs the capture's original audio through a different Whisper model without
|
||||
re-uploading or re-refining anything. Useful when:
|
||||
|
||||
- The default model mis-heard something and you want to try a larger model
|
||||
- You used Base for a noisy clip and want to rerun with Turbo
|
||||
- A non-English clip needs an explicit language hint
|
||||
|
||||
**Settings → Captures → Transcription** controls the default model and
|
||||
language lock for new captures. Retranscribe uses those defaults unless you
|
||||
override them per capture.
|
||||
|
||||
## Refine
|
||||
|
||||
Runs the raw transcript through the local LLM to produce a cleaned-up
|
||||
version. The flags on the capture are snapshotted when refinement first
|
||||
runs, so you can re-refine later with different flags without losing the raw
|
||||
transcript:
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| **Smart cleanup** | Remove fillers (`um`, `uh`, `like`), tidy punctuation and capitalization. |
|
||||
| **Remove self-corrections** | Keep the final version when the speaker backtracks ("actually, no, on Tuesday"). |
|
||||
| **Preserve technical terms** | Leave identifiers (`handleSubmit`, `npm install`) untouched. |
|
||||
|
||||
See the Refinement section of [Dictation](/overview/dictation#refinement) for
|
||||
how Voicebox strips Whisper loop hallucinations *before* the LLM sees the
|
||||
transcript — a capture can be re-refined any number of times without
|
||||
re-introducing "thanks for watching thanks for watching" echoes.
|
||||
|
||||
The refinement model picker (three bundled Qwen3 sizes) lives in
|
||||
**Settings → Captures → Refinement**.
|
||||
|
||||
## Play as voice
|
||||
|
||||
This is the capability no one else in the dictation category ships: take any
|
||||
capture and play it back as speech in any of your voice profiles. One
|
||||
dropdown over every profile, one click, and the capture's text runs through
|
||||
`/generate` with the selected voice.
|
||||
|
||||
Use cases:
|
||||
|
||||
- Hear your own dictation back in a cloned voice of someone you like
|
||||
- Send a message you dictated as an audio reply in a specific character
|
||||
- Quickly prototype a line for a story without retyping
|
||||
|
||||
Playback uses whatever engine the selected profile is bound to — the same
|
||||
rules as the Generate tab. There's no LLM in this path; the transcript goes
|
||||
through unchanged. If you want the agent-style "transform the content before
|
||||
speaking" flow, that's what the
|
||||
[personality modes](/overview/voice-personalities) do — and the same
|
||||
primitive is exposed to MCP-aware agents via the
|
||||
[MCP Server](/overview/mcp-server) so Claude Code, Cursor, or Cline can speak
|
||||
in one of your voices on their own.
|
||||
|
||||
<Callout type="info">
|
||||
The default voice for the Captures tab's Play-as action is set in
|
||||
**Settings → Captures → Playback → Default voice**. You can still override
|
||||
it per capture.
|
||||
</Callout>
|
||||
|
||||
## Send-to menu
|
||||
|
||||
Each capture has a Send-to menu for moving its content into other parts of
|
||||
Voicebox:
|
||||
|
||||
- **Copy transcript** — to clipboard
|
||||
- **Use as voice sample…** — promote this capture to a sample on a voice
|
||||
profile of your choice. Opens a profile picker (with "+ New voice" for
|
||||
cold starts) and a reference-text confirm dialog, because cloning needs
|
||||
the `reference_text` to match the audio verbatim. Edit as needed and
|
||||
save — the capture stays in the Captures tab untouched; the sample is a
|
||||
copy, not a move.
|
||||
|
||||
## Storage
|
||||
|
||||
The original audio is kept alongside the transcript in your Voicebox data
|
||||
directory. **Settings → Captures → Storage** shows the captures folder and can
|
||||
open it directly in your file manager.
|
||||
|
||||
Every capture's audio file and metadata row can be re-processed (retranscribe,
|
||||
refine, Play-as) as long as the audio file still exists.
|
||||
|
||||
## Short-recording guard
|
||||
|
||||
Audio clips under **300 ms** are short-circuited client-side and never
|
||||
uploaded. This prevents a fumbled chord tap from landing an empty capture.
|
||||
The threshold is tuned to filter accidents without cutting off intentional
|
||||
short dictations.
|
||||
|
||||
## Keyboard shortcuts
|
||||
|
||||
Inside the Captures tab:
|
||||
|
||||
| Keys | Action |
|
||||
|---|---|
|
||||
| `Space` | Play / pause the selected capture |
|
||||
| `↑` / `↓` | Previous / next capture in the list |
|
||||
| `Enter` | Open the selected capture in detail view |
|
||||
| `⌘ / Ctrl` + `C` (in detail view) | Copy the transcript |
|
||||
|
||||
## API surface
|
||||
|
||||
The Captures tab is backed by a small set of REST endpoints:
|
||||
|
||||
| Method | Endpoint | Use |
|
||||
|---|---|---|
|
||||
| `POST` | `/captures` | Upload audio + start the pipeline (STT, optional refinement, archival). |
|
||||
| `GET` | `/captures` | List captures. |
|
||||
| `GET` | `/captures/{id}` | Fetch one capture. |
|
||||
| `POST` | `/captures/{id}/retranscribe` | Rerun STT with a chosen model. |
|
||||
| `POST` | `/captures/{id}/refine` | Rerun refinement with chosen flags. |
|
||||
| `POST` | `/profiles/{id}/samples/from-capture/{capture_id}` | Promote a capture to a voice profile sample. |
|
||||
|
||||
These endpoints are stable and usable from your own scripts — see
|
||||
[Remote Mode](/overview/remote-mode) for running Voicebox as a server the rest
|
||||
of your machine can talk to.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
The global hotkey flow that feeds most captures.
|
||||
</Card>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
Per-profile compose button and persona rewrite toggle for captures you
|
||||
want to transform, not just transcribe.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Promote a capture into a voice sample on a profile.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
title: "Dictation"
|
||||
description: "Hold a key anywhere on your machine, speak, release — the transcript lands in whatever text field you had focused."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Dictation lets you turn speech into clean text anywhere on your computer. Hold
|
||||
a chord, talk, release — Voicebox transcribes what you said with Whisper,
|
||||
optionally cleans it up with a local LLM, and pastes the result into the text
|
||||
field you had focused when you started.
|
||||
|
||||
Everything happens on your hardware. No cloud, no accounts, no audio leaving
|
||||
the machine.
|
||||
|
||||
<Callout type="info">
|
||||
Dictation was introduced in **0.5.0** alongside the Captures tab and the
|
||||
per-profile personality modes. It's the "input" half of Voicebox's voice I/O
|
||||
loop — cloning and TTS are still the "output" half.
|
||||
</Callout>
|
||||
|
||||
## The flow
|
||||
|
||||
<Steps>
|
||||
<Step title="Hold the chord">
|
||||
Hold the push-to-talk chord anywhere on your machine. A small pill fades
|
||||
in over your current app.
|
||||
</Step>
|
||||
<Step title="Speak">
|
||||
The pill shows `Recording` with a live waveform and an elapsed-time
|
||||
counter. Speak naturally — you don't have to wait for anything.
|
||||
</Step>
|
||||
<Step title="Release">
|
||||
On release, the pill flips to `Transcribing`, then `Refining` if
|
||||
auto-refine is on, then disappears.
|
||||
</Step>
|
||||
<Step title="Text lands in your app">
|
||||
If auto-paste is enabled and Voicebox has Accessibility permission, the
|
||||
transcript pastes into the text field you had focused when you started
|
||||
talking — not wherever focus drifted while you were speaking.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
Either way, every capture also appears in the **Captures tab** with the
|
||||
original audio and the transcript paired together. See
|
||||
[Captures](/overview/captures) for what you can do with them after the fact.
|
||||
|
||||
## Push-to-talk and toggle modes
|
||||
|
||||
Voicebox ships two chord behaviors out of the box:
|
||||
|
||||
| Mode | Default (macOS) | Default (Windows) | Behavior |
|
||||
|---|---|---|---|
|
||||
| **Push-to-talk** | Right `⌘` + Right `⌥` | Right `Ctrl` + Right `Shift` | Recording stops when you release the chord. |
|
||||
| **Toggle-to-talk** | Push-to-talk + `Space` | Push-to-talk + `Space` | Recording keeps going until you tap the chord again. |
|
||||
|
||||
**Holding PTT and tapping `Space` mid-hold upgrades a hold into a toggled
|
||||
session** without a gap in the audio. This is the single most useful detail of
|
||||
the chord system — short bursts feel fast, long-form narration feels
|
||||
hands-free, and there's no decision up front about which mode you wanted.
|
||||
|
||||
## The on-screen pill
|
||||
|
||||
While you're dictating, a floating pill appears over the current app. It walks
|
||||
through the states of the capture cycle and shows live signals for each:
|
||||
|
||||
| State | What it shows |
|
||||
|---|---|
|
||||
| `Recording` | Live waveform + elapsed time. |
|
||||
| `Transcribing` | Thinking waveform while Whisper runs. |
|
||||
| `Refining` | Same thinking waveform while the LLM cleans up the transcript (only if auto-refine is on). |
|
||||
| Error | Red tint. Click the pill to copy the error to your clipboard. Auto-dismisses. |
|
||||
|
||||
The pill is transparent, always-on-top, and pre-created hidden at app start —
|
||||
so it appears instantly when you hit the chord, with no window flash.
|
||||
|
||||
## Customizing the chord
|
||||
|
||||
Open **Settings → Captures → Dictation** to change either chord.
|
||||
|
||||
- **Left vs right modifier badges.** When you hold keys into the chord
|
||||
picker, Voicebox records whether each modifier is the left or right variant.
|
||||
That means you can bind to just the right `⌥` while leaving the left `⌥`
|
||||
alone — useful if you want dictation on one hand and keep your
|
||||
other-hand shortcuts intact.
|
||||
- **Chord defaults are picked to stay out of your way.** On macOS, the
|
||||
defaults deliberately avoid left-hand `Cmd+Option` chords so
|
||||
`Cmd+Option+I` (devtools), `Cmd+Option+Esc` (force quit), and
|
||||
`Cmd+Option+Space` (Spotlight) all remain yours. On Windows, the defaults
|
||||
route around AltGr collisions on German / French / Spanish layouts where
|
||||
`Ctrl+Alt` synthesizes AltGr.
|
||||
- **Live reload.** Changing a chord in Settings takes effect immediately —
|
||||
no restart, no tab reload.
|
||||
|
||||
## Auto-paste into the focused app
|
||||
|
||||
Once transcription finishes, Voicebox can synthesize a native paste into
|
||||
whatever text field had focus when you started the chord. Your clipboard is
|
||||
saved before and restored after, so nothing you had copied goes missing.
|
||||
|
||||
| Platform | Mechanism |
|
||||
|---|---|
|
||||
| macOS | `CGEventPost` at the HID tap with a full `⌘V` key sequence, preceded by reactivating the original app via `NSRunningApplication`. |
|
||||
| Windows | `SendInput` with correct scan codes, plus a `SetForegroundWindow` + `AttachThreadInput` handshake to defeat foreground-lock when pasting into a window that wasn't frontmost at chord-start. |
|
||||
|
||||
**Focus is snapshotted at chord-start.** The paste targets the original field
|
||||
even if focus drifts during transcribe / refine — that's the "pastes where you
|
||||
were talking *from*, not where you're looking *now*" behavior.
|
||||
|
||||
<Callout type="info">
|
||||
Auto-paste is optional. If Accessibility permission isn't granted (macOS),
|
||||
or you prefer to keep synthetic input off, dictation still runs — transcripts
|
||||
land in the Captures tab and you can copy them manually. The setting lives
|
||||
inline next to the Accessibility prompt in Settings → Captures → Dictation,
|
||||
not as a global banner.
|
||||
</Callout>
|
||||
|
||||
## Refinement
|
||||
|
||||
If auto-refine is on, a local LLM cleans up the raw Whisper transcript
|
||||
before it's pasted. The goal is to remove verbal clutter without rewriting
|
||||
what you actually said.
|
||||
|
||||
What refinement typically fixes:
|
||||
|
||||
- Filler words (`um`, `uh`, `like` used as pauses, `you know`)
|
||||
- Self-corrections — the LLM keeps the final version and drops earlier
|
||||
attempts (`could you uh run the migration real quick, and then, yeah,
|
||||
check the logs` → `Could you run the migration, then check the logs?`)
|
||||
- Basic punctuation and capitalization
|
||||
- Whisper loop hallucinations — Voicebox strips repeated tokens (six or
|
||||
more identical tokens in a row, case-insensitive) *before* the LLM
|
||||
sees the transcript, so a small refinement model can't echo them back
|
||||
|
||||
What refinement deliberately preserves:
|
||||
|
||||
- Technical terms and code identifiers (`npm install`, `handleSubmit`)
|
||||
- Legitimate repetition (`no, no, no, no, no` has fewer than six identical
|
||||
tokens, so it survives)
|
||||
- Your intent — refinement is cleanup, not rewriting
|
||||
|
||||
Flags are snapshotted per capture, so you can re-refine the same raw
|
||||
transcript later with different flags without losing the original. The
|
||||
refinement model picker (**Settings → Captures → Refinement**) offers three
|
||||
bundled Qwen3 sizes:
|
||||
|
||||
| Model | Size | Best for |
|
||||
|---|---|---|
|
||||
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual dictation. |
|
||||
| Qwen3 1.7B | ~1.1 GB | Sweet spot when transcripts contain code identifiers. |
|
||||
| Qwen3 4B | ~2.5 GB | Full quality, slowest. |
|
||||
|
||||
This is the same local LLM used by the per-profile personality modes — one
|
||||
LLM in the app, not two. See [Voice Personalities](/overview/voice-personalities).
|
||||
|
||||
## Platform notes
|
||||
|
||||
### macOS
|
||||
|
||||
- **Accessibility permission** is required for auto-paste. The prompt lives
|
||||
inline next to the toggle in **Settings → Captures → Dictation**, with a
|
||||
deep link to **System Settings → Privacy & Security → Accessibility**.
|
||||
- **TSM crash mitigation.** The global hotkey listener runs on a background
|
||||
thread with `set_is_main_thread(false)` to sidestep a known
|
||||
macOS 14+ crash in the `rdev` library. If you hit an unexpected dictation
|
||||
failure on macOS, check the logs for TSM-related messages.
|
||||
|
||||
### Windows
|
||||
|
||||
- **UAC / UIPI caveat.** Synthetic paste into an *elevated* window from a
|
||||
non-elevated Voicebox is blocked by Windows itself. Run Voicebox elevated
|
||||
if you regularly dictate into elevated apps (e.g. an elevated terminal or
|
||||
Task Manager).
|
||||
- **Right-hand default chord** (`Ctrl+Shift`) avoids AltGr collisions on
|
||||
keyboard layouts where `Ctrl+Alt` is the compose key (German, French,
|
||||
Spanish, some others).
|
||||
|
||||
### Linux
|
||||
|
||||
- **Not yet in this release.** The Rust shim ships the macOS and Windows
|
||||
paths in 0.5.0. Linux `uinput` / AT-SPI support and the Wayland paste
|
||||
story are tracked in `docs/plans/VOICE_IO.md`.
|
||||
|
||||
## When auto-paste skips itself
|
||||
|
||||
A few cases where Voicebox deliberately does *not* synthesize a paste:
|
||||
|
||||
- **Focus was inside Voicebox** when the chord started. The transcript goes
|
||||
to the Captures tab so a dictation-into-Voicebox round-trip doesn't
|
||||
accidentally paste into the generate box.
|
||||
- **No text focus detected.** The transcript still lands in the Captures
|
||||
tab; copy it from there with one click.
|
||||
- **Accessibility permission not granted** on macOS. Same — Captures tab
|
||||
only.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
The paired audio + transcript archive every dictation lands in.
|
||||
</Card>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
The same local LLM powers per-profile compose and persona rewrite.
|
||||
</Card>
|
||||
<Card title="Transcription" href="/developer/transcription">
|
||||
Developer-level details on Whisper, Whisper Turbo, and the STT backend.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -1,23 +1,48 @@
|
||||
---
|
||||
title: "Introduction"
|
||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||
description: "Voicebox is the open-source, local-first AI voice studio — a free alternative to ElevenLabs and WisprFlow, running entirely on your machine."
|
||||
---
|
||||
|
||||
## What is Voicebox?
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is the **open-source, local-first AI voice studio**. It closes the
|
||||
voice I/O loop in both directions on one machine, with no cloud and no
|
||||
accounts:
|
||||
|
||||
- **Complete privacy** -- models and voice data stay on your machine
|
||||
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
|
||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** -- REST API for integrating voice synthesis into your own projects
|
||||
- **Native performance** -- built with Tauri (Rust), not Electron
|
||||
- **Runs everywhere** -- macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
|
||||
- **Humans talk** — hold a chord anywhere on your machine and your
|
||||
dictation lands as clean text in whatever text field you had focused
|
||||
- **Agents talk back** — any MCP-aware agent can call Voicebox to speak in
|
||||
one of your cloned voices
|
||||
- **Voices speak for themselves** — voice profiles can carry a personality
|
||||
that composes fresh lines or rewrites text before it's spoken
|
||||
|
||||
It's the free, local alternative to both ElevenLabs (voice cloning and TTS)
|
||||
and WisprFlow (voice dictation for agents and power users) — covering both
|
||||
sides of the same loop in one app, with a single model directory and LLM
|
||||
shared between input and output.
|
||||
|
||||
## What's in the app
|
||||
|
||||
- **Dictation** — global hotkey, push-to-talk and toggle modes, auto-paste
|
||||
into the focused field on macOS and Windows (see [Dictation](/overview/dictation))
|
||||
- **Captures tab** — paired audio + transcript archive, retranscribe,
|
||||
refine, play-as-voice, promote-to-sample (see [Captures](/overview/captures))
|
||||
- **Voice cloning** — 5 cloning engines covering 23 languages. Zero-shot
|
||||
cloning from a reference sample (see [Voice Cloning](/overview/voice-cloning))
|
||||
- **Preset voices** — 50+ curated voices via Kokoro and Qwen CustomVoice
|
||||
for when you don't want to clone (see [Preset Voices](/overview/preset-voices))
|
||||
- **Voice personalities** — optional free-form personality on any profile
|
||||
plus a compose button and persona-rewrite toggle powered by a local LLM (see
|
||||
[Voice Personalities](/overview/voice-personalities))
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus,
|
||||
compression, filters (Spotify's Pedalboard)
|
||||
- **Expressive speech** — paralinguistic tags like `[laugh]` and `[sigh]`
|
||||
via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** — auto-chunking with crossfade for long scripts
|
||||
- **Stories editor** — multi-track timeline for conversations and podcasts
|
||||
- **API-first** — REST + WebSocket API; MCP server for agent integrations
|
||||
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA / DirectML), Linux
|
||||
(ROCm / CPU), Intel Arc, Docker
|
||||
|
||||
## TTS Engines
|
||||
|
||||
@@ -30,9 +55,21 @@ Seven engines with different strengths, switchable per-generation:
|
||||
| **LuxTTS** | Cloned | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||
| **Chatterbox Multilingual** | Cloned | 23 | Broadest language coverage |
|
||||
| **Chatterbox Turbo** | Cloned | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model -- 700s+ coherent audio |
|
||||
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model — 700s+ coherent audio |
|
||||
| **Kokoro** | Preset (50 voices) | 9 | 82M parameters, CPU realtime, lowest VRAM of any engine |
|
||||
|
||||
## STT and local LLM
|
||||
|
||||
Voicebox also runs a full speech recognition and local LLM stack, shared
|
||||
between dictation, the Captures tab, and per-profile personality modes:
|
||||
|
||||
| Layer | Models |
|
||||
|---|---|
|
||||
| **STT** | Whisper Base / Small / Medium / Large / Turbo (PyTorch or MLX) |
|
||||
| **LLM** | Qwen3 0.6B / 1.7B / 4B (refinement + per-profile compose / persona-rewrite) |
|
||||
|
||||
No cloud fallback, no bring-your-own-API-key. Local is the product.
|
||||
|
||||
## GPU Support
|
||||
|
||||
| Platform | Backend | Notes |
|
||||
@@ -46,11 +83,13 @@ Seven engines with different strengths, switchable per-generation:
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Game development** -- generate dynamic dialogue for characters
|
||||
- **Content creation** -- produce podcasts and video voiceovers
|
||||
- **Accessibility** -- build text-to-speech tools for users who need them
|
||||
- **Voice assistants** -- create custom voice interfaces
|
||||
- **Production pipelines** -- automate voiceover workflows via the REST API
|
||||
- **Dictation for humans and agents** — speak instead of type, in any app
|
||||
- **Agent voice output** — any MCP-aware agent can speak in a cloned voice
|
||||
- **Game development** — generate dynamic dialogue for characters
|
||||
- **Content creation** — podcasts, video voiceovers, audiobooks
|
||||
- **Accessibility** — speech-to-text for any field, TTS with a voice you own
|
||||
- **Voice assistants** — custom voice interfaces without a cloud bill
|
||||
- **Production pipelines** — automate voice workflows via the REST API
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -61,8 +100,9 @@ Seven engines with different strengths, switchable per-generation:
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
|
||||
| STT | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Local LLM | Qwen3 0.6B / 1.7B / 4B (MLX or PyTorch) |
|
||||
| Effects | Pedalboard (Spotify) |
|
||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||
| Database | SQLite |
|
||||
| Audio | WaveSurfer.js, librosa |
|
||||
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: "MCP Server"
|
||||
description: "Let Claude Code, Cursor, Cline, or any MCP-aware agent speak in one of your cloned voices — locally, with no cloud."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox ships a built-in **Model Context Protocol** server so local AI
|
||||
agents can call your Voicebox install directly: speak text in a voice
|
||||
profile, transcribe audio, and list captures or profiles. The server runs
|
||||
inside the same process as the rest of Voicebox and is mounted at `/mcp`
|
||||
over Streamable HTTP.
|
||||
|
||||
Agent asks to speak → Voicebox plays audio on your speakers → an on-screen
|
||||
pill surfaces the voice name for the whole duration so you always see what's
|
||||
coming out of your machine.
|
||||
|
||||
<Callout type="info">
|
||||
MCP shipped in **0.5.0** alongside [Dictation](/overview/dictation) and
|
||||
[Voice Personalities](/overview/voice-personalities). The design goal is
|
||||
"local voice layer for every agent on your machine" — the same app that
|
||||
captures your voice can generate a response in any voice profile you've
|
||||
cloned.
|
||||
</Callout>
|
||||
|
||||
## Quick install
|
||||
|
||||
### Claude Code
|
||||
|
||||
```
|
||||
claude mcp add voicebox \
|
||||
--transport http \
|
||||
--url http://127.0.0.1:17493/mcp \
|
||||
--header "X-Voicebox-Client-Id: claude-code"
|
||||
```
|
||||
|
||||
### Cursor / Windsurf / VS Code MCP / any HTTP MCP client
|
||||
|
||||
Drop this into the client's MCP config (usually `.mcp.json` or a Settings UI):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"url": "http://127.0.0.1:17493/mcp",
|
||||
"headers": { "X-Voicebox-Client-Id": "cursor" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change `cursor` to whatever name you want the binding to show up as in
|
||||
Voicebox → Settings → MCP. The value is just an identifier for the
|
||||
per-client voice binding — not a secret, not a credential.
|
||||
|
||||
### Clients that only speak stdio
|
||||
|
||||
A stdio shim binary `voicebox-mcp` is bundled with the desktop app. Point
|
||||
the client at that binary's absolute path:
|
||||
|
||||
<Tabs items={["macOS", "Windows", "Linux"]}>
|
||||
<Tab value="macOS">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab value="Windows">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "C:\\Program Files\\Voicebox\\voicebox-mcp.exe",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab value="Linux">
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicebox": {
|
||||
"command": "/opt/voicebox/voicebox-mcp",
|
||||
"env": { "VOICEBOX_CLIENT_ID": "claude-desktop" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
The shim waits up to 30 seconds for the Voicebox backend to come up, then
|
||||
proxies JSON-RPC from stdio over Streamable HTTP. Voicebox must be running
|
||||
for the shim to connect.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Use |
|
||||
|---|---|
|
||||
| `voicebox.speak` | Speak text in a voice profile. Returns a `generation_id` to poll. |
|
||||
| `voicebox.transcribe` | Whisper transcription of base64 audio or an absolute local path. |
|
||||
| `voicebox.list_captures` | Recent captures with transcripts, paginated. |
|
||||
| `voicebox.list_profiles` | Available voice profiles (cloned + preset). |
|
||||
|
||||
### `voicebox.speak`
|
||||
|
||||
```ts
|
||||
voicebox.speak({
|
||||
text: "Deploy complete.",
|
||||
profile?: "Morgan", // name or id; falls back to per-client binding, then default
|
||||
engine?: "qwen", // qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro
|
||||
personality?: true, // rewrite via the profile's personality LLM before TTS; default comes from the per-client binding
|
||||
language?: "en",
|
||||
})
|
||||
```
|
||||
|
||||
Returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"generation_id": "…",
|
||||
"status": "generating",
|
||||
"profile": "Morgan",
|
||||
"source": "mcp",
|
||||
"poll_url": "/generate/<id>/status"
|
||||
}
|
||||
```
|
||||
|
||||
- **Plain TTS** — `personality: false` (or omitted + binding default is false). Text is spoken as-is.
|
||||
- **Persona mode** — `personality: true` and the profile must have a personality prompt set.
|
||||
The LLM rewrites the text in character before TTS. See [Voice Personalities](/overview/voice-personalities).
|
||||
|
||||
### `voicebox.transcribe`
|
||||
|
||||
```ts
|
||||
voicebox.transcribe({
|
||||
audio_base64?: "<base64>", // exactly one of these two
|
||||
audio_path?: "/absolute/path/to/file.wav",
|
||||
language?: "en",
|
||||
model?: "turbo", // base | small | medium | large | turbo
|
||||
})
|
||||
```
|
||||
|
||||
Returns `{ text, duration, language, model }`. 200 MB ceiling on either path.
|
||||
|
||||
### `voicebox.list_captures`
|
||||
|
||||
`{ limit?: 20, offset?: 0 }` → `{ captures: [...], total }`. `limit` is
|
||||
clamped to `1..=200`.
|
||||
|
||||
### `voicebox.list_profiles`
|
||||
|
||||
No args → `{ profiles: [{ id, name, voice_type, language, has_personality }] }`.
|
||||
|
||||
## Voice resolution
|
||||
|
||||
Every call to `voicebox.speak` (and `POST /speak`) resolves the voice profile
|
||||
in this order:
|
||||
|
||||
<Steps>
|
||||
<Step title="Explicit `profile` arg">
|
||||
Passed as a name (case-insensitive) or id. If the name/id doesn't match,
|
||||
the call errors — the server doesn't silently fall back.
|
||||
</Step>
|
||||
<Step title="Per-client binding">
|
||||
Looked up by the `X-Voicebox-Client-Id` header. Managed in
|
||||
**Voicebox → Settings → MCP**. Lets you pin Claude Code to Morgan,
|
||||
Cursor to Scarlett, etc.
|
||||
</Step>
|
||||
<Step title="Global default">
|
||||
`capture_settings.default_playback_voice_id` — same default voice the
|
||||
Captures tab's "Play as voice" action uses.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
If none of the three produce a profile the tool returns a helpful error
|
||||
pointing at Settings.
|
||||
|
||||
## Per-client bindings
|
||||
|
||||
Voicebox → Settings → MCP shows one row per `client_id` Voicebox has heard
|
||||
from, plus the config snippets you can copy into each agent. Each row
|
||||
carries:
|
||||
|
||||
| Field | Purpose |
|
||||
|---|---|
|
||||
| `label` | Display name in the Settings UI (e.g. "Claude Code"). |
|
||||
| `profile_id` | The voice this client uses when `profile` isn't passed. |
|
||||
| `default_engine` | Override the TTS engine for this client. |
|
||||
| `default_personality` | When true, `voicebox.speak` routes through the profile's personality LLM (rewrite) by default. |
|
||||
| `last_seen_at` | Last time the server saw a request from this client. |
|
||||
|
||||
`last_seen_at` is stamped automatically by middleware on every `/mcp/*`
|
||||
request — useful when you're not sure whether your config took.
|
||||
|
||||
## The speaking pill
|
||||
|
||||
Every agent-initiated speak surfaces the floating pill the same way
|
||||
[Dictation](/overview/dictation) does, in a new `Speaking` state showing the
|
||||
profile name and an elapsed timer. The pill is intentionally unmissable —
|
||||
silent background TTS is a trust hazard, so Voicebox always shows what's
|
||||
being spoken and in what voice.
|
||||
|
||||
Behind the scenes, the backend broadcasts `speak-start` and `speak-end`
|
||||
events on `GET /events/speak`, which `DictateWindow` subscribes to via SSE.
|
||||
The pill overrides the capture session when both would render — you can't
|
||||
hear two pills at once.
|
||||
|
||||
## Non-MCP REST surface
|
||||
|
||||
`POST /speak` is a thin wrapper on the same code path for callers that
|
||||
don't speak MCP — shell scripts, ACP, A2A, GitHub Actions, whatever.
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:17493/speak \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'X-Voicebox-Client-Id: ci' \
|
||||
-d '{"text":"Build complete.","profile":"Morgan"}'
|
||||
```
|
||||
|
||||
Body fields match the MCP tool: `text`, optional `profile`, `engine`,
|
||||
`personality`, `language`. Returns a `GenerationResponse` — the same shape as
|
||||
`POST /generate`.
|
||||
|
||||
## Debugging
|
||||
|
||||
Use the MCP Inspector to poke tools directly without plumbing through an
|
||||
agent:
|
||||
|
||||
```
|
||||
npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp
|
||||
```
|
||||
|
||||
Start with `voicebox.list_profiles` to confirm wiring, then
|
||||
`voicebox.speak` for end-to-end — you should hear audio and see the
|
||||
generation land in the Captures tab.
|
||||
|
||||
<Callout type="info">
|
||||
If an agent can't reach the server, the first thing to check is that
|
||||
Voicebox is running — the backend only listens while the desktop app is
|
||||
open. The stdio shim surfaces this as a JSON-RPC error on the client
|
||||
side after its 30-second health-wait window elapses.
|
||||
</Callout>
|
||||
|
||||
## Security
|
||||
|
||||
- **Localhost only.** The server binds to `127.0.0.1`. If you ever point
|
||||
Voicebox at a non-loopback interface (e.g. remote-mode over a trusted
|
||||
network), add a bearer token — it's on the roadmap but not in 0.5.0.
|
||||
- **No auth today.** Any process that can connect to your loopback can
|
||||
call MCP. That's the same trust boundary as the rest of Voicebox's REST
|
||||
API and is appropriate for a single-user local tool.
|
||||
- **`audio_path` reads are unrestricted** against the same trust
|
||||
boundary. If you're scripting against a shared host, prefer
|
||||
`audio_base64` so you don't have to think about path sandboxing.
|
||||
- **Voice cloning consent applies.** See [Voice Cloning](/overview/voice-cloning#limitations)
|
||||
— an agent being able to call `voicebox.speak` in someone's voice
|
||||
doesn't change the ethics of whose voices you clone.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- **Transport:** Streamable HTTP (Nov-2025 MCP spec, post-SSE). Claude
|
||||
Code, Cursor, Windsurf, and VS Code MCP extensions all support it.
|
||||
- **Package naming:** the backend package is `backend/mcp_server/`, not
|
||||
`mcp`, to avoid shadowing the PyPI `mcp` package FastMCP imports
|
||||
internally.
|
||||
- **Dependencies:** `fastmcp>=3.0,<4.0`, `sse-starlette>=2.0`.
|
||||
- **Lifespan:** mounting FastMCP requires the `lifespan=` kwarg on
|
||||
`FastAPI()` — the startup/shutdown event decorators are incompatible
|
||||
with FastMCP's Streamable HTTP session manager. The Voicebox app.py
|
||||
composes both into one async context manager.
|
||||
|
||||
For the full developer-facing tour of the code layout, see
|
||||
`backend/mcp_server/README.md` in the repo.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Voice Personalities" href="/overview/voice-personalities">
|
||||
Persona mode (`personality: true`) for agents that should
|
||||
transform text in-character before speaking.
|
||||
</Card>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
The pill that surfaces agent speech is the same one that surfaces
|
||||
your dictations — one mental model for both directions of the loop.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
Every agent-initiated speak lands in the Captures tab with its
|
||||
generated audio — replay, download, repurpose.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -7,8 +7,12 @@
|
||||
"docker",
|
||||
"quick-start",
|
||||
"gpu-acceleration",
|
||||
"dictation",
|
||||
"captures",
|
||||
"voice-cloning",
|
||||
"preset-voices",
|
||||
"voice-personalities",
|
||||
"mcp-server",
|
||||
"stories-editor",
|
||||
"recording-transcription",
|
||||
"generation-history",
|
||||
|
||||
@@ -1,64 +1,106 @@
|
||||
---
|
||||
title: "Recording & Transcription"
|
||||
description: "Record audio and transcribe speech with Whisper"
|
||||
description: "A map of the three places you can record and transcribe audio in Voicebox — dictation, captures, and voice-profile samples."
|
||||
---
|
||||
|
||||
## Recording
|
||||
## Overview
|
||||
|
||||
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
|
||||
Voicebox records and transcribes audio in three different contexts, each
|
||||
feeding a different surface in the app. This page is a map; follow the links
|
||||
for the detail.
|
||||
|
||||
### Features
|
||||
| Goal | Where | Docs |
|
||||
|---|---|---|
|
||||
| Speak and have your words land in another app | Global hotkey → Captures tab + auto-paste | [Dictation](/overview/dictation) |
|
||||
| Record a thought, a meeting, or a voice memo inside Voicebox | Captures tab | [Captures](/overview/captures) |
|
||||
| Record a clip to clone a voice from | Voices tab → profile samples | [Creating Voice Profiles](/overview/creating-voice-profiles) |
|
||||
|
||||
- **Microphone input** - Record from any audio input device
|
||||
- **System audio capture** - Record desktop audio (macOS/Windows)
|
||||
- **Waveform visualization** - See audio levels in real-time
|
||||
- **Multiple formats** - Export as WAV, MP3, or M4A
|
||||
All three paths share the same STT backend — it's the surrounding workflow
|
||||
that differs.
|
||||
|
||||
### How to Record
|
||||
## Dictation
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Input">
|
||||
Choose your microphone or system audio
|
||||
</Step>
|
||||
<Step title="Start Recording">
|
||||
Click the record button and speak clearly
|
||||
</Step>
|
||||
<Step title="Stop & Save">
|
||||
Click stop when finished
|
||||
</Step>
|
||||
<Step title="Use or Export">
|
||||
Use as voice sample or export to file
|
||||
</Step>
|
||||
</Steps>
|
||||
The 0.5.0 headline feature. Hold a chord anywhere on your machine, speak,
|
||||
release. The transcript lands in whatever text field you had focused,
|
||||
cleaned up by a local LLM if auto-refine is on. Captures accumulate in the
|
||||
Captures tab for later replay or re-transcription.
|
||||
|
||||
## Transcription
|
||||
Covered end-to-end in [Dictation](/overview/dictation).
|
||||
|
||||
Automatic speech-to-text powered by OpenAI's Whisper model.
|
||||
## Captures tab
|
||||
|
||||
### Features
|
||||
When you don't need to paste into another app — you just want a clean
|
||||
transcript of some audio — the Captures tab is the home. Record in-app,
|
||||
drop in a file (`.wav`, `.mp3`, `.m4a`, `.webm`, `.opus`, `.flac`), or dig
|
||||
through dictations that already landed there. Every capture keeps its
|
||||
original audio, can be retranscribed with a different model, and can be
|
||||
played back through any voice profile you have.
|
||||
|
||||
- **High accuracy** - Industry-leading speech recognition
|
||||
- **Multiple languages** - Supports 50+ languages
|
||||
- **Automatic detection** - Language auto-detection
|
||||
- **Timestamps** - Word-level timing information
|
||||
Covered in [Captures](/overview/captures).
|
||||
|
||||
### How to Transcribe
|
||||
## Voice profile samples
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Audio">
|
||||
Choose a recording or upload an audio file
|
||||
</Step>
|
||||
<Step title="Choose Language">
|
||||
Select language or use auto-detect
|
||||
</Step>
|
||||
<Step title="Transcribe">
|
||||
Click transcribe and wait for processing
|
||||
</Step>
|
||||
<Step title="Review & Export">
|
||||
Review text and export as needed
|
||||
</Step>
|
||||
</Steps>
|
||||
A separate flow, in the Voices tab. When you're creating a profile from an
|
||||
audio clip, the sample is what the cloning engine actually learns from —
|
||||
the `reference_text` on a sample must match the audio *verbatim*, which is
|
||||
why samples are a different data model from captures.
|
||||
|
||||
You can promote a capture to a sample from the Captures tab's Send-to menu
|
||||
("Use as voice sample…"), which opens a reference-text confirm dialog so
|
||||
you can correct the last ~10% of transcript accuracy before saving.
|
||||
|
||||
Covered in [Creating Voice Profiles](/overview/creating-voice-profiles).
|
||||
|
||||
## Transcription models
|
||||
|
||||
All three paths share the same Whisper models. Pick a default in
|
||||
**Settings → Captures → Transcription**; override per capture if you need
|
||||
to.
|
||||
|
||||
| Model | Size | When to pick it |
|
||||
|---|---|---|
|
||||
| Whisper Base | ~300 MB | Fast. Default. Good for clean speech. |
|
||||
| Whisper Small | ~500 MB | Better quality, still fast. |
|
||||
| Whisper Medium | ~1.5 GB | High quality. |
|
||||
| Whisper Large | ~3 GB | Best quality, slow on CPU. |
|
||||
| Whisper Turbo | ~1.5 GB | Large-tier quality, ~5× faster than Large. |
|
||||
|
||||
On Apple Silicon the model runs through **MLX-Whisper** (~8× faster than
|
||||
PyTorch). Everywhere else it runs through PyTorch `transformers`. The
|
||||
backend picks the right one — you don't configure it.
|
||||
|
||||
<Callout type="info">
|
||||
Transcription is useful for creating voice samples from existing audio or generating subtitles.
|
||||
For noisy clips, prefer **Turbo** or **Large**. Base can hallucinate on
|
||||
hard inputs — most famously the "thanks for watching" loop. Voicebox
|
||||
strips those loops deterministically before LLM refinement runs, so a
|
||||
capture can be cleanly re-refined even if the raw transcript has them.
|
||||
</Callout>
|
||||
|
||||
## Language
|
||||
|
||||
You can pass a language hint for short clips (under ~5 seconds) where
|
||||
Whisper's auto-detect is unreliable. Set a default language lock in
|
||||
**Settings → Captures → Transcription → Language**, or override per capture.
|
||||
|
||||
## Transcription API
|
||||
|
||||
Developer-level detail on the STT backend, model loading, preprocessing, and
|
||||
the `/transcribe` endpoint lives in the
|
||||
[Transcription developer guide](/developer/transcription). The Captures
|
||||
pipeline also exposes `/captures` as a higher-level endpoint that wraps
|
||||
STT + archival + optional refinement in one call — see
|
||||
[Captures](/overview/captures#api-surface).
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
Hold a chord anywhere on your machine, speak, release.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
The paired audio + transcript archive.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Record or upload samples for voice cloning.
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
title: "Voice Personalities"
|
||||
description: "Attach a personality to a voice profile, compose fresh in-character lines, and rewrite input text in their voice — all powered by a local LLM."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A **personality** is an optional free-form description attached to a voice
|
||||
profile — who this voice is, how they speak, what they care about. Set one
|
||||
and two new controls appear next to the generate button, both powered by a
|
||||
bundled Qwen3 LLM running entirely locally:
|
||||
|
||||
- **Compose** — drop a fresh in-character line into the textarea. Click
|
||||
again for a different take.
|
||||
- **Speak in character** — a toggle that rewrites your input text in the
|
||||
character's voice before TTS, preserving every idea.
|
||||
|
||||
The LLM produces the text. The voice profile speaks it. No cloud round-trip,
|
||||
no external API — the whole loop runs on your hardware.
|
||||
|
||||
<Callout type="info">
|
||||
Personalities shipped in **0.5.0**. The same local LLM doubles as the
|
||||
refinement model for [Dictation](/overview/dictation) — one LLM in the app,
|
||||
not two, sharing one model cache and one GPU-memory footprint.
|
||||
</Callout>
|
||||
|
||||
## Setting a personality
|
||||
|
||||
Open a voice profile's edit view. The **Personality** field is free-form text
|
||||
up to **2,000 characters**. Describe the voice however helps you — past
|
||||
lines they'd say, speech patterns, tone, boundaries.
|
||||
|
||||
Good descriptions tend to include:
|
||||
|
||||
- A one-line identity (who they are)
|
||||
- Speech patterns (rhythm, vocabulary, what they avoid)
|
||||
- Representative phrases — example lines show the LLM the target tone
|
||||
better than adjectives
|
||||
- What the character *wouldn't* do (they don't explain, they don't
|
||||
apologize, they refuse to break character, etc.)
|
||||
|
||||
You can set a personality on any voice profile type — cloned or preset. The
|
||||
three modes work identically regardless of engine.
|
||||
|
||||
## The two actions
|
||||
|
||||
Each action is tuned for a specific job and the LLM temperature is adjusted
|
||||
to match.
|
||||
|
||||
### Compose
|
||||
|
||||
Generate a fresh utterance in the character's voice, with no seed text.
|
||||
Click the shuffle button to drop a line straight into the generate
|
||||
textarea; click again for a different take.
|
||||
|
||||
- **When to use:** prototyping, sampling a character's voice, brainstorming
|
||||
a line without typing one first
|
||||
- **Temperature:** hot — variety is the point
|
||||
- **Typical output:** a short, punchy line that fits the character's
|
||||
register
|
||||
|
||||
### Speak in character (rewrite)
|
||||
|
||||
Flip the persona toggle and whatever you type (or dictate) gets rewritten in
|
||||
the character's voice before TTS — every idea preserved, only the phrasing
|
||||
changes. High-fidelity mode: the content doesn't change, only the voice does.
|
||||
|
||||
- **When to use:** turning a dictated memo into in-character speech; lifting
|
||||
a plain-English script into a specific voice without editing by hand
|
||||
- **Temperature:** cold — faithfulness wins
|
||||
- **Typical output:** same ideas, same order, different phrasing and cadence
|
||||
|
||||
## Speech-only framing
|
||||
|
||||
Both modes enforce **speech-only** output. The LLM is prompted to
|
||||
produce things a person would actually say out loud — no narration, no
|
||||
action tags (`*sighs*`, `[laughs]`), no meta-commentary, no markdown
|
||||
formatting, no stage directions.
|
||||
|
||||
This is deliberate: the output is going straight into TTS, and anything that
|
||||
isn't speakable ends up either ignored or read literally. The speech-only
|
||||
framing also makes the output land cleanly inside dialogue, so you can drop
|
||||
a Respond result straight into a Story.
|
||||
|
||||
## The local LLM
|
||||
|
||||
The bundled LLM is **Qwen3**, available in three sizes:
|
||||
|
||||
| Model | Download size | Best for |
|
||||
|---|---|---|
|
||||
| Qwen3 0.6B | ~400 MB | Default. Very fast, good for casual use. |
|
||||
| Qwen3 1.7B | ~1.1 GB | Sweet spot for character personalities with specific phrasing. |
|
||||
| Qwen3 4B | ~2.5 GB | Full quality. Slowest. Useful for very particular tone. |
|
||||
|
||||
The model runs through the same backend split Voicebox already uses for TTS
|
||||
— **MLX** (4-bit community quants) on Apple Silicon, **PyTorch** (transformers
|
||||
`AutoModelForCausalLM`) everywhere else. Downloads go through the same cache
|
||||
and model-management UI as TTS models.
|
||||
|
||||
Pick a size in **Settings → Captures → Refinement → Refinement model** — the
|
||||
personality modes reuse it. If you switch models, both refinement and
|
||||
personality output pick up the change on the next call.
|
||||
|
||||
## Using the controls
|
||||
|
||||
Both controls appear on the floating generate box when the selected profile
|
||||
has a personality set.
|
||||
|
||||
<Steps>
|
||||
<Step title="Compose — shuffle a line">
|
||||
Click the shuffle button. The LLM runs and the result fills the generate
|
||||
textarea. Edit if you want, then hit generate.
|
||||
</Step>
|
||||
<Step title="Speak in character — toggle persona rewrite">
|
||||
Type (or dictate) what you want said. Flip the wand toggle on. Hit
|
||||
generate — Voicebox runs the text through the personality LLM first,
|
||||
then TTS speaks the rewritten version. Leave the toggle off for plain
|
||||
TTS.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
Compose always gives you something different on re-click. The persona
|
||||
toggle, on the other hand, is a mode — it applies to every generate call
|
||||
until you flip it back off.
|
||||
|
||||
## Use cases
|
||||
|
||||
- **Agents that speak in a voice you own.** Combine the persona toggle with
|
||||
the built-in [MCP Server](/overview/mcp-server) so Claude Code, Cursor,
|
||||
Cline, or any MCP-aware agent can talk back through a profile with a
|
||||
personality. The agent calls `voicebox.speak({ text, profile, personality:
|
||||
true })` and Voicebox rewrites the text in character before speaking.
|
||||
- **Interactive characters.** Games, narrative tools, accessibility
|
||||
experiences. A character with a personality description plus a cloned
|
||||
voice becomes a reusable prop.
|
||||
- **Accessibility.** People who can't speak in their original voice can
|
||||
keep a personality description of how they used to sound and use the
|
||||
rewrite toggle to turn typed input into in-character speech.
|
||||
- **Creative drafting.** Write a plain outline, flip the persona toggle,
|
||||
generate line-by-line into the character's voice, drop the audio into a
|
||||
Story.
|
||||
|
||||
## API surface
|
||||
|
||||
Personalities are accessible via REST:
|
||||
|
||||
| Method | Endpoint | Body |
|
||||
|---|---|---|
|
||||
| `PUT` | `/profiles/{id}` | Include a `personality` field up to 2,000 chars to set it. |
|
||||
| `POST` | `/profiles/{id}/compose` | No body. Returns a fresh in-character utterance as text. |
|
||||
| `POST` | `/generate` | Include `personality: true` to run input text through the personality LLM before TTS. Same for `POST /speak`. |
|
||||
|
||||
`POST /generate` with `personality: true` is the same primitive MCP's
|
||||
`voicebox.speak` tool uses when you pass `personality: true`. Scripts and
|
||||
agents can use it directly.
|
||||
|
||||
## Limits and gotchas
|
||||
|
||||
- **The personality is a prompt, not a fine-tune.** The LLM will sometimes
|
||||
drift out of character, especially on Compose at high temperature. Click
|
||||
again for another take.
|
||||
- **Long personalities are not always better.** 2,000 chars is a ceiling,
|
||||
not a goal. A sharp 300-char description with two example lines
|
||||
typically outperforms a long one.
|
||||
- **Speech-only framing is enforced, but not bulletproof.** Very large
|
||||
prompts or unusual inputs can sneak an action tag through. If you see
|
||||
`[laughs]` in TTS output, it's usually a personality-field hint the
|
||||
model anchored onto — remove it from the description.
|
||||
- **Rewrite is stricter than Respond.** If the output is changing your
|
||||
meaning, you probably want Respond (or a wholesale Compose with context
|
||||
in the input), not Rewrite.
|
||||
|
||||
## Next steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Dictation" href="/overview/dictation">
|
||||
Dictate the input for Rewrite or Respond from anywhere on your machine.
|
||||
</Card>
|
||||
<Card title="Captures" href="/overview/captures">
|
||||
Captures feed personalities naturally — dictate a memo, rewrite it in
|
||||
a character voice, generate speech.
|
||||
</Card>
|
||||
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
|
||||
Add a personality to an existing profile.
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -0,0 +1,102 @@
|
||||
# macOS Notarization & Gatekeeper
|
||||
|
||||
**Status:** Diagnosis — Homebrew Cask CI rejects v0.4.5 on macOS 15 (Sequoia); fix pending
|
||||
**Touches:** `.github/workflows/release.yml`, Tauri bundler config, sidecar signing
|
||||
**Last reviewed:** 2026-04-24
|
||||
|
||||
## Context
|
||||
|
||||
Homebrew Cask PR [#260314](https://github.com/Homebrew/homebrew-cask/pull/260314) adds `brew install --cask voicebox`. CI is green on macOS 14 and macOS 26 (arm + intel) but fails on macOS 15 (arm + intel). The 0.4.3 release added DMG-level stapling to address this, and it didn't move CI — 0.4.5 still fails. A maintainer reproduced the failure in a fresh Sequoia VM.
|
||||
|
||||
This document is the working diagnosis plus the ordered fix plan.
|
||||
|
||||
## What the failing check actually does
|
||||
|
||||
The failing step is `brew audit --cask --online --signing --new voicebox`, not `brew install`. `brew install` succeeds end-to-end in CI (the log shows `Uninstalling Cask voicebox` after the install phase). The `--signing` audit:
|
||||
|
||||
1. Downloads the cask's `url`
|
||||
2. Mounts the DMG
|
||||
3. Runs `spctl --assess -t open --context context:primary-signature` against the `.app` inside
|
||||
|
||||
That policy tests the first-launch Gatekeeper path on the extracted bundle. It reads the `.app`'s own code signature and notarization ticket — the DMG wrapper is not involved. The staple added in 0.4.3 covers the DMG, so it has no effect on this audit.
|
||||
|
||||
## Why Sequoia and not Sonoma
|
||||
|
||||
`spctl -t open` on macOS 15 enforces checks that 14 tolerated:
|
||||
|
||||
- Secure timestamp required on hardened-runtime signatures. Untimestamped signatures pass on 14, fail on 15.
|
||||
- Deep verification of nested Mach-Os. If any embedded `.dylib` or helper binary carries an ad-hoc signature (or a signature with a different Team ID), 15 rejects the whole bundle; 14 often accepted it.
|
||||
- Hardened runtime must be set on every nested executable, not just the top-level app binary. Entitlements declared on the outer app do not propagate.
|
||||
|
||||
Local dev machines pass `spctl` because the first-party developer context and cached notarization tickets mask these failures. A fresh Sequoia VM with no prior trust state does not.
|
||||
|
||||
## Where the gap is likely to be
|
||||
|
||||
Voicebox ships PyInstaller sidecars declared in `tauri.conf.json` under `externalBin`:
|
||||
|
||||
- **0.4.x:** `voicebox-server` only (single `--onefile` Mach-O on macOS)
|
||||
- **0.5.0+:** `voicebox-server` and `voicebox-mcp` (`voicebox-mcp` is new in 0.5.0)
|
||||
|
||||
Tauri's bundler signs each `externalBin` with the configured identity but does not apply `--options=runtime` or `--timestamp` automatically, and does not merge the outer app's entitlements into the sidecar signature. The outer `Voicebox` binary is correctly signed with hardened runtime + `disable-library-validation`; the sidecars likely are not.
|
||||
|
||||
Order of likelihood:
|
||||
|
||||
1. Sidecar `voicebox-server` lacks hardened runtime or a secure timestamp in its signature.
|
||||
2. The sidecar inherits the identity but was signed before tauri-action's final notarization pass, so the notarization ticket doesn't actually cover it.
|
||||
3. Something inside the sidecar's PyInstaller archive unpacks to a `.dylib` at runtime that Gatekeeper inspects during assessment.
|
||||
|
||||
The 0.5.0 fix must cover both sidecars.
|
||||
|
||||
## Diagnostic commands
|
||||
|
||||
Run against a freshly downloaded release DMG (not a dev build, and from a machine that has never opened the app before):
|
||||
|
||||
```
|
||||
hdiutil attach Voicebox_0.4.5_aarch64.dmg
|
||||
xcrun stapler validate "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
spctl -a -vvv -t open --context context:primary-signature "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
codesign --verify --deep --strict --verbose=2 "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
codesign -dv --verbose=4 "/Volumes/Voicebox 0.4.5/Voicebox.app/Contents/MacOS/voicebox-server"
|
||||
```
|
||||
|
||||
The last command is the tell — look for `flags=0x10000(runtime)` and a `Timestamp=` line. If either is missing, the sidecar is the failure.
|
||||
|
||||
`spctl -t install` (what 0.4.3 verified with) is a different policy and can pass while `-t open` fails — any future verification should use `-t open --context context:primary-signature` to match what Homebrew's audit runs.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1 — Confirm the failure mode
|
||||
|
||||
Pull the 0.4.5 DMG on a fresh Sequoia environment or a VM snapshot with no trust state. Run the diagnostic block above. Record the exact failing command and its CSSMERR / rejection reason. This disambiguates between the three hypotheses before we change the workflow.
|
||||
|
||||
### Phase 2 — Sign sidecars explicitly in the release workflow
|
||||
|
||||
Between tauri-action's build step and the DMG-notarization step already in `release.yml`, add a step that re-signs every `externalBin` present under `Voicebox.app/Contents/MacOS/` with:
|
||||
|
||||
- `--options=runtime` (hardened runtime)
|
||||
- `--timestamp` (secure timestamp)
|
||||
- `--entitlements` pointing at `Entitlements.plist` or a sidecar-specific subset
|
||||
- The same `APPLE_SIGNING_IDENTITY` the outer app uses
|
||||
|
||||
Re-sign the outer `.app` afterward so its seal covers the updated nested signatures.
|
||||
|
||||
Covers `voicebox-server` on 0.4.x and both sidecars from 0.5.0 forward.
|
||||
|
||||
### Phase 3 — Re-notarize and staple the `.app`
|
||||
|
||||
After sidecars are re-signed the outer bundle's notarization ticket is stale. Submit the `.app` (zipped) to `notarytool`, wait, then `xcrun stapler staple Voicebox.app`. This puts the ticket directly on the `.app` so the `spctl -t open` audit passes without any online ticket lookup.
|
||||
|
||||
Then rebuild the DMG from the stapled `.app` and keep the existing DMG-level notarize/staple step — it still helps Finder drag-install.
|
||||
|
||||
### Phase 4 — CI verification gate in the release workflow
|
||||
|
||||
Before upload, run the same four diagnostic commands against the built artifact inside the workflow. If any fail, fail the release job rather than shipping a DMG that Homebrew (and Sequoia Finder users) will reject. This is the check that would have caught the 0.4.3 and 0.4.5 attempts before they cost PR review cycles.
|
||||
|
||||
### Phase 5 — Re-request Homebrew CI
|
||||
|
||||
Once a tagged release passes Phase 4 locally, push a cask update to #260314. Expect `test voicebox (macos-15, arm)` and `test voicebox (macos-15-intel, intel)` to go green.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does tauri-action v0.6 pass `APPLE_API_KEY_PATH` to the bundler's notarize path, or does it rely on the `~/.appstoreconnect/private_keys/AuthKey_*.p8` auto-discovery the staple step already sets up? If the former isn't working, tauri may be signing but never notarizing the `.app`, which would make the ticket absent entirely rather than stale. Worth a `grep -i notariz` on a full release job log.
|
||||
- If Phase 2 resolves the macOS 15 failure, revisit whether the 0.4.3 DMG staple step is still needed. It's cheap to keep and helps the Finder-open case, so default to leaving it.
|
||||
@@ -0,0 +1,347 @@
|
||||
# MCP Server — Voicebox Speed Run
|
||||
|
||||
**Status:** v1 shipped — HTTP transport, all 4 tools, per-client bindings, `POST /speak`, stdio shim (binary built, bundled into Tauri sidecar), Settings UI, speak-pill via SSE with Rust-side `dictate:show` handler so agent-initiated speech surfaces the pill on screen. `cargo check` clean, `tsc` clean, full Inspector round-trip verified.
|
||||
**Last reviewed:** 2026-04-23
|
||||
|
||||
## Status
|
||||
|
||||
### Shipped (backend)
|
||||
- **`fastmcp` + `sse-starlette`** pinned in `backend/requirements.txt`.
|
||||
- **`backend/mcp_server/`** package with `server.py`, `tools.py`, `context.py`, `resolve.py`, `events.py`, `README.md`. Named `mcp_server` (not `mcp`) to sidestep a shadowing conflict with the installed `mcp` PyPI package that FastMCP imports internally.
|
||||
- **Streamable HTTP mount at `/mcp`** via FastMCP's `http_app(transport='http')`. Sub-app lifespan composed with Voicebox's own startup/shutdown through an `@asynccontextmanager lifespan=` in `backend/app.py` (migrated away from the deprecated `@app.on_event` handlers).
|
||||
- **Four MCP tools**, dot-named to match the landing and ecosystem convention:
|
||||
- `voicebox.speak(text, profile?, engine?, personality?, language?)`
|
||||
- `voicebox.transcribe(audio_base64?, audio_path?, language?, model?)`
|
||||
- `voicebox.list_captures(limit, offset)`
|
||||
- `voicebox.list_profiles()`
|
||||
- **`ClientIdMiddleware`** pulls `X-Voicebox-Client-Id` into a `ContextVar` on every `/mcp*` request; auto-stamps `MCPClientBinding.last_seen_at`, auto-creating the row if the client is new.
|
||||
- **Profile resolution precedence** `explicit → per-client binding → capture_settings.default_playback_voice_id → error`. `services/profiles.get_profile_orm_by_name_or_id()` lets agents pass a voice by name ("Morgan") instead of UUID.
|
||||
- **`MCPClientBinding` table** (new) via `Base.metadata.create_all` — no migration needed.
|
||||
- **Bindings REST:** `GET|PUT /mcp/bindings`, `DELETE /mcp/bindings/{client_id}`.
|
||||
- **`POST /speak`** REST wrapper for non-MCP callers (shell / ACP / A2A). Same `resolve_profile` precedence, same code path as the MCP tool.
|
||||
- **Stdio shim** at `backend/mcp_shim/__main__.py` — ~200 lines of `httpx` proxy; reads env (`VOICEBOX_PORT`, `VOICEBOX_HOST`, `VOICEBOX_CLIENT_ID`), waits for `/health`, then streams JSON-RPC ↔ SSE. Rolled our own after the `mcp` SDK's session-management helpers mis-shook-hands. Smoke-tested: `initialize`, `tools/list`, and `tools/call` all round-trip cleanly.
|
||||
- **Pill SSE:** `GET /events/speak` (`sse-starlette`) emits `speak-start` from the MCP tool and `POST /speak`, `speak-end` from `services/generation.run_generation`'s finally block.
|
||||
- **PyInstaller:**
|
||||
- `backend/build_binary.py` `--shim` flag builds a minimal `voicebox-mcp` binary (torch/transformers/mlx/etc. explicitly excluded, target <20 MB).
|
||||
- The main server spec picks up `fastmcp`, `mcp`, `sse_starlette`, and `backend.mcp_server.*` via `--collect-all` / `--hidden-import`.
|
||||
- **`backend/mcp_server/README.md`** quickstart (Inspector, `.mcp.json` snippets, tool reference).
|
||||
|
||||
### Shipped (frontend)
|
||||
- **`Settings → MCP`** page (`app/src/components/ServerTab/MCPPage.tsx`):
|
||||
- Three copy-paste snippets auto-filled with the detected `serverUrl`: HTTP (recommended), Claude Code CLI one-liner, stdio fallback.
|
||||
- Default voice picker (bound to `capture_settings.default_playback_voice_id`, shared with Captures-tab "Play as voice").
|
||||
- Per-client bindings table with inline profile picker, remove button, and a connection-status indicator that refreshes every 10 s.
|
||||
- Add-binding form with client_id / label / profile dropdown.
|
||||
- **`useMCPBindings`** TanStack hook (optimistic delete, invalidate on upsert).
|
||||
- **`useSpeakEvents`** hook — auto-reconnecting `EventSource('/events/speak')`, tracks the active generation_id, exposes an elapsed-ms timer that ticks so the pill's clock advances.
|
||||
- **`CapturePill`** has a new `'speaking'` state + "Speaking" label + playing-bars mode.
|
||||
- **`DictateWindow`** subscribes to speak events and overrides `pillState` when an agent is speaking. Emits `dictate:show` on speak-start so the Rust side can surface the pill window.
|
||||
- Router + `ServerTab` tab bar wired to `/settings/mcp`.
|
||||
|
||||
### Shipped (native shell)
|
||||
- **`tauri.conf.json`** — `voicebox-mcp` added to `externalBin` (alongside `voicebox-server`).
|
||||
- **`dictate:show` listener** in `tauri/src-tauri/src/main.rs` — invokes a new `show_dictate_window(app_handle)` helper that mirrors the hotkey-monitor's position+show logic (undo click-through, reposition to top-center of the current monitor, show). Agent-initiated speech now pops the pill visible on screen.
|
||||
|
||||
### Validated end-to-end (this session, via curl)
|
||||
- `/mcp/` init → `tools/list` → `tools/call voicebox.speak` → actual audio plays (Jarvis, 1.68 s).
|
||||
- `POST /speak` with `X-Voicebox-Client-Id: claude-code` resolves to the bound Jarvis profile without passing `profile`.
|
||||
- `/events/speak` emits `ready`, `speak-start`, `speak-end` in order, generation_id threads through both.
|
||||
- Stdio shim: `echo {…} | python -m backend.mcp_shim` returns valid JSON-RPC for all 4 methods.
|
||||
- `last_seen_at` auto-stamps on first call; binding row auto-creates.
|
||||
- Frontend `tsc --noEmit`: clean.
|
||||
- `cargo check` on the Tauri crate: clean.
|
||||
|
||||
### Outstanding (must-do before release)
|
||||
- **CI build for shim on Windows/Linux** — `python backend/build_binary.py --shim` is wired up and built cleanly for `aarch64-apple-darwin` (18 MB, installed at `tauri/src-tauri/binaries/voicebox-mcp-aarch64-apple-darwin`, Tauri `cargo check` green). The Windows and Linux triples (`x86_64-pc-windows-msvc`, `x86_64-unknown-linux-gnu`) need the same build in their respective CI runners and artifacts dropped alongside the macOS binary.
|
||||
- **Windows/Linux paths in the stdio snippet** — the Settings page hardcodes the macOS path (`/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp`). Needs a per-OS switch (`%LOCALAPPDATA%\Programs\Voicebox\voicebox-mcp.exe`, Linux bundled-path), ideally with the Tauri shell resolving its own app path at runtime and injecting it into the snippet.
|
||||
|
||||
### Nice-to-have (follow-up passes)
|
||||
- **One-click install buttons** — write/merge into `~/.claude/settings.json`, `~/.cursor/mcp.json`, etc. via a Tauri command. Copy-paste works today; this is pure ergonomics.
|
||||
- **`.mcpb` desktop extension** for Claude Desktop (single file, double-click to install). Claude Desktop-only, so lower priority than the agent-harness crowd.
|
||||
- **Refactor the hotkey_monitor.rs show-logic** to call `show_dictate_window()` instead of duplicating the position+show block. Skipped at ship to avoid regressing the well-tested chord path.
|
||||
- **Source attribution on `Generation.source`** — currently `"manual" | "personality_speak"`; adding `"mcp"` / `"rest"` would let the Captures tab filter by MCP-originated rows.
|
||||
|
||||
## Context
|
||||
|
||||
Voicebox already ships the I/O surface (Captures, Generate, personality-driven `/profiles/{id}/speak`), but local AI agents can't reach any of it. This plan adds a Model Context Protocol server so Claude Code / Cursor / Cline can call `voicebox.speak`, `voicebox.transcribe`, `voicebox.list_captures`, and `voicebox.list_profiles` — turning Voicebox into the local voice layer for every agent on the user's machine (Phase 5 of `docs/plans/VOICE_IO.md`).
|
||||
|
||||
The shortest path to "Claude Code speaks in a cloned voice": mount **FastMCP** inside the existing FastAPI/uvicorn process at `/mcp` (Streamable HTTP), and users install it as a URL (`{"url": "http://127.0.0.1:17493/mcp"}`) — the ecosystem-idiomatic shape for a long-running local service. Per-client voice binding via a new `mcp_client_bindings` table + Settings UI, resolved from an `X-Voicebox-Client-Id` header. A **stdio shim binary** `voicebox-mcp` is bundled as a fallback sidecar for clients that can't speak HTTP MCP. A public `POST /speak` REST wrapper covers non-MCP callers (shell scripts, ACP, A2A). A `speaking` pill state gives agent-initiated audio visibility — trust-critical, non-negotiable.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Claude Code / Cursor / Windsurf / VS Code MCP
|
||||
│
|
||||
├─ HTTP (primary) ────────────────────┐
|
||||
│ {"url": ".../mcp"} │
|
||||
│ │
|
||||
└─ stdio (fallback) ───────────────▶ [voicebox-mcp shim binary]
|
||||
{"command": "/abs/path/voicebox-mcp"} (absolute path;
|
||||
│ Settings page
|
||||
│ copies it for you)
|
||||
▼
|
||||
uvicorn + FastAPI (port 17493)
|
||||
├─ /mcp (FastMCP, Streamable HTTP)
|
||||
└─ /speak (REST wrapper for non-MCP callers)
|
||||
└─ tools call existing services
|
||||
```
|
||||
|
||||
- **Transport:** Streamable HTTP as primary (Nov-2025 spec, post-SSE). Claude Code, Cursor, Windsurf, and the VS Code MCP extensions all support HTTP — it's the idiomatic shape for a long-running local service, which Voicebox already is.
|
||||
- **Stdio fallback:** `voicebox-mcp` binary bundled inside the app for clients that can't speak HTTP MCP. The Settings page renders the exact snippet with the detected absolute path — user copies, pastes, done. No PATH manipulation, no custom CLI wrapper.
|
||||
- **Identity:** HTTP clients set `X-Voicebox-Client-Id` header in their MCP config's `headers` block. Stdio clients set `VOICEBOX_CLIENT_ID` env var, which the shim forwards as the same HTTP header. Server reads it into a `ContextVar`.
|
||||
- **Profile resolution precedence:** explicit tool arg → per-client `MCPClientBinding.profile_id` → `capture_settings.default_playback_voice_id` → error.
|
||||
- **Port:** `17493`, matching `tauri/src-tauri/src/main.rs:63` (`SERVER_PORT` constant). Shim default with `VOICEBOX_PORT` env override.
|
||||
- **Non-MCP access:** `POST /speak` is a thin REST wrapper around the same tool path — one endpoint for shell scripts, ACP, A2A, and anything that isn't MCP-native.
|
||||
|
||||
## Library choice
|
||||
|
||||
- **`fastmcp`** (PyPI — verify on install whether the canonical import is `fastmcp` standalone or `mcp.server.fastmcp` from the consolidated `mcp` package; the API is identical).
|
||||
- **`sse-starlette`** for the `/events/speak` pill-state broadcast.
|
||||
- **`httpx` + `anyio`** already present — used by the shim.
|
||||
|
||||
## Data model
|
||||
|
||||
New table, **one row per client_id** (not a singleton — scales to unknown clients, maps 1:1 to the Settings UI list):
|
||||
|
||||
```python
|
||||
# backend/database/models.py
|
||||
class MCPClientBinding(Base):
|
||||
__tablename__ = "mcp_client_bindings"
|
||||
client_id = Column(String, primary_key=True) # "claude-code", "cursor", ...
|
||||
label = Column(String, nullable=True)
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=True)
|
||||
default_engine = Column(String, nullable=True)
|
||||
default_personality = Column(Boolean, nullable=False, default=False) # rewrite-before-speak default
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
```
|
||||
|
||||
Global default stays in `capture_settings.default_playback_voice_id` — no duplication. Migration: new `_migrate_mcp_client_bindings()` in `backend/database/migrations.py` using `CREATE TABLE IF NOT EXISTS`, mirroring the existing idempotent-add-column pattern.
|
||||
|
||||
## File plan
|
||||
|
||||
### Backend — new
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `backend/mcp/__init__.py` | Package marker |
|
||||
| `backend/mcp/server.py` | `build_mcp_server()` + `mount_into(app)`; registers tools, middleware, mount at `/mcp` |
|
||||
| `backend/mcp/tools.py` | The 4 `@mcp.tool()` functions — thin wrappers over existing services |
|
||||
| `backend/mcp/context.py` | `current_client_id: ContextVar[str \| None]` + Starlette middleware |
|
||||
| `backend/mcp/resolve.py` | `resolve_profile(explicit, client_id, db) -> VoiceProfile \| None` |
|
||||
| `backend/mcp/events.py` | In-memory `asyncio.Queue` pub/sub for speak-start / speak-end |
|
||||
| `backend/mcp/README.md` | MCP Inspector quickstart + `.mcp.json` snippets |
|
||||
| `backend/mcp_shim/__init__.py`, `__main__.py` | Stdio ↔ Streamable HTTP proxy (~150 lines) |
|
||||
| `backend/voicebox-mcp.spec` | PyInstaller spec for the shim (strips torch/transformers from `hiddenimports`) |
|
||||
| `backend/routes/speak.py` | `POST /speak {text, profile?, engine?, personality?, language?}` — REST wrapper around `resolve_profile()` + `generate_speech()` for non-MCP agents |
|
||||
|
||||
### Backend — modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `backend/app.py` | Migrate `@app.on_event("startup"/"shutdown")` (lines 185, 268) to `lifespan=` kwarg on `FastAPI()` using `AsyncExitStack`; call `mount_into(application)` after `register_routers`. Register `ClientIdMiddleware`. |
|
||||
| `backend/routes/profiles.py` | In `speak_in_character` (line 453): `events.publish("speak-start", {...})` on entry; completion hook publishes `speak-end`. Accept optional `source="mcp"` marker. |
|
||||
| `backend/services/generation.py` | `run_generation` completion path publishes `speak-end`. |
|
||||
| `backend/services/profiles.py` | New `async def get_profile_by_name_or_id(name_or_id, db)` — id lookup first, case-insensitive name fallback. |
|
||||
| `backend/database/models.py` | Add `MCPClientBinding`. |
|
||||
| `backend/database/migrations.py` | Add `_migrate_mcp_client_bindings`. |
|
||||
| `backend/models.py` | Add `MCPClientBindingResponse`, `MCPClientBindingUpdate`. |
|
||||
| `backend/routes/__init__.py` | Register `mcp_bindings_router`, `speak_router`, `events_router`. |
|
||||
| `backend/routes/mcp_bindings.py` (new) | REST CRUD for bindings (list, upsert, delete). |
|
||||
| `backend/routes/events.py` (new) | `GET /events/speak` — `EventSourceResponse` subscribed to the events queue. |
|
||||
| `backend/requirements.txt` | `+ fastmcp` (or `mcp>=1.0`), `+ sse-starlette` |
|
||||
| `backend/voicebox-server.spec` | `hiddenimports += ['mcp', 'mcp.server', 'fastmcp']` |
|
||||
| `backend/build_binary.py` | Second PyInstaller invocation for `voicebox-mcp.spec`; copy to `tauri/src-tauri/binaries/` with target-triple suffix |
|
||||
|
||||
### Frontend — new
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `app/src/components/ServerSettings/MCPBindings.tsx` | Settings section — default voice + per-client binding rows + `.mcp.json` copy-paste cheatsheet |
|
||||
| `app/src/lib/hooks/useMCPBindings.ts` | TanStack Query mirror of `useCaptureSettings` |
|
||||
| `app/src/lib/api/mcp.ts` | `listMCPBindings` / `upsertMCPBinding` / `deleteMCPBinding` |
|
||||
|
||||
### Frontend — modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `app/src/components/DictateWindow/DictateWindow.tsx` | Open `EventSource('/events/speak')`; on `speak-start` set pill to `speaking` with profile name; dismiss on `speak-end`. |
|
||||
| `app/src/components/CapturePill/CapturePill.tsx` | Add `speaking` branch — reuse the active waveform, swap status label to profile name. |
|
||||
| `app/src/lib/hooks/useCaptureRecordingSession.ts` | Union a `speaking` injection into the derived pill state. |
|
||||
| `app/src/lib/api/types.ts` | `MCPClientBinding`, `MCPClientBindingUpdate` types. |
|
||||
| `app/src/components/ServerSettings/index.tsx` | Register the new MCP section in the tab aggregator. |
|
||||
|
||||
### Tauri
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `tauri/src-tauri/tauri.conf.json` | `"externalBin": ["binaries/voicebox-server", "binaries/voicebox-mcp"]` |
|
||||
| `tauri/src-tauri/binaries/voicebox-mcp-<triple>` | Build artifact from PyInstaller |
|
||||
|
||||
## Tool signatures
|
||||
|
||||
All tools read `current_client_id.get()` (from middleware). Return JSON-serializable dicts.
|
||||
|
||||
Tools are registered with **dotted names** (`voicebox.speak`, etc.) to match the landing page and the industry convention (`filesystem.read_file`, `github.create_issue`). Python function names stay snake_case; the dot goes in the `name=` kwarg.
|
||||
|
||||
```python
|
||||
# backend/mcp/tools.py
|
||||
|
||||
@mcp.tool(name="voicebox.speak")
|
||||
async def speak(text: str,
|
||||
profile: str | None = None, # name OR id
|
||||
engine: str | None = None,
|
||||
personality: bool | None = None, # true → rewrite via profile's personality LLM before TTS
|
||||
language: str | None = None) -> dict:
|
||||
"""Speak text in a voice profile. Returns {generation_id, status, profile, poll}."""
|
||||
# resolve profile via precedence, delegate to generate_speech — the
|
||||
# route honors `personality=True` by running rewrite_as_profile on
|
||||
# the input before running the normal TTS pipeline.
|
||||
|
||||
@mcp.tool(name="voicebox.transcribe")
|
||||
async def transcribe(audio_base64: str | None = None,
|
||||
audio_path: str | None = None, # absolute local path
|
||||
language: str | None = None,
|
||||
model: str | None = None) -> dict:
|
||||
"""Transcribe audio. Exactly one of audio_base64/audio_path. Returns {text, duration, language}."""
|
||||
# validate path readable, size < 200 MB, then call services.transcribe.transcribe_bytes
|
||||
|
||||
@mcp.tool(name="voicebox.list_captures")
|
||||
async def list_captures(limit: int = 20, offset: int = 0) -> dict:
|
||||
"""Recent captures with transcripts. Returns {captures: [...]}"""
|
||||
|
||||
@mcp.tool(name="voicebox.list_profiles")
|
||||
async def list_profiles() -> dict:
|
||||
"""Available voice profiles. Returns {profiles: [{id, name, voice_type, has_personality}]}"""
|
||||
```
|
||||
|
||||
### `POST /speak` (non-MCP REST wrapper)
|
||||
|
||||
```python
|
||||
# backend/routes/speak.py
|
||||
@router.post("/speak", response_model=GenerationResponse)
|
||||
async def speak(data: SpeakRequest, request: Request, db: Session = Depends(get_db)):
|
||||
"""Same behavior as the MCP tool — for shell scripts, ACP, A2A, or anything non-MCP."""
|
||||
client_id = request.headers.get("X-Voicebox-Client-Id")
|
||||
profile = resolve_profile(data.profile, client_id, db)
|
||||
if profile is None: raise HTTPException(400, "No voice profile resolved.")
|
||||
req = GenerationRequest(profile_id=profile.id, text=data.text,
|
||||
language=data.language or "en",
|
||||
engine=data.engine or "qwen",
|
||||
personality=bool(data.personality))
|
||||
return await generate_speech(req, db)
|
||||
```
|
||||
|
||||
`SpeakRequest`: `{ text: str, profile: str | None, engine: str | None, personality: bool | None, language: str | None }`. Accepts name OR id for `profile` (via `resolve_profile`). `personality=None` means "use the per-client binding's `default_personality`"; explicit `true`/`false` always wins. Same precedence as the MCP tool so the two surfaces behave identically.
|
||||
|
||||
## Mount point (`backend/app.py`)
|
||||
|
||||
```python
|
||||
# After register_routers(application):
|
||||
from .mcp.server import mount_into
|
||||
mount_into(application)
|
||||
```
|
||||
|
||||
`mount_into` installs `ClientIdMiddleware` and calls `app.mount("/mcp", mcp.streamable_http_app())`.
|
||||
|
||||
**Lifespan migration is load-bearing** — FastMCP's session manager requires the `lifespan=` kwarg, not `@app.on_event`. Wrap the existing startup/shutdown bodies in an `@asynccontextmanager` using `contextlib.AsyncExitStack` so both Voicebox's init and FastMCP's session manager run. Verify dev + packaged build after the migration.
|
||||
|
||||
## Stdio shim (`backend/mcp_shim/__main__.py`)
|
||||
|
||||
1. Port: `int(os.environ.get("VOICEBOX_PORT", "17493"))`.
|
||||
2. Client id: `os.environ.get("VOICEBOX_CLIENT_ID", "unknown")`.
|
||||
3. Health probe `GET /health` with 30 s tolerance (torch imports slowly). On failure, emit JSON-RPC error on stdout, exit 1.
|
||||
4. Connect Streamable HTTP MCP client to `http://127.0.0.1:{port}/mcp` with `X-Voicebox-Client-Id: {client_id}` header.
|
||||
5. Proxy JSON-RPC bidirectionally — stdin → HTTP, SSE → stdout. Use `mcp` SDK's built-in stdio↔HTTP bridge if available; otherwise ~40 lines of asyncio.
|
||||
6. Stdout = JSON-RPC only. All logs to stderr.
|
||||
|
||||
PyInstaller spec keeps only `mcp`, `httpx`, `anyio`, `click` — target binary <20 MB.
|
||||
|
||||
## Pill `speaking` state
|
||||
|
||||
- `backend/mcp/events.py`: module-level `_subscribers: list[asyncio.Queue]` + `publish(kind, payload)` + `subscribe() -> Queue`.
|
||||
- `speak_in_character` publishes `speak-start` with `{generation_id, profile_id, profile_name, source}` immediately after `task_manager.start_generation`; `run_generation`'s completion path publishes `speak-end`.
|
||||
- `/events/speak` → `EventSourceResponse`.
|
||||
- `DictateWindow` opens `EventSource` next to existing `dictate:*` listeners, maps `speak-start/end` → pill `speaking` mode with profile name.
|
||||
- Optional filter: only show pill when `source === "mcp"` (avoids pill churn during manual speak flows). Settings toggle later.
|
||||
|
||||
## Settings UI (`MCPBindings.tsx`)
|
||||
|
||||
- **Global default voice** picker bound to `capture_settings.default_playback_voice_id` (reuses `useCaptureSettings`).
|
||||
- **Per-client table** — add/edit/remove rows of `{client_id, label, profile_id, default_engine, default_personality}`. Uses `useMCPBindings`.
|
||||
- **Connection cheatsheet** — two tabs, HTTP (default) and Stdio (fallback), with copy-to-clipboard snippets per known client:
|
||||
|
||||
HTTP form (primary):
|
||||
```json
|
||||
{"mcpServers": {"voicebox": {
|
||||
"url": "http://127.0.0.1:17493/mcp",
|
||||
"headers": {"X-Voicebox-Client-Id": "claude-code"}
|
||||
}}}
|
||||
```
|
||||
|
||||
Stdio form (fallback, absolute path auto-filled from detected app location):
|
||||
```json
|
||||
{"mcpServers": {"voicebox": {
|
||||
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
|
||||
"env": {"VOICEBOX_CLIENT_ID": "claude-code"}
|
||||
}}}
|
||||
```
|
||||
|
||||
Plus the Claude-Code-specific one-liner:
|
||||
```
|
||||
claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"
|
||||
```
|
||||
- **One-click install buttons** for known clients (v1: Claude Code via `claude mcp add` invocation, and a config-file writer for Cursor/Windsurf whose config locations are known). Each has a matching "Remove" button. Hide buttons for clients not detected on disk.
|
||||
- **Connection status** — small indicator next to each binding showing the last time that `client_id` actually called the server (rolling timestamp recorded by middleware), so users can tell their install worked.
|
||||
|
||||
## Ordered task list (shortest path first)
|
||||
|
||||
1. `fastmcp` + `sse-starlette` → `backend/requirements.txt`; install.
|
||||
2. Add `backend/mcp/{server,tools,context,resolve}.py` with the 4 tools registered as `voicebox.speak` etc. (no middleware yet — global default profile only).
|
||||
3. Migrate `app.py` to `lifespan=`; mount FastMCP at `/mcp`.
|
||||
4. **Milestone:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp` — call `voicebox.speak`, hear audio.
|
||||
5. Add `get_profile_by_name_or_id`; wire the tool's `profile` arg.
|
||||
6. `MCPClientBinding` model + migration; middleware; full `resolve_profile` precedence.
|
||||
7. `backend/routes/speak.py` — `POST /speak` REST wrapper, reusing `resolve_profile` + `speak_in_character`.
|
||||
8. `/mcp/bindings` REST + `MCPBindings.tsx` UI with HTTP and stdio copy-snippets, one-click install for detected clients, and connection-status indicators. **Users can install Voicebox as an MCP server after this step.**
|
||||
9. `backend/mcp_shim/__main__.py` + PyInstaller spec + `build_binary.py` second pass; register `voicebox-mcp` as a Tauri sidecar. (Fallback path goes live.)
|
||||
10. Events queue + `/events/speak` SSE + `DictateWindow` `speaking` pill state.
|
||||
11. `backend/mcp/README.md` quickstart.
|
||||
|
||||
Claude Code can call `voicebox.speak` after step 4 (direct HTTP, manual config). Step 8 makes that a one-click experience. Step 9 adds the stdio fallback for clients that don't speak HTTP MCP.
|
||||
|
||||
## Verification
|
||||
|
||||
- **Step 4 smoke:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp`. Call `voicebox.list_profiles`, then `voicebox.speak(text="hello from mcp")`. Audio plays; generation appears in History with `source="personality_speak"` (or new `source="mcp"` if we add one).
|
||||
- **REST wrapper:** `curl -X POST http://127.0.0.1:17493/speak -d '{"text":"hi","profile":"Morgan"}'` — same behavior, same pill surface.
|
||||
- **Per-client:** open two Inspector sessions with different `X-Voicebox-Client-Id` headers, bind each to a different profile in Settings, verify distinct voices without `profile` arg.
|
||||
- **Claude Code end-to-end (HTTP):** `claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"`, then ask Claude Code to speak. Pill shows `speaking: <profile>`, audio plays, capture appears in history.
|
||||
- **Stdio fallback:** manually paste the stdio snippet from Settings into a client's config, verify same behavior. `VOICEBOX_CLIENT_ID=claude-code python -m backend.mcp_shim` while backend is up; pipe a tools/list JSON-RPC in, verify response over stdout.
|
||||
- **Transcribe:** point at `/tmp/test.wav`; diff against `POST /transcribe` response.
|
||||
- **Failure modes:** kill backend mid-speak — shim must surface a JSON-RPC error, not deadlock. When backend isn't running, HTTP clients should get a clear connection-refused surfaced by the client.
|
||||
|
||||
## Risks / open decisions
|
||||
|
||||
- **`fastmcp` vs `mcp` package name** — confirm on `pip install`; APIs are near-identical, adjust imports.
|
||||
- **Lifespan migration** touches critical path (DB init, task queue, watchdog). Dev + packaged build both need a smoke after.
|
||||
- **Shim binary size** — if `mcp` pulls in enough dep weight that PyInstaller output is awkward, fall back to a Rust shim (Tauri shell is already Rust; JSON-RPC framing is trivial).
|
||||
- **Source attribution** — consider `source="mcp"` on the `Generation` model, or a dedicated `originator_client` column, if the Captures tab should filter MCP-originated generations.
|
||||
- **`audio_path` in `voicebox_transcribe`** — local-only today, but if the server ever binds beyond 127.0.0.1 we need to restrict reads to `data_dir` + user-whitelist.
|
||||
- **Auth** — none for now (127.0.0.1 only). If we bind outside, bearer token via `~/.voicebox/secret` + plumb through shim.
|
||||
- **HTTP MCP client support** — the plan leads with direct HTTP. Claude Code, Cursor, Windsurf, and VS Code MCP extensions all support it as of 2026, but if we discover an important client is stdio-only we still have the shim fallback ready.
|
||||
- **`.mcpb` desktop extension for Claude Desktop** (v2 polish) — Claude Desktop supports a double-clickable extension bundle format. Worth revisiting after v1 ships for an even cleaner install; skipped for now since Claude Desktop isn't the primary user (Claude Code + IDE users are).
|
||||
|
||||
## Critical files
|
||||
|
||||
- `backend/app.py`
|
||||
- `backend/routes/profiles.py`
|
||||
- `backend/routes/speak.py` (new)
|
||||
- `backend/database/models.py`
|
||||
- `backend/database/migrations.py`
|
||||
- `backend/services/generation.py`
|
||||
- `backend/build_binary.py`
|
||||
- `tauri/src-tauri/tauri.conf.json`
|
||||
- `tauri/src-tauri/src/main.rs` (port constant — no change, just reference)
|
||||
- `app/src/components/DictateWindow/DictateWindow.tsx`
|
||||
- `app/src/components/CapturePill/CapturePill.tsx`
|
||||
- `app/src/components/ServerSettings/`
|
||||
@@ -0,0 +1,675 @@
|
||||
# Voice I/O
|
||||
|
||||
**Status:** Shipping — phases 1, 2, 4, 7 (macOS) complete · 3 partial · 5, 6, 7 (Windows/Linux), 8 pending
|
||||
**Touches:** backend, Tauri shell, frontend, a new native shim crate
|
||||
**Last reviewed:** 2026-04-21
|
||||
|
||||
## Progress
|
||||
|
||||
### Shipped
|
||||
|
||||
**Phase 1 — Groundwork.** Audio tab retired from the sidebar; its device / channel
|
||||
config lives under Settings. Captures tab is live at `/captures` with no feature
|
||||
flag.
|
||||
|
||||
**Phase 2 — Local LLM backend.** `LLMBackend` protocol alongside the existing
|
||||
TTS/STT backends. `qwen_llm_backend.py`, `services/llm.py`, `routes/llm.py`, and
|
||||
a shared model-download / cache pipeline. Qwen3 0.6B / 1.7B / 4B registered and
|
||||
user-selectable via `capture_settings.llm_model`.
|
||||
|
||||
**Phase 4 — Captures tab.** List + detail view, source badges (dictation /
|
||||
recording / file), retranscribe, refine (flags + model resolved from a
|
||||
server-side `capture_settings` singleton), delete, and the Play-as-voice
|
||||
dropdown over every profile.
|
||||
|
||||
### Partial
|
||||
|
||||
**Phase 3 — In-app voice input.** `CapturesTab` dictates end-to-end via
|
||||
`useCaptureRecordingSession`, which the Phase 7 floating pill also consumes.
|
||||
Outstanding: a universal mic button on other text inputs (Generate form,
|
||||
profile descriptions, story titles, etc.), and the streaming
|
||||
`/transcribe/stream` WebSocket — today's flow is a single `POST /captures`
|
||||
with the complete audio blob.
|
||||
|
||||
**Phase 7 — External dictation shell (macOS).** Both halves shipped on macOS.
|
||||
|
||||
Hotkey half:
|
||||
|
||||
- `tauri/src-tauri/src/chord_engine.rs` — pure state machine. Unit tests green.
|
||||
- `tauri/src-tauri/src/hotkey_monitor.rs` — `rdev`-based global listener on a
|
||||
background thread, with `set_is_main_thread(false)` applied to sidestep the
|
||||
macOS 14+ TSM crash ([Narsil/rdev#165](https://github.com/Narsil/rdev/issues/165)).
|
||||
Right-hand-only defaults preserve left-hand Cmd+Option+I devtools.
|
||||
- Default bindings hardcoded: `Cmd+Option` (push-to-talk) and
|
||||
`Cmd+Option+Space` (toggle-to-talk). The PTT → Toggle upgrade transition is
|
||||
preserved — adding Space mid-hold promotes the session without interrupting
|
||||
audio.
|
||||
- `DictateWindow` — transparent, always-on-top, borderless 420×64 webview
|
||||
pre-created hidden at app setup. Shows on chord-start, hides on
|
||||
capture-cycle completion. Error state on the pill auto-dismisses and
|
||||
copies-to-clipboard on click.
|
||||
|
||||
Paste half (macOS):
|
||||
|
||||
- `clipboard.rs` — `NSPasteboard` snapshot that walks `pasteboardItems` and
|
||||
copies every `(uti, bytes)` pair so multi-type content (images, styled
|
||||
text, file refs) survives the round-trip. `save_clipboard`,
|
||||
`write_text`, `restore_clipboard`, `current_change_count`.
|
||||
- `synthetic_keys.rs` — `CGEventPost` at the HID tap with the full four-event
|
||||
Cmd+V sequence (Cmd down → V down w/ flag → V up w/ flag → Cmd up).
|
||||
- `focus_capture.rs` — `AXUIElementCreateSystemWide` +
|
||||
`AXUIElementCopyAttributeValue(kAXFocusedUIElement)` +
|
||||
`AXUIElementGetPid`, with the AX attribute key CFStrings built at
|
||||
runtime because they're CFSTR macros, not linkable symbols.
|
||||
`NSRunningApplication.activateWithOptions:` for re-activation.
|
||||
- `accessibility.rs` — `AXIsProcessTrusted` gate.
|
||||
- `paste_final_text` command — activate → 120 ms settle → save clip →
|
||||
write text → ⌘V → 400 ms → restore. Skips when focus was in Voicebox
|
||||
itself.
|
||||
- Focus rides the `dictate:start` event payload; `DictateWindow` holds the
|
||||
snapshot in a ref and consume-once-nulls on paste so a late-arriving
|
||||
refine from an earlier session can't misfire.
|
||||
- Dictation recording no longer hard-caps at 29 s — the limit still
|
||||
applies to voice-profile reference clips.
|
||||
|
||||
Outstanding: Windows `SendInput` / UIAutomation / `SetForegroundWindow`
|
||||
equivalents, Linux `uinput` / AT-SPI equivalents (and the Wayland story),
|
||||
first-run Accessibility prompt UI with deep-link to System Settings,
|
||||
direct-injection path for focus-was-inside-Voicebox (step 6 — dictating
|
||||
into our own Generate tab currently falls back to the capture list).
|
||||
|
||||
### Not started
|
||||
|
||||
- **Phase 5 — Agent voice output + persona loop.** No `/speak` endpoint, no
|
||||
`voicebox.speak` MCP tool, no per-agent voice binding, no persona metadata
|
||||
on profiles.
|
||||
- **Phase 6 — STT engine expansion.** Only Whisper (`mlx_backend.py`).
|
||||
Parakeet v3, Qwen3-ASR, Kyutai — all unregistered.
|
||||
- **Phase 8 — Pipeline routing, sinks, long-form.** No preset primitive, no
|
||||
MCP sink, no webhook sink, no dual-stream recorder, no summary transform.
|
||||
|
||||
### Additionally landed (not explicit in the original plan)
|
||||
|
||||
These fell out of the Phase 3/4/7 work but deserve their own mention:
|
||||
|
||||
- **Server-authoritative settings.** Singleton `capture_settings` and
|
||||
`generation_settings` tables. The client sends nothing but the audio; STT
|
||||
model, refine flags, refine LLM, and the auto-refine flag are all resolved
|
||||
server-side, so sibling Tauri webviews can't go stale.
|
||||
- **Backend audio normalisation.** `POST /captures` transcodes anything
|
||||
librosa can decode (webm/opus, m4a, etc.) to WAV before handing it to
|
||||
whisper, side-stepping miniaudio's format gaps inside mlx-audio.
|
||||
- **Short-recording guard.** Sub-300 ms blobs short-circuit client-side so a
|
||||
fumbled chord tap never uploads an empty webm.
|
||||
- **Refinement prompt.** Rewritten with firmer anti-chatbot framing and
|
||||
inline examples covering multi-sentence preservation and self-correction.
|
||||
|
||||
### Near-term outstanding
|
||||
|
||||
Called out in recent sessions but not yet in a phase:
|
||||
|
||||
- **Configurable chord bindings.** Pass 2 of the hotkey work — persist
|
||||
`push_to_talk_chord` / `toggle_to_talk_chord` in `capture_settings`,
|
||||
surface a chord-picker UI in `CapturesPage`, and wire a Tauri
|
||||
`update_chord_bindings` command so `HotkeyMonitor::update_bindings` picks
|
||||
up user changes live.
|
||||
- **Generate-tab empty-state explainer.** The parallel aside to the Captures
|
||||
explainer described in *Product surface → Parallel explainer on the
|
||||
Generate tab*. Lands alongside Phase 3's universal mic button so both tabs
|
||||
feel symmetric.
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox ships the output half of a voice I/O loop: clone a voice, generate
|
||||
speech, apply effects, compose multi-voice projects. The input half — speech to
|
||||
text, dictation, routing — exists today as a single Whisper model wired into the
|
||||
Recording & Transcription panel. This doc proposes making voice *input* a
|
||||
first-class pillar: more STT engines, a dictation shell (global hotkey, audio
|
||||
capture, paste, streaming), a local LLM backend, and a user-configurable
|
||||
pipeline from captured audio to whatever the user wants to do with it.
|
||||
|
||||
Positioning is the key move. **Voicebox becomes the local voice I/O layer for
|
||||
humans and AI agents** — a local alternative to cloud dictation tools, with the
|
||||
differentiator that we also do TTS and voice cloning. The same app that
|
||||
captures your voice can generate a response in any voice profile you've
|
||||
cloned. "Anything voice is Voicebox."
|
||||
|
||||
### Positioning shift
|
||||
|
||||
Before this plan, Voicebox was **"the open-source AI voice cloning studio."**
|
||||
Cloning was the headline capability.
|
||||
|
||||
After this plan, Voicebox is **"the open-source AI voice studio."** Cloning is
|
||||
one capability in a broader category that now spans input (STT, dictation),
|
||||
intelligence (local LLM, refinement, persona), output (TTS, cloning, effects,
|
||||
Stories), and routing. The word "cloning" drops out of the top-line descriptor
|
||||
because it's become a feature rather than the thesis.
|
||||
|
||||
### Competitive frame
|
||||
|
||||
Voicebox ends up covering the territory of two separately-funded, separately
|
||||
branded cloud incumbents that operate on opposite sides of the same voice I/O
|
||||
loop:
|
||||
|
||||
- **ElevenLabs** (~$3B+): voice cloning and TTS — the "agents speak" side
|
||||
- **WisprFlow** (~$70M raised): voice dictation for agents and power users —
|
||||
the "users talk" side
|
||||
|
||||
Both are cloud-only. Voicebox becomes the only local alternative to either,
|
||||
running in one app, with a single model directory and LLM shared between input
|
||||
and output. That bridging — dictation → LLM → TTS with a cloned voice in the
|
||||
middle — is the thing no single incumbent can match, because neither has the
|
||||
other half.
|
||||
|
||||
### Launch-time copy tasks
|
||||
|
||||
These are not engineering tasks but should ride the Phase 4 ship so marketing
|
||||
and positioning stay in sync with the product.
|
||||
|
||||
- **README.md** — drop "cloning" from the top-line descriptor. Add a section
|
||||
that explicitly frames Voicebox as "the open-source local alternative to
|
||||
WisprFlow and ElevenLabs." Competitive framing belongs in the README and on
|
||||
the landing page — not in-app (reads as defensive).
|
||||
- **voicebox.sh landing page** — same positioning shift.
|
||||
- **GitHub About / repo topics** — swap "voice-cloning" or similar tags for
|
||||
broader "voice-io," "local-tts," "local-stt," etc.
|
||||
- **Release notes** — the Phase 4 launch note is the "we're now voice I/O" moment.
|
||||
|
||||
## Why now
|
||||
|
||||
- Cross-platform local dictation is an empty category. The tools people love
|
||||
(Superwhisper, MacWhisper, Aiko) are macOS-only. WisprFlow and
|
||||
Willow are cloud. Our Windows install base is the wedge — first-class Windows
|
||||
support for a local dictation product is genuinely differentiated.
|
||||
- The `STTBackend` protocol already exists. The multi-engine registry pattern
|
||||
shipped with TTS makes adding Parakeet v3 and Qwen3-ASR a days-not-weeks
|
||||
effort on the backend side.
|
||||
- The **persona loop** — speak to an agent, have it reply in a cloned voice —
|
||||
is a feature only we can ship. Nobody with a dictation product has TTS; nobody
|
||||
with a TTS product has good dictation. The full duplex is ours.
|
||||
- Agent harnesses already pipe Voicebox TTS into their stacks. Giving those
|
||||
users STT from the same app closes the loop and makes Voicebox the default
|
||||
voice I/O layer for the agentic dev-tool crowd.
|
||||
- **Typing a 2,000-character TTS script is user-hostile.** The most immediate
|
||||
internal win is dictating directly into Voicebox's own generation form —
|
||||
speak the script, generate the voice. This dogfoods the whole STT pipeline
|
||||
without touching a single OS-level API.
|
||||
- **Voice-to-voice models are landing.** Moshi (Kyutai), GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM, Spirit LM (Meta) — end-to-end speech LLMs that
|
||||
take audio in and emit audio out are a near-term reality. The pipeline we're
|
||||
building today is the scaffolding they slot into tomorrow.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Cloud fallback or "bring your own API key" STT/LLM. Local is the product.
|
||||
- A separate tray-only dictation app. We extend Voicebox, not fork it.
|
||||
- Replacing the Stories editor with a notes layout. Long-form capture is a
|
||||
preset on top of the pipeline, not a new product surface.
|
||||
- Real-time translation UI. It can exist as a transform later, but it's not in
|
||||
this plan.
|
||||
- Full agent orchestration. We provide the voice rails; the agent lives
|
||||
elsewhere and talks to us via the developer API.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Three new backend concepts
|
||||
|
||||
**1. Expanded STT registry.** The existing `STTBackend` protocol abstracts
|
||||
Whisper today. Add:
|
||||
|
||||
- **Parakeet v3** — 25 languages, very fast, the current quality leader for
|
||||
non-English local STT. Python path via `nemo_toolkit` or `transformers`.
|
||||
- **Qwen3-ASR 0.6B int8** — 50+ languages, highest multilingual quality,
|
||||
cross-platform via `transformers`.
|
||||
- **Kyutai ASR** *(optional)* — streaming-first, small, CPU-friendly. Fills the
|
||||
"CPU-only laptop" tier.
|
||||
|
||||
All register via `ModelConfig` and use the same download, cache, and model
|
||||
management UI we already have for TTS. Zero special-casing.
|
||||
|
||||
**2. `LLMBackend` protocol.** Mirror of `TTSBackend` / `STTBackend`. First
|
||||
implementations are Qwen3 0.6B / 1.7B / 4B running on the same PyTorch + MLX
|
||||
infrastructure we already run. One runtime, one model cache, one GPU-memory
|
||||
story.
|
||||
|
||||
Why not `llama.cpp` or `ollama`: we already have the dependency surface and the
|
||||
model download UX. A second runtime fragments cache directories and model-status
|
||||
UI. If CPU-only Windows latency becomes a problem we can revisit.
|
||||
|
||||
**3. Streaming transcribe transport.** Add `/transcribe/stream` as a WebSocket
|
||||
endpoint alongside the existing HTTP `/transcribe`. Audio frames flow in,
|
||||
partial transcripts stream back. Same FastAPI process, same loaded models. This
|
||||
keeps dictation latency off the per-request JSON-encode critical path and lets
|
||||
us ship real-time partial transcripts later without a protocol change.
|
||||
|
||||
### The pipeline abstraction
|
||||
|
||||
Every captured audio event flows through the same shape:
|
||||
**Source → Transforms → Sink(s)**. Users configure presets that bind a source
|
||||
to a transform chain to one or more sinks.
|
||||
|
||||
```
|
||||
Source Transform Sink
|
||||
────────────────── ───────────────── ─────────────────
|
||||
Hold to speak ──┐ STT model Clipboard + paste
|
||||
Tap to toggle │ Refinement LLM Capture history
|
||||
Long-form recorder ├──▶ Persona LLM ──▶ File on disk
|
||||
File drop │ Translation (later) HTTP webhook
|
||||
API call (WS / HTTP) ──┘ MCP server sink
|
||||
TTS loopback (persona)
|
||||
Platform sinks (later)
|
||||
```
|
||||
|
||||
`Source → Transform → Sink` is internal, dataflow-style vocabulary (same shape
|
||||
as Unix pipes, Apache Beam, Kafka) — not user-facing. The UI surface will use
|
||||
Voicebox-native language (see open questions).
|
||||
|
||||
Concrete preset examples this shape enables:
|
||||
|
||||
- **Dictation** — hold-to-speak → Parakeet v3 → light refinement → clipboard + paste + history
|
||||
- **Code prompt** — dedicated hotkey → Whisper Turbo → technical-vocab refinement → MCP sink for Claude Code
|
||||
- **Agent voice reply** — hold-to-speak → STT → persona LLM → TTS with cloned profile → system audio out
|
||||
- **Long-form capture** — dual-stream recorder → chunked STT → summary LLM → markdown file + history
|
||||
|
||||
Every user-facing feature collapses into (source + transform chain + sinks).
|
||||
Meeting-style capture isn't a separate product; it's a preset. Competing tools
|
||||
hardcode integrations (Trello, Granola); we make routing user-configurable.
|
||||
|
||||
### Native shim crate
|
||||
|
||||
The parts Tauri doesn't handle cleanly, gathered in one Rust crate with a
|
||||
platform-agnostic API:
|
||||
|
||||
- **Global hotkey with modifier-only support.** Tauri's `global-shortcut`
|
||||
plugin requires full combos. We need "hold right-cmd" or "hold ctrl" as
|
||||
primitives. On macOS this means a CGEventTap on a background thread with
|
||||
polling fallback for dropped modifier events; on Windows a low-level keyboard
|
||||
hook; on Linux X11 + libinput, with Wayland as a known gap.
|
||||
- **Focus introspection.** Query the frontmost app and its focused element via
|
||||
OS accessibility APIs — `AXUIElement` on macOS, UIAutomation on Windows,
|
||||
AT-SPI on Linux. Check the element's role to decide between a direct
|
||||
injection, a clipboard + paste, and a clipboard-only fallback with a
|
||||
notification. A blind paste that only "works when a text field happens to
|
||||
be focused" is the easy default; we should make the decision deliberately.
|
||||
- **Simulated paste.** CGEvent on macOS, SendInput on Windows, uinput / ydotool
|
||||
on Linux. Wayland is the hard case and needs explicit handling.
|
||||
- **Atomic clipboard save/restore.** Save *all* items and *all* MIME
|
||||
representations before writing our transcript, restore atomically after
|
||||
paste. Pasting a transcript shouldn't clobber a user's in-progress rich-media
|
||||
clipboard.
|
||||
- **Frontmost-window context capture** *(later).* macOS Vision, Windows OCR,
|
||||
Linux tesseract. Optional feature to feed the refinement LLM disambiguation
|
||||
hints from the window being pasted into.
|
||||
|
||||
Main process owns this crate. Webview never sees platform differences.
|
||||
|
||||
### Target-aware delivery
|
||||
|
||||
The paste sink adapts to what's in focus. This is a single sink type with
|
||||
branching behavior, not four separate sinks.
|
||||
|
||||
| Target | Delivery strategy |
|
||||
|---|---|
|
||||
| Focused text field inside Voicebox | Direct React state update via event. No clipboard involved. |
|
||||
| Focused text field in another app | Accessibility-verified paste: save clipboard, write transcript, simulate paste, restore clipboard. |
|
||||
| No text focus detected | Clipboard only, toast notification ("Transcript copied — no text field focused"). |
|
||||
| Platform-specific special cases (terminal apps, specific editors) | Per-app overrides where the generic path misbehaves. |
|
||||
|
||||
### Where each concern lives
|
||||
|
||||
| Concern | Layer |
|
||||
|---|---|
|
||||
| STT / LLM / TTS inference | Python backend |
|
||||
| Model downloads, progress, cache | Python backend |
|
||||
| Pipeline runner (orchestrates transforms and sinks) | Python backend |
|
||||
| Audio capture from mic / system audio | Rust (Tauri side) |
|
||||
| Audio streaming over WebSocket to backend | Rust |
|
||||
| Global hotkey capture | Rust (native shim crate) |
|
||||
| Paste simulation, clipboard save/restore | Rust (native shim crate) |
|
||||
| Pipeline preset UI, capture history, settings | React |
|
||||
|
||||
Model work in Python. OS work in Rust. User config in React.
|
||||
|
||||
## Product surface
|
||||
|
||||
### A new tab (and a sidebar reshuffle)
|
||||
|
||||
The current sidebar is `Generate · Stories · Voices · Effects · Audio · Models ·
|
||||
Settings`. The existing Audio tab is output-device and channel routing
|
||||
config — infrastructure, not a creative workspace — and the Settings page
|
||||
already has a sub-tab pattern (`ServerSettings/`: Connection, Models, GPU,
|
||||
Update) that fits it naturally.
|
||||
|
||||
**Move Audio to a Settings sub-tab. Reclaim the sidebar slot for voice input.**
|
||||
|
||||
The new tab shows recent captures (audio + transcript paired), active presets,
|
||||
dictation settings, model pickers for STT and LLM. Exact name is an open
|
||||
question.
|
||||
|
||||
**Sidebar placement:** Captures sits at position 3, directly under Stories and
|
||||
above Voices. Creates an "input voice / output voice" adjacency — captured
|
||||
speech is one slot away from the voices you can play it back through, which
|
||||
mirrors the Phase 4 "Play as voice" feature's mental model. Full order:
|
||||
Generate · Stories · Captures · Voices · Effects · Models · Settings.
|
||||
|
||||
### Parallel explainer on the Generate tab
|
||||
|
||||
The Captures settings page gets a "What's different" aside that introduces
|
||||
Voicebox's dictation story. The Generate tab deserves a parallel — first-time
|
||||
users need to be told what voice generation is *for* in a post-Voice-I/O
|
||||
world, not just handed a text field.
|
||||
|
||||
Shape: an **empty-state card** rendered in the Generate tab when there's no
|
||||
generation history yet, disappearing once the user has generated anything.
|
||||
Teaches without claiming permanent real estate. Parallel bullets to the
|
||||
Captures aside so the two tabs feel like two sides of one product:
|
||||
|
||||
- **Clone any voice in seconds** — a short sample is enough
|
||||
- **Seven engines, 23 languages** — creative range, not a single model
|
||||
- **Agent-ready** — REST + WebSocket API, one checkbox away from giving any
|
||||
AI agent a voice
|
||||
|
||||
This lands in Phase 4 alongside the Captures tab, for visual and thematic
|
||||
symmetry. Not a persistent sidebar — the Generate tab is a workspace and
|
||||
should reclaim its space once the user is producing work.
|
||||
|
||||
### Archival by default
|
||||
|
||||
Every capture saves the original audio alongside the final transcript in a
|
||||
pattern that mirrors `data/generations/`. Optional retention setting. Free for
|
||||
us — the storage and UI patterns exist today for generations.
|
||||
|
||||
### Developer API, day one
|
||||
|
||||
The WebSocket transcribe endpoint is a first-class public API, documented
|
||||
alongside `/generate`. Pipeline presets are addressable by ID via
|
||||
`/pipelines/{id}/run` so agent harnesses and shell scripts can invoke
|
||||
user-configured flows. An MCP server sink ships built-in, so integrations with
|
||||
Claude Code, Cursor, Cline, etc. are one checkbox rather than a custom build.
|
||||
|
||||
### Agent voice output
|
||||
|
||||
Dictation is one half of the loop — user speaks, agent listens. The other half
|
||||
— agent speaks, user hears — is equally load-bearing and deserves a
|
||||
first-class primitive rather than being buried as a TTS loopback sink or a
|
||||
consumer read-aloud button.
|
||||
|
||||
The shape is a single new capability: any agent can call Voicebox to speak
|
||||
arbitrary text in a user-configured voice. The same pill that surfaces during
|
||||
dictation surfaces during agent speech, so the user always sees what's coming
|
||||
out of their machine.
|
||||
|
||||
```
|
||||
MCP tool: voicebox.speak({ text, profile?, style? })
|
||||
REST: POST /speak { text, profile_id?, style? }
|
||||
```
|
||||
|
||||
Both accept an optional voice profile (defaults to the user's configured
|
||||
default), an optional delivery-style string for engines that support it, play
|
||||
audio through system output, and surface the pill in a `speaking` state.
|
||||
|
||||
**Key design points:**
|
||||
|
||||
- **Pill is bidirectional.** States expand from `recording / transcribing /
|
||||
refining / rest` to include `speaking` — voice profile name, waveform in
|
||||
the profile's color, visible duration. Same floating surface for both
|
||||
directions so users have one mental model.
|
||||
- **Visibility is mandatory.** Silent background TTS is a trust hazard. Every
|
||||
agent-initiated `speak()` surfaces the pill. No headless "TTS daemon" mode.
|
||||
- **Per-source voice policy.** Settings let users bind specific MCP clients or
|
||||
API keys to specific voice profiles — Claude Code in "Morgan," Cursor in
|
||||
"Scarlett" — so users can tell which agent is talking without looking.
|
||||
- **Mute + rate limits.** One-toggle mute for all agent speech. Per-source
|
||||
rate limits prevent a runaway agent from monologuing.
|
||||
|
||||
This primitive is what makes "Voicebox as voice layer for every agent on your
|
||||
machine" a concrete shipping capability rather than marketing language. MCP,
|
||||
ACP, and A2A integrations all slot into it — none of those agent protocols
|
||||
need to know anything about TTS models, GPU placement, or voice profiles.
|
||||
They call `speak()`.
|
||||
|
||||
**Relationship to the persona loop.** The persona loop below is *one* use of
|
||||
`speak()` — STT → LLM → `speak(llm_reply)`. Other uses skip STT entirely: a
|
||||
long-running task announcing completion, a notification, an agent proactively
|
||||
asking the user a question. The primitive is deliberately simpler than the
|
||||
persona loop so it can serve both flows from the same API.
|
||||
|
||||
### Relationship to voice profile samples
|
||||
|
||||
A capture and a voice profile sample both hold `audio + text`, so there's an
|
||||
obvious temptation to unify them. Don't. The metadata and lifecycle
|
||||
differences are real:
|
||||
|
||||
| | Capture | Voice profile sample |
|
||||
|---|---|---|
|
||||
| Profile association | Standalone | Bound to one profile |
|
||||
| Text field | Raw transcript + optional LLM-refined version | Exact `reference_text` only |
|
||||
| LLM refinement | Often applied | Must not be applied — the reference text must match the audio verbatim or cloning breaks |
|
||||
| Volume | Dozens per day | ~5 per profile, semi-permanent |
|
||||
| Typical content | Whatever the user said | Often scripted phrases for cloning |
|
||||
|
||||
A unified table would mean nullable `profile_id`, nullable `refined_transcript`,
|
||||
nullable `reference_text` — a fat row that means different things in different
|
||||
states. Not worth the complexity.
|
||||
|
||||
**What to ship instead: a one-way promote action.** Capture → Sample, zero
|
||||
data-model churn. Thin endpoint:
|
||||
|
||||
```
|
||||
POST /profiles/{id}/samples/from-capture/{capture_id}
|
||||
```
|
||||
|
||||
Reads the capture's audio path and raw transcript, calls the existing
|
||||
`add_sample()` service with `reference_text` pre-filled from the transcript,
|
||||
lets the user edit the reference text in a dialog before saving (transcripts
|
||||
are usually 90% right but cloning wants 100%). The capture stays in the
|
||||
Captures tab untouched — the sample is a copy, not a move.
|
||||
|
||||
UI hook: the Captures tab's Send-to menu gains a **"Use as voice sample…"**
|
||||
option that opens a profile picker (with "+ New voice" for cold starts) and a
|
||||
reference-text confirm dialog.
|
||||
|
||||
The inverse direction (sample → capture) we deliberately skip. Samples are
|
||||
often scripted phrases used for cloning and they'd clutter the Captures list
|
||||
without adding value; also a subtle privacy surprise for users who don't
|
||||
expect their sample text browsable alongside real captures.
|
||||
|
||||
**Audio storage deduplication is a later optimization.** Today a promoted
|
||||
capture duplicates the audio file on disk. That's fine. Content-addressable
|
||||
storage (`data/audio/<sha256>.wav` with refcounting) can come in Phase 8 as
|
||||
housekeeping — it'd let a capture and a sample share one underlying file, but
|
||||
it's not user-visible and not necessary to ship the promote flow.
|
||||
|
||||
### The persona loop
|
||||
|
||||
One flow on top of the `speak()` primitive: STT → persona LLM →
|
||||
`speak(llm_reply)`. Voice profiles gain optional metadata — a natural-language
|
||||
personality description and default LLM behavior. The LLM runs text through
|
||||
the profile's voice context, then `speak()` generates TTS with the cloned
|
||||
profile. End-to-end voice-to-voice with a cloned identity transforming the
|
||||
content, not just reading it.
|
||||
|
||||
Use cases this unlocks:
|
||||
|
||||
- Agents that respond to spoken input in a specific voice
|
||||
- Interactive character experiences (games, narrative tools, accessibility)
|
||||
- Speech assistance for people who can't speak in their original voice
|
||||
|
||||
The shape — STT + LLM + TTS — also stages us for end-to-end speech LLMs which
|
||||
collapse all three into one transform. See *Voice-to-voice readiness* below.
|
||||
|
||||
### Voice-to-voice readiness
|
||||
|
||||
The STT → LLM → TTS chain that powers the persona loop is a staged approximation
|
||||
of voice-to-voice. A real end-to-end speech LLM (Moshi, GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM) replaces the three middle boxes with a single
|
||||
fused transform: audio in, audio out, no text in between. The pipeline shape
|
||||
accommodates this natively — register the model as a single `LLMBackend` (or
|
||||
a new `SpeechLLMBackend` if the protocol needs to differ), expose it as a
|
||||
transform type, and the same sinks work unchanged.
|
||||
|
||||
Framing this plan as "voice-to-voice scaffolding, with today's models as the
|
||||
staged fallback" is a strong pitch for agent-harness users who are already
|
||||
tracking these models.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Tab name.** Leaning **Captures** — neutral, extensible across dictation,
|
||||
long-form recordings, and uploaded audio without repainting the tab later.
|
||||
"Dictations" is narrower (office-productivity coded, doesn't fit meeting
|
||||
recordings). "Notes" is the wrong mental model — nobody opens Voicebox to
|
||||
write notes. "Transcriptions" is flat.
|
||||
2. **Refinement vocabulary.** The LLM-post-STT step needs a user-facing name.
|
||||
"Refine," "polish," "rewrite," "smart edit" are candidates. "Refinement" in
|
||||
this doc as a placeholder only.
|
||||
3. **Preset primitive.** What do we call a user-configured pipeline? "Intent"
|
||||
collides with the existing `instruct` field on TTS generation. "Flow" is
|
||||
Zapier-coded. "Route" is too networking. Needs its own pass.
|
||||
4. **Persona metadata shape.** Does personality live directly on the voice
|
||||
profile, or as a separate persona construct that wraps profile + LLM config?
|
||||
The first is simpler; the second scales better if we later want multiple
|
||||
personas per voice.
|
||||
5. **Long-form capture product surface.** Pure preset, or dedicated entry point
|
||||
in the new tab? Leaning preset, but long-form is the feature that most
|
||||
justifies its own landing page.
|
||||
6. **Hotkey primitive naming.** Hold-vs-tap needs Voicebox-native phrasing in
|
||||
UI copy. Settings can still use industry-standard terms.
|
||||
|
||||
## Ordered phases
|
||||
|
||||
The v1 prototype deliberately skips the hardest parts of the long-term plan
|
||||
(native OS shim, global hotkeys, paste injection, new STT models). Everything
|
||||
in Phase 1–4 is in-process code using Whisper (which we already ship) and the
|
||||
existing model infra. No CGEvent taps, no SendInput, no clipboard timing.
|
||||
The usual OS-level sprawl of a dictation stack is exactly what we sidestep
|
||||
by starting in-app.
|
||||
|
||||
### Phase 1 — Groundwork
|
||||
|
||||
- Move the Audio tab into a Settings sub-tab (`ServerSettings/` gains one
|
||||
more section). Audio is device/channel config, not a creative workspace.
|
||||
- Reserve the sidebar slot for the new Captures tab (name TBD but leaning
|
||||
Captures — see open questions).
|
||||
- Gate the Captures tab behind a feature flag so we can merge to `main` and
|
||||
iterate without shipping half-built UI to users.
|
||||
|
||||
### Phase 2 — Local LLM backend
|
||||
|
||||
`LLMBackend` protocol alongside `TTSBackend` / `STTBackend`. Register Qwen3
|
||||
0.6B / 1.7B / 4B via `ModelConfig`. Reuses the HF download path, cache
|
||||
directory, and model management UI. MLX (4-bit community quants) on Apple
|
||||
Silicon, PyTorch (transformers AutoModelForCausalLM) elsewhere, same as our
|
||||
TTS split.
|
||||
|
||||
No new runtime. No `llama.cpp`, no `ollama`, no fragmented model cache.
|
||||
|
||||
### Phase 3 — In-app voice input
|
||||
|
||||
A universal mic button on every Voicebox text input. Hold, speak, release —
|
||||
text lands in the focused field via direct React state update. No OS APIs
|
||||
involved; Voicebox owns the input.
|
||||
|
||||
Marquee use cases:
|
||||
|
||||
- **Generation form.** Dictate a 2,000-character TTS script instead of typing
|
||||
it. This alone justifies the feature.
|
||||
- **Voice profile descriptions.** Describe a voice's personality by speaking,
|
||||
which then becomes the input for Phase 4's persona loop.
|
||||
- **Story titles, preset names, any free-text field.** Free reuse.
|
||||
|
||||
Backend: add `/transcribe/stream` WebSocket endpoint. Audio frames in, partial
|
||||
transcripts out. Reuses the existing Whisper model in memory. Optionally routes
|
||||
through the LLM from Phase 2 for light refinement.
|
||||
|
||||
### Phase 4 — Captures tab
|
||||
|
||||
Graduates the tab out from behind the feature flag. Shows recent captures
|
||||
(audio + transcript pairs), lets the user replay, re-transcribe with a
|
||||
different model, edit the transcript, and send the output through the LLM.
|
||||
Archival is automatic — every capture saves audio alongside transcript.
|
||||
|
||||
**Includes the "Play as voice profile" action.** This is the simplest version
|
||||
of the persona loop and it lands here for free — no LLM involved, no new
|
||||
backend endpoints, just a Captures-tab button that sends the transcript text
|
||||
to the existing `/generate` endpoint with a user-selected voice profile and
|
||||
plays the result. Category-defining differentiator from the v1 prototype
|
||||
onward: Superwhisper and WisprFlow cannot do this because they have no TTS. Voicebox can, with one day of frontend wiring.
|
||||
|
||||
Keep it aggressively minimal on day one. A capture list, a detail view, a
|
||||
model picker, a Play-as-voice dropdown. Refinement prompt editing, correction
|
||||
dictionaries, per-source overrides — none of that ships here. They become
|
||||
Tier-2 work when someone actually asks for them.
|
||||
|
||||
### Phase 5 — Agent voice output + persona loop
|
||||
|
||||
Two features that together make "Voicebox as the voice layer for every agent
|
||||
on your machine" a shipping reality:
|
||||
|
||||
1. **`speak()` primitive.** New `POST /speak` endpoint and `voicebox.speak`
|
||||
MCP tool. Any agent calls Voicebox to speak arbitrary text in a
|
||||
user-configured voice; the pill surfaces in a `speaking` state. Settings
|
||||
UI for default voice, per-agent voice binding (Claude Code → Morgan,
|
||||
Cursor → Scarlett), and a global mute.
|
||||
2. **Persona loop.** Extends `speak()` with an LLM step — STT → persona LLM
|
||||
→ `speak(llm_reply)`. Voice profiles gain optional personality metadata
|
||||
and default LLM behavior. End-to-end voice-to-voice with a cloned
|
||||
identity transforming the content, not just reading it.
|
||||
|
||||
Phase 4 demoed the user-initiated direction of the loop (Play as voice). This
|
||||
phase ships the *agent*-initiated direction, which is the category-defining
|
||||
capability and the pitch that lands with agent-harness users. The persona
|
||||
loop is one flow on top of the `speak()` primitive — notifications, proactive
|
||||
agent questions, and task-completion announcements all use `speak()` directly
|
||||
without the LLM in the middle.
|
||||
|
||||
Launchable headline moment for the "local voice I/O" positioning.
|
||||
|
||||
### Phase 6 — STT engine expansion
|
||||
|
||||
Parakeet v3 and Qwen3-ASR register as additional `STTBackend` implementations.
|
||||
Optional: Kyutai ASR. Multilingual coverage upgrades (50+ languages). Whisper
|
||||
stays as the sensible default.
|
||||
|
||||
Deferred to here because Whisper is already good enough for v1 and the model
|
||||
picker UI exists. Adding rows to it doesn't change the product shape.
|
||||
|
||||
### Phase 7 — External dictation shell
|
||||
|
||||
Native shim crate (global hotkey with modifier-only support, focus
|
||||
introspection via OS accessibility APIs, paste simulation, atomic clipboard
|
||||
save/restore). Tauri-side audio capture streams to the same WebSocket endpoint
|
||||
Phase 3 already ships. Paste sink with target-aware delivery.
|
||||
|
||||
This is the feel-good phase. It's also the riskiest: paste timing, hotkey
|
||||
reliability, and cross-platform focus detection are all engineering problems
|
||||
that have to be nailed or the product doesn't work. Phase 3's success derisks
|
||||
the backend plumbing before we start it.
|
||||
|
||||
### Phase 8 — Pipeline routing, sinks, long-form
|
||||
|
||||
Multiple source types, user-configurable transform chains, multiple sinks per
|
||||
preset. MCP server sink (the agent-harness play). HTTP webhook sink. File
|
||||
sink. Developer-facing `/pipelines/{id}/run` endpoint. Preset editor UI in
|
||||
the Captures tab.
|
||||
|
||||
Dual-stream recorder (mic + system audio) as a source type. Chunked STT
|
||||
transform with overlap-based deduplication. Summary LLM transform. Long-form
|
||||
capture becomes a preset, not a new tab.
|
||||
|
||||
Platform-specific sinks (Apple Notes on macOS, Obsidian, etc.) as opt-in
|
||||
integrations behind the generic sink interface.
|
||||
|
||||
## Architectural prerequisites
|
||||
|
||||
Two pieces of existing `docs/PROJECT_STATUS.md` work become load-bearing here:
|
||||
|
||||
- **Platform support tiers** (#420, PR #465). Native shim capabilities vary by
|
||||
platform — Wayland paste is worse than X11, Windows system-audio capture has
|
||||
edge cases, frontmost-window OCR is platform-gated. Tier definitions let us
|
||||
ship confidently with honest user-facing expectations.
|
||||
- **Platform gating on `ModelConfig`** (bottleneck #6 in PROJECT_STATUS).
|
||||
Parakeet's Core ML path is Apple-only; the PyTorch path is Windows/Linux.
|
||||
Same gating mechanism that currently blocks shipping VoxCPM.
|
||||
|
||||
Neither needs to complete before Phase 1, but both should complete before
|
||||
Phase 4 when user-configurable pipelines surface the differences to end users.
|
||||
Reference in New Issue
Block a user