voicebox/docs/plans/VOICE_IO.md

Voice I/O

Status: Shipping — phases 1, 2, 4, 7 (macOS) complete · 3 partial · 5, 6, 7 (Windows/Linux), 8 pending Touches: backend, Tauri shell, frontend, a new native shim crate Last reviewed: 2026-04-21

Progress

Shipped

Phase 1 — Groundwork. Audio tab retired from the sidebar; its device / channel config lives under Settings. Captures tab is live at /captures with no feature flag.

Phase 2 — Local LLM backend. LLMBackend protocol alongside the existing TTS/STT backends. qwen_llm_backend.py, services/llm.py, routes/llm.py, and a shared model-download / cache pipeline. Qwen3 0.6B / 1.7B / 4B registered and user-selectable via capture_settings.llm_model.

Phase 4 — Captures tab. List + detail view, source badges (dictation / recording / file), retranscribe, refine (flags + model resolved from a server-side capture_settings singleton), delete, and the Play-as-voice dropdown over every profile.

Partial

Phase 3 — In-app voice input. CapturesTab dictates end-to-end via useCaptureRecordingSession, which the Phase 7 floating pill also consumes. Outstanding: a universal mic button on other text inputs (Generate form, profile descriptions, story titles, etc.), and the streaming /transcribe/stream WebSocket — today's flow is a single POST /captures with the complete audio blob.

Phase 7 — External dictation shell (macOS). Both halves shipped on macOS.

Hotkey half:

• tauri/src-tauri/src/chord_engine.rs — pure state machine. Unit tests green.
• tauri/src-tauri/src/hotkey_monitor.rs — rdev-based global listener on a background thread, with set_is_main_thread(false) applied to sidestep the macOS 14+ TSM crash (Narsil/rdev#165). Right-hand-only defaults preserve left-hand Cmd+Option+I devtools.
• Default bindings hardcoded: Cmd+Option (push-to-talk) and Cmd+Option+Space (toggle-to-talk). The PTT → Toggle upgrade transition is preserved — adding Space mid-hold promotes the session without interrupting audio.
• DictateWindow — transparent, always-on-top, borderless 420×64 webview pre-created hidden at app setup. Shows on chord-start, hides on capture-cycle completion. Error state on the pill auto-dismisses and copies-to-clipboard on click.

Paste half (macOS):

• clipboard.rs — NSPasteboard snapshot that walks pasteboardItems and copies every (uti, bytes) pair so multi-type content (images, styled text, file refs) survives the round-trip. save_clipboard, write_text, restore_clipboard, current_change_count.
• synthetic_keys.rs — CGEventPost at the HID tap with the full four-event Cmd+V sequence (Cmd down → V down w/ flag → V up w/ flag → Cmd up).
• focus_capture.rs — AXUIElementCreateSystemWide + AXUIElementCopyAttributeValue(kAXFocusedUIElement) + AXUIElementGetPid, with the AX attribute key CFStrings built at runtime because they're CFSTR macros, not linkable symbols. NSRunningApplication.activateWithOptions: for re-activation.
• accessibility.rs — AXIsProcessTrusted gate.
• paste_final_text command — activate → 120 ms settle → save clip → write text → ⌘V → 400 ms → restore. Skips when focus was in Voicebox itself.
• Focus rides the dictate:start event payload; DictateWindow holds the snapshot in a ref and consume-once-nulls on paste so a late-arriving refine from an earlier session can't misfire.
• Dictation recording no longer hard-caps at 29 s — the limit still applies to voice-profile reference clips.

Outstanding: Windows SendInput / UIAutomation / SetForegroundWindow equivalents, Linux uinput / AT-SPI equivalents (and the Wayland story), first-run Accessibility prompt UI with deep-link to System Settings, direct-injection path for focus-was-inside-Voicebox (step 6 — dictating into our own Generate tab currently falls back to the capture list).

Not started

• Phase 5 — Agent voice output + persona loop. No /speak endpoint, no voicebox.speak MCP tool, no per-agent voice binding, no persona metadata on profiles.
• Phase 6 — STT engine expansion. Only Whisper (mlx_backend.py). Parakeet v3, Qwen3-ASR, Kyutai — all unregistered.
• Phase 8 — Pipeline routing, sinks, long-form. No preset primitive, no MCP sink, no webhook sink, no dual-stream recorder, no summary transform.

Additionally landed (not explicit in the original plan)

These fell out of the Phase 3/4/7 work but deserve their own mention:

• Server-authoritative settings. Singleton capture_settings and generation_settings tables. The client sends nothing but the audio; STT model, refine flags, refine LLM, and the auto-refine flag are all resolved server-side, so sibling Tauri webviews can't go stale.
• Backend audio normalisation. POST /captures transcodes anything librosa can decode (webm/opus, m4a, etc.) to WAV before handing it to whisper, side-stepping miniaudio's format gaps inside mlx-audio.
• Short-recording guard. Sub-300 ms blobs short-circuit client-side so a fumbled chord tap never uploads an empty webm.
• Refinement prompt. Rewritten with firmer anti-chatbot framing and inline examples covering multi-sentence preservation and self-correction.

Near-term outstanding

Called out in recent sessions but not yet in a phase:

• Configurable chord bindings. Pass 2 of the hotkey work — persist push_to_talk_chord / toggle_to_talk_chord in capture_settings, surface a chord-picker UI in CapturesPage, and wire a Tauri update_chord_bindings command so HotkeyMonitor::update_bindings picks up user changes live.
• Generate-tab empty-state explainer. The parallel aside to the Captures explainer described in Product surface → Parallel explainer on the Generate tab. Lands alongside Phase 3's universal mic button so both tabs feel symmetric.

Overview

Voicebox ships the output half of a voice I/O loop: clone a voice, generate speech, apply effects, compose multi-voice projects. The input half — speech to text, dictation, routing — exists today as a single Whisper model wired into the Recording & Transcription panel. This doc proposes making voice input a first-class pillar: more STT engines, a dictation shell (global hotkey, audio capture, paste, streaming), a local LLM backend, and a user-configurable pipeline from captured audio to whatever the user wants to do with it.

Positioning is the key move. Voicebox becomes the local voice I/O layer for humans and AI agents — a local alternative to cloud dictation tools, with the differentiator that we also do TTS and voice cloning. The same app that captures your voice can generate a response in any voice profile you've cloned. "Anything voice is Voicebox."

Positioning shift

Before this plan, Voicebox was "the open-source AI voice cloning studio." Cloning was the headline capability.

After this plan, Voicebox is "the open-source AI voice studio." Cloning is one capability in a broader category that now spans input (STT, dictation), intelligence (local LLM, refinement, persona), output (TTS, cloning, effects, Stories), and routing. The word "cloning" drops out of the top-line descriptor because it's become a feature rather than the thesis.

Competitive frame

Voicebox ends up covering the territory of two separately-funded, separately branded cloud incumbents that operate on opposite sides of the same voice I/O loop:

• ElevenLabs (~$3B+): voice cloning and TTS — the "agents speak" side
• WisprFlow (~$70M raised): voice dictation for agents and power users — the "users talk" side

Both are cloud-only. Voicebox becomes the only local alternative to either, running in one app, with a single model directory and LLM shared between input and output. That bridging — dictation → LLM → TTS with a cloned voice in the middle — is the thing no single incumbent can match, because neither has the other half.

Launch-time copy tasks

These are not engineering tasks but should ride the Phase 4 ship so marketing and positioning stay in sync with the product.

• README.md — drop "cloning" from the top-line descriptor. Add a section that explicitly frames Voicebox as "the open-source local alternative to WisprFlow and ElevenLabs." Competitive framing belongs in the README and on the landing page — not in-app (reads as defensive).
• voicebox.sh landing page — same positioning shift.
• GitHub About / repo topics — swap "voice-cloning" or similar tags for broader "voice-io," "local-tts," "local-stt," etc.
• Release notes — the Phase 4 launch note is the "we're now voice I/O" moment.

Why now

• Cross-platform local dictation is an empty category. The tools people love (Superwhisper, MacWhisper, Aiko) are macOS-only. WisprFlow and Willow are cloud. Our Windows install base is the wedge — first-class Windows support for a local dictation product is genuinely differentiated.
• The STTBackend protocol already exists. The multi-engine registry pattern shipped with TTS makes adding Parakeet v3 and Qwen3-ASR a days-not-weeks effort on the backend side.
• The persona loop — speak to an agent, have it reply in a cloned voice — is a feature only we can ship. Nobody with a dictation product has TTS; nobody with a TTS product has good dictation. The full duplex is ours.
• Agent harnesses already pipe Voicebox TTS into their stacks. Giving those users STT from the same app closes the loop and makes Voicebox the default voice I/O layer for the agentic dev-tool crowd.
• Typing a 2,000-character TTS script is user-hostile. The most immediate internal win is dictating directly into Voicebox's own generation form — speak the script, generate the voice. This dogfoods the whole STT pipeline without touching a single OS-level API.
• Voice-to-voice models are landing. Moshi (Kyutai), GLM-4-Voice, Qwen2.5 Omni, Mini-Omni, Sesame CSM, Spirit LM (Meta) — end-to-end speech LLMs that take audio in and emit audio out are a near-term reality. The pipeline we're building today is the scaffolding they slot into tomorrow.

Non-goals

• Cloud fallback or "bring your own API key" STT/LLM. Local is the product.
• A separate tray-only dictation app. We extend Voicebox, not fork it.
• Replacing the Stories editor with a notes layout. Long-form capture is a preset on top of the pipeline, not a new product surface.
• Real-time translation UI. It can exist as a transform later, but it's not in this plan.
• Full agent orchestration. We provide the voice rails; the agent lives elsewhere and talks to us via the developer API.

Architecture

Three new backend concepts

1. Expanded STT registry. The existing STTBackend protocol abstracts Whisper today. Add:

• Parakeet v3 — 25 languages, very fast, the current quality leader for non-English local STT. Python path via nemo_toolkit or transformers.
• Qwen3-ASR 0.6B int8 — 50+ languages, highest multilingual quality, cross-platform via transformers.
• Kyutai ASR (optional) — streaming-first, small, CPU-friendly. Fills the "CPU-only laptop" tier.

All register via ModelConfig and use the same download, cache, and model management UI we already have for TTS. Zero special-casing.

2. LLMBackend protocol. Mirror of TTSBackend / STTBackend. First implementations are Qwen3 0.6B / 1.7B / 4B running on the same PyTorch + MLX infrastructure we already run. One runtime, one model cache, one GPU-memory story.

Why not llama.cpp or ollama: we already have the dependency surface and the model download UX. A second runtime fragments cache directories and model-status UI. If CPU-only Windows latency becomes a problem we can revisit.

3. Streaming transcribe transport. Add /transcribe/stream as a WebSocket endpoint alongside the existing HTTP /transcribe. Audio frames flow in, partial transcripts stream back. Same FastAPI process, same loaded models. This keeps dictation latency off the per-request JSON-encode critical path and lets us ship real-time partial transcripts later without a protocol change.

The pipeline abstraction

Every captured audio event flows through the same shape: Source → Transforms → Sink(s). Users configure presets that bind a source to a transform chain to one or more sinks.

Source                      Transform                    Sink
──────────────────          ─────────────────            ─────────────────
Hold to speak        ──┐    STT model                    Clipboard + paste
Tap to toggle          │    Refinement LLM               Capture history
Long-form recorder     ├──▶ Persona LLM              ──▶ File on disk
File drop              │    Translation (later)          HTTP webhook
API call (WS / HTTP) ──┘                                 MCP server sink
                                                         TTS loopback (persona)
                                                         Platform sinks (later)

Source → Transform → Sink is internal, dataflow-style vocabulary (same shape as Unix pipes, Apache Beam, Kafka) — not user-facing. The UI surface will use Voicebox-native language (see open questions).

Concrete preset examples this shape enables:

• Dictation — hold-to-speak → Parakeet v3 → light refinement → clipboard + paste + history
• Code prompt — dedicated hotkey → Whisper Turbo → technical-vocab refinement → MCP sink for Claude Code
• Agent voice reply — hold-to-speak → STT → persona LLM → TTS with cloned profile → system audio out
• Long-form capture — dual-stream recorder → chunked STT → summary LLM → markdown file + history

Every user-facing feature collapses into (source + transform chain + sinks). Meeting-style capture isn't a separate product; it's a preset. Competing tools hardcode integrations (Trello, Granola); we make routing user-configurable.

Native shim crate

The parts Tauri doesn't handle cleanly, gathered in one Rust crate with a platform-agnostic API:

• Global hotkey with modifier-only support. Tauri's global-shortcut plugin requires full combos. We need "hold right-cmd" or "hold ctrl" as primitives. On macOS this means a CGEventTap on a background thread with polling fallback for dropped modifier events; on Windows a low-level keyboard hook; on Linux X11 + libinput, with Wayland as a known gap.
• Focus introspection. Query the frontmost app and its focused element via OS accessibility APIs — AXUIElement on macOS, UIAutomation on Windows, AT-SPI on Linux. Check the element's role to decide between a direct injection, a clipboard + paste, and a clipboard-only fallback with a notification. A blind paste that only "works when a text field happens to be focused" is the easy default; we should make the decision deliberately.
• Simulated paste. CGEvent on macOS, SendInput on Windows, uinput / ydotool on Linux. Wayland is the hard case and needs explicit handling.
• Atomic clipboard save/restore. Save all items and all MIME representations before writing our transcript, restore atomically after paste. Pasting a transcript shouldn't clobber a user's in-progress rich-media clipboard.
• Frontmost-window context capture (later). macOS Vision, Windows OCR, Linux tesseract. Optional feature to feed the refinement LLM disambiguation hints from the window being pasted into.

Main process owns this crate. Webview never sees platform differences.

Target-aware delivery

The paste sink adapts to what's in focus. This is a single sink type with branching behavior, not four separate sinks.

Target	Delivery strategy
Focused text field inside Voicebox	Direct React state update via event. No clipboard involved.
Focused text field in another app	Accessibility-verified paste: save clipboard, write transcript, simulate paste, restore clipboard.
No text focus detected	Clipboard only, toast notification ("Transcript copied — no text field focused").
Platform-specific special cases (terminal apps, specific editors)	Per-app overrides where the generic path misbehaves.

Where each concern lives

Concern	Layer
STT / LLM / TTS inference	Python backend
Model downloads, progress, cache	Python backend
Pipeline runner (orchestrates transforms and sinks)	Python backend
Audio capture from mic / system audio	Rust (Tauri side)
Audio streaming over WebSocket to backend	Rust
Global hotkey capture	Rust (native shim crate)
Paste simulation, clipboard save/restore	Rust (native shim crate)
Pipeline preset UI, capture history, settings	React

Model work in Python. OS work in Rust. User config in React.

Product surface

A new tab (and a sidebar reshuffle)

The current sidebar is Generate · Stories · Voices · Effects · Audio · Models · Settings. The existing Audio tab is output-device and channel routing config — infrastructure, not a creative workspace — and the Settings page already has a sub-tab pattern (ServerSettings/: Connection, Models, GPU, Update) that fits it naturally.

Move Audio to a Settings sub-tab. Reclaim the sidebar slot for voice input.

The new tab shows recent captures (audio + transcript paired), active presets, dictation settings, model pickers for STT and LLM. Exact name is an open question.

Sidebar placement: Captures sits at position 3, directly under Stories and above Voices. Creates an "input voice / output voice" adjacency — captured speech is one slot away from the voices you can play it back through, which mirrors the Phase 4 "Play as voice" feature's mental model. Full order: Generate · Stories · Captures · Voices · Effects · Models · Settings.

Parallel explainer on the Generate tab

The Captures settings page gets a "What's different" aside that introduces Voicebox's dictation story. The Generate tab deserves a parallel — first-time users need to be told what voice generation is for in a post-Voice-I/O world, not just handed a text field.

Shape: an empty-state card rendered in the Generate tab when there's no generation history yet, disappearing once the user has generated anything. Teaches without claiming permanent real estate. Parallel bullets to the Captures aside so the two tabs feel like two sides of one product:

• Clone any voice in seconds — a short sample is enough
• Seven engines, 23 languages — creative range, not a single model
• Agent-ready — REST + WebSocket API, one checkbox away from giving any AI agent a voice

This lands in Phase 4 alongside the Captures tab, for visual and thematic symmetry. Not a persistent sidebar — the Generate tab is a workspace and should reclaim its space once the user is producing work.

Archival by default

Every capture saves the original audio alongside the final transcript in a pattern that mirrors data/generations/. Optional retention setting. Free for us — the storage and UI patterns exist today for generations.

Developer API, day one

The WebSocket transcribe endpoint is a first-class public API, documented alongside /generate. Pipeline presets are addressable by ID via /pipelines/{id}/run so agent harnesses and shell scripts can invoke user-configured flows. An MCP server sink ships built-in, so integrations with Claude Code, Cursor, Cline, etc. are one checkbox rather than a custom build.

Agent voice output

Dictation is one half of the loop — user speaks, agent listens. The other half — agent speaks, user hears — is equally load-bearing and deserves a first-class primitive rather than being buried as a TTS loopback sink or a consumer read-aloud button.

The shape is a single new capability: any agent can call Voicebox to speak arbitrary text in a user-configured voice. The same pill that surfaces during dictation surfaces during agent speech, so the user always sees what's coming out of their machine.

MCP tool:  voicebox.speak({ text, profile?, style? })
REST:      POST /speak { text, profile_id?, style? }

Both accept an optional voice profile (defaults to the user's configured default), an optional delivery-style string for engines that support it, play audio through system output, and surface the pill in a speaking state.

Key design points:

• Pill is bidirectional. States expand from recording / transcribing / refining / rest to include speaking — voice profile name, waveform in the profile's color, visible duration. Same floating surface for both directions so users have one mental model.
• Visibility is mandatory. Silent background TTS is a trust hazard. Every agent-initiated speak() surfaces the pill. No headless "TTS daemon" mode.
• Per-source voice policy. Settings let users bind specific MCP clients or API keys to specific voice profiles — Claude Code in "Morgan," Cursor in "Scarlett" — so users can tell which agent is talking without looking.
• Mute + rate limits. One-toggle mute for all agent speech. Per-source rate limits prevent a runaway agent from monologuing.

This primitive is what makes "Voicebox as voice layer for every agent on your machine" a concrete shipping capability rather than marketing language. MCP, ACP, and A2A integrations all slot into it — none of those agent protocols need to know anything about TTS models, GPU placement, or voice profiles. They call speak().

Relationship to the persona loop. The persona loop below is one use of speak() — STT → LLM → speak(llm_reply). Other uses skip STT entirely: a long-running task announcing completion, a notification, an agent proactively asking the user a question. The primitive is deliberately simpler than the persona loop so it can serve both flows from the same API.

Relationship to voice profile samples

A capture and a voice profile sample both hold audio + text, so there's an obvious temptation to unify them. Don't. The metadata and lifecycle differences are real:

	Capture	Voice profile sample
Profile association	Standalone	Bound to one profile
Text field	Raw transcript + optional LLM-refined version	Exact reference_text only
LLM refinement	Often applied	Must not be applied — the reference text must match the audio verbatim or cloning breaks
Volume	Dozens per day	~5 per profile, semi-permanent
Typical content	Whatever the user said	Often scripted phrases for cloning

A unified table would mean nullable profile_id, nullable refined_transcript, nullable reference_text — a fat row that means different things in different states. Not worth the complexity.

What to ship instead: a one-way promote action. Capture → Sample, zero data-model churn. Thin endpoint:

POST /profiles/{id}/samples/from-capture/{capture_id}

Reads the capture's audio path and raw transcript, calls the existing add_sample() service with reference_text pre-filled from the transcript, lets the user edit the reference text in a dialog before saving (transcripts are usually 90% right but cloning wants 100%). The capture stays in the Captures tab untouched — the sample is a copy, not a move.

UI hook: the Captures tab's Send-to menu gains a "Use as voice sample…" option that opens a profile picker (with "+ New voice" for cold starts) and a reference-text confirm dialog.

The inverse direction (sample → capture) we deliberately skip. Samples are often scripted phrases used for cloning and they'd clutter the Captures list without adding value; also a subtle privacy surprise for users who don't expect their sample text browsable alongside real captures.

Audio storage deduplication is a later optimization. Today a promoted capture duplicates the audio file on disk. That's fine. Content-addressable storage (data/audio/<sha256>.wav with refcounting) can come in Phase 8 as housekeeping — it'd let a capture and a sample share one underlying file, but it's not user-visible and not necessary to ship the promote flow.

The persona loop

One flow on top of the speak() primitive: STT → persona LLM → speak(llm_reply). Voice profiles gain optional metadata — a natural-language personality description and default LLM behavior. The LLM runs text through the profile's voice context, then speak() generates TTS with the cloned profile. End-to-end voice-to-voice with a cloned identity transforming the content, not just reading it.

Use cases this unlocks:

• Agents that respond to spoken input in a specific voice
• Interactive character experiences (games, narrative tools, accessibility)
• Speech assistance for people who can't speak in their original voice

The shape — STT + LLM + TTS — also stages us for end-to-end speech LLMs which collapse all three into one transform. See Voice-to-voice readiness below.

Voice-to-voice readiness

The STT → LLM → TTS chain that powers the persona loop is a staged approximation of voice-to-voice. A real end-to-end speech LLM (Moshi, GLM-4-Voice, Qwen2.5 Omni, Mini-Omni, Sesame CSM) replaces the three middle boxes with a single fused transform: audio in, audio out, no text in between. The pipeline shape accommodates this natively — register the model as a single LLMBackend (or a new SpeechLLMBackend if the protocol needs to differ), expose it as a transform type, and the same sinks work unchanged.

Framing this plan as "voice-to-voice scaffolding, with today's models as the staged fallback" is a strong pitch for agent-harness users who are already tracking these models.

Open questions

1. Tab name. Leaning Captures — neutral, extensible across dictation, long-form recordings, and uploaded audio without repainting the tab later. "Dictations" is narrower (office-productivity coded, doesn't fit meeting recordings). "Notes" is the wrong mental model — nobody opens Voicebox to write notes. "Transcriptions" is flat.
2. Refinement vocabulary. The LLM-post-STT step needs a user-facing name. "Refine," "polish," "rewrite," "smart edit" are candidates. "Refinement" in this doc as a placeholder only.
3. Preset primitive. What do we call a user-configured pipeline? "Intent" collides with the existing instruct field on TTS generation. "Flow" is Zapier-coded. "Route" is too networking. Needs its own pass.
4. Persona metadata shape. Does personality live directly on the voice profile, or as a separate persona construct that wraps profile + LLM config? The first is simpler; the second scales better if we later want multiple personas per voice.
5. Long-form capture product surface. Pure preset, or dedicated entry point in the new tab? Leaning preset, but long-form is the feature that most justifies its own landing page.
6. Hotkey primitive naming. Hold-vs-tap needs Voicebox-native phrasing in UI copy. Settings can still use industry-standard terms.

Ordered phases

The v1 prototype deliberately skips the hardest parts of the long-term plan (native OS shim, global hotkeys, paste injection, new STT models). Everything in Phase 1–4 is in-process code using Whisper (which we already ship) and the existing model infra. No CGEvent taps, no SendInput, no clipboard timing. The usual OS-level sprawl of a dictation stack is exactly what we sidestep by starting in-app.

Phase 1 — Groundwork

• Move the Audio tab into a Settings sub-tab (ServerSettings/ gains one more section). Audio is device/channel config, not a creative workspace.
• Reserve the sidebar slot for the new Captures tab (name TBD but leaning Captures — see open questions).
• Gate the Captures tab behind a feature flag so we can merge to main and iterate without shipping half-built UI to users.

Phase 2 — Local LLM backend

LLMBackend protocol alongside TTSBackend / STTBackend. Register Qwen3 0.6B / 1.7B / 4B via ModelConfig. Reuses the HF download path, cache directory, and model management UI. MLX (4-bit community quants) on Apple Silicon, PyTorch (transformers AutoModelForCausalLM) elsewhere, same as our TTS split.

No new runtime. No llama.cpp, no ollama, no fragmented model cache.

Phase 3 — In-app voice input

A universal mic button on every Voicebox text input. Hold, speak, release — text lands in the focused field via direct React state update. No OS APIs involved; Voicebox owns the input.

Marquee use cases:

• Generation form. Dictate a 2,000-character TTS script instead of typing it. This alone justifies the feature.
• Voice profile descriptions. Describe a voice's personality by speaking, which then becomes the input for Phase 4's persona loop.
• Story titles, preset names, any free-text field. Free reuse.

Backend: add /transcribe/stream WebSocket endpoint. Audio frames in, partial transcripts out. Reuses the existing Whisper model in memory. Optionally routes through the LLM from Phase 2 for light refinement.

Phase 4 — Captures tab

Graduates the tab out from behind the feature flag. Shows recent captures (audio + transcript pairs), lets the user replay, re-transcribe with a different model, edit the transcript, and send the output through the LLM. Archival is automatic — every capture saves audio alongside transcript.

Includes the "Play as voice profile" action. This is the simplest version of the persona loop and it lands here for free — no LLM involved, no new backend endpoints, just a Captures-tab button that sends the transcript text to the existing /generate endpoint with a user-selected voice profile and plays the result. Category-defining differentiator from the v1 prototype onward: Superwhisper and WisprFlow cannot do this because they have no TTS. Voicebox can, with one day of frontend wiring.

Keep it aggressively minimal on day one. A capture list, a detail view, a model picker, a Play-as-voice dropdown. Refinement prompt editing, correction dictionaries, per-source overrides — none of that ships here. They become Tier-2 work when someone actually asks for them.

Phase 5 — Agent voice output + persona loop

Two features that together make "Voicebox as the voice layer for every agent on your machine" a shipping reality:

1. speak() primitive. New POST /speak endpoint and voicebox.speak MCP tool. Any agent calls Voicebox to speak arbitrary text in a user-configured voice; the pill surfaces in a speaking state. Settings UI for default voice, per-agent voice binding (Claude Code → Morgan, Cursor → Scarlett), and a global mute.
2. Persona loop. Extends speak() with an LLM step — STT → persona LLM → speak(llm_reply). Voice profiles gain optional personality metadata and default LLM behavior. End-to-end voice-to-voice with a cloned identity transforming the content, not just reading it.

Phase 4 demoed the user-initiated direction of the loop (Play as voice). This phase ships the agent-initiated direction, which is the category-defining capability and the pitch that lands with agent-harness users. The persona loop is one flow on top of the speak() primitive — notifications, proactive agent questions, and task-completion announcements all use speak() directly without the LLM in the middle.

Launchable headline moment for the "local voice I/O" positioning.

Phase 6 — STT engine expansion

Parakeet v3 and Qwen3-ASR register as additional STTBackend implementations. Optional: Kyutai ASR. Multilingual coverage upgrades (50+ languages). Whisper stays as the sensible default.

Deferred to here because Whisper is already good enough for v1 and the model picker UI exists. Adding rows to it doesn't change the product shape.

Phase 7 — External dictation shell

Native shim crate (global hotkey with modifier-only support, focus introspection via OS accessibility APIs, paste simulation, atomic clipboard save/restore). Tauri-side audio capture streams to the same WebSocket endpoint Phase 3 already ships. Paste sink with target-aware delivery.

This is the feel-good phase. It's also the riskiest: paste timing, hotkey reliability, and cross-platform focus detection are all engineering problems that have to be nailed or the product doesn't work. Phase 3's success derisks the backend plumbing before we start it.

Phase 8 — Pipeline routing, sinks, long-form

Multiple source types, user-configurable transform chains, multiple sinks per preset. MCP server sink (the agent-harness play). HTTP webhook sink. File sink. Developer-facing /pipelines/{id}/run endpoint. Preset editor UI in the Captures tab.

Dual-stream recorder (mic + system audio) as a source type. Chunked STT transform with overlap-based deduplication. Summary LLM transform. Long-form capture becomes a preset, not a new tab.

Platform-specific sinks (Apple Notes on macOS, Obsidian, etc.) as opt-in integrations behind the generic sink interface.

Architectural prerequisites

Two pieces of existing docs/PROJECT_STATUS.md work become load-bearing here:

• Platform support tiers (#420, PR #465). Native shim capabilities vary by platform — Wayland paste is worse than X11, Windows system-audio capture has edge cases, frontmost-window OCR is platform-gated. Tier definitions let us ship confidently with honest user-facing expectations.
• Platform gating on ModelConfig (bottleneck #6 in PROJECT_STATUS). Parakeet's Core ML path is Apple-only; the PyTorch path is Windows/Linux. Same gating mechanism that currently blocks shipping VoxCPM.

Neither needs to complete before Phase 1, but both should complete before Phase 4 when user-configurable pipelines surface the differences to end users.