mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-18 06:10:43 -07:00
feat(capture): dictation, personalities, 0.5.0
Ships the Capture release end to end. Global-hotkey dictation with synthetic paste into the focused app on macOS and Windows, an on-screen pill across recording / transcribing / refining, customizable push-to- talk and toggle chords, and an accessibility-permission prompt scoped to Settings → Captures with inline re-check feedback. Voice profiles gain optional personalities that power compose / rewrite / respond actions via a local Qwen3 LLM — shared with refinement, so there is one local LLM in the app, not two. Refinement hardened with deterministic Whisper-loop collapse before the LLM sees the transcript, per-capture flag snapshots for re-runs, and a ten-transcript evaluation harness across every bundled refinement size. Version bump 0.4.5 → 0.5.0. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
James Pine
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
ed2eec591a
commit
87c582ad54
@@ -0,0 +1,675 @@
|
||||
# Voice I/O
|
||||
|
||||
**Status:** Shipping — phases 1, 2, 4, 7 (macOS) complete · 3 partial · 5, 6, 7 (Windows/Linux), 8 pending
|
||||
**Touches:** backend, Tauri shell, frontend, a new native shim crate
|
||||
**Last reviewed:** 2026-04-21
|
||||
|
||||
## Progress
|
||||
|
||||
### Shipped
|
||||
|
||||
**Phase 1 — Groundwork.** Audio tab retired from the sidebar; its device / channel
|
||||
config lives under Settings. Captures tab is live at `/captures` with no feature
|
||||
flag.
|
||||
|
||||
**Phase 2 — Local LLM backend.** `LLMBackend` protocol alongside the existing
|
||||
TTS/STT backends. `qwen_llm_backend.py`, `services/llm.py`, `routes/llm.py`, and
|
||||
a shared model-download / cache pipeline. Qwen3 0.6B / 1.7B / 4B registered and
|
||||
user-selectable via `capture_settings.llm_model`.
|
||||
|
||||
**Phase 4 — Captures tab.** List + detail view, source badges (dictation /
|
||||
recording / file), retranscribe, refine (flags + model resolved from a
|
||||
server-side `capture_settings` singleton), delete, and the Play-as-voice
|
||||
dropdown over every profile.
|
||||
|
||||
### Partial
|
||||
|
||||
**Phase 3 — In-app voice input.** `CapturesTab` dictates end-to-end via
|
||||
`useCaptureRecordingSession`, which the Phase 7 floating pill also consumes.
|
||||
Outstanding: a universal mic button on other text inputs (Generate form,
|
||||
profile descriptions, story titles, etc.), and the streaming
|
||||
`/transcribe/stream` WebSocket — today's flow is a single `POST /captures`
|
||||
with the complete audio blob.
|
||||
|
||||
**Phase 7 — External dictation shell (macOS).** Both halves shipped on macOS.
|
||||
|
||||
Hotkey half:
|
||||
|
||||
- `tauri/src-tauri/src/chord_engine.rs` — pure state machine. Unit tests green.
|
||||
- `tauri/src-tauri/src/hotkey_monitor.rs` — `rdev`-based global listener on a
|
||||
background thread, with `set_is_main_thread(false)` applied to sidestep the
|
||||
macOS 14+ TSM crash ([Narsil/rdev#165](https://github.com/Narsil/rdev/issues/165)).
|
||||
Right-hand-only defaults preserve left-hand Cmd+Option+I devtools.
|
||||
- Default bindings hardcoded: `Cmd+Option` (push-to-talk) and
|
||||
`Cmd+Option+Space` (toggle-to-talk). The PTT → Toggle upgrade transition is
|
||||
preserved — adding Space mid-hold promotes the session without interrupting
|
||||
audio.
|
||||
- `DictateWindow` — transparent, always-on-top, borderless 420×64 webview
|
||||
pre-created hidden at app setup. Shows on chord-start, hides on
|
||||
capture-cycle completion. Error state on the pill auto-dismisses and
|
||||
copies-to-clipboard on click.
|
||||
|
||||
Paste half (macOS):
|
||||
|
||||
- `clipboard.rs` — `NSPasteboard` snapshot that walks `pasteboardItems` and
|
||||
copies every `(uti, bytes)` pair so multi-type content (images, styled
|
||||
text, file refs) survives the round-trip. `save_clipboard`,
|
||||
`write_text`, `restore_clipboard`, `current_change_count`.
|
||||
- `synthetic_keys.rs` — `CGEventPost` at the HID tap with the full four-event
|
||||
Cmd+V sequence (Cmd down → V down w/ flag → V up w/ flag → Cmd up).
|
||||
- `focus_capture.rs` — `AXUIElementCreateSystemWide` +
|
||||
`AXUIElementCopyAttributeValue(kAXFocusedUIElement)` +
|
||||
`AXUIElementGetPid`, with the AX attribute key CFStrings built at
|
||||
runtime because they're CFSTR macros, not linkable symbols.
|
||||
`NSRunningApplication.activateWithOptions:` for re-activation.
|
||||
- `accessibility.rs` — `AXIsProcessTrusted` gate.
|
||||
- `paste_final_text` command — activate → 120 ms settle → save clip →
|
||||
write text → ⌘V → 400 ms → restore. Skips when focus was in Voicebox
|
||||
itself.
|
||||
- Focus rides the `dictate:start` event payload; `DictateWindow` holds the
|
||||
snapshot in a ref and consume-once-nulls on paste so a late-arriving
|
||||
refine from an earlier session can't misfire.
|
||||
- Dictation recording no longer hard-caps at 29 s — the limit still
|
||||
applies to voice-profile reference clips.
|
||||
|
||||
Outstanding: Windows `SendInput` / UIAutomation / `SetForegroundWindow`
|
||||
equivalents, Linux `uinput` / AT-SPI equivalents (and the Wayland story),
|
||||
first-run Accessibility prompt UI with deep-link to System Settings,
|
||||
direct-injection path for focus-was-inside-Voicebox (step 6 — dictating
|
||||
into our own Generate tab currently falls back to the capture list).
|
||||
|
||||
### Not started
|
||||
|
||||
- **Phase 5 — Agent voice output + persona loop.** No `/speak` endpoint, no
|
||||
`voicebox.speak` MCP tool, no per-agent voice binding, no persona metadata
|
||||
on profiles.
|
||||
- **Phase 6 — STT engine expansion.** Only Whisper (`mlx_backend.py`).
|
||||
Parakeet v3, Qwen3-ASR, Kyutai — all unregistered.
|
||||
- **Phase 8 — Pipeline routing, sinks, long-form.** No preset primitive, no
|
||||
MCP sink, no webhook sink, no dual-stream recorder, no summary transform.
|
||||
|
||||
### Additionally landed (not explicit in the original plan)
|
||||
|
||||
These fell out of the Phase 3/4/7 work but deserve their own mention:
|
||||
|
||||
- **Server-authoritative settings.** Singleton `capture_settings` and
|
||||
`generation_settings` tables. The client sends nothing but the audio; STT
|
||||
model, refine flags, refine LLM, and the auto-refine flag are all resolved
|
||||
server-side, so sibling Tauri webviews can't go stale.
|
||||
- **Backend audio normalisation.** `POST /captures` transcodes anything
|
||||
librosa can decode (webm/opus, m4a, etc.) to WAV before handing it to
|
||||
whisper, side-stepping miniaudio's format gaps inside mlx-audio.
|
||||
- **Short-recording guard.** Sub-300 ms blobs short-circuit client-side so a
|
||||
fumbled chord tap never uploads an empty webm.
|
||||
- **Refinement prompt.** Rewritten with firmer anti-chatbot framing and
|
||||
inline examples covering multi-sentence preservation and self-correction.
|
||||
|
||||
### Near-term outstanding
|
||||
|
||||
Called out in recent sessions but not yet in a phase:
|
||||
|
||||
- **Configurable chord bindings.** Pass 2 of the hotkey work — persist
|
||||
`push_to_talk_chord` / `toggle_to_talk_chord` in `capture_settings`,
|
||||
surface a chord-picker UI in `CapturesPage`, and wire a Tauri
|
||||
`update_chord_bindings` command so `HotkeyMonitor::update_bindings` picks
|
||||
up user changes live.
|
||||
- **Generate-tab empty-state explainer.** The parallel aside to the Captures
|
||||
explainer described in *Product surface → Parallel explainer on the
|
||||
Generate tab*. Lands alongside Phase 3's universal mic button so both tabs
|
||||
feel symmetric.
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox ships the output half of a voice I/O loop: clone a voice, generate
|
||||
speech, apply effects, compose multi-voice projects. The input half — speech to
|
||||
text, dictation, routing — exists today as a single Whisper model wired into the
|
||||
Recording & Transcription panel. This doc proposes making voice *input* a
|
||||
first-class pillar: more STT engines, a dictation shell (global hotkey, audio
|
||||
capture, paste, streaming), a local LLM backend, and a user-configurable
|
||||
pipeline from captured audio to whatever the user wants to do with it.
|
||||
|
||||
Positioning is the key move. **Voicebox becomes the local voice I/O layer for
|
||||
humans and AI agents** — a local alternative to cloud dictation tools, with the
|
||||
differentiator that we also do TTS and voice cloning. The same app that
|
||||
captures your voice can generate a response in any voice profile you've
|
||||
cloned. "Anything voice is Voicebox."
|
||||
|
||||
### Positioning shift
|
||||
|
||||
Before this plan, Voicebox was **"the open-source AI voice cloning studio."**
|
||||
Cloning was the headline capability.
|
||||
|
||||
After this plan, Voicebox is **"the open-source AI voice studio."** Cloning is
|
||||
one capability in a broader category that now spans input (STT, dictation),
|
||||
intelligence (local LLM, refinement, persona), output (TTS, cloning, effects,
|
||||
Stories), and routing. The word "cloning" drops out of the top-line descriptor
|
||||
because it's become a feature rather than the thesis.
|
||||
|
||||
### Competitive frame
|
||||
|
||||
Voicebox ends up covering the territory of two separately-funded, separately
|
||||
branded cloud incumbents that operate on opposite sides of the same voice I/O
|
||||
loop:
|
||||
|
||||
- **ElevenLabs** (~$3B+): voice cloning and TTS — the "agents speak" side
|
||||
- **WisprFlow** (~$70M raised): voice dictation for agents and power users —
|
||||
the "users talk" side
|
||||
|
||||
Both are cloud-only. Voicebox becomes the only local alternative to either,
|
||||
running in one app, with a single model directory and LLM shared between input
|
||||
and output. That bridging — dictation → LLM → TTS with a cloned voice in the
|
||||
middle — is the thing no single incumbent can match, because neither has the
|
||||
other half.
|
||||
|
||||
### Launch-time copy tasks
|
||||
|
||||
These are not engineering tasks but should ride the Phase 4 ship so marketing
|
||||
and positioning stay in sync with the product.
|
||||
|
||||
- **README.md** — drop "cloning" from the top-line descriptor. Add a section
|
||||
that explicitly frames Voicebox as "the open-source local alternative to
|
||||
WisprFlow and ElevenLabs." Competitive framing belongs in the README and on
|
||||
the landing page — not in-app (reads as defensive).
|
||||
- **voicebox.sh landing page** — same positioning shift.
|
||||
- **GitHub About / repo topics** — swap "voice-cloning" or similar tags for
|
||||
broader "voice-io," "local-tts," "local-stt," etc.
|
||||
- **Release notes** — the Phase 4 launch note is the "we're now voice I/O" moment.
|
||||
|
||||
## Why now
|
||||
|
||||
- Cross-platform local dictation is an empty category. The tools people love
|
||||
(Superwhisper, MacWhisper, Aiko) are macOS-only. WisprFlow and
|
||||
Willow are cloud. Our Windows install base is the wedge — first-class Windows
|
||||
support for a local dictation product is genuinely differentiated.
|
||||
- The `STTBackend` protocol already exists. The multi-engine registry pattern
|
||||
shipped with TTS makes adding Parakeet v3 and Qwen3-ASR a days-not-weeks
|
||||
effort on the backend side.
|
||||
- The **persona loop** — speak to an agent, have it reply in a cloned voice —
|
||||
is a feature only we can ship. Nobody with a dictation product has TTS; nobody
|
||||
with a TTS product has good dictation. The full duplex is ours.
|
||||
- Agent harnesses already pipe Voicebox TTS into their stacks. Giving those
|
||||
users STT from the same app closes the loop and makes Voicebox the default
|
||||
voice I/O layer for the agentic dev-tool crowd.
|
||||
- **Typing a 2,000-character TTS script is user-hostile.** The most immediate
|
||||
internal win is dictating directly into Voicebox's own generation form —
|
||||
speak the script, generate the voice. This dogfoods the whole STT pipeline
|
||||
without touching a single OS-level API.
|
||||
- **Voice-to-voice models are landing.** Moshi (Kyutai), GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM, Spirit LM (Meta) — end-to-end speech LLMs that
|
||||
take audio in and emit audio out are a near-term reality. The pipeline we're
|
||||
building today is the scaffolding they slot into tomorrow.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Cloud fallback or "bring your own API key" STT/LLM. Local is the product.
|
||||
- A separate tray-only dictation app. We extend Voicebox, not fork it.
|
||||
- Replacing the Stories editor with a notes layout. Long-form capture is a
|
||||
preset on top of the pipeline, not a new product surface.
|
||||
- Real-time translation UI. It can exist as a transform later, but it's not in
|
||||
this plan.
|
||||
- Full agent orchestration. We provide the voice rails; the agent lives
|
||||
elsewhere and talks to us via the developer API.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Three new backend concepts
|
||||
|
||||
**1. Expanded STT registry.** The existing `STTBackend` protocol abstracts
|
||||
Whisper today. Add:
|
||||
|
||||
- **Parakeet v3** — 25 languages, very fast, the current quality leader for
|
||||
non-English local STT. Python path via `nemo_toolkit` or `transformers`.
|
||||
- **Qwen3-ASR 0.6B int8** — 50+ languages, highest multilingual quality,
|
||||
cross-platform via `transformers`.
|
||||
- **Kyutai ASR** *(optional)* — streaming-first, small, CPU-friendly. Fills the
|
||||
"CPU-only laptop" tier.
|
||||
|
||||
All register via `ModelConfig` and use the same download, cache, and model
|
||||
management UI we already have for TTS. Zero special-casing.
|
||||
|
||||
**2. `LLMBackend` protocol.** Mirror of `TTSBackend` / `STTBackend`. First
|
||||
implementations are Qwen3 0.6B / 1.7B / 4B running on the same PyTorch + MLX
|
||||
infrastructure we already run. One runtime, one model cache, one GPU-memory
|
||||
story.
|
||||
|
||||
Why not `llama.cpp` or `ollama`: we already have the dependency surface and the
|
||||
model download UX. A second runtime fragments cache directories and model-status
|
||||
UI. If CPU-only Windows latency becomes a problem we can revisit.
|
||||
|
||||
**3. Streaming transcribe transport.** Add `/transcribe/stream` as a WebSocket
|
||||
endpoint alongside the existing HTTP `/transcribe`. Audio frames flow in,
|
||||
partial transcripts stream back. Same FastAPI process, same loaded models. This
|
||||
keeps dictation latency off the per-request JSON-encode critical path and lets
|
||||
us ship real-time partial transcripts later without a protocol change.
|
||||
|
||||
### The pipeline abstraction
|
||||
|
||||
Every captured audio event flows through the same shape:
|
||||
**Source → Transforms → Sink(s)**. Users configure presets that bind a source
|
||||
to a transform chain to one or more sinks.
|
||||
|
||||
```
|
||||
Source Transform Sink
|
||||
────────────────── ───────────────── ─────────────────
|
||||
Hold to speak ──┐ STT model Clipboard + paste
|
||||
Tap to toggle │ Refinement LLM Capture history
|
||||
Long-form recorder ├──▶ Persona LLM ──▶ File on disk
|
||||
File drop │ Translation (later) HTTP webhook
|
||||
API call (WS / HTTP) ──┘ MCP server sink
|
||||
TTS loopback (persona)
|
||||
Platform sinks (later)
|
||||
```
|
||||
|
||||
`Source → Transform → Sink` is internal, dataflow-style vocabulary (same shape
|
||||
as Unix pipes, Apache Beam, Kafka) — not user-facing. The UI surface will use
|
||||
Voicebox-native language (see open questions).
|
||||
|
||||
Concrete preset examples this shape enables:
|
||||
|
||||
- **Dictation** — hold-to-speak → Parakeet v3 → light refinement → clipboard + paste + history
|
||||
- **Code prompt** — dedicated hotkey → Whisper Turbo → technical-vocab refinement → MCP sink for Claude Code
|
||||
- **Agent voice reply** — hold-to-speak → STT → persona LLM → TTS with cloned profile → system audio out
|
||||
- **Long-form capture** — dual-stream recorder → chunked STT → summary LLM → markdown file + history
|
||||
|
||||
Every user-facing feature collapses into (source + transform chain + sinks).
|
||||
Meeting-style capture isn't a separate product; it's a preset. Competing tools
|
||||
hardcode integrations (Trello, Granola); we make routing user-configurable.
|
||||
|
||||
### Native shim crate
|
||||
|
||||
The parts Tauri doesn't handle cleanly, gathered in one Rust crate with a
|
||||
platform-agnostic API:
|
||||
|
||||
- **Global hotkey with modifier-only support.** Tauri's `global-shortcut`
|
||||
plugin requires full combos. We need "hold right-cmd" or "hold ctrl" as
|
||||
primitives. On macOS this means a CGEventTap on a background thread with
|
||||
polling fallback for dropped modifier events; on Windows a low-level keyboard
|
||||
hook; on Linux X11 + libinput, with Wayland as a known gap.
|
||||
- **Focus introspection.** Query the frontmost app and its focused element via
|
||||
OS accessibility APIs — `AXUIElement` on macOS, UIAutomation on Windows,
|
||||
AT-SPI on Linux. Check the element's role to decide between a direct
|
||||
injection, a clipboard + paste, and a clipboard-only fallback with a
|
||||
notification. A blind paste that only "works when a text field happens to
|
||||
be focused" is the easy default; we should make the decision deliberately.
|
||||
- **Simulated paste.** CGEvent on macOS, SendInput on Windows, uinput / ydotool
|
||||
on Linux. Wayland is the hard case and needs explicit handling.
|
||||
- **Atomic clipboard save/restore.** Save *all* items and *all* MIME
|
||||
representations before writing our transcript, restore atomically after
|
||||
paste. Pasting a transcript shouldn't clobber a user's in-progress rich-media
|
||||
clipboard.
|
||||
- **Frontmost-window context capture** *(later).* macOS Vision, Windows OCR,
|
||||
Linux tesseract. Optional feature to feed the refinement LLM disambiguation
|
||||
hints from the window being pasted into.
|
||||
|
||||
Main process owns this crate. Webview never sees platform differences.
|
||||
|
||||
### Target-aware delivery
|
||||
|
||||
The paste sink adapts to what's in focus. This is a single sink type with
|
||||
branching behavior, not four separate sinks.
|
||||
|
||||
| Target | Delivery strategy |
|
||||
|---|---|
|
||||
| Focused text field inside Voicebox | Direct React state update via event. No clipboard involved. |
|
||||
| Focused text field in another app | Accessibility-verified paste: save clipboard, write transcript, simulate paste, restore clipboard. |
|
||||
| No text focus detected | Clipboard only, toast notification ("Transcript copied — no text field focused"). |
|
||||
| Platform-specific special cases (terminal apps, specific editors) | Per-app overrides where the generic path misbehaves. |
|
||||
|
||||
### Where each concern lives
|
||||
|
||||
| Concern | Layer |
|
||||
|---|---|
|
||||
| STT / LLM / TTS inference | Python backend |
|
||||
| Model downloads, progress, cache | Python backend |
|
||||
| Pipeline runner (orchestrates transforms and sinks) | Python backend |
|
||||
| Audio capture from mic / system audio | Rust (Tauri side) |
|
||||
| Audio streaming over WebSocket to backend | Rust |
|
||||
| Global hotkey capture | Rust (native shim crate) |
|
||||
| Paste simulation, clipboard save/restore | Rust (native shim crate) |
|
||||
| Pipeline preset UI, capture history, settings | React |
|
||||
|
||||
Model work in Python. OS work in Rust. User config in React.
|
||||
|
||||
## Product surface
|
||||
|
||||
### A new tab (and a sidebar reshuffle)
|
||||
|
||||
The current sidebar is `Generate · Stories · Voices · Effects · Audio · Models ·
|
||||
Settings`. The existing Audio tab is output-device and channel routing
|
||||
config — infrastructure, not a creative workspace — and the Settings page
|
||||
already has a sub-tab pattern (`ServerSettings/`: Connection, Models, GPU,
|
||||
Update) that fits it naturally.
|
||||
|
||||
**Move Audio to a Settings sub-tab. Reclaim the sidebar slot for voice input.**
|
||||
|
||||
The new tab shows recent captures (audio + transcript paired), active presets,
|
||||
dictation settings, model pickers for STT and LLM. Exact name is an open
|
||||
question.
|
||||
|
||||
**Sidebar placement:** Captures sits at position 3, directly under Stories and
|
||||
above Voices. Creates an "input voice / output voice" adjacency — captured
|
||||
speech is one slot away from the voices you can play it back through, which
|
||||
mirrors the Phase 4 "Play as voice" feature's mental model. Full order:
|
||||
Generate · Stories · Captures · Voices · Effects · Models · Settings.
|
||||
|
||||
### Parallel explainer on the Generate tab
|
||||
|
||||
The Captures settings page gets a "What's different" aside that introduces
|
||||
Voicebox's dictation story. The Generate tab deserves a parallel — first-time
|
||||
users need to be told what voice generation is *for* in a post-Voice-I/O
|
||||
world, not just handed a text field.
|
||||
|
||||
Shape: an **empty-state card** rendered in the Generate tab when there's no
|
||||
generation history yet, disappearing once the user has generated anything.
|
||||
Teaches without claiming permanent real estate. Parallel bullets to the
|
||||
Captures aside so the two tabs feel like two sides of one product:
|
||||
|
||||
- **Clone any voice in seconds** — a short sample is enough
|
||||
- **Seven engines, 23 languages** — creative range, not a single model
|
||||
- **Agent-ready** — REST + WebSocket API, one checkbox away from giving any
|
||||
AI agent a voice
|
||||
|
||||
This lands in Phase 4 alongside the Captures tab, for visual and thematic
|
||||
symmetry. Not a persistent sidebar — the Generate tab is a workspace and
|
||||
should reclaim its space once the user is producing work.
|
||||
|
||||
### Archival by default
|
||||
|
||||
Every capture saves the original audio alongside the final transcript in a
|
||||
pattern that mirrors `data/generations/`. Optional retention setting. Free for
|
||||
us — the storage and UI patterns exist today for generations.
|
||||
|
||||
### Developer API, day one
|
||||
|
||||
The WebSocket transcribe endpoint is a first-class public API, documented
|
||||
alongside `/generate`. Pipeline presets are addressable by ID via
|
||||
`/pipelines/{id}/run` so agent harnesses and shell scripts can invoke
|
||||
user-configured flows. An MCP server sink ships built-in, so integrations with
|
||||
Claude Code, Cursor, Cline, etc. are one checkbox rather than a custom build.
|
||||
|
||||
### Agent voice output
|
||||
|
||||
Dictation is one half of the loop — user speaks, agent listens. The other half
|
||||
— agent speaks, user hears — is equally load-bearing and deserves a
|
||||
first-class primitive rather than being buried as a TTS loopback sink or a
|
||||
consumer read-aloud button.
|
||||
|
||||
The shape is a single new capability: any agent can call Voicebox to speak
|
||||
arbitrary text in a user-configured voice. The same pill that surfaces during
|
||||
dictation surfaces during agent speech, so the user always sees what's coming
|
||||
out of their machine.
|
||||
|
||||
```
|
||||
MCP tool: voicebox.speak({ text, profile?, style? })
|
||||
REST: POST /speak { text, profile_id?, style? }
|
||||
```
|
||||
|
||||
Both accept an optional voice profile (defaults to the user's configured
|
||||
default), an optional delivery-style string for engines that support it, play
|
||||
audio through system output, and surface the pill in a `speaking` state.
|
||||
|
||||
**Key design points:**
|
||||
|
||||
- **Pill is bidirectional.** States expand from `recording / transcribing /
|
||||
refining / rest` to include `speaking` — voice profile name, waveform in
|
||||
the profile's color, visible duration. Same floating surface for both
|
||||
directions so users have one mental model.
|
||||
- **Visibility is mandatory.** Silent background TTS is a trust hazard. Every
|
||||
agent-initiated `speak()` surfaces the pill. No headless "TTS daemon" mode.
|
||||
- **Per-source voice policy.** Settings let users bind specific MCP clients or
|
||||
API keys to specific voice profiles — Claude Code in "Morgan," Cursor in
|
||||
"Scarlett" — so users can tell which agent is talking without looking.
|
||||
- **Mute + rate limits.** One-toggle mute for all agent speech. Per-source
|
||||
rate limits prevent a runaway agent from monologuing.
|
||||
|
||||
This primitive is what makes "Voicebox as voice layer for every agent on your
|
||||
machine" a concrete shipping capability rather than marketing language. MCP,
|
||||
ACP, and A2A integrations all slot into it — none of those agent protocols
|
||||
need to know anything about TTS models, GPU placement, or voice profiles.
|
||||
They call `speak()`.
|
||||
|
||||
**Relationship to the persona loop.** The persona loop below is *one* use of
|
||||
`speak()` — STT → LLM → `speak(llm_reply)`. Other uses skip STT entirely: a
|
||||
long-running task announcing completion, a notification, an agent proactively
|
||||
asking the user a question. The primitive is deliberately simpler than the
|
||||
persona loop so it can serve both flows from the same API.
|
||||
|
||||
### Relationship to voice profile samples
|
||||
|
||||
A capture and a voice profile sample both hold `audio + text`, so there's an
|
||||
obvious temptation to unify them. Don't. The metadata and lifecycle
|
||||
differences are real:
|
||||
|
||||
| | Capture | Voice profile sample |
|
||||
|---|---|---|
|
||||
| Profile association | Standalone | Bound to one profile |
|
||||
| Text field | Raw transcript + optional LLM-refined version | Exact `reference_text` only |
|
||||
| LLM refinement | Often applied | Must not be applied — the reference text must match the audio verbatim or cloning breaks |
|
||||
| Volume | Dozens per day | ~5 per profile, semi-permanent |
|
||||
| Typical content | Whatever the user said | Often scripted phrases for cloning |
|
||||
|
||||
A unified table would mean nullable `profile_id`, nullable `refined_transcript`,
|
||||
nullable `reference_text` — a fat row that means different things in different
|
||||
states. Not worth the complexity.
|
||||
|
||||
**What to ship instead: a one-way promote action.** Capture → Sample, zero
|
||||
data-model churn. Thin endpoint:
|
||||
|
||||
```
|
||||
POST /profiles/{id}/samples/from-capture/{capture_id}
|
||||
```
|
||||
|
||||
Reads the capture's audio path and raw transcript, calls the existing
|
||||
`add_sample()` service with `reference_text` pre-filled from the transcript,
|
||||
lets the user edit the reference text in a dialog before saving (transcripts
|
||||
are usually 90% right but cloning wants 100%). The capture stays in the
|
||||
Captures tab untouched — the sample is a copy, not a move.
|
||||
|
||||
UI hook: the Captures tab's Send-to menu gains a **"Use as voice sample…"**
|
||||
option that opens a profile picker (with "+ New voice" for cold starts) and a
|
||||
reference-text confirm dialog.
|
||||
|
||||
The inverse direction (sample → capture) we deliberately skip. Samples are
|
||||
often scripted phrases used for cloning and they'd clutter the Captures list
|
||||
without adding value; also a subtle privacy surprise for users who don't
|
||||
expect their sample text browsable alongside real captures.
|
||||
|
||||
**Audio storage deduplication is a later optimization.** Today a promoted
|
||||
capture duplicates the audio file on disk. That's fine. Content-addressable
|
||||
storage (`data/audio/<sha256>.wav` with refcounting) can come in Phase 8 as
|
||||
housekeeping — it'd let a capture and a sample share one underlying file, but
|
||||
it's not user-visible and not necessary to ship the promote flow.
|
||||
|
||||
### The persona loop
|
||||
|
||||
One flow on top of the `speak()` primitive: STT → persona LLM →
|
||||
`speak(llm_reply)`. Voice profiles gain optional metadata — a natural-language
|
||||
personality description and default LLM behavior. The LLM runs text through
|
||||
the profile's voice context, then `speak()` generates TTS with the cloned
|
||||
profile. End-to-end voice-to-voice with a cloned identity transforming the
|
||||
content, not just reading it.
|
||||
|
||||
Use cases this unlocks:
|
||||
|
||||
- Agents that respond to spoken input in a specific voice
|
||||
- Interactive character experiences (games, narrative tools, accessibility)
|
||||
- Speech assistance for people who can't speak in their original voice
|
||||
|
||||
The shape — STT + LLM + TTS — also stages us for end-to-end speech LLMs which
|
||||
collapse all three into one transform. See *Voice-to-voice readiness* below.
|
||||
|
||||
### Voice-to-voice readiness
|
||||
|
||||
The STT → LLM → TTS chain that powers the persona loop is a staged approximation
|
||||
of voice-to-voice. A real end-to-end speech LLM (Moshi, GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM) replaces the three middle boxes with a single
|
||||
fused transform: audio in, audio out, no text in between. The pipeline shape
|
||||
accommodates this natively — register the model as a single `LLMBackend` (or
|
||||
a new `SpeechLLMBackend` if the protocol needs to differ), expose it as a
|
||||
transform type, and the same sinks work unchanged.
|
||||
|
||||
Framing this plan as "voice-to-voice scaffolding, with today's models as the
|
||||
staged fallback" is a strong pitch for agent-harness users who are already
|
||||
tracking these models.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Tab name.** Leaning **Captures** — neutral, extensible across dictation,
|
||||
long-form recordings, and uploaded audio without repainting the tab later.
|
||||
"Dictations" is narrower (office-productivity coded, doesn't fit meeting
|
||||
recordings). "Notes" is the wrong mental model — nobody opens Voicebox to
|
||||
write notes. "Transcriptions" is flat.
|
||||
2. **Refinement vocabulary.** The LLM-post-STT step needs a user-facing name.
|
||||
"Refine," "polish," "rewrite," "smart edit" are candidates. "Refinement" in
|
||||
this doc as a placeholder only.
|
||||
3. **Preset primitive.** What do we call a user-configured pipeline? "Intent"
|
||||
collides with the existing `instruct` field on TTS generation. "Flow" is
|
||||
Zapier-coded. "Route" is too networking. Needs its own pass.
|
||||
4. **Persona metadata shape.** Does personality live directly on the voice
|
||||
profile, or as a separate persona construct that wraps profile + LLM config?
|
||||
The first is simpler; the second scales better if we later want multiple
|
||||
personas per voice.
|
||||
5. **Long-form capture product surface.** Pure preset, or dedicated entry point
|
||||
in the new tab? Leaning preset, but long-form is the feature that most
|
||||
justifies its own landing page.
|
||||
6. **Hotkey primitive naming.** Hold-vs-tap needs Voicebox-native phrasing in
|
||||
UI copy. Settings can still use industry-standard terms.
|
||||
|
||||
## Ordered phases
|
||||
|
||||
The v1 prototype deliberately skips the hardest parts of the long-term plan
|
||||
(native OS shim, global hotkeys, paste injection, new STT models). Everything
|
||||
in Phase 1–4 is in-process code using Whisper (which we already ship) and the
|
||||
existing model infra. No CGEvent taps, no SendInput, no clipboard timing.
|
||||
The usual OS-level sprawl of a dictation stack is exactly what we sidestep
|
||||
by starting in-app.
|
||||
|
||||
### Phase 1 — Groundwork
|
||||
|
||||
- Move the Audio tab into a Settings sub-tab (`ServerSettings/` gains one
|
||||
more section). Audio is device/channel config, not a creative workspace.
|
||||
- Reserve the sidebar slot for the new Captures tab (name TBD but leaning
|
||||
Captures — see open questions).
|
||||
- Gate the Captures tab behind a feature flag so we can merge to `main` and
|
||||
iterate without shipping half-built UI to users.
|
||||
|
||||
### Phase 2 — Local LLM backend
|
||||
|
||||
`LLMBackend` protocol alongside `TTSBackend` / `STTBackend`. Register Qwen3
|
||||
0.6B / 1.7B / 4B via `ModelConfig`. Reuses the HF download path, cache
|
||||
directory, and model management UI. MLX (4-bit community quants) on Apple
|
||||
Silicon, PyTorch (transformers AutoModelForCausalLM) elsewhere, same as our
|
||||
TTS split.
|
||||
|
||||
No new runtime. No `llama.cpp`, no `ollama`, no fragmented model cache.
|
||||
|
||||
### Phase 3 — In-app voice input
|
||||
|
||||
A universal mic button on every Voicebox text input. Hold, speak, release —
|
||||
text lands in the focused field via direct React state update. No OS APIs
|
||||
involved; Voicebox owns the input.
|
||||
|
||||
Marquee use cases:
|
||||
|
||||
- **Generation form.** Dictate a 2,000-character TTS script instead of typing
|
||||
it. This alone justifies the feature.
|
||||
- **Voice profile descriptions.** Describe a voice's personality by speaking,
|
||||
which then becomes the input for Phase 4's persona loop.
|
||||
- **Story titles, preset names, any free-text field.** Free reuse.
|
||||
|
||||
Backend: add `/transcribe/stream` WebSocket endpoint. Audio frames in, partial
|
||||
transcripts out. Reuses the existing Whisper model in memory. Optionally routes
|
||||
through the LLM from Phase 2 for light refinement.
|
||||
|
||||
### Phase 4 — Captures tab
|
||||
|
||||
Graduates the tab out from behind the feature flag. Shows recent captures
|
||||
(audio + transcript pairs), lets the user replay, re-transcribe with a
|
||||
different model, edit the transcript, and send the output through the LLM.
|
||||
Archival is automatic — every capture saves audio alongside transcript.
|
||||
|
||||
**Includes the "Play as voice profile" action.** This is the simplest version
|
||||
of the persona loop and it lands here for free — no LLM involved, no new
|
||||
backend endpoints, just a Captures-tab button that sends the transcript text
|
||||
to the existing `/generate` endpoint with a user-selected voice profile and
|
||||
plays the result. Category-defining differentiator from the v1 prototype
|
||||
onward: Superwhisper and WisprFlow cannot do this because they have no TTS. Voicebox can, with one day of frontend wiring.
|
||||
|
||||
Keep it aggressively minimal on day one. A capture list, a detail view, a
|
||||
model picker, a Play-as-voice dropdown. Refinement prompt editing, correction
|
||||
dictionaries, per-source overrides — none of that ships here. They become
|
||||
Tier-2 work when someone actually asks for them.
|
||||
|
||||
### Phase 5 — Agent voice output + persona loop
|
||||
|
||||
Two features that together make "Voicebox as the voice layer for every agent
|
||||
on your machine" a shipping reality:
|
||||
|
||||
1. **`speak()` primitive.** New `POST /speak` endpoint and `voicebox.speak`
|
||||
MCP tool. Any agent calls Voicebox to speak arbitrary text in a
|
||||
user-configured voice; the pill surfaces in a `speaking` state. Settings
|
||||
UI for default voice, per-agent voice binding (Claude Code → Morgan,
|
||||
Cursor → Scarlett), and a global mute.
|
||||
2. **Persona loop.** Extends `speak()` with an LLM step — STT → persona LLM
|
||||
→ `speak(llm_reply)`. Voice profiles gain optional personality metadata
|
||||
and default LLM behavior. End-to-end voice-to-voice with a cloned
|
||||
identity transforming the content, not just reading it.
|
||||
|
||||
Phase 4 demoed the user-initiated direction of the loop (Play as voice). This
|
||||
phase ships the *agent*-initiated direction, which is the category-defining
|
||||
capability and the pitch that lands with agent-harness users. The persona
|
||||
loop is one flow on top of the `speak()` primitive — notifications, proactive
|
||||
agent questions, and task-completion announcements all use `speak()` directly
|
||||
without the LLM in the middle.
|
||||
|
||||
Launchable headline moment for the "local voice I/O" positioning.
|
||||
|
||||
### Phase 6 — STT engine expansion
|
||||
|
||||
Parakeet v3 and Qwen3-ASR register as additional `STTBackend` implementations.
|
||||
Optional: Kyutai ASR. Multilingual coverage upgrades (50+ languages). Whisper
|
||||
stays as the sensible default.
|
||||
|
||||
Deferred to here because Whisper is already good enough for v1 and the model
|
||||
picker UI exists. Adding rows to it doesn't change the product shape.
|
||||
|
||||
### Phase 7 — External dictation shell
|
||||
|
||||
Native shim crate (global hotkey with modifier-only support, focus
|
||||
introspection via OS accessibility APIs, paste simulation, atomic clipboard
|
||||
save/restore). Tauri-side audio capture streams to the same WebSocket endpoint
|
||||
Phase 3 already ships. Paste sink with target-aware delivery.
|
||||
|
||||
This is the feel-good phase. It's also the riskiest: paste timing, hotkey
|
||||
reliability, and cross-platform focus detection are all engineering problems
|
||||
that have to be nailed or the product doesn't work. Phase 3's success derisks
|
||||
the backend plumbing before we start it.
|
||||
|
||||
### Phase 8 — Pipeline routing, sinks, long-form
|
||||
|
||||
Multiple source types, user-configurable transform chains, multiple sinks per
|
||||
preset. MCP server sink (the agent-harness play). HTTP webhook sink. File
|
||||
sink. Developer-facing `/pipelines/{id}/run` endpoint. Preset editor UI in
|
||||
the Captures tab.
|
||||
|
||||
Dual-stream recorder (mic + system audio) as a source type. Chunked STT
|
||||
transform with overlap-based deduplication. Summary LLM transform. Long-form
|
||||
capture becomes a preset, not a new tab.
|
||||
|
||||
Platform-specific sinks (Apple Notes on macOS, Obsidian, etc.) as opt-in
|
||||
integrations behind the generic sink interface.
|
||||
|
||||
## Architectural prerequisites
|
||||
|
||||
Two pieces of existing `docs/PROJECT_STATUS.md` work become load-bearing here:
|
||||
|
||||
- **Platform support tiers** (#420, PR #465). Native shim capabilities vary by
|
||||
platform — Wayland paste is worse than X11, Windows system-audio capture has
|
||||
edge cases, frontmost-window OCR is platform-gated. Tier definitions let us
|
||||
ship confidently with honest user-facing expectations.
|
||||
- **Platform gating on `ModelConfig`** (bottleneck #6 in PROJECT_STATUS).
|
||||
Parakeet's Core ML path is Apple-only; the PyTorch path is Windows/Linux.
|
||||
Same gating mechanism that currently blocks shipping VoxCPM.
|
||||
|
||||
Neither needs to complete before Phase 1, but both should complete before
|
||||
Phase 4 when user-configurable pipelines surface the differences to end users.
|
||||
Reference in New Issue
Block a user