generated from Labyricorn/labyricorn-project-template
Initial commit (forked from jamiepine/voicebox)
This commit is contained in:
@@ -0,0 +1,168 @@
|
||||
# Voicebox Cloud Roadmap
|
||||
|
||||
The post-mobile commercial trajectory. Captures the strategic arc beyond `mobile/PLAN.md` — what Voicebox becomes once the mobile companion ships and we start layering optional cloud services on top of the local-first base.
|
||||
|
||||
The desktop app stays free. Paid surface is the cloud layer, gated behind a Voicebox account, designed so the server sees as little as possible.
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0 — Mobile companion (in progress)
|
||||
|
||||
See [`mobile/PLAN.md`](../../mobile/PLAN.md). Entirely local: paired-device keys live on the iPhone, traffic goes over Tailscale or LAN, no cloud account required. This is the wedge — it establishes the device-key primitive that every later phase reuses.
|
||||
|
||||
### Phase 1 — Backup & Sync (next big feature)
|
||||
|
||||
First introduction of a Voicebox cloud account. Server stores **only encrypted blobs**.
|
||||
|
||||
- **E2E encryption keyed off the device key** from the mobile pairing flow. Audio + transcript blobs are encrypted client-side before upload; the server never has the plaintext or the key.
|
||||
- **Quota by number of generations**, not by storage GB. Avoids "how many GB do you offer" framing and keeps tiering legible. (Word-count quotas are an alternative — closer to the ElevenLabs model — but generations are simpler to communicate.)
|
||||
- **What's synced:** captures (audio + transcripts), generations, voice profiles **as ciphertext**, settings.
|
||||
- **What's NOT synced:** voice profile audio in plaintext, refinement LLM context, anything that would let us reconstruct what a user said or who they sound like.
|
||||
- **Multi-device read:** the same paired-device key on a second device decrypts the backup. Recovery via printable key on first pairing.
|
||||
|
||||
The privacy framing is load-bearing. "We see encrypted blobs and that's it" is the commitment the rest of the cloud story rests on.
|
||||
|
||||
### Phase 2 — Private Voice Inference ("the OpenRouter for voice")
|
||||
|
||||
The big bet. Today there is no major neutral voice-inference provider — every cloud TTS service ships its own proprietary models. Open-source TTS models exist and keep getting better, but nobody runs them as a paid hosted catalog at scale.
|
||||
|
||||
Voicebox already has the distribution. The thesis is: the same users who chose local-first specifically to avoid sending voice data to ElevenLabs will pay a fair markup to run open-source voices on hosted GPUs **when they don't have local hardware** (mobile-only users, low-end laptops, "I just don't want to manage CUDA"), provided the privacy story stays consistent.
|
||||
|
||||
- **Catalog-first positioning.** Cloud can offer more voices than the desktop binary bundles (the bundle is already 500MB without CUDA, ~3GB with — there's a hard ceiling on what we can ship locally). Catalog grows over time.
|
||||
- **Pricing tiers (rough first cut):** $5 / $15 / $25 / month, plus Enterprise. Final numbers depend on benchmarking — see below.
|
||||
- **Unit economics work to do:** benchmark every open-source TTS engine in the lineup (Qwen3-TTS, Chatterbox Multilingual + Turbo, TADA, Kokoro, LuxTTS, plus future additions) for cost-per-generation on candidate hardware. Find the engines where our markup is comfortably below ElevenLabs's per-character cost.
|
||||
- **Privacy ceiling:** server-side inference cannot be cryptographically verified the way E2E backup can. The honest framing is "we don't log inputs, we don't train on your data, audited" — not "we mathematically can't see it." That's a real step down from Phase 1's guarantee, and the product has to be clear about it.
|
||||
- **Mobile + OS integrations.** Once cloud inference exists, the mobile app unlocks the same OS-level surfaces ElevenLabs has (keyboard-tied dictation, share-sheet TTS, Siri-equivalent). Local-first users still get them via paired desktop; cloud users get them without needing a desktop at all.
|
||||
|
||||
#### Inference architecture
|
||||
|
||||
**Compute layer.** Modal as the v1 platform — per-second billing, scale-to-zero, volume mounts for model weights, runs our existing Python code with a thin decorator. The ~30-50% premium over raw GPU cost is irrelevant at launch scale and small (less than one DevOps hire) at $1M ARR. Migrate engine-by-engine to bare-metal (Lambda Labs, Crusoe, CoreWeave) once any single engine has predictable demand. Hyperscalers (AWS / GCP) only for enterprise contracts that require it.
|
||||
|
||||
**Topology.**
|
||||
|
||||
```
|
||||
Client (desktop / mobile / API user)
|
||||
│
|
||||
▼ HTTPS, bearer auth
|
||||
Gateway ← R2: encrypted profile blobs
|
||||
│ ← D1 / Postgres: users, billing, quotas, profile metadata
|
||||
▼ internal RPC
|
||||
Per-engine Modal apps (Kokoro / Chatterbox / TADA / Whisper / …)
|
||||
```
|
||||
|
||||
**Gateway.** Cloudflare Workers + R2 + D1 for v1 — Workers handle auth and routing, R2 has no egress fees which matters when the payload is audio, D1 handles small relational state (users, quotas, profile metadata). Auth, billing, rate limits, profile resolution, engine routing, and log redaction all live in the gateway. Workers stay dumb: they receive a request with the profile envelope already in hand, run inference, stream audio back. The gateway is what makes engine migration painless — moving TADA to bare-metal later is a routing config change, not a client change.
|
||||
|
||||
**Model packaging.** Each `backend/backends/<engine>.py` class becomes a Modal `@app.cls` wrapper. Same inference code as desktop. Weights download on container build, live on a Modal Volume, get reused by warm containers. The PyInstaller-specific runtime hooks from 0.4.x (scipy / transformers / `torch._dynamo` workarounds for the frozen binary) factor into a `frozen.py` runtime hook the desktop build imports — cloud doesn't. Single source of truth for inference logic; two entry points for two runtimes.
|
||||
|
||||
**Streaming.** SSE over HTTPS, base64-encoded audio frames, interleaved status events (`queued` / `generating` / `done`), usage event at the end with `characters_consumed` and `seconds_generated`. Wire format identical to the desktop SSE pattern from 0.2.x — cloud is the same shape at a different URL.
|
||||
|
||||
**Latency budgets** (first audio chunk, warm / cold):
|
||||
|
||||
| Engine | Hardware | Warm | Cold | Pool strategy |
|
||||
| ---------------------------------- | --------- | ---- | ---- | ------------------------------ |
|
||||
| Kokoro, LuxTTS | CPU | <1s | ~5s | Scale-to-zero |
|
||||
| Chatterbox Turbo, Whisper Turbo | A10g / L4 | 1-3s | ~15s | Small warm pool, p95 sizing |
|
||||
| Qwen3-TTS, Chatterbox Multilingual | A10g / L4 | 2-5s | ~30s | Larger warm pool |
|
||||
| TADA-3B | A100 | ~5s | ~60s | Premium tier only, capped pool |
|
||||
|
||||
Scale-to-zero where cold start fits the budget. Hot engines need warm pools sized to p95 demand — that's where unit economics get sensitive. Reserved capacity only after a quarterly demand baseline.
|
||||
|
||||
**Profile pipeline.** Cloned voice → encrypted blob with user-account-key → uploaded to R2 cold storage → fetched into worker memory at job start → decrypted in memory only, never written to worker disk → discarded on worker idle. TTL applies at the R2 layer (cold storage retention); worker hot-path retention is bounded by warmup window. Embedding-only caching where the engine exposes a stable embedding interface; raw-audio caching is the fallback. Per-engine audit needed before launch — see open questions.
|
||||
|
||||
**Hybrid routing on the client.** Desktop, mobile, and MCP clients already speak `127.0.0.1:17493`. Add `VOICEBOX_API_URL` + `VOICEBOX_API_KEY` plus a routing function:
|
||||
|
||||
```
|
||||
if local_backend_reachable() and engine in local_engines:
|
||||
→ 127.0.0.1:17493
|
||||
else:
|
||||
→ api.voicebox.sh/v1
|
||||
```
|
||||
|
||||
Mobile-without-paired-desktop falls through to cloud automatically. Desktop without a usable GPU falls through for big engines, stays local for Kokoro. Same `voicebox.speak()` MCP call works either way. This is the differentiator versus ElevenLabs (cloud-only) and pure local-first competitors (no fallback).
|
||||
|
||||
#### Cloud-cached voice profiles
|
||||
|
||||
Inference latency makes it untenable to re-upload reference samples per call. Cloud caches the user's own voice profiles for the user's own inference, under tight guardrails:
|
||||
|
||||
- **Per-profile opt-in.** Profiles are local-only by default. A "Cloud-enabled" toggle (per-profile, never global, never automatic) is what triggers upload on first cloud generation. Mobile-without-paired-desktop is the main upgrade path here — without cached profiles, mobile cloud is preset-voices only.
|
||||
- **User-controlled TTL.** `Session only` / `24h` / `7d` / `30d` / `Never expire`. Conservative default (24h). Auto-purge on inactivity regardless of ceiling.
|
||||
- **Encrypted at rest under a user-account-key envelope.** Inference workers decrypt in memory only. Keys derived from the same identity primitive that backs Phase 1.
|
||||
- **Cache embeddings, not raw audio, where the engine supports it.** Speaker embeddings (Chatterbox-style) are derived vectors — cache *those* instead of the .wav. Smaller blast radius, not reconstructible to original speech. Per-engine audit needed before launch (Qwen3-TTS, Chatterbox Multilingual + Turbo, TADA all do speaker conditioning differently); raw-audio caching is the fallback when the engine doesn't expose a stable embedding interface.
|
||||
- **Consent attestation logged at upload.** "I have rights to this voice." Timestamped, retained. Doesn't shield from claims but it's the legal posture.
|
||||
- **Verifiable deletion.** `DELETE /v2/profiles/{id}/cloud-cache` from day one, enforced across replicas, surfaced in-app as a one-click action.
|
||||
- **Trust tier:** audited-no-log, encrypted at rest, user-controlled TTL — *not* the cryptographic guarantee Phase 1 backup carries. The product has to communicate this difference clearly so cached profiles don't bleed into the Phase 1 framing. The TTL control is the marketable differentiator versus ElevenLabs, which doesn't expose retention as a user lever at all.
|
||||
- **Legal line items.** GDPR Article 9 (biometrics are special category), BIPA ($1k–5k statutory damages per violation), Texas CUBI, Washington MHMD. Real consent flow, retention controls, deletion rights, breach notification, signed DPAs for enterprise. SOC 2 + pen test before this surface goes public — not optional.
|
||||
|
||||
### Phase 3 — Voice Marketplace (much later)
|
||||
|
||||
A marketplace where voice owners license their cloned voices for others to use, with revenue sharing. Possibly: "rent out your AI voice."
|
||||
|
||||
This is the only phase that requires hosting voice profiles, and it requires real licensing infrastructure first — consent verification, takedown flow, identity claims, revenue accounting. Until that exists, **Voicebox does not host voice profiles in cloud at all** (see constraint below). Marketplace is the long-term endgame, not the next quarter.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting constraints
|
||||
|
||||
### Voice profiles in cloud: owner-only, opt-in, time-bound
|
||||
|
||||
Three rules, in increasing strictness depending on phase:
|
||||
|
||||
- **Phase 1 (backup & sync):** profiles travel as ciphertext the server cannot decrypt. The server has no path to plaintext for any reason.
|
||||
- **Phase 2 (inference):** the user's own profiles can be cached for the user's own inference, but only with per-profile opt-in, user-controlled TTL, encryption at rest, and verifiable deletion. The server holds plaintext (or derived embeddings) under audited-no-log terms — a real downshift from Phase 1's cryptographic guarantee, and one the product has to communicate honestly.
|
||||
- **Phase 3 (marketplace):** hosting other users' voices for non-owners is gated on consent verification, licensing, takedown, and revenue accounting infrastructure. Until those exist, no profile is served to anyone but its owner. No shortcuts.
|
||||
|
||||
This protects two things at once:
|
||||
- **Legal posture.** Biometric voice data triggers GDPR Article 9, BIPA, Texas CUBI, Washington MHMD. The trust hierarchy above maps to the consent and retention story we can defend at each phase.
|
||||
- **Privacy positioning.** Phase 1 is "cryptographically can't see." Phase 2 is "audited won't see, with a timer you control." Both are honest, both sit above ElevenLabs's posture, and both have to be communicated as distinct trust tiers — not blurred together.
|
||||
|
||||
### Privacy is the moat, not a feature
|
||||
|
||||
The "private LLM users → ElevenLabs voice" workflow is incoherent: people pay to keep their text private and then hand their speech to a cloud vendor that trains on it. Voicebox is the consistent answer for that audience. Every cloud feature should be designed so a privacy-conscious user can adopt it without breaking that internal consistency — which is why Phase 1 is fully E2E and Phase 2 is "audited no-log" rather than "we have your audio but trust us."
|
||||
|
||||
### Revenue stack is multi-source
|
||||
|
||||
Subscriptions are not the only line. The full picture:
|
||||
|
||||
- **Subscriptions** — Phase 1 quotas + Phase 2 inference
|
||||
- **Corporate sponsorship** — `landing/src/app/sponsors/page.tsx`, $500/mo tier live in 0.5
|
||||
- **Individual donations** — Buy Me a Coffee
|
||||
- **Marketplace revenue share** — Phase 3, far off
|
||||
|
||||
Diversification matters because the desktop app stays free forever. Subscriptions never have to carry the whole product.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing & "ease it onto them"
|
||||
|
||||
The deliberate ordering is privacy-additive: each phase introduces the next layer of cloud only after the user has had time to trust the previous one.
|
||||
|
||||
1. **Mobile (entirely local)** — no account, no cloud, just a companion to the desktop you already trust.
|
||||
2. **Backup & sync (cloud, fully E2E)** — first cloud account. Server sees nothing. Trust is bootstrapped on "we built the math so we can't see your data even if we wanted to."
|
||||
3. **Private inference (cloud, audited no-log)** — second cloud surface. Honest about the ceiling: server-side inference can't carry the same cryptographic guarantee, but the operational commitment is no logs, no training, audited.
|
||||
4. **Marketplace (cloud, profiles hosted with consent)** — only after licensing infra. The most invasive surface, gated behind real verification.
|
||||
|
||||
Skipping ahead breaks the trust ladder. Don't ship marketplace before backup & sync is mature; don't ship hosted inference before users are comfortable holding accounts at all.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Quota unit.** Generations vs. words vs. characters. Generations is the cleanest to communicate; words/characters maps onto how ElevenLabs prices and might be required for inference billing. Could be different units per phase (generations for backup, characters for inference).
|
||||
2. **Recovery key UX.** First pairing in Phase 1 needs to print a recovery key. How prominent? Force-display vs. hide-behind-link?
|
||||
3. **Inference billing model.** Per-character (ElevenLabs-style), per-generation (simpler), per-second-of-output (closest to GPU cost). Pick before pricing tiers are finalized.
|
||||
4. **Bring-your-own-key for inference?** Some privacy-conscious users may prefer to provide their own GPU credits / API keys to a third-party host through us. Worth considering for Enterprise.
|
||||
5. **Marketplace consent verification.** What's the bar? Notarized release? Real-time liveness check? Out of scope for Phase 1-2 but informs how the device key is structured today.
|
||||
6. **Default cloud-cache TTL.** 24h is the proposed conservative default. Worth A/B testing against `Session only` for first-time users — the "auto-purge after this session" framing might be a stronger trust signal than any number.
|
||||
7. **Embedding vs. raw-audio caching, per engine.** Chatterbox produces stable speaker embeddings; Qwen3-TTS, TADA, and others use different conditioning strategies. Audit needed before launch — embedding-only caching shrinks the legal/privacy surface meaningfully, but only where the engine exposes a clean embedding interface.
|
||||
8. **Single gateway region or multi-region?** Cloudflare is global by default, but Modal apps are primarily us-east / us-west. EU users hitting US compute = +100ms first-token latency, and GDPR pushes toward EU compute regardless. v1 single-region or hold launch for EU?
|
||||
9. **SSE vs WebSocket for streaming.** SSE works through any proxy and is what desktop already uses, so the wire format is shared for free. WebSocket is bidirectional and unlocks "interrupt mid-generation" and live duplex features later. Default: SSE for v1, WS as a follow-on.
|
||||
10. **Cloud Whisper in the v1 bundle?** Phase 2 was framed as TTS-only ("OpenRouter for voice"), but mobile dictation hitting cloud Whisper instead of a paired desktop is the obvious mobile-only feature. Same launch bundle, or hold for Phase 2.5?
|
||||
11. **Billing integration.** Stripe Metered + customer portal (~2 weeks of work, 2.9% fee) vs self-hosted (saves the fee, adds significant ongoing work). Default: Stripe.
|
||||
|
||||
---
|
||||
|
||||
## How this connects to mobile V1
|
||||
|
||||
The encryption story starts with the device key minted during mobile pairing (`mobile/PLAN.md` → "Pairing & transport"). That same key — or a key derived from it — is what encrypts cloud blobs in Phase 1. Don't treat the mobile pairing key as a one-off; design it as the root of the user's lifetime encryption identity, with rotation + multi-device-add flows in mind even if those don't ship until Phase 1.
|
||||
@@ -0,0 +1,758 @@
|
||||
# Docker Deployment Guide
|
||||
|
||||
**Status:** In Development for v0.2.0
|
||||
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
|
||||
|
||||
## Overview
|
||||
|
||||
Docker support makes Voicebox easier to deploy, especially for:
|
||||
|
||||
- **Consistent Environments**: Same setup across dev/staging/prod
|
||||
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
|
||||
- **Server Deployments**: Run on headless Linux servers
|
||||
- **Multi-User Setups**: Isolate instances per user/team
|
||||
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Pre-Built Images (Recommended)
|
||||
|
||||
```bash
|
||||
# CPU-only version
|
||||
docker run -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest
|
||||
|
||||
# NVIDIA GPU version
|
||||
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
|
||||
# AMD GPU version (experimental)
|
||||
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-rocm
|
||||
```
|
||||
|
||||
Then open: `http://localhost:8000`
|
||||
|
||||
### Using Docker Compose (Easiest)
|
||||
|
||||
Create `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Building From Source
|
||||
|
||||
### Basic Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git \
|
||||
build-essential \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy application
|
||||
COPY backend/ /app/backend/
|
||||
COPY requirements.txt /app/
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8000
|
||||
|
||||
# Run server
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build and run:
|
||||
```bash
|
||||
docker build -t voicebox .
|
||||
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Multi-Stage Build (Optimized)
|
||||
|
||||
Smaller image size by separating build and runtime:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.optimized
|
||||
# Stage 1: Build dependencies
|
||||
FROM python:3.11-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git build-essential && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
-r requirements.txt
|
||||
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Stage 2: Runtime
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install only runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy installed packages from builder
|
||||
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
|
||||
|
||||
# Copy application code
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build:
|
||||
```bash
|
||||
docker build -f Dockerfile.optimized -t voicebox:slim .
|
||||
```
|
||||
|
||||
## GPU Support
|
||||
|
||||
### NVIDIA GPUs (CUDA)
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with CUDA support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with GPU:**
|
||||
```bash
|
||||
docker run --gpus all -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
voicebox:cuda
|
||||
```
|
||||
|
||||
**Docker Compose with GPU:**
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
image: voicebox:cuda
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
### AMD GPUs (ROCm) - Experimental
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM rocm/dev-ubuntu-22.04:6.0
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with ROCm support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Set ROCm environment variables
|
||||
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with AMD GPU:**
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video --ipc=host --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-p 8000:8000 -v voicebox-data:/app/data \
|
||||
voicebox:rocm
|
||||
```
|
||||
|
||||
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
|
||||
|
||||
## Volume Mounts
|
||||
|
||||
### Essential Volumes
|
||||
|
||||
```bash
|
||||
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
|
||||
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Development Volume Mounts
|
||||
|
||||
For development with hot-reload:
|
||||
|
||||
```bash
|
||||
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
|
||||
-v voicebox-data:/app/data \
|
||||
-e RELOAD=true \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Custom Model Storage
|
||||
|
||||
Use external model directory:
|
||||
|
||||
```bash
|
||||
docker run -v /path/to/models:/models \
|
||||
-e MODELS_DIR=/models \
|
||||
-v voicebox-data:/app/data \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Configure Voicebox via environment variables:
|
||||
|
||||
```bash
|
||||
docker run -e TTS_MODE=local \
|
||||
-e WHISPER_MODE=openai-api \
|
||||
-e OPENAI_API_KEY=sk-... \
|
||||
-e GPU_MEMORY_FRACTION=0.8 \
|
||||
-e LOG_LEVEL=info \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Available Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
|
||||
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
|
||||
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
|
||||
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
|
||||
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
|
||||
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
|
||||
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
|
||||
| `MODELS_DIR` | `/app/models` | Directory for local models |
|
||||
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
|
||||
| `RELOAD` | `false` | Enable hot-reload for development |
|
||||
|
||||
## Complete Docker Compose Examples
|
||||
|
||||
### Production Deployment
|
||||
|
||||
```yaml
|
||||
# docker-compose.prod.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
container_name: voicebox
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
- GPU_MEMORY_FRACTION=0.8
|
||||
- LOG_LEVEL=info
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
driver: local
|
||||
huggingface-cache:
|
||||
driver: local
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### Development Setup
|
||||
|
||||
```yaml
|
||||
# docker-compose.dev.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./backend:/app/backend:ro
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- RELOAD=true
|
||||
- LOG_LEVEL=debug
|
||||
- TTS_MODE=local
|
||||
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
### Multi-Service Stack
|
||||
|
||||
Full stack with reverse proxy and monitoring:
|
||||
|
||||
```yaml
|
||||
# docker-compose.stack.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# Main Voicebox app
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
# Nginx reverse proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
depends_on:
|
||||
- voicebox
|
||||
|
||||
# Prometheus monitoring (optional)
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
- prometheus-data:/prometheus
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
prometheus-data:
|
||||
```
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
|
||||
2. **Install Docker + nvidia-docker:**
|
||||
```bash
|
||||
# Amazon Linux 2
|
||||
sudo yum install -y docker
|
||||
sudo systemctl start docker
|
||||
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
|
||||
sudo apt-get update && sudo apt-get install -y nvidia-docker2
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
3. **Deploy:**
|
||||
```bash
|
||||
docker run --gpus all -d -p 80:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
```
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
Use GPU Droplet + Docker:
|
||||
|
||||
```bash
|
||||
# Create droplet via CLI
|
||||
doctl compute droplet create voicebox \
|
||||
--size gpu-h100x1-80gb \
|
||||
--image ubuntu-22-04-x64 \
|
||||
--region nyc3
|
||||
|
||||
# SSH and deploy
|
||||
ssh root@<droplet-ip>
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
docker run --gpus all -d -p 80:8000 voicebox:cuda
|
||||
```
|
||||
|
||||
### Google Cloud Run (CPU-only)
|
||||
|
||||
```bash
|
||||
# Build and push
|
||||
docker build -t gcr.io/your-project/voicebox .
|
||||
docker push gcr.io/your-project/voicebox
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy voicebox \
|
||||
--image gcr.io/your-project/voicebox \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--memory 4Gi \
|
||||
--cpu 2 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Fly.io
|
||||
|
||||
Create `fly.toml`:
|
||||
```toml
|
||||
app = "voicebox"
|
||||
|
||||
[build]
|
||||
image = "ghcr.io/jamiepine/voicebox:latest"
|
||||
|
||||
[[services]]
|
||||
http_checks = []
|
||||
internal_port = 8000
|
||||
protocol = "tcp"
|
||||
|
||||
[[services.ports]]
|
||||
port = 80
|
||||
handlers = ["http"]
|
||||
|
||||
[[services.ports]]
|
||||
port = 443
|
||||
handlers = ["tls", "http"]
|
||||
|
||||
[mounts]
|
||||
source = "voicebox_data"
|
||||
destination = "/app/data"
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
fly launch
|
||||
fly deploy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### GPU Not Detected
|
||||
|
||||
**Check NVIDIA Docker:**
|
||||
```bash
|
||||
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
If this fails, reinstall nvidia-docker2.
|
||||
|
||||
**Check AMD ROCm:**
|
||||
```bash
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
|
||||
```
|
||||
|
||||
### Permission Errors
|
||||
|
||||
Container can't write to volumes:
|
||||
```bash
|
||||
# Fix permissions
|
||||
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
Reduce GPU memory usage:
|
||||
```bash
|
||||
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
|
||||
```
|
||||
|
||||
Or use CPU-only:
|
||||
```bash
|
||||
docker run -e DEVICE=cpu voicebox
|
||||
```
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
Ensure HuggingFace cache is writable:
|
||||
```bash
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
Or use host cache:
|
||||
```bash
|
||||
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
Change host port:
|
||||
```bash
|
||||
docker run -p 8080:8000 voicebox # Use port 8080 instead
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Don't Run as Root
|
||||
|
||||
Create non-root user in Dockerfile:
|
||||
```dockerfile
|
||||
RUN useradd -m -u 1000 voicebox
|
||||
USER voicebox
|
||||
```
|
||||
|
||||
### 2. Use Secrets for API Keys
|
||||
|
||||
Don't put API keys in docker-compose.yml:
|
||||
|
||||
```bash
|
||||
# Use Docker secrets
|
||||
echo "sk-your-key" | docker secret create openai_key -
|
||||
|
||||
docker service create \
|
||||
--secret openai_key \
|
||||
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
|
||||
voicebox
|
||||
```
|
||||
|
||||
### 3. Network Isolation
|
||||
|
||||
Use internal networks for multi-container setups:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
networks:
|
||||
- internal
|
||||
nginx:
|
||||
networks:
|
||||
- internal
|
||||
- external
|
||||
ports:
|
||||
- "80:80"
|
||||
|
||||
networks:
|
||||
internal:
|
||||
internal: true
|
||||
external:
|
||||
```
|
||||
|
||||
### 4. Resource Limits
|
||||
|
||||
Prevent resource exhaustion:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '4'
|
||||
memory: 8G
|
||||
reservations:
|
||||
cpus: '2'
|
||||
memory: 4G
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### GPU Memory Management
|
||||
|
||||
```bash
|
||||
# Use 80% of GPU (default 90%)
|
||||
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
|
||||
|
||||
# Allow GPU memory growth (prevents OOM)
|
||||
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
|
||||
```
|
||||
|
||||
### Model Caching
|
||||
|
||||
Pre-download models to volume:
|
||||
|
||||
```bash
|
||||
# Download models first
|
||||
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
|
||||
voicebox python -c "
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
WhisperProcessor.from_pretrained('openai/whisper-base')
|
||||
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
|
||||
"
|
||||
|
||||
# Then run normally
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Multi-Worker Setup
|
||||
|
||||
Use uvicorn workers for better throughput:
|
||||
|
||||
```dockerfile
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
Built-in health endpoint:
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
Docker health check:
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Add metrics exporter:
|
||||
```python
|
||||
# backend/main.py
|
||||
from prometheus_fastapi_instrumentator import Instrumentator
|
||||
|
||||
Instrumentator().instrument(app).expose(app)
|
||||
```
|
||||
|
||||
Then scrape `/metrics` with Prometheus.
|
||||
|
||||
### Logs
|
||||
|
||||
View container logs:
|
||||
```bash
|
||||
docker logs -f voicebox
|
||||
|
||||
# Or with compose
|
||||
docker compose logs -f voicebox
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] Publish official images to GitHub Container Registry
|
||||
- [ ] Add Kubernetes Helm charts
|
||||
- [ ] Create Docker Desktop extension
|
||||
- [ ] Add automated vulnerability scanning
|
||||
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
|
||||
|
||||
## Contributing
|
||||
|
||||
Help improve Docker support:
|
||||
1. Test on different platforms (AMD GPU, ARM64, etc.)
|
||||
2. Submit Dockerfile optimizations
|
||||
3. Share deployment configurations
|
||||
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
|
||||
|
||||
## Resources
|
||||
|
||||
- [Docker Documentation](https://docs.docker.com)
|
||||
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
|
||||
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
|
||||
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
|
||||
@@ -0,0 +1,102 @@
|
||||
# macOS Notarization & Gatekeeper
|
||||
|
||||
**Status:** Diagnosis — Homebrew Cask CI rejects v0.4.5 on macOS 15 (Sequoia); fix pending
|
||||
**Touches:** `.github/workflows/release.yml`, Tauri bundler config, sidecar signing
|
||||
**Last reviewed:** 2026-04-24
|
||||
|
||||
## Context
|
||||
|
||||
Homebrew Cask PR [#260314](https://github.com/Homebrew/homebrew-cask/pull/260314) adds `brew install --cask voicebox`. CI is green on macOS 14 and macOS 26 (arm + intel) but fails on macOS 15 (arm + intel). The 0.4.3 release added DMG-level stapling to address this, and it didn't move CI — 0.4.5 still fails. A maintainer reproduced the failure in a fresh Sequoia VM.
|
||||
|
||||
This document is the working diagnosis plus the ordered fix plan.
|
||||
|
||||
## What the failing check actually does
|
||||
|
||||
The failing step is `brew audit --cask --online --signing --new voicebox`, not `brew install`. `brew install` succeeds end-to-end in CI (the log shows `Uninstalling Cask voicebox` after the install phase). The `--signing` audit:
|
||||
|
||||
1. Downloads the cask's `url`
|
||||
2. Mounts the DMG
|
||||
3. Runs `spctl --assess -t open --context context:primary-signature` against the `.app` inside
|
||||
|
||||
That policy tests the first-launch Gatekeeper path on the extracted bundle. It reads the `.app`'s own code signature and notarization ticket — the DMG wrapper is not involved. The staple added in 0.4.3 covers the DMG, so it has no effect on this audit.
|
||||
|
||||
## Why Sequoia and not Sonoma
|
||||
|
||||
`spctl -t open` on macOS 15 enforces checks that 14 tolerated:
|
||||
|
||||
- Secure timestamp required on hardened-runtime signatures. Untimestamped signatures pass on 14, fail on 15.
|
||||
- Deep verification of nested Mach-Os. If any embedded `.dylib` or helper binary carries an ad-hoc signature (or a signature with a different Team ID), 15 rejects the whole bundle; 14 often accepted it.
|
||||
- Hardened runtime must be set on every nested executable, not just the top-level app binary. Entitlements declared on the outer app do not propagate.
|
||||
|
||||
Local dev machines pass `spctl` because the first-party developer context and cached notarization tickets mask these failures. A fresh Sequoia VM with no prior trust state does not.
|
||||
|
||||
## Where the gap is likely to be
|
||||
|
||||
Voicebox ships PyInstaller sidecars declared in `tauri.conf.json` under `externalBin`:
|
||||
|
||||
- **0.4.x:** `voicebox-server` only (single `--onefile` Mach-O on macOS)
|
||||
- **0.5.0+:** `voicebox-server` and `voicebox-mcp` (`voicebox-mcp` is new in 0.5.0)
|
||||
|
||||
Tauri's bundler signs each `externalBin` with the configured identity but does not apply `--options=runtime` or `--timestamp` automatically, and does not merge the outer app's entitlements into the sidecar signature. The outer `Voicebox` binary is correctly signed with hardened runtime + `disable-library-validation`; the sidecars likely are not.
|
||||
|
||||
Order of likelihood:
|
||||
|
||||
1. Sidecar `voicebox-server` lacks hardened runtime or a secure timestamp in its signature.
|
||||
2. The sidecar inherits the identity but was signed before tauri-action's final notarization pass, so the notarization ticket doesn't actually cover it.
|
||||
3. Something inside the sidecar's PyInstaller archive unpacks to a `.dylib` at runtime that Gatekeeper inspects during assessment.
|
||||
|
||||
The 0.5.0 fix must cover both sidecars.
|
||||
|
||||
## Diagnostic commands
|
||||
|
||||
Run against a freshly downloaded release DMG (not a dev build, and from a machine that has never opened the app before):
|
||||
|
||||
```
|
||||
hdiutil attach Voicebox_0.4.5_aarch64.dmg
|
||||
xcrun stapler validate "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
spctl -a -vvv -t open --context context:primary-signature "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
codesign --verify --deep --strict --verbose=2 "/Volumes/Voicebox 0.4.5/Voicebox.app"
|
||||
codesign -dv --verbose=4 "/Volumes/Voicebox 0.4.5/Voicebox.app/Contents/MacOS/voicebox-server"
|
||||
```
|
||||
|
||||
The last command is the tell — look for `flags=0x10000(runtime)` and a `Timestamp=` line. If either is missing, the sidecar is the failure.
|
||||
|
||||
`spctl -t install` (what 0.4.3 verified with) is a different policy and can pass while `-t open` fails — any future verification should use `-t open --context context:primary-signature` to match what Homebrew's audit runs.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1 — Confirm the failure mode
|
||||
|
||||
Pull the 0.4.5 DMG on a fresh Sequoia environment or a VM snapshot with no trust state. Run the diagnostic block above. Record the exact failing command and its CSSMERR / rejection reason. This disambiguates between the three hypotheses before we change the workflow.
|
||||
|
||||
### Phase 2 — Sign sidecars explicitly in the release workflow
|
||||
|
||||
Between tauri-action's build step and the DMG-notarization step already in `release.yml`, add a step that re-signs every `externalBin` present under `Voicebox.app/Contents/MacOS/` with:
|
||||
|
||||
- `--options=runtime` (hardened runtime)
|
||||
- `--timestamp` (secure timestamp)
|
||||
- `--entitlements` pointing at `Entitlements.plist` or a sidecar-specific subset
|
||||
- The same `APPLE_SIGNING_IDENTITY` the outer app uses
|
||||
|
||||
Re-sign the outer `.app` afterward so its seal covers the updated nested signatures.
|
||||
|
||||
Covers `voicebox-server` on 0.4.x and both sidecars from 0.5.0 forward.
|
||||
|
||||
### Phase 3 — Re-notarize and staple the `.app`
|
||||
|
||||
After sidecars are re-signed the outer bundle's notarization ticket is stale. Submit the `.app` (zipped) to `notarytool`, wait, then `xcrun stapler staple Voicebox.app`. This puts the ticket directly on the `.app` so the `spctl -t open` audit passes without any online ticket lookup.
|
||||
|
||||
Then rebuild the DMG from the stapled `.app` and keep the existing DMG-level notarize/staple step — it still helps Finder drag-install.
|
||||
|
||||
### Phase 4 — CI verification gate in the release workflow
|
||||
|
||||
Before upload, run the same four diagnostic commands against the built artifact inside the workflow. If any fail, fail the release job rather than shipping a DMG that Homebrew (and Sequoia Finder users) will reject. This is the check that would have caught the 0.4.3 and 0.4.5 attempts before they cost PR review cycles.
|
||||
|
||||
### Phase 5 — Re-request Homebrew CI
|
||||
|
||||
Once a tagged release passes Phase 4 locally, push a cask update to #260314. Expect `test voicebox (macos-15, arm)` and `test voicebox (macos-15-intel, intel)` to go green.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does tauri-action v0.6 pass `APPLE_API_KEY_PATH` to the bundler's notarize path, or does it rely on the `~/.appstoreconnect/private_keys/AuthKey_*.p8` auto-discovery the staple step already sets up? If the former isn't working, tauri may be signing but never notarizing the `.app`, which would make the ticket absent entirely rather than stale. Worth a `grep -i notariz` on a full release job log.
|
||||
- If Phase 2 resolves the macOS 15 failure, revisit whether the 0.4.3 DMG staple step is still needed. It's cheap to keep and helps the Finder-open case, so default to leaving it.
|
||||
@@ -0,0 +1,347 @@
|
||||
# MCP Server — Voicebox Speed Run
|
||||
|
||||
**Status:** v1 shipped — HTTP transport, all 4 tools, per-client bindings, `POST /speak`, stdio shim (binary built, bundled into Tauri sidecar), Settings UI, speak-pill via SSE with Rust-side `dictate:show` handler so agent-initiated speech surfaces the pill on screen. `cargo check` clean, `tsc` clean, full Inspector round-trip verified.
|
||||
**Last reviewed:** 2026-04-23
|
||||
|
||||
## Status
|
||||
|
||||
### Shipped (backend)
|
||||
- **`fastmcp` + `sse-starlette`** pinned in `backend/requirements.txt`.
|
||||
- **`backend/mcp_server/`** package with `server.py`, `tools.py`, `context.py`, `resolve.py`, `events.py`, `README.md`. Named `mcp_server` (not `mcp`) to sidestep a shadowing conflict with the installed `mcp` PyPI package that FastMCP imports internally.
|
||||
- **Streamable HTTP mount at `/mcp`** via FastMCP's `http_app(transport='http')`. Sub-app lifespan composed with Voicebox's own startup/shutdown through an `@asynccontextmanager lifespan=` in `backend/app.py` (migrated away from the deprecated `@app.on_event` handlers).
|
||||
- **Four MCP tools**, dot-named to match the landing and ecosystem convention:
|
||||
- `voicebox.speak(text, profile?, engine?, personality?, language?)`
|
||||
- `voicebox.transcribe(audio_base64?, audio_path?, language?, model?)`
|
||||
- `voicebox.list_captures(limit, offset)`
|
||||
- `voicebox.list_profiles()`
|
||||
- **`ClientIdMiddleware`** pulls `X-Voicebox-Client-Id` into a `ContextVar` on every `/mcp*` request; auto-stamps `MCPClientBinding.last_seen_at`, auto-creating the row if the client is new.
|
||||
- **Profile resolution precedence** `explicit → per-client binding → capture_settings.default_playback_voice_id → error`. `services/profiles.get_profile_orm_by_name_or_id()` lets agents pass a voice by name ("Morgan") instead of UUID.
|
||||
- **`MCPClientBinding` table** (new) via `Base.metadata.create_all` — no migration needed.
|
||||
- **Bindings REST:** `GET|PUT /mcp/bindings`, `DELETE /mcp/bindings/{client_id}`.
|
||||
- **`POST /speak`** REST wrapper for non-MCP callers (shell / ACP / A2A). Same `resolve_profile` precedence, same code path as the MCP tool.
|
||||
- **Stdio shim** at `backend/mcp_shim/__main__.py` — ~200 lines of `httpx` proxy; reads env (`VOICEBOX_PORT`, `VOICEBOX_HOST`, `VOICEBOX_CLIENT_ID`), waits for `/health`, then streams JSON-RPC ↔ SSE. Rolled our own after the `mcp` SDK's session-management helpers mis-shook-hands. Smoke-tested: `initialize`, `tools/list`, and `tools/call` all round-trip cleanly.
|
||||
- **Pill SSE:** `GET /events/speak` (`sse-starlette`) emits `speak-start` from the MCP tool and `POST /speak`, `speak-end` from `services/generation.run_generation`'s finally block.
|
||||
- **PyInstaller:**
|
||||
- `backend/build_binary.py` `--shim` flag builds a minimal `voicebox-mcp` binary (torch/transformers/mlx/etc. explicitly excluded, target <20 MB).
|
||||
- The main server spec picks up `fastmcp`, `mcp`, `sse_starlette`, and `backend.mcp_server.*` via `--collect-all` / `--hidden-import`.
|
||||
- **`backend/mcp_server/README.md`** quickstart (Inspector, `.mcp.json` snippets, tool reference).
|
||||
|
||||
### Shipped (frontend)
|
||||
- **`Settings → MCP`** page (`app/src/components/ServerTab/MCPPage.tsx`):
|
||||
- Three copy-paste snippets auto-filled with the detected `serverUrl`: HTTP (recommended), Claude Code CLI one-liner, stdio fallback.
|
||||
- Default voice picker (bound to `capture_settings.default_playback_voice_id`, shared with Captures-tab "Play as voice").
|
||||
- Per-client bindings table with inline profile picker, remove button, and a connection-status indicator that refreshes every 10 s.
|
||||
- Add-binding form with client_id / label / profile dropdown.
|
||||
- **`useMCPBindings`** TanStack hook (optimistic delete, invalidate on upsert).
|
||||
- **`useSpeakEvents`** hook — auto-reconnecting `EventSource('/events/speak')`, tracks the active generation_id, exposes an elapsed-ms timer that ticks so the pill's clock advances.
|
||||
- **`CapturePill`** has a new `'speaking'` state + "Speaking" label + playing-bars mode.
|
||||
- **`DictateWindow`** subscribes to speak events and overrides `pillState` when an agent is speaking. Emits `dictate:show` on speak-start so the Rust side can surface the pill window.
|
||||
- Router + `ServerTab` tab bar wired to `/settings/mcp`.
|
||||
|
||||
### Shipped (native shell)
|
||||
- **`tauri.conf.json`** — `voicebox-mcp` added to `externalBin` (alongside `voicebox-server`).
|
||||
- **`dictate:show` listener** in `tauri/src-tauri/src/main.rs` — invokes a new `show_dictate_window(app_handle)` helper that mirrors the hotkey-monitor's position+show logic (undo click-through, reposition to top-center of the current monitor, show). Agent-initiated speech now pops the pill visible on screen.
|
||||
|
||||
### Validated end-to-end (this session, via curl)
|
||||
- `/mcp/` init → `tools/list` → `tools/call voicebox.speak` → actual audio plays (Jarvis, 1.68 s).
|
||||
- `POST /speak` with `X-Voicebox-Client-Id: claude-code` resolves to the bound Jarvis profile without passing `profile`.
|
||||
- `/events/speak` emits `ready`, `speak-start`, `speak-end` in order, generation_id threads through both.
|
||||
- Stdio shim: `echo {…} | python -m backend.mcp_shim` returns valid JSON-RPC for all 4 methods.
|
||||
- `last_seen_at` auto-stamps on first call; binding row auto-creates.
|
||||
- Frontend `tsc --noEmit`: clean.
|
||||
- `cargo check` on the Tauri crate: clean.
|
||||
|
||||
### Outstanding (must-do before release)
|
||||
- **CI build for shim on Windows/Linux** — `python backend/build_binary.py --shim` is wired up and built cleanly for `aarch64-apple-darwin` (18 MB, installed at `tauri/src-tauri/binaries/voicebox-mcp-aarch64-apple-darwin`, Tauri `cargo check` green). The Windows and Linux triples (`x86_64-pc-windows-msvc`, `x86_64-unknown-linux-gnu`) need the same build in their respective CI runners and artifacts dropped alongside the macOS binary.
|
||||
- **Windows/Linux paths in the stdio snippet** — the Settings page hardcodes the macOS path (`/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp`). Needs a per-OS switch (`%LOCALAPPDATA%\Programs\Voicebox\voicebox-mcp.exe`, Linux bundled-path), ideally with the Tauri shell resolving its own app path at runtime and injecting it into the snippet.
|
||||
|
||||
### Nice-to-have (follow-up passes)
|
||||
- **One-click install buttons** — write/merge into `~/.claude/settings.json`, `~/.cursor/mcp.json`, etc. via a Tauri command. Copy-paste works today; this is pure ergonomics.
|
||||
- **`.mcpb` desktop extension** for Claude Desktop (single file, double-click to install). Claude Desktop-only, so lower priority than the agent-harness crowd.
|
||||
- **Refactor the hotkey_monitor.rs show-logic** to call `show_dictate_window()` instead of duplicating the position+show block. Skipped at ship to avoid regressing the well-tested chord path.
|
||||
- **Source attribution on `Generation.source`** — currently `"manual" | "personality_speak"`; adding `"mcp"` / `"rest"` would let the Captures tab filter by MCP-originated rows.
|
||||
|
||||
## Context
|
||||
|
||||
Voicebox already ships the I/O surface (Captures, Generate, personality-driven `/profiles/{id}/speak`), but local AI agents can't reach any of it. This plan adds a Model Context Protocol server so Claude Code / Cursor / Cline can call `voicebox.speak`, `voicebox.transcribe`, `voicebox.list_captures`, and `voicebox.list_profiles` — turning Voicebox into the local voice layer for every agent on the user's machine (Phase 5 of `docs/plans/VOICE_IO.md`).
|
||||
|
||||
The shortest path to "Claude Code speaks in a cloned voice": mount **FastMCP** inside the existing FastAPI/uvicorn process at `/mcp` (Streamable HTTP), and users install it as a URL (`{"url": "http://127.0.0.1:17493/mcp"}`) — the ecosystem-idiomatic shape for a long-running local service. Per-client voice binding via a new `mcp_client_bindings` table + Settings UI, resolved from an `X-Voicebox-Client-Id` header. A **stdio shim binary** `voicebox-mcp` is bundled as a fallback sidecar for clients that can't speak HTTP MCP. A public `POST /speak` REST wrapper covers non-MCP callers (shell scripts, ACP, A2A). A `speaking` pill state gives agent-initiated audio visibility — trust-critical, non-negotiable.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Claude Code / Cursor / Windsurf / VS Code MCP
|
||||
│
|
||||
├─ HTTP (primary) ────────────────────┐
|
||||
│ {"url": ".../mcp"} │
|
||||
│ │
|
||||
└─ stdio (fallback) ───────────────▶ [voicebox-mcp shim binary]
|
||||
{"command": "/abs/path/voicebox-mcp"} (absolute path;
|
||||
│ Settings page
|
||||
│ copies it for you)
|
||||
▼
|
||||
uvicorn + FastAPI (port 17493)
|
||||
├─ /mcp (FastMCP, Streamable HTTP)
|
||||
└─ /speak (REST wrapper for non-MCP callers)
|
||||
└─ tools call existing services
|
||||
```
|
||||
|
||||
- **Transport:** Streamable HTTP as primary (Nov-2025 spec, post-SSE). Claude Code, Cursor, Windsurf, and the VS Code MCP extensions all support HTTP — it's the idiomatic shape for a long-running local service, which Voicebox already is.
|
||||
- **Stdio fallback:** `voicebox-mcp` binary bundled inside the app for clients that can't speak HTTP MCP. The Settings page renders the exact snippet with the detected absolute path — user copies, pastes, done. No PATH manipulation, no custom CLI wrapper.
|
||||
- **Identity:** HTTP clients set `X-Voicebox-Client-Id` header in their MCP config's `headers` block. Stdio clients set `VOICEBOX_CLIENT_ID` env var, which the shim forwards as the same HTTP header. Server reads it into a `ContextVar`.
|
||||
- **Profile resolution precedence:** explicit tool arg → per-client `MCPClientBinding.profile_id` → `capture_settings.default_playback_voice_id` → error.
|
||||
- **Port:** `17493`, matching `tauri/src-tauri/src/main.rs:63` (`SERVER_PORT` constant). Shim default with `VOICEBOX_PORT` env override.
|
||||
- **Non-MCP access:** `POST /speak` is a thin REST wrapper around the same tool path — one endpoint for shell scripts, ACP, A2A, and anything that isn't MCP-native.
|
||||
|
||||
## Library choice
|
||||
|
||||
- **`fastmcp`** (PyPI — verify on install whether the canonical import is `fastmcp` standalone or `mcp.server.fastmcp` from the consolidated `mcp` package; the API is identical).
|
||||
- **`sse-starlette`** for the `/events/speak` pill-state broadcast.
|
||||
- **`httpx` + `anyio`** already present — used by the shim.
|
||||
|
||||
## Data model
|
||||
|
||||
New table, **one row per client_id** (not a singleton — scales to unknown clients, maps 1:1 to the Settings UI list):
|
||||
|
||||
```python
|
||||
# backend/database/models.py
|
||||
class MCPClientBinding(Base):
|
||||
__tablename__ = "mcp_client_bindings"
|
||||
client_id = Column(String, primary_key=True) # "claude-code", "cursor", ...
|
||||
label = Column(String, nullable=True)
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=True)
|
||||
default_engine = Column(String, nullable=True)
|
||||
default_personality = Column(Boolean, nullable=False, default=False) # rewrite-before-speak default
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
```
|
||||
|
||||
Global default stays in `capture_settings.default_playback_voice_id` — no duplication. Migration: new `_migrate_mcp_client_bindings()` in `backend/database/migrations.py` using `CREATE TABLE IF NOT EXISTS`, mirroring the existing idempotent-add-column pattern.
|
||||
|
||||
## File plan
|
||||
|
||||
### Backend — new
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `backend/mcp/__init__.py` | Package marker |
|
||||
| `backend/mcp/server.py` | `build_mcp_server()` + `mount_into(app)`; registers tools, middleware, mount at `/mcp` |
|
||||
| `backend/mcp/tools.py` | The 4 `@mcp.tool()` functions — thin wrappers over existing services |
|
||||
| `backend/mcp/context.py` | `current_client_id: ContextVar[str \| None]` + Starlette middleware |
|
||||
| `backend/mcp/resolve.py` | `resolve_profile(explicit, client_id, db) -> VoiceProfile \| None` |
|
||||
| `backend/mcp/events.py` | In-memory `asyncio.Queue` pub/sub for speak-start / speak-end |
|
||||
| `backend/mcp/README.md` | MCP Inspector quickstart + `.mcp.json` snippets |
|
||||
| `backend/mcp_shim/__init__.py`, `__main__.py` | Stdio ↔ Streamable HTTP proxy (~150 lines) |
|
||||
| `backend/voicebox-mcp.spec` | PyInstaller spec for the shim (strips torch/transformers from `hiddenimports`) |
|
||||
| `backend/routes/speak.py` | `POST /speak {text, profile?, engine?, personality?, language?}` — REST wrapper around `resolve_profile()` + `generate_speech()` for non-MCP agents |
|
||||
|
||||
### Backend — modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `backend/app.py` | Migrate `@app.on_event("startup"/"shutdown")` (lines 185, 268) to `lifespan=` kwarg on `FastAPI()` using `AsyncExitStack`; call `mount_into(application)` after `register_routers`. Register `ClientIdMiddleware`. |
|
||||
| `backend/routes/profiles.py` | In `speak_in_character` (line 453): `events.publish("speak-start", {...})` on entry; completion hook publishes `speak-end`. Accept optional `source="mcp"` marker. |
|
||||
| `backend/services/generation.py` | `run_generation` completion path publishes `speak-end`. |
|
||||
| `backend/services/profiles.py` | New `async def get_profile_by_name_or_id(name_or_id, db)` — id lookup first, case-insensitive name fallback. |
|
||||
| `backend/database/models.py` | Add `MCPClientBinding`. |
|
||||
| `backend/database/migrations.py` | Add `_migrate_mcp_client_bindings`. |
|
||||
| `backend/models.py` | Add `MCPClientBindingResponse`, `MCPClientBindingUpdate`. |
|
||||
| `backend/routes/__init__.py` | Register `mcp_bindings_router`, `speak_router`, `events_router`. |
|
||||
| `backend/routes/mcp_bindings.py` (new) | REST CRUD for bindings (list, upsert, delete). |
|
||||
| `backend/routes/events.py` (new) | `GET /events/speak` — `EventSourceResponse` subscribed to the events queue. |
|
||||
| `backend/requirements.txt` | `+ fastmcp` (or `mcp>=1.0`), `+ sse-starlette` |
|
||||
| `backend/voicebox-server.spec` | `hiddenimports += ['mcp', 'mcp.server', 'fastmcp']` |
|
||||
| `backend/build_binary.py` | Second PyInstaller invocation for `voicebox-mcp.spec`; copy to `tauri/src-tauri/binaries/` with target-triple suffix |
|
||||
|
||||
### Frontend — new
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `app/src/components/ServerSettings/MCPBindings.tsx` | Settings section — default voice + per-client binding rows + `.mcp.json` copy-paste cheatsheet |
|
||||
| `app/src/lib/hooks/useMCPBindings.ts` | TanStack Query mirror of `useCaptureSettings` |
|
||||
| `app/src/lib/api/mcp.ts` | `listMCPBindings` / `upsertMCPBinding` / `deleteMCPBinding` |
|
||||
|
||||
### Frontend — modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `app/src/components/DictateWindow/DictateWindow.tsx` | Open `EventSource('/events/speak')`; on `speak-start` set pill to `speaking` with profile name; dismiss on `speak-end`. |
|
||||
| `app/src/components/CapturePill/CapturePill.tsx` | Add `speaking` branch — reuse the active waveform, swap status label to profile name. |
|
||||
| `app/src/lib/hooks/useCaptureRecordingSession.ts` | Union a `speaking` injection into the derived pill state. |
|
||||
| `app/src/lib/api/types.ts` | `MCPClientBinding`, `MCPClientBindingUpdate` types. |
|
||||
| `app/src/components/ServerSettings/index.tsx` | Register the new MCP section in the tab aggregator. |
|
||||
|
||||
### Tauri
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `tauri/src-tauri/tauri.conf.json` | `"externalBin": ["binaries/voicebox-server", "binaries/voicebox-mcp"]` |
|
||||
| `tauri/src-tauri/binaries/voicebox-mcp-<triple>` | Build artifact from PyInstaller |
|
||||
|
||||
## Tool signatures
|
||||
|
||||
All tools read `current_client_id.get()` (from middleware). Return JSON-serializable dicts.
|
||||
|
||||
Tools are registered with **dotted names** (`voicebox.speak`, etc.) to match the landing page and the industry convention (`filesystem.read_file`, `github.create_issue`). Python function names stay snake_case; the dot goes in the `name=` kwarg.
|
||||
|
||||
```python
|
||||
# backend/mcp/tools.py
|
||||
|
||||
@mcp.tool(name="voicebox.speak")
|
||||
async def speak(text: str,
|
||||
profile: str | None = None, # name OR id
|
||||
engine: str | None = None,
|
||||
personality: bool | None = None, # true → rewrite via profile's personality LLM before TTS
|
||||
language: str | None = None) -> dict:
|
||||
"""Speak text in a voice profile. Returns {generation_id, status, profile, poll}."""
|
||||
# resolve profile via precedence, delegate to generate_speech — the
|
||||
# route honors `personality=True` by running rewrite_as_profile on
|
||||
# the input before running the normal TTS pipeline.
|
||||
|
||||
@mcp.tool(name="voicebox.transcribe")
|
||||
async def transcribe(audio_base64: str | None = None,
|
||||
audio_path: str | None = None, # absolute local path
|
||||
language: str | None = None,
|
||||
model: str | None = None) -> dict:
|
||||
"""Transcribe audio. Exactly one of audio_base64/audio_path. Returns {text, duration, language}."""
|
||||
# validate path readable, size < 200 MB, then call services.transcribe.transcribe_bytes
|
||||
|
||||
@mcp.tool(name="voicebox.list_captures")
|
||||
async def list_captures(limit: int = 20, offset: int = 0) -> dict:
|
||||
"""Recent captures with transcripts. Returns {captures: [...]}"""
|
||||
|
||||
@mcp.tool(name="voicebox.list_profiles")
|
||||
async def list_profiles() -> dict:
|
||||
"""Available voice profiles. Returns {profiles: [{id, name, voice_type, has_personality}]}"""
|
||||
```
|
||||
|
||||
### `POST /speak` (non-MCP REST wrapper)
|
||||
|
||||
```python
|
||||
# backend/routes/speak.py
|
||||
@router.post("/speak", response_model=GenerationResponse)
|
||||
async def speak(data: SpeakRequest, request: Request, db: Session = Depends(get_db)):
|
||||
"""Same behavior as the MCP tool — for shell scripts, ACP, A2A, or anything non-MCP."""
|
||||
client_id = request.headers.get("X-Voicebox-Client-Id")
|
||||
profile = resolve_profile(data.profile, client_id, db)
|
||||
if profile is None: raise HTTPException(400, "No voice profile resolved.")
|
||||
req = GenerationRequest(profile_id=profile.id, text=data.text,
|
||||
language=data.language or "en",
|
||||
engine=data.engine or "qwen",
|
||||
personality=bool(data.personality))
|
||||
return await generate_speech(req, db)
|
||||
```
|
||||
|
||||
`SpeakRequest`: `{ text: str, profile: str | None, engine: str | None, personality: bool | None, language: str | None }`. Accepts name OR id for `profile` (via `resolve_profile`). `personality=None` means "use the per-client binding's `default_personality`"; explicit `true`/`false` always wins. Same precedence as the MCP tool so the two surfaces behave identically.
|
||||
|
||||
## Mount point (`backend/app.py`)
|
||||
|
||||
```python
|
||||
# After register_routers(application):
|
||||
from .mcp.server import mount_into
|
||||
mount_into(application)
|
||||
```
|
||||
|
||||
`mount_into` installs `ClientIdMiddleware` and calls `app.mount("/mcp", mcp.streamable_http_app())`.
|
||||
|
||||
**Lifespan migration is load-bearing** — FastMCP's session manager requires the `lifespan=` kwarg, not `@app.on_event`. Wrap the existing startup/shutdown bodies in an `@asynccontextmanager` using `contextlib.AsyncExitStack` so both Voicebox's init and FastMCP's session manager run. Verify dev + packaged build after the migration.
|
||||
|
||||
## Stdio shim (`backend/mcp_shim/__main__.py`)
|
||||
|
||||
1. Port: `int(os.environ.get("VOICEBOX_PORT", "17493"))`.
|
||||
2. Client id: `os.environ.get("VOICEBOX_CLIENT_ID", "unknown")`.
|
||||
3. Health probe `GET /health` with 30 s tolerance (torch imports slowly). On failure, emit JSON-RPC error on stdout, exit 1.
|
||||
4. Connect Streamable HTTP MCP client to `http://127.0.0.1:{port}/mcp` with `X-Voicebox-Client-Id: {client_id}` header.
|
||||
5. Proxy JSON-RPC bidirectionally — stdin → HTTP, SSE → stdout. Use `mcp` SDK's built-in stdio↔HTTP bridge if available; otherwise ~40 lines of asyncio.
|
||||
6. Stdout = JSON-RPC only. All logs to stderr.
|
||||
|
||||
PyInstaller spec keeps only `mcp`, `httpx`, `anyio`, `click` — target binary <20 MB.
|
||||
|
||||
## Pill `speaking` state
|
||||
|
||||
- `backend/mcp/events.py`: module-level `_subscribers: list[asyncio.Queue]` + `publish(kind, payload)` + `subscribe() -> Queue`.
|
||||
- `speak_in_character` publishes `speak-start` with `{generation_id, profile_id, profile_name, source}` immediately after `task_manager.start_generation`; `run_generation`'s completion path publishes `speak-end`.
|
||||
- `/events/speak` → `EventSourceResponse`.
|
||||
- `DictateWindow` opens `EventSource` next to existing `dictate:*` listeners, maps `speak-start/end` → pill `speaking` mode with profile name.
|
||||
- Optional filter: only show pill when `source === "mcp"` (avoids pill churn during manual speak flows). Settings toggle later.
|
||||
|
||||
## Settings UI (`MCPBindings.tsx`)
|
||||
|
||||
- **Global default voice** picker bound to `capture_settings.default_playback_voice_id` (reuses `useCaptureSettings`).
|
||||
- **Per-client table** — add/edit/remove rows of `{client_id, label, profile_id, default_engine, default_personality}`. Uses `useMCPBindings`.
|
||||
- **Connection cheatsheet** — two tabs, HTTP (default) and Stdio (fallback), with copy-to-clipboard snippets per known client:
|
||||
|
||||
HTTP form (primary):
|
||||
```json
|
||||
{"mcpServers": {"voicebox": {
|
||||
"url": "http://127.0.0.1:17493/mcp",
|
||||
"headers": {"X-Voicebox-Client-Id": "claude-code"}
|
||||
}}}
|
||||
```
|
||||
|
||||
Stdio form (fallback, absolute path auto-filled from detected app location):
|
||||
```json
|
||||
{"mcpServers": {"voicebox": {
|
||||
"command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp",
|
||||
"env": {"VOICEBOX_CLIENT_ID": "claude-code"}
|
||||
}}}
|
||||
```
|
||||
|
||||
Plus the Claude-Code-specific one-liner:
|
||||
```
|
||||
claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"
|
||||
```
|
||||
- **One-click install buttons** for known clients (v1: Claude Code via `claude mcp add` invocation, and a config-file writer for Cursor/Windsurf whose config locations are known). Each has a matching "Remove" button. Hide buttons for clients not detected on disk.
|
||||
- **Connection status** — small indicator next to each binding showing the last time that `client_id` actually called the server (rolling timestamp recorded by middleware), so users can tell their install worked.
|
||||
|
||||
## Ordered task list (shortest path first)
|
||||
|
||||
1. `fastmcp` + `sse-starlette` → `backend/requirements.txt`; install.
|
||||
2. Add `backend/mcp/{server,tools,context,resolve}.py` with the 4 tools registered as `voicebox.speak` etc. (no middleware yet — global default profile only).
|
||||
3. Migrate `app.py` to `lifespan=`; mount FastMCP at `/mcp`.
|
||||
4. **Milestone:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp` — call `voicebox.speak`, hear audio.
|
||||
5. Add `get_profile_by_name_or_id`; wire the tool's `profile` arg.
|
||||
6. `MCPClientBinding` model + migration; middleware; full `resolve_profile` precedence.
|
||||
7. `backend/routes/speak.py` — `POST /speak` REST wrapper, reusing `resolve_profile` + `speak_in_character`.
|
||||
8. `/mcp/bindings` REST + `MCPBindings.tsx` UI with HTTP and stdio copy-snippets, one-click install for detected clients, and connection-status indicators. **Users can install Voicebox as an MCP server after this step.**
|
||||
9. `backend/mcp_shim/__main__.py` + PyInstaller spec + `build_binary.py` second pass; register `voicebox-mcp` as a Tauri sidecar. (Fallback path goes live.)
|
||||
10. Events queue + `/events/speak` SSE + `DictateWindow` `speaking` pill state.
|
||||
11. `backend/mcp/README.md` quickstart.
|
||||
|
||||
Claude Code can call `voicebox.speak` after step 4 (direct HTTP, manual config). Step 8 makes that a one-click experience. Step 9 adds the stdio fallback for clients that don't speak HTTP MCP.
|
||||
|
||||
## Verification
|
||||
|
||||
- **Step 4 smoke:** `npx @modelcontextprotocol/inspector http://127.0.0.1:17493/mcp`. Call `voicebox.list_profiles`, then `voicebox.speak(text="hello from mcp")`. Audio plays; generation appears in History with `source="personality_speak"` (or new `source="mcp"` if we add one).
|
||||
- **REST wrapper:** `curl -X POST http://127.0.0.1:17493/speak -d '{"text":"hi","profile":"Morgan"}'` — same behavior, same pill surface.
|
||||
- **Per-client:** open two Inspector sessions with different `X-Voicebox-Client-Id` headers, bind each to a different profile in Settings, verify distinct voices without `profile` arg.
|
||||
- **Claude Code end-to-end (HTTP):** `claude mcp add voicebox --transport http --url http://127.0.0.1:17493/mcp --header "X-Voicebox-Client-Id: claude-code"`, then ask Claude Code to speak. Pill shows `speaking: <profile>`, audio plays, capture appears in history.
|
||||
- **Stdio fallback:** manually paste the stdio snippet from Settings into a client's config, verify same behavior. `VOICEBOX_CLIENT_ID=claude-code python -m backend.mcp_shim` while backend is up; pipe a tools/list JSON-RPC in, verify response over stdout.
|
||||
- **Transcribe:** point at `/tmp/test.wav`; diff against `POST /transcribe` response.
|
||||
- **Failure modes:** kill backend mid-speak — shim must surface a JSON-RPC error, not deadlock. When backend isn't running, HTTP clients should get a clear connection-refused surfaced by the client.
|
||||
|
||||
## Risks / open decisions
|
||||
|
||||
- **`fastmcp` vs `mcp` package name** — confirm on `pip install`; APIs are near-identical, adjust imports.
|
||||
- **Lifespan migration** touches critical path (DB init, task queue, watchdog). Dev + packaged build both need a smoke after.
|
||||
- **Shim binary size** — if `mcp` pulls in enough dep weight that PyInstaller output is awkward, fall back to a Rust shim (Tauri shell is already Rust; JSON-RPC framing is trivial).
|
||||
- **Source attribution** — consider `source="mcp"` on the `Generation` model, or a dedicated `originator_client` column, if the Captures tab should filter MCP-originated generations.
|
||||
- **`audio_path` in `voicebox_transcribe`** — local-only today, but if the server ever binds beyond 127.0.0.1 we need to restrict reads to `data_dir` + user-whitelist.
|
||||
- **Auth** — none for now (127.0.0.1 only). If we bind outside, bearer token via `~/.voicebox/secret` + plumb through shim.
|
||||
- **HTTP MCP client support** — the plan leads with direct HTTP. Claude Code, Cursor, Windsurf, and VS Code MCP extensions all support it as of 2026, but if we discover an important client is stdio-only we still have the shim fallback ready.
|
||||
- **`.mcpb` desktop extension for Claude Desktop** (v2 polish) — Claude Desktop supports a double-clickable extension bundle format. Worth revisiting after v1 ships for an even cleaner install; skipped for now since Claude Desktop isn't the primary user (Claude Code + IDE users are).
|
||||
|
||||
## Critical files
|
||||
|
||||
- `backend/app.py`
|
||||
- `backend/routes/profiles.py`
|
||||
- `backend/routes/speak.py` (new)
|
||||
- `backend/database/models.py`
|
||||
- `backend/database/migrations.py`
|
||||
- `backend/services/generation.py`
|
||||
- `backend/build_binary.py`
|
||||
- `tauri/src-tauri/tauri.conf.json`
|
||||
- `tauri/src-tauri/src/main.rs` (port constant — no change, just reference)
|
||||
- `app/src/components/DictateWindow/DictateWindow.tsx`
|
||||
- `app/src/components/CapturePill/CapturePill.tsx`
|
||||
- `app/src/components/ServerSettings/`
|
||||
@@ -0,0 +1,235 @@
|
||||
# OpenAI API Compatibility
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
|
||||
**Issue:** [#10 OpenAI API compatibility](https://github.com/jamiepine/voicebox/issues/10)
|
||||
|
||||
## Overview
|
||||
|
||||
This feature exposes OpenAI-compatible endpoints from Voicebox, allowing any tool, library, or application that speaks the OpenAI Audio API to use Voicebox as a drop-in local replacement.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph clients [External Clients]
|
||||
SDK[OpenAI SDK]
|
||||
Curl[curl / HTTP]
|
||||
Apps[Third-party Apps]
|
||||
end
|
||||
|
||||
subgraph voicebox [Voicebox Server]
|
||||
OpenAI["/v1/audio/* endpoints"]
|
||||
TTS[TTSModel]
|
||||
Whisper[WhisperModel]
|
||||
Profiles[Voice Profiles]
|
||||
end
|
||||
|
||||
SDK --> OpenAI
|
||||
Curl --> OpenAI
|
||||
Apps --> OpenAI
|
||||
OpenAI --> TTS
|
||||
OpenAI --> Whisper
|
||||
OpenAI --> Profiles
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **OpenAI SDK users**: `openai.audio.speech.create()` works with Voicebox
|
||||
- **LLM frameworks**: LangChain, AutoGen, etc. can use Voicebox for TTS
|
||||
- **Shell scripts**: `curl` commands copy-pasted from OpenAI docs work
|
||||
- **Existing integrations**: Any tool expecting OpenAI's API works without code changes
|
||||
|
||||
## Endpoints to Implement
|
||||
|
||||
### 1. `POST /v1/audio/speech` (TTS)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createSpeech
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "tts-1",
|
||||
"input": "Hello world!",
|
||||
"voice": "alloy",
|
||||
"response_format": "mp3",
|
||||
"speed": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
**Response:** Audio file (mp3, wav, opus, aac, flac, pcm)
|
||||
|
||||
**Voice Mapping Strategy:**
|
||||
|
||||
- `voice` parameter maps to Voicebox profile names (case-insensitive)
|
||||
- If no match, use a configurable default profile
|
||||
- Support special syntax: `voice: "profile:uuid"` for explicit profile ID
|
||||
|
||||
### 2. `POST /v1/audio/transcriptions` (Whisper)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createTranscription
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `file`: Audio file
|
||||
- `model`: "whisper-1"
|
||||
- `language`: Optional language hint
|
||||
- `response_format`: json, text, srt, verbose_json, vtt
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello world!"
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### New File: `backend/openai_compat.py`
|
||||
|
||||
Create a dedicated module with an APIRouter for OpenAI-compatible endpoints:
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Literal, Optional
|
||||
|
||||
router = APIRouter(prefix="/v1/audio", tags=["OpenAI Compatible"])
|
||||
|
||||
class SpeechRequest(BaseModel):
|
||||
model: str = "tts-1"
|
||||
input: str
|
||||
voice: str = "alloy"
|
||||
response_format: Literal["mp3", "wav", "opus", "aac", "flac", "pcm"] = "mp3"
|
||||
speed: float = 1.0
|
||||
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest, db: Session = Depends(get_db)):
|
||||
# 1. Map voice name to profile
|
||||
# 2. Generate audio using existing TTSModel
|
||||
# 3. Convert to requested format
|
||||
# 4. Return audio stream
|
||||
...
|
||||
|
||||
@router.post("/transcriptions")
|
||||
async def create_transcription(
|
||||
file: UploadFile = File(...),
|
||||
model: str = Form("whisper-1"),
|
||||
language: Optional[str] = Form(None),
|
||||
response_format: str = Form("json"),
|
||||
):
|
||||
# 1. Save uploaded file
|
||||
# 2. Transcribe using existing WhisperModel
|
||||
# 3. Return in requested format
|
||||
...
|
||||
```
|
||||
|
||||
### Voice Profile Resolution
|
||||
|
||||
Add helper in [backend/profiles.py](backend/profiles.py):
|
||||
|
||||
```python
|
||||
async def resolve_voice_for_openai(voice: str, db: Session) -> Optional[VoiceProfile]:
|
||||
"""
|
||||
Resolve OpenAI voice parameter to a Voicebox profile.
|
||||
|
||||
Priority:
|
||||
1. Exact profile name match (case-insensitive)
|
||||
2. Profile ID match (if voice starts with "profile:")
|
||||
3. Default profile from config
|
||||
4. First available profile
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### Audio Format Conversion
|
||||
|
||||
Add conversion utilities in [backend/utils/audio.py](backend/utils/audio.py):
|
||||
|
||||
```python
|
||||
def convert_audio_format(
|
||||
audio: np.ndarray,
|
||||
sample_rate: int,
|
||||
target_format: str, # mp3, wav, opus, aac, flac, pcm
|
||||
) -> bytes:
|
||||
"""Convert audio to target format using ffmpeg or pydub."""
|
||||
...
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to [backend/config.py](backend/config.py):
|
||||
|
||||
```python
|
||||
# OpenAI API Compatibility
|
||||
OPENAI_COMPAT_ENABLED = True
|
||||
OPENAI_COMPAT_DEFAULT_VOICE = None # Profile ID or name for default voice
|
||||
OPENAI_COMPAT_REQUIRE_AUTH = False # Require API key validation
|
||||
OPENAI_COMPAT_API_KEY = None # If set, validate against this
|
||||
```
|
||||
|
||||
### Integration with main.py
|
||||
|
||||
In [backend/main.py](backend/main.py), include the router:
|
||||
|
||||
```python
|
||||
from . import openai_compat
|
||||
|
||||
# Add OpenAI-compatible routes
|
||||
if config.OPENAI_COMPAT_ENABLED:
|
||||
app.include_router(openai_compat.router)
|
||||
```
|
||||
|
||||
## Streaming Support (Future Enhancement)
|
||||
|
||||
Initial implementation returns complete audio. Streaming can be added later:
|
||||
|
||||
```python
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest):
|
||||
if request.stream:
|
||||
return StreamingResponse(
|
||||
generate_audio_chunks(request),
|
||||
media_type=f"audio/{request.response_format}"
|
||||
)
|
||||
...
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Example usage after implementation:
|
||||
|
||||
```bash
|
||||
# TTS with curl
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Hello!", "voice": "MyProfile"}' \
|
||||
--output speech.mp3
|
||||
|
||||
# With OpenAI Python SDK
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.audio.speech.create(
|
||||
model="tts-1",
|
||||
voice="MyProfile",
|
||||
input="Hello world!"
|
||||
)
|
||||
response.stream_to_file("output.mp3")
|
||||
|
||||
# Transcription
|
||||
curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F file=@audio.mp3 \
|
||||
-F model="whisper-1"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Optional API key validation (for shared deployments)
|
||||
- Rate limiting on endpoints
|
||||
- Input length limits (same as existing `/generate` endpoint)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `pydub` or `ffmpeg-python` for audio format conversion (mp3, opus, etc.)
|
||||
- No changes to existing TTS/Whisper model code
|
||||
@@ -0,0 +1,675 @@
|
||||
# Voice I/O
|
||||
|
||||
**Status:** Shipping — phases 1, 2, 4, 7 (macOS) complete · 3 partial · 5, 6, 7 (Windows/Linux), 8 pending
|
||||
**Touches:** backend, Tauri shell, frontend, a new native shim crate
|
||||
**Last reviewed:** 2026-04-21
|
||||
|
||||
## Progress
|
||||
|
||||
### Shipped
|
||||
|
||||
**Phase 1 — Groundwork.** Audio tab retired from the sidebar; its device / channel
|
||||
config lives under Settings. Captures tab is live at `/captures` with no feature
|
||||
flag.
|
||||
|
||||
**Phase 2 — Local LLM backend.** `LLMBackend` protocol alongside the existing
|
||||
TTS/STT backends. `qwen_llm_backend.py`, `services/llm.py`, `routes/llm.py`, and
|
||||
a shared model-download / cache pipeline. Qwen3 0.6B / 1.7B / 4B registered and
|
||||
user-selectable via `capture_settings.llm_model`.
|
||||
|
||||
**Phase 4 — Captures tab.** List + detail view, source badges (dictation /
|
||||
recording / file), retranscribe, refine (flags + model resolved from a
|
||||
server-side `capture_settings` singleton), delete, and the Play-as-voice
|
||||
dropdown over every profile.
|
||||
|
||||
### Partial
|
||||
|
||||
**Phase 3 — In-app voice input.** `CapturesTab` dictates end-to-end via
|
||||
`useCaptureRecordingSession`, which the Phase 7 floating pill also consumes.
|
||||
Outstanding: a universal mic button on other text inputs (Generate form,
|
||||
profile descriptions, story titles, etc.), and the streaming
|
||||
`/transcribe/stream` WebSocket — today's flow is a single `POST /captures`
|
||||
with the complete audio blob.
|
||||
|
||||
**Phase 7 — External dictation shell (macOS).** Both halves shipped on macOS.
|
||||
|
||||
Hotkey half:
|
||||
|
||||
- `tauri/src-tauri/src/chord_engine.rs` — pure state machine. Unit tests green.
|
||||
- `tauri/src-tauri/src/hotkey_monitor.rs` — `rdev`-based global listener on a
|
||||
background thread, with `set_is_main_thread(false)` applied to sidestep the
|
||||
macOS 14+ TSM crash ([Narsil/rdev#165](https://github.com/Narsil/rdev/issues/165)).
|
||||
Right-hand-only defaults preserve left-hand Cmd+Option+I devtools.
|
||||
- Default bindings hardcoded: `Cmd+Option` (push-to-talk) and
|
||||
`Cmd+Option+Space` (toggle-to-talk). The PTT → Toggle upgrade transition is
|
||||
preserved — adding Space mid-hold promotes the session without interrupting
|
||||
audio.
|
||||
- `DictateWindow` — transparent, always-on-top, borderless 420×64 webview
|
||||
pre-created hidden at app setup. Shows on chord-start, hides on
|
||||
capture-cycle completion. Error state on the pill auto-dismisses and
|
||||
copies-to-clipboard on click.
|
||||
|
||||
Paste half (macOS):
|
||||
|
||||
- `clipboard.rs` — `NSPasteboard` snapshot that walks `pasteboardItems` and
|
||||
copies every `(uti, bytes)` pair so multi-type content (images, styled
|
||||
text, file refs) survives the round-trip. `save_clipboard`,
|
||||
`write_text`, `restore_clipboard`, `current_change_count`.
|
||||
- `synthetic_keys.rs` — `CGEventPost` at the HID tap with the full four-event
|
||||
Cmd+V sequence (Cmd down → V down w/ flag → V up w/ flag → Cmd up).
|
||||
- `focus_capture.rs` — `AXUIElementCreateSystemWide` +
|
||||
`AXUIElementCopyAttributeValue(kAXFocusedUIElement)` +
|
||||
`AXUIElementGetPid`, with the AX attribute key CFStrings built at
|
||||
runtime because they're CFSTR macros, not linkable symbols.
|
||||
`NSRunningApplication.activateWithOptions:` for re-activation.
|
||||
- `accessibility.rs` — `AXIsProcessTrusted` gate.
|
||||
- `paste_final_text` command — activate → 120 ms settle → save clip →
|
||||
write text → ⌘V → 400 ms → restore. Skips when focus was in Voicebox
|
||||
itself.
|
||||
- Focus rides the `dictate:start` event payload; `DictateWindow` holds the
|
||||
snapshot in a ref and consume-once-nulls on paste so a late-arriving
|
||||
refine from an earlier session can't misfire.
|
||||
- Dictation recording no longer hard-caps at 29 s — the limit still
|
||||
applies to voice-profile reference clips.
|
||||
|
||||
Outstanding: Windows `SendInput` / UIAutomation / `SetForegroundWindow`
|
||||
equivalents, Linux `uinput` / AT-SPI equivalents (and the Wayland story),
|
||||
first-run Accessibility prompt UI with deep-link to System Settings,
|
||||
direct-injection path for focus-was-inside-Voicebox (step 6 — dictating
|
||||
into our own Generate tab currently falls back to the capture list).
|
||||
|
||||
### Not started
|
||||
|
||||
- **Phase 5 — Agent voice output + persona loop.** No `/speak` endpoint, no
|
||||
`voicebox.speak` MCP tool, no per-agent voice binding, no persona metadata
|
||||
on profiles.
|
||||
- **Phase 6 — STT engine expansion.** Only Whisper (`mlx_backend.py`).
|
||||
Parakeet v3, Qwen3-ASR, Kyutai — all unregistered.
|
||||
- **Phase 8 — Pipeline routing, sinks, long-form.** No preset primitive, no
|
||||
MCP sink, no webhook sink, no dual-stream recorder, no summary transform.
|
||||
|
||||
### Additionally landed (not explicit in the original plan)
|
||||
|
||||
These fell out of the Phase 3/4/7 work but deserve their own mention:
|
||||
|
||||
- **Server-authoritative settings.** Singleton `capture_settings` and
|
||||
`generation_settings` tables. The client sends nothing but the audio; STT
|
||||
model, refine flags, refine LLM, and the auto-refine flag are all resolved
|
||||
server-side, so sibling Tauri webviews can't go stale.
|
||||
- **Backend audio normalisation.** `POST /captures` transcodes anything
|
||||
librosa can decode (webm/opus, m4a, etc.) to WAV before handing it to
|
||||
whisper, side-stepping miniaudio's format gaps inside mlx-audio.
|
||||
- **Short-recording guard.** Sub-300 ms blobs short-circuit client-side so a
|
||||
fumbled chord tap never uploads an empty webm.
|
||||
- **Refinement prompt.** Rewritten with firmer anti-chatbot framing and
|
||||
inline examples covering multi-sentence preservation and self-correction.
|
||||
|
||||
### Near-term outstanding
|
||||
|
||||
Called out in recent sessions but not yet in a phase:
|
||||
|
||||
- **Configurable chord bindings.** Pass 2 of the hotkey work — persist
|
||||
`push_to_talk_chord` / `toggle_to_talk_chord` in `capture_settings`,
|
||||
surface a chord-picker UI in `CapturesPage`, and wire a Tauri
|
||||
`update_chord_bindings` command so `HotkeyMonitor::update_bindings` picks
|
||||
up user changes live.
|
||||
- **Generate-tab empty-state explainer.** The parallel aside to the Captures
|
||||
explainer described in *Product surface → Parallel explainer on the
|
||||
Generate tab*. Lands alongside Phase 3's universal mic button so both tabs
|
||||
feel symmetric.
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox ships the output half of a voice I/O loop: clone a voice, generate
|
||||
speech, apply effects, compose multi-voice projects. The input half — speech to
|
||||
text, dictation, routing — exists today as a single Whisper model wired into the
|
||||
Recording & Transcription panel. This doc proposes making voice *input* a
|
||||
first-class pillar: more STT engines, a dictation shell (global hotkey, audio
|
||||
capture, paste, streaming), a local LLM backend, and a user-configurable
|
||||
pipeline from captured audio to whatever the user wants to do with it.
|
||||
|
||||
Positioning is the key move. **Voicebox becomes the local voice I/O layer for
|
||||
humans and AI agents** — a local alternative to cloud dictation tools, with the
|
||||
differentiator that we also do TTS and voice cloning. The same app that
|
||||
captures your voice can generate a response in any voice profile you've
|
||||
cloned. "Anything voice is Voicebox."
|
||||
|
||||
### Positioning shift
|
||||
|
||||
Before this plan, Voicebox was **"the open-source AI voice cloning studio."**
|
||||
Cloning was the headline capability.
|
||||
|
||||
After this plan, Voicebox is **"the open-source AI voice studio."** Cloning is
|
||||
one capability in a broader category that now spans input (STT, dictation),
|
||||
intelligence (local LLM, refinement, persona), output (TTS, cloning, effects,
|
||||
Stories), and routing. The word "cloning" drops out of the top-line descriptor
|
||||
because it's become a feature rather than the thesis.
|
||||
|
||||
### Competitive frame
|
||||
|
||||
Voicebox ends up covering the territory of two separately-funded, separately
|
||||
branded cloud incumbents that operate on opposite sides of the same voice I/O
|
||||
loop:
|
||||
|
||||
- **ElevenLabs** (~$3B+): voice cloning and TTS — the "agents speak" side
|
||||
- **WisprFlow** (~$70M raised): voice dictation for agents and power users —
|
||||
the "users talk" side
|
||||
|
||||
Both are cloud-only. Voicebox becomes the only local alternative to either,
|
||||
running in one app, with a single model directory and LLM shared between input
|
||||
and output. That bridging — dictation → LLM → TTS with a cloned voice in the
|
||||
middle — is the thing no single incumbent can match, because neither has the
|
||||
other half.
|
||||
|
||||
### Launch-time copy tasks
|
||||
|
||||
These are not engineering tasks but should ride the Phase 4 ship so marketing
|
||||
and positioning stay in sync with the product.
|
||||
|
||||
- **README.md** — drop "cloning" from the top-line descriptor. Add a section
|
||||
that explicitly frames Voicebox as "the open-source local alternative to
|
||||
WisprFlow and ElevenLabs." Competitive framing belongs in the README and on
|
||||
the landing page — not in-app (reads as defensive).
|
||||
- **voicebox.sh landing page** — same positioning shift.
|
||||
- **GitHub About / repo topics** — swap "voice-cloning" or similar tags for
|
||||
broader "voice-io," "local-tts," "local-stt," etc.
|
||||
- **Release notes** — the Phase 4 launch note is the "we're now voice I/O" moment.
|
||||
|
||||
## Why now
|
||||
|
||||
- Cross-platform local dictation is an empty category. The tools people love
|
||||
(Superwhisper, MacWhisper, Aiko) are macOS-only. WisprFlow and
|
||||
Willow are cloud. Our Windows install base is the wedge — first-class Windows
|
||||
support for a local dictation product is genuinely differentiated.
|
||||
- The `STTBackend` protocol already exists. The multi-engine registry pattern
|
||||
shipped with TTS makes adding Parakeet v3 and Qwen3-ASR a days-not-weeks
|
||||
effort on the backend side.
|
||||
- The **persona loop** — speak to an agent, have it reply in a cloned voice —
|
||||
is a feature only we can ship. Nobody with a dictation product has TTS; nobody
|
||||
with a TTS product has good dictation. The full duplex is ours.
|
||||
- Agent harnesses already pipe Voicebox TTS into their stacks. Giving those
|
||||
users STT from the same app closes the loop and makes Voicebox the default
|
||||
voice I/O layer for the agentic dev-tool crowd.
|
||||
- **Typing a 2,000-character TTS script is user-hostile.** The most immediate
|
||||
internal win is dictating directly into Voicebox's own generation form —
|
||||
speak the script, generate the voice. This dogfoods the whole STT pipeline
|
||||
without touching a single OS-level API.
|
||||
- **Voice-to-voice models are landing.** Moshi (Kyutai), GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM, Spirit LM (Meta) — end-to-end speech LLMs that
|
||||
take audio in and emit audio out are a near-term reality. The pipeline we're
|
||||
building today is the scaffolding they slot into tomorrow.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Cloud fallback or "bring your own API key" STT/LLM. Local is the product.
|
||||
- A separate tray-only dictation app. We extend Voicebox, not fork it.
|
||||
- Replacing the Stories editor with a notes layout. Long-form capture is a
|
||||
preset on top of the pipeline, not a new product surface.
|
||||
- Real-time translation UI. It can exist as a transform later, but it's not in
|
||||
this plan.
|
||||
- Full agent orchestration. We provide the voice rails; the agent lives
|
||||
elsewhere and talks to us via the developer API.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Three new backend concepts
|
||||
|
||||
**1. Expanded STT registry.** The existing `STTBackend` protocol abstracts
|
||||
Whisper today. Add:
|
||||
|
||||
- **Parakeet v3** — 25 languages, very fast, the current quality leader for
|
||||
non-English local STT. Python path via `nemo_toolkit` or `transformers`.
|
||||
- **Qwen3-ASR 0.6B int8** — 50+ languages, highest multilingual quality,
|
||||
cross-platform via `transformers`.
|
||||
- **Kyutai ASR** *(optional)* — streaming-first, small, CPU-friendly. Fills the
|
||||
"CPU-only laptop" tier.
|
||||
|
||||
All register via `ModelConfig` and use the same download, cache, and model
|
||||
management UI we already have for TTS. Zero special-casing.
|
||||
|
||||
**2. `LLMBackend` protocol.** Mirror of `TTSBackend` / `STTBackend`. First
|
||||
implementations are Qwen3 0.6B / 1.7B / 4B running on the same PyTorch + MLX
|
||||
infrastructure we already run. One runtime, one model cache, one GPU-memory
|
||||
story.
|
||||
|
||||
Why not `llama.cpp` or `ollama`: we already have the dependency surface and the
|
||||
model download UX. A second runtime fragments cache directories and model-status
|
||||
UI. If CPU-only Windows latency becomes a problem we can revisit.
|
||||
|
||||
**3. Streaming transcribe transport.** Add `/transcribe/stream` as a WebSocket
|
||||
endpoint alongside the existing HTTP `/transcribe`. Audio frames flow in,
|
||||
partial transcripts stream back. Same FastAPI process, same loaded models. This
|
||||
keeps dictation latency off the per-request JSON-encode critical path and lets
|
||||
us ship real-time partial transcripts later without a protocol change.
|
||||
|
||||
### The pipeline abstraction
|
||||
|
||||
Every captured audio event flows through the same shape:
|
||||
**Source → Transforms → Sink(s)**. Users configure presets that bind a source
|
||||
to a transform chain to one or more sinks.
|
||||
|
||||
```
|
||||
Source Transform Sink
|
||||
────────────────── ───────────────── ─────────────────
|
||||
Hold to speak ──┐ STT model Clipboard + paste
|
||||
Tap to toggle │ Refinement LLM Capture history
|
||||
Long-form recorder ├──▶ Persona LLM ──▶ File on disk
|
||||
File drop │ Translation (later) HTTP webhook
|
||||
API call (WS / HTTP) ──┘ MCP server sink
|
||||
TTS loopback (persona)
|
||||
Platform sinks (later)
|
||||
```
|
||||
|
||||
`Source → Transform → Sink` is internal, dataflow-style vocabulary (same shape
|
||||
as Unix pipes, Apache Beam, Kafka) — not user-facing. The UI surface will use
|
||||
Voicebox-native language (see open questions).
|
||||
|
||||
Concrete preset examples this shape enables:
|
||||
|
||||
- **Dictation** — hold-to-speak → Parakeet v3 → light refinement → clipboard + paste + history
|
||||
- **Code prompt** — dedicated hotkey → Whisper Turbo → technical-vocab refinement → MCP sink for Claude Code
|
||||
- **Agent voice reply** — hold-to-speak → STT → persona LLM → TTS with cloned profile → system audio out
|
||||
- **Long-form capture** — dual-stream recorder → chunked STT → summary LLM → markdown file + history
|
||||
|
||||
Every user-facing feature collapses into (source + transform chain + sinks).
|
||||
Meeting-style capture isn't a separate product; it's a preset. Competing tools
|
||||
hardcode integrations (Trello, Granola); we make routing user-configurable.
|
||||
|
||||
### Native shim crate
|
||||
|
||||
The parts Tauri doesn't handle cleanly, gathered in one Rust crate with a
|
||||
platform-agnostic API:
|
||||
|
||||
- **Global hotkey with modifier-only support.** Tauri's `global-shortcut`
|
||||
plugin requires full combos. We need "hold right-cmd" or "hold ctrl" as
|
||||
primitives. On macOS this means a CGEventTap on a background thread with
|
||||
polling fallback for dropped modifier events; on Windows a low-level keyboard
|
||||
hook; on Linux X11 + libinput, with Wayland as a known gap.
|
||||
- **Focus introspection.** Query the frontmost app and its focused element via
|
||||
OS accessibility APIs — `AXUIElement` on macOS, UIAutomation on Windows,
|
||||
AT-SPI on Linux. Check the element's role to decide between a direct
|
||||
injection, a clipboard + paste, and a clipboard-only fallback with a
|
||||
notification. A blind paste that only "works when a text field happens to
|
||||
be focused" is the easy default; we should make the decision deliberately.
|
||||
- **Simulated paste.** CGEvent on macOS, SendInput on Windows, uinput / ydotool
|
||||
on Linux. Wayland is the hard case and needs explicit handling.
|
||||
- **Atomic clipboard save/restore.** Save *all* items and *all* MIME
|
||||
representations before writing our transcript, restore atomically after
|
||||
paste. Pasting a transcript shouldn't clobber a user's in-progress rich-media
|
||||
clipboard.
|
||||
- **Frontmost-window context capture** *(later).* macOS Vision, Windows OCR,
|
||||
Linux tesseract. Optional feature to feed the refinement LLM disambiguation
|
||||
hints from the window being pasted into.
|
||||
|
||||
Main process owns this crate. Webview never sees platform differences.
|
||||
|
||||
### Target-aware delivery
|
||||
|
||||
The paste sink adapts to what's in focus. This is a single sink type with
|
||||
branching behavior, not four separate sinks.
|
||||
|
||||
| Target | Delivery strategy |
|
||||
|---|---|
|
||||
| Focused text field inside Voicebox | Direct React state update via event. No clipboard involved. |
|
||||
| Focused text field in another app | Accessibility-verified paste: save clipboard, write transcript, simulate paste, restore clipboard. |
|
||||
| No text focus detected | Clipboard only, toast notification ("Transcript copied — no text field focused"). |
|
||||
| Platform-specific special cases (terminal apps, specific editors) | Per-app overrides where the generic path misbehaves. |
|
||||
|
||||
### Where each concern lives
|
||||
|
||||
| Concern | Layer |
|
||||
|---|---|
|
||||
| STT / LLM / TTS inference | Python backend |
|
||||
| Model downloads, progress, cache | Python backend |
|
||||
| Pipeline runner (orchestrates transforms and sinks) | Python backend |
|
||||
| Audio capture from mic / system audio | Rust (Tauri side) |
|
||||
| Audio streaming over WebSocket to backend | Rust |
|
||||
| Global hotkey capture | Rust (native shim crate) |
|
||||
| Paste simulation, clipboard save/restore | Rust (native shim crate) |
|
||||
| Pipeline preset UI, capture history, settings | React |
|
||||
|
||||
Model work in Python. OS work in Rust. User config in React.
|
||||
|
||||
## Product surface
|
||||
|
||||
### A new tab (and a sidebar reshuffle)
|
||||
|
||||
The current sidebar is `Generate · Stories · Voices · Effects · Audio · Models ·
|
||||
Settings`. The existing Audio tab is output-device and channel routing
|
||||
config — infrastructure, not a creative workspace — and the Settings page
|
||||
already has a sub-tab pattern (`ServerSettings/`: Connection, Models, GPU,
|
||||
Update) that fits it naturally.
|
||||
|
||||
**Move Audio to a Settings sub-tab. Reclaim the sidebar slot for voice input.**
|
||||
|
||||
The new tab shows recent captures (audio + transcript paired), active presets,
|
||||
dictation settings, model pickers for STT and LLM. Exact name is an open
|
||||
question.
|
||||
|
||||
**Sidebar placement:** Captures sits at position 3, directly under Stories and
|
||||
above Voices. Creates an "input voice / output voice" adjacency — captured
|
||||
speech is one slot away from the voices you can play it back through, which
|
||||
mirrors the Phase 4 "Play as voice" feature's mental model. Full order:
|
||||
Generate · Stories · Captures · Voices · Effects · Models · Settings.
|
||||
|
||||
### Parallel explainer on the Generate tab
|
||||
|
||||
The Captures settings page gets a "What's different" aside that introduces
|
||||
Voicebox's dictation story. The Generate tab deserves a parallel — first-time
|
||||
users need to be told what voice generation is *for* in a post-Voice-I/O
|
||||
world, not just handed a text field.
|
||||
|
||||
Shape: an **empty-state card** rendered in the Generate tab when there's no
|
||||
generation history yet, disappearing once the user has generated anything.
|
||||
Teaches without claiming permanent real estate. Parallel bullets to the
|
||||
Captures aside so the two tabs feel like two sides of one product:
|
||||
|
||||
- **Clone any voice in seconds** — a short sample is enough
|
||||
- **Seven engines, 23 languages** — creative range, not a single model
|
||||
- **Agent-ready** — REST + WebSocket API, one checkbox away from giving any
|
||||
AI agent a voice
|
||||
|
||||
This lands in Phase 4 alongside the Captures tab, for visual and thematic
|
||||
symmetry. Not a persistent sidebar — the Generate tab is a workspace and
|
||||
should reclaim its space once the user is producing work.
|
||||
|
||||
### Archival by default
|
||||
|
||||
Every capture saves the original audio alongside the final transcript in a
|
||||
pattern that mirrors `data/generations/`. Optional retention setting. Free for
|
||||
us — the storage and UI patterns exist today for generations.
|
||||
|
||||
### Developer API, day one
|
||||
|
||||
The WebSocket transcribe endpoint is a first-class public API, documented
|
||||
alongside `/generate`. Pipeline presets are addressable by ID via
|
||||
`/pipelines/{id}/run` so agent harnesses and shell scripts can invoke
|
||||
user-configured flows. An MCP server sink ships built-in, so integrations with
|
||||
Claude Code, Cursor, Cline, etc. are one checkbox rather than a custom build.
|
||||
|
||||
### Agent voice output
|
||||
|
||||
Dictation is one half of the loop — user speaks, agent listens. The other half
|
||||
— agent speaks, user hears — is equally load-bearing and deserves a
|
||||
first-class primitive rather than being buried as a TTS loopback sink or a
|
||||
consumer read-aloud button.
|
||||
|
||||
The shape is a single new capability: any agent can call Voicebox to speak
|
||||
arbitrary text in a user-configured voice. The same pill that surfaces during
|
||||
dictation surfaces during agent speech, so the user always sees what's coming
|
||||
out of their machine.
|
||||
|
||||
```
|
||||
MCP tool: voicebox.speak({ text, profile?, style? })
|
||||
REST: POST /speak { text, profile_id?, style? }
|
||||
```
|
||||
|
||||
Both accept an optional voice profile (defaults to the user's configured
|
||||
default), an optional delivery-style string for engines that support it, play
|
||||
audio through system output, and surface the pill in a `speaking` state.
|
||||
|
||||
**Key design points:**
|
||||
|
||||
- **Pill is bidirectional.** States expand from `recording / transcribing /
|
||||
refining / rest` to include `speaking` — voice profile name, waveform in
|
||||
the profile's color, visible duration. Same floating surface for both
|
||||
directions so users have one mental model.
|
||||
- **Visibility is mandatory.** Silent background TTS is a trust hazard. Every
|
||||
agent-initiated `speak()` surfaces the pill. No headless "TTS daemon" mode.
|
||||
- **Per-source voice policy.** Settings let users bind specific MCP clients or
|
||||
API keys to specific voice profiles — Claude Code in "Morgan," Cursor in
|
||||
"Scarlett" — so users can tell which agent is talking without looking.
|
||||
- **Mute + rate limits.** One-toggle mute for all agent speech. Per-source
|
||||
rate limits prevent a runaway agent from monologuing.
|
||||
|
||||
This primitive is what makes "Voicebox as voice layer for every agent on your
|
||||
machine" a concrete shipping capability rather than marketing language. MCP,
|
||||
ACP, and A2A integrations all slot into it — none of those agent protocols
|
||||
need to know anything about TTS models, GPU placement, or voice profiles.
|
||||
They call `speak()`.
|
||||
|
||||
**Relationship to the persona loop.** The persona loop below is *one* use of
|
||||
`speak()` — STT → LLM → `speak(llm_reply)`. Other uses skip STT entirely: a
|
||||
long-running task announcing completion, a notification, an agent proactively
|
||||
asking the user a question. The primitive is deliberately simpler than the
|
||||
persona loop so it can serve both flows from the same API.
|
||||
|
||||
### Relationship to voice profile samples
|
||||
|
||||
A capture and a voice profile sample both hold `audio + text`, so there's an
|
||||
obvious temptation to unify them. Don't. The metadata and lifecycle
|
||||
differences are real:
|
||||
|
||||
| | Capture | Voice profile sample |
|
||||
|---|---|---|
|
||||
| Profile association | Standalone | Bound to one profile |
|
||||
| Text field | Raw transcript + optional LLM-refined version | Exact `reference_text` only |
|
||||
| LLM refinement | Often applied | Must not be applied — the reference text must match the audio verbatim or cloning breaks |
|
||||
| Volume | Dozens per day | ~5 per profile, semi-permanent |
|
||||
| Typical content | Whatever the user said | Often scripted phrases for cloning |
|
||||
|
||||
A unified table would mean nullable `profile_id`, nullable `refined_transcript`,
|
||||
nullable `reference_text` — a fat row that means different things in different
|
||||
states. Not worth the complexity.
|
||||
|
||||
**What to ship instead: a one-way promote action.** Capture → Sample, zero
|
||||
data-model churn. Thin endpoint:
|
||||
|
||||
```
|
||||
POST /profiles/{id}/samples/from-capture/{capture_id}
|
||||
```
|
||||
|
||||
Reads the capture's audio path and raw transcript, calls the existing
|
||||
`add_sample()` service with `reference_text` pre-filled from the transcript,
|
||||
lets the user edit the reference text in a dialog before saving (transcripts
|
||||
are usually 90% right but cloning wants 100%). The capture stays in the
|
||||
Captures tab untouched — the sample is a copy, not a move.
|
||||
|
||||
UI hook: the Captures tab's Send-to menu gains a **"Use as voice sample…"**
|
||||
option that opens a profile picker (with "+ New voice" for cold starts) and a
|
||||
reference-text confirm dialog.
|
||||
|
||||
The inverse direction (sample → capture) we deliberately skip. Samples are
|
||||
often scripted phrases used for cloning and they'd clutter the Captures list
|
||||
without adding value; also a subtle privacy surprise for users who don't
|
||||
expect their sample text browsable alongside real captures.
|
||||
|
||||
**Audio storage deduplication is a later optimization.** Today a promoted
|
||||
capture duplicates the audio file on disk. That's fine. Content-addressable
|
||||
storage (`data/audio/<sha256>.wav` with refcounting) can come in Phase 8 as
|
||||
housekeeping — it'd let a capture and a sample share one underlying file, but
|
||||
it's not user-visible and not necessary to ship the promote flow.
|
||||
|
||||
### The persona loop
|
||||
|
||||
One flow on top of the `speak()` primitive: STT → persona LLM →
|
||||
`speak(llm_reply)`. Voice profiles gain optional metadata — a natural-language
|
||||
personality description and default LLM behavior. The LLM runs text through
|
||||
the profile's voice context, then `speak()` generates TTS with the cloned
|
||||
profile. End-to-end voice-to-voice with a cloned identity transforming the
|
||||
content, not just reading it.
|
||||
|
||||
Use cases this unlocks:
|
||||
|
||||
- Agents that respond to spoken input in a specific voice
|
||||
- Interactive character experiences (games, narrative tools, accessibility)
|
||||
- Speech assistance for people who can't speak in their original voice
|
||||
|
||||
The shape — STT + LLM + TTS — also stages us for end-to-end speech LLMs which
|
||||
collapse all three into one transform. See *Voice-to-voice readiness* below.
|
||||
|
||||
### Voice-to-voice readiness
|
||||
|
||||
The STT → LLM → TTS chain that powers the persona loop is a staged approximation
|
||||
of voice-to-voice. A real end-to-end speech LLM (Moshi, GLM-4-Voice, Qwen2.5
|
||||
Omni, Mini-Omni, Sesame CSM) replaces the three middle boxes with a single
|
||||
fused transform: audio in, audio out, no text in between. The pipeline shape
|
||||
accommodates this natively — register the model as a single `LLMBackend` (or
|
||||
a new `SpeechLLMBackend` if the protocol needs to differ), expose it as a
|
||||
transform type, and the same sinks work unchanged.
|
||||
|
||||
Framing this plan as "voice-to-voice scaffolding, with today's models as the
|
||||
staged fallback" is a strong pitch for agent-harness users who are already
|
||||
tracking these models.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Tab name.** Leaning **Captures** — neutral, extensible across dictation,
|
||||
long-form recordings, and uploaded audio without repainting the tab later.
|
||||
"Dictations" is narrower (office-productivity coded, doesn't fit meeting
|
||||
recordings). "Notes" is the wrong mental model — nobody opens Voicebox to
|
||||
write notes. "Transcriptions" is flat.
|
||||
2. **Refinement vocabulary.** The LLM-post-STT step needs a user-facing name.
|
||||
"Refine," "polish," "rewrite," "smart edit" are candidates. "Refinement" in
|
||||
this doc as a placeholder only.
|
||||
3. **Preset primitive.** What do we call a user-configured pipeline? "Intent"
|
||||
collides with the existing `instruct` field on TTS generation. "Flow" is
|
||||
Zapier-coded. "Route" is too networking. Needs its own pass.
|
||||
4. **Persona metadata shape.** Does personality live directly on the voice
|
||||
profile, or as a separate persona construct that wraps profile + LLM config?
|
||||
The first is simpler; the second scales better if we later want multiple
|
||||
personas per voice.
|
||||
5. **Long-form capture product surface.** Pure preset, or dedicated entry point
|
||||
in the new tab? Leaning preset, but long-form is the feature that most
|
||||
justifies its own landing page.
|
||||
6. **Hotkey primitive naming.** Hold-vs-tap needs Voicebox-native phrasing in
|
||||
UI copy. Settings can still use industry-standard terms.
|
||||
|
||||
## Ordered phases
|
||||
|
||||
The v1 prototype deliberately skips the hardest parts of the long-term plan
|
||||
(native OS shim, global hotkeys, paste injection, new STT models). Everything
|
||||
in Phase 1–4 is in-process code using Whisper (which we already ship) and the
|
||||
existing model infra. No CGEvent taps, no SendInput, no clipboard timing.
|
||||
The usual OS-level sprawl of a dictation stack is exactly what we sidestep
|
||||
by starting in-app.
|
||||
|
||||
### Phase 1 — Groundwork
|
||||
|
||||
- Move the Audio tab into a Settings sub-tab (`ServerSettings/` gains one
|
||||
more section). Audio is device/channel config, not a creative workspace.
|
||||
- Reserve the sidebar slot for the new Captures tab (name TBD but leaning
|
||||
Captures — see open questions).
|
||||
- Gate the Captures tab behind a feature flag so we can merge to `main` and
|
||||
iterate without shipping half-built UI to users.
|
||||
|
||||
### Phase 2 — Local LLM backend
|
||||
|
||||
`LLMBackend` protocol alongside `TTSBackend` / `STTBackend`. Register Qwen3
|
||||
0.6B / 1.7B / 4B via `ModelConfig`. Reuses the HF download path, cache
|
||||
directory, and model management UI. MLX (4-bit community quants) on Apple
|
||||
Silicon, PyTorch (transformers AutoModelForCausalLM) elsewhere, same as our
|
||||
TTS split.
|
||||
|
||||
No new runtime. No `llama.cpp`, no `ollama`, no fragmented model cache.
|
||||
|
||||
### Phase 3 — In-app voice input
|
||||
|
||||
A universal mic button on every Voicebox text input. Hold, speak, release —
|
||||
text lands in the focused field via direct React state update. No OS APIs
|
||||
involved; Voicebox owns the input.
|
||||
|
||||
Marquee use cases:
|
||||
|
||||
- **Generation form.** Dictate a 2,000-character TTS script instead of typing
|
||||
it. This alone justifies the feature.
|
||||
- **Voice profile descriptions.** Describe a voice's personality by speaking,
|
||||
which then becomes the input for Phase 4's persona loop.
|
||||
- **Story titles, preset names, any free-text field.** Free reuse.
|
||||
|
||||
Backend: add `/transcribe/stream` WebSocket endpoint. Audio frames in, partial
|
||||
transcripts out. Reuses the existing Whisper model in memory. Optionally routes
|
||||
through the LLM from Phase 2 for light refinement.
|
||||
|
||||
### Phase 4 — Captures tab
|
||||
|
||||
Graduates the tab out from behind the feature flag. Shows recent captures
|
||||
(audio + transcript pairs), lets the user replay, re-transcribe with a
|
||||
different model, edit the transcript, and send the output through the LLM.
|
||||
Archival is automatic — every capture saves audio alongside transcript.
|
||||
|
||||
**Includes the "Play as voice profile" action.** This is the simplest version
|
||||
of the persona loop and it lands here for free — no LLM involved, no new
|
||||
backend endpoints, just a Captures-tab button that sends the transcript text
|
||||
to the existing `/generate` endpoint with a user-selected voice profile and
|
||||
plays the result. Category-defining differentiator from the v1 prototype
|
||||
onward: Superwhisper and WisprFlow cannot do this because they have no TTS. Voicebox can, with one day of frontend wiring.
|
||||
|
||||
Keep it aggressively minimal on day one. A capture list, a detail view, a
|
||||
model picker, a Play-as-voice dropdown. Refinement prompt editing, correction
|
||||
dictionaries, per-source overrides — none of that ships here. They become
|
||||
Tier-2 work when someone actually asks for them.
|
||||
|
||||
### Phase 5 — Agent voice output + persona loop
|
||||
|
||||
Two features that together make "Voicebox as the voice layer for every agent
|
||||
on your machine" a shipping reality:
|
||||
|
||||
1. **`speak()` primitive.** New `POST /speak` endpoint and `voicebox.speak`
|
||||
MCP tool. Any agent calls Voicebox to speak arbitrary text in a
|
||||
user-configured voice; the pill surfaces in a `speaking` state. Settings
|
||||
UI for default voice, per-agent voice binding (Claude Code → Morgan,
|
||||
Cursor → Scarlett), and a global mute.
|
||||
2. **Persona loop.** Extends `speak()` with an LLM step — STT → persona LLM
|
||||
→ `speak(llm_reply)`. Voice profiles gain optional personality metadata
|
||||
and default LLM behavior. End-to-end voice-to-voice with a cloned
|
||||
identity transforming the content, not just reading it.
|
||||
|
||||
Phase 4 demoed the user-initiated direction of the loop (Play as voice). This
|
||||
phase ships the *agent*-initiated direction, which is the category-defining
|
||||
capability and the pitch that lands with agent-harness users. The persona
|
||||
loop is one flow on top of the `speak()` primitive — notifications, proactive
|
||||
agent questions, and task-completion announcements all use `speak()` directly
|
||||
without the LLM in the middle.
|
||||
|
||||
Launchable headline moment for the "local voice I/O" positioning.
|
||||
|
||||
### Phase 6 — STT engine expansion
|
||||
|
||||
Parakeet v3 and Qwen3-ASR register as additional `STTBackend` implementations.
|
||||
Optional: Kyutai ASR. Multilingual coverage upgrades (50+ languages). Whisper
|
||||
stays as the sensible default.
|
||||
|
||||
Deferred to here because Whisper is already good enough for v1 and the model
|
||||
picker UI exists. Adding rows to it doesn't change the product shape.
|
||||
|
||||
### Phase 7 — External dictation shell
|
||||
|
||||
Native shim crate (global hotkey with modifier-only support, focus
|
||||
introspection via OS accessibility APIs, paste simulation, atomic clipboard
|
||||
save/restore). Tauri-side audio capture streams to the same WebSocket endpoint
|
||||
Phase 3 already ships. Paste sink with target-aware delivery.
|
||||
|
||||
This is the feel-good phase. It's also the riskiest: paste timing, hotkey
|
||||
reliability, and cross-platform focus detection are all engineering problems
|
||||
that have to be nailed or the product doesn't work. Phase 3's success derisks
|
||||
the backend plumbing before we start it.
|
||||
|
||||
### Phase 8 — Pipeline routing, sinks, long-form
|
||||
|
||||
Multiple source types, user-configurable transform chains, multiple sinks per
|
||||
preset. MCP server sink (the agent-harness play). HTTP webhook sink. File
|
||||
sink. Developer-facing `/pipelines/{id}/run` endpoint. Preset editor UI in
|
||||
the Captures tab.
|
||||
|
||||
Dual-stream recorder (mic + system audio) as a source type. Chunked STT
|
||||
transform with overlap-based deduplication. Summary LLM transform. Long-form
|
||||
capture becomes a preset, not a new tab.
|
||||
|
||||
Platform-specific sinks (Apple Notes on macOS, Obsidian, etc.) as opt-in
|
||||
integrations behind the generic sink interface.
|
||||
|
||||
## Architectural prerequisites
|
||||
|
||||
Two pieces of existing `docs/PROJECT_STATUS.md` work become load-bearing here:
|
||||
|
||||
- **Platform support tiers** (#420, PR #465). Native shim capabilities vary by
|
||||
platform — Wayland paste is worse than X11, Windows system-audio capture has
|
||||
edge cases, frontmost-window OCR is platform-gated. Tier definitions let us
|
||||
ship confidently with honest user-facing expectations.
|
||||
- **Platform gating on `ModelConfig`** (bottleneck #6 in PROJECT_STATUS).
|
||||
Parakeet's Core ML path is Apple-only; the PyTorch path is Windows/Linux.
|
||||
Same gating mechanism that currently blocks shipping VoxCPM.
|
||||
|
||||
Neither needs to complete before Phase 1, but both should complete before
|
||||
Phase 4 when user-configurable pipelines surface the differences to end users.
|
||||
Reference in New Issue
Block a user