From 798cd40f059eb87d613e8cbbbb33696013643c73 Mon Sep 17 00:00:00 2001 From: James Pine Date: Mon, 16 Mar 2026 02:59:24 -0700 Subject: [PATCH] delete stale planning docs --- docs/PR-ACCESSIBILITY.md | 70 -- docs/plans/EXTERNAL_PROVIDERS.md | 435 ----------- docs/plans/MLX_AUDIO.md | 396 ---------- docs/plans/PR33_CUDA_PROVIDER_REVIEW.md | 500 ------------ docs/plans/TTS_PROVIDER_ARCHITECTURE.md | 964 ------------------------ 5 files changed, 2365 deletions(-) delete mode 100644 docs/PR-ACCESSIBILITY.md delete mode 100644 docs/plans/EXTERNAL_PROVIDERS.md delete mode 100644 docs/plans/MLX_AUDIO.md delete mode 100644 docs/plans/PR33_CUDA_PROVIDER_REVIEW.md delete mode 100644 docs/plans/TTS_PROVIDER_ARCHITECTURE.md diff --git a/docs/PR-ACCESSIBILITY.md b/docs/PR-ACCESSIBILITY.md deleted file mode 100644 index d9e71ac9..00000000 --- a/docs/PR-ACCESSIBILITY.md +++ /dev/null @@ -1,70 +0,0 @@ -# Accessibility: screen reader and keyboard improvements - -## Summary - -Improvements to support screen reader and keyboard users across the main app surfaces: audio player, generation UI, voice selection, history, voices tab, model management, server tab, and stories. - -**Tested with NVDA and Narrator on Windows.** - ---- - -## What changed - -### Audio player (after generating audio) - -- **Play/Pause, Loop, Mute, Close** – `aria-label` added so each control is announced (e.g. "Play", "Pause", "Loop", "Mute", "Close player"). -- **Playback position slider** – `aria-label="Playback position"` and `aria-valuetext` with current/total time (e.g. "0:30 of 2:15"). -- **Volume** – Wrapped in a labelled group; volume slider has an associated screen-reader-only label and `aria-valuetext` for the level (e.g. "Volume level, 75%"). - -### Generation UI (text box and voice choice) - -- **Generate speech** (submit) and **Fine-tune instructions** (sliders) – Icon buttons now have `aria-label` (and state for fine-tune, e.g. "Fine-tune instructions, on"). - -### Voice selection (cards on Generate screen) - -- Each **voice card** is focusable (`tabIndex={0}`), has `role="button"`, and an `aria-label` (e.g. "Prashant, en. Select as voice for generation.") with `aria-pressed` when selected. -- **Enter/Space** on the card selects that voice; tab order is card → Export/Edit/Delete. - -### History list (generated samples) - -- Each **sample row** is focusable with `role="button"` and an `aria-label` (e.g. "Sample from [profile], [duration], [date]. Press Enter to play."); **Enter/Space** plays or restarts. -- **Transcript textarea** has `aria-label` (e.g. "Transcript for sample from [profile], [duration]") so when you focus on the text area, the sample is announced in context. - -### Voices tab (table) - -- Each **voice row** is focusable with `role="button"` and an `aria-label` (e.g. "[Name], [language], [N] generations, [N] samples. Press Enter to edit."); **Enter/Space** opens edit (except when focus is in a control). -- **Actions** dropdown trigger has `aria-label="Actions for [profile name]"`. - -### Model management - -- Each **model row** is a focusable region (`tabIndex={0}`, `role="group"`) with an `aria-label` (e.g. "[Model name], [status], [size]. Use Tab to reach Download or Delete."). -- **Download** and **Delete** (and Downloading) buttons have `aria-label` (e.g. "Download [name]", "Delete [name]"). - -### Server tab (panels) - -- **Server Connection**, **Server Status**, and **App Updates** cards are landmarks: `role="region"`, `aria-label`, and `tabIndex={0}` so each panel is focusable and announced (e.g. "Server Connection", "Server Status", "App Updates"). - -### Stories list - -- Each **story row** is a focusable control (`role="button"`, `tabIndex={0}`) with `aria-label` (e.g. "Story [name], [N] items, [date]. Press Enter to select."); **Enter/Space** selects the story. Actions button has `aria-label="Actions for [story name]"`. - -### Other controls - -- **Story list** – Actions (⋮) button: `aria-label="Actions for [story name]"`. -- **Story track editor** – Play/Pause, Stop, Split, Duplicate, Delete, Zoom in/out: `aria-label` on all icon buttons. -- **Voice profile samples** (SampleList, AudioSampleUpload, AudioSampleRecording, AudioSampleSystem) – Play/Pause and Stop: `aria-label` (e.g. "Play sample", "Pause", "Stop playback"). -- **SampleList** mini sample player – Seek slider has `aria-label="Sample playback position"` and `aria-valuetext` for time. - ---- - -## Testing - -- **Screen readers:** Tested with **NVDA** and **Narrator** on Windows. -- **Keyboard:** Tab order and Enter/Space activation verified for focusable rows and buttons. - ---- - -## Tech note - -- React + TypeScript; Radix UI primitives; labels added via `aria-label`, `aria-labelledby`, `aria-valuetext`, and `role`/`tabIndex` where needed. -- No new dependencies. diff --git a/docs/plans/EXTERNAL_PROVIDERS.md b/docs/plans/EXTERNAL_PROVIDERS.md deleted file mode 100644 index 3b1e7e21..00000000 --- a/docs/plans/EXTERNAL_PROVIDERS.md +++ /dev/null @@ -1,435 +0,0 @@ -# External Provider Support - -**Status:** Planned for v0.2.0 -**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...) - -## Overview - -External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for: - -- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server -- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI -- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc. -- **Team Sharing**: Multiple users share one GPU server running models -- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa - -## Architecture - -``` -┌─────────────────┐ HTTP/API ┌──────────────────┐ -│ Voicebox UI │ ───────────────────────> │ Your TTS Server │ -│ + Backend │ │ (Qwen3-TTS on │ -│ │ <─────────────────────── │ AMD/NVIDIA GPU)│ -│ - Profiles │ Audio + Metadata └──────────────────┘ -│ - History │ -│ - Audio Edit │ HTTP/API ┌──────────────────┐ -│ - UI │ ───────────────────────> │ Whisper Service │ -└─────────────────┘ │ (OpenAI API or │ - │ self-hosted) │ - └──────────────────┘ -``` - -**What Voicebox Still Handles:** -- Voice profile management -- Generation history -- Audio trimming/editing -- Multi-track story editor -- UI/UX layer - -**What External Providers Handle:** -- Model inference (TTS generation, transcription) -- GPU allocation -- Model loading/caching - -## Configuration - -### Environment Variables - -```bash -# TTS Provider -TTS_MODE=remote # local | remote -TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL -TTS_API_KEY=your-api-key # Optional authentication - -# Whisper Provider -WHISPER_MODE=openai-api # local | openai-api | remote -WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper -OPENAI_API_KEY=sk-... # For OpenAI Whisper API -``` - -### Voicebox Config UI (Planned) - -Settings page will include: -- Provider selection dropdowns -- URL/API key inputs -- Connection test button -- Latency/status indicators - -## Hosting External Services - -### Option 1: Simple FastAPI Server (Recommended) - -Create a lightweight server to expose your local Qwen3-TTS model: - -```python -# tts_server.py -from fastapi import FastAPI, UploadFile, File -from qwen_tts import Qwen3TTSModel -import numpy as np -import base64 - -app = FastAPI() -model = Qwen3TTSModel.from_pretrained( - "Qwen/Qwen3-TTS-12Hz-1.7B-Base", - device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm -) - -@app.post("/v1/generate") -async def generate( - text: str, - voice_prompt: dict, - language: str = "en", - seed: int = None -): - """Generate speech from text using voice prompt.""" - audio, sample_rate = model.generate_voice_clone( - text=text, - voice_clone_prompt=voice_prompt, - ) - - # Return as base64 for transport - audio_bytes = audio.tobytes() - return { - "audio": base64.b64encode(audio_bytes).decode(), - "sample_rate": sample_rate, - "dtype": str(audio.dtype) - } - -@app.post("/v1/create_voice_prompt") -async def create_voice_prompt( - audio: UploadFile = File(...), - reference_text: str = "" -): - """Create voice prompt from reference audio.""" - # Save uploaded audio temporarily - audio_path = f"/tmp/{audio.filename}" - with open(audio_path, "wb") as f: - f.write(await audio.read()) - - # Create voice prompt - voice_prompt = model.create_voice_clone_prompt( - ref_audio=audio_path, - ref_text=reference_text, - ) - - return {"voice_prompt": voice_prompt} - -@app.get("/health") -async def health(): - return { - "status": "healthy", - "model": "Qwen3-TTS-12Hz-1.7B-Base", - "device": str(model.device) - } - -if __name__ == "__main__": - import uvicorn - uvicorn.run(app, host="0.0.0.0", port=8000) -``` - -**Run it:** -```bash -# Install dependencies -pip install fastapi uvicorn qwen-tts torch - -# For AMD GPUs, use ROCm PyTorch: -pip install torch --index-url https://download.pytorch.org/whl/rocm6.4 - -# Start server -python tts_server.py -``` - -### Option 2: vLLM (If Supported) - -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --host 0.0.0.0 \ - --port 8000 \ - --gpu-memory-utilization 0.9 -``` - -### Option 3: Cloud Platforms - -**Modal.com Example:** -```python -import modal - -app = modal.App("qwen-tts") -image = modal.Image.debian_slim().pip_install("qwen-tts", "torch") - -@app.function(gpu="A10G", image=image) -@modal.web_endpoint(method="POST") -def generate(text: str, voice_prompt: dict): - from qwen_tts import Qwen3TTSModel - model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base") - audio, sr = model.generate_voice_clone(text, voice_prompt) - return {"audio": audio.tolist(), "sample_rate": sr} -``` - -Deploy: `modal deploy tts_server.py` -Get URL: `https://yourapp--generate.modal.run` - -## API Specification - -External TTS providers must implement these endpoints: - -### `POST /v1/generate` - -Generate speech from text. - -**Request:** -```json -{ - "text": "Hello, this is a test.", - "voice_prompt": { /* voice prompt object */ }, - "language": "en", - "seed": 12345 -} -``` - -**Response:** -```json -{ - "audio": "base64-encoded-audio-bytes", - "sample_rate": 24000, - "dtype": "float32" -} -``` - -### `POST /v1/create_voice_prompt` - -Create a voice prompt from reference audio. - -**Request:** (multipart/form-data) -- `audio`: Audio file upload -- `reference_text`: Transcript of the audio - -**Response:** -```json -{ - "voice_prompt": { /* voice prompt object */ } -} -``` - -### `GET /health` - -Health check endpoint. - -**Response:** -```json -{ - "status": "healthy", - "model": "Qwen3-TTS-12Hz-1.7B-Base", - "device": "cuda:0" -} -``` - -## Whisper External Providers - -### OpenAI Whisper API - -Simply set: -```bash -WHISPER_MODE=openai-api -OPENAI_API_KEY=sk-... -``` - -Voicebox will use OpenAI's Whisper API automatically. - -### Self-Hosted Whisper - -Run your own Whisper server: - -```python -# whisper_server.py -from fastapi import FastAPI, UploadFile, File -from transformers import WhisperProcessor, WhisperForConditionalGeneration -import librosa - -app = FastAPI() -processor = WhisperProcessor.from_pretrained("openai/whisper-base") -model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base") - -@app.post("/v1/transcribe") -async def transcribe(audio: UploadFile = File(...), language: str = None): - # Load audio - audio_path = f"/tmp/{audio.filename}" - with open(audio_path, "wb") as f: - f.write(await audio.read()) - - audio_data, sr = librosa.load(audio_path, sr=16000) - - # Process - inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt") - predicted_ids = model.generate(inputs["input_features"]) - transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] - - return {"text": transcription} -``` - -Configure Voicebox: -```bash -WHISPER_MODE=remote -WHISPER_REMOTE_URL=http://localhost:9000 -``` - -## Use Cases - -### 1. AMD GPU User with Existing Setup - -**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux. - -**Setup:** -1. Run `tts_server.py` on your AMD box (ROCm PyTorch) -2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000` -3. Use Voicebox UI for profiles, generation, editing -4. TTS happens on your AMD GPU - -### 2. Team Deployment - -**Scenario:** 5 team members, 1 GPU server. - -**Setup:** -1. Deploy TTS server on shared GPU box -2. Each person runs Voicebox desktop app locally -3. All point to same `TTS_REMOTE_URL` -4. Profiles and history stay local per user -5. GPU usage is shared - -### 3. Hybrid Local/Remote - -**Scenario:** Fast local Whisper, heavy TTS on cloud. - -**Setup:** -```bash -TTS_MODE=remote -TTS_REMOTE_URL=https://your-modal-app.modal.run - -WHISPER_MODE=local # Fast transcription on your CPU -``` - -### 4. OpenAI Whisper + Self-Hosted TTS - -**Scenario:** Use OpenAI's API for transcription, run TTS locally. - -**Setup:** -```bash -TTS_MODE=local - -WHISPER_MODE=openai-api -OPENAI_API_KEY=sk-... -``` - -## Security Considerations - -### Authentication - -Add API key authentication to your external server: - -```python -from fastapi import Header, HTTPException - -API_KEY = "your-secret-key" - -async def verify_api_key(x_api_key: str = Header(...)): - if x_api_key != API_KEY: - raise HTTPException(status_code=401, detail="Invalid API key") - -@app.post("/v1/generate", dependencies=[Depends(verify_api_key)]) -async def generate(...): - ... -``` - -Configure Voicebox: -```bash -TTS_API_KEY=your-secret-key -``` - -### Network Security - -- **VPN/Tailscale**: Use private network for remote servers -- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates -- **Firewall**: Restrict access to known IPs - -### Rate Limiting - -Protect your external server: - -```python -from slowapi import Limiter -from slowapi.util import get_remote_address - -limiter = Limiter(key_func=get_remote_address) -app.state.limiter = limiter - -@app.post("/v1/generate") -@limiter.limit("10/minute") -async def generate(...): - ... -``` - -## Performance Considerations - -### Latency - -External providers add network latency: -- **Local network**: ~10-50ms overhead (negligible) -- **Same datacenter**: ~1-5ms overhead -- **Cross-region cloud**: 50-200ms+ overhead - -For real-time applications, keep TTS server on local network or same cloud region. - -### Caching - -Implement response caching on external server: - -```python -from functools import lru_cache - -@lru_cache(maxsize=1000) -def get_cached_generation(text, voice_prompt_hash, language, seed): - return model.generate_voice_clone(text, voice_prompt) -``` - -### Load Balancing - -For high-traffic deployments, run multiple TTS servers behind a load balancer: - -``` -Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1) - ├──> TTS Server 2 (GPU 2) - └──> TTS Server 3 (GPU 3) -``` - -## Future Enhancements - -- [ ] **Provider Marketplace**: Built-in directory of compatible providers -- [ ] **Automatic Fallback**: If remote fails, fallback to local -- [ ] **Cost Tracking**: Monitor API usage and costs -- [ ] **Performance Metrics**: Latency, throughput dashboards -- [ ] **Multi-Provider**: Use different providers for different voices/languages - -## Contributing - -If you build an external provider, please share: -1. Server implementation -2. Performance benchmarks -3. Deployment guide - -Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions) - -## Questions? - -- **Discord**: [Join the community](https://discord.gg/...) -- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues) -- **Docs**: [Full documentation](https://voicebox.sh/docs) diff --git a/docs/plans/MLX_AUDIO.md b/docs/plans/MLX_AUDIO.md deleted file mode 100644 index 9c2a100a..00000000 --- a/docs/plans/MLX_AUDIO.md +++ /dev/null @@ -1,396 +0,0 @@ -# MLX Audio Integration - -**Status:** Validated ✅ -**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio) - -## Validation Results - -We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings: - -| Metric | Result | -|--------|--------| -| MLX Version | 0.30.4 | -| Model Load Time | ~1s (after initial download) | -| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) | -| Test Hardware | Apple Silicon Mac | - -### Model Mapping - -| voicebox (PyTorch) | mlx-audio (MLX) | -|--------------------|-----------------| -| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` | -| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) | - -### mlx-audio API - -The API uses a **generator-based streaming pattern**: - -```python -from mlx_audio.tts import load - -model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16") - -# generate() yields GenerationResult objects -for result in model.generate("Hello world"): - audio = result.audio # numpy array of samples - sample_rate = result.sample_rate # 24000 - rtf = result.real_time_factor # e.g., 0.55 -``` - -### Known Warnings (harmless) - -``` -You are using a model of type qwen3_tts to instantiate a model of type . -The tokenizer you are loading... with an incorrect regex pattern... -``` - -These warnings appear but don't affect functionality or output quality. - -### Demo Script - -Run `mlx-test/demo.py` to test: -```bash -cd mlx-test && source venv/bin/activate && python demo.py "Your text here" -``` - -## Problem - -Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs. - -This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads. - -## Solution - -Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use. - -**Key wins:** -- Native GPU acceleration on Apple Silicon (no more CPU fallback) -- Streaming TTS support (faster perceived latency) -- Memory optimizations (run larger models on less RAM) -- Fixed 0.6B silence bug that we currently ship -- Same Qwen3-TTS models (zero migration cost for users) - -## Architecture - -### Current Stack -``` -┌─────────────────────────┐ -│ PyTorch + Qwen3-TTS │ -│ (CPU only on macOS) │ -└─────────────────────────┘ -``` - -### Proposed Stack -``` -┌─────────────────────────────────────────┐ -│ Platform Detection at Runtime │ -└─────────────────────────────────────────┘ - │ - ├─── Apple Silicon (aarch64-darwin) - │ ┌─────────────────────────┐ - │ │ MLX Audio Backend │ - │ │ - Qwen3-TTS (mlx) │ - │ │ - Whisper (mlx) │ - │ │ - Streaming support │ - │ └─────────────────────────┘ - │ - └─── Other (x86_64, Windows, Linux) - ┌─────────────────────────┐ - │ PyTorch Backend │ - │ - Qwen3-TTS (pytorch) │ - │ - Whisper (pytorch) │ - │ - CUDA if available │ - └─────────────────────────┘ -``` - -## Implementation Phases - -### Phase 1: Platform Detection & Dependency Management - -Create a backend that switches between PyTorch and MLX based on runtime platform detection. - -**New files:** -- `backend/platform.py` - Detect Apple Silicon, return backend type -- `backend/backends/__init__.py` - Backend factory pattern -- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only) - -**Modified files:** -- `backend/requirements.txt` - Keep PyTorch as default -- `backend/main.py` - Import from backend factory instead of direct imports - -**Platform detection logic:** -```python -def get_backend_type() -> str: - """Detect best backend for current platform.""" - if platform.system() == "Darwin" and platform.machine() == "arm64": - # Apple Silicon detected - try: - import mlx - return "mlx" - except ImportError: - return "pytorch" # Fallback if mlx not installed - return "pytorch" -``` - -### Phase 2: MLX Backend Implementation - -Create parallel implementations of TTS and STT using mlx-audio. - -**New files:** -- `backend/backends/mlx_backend.py` - MLX inference engine -- `backend/backends/pytorch_backend.py` - Refactor current code into backend - -**Interface both backends must implement:** -```python -class TTSBackend(Protocol): - async def load_model(self, model_size: str) -> None: ... - async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ... - async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ... - async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ... - def unload_model(self) -> None: ... - -class STTBackend(Protocol): - async def load_model(self, model_size: str) -> None: ... - async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ... - def unload_model(self) -> None: ... -``` - -**MLX backend implementation notes:** - -mlx-audio's `generate()` returns a generator by default (streaming is built-in): - -```python -# MLX backend wrapper -from mlx_audio.tts import load - -class MLXTTSBackend: - def __init__(self): - self.model = None - - async def load_model(self, model_size: str) -> None: - model_map = { - "1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16", - # "0.6B": needs conversion to mlx format - } - self.model = load(model_map[model_size]) - - async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: - # Collect all chunks from generator - chunks = [] - for result in self.model.generate(text): # TODO: add voice_prompt support - chunks.append(np.array(result.audio)) - return np.concatenate(chunks), 24000 -``` - -**MLX-specific features to expose:** -- Streaming TTS (new endpoint: `/api/generate/stream`) -- Memory-optimized model loading -- Qwen3-ASR for transcription (in addition to Whisper) - -### Phase 3: API Layer Updates - -Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility. - -**Modified files:** -- `backend/main.py` - Add streaming endpoints -- `backend/tts.py` - Refactor to use backend abstraction -- `backend/transcribe.py` - Refactor to use backend abstraction - -**New endpoints:** -```python -@app.post("/api/generate/stream") -async def generate_stream(...) -> StreamingResponse: - """Stream TTS chunks as they're generated (MLX only).""" - backend = get_backend() - if not hasattr(backend, 'generate_streaming'): - raise HTTPException(501, "Streaming not supported on this backend") - return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav") -``` - -**Backward compatibility:** -- Keep all existing `/api/generate` endpoints unchanged -- PyTorch backend users see no behavior change -- MLX users automatically get faster inference, streaming is opt-in - -### Phase 4: Frontend Integration - -Add UI indicators for backend type and streaming progress. - -**Modified files:** -- `app/src/hooks/useGenerationForm.tsx` - Add streaming support -- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle -- `app/src/lib/api.ts` - Add streaming API client - -**UI additions:** -- Badge showing current backend ("MLX" or "PyTorch") -- Toggle for streaming mode (disabled if PyTorch) -- Real-time streaming playback (WaveSurfer progressive loading) - -### Phase 5: Build & Distribution - -Create separate installers for MLX (Apple Silicon) and PyTorch (Universal). - -**Modified files:** -- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds -- `.github/workflows/release.yml` - Build both variants - -**Build matrix:** -```yaml -- target: aarch64-apple-darwin - backend: mlx - installer: voicebox-macos-silicon-{version}.dmg - -- target: x86_64-apple-darwin - backend: pytorch - installer: voicebox-macos-intel-{version}.dmg - -- target: x86_64-pc-windows-msvc - backend: pytorch - installer: voicebox-windows-{version}.exe -``` - -**Installation flow:** -- Auto-detect architecture, recommend correct installer -- MLX installer includes `mlx-audio` in embedded Python -- PyTorch installer includes `torch` in embedded Python -- Both can coexist (different backend, same profile format) - -### Phase 6: Testing & Validation - -Ensure both backends produce compatible outputs. - -**New files:** -- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio -- `backend/tests/test_streaming.py` - Streaming-specific tests - -**Test scenarios:** -- Same voice prompt on both backends → similar (not identical) audio output -- Profile created on MLX → loads on PyTorch (and vice versa) -- Streaming chunks assemble into valid WAV file -- Model downloads work on both backends -- Memory usage stays within bounds - -### Phase 7: Documentation - -Update user-facing docs and developer guides. - -**New files:** -- `docs/developer/BACKENDS.md` - Guide for adding new backends -- `docs/overview/performance.md` - Backend comparison benchmarks - -**Modified files:** -- `README.md` - Note Apple Silicon acceleration -- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues - -**Key docs to write:** -- Which installer to download (architecture detection) -- Performance comparison (MLX vs PyTorch on same M2 hardware) -- How streaming mode works -- How to force PyTorch on Apple Silicon (for debugging) - -## Technical Decisions - -### Why Dual Backend Instead of MLX-Only? - -**Pros of dual backend:** -- Windows and Intel Mac users unaffected -- Easier testing (can compare outputs) -- Fallback if MLX has issues - -**Cons of dual backend:** -- More code to maintain -- Two dependency trees -- Build complexity (separate installers) - -**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback. - -### Why Separate Installers Instead of Runtime Detection? - -**Pros of separate installers:** -- Smaller bundle size (don't ship both PyTorch and MLX) -- Clearer to users which version they have -- Easier to debug (no "which backend am I running?" confusion) -- Can optimize each build for its target - -**Cons:** -- More installers to build and test -- Users might download the wrong one - -**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page. - -### Streaming vs Batch Generation - -MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be: -1. MLX-only feature (✅ chosen) -2. Implemented for both (lots of work) -3. Not exposed at all (wasted opportunity) - -**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend). - -## Migration Path - -Nothing needs migrating, macos users will just notice a speed-boost in inference - -**Data format compatibility:** -- Profiles (SQLite) → no schema changes needed -- Voice prompts (cached) → backend-agnostic (just numpy arrays) -- Audio files → unchanged - -## Performance Expectations - -### Measured Results (from validation) - -| Metric | MLX (measured) | PyTorch CPU (estimated) | -|--------|----------------|-------------------------| -| **6s audio generation** | ~3-4s | ~10-15s | -| **Real-time factor** | 0.5-0.6x | 2-3x | -| **Model load (cached)** | ~1s | ~3-5s | - -### TTS Generation (1.7B model, ~20s output) -- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time) -- **MLX (M2 Max):** ~8-12s (faster than real-time) -- **Improvement:** ~4-5x faster - -### Whisper Transcription (10s audio clip) -- **PyTorch CPU:** ~5-8s -- **MLX:** ~1-2s -- **Improvement:** ~3-4x faster - -### Memory Usage (1.7B model) -- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM) -- **MLX:** ~4-6GB (unified memory, better optimization) -- **Improvement:** ~40% less RAM - -Full benchmarks will be in `docs/overview/performance.md` after Phase 6. - -## Open Questions - -- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure -- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No -- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing -- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users. - -## Success Metrics - -How we'll know this worked: - -1. **Performance:** Apple Silicon users report generation faster than real-time -2. **Adoption:** >80% of macOS downloads are MLX build within 1 month -3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions) -4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance - -## Related Work - -- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly -- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API -- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data - -## Next Steps - -1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/` -2. Get approval on dual-backend architecture -3. Start Phase 1 (platform detection) - -## Questions? - -Feedback welcome in GitHub discussions or Discord. diff --git a/docs/plans/PR33_CUDA_PROVIDER_REVIEW.md b/docs/plans/PR33_CUDA_PROVIDER_REVIEW.md deleted file mode 100644 index 66554dd6..00000000 --- a/docs/plans/PR33_CUDA_PROVIDER_REVIEW.md +++ /dev/null @@ -1,500 +0,0 @@ -# PR #33 — CUDA Provider System Review - -> Branch: `external-provider-binaries` | Created: 2026-02-01 | 34 commits, 136 files, +10,266 lines -> Reviewed: 2026-03-12 - ---- - -## The Problem - -The CUDA PyTorch binary is ~2.4 GB. GitHub Releases has a 2 GB artifact limit. This means: - -- Windows/Linux users with NVIDIA GPUs cannot get GPU acceleration from official releases -- 19 open issues about "GPU not detected" — the single most reported problem category -- Users who want GPU must clone the repo and run from source -- Every app update forces re-download of the entire binary - -This is the #1 user pain point by volume. - ---- - -## What PR #33 Does - -Splits the monolithic Voicebox binary into two layers: - -``` -┌──────────────────────────────────────┐ -│ Main App (~150MB Win/Lin, ~300 Mac) │ -│ Tauri + React + FastAPI + Whisper │ -│ No PyTorch. MLX bundled on macOS. │ -├──────────────────────────────────────┤ -│ HTTP (localhost) │ -├──────────────────────────────────────┤ -│ Provider Binary (downloaded later) │ -│ PyTorch CPU (~300MB) │ -│ PyTorch CUDA (~2.4GB) │ -│ Hosted on Cloudflare R2 │ -└──────────────────────────────────────┘ -``` - -### New Backend Code - -| File | Purpose | -|------|---------| -| `backend/providers/__init__.py` (327 lines) | `ProviderManager` — lifecycle management, subprocess spawning, port allocation | -| `backend/providers/base.py` (97 lines) | `TTSProvider` Protocol definition | -| `backend/providers/bundled.py` (144 lines) | `BundledProvider` — wraps existing MLX/PyTorch backends for the new interface | -| `backend/providers/local.py` (191 lines) | `LocalProvider` — HTTP client that talks to external provider processes | -| `backend/providers/installer.py` (262 lines) | Download, extract, delete provider binaries | -| `backend/providers/types.py` (34 lines) | `ProviderType` enum, `ProviderInfo` dataclass | -| `backend/providers/checksums.py` (11 lines) | Checksum dict (currently empty) | - -### Provider Servers (Standalone Executables) - -| File | Purpose | -|------|---------| -| `providers/pytorch-cpu/main.py` (238 lines) | FastAPI server wrapping PyTorch CPU inference | -| `providers/pytorch-cuda/main.py` (238 lines) | FastAPI server wrapping PyTorch CUDA inference | -| `providers/pytorch-*/build.py` | PyInstaller build scripts | -| `providers/pytorch-*/requirements.txt` | Isolated dependencies | - -### Frontend - -| File | Purpose | -|------|---------| -| `app/src/components/ServerSettings/ProviderSettings.tsx` (400 lines) | Provider download/start/stop/delete UI | - -### Also Included (Scope Creep) - -The PR bundles several unrelated changes that inflate the diff: - -- `docs2/` — Entire documentation site rewrite (Fumadocs migration, ~3000 lines) -- `Dockerfile`, `Dockerfile.cuda`, `docker-compose.yml` — Docker support -- `landing/` — Banner removal -- UI refactors in Stories, History, Voice Profiles, Audio tab -- Linux audio capture module -- Various dependency bumps - ---- - -## Bug Report - -### Critical — Will Crash at Runtime - -#### C1. Provider `generate` endpoint can't parse requests - -**`providers/pytorch-cpu/main.py:91-97`** (same in pytorch-cuda) - -```python -@app.post("/tts/generate") -async def generate( - text: str, - voice_prompt: dict, - language: str = "auto", - seed: int = None, - model_size: str = "1.7B" -): -``` - -Parameters declared as function arguments. FastAPI interprets these as **query parameters**, not JSON body. But `LocalProvider.generate()` sends a JSON body via `httpx`: - -```python -# backend/providers/local.py:33-40 -response = await self.client.post("/tts/generate", json={ - "text": text, - "voice_prompt": voice_prompt, - ... -}) -``` - -**Result:** Every generation call to an external provider returns HTTP 422 (Validation Error). The generation path is completely broken for external providers. - -**Fix:** Use a Pydantic request body model: -```python -class GenerateRequest(BaseModel): - text: str - voice_prompt: dict - language: str = "auto" - seed: Optional[int] = None - model_size: str = "1.7B" - -@app.post("/tts/generate") -async def generate(data: GenerateRequest): -``` - -#### C2. Timeout error handler references undefined variables - -**`backend/providers/__init__.py:82-90`** - -```python -stdout_content = "" -stderr_content = "" -# ... threads write to stdout_queue / stderr_queue ... -except TimeoutError: - while not stdout_queue.empty(): - stdout_lines.append(stdout_queue.get_nowait()) # NameError - while not stderr_queue.empty(): - stderr_lines.append(stderr_queue.get_nowait()) # NameError -``` - -`stdout_lines` and `stderr_lines` are never defined. Every provider startup timeout will throw `NameError`, masking the real failure cause. Then `stdout_content` and `stderr_content` are logged but they're still empty strings — the queue data is never assigned back. - -#### C3. Sync `get_tts_model()` ignores external provider in async context - -**`backend/tts.py:15-29`** - -```python -def get_tts_model(): - manager = get_provider_manager() - loop = asyncio.get_event_loop() - if loop.is_running(): - # We're in an async context, but can't await here - return manager._get_default_provider() -``` - -FastAPI routes are async. This function is called from several code paths during generation. In async context it **always returns the bundled provider**, ignoring whatever external provider the user selected. The user downloads and starts a CUDA provider, but generation still runs on CPU. - -### Critical — Security - -#### C4. Path traversal via `tarfile.extractall()` (CVE-2007-4559) - -**`backend/providers/installer.py:115-118`** - -```python -with tarfile.open(archive_path, 'r:gz') as tar_ref: - tar_ref.extractall(providers_dir) -``` - -No member path filtering. A crafted `.tar.gz` from a compromised CDN can write files anywhere on disk via `../` entries. Python 3.12+ emits a deprecation warning for exactly this pattern. - -**Fix:** -```python -tar_ref.extractall(providers_dir, filter='data') # Python 3.12+ -``` - -Or manually validate each member: -```python -for member in tar_ref.getmembers(): - member_path = os.path.join(providers_dir, member.name) - if not os.path.commonpath([providers_dir, member_path]).startswith(str(providers_dir)): - raise ValueError(f"Path traversal attempt: {member.name}") -tar_ref.extractall(providers_dir) -``` - -#### C5. No checksum verification on downloaded binaries - -**`backend/providers/checksums.py`** - -```python -PROVIDER_CHECKSUMS = {} -``` - -Empty dict. `download_provider()` in `installer.py` never calls any verification function. Downloaded binaries are `chmod 0o755`'d and executed without integrity checks. A MitM or CDN compromise delivers arbitrary code. - -**Fix:** Populate checksums per release. Verify SHA-256 after download before extraction: -```python -import hashlib -sha256 = hashlib.sha256(archive_path.read_bytes()).hexdigest() -if sha256 != expected: - archive_path.unlink() - raise ValueError(f"Checksum mismatch for {provider_type}") -``` - -#### C6. Provider servers have no authentication - -**`providers/pytorch-cpu/main.py:18-23`** - -```python -app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - ... -) -``` - -Zero auth. Any local process — including browser JavaScript via localhost — can send requests to the provider on its ephemeral port. Port is discoverable by scanning. - -**Fix:** Generate a random token in the parent process, pass via environment variable to the child, validate in middleware: -```python -# Parent (ProviderManager) -token = secrets.token_urlsafe(32) -env = {**os.environ, "VOICEBOX_PROVIDER_TOKEN": token} -process = subprocess.Popen([...], env=env, ...) - -# Child (provider server) -EXPECTED_TOKEN = os.environ.get("VOICEBOX_PROVIDER_TOKEN") - -@app.middleware("http") -async def verify_token(request, call_next): - if request.headers.get("X-Provider-Token") != EXPECTED_TOKEN: - return JSONResponse(status_code=403, content={"error": "unauthorized"}) - return await call_next(request) -``` - -### Major — Will Cause Problems in Production - -#### M1. Leaked file handles on subprocess stdout/stderr - -**`backend/providers/__init__.py:68-73`** - -```python -process = subprocess.Popen( - [...], - stdout=open(stdout_log, 'w'), # leaked handle - stderr=open(stderr_log, 'w'), # leaked handle -) -``` - -File handles passed directly from `open()` without storing references. They close on GC, not deterministically. On Windows the log files stay locked and unreadable until the process exits. - -**Fix:** -```python -stdout_fh = open(stdout_log, 'w') -stderr_fh = open(stderr_log, 'w') -try: - process = subprocess.Popen([...], stdout=stdout_fh, stderr=stderr_fh) -finally: - stdout_fh.close() - stderr_fh.close() -``` - -#### M2. No subprocess crash detection or recovery - -**`backend/providers/__init__.py:56-110`** - -Once `start_provider()` succeeds, the `Popen` object is stored but never polled. If the provider process crashes mid-session: -- `LocalProvider` HTTP calls fail with `httpx.ConnectError` -- No auto-restart -- No health-check loop -- User sees cryptic "connection refused" errors -- Must manually restart provider from UI - -**Fix:** Background asyncio task that polls `process.poll()` every few seconds. On crash, update provider status and optionally auto-restart: -```python -async def _watch_provider_process(self): - while self._provider_process and self._provider_process.poll() is None: - await asyncio.sleep(5) - if self._provider_process and self._provider_process.returncode != 0: - logger.error(f"Provider crashed with code {self._provider_process.returncode}") - self.active_provider = self._default_provider - # Notify frontend via next health check -``` - -#### M3. Port allocation race condition (TOCTOU) - -**`backend/providers/__init__.py:145-149`** - -```python -def _get_free_port(self) -> int: - with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: - s.bind(('', 0)) - return s.getsockname()[1] - # Socket closed here — port is free but unprotected -``` - -Between this function returning and the provider process binding, another process can claim the port. On busy systems this causes "address already in use" failures. - -**Fix options:** -- Pass the socket fd to the child process (complex, platform-specific) -- Retry with a new port on bind failure (simplest) -- Use a fixed port range and try sequentially - -#### M4. `delete_provider()` leaves hundreds of MB behind - -**`backend/providers/installer.py:155-168`** - -```python -provider_path.unlink() # Deletes just the executable -``` - -PyInstaller `--onedir` produces a directory with the executable plus all shared libraries. `unlink()` only removes the binary file, leaving behind hundreds of MB of `.so`/`.dll`/`.dylib` files. - -**Fix:** -```python -provider_dir = provider_path.parent -shutil.rmtree(provider_dir) -``` - -#### M5. `LocalProvider.combine_voice_prompts()` bypasses the provider - -**`backend/providers/local.py:68-88`** - -This method imports from `..utils.audio` and processes locally instead of sending to the provider server. If the user chose an external provider because they lack local dependencies (e.g., no PyTorch on the machine), this will crash with `ImportError`. - -#### M6. Download errors silently swallowed - -**`backend/main.py:1640`** - -```python -asyncio.create_task(download_provider(provider_type)) -``` - -Fire-and-forget. If the download fails, the exception is logged as "Task exception was never retrieved." The frontend SSE progress stream may hang forever showing "downloading" without the error. - -**Fix:** Store the task, add an error callback: -```python -task = asyncio.create_task(download_provider(provider_type)) -task.add_done_callback(lambda t: t.exception() if not t.cancelled() else None) -``` -And propagate errors through the progress manager so the SSE stream surfaces them. - -#### M7. `LocalProvider.is_loaded()` always returns `True` - -**`backend/providers/local.py:105-108`** - -```python -def is_loaded(self) -> bool: - return True # Return True optimistically -``` - -Health/status checks always report the model as loaded for external providers, even when the provider hasn't loaded anything yet. This breaks the "download model if not cached" logic in the generation flow. - -#### M8. `instruct` parameter silently dropped - -**`backend/providers/local.py:33-40`** - -The `generate()` method accepts `instruct` but never includes it in the JSON payload. The provider server also hardcodes `instruct=None`. Delivery instructions silently do nothing for external providers. - -### Minor - -| # | Issue | Location | -|---|-------|----------| -| m1 | `pytorch-cpu/main.py` and `pytorch-cuda/main.py` are 95% identical | Both files | -| m2 | `build.py` scripts also nearly identical | Both build files | -| m3 | `navigator.platform` is deprecated | `ProviderSettings.tsx:20-23` | -| m4 | `console.log('currentProvider', ...)` left in | `ProviderSettings.tsx:151` | -| m5 | `ProviderType` enum defined but never used for validation | `types.py:10-15` | -| m6 | `list_installed()` reimplements platform detection | `__init__.py:129-143` | -| m7 | New `httpx.AsyncClient` created per health poll iteration | `__init__.py:151-165` | -| m8 | `load_model_async()` only stores size, doesn't actually preload | `local.py:95-99` | - ---- - -## Scope Creep - -The PR should be split. These are independent changes bundled in: - -| Change | Lines | Should Be Separate PR | -|--------|-------|-----------------------| -| `docs2/` site rewrite | ~3000 | Yes | -| Docker support (Dockerfile, compose, docs) | ~600 | Yes — overlaps with PR #161 | -| Landing page banner removal | ~30 | Yes | -| UI refactors (Stories, History, Voices, Audio) | ~400 | Yes | -| Linux audio capture module | ~10 | Yes | -| Dependency bumps | ~100 | Yes | - -**Core provider system** (the actual feature) is ~2500 lines across backend + frontend + provider servers. That's the reviewable scope. - ---- - -## What's Well-Designed - -These parts should survive any rewrite: - -1. **`TTSProvider` Protocol** (`base.py`) — Structural typing via `@runtime_checkable Protocol`. Right pattern. Comprehensive interface. - -2. **`BundledProvider` / `LocalProvider` split** — Clean separation between in-process and HTTP-based inference. The wrapper pattern in `BundledProvider` correctly delegates to existing `TTSBackend`. - -3. **R2 distribution strategy** — Provider binaries on Cloudflare R2, main app on GitHub Releases. Correct solution to the 2 GB limit. - -4. **Progress tracking** — SSE-based download progress integrated with the existing `ProgressManager`. Good UX. - -5. **Subprocess log files** — Writing provider stdout/stderr to log files in the data directory is pragmatic and debuggable. - -6. **Frontend `ProviderSettings.tsx`** — Clean component structure. Proper loading/disabled states, confirmation dialogs, platform-aware visibility. - -7. **CI split** — Separate `build-providers` and `release` jobs. Providers built and uploaded to R2 independently. - ---- - -## Options for Moving Forward - -### Option A — Fix and Slim PR #33 - -Strip the PR down to just the provider system (~2500 lines). Fix the 5 critical and 8 major bugs. Rebase onto current `main`. - -**Effort:** ~2-3 days focused work -**Pros:** Full auto-managed provider lifecycle. Foundation for multi-model. -**Cons:** Still complex. Process management is inherently fragile cross-platform. - -### Option B — Manual External Server Mode - -Skip subprocess management entirely. Ship a "Connect to External Server" feature: - -1. User downloads CUDA provider zip from `downloads.voicebox.sh` -2. User runs it manually (`./tts-provider-pytorch-cuda --port 8100`) -3. In Voicebox UI: paste `http://localhost:8100` as the TTS server URL -4. Voicebox routes generation to that URL via `LocalProvider` - -This reuses `LocalProvider` from PR #33 but removes: -- `ProviderManager` subprocess spawning (the buggiest part) -- `installer.py` download/extract logic (the security risks) -- Port allocation (user picks the port) -- Process lifecycle management (user's responsibility) - -**Effort:** ~1 day. `LocalProvider` + a URL input field + health check. -**Pros:** Simple, reliable, no process management bugs, no security surface. -**Cons:** Manual setup. Not seamless. But CUDA users are already technical (they run from source today). - -### Option C — Hybrid (Recommended) - -Ship Option B first as v0.2.0. Then iterate toward auto-management: - -**Phase 1 (v0.2.0):** Manual external server mode -- `LocalProvider` HTTP client (from PR #33, with the 422 bug fixed) -- Server URL input in Settings -- Health indicator -- CUDA provider published as standalone zip on R2 -- One page of docs: "download, unzip, run, paste URL" - -**Phase 2 (v0.2.x):** Auto-download + auto-start -- `installer.py` with checksum verification and safe extraction -- `ProviderManager` subprocess spawning with crash detection -- Provider settings UI with download/start/stop buttons - -**Phase 3 (v0.3.0):** Multi-model providers -- Provider per model family (not just per hardware) -- LuxTTS provider, Chatterbox provider, etc. -- Provider marketplace / registry - -This gets CUDA into users' hands immediately (Phase 1 is ~1 day) while building toward the full vision incrementally. Each phase is independently shippable and testable. - -### Option D — GitHub Workaround - -Avoid the provider architecture entirely. Host CUDA binaries on R2 and add a download link in the app that opens the user's browser. User downloads the full monolithic CUDA build, replaces their existing install. - -**Effort:** Minimal — just hosting + a link. -**Pros:** Zero architecture changes. -**Cons:** Doesn't solve: multi-model, independent app updates, or the re-download-everything-on-update problem. Kicks the can. - ---- - -## Recommendation - -**Option C (Hybrid)** is the strongest path. Specifically: - -1. **Now:** Close PR #33 as-is. It's too large, too buggy, and too stale to salvage as a single merge. - -2. **Extract:** Cherry-pick the good parts into small focused PRs: - - PR: `TTSProvider` Protocol + `BundledProvider` + `LocalProvider` (the abstractions) - - PR: Provider settings UI (the frontend) - - PR: `installer.py` + checksums (the download system) - - PR: CI changes for R2 upload (the distribution) - -3. **Ship Phase 1:** Manual external server mode. One small PR. Unblocks every CUDA user immediately. - -4. **Iterate:** Layer in auto-management once the manual mode is proven stable. - -The critical bugs in PR #33 (C1-C6) are all fixable, but the PR's size makes review unreliable. Splitting it ensures each piece gets proper attention and nothing ships broken. - ---- - -## Bug Summary - -| Severity | Count | Blocks Ship? | -|----------|-------|-------------| -| Critical (runtime crash) | 3 | Yes — C1, C2, C3 | -| Critical (security) | 3 | Yes — C4, C5, C6 | -| Major | 8 | Some — M1, M2, M3 are high risk | -| Minor | 8 | No | -| **Total** | **22** | | diff --git a/docs/plans/TTS_PROVIDER_ARCHITECTURE.md b/docs/plans/TTS_PROVIDER_ARCHITECTURE.md deleted file mode 100644 index 8d35a7e5..00000000 --- a/docs/plans/TTS_PROVIDER_ARCHITECTURE.md +++ /dev/null @@ -1,964 +0,0 @@ -# TTS Provider Architecture - -**Status:** Planned for v0.1.13 -**Created:** 2025-01-31 -**Problem:** GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads - ---- - -## Overview - -Split the monolithic backend into modular components: - -1. **Main App** (~150-200MB): Tauri + FastAPI backend + Whisper + UI/profiles/history -2. **TTS Providers** (downloadable plugins): Separate executables for model inference - -This architecture solves: - -- ✅ GitHub 2GB release artifact limit -- ✅ Frequent app updates without re-downloading large python binaries -- ✅ User choice of compute backend (CPU/GPU/Cloud) -- ✅ External provider support (OpenAI, custom servers) -- ✅ Future extensibility - ---- - -## Architecture Diagram - -``` -┌─────────────────────────────────────────────────────────┐ -│ Voicebox App (Tauri + Backend) ~150MB │ -│ ├─ UI Layer (React) │ -│ ├─ Backend (FastAPI) │ -│ │ ├─ Voice Profiles │ -│ │ ├─ Generation History │ -│ │ ├─ Audio Editing / Stories │ -│ │ └─ Provider Manager ◄──────────────┐ │ -│ └─ Whisper (bundled, tiny ~50MB) │ │ -└─────────────────────────────────────────┼────────────────┘ - │ - HTTP/IPC │ - │ - ┌────────────────────────────────┼─────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ -│ TTS Provider: │ │ TTS Provider: │ │ TTS Provider: │ -│ PyTorch CPU │ │ PyTorch CUDA │ │ MLX (Apple) │ -│ │ │ │ │ │ -│ ~300MB │ │ ~2.4GB │ │ ~800MB │ -│ │ │ │ │ │ -│ Local inference │ │ GPU inference │ │ Metal inference │ -└─────────────────┘ └─────────────────┘ └──────────────────┘ - │ │ │ - └────────────────────────┴─────────────────────┘ - │ - ┌─────────────▼──────────────┐ - │ Future Providers: │ - │ • Remote Server │ - │ • OpenAI API │ - │ • ElevenLabs │ - │ • Custom Docker Container │ - └────────────────────────────┘ -``` - ---- - -## Problem Statement - -### Current Architecture Issues - -**Monolithic Binary:** - -- CPU version: ~295MB -- CUDA version: ~2.37GB -- GitHub releases: 2GB file size limit (BLOCKED) -- Updates require re-downloading entire binary -- Poor UX: update app → restart → download CUDA update → restart again - -**User Pain Points:** - -1. Cannot release CUDA version on GitHub (over 2GB) -2. Every app update forces 2.4GB re-download for GPU users -3. No flexibility (can't use OpenAI, remote servers, etc.) -4. Wastes bandwidth for small bug fixes - ---- - -## Solution: Pluggable TTS Providers - -### Component Breakdown - -#### 1. Main App (voicebox.exe / .app / .AppImage) - -**Size:** ~100-150MB - -**Includes:** - -- Tauri runtime + React UI -- FastAPI backend (pure Python, no PyTorch) -- Whisper model (tiny, ~50MB) -- SQLite database -- Profile/history/audio editing logic -- Provider management system - -**Does NOT include:** - -- PyTorch (CPU or CUDA) -- TTS models (Qwen3-TTS) -- Heavy ML dependencies - -**Updates frequently:** UI fixes, feature additions, non-ML changes - ---- - -#### 2. TTS Provider: PyTorch CPU - -**Binary:** `tts-provider-pytorch-cpu.exe` -**Size:** ~200MB - -**Includes:** - -- PyTorch CPU build -- Qwen3-TTS package -- Transformers -- No CUDA libraries - -**Download source:** Cloudflare R2 -**Updates rarely:** Only when model code changes - ---- - -#### 3. TTS Provider: PyTorch CUDA - -**Binary:** `tts-provider-pytorch-cuda.exe` -**Size:** ~2.4GB - -**Includes:** - -- PyTorch CUDA build (cu121) -- Qwen3-TTS package -- CUDA runtime, cuDNN, cuBLAS -- Transformers - -**Download source:** Cloudflare R2 -**Platform:** Windows + Linux (NVIDIA GPU) -**Updates rarely:** Only when model code or CUDA version changes - ---- - -#### 4. TTS Provider: MLX - -**Binary:** `tts-provider-mlx` -**Size:** ~150MB - -**Includes:** - -- MLX framework -- MLX-optimized Qwen3-TTS -- Metal acceleration - -**Platform:** macOS only (Apple Silicon) -**Download source:** Cloudflare R2 - ---- - -#### 5. TTS Provider: Remote - -**Binary:** None (built-in config) -**Size:** 0MB - -**How it works:** - -- User provides URL to their own TTS server -- Backend proxies requests to that server -- Implements API spec from `EXTERNAL_PROVIDERS.md` - -**Use cases:** - -- AMD GPU users running their own server -- Team deployments with shared GPU server -- Cloud hosting (Modal, RunPod, Replicate) - ---- - -#### 6. TTS Provider: OpenAI - -**Binary:** None (API wrapper) -**Size:** 0MB - -**How it works:** - -- User provides OpenAI API key -- Backend wraps OpenAI Audio API -- Voice profiles map to OpenAI voices - -**Benefits:** - -- Zero local compute -- Pay-per-use -- Instant setup - ---- - -## Communication Protocol - -### Provider API Specification - -All TTS providers must implement these endpoints: - -#### POST /tts/generate - -Generate speech from text. - -**Request:** - -```json -{ - "text": "Hello world!", - "voice_prompt": { - /* voice prompt object */ - }, - "language": "en", - "seed": 12345, - "model_size": "1.7B" -} -``` - -**Response:** - -```json -{ - "audio": "base64-encoded-audio", - "sample_rate": 24000, - "duration": 2.5 -} -``` - -#### POST /tts/create_voice_prompt - -Create voice prompt from reference audio. - -**Request:** (multipart/form-data) - -- `audio`: Audio file -- `reference_text`: Transcript - -**Response:** - -```json -{ - "voice_prompt": { - /* serialized prompt */ - } -} -``` - -#### GET /tts/health - -Health check. - -**Response:** - -```json -{ - "status": "healthy", - "provider": "pytorch-cuda", - "version": "1.0.0", - "model": "Qwen3-TTS-12Hz-1.7B-Base", - "device": "cuda:0" -} -``` - -#### GET /tts/status - -Model status. - -**Response:** - -```json -{ - "model_loaded": true, - "model_size": "1.7B", - "available_sizes": ["0.6B", "1.7B"], - "gpu_available": true, - "vram_used_mb": 1234 -} -``` - ---- - -## Backend Implementation - -### Provider Manager - -**File:** `backend/providers/__init__.py` - -```python -class ProviderManager: - """Manages TTS provider lifecycle.""" - - def __init__(self): - self.active_provider: Optional[Provider] = None - self.config = load_provider_config() - - async def start_provider(self, provider_type: str) -> str: - """Start a TTS provider process.""" - if provider_type == "pytorch-cpu": - return await self._start_local_provider("tts-provider-pytorch-cpu.exe") - elif provider_type == "pytorch-cuda": - return await self._start_local_provider("tts-provider-pytorch-cuda.exe") - elif provider_type == "mlx": - return await self._start_local_provider("tts-provider-mlx") - elif provider_type == "remote": - return self.config["remote_url"] - elif provider_type == "openai": - return None # No subprocess, API wrapper - - async def _start_local_provider(self, binary_name: str) -> str: - """Start local provider subprocess.""" - provider_path = get_provider_binary_path(binary_name) - - if not provider_path.exists(): - raise ProviderNotInstalledException(binary_name) - - # Start subprocess on random port - port = get_free_port() - process = subprocess.Popen([ - str(provider_path), - "--port", str(port), - "--data-dir", str(config.get_data_dir()) - ]) - - # Wait for provider to be ready - await wait_for_provider_health(f"http://localhost:{port}") - - self.active_provider = Provider(process, port) - return f"http://localhost:{port}" - - async def stop_provider(self): - """Stop active provider.""" - if self.active_provider: - self.active_provider.process.terminate() - self.active_provider = None -``` - ---- - -### Provider Abstraction - -**File:** `backend/providers/base.py` - -```python -class TTSProvider(ABC): - """Abstract base for TTS providers.""" - - @abstractmethod - async def generate( - self, - text: str, - voice_prompt: dict, - language: str, - seed: Optional[int] - ) -> tuple[np.ndarray, int]: - """Generate speech audio.""" - pass - - @abstractmethod - async def create_voice_prompt( - self, - audio_path: str, - reference_text: str - ) -> dict: - """Create voice prompt from reference audio.""" - pass -``` - -**File:** `backend/providers/local.py` - -```python -class LocalProvider(TTSProvider): - """Provider that communicates with local subprocess via HTTP.""" - - def __init__(self, base_url: str): - self.base_url = base_url - self.client = httpx.AsyncClient() - - async def generate(self, text, voice_prompt, language, seed): - response = await self.client.post( - f"{self.base_url}/tts/generate", - json={ - "text": text, - "voice_prompt": voice_prompt, - "language": language, - "seed": seed - } - ) - data = response.json() - audio = np.frombuffer(base64.b64decode(data["audio"]), dtype=np.float32) - return audio, data["sample_rate"] -``` - -**File:** `backend/providers/openai.py` - -```python -class OpenAIProvider(TTSProvider): - """Provider that wraps OpenAI Audio API.""" - - def __init__(self, api_key: str): - self.client = OpenAI(api_key=api_key) - - async def generate(self, text, voice_prompt, language, seed): - # Map voice_prompt to OpenAI voice name - voice = map_profile_to_openai_voice(voice_prompt) - - response = await self.client.audio.speech.create( - model="tts-1", - voice=voice, - input=text - ) - - # Convert to numpy array - audio_data = response.content - audio, sr = load_audio_from_bytes(audio_data) - return audio, sr -``` - ---- - -## Provider Installation - -### Download Manager - -**File:** `backend/providers/installer.py` - -```python -class ProviderInstaller: - """Handles provider download and installation.""" - - async def download_provider(self, provider_type: str): - """Download provider binary from R2.""" - - binary_name = { - "pytorch-cpu": "tts-provider-pytorch-cpu.exe", - "pytorch-cuda": "tts-provider-pytorch-cuda.exe", - "mlx": "tts-provider-mlx" - }[provider_type] - - download_url = f"https://downloads.voicebox.sh/providers/v{PROVIDER_VERSION}/{binary_name}" - - # Download with progress tracking (reuse existing SSE system) - await download_with_progress( - url=download_url, - destination=get_provider_install_path(binary_name), - progress_key=f"provider-{provider_type}" - ) -``` - -**Provider Storage Location:** - -- Windows: `%APPDATA%/voicebox/providers/` -- macOS: `~/Library/Application Support/voicebox/providers/` -- Linux: `~/.local/share/voicebox/providers/` - ---- - -## Frontend Implementation - -### Provider Settings UI - -**Component:** `app/src/components/ServerSettings/ProviderSettings.tsx` - -```tsx -export function ProviderSettings() { - const [selectedProvider, setSelectedProvider] = - useState("auto"); - const {data: installedProviders} = useQuery({ - queryKey: ["providers", "installed"], - queryFn: () => apiClient.getInstalledProviders(), - }); - - return ( - - - TTS Provider - Choose how Voicebox generates speech - - - - {/* Auto-detect */} -
- - -
- - {/* PyTorch CUDA */} -
-
- - -
- {!installedProviders?.includes("pytorch-cuda") && gpuAvailable && ( - - )} -
- - {/* PyTorch CPU */} -
-
- - -
- {!installedProviders?.includes("pytorch-cpu") && ( - - )} -
- - {/* MLX (macOS only) */} - {isMacOS && ( -
-
- - -
- {!installedProviders?.includes("mlx") && ( - - )} -
- )} - - {/* Remote */} -
-
- - -
- {selectedProvider === "remote" && ( - - )} -
- - {/* OpenAI */} -
-
- - -
- {selectedProvider === "openai" && ( - - )} -
- - - - ); -} -``` - ---- - -## File Structure - -``` -voicebox/ -├── backend/ -│ ├── main.py # Main FastAPI app (no TTS code) -│ ├── providers/ -│ │ ├── __init__.py # ProviderManager -│ │ ├── base.py # TTSProvider ABC -│ │ ├── local.py # LocalProvider (subprocess) -│ │ ├── remote.py # RemoteProvider (HTTP) -│ │ ├── openai.py # OpenAIProvider (API wrapper) -│ │ └── installer.py # Provider download logic -│ ├── profiles.py # Voice profile management -│ ├── history.py # Generation history -│ ├── transcribe.py # Whisper (still bundled) -│ └── ... (other backend modules) -│ -├── providers/ -│ ├── pytorch-cpu/ -│ │ ├── main.py # FastAPI server for TTS -│ │ ├── tts_backend.py # PyTorch TTS logic -│ │ ├── requirements.txt # torch (CPU), qwen-tts, transformers -│ │ └── build.spec # PyInstaller spec -│ │ -│ ├── pytorch-cuda/ -│ │ ├── main.py # FastAPI server for TTS -│ │ ├── tts_backend.py # PyTorch TTS logic -│ │ ├── requirements.txt # torch+cu121, qwen-tts, transformers -│ │ └── build.spec # PyInstaller spec -│ │ -│ └── mlx/ -│ ├── main.py # FastAPI server for TTS -│ ├── mlx_backend.py # MLX TTS logic -│ ├── requirements.txt # mlx, qwen-tts-mlx -│ └── build.spec # PyInstaller spec -│ -├── app/ # Frontend (Tauri + React) -│ └── src/ -│ └── components/ -│ └── ServerSettings/ -│ └── ProviderSettings.tsx -│ -└── tauri/ - └── src-tauri/ - └── tauri.conf.json # No externalBin for providers -``` - ---- - -## Migration Path - -### Phase 1: Refactor Backend (No User Changes) - -**Goal:** Abstract TTS behind provider interface - -1. Create `backend/providers/` module structure -2. Implement `TTSProvider` abstract base class -3. Create `LocalProvider` wrapper for current PyTorch code -4. Modify `backend/tts.py` to use provider abstraction -5. Keep PyTorch bundled in main app - -**Result:** Code is prepared, but user experience unchanged - ---- - -### Phase 2: Build Provider Binaries - -**Goal:** Create standalone TTS provider executables - -1. Create separate PyInstaller specs for each provider -2. Build provider executables: - - `tts-provider-pytorch-cpu.exe` (~300MB) - - `tts-provider-pytorch-cuda.exe` (~2.4GB) - - `tts-provider-mlx` (~800MB, macOS) -3. Test subprocess communication -4. Upload providers to Cloudflare R2 - -**Result:** Provider binaries exist but aren't used yet - ---- - -### Phase 3: Remove PyTorch from Main App - -**Goal:** Split main app from providers - -1. Exclude PyTorch/Qwen3-TTS from main app PyInstaller spec -2. Main app now requires provider download -3. Update GitHub CI to build multiple artifacts: - - `voicebox-{version}-{platform}.exe` (~150MB) - - `tts-provider-pytorch-cpu-{version}.exe` - - `tts-provider-pytorch-cuda-{version}.exe` - - `tts-provider-mlx-{version}` (macOS) - -**Result:** Main app is small, providers downloaded separately - ---- - -### Phase 4: Add Provider UI - -**Goal:** User-facing provider management - -1. Create Provider Settings page -2. Implement provider download UI -3. Add provider status indicators -4. Show active provider in UI - -**Result:** Users can choose and download providers - ---- - -### Phase 5: External Providers - -**Goal:** Enable remote and cloud providers - -1. Implement `RemoteProvider` (HTTP client) -2. Implement `OpenAIProvider` (API wrapper) -3. Add provider configuration UI (URLs, API keys) -4. Document external provider API spec - -**Result:** Full provider ecosystem - ---- - -## Provider Versioning - -### Independent Versioning - -Providers have their own version numbers, independent of the main app: - -- **App version:** `v0.2.0` (frequent updates) -- **Provider version:** `v1.0.0` (rare updates) - -### Compatibility Matrix - -**Example:** - -| App Version | Min Provider Version | Max Provider Version | -| ----------- | -------------------- | -------------------- | -| v0.2.0 | v1.0.0 | v1.x.x | -| v0.3.0 | v1.0.0 | v1.x.x | -| v0.4.0 | v1.2.0 | v1.x.x | -| v1.0.0 | v2.0.0 | v2.x.x | - -**Backend checks compatibility:** - -```python -async def check_provider_compatibility(provider_version: str) -> bool: - """Check if provider version is compatible with current app.""" - min_version = "1.0.0" - max_version = "1.999.999" - return min_version <= provider_version < max_version -``` - -**UI shows warning if incompatible:** - -``` -⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+ -``` - ---- - -## User Flows - -### First-Time Setup - -1. User downloads and installs Voicebox (~150MB) -2. App launches → detects no TTS provider installed -3. Shows setup wizard: - - ``` - Choose your TTS provider: - - [ ] PyTorch CUDA (2.4GB) [Download] - ✓ Fastest on NVIDIA GPUs - ✗ Requires NVIDIA GPU - - [●] PyTorch CPU (300MB) [Download] - ✓ Works on any system - ✗ Slower inference - - [ ] MLX (800MB) [Download] - ✓ Fast on Apple Silicon - ✗ macOS only (M1/M2/M3) - - [ ] Remote Server - URL: ___________________ - - [ ] OpenAI API - API Key: ________________ - ``` - -4. User selects provider → downloads with progress bar -5. Provider installs to AppData/Application Support -6. App starts provider → ready to use - ---- - -### App Update Flow (No Provider Change) - -**Scenario:** Bug fix in UI, no backend changes - -1. User gets update notification: "Voicebox v0.2.1 available" -2. Downloads update (~150MB, not 2.4GB!) -3. Installs and restarts -4. **Provider stays the same** (no re-download needed) -5. App starts using existing provider - -**User experience:** Fast updates, no multi-GB downloads - ---- - -### Provider Update Flow - -**Scenario:** New Qwen3-TTS model version released - -1. User opens Settings → Provider tab -2. Sees notification: "Provider update available (v1.1.0)" -3. Clicks "Update Provider" -4. Downloads new provider binary -5. Old provider binary is replaced -6. Restart app to use new provider - -**Frequency:** Rare (only when TTS model/backend changes) - ---- - -### Switching Providers - -**Scenario:** User upgrades to NVIDIA GPU - -1. User goes to Settings → Provider -2. Selects "PyTorch CUDA" -3. Clicks "Download" → downloads 2.4GB -4. Download completes → restarts app -5. App now uses CUDA provider - ---- - -## Benefits - -| Benefit | Details | -| ----------------------------- | --------------------------------------------------------- | -| **GitHub Releases Work** | Main app ~150MB << 2GB limit | -| **Fast Updates** | UI/feature updates don't require re-downloading providers | -| **User Choice** | CPU, CUDA, MLX, OpenAI, remote server | -| **External Provider Support** | Users can run their own TTS servers | -| **Bandwidth Savings** | Only download provider once, app updates are small | -| **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) | -| **Team Deployments** | Multiple users share one remote provider | -| **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. | - ---- - -## Open Questions - -### 1. Provider Versioning - -**Question:** Should providers have independent versions or match app version? - -**Options:** - -- A. Independent (providers: v1.x, app: v0.2.x) -- B. Matched (both use v0.2.x) - -**Recommendation:** Independent versioning with compatibility matrix - ---- - -### 2. Auto-Update Providers - -**Question:** Should providers auto-update separately from app? - -**Options:** - -- A. Manual updates only (user clicks "Update Provider") -- B. Optional auto-update (user can enable) -- C. Always auto-update - -**Recommendation:** Optional auto-update (default off) - ---- - -### 3. Provider Discovery - -**Question:** How does app find installed providers? - -**Options:** - -- A. Check standard paths in AppData/Application Support -- B. Registry (Windows) / plist (macOS) -- C. Config file with provider locations - -**Recommendation:** Standard paths + config fallback - ---- - -### 4. Fallback Behavior - -**Question:** What if no provider is installed? - -**Options:** - -- A. Show setup wizard on first launch -- B. Block app until provider installed -- C. Allow app to run in "demo mode" (transcription only) - -**Recommendation:** Setup wizard on first launch - ---- - -### 5. Provider Auto-Start - -**Question:** Should provider start automatically with app? - -**Options:** - -- A. Always start selected provider on app launch -- B. Start on-demand (when user generates speech) -- C. User preference - -**Recommendation:** Auto-start (configurable in settings) - ---- - -## Future Enhancements - -- [ ] **Provider Marketplace:** Built-in directory of community providers -- [ ] **Multi-Provider Support:** Use different providers per voice/language -- [ ] **Provider Health Monitoring:** Automatic failover if provider crashes -- [ ] **Cost Tracking:** Monitor API usage for OpenAI/cloud providers -- [ ] **Performance Metrics:** Latency, throughput, VRAM usage dashboards -- [ ] **Docker Providers:** Run providers in Docker containers -- [ ] **Provider Plugins:** Load custom providers from user scripts - ---- - -## Related Documents - -- [EXTERNAL_PROVIDERS.md](./EXTERNAL_PROVIDERS.md) - External provider support plan -- [OPENAI_SUPPORT.md](./OPENAI_SUPPORT.md) - OpenAI API compatibility -- [github-2gb-limit-issue.md](../github-2gb-limit-issue.md) - Original problem -- [r2-setup.md](../r2-setup.md) - Cloudflare R2 configuration - ---- - -## Contributing - -If you want to build a custom TTS provider: - -1. Implement the provider API spec (see above) -2. Test with Voicebox locally -3. Package as executable (PyInstaller, Docker, etc.) -4. Share in GitHub Discussions - -**Questions?** - -- GitHub Issues: [voicebox/issues](https://github.com/jamiepine/voicebox/issues) -- Discord: Coming soon