From 2bc243f93e463d91552d86b4ac3af7d7011bebeb Mon Sep 17 00:00:00 2001 From: Jamie Pine Date: Sat, 31 Jan 2026 02:09:29 -0800 Subject: [PATCH] Add TTS Provider Architecture plan Solves GitHub 2GB limit + frequent update UX issues by splitting app into: - Main app (~150MB): UI + backend logic + Whisper - TTS Providers (plugins): Separate downloadable binaries - pytorch-cpu (~300MB) - pytorch-cuda (~2.4GB) - mlx (~800MB, macOS) - remote (connect to external server) - openai (API wrapper) Benefits: - Main app under GitHub 2GB limit - Updates don't require re-downloading providers - User choice of compute backend - External provider support for teams/cloud - Future-proof extensibility --- docs/plans/TTS_PROVIDER_ARCHITECTURE.md | 932 ++++++++++++++++++++++++ 1 file changed, 932 insertions(+) create mode 100644 docs/plans/TTS_PROVIDER_ARCHITECTURE.md diff --git a/docs/plans/TTS_PROVIDER_ARCHITECTURE.md b/docs/plans/TTS_PROVIDER_ARCHITECTURE.md new file mode 100644 index 00000000..37ad40f7 --- /dev/null +++ b/docs/plans/TTS_PROVIDER_ARCHITECTURE.md @@ -0,0 +1,932 @@ +# TTS Provider Architecture + +**Status:** Planned for v0.2.0 +**Created:** 2025-01-31 +**Problem:** GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads + +--- + +## Overview + +Split the monolithic backend into modular components: + +1. **Main App** (~150-200MB): Tauri + FastAPI backend + Whisper + UI/profiles/history +2. **TTS Providers** (downloadable plugins): Separate executables for model inference + +This architecture solves: +- ✅ GitHub 2GB release artifact limit +- ✅ Frequent app updates without re-downloading large ML models +- ✅ User choice of compute backend (CPU/GPU/Cloud) +- ✅ External provider support (OpenAI, custom servers) +- ✅ Future extensibility + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────┐ +│ Voicebox App (Tauri + Backend) ~150MB │ +│ ├─ UI Layer (React) │ +│ ├─ Backend (FastAPI) │ +│ │ ├─ Voice Profiles │ +│ │ ├─ Generation History │ +│ │ ├─ Audio Editing / Stories │ +│ │ └─ Provider Manager ◄──────────────┐ │ +│ └─ Whisper (bundled, tiny ~50MB) │ │ +└─────────────────────────────────────────┼────────────────┘ + │ + HTTP/IPC │ + │ + ┌────────────────────────────────┼─────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ +│ TTS Provider: │ │ TTS Provider: │ │ TTS Provider: │ +│ PyTorch CPU │ │ PyTorch CUDA │ │ MLX (Apple) │ +│ │ │ │ │ │ +│ ~300MB │ │ ~2.4GB │ │ ~800MB │ +│ │ │ │ │ │ +│ Local inference │ │ GPU inference │ │ Metal inference │ +└─────────────────┘ └─────────────────┘ └──────────────────┘ + │ │ │ + └────────────────────────┴─────────────────────┘ + │ + ┌─────────────▼──────────────┐ + │ Future Providers: │ + │ • Remote Server │ + │ • OpenAI API │ + │ • ElevenLabs │ + │ • Custom Docker Container │ + └────────────────────────────┘ +``` + +--- + +## Problem Statement + +### Current Architecture Issues + +**Monolithic Binary:** +- CPU version: ~295MB +- CUDA version: ~2.37GB +- GitHub releases: 2GB file size limit (BLOCKED) +- Updates require re-downloading entire binary +- Poor UX: update app → restart → download CUDA update → restart again + +**User Pain Points:** +1. Cannot release CUDA version on GitHub (over 2GB) +2. Every app update forces 2.4GB re-download for GPU users +3. No flexibility (can't use OpenAI, remote servers, etc.) +4. Wastes bandwidth for small bug fixes + +--- + +## Solution: Pluggable TTS Providers + +### Component Breakdown + +#### 1. Main App (voicebox.exe / .app / .AppImage) + +**Size:** ~150-200MB + +**Includes:** +- Tauri runtime + React UI +- FastAPI backend (pure Python, no PyTorch) +- Whisper model (tiny, ~50MB) +- SQLite database +- Profile/history/audio editing logic +- Provider management system + +**Does NOT include:** +- PyTorch (CPU or CUDA) +- TTS models (Qwen3-TTS) +- Heavy ML dependencies + +**Updates frequently:** UI fixes, feature additions, non-ML changes + +--- + +#### 2. TTS Provider: PyTorch CPU + +**Binary:** `tts-provider-pytorch-cpu.exe` +**Size:** ~300MB + +**Includes:** +- PyTorch CPU build +- Qwen3-TTS package +- Transformers +- No CUDA libraries + +**Download source:** Cloudflare R2 +**Updates rarely:** Only when model code changes + +--- + +#### 3. TTS Provider: PyTorch CUDA + +**Binary:** `tts-provider-pytorch-cuda.exe` +**Size:** ~2.4GB + +**Includes:** +- PyTorch CUDA build (cu121) +- Qwen3-TTS package +- CUDA runtime, cuDNN, cuBLAS +- Transformers + +**Download source:** Cloudflare R2 +**Platform:** Windows + Linux (NVIDIA GPU) +**Updates rarely:** Only when model code or CUDA version changes + +--- + +#### 4. TTS Provider: MLX + +**Binary:** `tts-provider-mlx` +**Size:** ~800MB + +**Includes:** +- MLX framework +- MLX-optimized Qwen3-TTS +- Metal acceleration + +**Platform:** macOS only (Apple Silicon) +**Download source:** Cloudflare R2 + +--- + +#### 5. TTS Provider: Remote + +**Binary:** None (built-in config) +**Size:** 0MB + +**How it works:** +- User provides URL to their own TTS server +- Backend proxies requests to that server +- Implements API spec from `EXTERNAL_PROVIDERS.md` + +**Use cases:** +- AMD GPU users running their own server +- Team deployments with shared GPU server +- Cloud hosting (Modal, RunPod, Replicate) + +--- + +#### 6. TTS Provider: OpenAI + +**Binary:** None (API wrapper) +**Size:** 0MB + +**How it works:** +- User provides OpenAI API key +- Backend wraps OpenAI Audio API +- Voice profiles map to OpenAI voices + +**Benefits:** +- Zero local compute +- Pay-per-use +- Instant setup + +--- + +## Communication Protocol + +### Provider API Specification + +All TTS providers must implement these endpoints: + +#### POST /tts/generate + +Generate speech from text. + +**Request:** +```json +{ + "text": "Hello world!", + "voice_prompt": { /* voice prompt object */ }, + "language": "en", + "seed": 12345, + "model_size": "1.7B" +} +``` + +**Response:** +```json +{ + "audio": "base64-encoded-audio", + "sample_rate": 24000, + "duration": 2.5 +} +``` + +#### POST /tts/create_voice_prompt + +Create voice prompt from reference audio. + +**Request:** (multipart/form-data) +- `audio`: Audio file +- `reference_text`: Transcript + +**Response:** +```json +{ + "voice_prompt": { /* serialized prompt */ } +} +``` + +#### GET /tts/health + +Health check. + +**Response:** +```json +{ + "status": "healthy", + "provider": "pytorch-cuda", + "version": "1.0.0", + "model": "Qwen3-TTS-12Hz-1.7B-Base", + "device": "cuda:0" +} +``` + +#### GET /tts/status + +Model status. + +**Response:** +```json +{ + "model_loaded": true, + "model_size": "1.7B", + "available_sizes": ["0.6B", "1.7B"], + "gpu_available": true, + "vram_used_mb": 1234 +} +``` + +--- + +## Backend Implementation + +### Provider Manager + +**File:** `backend/providers/__init__.py` + +```python +class ProviderManager: + """Manages TTS provider lifecycle.""" + + def __init__(self): + self.active_provider: Optional[Provider] = None + self.config = load_provider_config() + + async def start_provider(self, provider_type: str) -> str: + """Start a TTS provider process.""" + if provider_type == "pytorch-cpu": + return await self._start_local_provider("tts-provider-pytorch-cpu.exe") + elif provider_type == "pytorch-cuda": + return await self._start_local_provider("tts-provider-pytorch-cuda.exe") + elif provider_type == "mlx": + return await self._start_local_provider("tts-provider-mlx") + elif provider_type == "remote": + return self.config["remote_url"] + elif provider_type == "openai": + return None # No subprocess, API wrapper + + async def _start_local_provider(self, binary_name: str) -> str: + """Start local provider subprocess.""" + provider_path = get_provider_binary_path(binary_name) + + if not provider_path.exists(): + raise ProviderNotInstalledException(binary_name) + + # Start subprocess on random port + port = get_free_port() + process = subprocess.Popen([ + str(provider_path), + "--port", str(port), + "--data-dir", str(config.get_data_dir()) + ]) + + # Wait for provider to be ready + await wait_for_provider_health(f"http://localhost:{port}") + + self.active_provider = Provider(process, port) + return f"http://localhost:{port}" + + async def stop_provider(self): + """Stop active provider.""" + if self.active_provider: + self.active_provider.process.terminate() + self.active_provider = None +``` + +--- + +### Provider Abstraction + +**File:** `backend/providers/base.py` + +```python +class TTSProvider(ABC): + """Abstract base for TTS providers.""" + + @abstractmethod + async def generate( + self, + text: str, + voice_prompt: dict, + language: str, + seed: Optional[int] + ) -> tuple[np.ndarray, int]: + """Generate speech audio.""" + pass + + @abstractmethod + async def create_voice_prompt( + self, + audio_path: str, + reference_text: str + ) -> dict: + """Create voice prompt from reference audio.""" + pass +``` + +**File:** `backend/providers/local.py` + +```python +class LocalProvider(TTSProvider): + """Provider that communicates with local subprocess via HTTP.""" + + def __init__(self, base_url: str): + self.base_url = base_url + self.client = httpx.AsyncClient() + + async def generate(self, text, voice_prompt, language, seed): + response = await self.client.post( + f"{self.base_url}/tts/generate", + json={ + "text": text, + "voice_prompt": voice_prompt, + "language": language, + "seed": seed + } + ) + data = response.json() + audio = np.frombuffer(base64.b64decode(data["audio"]), dtype=np.float32) + return audio, data["sample_rate"] +``` + +**File:** `backend/providers/openai.py` + +```python +class OpenAIProvider(TTSProvider): + """Provider that wraps OpenAI Audio API.""" + + def __init__(self, api_key: str): + self.client = OpenAI(api_key=api_key) + + async def generate(self, text, voice_prompt, language, seed): + # Map voice_prompt to OpenAI voice name + voice = map_profile_to_openai_voice(voice_prompt) + + response = await self.client.audio.speech.create( + model="tts-1", + voice=voice, + input=text + ) + + # Convert to numpy array + audio_data = response.content + audio, sr = load_audio_from_bytes(audio_data) + return audio, sr +``` + +--- + +## Provider Installation + +### Download Manager + +**File:** `backend/providers/installer.py` + +```python +class ProviderInstaller: + """Handles provider download and installation.""" + + async def download_provider(self, provider_type: str): + """Download provider binary from R2.""" + + binary_name = { + "pytorch-cpu": "tts-provider-pytorch-cpu.exe", + "pytorch-cuda": "tts-provider-pytorch-cuda.exe", + "mlx": "tts-provider-mlx" + }[provider_type] + + download_url = f"https://downloads.voicebox.sh/providers/v{PROVIDER_VERSION}/{binary_name}" + + # Download with progress tracking (reuse existing SSE system) + await download_with_progress( + url=download_url, + destination=get_provider_install_path(binary_name), + progress_key=f"provider-{provider_type}" + ) +``` + +**Provider Storage Location:** +- Windows: `%APPDATA%/voicebox/providers/` +- macOS: `~/Library/Application Support/voicebox/providers/` +- Linux: `~/.local/share/voicebox/providers/` + +--- + +## Frontend Implementation + +### Provider Settings UI + +**Component:** `app/src/components/ServerSettings/ProviderSettings.tsx` + +```tsx +export function ProviderSettings() { + const [selectedProvider, setSelectedProvider] = useState('auto'); + const { data: installedProviders } = useQuery({ + queryKey: ['providers', 'installed'], + queryFn: () => apiClient.getInstalledProviders() + }); + + return ( + + + TTS Provider + + Choose how Voicebox generates speech + + + + + + {/* Auto-detect */} +
+ + +
+ + {/* PyTorch CUDA */} +
+
+ + +
+ {!installedProviders?.includes('pytorch-cuda') && gpuAvailable && ( + + )} +
+ + {/* PyTorch CPU */} +
+
+ + +
+ {!installedProviders?.includes('pytorch-cpu') && ( + + )} +
+ + {/* MLX (macOS only) */} + {isMacOS && ( +
+
+ + +
+ {!installedProviders?.includes('mlx') && ( + + )} +
+ )} + + {/* Remote */} +
+
+ + +
+ {selectedProvider === 'remote' && ( + + )} +
+ + {/* OpenAI */} +
+
+ + +
+ {selectedProvider === 'openai' && ( + + )} +
+ + + + + ); +} +``` + +--- + +## File Structure + +``` +voicebox/ +├── backend/ +│ ├── main.py # Main FastAPI app (no TTS code) +│ ├── providers/ +│ │ ├── __init__.py # ProviderManager +│ │ ├── base.py # TTSProvider ABC +│ │ ├── local.py # LocalProvider (subprocess) +│ │ ├── remote.py # RemoteProvider (HTTP) +│ │ ├── openai.py # OpenAIProvider (API wrapper) +│ │ └── installer.py # Provider download logic +│ ├── profiles.py # Voice profile management +│ ├── history.py # Generation history +│ ├── transcribe.py # Whisper (still bundled) +│ └── ... (other backend modules) +│ +├── providers/ +│ ├── pytorch-cpu/ +│ │ ├── main.py # FastAPI server for TTS +│ │ ├── tts_backend.py # PyTorch TTS logic +│ │ ├── requirements.txt # torch (CPU), qwen-tts, transformers +│ │ └── build.spec # PyInstaller spec +│ │ +│ ├── pytorch-cuda/ +│ │ ├── main.py # FastAPI server for TTS +│ │ ├── tts_backend.py # PyTorch TTS logic +│ │ ├── requirements.txt # torch+cu121, qwen-tts, transformers +│ │ └── build.spec # PyInstaller spec +│ │ +│ └── mlx/ +│ ├── main.py # FastAPI server for TTS +│ ├── mlx_backend.py # MLX TTS logic +│ ├── requirements.txt # mlx, qwen-tts-mlx +│ └── build.spec # PyInstaller spec +│ +├── app/ # Frontend (Tauri + React) +│ └── src/ +│ └── components/ +│ └── ServerSettings/ +│ └── ProviderSettings.tsx +│ +└── tauri/ + └── src-tauri/ + └── tauri.conf.json # No externalBin for providers +``` + +--- + +## Migration Path + +### Phase 1: Refactor Backend (No User Changes) + +**Goal:** Abstract TTS behind provider interface + +1. Create `backend/providers/` module structure +2. Implement `TTSProvider` abstract base class +3. Create `LocalProvider` wrapper for current PyTorch code +4. Modify `backend/tts.py` to use provider abstraction +5. Keep PyTorch bundled in main app + +**Result:** Code is prepared, but user experience unchanged + +--- + +### Phase 2: Build Provider Binaries + +**Goal:** Create standalone TTS provider executables + +1. Create separate PyInstaller specs for each provider +2. Build provider executables: + - `tts-provider-pytorch-cpu.exe` (~300MB) + - `tts-provider-pytorch-cuda.exe` (~2.4GB) + - `tts-provider-mlx` (~800MB, macOS) +3. Test subprocess communication +4. Upload providers to Cloudflare R2 + +**Result:** Provider binaries exist but aren't used yet + +--- + +### Phase 3: Remove PyTorch from Main App + +**Goal:** Split main app from providers + +1. Exclude PyTorch/Qwen3-TTS from main app PyInstaller spec +2. Main app now requires provider download +3. Update GitHub CI to build multiple artifacts: + - `voicebox-{version}-{platform}.exe` (~150MB) + - `tts-provider-pytorch-cpu-{version}.exe` + - `tts-provider-pytorch-cuda-{version}.exe` + - `tts-provider-mlx-{version}` (macOS) + +**Result:** Main app is small, providers downloaded separately + +--- + +### Phase 4: Add Provider UI + +**Goal:** User-facing provider management + +1. Create Provider Settings page +2. Implement provider download UI +3. Add provider status indicators +4. Show active provider in UI + +**Result:** Users can choose and download providers + +--- + +### Phase 5: External Providers + +**Goal:** Enable remote and cloud providers + +1. Implement `RemoteProvider` (HTTP client) +2. Implement `OpenAIProvider` (API wrapper) +3. Add provider configuration UI (URLs, API keys) +4. Document external provider API spec + +**Result:** Full provider ecosystem + +--- + +## Provider Versioning + +### Independent Versioning + +Providers have their own version numbers, independent of the main app: + +- **App version:** `v0.2.0` (frequent updates) +- **Provider version:** `v1.0.0` (rare updates) + +### Compatibility Matrix + +**Example:** + +| App Version | Min Provider Version | Max Provider Version | +|-------------|---------------------|---------------------| +| v0.2.0 | v1.0.0 | v1.x.x | +| v0.3.0 | v1.0.0 | v1.x.x | +| v0.4.0 | v1.2.0 | v1.x.x | +| v1.0.0 | v2.0.0 | v2.x.x | + +**Backend checks compatibility:** + +```python +async def check_provider_compatibility(provider_version: str) -> bool: + """Check if provider version is compatible with current app.""" + min_version = "1.0.0" + max_version = "1.999.999" + return min_version <= provider_version < max_version +``` + +**UI shows warning if incompatible:** +``` +⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+ +``` + +--- + +## User Flows + +### First-Time Setup + +1. User downloads and installs Voicebox (~150MB) +2. App launches → detects no TTS provider installed +3. Shows setup wizard: + ``` + Choose your TTS provider: + + [ ] PyTorch CUDA (2.4GB) [Download] + ✓ Fastest on NVIDIA GPUs + ✗ Requires NVIDIA GPU + + [●] PyTorch CPU (300MB) [Download] + ✓ Works on any system + ✗ Slower inference + + [ ] MLX (800MB) [Download] + ✓ Fast on Apple Silicon + ✗ macOS only (M1/M2/M3) + + [ ] Remote Server + URL: ___________________ + + [ ] OpenAI API + API Key: ________________ + ``` +4. User selects provider → downloads with progress bar +5. Provider installs to AppData/Application Support +6. App starts provider → ready to use + +--- + +### App Update Flow (No Provider Change) + +**Scenario:** Bug fix in UI, no backend changes + +1. User gets update notification: "Voicebox v0.2.1 available" +2. Downloads update (~150MB, not 2.4GB!) +3. Installs and restarts +4. **Provider stays the same** (no re-download needed) +5. App starts using existing provider + +**User experience:** Fast updates, no multi-GB downloads + +--- + +### Provider Update Flow + +**Scenario:** New Qwen3-TTS model version released + +1. User opens Settings → Provider tab +2. Sees notification: "Provider update available (v1.1.0)" +3. Clicks "Update Provider" +4. Downloads new provider binary +5. Old provider binary is replaced +6. Restart app to use new provider + +**Frequency:** Rare (only when TTS model/backend changes) + +--- + +### Switching Providers + +**Scenario:** User upgrades to NVIDIA GPU + +1. User goes to Settings → Provider +2. Selects "PyTorch CUDA" +3. Clicks "Download" → downloads 2.4GB +4. Download completes → restarts app +5. App now uses CUDA provider + +--- + +## Benefits + +| Benefit | Details | +|---------|---------| +| **GitHub Releases Work** | Main app ~150MB << 2GB limit | +| **Fast Updates** | UI/feature updates don't require re-downloading providers | +| **User Choice** | CPU, CUDA, MLX, OpenAI, remote server | +| **External Provider Support** | Users can run their own TTS servers | +| **Bandwidth Savings** | Only download provider once, app updates are small | +| **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) | +| **Team Deployments** | Multiple users share one remote provider | +| **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. | + +--- + +## Open Questions + +### 1. Provider Versioning + +**Question:** Should providers have independent versions or match app version? + +**Options:** +- A. Independent (providers: v1.x, app: v0.2.x) +- B. Matched (both use v0.2.x) + +**Recommendation:** Independent versioning with compatibility matrix + +--- + +### 2. Auto-Update Providers + +**Question:** Should providers auto-update separately from app? + +**Options:** +- A. Manual updates only (user clicks "Update Provider") +- B. Optional auto-update (user can enable) +- C. Always auto-update + +**Recommendation:** Optional auto-update (default off) + +--- + +### 3. Provider Discovery + +**Question:** How does app find installed providers? + +**Options:** +- A. Check standard paths in AppData/Application Support +- B. Registry (Windows) / plist (macOS) +- C. Config file with provider locations + +**Recommendation:** Standard paths + config fallback + +--- + +### 4. Fallback Behavior + +**Question:** What if no provider is installed? + +**Options:** +- A. Show setup wizard on first launch +- B. Block app until provider installed +- C. Allow app to run in "demo mode" (transcription only) + +**Recommendation:** Setup wizard on first launch + +--- + +### 5. Provider Auto-Start + +**Question:** Should provider start automatically with app? + +**Options:** +- A. Always start selected provider on app launch +- B. Start on-demand (when user generates speech) +- C. User preference + +**Recommendation:** Auto-start (configurable in settings) + +--- + +## Future Enhancements + +- [ ] **Provider Marketplace:** Built-in directory of community providers +- [ ] **Multi-Provider Support:** Use different providers per voice/language +- [ ] **Provider Health Monitoring:** Automatic failover if provider crashes +- [ ] **Cost Tracking:** Monitor API usage for OpenAI/cloud providers +- [ ] **Performance Metrics:** Latency, throughput, VRAM usage dashboards +- [ ] **Docker Providers:** Run providers in Docker containers +- [ ] **Provider Plugins:** Load custom providers from user scripts + +--- + +## Related Documents + +- [EXTERNAL_PROVIDERS.md](./EXTERNAL_PROVIDERS.md) - External provider support plan +- [OPENAI_SUPPORT.md](./OPENAI_SUPPORT.md) - OpenAI API compatibility +- [github-2gb-limit-issue.md](../github-2gb-limit-issue.md) - Original problem +- [r2-setup.md](../r2-setup.md) - Cloudflare R2 configuration + +--- + +## Contributing + +If you want to build a custom TTS provider: + +1. Implement the provider API spec (see above) +2. Test with Voicebox locally +3. Package as executable (PyInstaller, Docker, etc.) +4. Share in GitHub Discussions + +**Questions?** +- GitHub Issues: [voicebox/issues](https://github.com/jamiepine/voicebox/issues) +- Discord: Coming soon