# TTS Provider Architecture **Status:** Planned for v0.1.13 **Created:** 2025-01-31 **Problem:** GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads --- ## Overview Split the monolithic backend into modular components: 1. **Main App** (~150-200MB): Tauri + FastAPI backend + Whisper + UI/profiles/history 2. **TTS Providers** (downloadable plugins): Separate executables for model inference This architecture solves: - ✅ GitHub 2GB release artifact limit - ✅ Frequent app updates without re-downloading large python binaries - ✅ User choice of compute backend (CPU/GPU/Cloud) - ✅ External provider support (OpenAI, custom servers) - ✅ Future extensibility --- ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────┐ │ Voicebox App (Tauri + Backend) ~150MB │ │ ├─ UI Layer (React) │ │ ├─ Backend (FastAPI) │ │ │ ├─ Voice Profiles │ │ │ ├─ Generation History │ │ │ ├─ Audio Editing / Stories │ │ │ └─ Provider Manager ◄──────────────┐ │ │ └─ Whisper (bundled, tiny ~50MB) │ │ └─────────────────────────────────────────┼────────────────┘ │ HTTP/IPC │ │ ┌────────────────────────────────┼─────────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ │ TTS Provider: │ │ TTS Provider: │ │ TTS Provider: │ │ PyTorch CPU │ │ PyTorch CUDA │ │ MLX (Apple) │ │ │ │ │ │ │ │ ~300MB │ │ ~2.4GB │ │ ~800MB │ │ │ │ │ │ │ │ Local inference │ │ GPU inference │ │ Metal inference │ └─────────────────┘ └─────────────────┘ └──────────────────┘ │ │ │ └────────────────────────┴─────────────────────┘ │ ┌─────────────▼──────────────┐ │ Future Providers: │ │ • Remote Server │ │ • OpenAI API │ │ • ElevenLabs │ │ • Custom Docker Container │ └────────────────────────────┘ ``` --- ## Problem Statement ### Current Architecture Issues **Monolithic Binary:** - CPU version: ~295MB - CUDA version: ~2.37GB - GitHub releases: 2GB file size limit (BLOCKED) - Updates require re-downloading entire binary - Poor UX: update app → restart → download CUDA update → restart again **User Pain Points:** 1. Cannot release CUDA version on GitHub (over 2GB) 2. Every app update forces 2.4GB re-download for GPU users 3. No flexibility (can't use OpenAI, remote servers, etc.) 4. Wastes bandwidth for small bug fixes --- ## Solution: Pluggable TTS Providers ### Component Breakdown #### 1. Main App (voicebox.exe / .app / .AppImage) **Size:** ~100-150MB **Includes:** - Tauri runtime + React UI - FastAPI backend (pure Python, no PyTorch) - Whisper model (tiny, ~50MB) - SQLite database - Profile/history/audio editing logic - Provider management system **Does NOT include:** - PyTorch (CPU or CUDA) - TTS models (Qwen3-TTS) - Heavy ML dependencies **Updates frequently:** UI fixes, feature additions, non-ML changes --- #### 2. TTS Provider: PyTorch CPU **Binary:** `tts-provider-pytorch-cpu.exe` **Size:** ~200MB **Includes:** - PyTorch CPU build - Qwen3-TTS package - Transformers - No CUDA libraries **Download source:** Cloudflare R2 **Updates rarely:** Only when model code changes --- #### 3. TTS Provider: PyTorch CUDA **Binary:** `tts-provider-pytorch-cuda.exe` **Size:** ~2.4GB **Includes:** - PyTorch CUDA build (cu121) - Qwen3-TTS package - CUDA runtime, cuDNN, cuBLAS - Transformers **Download source:** Cloudflare R2 **Platform:** Windows + Linux (NVIDIA GPU) **Updates rarely:** Only when model code or CUDA version changes --- #### 4. TTS Provider: MLX **Binary:** `tts-provider-mlx` **Size:** ~150MB **Includes:** - MLX framework - MLX-optimized Qwen3-TTS - Metal acceleration **Platform:** macOS only (Apple Silicon) **Download source:** Cloudflare R2 --- #### 5. TTS Provider: Remote **Binary:** None (built-in config) **Size:** 0MB **How it works:** - User provides URL to their own TTS server - Backend proxies requests to that server - Implements API spec from `EXTERNAL_PROVIDERS.md` **Use cases:** - AMD GPU users running their own server - Team deployments with shared GPU server - Cloud hosting (Modal, RunPod, Replicate) --- #### 6. TTS Provider: OpenAI **Binary:** None (API wrapper) **Size:** 0MB **How it works:** - User provides OpenAI API key - Backend wraps OpenAI Audio API - Voice profiles map to OpenAI voices **Benefits:** - Zero local compute - Pay-per-use - Instant setup --- ## Communication Protocol ### Provider API Specification All TTS providers must implement these endpoints: #### POST /tts/generate Generate speech from text. **Request:** ```json { "text": "Hello world!", "voice_prompt": { /* voice prompt object */ }, "language": "en", "seed": 12345, "model_size": "1.7B" } ``` **Response:** ```json { "audio": "base64-encoded-audio", "sample_rate": 24000, "duration": 2.5 } ``` #### POST /tts/create_voice_prompt Create voice prompt from reference audio. **Request:** (multipart/form-data) - `audio`: Audio file - `reference_text`: Transcript **Response:** ```json { "voice_prompt": { /* serialized prompt */ } } ``` #### GET /tts/health Health check. **Response:** ```json { "status": "healthy", "provider": "pytorch-cuda", "version": "1.0.0", "model": "Qwen3-TTS-12Hz-1.7B-Base", "device": "cuda:0" } ``` #### GET /tts/status Model status. **Response:** ```json { "model_loaded": true, "model_size": "1.7B", "available_sizes": ["0.6B", "1.7B"], "gpu_available": true, "vram_used_mb": 1234 } ``` --- ## Backend Implementation ### Provider Manager **File:** `backend/providers/__init__.py` ```python class ProviderManager: """Manages TTS provider lifecycle.""" def __init__(self): self.active_provider: Optional[Provider] = None self.config = load_provider_config() async def start_provider(self, provider_type: str) -> str: """Start a TTS provider process.""" if provider_type == "pytorch-cpu": return await self._start_local_provider("tts-provider-pytorch-cpu.exe") elif provider_type == "pytorch-cuda": return await self._start_local_provider("tts-provider-pytorch-cuda.exe") elif provider_type == "mlx": return await self._start_local_provider("tts-provider-mlx") elif provider_type == "remote": return self.config["remote_url"] elif provider_type == "openai": return None # No subprocess, API wrapper async def _start_local_provider(self, binary_name: str) -> str: """Start local provider subprocess.""" provider_path = get_provider_binary_path(binary_name) if not provider_path.exists(): raise ProviderNotInstalledException(binary_name) # Start subprocess on random port port = get_free_port() process = subprocess.Popen([ str(provider_path), "--port", str(port), "--data-dir", str(config.get_data_dir()) ]) # Wait for provider to be ready await wait_for_provider_health(f"http://localhost:{port}") self.active_provider = Provider(process, port) return f"http://localhost:{port}" async def stop_provider(self): """Stop active provider.""" if self.active_provider: self.active_provider.process.terminate() self.active_provider = None ``` --- ### Provider Abstraction **File:** `backend/providers/base.py` ```python class TTSProvider(ABC): """Abstract base for TTS providers.""" @abstractmethod async def generate( self, text: str, voice_prompt: dict, language: str, seed: Optional[int] ) -> tuple[np.ndarray, int]: """Generate speech audio.""" pass @abstractmethod async def create_voice_prompt( self, audio_path: str, reference_text: str ) -> dict: """Create voice prompt from reference audio.""" pass ``` **File:** `backend/providers/local.py` ```python class LocalProvider(TTSProvider): """Provider that communicates with local subprocess via HTTP.""" def __init__(self, base_url: str): self.base_url = base_url self.client = httpx.AsyncClient() async def generate(self, text, voice_prompt, language, seed): response = await self.client.post( f"{self.base_url}/tts/generate", json={ "text": text, "voice_prompt": voice_prompt, "language": language, "seed": seed } ) data = response.json() audio = np.frombuffer(base64.b64decode(data["audio"]), dtype=np.float32) return audio, data["sample_rate"] ``` **File:** `backend/providers/openai.py` ```python class OpenAIProvider(TTSProvider): """Provider that wraps OpenAI Audio API.""" def __init__(self, api_key: str): self.client = OpenAI(api_key=api_key) async def generate(self, text, voice_prompt, language, seed): # Map voice_prompt to OpenAI voice name voice = map_profile_to_openai_voice(voice_prompt) response = await self.client.audio.speech.create( model="tts-1", voice=voice, input=text ) # Convert to numpy array audio_data = response.content audio, sr = load_audio_from_bytes(audio_data) return audio, sr ``` --- ## Provider Installation ### Download Manager **File:** `backend/providers/installer.py` ```python class ProviderInstaller: """Handles provider download and installation.""" async def download_provider(self, provider_type: str): """Download provider binary from R2.""" binary_name = { "pytorch-cpu": "tts-provider-pytorch-cpu.exe", "pytorch-cuda": "tts-provider-pytorch-cuda.exe", "mlx": "tts-provider-mlx" }[provider_type] download_url = f"https://downloads.voicebox.sh/providers/v{PROVIDER_VERSION}/{binary_name}" # Download with progress tracking (reuse existing SSE system) await download_with_progress( url=download_url, destination=get_provider_install_path(binary_name), progress_key=f"provider-{provider_type}" ) ``` **Provider Storage Location:** - Windows: `%APPDATA%/voicebox/providers/` - macOS: `~/Library/Application Support/voicebox/providers/` - Linux: `~/.local/share/voicebox/providers/` --- ## Frontend Implementation ### Provider Settings UI **Component:** `app/src/components/ServerSettings/ProviderSettings.tsx` ```tsx export function ProviderSettings() { const [selectedProvider, setSelectedProvider] = useState("auto"); const {data: installedProviders} = useQuery({ queryKey: ["providers", "installed"], queryFn: () => apiClient.getInstalledProviders(), }); return ( TTS Provider Choose how Voicebox generates speech {/* Auto-detect */}
{/* PyTorch CUDA */}
{!installedProviders?.includes("pytorch-cuda") && gpuAvailable && ( )}
{/* PyTorch CPU */}
{!installedProviders?.includes("pytorch-cpu") && ( )}
{/* MLX (macOS only) */} {isMacOS && (
{!installedProviders?.includes("mlx") && ( )}
)} {/* Remote */}
{selectedProvider === "remote" && ( )}
{/* OpenAI */}
{selectedProvider === "openai" && ( )}
); } ``` --- ## File Structure ``` voicebox/ ├── backend/ │ ├── main.py # Main FastAPI app (no TTS code) │ ├── providers/ │ │ ├── __init__.py # ProviderManager │ │ ├── base.py # TTSProvider ABC │ │ ├── local.py # LocalProvider (subprocess) │ │ ├── remote.py # RemoteProvider (HTTP) │ │ ├── openai.py # OpenAIProvider (API wrapper) │ │ └── installer.py # Provider download logic │ ├── profiles.py # Voice profile management │ ├── history.py # Generation history │ ├── transcribe.py # Whisper (still bundled) │ └── ... (other backend modules) │ ├── providers/ │ ├── pytorch-cpu/ │ │ ├── main.py # FastAPI server for TTS │ │ ├── tts_backend.py # PyTorch TTS logic │ │ ├── requirements.txt # torch (CPU), qwen-tts, transformers │ │ └── build.spec # PyInstaller spec │ │ │ ├── pytorch-cuda/ │ │ ├── main.py # FastAPI server for TTS │ │ ├── tts_backend.py # PyTorch TTS logic │ │ ├── requirements.txt # torch+cu121, qwen-tts, transformers │ │ └── build.spec # PyInstaller spec │ │ │ └── mlx/ │ ├── main.py # FastAPI server for TTS │ ├── mlx_backend.py # MLX TTS logic │ ├── requirements.txt # mlx, qwen-tts-mlx │ └── build.spec # PyInstaller spec │ ├── app/ # Frontend (Tauri + React) │ └── src/ │ └── components/ │ └── ServerSettings/ │ └── ProviderSettings.tsx │ └── tauri/ └── src-tauri/ └── tauri.conf.json # No externalBin for providers ``` --- ## Migration Path ### Phase 1: Refactor Backend (No User Changes) **Goal:** Abstract TTS behind provider interface 1. Create `backend/providers/` module structure 2. Implement `TTSProvider` abstract base class 3. Create `LocalProvider` wrapper for current PyTorch code 4. Modify `backend/tts.py` to use provider abstraction 5. Keep PyTorch bundled in main app **Result:** Code is prepared, but user experience unchanged --- ### Phase 2: Build Provider Binaries **Goal:** Create standalone TTS provider executables 1. Create separate PyInstaller specs for each provider 2. Build provider executables: - `tts-provider-pytorch-cpu.exe` (~300MB) - `tts-provider-pytorch-cuda.exe` (~2.4GB) - `tts-provider-mlx` (~800MB, macOS) 3. Test subprocess communication 4. Upload providers to Cloudflare R2 **Result:** Provider binaries exist but aren't used yet --- ### Phase 3: Remove PyTorch from Main App **Goal:** Split main app from providers 1. Exclude PyTorch/Qwen3-TTS from main app PyInstaller spec 2. Main app now requires provider download 3. Update GitHub CI to build multiple artifacts: - `voicebox-{version}-{platform}.exe` (~150MB) - `tts-provider-pytorch-cpu-{version}.exe` - `tts-provider-pytorch-cuda-{version}.exe` - `tts-provider-mlx-{version}` (macOS) **Result:** Main app is small, providers downloaded separately --- ### Phase 4: Add Provider UI **Goal:** User-facing provider management 1. Create Provider Settings page 2. Implement provider download UI 3. Add provider status indicators 4. Show active provider in UI **Result:** Users can choose and download providers --- ### Phase 5: External Providers **Goal:** Enable remote and cloud providers 1. Implement `RemoteProvider` (HTTP client) 2. Implement `OpenAIProvider` (API wrapper) 3. Add provider configuration UI (URLs, API keys) 4. Document external provider API spec **Result:** Full provider ecosystem --- ## Provider Versioning ### Independent Versioning Providers have their own version numbers, independent of the main app: - **App version:** `v0.2.0` (frequent updates) - **Provider version:** `v1.0.0` (rare updates) ### Compatibility Matrix **Example:** | App Version | Min Provider Version | Max Provider Version | | ----------- | -------------------- | -------------------- | | v0.2.0 | v1.0.0 | v1.x.x | | v0.3.0 | v1.0.0 | v1.x.x | | v0.4.0 | v1.2.0 | v1.x.x | | v1.0.0 | v2.0.0 | v2.x.x | **Backend checks compatibility:** ```python async def check_provider_compatibility(provider_version: str) -> bool: """Check if provider version is compatible with current app.""" min_version = "1.0.0" max_version = "1.999.999" return min_version <= provider_version < max_version ``` **UI shows warning if incompatible:** ``` ⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+ ``` --- ## User Flows ### First-Time Setup 1. User downloads and installs Voicebox (~150MB) 2. App launches → detects no TTS provider installed 3. Shows setup wizard: ``` Choose your TTS provider: [ ] PyTorch CUDA (2.4GB) [Download] ✓ Fastest on NVIDIA GPUs ✗ Requires NVIDIA GPU [●] PyTorch CPU (300MB) [Download] ✓ Works on any system ✗ Slower inference [ ] MLX (800MB) [Download] ✓ Fast on Apple Silicon ✗ macOS only (M1/M2/M3) [ ] Remote Server URL: ___________________ [ ] OpenAI API API Key: ________________ ``` 4. User selects provider → downloads with progress bar 5. Provider installs to AppData/Application Support 6. App starts provider → ready to use --- ### App Update Flow (No Provider Change) **Scenario:** Bug fix in UI, no backend changes 1. User gets update notification: "Voicebox v0.2.1 available" 2. Downloads update (~150MB, not 2.4GB!) 3. Installs and restarts 4. **Provider stays the same** (no re-download needed) 5. App starts using existing provider **User experience:** Fast updates, no multi-GB downloads --- ### Provider Update Flow **Scenario:** New Qwen3-TTS model version released 1. User opens Settings → Provider tab 2. Sees notification: "Provider update available (v1.1.0)" 3. Clicks "Update Provider" 4. Downloads new provider binary 5. Old provider binary is replaced 6. Restart app to use new provider **Frequency:** Rare (only when TTS model/backend changes) --- ### Switching Providers **Scenario:** User upgrades to NVIDIA GPU 1. User goes to Settings → Provider 2. Selects "PyTorch CUDA" 3. Clicks "Download" → downloads 2.4GB 4. Download completes → restarts app 5. App now uses CUDA provider --- ## Benefits | Benefit | Details | | ----------------------------- | --------------------------------------------------------- | | **GitHub Releases Work** | Main app ~150MB << 2GB limit | | **Fast Updates** | UI/feature updates don't require re-downloading providers | | **User Choice** | CPU, CUDA, MLX, OpenAI, remote server | | **External Provider Support** | Users can run their own TTS servers | | **Bandwidth Savings** | Only download provider once, app updates are small | | **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) | | **Team Deployments** | Multiple users share one remote provider | | **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. | --- ## Open Questions ### 1. Provider Versioning **Question:** Should providers have independent versions or match app version? **Options:** - A. Independent (providers: v1.x, app: v0.2.x) - B. Matched (both use v0.2.x) **Recommendation:** Independent versioning with compatibility matrix --- ### 2. Auto-Update Providers **Question:** Should providers auto-update separately from app? **Options:** - A. Manual updates only (user clicks "Update Provider") - B. Optional auto-update (user can enable) - C. Always auto-update **Recommendation:** Optional auto-update (default off) --- ### 3. Provider Discovery **Question:** How does app find installed providers? **Options:** - A. Check standard paths in AppData/Application Support - B. Registry (Windows) / plist (macOS) - C. Config file with provider locations **Recommendation:** Standard paths + config fallback --- ### 4. Fallback Behavior **Question:** What if no provider is installed? **Options:** - A. Show setup wizard on first launch - B. Block app until provider installed - C. Allow app to run in "demo mode" (transcription only) **Recommendation:** Setup wizard on first launch --- ### 5. Provider Auto-Start **Question:** Should provider start automatically with app? **Options:** - A. Always start selected provider on app launch - B. Start on-demand (when user generates speech) - C. User preference **Recommendation:** Auto-start (configurable in settings) --- ## Future Enhancements - [ ] **Provider Marketplace:** Built-in directory of community providers - [ ] **Multi-Provider Support:** Use different providers per voice/language - [ ] **Provider Health Monitoring:** Automatic failover if provider crashes - [ ] **Cost Tracking:** Monitor API usage for OpenAI/cloud providers - [ ] **Performance Metrics:** Latency, throughput, VRAM usage dashboards - [ ] **Docker Providers:** Run providers in Docker containers - [ ] **Provider Plugins:** Load custom providers from user scripts --- ## Related Documents - [EXTERNAL_PROVIDERS.md](./EXTERNAL_PROVIDERS.md) - External provider support plan - [OPENAI_SUPPORT.md](./OPENAI_SUPPORT.md) - OpenAI API compatibility - [github-2gb-limit-issue.md](../github-2gb-limit-issue.md) - Original problem - [r2-setup.md](../r2-setup.md) - Cloudflare R2 configuration --- ## Contributing If you want to build a custom TTS provider: 1. Implement the provider API spec (see above) 2. Test with Voicebox locally 3. Package as executable (PyInstaller, Docker, etc.) 4. Share in GitHub Discussions **Questions?** - GitHub Issues: [voicebox/issues](https://github.com/jamiepine/voicebox/issues) - Discord: Coming soon