Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-16 13:20:39 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
c9d7bc4f27a373a55e1b5bc5c49d9fc12f2ccee8
voicebox/docs/plans/TTS_PROVIDER_ARCHITECTURE.md
T
Jamie Pine 220333b3bb corrections
2026-01-31 02:15:45 -08:00

26 KiB
Raw Blame History

TTS Provider Architecture

Status: Planned for v0.1.13 Created: 2025-01-31 Problem: GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads

Overview

Split the monolithic backend into modular components:

  1. Main App (~150-200MB): Tauri + FastAPI backend + Whisper + UI/profiles/history
  2. TTS Providers (downloadable plugins): Separate executables for model inference

This architecture solves:

  • ✅ GitHub 2GB release artifact limit
  • ✅ Frequent app updates without re-downloading large python binaries
  • ✅ User choice of compute backend (CPU/GPU/Cloud)
  • ✅ External provider support (OpenAI, custom servers)
  • ✅ Future extensibility

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│  Voicebox App (Tauri + Backend)           ~150MB        │
│  ├─ UI Layer (React)                                    │
│  ├─ Backend (FastAPI)                                   │
│  │  ├─ Voice Profiles                                   │
│  │  ├─ Generation History                               │
│  │  ├─ Audio Editing / Stories                          │
│  │  └─ Provider Manager ◄──────────────┐                │
│  └─ Whisper (bundled, tiny ~50MB)      │                │
└─────────────────────────────────────────┼────────────────┘
                                          │
                            HTTP/IPC      │
                                          │
         ┌────────────────────────────────┼─────────────────┐
         │                                │                 │
         ▼                                ▼                 ▼
┌─────────────────┐      ┌─────────────────┐   ┌──────────────────┐
│ TTS Provider:   │      │ TTS Provider:   │   │ TTS Provider:    │
│ PyTorch CPU     │      │ PyTorch CUDA    │   │ MLX (Apple)      │
│                 │      │                 │   │                  │
│ ~300MB          │      │ ~2.4GB          │   │ ~800MB           │
│                 │      │                 │   │                  │
│ Local inference │      │ GPU inference   │   │ Metal inference  │
└─────────────────┘      └─────────────────┘   └──────────────────┘
         │                        │                     │
         └────────────────────────┴─────────────────────┘
                                  │
                    ┌─────────────▼──────────────┐
                    │  Future Providers:         │
                    │  • Remote Server           │
                    │  • OpenAI API              │
                    │  • ElevenLabs              │
                    │  • Custom Docker Container │
                    └────────────────────────────┘

Problem Statement

Current Architecture Issues

Monolithic Binary:

  • CPU version: ~295MB
  • CUDA version: ~2.37GB
  • GitHub releases: 2GB file size limit (BLOCKED)
  • Updates require re-downloading entire binary
  • Poor UX: update app → restart → download CUDA update → restart again

User Pain Points:

  1. Cannot release CUDA version on GitHub (over 2GB)
  2. Every app update forces 2.4GB re-download for GPU users
  3. No flexibility (can't use OpenAI, remote servers, etc.)
  4. Wastes bandwidth for small bug fixes

Solution: Pluggable TTS Providers

Component Breakdown

1. Main App (voicebox.exe / .app / .AppImage)

Size: ~100-150MB

Includes:

  • Tauri runtime + React UI
  • FastAPI backend (pure Python, no PyTorch)
  • Whisper model (tiny, ~50MB)
  • SQLite database
  • Profile/history/audio editing logic
  • Provider management system

Does NOT include:

  • PyTorch (CPU or CUDA)
  • TTS models (Qwen3-TTS)
  • Heavy ML dependencies

Updates frequently: UI fixes, feature additions, non-ML changes

2. TTS Provider: PyTorch CPU

Binary: tts-provider-pytorch-cpu.exe Size: ~200MB

Includes:

  • PyTorch CPU build
  • Qwen3-TTS package
  • Transformers
  • No CUDA libraries

Download source: Cloudflare R2 Updates rarely: Only when model code changes

3. TTS Provider: PyTorch CUDA

Binary: tts-provider-pytorch-cuda.exe Size: ~2.4GB

Includes:

  • PyTorch CUDA build (cu121)
  • Qwen3-TTS package
  • CUDA runtime, cuDNN, cuBLAS
  • Transformers

Download source: Cloudflare R2 Platform: Windows + Linux (NVIDIA GPU) Updates rarely: Only when model code or CUDA version changes

4. TTS Provider: MLX

Binary: tts-provider-mlx Size: ~150MB

Includes:

  • MLX framework
  • MLX-optimized Qwen3-TTS
  • Metal acceleration

Platform: macOS only (Apple Silicon) Download source: Cloudflare R2

5. TTS Provider: Remote

Binary: None (built-in config) Size: 0MB

How it works:

  • User provides URL to their own TTS server
  • Backend proxies requests to that server
  • Implements API spec from EXTERNAL_PROVIDERS.md

Use cases:

  • AMD GPU users running their own server
  • Team deployments with shared GPU server
  • Cloud hosting (Modal, RunPod, Replicate)

6. TTS Provider: OpenAI

Binary: None (API wrapper) Size: 0MB

How it works:

  • User provides OpenAI API key
  • Backend wraps OpenAI Audio API
  • Voice profiles map to OpenAI voices

Benefits:

  • Zero local compute
  • Pay-per-use
  • Instant setup

Communication Protocol

Provider API Specification

All TTS providers must implement these endpoints:

POST /tts/generate

Generate speech from text.

Request:

{
	"text": "Hello world!",
	"voice_prompt": {
		/* voice prompt object */
	},
	"language": "en",
	"seed": 12345,
	"model_size": "1.7B"
}

Response:

{
	"audio": "base64-encoded-audio",
	"sample_rate": 24000,
	"duration": 2.5
}

POST /tts/create_voice_prompt

Create voice prompt from reference audio.

Request: (multipart/form-data)

  • audio: Audio file
  • reference_text: Transcript

Response:

{
	"voice_prompt": {
		/* serialized prompt */
	}
}

GET /tts/health

Health check.

Response:

{
	"status": "healthy",
	"provider": "pytorch-cuda",
	"version": "1.0.0",
	"model": "Qwen3-TTS-12Hz-1.7B-Base",
	"device": "cuda:0"
}

GET /tts/status

Model status.

Response:

{
	"model_loaded": true,
	"model_size": "1.7B",
	"available_sizes": ["0.6B", "1.7B"],
	"gpu_available": true,
	"vram_used_mb": 1234
}

Backend Implementation

Provider Manager

File: backend/providers/__init__.py

class ProviderManager:
    """Manages TTS provider lifecycle."""

    def __init__(self):
        self.active_provider: Optional[Provider] = None
        self.config = load_provider_config()

    async def start_provider(self, provider_type: str) -> str:
        """Start a TTS provider process."""
        if provider_type == "pytorch-cpu":
            return await self._start_local_provider("tts-provider-pytorch-cpu.exe")
        elif provider_type == "pytorch-cuda":
            return await self._start_local_provider("tts-provider-pytorch-cuda.exe")
        elif provider_type == "mlx":
            return await self._start_local_provider("tts-provider-mlx")
        elif provider_type == "remote":
            return self.config["remote_url"]
        elif provider_type == "openai":
            return None  # No subprocess, API wrapper

    async def _start_local_provider(self, binary_name: str) -> str:
        """Start local provider subprocess."""
        provider_path = get_provider_binary_path(binary_name)

        if not provider_path.exists():
            raise ProviderNotInstalledException(binary_name)

        # Start subprocess on random port
        port = get_free_port()
        process = subprocess.Popen([
            str(provider_path),
            "--port", str(port),
            "--data-dir", str(config.get_data_dir())
        ])

        # Wait for provider to be ready
        await wait_for_provider_health(f"http://localhost:{port}")

        self.active_provider = Provider(process, port)
        return f"http://localhost:{port}"

    async def stop_provider(self):
        """Stop active provider."""
        if self.active_provider:
            self.active_provider.process.terminate()
            self.active_provider = None

Provider Abstraction

File: backend/providers/base.py

class TTSProvider(ABC):
    """Abstract base for TTS providers."""

    @abstractmethod
    async def generate(
        self,
        text: str,
        voice_prompt: dict,
        language: str,
        seed: Optional[int]
    ) -> tuple[np.ndarray, int]:
        """Generate speech audio."""
        pass

    @abstractmethod
    async def create_voice_prompt(
        self,
        audio_path: str,
        reference_text: str
    ) -> dict:
        """Create voice prompt from reference audio."""
        pass

File: backend/providers/local.py

class LocalProvider(TTSProvider):
    """Provider that communicates with local subprocess via HTTP."""

    def __init__(self, base_url: str):
        self.base_url = base_url
        self.client = httpx.AsyncClient()

    async def generate(self, text, voice_prompt, language, seed):
        response = await self.client.post(
            f"{self.base_url}/tts/generate",
            json={
                "text": text,
                "voice_prompt": voice_prompt,
                "language": language,
                "seed": seed
            }
        )
        data = response.json()
        audio = np.frombuffer(base64.b64decode(data["audio"]), dtype=np.float32)
        return audio, data["sample_rate"]

File: backend/providers/openai.py

class OpenAIProvider(TTSProvider):
    """Provider that wraps OpenAI Audio API."""

    def __init__(self, api_key: str):
        self.client = OpenAI(api_key=api_key)

    async def generate(self, text, voice_prompt, language, seed):
        # Map voice_prompt to OpenAI voice name
        voice = map_profile_to_openai_voice(voice_prompt)

        response = await self.client.audio.speech.create(
            model="tts-1",
            voice=voice,
            input=text
        )

        # Convert to numpy array
        audio_data = response.content
        audio, sr = load_audio_from_bytes(audio_data)
        return audio, sr

Provider Installation

Download Manager

File: backend/providers/installer.py

class ProviderInstaller:
    """Handles provider download and installation."""

    async def download_provider(self, provider_type: str):
        """Download provider binary from R2."""

        binary_name = {
            "pytorch-cpu": "tts-provider-pytorch-cpu.exe",
            "pytorch-cuda": "tts-provider-pytorch-cuda.exe",
            "mlx": "tts-provider-mlx"
        }[provider_type]

        download_url = f"https://downloads.voicebox.sh/providers/v{PROVIDER_VERSION}/{binary_name}"

        # Download with progress tracking (reuse existing SSE system)
        await download_with_progress(
            url=download_url,
            destination=get_provider_install_path(binary_name),
            progress_key=f"provider-{provider_type}"
        )

Provider Storage Location:

  • Windows: %APPDATA%/voicebox/providers/
  • macOS: ~/Library/Application Support/voicebox/providers/
  • Linux: ~/.local/share/voicebox/providers/

Frontend Implementation

Provider Settings UI

Component: app/src/components/ServerSettings/ProviderSettings.tsx

export function ProviderSettings() {
	const [selectedProvider, setSelectedProvider] =
		useState<ProviderType>("auto");
	const {data: installedProviders} = useQuery({
		queryKey: ["providers", "installed"],
		queryFn: () => apiClient.getInstalledProviders(),
	});

	return (
		<Card>
			<CardHeader>
				<CardTitle>TTS Provider</CardTitle>
				<CardDescription>Choose how Voicebox generates speech</CardDescription>
			</CardHeader>
			<CardContent>
				<RadioGroup
					value={selectedProvider}
					onValueChange={setSelectedProvider}
				>
					{/* Auto-detect */}
					<div className="flex items-center space-x-2">
						<RadioGroupItem value="auto" id="auto" />
						<Label htmlFor="auto">
							<div className="font-medium">Auto-detect (Recommended)</div>
							<div className="text-sm text-muted-foreground">
								Automatically choose the best available provider
							</div>
						</Label>
					</div>

					{/* PyTorch CUDA */}
					<div className="flex items-center justify-between">
						<div className="flex items-center space-x-2">
							<RadioGroupItem
								value="pytorch-cuda"
								id="cuda"
								disabled={!gpuAvailable}
							/>
							<Label htmlFor="cuda">
								<div className="font-medium">PyTorch CUDA (NVIDIA GPU)</div>
								<div className="text-sm text-muted-foreground">
									4-5x faster inference on NVIDIA GPUs
								</div>
							</Label>
						</div>
						{!installedProviders?.includes("pytorch-cuda") && gpuAvailable && (
							<Button
								onClick={() => downloadProvider("pytorch-cuda")}
								size="sm"
							>
								Download (2.4GB)
							</Button>
						)}
					</div>

					{/* PyTorch CPU */}
					<div className="flex items-center justify-between">
						<div className="flex items-center space-x-2">
							<RadioGroupItem value="pytorch-cpu" id="cpu" />
							<Label htmlFor="cpu">
								<div className="font-medium">PyTorch CPU</div>
								<div className="text-sm text-muted-foreground">
									Works on any system, slower inference
								</div>
							</Label>
						</div>
						{!installedProviders?.includes("pytorch-cpu") && (
							<Button onClick={() => downloadProvider("pytorch-cpu")} size="sm">
								Download (300MB)
							</Button>
						)}
					</div>

					{/* MLX (macOS only) */}
					{isMacOS && (
						<div className="flex items-center justify-between">
							<div className="flex items-center space-x-2">
								<RadioGroupItem value="mlx" id="mlx" />
								<Label htmlFor="mlx">
									<div className="font-medium">MLX (Apple Silicon)</div>
									<div className="text-sm text-muted-foreground">
										Optimized for M1/M2/M3 chips
									</div>
								</Label>
							</div>
							{!installedProviders?.includes("mlx") && (
								<Button onClick={() => downloadProvider("mlx")} size="sm">
									Download (800MB)
								</Button>
							)}
						</div>
					)}

					{/* Remote */}
					<div className="space-y-2">
						<div className="flex items-center space-x-2">
							<RadioGroupItem value="remote" id="remote" />
							<Label htmlFor="remote">
								<div className="font-medium">Remote Server</div>
								<div className="text-sm text-muted-foreground">
									Connect to your own TTS server
								</div>
							</Label>
						</div>
						{selectedProvider === "remote" && (
							<Input placeholder="http://your-server:8000" className="ml-6" />
						)}
					</div>

					{/* OpenAI */}
					<div className="space-y-2">
						<div className="flex items-center space-x-2">
							<RadioGroupItem value="openai" id="openai" />
							<Label htmlFor="openai">
								<div className="font-medium">OpenAI API</div>
								<div className="text-sm text-muted-foreground">
									Use OpenAI's TTS API (requires API key)
								</div>
							</Label>
						</div>
						{selectedProvider === "openai" && (
							<Input type="password" placeholder="sk-..." className="ml-6" />
						)}
					</div>
				</RadioGroup>
			</CardContent>
		</Card>
	);
}

File Structure

voicebox/
├── backend/
│   ├── main.py                    # Main FastAPI app (no TTS code)
│   ├── providers/
│   │   ├── __init__.py            # ProviderManager
│   │   ├── base.py                # TTSProvider ABC
│   │   ├── local.py               # LocalProvider (subprocess)
│   │   ├── remote.py              # RemoteProvider (HTTP)
│   │   ├── openai.py              # OpenAIProvider (API wrapper)
│   │   └── installer.py           # Provider download logic
│   ├── profiles.py                # Voice profile management
│   ├── history.py                 # Generation history
│   ├── transcribe.py              # Whisper (still bundled)
│   └── ... (other backend modules)
│
├── providers/
│   ├── pytorch-cpu/
│   │   ├── main.py                # FastAPI server for TTS
│   │   ├── tts_backend.py         # PyTorch TTS logic
│   │   ├── requirements.txt       # torch (CPU), qwen-tts, transformers
│   │   └── build.spec             # PyInstaller spec
│   │
│   ├── pytorch-cuda/
│   │   ├── main.py                # FastAPI server for TTS
│   │   ├── tts_backend.py         # PyTorch TTS logic
│   │   ├── requirements.txt       # torch+cu121, qwen-tts, transformers
│   │   └── build.spec             # PyInstaller spec
│   │
│   └── mlx/
│       ├── main.py                # FastAPI server for TTS
│       ├── mlx_backend.py         # MLX TTS logic
│       ├── requirements.txt       # mlx, qwen-tts-mlx
│       └── build.spec             # PyInstaller spec
│
├── app/                           # Frontend (Tauri + React)
│   └── src/
│       └── components/
│           └── ServerSettings/
│               └── ProviderSettings.tsx
│
└── tauri/
    └── src-tauri/
        └── tauri.conf.json        # No externalBin for providers

Migration Path

Phase 1: Refactor Backend (No User Changes)

Goal: Abstract TTS behind provider interface

  1. Create backend/providers/ module structure
  2. Implement TTSProvider abstract base class
  3. Create LocalProvider wrapper for current PyTorch code
  4. Modify backend/tts.py to use provider abstraction
  5. Keep PyTorch bundled in main app

Result: Code is prepared, but user experience unchanged

Phase 2: Build Provider Binaries

Goal: Create standalone TTS provider executables

  1. Create separate PyInstaller specs for each provider
  2. Build provider executables:
    • tts-provider-pytorch-cpu.exe (~300MB)
    • tts-provider-pytorch-cuda.exe (~2.4GB)
    • tts-provider-mlx (~800MB, macOS)
  3. Test subprocess communication
  4. Upload providers to Cloudflare R2

Result: Provider binaries exist but aren't used yet

Phase 3: Remove PyTorch from Main App

Goal: Split main app from providers

  1. Exclude PyTorch/Qwen3-TTS from main app PyInstaller spec
  2. Main app now requires provider download
  3. Update GitHub CI to build multiple artifacts:
    • voicebox-{version}-{platform}.exe (~150MB)
    • tts-provider-pytorch-cpu-{version}.exe
    • tts-provider-pytorch-cuda-{version}.exe
    • tts-provider-mlx-{version} (macOS)

Result: Main app is small, providers downloaded separately

Phase 4: Add Provider UI

Goal: User-facing provider management

  1. Create Provider Settings page
  2. Implement provider download UI
  3. Add provider status indicators
  4. Show active provider in UI

Result: Users can choose and download providers

Phase 5: External Providers

Goal: Enable remote and cloud providers

  1. Implement RemoteProvider (HTTP client)
  2. Implement OpenAIProvider (API wrapper)
  3. Add provider configuration UI (URLs, API keys)
  4. Document external provider API spec

Result: Full provider ecosystem

Provider Versioning

Independent Versioning

Providers have their own version numbers, independent of the main app:

  • App version: v0.2.0 (frequent updates)
  • Provider version: v1.0.0 (rare updates)

Compatibility Matrix

Example:

App Version Min Provider Version Max Provider Version
v0.2.0 v1.0.0 v1.x.x
v0.3.0 v1.0.0 v1.x.x
v0.4.0 v1.2.0 v1.x.x
v1.0.0 v2.0.0 v2.x.x

Backend checks compatibility:

async def check_provider_compatibility(provider_version: str) -> bool:
    """Check if provider version is compatible with current app."""
    min_version = "1.0.0"
    max_version = "1.999.999"
    return min_version <= provider_version < max_version

UI shows warning if incompatible:

⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+

User Flows

First-Time Setup

  1. User downloads and installs Voicebox (~150MB)

  2. App launches → detects no TTS provider installed

  3. Shows setup wizard:

    Choose your TTS provider:

[ ] PyTorch CUDA (2.4GB)    [Download]
    ✓ Fastest on NVIDIA GPUs
    ✗ Requires NVIDIA GPU

[●] PyTorch CPU (300MB)     [Download]
    ✓ Works on any system
    ✗ Slower inference

[ ] MLX (800MB)             [Download]
    ✓ Fast on Apple Silicon
    ✗ macOS only (M1/M2/M3)

[ ] Remote Server
    URL: ___________________

[ ] OpenAI API
    API Key: ________________

  4. User selects provider → downloads with progress bar

  5. Provider installs to AppData/Application Support

  6. App starts provider → ready to use

App Update Flow (No Provider Change)

Scenario: Bug fix in UI, no backend changes

  1. User gets update notification: "Voicebox v0.2.1 available"
  2. Downloads update (~150MB, not 2.4GB!)
  3. Installs and restarts
  4. Provider stays the same (no re-download needed)
  5. App starts using existing provider

User experience: Fast updates, no multi-GB downloads

Provider Update Flow

Scenario: New Qwen3-TTS model version released

  1. User opens Settings → Provider tab
  2. Sees notification: "Provider update available (v1.1.0)"
  3. Clicks "Update Provider"
  4. Downloads new provider binary
  5. Old provider binary is replaced
  6. Restart app to use new provider

Frequency: Rare (only when TTS model/backend changes)

Switching Providers

Scenario: User upgrades to NVIDIA GPU

  1. User goes to Settings → Provider
  2. Selects "PyTorch CUDA"
  3. Clicks "Download" → downloads 2.4GB
  4. Download completes → restarts app
  5. App now uses CUDA provider

Benefits

Benefit Details
GitHub Releases Work Main app ~150MB << 2GB limit
Fast Updates UI/feature updates don't require re-downloading providers
User Choice CPU, CUDA, MLX, OpenAI, remote server
External Provider Support Users can run their own TTS servers
Bandwidth Savings Only download provider once, app updates are small
Future-Proof Easy to add new providers (ElevenLabs, custom models)
Team Deployments Multiple users share one remote provider
Cloud-Ready Works with Modal, Replicate, RunPod, etc.

Open Questions

1. Provider Versioning

Question: Should providers have independent versions or match app version?

Options:

  • A. Independent (providers: v1.x, app: v0.2.x)
  • B. Matched (both use v0.2.x)

Recommendation: Independent versioning with compatibility matrix

2. Auto-Update Providers

Question: Should providers auto-update separately from app?

Options:

  • A. Manual updates only (user clicks "Update Provider")
  • B. Optional auto-update (user can enable)
  • C. Always auto-update

Recommendation: Optional auto-update (default off)

3. Provider Discovery

Question: How does app find installed providers?

Options:

  • A. Check standard paths in AppData/Application Support
  • B. Registry (Windows) / plist (macOS)
  • C. Config file with provider locations

Recommendation: Standard paths + config fallback

4. Fallback Behavior

Question: What if no provider is installed?

Options:

  • A. Show setup wizard on first launch
  • B. Block app until provider installed
  • C. Allow app to run in "demo mode" (transcription only)

Recommendation: Setup wizard on first launch

5. Provider Auto-Start

Question: Should provider start automatically with app?

Options:

  • A. Always start selected provider on app launch
  • B. Start on-demand (when user generates speech)
  • C. User preference

Recommendation: Auto-start (configurable in settings)

Future Enhancements

  • Provider Marketplace: Built-in directory of community providers
  • Multi-Provider Support: Use different providers per voice/language
  • Provider Health Monitoring: Automatic failover if provider crashes
  • Cost Tracking: Monitor API usage for OpenAI/cloud providers
  • Performance Metrics: Latency, throughput, VRAM usage dashboards
  • Docker Providers: Run providers in Docker containers
  • Provider Plugins: Load custom providers from user scripts

Related Documents

Contributing

If you want to build a custom TTS provider:

  1. Implement the provider API spec (see above)
  2. Test with Voicebox locally
  3. Package as executable (PyInstaller, Docker, etc.)
  4. Share in GitHub Discussions

Questions?

Reference in New Issue View Git Blame Copy Permalink