voicebox/providers/README.md

# TTS Provider Architecture

This document explains how Voicebox's modular TTS provider system works.

## Overview

Voicebox uses a **pluggable provider architecture** that separates the main application from TTS inference. This solves several problems:

- **GitHub's 2GB release limit** - CUDA builds are ~2.4GB, too large for GitHub releases
- **Faster app updates** - UI/feature updates don't require re-downloading heavy ML binaries
- **User choice** - Users can pick CPU, CUDA, or external providers based on their hardware

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│  Voicebox App                                               │
│  ├─ UI (React)                                              │
│  ├─ Backend (FastAPI)                                       │
│  │   ├─ Voice Profiles                                      │
│  │   ├─ Generation History                                  │
│  │   ├─ Whisper STT (bundled)                               │
│  │   └─ Provider Manager ◄────────────────┐                 │
│  │                                         │                │
│  └─ providers/                             │                │
│      ├─ bundled.py (wraps backends/)       │                │
│      └─ local.py (HTTP client)─────────────┼───┐            │
│                                            │   │            │
└────────────────────────────────────────────┼───┼────────────┘
                                             │   │
                        ┌────────────────────┘   │
                        │                        │ HTTP
                        ▼                        ▼
              ┌──────────────────┐    ┌──────────────────────┐
              │ backends/        │    │ Standalone Provider  │
              │ (bundled on Mac) │    │ (subprocess)         │
              │                  │    │                      │
              │ - mlx_backend    │    │ - FastAPI server     │
              │ - pytorch_backend│    │ - PyTorch + Qwen-TTS │
              └──────────────────┘    │ - Runs on localhost  │
                                      └──────────────────────┘
```

## Platform Behavior

| Platform              | App Size | TTS Backend     | Provider Download |
| --------------------- | -------- | --------------- | ----------------- |
| macOS (Apple Silicon) | ~300MB   | MLX bundled     | Not needed        |
| macOS (Intel)         | ~300MB   | PyTorch bundled | Not needed        |
| Windows               | ~150MB   | None bundled    | Required          |
| Linux                 | ~150MB   | None bundled    | Required          |

### macOS (Apple Silicon)

- MLX backend is **bundled** in the app
- Works immediately after install
- Uses Metal for GPU acceleration

### macOS (Intel)

- PyTorch backend is **bundled** in the app
- Works immediately after install
- Uses CPU inference

### Windows / Linux

- **No TTS bundled** - keeps app small (~150MB)
- On first use, prompts to download a provider
- Provider options:
  - **PyTorch CPU** (~300MB) - Works on any system
  - **PyTorch CUDA** (~2.4GB) - Fast inference on NVIDIA GPUs

## Directory Structure

```
voicebox/
├── backend/
│   ├── backends/                 # Actual TTS implementations
│   │   ├── __init__.py          # TTSBackend Protocol
│   │   ├── mlx_backend.py       # MLX implementation (macOS)
│   │   └── pytorch_backend.py   # PyTorch implementation
│   │
│   └── providers/               # Provider abstraction layer
│       ├── __init__.py          # ProviderManager
│       ├── base.py              # TTSProvider Protocol
│       ├── bundled.py           # Wraps backends/ for bundled use
│       ├── local.py             # HTTP client for subprocess providers
│       ├── installer.py         # Downloads providers from R2
│       └── types.py             # Shared types
│
└── providers/                   # Standalone provider builds
    ├── pytorch-cpu/
    │   ├── main.py              # FastAPI server
    │   ├── build.py             # PyInstaller build script
    │   └── requirements.txt
    │
    └── pytorch-cuda/
        ├── main.py              # FastAPI server
        │   build.py              # PyInstaller build script
        └── requirements.txt
```

## How Providers Work

### 1. BundledProvider (macOS)

On macOS, the `BundledProvider` directly calls the bundled `backends/` code:

```python
# backend/providers/bundled.py
class BundledProvider:
    def __init__(self):
        self._backend = get_tts_backend()  # MLX or PyTorch

    async def generate(self, text, voice_prompt, ...):
        return await self._backend.generate(text, voice_prompt, ...)
```

### 2. LocalProvider (Windows/Linux)

On Windows/Linux, the `LocalProvider` communicates with a standalone provider via HTTP:

```python
# backend/providers/local.py
class LocalProvider:
    def __init__(self, base_url: str):
        self.base_url = base_url  # e.g., "http://127.0.0.1:8765"

    async def generate(self, text, voice_prompt, ...):
        response = await self.client.post(
            f"{self.base_url}/tts/generate",
            json={"text": text, "voice_prompt": voice_prompt, ...}
        )
        # Decode audio from response
        return audio, sample_rate
```

### 3. Standalone Provider Server

The standalone providers are self-contained FastAPI servers:

```python
# providers/pytorch-cpu/main.py
@app.post("/tts/generate")
async def generate(text: str, voice_prompt: dict, ...):
    audio, sr = await backend.generate(text, voice_prompt, ...)
    return {"audio": base64_encode(audio), "sample_rate": sr}
```

## Provider API Specification

All providers (local or remote) must implement these HTTP endpoints:

### POST /tts/generate

Generate speech from text.

**Request:**

```json
{
	"text": "Hello world!",
	"voice_prompt": {
		/* voice embedding */
	},
	"language": "en",
	"seed": 12345,
	"model_size": "1.7B"
}
```

**Response:**

```json
{
	"audio": "base64-encoded-wav",
	"sample_rate": 24000,
	"duration": 2.5
}
```

### POST /tts/create_voice_prompt

Create voice embedding from reference audio.

**Request:** `multipart/form-data`

- `audio`: Audio file
- `reference_text`: Transcript

**Response:**

```json
{
	"voice_prompt": {
		/* voice embedding */
	},
	"was_cached": false
}
```

### GET /tts/health

Health check.

**Response:**

```json
{
	"status": "healthy",
	"provider": "pytorch-cuda",
	"version": "1.0.0",
	"model": "1.7B",
	"device": "cuda:0"
}
```

### GET /tts/status

Model status.

**Response:**

```json
{
	"model_loaded": true,
	"model_size": "1.7B",
	"available_sizes": ["0.6B", "1.7B"],
	"gpu_available": true,
	"vram_used_mb": 1234
}
```

## Provider Lifecycle

### Startup Flow (Windows/Linux)

```
1. App launches
2. ProviderManager checks for installed providers
3. If none installed:
   └─ Show setup wizard, prompt download
4. If installed:
   ├─ Start provider subprocess on random port
   ├─ Wait for /tts/health to return 200
   └─ Create LocalProvider with that URL
5. Generation requests go through LocalProvider → subprocess
```

### Download Flow

```
1. User clicks "Download PyTorch CUDA"
2. Installer downloads from Cloudflare R2:
   https://downloads.voicebox.sh/providers/v1.0.0/tts-provider-pytorch-cuda-windows.exe
3. Saved to:
   - Windows: %APPDATA%/voicebox/providers/
   - Linux: ~/.local/share/voicebox/providers/
4. Provider is now available to start
```

## Building Providers

### Prerequisites

- Python 3.12
- PyInstaller

### Build PyTorch CPU Provider

```bash
cd providers/pytorch-cpu
pip install -r requirements.txt
python build.py
# Output: dist/tts-provider-pytorch-cpu.exe
```

### Build PyTorch CUDA Provider

```bash
cd providers/pytorch-cuda
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
python build.py
# Output: dist/tts-provider-pytorch-cuda.exe (~2.4GB)
```

## Provider Versioning

Providers have **independent versions** from the app:

- **App version:** `v0.2.0` (frequent updates)
- **Provider version:** `v1.0.0` (rare updates)

Providers only need updates when:

- TTS model changes (new Qwen3-TTS version)
- API spec changes
- Bug fixes in inference code

The app checks provider compatibility on startup.

## Future Providers

The architecture supports additional providers:

- **Remote Server** - Connect to your own TTS server
- **OpenAI API** - Use OpenAI's TTS (requires API key)
- **ElevenLabs** - Cloud TTS service
- **Docker** - Run providers in containers

These would implement the same HTTP API spec.