mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-15 04:40:40 -07:00
Refactor documentation structure and dependencies for migration to Fumadocs
- Updated `.gitignore` to include new build and generated content directories. - Removed outdated Mintlify configuration files and documentation. - Introduced new `MIGRATION.md` to outline the transition from Mintlify to Fumadocs. - Added `mdx-components.tsx` for MDX component configuration and compatibility. - Updated `package.json` and `next.config.mjs` for new dependencies and Next.js configuration. - Created `source.config.ts` for content source configuration. - Added OpenAPI specification in `openapi.json` for API documentation. - Removed legacy files and adjusted project structure to align with Fumadocs conventions.
This commit is contained in:
Jamie Pine
@@ -1,758 +0,0 @@
|
||||
# Docker Deployment Guide
|
||||
|
||||
**Status:** In Development for v0.2.0
|
||||
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
|
||||
|
||||
## Overview
|
||||
|
||||
Docker support makes Voicebox easier to deploy, especially for:
|
||||
|
||||
- **Consistent Environments**: Same setup across dev/staging/prod
|
||||
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
|
||||
- **Server Deployments**: Run on headless Linux servers
|
||||
- **Multi-User Setups**: Isolate instances per user/team
|
||||
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Pre-Built Images (Recommended)
|
||||
|
||||
```bash
|
||||
# CPU-only version
|
||||
docker run -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest
|
||||
|
||||
# NVIDIA GPU version
|
||||
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
|
||||
# AMD GPU version (experimental)
|
||||
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-rocm
|
||||
```
|
||||
|
||||
Then open: `http://localhost:8000`
|
||||
|
||||
### Using Docker Compose (Easiest)
|
||||
|
||||
Create `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Building From Source
|
||||
|
||||
### Basic Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git \
|
||||
build-essential \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy application
|
||||
COPY backend/ /app/backend/
|
||||
COPY requirements.txt /app/
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8000
|
||||
|
||||
# Run server
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build and run:
|
||||
```bash
|
||||
docker build -t voicebox .
|
||||
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Multi-Stage Build (Optimized)
|
||||
|
||||
Smaller image size by separating build and runtime:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.optimized
|
||||
# Stage 1: Build dependencies
|
||||
FROM python:3.11-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git build-essential && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
-r requirements.txt
|
||||
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Stage 2: Runtime
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install only runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy installed packages from builder
|
||||
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
|
||||
|
||||
# Copy application code
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build:
|
||||
```bash
|
||||
docker build -f Dockerfile.optimized -t voicebox:slim .
|
||||
```
|
||||
|
||||
## GPU Support
|
||||
|
||||
### NVIDIA GPUs (CUDA)
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with CUDA support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with GPU:**
|
||||
```bash
|
||||
docker run --gpus all -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
voicebox:cuda
|
||||
```
|
||||
|
||||
**Docker Compose with GPU:**
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
image: voicebox:cuda
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
### AMD GPUs (ROCm) - Experimental
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM rocm/dev-ubuntu-22.04:6.0
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with ROCm support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Set ROCm environment variables
|
||||
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with AMD GPU:**
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video --ipc=host --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-p 8000:8000 -v voicebox-data:/app/data \
|
||||
voicebox:rocm
|
||||
```
|
||||
|
||||
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
|
||||
|
||||
## Volume Mounts
|
||||
|
||||
### Essential Volumes
|
||||
|
||||
```bash
|
||||
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
|
||||
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Development Volume Mounts
|
||||
|
||||
For development with hot-reload:
|
||||
|
||||
```bash
|
||||
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
|
||||
-v voicebox-data:/app/data \
|
||||
-e RELOAD=true \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Custom Model Storage
|
||||
|
||||
Use external model directory:
|
||||
|
||||
```bash
|
||||
docker run -v /path/to/models:/models \
|
||||
-e MODELS_DIR=/models \
|
||||
-v voicebox-data:/app/data \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Configure Voicebox via environment variables:
|
||||
|
||||
```bash
|
||||
docker run -e TTS_MODE=local \
|
||||
-e WHISPER_MODE=openai-api \
|
||||
-e OPENAI_API_KEY=sk-... \
|
||||
-e GPU_MEMORY_FRACTION=0.8 \
|
||||
-e LOG_LEVEL=info \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Available Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
|
||||
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
|
||||
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
|
||||
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
|
||||
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
|
||||
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
|
||||
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
|
||||
| `MODELS_DIR` | `/app/models` | Directory for local models |
|
||||
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
|
||||
| `RELOAD` | `false` | Enable hot-reload for development |
|
||||
|
||||
## Complete Docker Compose Examples
|
||||
|
||||
### Production Deployment
|
||||
|
||||
```yaml
|
||||
# docker-compose.prod.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
container_name: voicebox
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
- GPU_MEMORY_FRACTION=0.8
|
||||
- LOG_LEVEL=info
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
driver: local
|
||||
huggingface-cache:
|
||||
driver: local
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### Development Setup
|
||||
|
||||
```yaml
|
||||
# docker-compose.dev.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./backend:/app/backend:ro
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- RELOAD=true
|
||||
- LOG_LEVEL=debug
|
||||
- TTS_MODE=local
|
||||
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
### Multi-Service Stack
|
||||
|
||||
Full stack with reverse proxy and monitoring:
|
||||
|
||||
```yaml
|
||||
# docker-compose.stack.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# Main Voicebox app
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
# Nginx reverse proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
depends_on:
|
||||
- voicebox
|
||||
|
||||
# Prometheus monitoring (optional)
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
- prometheus-data:/prometheus
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
prometheus-data:
|
||||
```
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
|
||||
2. **Install Docker + nvidia-docker:**
|
||||
```bash
|
||||
# Amazon Linux 2
|
||||
sudo yum install -y docker
|
||||
sudo systemctl start docker
|
||||
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
|
||||
sudo apt-get update && sudo apt-get install -y nvidia-docker2
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
3. **Deploy:**
|
||||
```bash
|
||||
docker run --gpus all -d -p 80:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
```
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
Use GPU Droplet + Docker:
|
||||
|
||||
```bash
|
||||
# Create droplet via CLI
|
||||
doctl compute droplet create voicebox \
|
||||
--size gpu-h100x1-80gb \
|
||||
--image ubuntu-22-04-x64 \
|
||||
--region nyc3
|
||||
|
||||
# SSH and deploy
|
||||
ssh root@<droplet-ip>
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
docker run --gpus all -d -p 80:8000 voicebox:cuda
|
||||
```
|
||||
|
||||
### Google Cloud Run (CPU-only)
|
||||
|
||||
```bash
|
||||
# Build and push
|
||||
docker build -t gcr.io/your-project/voicebox .
|
||||
docker push gcr.io/your-project/voicebox
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy voicebox \
|
||||
--image gcr.io/your-project/voicebox \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--memory 4Gi \
|
||||
--cpu 2 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Fly.io
|
||||
|
||||
Create `fly.toml`:
|
||||
```toml
|
||||
app = "voicebox"
|
||||
|
||||
[build]
|
||||
image = "ghcr.io/jamiepine/voicebox:latest"
|
||||
|
||||
[[services]]
|
||||
http_checks = []
|
||||
internal_port = 8000
|
||||
protocol = "tcp"
|
||||
|
||||
[[services.ports]]
|
||||
port = 80
|
||||
handlers = ["http"]
|
||||
|
||||
[[services.ports]]
|
||||
port = 443
|
||||
handlers = ["tls", "http"]
|
||||
|
||||
[mounts]
|
||||
source = "voicebox_data"
|
||||
destination = "/app/data"
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
fly launch
|
||||
fly deploy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### GPU Not Detected
|
||||
|
||||
**Check NVIDIA Docker:**
|
||||
```bash
|
||||
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
If this fails, reinstall nvidia-docker2.
|
||||
|
||||
**Check AMD ROCm:**
|
||||
```bash
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
|
||||
```
|
||||
|
||||
### Permission Errors
|
||||
|
||||
Container can't write to volumes:
|
||||
```bash
|
||||
# Fix permissions
|
||||
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
Reduce GPU memory usage:
|
||||
```bash
|
||||
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
|
||||
```
|
||||
|
||||
Or use CPU-only:
|
||||
```bash
|
||||
docker run -e DEVICE=cpu voicebox
|
||||
```
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
Ensure HuggingFace cache is writable:
|
||||
```bash
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
Or use host cache:
|
||||
```bash
|
||||
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
Change host port:
|
||||
```bash
|
||||
docker run -p 8080:8000 voicebox # Use port 8080 instead
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Don't Run as Root
|
||||
|
||||
Create non-root user in Dockerfile:
|
||||
```dockerfile
|
||||
RUN useradd -m -u 1000 voicebox
|
||||
USER voicebox
|
||||
```
|
||||
|
||||
### 2. Use Secrets for API Keys
|
||||
|
||||
Don't put API keys in docker-compose.yml:
|
||||
|
||||
```bash
|
||||
# Use Docker secrets
|
||||
echo "sk-your-key" | docker secret create openai_key -
|
||||
|
||||
docker service create \
|
||||
--secret openai_key \
|
||||
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
|
||||
voicebox
|
||||
```
|
||||
|
||||
### 3. Network Isolation
|
||||
|
||||
Use internal networks for multi-container setups:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
networks:
|
||||
- internal
|
||||
nginx:
|
||||
networks:
|
||||
- internal
|
||||
- external
|
||||
ports:
|
||||
- "80:80"
|
||||
|
||||
networks:
|
||||
internal:
|
||||
internal: true
|
||||
external:
|
||||
```
|
||||
|
||||
### 4. Resource Limits
|
||||
|
||||
Prevent resource exhaustion:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '4'
|
||||
memory: 8G
|
||||
reservations:
|
||||
cpus: '2'
|
||||
memory: 4G
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### GPU Memory Management
|
||||
|
||||
```bash
|
||||
# Use 80% of GPU (default 90%)
|
||||
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
|
||||
|
||||
# Allow GPU memory growth (prevents OOM)
|
||||
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
|
||||
```
|
||||
|
||||
### Model Caching
|
||||
|
||||
Pre-download models to volume:
|
||||
|
||||
```bash
|
||||
# Download models first
|
||||
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
|
||||
voicebox python -c "
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
WhisperProcessor.from_pretrained('openai/whisper-base')
|
||||
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
|
||||
"
|
||||
|
||||
# Then run normally
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Multi-Worker Setup
|
||||
|
||||
Use uvicorn workers for better throughput:
|
||||
|
||||
```dockerfile
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
Built-in health endpoint:
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
Docker health check:
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Add metrics exporter:
|
||||
```python
|
||||
# backend/main.py
|
||||
from prometheus_fastapi_instrumentator import Instrumentator
|
||||
|
||||
Instrumentator().instrument(app).expose(app)
|
||||
```
|
||||
|
||||
Then scrape `/metrics` with Prometheus.
|
||||
|
||||
### Logs
|
||||
|
||||
View container logs:
|
||||
```bash
|
||||
docker logs -f voicebox
|
||||
|
||||
# Or with compose
|
||||
docker compose logs -f voicebox
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] Publish official images to GitHub Container Registry
|
||||
- [ ] Add Kubernetes Helm charts
|
||||
- [ ] Create Docker Desktop extension
|
||||
- [ ] Add automated vulnerability scanning
|
||||
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
|
||||
|
||||
## Contributing
|
||||
|
||||
Help improve Docker support:
|
||||
1. Test on different platforms (AMD GPU, ARM64, etc.)
|
||||
2. Submit Dockerfile optimizations
|
||||
3. Share deployment configurations
|
||||
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
|
||||
|
||||
## Resources
|
||||
|
||||
- [Docker Documentation](https://docs.docker.com)
|
||||
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
|
||||
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
|
||||
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
|
||||
@@ -1,435 +0,0 @@
|
||||
# External Provider Support
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...)
|
||||
|
||||
## Overview
|
||||
|
||||
External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for:
|
||||
|
||||
- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server
|
||||
- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI
|
||||
- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc.
|
||||
- **Team Sharing**: Multiple users share one GPU server running models
|
||||
- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ HTTP/API ┌──────────────────┐
|
||||
│ Voicebox UI │ ───────────────────────> │ Your TTS Server │
|
||||
│ + Backend │ │ (Qwen3-TTS on │
|
||||
│ │ <─────────────────────── │ AMD/NVIDIA GPU)│
|
||||
│ - Profiles │ Audio + Metadata └──────────────────┘
|
||||
│ - History │
|
||||
│ - Audio Edit │ HTTP/API ┌──────────────────┐
|
||||
│ - UI │ ───────────────────────> │ Whisper Service │
|
||||
└─────────────────┘ │ (OpenAI API or │
|
||||
│ self-hosted) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**What Voicebox Still Handles:**
|
||||
- Voice profile management
|
||||
- Generation history
|
||||
- Audio trimming/editing
|
||||
- Multi-track story editor
|
||||
- UI/UX layer
|
||||
|
||||
**What External Providers Handle:**
|
||||
- Model inference (TTS generation, transcription)
|
||||
- GPU allocation
|
||||
- Model loading/caching
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# TTS Provider
|
||||
TTS_MODE=remote # local | remote
|
||||
TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL
|
||||
TTS_API_KEY=your-api-key # Optional authentication
|
||||
|
||||
# Whisper Provider
|
||||
WHISPER_MODE=openai-api # local | openai-api | remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper
|
||||
OPENAI_API_KEY=sk-... # For OpenAI Whisper API
|
||||
```
|
||||
|
||||
### Voicebox Config UI (Planned)
|
||||
|
||||
Settings page will include:
|
||||
- Provider selection dropdowns
|
||||
- URL/API key inputs
|
||||
- Connection test button
|
||||
- Latency/status indicators
|
||||
|
||||
## Hosting External Services
|
||||
|
||||
### Option 1: Simple FastAPI Server (Recommended)
|
||||
|
||||
Create a lightweight server to expose your local Qwen3-TTS model:
|
||||
|
||||
```python
|
||||
# tts_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
import numpy as np
|
||||
import base64
|
||||
|
||||
app = FastAPI()
|
||||
model = Qwen3TTSModel.from_pretrained(
|
||||
"Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm
|
||||
)
|
||||
|
||||
@app.post("/v1/generate")
|
||||
async def generate(
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: int = None
|
||||
):
|
||||
"""Generate speech from text using voice prompt."""
|
||||
audio, sample_rate = model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
)
|
||||
|
||||
# Return as base64 for transport
|
||||
audio_bytes = audio.tobytes()
|
||||
return {
|
||||
"audio": base64.b64encode(audio_bytes).decode(),
|
||||
"sample_rate": sample_rate,
|
||||
"dtype": str(audio.dtype)
|
||||
}
|
||||
|
||||
@app.post("/v1/create_voice_prompt")
|
||||
async def create_voice_prompt(
|
||||
audio: UploadFile = File(...),
|
||||
reference_text: str = ""
|
||||
):
|
||||
"""Create voice prompt from reference audio."""
|
||||
# Save uploaded audio temporarily
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
# Create voice prompt
|
||||
voice_prompt = model.create_voice_clone_prompt(
|
||||
ref_audio=audio_path,
|
||||
ref_text=reference_text,
|
||||
)
|
||||
|
||||
return {"voice_prompt": voice_prompt}
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": str(model.device)
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
```
|
||||
|
||||
**Run it:**
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install fastapi uvicorn qwen-tts torch
|
||||
|
||||
# For AMD GPUs, use ROCm PyTorch:
|
||||
pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
|
||||
|
||||
# Start server
|
||||
python tts_server.py
|
||||
```
|
||||
|
||||
### Option 2: vLLM (If Supported)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
### Option 3: Cloud Platforms
|
||||
|
||||
**Modal.com Example:**
|
||||
```python
|
||||
import modal
|
||||
|
||||
app = modal.App("qwen-tts")
|
||||
image = modal.Image.debian_slim().pip_install("qwen-tts", "torch")
|
||||
|
||||
@app.function(gpu="A10G", image=image)
|
||||
@modal.web_endpoint(method="POST")
|
||||
def generate(text: str, voice_prompt: dict):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
|
||||
audio, sr = model.generate_voice_clone(text, voice_prompt)
|
||||
return {"audio": audio.tolist(), "sample_rate": sr}
|
||||
```
|
||||
|
||||
Deploy: `modal deploy tts_server.py`
|
||||
Get URL: `https://yourapp--generate.modal.run`
|
||||
|
||||
## API Specification
|
||||
|
||||
External TTS providers must implement these endpoints:
|
||||
|
||||
### `POST /v1/generate`
|
||||
|
||||
Generate speech from text.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"text": "Hello, this is a test.",
|
||||
"voice_prompt": { /* voice prompt object */ },
|
||||
"language": "en",
|
||||
"seed": 12345
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"audio": "base64-encoded-audio-bytes",
|
||||
"sample_rate": 24000,
|
||||
"dtype": "float32"
|
||||
}
|
||||
```
|
||||
|
||||
### `POST /v1/create_voice_prompt`
|
||||
|
||||
Create a voice prompt from reference audio.
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
- `audio`: Audio file upload
|
||||
- `reference_text`: Transcript of the audio
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"voice_prompt": { /* voice prompt object */ }
|
||||
}
|
||||
```
|
||||
|
||||
### `GET /health`
|
||||
|
||||
Health check endpoint.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": "cuda:0"
|
||||
}
|
||||
```
|
||||
|
||||
## Whisper External Providers
|
||||
|
||||
### OpenAI Whisper API
|
||||
|
||||
Simply set:
|
||||
```bash
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
Voicebox will use OpenAI's Whisper API automatically.
|
||||
|
||||
### Self-Hosted Whisper
|
||||
|
||||
Run your own Whisper server:
|
||||
|
||||
```python
|
||||
# whisper_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
import librosa
|
||||
|
||||
app = FastAPI()
|
||||
processor = WhisperProcessor.from_pretrained("openai/whisper-base")
|
||||
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
|
||||
|
||||
@app.post("/v1/transcribe")
|
||||
async def transcribe(audio: UploadFile = File(...), language: str = None):
|
||||
# Load audio
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
audio_data, sr = librosa.load(audio_path, sr=16000)
|
||||
|
||||
# Process
|
||||
inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt")
|
||||
predicted_ids = model.generate(inputs["input_features"])
|
||||
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
|
||||
|
||||
return {"text": transcription}
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
```bash
|
||||
WHISPER_MODE=remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. AMD GPU User with Existing Setup
|
||||
|
||||
**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux.
|
||||
|
||||
**Setup:**
|
||||
1. Run `tts_server.py` on your AMD box (ROCm PyTorch)
|
||||
2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000`
|
||||
3. Use Voicebox UI for profiles, generation, editing
|
||||
4. TTS happens on your AMD GPU
|
||||
|
||||
### 2. Team Deployment
|
||||
|
||||
**Scenario:** 5 team members, 1 GPU server.
|
||||
|
||||
**Setup:**
|
||||
1. Deploy TTS server on shared GPU box
|
||||
2. Each person runs Voicebox desktop app locally
|
||||
3. All point to same `TTS_REMOTE_URL`
|
||||
4. Profiles and history stay local per user
|
||||
5. GPU usage is shared
|
||||
|
||||
### 3. Hybrid Local/Remote
|
||||
|
||||
**Scenario:** Fast local Whisper, heavy TTS on cloud.
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
TTS_MODE=remote
|
||||
TTS_REMOTE_URL=https://your-modal-app.modal.run
|
||||
|
||||
WHISPER_MODE=local # Fast transcription on your CPU
|
||||
```
|
||||
|
||||
### 4. OpenAI Whisper + Self-Hosted TTS
|
||||
|
||||
**Scenario:** Use OpenAI's API for transcription, run TTS locally.
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
TTS_MODE=local
|
||||
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Authentication
|
||||
|
||||
Add API key authentication to your external server:
|
||||
|
||||
```python
|
||||
from fastapi import Header, HTTPException
|
||||
|
||||
API_KEY = "your-secret-key"
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(...)):
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
|
||||
@app.post("/v1/generate", dependencies=[Depends(verify_api_key)])
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
```bash
|
||||
TTS_API_KEY=your-secret-key
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- **VPN/Tailscale**: Use private network for remote servers
|
||||
- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates
|
||||
- **Firewall**: Restrict access to known IPs
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Protect your external server:
|
||||
|
||||
```python
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
|
||||
@app.post("/v1/generate")
|
||||
@limiter.limit("10/minute")
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Latency
|
||||
|
||||
External providers add network latency:
|
||||
- **Local network**: ~10-50ms overhead (negligible)
|
||||
- **Same datacenter**: ~1-5ms overhead
|
||||
- **Cross-region cloud**: 50-200ms+ overhead
|
||||
|
||||
For real-time applications, keep TTS server on local network or same cloud region.
|
||||
|
||||
### Caching
|
||||
|
||||
Implement response caching on external server:
|
||||
|
||||
```python
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=1000)
|
||||
def get_cached_generation(text, voice_prompt_hash, language, seed):
|
||||
return model.generate_voice_clone(text, voice_prompt)
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
For high-traffic deployments, run multiple TTS servers behind a load balancer:
|
||||
|
||||
```
|
||||
Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1)
|
||||
├──> TTS Server 2 (GPU 2)
|
||||
└──> TTS Server 3 (GPU 3)
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] **Provider Marketplace**: Built-in directory of compatible providers
|
||||
- [ ] **Automatic Fallback**: If remote fails, fallback to local
|
||||
- [ ] **Cost Tracking**: Monitor API usage and costs
|
||||
- [ ] **Performance Metrics**: Latency, throughput dashboards
|
||||
- [ ] **Multi-Provider**: Use different providers for different voices/languages
|
||||
|
||||
## Contributing
|
||||
|
||||
If you build an external provider, please share:
|
||||
1. Server implementation
|
||||
2. Performance benchmarks
|
||||
3. Deployment guide
|
||||
|
||||
Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions)
|
||||
|
||||
## Questions?
|
||||
|
||||
- **Discord**: [Join the community](https://discord.gg/...)
|
||||
- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues)
|
||||
- **Docs**: [Full documentation](https://voicebox.sh/docs)
|
||||
@@ -1,396 +0,0 @@
|
||||
# MLX Audio Integration
|
||||
|
||||
**Status:** Validated ✅
|
||||
**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio)
|
||||
|
||||
## Validation Results
|
||||
|
||||
We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings:
|
||||
|
||||
| Metric | Result |
|
||||
|--------|--------|
|
||||
| MLX Version | 0.30.4 |
|
||||
| Model Load Time | ~1s (after initial download) |
|
||||
| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) |
|
||||
| Test Hardware | Apple Silicon Mac |
|
||||
|
||||
### Model Mapping
|
||||
|
||||
| voicebox (PyTorch) | mlx-audio (MLX) |
|
||||
|--------------------|-----------------|
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` |
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) |
|
||||
|
||||
### mlx-audio API
|
||||
|
||||
The API uses a **generator-based streaming pattern**:
|
||||
|
||||
```python
|
||||
from mlx_audio.tts import load
|
||||
|
||||
model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16")
|
||||
|
||||
# generate() yields GenerationResult objects
|
||||
for result in model.generate("Hello world"):
|
||||
audio = result.audio # numpy array of samples
|
||||
sample_rate = result.sample_rate # 24000
|
||||
rtf = result.real_time_factor # e.g., 0.55
|
||||
```
|
||||
|
||||
### Known Warnings (harmless)
|
||||
|
||||
```
|
||||
You are using a model of type qwen3_tts to instantiate a model of type .
|
||||
The tokenizer you are loading... with an incorrect regex pattern...
|
||||
```
|
||||
|
||||
These warnings appear but don't affect functionality or output quality.
|
||||
|
||||
### Demo Script
|
||||
|
||||
Run `mlx-test/demo.py` to test:
|
||||
```bash
|
||||
cd mlx-test && source venv/bin/activate && python demo.py "Your text here"
|
||||
```
|
||||
|
||||
## Problem
|
||||
|
||||
Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs.
|
||||
|
||||
This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads.
|
||||
|
||||
## Solution
|
||||
|
||||
Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use.
|
||||
|
||||
**Key wins:**
|
||||
- Native GPU acceleration on Apple Silicon (no more CPU fallback)
|
||||
- Streaming TTS support (faster perceived latency)
|
||||
- Memory optimizations (run larger models on less RAM)
|
||||
- Fixed 0.6B silence bug that we currently ship
|
||||
- Same Qwen3-TTS models (zero migration cost for users)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Current Stack
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch + Qwen3-TTS │
|
||||
│ (CPU only on macOS) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### Proposed Stack
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Platform Detection at Runtime │
|
||||
└─────────────────────────────────────────┘
|
||||
│
|
||||
├─── Apple Silicon (aarch64-darwin)
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ MLX Audio Backend │
|
||||
│ │ - Qwen3-TTS (mlx) │
|
||||
│ │ - Whisper (mlx) │
|
||||
│ │ - Streaming support │
|
||||
│ └─────────────────────────┘
|
||||
│
|
||||
└─── Other (x86_64, Windows, Linux)
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch Backend │
|
||||
│ - Qwen3-TTS (pytorch) │
|
||||
│ - Whisper (pytorch) │
|
||||
│ - CUDA if available │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Platform Detection & Dependency Management
|
||||
|
||||
Create a backend that switches between PyTorch and MLX based on runtime platform detection.
|
||||
|
||||
**New files:**
|
||||
- `backend/platform.py` - Detect Apple Silicon, return backend type
|
||||
- `backend/backends/__init__.py` - Backend factory pattern
|
||||
- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only)
|
||||
|
||||
**Modified files:**
|
||||
- `backend/requirements.txt` - Keep PyTorch as default
|
||||
- `backend/main.py` - Import from backend factory instead of direct imports
|
||||
|
||||
**Platform detection logic:**
|
||||
```python
|
||||
def get_backend_type() -> str:
|
||||
"""Detect best backend for current platform."""
|
||||
if platform.system() == "Darwin" and platform.machine() == "arm64":
|
||||
# Apple Silicon detected
|
||||
try:
|
||||
import mlx
|
||||
return "mlx"
|
||||
except ImportError:
|
||||
return "pytorch" # Fallback if mlx not installed
|
||||
return "pytorch"
|
||||
```
|
||||
|
||||
### Phase 2: MLX Backend Implementation
|
||||
|
||||
Create parallel implementations of TTS and STT using mlx-audio.
|
||||
|
||||
**New files:**
|
||||
- `backend/backends/mlx_backend.py` - MLX inference engine
|
||||
- `backend/backends/pytorch_backend.py` - Refactor current code into backend
|
||||
|
||||
**Interface both backends must implement:**
|
||||
```python
|
||||
class TTSBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ...
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ...
|
||||
async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ...
|
||||
def unload_model(self) -> None: ...
|
||||
|
||||
class STTBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ...
|
||||
def unload_model(self) -> None: ...
|
||||
```
|
||||
|
||||
**MLX backend implementation notes:**
|
||||
|
||||
mlx-audio's `generate()` returns a generator by default (streaming is built-in):
|
||||
|
||||
```python
|
||||
# MLX backend wrapper
|
||||
from mlx_audio.tts import load
|
||||
|
||||
class MLXTTSBackend:
|
||||
def __init__(self):
|
||||
self.model = None
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
model_map = {
|
||||
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
|
||||
# "0.6B": needs conversion to mlx format
|
||||
}
|
||||
self.model = load(model_map[model_size])
|
||||
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]:
|
||||
# Collect all chunks from generator
|
||||
chunks = []
|
||||
for result in self.model.generate(text): # TODO: add voice_prompt support
|
||||
chunks.append(np.array(result.audio))
|
||||
return np.concatenate(chunks), 24000
|
||||
```
|
||||
|
||||
**MLX-specific features to expose:**
|
||||
- Streaming TTS (new endpoint: `/api/generate/stream`)
|
||||
- Memory-optimized model loading
|
||||
- Qwen3-ASR for transcription (in addition to Whisper)
|
||||
|
||||
### Phase 3: API Layer Updates
|
||||
|
||||
Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility.
|
||||
|
||||
**Modified files:**
|
||||
- `backend/main.py` - Add streaming endpoints
|
||||
- `backend/tts.py` - Refactor to use backend abstraction
|
||||
- `backend/transcribe.py` - Refactor to use backend abstraction
|
||||
|
||||
**New endpoints:**
|
||||
```python
|
||||
@app.post("/api/generate/stream")
|
||||
async def generate_stream(...) -> StreamingResponse:
|
||||
"""Stream TTS chunks as they're generated (MLX only)."""
|
||||
backend = get_backend()
|
||||
if not hasattr(backend, 'generate_streaming'):
|
||||
raise HTTPException(501, "Streaming not supported on this backend")
|
||||
return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav")
|
||||
```
|
||||
|
||||
**Backward compatibility:**
|
||||
- Keep all existing `/api/generate` endpoints unchanged
|
||||
- PyTorch backend users see no behavior change
|
||||
- MLX users automatically get faster inference, streaming is opt-in
|
||||
|
||||
### Phase 4: Frontend Integration
|
||||
|
||||
Add UI indicators for backend type and streaming progress.
|
||||
|
||||
**Modified files:**
|
||||
- `app/src/hooks/useGenerationForm.tsx` - Add streaming support
|
||||
- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle
|
||||
- `app/src/lib/api.ts` - Add streaming API client
|
||||
|
||||
**UI additions:**
|
||||
- Badge showing current backend ("MLX" or "PyTorch")
|
||||
- Toggle for streaming mode (disabled if PyTorch)
|
||||
- Real-time streaming playback (WaveSurfer progressive loading)
|
||||
|
||||
### Phase 5: Build & Distribution
|
||||
|
||||
Create separate installers for MLX (Apple Silicon) and PyTorch (Universal).
|
||||
|
||||
**Modified files:**
|
||||
- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds
|
||||
- `.github/workflows/release.yml` - Build both variants
|
||||
|
||||
**Build matrix:**
|
||||
```yaml
|
||||
- target: aarch64-apple-darwin
|
||||
backend: mlx
|
||||
installer: voicebox-macos-silicon-{version}.dmg
|
||||
|
||||
- target: x86_64-apple-darwin
|
||||
backend: pytorch
|
||||
installer: voicebox-macos-intel-{version}.dmg
|
||||
|
||||
- target: x86_64-pc-windows-msvc
|
||||
backend: pytorch
|
||||
installer: voicebox-windows-{version}.exe
|
||||
```
|
||||
|
||||
**Installation flow:**
|
||||
- Auto-detect architecture, recommend correct installer
|
||||
- MLX installer includes `mlx-audio` in embedded Python
|
||||
- PyTorch installer includes `torch` in embedded Python
|
||||
- Both can coexist (different backend, same profile format)
|
||||
|
||||
### Phase 6: Testing & Validation
|
||||
|
||||
Ensure both backends produce compatible outputs.
|
||||
|
||||
**New files:**
|
||||
- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio
|
||||
- `backend/tests/test_streaming.py` - Streaming-specific tests
|
||||
|
||||
**Test scenarios:**
|
||||
- Same voice prompt on both backends → similar (not identical) audio output
|
||||
- Profile created on MLX → loads on PyTorch (and vice versa)
|
||||
- Streaming chunks assemble into valid WAV file
|
||||
- Model downloads work on both backends
|
||||
- Memory usage stays within bounds
|
||||
|
||||
### Phase 7: Documentation
|
||||
|
||||
Update user-facing docs and developer guides.
|
||||
|
||||
**New files:**
|
||||
- `docs/developer/BACKENDS.md` - Guide for adding new backends
|
||||
- `docs/overview/performance.md` - Backend comparison benchmarks
|
||||
|
||||
**Modified files:**
|
||||
- `README.md` - Note Apple Silicon acceleration
|
||||
- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues
|
||||
|
||||
**Key docs to write:**
|
||||
- Which installer to download (architecture detection)
|
||||
- Performance comparison (MLX vs PyTorch on same M2 hardware)
|
||||
- How streaming mode works
|
||||
- How to force PyTorch on Apple Silicon (for debugging)
|
||||
|
||||
## Technical Decisions
|
||||
|
||||
### Why Dual Backend Instead of MLX-Only?
|
||||
|
||||
**Pros of dual backend:**
|
||||
- Windows and Intel Mac users unaffected
|
||||
- Easier testing (can compare outputs)
|
||||
- Fallback if MLX has issues
|
||||
|
||||
**Cons of dual backend:**
|
||||
- More code to maintain
|
||||
- Two dependency trees
|
||||
- Build complexity (separate installers)
|
||||
|
||||
**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback.
|
||||
|
||||
### Why Separate Installers Instead of Runtime Detection?
|
||||
|
||||
**Pros of separate installers:**
|
||||
- Smaller bundle size (don't ship both PyTorch and MLX)
|
||||
- Clearer to users which version they have
|
||||
- Easier to debug (no "which backend am I running?" confusion)
|
||||
- Can optimize each build for its target
|
||||
|
||||
**Cons:**
|
||||
- More installers to build and test
|
||||
- Users might download the wrong one
|
||||
|
||||
**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page.
|
||||
|
||||
### Streaming vs Batch Generation
|
||||
|
||||
MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be:
|
||||
1. MLX-only feature (✅ chosen)
|
||||
2. Implemented for both (lots of work)
|
||||
3. Not exposed at all (wasted opportunity)
|
||||
|
||||
**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend).
|
||||
|
||||
## Migration Path
|
||||
|
||||
Nothing needs migrating, macos users will just notice a speed-boost in inference
|
||||
|
||||
**Data format compatibility:**
|
||||
- Profiles (SQLite) → no schema changes needed
|
||||
- Voice prompts (cached) → backend-agnostic (just numpy arrays)
|
||||
- Audio files → unchanged
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
### Measured Results (from validation)
|
||||
|
||||
| Metric | MLX (measured) | PyTorch CPU (estimated) |
|
||||
|--------|----------------|-------------------------|
|
||||
| **6s audio generation** | ~3-4s | ~10-15s |
|
||||
| **Real-time factor** | 0.5-0.6x | 2-3x |
|
||||
| **Model load (cached)** | ~1s | ~3-5s |
|
||||
|
||||
### TTS Generation (1.7B model, ~20s output)
|
||||
- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time)
|
||||
- **MLX (M2 Max):** ~8-12s (faster than real-time)
|
||||
- **Improvement:** ~4-5x faster
|
||||
|
||||
### Whisper Transcription (10s audio clip)
|
||||
- **PyTorch CPU:** ~5-8s
|
||||
- **MLX:** ~1-2s
|
||||
- **Improvement:** ~3-4x faster
|
||||
|
||||
### Memory Usage (1.7B model)
|
||||
- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM)
|
||||
- **MLX:** ~4-6GB (unified memory, better optimization)
|
||||
- **Improvement:** ~40% less RAM
|
||||
|
||||
Full benchmarks will be in `docs/overview/performance.md` after Phase 6.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure
|
||||
- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No
|
||||
- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing
|
||||
- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
How we'll know this worked:
|
||||
|
||||
1. **Performance:** Apple Silicon users report generation faster than real-time
|
||||
2. **Adoption:** >80% of macOS downloads are MLX build within 1 month
|
||||
3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions)
|
||||
4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance
|
||||
|
||||
## Related Work
|
||||
|
||||
- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly
|
||||
- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API
|
||||
- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/`
|
||||
2. Get approval on dual-backend architecture
|
||||
3. Start Phase 1 (platform detection)
|
||||
|
||||
## Questions?
|
||||
|
||||
Feedback welcome in GitHub discussions or Discord.
|
||||
@@ -1,235 +0,0 @@
|
||||
# OpenAI API Compatibility
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
|
||||
**Issue:** [#10 OpenAI API compatibility](https://github.com/jamiepine/voicebox/issues/10)
|
||||
|
||||
## Overview
|
||||
|
||||
This feature exposes OpenAI-compatible endpoints from Voicebox, allowing any tool, library, or application that speaks the OpenAI Audio API to use Voicebox as a drop-in local replacement.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph clients [External Clients]
|
||||
SDK[OpenAI SDK]
|
||||
Curl[curl / HTTP]
|
||||
Apps[Third-party Apps]
|
||||
end
|
||||
|
||||
subgraph voicebox [Voicebox Server]
|
||||
OpenAI["/v1/audio/* endpoints"]
|
||||
TTS[TTSModel]
|
||||
Whisper[WhisperModel]
|
||||
Profiles[Voice Profiles]
|
||||
end
|
||||
|
||||
SDK --> OpenAI
|
||||
Curl --> OpenAI
|
||||
Apps --> OpenAI
|
||||
OpenAI --> TTS
|
||||
OpenAI --> Whisper
|
||||
OpenAI --> Profiles
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **OpenAI SDK users**: `openai.audio.speech.create()` works with Voicebox
|
||||
- **LLM frameworks**: LangChain, AutoGen, etc. can use Voicebox for TTS
|
||||
- **Shell scripts**: `curl` commands copy-pasted from OpenAI docs work
|
||||
- **Existing integrations**: Any tool expecting OpenAI's API works without code changes
|
||||
|
||||
## Endpoints to Implement
|
||||
|
||||
### 1. `POST /v1/audio/speech` (TTS)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createSpeech
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "tts-1",
|
||||
"input": "Hello world!",
|
||||
"voice": "alloy",
|
||||
"response_format": "mp3",
|
||||
"speed": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
**Response:** Audio file (mp3, wav, opus, aac, flac, pcm)
|
||||
|
||||
**Voice Mapping Strategy:**
|
||||
|
||||
- `voice` parameter maps to Voicebox profile names (case-insensitive)
|
||||
- If no match, use a configurable default profile
|
||||
- Support special syntax: `voice: "profile:uuid"` for explicit profile ID
|
||||
|
||||
### 2. `POST /v1/audio/transcriptions` (Whisper)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createTranscription
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `file`: Audio file
|
||||
- `model`: "whisper-1"
|
||||
- `language`: Optional language hint
|
||||
- `response_format`: json, text, srt, verbose_json, vtt
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello world!"
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### New File: `backend/openai_compat.py`
|
||||
|
||||
Create a dedicated module with an APIRouter for OpenAI-compatible endpoints:
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Literal, Optional
|
||||
|
||||
router = APIRouter(prefix="/v1/audio", tags=["OpenAI Compatible"])
|
||||
|
||||
class SpeechRequest(BaseModel):
|
||||
model: str = "tts-1"
|
||||
input: str
|
||||
voice: str = "alloy"
|
||||
response_format: Literal["mp3", "wav", "opus", "aac", "flac", "pcm"] = "mp3"
|
||||
speed: float = 1.0
|
||||
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest, db: Session = Depends(get_db)):
|
||||
# 1. Map voice name to profile
|
||||
# 2. Generate audio using existing TTSModel
|
||||
# 3. Convert to requested format
|
||||
# 4. Return audio stream
|
||||
...
|
||||
|
||||
@router.post("/transcriptions")
|
||||
async def create_transcription(
|
||||
file: UploadFile = File(...),
|
||||
model: str = Form("whisper-1"),
|
||||
language: Optional[str] = Form(None),
|
||||
response_format: str = Form("json"),
|
||||
):
|
||||
# 1. Save uploaded file
|
||||
# 2. Transcribe using existing WhisperModel
|
||||
# 3. Return in requested format
|
||||
...
|
||||
```
|
||||
|
||||
### Voice Profile Resolution
|
||||
|
||||
Add helper in [backend/profiles.py](backend/profiles.py):
|
||||
|
||||
```python
|
||||
async def resolve_voice_for_openai(voice: str, db: Session) -> Optional[VoiceProfile]:
|
||||
"""
|
||||
Resolve OpenAI voice parameter to a Voicebox profile.
|
||||
|
||||
Priority:
|
||||
1. Exact profile name match (case-insensitive)
|
||||
2. Profile ID match (if voice starts with "profile:")
|
||||
3. Default profile from config
|
||||
4. First available profile
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### Audio Format Conversion
|
||||
|
||||
Add conversion utilities in [backend/utils/audio.py](backend/utils/audio.py):
|
||||
|
||||
```python
|
||||
def convert_audio_format(
|
||||
audio: np.ndarray,
|
||||
sample_rate: int,
|
||||
target_format: str, # mp3, wav, opus, aac, flac, pcm
|
||||
) -> bytes:
|
||||
"""Convert audio to target format using ffmpeg or pydub."""
|
||||
...
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to [backend/config.py](backend/config.py):
|
||||
|
||||
```python
|
||||
# OpenAI API Compatibility
|
||||
OPENAI_COMPAT_ENABLED = True
|
||||
OPENAI_COMPAT_DEFAULT_VOICE = None # Profile ID or name for default voice
|
||||
OPENAI_COMPAT_REQUIRE_AUTH = False # Require API key validation
|
||||
OPENAI_COMPAT_API_KEY = None # If set, validate against this
|
||||
```
|
||||
|
||||
### Integration with main.py
|
||||
|
||||
In [backend/main.py](backend/main.py), include the router:
|
||||
|
||||
```python
|
||||
from . import openai_compat
|
||||
|
||||
# Add OpenAI-compatible routes
|
||||
if config.OPENAI_COMPAT_ENABLED:
|
||||
app.include_router(openai_compat.router)
|
||||
```
|
||||
|
||||
## Streaming Support (Future Enhancement)
|
||||
|
||||
Initial implementation returns complete audio. Streaming can be added later:
|
||||
|
||||
```python
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest):
|
||||
if request.stream:
|
||||
return StreamingResponse(
|
||||
generate_audio_chunks(request),
|
||||
media_type=f"audio/{request.response_format}"
|
||||
)
|
||||
...
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Example usage after implementation:
|
||||
|
||||
```bash
|
||||
# TTS with curl
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Hello!", "voice": "MyProfile"}' \
|
||||
--output speech.mp3
|
||||
|
||||
# With OpenAI Python SDK
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.audio.speech.create(
|
||||
model="tts-1",
|
||||
voice="MyProfile",
|
||||
input="Hello world!"
|
||||
)
|
||||
response.stream_to_file("output.mp3")
|
||||
|
||||
# Transcription
|
||||
curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F file=@audio.mp3 \
|
||||
-F model="whisper-1"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Optional API key validation (for shared deployments)
|
||||
- Rate limiting on endpoints
|
||||
- Input length limits (same as existing `/generate` endpoint)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `pydub` or `ffmpeg-python` for audio format conversion (mp3, opus, etc.)
|
||||
- No changes to existing TTS/Whisper model code
|
||||
@@ -1,964 +0,0 @@
|
||||
# TTS Provider Architecture
|
||||
|
||||
**Status:** Planned for v0.1.13
|
||||
**Created:** 2025-01-31
|
||||
**Problem:** GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Split the monolithic backend into modular components:
|
||||
|
||||
1. **Main App** (~150-200MB): Tauri + FastAPI backend + Whisper + UI/profiles/history
|
||||
2. **TTS Providers** (downloadable plugins): Separate executables for model inference
|
||||
|
||||
This architecture solves:
|
||||
|
||||
- ✅ GitHub 2GB release artifact limit
|
||||
- ✅ Frequent app updates without re-downloading large python binaries
|
||||
- ✅ User choice of compute backend (CPU/GPU/Cloud)
|
||||
- ✅ External provider support (OpenAI, custom servers)
|
||||
- ✅ Future extensibility
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Voicebox App (Tauri + Backend) ~150MB │
|
||||
│ ├─ UI Layer (React) │
|
||||
│ ├─ Backend (FastAPI) │
|
||||
│ │ ├─ Voice Profiles │
|
||||
│ │ ├─ Generation History │
|
||||
│ │ ├─ Audio Editing / Stories │
|
||||
│ │ └─ Provider Manager ◄──────────────┐ │
|
||||
│ └─ Whisper (bundled, tiny ~50MB) │ │
|
||||
└─────────────────────────────────────────┼────────────────┘
|
||||
│
|
||||
HTTP/IPC │
|
||||
│
|
||||
┌────────────────────────────────┼─────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐
|
||||
│ TTS Provider: │ │ TTS Provider: │ │ TTS Provider: │
|
||||
│ PyTorch CPU │ │ PyTorch CUDA │ │ MLX (Apple) │
|
||||
│ │ │ │ │ │
|
||||
│ ~300MB │ │ ~2.4GB │ │ ~800MB │
|
||||
│ │ │ │ │ │
|
||||
│ Local inference │ │ GPU inference │ │ Metal inference │
|
||||
└─────────────────┘ └─────────────────┘ └──────────────────┘
|
||||
│ │ │
|
||||
└────────────────────────┴─────────────────────┘
|
||||
│
|
||||
┌─────────────▼──────────────┐
|
||||
│ Future Providers: │
|
||||
│ • Remote Server │
|
||||
│ • OpenAI API │
|
||||
│ • ElevenLabs │
|
||||
│ • Custom Docker Container │
|
||||
└────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current Architecture Issues
|
||||
|
||||
**Monolithic Binary:**
|
||||
|
||||
- CPU version: ~295MB
|
||||
- CUDA version: ~2.37GB
|
||||
- GitHub releases: 2GB file size limit (BLOCKED)
|
||||
- Updates require re-downloading entire binary
|
||||
- Poor UX: update app → restart → download CUDA update → restart again
|
||||
|
||||
**User Pain Points:**
|
||||
|
||||
1. Cannot release CUDA version on GitHub (over 2GB)
|
||||
2. Every app update forces 2.4GB re-download for GPU users
|
||||
3. No flexibility (can't use OpenAI, remote servers, etc.)
|
||||
4. Wastes bandwidth for small bug fixes
|
||||
|
||||
---
|
||||
|
||||
## Solution: Pluggable TTS Providers
|
||||
|
||||
### Component Breakdown
|
||||
|
||||
#### 1. Main App (voicebox.exe / .app / .AppImage)
|
||||
|
||||
**Size:** ~100-150MB
|
||||
|
||||
**Includes:**
|
||||
|
||||
- Tauri runtime + React UI
|
||||
- FastAPI backend (pure Python, no PyTorch)
|
||||
- Whisper model (tiny, ~50MB)
|
||||
- SQLite database
|
||||
- Profile/history/audio editing logic
|
||||
- Provider management system
|
||||
|
||||
**Does NOT include:**
|
||||
|
||||
- PyTorch (CPU or CUDA)
|
||||
- TTS models (Qwen3-TTS)
|
||||
- Heavy ML dependencies
|
||||
|
||||
**Updates frequently:** UI fixes, feature additions, non-ML changes
|
||||
|
||||
---
|
||||
|
||||
#### 2. TTS Provider: PyTorch CPU
|
||||
|
||||
**Binary:** `tts-provider-pytorch-cpu.exe`
|
||||
**Size:** ~200MB
|
||||
|
||||
**Includes:**
|
||||
|
||||
- PyTorch CPU build
|
||||
- Qwen3-TTS package
|
||||
- Transformers
|
||||
- No CUDA libraries
|
||||
|
||||
**Download source:** Cloudflare R2
|
||||
**Updates rarely:** Only when model code changes
|
||||
|
||||
---
|
||||
|
||||
#### 3. TTS Provider: PyTorch CUDA
|
||||
|
||||
**Binary:** `tts-provider-pytorch-cuda.exe`
|
||||
**Size:** ~2.4GB
|
||||
|
||||
**Includes:**
|
||||
|
||||
- PyTorch CUDA build (cu121)
|
||||
- Qwen3-TTS package
|
||||
- CUDA runtime, cuDNN, cuBLAS
|
||||
- Transformers
|
||||
|
||||
**Download source:** Cloudflare R2
|
||||
**Platform:** Windows + Linux (NVIDIA GPU)
|
||||
**Updates rarely:** Only when model code or CUDA version changes
|
||||
|
||||
---
|
||||
|
||||
#### 4. TTS Provider: MLX
|
||||
|
||||
**Binary:** `tts-provider-mlx`
|
||||
**Size:** ~150MB
|
||||
|
||||
**Includes:**
|
||||
|
||||
- MLX framework
|
||||
- MLX-optimized Qwen3-TTS
|
||||
- Metal acceleration
|
||||
|
||||
**Platform:** macOS only (Apple Silicon)
|
||||
**Download source:** Cloudflare R2
|
||||
|
||||
---
|
||||
|
||||
#### 5. TTS Provider: Remote
|
||||
|
||||
**Binary:** None (built-in config)
|
||||
**Size:** 0MB
|
||||
|
||||
**How it works:**
|
||||
|
||||
- User provides URL to their own TTS server
|
||||
- Backend proxies requests to that server
|
||||
- Implements API spec from `EXTERNAL_PROVIDERS.md`
|
||||
|
||||
**Use cases:**
|
||||
|
||||
- AMD GPU users running their own server
|
||||
- Team deployments with shared GPU server
|
||||
- Cloud hosting (Modal, RunPod, Replicate)
|
||||
|
||||
---
|
||||
|
||||
#### 6. TTS Provider: OpenAI
|
||||
|
||||
**Binary:** None (API wrapper)
|
||||
**Size:** 0MB
|
||||
|
||||
**How it works:**
|
||||
|
||||
- User provides OpenAI API key
|
||||
- Backend wraps OpenAI Audio API
|
||||
- Voice profiles map to OpenAI voices
|
||||
|
||||
**Benefits:**
|
||||
|
||||
- Zero local compute
|
||||
- Pay-per-use
|
||||
- Instant setup
|
||||
|
||||
---
|
||||
|
||||
## Communication Protocol
|
||||
|
||||
### Provider API Specification
|
||||
|
||||
All TTS providers must implement these endpoints:
|
||||
|
||||
#### POST /tts/generate
|
||||
|
||||
Generate speech from text.
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello world!",
|
||||
"voice_prompt": {
|
||||
/* voice prompt object */
|
||||
},
|
||||
"language": "en",
|
||||
"seed": 12345,
|
||||
"model_size": "1.7B"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"audio": "base64-encoded-audio",
|
||||
"sample_rate": 24000,
|
||||
"duration": 2.5
|
||||
}
|
||||
```
|
||||
|
||||
#### POST /tts/create_voice_prompt
|
||||
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `audio`: Audio file
|
||||
- `reference_text`: Transcript
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"voice_prompt": {
|
||||
/* serialized prompt */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### GET /tts/health
|
||||
|
||||
Health check.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"provider": "pytorch-cuda",
|
||||
"version": "1.0.0",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": "cuda:0"
|
||||
}
|
||||
```
|
||||
|
||||
#### GET /tts/status
|
||||
|
||||
Model status.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_loaded": true,
|
||||
"model_size": "1.7B",
|
||||
"available_sizes": ["0.6B", "1.7B"],
|
||||
"gpu_available": true,
|
||||
"vram_used_mb": 1234
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backend Implementation
|
||||
|
||||
### Provider Manager
|
||||
|
||||
**File:** `backend/providers/__init__.py`
|
||||
|
||||
```python
|
||||
class ProviderManager:
|
||||
"""Manages TTS provider lifecycle."""
|
||||
|
||||
def __init__(self):
|
||||
self.active_provider: Optional[Provider] = None
|
||||
self.config = load_provider_config()
|
||||
|
||||
async def start_provider(self, provider_type: str) -> str:
|
||||
"""Start a TTS provider process."""
|
||||
if provider_type == "pytorch-cpu":
|
||||
return await self._start_local_provider("tts-provider-pytorch-cpu.exe")
|
||||
elif provider_type == "pytorch-cuda":
|
||||
return await self._start_local_provider("tts-provider-pytorch-cuda.exe")
|
||||
elif provider_type == "mlx":
|
||||
return await self._start_local_provider("tts-provider-mlx")
|
||||
elif provider_type == "remote":
|
||||
return self.config["remote_url"]
|
||||
elif provider_type == "openai":
|
||||
return None # No subprocess, API wrapper
|
||||
|
||||
async def _start_local_provider(self, binary_name: str) -> str:
|
||||
"""Start local provider subprocess."""
|
||||
provider_path = get_provider_binary_path(binary_name)
|
||||
|
||||
if not provider_path.exists():
|
||||
raise ProviderNotInstalledException(binary_name)
|
||||
|
||||
# Start subprocess on random port
|
||||
port = get_free_port()
|
||||
process = subprocess.Popen([
|
||||
str(provider_path),
|
||||
"--port", str(port),
|
||||
"--data-dir", str(config.get_data_dir())
|
||||
])
|
||||
|
||||
# Wait for provider to be ready
|
||||
await wait_for_provider_health(f"http://localhost:{port}")
|
||||
|
||||
self.active_provider = Provider(process, port)
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
async def stop_provider(self):
|
||||
"""Stop active provider."""
|
||||
if self.active_provider:
|
||||
self.active_provider.process.terminate()
|
||||
self.active_provider = None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Provider Abstraction
|
||||
|
||||
**File:** `backend/providers/base.py`
|
||||
|
||||
```python
|
||||
class TTSProvider(ABC):
|
||||
"""Abstract base for TTS providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str,
|
||||
seed: Optional[int]
|
||||
) -> tuple[np.ndarray, int]:
|
||||
"""Generate speech audio."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str
|
||||
) -> dict:
|
||||
"""Create voice prompt from reference audio."""
|
||||
pass
|
||||
```
|
||||
|
||||
**File:** `backend/providers/local.py`
|
||||
|
||||
```python
|
||||
class LocalProvider(TTSProvider):
|
||||
"""Provider that communicates with local subprocess via HTTP."""
|
||||
|
||||
def __init__(self, base_url: str):
|
||||
self.base_url = base_url
|
||||
self.client = httpx.AsyncClient()
|
||||
|
||||
async def generate(self, text, voice_prompt, language, seed):
|
||||
response = await self.client.post(
|
||||
f"{self.base_url}/tts/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"voice_prompt": voice_prompt,
|
||||
"language": language,
|
||||
"seed": seed
|
||||
}
|
||||
)
|
||||
data = response.json()
|
||||
audio = np.frombuffer(base64.b64decode(data["audio"]), dtype=np.float32)
|
||||
return audio, data["sample_rate"]
|
||||
```
|
||||
|
||||
**File:** `backend/providers/openai.py`
|
||||
|
||||
```python
|
||||
class OpenAIProvider(TTSProvider):
|
||||
"""Provider that wraps OpenAI Audio API."""
|
||||
|
||||
def __init__(self, api_key: str):
|
||||
self.client = OpenAI(api_key=api_key)
|
||||
|
||||
async def generate(self, text, voice_prompt, language, seed):
|
||||
# Map voice_prompt to OpenAI voice name
|
||||
voice = map_profile_to_openai_voice(voice_prompt)
|
||||
|
||||
response = await self.client.audio.speech.create(
|
||||
model="tts-1",
|
||||
voice=voice,
|
||||
input=text
|
||||
)
|
||||
|
||||
# Convert to numpy array
|
||||
audio_data = response.content
|
||||
audio, sr = load_audio_from_bytes(audio_data)
|
||||
return audio, sr
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provider Installation
|
||||
|
||||
### Download Manager
|
||||
|
||||
**File:** `backend/providers/installer.py`
|
||||
|
||||
```python
|
||||
class ProviderInstaller:
|
||||
"""Handles provider download and installation."""
|
||||
|
||||
async def download_provider(self, provider_type: str):
|
||||
"""Download provider binary from R2."""
|
||||
|
||||
binary_name = {
|
||||
"pytorch-cpu": "tts-provider-pytorch-cpu.exe",
|
||||
"pytorch-cuda": "tts-provider-pytorch-cuda.exe",
|
||||
"mlx": "tts-provider-mlx"
|
||||
}[provider_type]
|
||||
|
||||
download_url = f"https://downloads.voicebox.sh/providers/v{PROVIDER_VERSION}/{binary_name}"
|
||||
|
||||
# Download with progress tracking (reuse existing SSE system)
|
||||
await download_with_progress(
|
||||
url=download_url,
|
||||
destination=get_provider_install_path(binary_name),
|
||||
progress_key=f"provider-{provider_type}"
|
||||
)
|
||||
```
|
||||
|
||||
**Provider Storage Location:**
|
||||
|
||||
- Windows: `%APPDATA%/voicebox/providers/`
|
||||
- macOS: `~/Library/Application Support/voicebox/providers/`
|
||||
- Linux: `~/.local/share/voicebox/providers/`
|
||||
|
||||
---
|
||||
|
||||
## Frontend Implementation
|
||||
|
||||
### Provider Settings UI
|
||||
|
||||
**Component:** `app/src/components/ServerSettings/ProviderSettings.tsx`
|
||||
|
||||
```tsx
|
||||
export function ProviderSettings() {
|
||||
const [selectedProvider, setSelectedProvider] =
|
||||
useState<ProviderType>("auto");
|
||||
const {data: installedProviders} = useQuery({
|
||||
queryKey: ["providers", "installed"],
|
||||
queryFn: () => apiClient.getInstalledProviders(),
|
||||
});
|
||||
|
||||
return (
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>TTS Provider</CardTitle>
|
||||
<CardDescription>Choose how Voicebox generates speech</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<RadioGroup
|
||||
value={selectedProvider}
|
||||
onValueChange={setSelectedProvider}
|
||||
>
|
||||
{/* Auto-detect */}
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem value="auto" id="auto" />
|
||||
<Label htmlFor="auto">
|
||||
<div className="font-medium">Auto-detect (Recommended)</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Automatically choose the best available provider
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
|
||||
{/* PyTorch CUDA */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem
|
||||
value="pytorch-cuda"
|
||||
id="cuda"
|
||||
disabled={!gpuAvailable}
|
||||
/>
|
||||
<Label htmlFor="cuda">
|
||||
<div className="font-medium">PyTorch CUDA (NVIDIA GPU)</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
4-5x faster inference on NVIDIA GPUs
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
{!installedProviders?.includes("pytorch-cuda") && gpuAvailable && (
|
||||
<Button
|
||||
onClick={() => downloadProvider("pytorch-cuda")}
|
||||
size="sm"
|
||||
>
|
||||
Download (2.4GB)
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* PyTorch CPU */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem value="pytorch-cpu" id="cpu" />
|
||||
<Label htmlFor="cpu">
|
||||
<div className="font-medium">PyTorch CPU</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Works on any system, slower inference
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
{!installedProviders?.includes("pytorch-cpu") && (
|
||||
<Button onClick={() => downloadProvider("pytorch-cpu")} size="sm">
|
||||
Download (300MB)
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* MLX (macOS only) */}
|
||||
{isMacOS && (
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem value="mlx" id="mlx" />
|
||||
<Label htmlFor="mlx">
|
||||
<div className="font-medium">MLX (Apple Silicon)</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Optimized for M1/M2/M3 chips
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
{!installedProviders?.includes("mlx") && (
|
||||
<Button onClick={() => downloadProvider("mlx")} size="sm">
|
||||
Download (800MB)
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Remote */}
|
||||
<div className="space-y-2">
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem value="remote" id="remote" />
|
||||
<Label htmlFor="remote">
|
||||
<div className="font-medium">Remote Server</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Connect to your own TTS server
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
{selectedProvider === "remote" && (
|
||||
<Input placeholder="http://your-server:8000" className="ml-6" />
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* OpenAI */}
|
||||
<div className="space-y-2">
|
||||
<div className="flex items-center space-x-2">
|
||||
<RadioGroupItem value="openai" id="openai" />
|
||||
<Label htmlFor="openai">
|
||||
<div className="font-medium">OpenAI API</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Use OpenAI's TTS API (requires API key)
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
{selectedProvider === "openai" && (
|
||||
<Input type="password" placeholder="sk-..." className="ml-6" />
|
||||
)}
|
||||
</div>
|
||||
</RadioGroup>
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
voicebox/
|
||||
├── backend/
|
||||
│ ├── main.py # Main FastAPI app (no TTS code)
|
||||
│ ├── providers/
|
||||
│ │ ├── __init__.py # ProviderManager
|
||||
│ │ ├── base.py # TTSProvider ABC
|
||||
│ │ ├── local.py # LocalProvider (subprocess)
|
||||
│ │ ├── remote.py # RemoteProvider (HTTP)
|
||||
│ │ ├── openai.py # OpenAIProvider (API wrapper)
|
||||
│ │ └── installer.py # Provider download logic
|
||||
│ ├── profiles.py # Voice profile management
|
||||
│ ├── history.py # Generation history
|
||||
│ ├── transcribe.py # Whisper (still bundled)
|
||||
│ └── ... (other backend modules)
|
||||
│
|
||||
├── providers/
|
||||
│ ├── pytorch-cpu/
|
||||
│ │ ├── main.py # FastAPI server for TTS
|
||||
│ │ ├── tts_backend.py # PyTorch TTS logic
|
||||
│ │ ├── requirements.txt # torch (CPU), qwen-tts, transformers
|
||||
│ │ └── build.spec # PyInstaller spec
|
||||
│ │
|
||||
│ ├── pytorch-cuda/
|
||||
│ │ ├── main.py # FastAPI server for TTS
|
||||
│ │ ├── tts_backend.py # PyTorch TTS logic
|
||||
│ │ ├── requirements.txt # torch+cu121, qwen-tts, transformers
|
||||
│ │ └── build.spec # PyInstaller spec
|
||||
│ │
|
||||
│ └── mlx/
|
||||
│ ├── main.py # FastAPI server for TTS
|
||||
│ ├── mlx_backend.py # MLX TTS logic
|
||||
│ ├── requirements.txt # mlx, qwen-tts-mlx
|
||||
│ └── build.spec # PyInstaller spec
|
||||
│
|
||||
├── app/ # Frontend (Tauri + React)
|
||||
│ └── src/
|
||||
│ └── components/
|
||||
│ └── ServerSettings/
|
||||
│ └── ProviderSettings.tsx
|
||||
│
|
||||
└── tauri/
|
||||
└── src-tauri/
|
||||
└── tauri.conf.json # No externalBin for providers
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Phase 1: Refactor Backend (No User Changes)
|
||||
|
||||
**Goal:** Abstract TTS behind provider interface
|
||||
|
||||
1. Create `backend/providers/` module structure
|
||||
2. Implement `TTSProvider` abstract base class
|
||||
3. Create `LocalProvider` wrapper for current PyTorch code
|
||||
4. Modify `backend/tts.py` to use provider abstraction
|
||||
5. Keep PyTorch bundled in main app
|
||||
|
||||
**Result:** Code is prepared, but user experience unchanged
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Build Provider Binaries
|
||||
|
||||
**Goal:** Create standalone TTS provider executables
|
||||
|
||||
1. Create separate PyInstaller specs for each provider
|
||||
2. Build provider executables:
|
||||
- `tts-provider-pytorch-cpu.exe` (~300MB)
|
||||
- `tts-provider-pytorch-cuda.exe` (~2.4GB)
|
||||
- `tts-provider-mlx` (~800MB, macOS)
|
||||
3. Test subprocess communication
|
||||
4. Upload providers to Cloudflare R2
|
||||
|
||||
**Result:** Provider binaries exist but aren't used yet
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Remove PyTorch from Main App
|
||||
|
||||
**Goal:** Split main app from providers
|
||||
|
||||
1. Exclude PyTorch/Qwen3-TTS from main app PyInstaller spec
|
||||
2. Main app now requires provider download
|
||||
3. Update GitHub CI to build multiple artifacts:
|
||||
- `voicebox-{version}-{platform}.exe` (~150MB)
|
||||
- `tts-provider-pytorch-cpu-{version}.exe`
|
||||
- `tts-provider-pytorch-cuda-{version}.exe`
|
||||
- `tts-provider-mlx-{version}` (macOS)
|
||||
|
||||
**Result:** Main app is small, providers downloaded separately
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Add Provider UI
|
||||
|
||||
**Goal:** User-facing provider management
|
||||
|
||||
1. Create Provider Settings page
|
||||
2. Implement provider download UI
|
||||
3. Add provider status indicators
|
||||
4. Show active provider in UI
|
||||
|
||||
**Result:** Users can choose and download providers
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: External Providers
|
||||
|
||||
**Goal:** Enable remote and cloud providers
|
||||
|
||||
1. Implement `RemoteProvider` (HTTP client)
|
||||
2. Implement `OpenAIProvider` (API wrapper)
|
||||
3. Add provider configuration UI (URLs, API keys)
|
||||
4. Document external provider API spec
|
||||
|
||||
**Result:** Full provider ecosystem
|
||||
|
||||
---
|
||||
|
||||
## Provider Versioning
|
||||
|
||||
### Independent Versioning
|
||||
|
||||
Providers have their own version numbers, independent of the main app:
|
||||
|
||||
- **App version:** `v0.2.0` (frequent updates)
|
||||
- **Provider version:** `v1.0.0` (rare updates)
|
||||
|
||||
### Compatibility Matrix
|
||||
|
||||
**Example:**
|
||||
|
||||
| App Version | Min Provider Version | Max Provider Version |
|
||||
| ----------- | -------------------- | -------------------- |
|
||||
| v0.2.0 | v1.0.0 | v1.x.x |
|
||||
| v0.3.0 | v1.0.0 | v1.x.x |
|
||||
| v0.4.0 | v1.2.0 | v1.x.x |
|
||||
| v1.0.0 | v2.0.0 | v2.x.x |
|
||||
|
||||
**Backend checks compatibility:**
|
||||
|
||||
```python
|
||||
async def check_provider_compatibility(provider_version: str) -> bool:
|
||||
"""Check if provider version is compatible with current app."""
|
||||
min_version = "1.0.0"
|
||||
max_version = "1.999.999"
|
||||
return min_version <= provider_version < max_version
|
||||
```
|
||||
|
||||
**UI shows warning if incompatible:**
|
||||
|
||||
```
|
||||
⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## User Flows
|
||||
|
||||
### First-Time Setup
|
||||
|
||||
1. User downloads and installs Voicebox (~150MB)
|
||||
2. App launches → detects no TTS provider installed
|
||||
3. Shows setup wizard:
|
||||
|
||||
```
|
||||
Choose your TTS provider:
|
||||
|
||||
[ ] PyTorch CUDA (2.4GB) [Download]
|
||||
✓ Fastest on NVIDIA GPUs
|
||||
✗ Requires NVIDIA GPU
|
||||
|
||||
[●] PyTorch CPU (300MB) [Download]
|
||||
✓ Works on any system
|
||||
✗ Slower inference
|
||||
|
||||
[ ] MLX (800MB) [Download]
|
||||
✓ Fast on Apple Silicon
|
||||
✗ macOS only (M1/M2/M3)
|
||||
|
||||
[ ] Remote Server
|
||||
URL: ___________________
|
||||
|
||||
[ ] OpenAI API
|
||||
API Key: ________________
|
||||
```
|
||||
|
||||
4. User selects provider → downloads with progress bar
|
||||
5. Provider installs to AppData/Application Support
|
||||
6. App starts provider → ready to use
|
||||
|
||||
---
|
||||
|
||||
### App Update Flow (No Provider Change)
|
||||
|
||||
**Scenario:** Bug fix in UI, no backend changes
|
||||
|
||||
1. User gets update notification: "Voicebox v0.2.1 available"
|
||||
2. Downloads update (~150MB, not 2.4GB!)
|
||||
3. Installs and restarts
|
||||
4. **Provider stays the same** (no re-download needed)
|
||||
5. App starts using existing provider
|
||||
|
||||
**User experience:** Fast updates, no multi-GB downloads
|
||||
|
||||
---
|
||||
|
||||
### Provider Update Flow
|
||||
|
||||
**Scenario:** New Qwen3-TTS model version released
|
||||
|
||||
1. User opens Settings → Provider tab
|
||||
2. Sees notification: "Provider update available (v1.1.0)"
|
||||
3. Clicks "Update Provider"
|
||||
4. Downloads new provider binary
|
||||
5. Old provider binary is replaced
|
||||
6. Restart app to use new provider
|
||||
|
||||
**Frequency:** Rare (only when TTS model/backend changes)
|
||||
|
||||
---
|
||||
|
||||
### Switching Providers
|
||||
|
||||
**Scenario:** User upgrades to NVIDIA GPU
|
||||
|
||||
1. User goes to Settings → Provider
|
||||
2. Selects "PyTorch CUDA"
|
||||
3. Clicks "Download" → downloads 2.4GB
|
||||
4. Download completes → restarts app
|
||||
5. App now uses CUDA provider
|
||||
|
||||
---
|
||||
|
||||
## Benefits
|
||||
|
||||
| Benefit | Details |
|
||||
| ----------------------------- | --------------------------------------------------------- |
|
||||
| **GitHub Releases Work** | Main app ~150MB << 2GB limit |
|
||||
| **Fast Updates** | UI/feature updates don't require re-downloading providers |
|
||||
| **User Choice** | CPU, CUDA, MLX, OpenAI, remote server |
|
||||
| **External Provider Support** | Users can run their own TTS servers |
|
||||
| **Bandwidth Savings** | Only download provider once, app updates are small |
|
||||
| **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) |
|
||||
| **Team Deployments** | Multiple users share one remote provider |
|
||||
| **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. |
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
### 1. Provider Versioning
|
||||
|
||||
**Question:** Should providers have independent versions or match app version?
|
||||
|
||||
**Options:**
|
||||
|
||||
- A. Independent (providers: v1.x, app: v0.2.x)
|
||||
- B. Matched (both use v0.2.x)
|
||||
|
||||
**Recommendation:** Independent versioning with compatibility matrix
|
||||
|
||||
---
|
||||
|
||||
### 2. Auto-Update Providers
|
||||
|
||||
**Question:** Should providers auto-update separately from app?
|
||||
|
||||
**Options:**
|
||||
|
||||
- A. Manual updates only (user clicks "Update Provider")
|
||||
- B. Optional auto-update (user can enable)
|
||||
- C. Always auto-update
|
||||
|
||||
**Recommendation:** Optional auto-update (default off)
|
||||
|
||||
---
|
||||
|
||||
### 3. Provider Discovery
|
||||
|
||||
**Question:** How does app find installed providers?
|
||||
|
||||
**Options:**
|
||||
|
||||
- A. Check standard paths in AppData/Application Support
|
||||
- B. Registry (Windows) / plist (macOS)
|
||||
- C. Config file with provider locations
|
||||
|
||||
**Recommendation:** Standard paths + config fallback
|
||||
|
||||
---
|
||||
|
||||
### 4. Fallback Behavior
|
||||
|
||||
**Question:** What if no provider is installed?
|
||||
|
||||
**Options:**
|
||||
|
||||
- A. Show setup wizard on first launch
|
||||
- B. Block app until provider installed
|
||||
- C. Allow app to run in "demo mode" (transcription only)
|
||||
|
||||
**Recommendation:** Setup wizard on first launch
|
||||
|
||||
---
|
||||
|
||||
### 5. Provider Auto-Start
|
||||
|
||||
**Question:** Should provider start automatically with app?
|
||||
|
||||
**Options:**
|
||||
|
||||
- A. Always start selected provider on app launch
|
||||
- B. Start on-demand (when user generates speech)
|
||||
- C. User preference
|
||||
|
||||
**Recommendation:** Auto-start (configurable in settings)
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] **Provider Marketplace:** Built-in directory of community providers
|
||||
- [ ] **Multi-Provider Support:** Use different providers per voice/language
|
||||
- [ ] **Provider Health Monitoring:** Automatic failover if provider crashes
|
||||
- [ ] **Cost Tracking:** Monitor API usage for OpenAI/cloud providers
|
||||
- [ ] **Performance Metrics:** Latency, throughput, VRAM usage dashboards
|
||||
- [ ] **Docker Providers:** Run providers in Docker containers
|
||||
- [ ] **Provider Plugins:** Load custom providers from user scripts
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [EXTERNAL_PROVIDERS.md](./EXTERNAL_PROVIDERS.md) - External provider support plan
|
||||
- [OPENAI_SUPPORT.md](./OPENAI_SUPPORT.md) - OpenAI API compatibility
|
||||
- [github-2gb-limit-issue.md](../github-2gb-limit-issue.md) - Original problem
|
||||
- [r2-setup.md](../r2-setup.md) - Cloudflare R2 configuration
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
If you want to build a custom TTS provider:
|
||||
|
||||
1. Implement the provider API spec (see above)
|
||||
2. Test with Voicebox locally
|
||||
3. Package as executable (PyInstaller, Docker, etc.)
|
||||
4. Share in GitHub Discussions
|
||||
|
||||
**Questions?**
|
||||
|
||||
- GitHub Issues: [voicebox/issues](https://github.com/jamiepine/voicebox/issues)
|
||||
- Discord: Coming soon
|
||||
Reference in New Issue
Block a user