mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-17 22:00:40 -07:00
Add Docker support and update dependencies
- Introduced Docker support with CPU-only and GPU-enabled configurations via Dockerfiles and docker-compose files. - Added a .dockerignore file to exclude unnecessary files from Docker images. - Updated bun.lock and package.json to include new dependencies for icon handling. - Enhanced README with Docker usage instructions and deployment options. - Refactored components to utilize new icon libraries for improved UI consistency.
This commit is contained in:
Jamie Pine
@@ -0,0 +1,786 @@
|
||||
---
|
||||
title: "Docker Deployment Guide"
|
||||
description: "Docker deployment guide for Voicebox (In Development)"
|
||||
---
|
||||
|
||||
**Status:** In Development for v0.2.0
|
||||
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
|
||||
|
||||
## Overview
|
||||
|
||||
Docker support makes Voicebox easier to deploy, especially for:
|
||||
|
||||
- **Consistent Environments**: Same setup across dev/staging/prod
|
||||
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
|
||||
- **Server Deployments**: Run on headless Linux servers
|
||||
- **Multi-User Setups**: Isolate instances per user/team
|
||||
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Pre-Built Images (Recommended)
|
||||
|
||||
```bash
|
||||
# CPU-only version
|
||||
docker run -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest
|
||||
|
||||
# NVIDIA GPU version
|
||||
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
|
||||
# AMD GPU version (experimental)
|
||||
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-rocm
|
||||
```
|
||||
|
||||
Then open: `http://localhost:8000`
|
||||
|
||||
### Using Docker Compose (Easiest)
|
||||
|
||||
Create `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Building From Source
|
||||
|
||||
### Basic Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git \
|
||||
build-essential \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy application
|
||||
COPY backend/ /app/backend/
|
||||
COPY requirements.txt /app/
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8000
|
||||
|
||||
# Run server
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build and run:
|
||||
|
||||
```bash
|
||||
docker build -t voicebox .
|
||||
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Multi-Stage Build (Optimized)
|
||||
|
||||
Smaller image size by separating build and runtime:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.optimized
|
||||
# Stage 1: Build dependencies
|
||||
FROM python:3.11-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git build-essential && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
-r requirements.txt
|
||||
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Stage 2: Runtime
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install only runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy installed packages from builder
|
||||
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
|
||||
|
||||
# Copy application code
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build:
|
||||
|
||||
```bash
|
||||
docker build -f Dockerfile.optimized -t voicebox:slim .
|
||||
```
|
||||
|
||||
## GPU Support
|
||||
|
||||
### NVIDIA GPUs (CUDA)
|
||||
|
||||
**Dockerfile:**
|
||||
|
||||
```dockerfile
|
||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with CUDA support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with GPU:**
|
||||
|
||||
```bash
|
||||
docker run --gpus all -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
voicebox:cuda
|
||||
```
|
||||
|
||||
**Docker Compose with GPU:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
image: voicebox:cuda
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
### AMD GPUs (ROCm) - Experimental
|
||||
|
||||
**Dockerfile:**
|
||||
|
||||
```dockerfile
|
||||
FROM rocm/dev-ubuntu-22.04:6.0
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with ROCm support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Set ROCm environment variables
|
||||
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with AMD GPU:**
|
||||
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video --ipc=host --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-p 8000:8000 -v voicebox-data:/app/data \
|
||||
voicebox:rocm
|
||||
```
|
||||
|
||||
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
|
||||
|
||||
## Volume Mounts
|
||||
|
||||
### Essential Volumes
|
||||
|
||||
```bash
|
||||
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
|
||||
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Development Volume Mounts
|
||||
|
||||
For development with hot-reload:
|
||||
|
||||
```bash
|
||||
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
|
||||
-v voicebox-data:/app/data \
|
||||
-e RELOAD=true \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Custom Model Storage
|
||||
|
||||
Use external model directory:
|
||||
|
||||
```bash
|
||||
docker run -v /path/to/models:/models \
|
||||
-e MODELS_DIR=/models \
|
||||
-v voicebox-data:/app/data \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Configure Voicebox via environment variables:
|
||||
|
||||
```bash
|
||||
docker run -e TTS_MODE=local \
|
||||
-e WHISPER_MODE=openai-api \
|
||||
-e OPENAI_API_KEY=sk-... \
|
||||
-e GPU_MEMORY_FRACTION=0.8 \
|
||||
-e LOG_LEVEL=info \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Available Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| --------------------- | ------------- | -------------------------------------------------- |
|
||||
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
|
||||
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
|
||||
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
|
||||
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
|
||||
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
|
||||
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
|
||||
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
|
||||
| `MODELS_DIR` | `/app/models` | Directory for local models |
|
||||
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
|
||||
| `RELOAD` | `false` | Enable hot-reload for development |
|
||||
|
||||
## Complete Docker Compose Examples
|
||||
|
||||
### Production Deployment
|
||||
|
||||
```yaml
|
||||
# docker-compose.prod.yml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
container_name: voicebox
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
- GPU_MEMORY_FRACTION=0.8
|
||||
- LOG_LEVEL=info
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
driver: local
|
||||
huggingface-cache:
|
||||
driver: local
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### Development Setup
|
||||
|
||||
```yaml
|
||||
# docker-compose.dev.yml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./backend:/app/backend:ro
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- RELOAD=true
|
||||
- LOG_LEVEL=debug
|
||||
- TTS_MODE=local
|
||||
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
### Multi-Service Stack
|
||||
|
||||
Full stack with reverse proxy and monitoring:
|
||||
|
||||
```yaml
|
||||
# docker-compose.stack.yml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
# Main Voicebox app
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
# Nginx reverse proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
depends_on:
|
||||
- voicebox
|
||||
|
||||
# Prometheus monitoring (optional)
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
- prometheus-data:/prometheus
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
prometheus-data:
|
||||
```
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
|
||||
2. **Install Docker + nvidia-docker:**
|
||||
```bash
|
||||
# Amazon Linux 2
|
||||
sudo yum install -y docker
|
||||
sudo systemctl start docker
|
||||
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
|
||||
sudo apt-get update && sudo apt-get install -y nvidia-docker2
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
3. **Deploy:**
|
||||
```bash
|
||||
docker run --gpus all -d -p 80:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
```
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
Use GPU Droplet + Docker:
|
||||
|
||||
```bash
|
||||
# Create droplet via CLI
|
||||
doctl compute droplet create voicebox \
|
||||
--size gpu-h100x1-80gb \
|
||||
--image ubuntu-22-04-x64 \
|
||||
--region nyc3
|
||||
|
||||
# SSH and deploy
|
||||
ssh root@<droplet-ip>
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
docker run --gpus all -d -p 80:8000 voicebox:cuda
|
||||
```
|
||||
|
||||
### Google Cloud Run (CPU-only)
|
||||
|
||||
```bash
|
||||
# Build and push
|
||||
docker build -t gcr.io/your-project/voicebox .
|
||||
docker push gcr.io/your-project/voicebox
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy voicebox \
|
||||
--image gcr.io/your-project/voicebox \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--memory 4Gi \
|
||||
--cpu 2 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Fly.io
|
||||
|
||||
Create `fly.toml`:
|
||||
|
||||
```toml
|
||||
app = "voicebox"
|
||||
|
||||
[build]
|
||||
image = "ghcr.io/jamiepine/voicebox:latest"
|
||||
|
||||
[[services]]
|
||||
http_checks = []
|
||||
internal_port = 8000
|
||||
protocol = "tcp"
|
||||
|
||||
[[services.ports]]
|
||||
port = 80
|
||||
handlers = ["http"]
|
||||
|
||||
[[services.ports]]
|
||||
port = 443
|
||||
handlers = ["tls", "http"]
|
||||
|
||||
[mounts]
|
||||
source = "voicebox_data"
|
||||
destination = "/app/data"
|
||||
```
|
||||
|
||||
Deploy:
|
||||
|
||||
```bash
|
||||
fly launch
|
||||
fly deploy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### GPU Not Detected
|
||||
|
||||
**Check NVIDIA Docker:**
|
||||
|
||||
```bash
|
||||
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
If this fails, reinstall nvidia-docker2.
|
||||
|
||||
**Check AMD ROCm:**
|
||||
|
||||
```bash
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
|
||||
```
|
||||
|
||||
### Permission Errors
|
||||
|
||||
Container can't write to volumes:
|
||||
|
||||
```bash
|
||||
# Fix permissions
|
||||
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
Reduce GPU memory usage:
|
||||
|
||||
```bash
|
||||
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
|
||||
```
|
||||
|
||||
Or use CPU-only:
|
||||
|
||||
```bash
|
||||
docker run -e DEVICE=cpu voicebox
|
||||
```
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
Ensure HuggingFace cache is writable:
|
||||
|
||||
```bash
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
Or use host cache:
|
||||
|
||||
```bash
|
||||
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
Change host port:
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8000 voicebox # Use port 8080 instead
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Don't Run as Root
|
||||
|
||||
Create non-root user in Dockerfile:
|
||||
|
||||
```dockerfile
|
||||
RUN useradd -m -u 1000 voicebox
|
||||
USER voicebox
|
||||
```
|
||||
|
||||
### 2. Use Secrets for API Keys
|
||||
|
||||
Don't put API keys in docker-compose.yml:
|
||||
|
||||
```bash
|
||||
# Use Docker secrets
|
||||
echo "sk-your-key" | docker secret create openai_key -
|
||||
|
||||
docker service create \
|
||||
--secret openai_key \
|
||||
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
|
||||
voicebox
|
||||
```
|
||||
|
||||
### 3. Network Isolation
|
||||
|
||||
Use internal networks for multi-container setups:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
networks:
|
||||
- internal
|
||||
nginx:
|
||||
networks:
|
||||
- internal
|
||||
- external
|
||||
ports:
|
||||
- "80:80"
|
||||
|
||||
networks:
|
||||
internal:
|
||||
internal: true
|
||||
external:
|
||||
```
|
||||
|
||||
### 4. Resource Limits
|
||||
|
||||
Prevent resource exhaustion:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "4"
|
||||
memory: 8G
|
||||
reservations:
|
||||
cpus: "2"
|
||||
memory: 4G
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### GPU Memory Management
|
||||
|
||||
```bash
|
||||
# Use 80% of GPU (default 90%)
|
||||
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
|
||||
|
||||
# Allow GPU memory growth (prevents OOM)
|
||||
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
|
||||
```
|
||||
|
||||
### Model Caching
|
||||
|
||||
Pre-download models to volume:
|
||||
|
||||
```bash
|
||||
# Download models first
|
||||
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
|
||||
voicebox python -c "
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
WhisperProcessor.from_pretrained('openai/whisper-base')
|
||||
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
|
||||
"
|
||||
|
||||
# Then run normally
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Multi-Worker Setup
|
||||
|
||||
Use uvicorn workers for better throughput:
|
||||
|
||||
```dockerfile
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
Built-in health endpoint:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
Docker health check:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Add metrics exporter:
|
||||
|
||||
```python
|
||||
# backend/main.py
|
||||
from prometheus_fastapi_instrumentator import Instrumentator
|
||||
|
||||
Instrumentator().instrument(app).expose(app)
|
||||
```
|
||||
|
||||
Then scrape `/metrics` with Prometheus.
|
||||
|
||||
### Logs
|
||||
|
||||
View container logs:
|
||||
|
||||
```bash
|
||||
docker logs -f voicebox
|
||||
|
||||
# Or with compose
|
||||
docker compose logs -f voicebox
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] Publish official images to GitHub Container Registry
|
||||
- [ ] Add Kubernetes Helm charts
|
||||
- [ ] Create Docker Desktop extension
|
||||
- [ ] Add automated vulnerability scanning
|
||||
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
|
||||
|
||||
## Contributing
|
||||
|
||||
Help improve Docker support:
|
||||
|
||||
1. Test on different platforms (AMD GPU, ARM64, etc.)
|
||||
2. Submit Dockerfile optimizations
|
||||
3. Share deployment configurations
|
||||
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
|
||||
|
||||
## Resources
|
||||
|
||||
- [Docker Documentation](https://docs.docker.com)
|
||||
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
|
||||
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
|
||||
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
|
||||
@@ -0,0 +1,461 @@
|
||||
---
|
||||
title: "External Provider Support"
|
||||
description: "External provider support for Voicebox (Planned)"
|
||||
---
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...)
|
||||
|
||||
## Overview
|
||||
|
||||
External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for:
|
||||
|
||||
- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server
|
||||
- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI
|
||||
- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc.
|
||||
- **Team Sharing**: Multiple users share one GPU server running models
|
||||
- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ HTTP/API ┌──────────────────┐
|
||||
│ Voicebox UI │ ───────────────────────> │ Your TTS Server │
|
||||
│ + Backend │ │ (Qwen3-TTS on │
|
||||
│ │ <─────────────────────── │ AMD/NVIDIA GPU)│
|
||||
│ - Profiles │ Audio + Metadata └──────────────────┘
|
||||
│ - History │
|
||||
│ - Audio Edit │ HTTP/API ┌──────────────────┐
|
||||
│ - UI │ ───────────────────────> │ Whisper Service │
|
||||
└─────────────────┘ │ (OpenAI API or │
|
||||
│ self-hosted) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**What Voicebox Still Handles:**
|
||||
|
||||
- Voice profile management
|
||||
- Generation history
|
||||
- Audio trimming/editing
|
||||
- Multi-track story editor
|
||||
- UI/UX layer
|
||||
|
||||
**What External Providers Handle:**
|
||||
|
||||
- Model inference (TTS generation, transcription)
|
||||
- GPU allocation
|
||||
- Model loading/caching
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# TTS Provider
|
||||
TTS_MODE=remote # local | remote
|
||||
TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL
|
||||
TTS_API_KEY=your-api-key # Optional authentication
|
||||
|
||||
# Whisper Provider
|
||||
WHISPER_MODE=openai-api # local | openai-api | remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper
|
||||
OPENAI_API_KEY=sk-... # For OpenAI Whisper API
|
||||
```
|
||||
|
||||
### Voicebox Config UI (Planned)
|
||||
|
||||
Settings page will include:
|
||||
|
||||
- Provider selection dropdowns
|
||||
- URL/API key inputs
|
||||
- Connection test button
|
||||
- Latency/status indicators
|
||||
|
||||
## Hosting External Services
|
||||
|
||||
### Option 1: Simple FastAPI Server (Recommended)
|
||||
|
||||
Create a lightweight server to expose your local Qwen3-TTS model:
|
||||
|
||||
```python
|
||||
# tts_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
import numpy as np
|
||||
import base64
|
||||
|
||||
app = FastAPI()
|
||||
model = Qwen3TTSModel.from_pretrained(
|
||||
"Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm
|
||||
)
|
||||
|
||||
@app.post("/v1/generate")
|
||||
async def generate(
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: int = None
|
||||
):
|
||||
"""Generate speech from text using voice prompt."""
|
||||
audio, sample_rate = model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
)
|
||||
|
||||
# Return as base64 for transport
|
||||
audio_bytes = audio.tobytes()
|
||||
return {
|
||||
"audio": base64.b64encode(audio_bytes).decode(),
|
||||
"sample_rate": sample_rate,
|
||||
"dtype": str(audio.dtype)
|
||||
}
|
||||
|
||||
@app.post("/v1/create_voice_prompt")
|
||||
async def create_voice_prompt(
|
||||
audio: UploadFile = File(...),
|
||||
reference_text: str = ""
|
||||
):
|
||||
"""Create voice prompt from reference audio."""
|
||||
# Save uploaded audio temporarily
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
# Create voice prompt
|
||||
voice_prompt = model.create_voice_clone_prompt(
|
||||
ref_audio=audio_path,
|
||||
ref_text=reference_text,
|
||||
)
|
||||
|
||||
return {"voice_prompt": voice_prompt}
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": str(model.device)
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
```
|
||||
|
||||
**Run it:**
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install fastapi uvicorn qwen-tts torch
|
||||
|
||||
# For AMD GPUs, use ROCm PyTorch:
|
||||
pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
|
||||
|
||||
# Start server
|
||||
python tts_server.py
|
||||
```
|
||||
|
||||
### Option 2: vLLM (If Supported)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
### Option 3: Cloud Platforms
|
||||
|
||||
**Modal.com Example:**
|
||||
|
||||
```python
|
||||
import modal
|
||||
|
||||
app = modal.App("qwen-tts")
|
||||
image = modal.Image.debian_slim().pip_install("qwen-tts", "torch")
|
||||
|
||||
@app.function(gpu="A10G", image=image)
|
||||
@modal.web_endpoint(method="POST")
|
||||
def generate(text: str, voice_prompt: dict):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
|
||||
audio, sr = model.generate_voice_clone(text, voice_prompt)
|
||||
return {"audio": audio.tolist(), "sample_rate": sr}
|
||||
```
|
||||
|
||||
Deploy: `modal deploy tts_server.py`
|
||||
Get URL: `https://yourapp--generate.modal.run`
|
||||
|
||||
## API Specification
|
||||
|
||||
External TTS providers must implement these endpoints:
|
||||
|
||||
### `POST /v1/generate`
|
||||
|
||||
Generate speech from text.
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello, this is a test.",
|
||||
"voice_prompt": {
|
||||
/* voice prompt object */
|
||||
},
|
||||
"language": "en",
|
||||
"seed": 12345
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"audio": "base64-encoded-audio-bytes",
|
||||
"sample_rate": 24000,
|
||||
"dtype": "float32"
|
||||
}
|
||||
```
|
||||
|
||||
### `POST /v1/create_voice_prompt`
|
||||
|
||||
Create a voice prompt from reference audio.
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `audio`: Audio file upload
|
||||
- `reference_text`: Transcript of the audio
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"voice_prompt": {
|
||||
/* voice prompt object */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `GET /health`
|
||||
|
||||
Health check endpoint.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": "cuda:0"
|
||||
}
|
||||
```
|
||||
|
||||
## Whisper External Providers
|
||||
|
||||
### OpenAI Whisper API
|
||||
|
||||
Simply set:
|
||||
|
||||
```bash
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
Voicebox will use OpenAI's Whisper API automatically.
|
||||
|
||||
### Self-Hosted Whisper
|
||||
|
||||
Run your own Whisper server:
|
||||
|
||||
```python
|
||||
# whisper_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
import librosa
|
||||
|
||||
app = FastAPI()
|
||||
processor = WhisperProcessor.from_pretrained("openai/whisper-base")
|
||||
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
|
||||
|
||||
@app.post("/v1/transcribe")
|
||||
async def transcribe(audio: UploadFile = File(...), language: str = None):
|
||||
# Load audio
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
audio_data, sr = librosa.load(audio_path, sr=16000)
|
||||
|
||||
# Process
|
||||
inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt")
|
||||
predicted_ids = model.generate(inputs["input_features"])
|
||||
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
|
||||
|
||||
return {"text": transcription}
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
|
||||
```bash
|
||||
WHISPER_MODE=remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. AMD GPU User with Existing Setup
|
||||
|
||||
**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux.
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. Run `tts_server.py` on your AMD box (ROCm PyTorch)
|
||||
2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000`
|
||||
3. Use Voicebox UI for profiles, generation, editing
|
||||
4. TTS happens on your AMD GPU
|
||||
|
||||
### 2. Team Deployment
|
||||
|
||||
**Scenario:** 5 team members, 1 GPU server.
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. Deploy TTS server on shared GPU box
|
||||
2. Each person runs Voicebox desktop app locally
|
||||
3. All point to same `TTS_REMOTE_URL`
|
||||
4. Profiles and history stay local per user
|
||||
5. GPU usage is shared
|
||||
|
||||
### 3. Hybrid Local/Remote
|
||||
|
||||
**Scenario:** Fast local Whisper, heavy TTS on cloud.
|
||||
|
||||
**Setup:**
|
||||
|
||||
```bash
|
||||
TTS_MODE=remote
|
||||
TTS_REMOTE_URL=https://your-modal-app.modal.run
|
||||
|
||||
WHISPER_MODE=local # Fast transcription on your CPU
|
||||
```
|
||||
|
||||
### 4. OpenAI Whisper + Self-Hosted TTS
|
||||
|
||||
**Scenario:** Use OpenAI's API for transcription, run TTS locally.
|
||||
|
||||
**Setup:**
|
||||
|
||||
```bash
|
||||
TTS_MODE=local
|
||||
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Authentication
|
||||
|
||||
Add API key authentication to your external server:
|
||||
|
||||
```python
|
||||
from fastapi import Header, HTTPException
|
||||
|
||||
API_KEY = "your-secret-key"
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(...)):
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
|
||||
@app.post("/v1/generate", dependencies=[Depends(verify_api_key)])
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
|
||||
```bash
|
||||
TTS_API_KEY=your-secret-key
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- **VPN/Tailscale**: Use private network for remote servers
|
||||
- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates
|
||||
- **Firewall**: Restrict access to known IPs
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Protect your external server:
|
||||
|
||||
```python
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
|
||||
@app.post("/v1/generate")
|
||||
@limiter.limit("10/minute")
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Latency
|
||||
|
||||
External providers add network latency:
|
||||
|
||||
- **Local network**: ~10-50ms overhead (negligible)
|
||||
- **Same datacenter**: ~1-5ms overhead
|
||||
- **Cross-region cloud**: 50-200ms+ overhead
|
||||
|
||||
For real-time applications, keep TTS server on local network or same cloud region.
|
||||
|
||||
### Caching
|
||||
|
||||
Implement response caching on external server:
|
||||
|
||||
```python
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=1000)
|
||||
def get_cached_generation(text, voice_prompt_hash, language, seed):
|
||||
return model.generate_voice_clone(text, voice_prompt)
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
For high-traffic deployments, run multiple TTS servers behind a load balancer:
|
||||
|
||||
```
|
||||
Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1)
|
||||
├──> TTS Server 2 (GPU 2)
|
||||
└──> TTS Server 3 (GPU 3)
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] **Provider Marketplace**: Built-in directory of compatible providers
|
||||
- [ ] **Automatic Fallback**: If remote fails, fallback to local
|
||||
- [ ] **Cost Tracking**: Monitor API usage and costs
|
||||
- [ ] **Performance Metrics**: Latency, throughput dashboards
|
||||
- [ ] **Multi-Provider**: Use different providers for different voices/languages
|
||||
|
||||
## Contributing
|
||||
|
||||
If you build an external provider, please share:
|
||||
|
||||
1. Server implementation
|
||||
2. Performance benchmarks
|
||||
3. Deployment guide
|
||||
|
||||
Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions)
|
||||
|
||||
## Questions?
|
||||
|
||||
- **Discord**: [Join the community](https://discord.gg/...)
|
||||
- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues)
|
||||
- **Docs**: [Full documentation](https://voicebox.sh/docs)
|
||||
@@ -0,0 +1,431 @@
|
||||
---
|
||||
title: "MLX Audio Integration"
|
||||
description: "MLX Audio integration for Voicebox (Validated)"
|
||||
---
|
||||
|
||||
**Status:** Validated ✅
|
||||
**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio)
|
||||
|
||||
## Validation Results
|
||||
|
||||
We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings:
|
||||
|
||||
| Metric | Result |
|
||||
| --------------- | ------------------------------------------- |
|
||||
| MLX Version | 0.30.4 |
|
||||
| Model Load Time | ~1s (after initial download) |
|
||||
| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) |
|
||||
| Test Hardware | Apple Silicon Mac |
|
||||
|
||||
### Model Mapping
|
||||
|
||||
| voicebox (PyTorch) | mlx-audio (MLX) |
|
||||
| ------------------------------- | --------------------------------------------- |
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` |
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) |
|
||||
|
||||
### mlx-audio API
|
||||
|
||||
The API uses a **generator-based streaming pattern**:
|
||||
|
||||
```python
|
||||
from mlx_audio.tts import load
|
||||
|
||||
model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16")
|
||||
|
||||
# generate() yields GenerationResult objects
|
||||
for result in model.generate("Hello world"):
|
||||
audio = result.audio # numpy array of samples
|
||||
sample_rate = result.sample_rate # 24000
|
||||
rtf = result.real_time_factor # e.g., 0.55
|
||||
```
|
||||
|
||||
### Known Warnings (harmless)
|
||||
|
||||
```
|
||||
You are using a model of type qwen3_tts to instantiate a model of type .
|
||||
The tokenizer you are loading... with an incorrect regex pattern...
|
||||
```
|
||||
|
||||
These warnings appear but don't affect functionality or output quality.
|
||||
|
||||
### Demo Script
|
||||
|
||||
Run `mlx-test/demo.py` to test:
|
||||
|
||||
```bash
|
||||
cd mlx-test && source venv/bin/activate && python demo.py "Your text here"
|
||||
```
|
||||
|
||||
## Problem
|
||||
|
||||
Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs.
|
||||
|
||||
This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads.
|
||||
|
||||
## Solution
|
||||
|
||||
Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use.
|
||||
|
||||
**Key wins:**
|
||||
|
||||
- Native GPU acceleration on Apple Silicon (no more CPU fallback)
|
||||
- Streaming TTS support (faster perceived latency)
|
||||
- Memory optimizations (run larger models on less RAM)
|
||||
- Fixed 0.6B silence bug that we currently ship
|
||||
- Same Qwen3-TTS models (zero migration cost for users)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Current Stack
|
||||
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch + Qwen3-TTS │
|
||||
│ (CPU only on macOS) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### Proposed Stack
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Platform Detection at Runtime │
|
||||
└─────────────────────────────────────────┘
|
||||
│
|
||||
├─── Apple Silicon (aarch64-darwin)
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ MLX Audio Backend │
|
||||
│ │ - Qwen3-TTS (mlx) │
|
||||
│ │ - Whisper (mlx) │
|
||||
│ │ - Streaming support │
|
||||
│ └─────────────────────────┘
|
||||
│
|
||||
└─── Other (x86_64, Windows, Linux)
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch Backend │
|
||||
│ - Qwen3-TTS (pytorch) │
|
||||
│ - Whisper (pytorch) │
|
||||
│ - CUDA if available │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Platform Detection & Dependency Management
|
||||
|
||||
Create a backend that switches between PyTorch and MLX based on runtime platform detection.
|
||||
|
||||
**New files:**
|
||||
|
||||
- `backend/platform.py` - Detect Apple Silicon, return backend type
|
||||
- `backend/backends/__init__.py` - Backend factory pattern
|
||||
- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only)
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `backend/requirements.txt` - Keep PyTorch as default
|
||||
- `backend/main.py` - Import from backend factory instead of direct imports
|
||||
|
||||
**Platform detection logic:**
|
||||
|
||||
```python
|
||||
def get_backend_type() -> str:
|
||||
"""Detect best backend for current platform."""
|
||||
if platform.system() == "Darwin" and platform.machine() == "arm64":
|
||||
# Apple Silicon detected
|
||||
try:
|
||||
import mlx
|
||||
return "mlx"
|
||||
except ImportError:
|
||||
return "pytorch" # Fallback if mlx not installed
|
||||
return "pytorch"
|
||||
```
|
||||
|
||||
### Phase 2: MLX Backend Implementation
|
||||
|
||||
Create parallel implementations of TTS and STT using mlx-audio.
|
||||
|
||||
**New files:**
|
||||
|
||||
- `backend/backends/mlx_backend.py` - MLX inference engine
|
||||
- `backend/backends/pytorch_backend.py` - Refactor current code into backend
|
||||
|
||||
**Interface both backends must implement:**
|
||||
|
||||
```python
|
||||
class TTSBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ...
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ...
|
||||
async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ...
|
||||
def unload_model(self) -> None: ...
|
||||
|
||||
class STTBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ...
|
||||
def unload_model(self) -> None: ...
|
||||
```
|
||||
|
||||
**MLX backend implementation notes:**
|
||||
|
||||
mlx-audio's `generate()` returns a generator by default (streaming is built-in):
|
||||
|
||||
```python
|
||||
# MLX backend wrapper
|
||||
from mlx_audio.tts import load
|
||||
|
||||
class MLXTTSBackend:
|
||||
def __init__(self):
|
||||
self.model = None
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
model_map = {
|
||||
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
|
||||
# "0.6B": needs conversion to mlx format
|
||||
}
|
||||
self.model = load(model_map[model_size])
|
||||
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]:
|
||||
# Collect all chunks from generator
|
||||
chunks = []
|
||||
for result in self.model.generate(text): # TODO: add voice_prompt support
|
||||
chunks.append(np.array(result.audio))
|
||||
return np.concatenate(chunks), 24000
|
||||
```
|
||||
|
||||
**MLX-specific features to expose:**
|
||||
|
||||
- Streaming TTS (new endpoint: `/api/generate/stream`)
|
||||
- Memory-optimized model loading
|
||||
- Qwen3-ASR for transcription (in addition to Whisper)
|
||||
|
||||
### Phase 3: API Layer Updates
|
||||
|
||||
Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility.
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `backend/main.py` - Add streaming endpoints
|
||||
- `backend/tts.py` - Refactor to use backend abstraction
|
||||
- `backend/transcribe.py` - Refactor to use backend abstraction
|
||||
|
||||
**New endpoints:**
|
||||
|
||||
```python
|
||||
@app.post("/api/generate/stream")
|
||||
async def generate_stream(...) -> StreamingResponse:
|
||||
"""Stream TTS chunks as they're generated (MLX only)."""
|
||||
backend = get_backend()
|
||||
if not hasattr(backend, 'generate_streaming'):
|
||||
raise HTTPException(501, "Streaming not supported on this backend")
|
||||
return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav")
|
||||
```
|
||||
|
||||
**Backward compatibility:**
|
||||
|
||||
- Keep all existing `/api/generate` endpoints unchanged
|
||||
- PyTorch backend users see no behavior change
|
||||
- MLX users automatically get faster inference, streaming is opt-in
|
||||
|
||||
### Phase 4: Frontend Integration
|
||||
|
||||
Add UI indicators for backend type and streaming progress.
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `app/src/hooks/useGenerationForm.tsx` - Add streaming support
|
||||
- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle
|
||||
- `app/src/lib/api.ts` - Add streaming API client
|
||||
|
||||
**UI additions:**
|
||||
|
||||
- Badge showing current backend ("MLX" or "PyTorch")
|
||||
- Toggle for streaming mode (disabled if PyTorch)
|
||||
- Real-time streaming playback (WaveSurfer progressive loading)
|
||||
|
||||
### Phase 5: Build & Distribution
|
||||
|
||||
Create separate installers for MLX (Apple Silicon) and PyTorch (Universal).
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds
|
||||
- `.github/workflows/release.yml` - Build both variants
|
||||
|
||||
**Build matrix:**
|
||||
|
||||
```yaml
|
||||
- target: aarch64-apple-darwin
|
||||
backend: mlx
|
||||
installer: voicebox-macos-silicon-{version}.dmg
|
||||
|
||||
- target: x86_64-apple-darwin
|
||||
backend: pytorch
|
||||
installer: voicebox-macos-intel-{version}.dmg
|
||||
|
||||
- target: x86_64-pc-windows-msvc
|
||||
backend: pytorch
|
||||
installer: voicebox-windows-{version}.exe
|
||||
```
|
||||
|
||||
**Installation flow:**
|
||||
|
||||
- Auto-detect architecture, recommend correct installer
|
||||
- MLX installer includes `mlx-audio` in embedded Python
|
||||
- PyTorch installer includes `torch` in embedded Python
|
||||
- Both can coexist (different backend, same profile format)
|
||||
|
||||
### Phase 6: Testing & Validation
|
||||
|
||||
Ensure both backends produce compatible outputs.
|
||||
|
||||
**New files:**
|
||||
|
||||
- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio
|
||||
- `backend/tests/test_streaming.py` - Streaming-specific tests
|
||||
|
||||
**Test scenarios:**
|
||||
|
||||
- Same voice prompt on both backends → similar (not identical) audio output
|
||||
- Profile created on MLX → loads on PyTorch (and vice versa)
|
||||
- Streaming chunks assemble into valid WAV file
|
||||
- Model downloads work on both backends
|
||||
- Memory usage stays within bounds
|
||||
|
||||
### Phase 7: Documentation
|
||||
|
||||
Update user-facing docs and developer guides.
|
||||
|
||||
**New files:**
|
||||
|
||||
- `docs/developer/BACKENDS.md` - Guide for adding new backends
|
||||
- `docs/overview/performance.md` - Backend comparison benchmarks
|
||||
|
||||
**Modified files:**
|
||||
|
||||
- `README.md` - Note Apple Silicon acceleration
|
||||
- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues
|
||||
|
||||
**Key docs to write:**
|
||||
|
||||
- Which installer to download (architecture detection)
|
||||
- Performance comparison (MLX vs PyTorch on same M2 hardware)
|
||||
- How streaming mode works
|
||||
- How to force PyTorch on Apple Silicon (for debugging)
|
||||
|
||||
## Technical Decisions
|
||||
|
||||
### Why Dual Backend Instead of MLX-Only?
|
||||
|
||||
**Pros of dual backend:**
|
||||
|
||||
- Windows and Intel Mac users unaffected
|
||||
- Easier testing (can compare outputs)
|
||||
- Fallback if MLX has issues
|
||||
|
||||
**Cons of dual backend:**
|
||||
|
||||
- More code to maintain
|
||||
- Two dependency trees
|
||||
- Build complexity (separate installers)
|
||||
|
||||
**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback.
|
||||
|
||||
### Why Separate Installers Instead of Runtime Detection?
|
||||
|
||||
**Pros of separate installers:**
|
||||
|
||||
- Smaller bundle size (don't ship both PyTorch and MLX)
|
||||
- Clearer to users which version they have
|
||||
- Easier to debug (no "which backend am I running?" confusion)
|
||||
- Can optimize each build for its target
|
||||
|
||||
**Cons:**
|
||||
|
||||
- More installers to build and test
|
||||
- Users might download the wrong one
|
||||
|
||||
**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page.
|
||||
|
||||
### Streaming vs Batch Generation
|
||||
|
||||
MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be:
|
||||
|
||||
1. MLX-only feature (✅ chosen)
|
||||
2. Implemented for both (lots of work)
|
||||
3. Not exposed at all (wasted opportunity)
|
||||
|
||||
**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend).
|
||||
|
||||
## Migration Path
|
||||
|
||||
Nothing needs migrating, macos users will just notice a speed-boost in inference
|
||||
|
||||
**Data format compatibility:**
|
||||
|
||||
- Profiles (SQLite) → no schema changes needed
|
||||
- Voice prompts (cached) → backend-agnostic (just numpy arrays)
|
||||
- Audio files → unchanged
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
### Measured Results (from validation)
|
||||
|
||||
| Metric | MLX (measured) | PyTorch CPU (estimated) |
|
||||
| ----------------------- | -------------- | ----------------------- |
|
||||
| **6s audio generation** | ~3-4s | ~10-15s |
|
||||
| **Real-time factor** | 0.5-0.6x | 2-3x |
|
||||
| **Model load (cached)** | ~1s | ~3-5s |
|
||||
|
||||
### TTS Generation (1.7B model, ~20s output)
|
||||
|
||||
- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time)
|
||||
- **MLX (M2 Max):** ~8-12s (faster than real-time)
|
||||
- **Improvement:** ~4-5x faster
|
||||
|
||||
### Whisper Transcription (10s audio clip)
|
||||
|
||||
- **PyTorch CPU:** ~5-8s
|
||||
- **MLX:** ~1-2s
|
||||
- **Improvement:** ~3-4x faster
|
||||
|
||||
### Memory Usage (1.7B model)
|
||||
|
||||
- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM)
|
||||
- **MLX:** ~4-6GB (unified memory, better optimization)
|
||||
- **Improvement:** ~40% less RAM
|
||||
|
||||
Full benchmarks will be in `docs/overview/performance.md` after Phase 6.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure
|
||||
- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No
|
||||
- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing
|
||||
- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
How we'll know this worked:
|
||||
|
||||
1. **Performance:** Apple Silicon users report generation faster than real-time
|
||||
2. **Adoption:** >80% of macOS downloads are MLX build within 1 month
|
||||
3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions)
|
||||
4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance
|
||||
|
||||
## Related Work
|
||||
|
||||
- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly
|
||||
- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API
|
||||
- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/`
|
||||
2. Get approval on dual-backend architecture
|
||||
3. Start Phase 1 (platform detection)
|
||||
|
||||
## Questions?
|
||||
|
||||
Feedback welcome in GitHub discussions or Discord.
|
||||
@@ -0,0 +1,238 @@
|
||||
---
|
||||
title: "OpenAI API Compatibility"
|
||||
description: "OpenAI API compatibility for Voicebox (Planned)"
|
||||
---
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
|
||||
**Issue:** [#10 OpenAI API compatibility](https://github.com/jamiepine/voicebox/issues/10)
|
||||
|
||||
## Overview
|
||||
|
||||
This feature exposes OpenAI-compatible endpoints from Voicebox, allowing any tool, library, or application that speaks the OpenAI Audio API to use Voicebox as a drop-in local replacement.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph clients [External Clients]
|
||||
SDK[OpenAI SDK]
|
||||
Curl[curl / HTTP]
|
||||
Apps[Third-party Apps]
|
||||
end
|
||||
|
||||
subgraph voicebox [Voicebox Server]
|
||||
OpenAI["/v1/audio/* endpoints"]
|
||||
TTS[TTSModel]
|
||||
Whisper[WhisperModel]
|
||||
Profiles[Voice Profiles]
|
||||
end
|
||||
|
||||
SDK --> OpenAI
|
||||
Curl --> OpenAI
|
||||
Apps --> OpenAI
|
||||
OpenAI --> TTS
|
||||
OpenAI --> Whisper
|
||||
OpenAI --> Profiles
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **OpenAI SDK users**: `openai.audio.speech.create()` works with Voicebox
|
||||
- **LLM frameworks**: LangChain, AutoGen, etc. can use Voicebox for TTS
|
||||
- **Shell scripts**: `curl` commands copy-pasted from OpenAI docs work
|
||||
- **Existing integrations**: Any tool expecting OpenAI's API works without code changes
|
||||
|
||||
## Endpoints to Implement
|
||||
|
||||
### 1. `POST /v1/audio/speech` (TTS)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createSpeech
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "tts-1",
|
||||
"input": "Hello world!",
|
||||
"voice": "alloy",
|
||||
"response_format": "mp3",
|
||||
"speed": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
**Response:** Audio file (mp3, wav, opus, aac, flac, pcm)
|
||||
|
||||
**Voice Mapping Strategy:**
|
||||
|
||||
- `voice` parameter maps to Voicebox profile names (case-insensitive)
|
||||
- If no match, use a configurable default profile
|
||||
- Support special syntax: `voice: "profile:uuid"` for explicit profile ID
|
||||
|
||||
### 2. `POST /v1/audio/transcriptions` (Whisper)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createTranscription
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `file`: Audio file
|
||||
- `model`: "whisper-1"
|
||||
- `language`: Optional language hint
|
||||
- `response_format`: json, text, srt, verbose_json, vtt
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello world!"
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### New File: `backend/openai_compat.py`
|
||||
|
||||
Create a dedicated module with an APIRouter for OpenAI-compatible endpoints:
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Literal, Optional
|
||||
|
||||
router = APIRouter(prefix="/v1/audio", tags=["OpenAI Compatible"])
|
||||
|
||||
class SpeechRequest(BaseModel):
|
||||
model: str = "tts-1"
|
||||
input: str
|
||||
voice: str = "alloy"
|
||||
response_format: Literal["mp3", "wav", "opus", "aac", "flac", "pcm"] = "mp3"
|
||||
speed: float = 1.0
|
||||
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest, db: Session = Depends(get_db)):
|
||||
# 1. Map voice name to profile
|
||||
# 2. Generate audio using existing TTSModel
|
||||
# 3. Convert to requested format
|
||||
# 4. Return audio stream
|
||||
...
|
||||
|
||||
@router.post("/transcriptions")
|
||||
async def create_transcription(
|
||||
file: UploadFile = File(...),
|
||||
model: str = Form("whisper-1"),
|
||||
language: Optional[str] = Form(None),
|
||||
response_format: str = Form("json"),
|
||||
):
|
||||
# 1. Save uploaded file
|
||||
# 2. Transcribe using existing WhisperModel
|
||||
# 3. Return in requested format
|
||||
...
|
||||
```
|
||||
|
||||
### Voice Profile Resolution
|
||||
|
||||
Add helper in [backend/profiles.py](backend/profiles.py):
|
||||
|
||||
```python
|
||||
async def resolve_voice_for_openai(voice: str, db: Session) -> Optional[VoiceProfile]:
|
||||
"""
|
||||
Resolve OpenAI voice parameter to a Voicebox profile.
|
||||
|
||||
Priority:
|
||||
1. Exact profile name match (case-insensitive)
|
||||
2. Profile ID match (if voice starts with "profile:")
|
||||
3. Default profile from config
|
||||
4. First available profile
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### Audio Format Conversion
|
||||
|
||||
Add conversion utilities in [backend/utils/audio.py](backend/utils/audio.py):
|
||||
|
||||
```python
|
||||
def convert_audio_format(
|
||||
audio: np.ndarray,
|
||||
sample_rate: int,
|
||||
target_format: str, # mp3, wav, opus, aac, flac, pcm
|
||||
) -> bytes:
|
||||
"""Convert audio to target format using ffmpeg or pydub."""
|
||||
...
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to [backend/config.py](backend/config.py):
|
||||
|
||||
```python
|
||||
# OpenAI API Compatibility
|
||||
OPENAI_COMPAT_ENABLED = True
|
||||
OPENAI_COMPAT_DEFAULT_VOICE = None # Profile ID or name for default voice
|
||||
OPENAI_COMPAT_REQUIRE_AUTH = False # Require API key validation
|
||||
OPENAI_COMPAT_API_KEY = None # If set, validate against this
|
||||
```
|
||||
|
||||
### Integration with main.py
|
||||
|
||||
In [backend/main.py](backend/main.py), include the router:
|
||||
|
||||
```python
|
||||
from . import openai_compat
|
||||
|
||||
# Add OpenAI-compatible routes
|
||||
if config.OPENAI_COMPAT_ENABLED:
|
||||
app.include_router(openai_compat.router)
|
||||
```
|
||||
|
||||
## Streaming Support (Future Enhancement)
|
||||
|
||||
Initial implementation returns complete audio. Streaming can be added later:
|
||||
|
||||
```python
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest):
|
||||
if request.stream:
|
||||
return StreamingResponse(
|
||||
generate_audio_chunks(request),
|
||||
media_type=f"audio/{request.response_format}"
|
||||
)
|
||||
...
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Example usage after implementation:
|
||||
|
||||
```bash
|
||||
# TTS with curl
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Hello!", "voice": "MyProfile"}' \
|
||||
--output speech.mp3
|
||||
|
||||
# With OpenAI Python SDK
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.audio.speech.create(
|
||||
model="tts-1",
|
||||
voice="MyProfile",
|
||||
input="Hello world!"
|
||||
)
|
||||
response.stream_to_file("output.mp3")
|
||||
|
||||
# Transcription
|
||||
curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F file=@audio.mp3 \
|
||||
-F model="whisper-1"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Optional API key validation (for shared deployments)
|
||||
- Rate limiting on endpoints
|
||||
- Input length limits (same as existing `/generate` endpoint)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `pydub` or `ffmpeg-python` for audio format conversion (mp3, opus, etc.)
|
||||
- No changes to existing TTS/Whisper model code
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"title": "Plans",
|
||||
"pages": ["DOCKER_DEPLOYMENT", "EXTERNAL_PROVIDERS", "MLX_AUDIO", "OPENAI_SUPPORT"]
|
||||
}
|
||||
Reference in New Issue
Block a user