Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-20 07:10:40 -07:00
Code Issues Packages Projects Releases Wiki Activity

rewrite backend README, remove completed refactor plan, update style guide

Browse Source 
Replace the outdated backend README (473 lines of stale API docs and
pre-refactor file tree) with a concise architecture document covering
module structure, request flow, backend selection, API domain overview,
and development commands.

Delete REFACTOR_PLAN.md -- all phases are complete.

Update STYLE_GUIDE.md to remove refactor plan references and replace
the verbose target layout with the current actual structure.
This commit is contained in:
James Pine
2026-03-16 02:18:34 -07:00
parent 7c1ea0a1e1
commit 7b0fbfb567
3 changed files with 122 additions and 693 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/README.md
+107 -445
View File
@@ -1,473 +1,135 @@
# voicebox Backend
# Voicebox Backend
Production-quality FastAPI backend for Qwen3-TTS voice cloning.
FastAPI server powering voice cloning, speech generation, and audio processing. Runs locally as a Tauri sidecar or standalone via `python -m backend.main`.
## Features
## Running
- ✅ **Voice Profile Management** - Create, update, delete voice profiles with multi-sample support
- ✅ **Voice Cloning** - Generate speech using voice profiles with caching
- ✅ **Generation History** - Full history tracking with search and filtering
- ✅ **Transcription** - Whisper-based audio transcription
- ✅ **Multi-Sample Profiles** - Combine multiple reference samples for better quality
- ✅ **Voice Prompt Caching** - Dual memory + disk caching for fast generation
- ✅ **Audio Validation** - Automatic validation of reference audio quality
- ✅ **Model Management** - Lazy loading and VRAM management
```bash
# Via justfile (recommended)
just dev:server
# Standalone
python -m backend.main --host 127.0.0.1 --port 17493
# With custom data directory
python -m backend.main --data-dir /path/to/data
```
The server auto-initializes the SQLite database on first startup. Models are downloaded from HuggingFace on first use.
## Architecture
```
backend/
├── main.py # FastAPI app with all routes
├── server.py # PyInstaller entry point, CLI arg parsing
├── models.py # Pydantic request/response models
├── config.py # Data directory configuration
├── database.py # SQLAlchemy ORM models + migrations
├── platform_detect.py # Platform detection for backend selection
├── tts.py # TTS backend facade
├── transcribe.py # STT backend facade
├── profiles.py # Voice profile CRUD
├── history.py # Generation history CRUD
├── channels.py # Audio channel management
├── stories.py # Story/timeline management + audio export
├── effects.py # Effect preset CRUD
├── versions.py # Generation version management
├── export_import.py # ZIP export/import for profiles and generations
├── backends/ # Backend implementations
│ ├── __init__.py # Protocols, model config registry, factory functions
│ ├── mlx_backend.py # MLX backend (Apple Silicon)
│ ├── pytorch_backend.py # PyTorch backend (Windows/Linux/Intel)
│ ├── chatterbox_backend.py # Chatterbox Multilingual TTS
│ ├── chatterbox_turbo_backend.py # Chatterbox Turbo TTS
│ └── luxtts_backend.py # LuxTTS backend
└── utils/
├── audio.py # Audio load/save/normalize/validate/trim
├── cache.py # Voice prompt caching (memory + disk)
├── effects.py # Audio effects engine (pedalboard)
├── chunked_tts.py # Text chunking + audio concatenation
├── progress.py # SSE progress tracking
├── tasks.py # Active task tracking
├── hf_progress.py # HuggingFace download progress tracking
├── hf_offline_patch.py # HuggingFace offline mode patch (MLX)
└── images.py # Avatar image processing
app.py # FastAPI app factory, CORS, lifecycle events
main.py # Entry point (imports app, runs uvicorn)
config.py # Data directory paths and configuration
models.py # Pydantic request/response schemas
server.py # Tauri sidecar launcher, parent-pid watchdog
routes/ # Thin HTTP handlers — validation, delegation, response formatting
services/ # Business logic, CRUD, orchestration
backends/ # TTS/STT engine implementations (MLX, PyTorch, etc.)
database/ # ORM models, session management, migrations, seed data
utils/ # Shared utilities (audio, effects, caching, progress tracking)
```
### Backend Selection
Voicebox automatically selects the best backend based on platform:
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration (4-5x faster)
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU if available, CPU fallback)
The backend is detected at runtime via `platform_detect.py`. Both backends implement the same interface, so the API remains consistent across platforms.
## API Endpoints
### Health & Info
#### `GET /`
Root endpoint with version info.
#### `GET /health`
Health check with model status.
**Response:**
```json
{
"status": "healthy",
"model_loaded": true,
"gpu_available": true,
"gpu_type": "Metal (Apple Silicon via MLX)",
"backend_type": "mlx",
"vram_used_mb": null
}
```
**Backend Types:**
- `"mlx"` - MLX backend (Apple Silicon with Metal acceleration)
- `"pytorch"` - PyTorch backend (Windows/Linux/Intel Mac)
### Voice Profiles
**Note:** The database is automatically initialized when the server starts. No manual setup required.
#### `POST /profiles`
Create a new voice profile.
**Request:**
```json
{
"name": "My Voice",
"description": "Optional description",
"language": "en"
}
```
**Response:**
```json
{
"id": "uuid",
"name": "My Voice",
"description": "Optional description",
"language": "en",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z"
}
```
#### `GET /profiles`
List all voice profiles.
#### `GET /profiles/{profile_id}`
Get a specific profile.
#### `PUT /profiles/{profile_id}`
Update a profile.
#### `DELETE /profiles/{profile_id}`
Delete a profile and all associated samples.
#### `POST /profiles/{profile_id}/samples`
Add a sample to a profile.
**Form Data:**
- `file`: Audio file (WAV, MP3, etc.)
- `reference_text`: Transcript of the audio
**Response:**
```json
{
"id": "sample-uuid",
"profile_id": "profile-uuid",
"audio_path": "/path/to/sample.wav",
"reference_text": "This is my voice"
}
```
#### `GET /profiles/{profile_id}/samples`
List all samples for a profile.
#### `DELETE /profiles/samples/{sample_id}`
Delete a specific sample.
### Generation
#### `POST /generate`
Generate speech from text using a voice profile.
**Request:**
```json
{
"profile_id": "uuid",
"text": "Hello, this is a test.",
"language": "en",
"seed": 42
}
```
**Response:**
```json
{
"id": "generation-uuid",
"profile_id": "profile-uuid",
"text": "Hello, this is a test.",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 2.5,
"seed": 42,
"created_at": "2024-01-01T00:00:00Z"
}
```
### History
#### `GET /history`
List generation history with optional filters.
**Query Parameters:**
- `profile_id` (optional): Filter by profile
- `search` (optional): Search in text content
- `limit` (default: 50): Results per page
- `offset` (default: 0): Pagination offset
#### `GET /history/{generation_id}`
Get a specific generation.
#### `DELETE /history/{generation_id}`
Delete a generation.
#### `GET /history/stats`
Get generation statistics.
**Response:**
```json
{
"total_generations": 100,
"total_duration_seconds": 250.5,
"generations_by_profile": {
"profile-uuid-1": 50,
"profile-uuid-2": 50
}
}
```
### Audio Files
#### `GET /audio/{generation_id}`
Download generated audio file.
Returns WAV file with appropriate headers.
### Transcription
#### `POST /transcribe`
Transcribe audio file to text.
**Form Data:**
- `file`: Audio file
- `language` (optional): Language hint (en or zh)
**Response:**
```json
{
"text": "Transcribed text here",
"duration": 5.5
}
```
### Model Management
#### `POST /models/load`
Manually load TTS model.
**Query Parameters:**
- `model_size`: Model size (1.7B or 0.6B)
#### `POST /models/unload`
Unload TTS model to free memory.
## Database Schema
### profiles
- `id`: UUID primary key
- `name`: Profile name (unique)
- `description`: Optional description
- `language`: Language code (en/zh)
- `created_at`: Creation timestamp
- `updated_at`: Last update timestamp
### profile_samples
- `id`: UUID primary key
- `profile_id`: Foreign key to profiles
- `audio_path`: Path to audio file
- `reference_text`: Transcript
### generations
- `id`: UUID primary key
- `profile_id`: Foreign key to profiles
- `text`: Generated text
- `language`: Language code
- `audio_path`: Path to audio file
- `duration`: Duration in seconds
- `seed`: Random seed (optional)
- `created_at`: Creation timestamp
### projects
- `id`: UUID primary key
- `name`: Project name
- `data`: JSON data
- `created_at`: Creation timestamp
- `updated_at`: Last update timestamp
## File Structure
### Request flow
```
data/
├── profiles/
│ └── {profile_id}/
│ ├── {sample_id}.wav
│ └── ...
├── generations/
│ └── {generation_id}.wav
├── cache/
│ └── {hash}.prompt
├── projects/
│ └── {project_id}.json
└── voicebox.db
HTTP request
-> routes/ (validate input, parse params)
-> services/ (business logic, database queries, orchestration)
-> backends/ (TTS/STT inference)
-> utils/ (audio processing, effects, caching)
```
## Setup
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
### 1. Install Dependencies
```bash
pip install -r requirements.txt
```
**Note:** On Apple Silicon, also install MLX dependencies for faster inference:
```bash
pip install -r requirements-mlx.txt
```
### 2. Download Models (Automatic)
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
**No manual download required!** The models will be cached locally after the first download.
Available models:
- **1.7B** (recommended): `Qwen/Qwen3-TTS-12Hz-1.7B-Base` (~4GB)
- **0.6B** (faster): `Qwen/Qwen3-TTS-12Hz-0.6B-Base` (~2GB)
**Note:** The first generation will take longer as the model downloads. Subsequent generations will use the cached model.
#### Manual Download (Optional)
If you prefer to download models manually or have limited internet during runtime:
```bash
# Install huggingface-cli
pip install huggingface_hub
# Download 1.7B model
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
# Or use Python
python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen3-TTS-12Hz-1.7B-Base')"
```
Models are cached in `~/.cache/huggingface/hub/` by default.
### 4. Run Server
```bash
# Development (local only)
python -m backend.main
# Production (allow remote access)
python -m backend.main --host 0.0.0.0 --port 8000
```
## Usage Examples
The desktop app, web client, and current development workflow use `http://localhost:17493` by default.
If you launch the backend manually with a different host or port, substitute that address in the examples below.
### Creating a Voice Profile
```bash
# 1. Create profile
curl -X POST http://localhost:17493/profiles \
-H "Content-Type: application/json" \
-d '{"name": "My Voice", "language": "en"}'
# Response: {"id": "abc-123", ...}
# 2. Add sample
curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "[email protected]" \
-F "reference_text=This is my voice sample"
```
### Generating Speech
### Key modules
**services/generation.py** -- Single `run_generation()` function that handles all three generation modes (generate, retry, regenerate). Manages model loading, voice prompt creation, chunked inference, normalization, effects, and version persistence.
**services/task_queue.py** -- Serial generation queue. Ensures only one GPU inference runs at a time. Background tasks are tracked to prevent garbage collection.
**backends/__init__.py** -- Protocol definitions (`TTSBackend`, `STTBackend`), model config registry, and factory functions. Adding a new engine means implementing the protocol and registering a config entry.
**backends/base.py** -- Shared utilities used across all engine implementations: HuggingFace cache checks, device detection, voice prompt combination, progress tracking.
**database/** -- SQLAlchemy ORM models with a re-exporting `__init__.py` for backward compatibility. Migrations run automatically on startup.
### Backend selection
The server detects the best inference backend at startup:
| Platform | Backend | Acceleration |
|----------|---------|-------------|
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
| Linux (AMD) | PyTorch | ROCm |
| Intel Arc | PyTorch | IPEX / XPU |
| Windows (any GPU) | PyTorch | DirectML |
| Any | PyTorch | CPU fallback |
Detection is handled by `utils/platform_detect.py`. Both backends implement the same `TTSBackend` protocol, so the API layer is engine-agnostic.
## API
90 endpoints organized by domain. Full interactive documentation available at `http://localhost:17493/docs` when the server is running.
| Domain | Prefix | Description |
|--------|--------|-------------|
| Health | `/`, `/health` | Server status, GPU info, filesystem checks |
| Profiles | `/profiles` | Voice profile CRUD, samples, avatars, import/export |
| Channels | `/channels` | Audio channel management and voice assignment |
| Generation | `/generate` | TTS generation, retry, regenerate, status SSE |
| History | `/history` | Generation history, search, favorites, export |
| Transcription | `/transcribe` | Whisper-based audio-to-text |
| Stories | `/stories` | Multi-track timeline editor, audio export |
| Effects | `/effects` | Effect presets, preview, version management |
| Audio | `/audio`, `/samples` | Audio file serving |
| Models | `/models` | Load, unload, download, migrate, status |
| Tasks | `/tasks`, `/cache` | Active task tracking, cache management |
| CUDA | `/backend/cuda-*` | CUDA binary download and management |
### Quick examples
```bash
# Generate speech
curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \
-d '{
"profile_id": "abc-123",
"text": "Hello, this is a test.",
"language": "en",
"seed": 42
}'
-d '{"text": "Hello world", "profile_id": "...", "language": "en"}'
# Response: {"id": "gen-456", "audio_path": "/path/to/audio.wav", ...}
# List profiles
curl http://localhost:17493/profiles
# Download audio
curl http://localhost:17493/audio/gen-456 -o output.wav
# Stream generation status (SSE)
curl http://localhost:17493/generate/{id}/status
```
### Transcribing Audio
## Data directory
```
{data_dir}/
voicebox.db # SQLite database
profiles/{id}/ # Voice samples per profile
generations/ # Generated audio files
cache/ # Voice prompt cache (memory + disk)
backends/ # Downloaded CUDA binary (if applicable)
```
Default location is the OS-specific app data directory. Override with `--data-dir` or the `VOICEBOX_DATA_DIR` environment variable.
## Code quality
Linting and formatting are enforced by [ruff](https://docs.astral.sh/ruff/), configured in `pyproject.toml`. See `STYLE_GUIDE.md` for conventions.
```bash
curl -X POST http://localhost:17493/transcribe \
-F "[email protected]" \
-F "language=en"
# Response: {"text": "Transcribed text", "duration": 5.5}
just check-python # lint + format check
just fix-python # auto-fix lint issues + reformat
just test # run pytest
```
## Advanced Features
## Dependencies
### Multi-Sample Profiles
Add multiple samples to a profile for better quality:
```bash
# Add first sample
curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "[email protected]" \
-F "reference_text=First sample"
# Add second sample
curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "[email protected]" \
-F "reference_text=Second sample"
# Generation will automatically combine all samples
```
### Voice Prompt Caching
Voice prompts are automatically cached for faster generation:
- First generation: ~5-10 seconds (creates prompt)
- Subsequent generations: ~1-2 seconds (uses cached prompt)
Cache is stored in `data/cache/` and persists across server restarts.
### VRAM Management
Models are lazy-loaded and can be manually unloaded:
```bash
# Unload TTS model
curl -X POST http://localhost:17493/models/unload
# Load specific model size
curl -X POST "http://localhost:17493/models/load?model_size=0.6B"
```
## Error Handling
All endpoints return proper HTTP status codes:
- `200 OK`: Success
- `400 Bad Request`: Invalid input
- `404 Not Found`: Resource not found
- `500 Internal Server Error`: Server error
Error responses include details:
```json
{
"detail": "Profile not found"
}
```
## Performance Tips
1. **Use multi-sample profiles** - Better quality than single sample
2. **Let caching work** - Voice prompts are cached automatically
3. **Use 0.6B model on CPU** - Faster than 1.7B with acceptable quality
4. **Use 1.7B model on GPU** - Best quality, still fast
5. **Unload Whisper after transcription** - Frees VRAM for TTS
## TODO
- [ ] WebSocket support for generation progress
- [ ] Batch generation endpoint
- [ ] Voice design (text-to-voice)
- [ ] Authentication & rate limiting
## License
See main project LICENSE.
Runtime dependencies are in `requirements.txt`. macOS-only MLX dependencies are in `requirements-mlx.txt`. Dev tools (ruff, pytest) are installed automatically by `just setup-python`.