Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-18 22:30:40 -07:00
Code Issues Packages Projects Releases Wiki Activity

docs: audit mdx docs against multi-engine backend (#484)

Browse Source 
* docs: audit mdx docs against multi-engine backend and refresh stale content

Rewrote developer-facing docs that predated the TTSBackend Protocol /
ModelConfig registry refactor (architecture, tts-generation,
model-management, transcription). Updated user-facing docs to reflect all
seven shipped engines (Qwen, Qwen CustomVoice, LuxTTS, Chatterbox,
Chatterbox Turbo, TADA, Kokoro) instead of the outdated "5 engines" claim.

Also fixes:
- Stale app identifier (com.voicebox.app → sh.voicebox.app)
- CUDA backend update flow (now two-archive split, not N-way chunks)
- Whisper model list (removed tiny, added turbo)
- Broken /development/ and /guides/ route links
- Stale just commands and install steps (missing --no-deps chatterbox/tada)
- Removed ASCII art diagrams from README and stories.mdx
- History Generation schema sync with DB model

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: add DeepWiki badge to README

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: address PR review feedback

- architecture.mdx: fix backends/ file list (remove nonexistent qwen_backend.py, rename tada_backend.py → hume_backend.py)
- model-management.mdx: Kokoro language count 9 → 8 (matches ModelConfig)
- model-management.mdx: ProgressManager path services/ → utils/
- tts-generation.mdx: ModelConfig example uses field(default_factory=...) — mutable default would raise at runtime
- tts-generation.mdx: "1080p samples" → "on CUDA" (1080p is video, not audio)
- PROJECT_STATUS.md: replace ASCII architecture diagram with prose (matches no-ASCII-art rule)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(app): guard against undefined engine in FloatingGenerateBox preset check

form.getValues('engine') returns string | undefined; Set<string>.has()
rejects undefined under strict mode. Added a truthy guard before the
preset lookup.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Jamie Pine
2026-04-18 21:06:06 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent da6070155e
commit ae91aa9a88
31 changed files with 1387 additions and 2613 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/docs/README.md
+13 -42
View File
@@ -7,61 +7,32 @@ This directory contains the documentation for Voicebox, built with [Fumadocs](ht
## Development
### Prerequisites
Install Mintlify globally using bun:
```bash
bun add -g mintlify
```
Or use the helper script:
```bash
bun run install:mintlify
```
### Running Locally
From the `docs/` directory:
```bash
bun install
bun run dev
```
This will start the Mintlify dev server.
The docs will be available at `http://localhost:3000`
The docs will be available at `http://localhost:3000`.
### Structure
```
docs/
├── mint.json # Mintlify configuration
├── custom.css # Custom styles
├── overview/ # Getting started & feature docs
├── guides/ # User guides
├── api/ # API reference
├── development/ # Developer documentation
├── logo/ # Logo assets
└── public/ # Static assets
```
- `content/docs/overview/` — user-facing guides (installation, quick start, feature walkthroughs)
- `content/docs/developer/` — architecture, backend internals, and contributor guides
- `content/docs/api-reference/` — auto-generated from the backend's OpenAPI schema
- `content/docs/index.mdx` — landing page
- `public/` — static assets (images, screenshots, videos)
### Writing Docs
- Use `.mdx` files for all documentation pages
- Follow the existing structure in `mint.json` for navigation
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
- Navigation is generated from `content/docs/meta.json` files
- Fumadocs components available: `Callout`, `Cards` / `Card`, `Tabs` / `Tab`, `Steps` / `Step`, `Accordion` / `AccordionGroup`, `Files` / `Folder` / `File`
- API reference pages under `api-reference/` are regenerated from the backend's OpenAPI schema — don't edit them by hand
## Deployment
Docs are automatically deployed when changes are pushed to the main branch.
To manually deploy:
```bash
mintlify deploy
```
## Contributing
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
Docs are automatically deployed when changes land on `main`.
docs/content/docs/TROUBLESHOOTING.md
+8 -8
View File
@@ -100,9 +100,9 @@ chmod +x voicebox-*.AppImage
**Solutions:**
1. **Rebuild server binary**
```bash
bun run build:server
just build-server
```
The build script should automatically include MLX Metal shader libraries.
The build script automatically includes MLX Metal shader libraries on Apple Silicon.
2. **Check MLX installation**
```bash
@@ -219,9 +219,9 @@ chmod +x voicebox-*.AppImage
**Solutions:**
1. **Check data directory**
- macOS: `~/Library/Application Support/voicebox/`
- Windows: `%APPDATA%/voicebox/`
- Linux: `~/.local/share/voicebox/`
- macOS: `~/Library/Application Support/sh.voicebox.app/`
- Windows: `%APPDATA%/sh.voicebox.app/`
- Linux: `~/.config/sh.voicebox.app/`
2. **Check database**
- Database: `data/voicebox.db`
@@ -266,7 +266,7 @@ chmod +x voicebox-*.AppImage
cd tauri/src-tauri
cargo clean
cd ../..
bun run build
just build
```
### API client generation fails
@@ -274,7 +274,7 @@ chmod +x voicebox-*.AppImage
**Solutions:**
1. **Start backend server**
```bash
bun run dev:server
just dev-backend
```
2. **Check OpenAPI endpoint**
@@ -284,7 +284,7 @@ chmod +x voicebox-*.AppImage
3. **Regenerate client**
```bash
bun run generate:api
just generate-api
```
## Still Having Issues?
docs/content/docs/developer/architecture.mdx
+116 -94
View File
@@ -9,9 +9,9 @@ Voicebox uses a client-server architecture with a React frontend and Python back
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
**Backend Layer:** A Python FastAPI server that hosts the REST API, runs a pluggable registry of TTS and STT engines, manages the SQLite database, and handles audio processing.
These two layers communicate via HTTP, with the frontend making API requests to the backend.
These two layers communicate via HTTP on `localhost:17493`, with the frontend making API requests to the backend. In production the backend is compiled with PyInstaller and launched as a Tauri sidecar; in development it's run manually via `uvicorn`.
## Frontend Architecture
@@ -29,43 +29,33 @@ These two layers communicate via HTTP, with the frontend making API requests to
<Files>
<Folder name="app/src" defaultOpen>
<Folder name="components">
<File name="profiles/" />
<File name="generation/" />
<File name="stories/" />
<File name="shared/" />
<File name="Profiles/" />
<File name="Generation/" />
<File name="Stories/" />
<File name="ServerSettings/" />
</Folder>
<Folder name="lib">
<File name="api/" />
<File name="constants/" />
<File name="hooks/" />
<File name="utils/" />
</Folder>
<Folder name="hooks" />
<Folder name="stores" />
</Folder>
</Files>
### State Management
```typescript
// Example: Profile store
const useProfileStore = create((set) => ({
profiles: [],
selectedProfile: null,
setProfiles: (profiles) => set({ profiles }),
selectProfile: (id) => set({ selectedProfile: id })
}))
```
## Backend Architecture
### Tech Stack
- **Framework**: FastAPI (Python 3.11+)
- **TTS Model**: Qwen3-TTS
- **Transcription**: Whisper
- **Database**: SQLite
- **Audio**: librosa, soundfile
- **TTS Engines**: Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro
- **Transcription**: Whisper (PyTorch or MLX-Whisper)
- **Inference Backends**: MLX (Apple Silicon), PyTorch (CUDA / ROCm / XPU / DirectML / CPU)
- **Database**: SQLite via SQLAlchemy
- **Audio**: librosa, soundfile, Pedalboard
### API Structure
### Layout
<Files>
<Folder name="backend" defaultOpen>
@@ -74,21 +64,31 @@ const useProfileStore = create((set) => ({
<File name="config.py" />
<File name="models.py" />
<File name="server.py" />
<File name="build_binary.py" />
<Folder name="routes">
<File name="profiles.py" />
<File name="generate.py" />
<File name="history.py" />
<File name="..." />
<File name="models.py" />
<File name="channels.py" />
</Folder>
<Folder name="services">
<File name="generation.py" />
<File name="task_queue.py" />
<File name="..." />
<File name="profiles.py" />
<File name="channels.py" />
</Folder>
<Folder name="backends">
<File name="__init__.py" />
<File name="base.py" />
<File name="..." />
<File name="pytorch_backend.py" />
<File name="mlx_backend.py" />
<File name="qwen_custom_voice_backend.py" />
<File name="luxtts_backend.py" />
<File name="chatterbox_backend.py" />
<File name="chatterbox_turbo_backend.py" />
<File name="hume_backend.py" />
<File name="kokoro_backend.py" />
</Folder>
<Folder name="database">
<File name="models.py" />
@@ -97,49 +97,75 @@ const useProfileStore = create((set) => ({
<Folder name="utils">
<File name="audio.py" />
<File name="effects.py" />
<File name="..." />
</Folder>
</Folder>
</Files>
### Request Flow
HTTP request → **routes/** (validate input, parse params) → **services/** (business logic, orchestration) → **backends/** (TTS/STT inference) → **utils/** (audio processing)
An HTTP request enters a **route handler**, which validates input and delegates to a **service** function. The service calls into the appropriate **engine backend** via the registry, which runs the actual inference. Audio post-processing runs through **utils** (trim, resample, effects).
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
Route handlers are intentionally thin — they validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
### Multi-Engine Registry
The backend is designed so that adding a new TTS engine only requires touching the `backends/` directory and the central registry. There is no per-engine branching in routes or services.
- **`TTSBackend` Protocol** (`backends/__init__.py`) — defines the contract every engine implements: `load_model`, `create_voice_prompt`, `combine_voice_prompts`, `generate`, `unload_model`, `is_loaded`, `_get_model_path`.
- **`ModelConfig` dataclass** — central metadata record for each model variant: `model_name`, `display_name`, `engine`, `hf_repo_id`, `size_mb`, `needs_trim`, `languages`, `supports_instruct`, etc.
- **`TTS_ENGINES` dict** — maps engine name (`"qwen"`, `"kokoro"`, etc.) to display name.
- **`get_tts_backend_for_engine(engine)`** — thread-safe factory that lazily instantiates and caches the backend for an engine using double-checked locking.
Shipped engines:
| Engine key | Display name | Profile type |
|------------|--------------|--------------|
| `qwen` | Qwen TTS | Cloned |
| `qwen_custom_voice` | Qwen CustomVoice | Preset |
| `luxtts` | LuxTTS | Cloned |
| `chatterbox` | Chatterbox TTS | Cloned |
| `chatterbox_turbo` | Chatterbox Turbo | Cloned |
| `tada` | TADA | Cloned |
| `kokoro` | Kokoro | Preset |
See [TTS Engines](/developer/tts-engines) for the full contract and integration phases, and [PROJECT_STATUS.md](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for candidates under evaluation.
### Key Modules
- **app.py** — FastAPI app factory, CORS, lifecycle events
- **main.py** — Entry point (imports app, runs uvicorn)
- **server.py** — Tauri sidecar launcher, parent-pid watchdog
- **services/generation.py** — Single function handling all generation modes (generate, retry, regenerate)
- **services/task_queue.py** — Serial generation queue for GPU inference
- **backends/__init__.py** — Protocol definitions and backend factory
- **backends/base.py** — Shared utilities across all engine implementations
- **`app.py`** — FastAPI app factory, CORS, lifecycle events
- **`main.py`** — Entry point (imports app, runs uvicorn)
- **`server.py`** — Tauri sidecar launcher, parent-pid watchdog, frozen-build environment setup
- **`services/generation.py`** — Single function handling all generation modes (generate, retry, regenerate)
- **`services/task_queue.py`** — Serial generation queue for GPU inference
- **`backends/__init__.py`** — Protocol definitions, `ModelConfig` registry, and engine factory
- **`backends/base.py`** — Shared utilities across all engine implementations (device selection, progress tracking, output trimming)
### Backend Selection
### Inference Backend Selection
The server detects the best inference backend at startup:
The server detects the best inference backend at startup and uses it for all engines that support it:
| Platform | Backend | Acceleration |
|----------|---------|-------------|
|----------|---------|--------------|
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
| Windows / Linux (NVIDIA) | PyTorch | CUDA (cu128) |
| Linux (AMD) | PyTorch | ROCm |
| Intel Arc | PyTorch | IPEX / XPU |
| Windows (any GPU) | PyTorch | DirectML |
| Windows / Linux (Intel Arc) | PyTorch | XPU (IPEX) |
| Windows (other GPU) | PyTorch | DirectML |
| Any | PyTorch | CPU fallback |
See [GPU Acceleration](/overview/gpu-acceleration) for platform-specific notes and manual overrides.
### Data Model
The database uses three main tables:
Core tables (see `backend/database/models.py`):
**Profile Table:** Stores voice profiles with fields for id, name, and language.
- **`profiles`** — Voice profiles with `voice_type` discriminator (`cloned` | `preset` | `designed`), `preset_engine`, `preset_voice_id`, and `default_engine`.
- **`profile_samples`** — Reference audio clips + transcripts for cloned profiles. Empty for preset profiles.
- **`generations`** — Generated audio with text, engine, model, language, seed, and duration.
- **`generation_versions`** — Processed variants of a generation with different effects chains applied.
- **`audio_channels`** + **`channel_device_mappings`** + **`profile_channel_mappings`** — Multi-output routing.
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
See [Voice Profiles](/developer/voice-profiles) and [Effects Pipeline](/developer/effects-pipeline) for details.
## Desktop App (Tauri)
@@ -148,6 +174,7 @@ The database uses three main tables:
<Files>
<Folder name="tauri/src-tauri" defaultOpen>
<File name="Cargo.toml" />
<File name="tauri.conf.json" />
<File name="src/" />
<Folder name="binaries" />
</Folder>
@@ -158,82 +185,74 @@ The database uses three main tables:
- Launch Python backend as sidecar process
- Native file dialogs
- System tray integration
- Auto-updates
- OS-specific features
- Auto-updates (Tauri updater + custom CUDA backend swap)
- Parent-PID watchdog so the backend exits if the app crashes
## Build Process
### Development
```bash
# Frontend (Vite dev server)
cd app && bun run dev
# Backend (manual start)
cd backend && uvicorn main:app --reload
# Desktop app (connects to manual backend)
bun run dev
just dev # Starts backend + Tauri app
just dev-web # Starts backend + web app (no Tauri)
just dev-backend # Backend only
just dev-frontend # Tauri app only (backend must be running)
```
### Production
```bash
# Build everything (server binary + Tauri app)
bun run build
# Or build separately:
# 1. Build server binary (PyInstaller)
bun run build:server
# 2. Build Tauri app (includes server)
cd tauri && bun run tauri build
just build # CPU server binary + Tauri installer
just build-local # CPU + CUDA binaries + Tauri installer (Windows)
just build-server # Server binary only
just build-tauri # Tauri app only
```
See [Building](/developer/building) for what PyInstaller does and how the CUDA binary is split and packaged separately.
## Data Flow
### Generation Flow
When a user generates speech, the data flows through the following stages:
1. **User Input** - User enters text in a React component
2. **State Update** - Text is stored in Zustand state
3. **API Request** - React Query mutation triggers an API call via fetch
4. **Backend Processing** - FastAPI endpoint receives the request
5. **TTS Generation** - Qwen3-TTS model generates the audio
6. **Storage** - Audio file is saved to disk and a database record is created
7. **Response** - Backend returns the audio URL
8. **Cache Update** - React Query updates its cache with the response
9. **UI Update** - Component re-renders with new data
10. **Playback** - User can play the generated audio
1. **User Input** — text entered in a React component, engine + profile selected
2. **State Update** — Zustand generation form store records the request
3. **API Request** — React Query mutation hits `POST /generate`
4. **Route** — `routes/generate.py` validates input, dispatches to `services/generation.py`
5. **Voice Prompt** — the service creates or retrieves a cached voice prompt via the engine's backend
6. **Queue** — `services/task_queue.py` serializes generation to avoid GPU contention
7. **Inference** — the engine backend runs `generate()` and returns audio + sample rate
8. **Post-process** — optional trim (for engines that need it), effects chain applied per generation version
9. **Storage** — audio written to the generations directory, metadata saved to SQLite
10. **Response** — backend returns the generation record; frontend updates React Query cache and plays audio
## Performance Considerations
### Frontend
- **Code splitting** - Lazy load routes
- **Memoization** - React.memo for heavy components
- **Virtual scrolling** - For large lists
- **Debouncing** - Search and input handling
- **Code splitting** — lazy-load routes
- **Memoization** — `React.memo` for heavy components
- **Virtual scrolling** — for large lists
- **Debouncing** — search and input handling
### Backend
- **Async operations** - All I/O is async
- **Model caching** - Keep TTS model in memory
- **Voice prompt caching** - Reuse embeddings
- **Connection pooling** - Database connections
- **Async I/O** — all I/O is async; inference runs in `asyncio.to_thread`
- **Serial task queue** — avoids multiple engines fighting for the GPU
- **Voice prompt caching** — engine-specific, keyed by audio hash + reference text
- **Model pinning** — only one model per engine loaded at a time; switching unloads the previous one
- **Per-engine backend cache** — engines are only instantiated once per process
## Security
### Current
- Local-only by default
- Local-only by default (bound to `127.0.0.1:17493`)
- No authentication (localhost trust)
- File system sandboxing via Tauri
### Planned
- API key authentication
- API key authentication for remote mode
- User accounts
- Rate limiting
- HTTPS support
@@ -248,17 +267,20 @@ When a user generates speech, the data flows through the following stages:
### Remote Mode
- Backend on separate machine
- Frontend connects via HTTP
- Shared infrastructure possible
- Backend on a separate machine (Docker or bare host)
- Frontend (desktop or web) connects over HTTP
- See [Remote Mode](/overview/remote-mode) and [Docker](/overview/docker)
## Next Steps
<Cards>
<Card title="Development Setup" href="/development/setup">
<Card title="Development Setup" href="/developer/setup">
Set up your dev environment
</Card>
<Card title="Contributing" href="/development/contributing">
<Card title="TTS Engines" href="/developer/tts-engines">
How to add a new engine
</Card>
<Card title="Contributing" href="/developer/contributing">
Contribute to Voicebox
</Card>
</Cards>
docs/content/docs/developer/autoupdater.mdx
+33 -41
View File
@@ -145,64 +145,55 @@ The updater only works in production Tauri builds. It doesn't run during `just d
## CUDA Backend Updates
The CUDA-enabled backend is distributed separately from the main app due to its large size (~2.43 GB). Unlike the Tauri auto-updater, this uses a custom download system built into the Python backend.
The CUDA-enabled backend is distributed separately from the main app because bundling CUDA would bloat the installer by several gigabytes for users who don't have an NVIDIA GPU. Unlike the Tauri auto-updater, the CUDA backend uses a custom download system built into the Python server.
**Size comparison:**
- Standard app bundle: ~410 MB
- CUDA backend binary: ~2.43 GB (6× larger)
**Size comparison (approximate):**
- Standard CPU bundle (in the installer): ~200–400 MB
- CUDA server core: ~945 MB (versioned with each Voicebox release)
- CUDA libs (NVIDIA runtime DLLs): ~1.7 GB (versioned independently, cached across upgrades)
### Why Split?
### Two-archive split
GitHub Releases has file size limits, and the CUDA-enabled `voicebox-server` binary is too large to include in the main Tauri bundle. Instead:
Since v0.4, the CUDA binary is packaged as **two archives** instead of one:
- **Standard release**: Includes CPU-only backend (~50MB)
- **CUDA release**: Split into multiple parts and downloaded on-demand by users who need GPU acceleration
- **Server core** (`voicebox-server-cuda.tar.gz`) — the Python server + PyTorch code, changes every release.
- **CUDA libs** (`cuda-libs-cu128-v1.tar.gz`) — the heavy NVIDIA CUDA/cuDNN DLLs, only re-downloaded when the CUDA toolkit major version changes.
This means most Voicebox upgrades only re-download the ~945 MB server core, not the full ~2.5 GB bundle.
### Download Process
When a user clicks "Enable CUDA" in the settings:
When a user clicks "Install CUDA backend" in Settings → GPU:
1. **Manifest Fetch** - Backend fetches `{version}/voicebox-server-cuda.manifest` from GitHub Releases
2. **Part Download** - Downloads each split part sequentially (e.g., `voicebox-server-cuda.part1`, `.part2`, etc.)
3. **Assembly** - Concatenates parts into a single binary
4. **Verification** - SHA-256 checksum verification (optional, if `.sha256` file exists)
5. **Placement** - Binary moved to `{data_dir}/backends/voicebox-server-cuda.exe`
6. **Restart** - Backend must restart to use the CUDA binary
1. **Server-core archive** — Downloaded from GitHub Releases and extracted.
2. **CUDA libs archive** — Downloaded separately (or reused if the installed version still matches).
3. **Verification** — SHA-256 checksum verification for integrity.
4. **Placement** — Extracted into `{data_dir}/backends/cuda/`.
5. **Restart** — The Voicebox server restarts and swaps in the CUDA backend.
### Auto-Update on Startup
On server startup, `check_and_update_cuda_binary()` compares the installed CUDA binary version with the app version:
```python
# backend/services/cuda.py
cuda_version = get_cuda_binary_version() # runs `voicebox-server-cuda --version`
current_version = __version__
if cuda_version != current_version:
await download_cuda_binary() # Auto-download in background
```
If versions mismatch, the backend automatically downloads the matching CUDA binary version without user intervention.
On startup, the backend compares the installed CUDA server-core version with the current app version. If they differ, the core archive is pulled in the background. If the libs version pinned by the new release also differs (rare — e.g. on a cu126 → cu128 bump), the user is prompted to confirm the larger download.
### Storage Location
Downloaded CUDA binaries are stored in the app's data directory:
Downloaded CUDA binaries live in the app's data directory:
```
{data_dir}/
backends/
voicebox-server-cuda.exe # Windows
voicebox-server-cuda # macOS/Linux
{data_dir}/backends/cuda/
voicebox-server-cuda.exe # Windows
voicebox-server-cuda # macOS/Linux
<NVIDIA CUDA runtime DLLs>
```
### API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/backend/cuda-status` | GET | Check if CUDA binary available/active |
| `/backend/download-cuda` | POST | Start download |
| `/backend/cuda-status` | GET | Check if the CUDA backend is available/active and which versions are installed |
| `/backend/download-cuda` | POST | Trigger server-core + libs download |
| `/backend/cuda-progress` | GET | SSE stream of download progress |
| `/backend/cuda` | DELETE | Remove downloaded binary |
| `/backend/cuda` | DELETE | Remove the downloaded CUDA backend |
### Progress Tracking
@@ -212,15 +203,16 @@ Downloads report progress via Server-Sent Events (SSE):
GET /backend/cuda-progress
event: progress
data: {"current": 52428800, "total": 104857600, "filename": "Downloading CUDA backend (2/4)", "status": "downloading"}
data: {"current": 52428800, "total": 945000000, "filename": "voicebox-server-cuda.tar.gz", "status": "downloading"}
```
The frontend subscribes to this endpoint to show real-time download progress in the UI.
The frontend subscribes to this endpoint to show real-time progress, including which archive (server core vs libs) is currently downloading.
### Release Artifacts
For each release, these CUDA-related files are uploaded to GitHub:
For each CUDA-capable release, these files are uploaded to GitHub:
- `voicebox-server-cuda.manifest` - List of split part filenames
- `voicebox-server-cuda.part1` through `voicebox-server-cuda.partN` - Binary chunks
- `voicebox-server-cuda.sha256` - SHA-256 checksum for integrity verification
- `voicebox-server-cuda.tar.gz` — server-core archive
- `voicebox-server-cuda.tar.gz.sha256` — checksum
- `cuda-libs-cu128-v1.tar.gz` — CUDA runtime libs (only when the libs version bumps)
- `cuda-libs-cu128-v1.tar.gz.sha256` — checksum
docs/content/docs/developer/contributing.mdx
+18 -10
View File
@@ -17,9 +17,10 @@ Thank you for your interest in contributing to Voicebox! This guide will help yo
Before you start contributing, make sure you have:
1. **Read the documentation** to understand how Voicebox works
2. **Set up your development environment** - see [Development Setup](/development/setup)
2. **Set up your development environment** — see [Development Setup](/developer/setup)
3. **Explored the codebase** to understand the project structure
4. **Checked existing issues** to see if someone else is working on something similar
4. **Checked [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md)** — the living engineering roadmap that tracks prioritized tasks (Tier 1 → 3), architectural bottlenecks, and candidate TTS engines under evaluation (including why some are backlogged)
5. **Checked existing issues** to see if someone else is working on something similar
## Ways to Contribute
@@ -173,10 +174,15 @@ When creating a pull request:
<File name="stores/" />
</Folder>
<Folder name="backend">
<File name="app.py" />
<File name="main.py" />
<File name="tts.py" />
<File name="database.py" />
<File name="server.py" />
<File name="models.py" />
<Folder name="routes" />
<Folder name="services" />
<Folder name="backends" />
<Folder name="database" />
<Folder name="utils" />
</Folder>
<Folder name="tauri">
<File name="src-tauri/" />
@@ -197,9 +203,10 @@ When creating a pull request:
### New Features
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
- Check [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) and the [roadmap](https://github.com/jamiepine/voicebox#roadmap) before proposing work — the status doc lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
- Discuss major features in an issue first
- Keep features focused and well-scoped
- Adding a new TTS engine? See [TTS Engines](/developer/tts-engines) for the phased workflow
### Documentation
@@ -253,7 +260,7 @@ When adding new API endpoints:
<Step title="Regenerate Client">
```bash
bun run generate:api
just generate-api
```
This updates the TypeScript client with type-safe bindings.
@@ -263,7 +270,7 @@ When adding new API endpoints:
The API documentation is automatically generated from the OpenAPI schema. Ensure your endpoint has proper docstrings and type hints, then regenerate the docs:
```bash
bun run generate:api
just generate-api
```
</Step>
</Steps>
@@ -324,8 +331,9 @@ By contributing, you agree that your contributions will be licensed under the MI
If you have questions:
1. Check the [documentation](/overview/introduction)
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
3. Open a new issue or discussion
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
2. Read [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for current engineering priorities
3. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
4. Open a new issue or discussion
5. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
Thank you for contributing to Voicebox! 🎉
docs/content/docs/developer/history.mdx
+18 -6
View File
@@ -15,17 +15,24 @@ The history module tracks all generated audio, providing a searchable record of
class Generation(Base):
__tablename__ = "generations"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
text = Column(Text, nullable=False)
language = Column(String, default="en")
audio_path = Column(String, nullable=False)
duration = Column(Float, nullable=False)
audio_path = Column(String, nullable=True)
duration = Column(Float, nullable=True)
seed = Column(Integer)
instruct = Column(Text)
created_at = Column(DateTime)
engine = Column(String, default="qwen")
model_size = Column(String, nullable=True)
status = Column(String, default="completed") # pending | completed | failed
error = Column(Text, nullable=True)
is_favorited = Column(Boolean, default=False)
created_at = Column(DateTime, default=datetime.utcnow)
```
Each generation can also have multiple **generation versions** — processed variants with different effects chains applied. The original (`clean`) version plus any number of processed versions live in a separate `generation_versions` table. See [Effects Pipeline](/developer/effects-pipeline).
## File Storage
Generated audio is stored in:
@@ -230,7 +237,12 @@ GET /history?profile_id=uuid&search=hello&limit=50&offset=0
"duration": 1.5,
"seed": 42,
"instruct": null,
"created_at": "2024-01-15T10:30:00Z"
"engine": "qwen",
"model_size": "1.7B",
"status": "completed",
"error": null,
"is_favorited": false,
"created_at": "2026-04-18T10:30:00Z"
}
],
"total": 150
docs/content/docs/developer/model-management.mdx
+152 -294
View File
@@ -1,341 +1,199 @@
---
title: "Model Management"
description: "How model downloading, loading, and status tracking works in Voicebox"
description: "How model downloading, loading, and status tracking works across all engines"
---
## Overview
Voicebox manages two types of models:
Voicebox manages two categories of models:
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
**TTS Models** — Seven engines covering zero-shot cloning and preset voices. Each engine may have one or more size variants.
**ASR Models:** Whisper for transcription (tiny through large).
**ASR Models** — Whisper for transcription. Five sizes, plus MLX-Whisper on Apple Silicon for ~8× faster transcription.
Models are downloaded from HuggingFace Hub on first use and cached locally.
Every model is described by a `ModelConfig` entry in `backend/backends/__init__.py`. Models are downloaded from HuggingFace Hub on first use and cached in the platform-standard HF cache.
## Available Models
## Available TTS Models
### TTS Models
| Model | Engine | HuggingFace Repo | Size | VRAM | Languages |
|-------|--------|------------------|------|------|-----------|
| **Qwen TTS 1.7B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 3.5 GB | ~6 GB | 10 |
| **Qwen TTS 0.6B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | 1.2 GB | ~2 GB | 10 |
| **Qwen CustomVoice 1.7B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 3.5 GB | ~6 GB | 10 |
| **Qwen CustomVoice 0.6B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | 1.2 GB | ~2 GB | 10 |
| **LuxTTS** | `luxtts` | `YatharthS/LuxTTS` | 300 MB | ~1 GB | English |
| **Chatterbox Multilingual** | `chatterbox` | `ResembleAI/chatterbox` | 3.2 GB | ~3 GB | 23 |
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 8 |
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
### Whisper Models
## Available Whisper Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
| base | `openai/whisper-base` | ~300MB | ~1GB |
| small | `openai/whisper-small` | ~500MB | ~2GB |
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
| large | `openai/whisper-large` | ~3GB | ~10GB |
| Model | HuggingFace Repo | Size |
|-------|------------------|------|
| **Whisper Base** | `openai/whisper-base` | ~300 MB |
| **Whisper Small** | `openai/whisper-small` | ~500 MB |
| **Whisper Medium** | `openai/whisper-medium` | ~1.5 GB |
| **Whisper Large** | `openai/whisper-large-v3` | ~3 GB |
| **Whisper Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB |
On Apple Silicon, MLX-Whisper is preferred automatically — see [Transcription](/developer/transcription).
## Model Storage
Models are cached in the HuggingFace cache directory:
Models live in the platform HuggingFace cache:
<Files>
<Folder name="~/.cache/huggingface/hub" defaultOpen>
<File name="models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/" />
<File name="models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/" />
<File name="models--openai--whisper-base/" />
</Folder>
</Files>
| Platform | Path |
|----------|------|
| macOS | `~/.cache/huggingface/hub/` |
| Linux | `~/.cache/huggingface/hub/` |
| Windows | `%USERPROFILE%\.cache\huggingface\hub\` |
| Docker | `/home/voicebox/.cache/huggingface/hub` (volume-mounted) |
Set `VOICEBOX_MODELS_DIR` to override.
## Progress Tracking
### Progress Manager
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
Tracks download progress across all models:
**`ProgressManager`** (`backend/utils/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
**SSE endpoint** — `GET /models/progress/{model_name}` streams updates until `status` is `complete` or `error`.
```python
class ProgressManager:
def __init__(self):
self._progress = {} # model_name -> progress_info
def update_progress(
self,
model_name: str,
current: int,
total: int,
filename: str,
status: str,
):
self._progress[model_name] = {
"current": current,
"total": total,
"filename": filename,
"status": status, # downloading, complete, error
"updated_at": datetime.utcnow(),
}
def get_progress(self, model_name: str) -> Optional[dict]:
return self._progress.get(model_name)
```
### HuggingFace Progress Callback
Hooks into HuggingFace's download system:
```python
class HFProgressTracker:
def __init__(self, callback):
self.callback = callback
@contextmanager
def patch_download(self):
"""Context manager to intercept HF downloads."""
original_download = hf_hub_download
def patched_download(*args, **kwargs):
# Intercept progress
result = original_download(*args, **kwargs)
self.callback(progress_info)
return result
# Apply patch
with patch('huggingface_hub.hf_hub_download', patched_download):
yield
```
### Server-Sent Events (SSE)
Progress is streamed to the frontend:
```python
@app.get("/models/progress/{model_name}")
async def get_model_progress(model_name: str):
async def event_generator():
while True:
progress = progress_manager.get_progress(model_name)
if progress:
yield f"data: {json.dumps(progress)}\n\n"
if progress and progress["status"] in ["complete", "error"]:
break
await asyncio.sleep(0.5)
return StreamingResponse(
event_generator(),
media_type="text/event-stream"
)
```
## Task Manager
Tracks active downloads and generations:
```python
class TaskManager:
def __init__(self):
self._active_downloads = {}
self._active_generations = {}
def start_download(self, model_name: str):
self._active_downloads[model_name] = {
"status": "downloading",
"started_at": datetime.utcnow(),
}
def complete_download(self, model_name: str):
if model_name in self._active_downloads:
del self._active_downloads[model_name]
def get_active_tasks(self) -> dict:
return {
"downloads": list(self._active_downloads.values()),
"generations": list(self._active_generations.values()),
}
# Frontend
const eventSource = new EventSource(`/models/progress/${modelName}`);
eventSource.onmessage = (event) => {
const { current, total, status } = JSON.parse(event.data);
updateProgressBar(current / total);
if (status === "complete") eventSource.close();
};
```
## Model Status
Check which models are downloaded and loaded:
```python
@app.get("/models/status")
async def get_model_status() -> ModelStatusListResponse:
models = []
# Check TTS models
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
downloaded = is_model_downloaded(hf_id)
loaded = tts_model._current_model_size == size
models.append(ModelStatus(
model_name=f"qwen-tts-{size}",
display_name=f"Qwen3-TTS {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=loaded,
))
# Check Whisper models
for size in ["tiny", "base", "small", "medium", "large"]:
hf_id = f"openai/whisper-{size}"
downloaded = is_model_downloaded(hf_id)
models.append(ModelStatus(
model_name=f"whisper-{size}",
display_name=f"Whisper {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=False, # Whisper is loaded on-demand
))
return ModelStatusListResponse(models=models)
```
## Manual Model Operations
### Load Model
```python
@app.post("/models/load")
async def load_model(model_size: str = "1.7B"):
tts_model = get_tts_model()
await tts_model.load_model_async(model_size)
return {"status": "loaded", "model_size": model_size}
```
### Unload Model
```python
@app.post("/models/unload")
async def unload_model():
tts_model = get_tts_model()
tts_model.unload_model()
return {"status": "unloaded"}
```
### Trigger Download
```python
@app.post("/models/download")
async def trigger_model_download(request: ModelDownloadRequest):
# This triggers the download in background
# Progress is tracked via /models/progress/{model_name}
if request.model_name.startswith("qwen-tts"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_tts_model(size))
elif request.model_name.startswith("whisper"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_whisper_model(size))
return {"status": "downloading"}
```
### Delete Model
```python
@app.delete("/models/{model_name}")
async def delete_model(model_name: str):
# Find and delete from HuggingFace cache
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
for model_dir in model_dirs:
shutil.rmtree(model_dir)
return {"status": "deleted"}
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Get status of all models |
| POST | `/models/load` | Load TTS model |
| POST | `/models/unload` | Unload TTS model |
| POST | `/models/download` | Trigger model download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete downloaded model |
| GET | `/tasks/active` | Get active downloads/generations |
## Response Schemas
### ModelStatus
`GET /models/status` returns every registered model's current state:
```json
{
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen3-TTS 1.7B",
"downloaded": true,
"size_mb": 3400,
"loaded": true
}
```
### ActiveTasksResponse
```json
{
"downloads": [
"models": [
{
"model_name": "whisper-medium",
"status": "downloading",
"started_at": "2024-01-15T10:30:00Z"
}
],
"generations": [
{
"task_id": "uuid",
"profile_id": "uuid",
"text_preview": "Hello world...",
"started_at": "2024-01-15T10:30:00Z"
}
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen TTS 1.7B",
"engine": "qwen",
"downloaded": true,
"size_mb": 3500,
"loaded": true
},
...
]
}
```
## Frontend Integration
The handler iterates `get_all_model_configs()` and calls `check_model_loaded(config)` for each entry, so new engines appear automatically once they're registered in `ModelConfig`.
### Progress Display
## Manual Model Operations
```typescript
// Subscribe to download progress via SSE
const eventSource = new EventSource(`/models/progress/${modelName}`);
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Status of every registered model |
| POST | `/models/load` | Load a TTS model into memory |
| POST | `/models/unload` | Unload a TTS model from memory |
| POST | `/models/download` | Trigger a background download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete a downloaded model from cache |
eventSource.onmessage = (event) => {
const progress = JSON.parse(event.data);
updateProgressBar(progress.current / progress.total);
if (progress.status === 'complete') {
eventSource.close();
}
};
### Load
```http
POST /models/load
{
"model_name": "qwen-tts-1.7B"
}
```
### Model Status UI
The route looks up the config, dispatches to `get_model_load_func(config)`, and returns once the model is ready.
```typescript
// Fetch model status
const { data: models } = useQuery({
queryKey: ['models', 'status'],
queryFn: () => api.getModelStatus(),
});
### Unload
// Display download/load buttons based on status
models.map(model => (
<ModelCard
name={model.display_name}
downloaded={model.downloaded}
loaded={model.loaded}
onDownload={() => triggerDownload(model.model_name)}
onLoad={() => loadModel(model.model_name)}
/>
));
```http
POST /models/unload
{
"model_name": "chatterbox-tts"
}
```
Calls `unload_model_by_config(config)`, which routes to the right backend's `unload_model()` and frees GPU memory.
### Download
```http
POST /models/download
{
"model_name": "kokoro"
}
```
Fires off an async download task. Progress is available via the SSE endpoint. Download is triggered automatically on first generation, so this is only needed for pre-warming.
## Preset Voice Seeding
For engines that use preset voices (Kokoro, Qwen CustomVoice), the backend auto-creates a voice profile per preset voice after the model is downloaded. This is driven by `seed_preset_profiles(engine)` in `backend/services/profiles.py`, called from the models route once download completes.
Preset profiles have:
- `voice_type = "preset"`
- `preset_engine` = engine name (`"kokoro"`, `"qwen_custom_voice"`)
- `preset_voice_id` = engine-specific voice ID (`"am_adam"`, `"f000001"`, etc.)
- No `profile_samples` rows — no audio to store
See [Voice Profiles](/developer/voice-profiles) for the schema.
## Adding a New Model
To add a new size variant of an existing engine, just add another `ModelConfig`:
```python
ModelConfig(
model_name="qwen-tts-3B",
display_name="Qwen TTS 3B",
engine="qwen",
hf_repo_id="Qwen/Qwen3-TTS-12Hz-3B-Base",
model_size="3B",
size_mb=7000,
languages=["zh", "en", ...],
),
```
The frontend picks it up via `/models/status`; download/load flow works without further changes.
Adding a whole new engine is a bigger lift — see [TTS Engines](/developer/tts-engines) for the full phased workflow.
## Error Handling
| Error | Cause | Solution |
|-------|-------|----------|
| Download failed | Network issue | Retry download |
| OOM on load | Model too large | Use smaller model |
| Model not found | Cache corrupted | Re-download |
| Slow download | HF rate limit | Wait and retry |
| Error | Cause | Fix |
|-------|-------|-----|
| Download failed | Network / HF rate limit | Retry |
| OOM on load | Not enough VRAM | Use a smaller variant, unload other engines |
| Model not found | Corrupt cache | Re-download via `/models/download` |
| Stuck progress bar in frozen build | `huggingface_hub` tqdm silenced | `HFProgressTracker` force-enables the internal counter |
| GPU architecture unsupported | PyTorch wheel doesn't target your GPU | See [GPU Acceleration](/overview/gpu-acceleration) |
## Next Steps
<Cards>
<Card title="TTS Generation" href="/developer/tts-generation">
How generation flows through the registry
</Card>
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new engine end-to-end
</Card>
<Card title="Transcription" href="/developer/transcription">
Whisper and MLX-Whisper integration
</Card>
</Cards>
docs/content/docs/developer/setup.mdx
+75 -24
View File
@@ -59,24 +59,64 @@ Ensure you have these installed:
## Just Commands
Run `just --list` to see all available commands:
Run `just --list` to see all available commands. Highlights:
### Setup
| Command | Description |
|---------|-------------|
| `just setup` | Full setup (Python venv + JS deps + dev sidecar). Detects Apple Silicon for MLX and NVIDIA/Intel Arc on Windows for accelerated PyTorch. |
| `just setup-python` | Python venv + dependencies only |
| `just setup-js` | `bun install` only |
### Development
| Command | Description |
|---------|-------------|
| `just dev` | Start backend + Tauri desktop app (reuses a running backend if one exists) |
| `just dev-web` | Start backend + web app (no Tauri/Rust build) |
| `just dev-backend` | Backend only |
| `just dev-frontend` | Tauri app only (backend must already be running) |
| `just kill` | Stop all dev processes |
### Build
| Command | Description |
|---------|-------------|
| `just build` | CPU server binary + Tauri installer |
| `just build-local` | **Windows:** CPU + CUDA server binaries + Tauri installer |
| `just build-server` | CPU server binary only |
| `just build-server-cuda` | **Windows:** CUDA server binary only, placed in `%APPDATA%/sh.voicebox.app/backends/cuda` for local testing |
| `just build-tauri` | Tauri app only |
| `just build-web` | Web app only |
### Quality
| Command | Description |
|---------|-------------|
| `just check` | Lint + format + typecheck (Biome + ruff) |
| `just fix` | Auto-fix lint + format issues |
| `just lint` / `just format` | Lint or format only |
| `just test` | Run Python tests (pytest) |
| `just test-models` | End-to-end generation against every TTS engine using the frozen binary |
### Database
| Command | Description |
|---------|-------------|
| `just setup` | Full setup (Python venv + JS deps) |
| `just dev` | Start backend + desktop app |
| `just dev-web` | Start backend + web app (no Tauri) |
| `just dev-backend` | Start backend only |
| `just dev-frontend` | Start desktop app only (backend must be running) |
| `just build` | Build desktop app for production |
| `just build-web` | Build web app for production |
| `just check` | Run all checks (JS + Python lint + format) |
| `just fix` | Fix lint + format issues |
| `just test` | Run Python tests |
| `just db-init` | Initialize SQLite database |
| `just db-reset` | Reset database (delete + reinit) |
| `just clean` | Clean build artifacts |
| `just clean-all` | Nuclear clean (includes node_modules) |
| `just db-reset` | Delete and reinitialize the database |
### Utilities
| Command | Description |
|---------|-------------|
| `just generate-api` | Generate TypeScript API client from the backend's OpenAPI schema |
| `just docs` | Open `http://localhost:17493/docs` in your browser |
| `just logs` | Tail backend logs |
| `just clean` | Remove build artifacts |
| `just clean-python` | Remove the Python venv + `__pycache__` |
| `just clean-all` | Nuclear clean (includes all `node_modules`) |
## Project Structure
@@ -133,10 +173,12 @@ HTTP request → **routes/** (validate input) → **services/** (business logic)
## Model Downloads
Models are automatically downloaded from HuggingFace Hub on first use:
Models are automatically downloaded from HuggingFace Hub on first use, with live progress streamed to the UI:
- **Whisper** (transcription): Auto-downloads on first transcription
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
- **Whisper** (transcription) — auto-downloads on first transcription
- **TTS engines** — auto-download on first generation. Sizes range from 82 M (Kokoro, ~350 MB) to 3 B (TADA, ~8 GB)
See [Model Management](/developer/model-management) for the full list.
<Callout type="warn">
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
@@ -150,7 +192,7 @@ After starting the backend server, generate the TypeScript API client:
just generate-api
```
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`.
## Manual Setup (Advanced)
@@ -186,8 +228,17 @@ pip install -r requirements.txt
# Apple Silicon: install MLX dependencies
pip install -r requirements-mlx.txt
# Install Qwen3-TTS
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
pip install --no-deps chatterbox-tts
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
pip install --no-deps hume-tada
# Install Qwen3-TTS from source
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
# PyInstaller and linting tools
pip install pyinstaller ruff pytest pytest-asyncio
```
### 3. Start Development
@@ -208,17 +259,17 @@ bun run tauri dev
## Next Steps
<Cards>
<Card title="Architecture" href="/development/architecture">
<Card title="Architecture" href="/developer/architecture">
Understand the system architecture
</Card>
<Card title="Contributing" href="/development/contributing">
<Card title="Contributing" href="/developer/contributing">
Read the contribution guidelines
</Card>
<Card title="Building" href="/development/building">
<Card title="Building" href="/developer/building">
Learn how to build production releases
</Card>
<Card title="API Reference" href="/api-reference">
Explore the REST API
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new TTS engine end-to-end
</Card>
</Cards>
docs/content/docs/developer/stories.mdx
+3 -22
View File
@@ -50,34 +50,15 @@ class StoryItem(Base):
### Start Time
`start_time_ms` defines when an item begins on the timeline:
```
Timeline (ms): 0----1000----2000----3000----4000
Item 1: [======]
Item 2: [==========]
Item 3: [====]
```
`start_time_ms` is the absolute position on the timeline where an item begins playing. Items on the same track cannot overlap; items on different tracks can.
### Tracks
Multiple tracks allow overlapping audio:
```
Track 0: [Item 1] [Item 3]
Track 1: [Item 2]
```
A `track` is an integer (0-indexed) that identifies the horizontal row an item sits on. Audio on separate tracks plays concurrently, so tracks are the primary way to layer multiple voices or sound effects.
### Trimming
Trim values cut audio from the start or end without destroying the original:
```
Original: [=========AUDIO=========]
trim_start: ^^
trim_end: ^^
Result: [=====AUDIO=====]
```
`trim_start_ms` and `trim_end_ms` hide the leading/trailing portions of the source generation without modifying the underlying audio file. The effective playback length is `generation.duration * 1000 - trim_start_ms - trim_end_ms`. Trimming is non-destructive — the same generation can be trimmed differently in different stories.
## Core Operations
docs/content/docs/developer/transcription.mdx
+94 -233
View File
@@ -5,250 +5,91 @@ description: "How Whisper-based audio transcription works in Voicebox"
## Overview
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
Voicebox uses OpenAI's Whisper for automatic speech recognition (ASR). Transcription powers two flows:
1. **Reference-text auto-fill** — when a user records or uploads a voice sample, the backend transcribes it and populates the `reference_text` field so cloning can use it.
2. **On-demand transcription** — a user-facing `/transcribe` endpoint for arbitrary audio.
On Apple Silicon, the transcription path runs through **MLX-Whisper** (from `mlx-audio`) for ~8× faster inference than PyTorch. Everywhere else it runs through PyTorch's `transformers` Whisper.
## Architecture
The transcription system is built around the `WhisperModel` class:
**Model Loading:** Lazy loading with HuggingFace Hub download.
**Audio Processing:** Resampling and preprocessing for Whisper.
**Inference:** Running transcription with optional language hints.
## WhisperModel Class
Transcription is wired through the same backend abstraction as TTS. The `STTBackend` protocol lives in `backend/backends/__init__.py`:
```python
class WhisperModel:
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
@runtime_checkable
class STTBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
model_size: Optional[str] = None,
) -> str: ...
def unload_model(self) -> None: ...
def is_loaded(self) -> bool: ...
```
### Model Sizes
Two implementations ship today:
| Size | Parameters | VRAM | Speed | Quality |
|------|------------|------|-------|---------|
| tiny | 39M | ~1GB | Fastest | Basic |
| base | 74M | ~1GB | Fast | Good |
| small | 244M | ~2GB | Medium | Better |
| medium | 769M | ~5GB | Slow | High |
| large | 1550M | ~10GB | Slowest | Best |
- **`MLXSTTBackend`** (`backends/mlx_backend.py`) — uses `mlx_audio.stt.load()`. Default on Apple Silicon.
- **`PyTorchSTTBackend`** (`backends/pytorch_backend.py`) — uses `transformers.WhisperForConditionalGeneration`. Default everywhere else.
Default is `base` for balance of speed and quality.
`get_stt_backend()` picks the right one based on `get_backend_type()`. `backend/services/transcribe.py` is a thin wrapper that delegates to the backend.
## Model Sizes
Five Whisper variants are registered in `ModelConfig`:
| Model | HuggingFace Repo | Size | Notes |
|-------|------------------|------|-------|
| **Base** | `openai/whisper-base` | ~300 MB | Default; fast, decent quality |
| **Small** | `openai/whisper-small` | ~500 MB | Better quality, still fast |
| **Medium** | `openai/whisper-medium` | ~1.5 GB | High quality |
| **Large** | `openai/whisper-large-v3` | ~3 GB | Best quality, slow on CPU |
| **Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB | Large-tier quality, ~5× faster than Large |
The `tiny` model is **not** exposed — the quality gap to `base` wasn't worth the download.
`Turbo` + MLX-Whisper on Apple Silicon dropped user-facing transcription latency from ~20s to ~2-3s in v0.1.10.
## Language Hints
Whisper can auto-detect language, but providing a hint improves accuracy on short clips:
```python
text = await backend.transcribe(audio_path, language="en")
```
Accepted language codes are the standard Whisper set (99+ languages). The frontend typically passes the profile's language if available, or lets Whisper detect otherwise.
## Model Loading
Models are downloaded from HuggingFace Hub:
Both backends are lazy: the model is loaded on first use and cached in memory. Switching sizes unloads the previous model.
On MLX, the model is loaded via `mlx_audio.stt.load(hf_repo)`. On PyTorch, via:
```python
def load_model(self, model_size: Optional[str] = None):
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
# Track download progress
progress_manager = get_progress_manager()
task_manager = get_task_manager()
task_manager.start_download(f"whisper-{model_size}")
# Load processor and model
with tracker.patch_download():
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
# Mark complete
progress_manager.mark_complete(f"whisper-{model_size}")
task_manager.complete_download(f"whisper-{model_size}")
WhisperProcessor.from_pretrained(hf_repo)
WhisperForConditionalGeneration.from_pretrained(hf_repo).to(device)
```
### Async Loading
Like TTS, loading runs in a thread pool:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self.model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Transcription
### Basic Transcription
```python
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
await self.load_model_async()
def _transcribe_sync():
# Load and resample to 16kHz (Whisper requirement)
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language hint if provided
forced_decoder_ids = None
if language:
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language,
task="transcribe",
)
# Generate
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
return await asyncio.to_thread(_transcribe_sync)
```
### Supported Languages
Whisper supports 99+ languages. Common ones in Voicebox:
| Code | Language |
|------|----------|
| en | English |
| zh | Chinese |
| ja | Japanese |
| ko | Korean |
| de | German |
| fr | French |
| ru | Russian |
| pt | Portuguese |
| es | Spanish |
| it | Italian |
### Language Detection
When no language is specified, Whisper auto-detects:
```python
# Without language hint - auto-detect
transcription = await whisper.transcribe(audio_path)
# With language hint - more accurate for short clips
transcription = await whisper.transcribe(audio_path, language="en")
```
## Transcription with Timestamps
For advanced use cases, word-level timestamps are available:
```python
async def transcribe_with_timestamps(
self,
audio_path: str,
language: Optional[str] = None,
) -> List[Dict[str, any]]:
await self.load_model_async()
def _transcribe_timestamps_sync():
audio, sr = load_audio(audio_path, sample_rate=16000)
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
return_timestamps=True,
)
# Parse timestamps
return [
{
"text": transcription,
"start": 0.0,
"end": len(audio) / sr,
}
]
return await asyncio.to_thread(_transcribe_timestamps_sync)
```
## Memory Management
### Unloading
Free memory when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Global Instance
A singleton pattern manages the model:
```python
_whisper_model: Optional[WhisperModel] = None
def get_whisper_model() -> WhisperModel:
global _whisper_model
if _whisper_model is None:
_whisper_model = WhisperModel()
return _whisper_model
```
Both load paths use `model_load_progress()` from `backends/base.py` so the frontend sees live download progress on the first use.
## Audio Preprocessing
### Resampling
Whisper expects mono 16 kHz audio. The audio utility in `backend/utils/audio.py` handles resampling and format conversion transparently:
Whisper requires 16kHz audio:
- **Formats:** WAV, MP3, FLAC, OGG, M4A (via soundfile / librosa)
- **Target:** mono, 16 kHz, float32
```python
audio, sr = load_audio(audio_path, sample_rate=16000)
```
### Format Support
The `load_audio` utility handles:
- WAV
- MP3
- FLAC
- OGG
- M4A
All formats are converted to mono 16kHz.
Files longer than Whisper's 30-second window are handled by the underlying library's chunking logic — no explicit splitting in Voicebox code.
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/transcribe` | Transcribe audio file |
| POST | `/transcribe` | Transcribe an uploaded audio file |
### Request
@@ -259,7 +100,8 @@ POST /transcribe
Content-Type: multipart/form-data
file: <audio_file>
language: en (optional)
language: en # optional
model_size: base # optional (default: "base")
```
### Response
@@ -275,25 +117,44 @@ language: en (optional)
### Reference Text for Voice Cloning
1. User records audio sample
2. Audio is sent to `/transcribe`
3. Transcription becomes `reference_text`
4. Both are added to voice profile
Adding a voice sample triggers transcription automatically:
1. User uploads or records audio.
2. The backend writes the audio file and calls `/transcribe` internally (or the frontend calls it separately).
3. The returned text becomes `reference_text` on the new `profile_samples` row.
4. Cloning engines that need reference text (Chatterbox, TADA, etc.) read it from there.
### Quality Tips
- Provide language hint for short audio
- Use clean audio with minimal noise
- Longer audio (>5s) improves accuracy
- Consider `small` or `medium` model for better quality
- Provide a language hint for short clips (under 5 seconds) — auto-detection is unreliable on little audio.
- Use Turbo or Large for noisy audio — Base can hallucinate on hard inputs.
- Prefer clean audio; transcription errors become reference-text errors, which become cloning errors.
## Memory Management
`unload_model()` drops the model reference and clears the CUDA cache if applicable. `/models/unload` wires this up for manual control.
A singleton per backend is returned by `get_stt_backend()` — multiple callers share one Whisper instance.
## Error Handling
Common issues:
| Error | Cause | Solution |
|-------|-------|----------|
| Model not found | First run, download failed | Retry with network |
| OOM | Model too large | Use smaller model |
| Empty result | No speech detected | Check audio has speech |
| Wrong language | Auto-detect failed | Provide language hint |
| Model not found | First run + network failure | Retry; check connectivity |
| OOM on load | Large model on low-VRAM GPU | Switch to Small or Turbo |
| Empty result | No speech in audio | Confirm input has voice; check trim |
| Wrong language | Auto-detect misfired | Pass `language` hint |
## Next Steps
<Cards>
<Card title="Model Management" href="/developer/model-management">
Download / load / unload any model
</Card>
<Card title="Voice Profiles" href="/developer/voice-profiles">
How reference text is stored alongside samples
</Card>
<Card title="GPU Acceleration" href="/overview/gpu-acceleration">
Platform-specific acceleration including MLX-Whisper
</Card>
</Cards>
docs/content/docs/developer/tts-engines.mdx
+17 -10
View File
@@ -604,18 +604,25 @@ for name, mod in [("dac", types.ModuleType("dac")),
- Do NOT use `@torch.jit.script` in the shim (see above)
- Only reimplement what the model actually uses — check the import chain carefully
## Upcoming Engines
## Candidate Engines
Based on the current model landscape, these are candidates for future integration:
The [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) file is the canonical, living list of candidates under evaluation — including why some have been backlogged (e.g. VoxCPM, which is effectively CUDA-only upstream).
| Model | Languages | Size | Key Features | Status |
|-------|-----------|------|--------------|--------|
| **CosyVoice2-0.5B** | Multilingual | ~500MB | Instruct support (`inference_instruct2()`) | Ready |
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
At a glance, current top candidates:
| Model | Tier | Size | Cross-platform? | Key Features |
|-------|------|------|-----------------|--------------|
| **MOSS-TTS-Nano** | 1 | 0.1 B | Yes (CPU realtime) | 48 kHz stereo, Apache 2.0, released 2026-04-13 |
| **Voxtral TTS** | 2 | 4 B | Likely | `mistralai/Voxtral-4B-TTS-2603` — presets + cloning |
| **VibeVoice** | 2 | ~500 M | Yes | Podcast-style multi-speaker dialogue |
| **Dia2** | 3 | TBD | TBD | Successor to the original Dia |
| **Fish Audio S2 Pro** | 3 | Medium | Yes | Word-level control via inline text |
**Backlogged:**
- **VoxCPM** (2B, Apache 2.0) — CUDA ≥12 required upstream; MPS broken in issues #232/#248; CPU path rejected by maintainers (#256). Keep watching for a PR that relaxes the device requirement.
Update `PROJECT_STATUS.md` when you pick one up or mark one as shipped/backlogged.
## Implementation Checklist
docs/content/docs/developer/tts-generation.mdx
+167 -199
View File
@@ -1,283 +1,251 @@
---
title: "TTS Generation"
description: "How text-to-speech generation works in Voicebox"
description: "How text-to-speech generation works across Voicebox's multi-engine backend"
---
## Overview
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
Voicebox ships seven TTS engines — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, TADA, and Kokoro — behind a single `TTSBackend` Protocol. All of them expose the same async interface so the routes and services don't need per-engine branching.
## Architecture
This page covers how generation flows through that abstraction. For the step-by-step guide to adding a new engine, see [TTS Engines](/developer/tts-engines).
The TTS system is built around the `TTSModel` class which manages:
## The `TTSBackend` Protocol
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
**Voice Prompts:** Converting reference audio into embeddings.
**Generation:** Synthesizing speech from text using voice prompts.
## TTSModel Class
Every engine implements the same contract (defined in `backend/backends/__init__.py`):
```python
class TTSModel:
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device() # cuda, mps, or cpu
@runtime_checkable
class TTSBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def create_voice_prompt(
self, audio_path: str, reference_text: str, use_cache: bool = True
) -> Tuple[dict, bool]: ...
async def combine_voice_prompts(
self, audio_paths: List[str], reference_texts: List[str]
) -> Tuple[np.ndarray, str]: ...
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]: ...
def unload_model(self) -> None: ...
def is_loaded(self) -> bool: ...
```
### Device Selection
## The `ModelConfig` Registry
The model automatically selects the best available device:
Each downloadable model variant is described by a `ModelConfig` dataclass:
```python
def _get_device(self) -> str:
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return "cpu" # MPS can have issues, use CPU for stability
return "cpu"
@dataclass
class ModelConfig:
model_name: str # "luxtts", "qwen-tts-1.7B", "kokoro"
display_name: str # "LuxTTS (Fast, CPU-friendly)"
engine: str # "luxtts", "qwen", "kokoro"
hf_repo_id: str # "YatharthS/LuxTTS"
model_size: str = "default"
size_mb: int = 0
needs_trim: bool = False
supports_instruct: bool = False
languages: list[str] = field(default_factory=lambda: ["en"])
```
## Model Loading
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
Models are downloaded from HuggingFace Hub on first use:
- `get_all_model_configs()` — every TTS + STT variant
- `get_tts_model_configs()` — only TTS variants
- `get_model_config(model_name)` — lookup by name
- `engine_needs_trim(engine)` — whether output should run through `trim_tts_output()`
- `load_engine_model(engine, model_size)` — downloads + loads, handles engines with multiple sizes
- `get_tts_backend_for_engine(engine)` — thread-safe backend factory with double-checked locking
The `TTS_ENGINES` dict is the canonical list of shipped engine names:
```python
def load_model(self, model_size: Optional[str] = None):
# Model IDs on HuggingFace Hub
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
# Load with progress tracking
with tracker.patch_download():
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16, # float32 on CPU
)
TTS_ENGINES = {
"qwen": "Qwen TTS",
"qwen_custom_voice": "Qwen CustomVoice",
"luxtts": "LuxTTS",
"chatterbox": "Chatterbox TTS",
"chatterbox_turbo": "Chatterbox Turbo",
"tada": "TADA",
"kokoro": "Kokoro",
}
```
### Async Loading
## Voice Prompt Patterns
Loading runs in a thread pool to avoid blocking the event loop:
Each engine chooses how to represent a voice in the prompt dict returned from `create_voice_prompt()`. Three patterns are in use today:
**Pattern A — Pre-computed tensors** (Qwen3-TTS, LuxTTS)
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self._current_model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
encoded = model.encode_prompt(audio_path)
return encoded, False # (prompt_dict, was_cached)
```
## Voice Prompt Creation
Voice prompts are created from reference audio and cached for reuse:
**Pattern B — Deferred file paths** (Chatterbox, Chatterbox Turbo, TADA)
```python
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
await self.load_model_async()
# Check cache
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached = get_cached_voice_prompt(cache_key)
if cached:
return cached, True
# Create prompt (blocking, run in thread pool)
voice_prompt = await asyncio.to_thread(
self.model.create_voice_clone_prompt,
ref_audio=audio_path,
ref_text=reference_text,
)
# Cache the result
cache_voice_prompt(cache_key, voice_prompt)
return voice_prompt, False
return {"ref_audio": audio_path, "ref_text": reference_text}, False
```
### Combining Multiple Samples
When a profile has multiple samples, they're combined:
**Pattern C — Preset voice pointer** (Kokoro, Qwen CustomVoice)
```python
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate and normalize
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
return {
"voice_type": "preset",
"preset_engine": "kokoro",
"preset_voice_id": "am_adam",
}, False
```
## Speech Generation
Pattern C is the shape used for profiles where `voice_type == "preset"` — there's no cloning step; the engine looks up a baked-in voice by ID.
The core generation function:
Engines that cache voice prompts prefix their cache keys to avoid collisions:
```python
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
await self.load_model_async()
def _generate_sync():
# Set seed for reproducibility
if seed is not None:
torch.manual_seed(seed)
# Generate audio
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct, # Natural language delivery control
)
return wavs[0], sample_rate
# Run in thread pool
return await asyncio.to_thread(_generate_sync)
cache_key = f"{engine}_{hash(audio_path, reference_text)}"
```
### Instruct Feature
## Device Selection
The `instruct` parameter allows natural language control over speech delivery:
Engines pick their device through `get_torch_device()` in `backends/base.py`, which layers:
1. `VOICEBOX_FORCE_CPU` environment override
2. CUDA (if compiled and available)
3. XPU (Intel Arc via IPEX)
4. MPS (Apple Silicon) — **only for engines that support it**; some (Chatterbox, older Qwen paths) skip MPS and fall back to CPU due to upstream operator gaps
5. CPU
Qwen TTS uses MLX directly on Apple Silicon instead of going through PyTorch — see `mlx_backend.py`.
## Generation Flow
The request path from frontend to audio file:
1. **Request** — `POST /generate` with `GenerationRequest`:
```json
{
"profile_id": "uuid",
"text": "...",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "warm, slightly amused",
"engine": "qwen",
"max_chunk_chars": 800
}
```
The `engine` field is validated against the regex `^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$`.
2. **Route** — `routes/generate.py` validates input and delegates.
3. **Service** — `services/generation.py` fetches the profile, resolves the engine backend via `get_tts_backend_for_engine(engine)`, and ensures the model is loaded (downloading it on first use with live progress).
4. **Voice prompt** — the service calls `create_voice_prompt()` (or the preset equivalent). For cloned profiles with multiple samples, it calls `combine_voice_prompts()` first to merge reference audio.
5. **Queue** — the request is serialized through `services/task_queue.py` to avoid multiple generations fighting for the GPU.
6. **Inference** — the engine's `generate()` returns `(audio_array, sample_rate)`.
7. **Post-process** — if `engine_needs_trim(engine)` is True, `trim_tts_output()` strips trailing silence. Effects chains (if any) are applied per generation version, not the clean version.
8. **Persist** — audio is written to the generations directory, a row is inserted into the `generations` table, and the response includes the generation metadata.
## Chunking for Long Text
Text longer than `max_chunk_chars` (default 800, range 100–5000) is split at sentence boundaries, generated in sequence, and crossfaded together. The chunking behavior is engine-agnostic — it lives in the service layer, not in individual backends.
## Instruct Mode
Two engines support natural-language delivery control via the `instruct` kwarg:
- **Qwen CustomVoice** — `supports_instruct=True`, fully wired to the model's instruct head.
- **Qwen Base** — silently drops the instruct text (`supports_instruct=False`). The frontend hides the instruct input for Base profiles.
```python
# Examples:
instruct = "Speak slowly and clearly"
instruct = "Sound excited and enthusiastic"
instruct = "Whisper softly"
# Good instruct prompts:
"warm and conversational, slight smile"
"whisper, intimate and close"
"authoritative, broadcast quality"
```
## Caching Strategy
Voice prompts are cached to avoid recomputation:
```python
def get_cache_key(audio_path: str, reference_text: str) -> str:
"""Generate cache key from audio hash and text."""
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
return f"{audio_hash}_{text_hash}"
```
Cache is stored in `data/cache/voice_prompts/`.
Other engines ignore `instruct` entirely.
## Memory Management
### Unloading Models
Free VRAM/RAM when not needed:
Models are loaded lazily on first use and kept in memory. Switching between model sizes (e.g. Qwen 1.7B ↔ 0.6B) unloads the previous model before loading the new one to avoid OOM:
```python
def unload_model(self):
if self.model is not None:
del self.model
self.model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Model Switching
When switching between model sizes (1.7B ↔ 0.6B):
```python
# Unload existing model first
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
```
## Generation Flow
1. **Request** → Validate text and profile ID
2. **Profile** → Load profile samples from database
3. **Voice Prompt** → Create or retrieve cached prompt
4. **Generate** → Run TTS inference
5. **Save** → Write audio to generations directory
6. **Record** → Create history entry in database
7. **Response** → Return audio path and metadata
The model management API (`/models/load`, `/models/unload`) lets users free VRAM manually — see [Model Management](/developer/model-management).
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/generate` | Generate speech from text |
| GET | `/audio/{id}` | Serve generated audio file |
| GET | `/audio/{generation_id}` | Serve generated audio file |
### Request Schema
```json
{
"profile_id": "uuid",
"text": "Text to synthesize",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "Speak clearly"
}
```
### Response Schema
### Response schema
```json
{
"id": "generation_uuid",
"profile_id": "profile_uuid",
"text": "Text to synthesize",
"text": "...",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 3.5,
"seed": 42,
"instruct": "Speak clearly",
"created_at": "2024-01-15T10:30:00Z"
"engine": "qwen",
"model_size": "1.7B",
"instruct": "...",
"created_at": "2026-04-18T10:30:00Z"
}
```
## Performance Considerations
### GPU Acceleration
- **CUDA** is the fastest backend for every PyTorch-based engine. Apple Silicon MLX is competitive with CUDA for Qwen TTS specifically.
- **Serial queue** — only one generation runs at a time per process; concurrent requests are queued.
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
- CUDA provides fastest inference
- MPS (Apple Silicon) has stability issues, uses CPU fallback
- CPU inference is slower but always works
### Per-engine VRAM (approximate, on CUDA)
### Batch Size
| Engine | VRAM |
|--------|------|
| Kokoro | ~150 MB |
| LuxTTS | ~1 GB |
| Chatterbox Turbo | ~1.5 GB |
| Qwen 0.6B / Qwen CustomVoice 0.6B | ~2 GB |
| Chatterbox Multilingual | ~3 GB |
| Qwen 1.7B / Qwen CustomVoice 1.7B | ~6 GB |
| TADA 1B | ~4 GB |
| TADA 3B | ~8 GB |
Currently generates one utterance at a time. For long texts, consider:
- Splitting into sentences
- Sequential generation
- Concatenating results
## Next Steps
### Memory Usage
| Model | VRAM/RAM Required |
|-------|-------------------|
| 0.6B | ~2GB |
| 1.7B | ~6GB |
<Cards>
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new engine — full phased workflow
</Card>
<Card title="Model Management" href="/developer/model-management">
Downloading, loading, and unloading models
</Card>
<Card title="Voice Profiles" href="/developer/voice-profiles">
Cloned vs preset profile schema
</Card>
</Cards>
docs/content/docs/developer/voice-profiles.mdx
+2 -2
View File
@@ -179,8 +179,8 @@ async def create_voice_prompt_for_profile(
Reference audio is validated before being accepted:
- **Duration:** 3-30 seconds recommended
- **Format:** WAV, MP3, FLAC, OGG supported
- **Sample Rate:** Resampled to 24kHz
- **Format:** WAV, MP3, FLAC, OGG, M4A supported
- **Sample Rate:** Engine-specific — the audio utility resamples to whatever the active engine expects (Whisper uses 16 kHz, most TTS engines use 24 kHz, LuxTTS outputs 48 kHz). Resampling happens on the fly; the stored sample retains its original rate.
- **Channels:** Converted to mono if stereo
## Export/Import
docs/content/docs/index.mdx
+4 -3
View File
@@ -3,15 +3,16 @@ title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
docs/content/docs/overview/creating-voice-profiles.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ Use this when you want to replicate a specific person's voice from a recording.
10-30 seconds of clear speech, minimal background noise. See [Voice Cloning](/overview/voice-cloning) for the engine catalog.
</Step>
<Step title="Create Profile">
**Profiles** → **+ New Profile** → choose a cloning engine (Qwen3-TTS, Chatterbox, LuxTTS, or TADA)
**Profiles** → **+ New Profile** → choose a cloning engine (Qwen3-TTS, Chatterbox Multilingual, Chatterbox Turbo, LuxTTS, or TADA)
</Step>
<Step title="Upload or Record Sample">
Drag in an audio file, or record directly with the in-app recorder
docs/content/docs/overview/generation-history.mdx
+3 -3
View File
@@ -79,9 +79,9 @@ Drag generations to the Stories Editor timeline.
History is stored locally:
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
- **Linux**: `~/.config/com.voicebox.app/data/`
- **macOS**: `~/Library/Application Support/sh.voicebox.app/data/`
- **Windows**: `%APPDATA%/sh.voicebox.app/data/`
- **Linux**: `~/.config/sh.voicebox.app/data/`
<Callout type="warn">
Deleting the data directory will remove all history. Export important files first.
docs/content/docs/overview/installation.mdx
+4 -4
View File
@@ -68,11 +68,11 @@ Voicebox is available for macOS and Windows, with Linux builds coming soon.
When you launch Voicebox for the first time:
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
1. **Model Download** — The TTS engine you generate with first will download its model automatically. Sizes range from ~350 MB (Kokoro) to ~8 GB (TADA 3B). Most users start with Qwen 1.7B (~3.5 GB).
2. **Data Directory** — Voice profiles and generated audio are stored in:
- macOS: `~/Library/Application Support/com.voicebox.app/`
- Windows: `%APPDATA%/com.voicebox.app/`
- Linux: `~/.config/com.voicebox.app/`
- macOS: `~/Library/Application Support/sh.voicebox.app/`
- Windows: `%APPDATA%/sh.voicebox.app/`
- Linux: `~/.config/sh.voicebox.app/`
3. **Backend Server** — The bundled Python server starts automatically
docs/content/docs/overview/introduction.mdx
+15 -12
View File
@@ -5,13 +5,14 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** -- models and voice data stay on your machine
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
@@ -20,15 +21,17 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## TTS Engines
Five engines with different strengths, switchable per-generation:
Seven engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions |
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
| Engine | Profile Type | Languages | Strengths |
|--------|--------------|-----------|-----------|
| **Qwen3-TTS** (0.6B / 1.7B) | Cloned | 10 | High-quality multilingual cloning |
| **Qwen CustomVoice** (0.6B / 1.7B) | Preset (9 voices) | 10 | Natural-language delivery control (tone, emotion, pace) |
| **LuxTTS** | Cloned | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | Cloned | 23 | Broadest language coverage |
| **Chatterbox Turbo** | Cloned | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model -- 700s+ coherent audio |
| **Kokoro** | Preset (50 voices) | 9 | 82M parameters, CPU realtime, lowest VRAM of any engine |
## GPU Support
@@ -57,7 +60,7 @@ Five engines with different strengths, switchable per-generation:
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
docs/content/docs/overview/remote-mode.mdx
+9 -1
View File
@@ -31,8 +31,16 @@ In Remote Mode, the Voicebox desktop app (running on your local machine) communi
# Install Python dependencies
pip install -r requirements.txt
# Engines with incompatible transitive pins — install with --no-deps
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
# Qwen3-TTS from source
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
Or just run `just setup` from the repo root, which handles all of this.
</Step>
<Step title="Start the Server">
@@ -135,4 +143,4 @@ Expected performance on various GPUs:
## Troubleshooting
See the [Troubleshooting Guide](/guides/troubleshooting#remote-mode-issues) for common remote mode issues.
See the [Troubleshooting Guide](/overview/troubleshooting) for common issues.
docs/content/docs/overview/troubleshooting.mdx
+23 -14
View File
@@ -66,7 +66,7 @@ Windows SmartScreen may warn that the app is unrecognized.
```bash
# macOS/Linux
chmod +x ~/Library/Application\ Support/com.voicebox.app/backend/voicebox-server
chmod +x ~/Library/Application\ Support/sh.voicebox.app/backend/voicebox-server
```
</Accordion>
@@ -75,12 +75,12 @@ Windows SmartScreen may warn that the app is unrecognized.
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
tail -f ~/Library/Application\ Support/sh.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
type %APPDATA%\sh.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
@@ -105,12 +105,13 @@ Windows SmartScreen may warn that the app is unrecognized.
- Progress indicator stuck at "Loading model..."
**Explanation:**
This is expected behavior. The first generation downloads the Qwen3-TTS model (~2-4GB) and initializes it.
This is expected behavior. The first generation downloads the selected TTS engine's model and initializes it. Sizes range from 350 MB (Kokoro) to 8 GB (TADA 3B).
**Solution:**
- Wait for the initial download to complete
- Subsequent generations will be much faster
- Wait for the initial download to complete (progress is shown in Settings → Models)
- Subsequent generations reuse the cached model and are much faster
- Check your internet connection
- For low-bandwidth setups, start with Kokoro (~350 MB) or LuxTTS (~300 MB)
### Poor Voice Quality
@@ -202,7 +203,7 @@ This is expected behavior. The first generation downloads the Qwen3-TTS model (~
### Backend Won't Start in Dev Mode
**Symptoms:**
- `bun run dev:server` fails
- `just dev-backend` or `just dev` fails
- Import errors or module not found
**Solutions:**
@@ -233,11 +234,19 @@ This is expected behavior. The first generation downloads the Qwen3-TTS model (~
</Accordion>
<Accordion title="Dependencies">
Reinstall dependencies:
Reinstall dependencies — easiest via `just`:
```bash
just setup
```
Or manually:
```bash
cd backend
pip install -r requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Accordion>
@@ -310,8 +319,8 @@ bun run tauri build
- Delete the lock file:
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-wal
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db-wal
```
### Corrupted Database
@@ -328,10 +337,10 @@ bun run tauri build
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db
# Windows
del %APPDATA%\com.voicebox.app\data\voicebox.db
del %APPDATA%\sh.voicebox.app\data\voicebox.db
```
Restart the app to create a fresh database.
@@ -357,10 +366,10 @@ Restart the app to create a fresh database.
- Different voice output
**Solutions:**
Clear the model cache and re-download:
Clear the model cache and re-download. Replace the `Qwen*` glob with the engine org prefix for other engines (`ResembleAI*` for Chatterbox, `HumeAI*` for TADA, `hexgrad*` for Kokoro, etc.) or use `DELETE /models/{name}` via the API.
```bash
# macOS
# macOS / Linux
rm -rf ~/.cache/huggingface/hub/models--Qwen*
# Windows