mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-15 12:50:42 -07:00
ed54347e81
* fix runaway MLX Qwen audio chunks * test: tighten runaway retry coverage --------- Co-authored-by: huanghua01 <[email protected]>
254 lines
8.9 KiB
Plaintext
254 lines
8.9 KiB
Plaintext
---
|
||
title: "TTS Generation"
|
||
description: "How text-to-speech generation works across Voicebox's multi-engine backend"
|
||
---
|
||
|
||
## Overview
|
||
|
||
Voicebox ships seven TTS engines — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, TADA, and Kokoro — behind a single `TTSBackend` Protocol. All of them expose the same async interface so the routes and services don't need per-engine branching.
|
||
|
||
This page covers how generation flows through that abstraction. For the step-by-step guide to adding a new engine, see [TTS Engines](/developer/tts-engines).
|
||
|
||
## The `TTSBackend` Protocol
|
||
|
||
Every engine implements the same contract (defined in `backend/backends/__init__.py`):
|
||
|
||
```python
|
||
@runtime_checkable
|
||
class TTSBackend(Protocol):
|
||
async def load_model(self, model_size: str) -> None: ...
|
||
async def create_voice_prompt(
|
||
self, audio_path: str, reference_text: str, use_cache: bool = True
|
||
) -> Tuple[dict, bool]: ...
|
||
async def combine_voice_prompts(
|
||
self, audio_paths: List[str], reference_texts: List[str]
|
||
) -> Tuple[np.ndarray, str]: ...
|
||
async def generate(
|
||
self,
|
||
text: str,
|
||
voice_prompt: dict,
|
||
language: str = "en",
|
||
seed: Optional[int] = None,
|
||
instruct: Optional[str] = None,
|
||
) -> Tuple[np.ndarray, int]: ...
|
||
def unload_model(self) -> None: ...
|
||
def is_loaded(self) -> bool: ...
|
||
```
|
||
|
||
## The `ModelConfig` Registry
|
||
|
||
Each downloadable model variant is described by a `ModelConfig` dataclass:
|
||
|
||
```python
|
||
@dataclass
|
||
class ModelConfig:
|
||
model_name: str # "luxtts", "qwen-tts-1.7B", "kokoro"
|
||
display_name: str # "LuxTTS (Fast, CPU-friendly)"
|
||
engine: str # "luxtts", "qwen", "kokoro"
|
||
hf_repo_id: str # "YatharthS/LuxTTS"
|
||
model_size: str = "default"
|
||
size_mb: int = 0
|
||
needs_trim: bool = False
|
||
retries_runaway: bool = False
|
||
supports_instruct: bool = False
|
||
languages: list[str] = field(default_factory=lambda: ["en"])
|
||
```
|
||
|
||
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
|
||
|
||
- `get_all_model_configs()` — every TTS + STT variant
|
||
- `get_tts_model_configs()` — only TTS variants
|
||
- `get_model_config(model_name)` — lookup by name
|
||
- `engine_needs_trim(engine)` — whether output should run through `trim_tts_output()`
|
||
- `engine_retries_runaway(engine)` — whether unstable output should be retried as smaller chunks
|
||
- `load_engine_model(engine, model_size)` — downloads + loads, handles engines with multiple sizes
|
||
- `get_tts_backend_for_engine(engine)` — thread-safe backend factory with double-checked locking
|
||
|
||
The `TTS_ENGINES` dict is the canonical list of shipped engine names:
|
||
|
||
```python
|
||
TTS_ENGINES = {
|
||
"qwen": "Qwen TTS",
|
||
"qwen_custom_voice": "Qwen CustomVoice",
|
||
"luxtts": "LuxTTS",
|
||
"chatterbox": "Chatterbox TTS",
|
||
"chatterbox_turbo": "Chatterbox Turbo",
|
||
"tada": "TADA",
|
||
"kokoro": "Kokoro",
|
||
}
|
||
```
|
||
|
||
## Voice Prompt Patterns
|
||
|
||
Each engine chooses how to represent a voice in the prompt dict returned from `create_voice_prompt()`. Three patterns are in use today:
|
||
|
||
**Pattern A — Pre-computed tensors** (Qwen3-TTS, LuxTTS)
|
||
|
||
```python
|
||
encoded = model.encode_prompt(audio_path)
|
||
return encoded, False # (prompt_dict, was_cached)
|
||
```
|
||
|
||
**Pattern B — Deferred file paths** (Chatterbox, Chatterbox Turbo, TADA)
|
||
|
||
```python
|
||
return {"ref_audio": audio_path, "ref_text": reference_text}, False
|
||
```
|
||
|
||
**Pattern C — Preset voice pointer** (Kokoro, Qwen CustomVoice)
|
||
|
||
```python
|
||
return {
|
||
"voice_type": "preset",
|
||
"preset_engine": "kokoro",
|
||
"preset_voice_id": "am_adam",
|
||
}, False
|
||
```
|
||
|
||
Pattern C is the shape used for profiles where `voice_type == "preset"` — there's no cloning step; the engine looks up a baked-in voice by ID.
|
||
|
||
Engines that cache voice prompts prefix their cache keys to avoid collisions:
|
||
|
||
```python
|
||
cache_key = f"{engine}_{hash(audio_path, reference_text)}"
|
||
```
|
||
|
||
## Device Selection
|
||
|
||
Engines pick their device through `get_torch_device()` in `backends/base.py`, which layers:
|
||
|
||
1. `VOICEBOX_FORCE_CPU` environment override
|
||
2. CUDA (if compiled and available)
|
||
3. XPU (Intel Arc via IPEX)
|
||
4. MPS (Apple Silicon) — **only for engines that support it**; some (Chatterbox, older Qwen paths) skip MPS and fall back to CPU due to upstream operator gaps
|
||
5. CPU
|
||
|
||
Qwen TTS uses MLX directly on Apple Silicon instead of going through PyTorch — see `mlx_backend.py`.
|
||
|
||
## Generation Flow
|
||
|
||
The request path from frontend to audio file:
|
||
|
||
1. **Request** — `POST /generate` with `GenerationRequest`:
|
||
```json
|
||
{
|
||
"profile_id": "uuid",
|
||
"text": "...",
|
||
"language": "en",
|
||
"seed": 42,
|
||
"model_size": "1.7B",
|
||
"instruct": "warm, slightly amused",
|
||
"engine": "qwen",
|
||
"max_chunk_chars": 800
|
||
}
|
||
```
|
||
The `engine` field is validated against the regex `^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$`.
|
||
|
||
2. **Route** — `routes/generate.py` validates input and delegates.
|
||
|
||
3. **Service** — `services/generation.py` fetches the profile, resolves the engine backend via `get_tts_backend_for_engine(engine)`, and ensures the model is loaded (downloading it on first use with live progress).
|
||
|
||
4. **Voice prompt** — the service calls `create_voice_prompt()` (or the preset equivalent). For cloned profiles with multiple samples, it calls `combine_voice_prompts()` first to merge reference audio.
|
||
|
||
5. **Queue** — the request is serialized through `services/task_queue.py` to avoid multiple generations fighting for the GPU.
|
||
|
||
6. **Inference** — the engine's `generate()` returns `(audio_array, sample_rate)`.
|
||
|
||
7. **Validate and post-process** — engines with `retries_runaway=True` retry unstable output as smaller chunks. If `engine_needs_trim(engine)` is True, `trim_tts_output()` strips trailing silence. Effects chains (if any) are applied per generation version, not the clean version.
|
||
|
||
8. **Persist** — audio is written to the generations directory, a row is inserted into the `generations` table, and the response includes the generation metadata.
|
||
|
||
## Chunking for Long Text
|
||
|
||
Text longer than `max_chunk_chars` (default 800, range 100–5000) is split at sentence boundaries, generated in sequence, and crossfaded together. The chunking behavior is engine-agnostic — it lives in the service layer, not in individual backends.
|
||
|
||
## Instruct Mode
|
||
|
||
Two engines support natural-language delivery control via the `instruct` kwarg:
|
||
|
||
- **Qwen CustomVoice** — `supports_instruct=True`, fully wired to the model's instruct head.
|
||
- **Qwen Base** — silently drops the instruct text (`supports_instruct=False`). The frontend hides the instruct input for Base profiles.
|
||
|
||
```python
|
||
# Good instruct prompts:
|
||
"warm and conversational, slight smile"
|
||
"whisper, intimate and close"
|
||
"authoritative, broadcast quality"
|
||
```
|
||
|
||
Other engines ignore `instruct` entirely.
|
||
|
||
## Memory Management
|
||
|
||
Models are loaded lazily on first use and kept in memory. Switching between model sizes (e.g. Qwen 1.7B ↔ 0.6B) unloads the previous model before loading the new one to avoid OOM:
|
||
|
||
```python
|
||
def unload_model(self):
|
||
if self.model is not None:
|
||
del self.model
|
||
self.model = None
|
||
if torch.cuda.is_available():
|
||
torch.cuda.empty_cache()
|
||
```
|
||
|
||
The model management API (`/models/load`, `/models/unload`) lets users free VRAM manually — see [Model Management](/developer/model-management).
|
||
|
||
## API Endpoints
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| POST | `/generate` | Generate speech from text |
|
||
| GET | `/audio/{generation_id}` | Serve generated audio file |
|
||
|
||
### Response schema
|
||
|
||
```json
|
||
{
|
||
"id": "generation_uuid",
|
||
"profile_id": "profile_uuid",
|
||
"text": "...",
|
||
"language": "en",
|
||
"audio_path": "/path/to/audio.wav",
|
||
"duration": 3.5,
|
||
"seed": 42,
|
||
"engine": "qwen",
|
||
"model_size": "1.7B",
|
||
"instruct": "...",
|
||
"created_at": "2026-04-18T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
## Performance Considerations
|
||
|
||
- **CUDA** is the fastest backend for every PyTorch-based engine. Apple Silicon MLX is competitive with CUDA for Qwen TTS specifically.
|
||
- **Serial queue** — only one generation runs at a time per process; concurrent requests are queued.
|
||
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
|
||
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
|
||
|
||
### Per-engine VRAM (approximate, on CUDA)
|
||
|
||
| Engine | VRAM |
|
||
|--------|------|
|
||
| Kokoro | ~150 MB |
|
||
| LuxTTS | ~1 GB |
|
||
| Chatterbox Turbo | ~1.5 GB |
|
||
| Qwen 0.6B / Qwen CustomVoice 0.6B | ~2 GB |
|
||
| Chatterbox Multilingual | ~3 GB |
|
||
| Qwen 1.7B / Qwen CustomVoice 1.7B | ~6 GB |
|
||
| TADA 1B | ~4 GB |
|
||
| TADA 3B | ~8 GB |
|
||
|
||
## Next Steps
|
||
|
||
<Cards>
|
||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||
Add a new engine — full phased workflow
|
||
</Card>
|
||
<Card title="Model Management" href="/developer/model-management">
|
||
Downloading, loading, and unloading models
|
||
</Card>
|
||
<Card title="Voice Profiles" href="/developer/voice-profiles">
|
||
Cloned vs preset profile schema
|
||
</Card>
|
||
</Cards>
|