refactor start

2026-09-18 14:20:42 -07:00 · 2026-03-16 00:52:23 -07:00
parent 82cd4bf2ef
commit 4e84415da7
9 changed files with 1009 additions and 541 deletions
@@ -0,0 +1,363 @@
+# Adding a TTS Engine to Voicebox
+
+Guide for adding new TTS model backends. Based on the implementation of LuxTTS (#254), Chatterbox Multilingual (#257), Chatterbox Turbo (#258), and the PyInstaller fixes in v0.2.3.
+
+---
+
+## Overview
+
+Adding an engine touches ~12 files across 4 layers (down from ~19 after the model config registry refactor). The backend protocol work is straightforward — the real time sink is dependency hell, upstream library bugs, and PyInstaller bundling.
+
+---
+
+## Phase 1: Backend Implementation
+
+### 1.1 Create the backend file
+
+`backend/backends/<engine>_backend.py` (~200-300 lines)
+
+Implement the `TTSBackend` protocol from `backend/backends/__init__.py`:
+
+```python
+class YourBackend:
+    """Must satisfy the TTSBackend protocol."""
+
+    async def load_model(self, model_size: str = "default") -> None: ...
+    async def create_voice_prompt(self, audio_path: str, reference_text: str, use_cache: bool = True) -> tuple[dict, bool]: ...
+    async def combine_voice_prompts(self, audio_paths: list[str], ref_texts: list[str]) -> tuple[np.ndarray, str]: ...
+    async def generate(self, text: str, voice_prompt: dict, language: str = "en", seed: int | None = None, instruct: str | None = None) -> tuple[np.ndarray, int]: ...
+    def unload_model(self) -> None: ...
+    def is_loaded(self) -> bool: ...
+    def _get_model_path(self, model_size: str) -> str: ...
+```
+
+Key decisions per engine:
+
+| Decision | Options | Examples |
+|----------|---------|---------|
+| **Voice prompt storage** | Pre-computed tensors vs deferred file paths | Qwen PyTorch stores tensor dicts; Chatterbox stores `{"ref_audio": path, "ref_text": text}` |
+| **Caching** | Use voice prompt cache or skip it | LuxTTS caches with `luxtts_` prefix; Chatterbox skips caching entirely |
+| **Device selection** | CUDA / MPS / CPU | Chatterbox forces CPU on macOS (MPS tensor bugs); LuxTTS supports MPS |
+| **Model download** | Library handles it vs manual `snapshot_download` | Turbo uses manual download to bypass upstream `token=True` bug |
+| **Sample rate** | Engine-specific | LuxTTS outputs 48kHz, everything else is 24kHz |
+
+### 1.2 Voice prompt patterns
+
+There are three patterns in use. Pick the one that fits your model:
+
+**Pattern A: Pre-computed tensors** (Qwen PyTorch, LuxTTS)
+```python
+# create_voice_prompt returns opaque dict of tensors
+# Cached via torch.save(), reused across generations
+encoded = model.encode_prompt(audio_path)
+return encoded, False  # (prompt_dict, was_cached)
+```
+
+**Pattern B: Deferred file paths** (Chatterbox, MLX)
+```python
+# Just store paths, process at generation time
+return {"ref_audio": audio_path, "ref_text": reference_text}, False
+```
+
+**Pattern C: Hybrid** (possible for new engines)
+```python
+# Pre-compute speaker embeddings, store alongside paths
+embedding = model.extract_speaker(audio_path)
+return {"embedding": embedding, "ref_audio": audio_path}, False
+```
+
+If caching, prefix your cache keys to avoid collisions with other engines using the same reference audio:
+```python
+cache_key = "yourengine_" + get_cache_key(audio_path, reference_text)
+```
+
+### 1.3 Register the engine
+
+In `backend/backends/__init__.py`, three things:
+
+**1. Add a `ModelConfig` entry** in `_get_non_qwen_tts_configs()`:
+
+```python
+ModelConfig(
+    model_name="your-engine",
+    display_name="Your Engine",
+    engine="your_engine",
+    hf_repo_id="org/model-repo",
+    size_mb=3200,
+    needs_trim=False,  # set True if output needs trim_tts_output()
+    languages=["en", "fr", "de"],
+),
+```
+
+This single entry replaces what used to be 6+ scattered dicts in `main.py`. The registry helpers (`get_model_config()`, `check_model_loaded()`, `engine_needs_trim()`, etc.) all derive from this config automatically.
+
+**2. Add to `TTS_ENGINES` dict:**
+
+```python
+TTS_ENGINES = {
+    ...
+    "your_engine": "Your Engine",
+}
+```
+
+**3. Add an elif branch in `get_tts_backend_for_engine()`:**
+
+```python
+elif engine == "your_engine":
+    from .your_backend import YourBackend
+    backend = YourBackend()
+```
+
+The import is deferred so platform-specific deps aren't loaded until the engine is first requested.
+
+### 1.4 Update request models
+
+In `backend/models.py`:
+
+- Add engine name to `GenerationRequest.engine` regex pattern
+- Add any new language codes to the language regex on both `GenerationRequest` and `VoiceProfileCreate`
+
+---
+
+## Phase 2: API Integration (`main.py`)
+
+With the model config registry, `main.py` has **zero per-engine dispatch points**. All endpoints use registry helpers like `get_model_config()`, `load_engine_model()`, `engine_needs_trim()`, `check_model_loaded()`, etc.
+
+**You don't need to touch `main.py` at all** unless your engine needs custom behavior in the generate endpoint (e.g. a new post-processing step beyond `trim_tts_output`).
+
+### 2.1 What the registry handles automatically
+
+| Endpoint | Registry function used |
+|----------|----------------------|
+| `POST /generate` | `load_engine_model(engine, size)` + `engine_needs_trim(engine)` |
+| `POST /generate/stream` | `ensure_model_cached_or_raise(engine, size)` + `load_engine_model()` |
+| `GET /models/status` | `get_all_model_configs()` + `check_model_loaded(config)` |
+| `POST /models/download` | `get_model_config(name)` + `get_model_load_func(config)` |
+| `POST /models/{name}/unload` | `get_model_config(name)` + `unload_model_by_config(config)` |
+| `DELETE /models/{name}` | `get_model_config(name)` + `unload_model_by_config(config)` |
+
+### 2.2 Post-processing
+
+If your model produces trailing silence or hallucinated audio, set `needs_trim=True` on your `ModelConfig`. The generate endpoint checks `engine_needs_trim(engine)` and applies `trim_tts_output()` automatically.
+
+---
+
+## Phase 3: Frontend Integration
+
+### 3.1 TypeScript types
+
+In `app/src/lib/api/types.ts`:
+- Add to the `engine` union type on `GenerationRequest`
+
+### 3.2 Language maps
+
+In `app/src/lib/constants/languages.ts`:
+- Add entry to `ENGINE_LANGUAGES` record
+- Add any new language codes to `ALL_LANGUAGES` if needed
+
+### 3.3 Engine/model selector (shared component)
+
+The model selector is a shared component — update one file:
+
+- `app/src/components/Generation/EngineModelSelector.tsx`
+
+Add an entry to `ENGINE_OPTIONS` and `ENGINE_DESCRIPTIONS`. If the engine is English-only, add it to `ENGLISH_ONLY_ENGINES`. The `handleEngineChange()` function handles language validation automatically (resets to first available language if the current one isn't supported).
+
+Both `GenerationForm.tsx` and `FloatingGenerateBox.tsx` use `<EngineModelSelector>` — no changes needed in either.
+
+Handle engine-specific UI conditionals in the form components if needed:
+- Hide instruct field for engines that don't support it
+- Show engine-specific controls (e.g. `ParalinguisticInput` for Turbo)
+
+### 3.4 Form hook
+
+In `app/src/lib/hooks/useGenerationForm.ts`:
+- Add to Zod schema enum for `engine`
+- Add engine-to-model-name mapping (e.g. `"your_engine"` → `"your-engine"`)
+- Update payload construction to conditionally include engine-specific fields
+
+### 3.5 Model management
+
+In `app/src/components/ServerSettings/ModelManagement.tsx`:
+- Add description to `MODEL_DESCRIPTIONS` record
+- The model list auto-renders from `/models/status` data
+
+---
+
+## Phase 4: Dependencies
+
+### 4.1 Python dependencies
+
+Add to `backend/requirements.txt`. Watch for:
+
+**Pinned dependency conflicts** — If the model package pins old versions of numpy, torch, or transformers, install with `--no-deps` and list sub-dependencies manually. This is what Chatterbox requires:
+```
+# In justfile/Makefile (NOT requirements.txt):
+pip install --no-deps chatterbox-tts
+
+# In requirements.txt — list the transitive deps:
+conformer
+diffusers
+omegaconf
+# ... etc
+```
+
+**Non-PyPI packages** — Some deps only exist as git repos:
+```
+linacodec @ git+https://github.com/user/repo.git
+Zipvoice @ git+https://github.com/user/repo.git
+```
+
+**Custom package indexes** — Some packages need `--find-links`:
+```
+--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
+```
+
+### 4.2 Identifying hidden sub-dependencies
+
+When using `--no-deps`, you need to manually figure out what the package actually imports. There's no shortcut:
+
+1. Install the package normally in a throwaway venv
+2. Run `pip show <package>` to get its `Requires:` list
+3. Cross-reference against what's already in our requirements.txt
+4. Test that the engine loads and generates without import errors
+
+---
+
+## Phase 5: PyInstaller Bundling
+
+This is where most of the pain lives. If your model's Python package or its dependencies use any of the following at runtime, PyInstaller won't bundle them automatically:
+
+### 5.1 Common PyInstaller issues
+
+| Issue | Symptom | Fix |
+|-------|---------|-----|
+| **`inspect.getsource()` at import time** | "could not get source code" | `--collect-all <package>` (bundles `.py` source files, not just bytecode) |
+| **Data files (yaml, .pth.tar, lang dicts)** | FileNotFoundError at runtime | `--collect-all <package>` or `--collect-data <package>` |
+| **Native data paths (espeak-ng, etc.)** | Library looks at `/usr/share/...` | Set env var in frozen builds: `os.environ["ESPEAK_DATA_PATH"] = bundled_path` |
+| **`importlib.metadata` lookups** | "No package metadata found" | `--copy-metadata <package>` |
+| **Dynamic imports** | ModuleNotFoundError | `--hidden-import <module>` |
+| **`typeguard` / `@typechecked`** | Calls `inspect.getsource()` on decorated functions | `--collect-all` for the decorated package |
+
+### 5.2 Testing frozen builds
+
+You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary. The flow:
+
+1. Build the binary: `just build` or the PyInstaller spec
+2. Run it and try to download + load + generate with the new engine
+3. Check stderr for the actual error (macOS/Linux: stdout/stderr go to Tauri sidecar logs)
+4. Fix, rebuild, repeat
+
+### 5.3 Real examples from v0.2.3
+
+These were all models that worked perfectly in dev:
+
+- **LuxTTS**: `typeguard`'s `@typechecked` calls `inspect.getsource()` at import → needed `--collect-all inflect`. `piper_phonemize` bundles `espeak-ng-data/` → needed `--collect-all piper_phonemize` + `ESPEAK_DATA_PATH` env var
+- **Chatterbox**: `resemble-perth` bundles a pretrained watermark model (`.pth.tar`, `hparams.yaml`) → needed `--collect-all perth`
+- **Both**: `huggingface_hub` silently disables tqdm based on logger level → progress bars showed 0% in frozen builds until we force-enabled the internal counter
+
+---
+
+## Phase 6: Common Upstream Workarounds
+
+Almost every model library has bugs you'll need to work around. Here's the catalog:
+
+### 6.1 torch.load device mismatch
+
+If model weights were saved on CUDA but you're loading on CPU/MPS:
+```python
+_original_torch_load = torch.load
+def _patched_torch_load(*args, **kwargs):
+    kwargs.setdefault("map_location", "cpu")
+    return _original_torch_load(*args, **kwargs)
+torch.load = _patched_torch_load
+```
+Used by both Chatterbox backends. Use a threading lock if patching globally.
+
+### 6.2 Float64/Float32 dtype mismatch
+
+`librosa` returns float64, model weights are float32. Patch the offending methods:
+```python
+original_fn = SomeClass.some_method
+def patched_fn(self, *args, **kwargs):
+    result = original_fn(self, *args, **kwargs)
+    return result.float()  # float64 → float32
+SomeClass.some_method = patched_fn
+```
+Used by Chatterbox for `S3Tokenizer.log_mel_spectrogram` and `VoiceEncoder.forward`.
+
+### 6.3 Transformers attention implementation
+
+If the model uses `output_attentions=True` with transformers >= 4.36:
+```python
+for module in model.modules():
+    if hasattr(module, '_attn_implementation'):
+        module._attn_implementation = "eager"
+```
+SDPA (the new default) doesn't support `output_attentions`. Force eager attention.
+
+### 6.4 HuggingFace token bug
+
+Some models' `from_pretrained()` passes `token=True` which requires a stored HF token even for public repos:
+```python
+from huggingface_hub import snapshot_download
+local_path = snapshot_download(repo_id=REPO, token=None)
+model = ModelClass.from_local(local_path, device=device)
+```
+Used by Chatterbox Turbo.
+
+### 6.5 MPS tensor issues
+
+MPS (Apple Silicon GPU) has incomplete operator coverage. If generation crashes on MPS:
+```python
+def _get_device(self):
+    if torch.cuda.is_available():
+        return "cuda"
+    return "cpu"  # Skip MPS entirely
+```
+Used by both Chatterbox backends. LuxTTS works fine on MPS.
+
+### 6.6 HuggingFace progress tracking
+
+To get download progress bars in the UI, wrap model loading with `HFProgressTracker`:
+```python
+from backend.utils.hf_progress import HFProgressTracker
+tracker = HFProgressTracker(model_name, progress_manager)
+with tracker.patch_download():
+    model = ModelClass.from_pretrained(repo_id)
+```
+The tracker monkey-patches tqdm to intercept HuggingFace's internal progress bars. Must be set up BEFORE importing the model library if it imports HF at module level.
+
+---
+
+## Checklist
+
+### Backend
+- [ ] `backend/backends/<engine>_backend.py` — implements TTSBackend protocol
+- [ ] `backend/backends/__init__.py` — `ModelConfig` entry + `TTS_ENGINES` + `get_tts_backend_for_engine()` elif
+- [ ] `backend/models.py` — engine name in regex, any new language codes
+- [ ] `backend/requirements.txt` — dependencies added (check for `--no-deps` needs)
+- [ ] `justfile` / `Makefile` — `--no-deps` install step if needed
+
+### API (`backend/main.py`)
+No changes needed — the model config registry handles all dispatch automatically.
+
+### Frontend
+- [ ] `app/src/lib/api/types.ts` — engine union type
+- [ ] `app/src/lib/constants/languages.ts` — `ENGINE_LANGUAGES` entry
+- [ ] `app/src/components/Generation/EngineModelSelector.tsx` — `ENGINE_OPTIONS` + `ENGINE_DESCRIPTIONS` + `ENGLISH_ONLY_ENGINES`
+- [ ] `app/src/lib/hooks/useGenerationForm.ts` — Zod schema + model mapping
+- [ ] `app/src/components/ServerSettings/ModelManagement.tsx` — model description
+
+### Production
+- [ ] PyInstaller spec — `--collect-all`, `--hidden-import`, `--copy-metadata` as needed
+- [ ] Test in frozen binary — download, load, generate all work
+- [ ] Download progress — `HFProgressTracker` wired up, progress shows in UI
+
+### Upstream workarounds (check which apply)
+- [ ] torch.load device mapping (CUDA weights on CPU)
+- [ ] Float64→Float32 patches (librosa interaction)
+- [ ] Eager attention forcing (transformers >= 4.36)
+- [ ] HF token bypass (snapshot_download + from_local)
+- [ ] MPS skip (if operators not supported)
+- [ ] espeak-ng / native data path env vars
@@ -50,9 +50,10 @@

 | Layer | File | Purpose |
 |-------|------|---------|
-| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2100 lines) |
-| TTS protocol | `backend/backends/__init__.py:14-81` | `TTSBackend` Protocol definition |
-| TTS factory | `backend/backends/__init__.py:138-178` | Thread-safe engine registry (double-checked locking) |
+| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2850 lines) |
+| TTS protocol | `backend/backends/__init__.py:32-101` | `TTSBackend` Protocol definition |
+| Model registry | `backend/backends/__init__.py:17-29,153-366` | `ModelConfig` dataclass + registry helpers |
+| TTS factory | `backend/backends/__init__.py:382-426` | Thread-safe engine registry (double-checked locking) |
 | PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
 | MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
 | LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
@@ -64,6 +65,7 @@
 | Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
 | Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
 | Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
+| Engine selector | `app/src/components/Generation/EngineModelSelector.tsx` | Shared engine/model dropdown |
 | Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
 | Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
 | Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
@@ -101,8 +103,10 @@ POST /generate
 - Multi-engine TTS architecture with thread-safe backend registry (PR #254)
 - LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
 - Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Delivery instructions (instruct parameter, Qwen only)
+- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
 - Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
+- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
+- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms

 **Infrastructure:**
 - CUDA backend swap via binary download and restart (PR #252)
@@ -125,13 +129,13 @@ POST /generate

 ### TTS Engine Comparison

-| Engine | Model Name | Languages | Size | Key Features |
-|--------|-----------|-----------|------|-------------|
-| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Instruct mode, highest quality |
-| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | 10 | ~1.2 GB | Lighter, faster |
-| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast |
-| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual |
-| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency |
+| Engine | Model Name | Languages | Size | Key Features | Instruct Support |
+|--------|-----------|-----------|------|-------------|-----------------|
+| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
+| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | 10 | ~1.2 GB | Lighter, faster | None |
+| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
+| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
+| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |

 ### Multi-Engine Architecture (Shipped)

@@ -149,6 +153,7 @@ The singleton TTS backend blocker described in the previous version of this doc
 - **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
 - **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
 - **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
+- **Instruct parameter is non-functional** (#224): The UI exposes an instruct text field, but it's silently dropped by every backend. The Qwen3-TTS Base model we ship only supports voice cloning — instruct requires the separate CustomVoice model variant (`Qwen3-TTS-12Hz-1.7B-CustomVoice`), which uses predefined speakers instead of ref audio. The instruct UI should be hidden until a backend with real support is integrated.
 - **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
 - **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.

@@ -323,41 +328,43 @@ Notable requests:

 ### Models Worth Supporting (2026 SOTA — updated March 13)

-| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Integration Ease | Status |
-|-------|---------|-------|-------------|-----------|------|-----------------|--------|
-| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | **Shipped** | v0.1.13 |
-| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | **Shipped** | PR #254 |
-| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | **Shipped** | PR #257 |
-| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | **PR #258** | In review |
-| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Needs vetting | MIT, 700s+ coherent, synced transcript output |
-| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | Needs vetting | Apache 2.0, multi-speaker dialogue, text-to-voice design (no ref audio) |
-| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion, LoRA-friendly |
-| **Pocket TTS** | Zero-shot + streaming | >1× RT on CPU | — | English | ~100M params, CPU-first | Needs vetting | MIT, Kyutai Labs, no GPU required |
-| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Ready | Apache 2.0, multi-engine arch in place |
-| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Ready | Multi-engine arch in place |
-| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | Ready | Multi-engine arch in place |
-| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Ready | Multi-engine arch in place |
+| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct Support | Integration Ease | Status |
+|-------|---------|-------|-------------|-----------|------|-----------------|-----------------|--------|
+| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None (Base); Yes (CustomVoice variant, predefined speakers only) | **Shipped** | v0.1.13 |
+| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | **Shipped** | PR #254 |
+| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` float | **Shipped** | PR #257 |
+| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **PR #258** | In review |
+| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
+| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
+| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
+| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output |
+| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
+| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
+| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
+| **Pocket TTS** | Zero-shot + streaming | >1× RT on CPU | — | English | ~100M params, CPU-first | None | Needs vetting | MIT, Kyutai Labs, no GPU required |

 #### Notes on New Candidates (March 2026)

- **HumeAI TADA** — Text-Audio Dual Alignment arch. Near-zero hallucinations/drift, free synced transcript. 700+ seconds coherent audio. Best candidate for Stories long-form reliability. [HF: HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **MOSS-TTS** — Modular suite: flagship cloning, MOSS-TTSD (multi-speaker dialogue), MOSS-VoiceGenerator (create voices from text descriptions, no ref audio). Unique UX for Stories voice design. [GitHub: OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS)
- **VoxCPM 1.5** — Tokenizer-free continuous diffusion + autoregressive. No discrete token artifacts. Context-aware prosody/emotion, real-time streaming, LoRA fine-tuning. Trained on 1.8M+ hours. [GitHub: OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs (Moshi team). Runs >1× realtime without GPU. Broadens hardware support significantly. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
+- **CosyVoice2-0.5B** — Best candidate for instruct support. `inference_instruct2()` accepts a text instruct parameter for emotions, speed, volume, dialects — and it works alongside voice cloning. This is the closest match to what users expect from our instruct UI. [HF: FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B)
+- **HumeAI TADA** — Text-Audio Dual Alignment arch. Near-zero hallucinations/drift, free synced transcript. 700+ seconds coherent audio. Best candidate for Stories long-form reliability. Prosody/emotion is automatic from text context, not user-controllable. [HF: HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
+- **MOSS-TTS** — Modular suite: flagship cloning, MOSS-TTSD (multi-speaker dialogue), MOSS-VoiceGenerator (create voices from text descriptions). VoiceGenerator unifies timbre design and style control via text prompts, usable as a layer for downstream TTS including cloning. [HF: OpenMOSS-Team/MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | [GitHub: OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS)
+- **Fish Speech** — Word-level fine-grained control using plain language descriptions inline in the script. Works with cloning. Note: Fish Audio S2 has a restrictive research license (commercial use requires approval), but the open-source Fish Speech model may differ. Needs license clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
+- **VoxCPM 1.5** — Tokenizer-free continuous diffusion + autoregressive. No discrete token artifacts. Prosody/emotion is context-aware but automatic, not explicitly controllable via text prompt. Real-time streaming, LoRA fine-tuning. Trained on 1.8M+ hours. [GitHub: OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM)
+- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs (Moshi team). Runs >1× realtime without GPU. No style control. Broadens hardware support significantly. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
 - **Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control)
- **Skipped:** Fish Audio S2 — restrictive research license (commercial use requires approval), despite strong features

 ### Adding a New Engine (Now Straightforward)

-With the multi-engine architecture shipped, adding a new TTS engine requires:
+With the model config registry and shared `EngineModelSelector` component, adding a new TTS engine requires:

 1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
-2. **Register in `backend/backends/__init__.py`** — add to `TTS_ENGINES` dict + factory function
+2. **Register in `backend/backends/__init__.py`** — add `ModelConfig` entry + `TTS_ENGINES` entry + factory elif
 3. **Update `backend/models.py`** — add engine name to regex
-4. **Update `backend/main.py`** — add engine cases in generate, stream, model-status, download, delete (5 dispatch points)
-5. **Update frontend** — add to engine union type, form schema, model dropdown, language map (5-6 files)
+4. **Update frontend** — add to engine union type, `EngineModelSelector` options, form schema, language map (4 files)

-Total effort: **~1 day** for a well-documented model with a PyPI package.
+`main.py` requires **zero changes** — the registry handles all dispatch automatically.
+
+Total effort: **~1 day** for a well-documented model with a PyPI package. See `docs/plans/ADDING_TTS_ENGINES.md` for the full guide.

 ---

@@ -367,13 +374,13 @@ Total effort: **~1 day** for a well-documented model with a PyPI package.

 The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.

-### 2. `main.py` is 2100+ Lines
+### ~~2. `main.py` Dispatch Point Duplication~~ — RESOLVED

-All API routes, all model configs, all business logic in one file. Five separate dispatch points for each engine. Any new engine touches this file in 5 places. A model config registry pattern would reduce duplication.
+Previously, each engine required updates to 6+ hardcoded dispatch maps across `main.py` (~320 lines of if/elif chains). A model config registry in `backend/backends/__init__.py` now centralizes all model metadata (`ModelConfig` dataclass) with helper functions (`load_engine_model()`, `check_model_loaded()`, `engine_needs_trim()`, etc.). Adding a new engine requires zero changes to `main.py`.

-### 3. Model Config is Scattered (Improved)
+### ~~3. Model Config is Scattered~~ — RESOLVED

-Model identifiers are still duplicated across `main.py` (3 dicts), backend files, frontend components, and the languages constant. However, the pattern is now consistent and well-understood. A centralized model registry would help but isn't blocking.
+Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the `ModelConfig` registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in `EngineModelSelector.tsx`.

 ### 4. Voice Prompt Cache Assumes PyTorch Tensors

@@ -410,7 +417,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
 | 1 | **#253** — 48kHz speech tokenizer | Quality improvement | Medium |
 | 2 | **#161** — Docker deployment | Server/headless users | Medium |
 | 3 | **#154** — Audiobook tab | Long-form users | Medium |
-| 4 | **Model config registry** | Reduce 5-dispatch-point duplication in main.py | Medium |
+| 4 | ~~**Model config registry**~~ | ~~Reduce dispatch duplication in main.py~~ | **Done** |
 | 5 | **#225** — Custom HuggingFace models | User-supplied models | High (needs rework for multi-engine) |

 ### Tier 3 — Future (v0.3.0+)
@@ -421,7 +428,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
 | 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model, broadens hardware support. Kyutai ships clean code. Needs API vetting. |
 | 3 | **MOSS-TTS** | Text-to-voice design (no ref audio) is unique. Multi-speaker dialogue for Stories. Needs thorough API vetting. |
 | 4 | **Kokoro-82M** | 82M params, CPU realtime, Apache 2.0. Easy win. |
-| 5 | **Model config registry refactor** | Reduce 5-dispatch-point duplication in main.py — do before adding 3+ more engines |
+| 5 | ~~**Model config registry refactor**~~ | **Done** — consolidated in `backend/backends/__init__.py` + `EngineModelSelector.tsx` |
 | 6 | XTTS-v2 / Fish Speech / CosyVoice | Multi-engine arch is ready; just needs backend implementation |
 | 7 | **VoxCPM 1.5** | Tokenizer-free streaming, interesting but uncertain integration surface |
 | 8 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable |