Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-16 13:20:39 -07:00
Code Issues Packages Projects Releases Wiki Activity

fix TorchScript error in frozen builds and update docs for TADA

Browse Source 
Remove @torch.jit.script from the DAC shim's snake() function —
TorchScript calls inspect.getsource() which fails in PyInstaller
binaries (no .py source files).

Update all user-facing docs: 4 → 5 TTS engines, add TADA row to
every engine comparison table, mark TADA as Shipped in the upcoming
engines list, update architecture diagrams and tech stack tables.
This commit is contained in:
James Pine
2026-03-17 03:28:58 -07:00
parent 5774a168a9
commit 273483ffcf
6 changed files with 30 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+6 -5
View File
@@ -59,10 +59,10 @@
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** — models and voice data stay on your machine
- **4 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -93,7 +93,7 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
### Multi-Engine Voice Cloning
Four TTS engines with different strengths, switchable per-generation:
Five TTS engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
@@ -101,6 +101,7 @@ Four TTS engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
### Emotions & Paralinguistic Tags
@@ -230,7 +231,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
@@ -245,7 +246,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| ----------------------- | ---------------------------------------------- |
| **Real-time Streaming** | Stream audio as it generates, word by word |
| **Voice Design** | Create new voices from text descriptions |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **Plugin Architecture** | Extend with custom models and effects |
| **Mobile Companion** | Control Voicebox from your phone |
backend/utils/dac_shim.py
+3 -1
View File
@@ -24,7 +24,9 @@ import torch.nn as nn
# ── Snake activation (from dac/nn/layers.py) ────────────────────────
@torch.jit.script
# NOTE: The original DAC code uses @torch.jit.script here for a 1.4x
# speedup. We omit it because TorchScript calls inspect.getsource()
# which fails inside a PyInstaller frozen binary (no .py source files).
def snake(x: torch.Tensor, alpha: torch.Tensor) -> torch.Tensor:
shape = x.shape
x = x.reshape(shape[0], shape[1], -1)
docs/content/docs/developer/tts-engines.mdx
+1 -1
View File
@@ -490,7 +490,7 @@ Based on the current model landscape, these are candidates for future integratio
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Needs vetting |
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Shipped |
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
docs/content/docs/index.mdx
+2 -2
View File
@@ -3,12 +3,12 @@ title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
docs/content/docs/overview/introduction.mdx
+5 -4
View File
@@ -5,10 +5,10 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -20,7 +20,7 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## TTS Engines
Four engines with different strengths, switchable per-generation:
Five engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
@@ -28,6 +28,7 @@ Four engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
## GPU Support
@@ -56,7 +57,7 @@ Four engines with different strengths, switchable per-generation:
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
docs/notes/PROJECT_STATUS.md
+13 -5
View File
@@ -36,6 +36,10 @@
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │ │ │ TADA │ │ │ │
│ │ │ │(1B / 3B) │ │ │ │
│ │ │ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ ┌───────────┐ ┌─────────┐ │ │
│ │ │ STTBackend│ │ Profiles│ │ │
@@ -59,6 +63,7 @@
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
@@ -78,7 +83,7 @@
```
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo)
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
@@ -104,7 +109,8 @@ POST /generate
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296)
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo, TADA 1B, TADA 3B)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
@@ -136,6 +142,8 @@ POST /generate
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
### Multi-Engine Architecture (Shipped)
@@ -143,7 +151,7 @@ The singleton TTS backend blocker described in the previous version of this doc
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'`
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'`
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
@@ -337,7 +345,7 @@ Notable requests:
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output |
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | **Shipped** | PR #296, MIT, 700s+ coherent |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
@@ -475,7 +483,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
| `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
| `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model |