diff --git a/README.md b/README.md index 614e4f21..1acc9a4f 100644 --- a/README.md +++ b/README.md @@ -59,10 +59,10 @@ ## What is Voicebox? -Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. +Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. - **Complete privacy** — models and voice data stay on your machine -- **4 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo +- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA - **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more - **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters - **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo @@ -93,7 +93,7 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al ### Multi-Engine Voice Cloning -Four TTS engines with different strengths, switchable per-generation: +Five TTS engines with different strengths, switchable per-generation: | Engine | Languages | Strengths | | --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- | @@ -101,6 +101,7 @@ Four TTS engines with different strengths, switchable per-generation: | **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU | | **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more | | **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags | +| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment | ### Emotions & Paralinguistic Tags @@ -230,7 +231,7 @@ Full API documentation available at `http://localhost:17493/docs`. | Frontend | React, TypeScript, Tailwind CSS | | State | Zustand, React Query | | Backend | FastAPI (Python) | -| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo | +| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA | | Effects | Pedalboard (Spotify) | | Transcription | Whisper / Whisper Turbo (PyTorch or MLX) | | Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) | @@ -245,7 +246,7 @@ Full API documentation available at `http://localhost:17493/docs`. | ----------------------- | ---------------------------------------------- | | **Real-time Streaming** | Stream audio as it generates, word by word | | **Voice Design** | Create new voices from text descriptions | -| **More Models** | XTTS, Bark, and other open-source voice models | +| **More Models** | XTTS, Bark, and other open-source voice models | | **Plugin Architecture** | Extend with custom models and effects | | **Mobile Companion** | Control Voicebox from your phone | diff --git a/backend/utils/dac_shim.py b/backend/utils/dac_shim.py index ea1294fc..89968c18 100644 --- a/backend/utils/dac_shim.py +++ b/backend/utils/dac_shim.py @@ -24,7 +24,9 @@ import torch.nn as nn # ── Snake activation (from dac/nn/layers.py) ──────────────────────── -@torch.jit.script +# NOTE: The original DAC code uses @torch.jit.script here for a 1.4x +# speedup. We omit it because TorchScript calls inspect.getsource() +# which fails inside a PyInstaller frozen binary (no .py source files). def snake(x: torch.Tensor, alpha: torch.Tensor) -> torch.Tensor: shape = x.shape x = x.reshape(shape[0], shape[1], -1) diff --git a/docs/content/docs/developer/tts-engines.mdx b/docs/content/docs/developer/tts-engines.mdx index f95e7a71..90135a37 100644 --- a/docs/content/docs/developer/tts-engines.mdx +++ b/docs/content/docs/developer/tts-engines.mdx @@ -490,7 +490,7 @@ Based on the current model landscape, these are candidates for future integratio | **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready | | **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready | | **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready | -| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Needs vetting | +| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Shipped | | **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting | | **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting | diff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdx index cbe120c3..9a7da078 100644 --- a/docs/content/docs/index.mdx +++ b/docs/content/docs/index.mdx @@ -3,12 +3,12 @@ title: "Voicebox Documentation" description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs." --- -Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. +Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. ![Voicebox App Screenshot](/images/app-screenshot-1.webp) - **Complete privacy** -- models and voice data stay on your machine -- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo +- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA - **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more - **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters - **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo diff --git a/docs/content/docs/overview/introduction.mdx b/docs/content/docs/overview/introduction.mdx index 134f1d71..d740a61e 100644 --- a/docs/content/docs/overview/introduction.mdx +++ b/docs/content/docs/overview/introduction.mdx @@ -5,10 +5,10 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open- ## What is Voicebox? -Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. +Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor. - **Complete privacy** -- models and voice data stay on your machine -- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo +- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA - **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more - **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters - **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo @@ -20,7 +20,7 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt ## TTS Engines -Four engines with different strengths, switchable per-generation: +Five engines with different strengths, switchable per-generation: | Engine | Languages | Strengths | |--------|-----------|-----------| @@ -28,6 +28,7 @@ Four engines with different strengths, switchable per-generation: | **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU | | **Chatterbox Multilingual** | 23 | Broadest language coverage | | **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags | +| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio | ## GPU Support @@ -56,7 +57,7 @@ Four engines with different strengths, switchable per-generation: | Frontend | React, TypeScript, Tailwind CSS | | State | Zustand, React Query | | Backend | FastAPI (Python) | -| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo | +| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA | | Effects | Pedalboard (Spotify) | | Transcription | Whisper / Whisper Turbo (PyTorch or MLX) | | Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) | diff --git a/docs/notes/PROJECT_STATUS.md b/docs/notes/PROJECT_STATUS.md index 628dfa0a..71c06453 100644 --- a/docs/notes/PROJECT_STATUS.md +++ b/docs/notes/PROJECT_STATUS.md @@ -36,6 +36,10 @@ │ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │ │ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │ │ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │ +│ │ │ ┌──────────┐ │ │ │ +│ │ │ │ TADA │ │ │ │ +│ │ │ │(1B / 3B) │ │ │ │ +│ │ │ └──────────┘ │ │ │ │ │ └─────────────────────────────────────────┘ │ │ │ │ ┌───────────┐ ┌─────────┐ │ │ │ │ │ STTBackend│ │ Profiles│ │ │ @@ -59,6 +63,7 @@ | LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly | | Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages | | Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags | +| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual | | Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch | | API types | `backend/models.py` | Pydantic request/response models | | HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) | @@ -78,7 +83,7 @@ ``` POST /generate 1. Look up voice profile from DB - 2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo) + 2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada) 3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine 4. Check model cache → if missing, trigger background download, return HTTP 202 5. Load model (lazy): tts_backend.load_model(model_size) @@ -104,7 +109,8 @@ POST /generate - LuxTTS integration — fast, CPU-friendly English TTS (PR #254) - Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257) - Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations) -- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo) +- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296) +- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo, TADA 1B, TADA 3B) - Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py` - Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms @@ -136,6 +142,8 @@ POST /generate | LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None | | Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness | | Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param | +| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None | +| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None | ### Multi-Engine Architecture (Shipped) @@ -143,7 +151,7 @@ The singleton TTS backend blocker described in the previous version of this doc - **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking - **Per-engine backend instances** — each engine gets its own singleton, loaded lazily -- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'` +- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'` - **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages - **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend - **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination) @@ -337,7 +345,7 @@ Notable requests: | **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate | | **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place | | **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue | -| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output | +| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | **Shipped** | PR #296, MIT, 700s+ coherent | | **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion | | **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place | | **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place | @@ -475,7 +483,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe | `/history/{id}/export` | GET | Export generation ZIP | | `/history/{id}/export-audio` | GET | Export audio only | | `/transcribe` | POST | Transcribe audio (Whisper) | -| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) | +| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) | | `/models/download` | POST | Trigger model download | | `/models/download/cancel` | POST | Cancel/dismiss download | | `/models/{name}` | DELETE | Delete downloaded model |