Compare commits

...
Author SHA1 Message Date
James Pine d6f48ace3e Mirror the regular /generate endpoint behavior more closely 2026-03-19 16:11:38 -07:00
James Pine 0fc2192204 fix: resolve relative paths using configured data dir, not CWD 2026-03-19 10:37:11 -07:00
James Pine 9e726ad048 fix: remove engine dropdown filtering — profile grid handles it 2026-03-19 10:14:33 -07:00
James Pine 3584283d84 feat: Kokoro 82M TTS engine + voice profile type system
Add Kokoro-82M as a new TTS engine — 82M params, CPU realtime, 8 languages,
Apache 2.0. Unlike cloning engines, Kokoro uses pre-built voice styles, which
required a new profile type system to support non-cloning engines cleanly.

Kokoro engine:
- New kokoro_backend.py implementing TTSBackend protocol
- 50 built-in voices across en/es/fr/hi/it/pt/ja/zh
- KPipeline API with language-aware G2P routing via misaki
- PyInstaller bundling for misaki, language_tags, espeakng_loader, en_core_web_sm

Voice profile type system:
- New voice_type column: 'cloned' | 'preset' | 'designed' (future)
- Preset profiles store engine + voice ID instead of audio samples
- default_engine field on profiles — auto-selects engine on profile pick
- Create Voice dialog: toggle between 'Clone from audio' and 'Built-in voice'
- Edit dialog shows preset voice info instead of sample list for preset profiles
- Engine selector locks to preset engine when preset profile is selected
- Profile grid filters by engine — shows Kokoro voices when Kokoro selected
- Custom empty state when no preset profiles exist for selected engine

Bug fixes:
- Fix relative audio paths in DB causing 404s in production builds
- config.set_data_dir() now resolves to absolute paths
- Startup migration converts existing relative paths to absolute

Also updates PROJECT_STATUS.md and tts-engines.mdx developer guide.
2026-03-19 10:09:48 -07:00
Jamie PineandGitHub ffc1b54812 Merge pull request #316 from jamiepine/fix/cuda-cu128-upgrade
Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI
2026-03-18 07:58:12 -07:00
James Pine fc5ed1ff40 upgrade CUDA backend from cu126 to cu128 and fix GPU settings UI
Upgrade CUDA toolkit from 12.6 (cu126) to 12.8 (cu128) for proper
RTX 50-series (Blackwell) GPU support. Users with RTX 5070/5080/5090
were reporting CUDA detection failures with cu126.

Also fix the GPU Acceleration settings panel where the 'Switch to CPU
Backend' button was unreachable — it was inside a conditional block
that required !isCurrentlyCuda, making it impossible to switch back
to CPU once running on CUDA.

Closes #315
2026-03-18 07:47:39 -07:00
Jamie PineandGitHub c9f38dd496 Merge pull request #305 from jamiepine/fix/qwen-tts-pyinstaller-source-files
fix: bundle qwen_tts source files in PyInstaller build
2026-03-17 09:24:42 -07:00
James Pine 58b19e4e9f fix: bundle qwen_tts source files in PyInstaller build
Replace --collect-submodules + --collect-data with --collect-all for
qwen_tts. The qwen_tts runtime expects physical .py source files
(e.g. modeling_qwen3_tts.py) under _MEIPASS, which only --collect-all
provides. This is the same pattern used for inflect/typeguard.

Fixes #212
2026-03-17 09:23:30 -07:00
Jamie PineandGitHub 0245c31dba Merge pull request #298 from jamiepine/feat/cuda-libs-addon
feat: split CUDA backend into independently versioned server + libs archives
2026-03-17 09:17:31 -07:00
Jamie Pine 81864e831a fix: always clean up temp archive on failure and fix justfile data dir path
- Wrap download/verify/extract in try/finally so .download-*.tmp is
  always deleted, even on mid-download or extraction failures
- Fix justfile build-server-cuda to use sh.voicebox.app (production path)
2026-03-17 09:15:23 -07:00
Jamie Pine 7bd72ea9f7 Bump version: 0.3.0 → 0.3.1 2026-03-17 07:50:12 -07:00
Jamie Pine f96eae2567 fix: address PR review feedback from CodeRabbit
- Upgrade softprops/action-gh-release@v1 to @v2 (Node 16 EOL)
- Fail-fast on checksum fetch failure instead of extracting unverified archives
- Abort packaging if no NVIDIA files found (prevents empty cuda-libs archive)
- Fix nvidia/ path detection bug (list membership vs substring check)
- Fix justfile Copy-Item nesting (copy contents, not the directory itself)
2026-03-17 06:53:54 -07:00
Jamie Pine 7d53699c96 fix: update build-server-cuda to copy onedir folder instead of single exe 2026-03-17 06:12:30 -07:00
Jamie Pine 28e91ce2c1 chore: add .spec and nul to .gitignore 2026-03-17 04:58:13 -07:00
Jamie Pine 88be097b62 fix: update package_cuda.py for PyInstaller 6.18 layout and remove split_binary.py
- Fix is_nvidia_file() to match NVIDIA DLLs in _internal/torch/lib/
  (PyInstaller 6.18 + torch 2.10 no longer uses nvidia/ subdirectories)
- Remove deprecated split_binary.py (both archives are under 2GB)
- Update torch_compat range to >=2.6.0,<2.11.0
- Update build docs for new dual-archive packaging flow
2026-03-17 04:57:05 -07:00
James Pine 564d787927 feat: split CUDA backend into independently versioned server + libs archives
Switch CUDA builds from PyInstaller --onefile to --onedir and split the
output into two separately versioned archives:

1. Server core (~200-400MB) — versioned with the app, redownloaded on
   every app update
2. CUDA libs (~2GB) — versioned independently (cu126-v1), only
   redownloaded when the CUDA toolkit or torch version changes

This eliminates the ~2.4GB full redownload on every version bump.
After initial setup, most app updates only need ~200-400MB.

Closes #297
2026-03-17 04:04:17 -07:00
James Pine 2c1ee94891 docs: add TADA learnings to TTS engine guide and CUDA libs addon plan
Enrich tts-engines.mdx with patterns discovered during TADA integration:
- Phase 0.2: new greps for @torch.jit.script, torchaudio.load, gated repos
- Phase 3.4: model naming inconsistency warning
- Phase 5.2: TADA shim failure added to lessons table
- Phase 6: four new workaround sections (gated repos, torchcodec,
  torch.jit.script, toxic dependency shim pattern)
- Checklist: four new items matching the new scan patterns
- Remove TADA from upcoming engines (now shipped)

Add CUDA_LIBS_ADDON.md exploring --onedir split to avoid 2.4GB
redownloads on every version bump.
2026-03-17 03:53:40 -07:00
Jamie PineandGitHub e789c937ad Merge pull request #296 from jamiepine/feat/add-tada-tts-engine
Add HumeAI TADA TTS engine (1B English + 3B Multilingual)
2026-03-17 03:47:47 -07:00
James Pine 273483ffcf fix TorchScript error in frozen builds and update docs for TADA
Remove @torch.jit.script from the DAC shim's snake() function —
TorchScript calls inspect.getsource() which fails in PyInstaller
binaries (no .py source files).

Update all user-facing docs: 4 → 5 TTS engines, add TADA row to
every engine comparison table, mark TADA as Shipped in the upcoming
engines list, update architecture diagrams and tech stack tables.
2026-03-17 03:28:58 -07:00
James Pine 5774a168a9 fix TADA 3B model name: tada-3b -> tada-3b-ml 2026-03-17 03:17:53 -07:00
James Pine 6bf40bd2d0 fix tokenizer patch corrupting AutoTokenizer for other engines
Replace the monkey-patch on AutoTokenizer.from_pretrained (which broke
the classmethod descriptor and caused 'Tokenizer not loaded' errors
when loading Qwen after TADA) with two targeted config patches:
- Set AlignerConfig.tokenizer_name to the local ungated tokenizer path
- Pre-load TadaConfig, inject tokenizer_name, pass config= to from_pretrained

No global state is modified; other engines are unaffected.
2026-03-17 03:15:57 -07:00
James Pine 12cda2e090 fix torchcodec error by using soundfile instead of torchaudio.load
torchaudio 2.10+ switched its default audio loading backend to
torchcodec, which isn't installed. Replace torchaudio.load() with
soundfile.read() in create_voice_prompt(). TADA's internal use of
torchaudio.functional.resample() is unaffected (pure PyTorch math,
no torchcodec dependency).
2026-03-17 02:25:05 -07:00
James Pine 7a90290a76 fix gated Llama tokenizer error by redirecting to ungated mirror
TADA hardcodes 'meta-llama/Llama-3.2-1B' as its tokenizer source in
both the Aligner and TadaForCausalLM.from_pretrained(). That repo is
gated and requires accepting Meta's license on HuggingFace.

Monkey-patch AutoTokenizer.from_pretrained during model loading to
redirect Llama tokenizer requests to 'unsloth/Llama-3.2-1B', an
ungated mirror with identical tokenizer files. The patch is scoped
to model loading only and restored immediately after.
2026-03-17 02:22:26 -07:00
James Pine b02ce8e2f3 replace descript-audio-codec with lightweight DAC shim
The real descript-audio-codec package pulls in descript-audiotools,
which transitively requires onnx, tensorboard, protobuf, matplotlib,
pystoi, and other heavy dependencies. onnx fails to build from source
on macOS due to CMake version incompatibility.

TADA only uses Snake1d (a 7-line PyTorch module) from DAC. This commit
adds a shim in backend/utils/dac_shim.py that registers fake dac.*
modules in sys.modules with just the Snake1d class, completely
eliminating the DAC/audiotools dependency chain.
2026-03-17 02:16:33 -07:00
James Pine 4e7772a21d add HumeAI TADA TTS engine (1B English + 3B Multilingual)
Integrates HumeAI's TADA (Text-Acoustic Dual Alignment) speech-language
model as a new TTS engine. TADA uses a novel 1:1 token-audio alignment
that produces coherent speech over long sequences (700s+).

Two model variants:
- tada-1b: English-only, ~4GB, built on Llama 3.2 1B
- tada-3b-ml: 10 languages, ~8GB, built on Llama 3.2 3B

Backend uses the Encoder for voice prompt encoding with caching, and
TadaForCausalLM with flow-matching diffusion for generation. Supports
bf16 inference on CUDA, forces CPU on macOS (MPS compatibility).

Installed with --no-deps due to torch>=2.7 pin conflict; descript-audio-codec
and torchaudio added as explicit sub-dependencies.
2026-03-17 01:55:15 -07:00
James Pine 51fb320b8c readme 2026-03-17 01:24:48 -07:00
James Pine 8ac202aa58 docs for adding new engines 2026-03-17 01:13:30 -07:00
Jamie Pine ac68052945 create stub resource files when actool output is missing
Older Xcode versions don't produce Assets.car from .icon assets.
Fall back to empty stubs for all platforms so the bundler succeeds.
2026-03-17 01:07:54 -07:00
Jamie Pine e601fd2ca4 generate icon assets at build time instead of tracking them
build.rs now generates voicebox.icns via sips + iconutil alongside the
existing actool Assets.car compilation. On non-macOS, empty stub files
are created so Tauri's resource bundler doesn't fail on missing paths.
2026-03-17 00:51:12 -07:00
Jamie Pine 7b25e0ba0b Bump version: 0.2.3 → 0.3.0 2026-03-17 00:25:56 -07:00
Jamie PineandGitHub a6817cd082 Merge pull request #295 from jamiepine/fix/misc-bugs
fix: batch of bug fixes from issue tracker
2026-03-17 00:08:17 -07:00
55 changed files with 3370 additions and 619 deletions
+120
View File
@@ -0,0 +1,120 @@
---
name: add-tts-engine
description: Use this skill to add a new TTS engine to Voicebox. It walks through dependency research, backend implementation, frontend wiring, PyInstaller bundling, and frozen-build testing. Always start with Phase 0 (dependency audit) before writing any code.
---
# Add TTS Engine
## Goal
Integrate a new text-to-speech engine into Voicebox end-to-end: dependency research, backend protocol implementation, frontend UI wiring, PyInstaller bundling, and frozen-build verification. The user should only need to test the final build locally.
## Reference Doc
The full phased guide lives at `docs/content/docs/developer/tts-engines.mdx`. **Read this file in its entirety before starting.** It contains:
- Phase 0: Dependency research (mandatory before writing code)
- Phase 1: Backend implementation (`TTSBackend` protocol)
- Phase 2: Route and service integration (usually zero changes)
- Phase 3: Frontend integration (5 files)
- Phase 4: Dependencies (`requirements.txt`, justfile, CI, Docker)
- Phase 5: PyInstaller bundling (`build_binary.py` + `server.py`)
- Phase 6: Common upstream workarounds
- Implementation checklist (gate between phases)
## Workflow
### 1. Read the guide
```bash
# Read the full TTS engines doc
cat docs/content/docs/developer/tts-engines.mdx
```
Internalize all phases, especially Phase 0 and Phase 5. The v0.2.3 release was three patch releases because Phase 0 was skipped.
### 2. Dependency research (Phase 0)
Clone the model library into a temporary directory and audit it. Do NOT skip this.
```bash
mkdir /tmp/engine-research && cd /tmp/engine-research
git clone <model-library-url>
```
Run the grep searches from Phase 0.2 in the guide against the cloned source and its transitive dependencies. Produce a written dependency audit covering:
1. PyPI vs non-PyPI packages
2. PyInstaller directives needed (`--collect-all`, `--copy-metadata`, `--hidden-import`)
3. Runtime data files that must be bundled
4. Native library paths that need env var overrides in frozen builds
5. Monkey-patches needed (`torch.load`, float64, MPS, HF token)
6. Sample rate
7. Model download method (`from_pretrained` vs `snapshot_download` + `from_local`)
Test model loading and generation on CPU in the throwaway venv before proceeding.
### 3. Implement (Phases 1–4)
Follow the guide's phases in order. Key files to modify:
**Backend (Phase 1):**
- Create `backend/backends/<engine>_backend.py`
- Register in `backend/backends/__init__.py` (ModelConfig + TTS_ENGINES + factory)
- Update regex in `backend/models.py`
**Frontend (Phase 3):**
- `app/src/lib/api/types.ts` — engine union type
- `app/src/lib/constants/languages.ts` — ENGINE_LANGUAGES
- `app/src/components/Generation/EngineModelSelector.tsx` — ENGINE_OPTIONS, ENGINE_DESCRIPTIONS
- `app/src/lib/hooks/useGenerationForm.ts` — Zod schema, model-name mapping
- `app/src/components/ServerSettings/ModelManagement.tsx` — MODEL_DESCRIPTIONS
**Dependencies (Phase 4):**
- `backend/requirements.txt`
- `justfile` (setup-python, setup-python-release targets)
- `.github/workflows/release.yml`
- `Dockerfile` (if applicable)
### 4. PyInstaller bundling (Phase 5)
Register the engine in `backend/build_binary.py`:
- `--hidden-import` for the backend module and model package
- `--collect-all` for packages using `inspect.getsource`, shipping data files, or native libraries
- `--copy-metadata` for packages using `importlib.metadata`
If the engine has native data paths, add `os.environ.setdefault()` in `backend/server.py` inside the `if getattr(sys, 'frozen', False):` block.
### 5. Verify in dev mode
```bash
just dev
```
Test the full chain: model download → load → generate → voice cloning.
### 6. Use the checklist
Walk through the Implementation Checklist at the bottom of `tts-engines.mdx`. Every item must be checked before handing the build to the user.
## Key Lessons (from v0.2.3)
These are the most common failure modes. Phase 0 research catches all of them:
| Pattern | Symptom in Frozen Build | Fix |
|---------|------------------------|-----|
| `@typechecked` / `inspect.getsource()` | "could not get source code" | `--collect-all <package>` |
| Package ships pretrained model files | `FileNotFoundError` for `.pth.tar`, `.yaml` | `--collect-all <package>` |
| C library with hardcoded system paths | `FileNotFoundError` for `/usr/share/...` | `--collect-all` + env var in `server.py` |
| `importlib.metadata.version()` | "No package metadata found" | `--copy-metadata <package>` |
| `torch.load` without `map_location` | CUDA device not available on CPU build | Monkey-patch `torch.load` |
| `torch.from_numpy` on float64 data | dtype mismatch RuntimeError | Cast to `.float()` |
| `token=True` in HF download calls | Auth failure without stored HF token | Use `snapshot_download(token=None)` + `from_local()` |
## Notes
- The route and service layers have zero per-engine dispatch points. `main.py` requires zero changes.
- The model config registry in `backends/__init__.py` handles all dispatch automatically.
- Use `get_torch_device()` and `model_load_progress()` from `backends/base.py` — don't reimplement device detection or progress tracking.
- Always test with a **clean HuggingFace cache** (no pre-downloaded models from dev).
- Do NOT push or create a release. Hand the build to the user for local testing.
+1 -1
View File
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.2.3
current_version = 0.3.1
commit = True
tag = True
tag_name = v{new_version}
+21 -15
View File
@@ -62,6 +62,7 @@ jobs:
pip install pyinstaller
pip install -r backend/requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
- name: Install MLX dependencies (Apple Silicon only)
if: matrix.backend == 'mlx'
@@ -188,43 +189,48 @@ jobs:
pip install pyinstaller
pip install -r backend/requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
- name: Install PyTorch with CUDA 12.6
- name: Install PyTorch with CUDA 12.8
run: |
pip install torch --index-url https://download.pytorch.org/whl/cu126 --force-reinstall --no-deps
pip install torchaudio --index-url https://download.pytorch.org/whl/cu126 --force-reinstall --no-deps
pip install torch --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
pip install torchaudio --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
- name: Verify CUDA support in torch
run: |
python -c "import torch; print(f'CUDA available in build: {torch.cuda.is_available()}'); print(f'CUDA version: {torch.version.cuda}')"
- name: Build CUDA server binary
- name: Build CUDA server binary (onedir)
shell: bash
working-directory: backend
run: python build_binary.py --cuda
- name: Split binary for GitHub Releases
- name: Package into server core + CUDA libs archives
shell: bash
run: |
python scripts/split_binary.py \
backend/dist/voicebox-server-cuda.exe \
--output release-assets/
python scripts/package_cuda.py \
backend/dist/voicebox-server-cuda/ \
--output release-assets/ \
--cuda-libs-version cu128-v1 \
--torch-compat ">=2.7.0,<2.11.0"
- name: Upload split parts to GitHub Release
- name: Upload archives to GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v1
uses: softprops/action-gh-release@v2
with:
files: |
release-assets/voicebox-server-cuda.part*.exe
release-assets/voicebox-server-cuda.sha256
release-assets/voicebox-server-cuda.manifest
release-assets/voicebox-server-cuda.tar.gz
release-assets/voicebox-server-cuda.tar.gz.sha256
release-assets/cuda-libs-cu128-v1.tar.gz
release-assets/cuda-libs-cu128-v1.tar.gz.sha256
release-assets/cuda-libs.json
draft: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload binary as workflow artifact
- name: Upload onedir as workflow artifact
uses: actions/upload-artifact@v4
with:
name: voicebox-server-cuda-windows
path: backend/dist/voicebox-server-cuda.exe
path: backend/dist/voicebox-server-cuda/
retention-days: 7
+7
View File
@@ -51,6 +51,13 @@ app/openapi.json
tauri/src-tauri/binaries/*
tauri/src-tauri/gen/Assets.car
tauri/src-tauri/gen/voicebox.icns
tauri/src-tauri/gen/partial.plist
# PyInstaller
*.spec
# Windows artifacts
nul
# Temporary
tmp/
+29 -2
View File
@@ -7,9 +7,25 @@
## [Unreleased]
This release rewrites the backend into a modular architecture, migrates the documentation site to Fumadocs, and ships a batch of bug fixes and UI polish across the stack.
## [0.3.0] - 2026-03-17
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, model loading status is now visible in the UI, effects presets get a dropdown, and several race conditions and accessibility gaps are closed.
This release rewrites the backend into a modular architecture, overhauls the settings UI into routed sub-pages, fixes audio player freezing, migrates documentation to Fumadocs, and ships a batch of bug fixes targeting the most-reported issues from the tracker.
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, settings have been split into dedicated routed pages with server logs, a changelog viewer, and an about page. The audio player no longer freezes mid-playback, and model loading status is now visible in the UI. Seven user-reported bugs have been fixed, including server crashes during sample uploads, generation list staleness, cryptic error messages, and CUDA support for RTX 50-series GPUs.
### Settings Overhaul ([#294](https://github.com/jamiepine/voicebox/pull/294))
- Split settings into routed sub-tabs: General, Generation, GPU, Logs, Changelog, About
- Added live server log viewer with auto-scroll
- Added in-app changelog page that parses `CHANGELOG.md` at build time
- Added About page with version info, license, and generation folder quick-open
- Extracted reusable `SettingRow` component for consistent setting layouts
### Audio Player Fix ([#293](https://github.com/jamiepine/voicebox/pull/293))
- Fixed audio player freezing during playback
- Improved playback UX with better state management and listener cleanup
- Fixed restart race condition during regeneration
- Added stable keys for audio element re-rendering
- Improved accessibility across player controls
### Backend Refactor ([#285](https://github.com/jamiepine/voicebox/pull/285))
- Extracted all routes from `main.py` into 13 domain routers under `backend/routes/` — `main.py` dropped from ~3,100 lines to ~10
@@ -40,6 +56,17 @@ The backend's 3,000-line monolith `main.py` has been decomposed into domain rout
- Softened select focus indicator opacity
- Addressed 4 critical and 12 major issues from CodeRabbit review
### Bug Fixes ([#295](https://github.com/jamiepine/voicebox/pull/295))
- Fixed sample uploads crashing the server — audio decoding now runs in a thread pool instead of blocking the async event loop ([#278](https://github.com/jamiepine/voicebox/issues/278))
- Fixed generation list not updating when a generation completes — switched to `refetchQueries` for reliable cache busting, added SSE error fallback, and page reset on completion ([#231](https://github.com/jamiepine/voicebox/issues/231))
- Fixed error toasts showing `[object Object]` instead of the actual error message ([#290](https://github.com/jamiepine/voicebox/issues/290))
- Added Whisper model selection (`base`, `small`, `medium`, `large`, `turbo`) and expanded language support to the `/transcribe` endpoint ([#233](https://github.com/jamiepine/voicebox/issues/233))
- Upgraded CUDA backend build from cu121 to cu126 for RTX 50-series (Blackwell) GPU support ([#289](https://github.com/jamiepine/voicebox/issues/289))
- Handled client disconnects in SSE and streaming endpoints to suppress `[Errno 32] Broken Pipe` errors ([#248](https://github.com/jamiepine/voicebox/issues/248))
- Fixed Docker build failure from pip hash mismatch on Qwen3-TTS dependencies ([#286](https://github.com/jamiepine/voicebox/issues/286))
- Added 50 MB upload size limit with chunked reads to prevent unbounded memory allocation on sample uploads
- Eliminated redundant double audio decode in sample processing pipeline
### Platform Fixes
- Replaced `netstat` with `TcpStream` + PowerShell for Windows port detection ([#277](https://github.com/jamiepine/voicebox/pull/277))
- Fixed Docker frontend build and cleaned up Docker docs
+2
View File
@@ -35,6 +35,8 @@ RUN pip install --no-cache-dir --upgrade pip
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
RUN pip install --no-cache-dir --prefix=/install --no-deps chatterbox-tts
RUN pip install --no-cache-dir --prefix=/install --no-deps hume-tada
RUN pip install --no-cache-dir --prefix=/install \
git+https://github.com/QwenLM/Qwen3-TTS.git
+12 -5
View File
@@ -59,10 +59,10 @@
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** — models and voice data stay on your machine
- **4 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -93,7 +93,7 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
### Multi-Engine Voice Cloning
Four TTS engines with different strengths, switchable per-generation:
Five TTS engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
@@ -101,6 +101,7 @@ Four TTS engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
### Emotions & Paralinguistic Tags
@@ -230,7 +231,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
@@ -245,7 +246,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| ----------------------- | ---------------------------------------------- |
| **Real-time Streaming** | Stream audio as it generates, word by word |
| **Voice Design** | Create new voices from text descriptions |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **Plugin Architecture** | Extend with custom models and effects |
| **Mobile Companion** | Control Voicebox from your phone |
@@ -276,6 +277,12 @@ just build # Build CPU server binary + Tauri app
just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app
```
### Adding New Voice Models
The multi-engine architecture makes adding new TTS engines straightforward. A [step-by-step guide](docs/content/docs/developer/tts-engines.mdx) covers the full process: dependency research, backend protocol implementation, frontend wiring, and PyInstaller bundling.
The guide is optimized for AI coding agents. An [agent skill](.agents/skills/add-tts-engine/SKILL.md) can pick up a model name and handle the entire integration autonomously — you just test the build locally.
### Project Structure
```
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@voicebox/app",
"version": "0.2.3",
"version": "0.3.1",
"private": true,
"type": "module",
"scripts": {
@@ -7,6 +7,7 @@ import {
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import type { VoiceProfileResponse } from '@/lib/api/types';
import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
import type { GenerationFormValues } from '@/lib/hooks/useGenerationForm';
@@ -15,11 +16,14 @@ import type { GenerationFormValues } from '@/lib/hooks/useGenerationForm';
* Adding a new engine means adding one entry here.
*/
const ENGINE_OPTIONS = [
{ value: 'qwen:1.7B', label: 'Qwen3-TTS 1.7B' },
{ value: 'qwen:0.6B', label: 'Qwen3-TTS 0.6B' },
{ value: 'luxtts', label: 'LuxTTS' },
{ value: 'chatterbox', label: 'Chatterbox' },
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
{ value: 'qwen:1.7B', label: 'Qwen3-TTS 1.7B', engine: 'qwen' },
{ value: 'qwen:0.6B', label: 'Qwen3-TTS 0.6B', engine: 'qwen' },
{ value: 'luxtts', label: 'LuxTTS', engine: 'luxtts' },
{ value: 'chatterbox', label: 'Chatterbox', engine: 'chatterbox' },
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo', engine: 'chatterbox_turbo' },
{ value: 'tada:1B', label: 'TADA 1B', engine: 'tada' },
{ value: 'tada:3B', label: 'TADA 3B Multilingual', engine: 'tada' },
{ value: 'kokoro', label: 'Kokoro 82M', engine: 'kokoro' },
] as const;
const ENGINE_DESCRIPTIONS: Record<string, string> = {
@@ -27,13 +31,27 @@ const ENGINE_DESCRIPTIONS: Record<string, string> = {
luxtts: 'Fast, English-focused',
chatterbox: '23 languages, incl. Hebrew',
chatterbox_turbo: 'English, [laugh] [cough] tags',
tada: 'HumeAI, 700s+ coherent audio',
kokoro: '82M params, CPU realtime, 8 langs',
};
/** Engines that only support English and should force language to 'en' on select. */
const ENGLISH_ONLY_ENGINES = new Set(['luxtts', 'chatterbox_turbo']);
/** Engines that support cloned (reference audio) profiles. */
const CLONING_ENGINES = new Set(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada']);
/**
* All engine options are always available. The profile grid already
* filters by engine, so the dropdown doesn't need to restrict options.
*/
function getAvailableOptions(_selectedProfile?: VoiceProfileResponse | null) {
return ENGINE_OPTIONS;
}
function getSelectValue(engine: string, modelSize?: string): string {
if (engine === 'qwen') return `qwen:${modelSize || '1.7B'}`;
if (engine === 'tada') return `tada:${modelSize || '1B'}`;
return engine;
}
@@ -48,6 +66,20 @@ function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: st
if (!available.some((l) => l.value === currentLang)) {
form.setValue('language', available[0]?.value ?? 'en');
}
} else if (value.startsWith('tada:')) {
const [, modelSize] = value.split(':');
form.setValue('engine', 'tada');
form.setValue('modelSize', modelSize as '1B' | '3B');
// TADA 1B is English-only; 3B is multilingual
if (modelSize === '1B') {
form.setValue('language', 'en');
} else {
const currentLang = form.getValues('language');
const available = getLanguageOptionsForEngine('tada');
if (!available.some((l) => l.value === currentLang)) {
form.setValue('language', available[0]?.value ?? 'en');
}
}
} else {
form.setValue('engine', value as GenerationFormValues['engine']);
form.setValue('modelSize', undefined as unknown as '1.7B' | '0.6B');
@@ -67,12 +99,21 @@ function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: st
interface EngineModelSelectorProps {
form: UseFormReturn<GenerationFormValues>;
compact?: boolean;
selectedProfile?: VoiceProfileResponse | null;
}
export function EngineModelSelector({ form, compact }: EngineModelSelectorProps) {
export function EngineModelSelector({ form, compact, selectedProfile }: EngineModelSelectorProps) {
const engine = form.watch('engine') || 'qwen';
const modelSize = form.watch('modelSize');
const selectValue = getSelectValue(engine, modelSize);
const availableOptions = getAvailableOptions(selectedProfile);
// If current engine isn't in available options, auto-switch to first available
const currentEngineAvailable = availableOptions.some((opt) => opt.value === selectValue);
if (!currentEngineAvailable && availableOptions.length > 0) {
// Defer to avoid setting state during render
setTimeout(() => handleEngineChange(form, availableOptions[0].value), 0);
}
const itemClass = compact ? 'text-xs text-muted-foreground' : undefined;
const triggerClass = compact
@@ -87,7 +128,7 @@ export function EngineModelSelector({ form, compact }: EngineModelSelectorProps)
</SelectTrigger>
</FormControl>
<SelectContent>
{ENGINE_OPTIONS.map((opt) => (
{availableOptions.map((opt) => (
<SelectItem key={opt.value} value={opt.value} className={itemClass}>
{opt.label}
</SelectItem>
@@ -101,3 +142,17 @@ export function EngineModelSelector({ form, compact }: EngineModelSelectorProps)
export function getEngineDescription(engine: string): string {
return ENGINE_DESCRIPTIONS[engine] ?? '';
}
/**
* Check if a profile is compatible with the currently selected engine.
* Useful for UI hints.
*/
export function isProfileCompatibleWithEngine(
profile: VoiceProfileResponse,
engine: string,
): boolean {
const voiceType = profile.voice_type || 'cloned';
if (voiceType === 'preset') return profile.preset_engine === engine;
if (voiceType === 'cloned') return CLONING_ENGINES.has(engine);
return true; // designed — future
}
@@ -36,6 +36,7 @@ export function FloatingGenerateBox({
}: FloatingGenerateBoxProps) {
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
const setSelectedEngine = useUIStore((state) => state.setSelectedEngine);
const { data: selectedProfile } = useProfile(selectedProfileId || '');
const { data: profiles } = useProfiles();
const [isExpanded, setIsExpanded] = useState(false);
@@ -67,7 +68,12 @@ export function FloatingGenerateBox({
}
},
getEffectsChain: () => {
if (!selectedPresetId || !effectPresets) return undefined;
if (!selectedPresetId) return undefined;
// Profile's own effects chain (no matching preset)
if (selectedPresetId === '_profile') {
return selectedProfile?.effects_chain ?? undefined;
}
if (!effectPresets) return undefined;
const preset = effectPresets.find((p) => p.id === selectedPresetId);
return preset?.effects_chain;
},
@@ -110,12 +116,56 @@ export function FloatingGenerateBox({
}
}, [selectedProfileId, profiles, setSelectedProfileId]);
// Sync generation form language with selected profile's language
// Sync engine selection to global store so ProfileList can filter
const watchedEngine = form.watch('engine');
useEffect(() => {
if (watchedEngine) {
setSelectedEngine(watchedEngine);
}
}, [watchedEngine, setSelectedEngine]);
// Sync generation form language, engine, and effects with selected profile
useEffect(() => {
if (selectedProfile?.language) {
form.setValue('language', selectedProfile.language as LanguageCode);
}
}, [selectedProfile, form]);
// Auto-switch engine if profile has a default
if (selectedProfile?.default_engine) {
form.setValue(
'engine',
selectedProfile.default_engine as
| 'qwen'
| 'luxtts'
| 'chatterbox'
| 'chatterbox_turbo'
| 'tada'
| 'kokoro',
);
}
// Pre-fill effects from profile defaults
if (
selectedProfile?.effects_chain &&
selectedProfile.effects_chain.length > 0 &&
effectPresets
) {
// Try to match against a known preset
const profileChainJson = JSON.stringify(selectedProfile.effects_chain);
const matchingPreset = effectPresets.find(
(p) => JSON.stringify(p.effects_chain) === profileChainJson,
);
if (matchingPreset) {
setSelectedPresetId(matchingPreset.id);
} else {
// No matching preset — use special value to pass profile chain directly
setSelectedPresetId('_profile');
}
} else if (
selectedProfile &&
(!selectedProfile.effects_chain || selectedProfile.effects_chain.length === 0)
) {
setSelectedPresetId(null);
}
}, [selectedProfile, effectPresets, form]);
// Auto-resize textarea based on content (only when expanded)
useEffect(() => {
@@ -358,7 +408,7 @@ export function FloatingGenerateBox({
/>
<FormItem className="flex-1 space-y-0">
<EngineModelSelector form={form} compact />
<EngineModelSelector form={form} compact selectedProfile={selectedProfile} />
</FormItem>
<FormItem className="flex-1 space-y-0">
@@ -375,6 +425,12 @@ export function FloatingGenerateBox({
<SelectItem value="none" className="text-xs">
No effects
</SelectItem>
{selectedProfile?.effects_chain &&
selectedProfile.effects_chain.length > 0 && (
<SelectItem value="_profile" className="text-xs">
Profile default
</SelectItem>
)}
{effectPresets?.map((preset) => (
<SelectItem key={preset.id} value={preset.id} className="text-xs">
{preset.name}
@@ -118,7 +118,7 @@ export function GenerationForm() {
<div className="grid gap-4 md:grid-cols-3">
<FormItem>
<FormLabel>Model</FormLabel>
<EngineModelSelector form={form} />
<EngineModelSelector form={form} selectedProfile={selectedProfile} />
<FormDescription>
{getEngineDescription(form.watch('engine') || 'qwen')}
</FormDescription>
@@ -243,7 +243,40 @@ export function GpuAcceleration() {
{/* Native GPU detected - no CUDA download needed */}
{/* CUDA download section - only show when no GPU is active (native or CUDA) */}
{/* Currently running CUDA - show switch back to CPU */}
{isCurrentlyCuda && platform.metadata.isTauri && (
<>
{restartPhase !== 'idle' ? (
<div className="flex items-center gap-2 p-3 rounded-lg bg-primary/5 border">
<Loader2 className="h-4 w-4 animate-spin" />
<span className="text-sm">
{restartPhase === 'stopping' && 'Stopping server...'}
{restartPhase === 'waiting' && 'Restarting server...'}
{restartPhase === 'ready' && 'Server restarted successfully!'}
</span>
</div>
) : (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
re-download later).
</p>
<Button onClick={handleSwitchToCpu} variant="outline" className="w-full" size="sm">
<RotateCw className="h-4 w-4 mr-2" />
Switch to CPU Backend
</Button>
</div>
)}
{error && (
<div className="flex items-center gap-2 text-sm text-destructive">
<AlertCircle className="h-4 w-4 shrink-0" />
<span>{error}</span>
</div>
)}
</>
)}
{/* CUDA download/manage section - show when no native GPU and not currently running CUDA */}
{!hasNativeGpu && !isCurrentlyCuda && (
<>
{/* Download progress (manual download or auto-update) */}
@@ -315,7 +348,7 @@ export function GpuAcceleration() {
)}
{/* Downloaded but not active - show switch button */}
{cudaAvailable && !isCurrentlyCuda && platform.metadata.isTauri && (
{cudaAvailable && platform.metadata.isTauri && (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
CUDA backend is downloaded and ready. Restart the server to enable GPU
@@ -328,27 +361,8 @@ export function GpuAcceleration() {
</div>
)}
{/* Currently active - show switch back to CPU */}
{isCurrentlyCuda && platform.metadata.isTauri && (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
re-download later).
</p>
<Button
onClick={handleSwitchToCpu}
variant="outline"
className="w-full"
size="sm"
>
<RotateCw className="h-4 w-4 mr-2" />
Switch to CPU Backend
</Button>
</div>
)}
{/* Delete option when downloaded (and not active) */}
{cudaAvailable && !isCurrentlyCuda && (
{cudaAvailable && (
<Button
onClick={handleDelete}
variant="ghost"
@@ -62,6 +62,12 @@ const MODEL_DESCRIPTIONS: Record<string, string> = {
'Production-grade open source TTS by Resemble AI. Supports 23 languages with voice cloning and emotion exaggeration control.',
'chatterbox-turbo':
'Streamlined 350M parameter TTS by Resemble AI. High-quality English speech with less compute and VRAM than larger models.',
'tada-1b':
'HumeAI TADA 1B — English speech-language model built on Llama 3.2 1B. Generates 700s+ of coherent audio with synchronized text-acoustic alignment.',
'tada-3b-ml':
'HumeAI TADA 3B Multilingual — built on Llama 3.2 3B. Supports 10 languages with high-fidelity voice cloning via text-acoustic dual alignment.',
kokoro:
'Kokoro 82M by hexgrad. Tiny 82M-parameter TTS that runs at CPU realtime. Supports 8 languages with pre-built voice styles. Apache 2.0 licensed.',
'whisper-base':
'Smallest Whisper model (74M parameters). Fast transcription with moderate accuracy.',
'whisper-small':
@@ -391,7 +397,9 @@ export function ModelManagement() {
(m) =>
m.model_name.startsWith('qwen-tts') ||
m.model_name.startsWith('luxtts') ||
m.model_name.startsWith('chatterbox'),
m.model_name.startsWith('chatterbox') ||
m.model_name.startsWith('tada') ||
m.model_name.startsWith('kokoro'),
) ?? [];
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
@@ -97,6 +97,16 @@ export function ProfileCard({ profile }: ProfileCardProps) {
<Badge variant="outline" className="text-xs h-5 px-1.5 text-muted-foreground">
{profile.language}
</Badge>
{profile.voice_type === 'preset' && (
<Badge variant="secondary" className="text-xs h-5 px-1.5">
{profile.preset_engine}
</Badge>
)}
{profile.voice_type === 'designed' && (
<Badge variant="secondary" className="text-xs h-5 px-1.5">
designed
</Badge>
)}
{profile.effects_chain && profile.effects_chain.length > 0 && (
<Sparkles className="h-3.5 w-3.5 text-accent fill-accent" />
)}
+358 -129
View File
@@ -1,9 +1,11 @@
import { zodResolver } from '@hookform/resolvers/zod';
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
import { useQuery } from '@tanstack/react-query';
import { Edit2, Mic, Monitor, Music, Upload, X } from 'lucide-react';
import { useEffect, useRef, useState } from 'react';
import { useForm } from 'react-hook-form';
import * as z from 'zod';
import { EffectsChainEditor } from '@/components/Effects/EffectsChainEditor';
import { Badge } from '@/components/ui/badge';
import { Button } from '@/components/ui/button';
import {
Dialog,
@@ -15,6 +17,7 @@ import {
import {
Form,
FormControl,
FormDescription,
FormField,
FormItem,
FormLabel,
@@ -32,7 +35,7 @@ import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
import { Textarea } from '@/components/ui/textarea';
import { useToast } from '@/components/ui/use-toast';
import { apiClient } from '@/lib/api/client';
import type { EffectConfig } from '@/lib/api/types';
import type { EffectConfig, PresetVoice, VoiceType } from '@/lib/api/types';
import { LANGUAGE_CODES, LANGUAGE_OPTIONS, type LanguageCode } from '@/lib/constants/languages';
import { useAudioPlayer } from '@/lib/hooks/useAudioPlayer';
import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
@@ -120,16 +123,20 @@ export function ProfileForm() {
const deleteAvatar = useDeleteAvatar();
const transcribe = useTranscription();
const { toast } = useToast();
const [voiceSource, setVoiceSource] = useState<'clone' | 'builtin'>('clone');
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
const [audioDuration, setAudioDuration] = useState<number | null>(null);
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
const [selectedPresetEngine, setSelectedPresetEngine] = useState<string>('kokoro');
const [selectedPresetVoiceId, setSelectedPresetVoiceId] = useState<string>('');
const avatarInputRef = useRef<HTMLInputElement>(null);
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
const isCreating = !editingProfileId;
const serverUrl = useServerStore((state) => state.serverUrl);
const [profileEffectsChain, setProfileEffectsChain] = useState<EffectConfig[]>([]);
const [effectsDirty, setEffectsDirty] = useState(false);
const [defaultEngine, setDefaultEngine] = useState<string>('');
const form = useForm<ProfileFormValues>({
resolver: zodResolver(profileSchema),
@@ -239,6 +246,20 @@ export function ProfileForm() {
},
});
// Fetch available preset voices for the selected engine
const presetEngineToQuery = isCreating
? selectedPresetEngine
: (editingProfile?.preset_engine ?? '');
const { data: presetVoicesData } = useQuery({
queryKey: ['presetVoices', presetEngineToQuery],
queryFn: () => apiClient.listPresetVoices(presetEngineToQuery),
enabled:
!!presetEngineToQuery &&
((voiceSource === 'builtin' && isCreating) ||
(!isCreating && editingProfile?.voice_type === 'preset')),
});
const presetVoices = presetVoicesData?.voices ?? [];
// Show recording errors
useEffect(() => {
if (recordingError) {
@@ -287,6 +308,7 @@ export function ProfileForm() {
});
setProfileEffectsChain(editingProfile.effects_chain ?? []);
setEffectsDirty(false);
setDefaultEngine(editingProfile.default_engine ?? '');
} else if (profileFormDraft && open) {
// Restore from draft when opening in create mode
form.reset({
@@ -415,13 +437,14 @@ export function ProfileForm() {
async function onSubmit(data: ProfileFormValues) {
try {
if (editingProfileId) {
// Editing: just update profile
// Editing: update profile
await updateProfile.mutateAsync({
profileId: editingProfileId,
data: {
name: data.name,
description: data.description,
language: data.language,
default_engine: defaultEngine || undefined,
},
});
@@ -464,8 +487,50 @@ export function ProfileForm() {
title: 'Voice updated',
description: `"${data.name}" has been updated successfully.`,
});
} else if (voiceSource === 'builtin') {
// Creating preset profile from built-in voice
if (!selectedPresetVoiceId) {
toast({
title: 'No voice selected',
description: 'Please select a built-in voice.',
variant: 'destructive',
});
return;
}
const profile = await createProfile.mutateAsync({
name: data.name,
description: data.description,
language: data.language,
voice_type: 'preset' as VoiceType,
preset_engine: selectedPresetEngine,
preset_voice_id: selectedPresetVoiceId,
default_engine: selectedPresetEngine,
});
// Handle avatar upload if provided
if (data.avatarFile) {
try {
await uploadAvatar.mutateAsync({
profileId: profile.id,
file: data.avatarFile,
});
} catch (avatarError) {
toast({
title: 'Avatar upload failed',
description:
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
variant: 'destructive',
});
}
}
toast({
title: 'Profile created',
description: `"${data.name}" has been created with a built-in voice.`,
});
} else {
// Creating: require sample file and reference text
// Creating cloned profile: require sample file and reference text
const sampleFile = form.getValues('sampleFile');
const referenceText = form.getValues('referenceText');
@@ -528,6 +593,7 @@ export function ProfileForm() {
name: data.name,
description: data.description,
language: data.language,
default_engine: defaultEngine || undefined,
});
// Convert non-WAV uploads to WAV so the backend can always use soundfile.
@@ -642,16 +708,16 @@ export function ProfileForm() {
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-hidden">
<div className="max-w-5xl h-[85vh] mx-auto my-auto w-full flex flex-col overflow-hidden">
<DialogHeader>
<DialogTitle className="text-2xl">
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
{editingProfileId ? 'Edit Voice' : 'Create Voice'}
</DialogTitle>
<DialogDescription>
{editingProfileId
? 'Update your voice profile details and manage samples.'
: 'Create a new voice profile with an audio sample to clone the voice.'}
: 'Create a new voice profile from an audio sample or a built-in voice.'}
</DialogDescription>
{isCreating && profileFormDraft && (
<div className="flex items-center gap-2 pt-2">
@@ -682,143 +748,275 @@ export function ProfileForm() {
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
<div className="grid gap-6 grid-cols-2 flex-1 min-h-0 overflow-hidden">
{/* Left column: Sample management */}
<div className="space-y-4 border-r pr-6">
<div className="space-y-4 border-r pr-6 overflow-y-auto min-h-0">
{isCreating ? (
<>
<Tabs
className="pt-4"
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsList
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
</TabsTrigger>
)}
</TabsList>
{/* Voice source selector */}
<div className="flex pt-4 pb-2">
<div className="inline-flex rounded-lg border border-border p-0.5 bg-muted/50">
<button
type="button"
onClick={() => setVoiceSource('clone')}
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
voiceSource === 'clone'
? 'bg-accent text-accent-foreground shadow-sm'
: 'text-muted-foreground hover:text-foreground'
}`}
>
<Mic className="h-3.5 w-3.5" />
Clone from audio
</button>
<button
type="button"
onClick={() => setVoiceSource('builtin')}
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
voiceSource === 'builtin'
? 'bg-accent text-accent-foreground shadow-sm'
: 'text-muted-foreground hover:text-foreground'
}`}
>
<Music className="h-3.5 w-3.5" />
Built-in voice
</button>
</div>
</div>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null &&
audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
)}
/>
</TabsContent>
{voiceSource === 'builtin' ? (
<div className="space-y-4">
<FormDescription>
Choose a pre-built voice. These don't require an audio sample.
</FormDescription>
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleRecording
file={selectedFile}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
)}
</Tabs>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
{/* Engine selector */}
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
/>
</FormControl>
<FormMessage />
<FormLabel>Engine</FormLabel>
<Select
value={selectedPresetEngine}
onValueChange={setSelectedPresetEngine}
>
<FormControl>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value="kokoro">Kokoro 82M</SelectItem>
</SelectContent>
</Select>
</FormItem>
)}
/>
{/* Voice picker */}
<FormItem>
<FormLabel>Voice</FormLabel>
<div className="grid grid-cols-2 gap-1.5 max-h-[340px] overflow-y-auto pr-1">
{presetVoices.map((voice: PresetVoice) => (
<button
key={voice.voice_id}
type="button"
onClick={() => {
setSelectedPresetVoiceId(voice.voice_id);
// Auto-set language from voice
if (voice.language) {
form.setValue('language', voice.language as LanguageCode);
}
}}
className={`text-left px-3 py-2 rounded-md border text-sm transition-colors ${
selectedPresetVoiceId === voice.voice_id
? 'border-accent bg-accent/10 text-accent-foreground'
: 'border-border hover:bg-muted'
}`}
>
<div className="font-medium">{voice.name}</div>
<div className="flex gap-1.5 mt-0.5">
<Badge variant="outline" className="text-[10px] h-4 px-1">
{voice.gender}
</Badge>
<Badge variant="outline" className="text-[10px] h-4 px-1">
{voice.language}
</Badge>
</div>
</button>
))}
</div>
</FormItem>
</div>
) : (
<>
<Tabs
className="pt-0"
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsList
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
</TabsTrigger>
)}
</TabsList>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null &&
audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
)}
/>
</TabsContent>
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleRecording
file={selectedFile}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
)}
</Tabs>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</>
)}
</>
) : (
// Show sample list when editing
editingProfileId && (
// Editing mode
editingProfileId &&
editingProfile &&
(editingProfile.voice_type === 'preset' ? (
<div className="space-y-4 pt-4">
<div className="rounded-lg border border-border p-4 space-y-3">
<div className="text-sm font-medium text-muted-foreground">
Built-in Voice
</div>
<div className="flex items-center gap-3">
<div className="text-lg font-semibold">
{presetVoices.find(
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
)?.name ?? editingProfile.preset_voice_id}
</div>
<Badge variant="secondary" className="text-xs">
{editingProfile.preset_engine}
</Badge>
</div>
{(() => {
const voice = presetVoices.find(
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
);
return voice ? (
<div className="flex gap-1.5">
<Badge variant="outline" className="text-xs">
{voice.gender}
</Badge>
<Badge variant="outline" className="text-xs">
{voice.language}
</Badge>
</div>
) : null;
})()}
</div>
<p className="text-xs text-muted-foreground">
This profile uses a built-in voice. The voice cannot be changed after
creation.
</p>
</div>
) : (
<div>
<SampleList profileId={editingProfileId} />
</div>
)
))
)}
</div>
{/* Right column: Profile info */}
<div className="space-y-4">
<div className="space-y-4 overflow-y-auto min-h-0">
{/* Avatar Upload */}
<FormField
control={form.control}
@@ -924,6 +1122,37 @@ export function ProfileForm() {
)}
/>
<FormItem>
<FormLabel>Default Engine</FormLabel>
<Select
value={defaultEngine || '_none'}
onValueChange={(v) => {
setDefaultEngine(v === '_none' ? '' : v);
}}
disabled={
voiceSource === 'builtin' || editingProfile?.voice_type === 'preset'
}
>
<FormControl>
<SelectTrigger>
<SelectValue placeholder="No preference" />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value="_none">No preference</SelectItem>
<SelectItem value="qwen">Qwen3-TTS</SelectItem>
<SelectItem value="luxtts">LuxTTS</SelectItem>
<SelectItem value="chatterbox">Chatterbox</SelectItem>
<SelectItem value="chatterbox_turbo">Chatterbox Turbo</SelectItem>
<SelectItem value="tada">TADA</SelectItem>
<SelectItem value="kokoro">Kokoro 82M</SelectItem>
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Auto-selects this engine when the profile is chosen.
</p>
</FormItem>
{editingProfileId && (
<div className="space-y-2">
<FormLabel>Default Effects</FormLabel>
@@ -1,4 +1,4 @@
import { Mic, Sparkles } from 'lucide-react';
import { Mic, Music, Sparkles } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { Card, CardContent } from '@/components/ui/card';
import { useProfiles } from '@/lib/hooks/useProfiles';
@@ -6,9 +6,18 @@ import { useUIStore } from '@/stores/uiStore';
import { ProfileCard } from './ProfileCard';
import { ProfileForm } from './ProfileForm';
/** Engines that use preset (built-in) voices instead of cloned profiles. */
const PRESET_ENGINES = new Set(['kokoro']);
/** Human-readable engine names for empty state messages. */
const ENGINE_NAMES: Record<string, string> = {
kokoro: 'Kokoro',
};
export function ProfileList() {
const { data: profiles, isLoading, error } = useProfiles();
const setDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
const selectedEngine = useUIStore((state) => state.selectedEngine);
if (isLoading) {
return null;
@@ -23,6 +32,12 @@ export function ProfileList() {
}
const allProfiles = profiles || [];
const isPresetEngine = PRESET_ENGINES.has(selectedEngine);
// Filter profiles based on selected engine
const filteredProfiles = isPresetEngine
? allProfiles.filter((p) => p.voice_type === 'preset' && p.preset_engine === selectedEngine)
: allProfiles.filter((p) => p.voice_type !== 'preset');
return (
<div className="flex flex-col">
@@ -40,9 +55,25 @@ export function ProfileList() {
</Button>
</CardContent>
</Card>
) : filteredProfiles.length === 0 && isPresetEngine ? (
<Card>
<CardContent className="flex flex-col items-center justify-center py-12">
<Music className="h-12 w-12 text-muted-foreground mb-4" />
<p className="text-muted-foreground mb-2">
No {ENGINE_NAMES[selectedEngine] ?? selectedEngine} voices created yet.
</p>
<p className="text-sm text-muted-foreground mb-4">
The default voice will be used. Create a profile to choose a specific voice.
</p>
<Button onClick={() => setDialogOpen(true)}>
<Sparkles className="mr-2 h-4 w-4" />
Create {ENGINE_NAMES[selectedEngine] ?? selectedEngine} Voice
</Button>
</CardContent>
</Card>
) : (
<div className="flex gap-4 overflow-x-auto p-1 pb-1 lg:grid lg:grid-cols-3 lg:auto-rows-auto lg:overflow-x-visible lg:pb-[150px]">
{allProfiles.map((profile) => (
{filteredProfiles.map((profile) => (
<div key={profile.id} className="shrink-0 w-[200px] lg:w-auto lg:shrink">
<ProfileCard profile={profile} />
</div>
+11
View File
@@ -17,6 +17,7 @@ import type {
HistoryResponse,
ModelDownloadRequest,
ModelStatusListResponse,
PresetVoice,
ProfileSampleResponse,
StoryCreate,
StoryDetailResponse,
@@ -97,6 +98,16 @@ class ApiClient {
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`);
}
async listPresetVoices(engine: string): Promise<{ engine: string; voices: PresetVoice[] }> {
return this.request<{ engine: string; voices: PresetVoice[] }>(`/profiles/presets/${engine}`);
}
async seedPresetProfiles(
engine: string,
): Promise<{ engine: string; created: number; total_available: number }> {
return this.request(`/profiles/presets/${engine}/seed`, { method: 'POST' });
}
async updateProfile(profileId: string, data: VoiceProfileCreate): Promise<VoiceProfileResponse> {
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`, {
method: 'PUT',
+21 -2
View File
@@ -1,10 +1,17 @@
// API Types matching backend Pydantic models
import type { LanguageCode } from '@/lib/constants/languages';
export type VoiceType = 'cloned' | 'preset' | 'designed';
export interface VoiceProfileCreate {
name: string;
description?: string;
language: LanguageCode;
voice_type?: VoiceType;
preset_engine?: string;
preset_voice_id?: string;
design_prompt?: string;
default_engine?: string;
}
export interface VoiceProfileResponse {
@@ -14,12 +21,24 @@ export interface VoiceProfileResponse {
language: string;
avatar_path?: string;
effects_chain?: EffectConfig[];
voice_type: VoiceType;
preset_engine?: string;
preset_voice_id?: string;
design_prompt?: string;
default_engine?: string;
generation_count: number;
sample_count: number;
created_at: string;
updated_at: string;
}
export interface PresetVoice {
voice_id: string;
name: string;
gender: 'male' | 'female';
language: string;
}
export interface ProfileSampleCreate {
reference_text: string;
}
@@ -42,8 +61,8 @@ export interface GenerationRequest {
text: string;
language: LanguageCode;
seed?: number;
model_size?: '1.7B' | '0.6B';
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo';
model_size?: '1.7B' | '0.6B' | '1B' | '3B';
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro';
instruct?: string;
max_chunk_chars?: number;
crossfade_ms?: number;
+3
View File
@@ -5,6 +5,7 @@
* LuxTTS is English-only.
* Chatterbox Multilingual supports 23 languages.
* Chatterbox Turbo is English-only.
* Kokoro supports 8 languages.
*/
/** All languages that any engine supports. */
@@ -66,6 +67,8 @@ export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
'zh',
],
chatterbox_turbo: ['en'],
tada: ['en', 'ar', 'zh', 'de', 'es', 'fr', 'it', 'ja', 'pl', 'pt'],
kokoro: ['en', 'es', 'fr', 'hi', 'it', 'pt', 'ja', 'zh'],
} as const;
/** Helper: get language options for a given engine. */
+21 -9
View File
@@ -15,9 +15,9 @@ const generationSchema = z.object({
text: z.string().min(1, '').max(50000),
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
seed: z.number().int().optional(),
modelSize: z.enum(['1.7B', '0.6B']).optional(),
modelSize: z.enum(['1.7B', '0.6B', '1B', '3B']).optional(),
instruct: z.string().max(500).optional(),
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo']).optional(),
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada', 'kokoro']).optional(),
});
export type GenerationFormValues = z.infer<typeof generationSchema>;
@@ -79,7 +79,13 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
? 'chatterbox-tts'
: engine === 'chatterbox_turbo'
? 'chatterbox-turbo'
: `qwen-tts-${data.modelSize}`;
: engine === 'tada'
? data.modelSize === '3B'
? 'tada-3b-ml'
: 'tada-1b'
: engine === 'kokoro'
? 'kokoro'
: `qwen-tts-${data.modelSize}`;
const displayName =
engine === 'luxtts'
? 'LuxTTS'
@@ -87,9 +93,15 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
? 'Chatterbox TTS'
: engine === 'chatterbox_turbo'
? 'Chatterbox Turbo'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
: engine === 'tada'
? data.modelSize === '3B'
? 'TADA 3B Multilingual'
: 'TADA 1B'
: engine === 'kokoro'
? 'Kokoro 82M'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
// Check if model needs downloading
try {
@@ -104,7 +116,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
console.error('Failed to check model status:', error);
}
const isQwen = engine === 'qwen';
const hasModelSizes = engine === 'qwen' || engine === 'tada';
const effectsChain = options.getEffectsChain?.();
// This now returns immediately with status="generating"
const result = await generation.mutateAsync({
@@ -112,9 +124,9 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
text: data.text,
language: data.language,
seed: data.seed,
model_size: isQwen ? data.modelSize : undefined,
model_size: hasModelSizes ? data.modelSize : undefined,
engine,
instruct: isQwen ? data.instruct || undefined : undefined,
instruct: engine === 'qwen' ? data.instruct || undefined : undefined,
max_chunk_chars: maxChunkChars,
crossfade_ms: crossfadeMs,
normalize: normalizeAudio,
+7
View File
@@ -31,6 +31,10 @@ interface UIStore {
selectedProfileId: string | null;
setSelectedProfileId: (id: string | null) => void;
// Currently selected engine (synced from generation form)
selectedEngine: string;
setSelectedEngine: (engine: string) => void;
// Selected voice in Voices tab inspector
selectedVoiceId: string | null;
setSelectedVoiceId: (id: string | null) => void;
@@ -59,6 +63,9 @@ export const useUIStore = create<UIStore>((set) => ({
selectedProfileId: null,
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
selectedEngine: 'qwen',
setSelectedEngine: (engine) => set({ selectedEngine: engine }),
selectedVoiceId: null,
setSelectedVoiceId: (id) => set({ selectedVoiceId: id }),
+1 -1
View File
@@ -1,3 +1,3 @@
# Backend package
__version__ = "0.2.3"
__version__ = "0.3.1"
+40 -2
View File
@@ -166,6 +166,8 @@ TTS_ENGINES = {
"luxtts": "LuxTTS",
"chatterbox": "Chatterbox TTS",
"chatterbox_turbo": "Chatterbox Turbo",
"tada": "TADA",
"kokoro": "Kokoro",
}
@@ -259,6 +261,32 @@ def _get_non_qwen_tts_configs() -> list[ModelConfig]:
needs_trim=True,
languages=["en"],
),
ModelConfig(
model_name="tada-1b",
display_name="TADA 1B (English)",
engine="tada",
hf_repo_id="HumeAI/tada-1b",
model_size="1B",
size_mb=4000,
languages=["en"],
),
ModelConfig(
model_name="tada-3b-ml",
display_name="TADA 3B Multilingual",
engine="tada",
hf_repo_id="HumeAI/tada-3b-ml",
model_size="3B",
size_mb=8000,
languages=["en", "ar", "zh", "de", "es", "fr", "it", "ja", "pl", "pt"],
),
ModelConfig(
model_name="kokoro",
display_name="Kokoro 82M",
engine="kokoro",
hf_repo_id="hexgrad/Kokoro-82M",
size_mb=350,
languages=["en", "es", "fr", "hi", "it", "pt", "ja", "zh"],
),
]
@@ -339,10 +367,12 @@ def engine_has_model_sizes(engine: str) -> bool:
async def load_engine_model(engine: str, model_size: str = "default") -> None:
"""Load a model for the given engine, handling the Qwen model_size special case."""
"""Load a model for the given engine, handling engines with multiple model sizes."""
backend = get_tts_backend_for_engine(engine)
if engine == "qwen":
await backend.load_model_async(model_size)
elif engine == "tada":
await backend.load_model(model_size)
else:
await backend.load_model()
@@ -358,7 +388,7 @@ async def ensure_model_cached_or_raise(engine: str, model_size: str = "default")
cfg = c
break
if engine == "qwen":
if engine in ("qwen", "tada"):
if not backend._is_model_cached(model_size):
raise HTTPException(
status_code=400,
@@ -490,6 +520,14 @@ def get_tts_backend_for_engine(engine: str) -> TTSBackend:
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
backend = ChatterboxTurboTTSBackend()
elif engine == "tada":
from .hume_backend import HumeTadaBackend
backend = HumeTadaBackend()
elif engine == "kokoro":
from .kokoro_backend import KokoroTTSBackend
backend = KokoroTTSBackend()
else:
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
+347
View File
@@ -0,0 +1,347 @@
"""
HumeAI TADA TTS backend implementation.
Wraps HumeAI's TADA (Text-Acoustic Dual Alignment) model for
high-quality voice cloning. Two model variants:
- tada-1b: English-only, ~2B params (Llama 3.2 1B base)
- tada-3b-ml: Multilingual, ~4B params (Llama 3.2 3B base)
Both use a shared encoder/codec (HumeAI/tada-codec). The encoder
produces 1:1 aligned token embeddings from reference audio, and the
causal LM generates speech via flow-matching diffusion.
24kHz output, bf16 inference on CUDA, fp32 on CPU.
"""
import asyncio
import logging
import threading
from typing import ClassVar, List, Optional, Tuple
import numpy as np
from . import TTSBackend
from .base import (
is_model_cached,
get_torch_device,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
logger = logging.getLogger(__name__)
# HuggingFace repos
TADA_CODEC_REPO = "HumeAI/tada-codec"
TADA_1B_REPO = "HumeAI/tada-1b"
TADA_3B_ML_REPO = "HumeAI/tada-3b-ml"
TADA_MODEL_REPOS = {
"1B": TADA_1B_REPO,
"3B": TADA_3B_ML_REPO,
}
# Key weight files for cache detection
_TADA_MODEL_WEIGHT_FILES = [
"model.safetensors",
]
_TADA_CODEC_WEIGHT_FILES = [
"encoder/model.safetensors",
]
class HumeTadaBackend:
"""HumeAI TADA TTS backend for high-quality voice cloning."""
_load_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self):
self.model = None
self.encoder = None
self.model_size = "1B" # default to 1B
self._device = None
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
# Force CPU on macOS — MPS has issues with flow matching
# and large vocab lm_head (>65536 output channels)
return get_torch_device(force_cpu_on_mac=True)
def is_loaded(self) -> bool:
return self.model is not None
def _get_model_path(self, model_size: str = "1B") -> str:
return TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
def _is_model_cached(self, model_size: str = "1B") -> bool:
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
model_cached = is_model_cached(repo, required_files=_TADA_MODEL_WEIGHT_FILES)
codec_cached = is_model_cached(TADA_CODEC_REPO, required_files=_TADA_CODEC_WEIGHT_FILES)
return model_cached and codec_cached
async def load_model(self, model_size: str = "1B") -> None:
"""Load the TADA model and encoder."""
if self.model is not None and self.model_size == model_size:
return
async with self._model_load_lock:
if self.model is not None and self.model_size == model_size:
return
# Unload existing model if switching sizes
if self.model is not None:
self.unload_model()
self.model_size = model_size
await asyncio.to_thread(self._load_model_sync, model_size)
def _load_model_sync(self, model_size: str = "1B"):
"""Synchronous model loading with progress tracking."""
model_name = f"tada-{model_size.lower()}"
is_cached = self._is_model_cached(model_size)
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
with model_load_progress(model_name, is_cached):
# Install DAC shim before importing tada — tada's encoder/decoder
# import dac.nn.layers.Snake1d which requires the descript-audio-codec
# package. The real package pulls in onnx/tensorboard/matplotlib via
# descript-audiotools, so we use a lightweight shim instead.
from ..utils.dac_shim import install_dac_shim
install_dac_shim()
import torch
from huggingface_hub import snapshot_download
device = self._get_device()
self._device = device
logger.info(f"Loading HumeAI TADA {model_size} on {device}...")
# Download codec (encoder + decoder) if not cached
logger.info("Downloading TADA codec...")
snapshot_download(
repo_id=TADA_CODEC_REPO,
token=None,
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin"],
)
# Download model weights if not cached
logger.info(f"Downloading TADA {model_size} model...")
snapshot_download(
repo_id=repo,
token=None,
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin", "*.model"],
)
# TADA hardcodes "meta-llama/Llama-3.2-1B" as the tokenizer
# source in its Aligner and TadaForCausalLM.from_pretrained().
# That repo is gated (requires Meta license acceptance).
# Download the tokenizer from an ungated mirror and get its
# local cache path so we can point TADA at it directly.
logger.info("Downloading Llama tokenizer (ungated mirror)...")
tokenizer_path = snapshot_download(
repo_id="unsloth/Llama-3.2-1B",
token=None,
allow_patterns=["tokenizer*", "special_tokens*"],
)
# Determine dtype — use bf16 on CUDA for ~50% memory savings
if device == "cuda" and torch.cuda.is_bf16_supported():
model_dtype = torch.bfloat16
else:
model_dtype = torch.float32
# Patch the Aligner config class to use the local tokenizer
# path instead of the gated "meta-llama/Llama-3.2-1B" default.
# This avoids monkey-patching AutoTokenizer.from_pretrained
# which corrupts the classmethod descriptor for other engines.
from tada.modules.aligner import AlignerConfig
AlignerConfig.tokenizer_name = tokenizer_path
# Load encoder (only needed for voice prompt encoding)
from tada.modules.encoder import Encoder
logger.info("Loading TADA encoder...")
self.encoder = Encoder.from_pretrained(
TADA_CODEC_REPO, subfolder="encoder"
).to(device)
self.encoder.eval()
# Load the causal LM (includes decoder for wav generation).
# TadaForCausalLM.from_pretrained() calls
# getattr(config, "tokenizer_name", "meta-llama/Llama-3.2-1B")
# which hits the gated repo. Pre-load the config from HF,
# inject the local tokenizer path, then pass it in.
from tada.modules.tada import TadaForCausalLM, TadaConfig
logger.info(f"Loading TADA {model_size} model...")
config = TadaConfig.from_pretrained(repo)
config.tokenizer_name = tokenizer_path
self.model = TadaForCausalLM.from_pretrained(
repo, config=config, torch_dtype=model_dtype
).to(device)
self.model.eval()
logger.info(f"HumeAI TADA {model_size} loaded successfully on {device}")
def unload_model(self) -> None:
"""Unload model and encoder to free memory."""
if self.model is not None:
del self.model
self.model = None
if self.encoder is not None:
del self.encoder
self.encoder = None
self._device = None
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
logger.info("HumeAI TADA unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio using TADA's encoder.
TADA's encoder performs forced alignment between audio and text tokens,
producing an EncoderOutput with 1:1 token-audio alignment. If no
reference_text is provided, the encoder uses built-in ASR (English only).
We serialize the EncoderOutput to a dict for caching.
"""
await self.load_model(self.model_size)
cache_key = (
"tada_" + get_cache_key(audio_path, reference_text)
) if use_cache else None
if cache_key:
cached = get_cached_voice_prompt(cache_key)
if cached is not None and isinstance(cached, dict):
return cached, True
def _encode_sync():
import torch
import soundfile as sf
device = self._device
# Load audio with soundfile (torchaudio 2.10+ requires torchcodec)
audio_np, sr = sf.read(str(audio_path), dtype="float32")
audio = torch.from_numpy(audio_np).float()
if audio.ndim == 1:
audio = audio.unsqueeze(0) # (samples,) -> (1, samples)
else:
audio = audio.T # (samples, channels) -> (channels, samples)
audio = audio.to(device)
# Encode with forced alignment
text_arg = [reference_text] if reference_text else None
prompt = self.encoder(
audio, text=text_arg, sample_rate=sr
)
# Serialize EncoderOutput to a dict of CPU tensors for caching
prompt_dict = {}
for field_name in prompt.__dataclass_fields__:
val = getattr(prompt, field_name)
if isinstance(val, torch.Tensor):
prompt_dict[field_name] = val.detach().cpu()
elif isinstance(val, list):
prompt_dict[field_name] = val
elif isinstance(val, (int, float)):
prompt_dict[field_name] = val
else:
prompt_dict[field_name] = val
return prompt_dict
encoded = await asyncio.to_thread(_encode_sync)
if cache_key:
cache_voice_prompt(cache_key, encoded)
return encoded, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
return await _combine_voice_prompts(audio_paths, reference_texts, sample_rate=24000)
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text using HumeAI TADA.
Args:
text: Text to synthesize
voice_prompt: Serialized EncoderOutput dict from create_voice_prompt()
language: Language code (en, ar, de, es, fr, it, ja, pl, pt, zh)
seed: Random seed for reproducibility
instruct: Not supported by TADA (ignored)
Returns:
Tuple of (audio_array, sample_rate=24000)
"""
await self.load_model(self.model_size)
def _generate_sync():
import torch
from tada.modules.encoder import EncoderOutput
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
device = self._device
# Reconstruct EncoderOutput from the cached dict
restored = {}
for k, v in voice_prompt.items():
if isinstance(v, torch.Tensor):
# Move to device and match model dtype for float tensors
if v.is_floating_point():
model_dtype = next(self.model.parameters()).dtype
restored[k] = v.to(device=device, dtype=model_dtype)
else:
restored[k] = v.to(device=device)
else:
restored[k] = v
prompt = EncoderOutput(**restored)
# For non-English with the 3B-ML model, we could reload the
# encoder with the language-specific aligner. However, the
# generation itself is language-agnostic — only the encoder's
# aligner changes. Since we encode at create_voice_prompt time,
# the language is already baked in. For simplicity, we don't
# reload the encoder here.
logger.info(f"[TADA] Generating ({language}), text length: {len(text)}")
output = self.model.generate(
prompt=prompt,
text=text,
)
# output.audio is a list of tensors (one per batch item)
if output.audio and output.audio[0] is not None:
audio_tensor = output.audio[0]
audio = audio_tensor.detach().cpu().numpy().squeeze().astype(np.float32)
else:
logger.warning("[TADA] Generation produced no audio")
audio = np.zeros(24000, dtype=np.float32)
return audio, 24000
return await asyncio.to_thread(_generate_sync)
+288
View File
@@ -0,0 +1,288 @@
"""
Kokoro TTS backend implementation.
Wraps the Kokoro-82M model for fast, lightweight text-to-speech.
82M parameters, CPU realtime, 24kHz output, Apache 2.0 license.
Kokoro uses pre-built voice style vectors (not traditional zero-shot cloning
from arbitrary audio). Voice prompts are stored as deferred references to
HF-hosted voice .pt files.
Languages supported (via misaki G2P):
- American English (a), British English (b)
- Spanish (e), French (f), Hindi (h), Italian (i), Portuguese (p)
- Japanese (j) — requires misaki[ja]
- Chinese (z) — requires misaki[zh]
"""
import asyncio
import logging
import os
from typing import Optional
import numpy as np
from . import TTSBackend
from .base import (
get_torch_device,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
logger = logging.getLogger(__name__)
# HuggingFace repo for model + voice detection
KOKORO_HF_REPO = "hexgrad/Kokoro-82M"
KOKORO_SAMPLE_RATE = 24000
# Default voice if none specified
KOKORO_DEFAULT_VOICE = "af_heart"
# All available Kokoro voices: (voice_id, display_name, gender, lang_code)
KOKORO_VOICES = [
# American English female
("af_alloy", "Alloy", "female", "en"),
("af_aoede", "Aoede", "female", "en"),
("af_bella", "Bella", "female", "en"),
("af_heart", "Heart", "female", "en"),
("af_jessica", "Jessica", "female", "en"),
("af_kore", "Kore", "female", "en"),
("af_nicole", "Nicole", "female", "en"),
("af_nova", "Nova", "female", "en"),
("af_river", "River", "female", "en"),
("af_sarah", "Sarah", "female", "en"),
("af_sky", "Sky", "female", "en"),
# American English male
("am_adam", "Adam", "male", "en"),
("am_echo", "Echo", "male", "en"),
("am_eric", "Eric", "male", "en"),
("am_fenrir", "Fenrir", "male", "en"),
("am_liam", "Liam", "male", "en"),
("am_michael", "Michael", "male", "en"),
("am_onyx", "Onyx", "male", "en"),
("am_puck", "Puck", "male", "en"),
("am_santa", "Santa", "male", "en"),
# British English female
("bf_alice", "Alice", "female", "en"),
("bf_emma", "Emma", "female", "en"),
("bf_isabella", "Isabella", "female", "en"),
("bf_lily", "Lily", "female", "en"),
# British English male
("bm_daniel", "Daniel", "male", "en"),
("bm_fable", "Fable", "male", "en"),
("bm_george", "George", "male", "en"),
("bm_lewis", "Lewis", "male", "en"),
# Spanish
("ef_dora", "Dora", "female", "es"),
("em_alex", "Alex", "male", "es"),
("em_santa", "Santa", "male", "es"),
# French
("ff_siwis", "Siwis", "female", "fr"),
# Hindi
("hf_alpha", "Alpha", "female", "hi"),
("hf_beta", "Beta", "female", "hi"),
("hm_omega", "Omega", "male", "hi"),
("hm_psi", "Psi", "male", "hi"),
# Italian
("if_sara", "Sara", "female", "it"),
("im_nicola", "Nicola", "male", "it"),
# Japanese
("jf_alpha", "Alpha", "female", "ja"),
("jf_gongitsune", "Gongitsune", "female", "ja"),
("jf_nezumi", "Nezumi", "female", "ja"),
("jf_tebukuro", "Tebukuro", "female", "ja"),
("jm_kumo", "Kumo", "male", "ja"),
# Portuguese
("pf_dora", "Dora", "female", "pt"),
("pm_alex", "Alex", "male", "pt"),
("pm_santa", "Santa", "male", "pt"),
# Chinese
("zf_xiaobei", "Xiaobei", "female", "zh"),
("zf_xiaoni", "Xiaoni", "female", "zh"),
("zf_xiaoxiao", "Xiaoxiao", "female", "zh"),
("zf_xiaoyi", "Xiaoyi", "female", "zh"),
]
# Map our ISO language codes to Kokoro lang_code characters
LANG_CODE_MAP = {
"en": "a", # American English
"es": "e",
"fr": "f",
"hi": "h",
"it": "i",
"pt": "p",
"ja": "j",
"zh": "z",
}
class KokoroTTSBackend:
"""Kokoro-82M TTS backend — tiny, fast, CPU-friendly."""
def __init__(self):
self._model = None
self._pipelines: dict = {} # lang_code -> KPipeline
self._device: Optional[str] = None
self.model_size = "default"
def _get_device(self) -> str:
"""Select device. Kokoro supports CUDA and CPU. MPS needs fallback env var."""
device = get_torch_device(allow_mps=False)
# Kokoro can use MPS but requires PYTORCH_ENABLE_MPS_FALLBACK=1
# For now, skip MPS to avoid user confusion — CPU is already realtime
return device
@property
def device(self) -> str:
if self._device is None:
self._device = self._get_device()
return self._device
def is_loaded(self) -> bool:
return self._model is not None
def _get_model_path(self, model_size: str) -> str:
return KOKORO_HF_REPO
def _is_model_cached(self, model_size: str = "default") -> bool:
"""Check if Kokoro model files are cached locally."""
from .base import is_model_cached
return is_model_cached(
KOKORO_HF_REPO,
required_files=["config.json", "kokoro-v1_0.pth"],
)
async def load_model(self, model_size: str = "default") -> None:
"""Load the Kokoro model."""
if self._model is not None:
return
await asyncio.to_thread(self._load_model_sync)
def _load_model_sync(self):
"""Synchronous model loading."""
model_name = "kokoro"
is_cached = self._is_model_cached()
with model_load_progress(model_name, is_cached):
from kokoro import KModel
device = self.device
logger.info(f"Loading Kokoro-82M on {device}...")
self._model = KModel(repo_id=KOKORO_HF_REPO).to(device).eval()
logger.info("Kokoro-82M loaded successfully")
def _get_pipeline(self, lang_code: str):
"""Get or create a KPipeline for the given language code."""
kokoro_lang = LANG_CODE_MAP.get(lang_code, "a")
if kokoro_lang not in self._pipelines:
from kokoro import KPipeline
# Create pipeline with our existing model (no redundant model loading)
self._pipelines[kokoro_lang] = KPipeline(
lang_code=kokoro_lang,
repo_id=KOKORO_HF_REPO,
model=self._model,
)
return self._pipelines[kokoro_lang]
def unload_model(self) -> None:
"""Unload model to free memory."""
if self._model is not None:
del self._model
self._model = None
self._pipelines.clear()
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
logger.info("Kokoro unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> tuple[dict, bool]:
"""
Create voice prompt for Kokoro.
Kokoro doesn't do traditional voice cloning from arbitrary audio.
When called for a cloned profile (fallback), uses the default voice.
For preset profiles, the voice_prompt dict is built by the profile
service and bypasses this method entirely.
"""
return {
"voice_type": "preset",
"preset_engine": "kokoro",
"preset_voice_id": KOKORO_DEFAULT_VOICE,
}, False
async def combine_voice_prompts(
self,
audio_paths: list[str],
reference_texts: list[str],
) -> tuple[np.ndarray, str]:
"""Combine voice prompts — uses base implementation for audio concatenation."""
return await _combine_voice_prompts(
audio_paths, reference_texts, sample_rate=KOKORO_SAMPLE_RATE
)
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> tuple[np.ndarray, int]:
"""
Generate audio from text using Kokoro.
Args:
text: Text to synthesize
voice_prompt: Dict with kokoro_voice key
language: Language code
seed: Random seed for reproducibility
instruct: Not supported by Kokoro (ignored)
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model()
voice_name = voice_prompt.get("preset_voice_id") or voice_prompt.get("kokoro_voice") or KOKORO_DEFAULT_VOICE
def _generate_sync():
import torch
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
pipeline = self._get_pipeline(language)
# Generate all chunks and concatenate
audio_chunks = []
for result in pipeline(text, voice=voice_name, speed=1.0):
if result.audio is not None:
chunk = result.audio
if isinstance(chunk, torch.Tensor):
chunk = chunk.detach().cpu().numpy()
audio_chunks.append(chunk.squeeze())
if not audio_chunks:
# Return 1 second of silence as fallback
return np.zeros(KOKORO_SAMPLE_RATE, dtype=np.float32), KOKORO_SAMPLE_RATE
audio = np.concatenate(audio_chunks)
return audio.astype(np.float32), KOKORO_SAMPLE_RATE
return await asyncio.to_thread(_generate_sync)
+85 -5
View File
@@ -34,9 +34,15 @@ def build_server(cuda=False):
binary_name = "voicebox-server-cuda" if cuda else "voicebox-server"
# PyInstaller arguments
# CUDA builds use --onedir so we can split the output into two archives:
# 1. Server core (~200-400MB) — versioned with the app
# 2. CUDA libs (~2GB) — versioned independently (only redownloaded on
# CUDA toolkit / torch major version changes)
# CPU builds remain --onefile for simplicity.
pack_mode = "--onedir" if cuda else "--onefile"
args = [
"server.py", # Use server.py as entry point instead of main.py
"--onefile",
pack_mode,
"--name",
binary_name,
]
@@ -165,9 +171,9 @@ def build_server(cuda=False):
"tqdm",
"--hidden-import",
"requests",
"--collect-submodules",
"qwen_tts",
"--collect-data",
# qwen_tts uses inspect.getsource() at runtime to locate
# modeling_qwen3_tts.py — needs physical .py source files bundled
"--collect-all",
"qwen_tts",
# Fix for pkg_resources and jaraco namespace packages
"--hidden-import",
@@ -186,6 +192,80 @@ def build_server(cuda=False):
# needed by LuxTTS for text-to-phoneme conversion
"--collect-all",
"piper_phonemize",
# HumeAI TADA — speech-language model using Llama + flow matching
"--hidden-import",
"backend.backends.hume_backend",
"--hidden-import",
"tada",
"--hidden-import",
"tada.modules",
"--hidden-import",
"tada.modules.tada",
"--hidden-import",
"tada.modules.encoder",
"--hidden-import",
"tada.modules.decoder",
"--hidden-import",
"tada.modules.aligner",
"--hidden-import",
"tada.modules.acoustic_spkr_verf",
"--hidden-import",
"tada.nn",
"--hidden-import",
"tada.nn.vibevoice",
"--hidden-import",
"tada.utils",
"--hidden-import",
"tada.utils.gray_code",
"--hidden-import",
"tada.utils.text",
# DAC shim — provides dac.nn.layers.Snake1d without the real
# descript-audio-codec package (which pulls onnx/tensorboard via
# descript-audiotools). The shim is in backend/utils/dac_shim.py.
"--hidden-import",
"backend.utils.dac_shim",
"--hidden-import",
"torchaudio",
"--collect-submodules",
"tada",
# Kokoro 82M — lightweight TTS engine using misaki G2P
"--hidden-import",
"backend.backends.kokoro_backend",
"--hidden-import",
"kokoro",
"--hidden-import",
"kokoro.pipeline",
"--hidden-import",
"kokoro.model",
"--hidden-import",
"kokoro.istftnet",
"--hidden-import",
"kokoro.modules",
"--hidden-import",
"kokoro.custom_stft",
# misaki ships G2P data files (dictionaries, phoneme tables)
# that must be bundled for espeak/en/ja/zh G2P to work
"--collect-all",
"misaki",
# language_tags ships JSON data files (index.json etc.) loaded at
# runtime via: misaki → phonemizer → segments → csvw → language_tags
"--collect-all",
"language_tags",
# espeakng_loader ships the entire espeak-ng-data directory (369 files)
# loaded at import time by misaki.espeak via get_data_path()
"--collect-all",
"espeakng_loader",
# spacy en_core_web_sm model — misaki.en tries to spacy.cli.download()
# at runtime if not found, which calls pip as a subprocess and crashes
# the frozen binary. Bundle the model so spacy.util.is_package() passes.
"--collect-all",
"en_core_web_sm",
"--copy-metadata",
"en_core_web_sm",
"--hidden-import",
"en_core_web_sm",
"--hidden-import",
"loguru",
]
)
@@ -328,7 +408,7 @@ def build_server(cuda=False):
"torchvision",
"torchaudio",
"--index-url",
"https://download.pytorch.org/whl/cu126",
"https://download.pytorch.org/whl/cu128",
"--force-reinstall",
"-q",
],
+3 -3
View File
@@ -19,7 +19,7 @@ if _custom_models_dir:
logger.info("Model download path set to: %s", _custom_models_dir)
# Default data directory (used in development)
_data_dir = Path("data")
_data_dir = Path("data").resolve()
def set_data_dir(path: str | Path):
@@ -30,9 +30,9 @@ def set_data_dir(path: str | Path):
path: Path to the data directory
"""
global _data_dir
_data_dir = Path(path)
_data_dir = Path(path).resolve()
_data_dir.mkdir(parents=True, exist_ok=True)
logger.info("Data directory set to: %s", _data_dir.absolute())
logger.info("Data directory set to: %s", _data_dir)
def get_data_dir() -> Path:
+76
View File
@@ -34,6 +34,7 @@ def run_migrations(engine) -> None:
_migrate_generations(engine, inspector, tables)
_migrate_effect_presets(engine, inspector, tables)
_migrate_generation_versions(engine, inspector, tables)
_resolve_relative_paths(engine, tables)
# -- helpers ---------------------------------------------------------------
@@ -134,6 +135,17 @@ def _migrate_profiles(engine, inspector, tables: set[str]) -> None:
_add_column(engine, "profiles", "avatar_path VARCHAR", "avatar_path")
if "effects_chain" not in columns:
_add_column(engine, "profiles", "effects_chain TEXT", "effects_chain")
# Voice type system — v0.3.x
if "voice_type" not in columns:
_add_column(engine, "profiles", "voice_type VARCHAR DEFAULT 'cloned'", "voice_type")
if "preset_engine" not in columns:
_add_column(engine, "profiles", "preset_engine VARCHAR", "preset_engine")
if "preset_voice_id" not in columns:
_add_column(engine, "profiles", "preset_voice_id VARCHAR", "preset_voice_id")
if "design_prompt" not in columns:
_add_column(engine, "profiles", "design_prompt TEXT", "design_prompt")
if "default_engine" not in columns:
_add_column(engine, "profiles", "default_engine VARCHAR", "default_engine")
def _migrate_generations(engine, inspector, tables: set[str]) -> None:
@@ -168,3 +180,67 @@ def _migrate_generation_versions(engine, inspector, tables: set[str]) -> None:
columns = _get_columns(inspector, "generation_versions")
if "source_version_id" not in columns:
_add_column(engine, "generation_versions", "source_version_id VARCHAR", "source_version_id")
def _resolve_relative_paths(engine, tables: set[str]) -> None:
"""Resolve any relative file paths in the database to absolute paths.
Earlier versions stored paths relative to CWD (e.g. "data/generations/abc.wav").
These break when the production binary's CWD differs from the data directory.
This migration converts them to absolute paths using the configured data dir.
Idempotent: absolute paths are left untouched.
Strategy: paths like "data/generations/abc.wav" are rebased onto the
configured data directory. If the path starts with "data/", strip that
prefix and prepend get_data_dir(). Otherwise, try resolving relative to
CWD as a fallback.
"""
from pathlib import Path
from ..config import get_data_dir
data_dir = get_data_dir()
path_columns = [
("generations", "audio_path"),
("generation_versions", "audio_path"),
("profile_samples", "audio_path"),
("profiles", "avatar_path"),
]
total_fixed = 0
with engine.connect() as conn:
for table, column in path_columns:
if table not in tables:
continue
rows = conn.execute(
text(f"SELECT id, {column} FROM {table} WHERE {column} IS NOT NULL")
).fetchall()
for row_id, path_val in rows:
if not path_val:
continue
p = Path(path_val)
if p.is_absolute():
continue
# Try rebasing: "data/generations/abc.wav" → data_dir / "generations/abc.wav"
parts = p.parts
if parts and parts[0] == "data":
rebased = data_dir / Path(*parts[1:])
else:
rebased = data_dir / p
if rebased.exists():
resolved = rebased
else:
# Fallback: resolve relative to CWD
resolved = p.resolve()
if resolved.exists():
conn.execute(
text(f"UPDATE {table} SET {column} = :path WHERE id = :id"),
{"path": str(resolved), "id": row_id},
)
total_fixed += 1
if total_fixed > 0:
conn.commit()
logger.info("Resolved %d relative file paths to absolute", total_fixed)
+15 -1
View File
@@ -10,7 +10,13 @@ Base = declarative_base()
class VoiceProfile(Base):
"""Voice profile."""
"""Voice profile.
voice_type discriminates three flavours:
- "cloned" — traditional reference-audio profiles (all cloning engines)
- "preset" — engine-specific pre-built voice (e.g. Kokoro voices)
- "designed" — text-described voice (e.g. Qwen CustomVoice, future)
"""
__tablename__ = "profiles"
@@ -20,6 +26,14 @@ class VoiceProfile(Base):
language = Column(String, default="en")
avatar_path = Column(String, nullable=True)
effects_chain = Column(Text, nullable=True)
# Voice type system — added v0.3.x
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
design_prompt = Column(Text, nullable=True) # text description — only for designed
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
+12 -2
View File
@@ -15,6 +15,11 @@ class VoiceProfileCreate(BaseModel):
language: str = Field(
default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$"
)
voice_type: Optional[str] = Field(default="cloned", pattern="^(cloned|preset|designed)$")
preset_engine: Optional[str] = Field(None, max_length=50)
preset_voice_id: Optional[str] = Field(None, max_length=100)
design_prompt: Optional[str] = Field(None, max_length=2000)
default_engine: Optional[str] = Field(None, max_length=50)
class VoiceProfileResponse(BaseModel):
@@ -26,6 +31,11 @@ class VoiceProfileResponse(BaseModel):
language: str
avatar_path: Optional[str] = None
effects_chain: Optional[List["EffectConfig"]] = None
voice_type: str = "cloned"
preset_engine: Optional[str] = None
preset_voice_id: Optional[str] = None
design_prompt: Optional[str] = None
default_engine: Optional[str] = None
generation_count: int = 0
sample_count: int = 0
created_at: datetime
@@ -66,9 +76,9 @@ class GenerationRequest(BaseModel):
text: str = Field(..., min_length=1, max_length=50000)
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$")
seed: Optional[int] = Field(None, ge=0)
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B)$")
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B|1B|3B)$")
instruct: Optional[str] = Field(None, max_length=500)
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo)$")
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$")
max_chunk_chars: int = Field(
default=800, ge=100, le=5000, description="Max characters per chunk for long text splitting"
)
+15 -1
View File
@@ -8,7 +8,7 @@ sqlalchemy>=2.0.0
alembic>=1.13.0
# ML models
torch>=2.1.0
torch>=2.7.0
transformers>=4.36.0,<=4.57.6
accelerate>=0.26.0
huggingface_hub>=0.20.0
@@ -33,6 +33,20 @@ s3tokenizer
spacy-pkuseg
pyloudnorm
# HumeAI TADA sub-dependencies (hume-tada itself is installed
# --no-deps in the setup script because it pins torch>=2.7,<2.8.
# descript-audio-codec is NOT installed — it pulls onnx/tensorboard
# via descript-audiotools. A lightweight shim in utils/dac_shim.py
# provides the only class TADA uses: Snake1d.)
torchaudio
# Kokoro TTS (lightweight 82M-param engine)
kokoro>=0.9.4
misaki[en]>=0.9.4
# spacy model for misaki English G2P — must be pre-installed or misaki
# tries spacy.cli.download() at runtime which crashes frozen builds
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
# Audio processing
librosa>=0.10.0
soundfile>=0.12.0
+25 -1
View File
@@ -230,7 +230,15 @@ async def stream_speech(
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
engine = data.engine or "qwen"
# Mirror the regular /generate endpoint behavior more closely:
# if the caller doesn't specify an engine, prefer the profile's default
# engine (or preset engine) before falling back to qwen.
engine = (
data.engine
or getattr(profile, "default_engine", None)
or getattr(profile, "preset_engine", None)
or "qwen"
)
tts_model = get_tts_backend_for_engine(engine)
model_size = data.model_size or "1.7B"
@@ -263,6 +271,22 @@ async def stream_speech(
trim_fn=trim_fn,
)
effects_chain_config = None
if data.effects_chain is not None:
effects_chain_config = [e.model_dump() for e in data.effects_chain]
elif profile.effects_chain:
import json as _json
try:
effects_chain_config = _json.loads(profile.effects_chain)
except Exception:
effects_chain_config = None
if effects_chain_config:
from ..utils.effects import apply_effects
audio = apply_effects(audio, sample_rate, effects_chain_config)
if data.normalize:
from ..utils.audio import normalize_audio
+97
View File
@@ -1,9 +1,13 @@
"""Voice profile endpoints."""
import io
import json as _json
import logging
import tempfile
import uuid
from datetime import datetime
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
from fastapi.responses import FileResponse, StreamingResponse
@@ -15,6 +19,8 @@ from ..database import VoiceProfile as DBVoiceProfile, get_db
from ..services import channels, export_import, profiles
from ..services.profiles import _profile_to_response
logger = logging.getLogger(__name__)
router = APIRouter()
@@ -62,6 +68,97 @@ async def import_profile(
raise HTTPException(status_code=500, detail=str(e))
# ── Preset Voice Endpoints ───────────────────────────────────────────
# These MUST be declared before /profiles/{profile_id} to avoid the
# wildcard swallowing "presets" as a profile_id.
@router.get("/profiles/presets/{engine}")
async def list_preset_voices(engine: str):
"""List available preset voices for an engine."""
if engine == "kokoro":
from ..backends.kokoro_backend import KOKORO_VOICES
return {
"engine": engine,
"voices": [
{
"voice_id": vid,
"name": name,
"gender": gender,
"language": lang,
}
for vid, name, gender, lang in KOKORO_VOICES
],
}
return {"engine": engine, "voices": []}
@router.post("/profiles/presets/{engine}/seed")
async def seed_preset_profiles_route(
engine: str,
db: Session = Depends(get_db),
):
"""Seed preset voice profiles for an engine.
Creates profiles for all available preset voices that don't already exist.
Returns the count of newly created profiles.
"""
if engine != "kokoro":
raise HTTPException(status_code=400, detail=f"No presets available for engine: {engine}")
try:
from ..backends.kokoro_backend import KOKORO_VOICES
created = 0
for voice_id, display_name, gender, lang in KOKORO_VOICES:
profile_name = display_name
# Disambiguate duplicate display names across languages
# (e.g. "Alpha" exists in Hindi and Japanese, "Dora" in Spanish and Portuguese)
dupes = [v for v in KOKORO_VOICES if v[1] == display_name]
if len(dupes) > 1:
lang_labels = {"en": "English", "es": "Spanish", "fr": "French", "hi": "Hindi",
"it": "Italian", "pt": "Portuguese", "ja": "Japanese", "zh": "Chinese"}
profile_name = f"{display_name} {lang_labels.get(lang, lang)}"
# Skip if preset already exists
existing = (
db.query(DBVoiceProfile)
.filter_by(preset_engine="kokoro", preset_voice_id=voice_id)
.first()
)
if existing:
continue
# Skip name collisions
if db.query(DBVoiceProfile).filter_by(name=profile_name).first():
continue
profile = DBVoiceProfile(
id=str(uuid.uuid4()),
name=profile_name,
description=f"Kokoro preset voice — {display_name} ({gender})",
language=lang,
voice_type="preset",
preset_engine="kokoro",
preset_voice_id=voice_id,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
)
db.add(profile)
created += 1
if created > 0:
db.commit()
logger.info(f"Seeded {created} Kokoro preset profiles")
return {"engine": engine, "created": created, "total_available": len(KOKORO_VOICES)}
except Exception as e:
logger.exception(f"Failed to seed Kokoro profiles: {e}")
raise HTTPException(status_code=500, detail=str(e))
@router.get("/profiles/{profile_id}", response_model=models.VoiceProfileResponse)
async def get_profile(
profile_id: str,
+266 -119
View File
@@ -1,16 +1,22 @@
"""
CUDA backend binary download, assembly, and verification.
CUDA backend download, assembly, and verification.
Downloads split parts of the CUDA-enabled voicebox-server binary from
GitHub Releases, reassembles them, verifies integrity via SHA-256,
and places the binary in the app's data directory for use on next
backend restart.
Downloads two archives from GitHub Releases:
1. Server core (voicebox-server-cuda.tar.gz) — the exe + non-NVIDIA deps,
versioned with the app.
2. CUDA libs (cuda-libs-{version}.tar.gz) — NVIDIA runtime libraries,
versioned independently (only redownloaded on CUDA toolkit bump).
Both archives are extracted into {data_dir}/backends/cuda/ which forms the
complete PyInstaller --onedir directory structure that torch expects.
"""
import hashlib
import json
import logging
import os
import sys
import tarfile
from pathlib import Path
from typing import Optional
@@ -24,6 +30,10 @@ GITHUB_RELEASES_URL = "https://github.com/jamiepine/voicebox/releases/download"
PROGRESS_KEY = "cuda-backend"
# The current expected CUDA libs version. Bump this when we change the
# CUDA toolkit version or torch's CUDA dependency changes (e.g. cu126 -> cu128).
CUDA_LIBS_VERSION = "cu128-v1"
def get_backends_dir() -> Path:
"""Directory where downloaded backend binaries are stored."""
@@ -32,21 +42,46 @@ def get_backends_dir() -> Path:
return d
def get_cuda_binary_name() -> str:
"""Platform-specific CUDA binary filename."""
def get_cuda_dir() -> Path:
"""Directory where the CUDA backend (onedir) is extracted."""
d = get_backends_dir() / "cuda"
d.mkdir(parents=True, exist_ok=True)
return d
def get_cuda_exe_name() -> str:
"""Platform-specific CUDA executable filename."""
if sys.platform == "win32":
return "voicebox-server-cuda.exe"
return "voicebox-server-cuda"
def get_cuda_binary_path() -> Optional[Path]:
"""Return path to CUDA binary if it exists."""
p = get_backends_dir() / get_cuda_binary_name()
"""Return path to the CUDA executable if it exists inside the onedir."""
p = get_cuda_dir() / get_cuda_exe_name()
if p.exists():
return p
return None
def get_cuda_libs_manifest_path() -> Path:
"""Path to the cuda-libs.json manifest inside the CUDA dir."""
return get_cuda_dir() / "cuda-libs.json"
def get_installed_cuda_libs_version() -> Optional[str]:
"""Read the installed CUDA libs version from cuda-libs.json, or None."""
manifest_path = get_cuda_libs_manifest_path()
if not manifest_path.exists():
return None
try:
data = json.loads(manifest_path.read_text())
return data.get("version")
except Exception as e:
logger.warning(f"Could not read cuda-libs.json: {e}")
return None
def is_cuda_active() -> bool:
"""Check if the current process is the CUDA binary.
@@ -60,25 +95,151 @@ def get_cuda_status() -> dict:
progress_manager = get_progress_manager()
cuda_path = get_cuda_binary_path()
progress = progress_manager.get_progress(PROGRESS_KEY)
cuda_libs_version = get_installed_cuda_libs_version()
return {
"available": cuda_path is not None,
"active": is_cuda_active(),
"binary_path": str(cuda_path) if cuda_path else None,
"cuda_libs_version": cuda_libs_version,
"downloading": progress is not None and progress.get("status") == "downloading",
"download_progress": progress,
}
async def download_cuda_binary(version: Optional[str] = None):
"""Download the CUDA backend binary from GitHub Releases.
def _needs_server_download(version: Optional[str] = None) -> bool:
"""Check if the server core archive needs to be (re)downloaded."""
cuda_path = get_cuda_binary_path()
if not cuda_path:
return True
# Check if the binary version matches the expected app version
installed = get_cuda_binary_version()
expected = version or __version__
if expected.startswith("v"):
expected = expected[1:]
return installed != expected
Downloads split parts listed in a manifest file, concatenates them,
and verifies the SHA-256 checksum for integrity. Atomic write
(temp file -> rename).
def _needs_cuda_libs_download() -> bool:
"""Check if the CUDA libs archive needs to be (re)downloaded."""
installed = get_installed_cuda_libs_version()
if installed is None:
return True
return installed != CUDA_LIBS_VERSION
async def _download_and_extract_archive(
client,
url: str,
sha256_url: Optional[str],
dest_dir: Path,
label: str,
progress_offset: int,
total_size: int,
):
"""Download a .tar.gz archive and extract it into dest_dir.
Args:
version: Version tag (e.g. "v0.2.0"). Defaults to current app version.
client: httpx.AsyncClient
url: URL of the .tar.gz archive
sha256_url: URL of the .sha256 checksum file (optional)
dest_dir: Directory to extract into
label: Human-readable label for progress updates
progress_offset: Byte offset for progress reporting (when downloading
multiple archives sequentially)
total_size: Total bytes across all downloads (for progress bar)
"""
progress = get_progress_manager()
temp_path = dest_dir / f".download-{label.replace(' ', '-')}.tmp"
# Clean up leftover partial download
if temp_path.exists():
temp_path.unlink()
# Fetch expected checksum (fail-fast: never extract an unverified archive)
expected_sha = None
if sha256_url:
try:
sha_resp = await client.get(sha256_url)
sha_resp.raise_for_status()
expected_sha = sha_resp.text.strip().split()[0]
logger.info(f"{label}: expected SHA-256: {expected_sha[:16]}...")
except Exception as e:
raise RuntimeError(f"{label}: failed to fetch checksum from {sha256_url}") from e
# Stream download, verify, and extract — always clean up temp file
downloaded = 0
try:
async with client.stream("GET", url) as response:
response.raise_for_status()
with open(temp_path, "wb") as f:
async for chunk in response.aiter_bytes(chunk_size=1024 * 1024):
f.write(chunk)
downloaded += len(chunk)
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Downloading {label}",
status="downloading",
)
# Verify integrity
if expected_sha:
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Verifying {label}...",
status="downloading",
)
sha256 = hashlib.sha256()
with open(temp_path, "rb") as f:
while True:
data = f.read(1024 * 1024)
if not data:
break
sha256.update(data)
actual = sha256.hexdigest()
if actual != expected_sha:
raise ValueError(
f"{label} integrity check failed: expected {expected_sha[:16]}..., got {actual[:16]}..."
)
logger.info(f"{label}: integrity verified")
# Extract (use data filter for path traversal protection on Python 3.12+)
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Extracting {label}...",
status="downloading",
)
with tarfile.open(temp_path, "r:gz") as tar:
if sys.version_info >= (3, 12):
tar.extractall(path=dest_dir, filter="data")
else:
tar.extractall(path=dest_dir)
logger.info(f"{label}: extracted to {dest_dir}")
finally:
if temp_path.exists():
temp_path.unlink()
return downloaded
async def download_cuda_binary(version: Optional[str] = None):
"""Download the CUDA backend (server core + CUDA libs if needed).
Downloads both archives from GitHub Releases, extracts them into
{data_dir}/backends/cuda/, and writes the cuda-libs.json manifest.
Only downloads what's needed:
- Server core: always redownloaded (versioned with app)
- CUDA libs: only if missing or version mismatch
Args:
version: Version tag (e.g. "v0.3.0"). Defaults to current app version.
"""
import httpx
@@ -86,114 +247,91 @@ async def download_cuda_binary(version: Optional[str] = None):
version = f"v{__version__}"
progress = get_progress_manager()
binary_name = get_cuda_binary_name()
dest_dir = get_backends_dir()
final_path = dest_dir / binary_name
temp_path = dest_dir / f"{binary_name}.download"
cuda_dir = get_cuda_dir()
# Clean up any leftover partial download
if temp_path.exists():
temp_path.unlink()
need_server = _needs_server_download(version)
need_libs = _needs_cuda_libs_download()
logger.info(f"Starting CUDA backend download for {version}")
if not need_server and not need_libs:
logger.info("CUDA backend is up to date, nothing to download")
return
logger.info(
f"Starting CUDA backend download for {version} "
f"(server={'yes' if need_server else 'cached'}, "
f"libs={'yes' if need_libs else 'cached'})"
)
progress.update_progress(
PROGRESS_KEY, current=0, total=0,
filename="Fetching manifest...", status="downloading",
PROGRESS_KEY,
current=0,
total=0,
filename="Preparing download...",
status="downloading",
)
base_url = f"{GITHUB_RELEASES_URL}/{version}"
stem = Path(binary_name).stem # voicebox-server-cuda
server_archive = "voicebox-server-cuda.tar.gz"
libs_archive = f"cuda-libs-{CUDA_LIBS_VERSION}.tar.gz"
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=30.0) as client:
# Fetch the manifest (list of split part filenames)
manifest_url = f"{base_url}/{stem}.manifest"
manifest_resp = await client.get(manifest_url)
manifest_resp.raise_for_status()
parts = [p.strip() for p in manifest_resp.text.strip().splitlines() if p.strip()]
if not parts:
raise ValueError("Empty manifest — no split parts found")
logger.info(f"Found {len(parts)} split parts to download")
# Fetch expected checksum (optional — for integrity verification)
expected_sha = None
try:
sha_url = f"{base_url}/{stem}.sha256"
sha_resp = await client.get(sha_url)
if sha_resp.status_code == 200:
# Format: "sha256hex filename\n"
expected_sha = sha_resp.text.strip().split()[0]
logger.info(f"Expected SHA-256: {expected_sha[:16]}...")
except Exception as e:
logger.warning(f"Could not fetch checksum file — skipping verification: {e}")
# Get total size across all parts by issuing HEAD requests
# Estimate total download size
total_size = 0
for part_name in parts:
if need_server:
try:
head_resp = await client.head(f"{base_url}/{part_name}")
content_length = int(head_resp.headers.get("content-length", 0))
total_size += content_length
head = await client.head(f"{base_url}/{server_archive}")
total_size += int(head.headers.get("content-length", 0))
except Exception:
pass
if need_libs:
try:
head = await client.head(f"{base_url}/{libs_archive}")
total_size += int(head.headers.get("content-length", 0))
except Exception:
pass
logger.info(f"Total download size: {total_size / 1024 / 1024:.1f} MB")
# Download and concatenate parts
total_downloaded = 0
with open(temp_path, "wb") as f:
for i, part_name in enumerate(parts):
part_url = f"{base_url}/{part_name}"
logger.info(f"Downloading part {i + 1}/{len(parts)}: {part_name}")
offset = 0
async with client.stream("GET", part_url) as response:
response.raise_for_status()
async for chunk in response.aiter_bytes(chunk_size=1024 * 1024):
f.write(chunk)
total_downloaded += len(chunk)
progress.update_progress(
PROGRESS_KEY, current=total_downloaded, total=total_size,
filename=f"Downloading CUDA backend ({i + 1}/{len(parts)})",
status="downloading",
)
# Verify integrity if checksum was available
if expected_sha:
progress.update_progress(
PROGRESS_KEY, current=total_downloaded, total=total_downloaded,
filename="Verifying integrity...", status="downloading",
)
sha256 = hashlib.sha256()
with open(temp_path, "rb") as f:
while True:
chunk = f.read(1024 * 1024)
if not chunk:
break
sha256.update(chunk)
actual = sha256.hexdigest()
if actual != expected_sha:
raise ValueError(
f"Integrity check failed: expected {expected_sha[:16]}..., "
f"got {actual[:16]}..."
# Download server core
if need_server:
server_downloaded = await _download_and_extract_archive(
client,
url=f"{base_url}/{server_archive}",
sha256_url=f"{base_url}/{server_archive}.sha256",
dest_dir=cuda_dir,
label="CUDA server",
progress_offset=offset,
total_size=total_size,
)
logger.info(f"Integrity verified: {actual[:16]}...")
offset += server_downloaded
# Atomic move into place (replace handles existing target on all platforms)
temp_path.replace(final_path)
# Make executable on Unix
exe_path = cuda_dir / get_cuda_exe_name()
if sys.platform != "win32" and exe_path.exists():
exe_path.chmod(0o755)
# Make executable on Unix
if sys.platform != "win32":
final_path.chmod(0o755)
# Download CUDA libs
if need_libs:
await _download_and_extract_archive(
client,
url=f"{base_url}/{libs_archive}",
sha256_url=f"{base_url}/{libs_archive}.sha256",
dest_dir=cuda_dir,
label="CUDA libraries",
progress_offset=offset,
total_size=total_size,
)
logger.info(f"CUDA backend downloaded to {final_path}")
# Write local cuda-libs.json manifest
manifest = {"version": CUDA_LIBS_VERSION}
get_cuda_libs_manifest_path().write_text(json.dumps(manifest, indent=2) + "\n")
logger.info(f"CUDA backend ready at {cuda_dir}")
progress.mark_complete(PROGRESS_KEY)
except Exception as e:
# Clean up on failure
if temp_path.exists():
temp_path.unlink()
logger.error(f"CUDA backend download failed: {e}")
progress.mark_error(PROGRESS_KEY, str(e))
raise
@@ -202,15 +340,19 @@ async def download_cuda_binary(version: Optional[str] = None):
def get_cuda_binary_version() -> Optional[str]:
"""Get the version of the installed CUDA binary, or None if not installed."""
import subprocess
cuda_path = get_cuda_binary_path()
if not cuda_path:
return None
try:
result = subprocess.run(
[str(cuda_path), "--version"],
capture_output=True, text=True, timeout=30,
capture_output=True,
text=True,
timeout=30,
cwd=str(cuda_path.parent), # Run from the onedir directory
)
# Output format: "voicebox-server 0.2.0"
# Output format: "voicebox-server 0.3.0"
for line in result.stdout.strip().splitlines():
if "voicebox-server" in line:
return line.split()[-1]
@@ -222,26 +364,29 @@ def get_cuda_binary_version() -> Optional[str]:
async def check_and_update_cuda_binary():
"""Check if the CUDA binary is outdated and auto-download if so.
Called on server startup. If a CUDA binary exists but its version
doesn't match the current app version, triggers a background download
of the updated CUDA binary. The download progress is visible to the
frontend via the existing SSE progress endpoint.
Called on server startup. Checks both server version and CUDA libs
version. Downloads only what's needed.
"""
cuda_path = get_cuda_binary_path()
if not cuda_path:
return # No CUDA binary installed, nothing to update
cuda_version = get_cuda_binary_version()
current_version = __version__
need_server = _needs_server_download()
need_libs = _needs_cuda_libs_download()
if cuda_version == current_version:
logger.info(f"CUDA binary is up to date (v{current_version})")
if not need_server and not need_libs:
logger.info(f"CUDA binary is up to date (server=v{__version__}, libs={get_installed_cuda_libs_version()})")
return
logger.info(
f"CUDA binary version mismatch: binary=v{cuda_version}, app=v{current_version}. "
f"Auto-downloading updated CUDA backend..."
)
reasons = []
if need_server:
cuda_version = get_cuda_binary_version()
reasons.append(f"server v{cuda_version} != v{__version__}")
if need_libs:
installed_libs = get_installed_cuda_libs_version()
reasons.append(f"libs {installed_libs} != {CUDA_LIBS_VERSION}")
logger.info(f"CUDA backend needs update ({', '.join(reasons)}). Auto-downloading...")
try:
await download_cuda_binary()
@@ -250,10 +395,12 @@ async def check_and_update_cuda_binary():
async def delete_cuda_binary() -> bool:
"""Delete the downloaded CUDA binary. Returns True if deleted."""
path = get_cuda_binary_path()
if path and path.exists():
path.unlink()
logger.info(f"Deleted CUDA binary: {path}")
"""Delete the downloaded CUDA backend directory. Returns True if deleted."""
import shutil
cuda_dir = get_cuda_dir()
if cuda_dir.exists() and any(cuda_dir.iterdir()):
shutil.rmtree(cuda_dir)
logger.info(f"Deleted CUDA backend directory: {cuda_dir}")
return True
return False
+52 -2
View File
@@ -2,6 +2,7 @@
Voice profile management module.
"""
import logging
from typing import List, Optional
from datetime import datetime
import uuid
@@ -10,6 +11,8 @@ from pathlib import Path
from sqlalchemy.orm import Session
from sqlalchemy import func, select
logger = logging.getLogger(__name__)
from ..models import (
VoiceProfileCreate,
VoiceProfileResponse,
@@ -52,6 +55,11 @@ def _profile_to_response(
language=profile.language,
avatar_path=profile.avatar_path,
effects_chain=effects_chain,
voice_type=getattr(profile, "voice_type", None) or "cloned",
preset_engine=getattr(profile, "preset_engine", None),
preset_voice_id=getattr(profile, "preset_voice_id", None),
design_prompt=getattr(profile, "design_prompt", None),
default_engine=getattr(profile, "default_engine", None),
generation_count=generation_count,
sample_count=sample_count,
created_at=profile.created_at,
@@ -80,11 +88,22 @@ async def create_profile(
if existing_profile:
raise ValueError(f"A profile with the name '{data.name}' already exists. Please choose a different name.")
# Auto-set default_engine for preset profiles
default_engine = data.default_engine
voice_type = data.voice_type or "cloned"
if voice_type == "preset" and data.preset_engine and not default_engine:
default_engine = data.preset_engine
db_profile = DBVoiceProfile(
id=str(uuid.uuid4()),
name=data.name,
description=data.description,
language=data.language,
voice_type=voice_type,
preset_engine=data.preset_engine,
preset_voice_id=data.preset_voice_id,
design_prompt=data.design_prompt,
default_engine=default_engine,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
)
@@ -265,6 +284,8 @@ async def update_profile(
profile.name = data.name
profile.description = data.description
profile.language = data.language
if data.default_engine is not None:
profile.default_engine = data.default_engine or None # empty string → NULL
profile.updated_at = datetime.utcnow()
db.commit()
@@ -382,19 +403,45 @@ async def create_voice_prompt_for_profile(
engine: str = "qwen",
) -> dict:
"""
Create a combined voice prompt from all samples in a profile.
Create a voice prompt from a profile.
For cloned profiles: combines all audio samples into a voice prompt.
For preset profiles: returns the engine-specific preset voice reference.
For designed profiles: returns the text design prompt (future).
Args:
profile_id: Profile ID
db: Database session
use_cache: Whether to use cached prompts
engine: TTS engine to create prompt for ("qwen" or "luxtts")
engine: TTS engine to create prompt for
Returns:
Voice prompt dictionary
"""
from ..backends import get_tts_backend_for_engine
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile not found: {profile_id}")
voice_type = getattr(profile, "voice_type", None) or "cloned"
# ── Preset profiles: return engine-specific voice reference ──
if voice_type == "preset":
return {
"voice_type": "preset",
"preset_engine": profile.preset_engine,
"preset_voice_id": profile.preset_voice_id,
}
# ── Designed profiles: return text description (future) ──
if voice_type == "designed":
return {
"voice_type": "designed",
"design_prompt": profile.design_prompt,
}
# ── Cloned profiles: create from audio samples ──
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
if not samples:
@@ -524,3 +571,6 @@ async def delete_avatar(
db.commit()
return True
+95
View File
@@ -0,0 +1,95 @@
"""
Minimal shim for descript-audio-codec (DAC).
TADA only imports Snake1d from dac.nn.layers and dac.model.dac.
The real DAC package pulls in descript-audiotools which depends on
onnx, tensorboard, protobuf, matplotlib, pystoi, etc. — none of
which are needed for TADA's runtime use of Snake1d.
This shim provides the exact Snake1d implementation (MIT-licensed,
from https://github.com/descriptinc/descript-audio-codec) so we can
avoid the entire audiotools dependency chain.
If the real DAC package is installed, this module is never used —
Python's import system will find the site-packages version first.
Install this shim only when descript-audio-codec is NOT installed.
"""
import sys
import types
import torch
import torch.nn as nn
# ── Snake activation (from dac/nn/layers.py) ────────────────────────
# NOTE: The original DAC code uses @torch.jit.script here for a 1.4x
# speedup. We omit it because TorchScript calls inspect.getsource()
# which fails inside a PyInstaller frozen binary (no .py source files).
def snake(x: torch.Tensor, alpha: torch.Tensor) -> torch.Tensor:
shape = x.shape
x = x.reshape(shape[0], shape[1], -1)
x = x + (alpha + 1e-9).reciprocal() * torch.sin(alpha * x).pow(2)
x = x.reshape(shape)
return x
class Snake1d(nn.Module):
def __init__(self, channels: int):
super().__init__()
self.alpha = nn.Parameter(torch.ones(1, channels, 1))
def forward(self, x: torch.Tensor) -> torch.Tensor:
return snake(x, self.alpha)
# ── Register as dac.nn.layers and dac.model.dac ─────────────────────
def install_dac_shim() -> None:
"""Register fake dac package modules in sys.modules.
Only installs the shim if 'dac' is not already importable
(i.e. the real descript-audio-codec is not installed).
"""
try:
import dac # noqa: F401 — real package exists, do nothing
return
except ImportError:
pass
# Create the module tree: dac -> dac.nn -> dac.nn.layers
# -> dac.model -> dac.model.dac
dac_pkg = types.ModuleType("dac")
dac_pkg.__path__ = [] # make it a package
dac_pkg.__package__ = "dac"
dac_nn = types.ModuleType("dac.nn")
dac_nn.__path__ = []
dac_nn.__package__ = "dac.nn"
dac_nn_layers = types.ModuleType("dac.nn.layers")
dac_nn_layers.__package__ = "dac.nn"
dac_nn_layers.Snake1d = Snake1d
dac_nn_layers.snake = snake
dac_model = types.ModuleType("dac.model")
dac_model.__path__ = []
dac_model.__package__ = "dac.model"
dac_model_dac = types.ModuleType("dac.model.dac")
dac_model_dac.__package__ = "dac.model"
dac_model_dac.Snake1d = Snake1d
# Wire up submodules
dac_pkg.nn = dac_nn
dac_pkg.model = dac_model
dac_nn.layers = dac_nn_layers
dac_model.dac = dac_model_dac
# Register in sys.modules
sys.modules["dac"] = dac_pkg
sys.modules["dac.nn"] = dac_nn
sys.modules["dac.nn.layers"] = dac_nn_layers
sys.modules["dac.model"] = dac_model
sys.modules["dac.model.dac"] = dac_model_dac
+13 -4
View File
@@ -1,13 +1,11 @@
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.hooks import collect_submodules
from PyInstaller.utils.hooks import collect_all
from PyInstaller.utils.hooks import copy_metadata
datas = []
binaries = []
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += collect_data_files('qwen_tts')
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'kokoro', 'kokoro.pipeline', 'kokoro.model', 'kokoro.istftnet', 'kokoro.modules', 'kokoro.custom_stft', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += copy_metadata('qwen-tts')
datas += copy_metadata('requests')
datas += copy_metadata('transformers')
@@ -15,8 +13,9 @@ datas += copy_metadata('huggingface-hub')
datas += copy_metadata('tokenizers')
datas += copy_metadata('safetensors')
datas += copy_metadata('tqdm')
hiddenimports += collect_submodules('qwen_tts')
datas += copy_metadata('en_core_web_sm')
hiddenimports += collect_submodules('jaraco')
hiddenimports += collect_submodules('tada')
hiddenimports += collect_submodules('mlx')
hiddenimports += collect_submodules('mlx_audio')
tmp_ret = collect_all('zipvoice')
@@ -27,12 +26,22 @@ tmp_ret = collect_all('lazy_loader')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('librosa')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('qwen_tts')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('inflect')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('perth')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('piper_phonemize')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('misaki')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('language_tags')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('espeakng_loader')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('en_core_web_sm')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('mlx')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('mlx_audio')
+7 -5
View File
@@ -159,12 +159,14 @@ Tauri looks for `voicebox-server-${PLATFORM}` in `src-tauri/binaries/` and bundl
The `build-cuda-windows` job runs separately:
1. Install PyTorch with CUDA 12.1
2. Build with `build_binary.py --cuda`
3. Split binary with `scripts/split_binary.py`
4. Upload parts as release artifacts
1. Install PyTorch with CUDA 12.8
2. Build with `build_binary.py --cuda` (produces `--onedir` output)
3. Package with `scripts/package_cuda.py` into two archives:
- `voicebox-server-cuda.tar.gz` — server core (~945 MB)
- `cuda-libs-cu128-v1.tar.gz` — NVIDIA runtime libraries (~1.7 GB, cached independently)
4. Upload archives as release artifacts
This binary is downloaded on-demand by users who enable CUDA in settings.
This binary is downloaded on-demand by users who enable CUDA in settings. The CUDA libs archive is only re-downloaded when the CUDA toolkit version changes, not on every app update.
## Troubleshooting
+468 -29
View File
@@ -3,8 +3,12 @@ title: "TTS Engines"
description: "How to add new text-to-speech engines to Voicebox"
---
> **For humans:** This doc is optimized for AI agents to implement new TTS engines autonomously. It's structured as a phased workflow with explicit gates and a checklist so an agent can do the full integration — dependency research, backend, frontend, bundling — and hand you a draft release or prod build to test locally. It's also a useful reference if you're doing it yourself.
Adding an engine touches ~10 files across 4 layers. The backend protocol work is straightforward — the real time sink is dependency hell, upstream library bugs, and PyInstaller bundling.
**Do not start writing code until you complete Phase 0.** The v0.2.3 release was three patch releases of PyInstaller fixes because dependency research was skipped. Every issue — `inspect.getsource()` failures, missing native data files, metadata lookups, dtype mismatches — was discoverable by reading the model library's source code before integration began.
## Architecture Overview
The backend is split into layers:
@@ -18,6 +22,133 @@ The backend is split into layers:
New engines only need to touch `backends/` and `models.py` on the backend side — the route and service layers use a model config registry that handles dispatch automatically.
## Phase 0: Dependency Research
**This phase is mandatory.** Clone the model library and its key dependencies into a temporary directory and inspect them before writing any integration code. The goal is to produce a dependency audit that identifies every PyInstaller-incompatible pattern, every native data file, and every upstream bug you'll need to work around.
### 0.1 Clone and Inspect the Model Library
```bash
# Create a throwaway workspace
mkdir /tmp/engine-research && cd /tmp/engine-research
# Clone the model library
git clone https://github.com/org/model-library.git
cd model-library
```
**Read these files first, in order:**
1. **`setup.py` / `setup.cfg` / `pyproject.toml`** — Check pinned dependency versions. If the library pins `torch==2.6.0` or `numpy<1.26`, you'll need `--no-deps` installation and manual sub-dependency listing (this is what happened with `chatterbox-tts`).
2. **`__init__.py` and the main model class** — Trace the import chain. Look for:
- `from_pretrained()` — does it call `huggingface_hub` internally? Does it pass `token=True` (which crashes without a stored HF token)?
- `from_local()` — does it exist? You may need manual `snapshot_download()` + `from_local()` to bypass download bugs.
- Device handling — does it default to CUDA? Does it support MPS? Many libraries crash on MPS with unsupported operators.
3. **All `import` statements** — Recursively trace what the library imports. You're looking for:
- `inspect.getsource()` anywhere in the chain (search all `.py` files)
- `typeguard` / `@typechecked` decorators (these call `inspect.getsource()` at import time)
- `importlib.metadata.version()` or `pkg_resources.get_distribution()` (need `--copy-metadata`)
- `lazy_loader` (needs `--collect-all` to bundle `.pyi` stubs)
### 0.2 Scan for PyInstaller-Incompatible Patterns
Run these searches against the cloned library **and** its transitive dependencies:
```bash
# inspect.getsource — will crash in frozen binary without --collect-all
grep -r "inspect.getsource\|getsource(" .
# typeguard / @typechecked — calls inspect.getsource at import time
grep -r "@typechecked\|from typeguard" .
# importlib.metadata — needs --copy-metadata
grep -r "importlib.metadata\|pkg_resources.get_distribution\|pkg_resources.require" .
# Data files loaded at runtime — need --collect-all or --collect-data
grep -r "Path(__file__).parent\|os.path.dirname(__file__)\|resources_path\|pkg_resources.resource_filename" .
# Native library paths — may need env var override in frozen builds
grep -r "/usr/share\|/usr/lib\|/usr/local\|espeak\|phonemize" .
# torch.load without map_location — will crash on CPU-only builds
grep -r "torch.load(" . | grep -v "map_location"
# HuggingFace token bugs
grep -r 'token=True\|token=os.getenv' .
# Float64/Float32 assumptions — librosa returns float64, many models assume float32
grep -r "torch.from_numpy\|\.double()\|float64" .
# @torch.jit.script — calls inspect.getsource(), crashes in frozen builds
grep -r "@torch.jit.script\|torch.jit.script" .
# torchaudio.load — requires torchcodec in torchaudio 2.10+, use soundfile.read() instead
grep -r "torchaudio.load\|torchaudio.save" .
# Gated HuggingFace repos — models that hardcode gated repos as tokenizer/config sources
grep -r "from_pretrained\|tokenizer_name\|AutoTokenizer" . | grep -i "llama\|meta-llama\|gated"
```
### 0.3 Install and Trace in a Throwaway Venv
```bash
# Create isolated venv
python -m venv /tmp/engine-venv
source /tmp/engine-venv/bin/activate
# Install the package (try normally first)
pip install model-package
# Check if it conflicts with our stack
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
# If this fails, you need --no-deps:
pip install --no-deps model-package
# Get the full dependency tree
pip show model-package # Check Requires: field
pip show -f model-package # List all installed files (look for data files)
# Check for non-PyPI dependencies
pip install model-package 2>&1 | grep -i "no matching distribution"
```
### 0.4 Test Model Loading on CPU
Before writing any integration code, verify the model works on CPU in a plain Python script:
```python
import torch
# Force CPU to catch map_location bugs early
model = ModelClass.from_pretrained("org/model", device="cpu")
# Test with a float32 audio array (not float64)
import numpy as np
audio = np.random.randn(16000).astype(np.float32)
output = model.generate("Hello world", audio)
print(f"Output shape: {output.shape}, dtype: {output.dtype}, sample rate: {model.sample_rate}")
```
If this crashes, you've found a bug you'll need to monkey-patch. Common ones:
- `RuntimeError: expected scalar type Float but found Double` → needs float32 cast
- `RuntimeError: map_location` → needs `torch.load` patch
- `RuntimeError: Unsupported operator aten::...` → needs MPS skip
### 0.5 Produce a Dependency Audit
Before proceeding to Phase 1, write down:
1. **PyPI vs non-PyPI deps** — which packages need `--find-links`, `git+https://`, or `--no-deps`?
2. **PyInstaller directives needed** — which packages need `--collect-all`, `--copy-metadata`, `--hidden-import`?
3. **Runtime data files** — which packages ship data files (YAML, pretrained weights, phoneme tables, shader libraries) that must be bundled?
4. **Native library paths** — which packages look for data at system paths that won't exist in a frozen binary?
5. **Monkey-patches needed** — `torch.load` map_location, float64→float32 casts, MPS skip, HF token bypass, etc.
6. **Sample rate** — what does the engine output? (24kHz, 44.1kHz, 48kHz)
7. **Model download method** — `from_pretrained()` with library-managed download, or manual `snapshot_download()` + `from_local()`?
This audit becomes your implementation plan for Phases 1, 4, and 5.
## Phase 1: Backend Implementation
### 1.1 Create the Backend File
@@ -148,61 +279,210 @@ In `app/src/lib/hooks/useGenerationForm.ts`:
- Add engine-to-model-name mapping
- Update payload construction for engine-specific fields
**Watch out for model naming inconsistencies.** The HuggingFace repo name, the model size label, and the API model name don't always follow predictable patterns. For example, TADA's 3B model is named `tada-3b-ml` (not `tada-3b`), because it's a multilingual variant. Always check the actual repo names and build the frontend model name mapping from those, not from assumptions like `{engine}-{size}`.
### 3.5 Model Management
In `app/src/components/ServerSettings/ModelManagement.tsx`:
- Add description to `MODEL_DESCRIPTIONS` record
- Add model name to `voiceModels` filter condition
### 3.6 Non-Cloning Engines (Preset Voices)
If your engine uses **pre-built voices** instead of zero-shot cloning from reference audio (e.g. Kokoro), additional integration is needed:
**Backend:**
- In `kokoro_backend.py` (or your engine), define a `VOICES` list of `(voice_id, display_name, gender, language)` tuples
- `create_voice_prompt()` should return `{"voice_type": "preset", "preset_engine": "<engine>", "preset_voice_id": "<id>"}`
- `generate()` should read `voice_prompt.get("preset_voice_id")` to select the voice
- Add a `seed_preset_profiles("<engine>")` call in `backend/routes/models.py` after model download completes
- The `seed_preset_profiles()` function in `backend/services/profiles.py` creates DB profiles with `voice_type="preset"`
**Frontend:**
- The `EngineModelSelector` filters options based on `selectedProfile.voice_type`:
- `"cloned"` profiles → only cloning engines shown (Kokoro hidden)
- `"preset"` profiles → only the preset's engine shown
- Profile cards show the engine name as a badge for preset profiles
- When a preset profile is selected, the engine auto-switches
**Profile schema fields for presets:**
- `voice_type: "preset"` (vs `"cloned"` for traditional profiles)
- `preset_engine: "<engine>"` — which engine owns this voice
- `preset_voice_id: "<id>"` — the engine-specific voice identifier
**For future "designed" voices** (text description instead of audio, e.g. Qwen CustomVoice):
- Use `voice_type: "designed"` with `design_prompt` field
- `create_voice_prompt_for_profile()` already returns the design prompt for this type
## Phase 4: Dependencies
Use the dependency audit from Phase 0 to drive this phase. You should already know what packages are needed, which conflict, and which require special installation.
### 4.1 Python Dependencies
Add to `backend/requirements.txt`. Watch for:
Add to `backend/requirements.txt`. There are three installation patterns, depending on what Phase 0 revealed:
**Pinned dependency conflicts** — If the model package pins old versions, install with `--no-deps`:
**Normal PyPI packages:**
```
some-model-package>=1.0.0
```
**Pinned dependency conflicts (`--no-deps`)** — If the model package pins old versions of torch/numpy/transformers, install with `--no-deps` and list sub-dependencies manually. This is the pattern used for `chatterbox-tts`:
```bash
# In justfile / CI setup:
pip install --no-deps chatterbox-tts
# In requirements.txt — list each actual sub-dependency:
conformer>=0.3.2
diffusers>=0.31.0
omegaconf>=2.3.0
resemble-perth>=0.0.2
s3tokenizer>=0.1.6
```
Then list sub-dependencies manually in `requirements.txt`.
To identify sub-deps: `pip show chatterbox-tts` → `Requires:` field, then cross-reference against existing `requirements.txt` to avoid duplicates.
**Non-PyPI packages:**
```
linacodec @ git+https://github.com/user/repo.git
**Non-PyPI packages** — Some libraries only exist on GitHub or require custom indexes:
```
# Git-only packages (no PyPI release)
linacodec @ git+https://github.com/ysharma3501/LinaCodec.git
Zipvoice @ git+https://github.com/ysharma3501/LuxTTS.git
**Custom package indexes:**
```
# Custom package indexes (C extensions with platform-specific wheels)
--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
piper-phonemize>=1.2.0
```
### 4.2 Identifying Hidden Sub-Dependencies
### 4.2 Dependency Conflict Resolution
1. Install the package normally in a throwaway venv
2. Run `pip show <package>` to get its `Requires:` list
3. Cross-reference against existing requirements.txt
4. Test that the engine loads and generates
Check for conflicts with the existing stack before adding anything:
## Phase 5: PyInstaller Bundling
```bash
# Our current stack pins (approximate):
# Python 3.12+, torch>=2.10, transformers>=4.57, numpy>=1.26
This is where most of the pain lives. Common issues:
# Test compatibility
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
| Issue | Symptom | Fix |
|-------|---------|-----|
| `inspect.getsource()` at import | "could not get source code" | `--collect-all <package>` |
| Data files (yaml, .pth.tar) | FileNotFoundError at runtime | `--collect-all <package>` |
| Native data paths (espeak-ng) | Library looks at `/usr/share/...` | Set env var in frozen builds |
| `importlib.metadata` lookups | "No package metadata found" | `--copy-metadata <package>` |
| Dynamic imports | ModuleNotFoundError | `--hidden-import <module>` |
# If it fails, check what the package pins:
pip show model-package | grep Requires
# Look at setup.py/pyproject.toml for version constraints
```
### Testing Frozen Builds
**Known incompatible patterns in the wild:**
- `torch==2.6.0` — many older packages pin this
- `numpy<1.26` — conflicts with Python 3.12+
- `transformers==4.46.3` — many packages pin old transformers
- `onnxruntime` pinned versions — often conflict with torch
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary.
### 4.3 Update Installation Scripts
Dependencies must be added in multiple places:
| File | What to add |
|------|------------|
| `backend/requirements.txt` | Package and version constraint |
| `justfile` | `--no-deps` install line if needed (in `setup-python` and `setup-python-release` targets) |
| `.github/workflows/release.yml` | Same `--no-deps` line in CI build steps |
| `Dockerfile` | Same install commands for Docker builds |
## Phase 5: PyInstaller Bundling (`build_binary.py`)
This is where most of the pain lives. **The v0.2.3 release was entirely dedicated to fixing bundling issues** — every new engine that shipped in v0.2.1 (LuxTTS, Chatterbox, Chatterbox Turbo) worked in dev but failed in production builds. Don't skip this phase.
### 5.1 Register Your Engine in `build_binary.py`
Every new engine needs entries in `backend/build_binary.py`. This file drives PyInstaller and is the single most common source of "works in dev, breaks in prod" bugs. You need to decide which PyInstaller directives your engine's dependencies require:
| Directive | What It Does | When You Need It |
|-----------|-------------|-----------------|
| `--hidden-import <module>` | Includes a module PyInstaller can't detect via static analysis | Dynamic imports, lazy imports, plugin architectures |
| `--collect-all <package>` | Bundles source `.py` files, data files, AND native libraries | Packages that call `inspect.getsource()` at import time (e.g. `inflect` via `typeguard`'s `@typechecked`), or that ship pretrained model files (e.g. `perth` ships `.pth.tar` + `hparams.yaml`) |
| `--collect-data <package>` | Bundles only data files (not source or native libs) | Packages with YAML configs, vocab files, etc. |
| `--collect-submodules <package>` | Bundles all submodules | Packages with deep module trees that PyInstaller misses |
| `--copy-metadata <package>` | Copies `importlib.metadata` info | Packages that call `importlib.metadata.version()` or `pkg_resources.get_distribution()` at runtime. Already required for: `requests`, `transformers`, `huggingface-hub`, `tokenizers`, `safetensors`, `tqdm` |
**Example: adding hidden imports and collect-all for a new engine:**
```python
# In build_binary.py, inside the args list:
"--hidden-import",
"backend.backends.your_engine_backend",
"--hidden-import",
"your_engine_package",
"--hidden-import",
"your_engine_package.inference",
"--collect-all",
"some_dependency_that_uses_inspect_getsource",
"--copy-metadata",
"some_dependency_that_checks_its_own_version",
```
### 5.2 Lessons from v0.2.3 — Real Failures and Their Fixes
These are actual production failures from shipping new engines. Every one of these passed `python -m uvicorn` in dev:
| Engine | Failure | Root Cause | Fix |
|--------|---------|-----------|-----|
| LuxTTS | `"could not get source code"` on import | `inflect` uses `typeguard`'s `@typechecked` which calls `inspect.getsource()` — needs `.py` source files, not just bytecode | `--collect-all inflect` |
| LuxTTS | `espeak-ng-data` not found | `piper_phonemize` C library looks for data at `/usr/share/espeak-ng-data/` which doesn't exist in the bundle | `--collect-all piper_phonemize` + set `ESPEAK_DATA_PATH` env var at runtime (see 5.3) |
| LuxTTS | `inspect.getsource` error in Vocos codec | `linacodec` and `zipvoice` use source introspection | `--collect-all linacodec` + `--collect-all zipvoice` |
| Chatterbox | `FileNotFoundError` for watermark model | `perth` ships pretrained model files (`hparams.yaml`, `.pth.tar`) that PyInstaller doesn't bundle by default | `--collect-all perth` |
| All engines | `importlib.metadata` failures | Frozen binary doesn't include package metadata for `huggingface-hub`, `transformers`, etc. | `--copy-metadata` for each affected package |
| All engines | Download progress bars stuck at 0% | `huggingface_hub` silently disables tqdm progress bars based on logger level in frozen builds — our progress tracker never receives byte updates | Force-enable tqdm's internal counter in `HFProgressTracker` |
| TADA | `inspect.getsource` error in DAC's `Snake1d` | `@torch.jit.script` calls `inspect.getsource()` which fails without `.py` source files | Wrote a lightweight shim (`dac_shim.py`) reimplementing `Snake1d` without `@torch.jit.script`, registered fake `dac.*` modules in `sys.modules` |
| All engines | `NameError: name 'obj' is not defined` on macOS | Python 3.12.0 has a [CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects | Upgrade to Python 3.12.13+ |
| All engines | `resource_tracker` subprocess crash | `multiprocessing` in frozen binaries needs `freeze_support()` called before anything else | Added to `server.py` entry point |
### 5.3 Runtime Frozen-Build Handling (`server.py`)
Some fixes can't live in `build_binary.py` — they need runtime detection. The entry point `backend/server.py` handles these before any heavy imports:
```python
# 1. freeze_support() — MUST be called before any multiprocessing use
import multiprocessing
multiprocessing.freeze_support()
# 2. Native data paths — redirect C libraries to bundled data
if getattr(sys, 'frozen', False):
_meipass = getattr(sys, '_MEIPASS', os.path.dirname(sys.executable))
_espeak_data = os.path.join(_meipass, 'piper_phonemize', 'espeak-ng-data')
if os.path.isdir(_espeak_data):
os.environ.setdefault('ESPEAK_DATA_PATH', _espeak_data)
# 3. stdout/stderr safety — PyInstaller --noconsole on Windows sets these to None
if not _is_writable(sys.stdout):
sys.stdout = open(os.devnull, 'w')
```
If your engine's dependencies include native libraries that look for data at system paths (like espeak-ng does), you'll need to add a similar `os.environ.setdefault()` block here.
### 5.4 CUDA vs CPU Build Branching
`build_binary.py` produces two different binaries:
- **`voicebox-server`** (CPU) — excludes all `nvidia.*` packages to avoid bundling ~3 GB of CUDA DLLs
- **`voicebox-server-cuda`** — includes `torch.cuda` and `torch.backends.cudnn`
On Windows, if the build environment has CUDA torch installed but you're building the CPU binary, the script temporarily swaps to CPU-only torch and restores CUDA torch afterward. This prevents PyInstaller from accidentally bundling CUDA libraries into the CPU build.
New engine imports go in the **common section** (not the CUDA or MLX conditional blocks) unless your engine has platform-specific dependencies.
### 5.5 MLX Conditional Inclusion
Apple Silicon builds conditionally include MLX hidden imports and `--collect-all mlx` / `--collect-all mlx_audio`. If your engine has an MLX-specific backend variant, add its imports inside the `if is_apple_silicon() and not cuda:` block.
### 5.6 Testing Frozen Builds
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary. The v0.2.3 release required **three patch releases** (v0.2.1 → v0.2.2 → v0.2.3) to get all engines working in production.
1. Build: `just build`
2. Run and try download + load + generate
3. Check stderr for the actual error
4. Fix, rebuild, repeat
2. Launch the binary directly (not via `python -m`)
3. Test the **full chain**: download → load → generate → progress tracking
4. Check stderr for the actual error (logs go to stderr for Tauri sidecar capture)
5. Fix, rebuild, repeat
**Common gotcha:** testing only generation with a pre-cached model from your dev install. Always test with a clean model cache to verify downloads work too.
## Phase 6: Common Upstream Workarounds
@@ -240,6 +520,90 @@ def _get_device(self):
return "cpu" # Skip MPS
```
### Gated HuggingFace repos as hardcoded config sources
Some models hardcode a gated HuggingFace repo as their tokenizer or config source (e.g., TADA hardcodes `"meta-llama/Llama-3.2-1B"` in both its `AlignerConfig` and `TadaConfig`). This silently fails without HF authentication.
**Fix:** Download from an ungated mirror and patch the config objects directly:
```python
# Download tokenizer from ungated mirror
UNGATED_TOKENIZER = "unsloth/Llama-3.2-1B"
tokenizer_path = snapshot_download(UNGATED_TOKENIZER, token=None)
# Patch the model config to use the local path instead of the gated repo
config = ModelConfig.from_pretrained(model_path)
config.tokenizer_name = tokenizer_path
model = ModelClass.from_pretrained(model_path, config=config)
```
**Do NOT monkey-patch `AutoTokenizer.from_pretrained`** — it's a classmethod, and replacing it corrupts the descriptor, which breaks other engines that use different tokenizers (e.g., Qwen uses a Qwen tokenizer via `AutoTokenizer`). Always patch at the config level, not the class method level.
### `torchaudio.load()` requires `torchcodec` in 2.10+
As of `torchaudio>=2.10`, `torchaudio.load()` requires the `torchcodec` package for audio I/O. If your engine or backend code uses `torchaudio.load()`, replace it with `soundfile`:
```python
# Before (breaks without torchcodec):
import torchaudio
waveform, sr = torchaudio.load("audio.wav")
# After:
import soundfile as sf
import torch
data, sr = sf.read("audio.wav", dtype="float32")
waveform = torch.from_numpy(data).unsqueeze(0)
```
Note: `torchaudio.functional.resample()` and other pure-PyTorch math functions work fine without `torchcodec` — only the I/O functions are affected.
### `@torch.jit.script` breaks in frozen builds
`torch.jit.script` calls `inspect.getsource()` to parse the decorated function's source code. In a PyInstaller binary, `.py` source files aren't available, so this crashes at import time.
**Fix:** Remove or avoid `@torch.jit.script` decorators. If the decorated function comes from an upstream dependency, write a shim that reimplements the function without the decorator (see "Toxic dependency chains" below).
### Toxic dependency chains — the shim pattern
Sometimes a model library depends on a package with a massive, hostile transitive dependency tree, but only uses a tiny piece of it. When the dependency chain is unbuildable or would pull in dozens of unwanted packages, the right move is to write a lightweight shim.
**Example:** TADA depends on `descript-audio-codec` (DAC), which pulls in `descript-audiotools` -> `onnx`, `tensorboard`, `protobuf`, `matplotlib`, `pystoi`, etc. The `onnx` package fails to build from source on macOS. But TADA only uses `Snake1d` from DAC — a 7-line PyTorch module.
**Solution:** Create a shim at `backend/utils/dac_shim.py` that registers fake modules in `sys.modules`:
```python
import sys
import types
import torch
from torch import nn
def snake(x, alpha):
"""Snake activation — reimplemented without @torch.jit.script."""
return x + (1.0 / (alpha + 1e-9)) * torch.sin(alpha * x).pow(2)
class Snake1d(nn.Module):
def __init__(self, channels):
super().__init__()
self.alpha = nn.Parameter(torch.ones(1, channels, 1))
def forward(self, x):
return snake(x, self.alpha)
# Register fake dac.* modules so "from dac.nn.layers import Snake1d" works
_nn = types.ModuleType("dac.nn")
_layers = types.ModuleType("dac.nn.layers")
_layers.Snake1d = Snake1d
_nn.layers = _layers
for name, mod in [("dac", types.ModuleType("dac")),
("dac.nn", _nn), ("dac.nn.layers", _layers)]:
sys.modules[name] = mod
```
**Key rules for shims:**
- Import the shim **before** importing the model library (so it finds the fake modules first)
- Do NOT use `@torch.jit.script` in the shim (see above)
- Only reimplement what the model actually uses — check the import chain carefully
## Upcoming Engines
Based on the current model landscape, these are candidates for future integration:
@@ -250,8 +614,83 @@ Based on the current model landscape, these are candidates for future integratio
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Needs vetting |
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
The multi-engine architecture is now in place, making new model integration straightforward (~1 day for a well-documented model with a PyPI package).
## Implementation Checklist
Use this as a gate between phases. Do not proceed to the next phase until every item in the current phase is checked.
### Phase 0: Dependency Research
- [ ] Cloned model library source into a temp directory
- [ ] Read `setup.py` / `pyproject.toml` — noted pinned dependency versions
- [ ] Traced all imports from the model class through to leaf dependencies
- [ ] Searched for `inspect.getsource`, `@typechecked`, `typeguard` in the full dependency tree
- [ ] Searched for `importlib.metadata`, `pkg_resources.get_distribution` in the dependency tree
- [ ] Searched for `Path(__file__).parent`, `os.path.dirname(__file__)`, hardcoded system paths
- [ ] Searched for `torch.load` calls missing `map_location`
- [ ] Searched for `torch.from_numpy` without `.float()` cast
- [ ] Searched for `token=True` or `token=os.getenv("HF_TOKEN")` in HuggingFace calls
- [ ] Searched for `@torch.jit.script` / `torch.jit.script` (crashes in frozen builds)
- [ ] Searched for `torchaudio.load` / `torchaudio.save` (requires `torchcodec` in 2.10+)
- [ ] Searched for hardcoded gated HuggingFace repo names (e.g., `meta-llama/*`)
- [ ] Evaluated whether any dependency is used minimally enough to shim instead of install
- [ ] Tested model loading and generation on CPU in a throwaway venv
- [ ] Tested with a clean HuggingFace cache (no pre-downloaded models)
- [ ] Produced a written dependency audit documenting all findings
### Phase 1: Backend Implementation
- [ ] Created `backend/backends/<engine>_backend.py` implementing `TTSBackend` protocol
- [ ] Chose voice prompt pattern (pre-computed tensors vs deferred file paths)
- [ ] Implemented all monkey-patches identified in Phase 0
- [ ] Used `get_torch_device()` from `backends/base.py` for device selection
- [ ] Used `model_load_progress()` from `backends/base.py` for download/load tracking
- [ ] Tested: model downloads correctly
- [ ] Tested: model loads on CPU
- [ ] Tested: generation produces valid audio
- [ ] Tested: voice cloning from reference audio works
- [ ] Registered `ModelConfig` in `backends/__init__.py`
- [ ] Added to `TTS_ENGINES` dict
- [ ] Added factory branch in `get_tts_backend_for_engine()`
- [ ] Updated engine regex in `backend/models.py`
### Phase 2–3: Route, Service, and Frontend
- [ ] Confirmed zero changes needed in routes/services (or documented why custom behavior is needed)
- [ ] Added engine to TypeScript union type in `app/src/lib/api/types.ts`
- [ ] Added language map entry in `app/src/lib/constants/languages.ts`
- [ ] Added to `ENGINE_OPTIONS` and `ENGINE_DESCRIPTIONS` in `EngineModelSelector.tsx`
- [ ] Added to Zod schema and model-name mapping in `useGenerationForm.ts`
- [ ] Added description in `ModelManagement.tsx`
### Phase 4: Dependencies
- [ ] Added packages to `backend/requirements.txt`
- [ ] If `--no-deps` needed: listed sub-dependencies explicitly
- [ ] If git-only packages: added `@ git+https://...` entries
- [ ] If custom index needed: added `--find-links` line
- [ ] Updated `justfile` setup targets
- [ ] Updated `.github/workflows/release.yml` build steps
- [ ] Updated `Dockerfile` if applicable
- [ ] Verified `pip install` succeeds in a clean venv with existing requirements
### Phase 5: PyInstaller Bundling
- [ ] Added `--hidden-import` entries in `build_binary.py` for:
- [ ] `backend.backends.<engine>_backend`
- [ ] The model package and its key submodules
- [ ] Added `--collect-all` for any packages that:
- [ ] Use `inspect.getsource()` / `@typechecked`
- [ ] Ship pretrained model data files (`.pth.tar`, `.yaml`, etc.)
- [ ] Ship native data files (phoneme tables, shader libraries, etc.)
- [ ] Added `--copy-metadata` for any packages that use `importlib.metadata`
- [ ] If engine has native data paths: added `os.environ.setdefault()` in `server.py`
- [ ] Built frozen binary with `just build`
- [ ] Tested in frozen binary with **clean model cache** (not pre-cached from dev):
- [ ] Model download works with real-time progress
- [ ] Model loading works
- [ ] Generation produces valid audio
- [ ] No errors in stderr logs
### Phase 6: Final Verification
- [ ] Engine works in dev mode (`just dev`)
- [ ] Engine works in frozen binary (`just build` → run binary directly)
- [ ] Tested on target platform (macOS for MLX, Windows/Linux for CUDA)
- [ ] No regressions in existing engines
+2 -2
View File
@@ -3,12 +3,12 @@ title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
+5 -4
View File
@@ -5,10 +5,10 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -20,7 +20,7 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## TTS Engines
Four engines with different strengths, switchable per-generation:
Five engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
@@ -28,6 +28,7 @@ Four engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
## GPU Support
@@ -56,7 +57,7 @@ Four engines with different strengths, switchable per-generation:
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
+115 -130
View File
@@ -1,6 +1,6 @@
# Voicebox Project Status & Roadmap
> Last updated: 2026-03-13 | Current version: **v0.1.13** | 13.1k stars | ~176 open issues | 25 open PRs
> Last updated: 2026-03-18 | Current version: **v0.3.0** | 13.4k stars | ~136 open issues | 9 open PRs
---
@@ -36,6 +36,10 @@
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │ │ │ TADA │ │ │ │
│ │ │ │(1B / 3B) │ │ │ │
│ │ │ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ ┌───────────┐ ┌─────────┐ │ │
│ │ │ STTBackend│ │ Profiles│ │ │
@@ -59,6 +63,7 @@
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
@@ -78,7 +83,7 @@
```
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo)
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
@@ -95,7 +100,7 @@ POST /generate
## Current State
### What's Shipped (v0.1.13 + recent merges)
### What's Shipped (v0.3.0)
**Core TTS:**
- Qwen3-TTS voice cloning (1.7B and 0.6B models)
@@ -103,29 +108,42 @@ POST /generate
- Multi-engine TTS architecture with thread-safe backend registry (PR #254)
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296)
- Chunked TTS generation for long text — engine-agnostic, removes ~500 char limit (PR #266)
- Async generation queue (PR #269)
- Post-processing audio effects system (PR #271)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
**Infrastructure:**
- CUDA backend swap via binary download and restart (PR #252)
- GPU acceleration settings UI
- CUDA backend swap via binary download and restart (PR #252), upgraded to cu128 (PR #316)
- CUDA backend split into independently versioned server + libs archives (PR #298)
- Docker + web deployment (PR #161)
- Backend refactor: modular architecture, style guide, tooling (PR #285)
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
- Windows support: CUDA detection, cross-platform justfile, clean server shutdown (PR #272)
- Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large variants)
- Model management UI with inline download progress bars (HFProgressTracker)
- Model management UI with inline download progress bars + folder migration (PR #268)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching
- Streaming generation endpoint (MLX only)
- Duplicate profile name validation (PR #175)
- Linux NVIDIA GBM buffer + WebKitGTK microphone fix (PR #210)
- Audio player freeze fix + UX improvements (PR #293)
- CORS restriction to known local origins (PR #88)
### Abandoned Integrations
| Model | PR | Reason |
|-------|----|--------|
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. |
### What's In-Flight
| Feature | Branch/PR | Status |
|---------|-----------|--------|
| Chatterbox Turbo + per-engine language lists | `feat/chatterbox-turbo` / PR #258 | Open, ready for review |
| Kokoro 82M TTS engine | WIP | In development — 82M CPU-realtime engine, 8 languages |
### TTS Engine Comparison
@@ -136,6 +154,9 @@ POST /generate
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
| Kokoro 82M | `kokoro` | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
### Multi-Engine Architecture (Shipped)
@@ -143,7 +164,7 @@ The singleton TTS backend blocker described in the previous version of this doc
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'`
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'`
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
@@ -165,69 +186,41 @@ The singleton TTS backend blocker described in the previous version of this doc
| PR | Title | Merged |
|----|-------|--------|
| **#257** | feat: Chatterbox TTS engine with multilingual voice cloning | 2026-03-13 |
| **#254** | feat: LuxTTS integration — multi-engine TTS support | 2026-03-13 |
| **#252** | feat: CUDA backend swap via binary download and restart | 2026-03-13 |
| **#238** | Download cancel/clear UI, fixed model downloading | 2026-03-13 |
| **#250** | docs: align local API port examples | 2026-03-13 |
| **#210** | fix: Linux NVIDIA GBM buffer crash | 2026-03-13 |
| **#175** | Fix #134: duplicate profile name validation | 2026-03-13 |
| **#316** | Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI | 2026-03-18 |
| **#305** | fix: bundle qwen_tts source files in PyInstaller build | 2026-03-17 |
| **#298** | feat: split CUDA backend into independently versioned server + libs archives | 2026-03-17 |
| **#296** | Add HumeAI TADA TTS engine (1B English + 3B Multilingual) | 2026-03-17 |
| **#295** | fix: batch of bug fixes from issue tracker | 2026-03-17 |
| **#293** | Fix audio player freezing and improve UX | 2026-03-17 |
| **#294** | Settings overhaul: routed sub-tabs, server logs, changelog, about page | 2026-03-16 |
| **#288** | Better docs | 2026-03-16 |
| **#285** | Backend refactor: modular architecture, style guide, tooling | 2026-03-16 |
| **#274** | Landing page v0.2.0 redesign | 2026-03-15 |
| **#272** | Windows support: CUDA detection, cross-platform justfile, clean server shutdown | 2026-03-15 |
| **#271** | Add post-processing audio effects system | 2026-03-14 |
| **#269** | feat: async generation queue | 2026-03-13 |
| **#268** | feat: model management improvements and folder migration | 2026-03-13 |
| **#266** | feat: chunked TTS generation for long text (engine-agnostic) | 2026-03-13 |
| **#265** | feat: paralinguistic tag autocomplete for Chatterbox Turbo | 2026-03-13 |
| **#264** | fix: Chatterbox float64 dtype mismatch + model unload button | 2026-03-13 |
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | 2026-03-13 |
| **#230** | docs: fix README grammar | 2026-03-13 |
| **#161** | feat: Docker + web deployment | 2026-03-13 |
| **#88** | security: restrict CORS to known local origins | 2026-03-13 |
### In-Flight (Our Work)
### Currently Open (9 PRs)
| PR | Title | Status | Notes |
|----|-------|--------|-------|
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | Open | Ready for review. Adds Turbo engine + dynamic language dropdown. |
### Merge-Ready / Near-Ready (Bug Fixes & Small Features)
| PR | Title | Risk | Notes |
|----|-------|------|-------|
| **#230** | docs: fix README grammar | None | Docs-only |
| **#243** | a11y: screen reader and keyboard improvements | Low | Accessibility, no backend changes |
| **#178** | Fix #168 #140: generation error handling | Low | Error handling improvements |
| **#152** | Fix: prevent crashes when HuggingFace unreachable | Medium | Monkey-patches HF hub; solves real offline bug (#150, #151) |
| **#218** | fix: unify qwen tts cache dir on Windows | Low | Windows-specific path fix |
| **#214** | fix: panic on launch from tokio::spawn | Low | Rust-side Tauri fix |
| **#88** | security: restrict CORS to known local origins | Low | Security hardening |
| **#133** | feat: network access toggle | Low | Wires up existing plumbing |
### Significant Feature PRs
| PR | Title | Complexity | Notes |
|----|-------|-----------|-------|
| **#253** | Enhance speech tokenizer with 48kHz version | Medium | Qwen tokenizer upgrade |
| **#97** | fix: pass language parameter to TTS models | Medium | May be partially obsoleted by multi-engine work — needs review |
| **#99** | feat: chunked TTS with quality selector | Medium | Solves 500-char limit. Addresses #191, #203, #69, #111. |
| **#154** | feat: Audiobook tab | Medium | Full audiobook workflow. Depends on #99 concepts. |
| **#91** | fix: CoreAudio device enumeration | Medium | macOS audio device handling |
### Architectural PRs (Need Careful Review)
| PR | Title | Complexity | Notes |
|----|-------|-----------|-------|
| **#225** | feat: custom HuggingFace model support | High | Arbitrary HF repo loading. May need rework given multi-engine arch is now shipped. |
| **#194** | feat: Hebrew + Chatterbox TTS | High | **Superseded** by PR #257 which shipped Chatterbox multilingual (23 langs incl. Hebrew). May be closeable. |
| **#195** | feat: per-profile LoRA fine-tuning | Very High | Training pipeline, adapter management, 15 new endpoints. Depends on #194 (now superseded). |
| **#161** | feat: Docker + web deployment | High | 3-stage Dockerfile, SPA serving. Independent of TTS engine work. |
| **#124** / **#123** | Docker (simpler attempts) | Low-Medium | Overlap with #161 |
| **#227** | fix: harden input validation & file safety | Medium | Coupled to #225 (custom models) |
### PRs That Need Author Action / Are Stale
| PR | Title | Notes |
|----|-------|-------|
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Build system, needs review |
| **#215** | Update prerequisites with Tauri deps | Branch is `main` — will have conflicts |
| **#89** | Linux Support | Branch is `main` — will have conflicts. Broad scope. |
| **#83** | Update download links for v0.1.12 | Outdated (we're on v0.1.13) |
### PRs Likely Superseded
| PR | Superseded By | Notes |
|----|--------------|-------|
| **#194** (Hebrew + Chatterbox) | PR #257 (merged) | #257 ships Chatterbox multilingual with 23 languages including Hebrew. #194 took a different approach (route by language). Can likely be closed. |
| **#33** (External provider binaries) | PR #252 (merged) | #252 shipped CUDA backend swap. #33's broader provider architecture may still have value but needs reassessment. |
| **#311** | feat: add CosyVoice2/3 TTS engine | **Will close** | Model quality too poor. See Abandoned Integrations. |
| **#253** | Enhance speech tokenizer with 48kHz version | Community PR | Qwen tokenizer upgrade. Worth reviewing. |
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Superseded | Our PR #305 shipped this. Can close. |
| **#227** | fix: harden input validation & file safety | Community PR | Coupled to #225 (custom models). |
| **#225** | feat: custom HuggingFace model support | Community PR | Needs rework for multi-engine arch. |
| **#218** | fix: unify qwen tts cache dir on Windows | Community PR | Windows-specific path fix. Still relevant. |
| **#195** | feat: per-profile LoRA fine-tuning | Draft | Complex. 15 new endpoints. |
| **#154** | feat: Audiobook tab | Community PR | Chunked generation now shipped (#266). |
| **#91** | fix: CoreAudio device enumeration | Draft | macOS audio device handling. |
---
@@ -272,7 +265,7 @@ Strong demand for: Hindi (#245), Indonesian (#247), Dutch (#236), Hebrew (#199),
| #132 | LavaSR (transcription) |
| #76 | (General model expansion) |
Community also requests: XTTS-v2, Fish Speech, CosyVoice, Kokoro. The multi-engine architecture is now in place, making new model integration significantly easier.
Community also requests: XTTS-v2, Fish Speech, Kokoro. CosyVoice was tried and abandoned. The multi-engine architecture is in place, making new model integration straightforward.
### Long-Form / Chunking (5 issues)
@@ -280,7 +273,7 @@ Users hitting the ~500 character practical limit.
**Key issues:** #234 (queue system), #203 (500 char limit), #191 (auto-split), #111, #69
**Fix path:** PR #99 (chunked TTS + quality selector) directly addresses this. PR #154 (Audiobook tab) builds on it.
**Fix path:** **Mostly resolved.** PR #266 (engine-agnostic chunked TTS) and PR #269 (async generation queue) are both merged. PR #154 (Audiobook tab) is still open.
### Feature Requests (23 issues)
@@ -318,7 +311,7 @@ Notable requests:
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **PR exists** (#161) | Waiting on review |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
@@ -326,31 +319,31 @@ Notable requests:
## New Model Integration — Landscape
### Models Worth Supporting (2026 SOTA — updated March 13)
### Models Worth Supporting (2026 SOTA — updated March 18)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct Support | Integration Ease | Status |
|-------|---------|-------|-------------|-----------|------|-----------------|-----------------|--------|
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None (Base); Yes (CustomVoice variant, predefined speakers only) | **Shipped** | v0.1.13 |
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | **Shipped** | PR #254 |
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` float | **Shipped** | PR #257 |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **PR #258** | In review |
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
| **Pocket TTS** | Zero-shot + streaming | >1× RT on CPU | — | English | ~100M params, CPU-first | None | Needs vetting | MIT, Kyutai Labs, no GPU required |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **Shipped** | PR #258 |
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody | **Shipped** | PR #296 |
| **Kokoro-82M** | Pre-built voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | **In progress** | Apache 2.0, pip install, ~350MB |
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Yes — `inference_instruct2()` | **Abandoned** | PR #311 — poor output quality |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Needs license clarification |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Mature pip package |
| **Pocket TTS** | Zero-shot + streaming | >1x RT on CPU | — | English | ~100M params, CPU-first | None | Ready | MIT, Kyutai Labs |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0 |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0 |
#### Notes on New Candidates (March 2026)
#### Notes on Candidates (March 2026)
- **CosyVoice2-0.5B** — Best candidate for instruct support. `inference_instruct2()` accepts a text instruct parameter for emotions, speed, volume, dialects — and it works alongside voice cloning. This is the closest match to what users expect from our instruct UI. [HF: FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B)
- **HumeAI TADA** — Text-Audio Dual Alignment arch. Near-zero hallucinations/drift, free synced transcript. 700+ seconds coherent audio. Best candidate for Stories long-form reliability. Prosody/emotion is automatic from text context, not user-controllable. [HF: HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **MOSS-TTS** — Modular suite: flagship cloning, MOSS-TTSD (multi-speaker dialogue), MOSS-VoiceGenerator (create voices from text descriptions). VoiceGenerator unifies timbre design and style control via text prompts, usable as a layer for downstream TTS including cloning. [HF: OpenMOSS-Team/MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | [GitHub: OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS)
- **Fish Speech** — Word-level fine-grained control using plain language descriptions inline in the script. Works with cloning. Note: Fish Audio S2 has a restrictive research license (commercial use requires approval), but the open-source Fish Speech model may differ. Needs license clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
- **VoxCPM 1.5** — Tokenizer-free continuous diffusion + autoregressive. No discrete token artifacts. Prosody/emotion is context-aware but automatic, not explicitly controllable via text prompt. Real-time streaming, LoRA fine-tuning. Trained on 1.8M+ hours. [GitHub: OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs (Moshi team). Runs >1× realtime without GPU. No style control. Broadens hardware support significantly. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **CosyVoice2-0.5B** — **Tried and abandoned** (PR #311). Despite having the best instruct API, output quality was poor. No PyPI package, needed 5+ shims, heavy deps. Not worth it.
- **HumeAI TADA** — **Shipped** (PR #296). 700+ seconds coherent audio. [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **Kokoro-82M** — **In progress.** 82M params, CPU realtime, Apache 2.0, clean `pip install kokoro`. Uses pre-built voice styles (not zero-shot cloning from arbitrary audio). [GitHub: hexgrad/kokoro](https://github.com/hexgrad/kokoro)
- **Fish Speech** — Word-level fine-grained control. License needs clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
- **XTTS-v2** — Coqui's multilingual cloning. 17+ languages, pip-installable. [GitHub: coqui-ai/TTS](https://github.com/coqui-ai/TTS)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control)
### Adding a New Engine (Now Straightforward)
@@ -394,49 +387,44 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
## Recommended Priorities
### Tier 1 — Ship Now (Low Risk)
### Tier 1 — Ship Now
| Priority | PR/Item | Impact | Effort |
|----------|---------|--------|--------|
| 1 | **#258** — Chatterbox Turbo + per-engine languages | Paralinguistic tags, proper language filtering | Review only |
| 2 | **#152** — Offline mode crash fix | Fixes #150, #151 | Low |
| 3 | **#99** — Chunked TTS + quality selector | Removes 500-char limit, addresses 5 issues | Medium |
| 4 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 5 | **#178** — Generation error handling | Error UX | Low |
| 6 | **#230** — Docs fixes | Zero risk | None |
| 7 | **#133** — Network access toggle | Wires up existing code | Low |
| 8 | **#88** — CORS restriction | Security improvement | Low |
| 9 | **#214** — Tauri window close panic fix | Stability | Low |
| 10 | Triage GPU issues | Many may be resolved by CUDA swap (#252) | Low |
| 11 | Close superseded PRs | #194 (superseded by #257), #83 (outdated) | None |
| 1 | **Kokoro 82M** — finish integration | New engine, CPU-friendly, 8 langs | Low (nearly done) |
| 2 | Close PR #311 (CosyVoice) and #237 (superseded by #305) | Housekeeping | None |
| 3 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 4 | **#253** — 48kHz speech tokenizer | Quality improvement for Qwen | Medium |
### Tier 2 — Next Release (v0.2.0)
### Tier 2 — Feature Work
| Priority | Item | Impact | Effort |
|----------|------|--------|--------|
| 1 | **#253** — 48kHz speech tokenizer | Quality improvement | Medium |
| 2 | **#161** — Docker deployment | Server/headless users | Medium |
| 3 | **#154** — Audiobook tab | Long-form users | Medium |
| 4 | ~~**Model config registry**~~ | ~~Reduce dispatch duplication in main.py~~ | **Done** |
| 5 | **#225** — Custom HuggingFace models | User-supplied models | High (needs rework for multi-engine) |
| 1 | **#154** — Audiobook tab | Long-form users. Chunking + queue now shipped. | Medium |
| 2 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
| 3 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable | Low |
| 4 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
| 5 | Streaming for non-MLX engines | Currently MLX-only | Medium |
### Tier 3 — Future (v0.3.0+)
### Tier 3 — Future Engines
| Priority | Item | Notes |
|----------|------|-------|
| 1 | **HumeAI TADA** | Long-form reliability for Stories, synced transcripts. Addresses #234, #203, #191, #111, #69. Needs API vetting. |
| 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model, broadens hardware support. Kyutai ships clean code. Needs API vetting. |
| 3 | **MOSS-TTS** | Text-to-voice design (no ref audio) is unique. Multi-speaker dialogue for Stories. Needs thorough API vetting. |
| 4 | **Kokoro-82M** | 82M params, CPU realtime, Apache 2.0. Easy win. |
| 5 | ~~**Model config registry refactor**~~ | **Done** — consolidated in `backend/backends/__init__.py` + `EngineModelSelector.tsx` |
| 6 | XTTS-v2 / Fish Speech / CosyVoice | Multi-engine arch is ready; just needs backend implementation |
| 7 | **VoxCPM 1.5** | Tokenizer-free streaming, interesting but uncertain integration surface |
| 8 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable |
| 9 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine |
| 10 | External/remote providers | Depends on use case demand |
| 11 | GGUF support (#226) | Depends on model ecosystem maturity |
| 12 | Queue system (#234) | Batch generation |
| 13 | Streaming for non-MLX engines | Currently MLX-only |
| 1 | **Fish Speech** | 50+ langs, word-level instruct. License TBD. |
| 2 | **XTTS-v2** | 17+ langs, mature pip package. Best multilingual cloning. |
| 3 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. |
| 4 | **MOSS-TTS** | Text-to-voice design. Multi-speaker dialogue for Stories. |
| 5 | **VoxCPM 1.5** | Tokenizer-free streaming. Uncertain integration surface. |
### ~~Previously Prioritized — Now Done~~
- ~~#258 — Chatterbox Turbo~~ **Merged**
- ~~#99 — Chunked TTS~~ **Superseded by #266, merged**
- ~~#88 — CORS restriction~~ **Merged**
- ~~#161 — Docker deployment~~ **Merged**
- ~~#234 — Queue system~~ **Addressed by #269, merged**
- ~~HumeAI TADA~~ **Shipped** (PR #296)
- ~~Kokoro-82M~~ **In progress**
---
@@ -444,13 +432,10 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
| Branch | PR | Status | Notes |
|--------|-----|--------|-------|
| `feat/chatterbox-turbo` | #258 | Open | Chatterbox Turbo + per-engine languages |
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
| `external-provider-binaries` | #33 | Superseded by #252 | Original CUDA provider approach |
| `feat/dual-server-binaries` | — | No PR | Related to provider split |
| `fix-multi-sample` | — | No PR | Voice profile multi-sample fix |
| `fix-dl-notification-...` | — | No PR | Model download UX |
---
@@ -475,7 +460,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
| `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
| `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model |
+173
View File
@@ -0,0 +1,173 @@
# CUDA Libs as a Bolt-On Addon
## Problem
Every time we bump `__version__` (even for a UI tweak or bugfix), the exact-match version check in both `main.rs:222` and `cuda.py:237` invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime.
## Why This Is Hard With `--onefile`
The core tension is PyInstaller `--onefile` mode (`build_binary.py:39`). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA `.dll`/`.so` files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server.
## Options
### Option A: Switch to `--onedir` for the CUDA Build (Recommended)
Instead of `--onefile`, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives:
1. **`voicebox-server-cuda` executable + non-NVIDIA deps** (~200-400MB) -- versioned with the app, redownloaded on every app update.
2. **`cuda-libs-cu126.tar.gz`** (~2GB) -- the `nvidia.*` packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., `cuda-libs-cu126-v1`). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes.
#### How it would work at runtime
- Tauri downloads the server binary archive and extracts it to `{data_dir}/backends/cuda/`
- On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory
- The CUDA server exe finds the `.dll`/`.so` files next to it (standard PyInstaller onedir behavior)
- Version check becomes two checks: server version + cuda-libs version
#### Independent versioning
Add a `cuda-libs.json` manifest:
```json
{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"}
```
The server checks this on startup. The Tauri side checks it before launching. Only bump `cu126-v1` -> `cu126-v2` when we actually change the CUDA toolkit or torch major version.
#### Build pipeline changes
The CI `build-cuda-windows` job would build with `--onedir`, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset.
#### Download experience
- First-time CUDA setup: ~2.4GB total (same as today)
- Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached
- CUDA toolkit bump: ~2GB for just the libs
#### Pros
- PyInstaller `--onedir` natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory
- The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (`nvidia/cublas/lib/`, etc.)
- CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version
- Server updates become ~200-400MB instead of ~2.4GB
- No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree
#### Cons
- Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up
- Need to modify download/assembly logic in `cuda.py` to handle two separate archives
- The Tauri side (`main.rs`) needs to point at an exe inside a directory rather than a standalone binary
- Users who manually manage the file may find the folder structure confusing
#### TTS engine compatibility
No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way.
---
### Option B: Keep `--onefile` but Externalize CUDA Libs via Library Path
Keep the server as a single `--onefile` binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to `{data_dir}/backends/cuda-libs/`. Before launching, set the library search path to include that directory.
**Important caveat:** The CPU torch wheel (`whl/cpu`) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally.
#### How it would work
- Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded
- Ship `cuda-libs-cu126-v1.tar.gz` separately (~2GB of `.dll`/`.so` files)
- When launching, Tauri sets `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux) to include the cuda-libs directory
#### Pros
- Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously
- True bolt-on CUDA libs with fully independent versioning
- Server updates are always small (~150MB for the onefile binary)
#### Cons
- **Fragile on Windows.** PyInstaller `--onefile` extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario.
- `os.add_dll_directory()` only affects `LoadLibraryEx` with `LOAD_LIBRARY_SEARCH_USER_DIRS` flag -- not all DLL loads go through this path
- PyInstaller's onefile bootloader may configure DLL search paths before Python code runs
- Could work on Linux but is fragile on Windows
---
### Option C: Hybrid -- `--onefile` Server + Dynamic CUDA Lib Loading at Runtime
Build the server as `--onefile` with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using `ctypes.CDLL` or `os.add_dll_directory()`.
In `server.py`, before any torch imports:
```python
cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS")
if cuda_libs_dir and os.path.isdir(cuda_libs_dir):
if sys.platform == "win32":
os.add_dll_directory(cuda_libs_dir)
os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "")
else:
os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "")
```
#### Pros
- Single server binary, true bolt-on CUDA libs
- Clean separation of concerns
- Independent versioning
#### Cons
- Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer
- On Windows, `os.add_dll_directory()` may not cover all DLL load paths
- PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs
- Most complex to get right and maintain
## Recommendation
**Option A (`--onedir` with split archives)** is the most reliable path:
1. **It actually works.** `--onedir` puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install.
2. **Natural separation.** PyInstaller's `--onedir` output already separates the NVIDIA `.dll`/`.so` files into `nvidia/` subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building.
3. **Independent versioning is straightforward.** A `cuda-libs.json` manifest controls when redownloads are needed.
4. **Build pipeline simplification.** Build CUDA libs archive less frequently, store as a pinned release asset.
The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in `cuda.py` with manifests and split parts. Extending that to handle two archives is incremental work.
## Tauri Compatibility (Validated)
Tauri handles PyInstaller `--onedir` with no issues. The key insight is that we're **not** using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing `cuda.py` + `main.rs` flow). For runtime-launched processes, Tauri's `tauri::shell::Command` supports arbitrary directories natively.
### The critical change in `main.rs`
The only Tauri-side change needed is adding `.current_dir()` when spawning the CUDA backend:
```rust
let cuda_dir = data_dir.join("backends/cuda");
let exe_path = cuda_dir.join("voicebox-server-cuda.exe");
let mut cmd = app.shell().command(exe_path.to_str().unwrap());
cmd = cmd.current_dir(&cuda_dir); // PyInstaller finds all DLLs relative to exe
cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);
```
`.current_dir()` tells the PyInstaller bootloader that everything (DLLs, `nvidia/cublas/lib/`, `_internal/`, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal `pip install` or dev environment -- no `LD_LIBRARY_PATH` hacks, no `os.add_dll_directory` gymnastics.
### Community evidence
- Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206).
- The shell plugin explicitly supports `cwd` in both Rust and JS APIs.
- No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage.
### Known gotcha: process termination on Windows
PyInstaller onedir creates a parent bootloader + child Python process on Windows. `child.kill()` only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use `taskkill /F /T` for clean shutdown. This is not a blocker -- our existing `--parent-pid` watchdog mechanism in `server.py` already handles orphan cleanup.
## Next Steps
1. Prototype: Build the current CUDA binary with `--onedir` and verify torch CUDA works from the output directory
2. Measure the size split: how much is NVIDIA libs vs everything else
3. Design the two-archive download flow and dual version checking
4. Update `cuda.py` for dual-archive extraction (server core + cuda-libs)
5. Update `main.rs`: change launch path to `backends/cuda/` dir + add `.current_dir()`
6. Add `ensure_cuda_structure()` helper in Rust to verify exe + nvidia/ subdirs exist before spawning
7. Update CI pipeline: `build-cuda-windows` produces two archives instead of split parts
8. ~~Update `split_binary.py` or replace with archive-based distribution~~ Done: replaced with `package_cuda.py`
+8 -4
View File
@@ -46,6 +46,8 @@ setup-python:
{{ pip }} install -r {{ backend_dir }}/requirements.txt
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
{{ pip }} install --no-deps chatterbox-tts
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
{{ pip }} install --no-deps hume-tada
# Apple Silicon: install MLX backend
if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then
echo "Detected Apple Silicon — installing MLX dependencies..."
@@ -70,10 +72,11 @@ setup-python:
$hasNvidia = $null -ne (Get-WmiObject Win32_VideoController | Where-Object { $_.Name -match 'NVIDIA' })
if ($hasNvidia) { \
Write-Host "NVIDIA GPU detected — installing PyTorch with CUDA support..."; \
& "{{ pip }}" install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126; \
& "{{ pip }}" install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128; \
}
& "{{ pip }}" install -r {{ backend_dir }}/requirements.txt
& "{{ pip }}" install --no-deps chatterbox-tts
& "{{ pip }}" install --no-deps hume-tada
& "{{ pip }}" install git+https://github.com/QwenLM/Qwen3-TTS.git
& "{{ pip }}" install pyinstaller ruff pytest pytest-asyncio -q
Write-Host "Python environment ready."
@@ -205,10 +208,11 @@ build-server-cuda: _ensure-venv
$env:PATH = "{{ venv_bin }};$env:PATH"; \
& "{{ python }}" backend/build_binary.py --cuda; \
if ($LASTEXITCODE -ne 0) { throw "build_binary.py --cuda failed with exit code $LASTEXITCODE" }; \
$dest = "$env:APPDATA/com.voicebox.app/backends"; \
$dest = "$env:APPDATA/sh.voicebox.app/backends/cuda"; \
if (Test-Path $dest) { Remove-Item -Recurse -Force $dest }; \
New-Item -ItemType Directory -Path $dest -Force | Out-Null; \
Copy-Item "backend/dist/voicebox-server-cuda.exe" "$dest/voicebox-server-cuda.exe" -Force; \
Write-Host "Copied CUDA binary to $dest"
Copy-Item "backend/dist/voicebox-server-cuda/*" $dest -Recurse -Force; \
Write-Host "Copied CUDA backend to $dest"
# Build everything locally: CPU server + CUDA server + installable Tauri app
[windows]
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@voicebox/landing",
"version": "0.2.3",
"version": "0.3.1",
"description": "Landing page for voicebox.sh",
"scripts": {
"dev": "bun --bun next dev --turbo",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "voicebox",
"version": "0.2.3",
"version": "0.3.1",
"private": true,
"workspaces": [
"app",
+232
View File
@@ -0,0 +1,232 @@
"""
Package the PyInstaller --onedir CUDA build into two archives.
Takes the PyInstaller --onedir output directory and splits it into:
1. voicebox-server-cuda.tar.gz — server core (exe + non-NVIDIA deps)
2. cuda-libs-cu128.tar.gz — NVIDIA runtime libraries only
3. cuda-libs.json — version manifest for the CUDA libs
Usage:
python scripts/package_cuda.py backend/dist/voicebox-server-cuda/
python scripts/package_cuda.py backend/dist/voicebox-server-cuda/ --output release-assets/
python scripts/package_cuda.py backend/dist/voicebox-server-cuda/ --cuda-libs-version cu128-v1
"""
import argparse
import hashlib
import json
import sys
import tarfile
from pathlib import Path
# DLL name prefixes that identify NVIDIA CUDA runtime libraries.
# These DLLs may appear in different locations depending on the torch
# and PyInstaller version:
# - nvidia/ subdirectories (older torch with separate nvidia-* packages)
# - _internal/torch/lib/ (torch 2.10+ bundles NVIDIA DLLs directly)
# - Top-level directory (some PyInstaller versions)
NVIDIA_DLL_PREFIXES = (
"cublas",
"cublaslt",
"cudart",
"cudnn",
"cufft",
"cufftw",
"curand",
"cusolver",
"cusolvermg",
"cusparse",
"nvjitlink",
"nvrtc",
"nccl",
"caffe2_nvrtc",
)
# Files to keep in the server core even if they match NVIDIA prefixes.
# These are small Python modules or stubs, not the large runtime DLLs.
NVIDIA_KEEP_IN_CORE = {
"torch/cuda/nccl.py",
"torch/_inductor/codegen/cuda/cutlass_lib_extensions/cutlass_mock_imports/cuda/cudart.py",
}
def is_nvidia_file(rel_path: str) -> bool:
"""Check if a relative path belongs to the NVIDIA CUDA libs.
Identifies large NVIDIA runtime DLLs (.dll/.so) regardless of where
PyInstaller placed them. Excludes small Python stubs that happen to
share NVIDIA-related names.
"""
rel_lower = rel_path.lower().replace("\\", "/")
# Never split out Python source files or small stubs
if rel_lower in NVIDIA_KEEP_IN_CORE:
return False
# Files under nvidia/ subdirectory tree (older torch layout)
if rel_lower.startswith("nvidia/") or "/nvidia/" in rel_lower:
# Only DLLs/shared objects — not .py, .dist-info, etc.
if rel_lower.endswith((".dll", ".so")):
return True
# Include entire nvidia/ namespace package tree
for part in rel_lower.split("/"):
if part == "nvidia":
return True
# NVIDIA DLLs anywhere in the tree (e.g. _internal/torch/lib/cublas64_12.dll)
name = rel_lower.rsplit("/", 1)[-1]
if name.endswith(".dll") or name.endswith(".so"):
name_no_ext = name.rsplit(".", 1)[0]
for prefix in NVIDIA_DLL_PREFIXES:
if name_no_ext.startswith(prefix):
return True
return False
def sha256_file(path: Path) -> str:
"""Compute SHA-256 hex digest of a file."""
h = hashlib.sha256()
with open(path, "rb") as f:
while True:
chunk = f.read(1024 * 1024)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
def package(
onedir_path: Path,
output_dir: Path,
cuda_libs_version: str,
torch_compat: str,
):
output_dir.mkdir(parents=True, exist_ok=True)
# Collect all files in the onedir output, split into core vs nvidia
core_files = []
nvidia_files = []
for item in sorted(onedir_path.rglob("*")):
if item.is_dir():
continue
rel = item.relative_to(onedir_path)
rel_str = str(rel)
if is_nvidia_file(rel_str):
nvidia_files.append((rel_str, item))
else:
core_files.append((rel_str, item))
core_size = sum(f.stat().st_size for _, f in core_files)
nvidia_size = sum(f.stat().st_size for _, f in nvidia_files)
print(f"Input directory: {onedir_path}")
print(f"Core files: {len(core_files)} ({core_size / (1024**2):.1f} MB)")
print(f"NVIDIA files: {len(nvidia_files)} ({nvidia_size / (1024**2):.1f} MB)")
if not nvidia_files:
print(
f"ERROR: No NVIDIA files found in {onedir_path}. "
"Refusing to create an empty CUDA libs archive.",
file=sys.stderr,
)
print(
"Make sure you built with --cuda and the NVIDIA packages are present.",
file=sys.stderr,
)
sys.exit(1)
# Create server core archive
# Files are stored relative to the archive root (no parent directory prefix)
# so extracting to backends/cuda/ puts everything at the right level.
server_archive = output_dir / "voicebox-server-cuda.tar.gz"
print(f"\nCreating server core archive: {server_archive.name}")
with tarfile.open(server_archive, "w:gz") as tar:
for rel_str, full_path in core_files:
tar.add(full_path, arcname=rel_str)
server_sha = sha256_file(server_archive)
(output_dir / "voicebox-server-cuda.tar.gz.sha256").write_text(
f"{server_sha} voicebox-server-cuda.tar.gz\n"
)
print(f" Size: {server_archive.stat().st_size / (1024**2):.1f} MB")
print(f" SHA-256: {server_sha[:16]}...")
# Create CUDA libs archive
cuda_libs_archive = output_dir / f"cuda-libs-{cuda_libs_version}.tar.gz"
print(f"\nCreating CUDA libs archive: {cuda_libs_archive.name}")
with tarfile.open(cuda_libs_archive, "w:gz") as tar:
for rel_str, full_path in nvidia_files:
tar.add(full_path, arcname=rel_str)
cuda_sha = sha256_file(cuda_libs_archive)
(output_dir / f"cuda-libs-{cuda_libs_version}.tar.gz.sha256").write_text(
f"{cuda_sha} cuda-libs-{cuda_libs_version}.tar.gz\n"
)
print(f" Size: {cuda_libs_archive.stat().st_size / (1024**2):.1f} MB")
print(f" SHA-256: {cuda_sha[:16]}...")
# Write cuda-libs.json manifest
manifest = {
"version": cuda_libs_version,
"torch_compat": torch_compat,
"archive": cuda_libs_archive.name,
"sha256": cuda_sha,
}
manifest_path = output_dir / "cuda-libs.json"
manifest_path.write_text(json.dumps(manifest, indent=2) + "\n")
print(f"\nManifest: {manifest_path.name}")
print(json.dumps(manifest, indent=2))
# Summary
total_input = core_size + nvidia_size
total_output = server_archive.stat().st_size + cuda_libs_archive.stat().st_size
print(f"\nTotal input: {total_input / (1024**3):.2f} GB")
print(f"Total output: {total_output / (1024**3):.2f} GB (compressed)")
print(
f"Server core: {server_archive.stat().st_size / (1024**2):.1f} MB (redownloaded on app update)"
)
print(
f"CUDA libs: {cuda_libs_archive.stat().st_size / (1024**2):.1f} MB (cached until CUDA toolkit bump)"
)
def main():
parser = argparse.ArgumentParser(
description="Package PyInstaller --onedir CUDA build into server + CUDA libs archives"
)
parser.add_argument(
"input",
type=Path,
help="Path to PyInstaller --onedir output directory (e.g. backend/dist/voicebox-server-cuda/)",
)
parser.add_argument(
"--output",
type=Path,
default=None,
help="Output directory for archives (default: same as input parent)",
)
parser.add_argument(
"--cuda-libs-version",
type=str,
default="cu128-v1",
help="Version string for the CUDA libs archive (default: cu128-v1)",
)
parser.add_argument(
"--torch-compat",
type=str,
default=">=2.7.0,<2.11.0",
help="Torch version compatibility range (default: >=2.6.0,<2.11.0)",
)
args = parser.parse_args()
if not args.input.is_dir():
print(f"Error: {args.input} is not a directory", file=sys.stderr)
print("Expected a PyInstaller --onedir output directory.", file=sys.stderr)
sys.exit(1)
output_dir = args.output or args.input.parent
package(args.input, output_dir, args.cuda_libs_version, args.torch_compat)
if __name__ == "__main__":
main()
-82
View File
@@ -1,82 +0,0 @@
"""
Split a large binary into chunks for GitHub Releases (<2 GB each).
Usage:
python scripts/split_binary.py backend/dist/voicebox-server-cuda.exe
python scripts/split_binary.py backend/dist/voicebox-server-cuda.exe --chunk-size 1900000000
python scripts/split_binary.py backend/dist/voicebox-server-cuda.exe --output release-assets/
The script produces:
- voicebox-server-cuda.part00.exe, .part01.exe, ... (binary chunks)
- voicebox-server-cuda.sha256 (SHA-256 checksum of the complete file)
- voicebox-server-cuda.manifest (ordered list of part filenames)
"""
import argparse
import hashlib
import sys
from pathlib import Path
def split(input_path: Path, chunk_size: int, output_dir: Path):
output_dir.mkdir(parents=True, exist_ok=True)
data = input_path.read_bytes()
total_size = len(data)
# Write SHA-256 of the complete file
sha256 = hashlib.sha256(data).hexdigest()
checksum_file = output_dir / f"{input_path.stem}.sha256"
checksum_file.write_text(f"{sha256} {input_path.name}\n")
# Split into chunks
parts = []
for i in range(0, total_size, chunk_size):
part_index = len(parts)
part_name = f"{input_path.stem}.part{part_index:02d}{input_path.suffix}"
part_path = output_dir / part_name
part_path.write_bytes(data[i:i + chunk_size])
parts.append(part_name)
# Write manifest (ordered list of part filenames)
manifest_file = output_dir / f"{input_path.stem}.manifest"
manifest_file.write_text("\n".join(parts) + "\n")
print(f"Input: {input_path} ({total_size / (1024**3):.2f} GB)")
print(f"Output: {output_dir}/")
print(f"Parts: {len(parts)} (chunk size: {chunk_size / (1024**3):.2f} GB)")
print(f"SHA-256: {sha256}")
print(f"Manifest: {manifest_file.name}")
for p in parts:
size = (output_dir / p).stat().st_size
print(f" {p} ({size / (1024**3):.2f} GB)")
def main():
parser = argparse.ArgumentParser(
description="Split a large binary into chunks for GitHub Releases"
)
parser.add_argument("input", type=Path, help="Path to the binary file to split")
parser.add_argument(
"--chunk-size",
type=int,
default=1_900_000_000, # 1.9 GB — safely under 2 GB GitHub limit
help="Maximum chunk size in bytes (default: 1.9 GB)",
)
parser.add_argument(
"--output",
type=Path,
default=None,
help="Output directory (default: same directory as input)",
)
args = parser.parse_args()
if not args.input.exists():
print(f"Error: {args.input} does not exist", file=sys.stderr)
sys.exit(1)
output_dir = args.output or args.input.parent
split(args.input, args.chunk_size, output_dir)
if __name__ == "__main__":
main()
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@voicebox/tauri",
"private": true,
"version": "0.2.3",
"version": "0.3.1",
"type": "module",
"scripts": {
"dev": "vite",
+1 -1
View File
@@ -5041,7 +5041,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "voicebox"
version = "0.2.3"
version = "0.3.1"
dependencies = [
"base64 0.22.1",
"core-foundation-sys",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "voicebox"
version = "0.2.3"
version = "0.3.1"
description = "A production-quality desktop app for Qwen3-TTS voice cloning and generation"
authors = ["you"]
license = ""
+82 -4
View File
@@ -23,15 +23,15 @@ fn main() {
}
}
let project_root = env!("CARGO_MANIFEST_DIR");
let gen_dir = format!("{}/gen", project_root);
std::fs::create_dir_all(&gen_dir).expect("Failed to create gen directory");
// Compile macOS Liquid Glass icon
#[cfg(target_os = "macos")]
{
let project_root = env!("CARGO_MANIFEST_DIR");
// voicebox.icon is in tauri/assets/voicebox.icon (one level up from src-tauri)
let icon_source = format!("{}/../assets/voicebox.icon", project_root);
let gen_dir = format!("{}/gen", project_root);
std::fs::create_dir_all(&gen_dir).expect("Failed to create gen directory");
if std::path::Path::new(&icon_source).exists() {
println!("cargo:rerun-if-changed={}", icon_source);
@@ -76,6 +76,71 @@ fn main() {
panic!("Icon compilation failed");
}
}
// Generate voicebox.icns from the source PNG via sips + iconutil
let icns_path = format!("{}/voicebox.icns", gen_dir);
if !std::path::Path::new(&icns_path).exists() {
let source_png = format!("{}/Assets/Voicebox.png", icon_source);
if std::path::Path::new(&source_png).exists() {
let iconset_dir = format!("{}/voicebox.iconset", gen_dir);
std::fs::create_dir_all(&iconset_dir).ok();
let sizes: &[(u32, &str)] = &[
(16, "icon_16x16.png"),
(32, "[email protected]"),
(32, "icon_32x32.png"),
(64, "[email protected]"),
(128, "icon_128x128.png"),
(256, "[email protected]"),
(256, "icon_256x256.png"),
(512, "[email protected]"),
(512, "icon_512x512.png"),
(1024, "[email protected]"),
];
for (size, name) in sizes {
let dest = format!("{}/{}", iconset_dir, name);
let status = Command::new("sips")
.args([
"-z",
&size.to_string(),
&size.to_string(),
&source_png,
"--out",
&dest,
])
.output();
if let Ok(out) = status {
if !out.status.success() {
eprintln!(
"sips failed for {}: {}",
name,
String::from_utf8_lossy(&out.stderr)
);
}
}
}
let iconutil_output = Command::new("iconutil")
.args(["-c", "icns", "-o", &icns_path, &iconset_dir])
.output();
match iconutil_output {
Ok(out) if out.status.success() => {
println!("Generated voicebox.icns");
}
Ok(out) => {
eprintln!("iconutil failed: {}", String::from_utf8_lossy(&out.stderr));
}
Err(e) => {
eprintln!("Failed to run iconutil: {}", e);
}
}
// Clean up iconset directory
std::fs::remove_dir_all(&iconset_dir).ok();
}
}
} else {
println!(
"cargo:warning=Icon source not found at {}, skipping icon compilation",
@@ -84,5 +149,18 @@ fn main() {
}
}
// Ensure all resource files exist so Tauri's bundler doesn't fail.
// On non-macOS these are always stubs. On macOS, actool may not produce
// Assets.car if the Xcode version doesn't support the .icon format.
{
let required = ["Assets.car", "voicebox.icns", "partial.plist"];
for name in required {
let path = format!("{}/{}", gen_dir, name);
if !std::path::Path::new(&path).exists() {
std::fs::write(&path, b"").ok();
}
}
}
tauri_build::build()
}
+16 -10
View File
@@ -197,22 +197,24 @@ async fn start_server(
println!("Data directory: {:?}", data_dir);
println!("Remote mode: {}", remote.unwrap_or(false));
// Check for CUDA backend binary in data directory
// Check for CUDA backend in data directory (onedir layout: backends/cuda/)
let cuda_binary = {
let backends_dir = data_dir.join("backends");
let cuda_dir = data_dir.join("backends").join("cuda");
let cuda_name = if cfg!(windows) {
"voicebox-server-cuda.exe"
} else {
"voicebox-server-cuda"
};
let path = backends_dir.join(cuda_name);
if path.exists() {
println!("Found CUDA backend binary at {:?}", path);
let exe_path = cuda_dir.join(cuda_name);
if exe_path.exists() {
println!("Found CUDA backend at {:?}", cuda_dir);
// Version check: run --version and compare to app version
// Version check: run --version from the onedir directory so
// PyInstaller can find its support files for the fast --version path
let app_version = app.config().version.clone().unwrap_or_default();
let version_ok = match std::process::Command::new(&path)
let version_ok = match std::process::Command::new(&exe_path)
.arg("--version")
.current_dir(&cuda_dir)
.output()
{
Ok(output) => {
@@ -237,7 +239,7 @@ async fn start_server(
};
if version_ok {
Some(path)
Some(exe_path)
} else {
None
}
@@ -300,10 +302,14 @@ async fn start_server(
println!("Custom models directory: {}", dir);
}
// If CUDA binary exists, launch it directly instead of the bundled sidecar
// If CUDA binary exists, launch it from the onedir directory.
// .current_dir() is critical: PyInstaller onedir expects all DLLs and
// support files (nvidia/, _internal/, etc.) relative to the exe.
let spawn_result = if let Some(ref cuda_path) = cuda_binary {
println!("Launching CUDA backend: {:?}", cuda_path);
let cuda_dir = cuda_path.parent().unwrap();
println!("Launching CUDA backend: {:?} (cwd: {:?})", cuda_path, cuda_dir);
let mut cmd = app.shell().command(cuda_path.to_str().unwrap());
cmd = cmd.current_dir(cuda_dir);
cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);
if is_remote {
cmd = cmd.args(["--host", "0.0.0.0"]);
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "Voicebox",
"version": "0.2.3",
"version": "0.3.1",
"identifier": "sh.voicebox.app",
"build": {
"beforeDevCommand": "bun run dev",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@voicebox/web",
"private": true,
"version": "0.2.3",
"version": "0.3.1",
"type": "module",
"scripts": {
"dev": "vite",