mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-28 22:55:15 -07:00
Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
273483ffcf | ||
|
|
5774a168a9 | ||
|
|
6bf40bd2d0 | ||
|
|
12cda2e090 | ||
|
|
7a90290a76 | ||
|
|
b02ce8e2f3 | ||
|
|
4e7772a21d | ||
|
|
51fb320b8c | ||
|
|
8ac202aa58 | ||
|
|
ac68052945 | ||
|
|
e601fd2ca4 | ||
|
|
7b25e0ba0b | ||
|
|
a6817cd082 |
@@ -0,0 +1,120 @@
|
|||||||
|
---
|
||||||
|
name: add-tts-engine
|
||||||
|
description: Use this skill to add a new TTS engine to Voicebox. It walks through dependency research, backend implementation, frontend wiring, PyInstaller bundling, and frozen-build testing. Always start with Phase 0 (dependency audit) before writing any code.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Add TTS Engine
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Integrate a new text-to-speech engine into Voicebox end-to-end: dependency research, backend protocol implementation, frontend UI wiring, PyInstaller bundling, and frozen-build verification. The user should only need to test the final build locally.
|
||||||
|
|
||||||
|
## Reference Doc
|
||||||
|
|
||||||
|
The full phased guide lives at `docs/content/docs/developer/tts-engines.mdx`. **Read this file in its entirety before starting.** It contains:
|
||||||
|
|
||||||
|
- Phase 0: Dependency research (mandatory before writing code)
|
||||||
|
- Phase 1: Backend implementation (`TTSBackend` protocol)
|
||||||
|
- Phase 2: Route and service integration (usually zero changes)
|
||||||
|
- Phase 3: Frontend integration (5 files)
|
||||||
|
- Phase 4: Dependencies (`requirements.txt`, justfile, CI, Docker)
|
||||||
|
- Phase 5: PyInstaller bundling (`build_binary.py` + `server.py`)
|
||||||
|
- Phase 6: Common upstream workarounds
|
||||||
|
- Implementation checklist (gate between phases)
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
### 1. Read the guide
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Read the full TTS engines doc
|
||||||
|
cat docs/content/docs/developer/tts-engines.mdx
|
||||||
|
```
|
||||||
|
|
||||||
|
Internalize all phases, especially Phase 0 and Phase 5. The v0.2.3 release was three patch releases because Phase 0 was skipped.
|
||||||
|
|
||||||
|
### 2. Dependency research (Phase 0)
|
||||||
|
|
||||||
|
Clone the model library into a temporary directory and audit it. Do NOT skip this.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir /tmp/engine-research && cd /tmp/engine-research
|
||||||
|
git clone <model-library-url>
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the grep searches from Phase 0.2 in the guide against the cloned source and its transitive dependencies. Produce a written dependency audit covering:
|
||||||
|
|
||||||
|
1. PyPI vs non-PyPI packages
|
||||||
|
2. PyInstaller directives needed (`--collect-all`, `--copy-metadata`, `--hidden-import`)
|
||||||
|
3. Runtime data files that must be bundled
|
||||||
|
4. Native library paths that need env var overrides in frozen builds
|
||||||
|
5. Monkey-patches needed (`torch.load`, float64, MPS, HF token)
|
||||||
|
6. Sample rate
|
||||||
|
7. Model download method (`from_pretrained` vs `snapshot_download` + `from_local`)
|
||||||
|
|
||||||
|
Test model loading and generation on CPU in the throwaway venv before proceeding.
|
||||||
|
|
||||||
|
### 3. Implement (Phases 1–4)
|
||||||
|
|
||||||
|
Follow the guide's phases in order. Key files to modify:
|
||||||
|
|
||||||
|
**Backend (Phase 1):**
|
||||||
|
- Create `backend/backends/<engine>_backend.py`
|
||||||
|
- Register in `backend/backends/__init__.py` (ModelConfig + TTS_ENGINES + factory)
|
||||||
|
- Update regex in `backend/models.py`
|
||||||
|
|
||||||
|
**Frontend (Phase 3):**
|
||||||
|
- `app/src/lib/api/types.ts` — engine union type
|
||||||
|
- `app/src/lib/constants/languages.ts` — ENGINE_LANGUAGES
|
||||||
|
- `app/src/components/Generation/EngineModelSelector.tsx` — ENGINE_OPTIONS, ENGINE_DESCRIPTIONS
|
||||||
|
- `app/src/lib/hooks/useGenerationForm.ts` — Zod schema, model-name mapping
|
||||||
|
- `app/src/components/ServerSettings/ModelManagement.tsx` — MODEL_DESCRIPTIONS
|
||||||
|
|
||||||
|
**Dependencies (Phase 4):**
|
||||||
|
- `backend/requirements.txt`
|
||||||
|
- `justfile` (setup-python, setup-python-release targets)
|
||||||
|
- `.github/workflows/release.yml`
|
||||||
|
- `Dockerfile` (if applicable)
|
||||||
|
|
||||||
|
### 4. PyInstaller bundling (Phase 5)
|
||||||
|
|
||||||
|
Register the engine in `backend/build_binary.py`:
|
||||||
|
- `--hidden-import` for the backend module and model package
|
||||||
|
- `--collect-all` for packages using `inspect.getsource`, shipping data files, or native libraries
|
||||||
|
- `--copy-metadata` for packages using `importlib.metadata`
|
||||||
|
|
||||||
|
If the engine has native data paths, add `os.environ.setdefault()` in `backend/server.py` inside the `if getattr(sys, 'frozen', False):` block.
|
||||||
|
|
||||||
|
### 5. Verify in dev mode
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Test the full chain: model download → load → generate → voice cloning.
|
||||||
|
|
||||||
|
### 6. Use the checklist
|
||||||
|
|
||||||
|
Walk through the Implementation Checklist at the bottom of `tts-engines.mdx`. Every item must be checked before handing the build to the user.
|
||||||
|
|
||||||
|
## Key Lessons (from v0.2.3)
|
||||||
|
|
||||||
|
These are the most common failure modes. Phase 0 research catches all of them:
|
||||||
|
|
||||||
|
| Pattern | Symptom in Frozen Build | Fix |
|
||||||
|
|---------|------------------------|-----|
|
||||||
|
| `@typechecked` / `inspect.getsource()` | "could not get source code" | `--collect-all <package>` |
|
||||||
|
| Package ships pretrained model files | `FileNotFoundError` for `.pth.tar`, `.yaml` | `--collect-all <package>` |
|
||||||
|
| C library with hardcoded system paths | `FileNotFoundError` for `/usr/share/...` | `--collect-all` + env var in `server.py` |
|
||||||
|
| `importlib.metadata.version()` | "No package metadata found" | `--copy-metadata <package>` |
|
||||||
|
| `torch.load` without `map_location` | CUDA device not available on CPU build | Monkey-patch `torch.load` |
|
||||||
|
| `torch.from_numpy` on float64 data | dtype mismatch RuntimeError | Cast to `.float()` |
|
||||||
|
| `token=True` in HF download calls | Auth failure without stored HF token | Use `snapshot_download(token=None)` + `from_local()` |
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The route and service layers have zero per-engine dispatch points. `main.py` requires zero changes.
|
||||||
|
- The model config registry in `backends/__init__.py` handles all dispatch automatically.
|
||||||
|
- Use `get_torch_device()` and `model_load_progress()` from `backends/base.py` — don't reimplement device detection or progress tracking.
|
||||||
|
- Always test with a **clean HuggingFace cache** (no pre-downloaded models from dev).
|
||||||
|
- Do NOT push or create a release. Hand the build to the user for local testing.
|
||||||
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
[bumpversion]
|
[bumpversion]
|
||||||
current_version = 0.2.3
|
current_version = 0.3.0
|
||||||
commit = True
|
commit = True
|
||||||
tag = True
|
tag = True
|
||||||
tag_name = v{new_version}
|
tag_name = v{new_version}
|
||||||
|
|||||||
@@ -62,6 +62,7 @@ jobs:
|
|||||||
pip install pyinstaller
|
pip install pyinstaller
|
||||||
pip install -r backend/requirements.txt
|
pip install -r backend/requirements.txt
|
||||||
pip install --no-deps chatterbox-tts
|
pip install --no-deps chatterbox-tts
|
||||||
|
pip install --no-deps hume-tada
|
||||||
|
|
||||||
- name: Install MLX dependencies (Apple Silicon only)
|
- name: Install MLX dependencies (Apple Silicon only)
|
||||||
if: matrix.backend == 'mlx'
|
if: matrix.backend == 'mlx'
|
||||||
@@ -188,6 +189,7 @@ jobs:
|
|||||||
pip install pyinstaller
|
pip install pyinstaller
|
||||||
pip install -r backend/requirements.txt
|
pip install -r backend/requirements.txt
|
||||||
pip install --no-deps chatterbox-tts
|
pip install --no-deps chatterbox-tts
|
||||||
|
pip install --no-deps hume-tada
|
||||||
|
|
||||||
- name: Install PyTorch with CUDA 12.6
|
- name: Install PyTorch with CUDA 12.6
|
||||||
run: |
|
run: |
|
||||||
|
|||||||
@@ -51,6 +51,7 @@ app/openapi.json
|
|||||||
tauri/src-tauri/binaries/*
|
tauri/src-tauri/binaries/*
|
||||||
tauri/src-tauri/gen/Assets.car
|
tauri/src-tauri/gen/Assets.car
|
||||||
tauri/src-tauri/gen/voicebox.icns
|
tauri/src-tauri/gen/voicebox.icns
|
||||||
|
tauri/src-tauri/gen/partial.plist
|
||||||
|
|
||||||
# Temporary
|
# Temporary
|
||||||
tmp/
|
tmp/
|
||||||
|
|||||||
+29
-2
@@ -7,9 +7,25 @@
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
This release rewrites the backend into a modular architecture, migrates the documentation site to Fumadocs, and ships a batch of bug fixes and UI polish across the stack.
|
## [0.3.0] - 2026-03-17
|
||||||
|
|
||||||
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, model loading status is now visible in the UI, effects presets get a dropdown, and several race conditions and accessibility gaps are closed.
|
This release rewrites the backend into a modular architecture, overhauls the settings UI into routed sub-pages, fixes audio player freezing, migrates documentation to Fumadocs, and ships a batch of bug fixes targeting the most-reported issues from the tracker.
|
||||||
|
|
||||||
|
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, settings have been split into dedicated routed pages with server logs, a changelog viewer, and an about page. The audio player no longer freezes mid-playback, and model loading status is now visible in the UI. Seven user-reported bugs have been fixed, including server crashes during sample uploads, generation list staleness, cryptic error messages, and CUDA support for RTX 50-series GPUs.
|
||||||
|
|
||||||
|
### Settings Overhaul ([#294](https://github.com/jamiepine/voicebox/pull/294))
|
||||||
|
- Split settings into routed sub-tabs: General, Generation, GPU, Logs, Changelog, About
|
||||||
|
- Added live server log viewer with auto-scroll
|
||||||
|
- Added in-app changelog page that parses `CHANGELOG.md` at build time
|
||||||
|
- Added About page with version info, license, and generation folder quick-open
|
||||||
|
- Extracted reusable `SettingRow` component for consistent setting layouts
|
||||||
|
|
||||||
|
### Audio Player Fix ([#293](https://github.com/jamiepine/voicebox/pull/293))
|
||||||
|
- Fixed audio player freezing during playback
|
||||||
|
- Improved playback UX with better state management and listener cleanup
|
||||||
|
- Fixed restart race condition during regeneration
|
||||||
|
- Added stable keys for audio element re-rendering
|
||||||
|
- Improved accessibility across player controls
|
||||||
|
|
||||||
### Backend Refactor ([#285](https://github.com/jamiepine/voicebox/pull/285))
|
### Backend Refactor ([#285](https://github.com/jamiepine/voicebox/pull/285))
|
||||||
- Extracted all routes from `main.py` into 13 domain routers under `backend/routes/` — `main.py` dropped from ~3,100 lines to ~10
|
- Extracted all routes from `main.py` into 13 domain routers under `backend/routes/` — `main.py` dropped from ~3,100 lines to ~10
|
||||||
@@ -40,6 +56,17 @@ The backend's 3,000-line monolith `main.py` has been decomposed into domain rout
|
|||||||
- Softened select focus indicator opacity
|
- Softened select focus indicator opacity
|
||||||
- Addressed 4 critical and 12 major issues from CodeRabbit review
|
- Addressed 4 critical and 12 major issues from CodeRabbit review
|
||||||
|
|
||||||
|
### Bug Fixes ([#295](https://github.com/jamiepine/voicebox/pull/295))
|
||||||
|
- Fixed sample uploads crashing the server — audio decoding now runs in a thread pool instead of blocking the async event loop ([#278](https://github.com/jamiepine/voicebox/issues/278))
|
||||||
|
- Fixed generation list not updating when a generation completes — switched to `refetchQueries` for reliable cache busting, added SSE error fallback, and page reset on completion ([#231](https://github.com/jamiepine/voicebox/issues/231))
|
||||||
|
- Fixed error toasts showing `[object Object]` instead of the actual error message ([#290](https://github.com/jamiepine/voicebox/issues/290))
|
||||||
|
- Added Whisper model selection (`base`, `small`, `medium`, `large`, `turbo`) and expanded language support to the `/transcribe` endpoint ([#233](https://github.com/jamiepine/voicebox/issues/233))
|
||||||
|
- Upgraded CUDA backend build from cu121 to cu126 for RTX 50-series (Blackwell) GPU support ([#289](https://github.com/jamiepine/voicebox/issues/289))
|
||||||
|
- Handled client disconnects in SSE and streaming endpoints to suppress `[Errno 32] Broken Pipe` errors ([#248](https://github.com/jamiepine/voicebox/issues/248))
|
||||||
|
- Fixed Docker build failure from pip hash mismatch on Qwen3-TTS dependencies ([#286](https://github.com/jamiepine/voicebox/issues/286))
|
||||||
|
- Added 50 MB upload size limit with chunked reads to prevent unbounded memory allocation on sample uploads
|
||||||
|
- Eliminated redundant double audio decode in sample processing pipeline
|
||||||
|
|
||||||
### Platform Fixes
|
### Platform Fixes
|
||||||
- Replaced `netstat` with `TcpStream` + PowerShell for Windows port detection ([#277](https://github.com/jamiepine/voicebox/pull/277))
|
- Replaced `netstat` with `TcpStream` + PowerShell for Windows port detection ([#277](https://github.com/jamiepine/voicebox/pull/277))
|
||||||
- Fixed Docker frontend build and cleaned up Docker docs
|
- Fixed Docker frontend build and cleaned up Docker docs
|
||||||
|
|||||||
@@ -35,6 +35,8 @@ RUN pip install --no-cache-dir --upgrade pip
|
|||||||
|
|
||||||
COPY backend/requirements.txt .
|
COPY backend/requirements.txt .
|
||||||
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
||||||
|
RUN pip install --no-cache-dir --prefix=/install --no-deps chatterbox-tts
|
||||||
|
RUN pip install --no-cache-dir --prefix=/install --no-deps hume-tada
|
||||||
RUN pip install --no-cache-dir --prefix=/install \
|
RUN pip install --no-cache-dir --prefix=/install \
|
||||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||||
|
|
||||||
|
|||||||
@@ -59,10 +59,10 @@
|
|||||||
|
|
||||||
## What is Voicebox?
|
## What is Voicebox?
|
||||||
|
|
||||||
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||||
|
|
||||||
- **Complete privacy** — models and voice data stay on your machine
|
- **Complete privacy** — models and voice data stay on your machine
|
||||||
- **4 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
|
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
|
||||||
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
|
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
|
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
|
||||||
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
||||||
@@ -93,7 +93,7 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
|
|||||||
|
|
||||||
### Multi-Engine Voice Cloning
|
### Multi-Engine Voice Cloning
|
||||||
|
|
||||||
Four TTS engines with different strengths, switchable per-generation:
|
Five TTS engines with different strengths, switchable per-generation:
|
||||||
|
|
||||||
| Engine | Languages | Strengths |
|
| Engine | Languages | Strengths |
|
||||||
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
@@ -101,6 +101,7 @@ Four TTS engines with different strengths, switchable per-generation:
|
|||||||
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||||
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
|
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
|
||||||
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||||
|
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
|
||||||
|
|
||||||
### Emotions & Paralinguistic Tags
|
### Emotions & Paralinguistic Tags
|
||||||
|
|
||||||
@@ -230,7 +231,7 @@ Full API documentation available at `http://localhost:17493/docs`.
|
|||||||
| Frontend | React, TypeScript, Tailwind CSS |
|
| Frontend | React, TypeScript, Tailwind CSS |
|
||||||
| State | Zustand, React Query |
|
| State | Zustand, React Query |
|
||||||
| Backend | FastAPI (Python) |
|
| Backend | FastAPI (Python) |
|
||||||
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
|
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
|
||||||
| Effects | Pedalboard (Spotify) |
|
| Effects | Pedalboard (Spotify) |
|
||||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||||
@@ -245,7 +246,7 @@ Full API documentation available at `http://localhost:17493/docs`.
|
|||||||
| ----------------------- | ---------------------------------------------- |
|
| ----------------------- | ---------------------------------------------- |
|
||||||
| **Real-time Streaming** | Stream audio as it generates, word by word |
|
| **Real-time Streaming** | Stream audio as it generates, word by word |
|
||||||
| **Voice Design** | Create new voices from text descriptions |
|
| **Voice Design** | Create new voices from text descriptions |
|
||||||
| **More Models** | XTTS, Bark, and other open-source voice models |
|
| **More Models** | XTTS, Bark, and other open-source voice models |
|
||||||
| **Plugin Architecture** | Extend with custom models and effects |
|
| **Plugin Architecture** | Extend with custom models and effects |
|
||||||
| **Mobile Companion** | Control Voicebox from your phone |
|
| **Mobile Companion** | Control Voicebox from your phone |
|
||||||
|
|
||||||
@@ -276,6 +277,12 @@ just build # Build CPU server binary + Tauri app
|
|||||||
just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app
|
just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Adding New Voice Models
|
||||||
|
|
||||||
|
The multi-engine architecture makes adding new TTS engines straightforward. A [step-by-step guide](docs/content/docs/developer/tts-engines.mdx) covers the full process: dependency research, backend protocol implementation, frontend wiring, and PyInstaller bundling.
|
||||||
|
|
||||||
|
The guide is optimized for AI coding agents. An [agent skill](.agents/skills/add-tts-engine/SKILL.md) can pick up a model name and handle the entire integration autonomously — you just test the build locally.
|
||||||
|
|
||||||
### Project Structure
|
### Project Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@voicebox/app",
|
"name": "@voicebox/app",
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ const ENGINE_OPTIONS = [
|
|||||||
{ value: 'luxtts', label: 'LuxTTS' },
|
{ value: 'luxtts', label: 'LuxTTS' },
|
||||||
{ value: 'chatterbox', label: 'Chatterbox' },
|
{ value: 'chatterbox', label: 'Chatterbox' },
|
||||||
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
|
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
|
||||||
|
{ value: 'tada:1B', label: 'TADA 1B' },
|
||||||
|
{ value: 'tada:3B', label: 'TADA 3B Multilingual' },
|
||||||
] as const;
|
] as const;
|
||||||
|
|
||||||
const ENGINE_DESCRIPTIONS: Record<string, string> = {
|
const ENGINE_DESCRIPTIONS: Record<string, string> = {
|
||||||
@@ -27,6 +29,7 @@ const ENGINE_DESCRIPTIONS: Record<string, string> = {
|
|||||||
luxtts: 'Fast, English-focused',
|
luxtts: 'Fast, English-focused',
|
||||||
chatterbox: '23 languages, incl. Hebrew',
|
chatterbox: '23 languages, incl. Hebrew',
|
||||||
chatterbox_turbo: 'English, [laugh] [cough] tags',
|
chatterbox_turbo: 'English, [laugh] [cough] tags',
|
||||||
|
tada: 'HumeAI, 700s+ coherent audio',
|
||||||
};
|
};
|
||||||
|
|
||||||
/** Engines that only support English and should force language to 'en' on select. */
|
/** Engines that only support English and should force language to 'en' on select. */
|
||||||
@@ -34,6 +37,7 @@ const ENGLISH_ONLY_ENGINES = new Set(['luxtts', 'chatterbox_turbo']);
|
|||||||
|
|
||||||
function getSelectValue(engine: string, modelSize?: string): string {
|
function getSelectValue(engine: string, modelSize?: string): string {
|
||||||
if (engine === 'qwen') return `qwen:${modelSize || '1.7B'}`;
|
if (engine === 'qwen') return `qwen:${modelSize || '1.7B'}`;
|
||||||
|
if (engine === 'tada') return `tada:${modelSize || '1B'}`;
|
||||||
return engine;
|
return engine;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -48,6 +52,20 @@ function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: st
|
|||||||
if (!available.some((l) => l.value === currentLang)) {
|
if (!available.some((l) => l.value === currentLang)) {
|
||||||
form.setValue('language', available[0]?.value ?? 'en');
|
form.setValue('language', available[0]?.value ?? 'en');
|
||||||
}
|
}
|
||||||
|
} else if (value.startsWith('tada:')) {
|
||||||
|
const [, modelSize] = value.split(':');
|
||||||
|
form.setValue('engine', 'tada');
|
||||||
|
form.setValue('modelSize', modelSize as '1B' | '3B');
|
||||||
|
// TADA 1B is English-only; 3B is multilingual
|
||||||
|
if (modelSize === '1B') {
|
||||||
|
form.setValue('language', 'en');
|
||||||
|
} else {
|
||||||
|
const currentLang = form.getValues('language');
|
||||||
|
const available = getLanguageOptionsForEngine('tada');
|
||||||
|
if (!available.some((l) => l.value === currentLang)) {
|
||||||
|
form.setValue('language', available[0]?.value ?? 'en');
|
||||||
|
}
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
form.setValue('engine', value as GenerationFormValues['engine']);
|
form.setValue('engine', value as GenerationFormValues['engine']);
|
||||||
form.setValue('modelSize', undefined as unknown as '1.7B' | '0.6B');
|
form.setValue('modelSize', undefined as unknown as '1.7B' | '0.6B');
|
||||||
|
|||||||
@@ -62,6 +62,10 @@ const MODEL_DESCRIPTIONS: Record<string, string> = {
|
|||||||
'Production-grade open source TTS by Resemble AI. Supports 23 languages with voice cloning and emotion exaggeration control.',
|
'Production-grade open source TTS by Resemble AI. Supports 23 languages with voice cloning and emotion exaggeration control.',
|
||||||
'chatterbox-turbo':
|
'chatterbox-turbo':
|
||||||
'Streamlined 350M parameter TTS by Resemble AI. High-quality English speech with less compute and VRAM than larger models.',
|
'Streamlined 350M parameter TTS by Resemble AI. High-quality English speech with less compute and VRAM than larger models.',
|
||||||
|
'tada-1b':
|
||||||
|
'HumeAI TADA 1B — English speech-language model built on Llama 3.2 1B. Generates 700s+ of coherent audio with synchronized text-acoustic alignment.',
|
||||||
|
'tada-3b-ml':
|
||||||
|
'HumeAI TADA 3B Multilingual — built on Llama 3.2 3B. Supports 10 languages with high-fidelity voice cloning via text-acoustic dual alignment.',
|
||||||
'whisper-base':
|
'whisper-base':
|
||||||
'Smallest Whisper model (74M parameters). Fast transcription with moderate accuracy.',
|
'Smallest Whisper model (74M parameters). Fast transcription with moderate accuracy.',
|
||||||
'whisper-small':
|
'whisper-small':
|
||||||
@@ -391,7 +395,8 @@ export function ModelManagement() {
|
|||||||
(m) =>
|
(m) =>
|
||||||
m.model_name.startsWith('qwen-tts') ||
|
m.model_name.startsWith('qwen-tts') ||
|
||||||
m.model_name.startsWith('luxtts') ||
|
m.model_name.startsWith('luxtts') ||
|
||||||
m.model_name.startsWith('chatterbox'),
|
m.model_name.startsWith('chatterbox') ||
|
||||||
|
m.model_name.startsWith('tada'),
|
||||||
) ?? [];
|
) ?? [];
|
||||||
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
|
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
|
||||||
|
|
||||||
|
|||||||
@@ -42,8 +42,8 @@ export interface GenerationRequest {
|
|||||||
text: string;
|
text: string;
|
||||||
language: LanguageCode;
|
language: LanguageCode;
|
||||||
seed?: number;
|
seed?: number;
|
||||||
model_size?: '1.7B' | '0.6B';
|
model_size?: '1.7B' | '0.6B' | '1B' | '3B';
|
||||||
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo';
|
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada';
|
||||||
instruct?: string;
|
instruct?: string;
|
||||||
max_chunk_chars?: number;
|
max_chunk_chars?: number;
|
||||||
crossfade_ms?: number;
|
crossfade_ms?: number;
|
||||||
|
|||||||
@@ -66,6 +66,7 @@ export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
|
|||||||
'zh',
|
'zh',
|
||||||
],
|
],
|
||||||
chatterbox_turbo: ['en'],
|
chatterbox_turbo: ['en'],
|
||||||
|
tada: ['en', 'ar', 'zh', 'de', 'es', 'fr', 'it', 'ja', 'pl', 'pt'],
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
/** Helper: get language options for a given engine. */
|
/** Helper: get language options for a given engine. */
|
||||||
|
|||||||
@@ -15,9 +15,9 @@ const generationSchema = z.object({
|
|||||||
text: z.string().min(1, '').max(50000),
|
text: z.string().min(1, '').max(50000),
|
||||||
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
||||||
seed: z.number().int().optional(),
|
seed: z.number().int().optional(),
|
||||||
modelSize: z.enum(['1.7B', '0.6B']).optional(),
|
modelSize: z.enum(['1.7B', '0.6B', '1B', '3B']).optional(),
|
||||||
instruct: z.string().max(500).optional(),
|
instruct: z.string().max(500).optional(),
|
||||||
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo']).optional(),
|
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada']).optional(),
|
||||||
});
|
});
|
||||||
|
|
||||||
export type GenerationFormValues = z.infer<typeof generationSchema>;
|
export type GenerationFormValues = z.infer<typeof generationSchema>;
|
||||||
@@ -79,7 +79,11 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
|||||||
? 'chatterbox-tts'
|
? 'chatterbox-tts'
|
||||||
: engine === 'chatterbox_turbo'
|
: engine === 'chatterbox_turbo'
|
||||||
? 'chatterbox-turbo'
|
? 'chatterbox-turbo'
|
||||||
: `qwen-tts-${data.modelSize}`;
|
: engine === 'tada'
|
||||||
|
? data.modelSize === '3B'
|
||||||
|
? 'tada-3b-ml'
|
||||||
|
: 'tada-1b'
|
||||||
|
: `qwen-tts-${data.modelSize}`;
|
||||||
const displayName =
|
const displayName =
|
||||||
engine === 'luxtts'
|
engine === 'luxtts'
|
||||||
? 'LuxTTS'
|
? 'LuxTTS'
|
||||||
@@ -87,9 +91,13 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
|||||||
? 'Chatterbox TTS'
|
? 'Chatterbox TTS'
|
||||||
: engine === 'chatterbox_turbo'
|
: engine === 'chatterbox_turbo'
|
||||||
? 'Chatterbox Turbo'
|
? 'Chatterbox Turbo'
|
||||||
: data.modelSize === '1.7B'
|
: engine === 'tada'
|
||||||
? 'Qwen TTS 1.7B'
|
? data.modelSize === '3B'
|
||||||
: 'Qwen TTS 0.6B';
|
? 'TADA 3B Multilingual'
|
||||||
|
: 'TADA 1B'
|
||||||
|
: data.modelSize === '1.7B'
|
||||||
|
? 'Qwen TTS 1.7B'
|
||||||
|
: 'Qwen TTS 0.6B';
|
||||||
|
|
||||||
// Check if model needs downloading
|
// Check if model needs downloading
|
||||||
try {
|
try {
|
||||||
@@ -104,7 +112,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
|||||||
console.error('Failed to check model status:', error);
|
console.error('Failed to check model status:', error);
|
||||||
}
|
}
|
||||||
|
|
||||||
const isQwen = engine === 'qwen';
|
const hasModelSizes = engine === 'qwen' || engine === 'tada';
|
||||||
const effectsChain = options.getEffectsChain?.();
|
const effectsChain = options.getEffectsChain?.();
|
||||||
// This now returns immediately with status="generating"
|
// This now returns immediately with status="generating"
|
||||||
const result = await generation.mutateAsync({
|
const result = await generation.mutateAsync({
|
||||||
@@ -112,9 +120,9 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
|||||||
text: data.text,
|
text: data.text,
|
||||||
language: data.language,
|
language: data.language,
|
||||||
seed: data.seed,
|
seed: data.seed,
|
||||||
model_size: isQwen ? data.modelSize : undefined,
|
model_size: hasModelSizes ? data.modelSize : undefined,
|
||||||
engine,
|
engine,
|
||||||
instruct: isQwen ? data.instruct || undefined : undefined,
|
instruct: engine === 'qwen' ? data.instruct || undefined : undefined,
|
||||||
max_chunk_chars: maxChunkChars,
|
max_chunk_chars: maxChunkChars,
|
||||||
crossfade_ms: crossfadeMs,
|
crossfade_ms: crossfadeMs,
|
||||||
normalize: normalizeAudio,
|
normalize: normalizeAudio,
|
||||||
|
|||||||
+1
-1
@@ -1,3 +1,3 @@
|
|||||||
# Backend package
|
# Backend package
|
||||||
|
|
||||||
__version__ = "0.2.3"
|
__version__ = "0.3.0"
|
||||||
|
|||||||
@@ -166,6 +166,7 @@ TTS_ENGINES = {
|
|||||||
"luxtts": "LuxTTS",
|
"luxtts": "LuxTTS",
|
||||||
"chatterbox": "Chatterbox TTS",
|
"chatterbox": "Chatterbox TTS",
|
||||||
"chatterbox_turbo": "Chatterbox Turbo",
|
"chatterbox_turbo": "Chatterbox Turbo",
|
||||||
|
"tada": "TADA",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -259,6 +260,24 @@ def _get_non_qwen_tts_configs() -> list[ModelConfig]:
|
|||||||
needs_trim=True,
|
needs_trim=True,
|
||||||
languages=["en"],
|
languages=["en"],
|
||||||
),
|
),
|
||||||
|
ModelConfig(
|
||||||
|
model_name="tada-1b",
|
||||||
|
display_name="TADA 1B (English)",
|
||||||
|
engine="tada",
|
||||||
|
hf_repo_id="HumeAI/tada-1b",
|
||||||
|
model_size="1B",
|
||||||
|
size_mb=4000,
|
||||||
|
languages=["en"],
|
||||||
|
),
|
||||||
|
ModelConfig(
|
||||||
|
model_name="tada-3b-ml",
|
||||||
|
display_name="TADA 3B Multilingual",
|
||||||
|
engine="tada",
|
||||||
|
hf_repo_id="HumeAI/tada-3b-ml",
|
||||||
|
model_size="3B",
|
||||||
|
size_mb=8000,
|
||||||
|
languages=["en", "ar", "zh", "de", "es", "fr", "it", "ja", "pl", "pt"],
|
||||||
|
),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -339,10 +358,12 @@ def engine_has_model_sizes(engine: str) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
async def load_engine_model(engine: str, model_size: str = "default") -> None:
|
async def load_engine_model(engine: str, model_size: str = "default") -> None:
|
||||||
"""Load a model for the given engine, handling the Qwen model_size special case."""
|
"""Load a model for the given engine, handling engines with multiple model sizes."""
|
||||||
backend = get_tts_backend_for_engine(engine)
|
backend = get_tts_backend_for_engine(engine)
|
||||||
if engine == "qwen":
|
if engine == "qwen":
|
||||||
await backend.load_model_async(model_size)
|
await backend.load_model_async(model_size)
|
||||||
|
elif engine == "tada":
|
||||||
|
await backend.load_model(model_size)
|
||||||
else:
|
else:
|
||||||
await backend.load_model()
|
await backend.load_model()
|
||||||
|
|
||||||
@@ -358,7 +379,7 @@ async def ensure_model_cached_or_raise(engine: str, model_size: str = "default")
|
|||||||
cfg = c
|
cfg = c
|
||||||
break
|
break
|
||||||
|
|
||||||
if engine == "qwen":
|
if engine in ("qwen", "tada"):
|
||||||
if not backend._is_model_cached(model_size):
|
if not backend._is_model_cached(model_size):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=400,
|
status_code=400,
|
||||||
@@ -490,6 +511,10 @@ def get_tts_backend_for_engine(engine: str) -> TTSBackend:
|
|||||||
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
|
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
|
||||||
|
|
||||||
backend = ChatterboxTurboTTSBackend()
|
backend = ChatterboxTurboTTSBackend()
|
||||||
|
elif engine == "tada":
|
||||||
|
from .hume_backend import HumeTadaBackend
|
||||||
|
|
||||||
|
backend = HumeTadaBackend()
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
|
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,347 @@
|
|||||||
|
"""
|
||||||
|
HumeAI TADA TTS backend implementation.
|
||||||
|
|
||||||
|
Wraps HumeAI's TADA (Text-Acoustic Dual Alignment) model for
|
||||||
|
high-quality voice cloning. Two model variants:
|
||||||
|
- tada-1b: English-only, ~2B params (Llama 3.2 1B base)
|
||||||
|
- tada-3b-ml: Multilingual, ~4B params (Llama 3.2 3B base)
|
||||||
|
|
||||||
|
Both use a shared encoder/codec (HumeAI/tada-codec). The encoder
|
||||||
|
produces 1:1 aligned token embeddings from reference audio, and the
|
||||||
|
causal LM generates speech via flow-matching diffusion.
|
||||||
|
|
||||||
|
24kHz output, bf16 inference on CUDA, fp32 on CPU.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import threading
|
||||||
|
from typing import ClassVar, List, Optional, Tuple
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from . import TTSBackend
|
||||||
|
from .base import (
|
||||||
|
is_model_cached,
|
||||||
|
get_torch_device,
|
||||||
|
combine_voice_prompts as _combine_voice_prompts,
|
||||||
|
model_load_progress,
|
||||||
|
)
|
||||||
|
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# HuggingFace repos
|
||||||
|
TADA_CODEC_REPO = "HumeAI/tada-codec"
|
||||||
|
TADA_1B_REPO = "HumeAI/tada-1b"
|
||||||
|
TADA_3B_ML_REPO = "HumeAI/tada-3b-ml"
|
||||||
|
|
||||||
|
TADA_MODEL_REPOS = {
|
||||||
|
"1B": TADA_1B_REPO,
|
||||||
|
"3B": TADA_3B_ML_REPO,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Key weight files for cache detection
|
||||||
|
_TADA_MODEL_WEIGHT_FILES = [
|
||||||
|
"model.safetensors",
|
||||||
|
]
|
||||||
|
|
||||||
|
_TADA_CODEC_WEIGHT_FILES = [
|
||||||
|
"encoder/model.safetensors",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class HumeTadaBackend:
|
||||||
|
"""HumeAI TADA TTS backend for high-quality voice cloning."""
|
||||||
|
|
||||||
|
_load_lock: ClassVar[threading.Lock] = threading.Lock()
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.model = None
|
||||||
|
self.encoder = None
|
||||||
|
self.model_size = "1B" # default to 1B
|
||||||
|
self._device = None
|
||||||
|
self._model_load_lock = asyncio.Lock()
|
||||||
|
|
||||||
|
def _get_device(self) -> str:
|
||||||
|
# Force CPU on macOS — MPS has issues with flow matching
|
||||||
|
# and large vocab lm_head (>65536 output channels)
|
||||||
|
return get_torch_device(force_cpu_on_mac=True)
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
return self.model is not None
|
||||||
|
|
||||||
|
def _get_model_path(self, model_size: str = "1B") -> str:
|
||||||
|
return TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||||
|
|
||||||
|
def _is_model_cached(self, model_size: str = "1B") -> bool:
|
||||||
|
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||||
|
model_cached = is_model_cached(repo, required_files=_TADA_MODEL_WEIGHT_FILES)
|
||||||
|
codec_cached = is_model_cached(TADA_CODEC_REPO, required_files=_TADA_CODEC_WEIGHT_FILES)
|
||||||
|
return model_cached and codec_cached
|
||||||
|
|
||||||
|
async def load_model(self, model_size: str = "1B") -> None:
|
||||||
|
"""Load the TADA model and encoder."""
|
||||||
|
if self.model is not None and self.model_size == model_size:
|
||||||
|
return
|
||||||
|
async with self._model_load_lock:
|
||||||
|
if self.model is not None and self.model_size == model_size:
|
||||||
|
return
|
||||||
|
# Unload existing model if switching sizes
|
||||||
|
if self.model is not None:
|
||||||
|
self.unload_model()
|
||||||
|
self.model_size = model_size
|
||||||
|
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||||
|
|
||||||
|
def _load_model_sync(self, model_size: str = "1B"):
|
||||||
|
"""Synchronous model loading with progress tracking."""
|
||||||
|
model_name = f"tada-{model_size.lower()}"
|
||||||
|
is_cached = self._is_model_cached(model_size)
|
||||||
|
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||||
|
|
||||||
|
with model_load_progress(model_name, is_cached):
|
||||||
|
# Install DAC shim before importing tada — tada's encoder/decoder
|
||||||
|
# import dac.nn.layers.Snake1d which requires the descript-audio-codec
|
||||||
|
# package. The real package pulls in onnx/tensorboard/matplotlib via
|
||||||
|
# descript-audiotools, so we use a lightweight shim instead.
|
||||||
|
from ..utils.dac_shim import install_dac_shim
|
||||||
|
install_dac_shim()
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from huggingface_hub import snapshot_download
|
||||||
|
|
||||||
|
device = self._get_device()
|
||||||
|
self._device = device
|
||||||
|
logger.info(f"Loading HumeAI TADA {model_size} on {device}...")
|
||||||
|
|
||||||
|
# Download codec (encoder + decoder) if not cached
|
||||||
|
logger.info("Downloading TADA codec...")
|
||||||
|
snapshot_download(
|
||||||
|
repo_id=TADA_CODEC_REPO,
|
||||||
|
token=None,
|
||||||
|
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Download model weights if not cached
|
||||||
|
logger.info(f"Downloading TADA {model_size} model...")
|
||||||
|
snapshot_download(
|
||||||
|
repo_id=repo,
|
||||||
|
token=None,
|
||||||
|
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin", "*.model"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# TADA hardcodes "meta-llama/Llama-3.2-1B" as the tokenizer
|
||||||
|
# source in its Aligner and TadaForCausalLM.from_pretrained().
|
||||||
|
# That repo is gated (requires Meta license acceptance).
|
||||||
|
# Download the tokenizer from an ungated mirror and get its
|
||||||
|
# local cache path so we can point TADA at it directly.
|
||||||
|
logger.info("Downloading Llama tokenizer (ungated mirror)...")
|
||||||
|
tokenizer_path = snapshot_download(
|
||||||
|
repo_id="unsloth/Llama-3.2-1B",
|
||||||
|
token=None,
|
||||||
|
allow_patterns=["tokenizer*", "special_tokens*"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Determine dtype — use bf16 on CUDA for ~50% memory savings
|
||||||
|
if device == "cuda" and torch.cuda.is_bf16_supported():
|
||||||
|
model_dtype = torch.bfloat16
|
||||||
|
else:
|
||||||
|
model_dtype = torch.float32
|
||||||
|
|
||||||
|
# Patch the Aligner config class to use the local tokenizer
|
||||||
|
# path instead of the gated "meta-llama/Llama-3.2-1B" default.
|
||||||
|
# This avoids monkey-patching AutoTokenizer.from_pretrained
|
||||||
|
# which corrupts the classmethod descriptor for other engines.
|
||||||
|
from tada.modules.aligner import AlignerConfig
|
||||||
|
AlignerConfig.tokenizer_name = tokenizer_path
|
||||||
|
|
||||||
|
# Load encoder (only needed for voice prompt encoding)
|
||||||
|
from tada.modules.encoder import Encoder
|
||||||
|
logger.info("Loading TADA encoder...")
|
||||||
|
self.encoder = Encoder.from_pretrained(
|
||||||
|
TADA_CODEC_REPO, subfolder="encoder"
|
||||||
|
).to(device)
|
||||||
|
self.encoder.eval()
|
||||||
|
|
||||||
|
# Load the causal LM (includes decoder for wav generation).
|
||||||
|
# TadaForCausalLM.from_pretrained() calls
|
||||||
|
# getattr(config, "tokenizer_name", "meta-llama/Llama-3.2-1B")
|
||||||
|
# which hits the gated repo. Pre-load the config from HF,
|
||||||
|
# inject the local tokenizer path, then pass it in.
|
||||||
|
from tada.modules.tada import TadaForCausalLM, TadaConfig
|
||||||
|
logger.info(f"Loading TADA {model_size} model...")
|
||||||
|
config = TadaConfig.from_pretrained(repo)
|
||||||
|
config.tokenizer_name = tokenizer_path
|
||||||
|
self.model = TadaForCausalLM.from_pretrained(
|
||||||
|
repo, config=config, torch_dtype=model_dtype
|
||||||
|
).to(device)
|
||||||
|
self.model.eval()
|
||||||
|
|
||||||
|
logger.info(f"HumeAI TADA {model_size} loaded successfully on {device}")
|
||||||
|
|
||||||
|
def unload_model(self) -> None:
|
||||||
|
"""Unload model and encoder to free memory."""
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
self.model = None
|
||||||
|
if self.encoder is not None:
|
||||||
|
del self.encoder
|
||||||
|
self.encoder = None
|
||||||
|
|
||||||
|
self._device = None
|
||||||
|
|
||||||
|
import torch
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
|
||||||
|
logger.info("HumeAI TADA unloaded")
|
||||||
|
|
||||||
|
async def create_voice_prompt(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
use_cache: bool = True,
|
||||||
|
) -> Tuple[dict, bool]:
|
||||||
|
"""
|
||||||
|
Create voice prompt from reference audio using TADA's encoder.
|
||||||
|
|
||||||
|
TADA's encoder performs forced alignment between audio and text tokens,
|
||||||
|
producing an EncoderOutput with 1:1 token-audio alignment. If no
|
||||||
|
reference_text is provided, the encoder uses built-in ASR (English only).
|
||||||
|
|
||||||
|
We serialize the EncoderOutput to a dict for caching.
|
||||||
|
"""
|
||||||
|
await self.load_model(self.model_size)
|
||||||
|
|
||||||
|
cache_key = (
|
||||||
|
"tada_" + get_cache_key(audio_path, reference_text)
|
||||||
|
) if use_cache else None
|
||||||
|
|
||||||
|
if cache_key:
|
||||||
|
cached = get_cached_voice_prompt(cache_key)
|
||||||
|
if cached is not None and isinstance(cached, dict):
|
||||||
|
return cached, True
|
||||||
|
|
||||||
|
def _encode_sync():
|
||||||
|
import torch
|
||||||
|
import soundfile as sf
|
||||||
|
|
||||||
|
device = self._device
|
||||||
|
|
||||||
|
# Load audio with soundfile (torchaudio 2.10+ requires torchcodec)
|
||||||
|
audio_np, sr = sf.read(str(audio_path), dtype="float32")
|
||||||
|
audio = torch.from_numpy(audio_np).float()
|
||||||
|
if audio.ndim == 1:
|
||||||
|
audio = audio.unsqueeze(0) # (samples,) -> (1, samples)
|
||||||
|
else:
|
||||||
|
audio = audio.T # (samples, channels) -> (channels, samples)
|
||||||
|
audio = audio.to(device)
|
||||||
|
|
||||||
|
# Encode with forced alignment
|
||||||
|
text_arg = [reference_text] if reference_text else None
|
||||||
|
prompt = self.encoder(
|
||||||
|
audio, text=text_arg, sample_rate=sr
|
||||||
|
)
|
||||||
|
|
||||||
|
# Serialize EncoderOutput to a dict of CPU tensors for caching
|
||||||
|
prompt_dict = {}
|
||||||
|
for field_name in prompt.__dataclass_fields__:
|
||||||
|
val = getattr(prompt, field_name)
|
||||||
|
if isinstance(val, torch.Tensor):
|
||||||
|
prompt_dict[field_name] = val.detach().cpu()
|
||||||
|
elif isinstance(val, list):
|
||||||
|
prompt_dict[field_name] = val
|
||||||
|
elif isinstance(val, (int, float)):
|
||||||
|
prompt_dict[field_name] = val
|
||||||
|
else:
|
||||||
|
prompt_dict[field_name] = val
|
||||||
|
return prompt_dict
|
||||||
|
|
||||||
|
encoded = await asyncio.to_thread(_encode_sync)
|
||||||
|
|
||||||
|
if cache_key:
|
||||||
|
cache_voice_prompt(cache_key, encoded)
|
||||||
|
|
||||||
|
return encoded, False
|
||||||
|
|
||||||
|
async def combine_voice_prompts(
|
||||||
|
self,
|
||||||
|
audio_paths: List[str],
|
||||||
|
reference_texts: List[str],
|
||||||
|
) -> Tuple[np.ndarray, str]:
|
||||||
|
return await _combine_voice_prompts(audio_paths, reference_texts, sample_rate=24000)
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice_prompt: dict,
|
||||||
|
language: str = "en",
|
||||||
|
seed: Optional[int] = None,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> Tuple[np.ndarray, int]:
|
||||||
|
"""
|
||||||
|
Generate audio from text using HumeAI TADA.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to synthesize
|
||||||
|
voice_prompt: Serialized EncoderOutput dict from create_voice_prompt()
|
||||||
|
language: Language code (en, ar, de, es, fr, it, ja, pl, pt, zh)
|
||||||
|
seed: Random seed for reproducibility
|
||||||
|
instruct: Not supported by TADA (ignored)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (audio_array, sample_rate=24000)
|
||||||
|
"""
|
||||||
|
await self.load_model(self.model_size)
|
||||||
|
|
||||||
|
def _generate_sync():
|
||||||
|
import torch
|
||||||
|
from tada.modules.encoder import EncoderOutput
|
||||||
|
|
||||||
|
if seed is not None:
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.manual_seed(seed)
|
||||||
|
|
||||||
|
device = self._device
|
||||||
|
|
||||||
|
# Reconstruct EncoderOutput from the cached dict
|
||||||
|
restored = {}
|
||||||
|
for k, v in voice_prompt.items():
|
||||||
|
if isinstance(v, torch.Tensor):
|
||||||
|
# Move to device and match model dtype for float tensors
|
||||||
|
if v.is_floating_point():
|
||||||
|
model_dtype = next(self.model.parameters()).dtype
|
||||||
|
restored[k] = v.to(device=device, dtype=model_dtype)
|
||||||
|
else:
|
||||||
|
restored[k] = v.to(device=device)
|
||||||
|
else:
|
||||||
|
restored[k] = v
|
||||||
|
|
||||||
|
prompt = EncoderOutput(**restored)
|
||||||
|
|
||||||
|
# For non-English with the 3B-ML model, we could reload the
|
||||||
|
# encoder with the language-specific aligner. However, the
|
||||||
|
# generation itself is language-agnostic — only the encoder's
|
||||||
|
# aligner changes. Since we encode at create_voice_prompt time,
|
||||||
|
# the language is already baked in. For simplicity, we don't
|
||||||
|
# reload the encoder here.
|
||||||
|
|
||||||
|
logger.info(f"[TADA] Generating ({language}), text length: {len(text)}")
|
||||||
|
|
||||||
|
output = self.model.generate(
|
||||||
|
prompt=prompt,
|
||||||
|
text=text,
|
||||||
|
)
|
||||||
|
|
||||||
|
# output.audio is a list of tensors (one per batch item)
|
||||||
|
if output.audio and output.audio[0] is not None:
|
||||||
|
audio_tensor = output.audio[0]
|
||||||
|
audio = audio_tensor.detach().cpu().numpy().squeeze().astype(np.float32)
|
||||||
|
else:
|
||||||
|
logger.warning("[TADA] Generation produced no audio")
|
||||||
|
audio = np.zeros(24000, dtype=np.float32)
|
||||||
|
|
||||||
|
return audio, 24000
|
||||||
|
|
||||||
|
return await asyncio.to_thread(_generate_sync)
|
||||||
@@ -186,6 +186,42 @@ def build_server(cuda=False):
|
|||||||
# needed by LuxTTS for text-to-phoneme conversion
|
# needed by LuxTTS for text-to-phoneme conversion
|
||||||
"--collect-all",
|
"--collect-all",
|
||||||
"piper_phonemize",
|
"piper_phonemize",
|
||||||
|
# HumeAI TADA — speech-language model using Llama + flow matching
|
||||||
|
"--hidden-import",
|
||||||
|
"backend.backends.hume_backend",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules.tada",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules.encoder",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules.decoder",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules.aligner",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.modules.acoustic_spkr_verf",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.nn",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.nn.vibevoice",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.utils",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.utils.gray_code",
|
||||||
|
"--hidden-import",
|
||||||
|
"tada.utils.text",
|
||||||
|
# DAC shim — provides dac.nn.layers.Snake1d without the real
|
||||||
|
# descript-audio-codec package (which pulls onnx/tensorboard via
|
||||||
|
# descript-audiotools). The shim is in backend/utils/dac_shim.py.
|
||||||
|
"--hidden-import",
|
||||||
|
"backend.utils.dac_shim",
|
||||||
|
"--hidden-import",
|
||||||
|
"torchaudio",
|
||||||
|
"--collect-submodules",
|
||||||
|
"tada",
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -66,9 +66,9 @@ class GenerationRequest(BaseModel):
|
|||||||
text: str = Field(..., min_length=1, max_length=50000)
|
text: str = Field(..., min_length=1, max_length=50000)
|
||||||
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$")
|
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$")
|
||||||
seed: Optional[int] = Field(None, ge=0)
|
seed: Optional[int] = Field(None, ge=0)
|
||||||
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B)$")
|
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B|1B|3B)$")
|
||||||
instruct: Optional[str] = Field(None, max_length=500)
|
instruct: Optional[str] = Field(None, max_length=500)
|
||||||
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo)$")
|
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo|tada)$")
|
||||||
max_chunk_chars: int = Field(
|
max_chunk_chars: int = Field(
|
||||||
default=800, ge=100, le=5000, description="Max characters per chunk for long text splitting"
|
default=800, ge=100, le=5000, description="Max characters per chunk for long text splitting"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -33,6 +33,13 @@ s3tokenizer
|
|||||||
spacy-pkuseg
|
spacy-pkuseg
|
||||||
pyloudnorm
|
pyloudnorm
|
||||||
|
|
||||||
|
# HumeAI TADA sub-dependencies (hume-tada itself is installed
|
||||||
|
# --no-deps in the setup script because it pins torch>=2.7,<2.8.
|
||||||
|
# descript-audio-codec is NOT installed — it pulls onnx/tensorboard
|
||||||
|
# via descript-audiotools. A lightweight shim in utils/dac_shim.py
|
||||||
|
# provides the only class TADA uses: Snake1d.)
|
||||||
|
torchaudio
|
||||||
|
|
||||||
# Audio processing
|
# Audio processing
|
||||||
librosa>=0.10.0
|
librosa>=0.10.0
|
||||||
soundfile>=0.12.0
|
soundfile>=0.12.0
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
"""
|
||||||
|
Minimal shim for descript-audio-codec (DAC).
|
||||||
|
|
||||||
|
TADA only imports Snake1d from dac.nn.layers and dac.model.dac.
|
||||||
|
The real DAC package pulls in descript-audiotools which depends on
|
||||||
|
onnx, tensorboard, protobuf, matplotlib, pystoi, etc. — none of
|
||||||
|
which are needed for TADA's runtime use of Snake1d.
|
||||||
|
|
||||||
|
This shim provides the exact Snake1d implementation (MIT-licensed,
|
||||||
|
from https://github.com/descriptinc/descript-audio-codec) so we can
|
||||||
|
avoid the entire audiotools dependency chain.
|
||||||
|
|
||||||
|
If the real DAC package is installed, this module is never used —
|
||||||
|
Python's import system will find the site-packages version first.
|
||||||
|
Install this shim only when descript-audio-codec is NOT installed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
|
||||||
|
|
||||||
|
# ── Snake activation (from dac/nn/layers.py) ────────────────────────
|
||||||
|
|
||||||
|
# NOTE: The original DAC code uses @torch.jit.script here for a 1.4x
|
||||||
|
# speedup. We omit it because TorchScript calls inspect.getsource()
|
||||||
|
# which fails inside a PyInstaller frozen binary (no .py source files).
|
||||||
|
def snake(x: torch.Tensor, alpha: torch.Tensor) -> torch.Tensor:
|
||||||
|
shape = x.shape
|
||||||
|
x = x.reshape(shape[0], shape[1], -1)
|
||||||
|
x = x + (alpha + 1e-9).reciprocal() * torch.sin(alpha * x).pow(2)
|
||||||
|
x = x.reshape(shape)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class Snake1d(nn.Module):
|
||||||
|
def __init__(self, channels: int):
|
||||||
|
super().__init__()
|
||||||
|
self.alpha = nn.Parameter(torch.ones(1, channels, 1))
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
return snake(x, self.alpha)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Register as dac.nn.layers and dac.model.dac ─────────────────────
|
||||||
|
|
||||||
|
def install_dac_shim() -> None:
|
||||||
|
"""Register fake dac package modules in sys.modules.
|
||||||
|
|
||||||
|
Only installs the shim if 'dac' is not already importable
|
||||||
|
(i.e. the real descript-audio-codec is not installed).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
import dac # noqa: F401 — real package exists, do nothing
|
||||||
|
return
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Create the module tree: dac -> dac.nn -> dac.nn.layers
|
||||||
|
# -> dac.model -> dac.model.dac
|
||||||
|
dac_pkg = types.ModuleType("dac")
|
||||||
|
dac_pkg.__path__ = [] # make it a package
|
||||||
|
dac_pkg.__package__ = "dac"
|
||||||
|
|
||||||
|
dac_nn = types.ModuleType("dac.nn")
|
||||||
|
dac_nn.__path__ = []
|
||||||
|
dac_nn.__package__ = "dac.nn"
|
||||||
|
|
||||||
|
dac_nn_layers = types.ModuleType("dac.nn.layers")
|
||||||
|
dac_nn_layers.__package__ = "dac.nn"
|
||||||
|
dac_nn_layers.Snake1d = Snake1d
|
||||||
|
dac_nn_layers.snake = snake
|
||||||
|
|
||||||
|
dac_model = types.ModuleType("dac.model")
|
||||||
|
dac_model.__path__ = []
|
||||||
|
dac_model.__package__ = "dac.model"
|
||||||
|
|
||||||
|
dac_model_dac = types.ModuleType("dac.model.dac")
|
||||||
|
dac_model_dac.__package__ = "dac.model"
|
||||||
|
dac_model_dac.Snake1d = Snake1d
|
||||||
|
|
||||||
|
# Wire up submodules
|
||||||
|
dac_pkg.nn = dac_nn
|
||||||
|
dac_pkg.model = dac_model
|
||||||
|
dac_nn.layers = dac_nn_layers
|
||||||
|
dac_model.dac = dac_model_dac
|
||||||
|
|
||||||
|
# Register in sys.modules
|
||||||
|
sys.modules["dac"] = dac_pkg
|
||||||
|
sys.modules["dac.nn"] = dac_nn
|
||||||
|
sys.modules["dac.nn.layers"] = dac_nn_layers
|
||||||
|
sys.modules["dac.model"] = dac_model
|
||||||
|
sys.modules["dac.model.dac"] = dac_model_dac
|
||||||
@@ -3,8 +3,12 @@ title: "TTS Engines"
|
|||||||
description: "How to add new text-to-speech engines to Voicebox"
|
description: "How to add new text-to-speech engines to Voicebox"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
> **For humans:** This doc is optimized for AI agents to implement new TTS engines autonomously. It's structured as a phased workflow with explicit gates and a checklist so an agent can do the full integration — dependency research, backend, frontend, bundling — and hand you a draft release or prod build to test locally. It's also a useful reference if you're doing it yourself.
|
||||||
|
|
||||||
Adding an engine touches ~10 files across 4 layers. The backend protocol work is straightforward — the real time sink is dependency hell, upstream library bugs, and PyInstaller bundling.
|
Adding an engine touches ~10 files across 4 layers. The backend protocol work is straightforward — the real time sink is dependency hell, upstream library bugs, and PyInstaller bundling.
|
||||||
|
|
||||||
|
**Do not start writing code until you complete Phase 0.** The v0.2.3 release was three patch releases of PyInstaller fixes because dependency research was skipped. Every issue — `inspect.getsource()` failures, missing native data files, metadata lookups, dtype mismatches — was discoverable by reading the model library's source code before integration began.
|
||||||
|
|
||||||
## Architecture Overview
|
## Architecture Overview
|
||||||
|
|
||||||
The backend is split into layers:
|
The backend is split into layers:
|
||||||
@@ -18,6 +22,124 @@ The backend is split into layers:
|
|||||||
|
|
||||||
New engines only need to touch `backends/` and `models.py` on the backend side — the route and service layers use a model config registry that handles dispatch automatically.
|
New engines only need to touch `backends/` and `models.py` on the backend side — the route and service layers use a model config registry that handles dispatch automatically.
|
||||||
|
|
||||||
|
## Phase 0: Dependency Research
|
||||||
|
|
||||||
|
**This phase is mandatory.** Clone the model library and its key dependencies into a temporary directory and inspect them before writing any integration code. The goal is to produce a dependency audit that identifies every PyInstaller-incompatible pattern, every native data file, and every upstream bug you'll need to work around.
|
||||||
|
|
||||||
|
### 0.1 Clone and Inspect the Model Library
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create a throwaway workspace
|
||||||
|
mkdir /tmp/engine-research && cd /tmp/engine-research
|
||||||
|
|
||||||
|
# Clone the model library
|
||||||
|
git clone https://github.com/org/model-library.git
|
||||||
|
cd model-library
|
||||||
|
```
|
||||||
|
|
||||||
|
**Read these files first, in order:**
|
||||||
|
|
||||||
|
1. **`setup.py` / `setup.cfg` / `pyproject.toml`** — Check pinned dependency versions. If the library pins `torch==2.6.0` or `numpy<1.26`, you'll need `--no-deps` installation and manual sub-dependency listing (this is what happened with `chatterbox-tts`).
|
||||||
|
|
||||||
|
2. **`__init__.py` and the main model class** — Trace the import chain. Look for:
|
||||||
|
- `from_pretrained()` — does it call `huggingface_hub` internally? Does it pass `token=True` (which crashes without a stored HF token)?
|
||||||
|
- `from_local()` — does it exist? You may need manual `snapshot_download()` + `from_local()` to bypass download bugs.
|
||||||
|
- Device handling — does it default to CUDA? Does it support MPS? Many libraries crash on MPS with unsupported operators.
|
||||||
|
|
||||||
|
3. **All `import` statements** — Recursively trace what the library imports. You're looking for:
|
||||||
|
- `inspect.getsource()` anywhere in the chain (search all `.py` files)
|
||||||
|
- `typeguard` / `@typechecked` decorators (these call `inspect.getsource()` at import time)
|
||||||
|
- `importlib.metadata.version()` or `pkg_resources.get_distribution()` (need `--copy-metadata`)
|
||||||
|
- `lazy_loader` (needs `--collect-all` to bundle `.pyi` stubs)
|
||||||
|
|
||||||
|
### 0.2 Scan for PyInstaller-Incompatible Patterns
|
||||||
|
|
||||||
|
Run these searches against the cloned library **and** its transitive dependencies:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# inspect.getsource — will crash in frozen binary without --collect-all
|
||||||
|
grep -r "inspect.getsource\|getsource(" .
|
||||||
|
|
||||||
|
# typeguard / @typechecked — calls inspect.getsource at import time
|
||||||
|
grep -r "@typechecked\|from typeguard" .
|
||||||
|
|
||||||
|
# importlib.metadata — needs --copy-metadata
|
||||||
|
grep -r "importlib.metadata\|pkg_resources.get_distribution\|pkg_resources.require" .
|
||||||
|
|
||||||
|
# Data files loaded at runtime — need --collect-all or --collect-data
|
||||||
|
grep -r "Path(__file__).parent\|os.path.dirname(__file__)\|resources_path\|pkg_resources.resource_filename" .
|
||||||
|
|
||||||
|
# Native library paths — may need env var override in frozen builds
|
||||||
|
grep -r "/usr/share\|/usr/lib\|/usr/local\|espeak\|phonemize" .
|
||||||
|
|
||||||
|
# torch.load without map_location — will crash on CPU-only builds
|
||||||
|
grep -r "torch.load(" . | grep -v "map_location"
|
||||||
|
|
||||||
|
# HuggingFace token bugs
|
||||||
|
grep -r 'token=True\|token=os.getenv' .
|
||||||
|
|
||||||
|
# Float64/Float32 assumptions — librosa returns float64, many models assume float32
|
||||||
|
grep -r "torch.from_numpy\|\.double()\|float64" .
|
||||||
|
```
|
||||||
|
|
||||||
|
### 0.3 Install and Trace in a Throwaway Venv
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create isolated venv
|
||||||
|
python -m venv /tmp/engine-venv
|
||||||
|
source /tmp/engine-venv/bin/activate
|
||||||
|
|
||||||
|
# Install the package (try normally first)
|
||||||
|
pip install model-package
|
||||||
|
|
||||||
|
# Check if it conflicts with our stack
|
||||||
|
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
|
||||||
|
# If this fails, you need --no-deps:
|
||||||
|
pip install --no-deps model-package
|
||||||
|
|
||||||
|
# Get the full dependency tree
|
||||||
|
pip show model-package # Check Requires: field
|
||||||
|
pip show -f model-package # List all installed files (look for data files)
|
||||||
|
|
||||||
|
# Check for non-PyPI dependencies
|
||||||
|
pip install model-package 2>&1 | grep -i "no matching distribution"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 0.4 Test Model Loading on CPU
|
||||||
|
|
||||||
|
Before writing any integration code, verify the model works on CPU in a plain Python script:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
# Force CPU to catch map_location bugs early
|
||||||
|
model = ModelClass.from_pretrained("org/model", device="cpu")
|
||||||
|
|
||||||
|
# Test with a float32 audio array (not float64)
|
||||||
|
import numpy as np
|
||||||
|
audio = np.random.randn(16000).astype(np.float32)
|
||||||
|
output = model.generate("Hello world", audio)
|
||||||
|
print(f"Output shape: {output.shape}, dtype: {output.dtype}, sample rate: {model.sample_rate}")
|
||||||
|
```
|
||||||
|
|
||||||
|
If this crashes, you've found a bug you'll need to monkey-patch. Common ones:
|
||||||
|
- `RuntimeError: expected scalar type Float but found Double` → needs float32 cast
|
||||||
|
- `RuntimeError: map_location` → needs `torch.load` patch
|
||||||
|
- `RuntimeError: Unsupported operator aten::...` → needs MPS skip
|
||||||
|
|
||||||
|
### 0.5 Produce a Dependency Audit
|
||||||
|
|
||||||
|
Before proceeding to Phase 1, write down:
|
||||||
|
|
||||||
|
1. **PyPI vs non-PyPI deps** — which packages need `--find-links`, `git+https://`, or `--no-deps`?
|
||||||
|
2. **PyInstaller directives needed** — which packages need `--collect-all`, `--copy-metadata`, `--hidden-import`?
|
||||||
|
3. **Runtime data files** — which packages ship data files (YAML, pretrained weights, phoneme tables, shader libraries) that must be bundled?
|
||||||
|
4. **Native library paths** — which packages look for data at system paths that won't exist in a frozen binary?
|
||||||
|
5. **Monkey-patches needed** — `torch.load` map_location, float64→float32 casts, MPS skip, HF token bypass, etc.
|
||||||
|
6. **Sample rate** — what does the engine output? (24kHz, 44.1kHz, 48kHz)
|
||||||
|
7. **Model download method** — `from_pretrained()` with library-managed download, or manual `snapshot_download()` + `from_local()`?
|
||||||
|
|
||||||
|
This audit becomes your implementation plan for Phases 1, 4, and 5.
|
||||||
|
|
||||||
## Phase 1: Backend Implementation
|
## Phase 1: Backend Implementation
|
||||||
|
|
||||||
### 1.1 Create the Backend File
|
### 1.1 Create the Backend File
|
||||||
@@ -155,54 +277,172 @@ In `app/src/components/ServerSettings/ModelManagement.tsx`:
|
|||||||
|
|
||||||
## Phase 4: Dependencies
|
## Phase 4: Dependencies
|
||||||
|
|
||||||
|
Use the dependency audit from Phase 0 to drive this phase. You should already know what packages are needed, which conflict, and which require special installation.
|
||||||
|
|
||||||
### 4.1 Python Dependencies
|
### 4.1 Python Dependencies
|
||||||
|
|
||||||
Add to `backend/requirements.txt`. Watch for:
|
Add to `backend/requirements.txt`. There are three installation patterns, depending on what Phase 0 revealed:
|
||||||
|
|
||||||
**Pinned dependency conflicts** — If the model package pins old versions, install with `--no-deps`:
|
**Normal PyPI packages:**
|
||||||
|
```
|
||||||
|
some-model-package>=1.0.0
|
||||||
|
```
|
||||||
|
|
||||||
|
**Pinned dependency conflicts (`--no-deps`)** — If the model package pins old versions of torch/numpy/transformers, install with `--no-deps` and list sub-dependencies manually. This is the pattern used for `chatterbox-tts`:
|
||||||
```bash
|
```bash
|
||||||
|
# In justfile / CI setup:
|
||||||
pip install --no-deps chatterbox-tts
|
pip install --no-deps chatterbox-tts
|
||||||
|
|
||||||
|
# In requirements.txt — list each actual sub-dependency:
|
||||||
|
conformer>=0.3.2
|
||||||
|
diffusers>=0.31.0
|
||||||
|
omegaconf>=2.3.0
|
||||||
|
resemble-perth>=0.0.2
|
||||||
|
s3tokenizer>=0.1.6
|
||||||
```
|
```
|
||||||
|
|
||||||
Then list sub-dependencies manually in `requirements.txt`.
|
To identify sub-deps: `pip show chatterbox-tts` → `Requires:` field, then cross-reference against existing `requirements.txt` to avoid duplicates.
|
||||||
|
|
||||||
**Non-PyPI packages:**
|
**Non-PyPI packages** — Some libraries only exist on GitHub or require custom indexes:
|
||||||
```
|
|
||||||
linacodec @ git+https://github.com/user/repo.git
|
|
||||||
```
|
```
|
||||||
|
# Git-only packages (no PyPI release)
|
||||||
|
linacodec @ git+https://github.com/ysharma3501/LinaCodec.git
|
||||||
|
Zipvoice @ git+https://github.com/ysharma3501/LuxTTS.git
|
||||||
|
|
||||||
**Custom package indexes:**
|
# Custom package indexes (C extensions with platform-specific wheels)
|
||||||
```
|
|
||||||
--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
|
--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
|
||||||
|
piper-phonemize>=1.2.0
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4.2 Identifying Hidden Sub-Dependencies
|
### 4.2 Dependency Conflict Resolution
|
||||||
|
|
||||||
1. Install the package normally in a throwaway venv
|
Check for conflicts with the existing stack before adding anything:
|
||||||
2. Run `pip show <package>` to get its `Requires:` list
|
|
||||||
3. Cross-reference against existing requirements.txt
|
|
||||||
4. Test that the engine loads and generates
|
|
||||||
|
|
||||||
## Phase 5: PyInstaller Bundling
|
```bash
|
||||||
|
# Our current stack pins (approximate):
|
||||||
|
# Python 3.12+, torch>=2.10, transformers>=4.57, numpy>=1.26
|
||||||
|
|
||||||
This is where most of the pain lives. Common issues:
|
# Test compatibility
|
||||||
|
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
|
||||||
|
|
||||||
| Issue | Symptom | Fix |
|
# If it fails, check what the package pins:
|
||||||
|-------|---------|-----|
|
pip show model-package | grep Requires
|
||||||
| `inspect.getsource()` at import | "could not get source code" | `--collect-all <package>` |
|
# Look at setup.py/pyproject.toml for version constraints
|
||||||
| Data files (yaml, .pth.tar) | FileNotFoundError at runtime | `--collect-all <package>` |
|
```
|
||||||
| Native data paths (espeak-ng) | Library looks at `/usr/share/...` | Set env var in frozen builds |
|
|
||||||
| `importlib.metadata` lookups | "No package metadata found" | `--copy-metadata <package>` |
|
|
||||||
| Dynamic imports | ModuleNotFoundError | `--hidden-import <module>` |
|
|
||||||
|
|
||||||
### Testing Frozen Builds
|
**Known incompatible patterns in the wild:**
|
||||||
|
- `torch==2.6.0` — many older packages pin this
|
||||||
|
- `numpy<1.26` — conflicts with Python 3.12+
|
||||||
|
- `transformers==4.46.3` — many packages pin old transformers
|
||||||
|
- `onnxruntime` pinned versions — often conflict with torch
|
||||||
|
|
||||||
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary.
|
### 4.3 Update Installation Scripts
|
||||||
|
|
||||||
|
Dependencies must be added in multiple places:
|
||||||
|
|
||||||
|
| File | What to add |
|
||||||
|
|------|------------|
|
||||||
|
| `backend/requirements.txt` | Package and version constraint |
|
||||||
|
| `justfile` | `--no-deps` install line if needed (in `setup-python` and `setup-python-release` targets) |
|
||||||
|
| `.github/workflows/release.yml` | Same `--no-deps` line in CI build steps |
|
||||||
|
| `Dockerfile` | Same install commands for Docker builds |
|
||||||
|
|
||||||
|
## Phase 5: PyInstaller Bundling (`build_binary.py`)
|
||||||
|
|
||||||
|
This is where most of the pain lives. **The v0.2.3 release was entirely dedicated to fixing bundling issues** — every new engine that shipped in v0.2.1 (LuxTTS, Chatterbox, Chatterbox Turbo) worked in dev but failed in production builds. Don't skip this phase.
|
||||||
|
|
||||||
|
### 5.1 Register Your Engine in `build_binary.py`
|
||||||
|
|
||||||
|
Every new engine needs entries in `backend/build_binary.py`. This file drives PyInstaller and is the single most common source of "works in dev, breaks in prod" bugs. You need to decide which PyInstaller directives your engine's dependencies require:
|
||||||
|
|
||||||
|
| Directive | What It Does | When You Need It |
|
||||||
|
|-----------|-------------|-----------------|
|
||||||
|
| `--hidden-import <module>` | Includes a module PyInstaller can't detect via static analysis | Dynamic imports, lazy imports, plugin architectures |
|
||||||
|
| `--collect-all <package>` | Bundles source `.py` files, data files, AND native libraries | Packages that call `inspect.getsource()` at import time (e.g. `inflect` via `typeguard`'s `@typechecked`), or that ship pretrained model files (e.g. `perth` ships `.pth.tar` + `hparams.yaml`) |
|
||||||
|
| `--collect-data <package>` | Bundles only data files (not source or native libs) | Packages with YAML configs, vocab files, etc. |
|
||||||
|
| `--collect-submodules <package>` | Bundles all submodules | Packages with deep module trees that PyInstaller misses |
|
||||||
|
| `--copy-metadata <package>` | Copies `importlib.metadata` info | Packages that call `importlib.metadata.version()` or `pkg_resources.get_distribution()` at runtime. Already required for: `requests`, `transformers`, `huggingface-hub`, `tokenizers`, `safetensors`, `tqdm` |
|
||||||
|
|
||||||
|
**Example: adding hidden imports and collect-all for a new engine:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# In build_binary.py, inside the args list:
|
||||||
|
"--hidden-import",
|
||||||
|
"backend.backends.your_engine_backend",
|
||||||
|
"--hidden-import",
|
||||||
|
"your_engine_package",
|
||||||
|
"--hidden-import",
|
||||||
|
"your_engine_package.inference",
|
||||||
|
"--collect-all",
|
||||||
|
"some_dependency_that_uses_inspect_getsource",
|
||||||
|
"--copy-metadata",
|
||||||
|
"some_dependency_that_checks_its_own_version",
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Lessons from v0.2.3 — Real Failures and Their Fixes
|
||||||
|
|
||||||
|
These are actual production failures from shipping new engines. Every one of these passed `python -m uvicorn` in dev:
|
||||||
|
|
||||||
|
| Engine | Failure | Root Cause | Fix |
|
||||||
|
|--------|---------|-----------|-----|
|
||||||
|
| LuxTTS | `"could not get source code"` on import | `inflect` uses `typeguard`'s `@typechecked` which calls `inspect.getsource()` — needs `.py` source files, not just bytecode | `--collect-all inflect` |
|
||||||
|
| LuxTTS | `espeak-ng-data` not found | `piper_phonemize` C library looks for data at `/usr/share/espeak-ng-data/` which doesn't exist in the bundle | `--collect-all piper_phonemize` + set `ESPEAK_DATA_PATH` env var at runtime (see 5.3) |
|
||||||
|
| LuxTTS | `inspect.getsource` error in Vocos codec | `linacodec` and `zipvoice` use source introspection | `--collect-all linacodec` + `--collect-all zipvoice` |
|
||||||
|
| Chatterbox | `FileNotFoundError` for watermark model | `perth` ships pretrained model files (`hparams.yaml`, `.pth.tar`) that PyInstaller doesn't bundle by default | `--collect-all perth` |
|
||||||
|
| All engines | `importlib.metadata` failures | Frozen binary doesn't include package metadata for `huggingface-hub`, `transformers`, etc. | `--copy-metadata` for each affected package |
|
||||||
|
| All engines | Download progress bars stuck at 0% | `huggingface_hub` silently disables tqdm progress bars based on logger level in frozen builds — our progress tracker never receives byte updates | Force-enable tqdm's internal counter in `HFProgressTracker` |
|
||||||
|
| All engines | `NameError: name 'obj' is not defined` on macOS | Python 3.12.0 has a [CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects | Upgrade to Python 3.12.13+ |
|
||||||
|
| All engines | `resource_tracker` subprocess crash | `multiprocessing` in frozen binaries needs `freeze_support()` called before anything else | Added to `server.py` entry point |
|
||||||
|
|
||||||
|
### 5.3 Runtime Frozen-Build Handling (`server.py`)
|
||||||
|
|
||||||
|
Some fixes can't live in `build_binary.py` — they need runtime detection. The entry point `backend/server.py` handles these before any heavy imports:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 1. freeze_support() — MUST be called before any multiprocessing use
|
||||||
|
import multiprocessing
|
||||||
|
multiprocessing.freeze_support()
|
||||||
|
|
||||||
|
# 2. Native data paths — redirect C libraries to bundled data
|
||||||
|
if getattr(sys, 'frozen', False):
|
||||||
|
_meipass = getattr(sys, '_MEIPASS', os.path.dirname(sys.executable))
|
||||||
|
_espeak_data = os.path.join(_meipass, 'piper_phonemize', 'espeak-ng-data')
|
||||||
|
if os.path.isdir(_espeak_data):
|
||||||
|
os.environ.setdefault('ESPEAK_DATA_PATH', _espeak_data)
|
||||||
|
|
||||||
|
# 3. stdout/stderr safety — PyInstaller --noconsole on Windows sets these to None
|
||||||
|
if not _is_writable(sys.stdout):
|
||||||
|
sys.stdout = open(os.devnull, 'w')
|
||||||
|
```
|
||||||
|
|
||||||
|
If your engine's dependencies include native libraries that look for data at system paths (like espeak-ng does), you'll need to add a similar `os.environ.setdefault()` block here.
|
||||||
|
|
||||||
|
### 5.4 CUDA vs CPU Build Branching
|
||||||
|
|
||||||
|
`build_binary.py` produces two different binaries:
|
||||||
|
|
||||||
|
- **`voicebox-server`** (CPU) — excludes all `nvidia.*` packages to avoid bundling ~3 GB of CUDA DLLs
|
||||||
|
- **`voicebox-server-cuda`** — includes `torch.cuda` and `torch.backends.cudnn`
|
||||||
|
|
||||||
|
On Windows, if the build environment has CUDA torch installed but you're building the CPU binary, the script temporarily swaps to CPU-only torch and restores CUDA torch afterward. This prevents PyInstaller from accidentally bundling CUDA libraries into the CPU build.
|
||||||
|
|
||||||
|
New engine imports go in the **common section** (not the CUDA or MLX conditional blocks) unless your engine has platform-specific dependencies.
|
||||||
|
|
||||||
|
### 5.5 MLX Conditional Inclusion
|
||||||
|
|
||||||
|
Apple Silicon builds conditionally include MLX hidden imports and `--collect-all mlx` / `--collect-all mlx_audio`. If your engine has an MLX-specific backend variant, add its imports inside the `if is_apple_silicon() and not cuda:` block.
|
||||||
|
|
||||||
|
### 5.6 Testing Frozen Builds
|
||||||
|
|
||||||
|
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary. The v0.2.3 release required **three patch releases** (v0.2.1 → v0.2.2 → v0.2.3) to get all engines working in production.
|
||||||
|
|
||||||
1. Build: `just build`
|
1. Build: `just build`
|
||||||
2. Run and try download + load + generate
|
2. Launch the binary directly (not via `python -m`)
|
||||||
3. Check stderr for the actual error
|
3. Test the **full chain**: download → load → generate → progress tracking
|
||||||
4. Fix, rebuild, repeat
|
4. Check stderr for the actual error (logs go to stderr for Tauri sidecar capture)
|
||||||
|
5. Fix, rebuild, repeat
|
||||||
|
|
||||||
|
**Common gotcha:** testing only generation with a pre-cached model from your dev install. Always test with a clean model cache to verify downloads work too.
|
||||||
|
|
||||||
## Phase 6: Common Upstream Workarounds
|
## Phase 6: Common Upstream Workarounds
|
||||||
|
|
||||||
@@ -250,8 +490,80 @@ Based on the current model landscape, these are candidates for future integratio
|
|||||||
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
|
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
|
||||||
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
|
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
|
||||||
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
|
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
|
||||||
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Needs vetting |
|
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Shipped |
|
||||||
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
|
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
|
||||||
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
|
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
|
||||||
|
|
||||||
The multi-engine architecture is now in place, making new model integration straightforward (~1 day for a well-documented model with a PyPI package).
|
## Implementation Checklist
|
||||||
|
|
||||||
|
Use this as a gate between phases. Do not proceed to the next phase until every item in the current phase is checked.
|
||||||
|
|
||||||
|
### Phase 0: Dependency Research
|
||||||
|
- [ ] Cloned model library source into a temp directory
|
||||||
|
- [ ] Read `setup.py` / `pyproject.toml` — noted pinned dependency versions
|
||||||
|
- [ ] Traced all imports from the model class through to leaf dependencies
|
||||||
|
- [ ] Searched for `inspect.getsource`, `@typechecked`, `typeguard` in the full dependency tree
|
||||||
|
- [ ] Searched for `importlib.metadata`, `pkg_resources.get_distribution` in the dependency tree
|
||||||
|
- [ ] Searched for `Path(__file__).parent`, `os.path.dirname(__file__)`, hardcoded system paths
|
||||||
|
- [ ] Searched for `torch.load` calls missing `map_location`
|
||||||
|
- [ ] Searched for `torch.from_numpy` without `.float()` cast
|
||||||
|
- [ ] Searched for `token=True` or `token=os.getenv("HF_TOKEN")` in HuggingFace calls
|
||||||
|
- [ ] Tested model loading and generation on CPU in a throwaway venv
|
||||||
|
- [ ] Tested with a clean HuggingFace cache (no pre-downloaded models)
|
||||||
|
- [ ] Produced a written dependency audit documenting all findings
|
||||||
|
|
||||||
|
### Phase 1: Backend Implementation
|
||||||
|
- [ ] Created `backend/backends/<engine>_backend.py` implementing `TTSBackend` protocol
|
||||||
|
- [ ] Chose voice prompt pattern (pre-computed tensors vs deferred file paths)
|
||||||
|
- [ ] Implemented all monkey-patches identified in Phase 0
|
||||||
|
- [ ] Used `get_torch_device()` from `backends/base.py` for device selection
|
||||||
|
- [ ] Used `model_load_progress()` from `backends/base.py` for download/load tracking
|
||||||
|
- [ ] Tested: model downloads correctly
|
||||||
|
- [ ] Tested: model loads on CPU
|
||||||
|
- [ ] Tested: generation produces valid audio
|
||||||
|
- [ ] Tested: voice cloning from reference audio works
|
||||||
|
- [ ] Registered `ModelConfig` in `backends/__init__.py`
|
||||||
|
- [ ] Added to `TTS_ENGINES` dict
|
||||||
|
- [ ] Added factory branch in `get_tts_backend_for_engine()`
|
||||||
|
- [ ] Updated engine regex in `backend/models.py`
|
||||||
|
|
||||||
|
### Phase 2–3: Route, Service, and Frontend
|
||||||
|
- [ ] Confirmed zero changes needed in routes/services (or documented why custom behavior is needed)
|
||||||
|
- [ ] Added engine to TypeScript union type in `app/src/lib/api/types.ts`
|
||||||
|
- [ ] Added language map entry in `app/src/lib/constants/languages.ts`
|
||||||
|
- [ ] Added to `ENGINE_OPTIONS` and `ENGINE_DESCRIPTIONS` in `EngineModelSelector.tsx`
|
||||||
|
- [ ] Added to Zod schema and model-name mapping in `useGenerationForm.ts`
|
||||||
|
- [ ] Added description in `ModelManagement.tsx`
|
||||||
|
|
||||||
|
### Phase 4: Dependencies
|
||||||
|
- [ ] Added packages to `backend/requirements.txt`
|
||||||
|
- [ ] If `--no-deps` needed: listed sub-dependencies explicitly
|
||||||
|
- [ ] If git-only packages: added `@ git+https://...` entries
|
||||||
|
- [ ] If custom index needed: added `--find-links` line
|
||||||
|
- [ ] Updated `justfile` setup targets
|
||||||
|
- [ ] Updated `.github/workflows/release.yml` build steps
|
||||||
|
- [ ] Updated `Dockerfile` if applicable
|
||||||
|
- [ ] Verified `pip install` succeeds in a clean venv with existing requirements
|
||||||
|
|
||||||
|
### Phase 5: PyInstaller Bundling
|
||||||
|
- [ ] Added `--hidden-import` entries in `build_binary.py` for:
|
||||||
|
- [ ] `backend.backends.<engine>_backend`
|
||||||
|
- [ ] The model package and its key submodules
|
||||||
|
- [ ] Added `--collect-all` for any packages that:
|
||||||
|
- [ ] Use `inspect.getsource()` / `@typechecked`
|
||||||
|
- [ ] Ship pretrained model data files (`.pth.tar`, `.yaml`, etc.)
|
||||||
|
- [ ] Ship native data files (phoneme tables, shader libraries, etc.)
|
||||||
|
- [ ] Added `--copy-metadata` for any packages that use `importlib.metadata`
|
||||||
|
- [ ] If engine has native data paths: added `os.environ.setdefault()` in `server.py`
|
||||||
|
- [ ] Built frozen binary with `just build`
|
||||||
|
- [ ] Tested in frozen binary with **clean model cache** (not pre-cached from dev):
|
||||||
|
- [ ] Model download works with real-time progress
|
||||||
|
- [ ] Model loading works
|
||||||
|
- [ ] Generation produces valid audio
|
||||||
|
- [ ] No errors in stderr logs
|
||||||
|
|
||||||
|
### Phase 6: Final Verification
|
||||||
|
- [ ] Engine works in dev mode (`just dev`)
|
||||||
|
- [ ] Engine works in frozen binary (`just build` → run binary directly)
|
||||||
|
- [ ] Tested on target platform (macOS for MLX, Windows/Linux for CUDA)
|
||||||
|
- [ ] No regressions in existing engines
|
||||||
|
|||||||
@@ -3,12 +3,12 @@ title: "Voicebox Documentation"
|
|||||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||||
---
|
---
|
||||||
|
|
||||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- **Complete privacy** -- models and voice data stay on your machine
|
- **Complete privacy** -- models and voice data stay on your machine
|
||||||
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
|
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
|
||||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
||||||
|
|||||||
@@ -5,10 +5,10 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
|
|||||||
|
|
||||||
## What is Voicebox?
|
## What is Voicebox?
|
||||||
|
|
||||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||||
|
|
||||||
- **Complete privacy** -- models and voice data stay on your machine
|
- **Complete privacy** -- models and voice data stay on your machine
|
||||||
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
|
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
|
||||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
||||||
@@ -20,7 +20,7 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
|
|||||||
|
|
||||||
## TTS Engines
|
## TTS Engines
|
||||||
|
|
||||||
Four engines with different strengths, switchable per-generation:
|
Five engines with different strengths, switchable per-generation:
|
||||||
|
|
||||||
| Engine | Languages | Strengths |
|
| Engine | Languages | Strengths |
|
||||||
|--------|-----------|-----------|
|
|--------|-----------|-----------|
|
||||||
@@ -28,6 +28,7 @@ Four engines with different strengths, switchable per-generation:
|
|||||||
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||||
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
|
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
|
||||||
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||||
|
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
|
||||||
|
|
||||||
## GPU Support
|
## GPU Support
|
||||||
|
|
||||||
@@ -56,7 +57,7 @@ Four engines with different strengths, switchable per-generation:
|
|||||||
| Frontend | React, TypeScript, Tailwind CSS |
|
| Frontend | React, TypeScript, Tailwind CSS |
|
||||||
| State | Zustand, React Query |
|
| State | Zustand, React Query |
|
||||||
| Backend | FastAPI (Python) |
|
| Backend | FastAPI (Python) |
|
||||||
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
|
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
|
||||||
| Effects | Pedalboard (Spotify) |
|
| Effects | Pedalboard (Spotify) |
|
||||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||||
|
|||||||
@@ -36,6 +36,10 @@
|
|||||||
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
|
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
|
||||||
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
|
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
|
||||||
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
|
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
|
||||||
|
│ │ │ ┌──────────┐ │ │ │
|
||||||
|
│ │ │ │ TADA │ │ │ │
|
||||||
|
│ │ │ │(1B / 3B) │ │ │ │
|
||||||
|
│ │ │ └──────────┘ │ │ │
|
||||||
│ │ └─────────────────────────────────────────┘ │ │
|
│ │ └─────────────────────────────────────────┘ │ │
|
||||||
│ │ ┌───────────┐ ┌─────────┐ │ │
|
│ │ ┌───────────┐ ┌─────────┐ │ │
|
||||||
│ │ │ STTBackend│ │ Profiles│ │ │
|
│ │ │ STTBackend│ │ Profiles│ │ │
|
||||||
@@ -59,6 +63,7 @@
|
|||||||
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
|
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
|
||||||
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
|
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
|
||||||
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
|
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
|
||||||
|
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
|
||||||
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
|
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
|
||||||
| API types | `backend/models.py` | Pydantic request/response models |
|
| API types | `backend/models.py` | Pydantic request/response models |
|
||||||
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
|
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
|
||||||
@@ -78,7 +83,7 @@
|
|||||||
```
|
```
|
||||||
POST /generate
|
POST /generate
|
||||||
1. Look up voice profile from DB
|
1. Look up voice profile from DB
|
||||||
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo)
|
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada)
|
||||||
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
|
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
|
||||||
4. Check model cache → if missing, trigger background download, return HTTP 202
|
4. Check model cache → if missing, trigger background download, return HTTP 202
|
||||||
5. Load model (lazy): tts_backend.load_model(model_size)
|
5. Load model (lazy): tts_backend.load_model(model_size)
|
||||||
@@ -104,7 +109,8 @@ POST /generate
|
|||||||
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
|
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
|
||||||
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
|
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
|
||||||
- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
|
- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
|
||||||
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
|
- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296)
|
||||||
|
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo, TADA 1B, TADA 3B)
|
||||||
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
|
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
|
||||||
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
|
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
|
||||||
|
|
||||||
@@ -136,6 +142,8 @@ POST /generate
|
|||||||
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
|
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
|
||||||
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
|
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
|
||||||
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
|
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
|
||||||
|
| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
|
||||||
|
| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
|
||||||
|
|
||||||
### Multi-Engine Architecture (Shipped)
|
### Multi-Engine Architecture (Shipped)
|
||||||
|
|
||||||
@@ -143,7 +151,7 @@ The singleton TTS backend blocker described in the previous version of this doc
|
|||||||
|
|
||||||
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
|
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
|
||||||
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
|
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
|
||||||
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'`
|
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'`
|
||||||
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
|
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
|
||||||
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
|
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
|
||||||
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
|
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
|
||||||
@@ -337,7 +345,7 @@ Notable requests:
|
|||||||
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
|
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
|
||||||
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
|
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
|
||||||
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
|
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
|
||||||
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output |
|
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | **Shipped** | PR #296, MIT, 700s+ coherent |
|
||||||
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
|
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
|
||||||
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
|
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
|
||||||
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
|
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
|
||||||
@@ -475,7 +483,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
|
|||||||
| `/history/{id}/export` | GET | Export generation ZIP |
|
| `/history/{id}/export` | GET | Export generation ZIP |
|
||||||
| `/history/{id}/export-audio` | GET | Export audio only |
|
| `/history/{id}/export-audio` | GET | Export audio only |
|
||||||
| `/transcribe` | POST | Transcribe audio (Whisper) |
|
| `/transcribe` | POST | Transcribe audio (Whisper) |
|
||||||
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) |
|
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
|
||||||
| `/models/download` | POST | Trigger model download |
|
| `/models/download` | POST | Trigger model download |
|
||||||
| `/models/download/cancel` | POST | Cancel/dismiss download |
|
| `/models/download/cancel` | POST | Cancel/dismiss download |
|
||||||
| `/models/{name}` | DELETE | Delete downloaded model |
|
| `/models/{name}` | DELETE | Delete downloaded model |
|
||||||
|
|||||||
@@ -46,6 +46,8 @@ setup-python:
|
|||||||
{{ pip }} install -r {{ backend_dir }}/requirements.txt
|
{{ pip }} install -r {{ backend_dir }}/requirements.txt
|
||||||
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
|
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
|
||||||
{{ pip }} install --no-deps chatterbox-tts
|
{{ pip }} install --no-deps chatterbox-tts
|
||||||
|
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
|
||||||
|
{{ pip }} install --no-deps hume-tada
|
||||||
# Apple Silicon: install MLX backend
|
# Apple Silicon: install MLX backend
|
||||||
if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then
|
if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then
|
||||||
echo "Detected Apple Silicon — installing MLX dependencies..."
|
echo "Detected Apple Silicon — installing MLX dependencies..."
|
||||||
@@ -74,6 +76,7 @@ setup-python:
|
|||||||
}
|
}
|
||||||
& "{{ pip }}" install -r {{ backend_dir }}/requirements.txt
|
& "{{ pip }}" install -r {{ backend_dir }}/requirements.txt
|
||||||
& "{{ pip }}" install --no-deps chatterbox-tts
|
& "{{ pip }}" install --no-deps chatterbox-tts
|
||||||
|
& "{{ pip }}" install --no-deps hume-tada
|
||||||
& "{{ pip }}" install git+https://github.com/QwenLM/Qwen3-TTS.git
|
& "{{ pip }}" install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||||
& "{{ pip }}" install pyinstaller ruff pytest pytest-asyncio -q
|
& "{{ pip }}" install pyinstaller ruff pytest pytest-asyncio -q
|
||||||
Write-Host "Python environment ready."
|
Write-Host "Python environment ready."
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@voicebox/landing",
|
"name": "@voicebox/landing",
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"description": "Landing page for voicebox.sh",
|
"description": "Landing page for voicebox.sh",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "bun --bun next dev --turbo",
|
"dev": "bun --bun next dev --turbo",
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "voicebox",
|
"name": "voicebox",
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"workspaces": [
|
"workspaces": [
|
||||||
"app",
|
"app",
|
||||||
|
|||||||
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "@voicebox/tauri",
|
"name": "@voicebox/tauri",
|
||||||
"private": true,
|
"private": true,
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "voicebox"
|
name = "voicebox"
|
||||||
version = "0.2.3"
|
version = "0.3.0"
|
||||||
description = "A production-quality desktop app for Qwen3-TTS voice cloning and generation"
|
description = "A production-quality desktop app for Qwen3-TTS voice cloning and generation"
|
||||||
authors = ["you"]
|
authors = ["you"]
|
||||||
license = ""
|
license = ""
|
||||||
|
|||||||
@@ -23,15 +23,15 @@ fn main() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
let project_root = env!("CARGO_MANIFEST_DIR");
|
||||||
|
let gen_dir = format!("{}/gen", project_root);
|
||||||
|
std::fs::create_dir_all(&gen_dir).expect("Failed to create gen directory");
|
||||||
|
|
||||||
// Compile macOS Liquid Glass icon
|
// Compile macOS Liquid Glass icon
|
||||||
#[cfg(target_os = "macos")]
|
#[cfg(target_os = "macos")]
|
||||||
{
|
{
|
||||||
let project_root = env!("CARGO_MANIFEST_DIR");
|
|
||||||
// voicebox.icon is in tauri/assets/voicebox.icon (one level up from src-tauri)
|
// voicebox.icon is in tauri/assets/voicebox.icon (one level up from src-tauri)
|
||||||
let icon_source = format!("{}/../assets/voicebox.icon", project_root);
|
let icon_source = format!("{}/../assets/voicebox.icon", project_root);
|
||||||
let gen_dir = format!("{}/gen", project_root);
|
|
||||||
|
|
||||||
std::fs::create_dir_all(&gen_dir).expect("Failed to create gen directory");
|
|
||||||
|
|
||||||
if std::path::Path::new(&icon_source).exists() {
|
if std::path::Path::new(&icon_source).exists() {
|
||||||
println!("cargo:rerun-if-changed={}", icon_source);
|
println!("cargo:rerun-if-changed={}", icon_source);
|
||||||
@@ -76,6 +76,71 @@ fn main() {
|
|||||||
panic!("Icon compilation failed");
|
panic!("Icon compilation failed");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Generate voicebox.icns from the source PNG via sips + iconutil
|
||||||
|
let icns_path = format!("{}/voicebox.icns", gen_dir);
|
||||||
|
if !std::path::Path::new(&icns_path).exists() {
|
||||||
|
let source_png = format!("{}/Assets/Voicebox.png", icon_source);
|
||||||
|
if std::path::Path::new(&source_png).exists() {
|
||||||
|
let iconset_dir = format!("{}/voicebox.iconset", gen_dir);
|
||||||
|
std::fs::create_dir_all(&iconset_dir).ok();
|
||||||
|
|
||||||
|
let sizes: &[(u32, &str)] = &[
|
||||||
|
(16, "icon_16x16.png"),
|
||||||
|
(32, "[email protected]"),
|
||||||
|
(32, "icon_32x32.png"),
|
||||||
|
(64, "[email protected]"),
|
||||||
|
(128, "icon_128x128.png"),
|
||||||
|
(256, "[email protected]"),
|
||||||
|
(256, "icon_256x256.png"),
|
||||||
|
(512, "[email protected]"),
|
||||||
|
(512, "icon_512x512.png"),
|
||||||
|
(1024, "[email protected]"),
|
||||||
|
];
|
||||||
|
|
||||||
|
for (size, name) in sizes {
|
||||||
|
let dest = format!("{}/{}", iconset_dir, name);
|
||||||
|
let status = Command::new("sips")
|
||||||
|
.args([
|
||||||
|
"-z",
|
||||||
|
&size.to_string(),
|
||||||
|
&size.to_string(),
|
||||||
|
&source_png,
|
||||||
|
"--out",
|
||||||
|
&dest,
|
||||||
|
])
|
||||||
|
.output();
|
||||||
|
if let Ok(out) = status {
|
||||||
|
if !out.status.success() {
|
||||||
|
eprintln!(
|
||||||
|
"sips failed for {}: {}",
|
||||||
|
name,
|
||||||
|
String::from_utf8_lossy(&out.stderr)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let iconutil_output = Command::new("iconutil")
|
||||||
|
.args(["-c", "icns", "-o", &icns_path, &iconset_dir])
|
||||||
|
.output();
|
||||||
|
|
||||||
|
match iconutil_output {
|
||||||
|
Ok(out) if out.status.success() => {
|
||||||
|
println!("Generated voicebox.icns");
|
||||||
|
}
|
||||||
|
Ok(out) => {
|
||||||
|
eprintln!("iconutil failed: {}", String::from_utf8_lossy(&out.stderr));
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("Failed to run iconutil: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clean up iconset directory
|
||||||
|
std::fs::remove_dir_all(&iconset_dir).ok();
|
||||||
|
}
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
println!(
|
println!(
|
||||||
"cargo:warning=Icon source not found at {}, skipping icon compilation",
|
"cargo:warning=Icon source not found at {}, skipping icon compilation",
|
||||||
@@ -84,5 +149,18 @@ fn main() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Ensure all resource files exist so Tauri's bundler doesn't fail.
|
||||||
|
// On non-macOS these are always stubs. On macOS, actool may not produce
|
||||||
|
// Assets.car if the Xcode version doesn't support the .icon format.
|
||||||
|
{
|
||||||
|
let required = ["Assets.car", "voicebox.icns", "partial.plist"];
|
||||||
|
for name in required {
|
||||||
|
let path = format!("{}/{}", gen_dir, name);
|
||||||
|
if !std::path::Path::new(&path).exists() {
|
||||||
|
std::fs::write(&path, b"").ok();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
tauri_build::build()
|
tauri_build::build()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"$schema": "https://schema.tauri.app/config/2",
|
"$schema": "https://schema.tauri.app/config/2",
|
||||||
"productName": "Voicebox",
|
"productName": "Voicebox",
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"identifier": "sh.voicebox.app",
|
"identifier": "sh.voicebox.app",
|
||||||
"build": {
|
"build": {
|
||||||
"beforeDevCommand": "bun run dev",
|
"beforeDevCommand": "bun run dev",
|
||||||
|
|||||||
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "@voicebox/web",
|
"name": "@voicebox/web",
|
||||||
"private": true,
|
"private": true,
|
||||||
"version": "0.2.3",
|
"version": "0.3.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
|
|||||||
Reference in New Issue
Block a user