docs: add TADA learnings to TTS engine guide and CUDA libs addon plan

Enrich tts-engines.mdx with patterns discovered during TADA integration: - Phase 0.2: new greps for @torch.jit.script, torchaudio.load, gated repos - Phase 3.4: model naming inconsistency warning - Phase 5.2: TADA shim failure added to lessons table - Phase 6: four new workaround sections (gated repos, torchcodec, torch.jit.script, toxic dependency shim pattern) - Checklist: four new items matching the new scan patterns - Remove TADA from upcoming engines (now shipped) Add CUDA_LIBS_ADDON.md exploring --onedir split to avoid 2.4GB redownloads on every version bump.
2026-09-15 04:40:40 -07:00 · 2026-03-17 03:53:40 -07:00
parent e789c937ad
commit 2c1ee94891
2 changed files with 273 additions and 1 deletions
@@ -80,6 +80,15 @@ grep -r 'token=True\|token=os.getenv' .

 # Float64/Float32 assumptions — librosa returns float64, many models assume float32
 grep -r "torch.from_numpy\|\.double()\|float64" .
+
+# @torch.jit.script — calls inspect.getsource(), crashes in frozen builds
+grep -r "@torch.jit.script\|torch.jit.script" .
+
+# torchaudio.load — requires torchcodec in torchaudio 2.10+, use soundfile.read() instead
+grep -r "torchaudio.load\|torchaudio.save" .
+
+# Gated HuggingFace repos — models that hardcode gated repos as tokenizer/config sources
+grep -r "from_pretrained\|tokenizer_name\|AutoTokenizer" . | grep -i "llama\|meta-llama\|gated"
 ```

 ### 0.3 Install and Trace in a Throwaway Venv
@@ -270,6 +279,8 @@ In `app/src/lib/hooks/useGenerationForm.ts`:
 - Add engine-to-model-name mapping
 - Update payload construction for engine-specific fields

+**Watch out for model naming inconsistencies.** The HuggingFace repo name, the model size label, and the API model name don't always follow predictable patterns. For example, TADA's 3B model is named `tada-3b-ml` (not `tada-3b`), because it's a multilingual variant. Always check the actual repo names and build the frontend model name mapping from those, not from assumptions like `{engine}-{size}`.
+
 ### 3.5 Model Management

 In `app/src/components/ServerSettings/ModelManagement.tsx`:
@@ -391,6 +402,7 @@ These are actual production failures from shipping new engines. Every one of the
 | Chatterbox | `FileNotFoundError` for watermark model | `perth` ships pretrained model files (`hparams.yaml`, `.pth.tar`) that PyInstaller doesn't bundle by default | `--collect-all perth` |
 | All engines | `importlib.metadata` failures | Frozen binary doesn't include package metadata for `huggingface-hub`, `transformers`, etc. | `--copy-metadata` for each affected package |
 | All engines | Download progress bars stuck at 0% | `huggingface_hub` silently disables tqdm progress bars based on logger level in frozen builds — our progress tracker never receives byte updates | Force-enable tqdm's internal counter in `HFProgressTracker` |
+| TADA | `inspect.getsource` error in DAC's `Snake1d` | `@torch.jit.script` calls `inspect.getsource()` which fails without `.py` source files | Wrote a lightweight shim (`dac_shim.py`) reimplementing `Snake1d` without `@torch.jit.script`, registered fake `dac.*` modules in `sys.modules` |
 | All engines | `NameError: name 'obj' is not defined` on macOS | Python 3.12.0 has a [CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects | Upgrade to Python 3.12.13+ |
 | All engines | `resource_tracker` subprocess crash | `multiprocessing` in frozen binaries needs `freeze_support()` called before anything else | Added to `server.py` entry point |

@@ -480,6 +492,90 @@ def _get_device(self):
    return "cpu"  # Skip MPS
 ```

+### Gated HuggingFace repos as hardcoded config sources
+
+Some models hardcode a gated HuggingFace repo as their tokenizer or config source (e.g., TADA hardcodes `"meta-llama/Llama-3.2-1B"` in both its `AlignerConfig` and `TadaConfig`). This silently fails without HF authentication.
+
+**Fix:** Download from an ungated mirror and patch the config objects directly:
+
+```python
+# Download tokenizer from ungated mirror
+UNGATED_TOKENIZER = "unsloth/Llama-3.2-1B"
+tokenizer_path = snapshot_download(UNGATED_TOKENIZER, token=None)
+
+# Patch the model config to use the local path instead of the gated repo
+config = ModelConfig.from_pretrained(model_path)
+config.tokenizer_name = tokenizer_path
+model = ModelClass.from_pretrained(model_path, config=config)
+```
+
+**Do NOT monkey-patch `AutoTokenizer.from_pretrained`** — it's a classmethod, and replacing it corrupts the descriptor, which breaks other engines that use different tokenizers (e.g., Qwen uses a Qwen tokenizer via `AutoTokenizer`). Always patch at the config level, not the class method level.
+
+### `torchaudio.load()` requires `torchcodec` in 2.10+
+
+As of `torchaudio>=2.10`, `torchaudio.load()` requires the `torchcodec` package for audio I/O. If your engine or backend code uses `torchaudio.load()`, replace it with `soundfile`:
+
+```python
+# Before (breaks without torchcodec):
+import torchaudio
+waveform, sr = torchaudio.load("audio.wav")
+
+# After:
+import soundfile as sf
+import torch
+data, sr = sf.read("audio.wav", dtype="float32")
+waveform = torch.from_numpy(data).unsqueeze(0)
+```
+
+Note: `torchaudio.functional.resample()` and other pure-PyTorch math functions work fine without `torchcodec` — only the I/O functions are affected.
+
+### `@torch.jit.script` breaks in frozen builds
+
+`torch.jit.script` calls `inspect.getsource()` to parse the decorated function's source code. In a PyInstaller binary, `.py` source files aren't available, so this crashes at import time.
+
+**Fix:** Remove or avoid `@torch.jit.script` decorators. If the decorated function comes from an upstream dependency, write a shim that reimplements the function without the decorator (see "Toxic dependency chains" below).
+
+### Toxic dependency chains — the shim pattern
+
+Sometimes a model library depends on a package with a massive, hostile transitive dependency tree, but only uses a tiny piece of it. When the dependency chain is unbuildable or would pull in dozens of unwanted packages, the right move is to write a lightweight shim.
+
+**Example:** TADA depends on `descript-audio-codec` (DAC), which pulls in `descript-audiotools` -> `onnx`, `tensorboard`, `protobuf`, `matplotlib`, `pystoi`, etc. The `onnx` package fails to build from source on macOS. But TADA only uses `Snake1d` from DAC — a 7-line PyTorch module.
+
+**Solution:** Create a shim at `backend/utils/dac_shim.py` that registers fake modules in `sys.modules`:
+
+```python
+import sys
+import types
+import torch
+from torch import nn
+
+def snake(x, alpha):
+    """Snake activation — reimplemented without @torch.jit.script."""
+    return x + (1.0 / (alpha + 1e-9)) * torch.sin(alpha * x).pow(2)
+
+class Snake1d(nn.Module):
+    def __init__(self, channels):
+        super().__init__()
+        self.alpha = nn.Parameter(torch.ones(1, channels, 1))
+    def forward(self, x):
+        return snake(x, self.alpha)
+
+# Register fake dac.* modules so "from dac.nn.layers import Snake1d" works
+_nn = types.ModuleType("dac.nn")
+_layers = types.ModuleType("dac.nn.layers")
+_layers.Snake1d = Snake1d
+_nn.layers = _layers
+
+for name, mod in [("dac", types.ModuleType("dac")),
+                   ("dac.nn", _nn), ("dac.nn.layers", _layers)]:
+    sys.modules[name] = mod
+```
+
+**Key rules for shims:**
+- Import the shim **before** importing the model library (so it finds the fake modules first)
+- Do NOT use `@torch.jit.script` in the shim (see above)
+- Only reimplement what the model actually uses — check the import chain carefully
+
 ## Upcoming Engines

 Based on the current model landscape, these are candidates for future integration:
@@ -490,7 +586,6 @@ Based on the current model landscape, these are candidates for future integratio
 | **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
 | **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
 | **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
-| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Shipped |
 | **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
 | **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |

@@ -508,6 +603,10 @@ Use this as a gate between phases. Do not proceed to the next phase until every
 - [ ] Searched for `torch.load` calls missing `map_location`
 - [ ] Searched for `torch.from_numpy` without `.float()` cast
 - [ ] Searched for `token=True` or `token=os.getenv("HF_TOKEN")` in HuggingFace calls
+- [ ] Searched for `@torch.jit.script` / `torch.jit.script` (crashes in frozen builds)
+- [ ] Searched for `torchaudio.load` / `torchaudio.save` (requires `torchcodec` in 2.10+)
+- [ ] Searched for hardcoded gated HuggingFace repo names (e.g., `meta-llama/*`)
+- [ ] Evaluated whether any dependency is used minimally enough to shim instead of install
 - [ ] Tested model loading and generation on CPU in a throwaway venv
 - [ ] Tested with a clean HuggingFace cache (no pre-downloaded models)
 - [ ] Produced a written dependency audit documenting all findings
@@ -0,0 +1,173 @@
+# CUDA Libs as a Bolt-On Addon
+
+## Problem
+
+Every time we bump `__version__` (even for a UI tweak or bugfix), the exact-match version check in both `main.rs:222` and `cuda.py:237` invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime.
+
+## Why This Is Hard With `--onefile`
+
+The core tension is PyInstaller `--onefile` mode (`build_binary.py:39`). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA `.dll`/`.so` files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server.
+
+## Options
+
+### Option A: Switch to `--onedir` for the CUDA Build (Recommended)
+
+Instead of `--onefile`, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives:
+
+1. **`voicebox-server-cuda` executable + non-NVIDIA deps** (~200-400MB) -- versioned with the app, redownloaded on every app update.
+2. **`cuda-libs-cu126.tar.gz`** (~2GB) -- the `nvidia.*` packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., `cuda-libs-cu126-v1`). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes.
+
+#### How it would work at runtime
+
+- Tauri downloads the server binary archive and extracts it to `{data_dir}/backends/cuda/`
+- On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory
+- The CUDA server exe finds the `.dll`/`.so` files next to it (standard PyInstaller onedir behavior)
+- Version check becomes two checks: server version + cuda-libs version
+
+#### Independent versioning
+
+Add a `cuda-libs.json` manifest:
+
+```json
+{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"}
+```
+
+The server checks this on startup. The Tauri side checks it before launching. Only bump `cu126-v1` -> `cu126-v2` when we actually change the CUDA toolkit or torch major version.
+
+#### Build pipeline changes
+
+The CI `build-cuda-windows` job would build with `--onedir`, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset.
+
+#### Download experience
+
+- First-time CUDA setup: ~2.4GB total (same as today)
+- Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached
+- CUDA toolkit bump: ~2GB for just the libs
+
+#### Pros
+
+- PyInstaller `--onedir` natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory
+- The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (`nvidia/cublas/lib/`, etc.)
+- CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version
+- Server updates become ~200-400MB instead of ~2.4GB
+- No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree
+
+#### Cons
+
+- Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up
+- Need to modify download/assembly logic in `cuda.py` to handle two separate archives
+- The Tauri side (`main.rs`) needs to point at an exe inside a directory rather than a standalone binary
+- Users who manually manage the file may find the folder structure confusing
+
+#### TTS engine compatibility
+
+No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way.
+
+---
+
+### Option B: Keep `--onefile` but Externalize CUDA Libs via Library Path
+
+Keep the server as a single `--onefile` binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to `{data_dir}/backends/cuda-libs/`. Before launching, set the library search path to include that directory.
+
+**Important caveat:** The CPU torch wheel (`whl/cpu`) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally.
+
+#### How it would work
+
+- Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded
+- Ship `cuda-libs-cu126-v1.tar.gz` separately (~2GB of `.dll`/`.so` files)
+- When launching, Tauri sets `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux) to include the cuda-libs directory
+
+#### Pros
+
+- Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously
+- True bolt-on CUDA libs with fully independent versioning
+- Server updates are always small (~150MB for the onefile binary)
+
+#### Cons
+
+- **Fragile on Windows.** PyInstaller `--onefile` extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario.
+- `os.add_dll_directory()` only affects `LoadLibraryEx` with `LOAD_LIBRARY_SEARCH_USER_DIRS` flag -- not all DLL loads go through this path
+- PyInstaller's onefile bootloader may configure DLL search paths before Python code runs
+- Could work on Linux but is fragile on Windows
+
+---
+
+### Option C: Hybrid -- `--onefile` Server + Dynamic CUDA Lib Loading at Runtime
+
+Build the server as `--onefile` with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using `ctypes.CDLL` or `os.add_dll_directory()`.
+
+In `server.py`, before any torch imports:
+
+```python
+cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS")
+if cuda_libs_dir and os.path.isdir(cuda_libs_dir):
+    if sys.platform == "win32":
+        os.add_dll_directory(cuda_libs_dir)
+        os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "")
+    else:
+        os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "")
+```
+
+#### Pros
+
+- Single server binary, true bolt-on CUDA libs
+- Clean separation of concerns
+- Independent versioning
+
+#### Cons
+
+- Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer
+- On Windows, `os.add_dll_directory()` may not cover all DLL load paths
+- PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs
+- Most complex to get right and maintain
+
+## Recommendation
+
+**Option A (`--onedir` with split archives)** is the most reliable path:
+
+1. **It actually works.** `--onedir` puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install.
+2. **Natural separation.** PyInstaller's `--onedir` output already separates the NVIDIA `.dll`/`.so` files into `nvidia/` subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building.
+3. **Independent versioning is straightforward.** A `cuda-libs.json` manifest controls when redownloads are needed.
+4. **Build pipeline simplification.** Build CUDA libs archive less frequently, store as a pinned release asset.
+
+The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in `cuda.py` with manifests and split parts. Extending that to handle two archives is incremental work.
+
+## Tauri Compatibility (Validated)
+
+Tauri handles PyInstaller `--onedir` with no issues. The key insight is that we're **not** using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing `cuda.py` + `main.rs` flow). For runtime-launched processes, Tauri's `tauri::shell::Command` supports arbitrary directories natively.
+
+### The critical change in `main.rs`
+
+The only Tauri-side change needed is adding `.current_dir()` when spawning the CUDA backend:
+
+```rust
+let cuda_dir = data_dir.join("backends/cuda");
+let exe_path = cuda_dir.join("voicebox-server-cuda.exe");
+
+let mut cmd = app.shell().command(exe_path.to_str().unwrap());
+cmd = cmd.current_dir(&cuda_dir);  // PyInstaller finds all DLLs relative to exe
+cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);
+```
+
+`.current_dir()` tells the PyInstaller bootloader that everything (DLLs, `nvidia/cublas/lib/`, `_internal/`, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal `pip install` or dev environment -- no `LD_LIBRARY_PATH` hacks, no `os.add_dll_directory` gymnastics.
+
+### Community evidence
+
+- Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206).
+- The shell plugin explicitly supports `cwd` in both Rust and JS APIs.
+- No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage.
+
+### Known gotcha: process termination on Windows
+
+PyInstaller onedir creates a parent bootloader + child Python process on Windows. `child.kill()` only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use `taskkill /F /T` for clean shutdown. This is not a blocker -- our existing `--parent-pid` watchdog mechanism in `server.py` already handles orphan cleanup.
+
+## Next Steps
+
+1. Prototype: Build the current CUDA binary with `--onedir` and verify torch CUDA works from the output directory
+2. Measure the size split: how much is NVIDIA libs vs everything else
+3. Design the two-archive download flow and dual version checking
+4. Update `cuda.py` for dual-archive extraction (server core + cuda-libs)
+5. Update `main.rs`: change launch path to `backends/cuda/` dir + add `.current_dir()`
+6. Add `ensure_cuda_structure()` helper in Rust to verify exe + nvidia/ subdirs exist before spawning
+7. Update CI pipeline: `build-cuda-windows` produces two archives instead of split parts
+8. Update `split_binary.py` or replace with archive-based distribution