diff --git a/docs/content/docs/developer/tts-engines.mdx b/docs/content/docs/developer/tts-engines.mdx index 90135a37..dc749b02 100644 --- a/docs/content/docs/developer/tts-engines.mdx +++ b/docs/content/docs/developer/tts-engines.mdx @@ -80,6 +80,15 @@ grep -r 'token=True\|token=os.getenv' . # Float64/Float32 assumptions — librosa returns float64, many models assume float32 grep -r "torch.from_numpy\|\.double()\|float64" . + +# @torch.jit.script — calls inspect.getsource(), crashes in frozen builds +grep -r "@torch.jit.script\|torch.jit.script" . + +# torchaudio.load — requires torchcodec in torchaudio 2.10+, use soundfile.read() instead +grep -r "torchaudio.load\|torchaudio.save" . + +# Gated HuggingFace repos — models that hardcode gated repos as tokenizer/config sources +grep -r "from_pretrained\|tokenizer_name\|AutoTokenizer" . | grep -i "llama\|meta-llama\|gated" ``` ### 0.3 Install and Trace in a Throwaway Venv @@ -270,6 +279,8 @@ In `app/src/lib/hooks/useGenerationForm.ts`: - Add engine-to-model-name mapping - Update payload construction for engine-specific fields +**Watch out for model naming inconsistencies.** The HuggingFace repo name, the model size label, and the API model name don't always follow predictable patterns. For example, TADA's 3B model is named `tada-3b-ml` (not `tada-3b`), because it's a multilingual variant. Always check the actual repo names and build the frontend model name mapping from those, not from assumptions like `{engine}-{size}`. + ### 3.5 Model Management In `app/src/components/ServerSettings/ModelManagement.tsx`: @@ -391,6 +402,7 @@ These are actual production failures from shipping new engines. Every one of the | Chatterbox | `FileNotFoundError` for watermark model | `perth` ships pretrained model files (`hparams.yaml`, `.pth.tar`) that PyInstaller doesn't bundle by default | `--collect-all perth` | | All engines | `importlib.metadata` failures | Frozen binary doesn't include package metadata for `huggingface-hub`, `transformers`, etc. | `--copy-metadata` for each affected package | | All engines | Download progress bars stuck at 0% | `huggingface_hub` silently disables tqdm progress bars based on logger level in frozen builds — our progress tracker never receives byte updates | Force-enable tqdm's internal counter in `HFProgressTracker` | +| TADA | `inspect.getsource` error in DAC's `Snake1d` | `@torch.jit.script` calls `inspect.getsource()` which fails without `.py` source files | Wrote a lightweight shim (`dac_shim.py`) reimplementing `Snake1d` without `@torch.jit.script`, registered fake `dac.*` modules in `sys.modules` | | All engines | `NameError: name 'obj' is not defined` on macOS | Python 3.12.0 has a [CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects | Upgrade to Python 3.12.13+ | | All engines | `resource_tracker` subprocess crash | `multiprocessing` in frozen binaries needs `freeze_support()` called before anything else | Added to `server.py` entry point | @@ -480,6 +492,90 @@ def _get_device(self): return "cpu" # Skip MPS ``` +### Gated HuggingFace repos as hardcoded config sources + +Some models hardcode a gated HuggingFace repo as their tokenizer or config source (e.g., TADA hardcodes `"meta-llama/Llama-3.2-1B"` in both its `AlignerConfig` and `TadaConfig`). This silently fails without HF authentication. + +**Fix:** Download from an ungated mirror and patch the config objects directly: + +```python +# Download tokenizer from ungated mirror +UNGATED_TOKENIZER = "unsloth/Llama-3.2-1B" +tokenizer_path = snapshot_download(UNGATED_TOKENIZER, token=None) + +# Patch the model config to use the local path instead of the gated repo +config = ModelConfig.from_pretrained(model_path) +config.tokenizer_name = tokenizer_path +model = ModelClass.from_pretrained(model_path, config=config) +``` + +**Do NOT monkey-patch `AutoTokenizer.from_pretrained`** — it's a classmethod, and replacing it corrupts the descriptor, which breaks other engines that use different tokenizers (e.g., Qwen uses a Qwen tokenizer via `AutoTokenizer`). Always patch at the config level, not the class method level. + +### `torchaudio.load()` requires `torchcodec` in 2.10+ + +As of `torchaudio>=2.10`, `torchaudio.load()` requires the `torchcodec` package for audio I/O. If your engine or backend code uses `torchaudio.load()`, replace it with `soundfile`: + +```python +# Before (breaks without torchcodec): +import torchaudio +waveform, sr = torchaudio.load("audio.wav") + +# After: +import soundfile as sf +import torch +data, sr = sf.read("audio.wav", dtype="float32") +waveform = torch.from_numpy(data).unsqueeze(0) +``` + +Note: `torchaudio.functional.resample()` and other pure-PyTorch math functions work fine without `torchcodec` — only the I/O functions are affected. + +### `@torch.jit.script` breaks in frozen builds + +`torch.jit.script` calls `inspect.getsource()` to parse the decorated function's source code. In a PyInstaller binary, `.py` source files aren't available, so this crashes at import time. + +**Fix:** Remove or avoid `@torch.jit.script` decorators. If the decorated function comes from an upstream dependency, write a shim that reimplements the function without the decorator (see "Toxic dependency chains" below). + +### Toxic dependency chains — the shim pattern + +Sometimes a model library depends on a package with a massive, hostile transitive dependency tree, but only uses a tiny piece of it. When the dependency chain is unbuildable or would pull in dozens of unwanted packages, the right move is to write a lightweight shim. + +**Example:** TADA depends on `descript-audio-codec` (DAC), which pulls in `descript-audiotools` -> `onnx`, `tensorboard`, `protobuf`, `matplotlib`, `pystoi`, etc. The `onnx` package fails to build from source on macOS. But TADA only uses `Snake1d` from DAC — a 7-line PyTorch module. + +**Solution:** Create a shim at `backend/utils/dac_shim.py` that registers fake modules in `sys.modules`: + +```python +import sys +import types +import torch +from torch import nn + +def snake(x, alpha): + """Snake activation — reimplemented without @torch.jit.script.""" + return x + (1.0 / (alpha + 1e-9)) * torch.sin(alpha * x).pow(2) + +class Snake1d(nn.Module): + def __init__(self, channels): + super().__init__() + self.alpha = nn.Parameter(torch.ones(1, channels, 1)) + def forward(self, x): + return snake(x, self.alpha) + +# Register fake dac.* modules so "from dac.nn.layers import Snake1d" works +_nn = types.ModuleType("dac.nn") +_layers = types.ModuleType("dac.nn.layers") +_layers.Snake1d = Snake1d +_nn.layers = _layers + +for name, mod in [("dac", types.ModuleType("dac")), + ("dac.nn", _nn), ("dac.nn.layers", _layers)]: + sys.modules[name] = mod +``` + +**Key rules for shims:** +- Import the shim **before** importing the model library (so it finds the fake modules first) +- Do NOT use `@torch.jit.script` in the shim (see above) +- Only reimplement what the model actually uses — check the import chain carefully + ## Upcoming Engines Based on the current model landscape, these are candidates for future integration: @@ -490,7 +586,6 @@ Based on the current model landscape, these are candidates for future integratio | **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready | | **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready | | **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready | -| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Shipped | | **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting | | **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting | @@ -508,6 +603,10 @@ Use this as a gate between phases. Do not proceed to the next phase until every - [ ] Searched for `torch.load` calls missing `map_location` - [ ] Searched for `torch.from_numpy` without `.float()` cast - [ ] Searched for `token=True` or `token=os.getenv("HF_TOKEN")` in HuggingFace calls +- [ ] Searched for `@torch.jit.script` / `torch.jit.script` (crashes in frozen builds) +- [ ] Searched for `torchaudio.load` / `torchaudio.save` (requires `torchcodec` in 2.10+) +- [ ] Searched for hardcoded gated HuggingFace repo names (e.g., `meta-llama/*`) +- [ ] Evaluated whether any dependency is used minimally enough to shim instead of install - [ ] Tested model loading and generation on CPU in a throwaway venv - [ ] Tested with a clean HuggingFace cache (no pre-downloaded models) - [ ] Produced a written dependency audit documenting all findings diff --git a/docs/plans/CUDA_LIBS_ADDON.md b/docs/plans/CUDA_LIBS_ADDON.md new file mode 100644 index 00000000..440d7950 --- /dev/null +++ b/docs/plans/CUDA_LIBS_ADDON.md @@ -0,0 +1,173 @@ +# CUDA Libs as a Bolt-On Addon + +## Problem + +Every time we bump `__version__` (even for a UI tweak or bugfix), the exact-match version check in both `main.rs:222` and `cuda.py:237` invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime. + +## Why This Is Hard With `--onefile` + +The core tension is PyInstaller `--onefile` mode (`build_binary.py:39`). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA `.dll`/`.so` files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server. + +## Options + +### Option A: Switch to `--onedir` for the CUDA Build (Recommended) + +Instead of `--onefile`, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives: + +1. **`voicebox-server-cuda` executable + non-NVIDIA deps** (~200-400MB) -- versioned with the app, redownloaded on every app update. +2. **`cuda-libs-cu126.tar.gz`** (~2GB) -- the `nvidia.*` packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., `cuda-libs-cu126-v1`). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes. + +#### How it would work at runtime + +- Tauri downloads the server binary archive and extracts it to `{data_dir}/backends/cuda/` +- On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory +- The CUDA server exe finds the `.dll`/`.so` files next to it (standard PyInstaller onedir behavior) +- Version check becomes two checks: server version + cuda-libs version + +#### Independent versioning + +Add a `cuda-libs.json` manifest: + +```json +{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"} +``` + +The server checks this on startup. The Tauri side checks it before launching. Only bump `cu126-v1` -> `cu126-v2` when we actually change the CUDA toolkit or torch major version. + +#### Build pipeline changes + +The CI `build-cuda-windows` job would build with `--onedir`, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset. + +#### Download experience + +- First-time CUDA setup: ~2.4GB total (same as today) +- Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached +- CUDA toolkit bump: ~2GB for just the libs + +#### Pros + +- PyInstaller `--onedir` natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory +- The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (`nvidia/cublas/lib/`, etc.) +- CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version +- Server updates become ~200-400MB instead of ~2.4GB +- No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree + +#### Cons + +- Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up +- Need to modify download/assembly logic in `cuda.py` to handle two separate archives +- The Tauri side (`main.rs`) needs to point at an exe inside a directory rather than a standalone binary +- Users who manually manage the file may find the folder structure confusing + +#### TTS engine compatibility + +No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way. + +--- + +### Option B: Keep `--onefile` but Externalize CUDA Libs via Library Path + +Keep the server as a single `--onefile` binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to `{data_dir}/backends/cuda-libs/`. Before launching, set the library search path to include that directory. + +**Important caveat:** The CPU torch wheel (`whl/cpu`) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally. + +#### How it would work + +- Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded +- Ship `cuda-libs-cu126-v1.tar.gz` separately (~2GB of `.dll`/`.so` files) +- When launching, Tauri sets `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux) to include the cuda-libs directory + +#### Pros + +- Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously +- True bolt-on CUDA libs with fully independent versioning +- Server updates are always small (~150MB for the onefile binary) + +#### Cons + +- **Fragile on Windows.** PyInstaller `--onefile` extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario. +- `os.add_dll_directory()` only affects `LoadLibraryEx` with `LOAD_LIBRARY_SEARCH_USER_DIRS` flag -- not all DLL loads go through this path +- PyInstaller's onefile bootloader may configure DLL search paths before Python code runs +- Could work on Linux but is fragile on Windows + +--- + +### Option C: Hybrid -- `--onefile` Server + Dynamic CUDA Lib Loading at Runtime + +Build the server as `--onefile` with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using `ctypes.CDLL` or `os.add_dll_directory()`. + +In `server.py`, before any torch imports: + +```python +cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS") +if cuda_libs_dir and os.path.isdir(cuda_libs_dir): + if sys.platform == "win32": + os.add_dll_directory(cuda_libs_dir) + os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "") + else: + os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "") +``` + +#### Pros + +- Single server binary, true bolt-on CUDA libs +- Clean separation of concerns +- Independent versioning + +#### Cons + +- Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer +- On Windows, `os.add_dll_directory()` may not cover all DLL load paths +- PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs +- Most complex to get right and maintain + +## Recommendation + +**Option A (`--onedir` with split archives)** is the most reliable path: + +1. **It actually works.** `--onedir` puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install. +2. **Natural separation.** PyInstaller's `--onedir` output already separates the NVIDIA `.dll`/`.so` files into `nvidia/` subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building. +3. **Independent versioning is straightforward.** A `cuda-libs.json` manifest controls when redownloads are needed. +4. **Build pipeline simplification.** Build CUDA libs archive less frequently, store as a pinned release asset. + +The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in `cuda.py` with manifests and split parts. Extending that to handle two archives is incremental work. + +## Tauri Compatibility (Validated) + +Tauri handles PyInstaller `--onedir` with no issues. The key insight is that we're **not** using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing `cuda.py` + `main.rs` flow). For runtime-launched processes, Tauri's `tauri::shell::Command` supports arbitrary directories natively. + +### The critical change in `main.rs` + +The only Tauri-side change needed is adding `.current_dir()` when spawning the CUDA backend: + +```rust +let cuda_dir = data_dir.join("backends/cuda"); +let exe_path = cuda_dir.join("voicebox-server-cuda.exe"); + +let mut cmd = app.shell().command(exe_path.to_str().unwrap()); +cmd = cmd.current_dir(&cuda_dir); // PyInstaller finds all DLLs relative to exe +cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]); +``` + +`.current_dir()` tells the PyInstaller bootloader that everything (DLLs, `nvidia/cublas/lib/`, `_internal/`, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal `pip install` or dev environment -- no `LD_LIBRARY_PATH` hacks, no `os.add_dll_directory` gymnastics. + +### Community evidence + +- Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206). +- The shell plugin explicitly supports `cwd` in both Rust and JS APIs. +- No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage. + +### Known gotcha: process termination on Windows + +PyInstaller onedir creates a parent bootloader + child Python process on Windows. `child.kill()` only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use `taskkill /F /T` for clean shutdown. This is not a blocker -- our existing `--parent-pid` watchdog mechanism in `server.py` already handles orphan cleanup. + +## Next Steps + +1. Prototype: Build the current CUDA binary with `--onedir` and verify torch CUDA works from the output directory +2. Measure the size split: how much is NVIDIA libs vs everything else +3. Design the two-archive download flow and dual version checking +4. Update `cuda.py` for dual-archive extraction (server core + cuda-libs) +5. Update `main.rs`: change launch path to `backends/cuda/` dir + add `.current_dir()` +6. Add `ensure_cuda_structure()` helper in Rust to verify exe + nvidia/ subdirs exist before spawning +7. Update CI pipeline: `build-cuda-windows` produces two archives instead of split parts +8. Update `split_binary.py` or replace with archive-based distribution