voicebox/docs/plans/CUDA_LIBS_ADDON.md

CUDA Libs as a Bolt-On Addon

Problem

Every time we bump __version__ (even for a UI tweak or bugfix), the exact-match version check in both main.rs:222 and cuda.py:237 invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime.

Why This Is Hard With --onefile

The core tension is PyInstaller --onefile mode (build_binary.py:39). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA .dll/.so files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server.

Options

Option A: Switch to --onedir for the CUDA Build (Recommended)

Instead of --onefile, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives:

1. voicebox-server-cuda executable + non-NVIDIA deps (~200-400MB) -- versioned with the app, redownloaded on every app update.
2. cuda-libs-cu126.tar.gz (~2GB) -- the nvidia.* packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., cuda-libs-cu126-v1). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes.

How it would work at runtime

• Tauri downloads the server binary archive and extracts it to {data_dir}/backends/cuda/
• On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory
• The CUDA server exe finds the .dll/.so files next to it (standard PyInstaller onedir behavior)
• Version check becomes two checks: server version + cuda-libs version

Independent versioning

Add a cuda-libs.json manifest:

{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"}

The server checks this on startup. The Tauri side checks it before launching. Only bump cu126-v1 -> cu126-v2 when we actually change the CUDA toolkit or torch major version.

Build pipeline changes

The CI build-cuda-windows job would build with --onedir, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset.

Download experience

• First-time CUDA setup: ~2.4GB total (same as today)
• Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached
• CUDA toolkit bump: ~2GB for just the libs

Pros

• PyInstaller --onedir natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory
• The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (nvidia/cublas/lib/, etc.)
• CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version
• Server updates become ~200-400MB instead of ~2.4GB
• No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree

Cons

• Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up
• Need to modify download/assembly logic in cuda.py to handle two separate archives
• The Tauri side (main.rs) needs to point at an exe inside a directory rather than a standalone binary
• Users who manually manage the file may find the folder structure confusing

TTS engine compatibility

No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way.

---

Option B: Keep --onefile but Externalize CUDA Libs via Library Path

Keep the server as a single --onefile binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to {data_dir}/backends/cuda-libs/. Before launching, set the library search path to include that directory.

Important caveat: The CPU torch wheel (whl/cpu) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally.

How it would work

• Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded
• Ship cuda-libs-cu126-v1.tar.gz separately (~2GB of .dll/.so files)
• When launching, Tauri sets PATH (Windows) or LD_LIBRARY_PATH (Linux) to include the cuda-libs directory

Pros

• Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously
• True bolt-on CUDA libs with fully independent versioning
• Server updates are always small (~150MB for the onefile binary)

Cons

• Fragile on Windows. PyInstaller --onefile extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario.
• os.add_dll_directory() only affects LoadLibraryEx with LOAD_LIBRARY_SEARCH_USER_DIRS flag -- not all DLL loads go through this path
• PyInstaller's onefile bootloader may configure DLL search paths before Python code runs
• Could work on Linux but is fragile on Windows

---

Option C: Hybrid -- --onefile Server + Dynamic CUDA Lib Loading at Runtime

Build the server as --onefile with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using ctypes.CDLL or os.add_dll_directory().

In server.py, before any torch imports:

cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS")
if cuda_libs_dir and os.path.isdir(cuda_libs_dir):
    if sys.platform == "win32":
        os.add_dll_directory(cuda_libs_dir)
        os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "")
    else:
        os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "")

Pros

• Single server binary, true bolt-on CUDA libs
• Clean separation of concerns
• Independent versioning

Cons

• Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer
• On Windows, os.add_dll_directory() may not cover all DLL load paths
• PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs
• Most complex to get right and maintain

Recommendation

Option A (--onedir with split archives) is the most reliable path:

1. It actually works. --onedir puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install.
2. Natural separation. PyInstaller's --onedir output already separates the NVIDIA .dll/.so files into nvidia/ subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building.
3. Independent versioning is straightforward. A cuda-libs.json manifest controls when redownloads are needed.
4. Build pipeline simplification. Build CUDA libs archive less frequently, store as a pinned release asset.

The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in cuda.py with manifests and split parts. Extending that to handle two archives is incremental work.

Tauri Compatibility (Validated)

Tauri handles PyInstaller --onedir with no issues. The key insight is that we're not using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing cuda.py + main.rs flow). For runtime-launched processes, Tauri's tauri::shell::Command supports arbitrary directories natively.

The critical change in main.rs

The only Tauri-side change needed is adding .current_dir() when spawning the CUDA backend:

let cuda_dir = data_dir.join("backends/cuda");
let exe_path = cuda_dir.join("voicebox-server-cuda.exe");

let mut cmd = app.shell().command(exe_path.to_str().unwrap());
cmd = cmd.current_dir(&cuda_dir);  // PyInstaller finds all DLLs relative to exe
cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);

.current_dir() tells the PyInstaller bootloader that everything (DLLs, nvidia/cublas/lib/, _internal/, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal pip install or dev environment -- no LD_LIBRARY_PATH hacks, no os.add_dll_directory gymnastics.

Community evidence

• Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206).
• The shell plugin explicitly supports cwd in both Rust and JS APIs.
• No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage.

Known gotcha: process termination on Windows

PyInstaller onedir creates a parent bootloader + child Python process on Windows. child.kill() only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use taskkill /F /T for clean shutdown. This is not a blocker -- our existing --parent-pid watchdog mechanism in server.py already handles orphan cleanup.

Next Steps

1. Prototype: Build the current CUDA binary with --onedir and verify torch CUDA works from the output directory
2. Measure the size split: how much is NVIDIA libs vs everything else
3. Design the two-archive download flow and dual version checking
4. Update cuda.py for dual-archive extraction (server core + cuda-libs)
5. Update main.rs: change launch path to backends/cuda/ dir + add .current_dir()
6. Add ensure_cuda_structure() helper in Rust to verify exe + nvidia/ subdirs exist before spawning
7. Update CI pipeline: build-cuda-windows produces two archives instead of split parts
8. Update split_binary.py or replace with archive-based distribution Done: replaced with package_cuda.py