voicebox/docs/dual-server-binaries.md

Dual Server Binary System

Overview

Voicebox now uses a dual-binary approach to manage the size difference between CPU-only and CUDA-enabled builds:

• CPU Binary (~500MB): Ships with the installer by default
• CUDA Binary (~3GB): Downloaded on-demand for GPU users

Problem Solved

Previously, bundling PyTorch with CUDA support created a 3GB server binary, which:

• Made the installer too large (failed CI builds with WiX)
• Forced all users to download CUDA libraries even without NVIDIA GPUs
• Created poor user experience

Solution

Build Process

Two separate binaries are built:

1. voicebox-server.exe (CPU)

   • Built with: pip install torch --index-url https://download.pytorch.org/whl/cpu
   • Size: ~500MB
   • Works on all Windows machines
   • Included in the installer by default

2. voicebox-server-cuda.exe (CUDA)

   • Built with: pip install torch --index-url https://download.pytorch.org/whl/cu121
   • Size: ~3GB
   • Requires NVIDIA GPU + drivers
   • Uploaded as separate GitHub Release asset

User Experience

First Launch:

1. User installs app (~500MB download)
2. App starts with CPU server
3. If NVIDIA GPU detected:
   • Show notification: "Download CUDA support for 4-5x faster inference?"
   • User clicks "Download"
   • Download voicebox-server-cuda.exe from GitHub (~3GB)
   • Save to %APPDATA%/voicebox/binaries/
   • Restart server with CUDA version

Settings Panel:

• Toggle between CPU/CUDA modes
• Download CUDA if not already installed
• Show current inference backend

Build Scripts

Windows:

cd backend

# Build CPU only
build_cpu.bat

# Build CUDA only
build_cuda.bat

# Build both
build_both.bat

Unix (macOS/Linux):

cd backend

# Build CPU only
./build_cpu.sh

CI/CD Workflow

GitHub Actions (.github/workflows/release.yml):

1. Install CPU PyTorch
2. Build CPU server → Copy to Tauri binaries
3. Install CUDA PyTorch
4. Build CUDA server → Save for upload
5. Build Tauri app (bundles CPU server)
6. Upload CUDA server as separate release asset

File Structure

Release Assets:
├── Voicebox_0.1.12_x64_en-US.msi           (~500MB - includes CPU server)
├── voicebox-server-cuda-x86_64-pc-windows-msvc.exe   (~3GB - optional download)
└── latest.json                              (updater manifest)

Implementation Details

Modified Files

1. backend/build_binary.py

   • Added variant parameter ('cpu' or 'cuda')
   • Outputs different binary names based on variant

2. backend/build_cpu.bat (new)

   • Installs CPU PyTorch
   • Builds CPU binary
   • Restores CUDA PyTorch for dev

3. backend/build_cuda.bat (new)

   • Ensures CUDA PyTorch is installed
   • Builds CUDA binary

4. .github/workflows/release.yml

   • Build CPU binary first (for installer)
   • Build CUDA binary second (for upload)
   • Upload CUDA binary as additional release asset
   • Updated release notes to explain GPU acceleration

Future Frontend Work

TODO: Implement CUDA download in the app

Location: tauri/src/

Features needed:

1. GPU detection on startup
2. Download manager for CUDA binary
3. Server binary path switcher
4. Settings UI for CPU/CUDA toggle
5. Progress indicator for 3GB download

API endpoints needed (already exist):

• /health - Shows GPU availability
• Server restart mechanism

Benefits

✓ Smaller installer: ~500MB instead of 3GB ✓ Faster CI builds: WiX can handle 500MB easily ✓ User choice: CPU users don't download unnecessary files ✓ Better UX: Optional performance upgrade for GPU users ✓ Cost savings: Reduced bandwidth for users without GPUs

Testing

Test CPU build:

cd backend
python build_binary.py cpu
./dist/voicebox-server.exe --version

Test CUDA build:

cd backend
python build_binary.py cuda
./dist/voicebox-server-cuda.exe --version

Verify size:

ls -lh backend/dist/
# Should see:
# voicebox-server.exe       ~500MB
# voicebox-server-cuda.exe  ~3GB

Test server startup:

# CPU version
./backend/dist/voicebox-server.exe
# Check logs: Should show CPU inference

# CUDA version (requires NVIDIA GPU)
./backend/dist/voicebox-server-cuda.exe
# Check logs: Should show CUDA inference