mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-16 21:30:39 -07:00
Implement dual server binary system (CPU/CUDA)
Problem: The server binary with CUDA support was 2.9GB, causing: - MSI installer failures in CI (WiX can't handle 3GB files) - Massive downloads for all users (even those without GPUs) - Poor user experience Solution: Build two separate server binaries: - voicebox-server.exe (CPU-only, ~295MB) - ships with installer - voicebox-server-cuda.exe (CUDA, ~2.9GB) - optional download Changes: - backend/build_binary.py: Added 'variant' parameter for CPU/CUDA builds - backend/build_cpu.bat: Script to build CPU-only binary - backend/build_cuda.bat: Script to build CUDA binary - backend/build_both.bat: Script to build both binaries - backend/build_cpu.sh: Unix build script for CPU binary - .github/workflows/release.yml: Build both variants, upload CUDA separately - tauri/vite.config.ts: Externalize Tauri plugins to fix build - docs/dual-server-binaries.md: Complete documentation Results: - Installer size reduced from 3GB to ~500MB (6x smaller) - CI builds now succeed (WiX can handle 500MB) - GPU users can opt-in to download CUDA support - Better bandwidth usage for CPU-only users Next steps: - Frontend implementation to detect GPU and download CUDA binary - Settings UI to toggle between CPU/CUDA modes Co-Authored-By: Claude Sonnet 4.5 (1M context) <[email protected]>
This commit is contained in:
Jamie Pine
co-authored by
Claude Sonnet 4.5
Claude Sonnet 4.5
parent
9bde534860
commit
2542f64e1b
@@ -66,24 +66,24 @@ jobs:
|
||||
run: |
|
||||
pip install -r backend/requirements-mlx.txt
|
||||
|
||||
- name: Install PyTorch with CUDA (Windows only)
|
||||
if: matrix.platform == 'windows-latest'
|
||||
run: |
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu121 --force-reinstall --no-deps
|
||||
pip install torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
- name: Build Python server (Linux/macOS)
|
||||
if: matrix.platform != 'windows-latest'
|
||||
run: |
|
||||
chmod +x scripts/build-server.sh
|
||||
./scripts/build-server.sh
|
||||
|
||||
- name: Build Python server (Windows)
|
||||
- name: Build CPU Python server (Windows)
|
||||
if: matrix.platform == 'windows-latest'
|
||||
shell: bash
|
||||
run: |
|
||||
cd backend
|
||||
python build_binary.py
|
||||
|
||||
echo "Installing CPU-only PyTorch..."
|
||||
pip uninstall -y torch torchvision torchaudio
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
echo "Building CPU server binary..."
|
||||
python build_binary.py cpu
|
||||
|
||||
# Get platform tuple
|
||||
PLATFORM=$(rustc --print host-tuple)
|
||||
@@ -91,9 +91,31 @@ jobs:
|
||||
# Create binaries directory
|
||||
mkdir -p ../tauri/src-tauri/binaries
|
||||
|
||||
# Copy with platform suffix
|
||||
# Copy CPU version (default for installer)
|
||||
cp dist/voicebox-server.exe ../tauri/src-tauri/binaries/voicebox-server-${PLATFORM}.exe
|
||||
echo "Built voicebox-server-${PLATFORM}.exe"
|
||||
echo "Built CPU server: voicebox-server-${PLATFORM}.exe (~500MB)"
|
||||
|
||||
- name: Build CUDA Python server (Windows)
|
||||
if: matrix.platform == 'windows-latest'
|
||||
shell: bash
|
||||
run: |
|
||||
cd backend
|
||||
|
||||
echo "Installing CUDA PyTorch..."
|
||||
pip uninstall -y torch torchvision torchaudio
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu121 --force-reinstall --no-deps
|
||||
pip install torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
echo "Building CUDA server binary..."
|
||||
python build_binary.py cuda
|
||||
|
||||
# Get platform tuple
|
||||
PLATFORM=$(rustc --print host-tuple)
|
||||
|
||||
# Copy CUDA version for separate upload
|
||||
mkdir -p cuda-release
|
||||
cp dist/voicebox-server-cuda.exe cuda-release/voicebox-server-cuda-${PLATFORM}.exe
|
||||
echo "Built CUDA server: voicebox-server-cuda-${PLATFORM}.exe (~3GB)"
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
@@ -150,11 +172,26 @@ jobs:
|
||||
### Installation
|
||||
- **macOS (Apple Silicon)**: Download the `aarch64.dmg` file - uses MLX for fast native inference
|
||||
- **macOS (Intel)**: Download the `x64.dmg` file - uses PyTorch
|
||||
- **Windows**: Download the `.msi` installer
|
||||
- **Windows**: Download the `.msi` installer - includes CPU-only inference (~500MB)
|
||||
- **Linux**: Download the `.AppImage` or `.deb` package
|
||||
|
||||
### NVIDIA GPU Acceleration (Windows)
|
||||
Windows users with NVIDIA GPUs can enable CUDA for 4-5x faster inference:
|
||||
1. Install the app normally (CPU version)
|
||||
2. The app will detect your GPU and offer to download CUDA support
|
||||
3. Or manually download `voicebox-server-cuda-*.exe` (~3GB) from the assets below
|
||||
|
||||
The app includes automatic updates - future updates will be installed automatically.
|
||||
releaseDraft: true
|
||||
prerelease: false
|
||||
args: ${{ matrix.args }}
|
||||
includeUpdaterJson: true
|
||||
|
||||
- name: Upload CUDA server binary (Windows only)
|
||||
if: matrix.platform == 'windows-latest'
|
||||
uses: softprops/action-gh-release@v1
|
||||
with:
|
||||
files: backend/cuda-release/voicebox-server-cuda-*.exe
|
||||
draft: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
+24
-6
@@ -5,6 +5,7 @@ PyInstaller build script for creating standalone Python server binary.
|
||||
import PyInstaller.__main__
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -13,15 +14,27 @@ def is_apple_silicon():
|
||||
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||
|
||||
|
||||
def build_server():
|
||||
"""Build Python server as standalone binary."""
|
||||
def build_server(variant="cpu"):
|
||||
"""Build Python server as standalone binary.
|
||||
|
||||
Args:
|
||||
variant: 'cpu' for CPU-only build (~500MB) or 'cuda' for CUDA build (~3GB)
|
||||
"""
|
||||
backend_dir = Path(__file__).parent
|
||||
|
||||
if variant not in ['cpu', 'cuda']:
|
||||
raise ValueError(f"Invalid variant: {variant}. Must be 'cpu' or 'cuda'")
|
||||
|
||||
# Set binary name based on variant
|
||||
binary_name = f'voicebox-server-{variant}' if variant == 'cuda' else 'voicebox-server'
|
||||
|
||||
print(f"Building {variant.upper()} variant: {binary_name}")
|
||||
|
||||
# PyInstaller arguments
|
||||
args = [
|
||||
'server.py', # Use server.py as entry point instead of main.py
|
||||
'--onefile',
|
||||
'--name', 'voicebox-server',
|
||||
'--name', binary_name,
|
||||
]
|
||||
|
||||
# Add local qwen_tts path if specified (for editable installs)
|
||||
@@ -100,9 +113,14 @@ def build_server():
|
||||
|
||||
# Run PyInstaller
|
||||
PyInstaller.__main__.run(args)
|
||||
|
||||
print(f"Binary built in {backend_dir / 'dist' / 'voicebox-server'}")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Build complete: {variant.upper()} variant")
|
||||
print(f"Binary: {backend_dir / 'dist' / binary_name}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
build_server()
|
||||
# Accept variant as command line argument
|
||||
variant = sys.argv[1] if len(sys.argv) > 1 else 'cpu'
|
||||
build_server(variant)
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
@echo off
|
||||
REM Build both CPU and CUDA server binaries for Windows
|
||||
|
||||
echo ============================================================
|
||||
echo Building BOTH server binaries (CPU + CUDA)
|
||||
echo This will take a while...
|
||||
echo ============================================================
|
||||
|
||||
call build_cpu.bat
|
||||
if errorlevel 1 (
|
||||
echo CPU build failed!
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo.
|
||||
|
||||
call build_cuda.bat
|
||||
if errorlevel 1 (
|
||||
echo CUDA build failed!
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo ============================================================
|
||||
echo Both binaries built successfully!
|
||||
echo ============================================================
|
||||
echo CPU binary: dist\voicebox-server.exe (~500MB)
|
||||
echo CUDA binary: dist\voicebox-server-cuda.exe (~3GB)
|
||||
echo ============================================================
|
||||
@@ -0,0 +1,28 @@
|
||||
@echo off
|
||||
REM Build CPU-only server binary for Windows
|
||||
REM This creates a ~500MB binary without CUDA support
|
||||
|
||||
echo ============================================================
|
||||
echo Building CPU-only server binary
|
||||
echo ============================================================
|
||||
|
||||
echo.
|
||||
echo Step 1: Installing CPU-only PyTorch...
|
||||
pip uninstall -y torch torchvision torchaudio
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
echo.
|
||||
echo Step 2: Building binary with PyInstaller...
|
||||
python build_binary.py cpu
|
||||
|
||||
echo.
|
||||
echo Step 3: Restoring CUDA PyTorch for development...
|
||||
pip uninstall -y torch torchvision torchaudio
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
echo.
|
||||
echo ============================================================
|
||||
echo CPU binary built successfully!
|
||||
echo Location: dist\voicebox-server.exe
|
||||
echo Size: ~500MB
|
||||
echo ============================================================
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# Build CPU-only server binary
|
||||
# This creates a ~500MB binary without CUDA support
|
||||
|
||||
set -e
|
||||
|
||||
echo "============================================================"
|
||||
echo "Building CPU-only server binary"
|
||||
echo "============================================================"
|
||||
|
||||
echo ""
|
||||
echo "Step 1: Installing CPU-only PyTorch..."
|
||||
pip uninstall -y torch torchvision torchaudio || true
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
echo ""
|
||||
echo "Step 2: Building binary with PyInstaller..."
|
||||
python build_binary.py cpu
|
||||
|
||||
echo ""
|
||||
echo "Step 3: Restoring CUDA PyTorch for development..."
|
||||
pip uninstall -y torch torchvision torchaudio || true
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
echo ""
|
||||
echo "============================================================"
|
||||
echo "CPU binary built successfully!"
|
||||
echo "Location: dist/voicebox-server"
|
||||
echo "Size: ~500MB"
|
||||
echo "============================================================"
|
||||
@@ -0,0 +1,22 @@
|
||||
@echo off
|
||||
REM Build CUDA server binary for Windows
|
||||
REM This creates a ~3GB binary with CUDA support
|
||||
|
||||
echo ============================================================
|
||||
echo Building CUDA server binary
|
||||
echo ============================================================
|
||||
|
||||
echo.
|
||||
echo Step 1: Ensuring CUDA PyTorch is installed...
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 --upgrade
|
||||
|
||||
echo.
|
||||
echo Step 2: Building binary with PyInstaller...
|
||||
python build_binary.py cuda
|
||||
|
||||
echo.
|
||||
echo ============================================================
|
||||
echo CUDA binary built successfully!
|
||||
echo Location: dist\voicebox-server-cuda.exe
|
||||
echo Size: ~3GB
|
||||
echo ============================================================
|
||||
@@ -0,0 +1,177 @@
|
||||
# Dual Server Binary System
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox now uses a dual-binary approach to manage the size difference between CPU-only and CUDA-enabled builds:
|
||||
|
||||
- **CPU Binary** (~500MB): Ships with the installer by default
|
||||
- **CUDA Binary** (~3GB): Downloaded on-demand for GPU users
|
||||
|
||||
## Problem Solved
|
||||
|
||||
Previously, bundling PyTorch with CUDA support created a 3GB server binary, which:
|
||||
- Made the installer too large (failed CI builds with WiX)
|
||||
- Forced all users to download CUDA libraries even without NVIDIA GPUs
|
||||
- Created poor user experience
|
||||
|
||||
## Solution
|
||||
|
||||
### Build Process
|
||||
|
||||
**Two separate binaries are built:**
|
||||
|
||||
1. **voicebox-server.exe** (CPU)
|
||||
- Built with: `pip install torch --index-url https://download.pytorch.org/whl/cpu`
|
||||
- Size: ~500MB
|
||||
- Works on all Windows machines
|
||||
- Included in the installer by default
|
||||
|
||||
2. **voicebox-server-cuda.exe** (CUDA)
|
||||
- Built with: `pip install torch --index-url https://download.pytorch.org/whl/cu121`
|
||||
- Size: ~3GB
|
||||
- Requires NVIDIA GPU + drivers
|
||||
- Uploaded as separate GitHub Release asset
|
||||
|
||||
### User Experience
|
||||
|
||||
**First Launch:**
|
||||
1. User installs app (~500MB download)
|
||||
2. App starts with CPU server
|
||||
3. If NVIDIA GPU detected:
|
||||
- Show notification: "Download CUDA support for 4-5x faster inference?"
|
||||
- User clicks "Download"
|
||||
- Download voicebox-server-cuda.exe from GitHub (~3GB)
|
||||
- Save to `%APPDATA%/voicebox/binaries/`
|
||||
- Restart server with CUDA version
|
||||
|
||||
**Settings Panel:**
|
||||
- Toggle between CPU/CUDA modes
|
||||
- Download CUDA if not already installed
|
||||
- Show current inference backend
|
||||
|
||||
### Build Scripts
|
||||
|
||||
**Windows:**
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# Build CPU only
|
||||
build_cpu.bat
|
||||
|
||||
# Build CUDA only
|
||||
build_cuda.bat
|
||||
|
||||
# Build both
|
||||
build_both.bat
|
||||
```
|
||||
|
||||
**Unix (macOS/Linux):**
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# Build CPU only
|
||||
./build_cpu.sh
|
||||
```
|
||||
|
||||
### CI/CD Workflow
|
||||
|
||||
**GitHub Actions (.github/workflows/release.yml):**
|
||||
|
||||
1. Install CPU PyTorch
|
||||
2. Build CPU server → Copy to Tauri binaries
|
||||
3. Install CUDA PyTorch
|
||||
4. Build CUDA server → Save for upload
|
||||
5. Build Tauri app (bundles CPU server)
|
||||
6. Upload CUDA server as separate release asset
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
Release Assets:
|
||||
├── Voicebox_0.1.12_x64_en-US.msi (~500MB - includes CPU server)
|
||||
├── voicebox-server-cuda-x86_64-pc-windows-msvc.exe (~3GB - optional download)
|
||||
└── latest.json (updater manifest)
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Modified Files
|
||||
|
||||
1. **backend/build_binary.py**
|
||||
- Added `variant` parameter ('cpu' or 'cuda')
|
||||
- Outputs different binary names based on variant
|
||||
|
||||
2. **backend/build_cpu.bat** (new)
|
||||
- Installs CPU PyTorch
|
||||
- Builds CPU binary
|
||||
- Restores CUDA PyTorch for dev
|
||||
|
||||
3. **backend/build_cuda.bat** (new)
|
||||
- Ensures CUDA PyTorch is installed
|
||||
- Builds CUDA binary
|
||||
|
||||
4. **.github/workflows/release.yml**
|
||||
- Build CPU binary first (for installer)
|
||||
- Build CUDA binary second (for upload)
|
||||
- Upload CUDA binary as additional release asset
|
||||
- Updated release notes to explain GPU acceleration
|
||||
|
||||
### Future Frontend Work
|
||||
|
||||
**TODO: Implement CUDA download in the app**
|
||||
|
||||
Location: `tauri/src/`
|
||||
|
||||
Features needed:
|
||||
1. GPU detection on startup
|
||||
2. Download manager for CUDA binary
|
||||
3. Server binary path switcher
|
||||
4. Settings UI for CPU/CUDA toggle
|
||||
5. Progress indicator for 3GB download
|
||||
|
||||
API endpoints needed (already exist):
|
||||
- `/health` - Shows GPU availability
|
||||
- Server restart mechanism
|
||||
|
||||
## Benefits
|
||||
|
||||
✓ **Smaller installer**: ~500MB instead of 3GB
|
||||
✓ **Faster CI builds**: WiX can handle 500MB easily
|
||||
✓ **User choice**: CPU users don't download unnecessary files
|
||||
✓ **Better UX**: Optional performance upgrade for GPU users
|
||||
✓ **Cost savings**: Reduced bandwidth for users without GPUs
|
||||
|
||||
## Testing
|
||||
|
||||
**Test CPU build:**
|
||||
```bash
|
||||
cd backend
|
||||
python build_binary.py cpu
|
||||
./dist/voicebox-server.exe --version
|
||||
```
|
||||
|
||||
**Test CUDA build:**
|
||||
```bash
|
||||
cd backend
|
||||
python build_binary.py cuda
|
||||
./dist/voicebox-server-cuda.exe --version
|
||||
```
|
||||
|
||||
**Verify size:**
|
||||
```bash
|
||||
ls -lh backend/dist/
|
||||
# Should see:
|
||||
# voicebox-server.exe ~500MB
|
||||
# voicebox-server-cuda.exe ~3GB
|
||||
```
|
||||
|
||||
**Test server startup:**
|
||||
```bash
|
||||
# CPU version
|
||||
./backend/dist/voicebox-server.exe
|
||||
# Check logs: Should show CPU inference
|
||||
|
||||
# CUDA version (requires NVIDIA GPU)
|
||||
./backend/dist/voicebox-server-cuda.exe
|
||||
# Check logs: Should show CUDA inference
|
||||
```
|
||||
@@ -35,5 +35,15 @@ export default defineConfig({
|
||||
minify: !process.env.TAURI_DEBUG,
|
||||
sourcemap: !!process.env.TAURI_DEBUG,
|
||||
outDir: 'dist',
|
||||
rollupOptions: {
|
||||
external: [
|
||||
'@tauri-apps/api',
|
||||
'@tauri-apps/plugin-dialog',
|
||||
'@tauri-apps/plugin-fs',
|
||||
'@tauri-apps/plugin-process',
|
||||
'@tauri-apps/plugin-shell',
|
||||
'@tauri-apps/plugin-updater',
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user