Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-15 21:00:42 -07:00
Code Issues Packages Projects Releases Wiki Activity

Refactor documentation structure and dependencies for migration to Fumadocs

Browse Source 
- Updated `.gitignore` to include new build and generated content directories.
- Removed outdated Mintlify configuration files and documentation.
- Introduced new `MIGRATION.md` to outline the transition from Mintlify to Fumadocs.
- Added `mdx-components.tsx` for MDX component configuration and compatibility.
- Updated `package.json` and `next.config.mjs` for new dependencies and Next.js configuration.
- Created `source.config.ts` for content source configuration.
- Added OpenAPI specification in `openapi.json` for API documentation.
- Removed legacy files and adjusted project structure to align with Fumadocs conventions.
This commit is contained in:
Jamie Pine
2026-02-02 23:29:35 -08:00
parent 788a04f265
commit 2e6efa00a2
145 changed files with 355 additions and 11214 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/docs/AUTOUPDATER.md
+195
View File
@@ -0,0 +1,195 @@
---
title: "Auto-Updater Documentation"
description: "How Voicebox automatic updates work for users and developers"
---
Voicebox includes automatic updates powered by Tauri's updater plugin. This document explains how it works for both users and developers.
## 1. Generate Signing Keys
Run this command to generate your signing keypair:
```bash
cd tauri && bun tauri signer generate -w ~/.tauri/voicebox.key
```
This creates:
- **Private key**: `~/.tauri/voicebox.key` (keep this secret!)
- **Public key**: `~/.tauri/voicebox.key.pub`
## 2. Update Configuration
Copy the content from `~/.tauri/voicebox.key.pub` and replace the placeholder in `tauri/src-tauri/tauri.conf.json`:
```json
{
"plugins": {
"updater": {
"pubkey": "PASTE_PUBLIC_KEY_CONTENT_HERE",
"endpoints": [
"https://github.com/YOUR_USERNAME/voicebox/releases/latest/download/latest.json"
]
}
}
}
```
Update the endpoint URL with your actual GitHub username/organization.
## 3. Building with Signatures
When building releases, set these environment variables:
**macOS/Linux:**
```bash
export TAURI_SIGNING_PRIVATE_KEY="$(cat ~/.tauri/voicebox.key)"
export TAURI_SIGNING_PRIVATE_KEY_PASSWORD=""
bun run build
```
**Windows PowerShell:**
```powershell
$env:TAURI_SIGNING_PRIVATE_KEY = Get-Content ~/.tauri/voicebox.key -Raw
$env:TAURI_SIGNING_PRIVATE_KEY_PASSWORD = ""
bun run build
```
## 4. GitHub Release Setup
When you create a GitHub release, the build process will generate:
- Installers for each platform
- `.sig` signature files
- `latest.json` update manifest
### Manual Release Process
1. Build the app with signing keys set
2. Create a new GitHub release
3. Upload all files from `tauri/src-tauri/target/release/bundle/`
4. Create `latest.json` in your release assets:
```json
{
"version": "0.2.0",
"notes": "Bug fixes and improvements",
"pub_date": "2026-01-25T12:00:00Z",
"platforms": {
"darwin-aarch64": {
"signature": "CONTENT_FROM_.app.tar.gz.sig",
"url": "https://github.com/YOUR_USERNAME/voicebox/releases/download/v0.2.0/voicebox_0.2.0_aarch64.dmg"
},
"darwin-x86_64": {
"signature": "CONTENT_FROM_.app.tar.gz.sig",
"url": "https://github.com/YOUR_USERNAME/voicebox/releases/download/v0.2.0/voicebox_0.2.0_x64.dmg"
},
"linux-x86_64": {
"signature": "CONTENT_FROM_.AppImage.sig",
"url": "https://github.com/YOUR_USERNAME/voicebox/releases/download/v0.2.0/voicebox_0.2.0_amd64.AppImage"
},
"windows-x86_64": {
"signature": "CONTENT_FROM_.msi.sig",
"url": "https://github.com/YOUR_USERNAME/voicebox/releases/download/v0.2.0/voicebox_0.2.0_x64_en-US.msi"
}
}
}
```
### Automated GitHub Actions (Recommended)
Create `.github/workflows/release.yml`:
```yaml
name: Release
on:
push:
tags:
- 'v*'
jobs:
release:
strategy:
matrix:
platform: [macos-latest, ubuntu-22.04, windows-latest]
runs-on: ${{ matrix.platform }}
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v1
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
- name: Install dependencies (Ubuntu)
if: matrix.platform == 'ubuntu-22.04'
run: |
sudo apt-get update
sudo apt-get install -y libwebkit2gtk-4.1-dev libappindicator3-dev librsvg2-dev patchelf
- name: Install dependencies
run: bun install
- name: Build
env:
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
TAURI_SIGNING_PRIVATE_KEY_PASSWORD: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY_PASSWORD }}
run: bun run build
- name: Upload Release
uses: softprops/action-gh-release@v1
with:
files: tauri/src-tauri/target/release/bundle/**/*
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
Add your private key to GitHub secrets:
- Go to Settings → Secrets and variables → Actions
- Add `TAURI_SIGNING_PRIVATE_KEY` with the content of `~/.tauri/voicebox.key`
- Add `TAURI_SIGNING_PRIVATE_KEY_PASSWORD` (empty string if no password)
## Frontend Integration
The frontend integration is complete with automatic update notifications and manual update checks:
- **Update Notification Banner** - Appears automatically when updates are available
- **Settings Panel** - Manual "Check for Updates" button in Settings tab
- **Update Hook** - React hook handles all update operations
See `docs/AUTOUPDATER_QUICKSTART.md` for a quick setup guide.
## Security Notes
- Never commit your private key to version control
- Store private keys securely (use GitHub secrets for CI/CD)
- The public key in `tauri.conf.json` is safe to commit
- Updates are cryptographically verified before installation
- HTTP endpoints are blocked by default (HTTPS only)
## Testing Updates
1. Build version 0.1.0 and install it
2. Update version in `tauri.conf.json` to 0.2.0
3. Build version 0.2.0 with signatures
4. Create a local server or GitHub release with `latest.json`
5. Run version 0.1.0 and trigger update check
6. Verify update downloads and installs correctly
## Troubleshooting
**"Invalid signature" error:**
- Verify public key matches the private key used to sign
- Ensure signature files (.sig) are uploaded correctly
**"No update available" when one exists:**
- Check endpoint URL is correct
- Verify `latest.json` format matches specification
- Ensure version in latest.json is higher than current version
**Build fails with signing:**
- Confirm environment variables are set correctly
- Check private key file exists and is readable
- Verify private key format (should start with `dW50cnVzdGVkIGNvbW1lbnQ6`)
docs/content/docs/AUTOUPDATER_QUICKSTART.md
+119
View File
@@ -0,0 +1,119 @@
---
title: "Autoupdater Quick Start"
description: "Quick guide to activate the Tauri v2 autoupdater"
---
The Tauri v2 autoupdater has been fully configured and integrated. Follow these steps to activate it.
## What's Already Done
✅ Rust plugin installed and initialized
✅ Tauri configuration set up with updater settings
✅ Permissions granted for update operations
✅ GitHub Actions workflow updated with signing support
✅ Frontend components created and integrated
✅ Update notifications on app startup
✅ Manual update check in Settings tab
## Required Steps (5 minutes)
### 1. Generate Signing Keys
```bash
bun run generate:keys
```
This creates:
- Private key: `~/.tauri/voicebox.key` (keep secret!)
- Public key: `~/.tauri/voicebox.key.pub` (safe to share)
### 2. Update Tauri Config
Open `tauri/src-tauri/tauri.conf.json` and:
1. Replace `"REPLACE_WITH_YOUR_PUBLIC_KEY"` with the content from `~/.tauri/voicebox.key.pub`
2. Update the endpoint URL with your GitHub username:
```json
"endpoints": [
"https://github.com/YOUR_USERNAME/voicebox/releases/latest/download/latest.json"
]
```
### 3. Add GitHub Secrets
Go to your repo Settings → Secrets and variables → Actions:
1. Add `TAURI_SIGNING_PRIVATE_KEY`:
```bash
cat ~/.tauri/voicebox.key
```
Copy the entire output and paste as the secret value
2. Add `TAURI_SIGNING_PRIVATE_KEY_PASSWORD`:
Leave empty (or add your password if you set one)
### 4. Test the Setup
To test locally before creating a release:
```bash
bun run build:release
```
This will verify your keys are set up correctly.
## How It Works
### For Users
1. App checks for updates on startup (only in Tauri builds)
2. If an update is available, a banner appears at the top
3. Users can click "Install Now" to download and install
4. App restarts automatically after installation
### For Developers
1. Create a new git tag: `git tag v0.2.0 && git push --tags`
2. GitHub Actions builds signed releases for all platforms
3. Uploads installers and generates `latest.json` manifest
4. Users running older versions will be notified automatically
## UI Components
### Update Notification Banner
- Shows at top of app when update is available
- Appears automatically on startup
- Displays download/install progress
### Settings Panel
- Located in Settings tab
- Shows current version
- Manual "Check for Updates" button
- Update status and progress
## Troubleshooting
**"Public key not configured"**
- Make sure you copied the entire content from `voicebox.key.pub`
- The key should start with `dW50cnVzdGVkIGNvbW1lbnQ6`
**"Failed to check for updates"**
- Endpoint URL might be incorrect
- No releases published yet (expected for first setup)
**Build fails with signing error**
- Check that GitHub secrets are set correctly
- Verify private key file exists at `~/.tauri/voicebox.key`
## Next Release Workflow
1. Update version in `tauri/src-tauri/tauri.conf.json`
2. Commit changes
3. Create and push tag: `git tag v0.2.0 && git push --tags`
4. GitHub Actions will automatically build and create a draft release
5. Review the release and publish it
6. Users will be notified of the update
## See Also
- Full documentation: `docs/AUTOUPDATER.md`
- Build script: `scripts/prepare-release.sh`
- GitHub workflow: `.github/workflows/release.yml`
docs/content/docs/README.md
+67
View File
@@ -0,0 +1,67 @@
---
title: "Documentation README"
description: "Voicebox documentation development guide"
---
This directory contains the documentation for Voicebox, built with [Fumadocs](https://fumadocs.dev).
## Development
### Prerequisites
Install Mintlify globally using bun:
```bash
bun add -g mintlify
```
Or use the helper script:
```bash
bun run install:mintlify
```
### Running Locally
```bash
bun run dev
```
This will start the Mintlify dev server.
The docs will be available at `http://localhost:3000`
### Structure
```
docs/
├── mint.json # Mintlify configuration
├── custom.css # Custom styles
├── overview/ # Getting started & feature docs
├── guides/ # User guides
├── api/ # API reference
├── development/ # Developer documentation
├── logo/ # Logo assets
└── public/ # Static assets
```
### Writing Docs
- Use `.mdx` files for all documentation pages
- Follow the existing structure in `mint.json` for navigation
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
## Deployment
Docs are automatically deployed when changes are pushed to the main branch.
To manually deploy:
```bash
mintlify deploy
```
## Contributing
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
docs/content/docs/TROUBLESHOOTING.md
+311
View File
@@ -0,0 +1,311 @@
---
title: "Troubleshooting Guide"
description: "Common issues and solutions for Voicebox"
---
Common issues and solutions for Voicebox.
## Installation Issues
### macOS: "Voicebox cannot be opened because it is from an unidentified developer"
**Solution:**
1. Right-click the `.dmg` file
2. Select "Open"
3. Click "Open" in the security dialog
4. Alternatively, go to System Settings → Privacy & Security → Allow Voicebox
### Windows: "Windows protected your PC"
**Solution:**
1. Click "More info"
2. Click "Run anyway"
3. Windows Defender may flag new software; this is normal for unsigned apps
### Linux: AppImage won't run
**Solution:**
```bash
chmod +x voicebox-*.AppImage
./voicebox-*.AppImage
```
## Runtime Issues
### Server won't start
**Symptoms:** App opens but shows "Server not connected"
**Solutions:**
1. **Check Python installation**
```bash
python --version # Should be 3.11+
```
2. **Check server binary exists**
- Look in `tauri/src-tauri/binaries/` for your platform
- Binary should match your system architecture
3. **Check permissions**
```bash
# macOS/Linux
chmod +x tauri/src-tauri/binaries/voicebox-server-*
```
4. **Check logs**
- macOS: Open Console.app and search for "voicebox"
- Linux: Check `~/.local/share/voicebox/` for logs
- Windows: Check Event Viewer
### "Model download failed"
**Symptoms:** First generation fails with download error
**Solutions:**
1. **Check internet connection**
- Models download from HuggingFace Hub (~2-4GB)
- First download may take several minutes
2. **Check disk space**
- Models are cached in `~/.cache/huggingface/`
- Ensure at least 5GB free space
3. **Manual download** (if automatic fails)
```bash
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
```
### "Out of memory" errors
**Symptoms:** Generation fails with CUDA/VRAM errors
**Solutions:**
1. **Use smaller model**
- Switch to 0.6B model instead of 1.7B
- Settings → Model Management → Load 0.6B
2. **Close other applications**
- Free up GPU memory
- Close browser tabs, other ML apps
3. **Use CPU mode**
- Slower but works without GPU
- Backend automatically falls back to CPU
### MLX "Failed to load the default metallib" error (Apple Silicon)
**Symptoms:** Generation fails with "library not found" or "metallib" errors
**Solutions:**
1. **Rebuild server binary**
```bash
bun run build:server
```
The build script should automatically include MLX Metal shader libraries.
2. **Check MLX installation**
```bash
pip install -r backend/requirements-mlx.txt
```
3. **Verify backend detection**
- Check server logs for "Backend: MLX"
- If showing "Backend: PYTORCH", MLX may not be installed correctly
### Audio playback issues
**Symptoms:** Generated audio won't play
**Solutions:**
1. **Check audio format**
- Audio is saved as WAV files
- Ensure your system supports WAV playback
2. **Try downloading audio**
- Right-click → Download
- Play in external player
3. **Check browser permissions** (web version)
- Allow audio autoplay in browser settings
### Slow generation
**Symptoms:** Generation takes >30 seconds
**Solutions:**
1. **Check backend type** (Apple Silicon)
- Check Settings → Server Status
- Should show "Backend: MLX" on Apple Silicon
- If showing "Backend: PYTORCH", install MLX: `pip install -r backend/requirements-mlx.txt`
- MLX provides 4-5x faster inference on Apple Silicon
2. **Use GPU** (if available)
- Check Settings → Server Status
- Should show "GPU available: true"
- Apple Silicon: Should show "Metal (Apple Silicon via MLX)"
- Windows/Linux: Should show "CUDA" if GPU available
3. **Enable caching**
- Voice prompts are cached automatically
- Second generation with same voice should be faster
4. **Use smaller model**
- 0.6B model is faster than 1.7B
- Quality difference is minimal for most voices
5. **Check system resources**
- Close other CPU/GPU intensive apps
- Ensure adequate RAM (8GB+ recommended)
## API Issues
### "Connection refused" when using API
**Solutions:**
1. **Check server is running**
```bash
curl http://localhost:8000/health
```
2. **Check remote mode**
- If connecting remotely, ensure server is started with `--host 0.0.0.0`
- Check firewall settings
3. **Check port availability**
- Default port is 8000
- Ensure no other service is using it
### CORS errors in browser
**Solutions:**
1. **Use desktop app** (recommended)
- Desktop app doesn't have CORS restrictions
2. **Configure CORS** (for web deployment)
- Update `backend/main.py` CORS settings
- Add your domain to allowed origins
## Update Issues
### "Update check failed"
**Solutions:**
1. **Check internet connection**
- Updates are fetched from GitHub releases
2. **Check GitHub access**
- Ensure `github.com` is accessible
- Check firewall/proxy settings
3. **Manual update**
- Download latest release from GitHub
- Install manually
### "Invalid signature" error
**Solutions:**
1. **Re-download installer**
- Signature may be corrupted
- Download fresh copy from GitHub
2. **Check release integrity**
- Verify `.sig` file matches installer
- Report issue if signature is invalid
## Data Issues
### Profiles disappeared
**Solutions:**
1. **Check data directory**
- macOS: `~/Library/Application Support/voicebox/`
- Windows: `%APPDATA%/voicebox/`
- Linux: `~/.local/share/voicebox/`
2. **Check database**
- Database: `data/voicebox.db`
- Ensure file exists and is readable
3. **Restore from backup**
- Profiles can be exported/imported
- Check for backup files
### "Database locked" error
**Solutions:**
1. **Close other instances**
- Ensure only one Voicebox instance is running
2. **Restart app**
- Close and reopen Voicebox
3. **Check file permissions**
- Ensure database file is writable
- Check directory permissions
## Development Issues
### Build fails
**Solutions:**
1. **Check Rust installation**
```bash
rustc --version
rustup update
```
2. **Check Tauri dependencies**
```bash
cd tauri
bun install
```
3. **Clean build**
```bash
cd tauri/src-tauri
cargo clean
cd ../..
bun run build
```
### API client generation fails
**Solutions:**
1. **Start backend server**
```bash
bun run dev:server
```
2. **Check OpenAPI endpoint**
```bash
curl http://localhost:8000/openapi.json
```
3. **Regenerate client**
```bash
bun run generate:api
```
## Still Having Issues?
1. **Check existing issues**
- Search GitHub issues for similar problems
- Check closed issues for solutions
2. **Create new issue**
- Include:
- OS and version
- Voicebox version
- Steps to reproduce
- Error messages/logs
- Screenshots (if applicable)
3. **Get help**
- Check documentation in `docs/`
- Review `backend/README.md` for API details
- See `CONTRIBUTING.md` for development help
---
For more help, open an issue on [GitHub](https://github.com/jamiepine/voicebox/issues).
docs/content/docs/api-reference/meta.json
+4
View File
@@ -0,0 +1,4 @@
{
"title": "API Reference",
"pages": ["unknown"]
}
docs/content/docs/api-reference/unknown/add_profile_sample_profiles__profile_id__samples_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Add Profile Sample
description: Add a sample to a voice profile.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Add a sample to a voice profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/{profile_id}/samples","method":"post"}]} />
docs/content/docs/api-reference/unknown/create_profile_profiles_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Create Profile
description: Create a new voice profile.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Create a new voice profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles","method":"post"}]} />
docs/content/docs/api-reference/unknown/delete_generation_history__generation_id__delete.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Delete Generation
description: Delete a generation.
full: true
_openapi:
method: DELETE
toc: []
structuredData:
headings: []
contents:
- content: Delete a generation.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/history/{generation_id}","method":"delete"}]} />
docs/content/docs/api-reference/unknown/delete_profile_profiles__profile_id__delete.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Delete Profile
description: Delete a voice profile.
full: true
_openapi:
method: DELETE
toc: []
structuredData:
headings: []
contents:
- content: Delete a voice profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/{profile_id}","method":"delete"}]} />
docs/content/docs/api-reference/unknown/delete_profile_sample_profiles_samples__sample_id__delete.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Delete Profile Sample
description: Delete a profile sample.
full: true
_openapi:
method: DELETE
toc: []
structuredData:
headings: []
contents:
- content: Delete a profile sample.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/samples/{sample_id}","method":"delete"}]} />
docs/content/docs/api-reference/unknown/generate_speech_generate_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Generate Speech
description: Generate speech from text using a voice profile.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Generate speech from text using a voice profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/generate","method":"post"}]} />
docs/content/docs/api-reference/unknown/get_audio_audio__generation_id__get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Audio
description: Serve generated audio file.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Serve generated audio file.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/audio/{generation_id}","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_generation_history__generation_id__get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Generation
description: Get a generation by ID.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get a generation by ID.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/history/{generation_id}","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_model_progress_models_progress__model_name__get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Model Progress
description: Get model download progress via Server-Sent Events.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get model download progress via Server-Sent Events.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/models/progress/{model_name}","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_model_status_models_status_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Model Status
description: Get status of all available models.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get status of all available models.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/models/status","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_profile_profiles__profile_id__get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Profile
description: Get a voice profile by ID.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get a voice profile by ID.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/{profile_id}","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_profile_samples_profiles__profile_id__samples_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Profile Samples
description: Get all samples for a profile.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get all samples for a profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/{profile_id}/samples","method":"get"}]} />
docs/content/docs/api-reference/unknown/get_stats_history_stats_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Get Stats
description: Get generation statistics.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Get generation statistics.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/history/stats","method":"get"}]} />
docs/content/docs/api-reference/unknown/health_health_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Health
description: Health check endpoint.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Health check endpoint.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/health","method":"get"}]} />
docs/content/docs/api-reference/unknown/list_history_history_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: List History
description: List generation history with optional filters.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: List generation history with optional filters.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/history","method":"get"}]} />
docs/content/docs/api-reference/unknown/list_profiles_profiles_get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: List Profiles
description: List all voice profiles.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: List all voice profiles.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles","method":"get"}]} />
docs/content/docs/api-reference/unknown/load_model_models_load_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Load Model
description: Manually load TTS model.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Manually load TTS model.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/models/load","method":"post"}]} />
docs/content/docs/api-reference/unknown/meta.json
+27
View File
@@ -0,0 +1,27 @@
{
"title": "Endpoints",
"pages": [
"root__get",
"health_health_get",
"list_profiles_profiles_get",
"create_profile_profiles_post",
"get_profile_profiles__profile_id__get",
"update_profile_profiles__profile_id__put",
"delete_profile_profiles__profile_id__delete",
"add_profile_sample_profiles__profile_id__samples_post",
"get_profile_samples_profiles__profile_id__samples_get",
"delete_profile_sample_profiles_samples__sample_id__delete",
"generate_speech_generate_post",
"list_history_history_get",
"get_generation_history__generation_id__get",
"delete_generation_history__generation_id__delete",
"get_stats_history_stats_get",
"transcribe_audio_transcribe_post",
"get_audio_audio__generation_id__get",
"load_model_models_load_post",
"unload_model_models_unload_post",
"get_model_progress_models_progress__model_name__get",
"get_model_status_models_status_get",
"trigger_model_download_models_download_post"
]
}
docs/content/docs/api-reference/unknown/root__get.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Root
description: Root endpoint.
full: true
_openapi:
method: GET
toc: []
structuredData:
headings: []
contents:
- content: Root endpoint.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/","method":"get"}]} />
docs/content/docs/api-reference/unknown/transcribe_audio_transcribe_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Transcribe Audio
description: Transcribe audio file to text.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Transcribe audio file to text.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/transcribe","method":"post"}]} />
docs/content/docs/api-reference/unknown/trigger_model_download_models_download_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Trigger Model Download
description: Trigger download of a specific model.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Trigger download of a specific model.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/models/download","method":"post"}]} />
docs/content/docs/api-reference/unknown/unload_model_models_unload_post.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Unload Model
description: Unload TTS model to free memory.
full: true
_openapi:
method: POST
toc: []
structuredData:
headings: []
contents:
- content: Unload TTS model to free memory.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/models/unload","method":"post"}]} />
docs/content/docs/api-reference/unknown/update_profile_profiles__profile_id__put.mdx
+16
View File
@@ -0,0 +1,16 @@
---
title: Update Profile
description: Update a voice profile.
full: true
_openapi:
method: PUT
toc: []
structuredData:
headings: []
contents:
- content: Update a voice profile.
---
{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}
<APIPage document={"./openapi.json"} operations={[{"path":"/profiles/{profile_id}","method":"put"}]} />
docs/content/docs/developer/architecture.mdx
+206
View File
@@ -0,0 +1,206 @@
---
title: "Architecture"
description: "Understanding Voicebox's technical architecture"
---
## System Overview
Voicebox uses a client-server architecture with a React frontend and Python backend. The desktop app is built with Tauri and contains two main layers:
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
These two layers communicate via HTTP, with the frontend making API requests to the backend.
## Frontend Architecture
### Tech Stack
- **Framework**: React 18 with TypeScript
- **State Management**: Zustand stores
- **Data Fetching**: React Query (TanStack Query)
- **Styling**: Tailwind CSS
- **Audio**: WaveSurfer.js
- **Desktop**: Tauri (Rust)
### Component Structure
```
app/src/
├── components/ # React components
│ ├── profiles/ # Voice profile UI
│ ├── generation/ # Speech generation UI
│ ├── stories/ # Timeline editor
│ └── shared/ # Reusable components
├── lib/ # Utilities
│ ├── api/ # Generated API client
│ └── utils/ # Helper functions
├── hooks/ # React hooks
└── stores/ # Zustand state stores
```
### State Management
```typescript
// Example: Profile store
const useProfileStore = create((set) => ({
profiles: [],
selectedProfile: null,
setProfiles: (profiles) => set({ profiles }),
selectProfile: (id) => set({ selectedProfile: id })
}))
```
## Backend Architecture
### Tech Stack
- **Framework**: FastAPI (Python 3.11+)
- **TTS Model**: Qwen3-TTS
- **Transcription**: Whisper
- **Database**: SQLite
- **Audio**: librosa, soundfile
### API Structure
```python
# main.py - API routes
@app.post("/generate")
async def generate_speech(request: GenerateRequest):
# 1. Validate request
# 2. Load voice profile
# 3. Generate audio with TTS
# 4. Save to database
# 5. Return response
```
### Data Model
The database uses three main tables:
**Profile Table:** Stores voice profiles with fields for id, name, and language.
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
## Desktop App (Tauri)
### Rust Backend
```rust
// Sidecar process management
// File system access
// Native integrations
```
### Responsibilities
- Launch Python backend as sidecar process
- Native file dialogs
- System tray integration
- Auto-updates
- OS-specific features
## Build Process
### Development
```bash
# Frontend (Vite dev server)
cd app && bun run dev
# Backend (manual start)
cd backend && uvicorn main:app --reload
# Desktop app (connects to manual backend)
bun run dev
```
### Production
```bash
# Build everything (server binary + Tauri app)
bun run build
# Or build separately:
# 1. Build server binary (PyInstaller)
bun run build:server
# 2. Build Tauri app (includes server)
cd tauri && bun run tauri build
```
## Data Flow
### Generation Flow
When a user generates speech, the data flows through the following stages:
1. **User Input** - User enters text in a React component
2. **State Update** - Text is stored in Zustand state
3. **API Request** - React Query mutation triggers an API call via fetch
4. **Backend Processing** - FastAPI endpoint receives the request
5. **TTS Generation** - Qwen3-TTS model generates the audio
6. **Storage** - Audio file is saved to disk and a database record is created
7. **Response** - Backend returns the audio URL
8. **Cache Update** - React Query updates its cache with the response
9. **UI Update** - Component re-renders with new data
10. **Playback** - User can play the generated audio
## Performance Considerations
### Frontend
- **Code splitting** - Lazy load routes
- **Memoization** - React.memo for heavy components
- **Virtual scrolling** - For large lists
- **Debouncing** - Search and input handling
### Backend
- **Async operations** - All I/O is async
- **Model caching** - Keep TTS model in memory
- **Voice prompt caching** - Reuse embeddings
- **Connection pooling** - Database connections
## Security
### Current
- Local-only by default
- No authentication (localhost trust)
- File system sandboxing via Tauri
### Planned
- API key authentication
- User accounts
- Rate limiting
- HTTPS support
## Deployment Modes
### Local Mode
- Backend runs as sidecar
- All data stays on device
- No network required
### Remote Mode
- Backend on separate machine
- Frontend connects via HTTP
- Shared infrastructure possible
## Next Steps
<CardGroup cols={2}>
<Card title="Development Setup" icon="code" href="/development/setup">
Set up your dev environment
</Card>
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
Contribute to Voicebox
</Card>
</CardGroup>
docs/content/docs/developer/audio-channels.mdx
+310
View File
@@ -0,0 +1,310 @@
---
title: "Audio Channels"
description: "How audio output routing works in Voicebox"
---
## Overview
Audio channels allow routing voice output to different audio devices. This is useful for multi-output setups where different voices should play through different speakers or applications.
## Architecture
**Channel:** A named audio bus that can be assigned to output devices.
**Device Mapping:** Links channels to OS audio device identifiers.
**Profile Mapping:** Links voice profiles to channels (many-to-many).
## Data Model
### AudioChannel Table
```python
class AudioChannel(Base):
__tablename__ = "audio_channels"
id = Column(String, primary_key=True)
name = Column(String, nullable=False)
is_default = Column(Boolean, default=False)
created_at = Column(DateTime)
```
### ChannelDeviceMapping Table
```python
class ChannelDeviceMapping(Base):
__tablename__ = "channel_device_mappings"
id = Column(String, primary_key=True)
channel_id = Column(String, ForeignKey("audio_channels.id"))
device_id = Column(String) # OS device identifier
```
### ProfileChannelMapping Table
```python
class ProfileChannelMapping(Base):
__tablename__ = "profile_channel_mappings"
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
```
## Default Channel
A default channel is created on database initialization:
```python
def init_db():
# Create default channel if it doesn't exist
default_channel = db.query(AudioChannel).filter(
AudioChannel.is_default == True
).first()
if not default_channel:
default_channel = AudioChannel(
id=str(uuid.uuid4()),
name="Default",
is_default=True
)
db.add(default_channel)
# Assign all existing profiles to default channel
profiles = db.query(VoiceProfile).all()
for profile in profiles:
mapping = ProfileChannelMapping(
profile_id=profile.id,
channel_id=default_channel.id
)
db.add(mapping)
```
## Core Operations
### Creating a Channel
```python
async def create_channel(
data: AudioChannelCreate,
db: Session,
) -> AudioChannelResponse:
# Check name uniqueness
existing = db.query(DBAudioChannel).filter_by(name=data.name).first()
if existing:
raise ValueError(f"Channel with name '{data.name}' already exists")
# Create channel
channel = DBAudioChannel(
id=str(uuid.uuid4()),
name=data.name,
is_default=False,
)
db.add(channel)
# Add device mappings
for device_id in data.device_ids:
mapping = DBChannelDeviceMapping(
id=str(uuid.uuid4()),
channel_id=channel.id,
device_id=device_id,
)
db.add(mapping)
db.commit()
```
### Updating a Channel
```python
async def update_channel(
channel_id: str,
data: AudioChannelUpdate,
db: Session,
) -> AudioChannelResponse:
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
# Cannot modify default channel
if channel.is_default:
raise ValueError("Cannot modify the default channel")
# Update name
if data.name is not None:
channel.name = data.name
# Update device mappings
if data.device_ids is not None:
# Delete existing
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
# Add new
for device_id in data.device_ids:
mapping = DBChannelDeviceMapping(
channel_id=channel.id,
device_id=device_id,
)
db.add(mapping)
db.commit()
```
### Deleting a Channel
```python
async def delete_channel(channel_id: str, db: Session) -> bool:
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
# Cannot delete default channel
if channel.is_default:
raise ValueError("Cannot delete the default channel")
# Delete device mappings
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
# Delete profile-channel mappings
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
# Delete channel
db.delete(channel)
db.commit()
```
## Voice Assignment
### Assigning Voices to Channel
```python
async def set_channel_voices(
channel_id: str,
data: ChannelVoiceAssignment,
db: Session,
) -> None:
# Verify channel exists
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
if not channel:
raise ValueError(f"Channel {channel_id} not found")
# Verify all profiles exist
for profile_id in data.profile_ids:
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile {profile_id} not found")
# Delete existing mappings
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
# Add new mappings
for profile_id in data.profile_ids:
mapping = DBProfileChannelMapping(
profile_id=profile_id,
channel_id=channel_id,
)
db.add(mapping)
db.commit()
```
### Assigning Channels to Voice
```python
async def set_profile_channels(
profile_id: str,
data: ProfileChannelAssignment,
db: Session,
) -> None:
# Verify profile exists
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile {profile_id} not found")
# Delete existing mappings
db.query(DBProfileChannelMapping).filter_by(profile_id=profile_id).delete()
# Add new mappings
for channel_id in data.channel_ids:
mapping = DBProfileChannelMapping(
profile_id=profile_id,
channel_id=channel_id,
)
db.add(mapping)
db.commit()
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/channels` | List all channels |
| POST | `/channels` | Create a channel |
| GET | `/channels/{id}` | Get channel by ID |
| PUT | `/channels/{id}` | Update channel |
| DELETE | `/channels/{id}` | Delete channel |
| GET | `/channels/{id}/voices` | Get assigned voices |
| PUT | `/channels/{id}/voices` | Set assigned voices |
| GET | `/profiles/{id}/channels` | Get profile's channels |
| PUT | `/profiles/{id}/channels` | Set profile's channels |
## Request/Response Schemas
### AudioChannelCreate
```json
{
"name": "Speakers",
"device_ids": ["device_uuid_1", "device_uuid_2"]
}
```
### AudioChannelResponse
```json
{
"id": "channel_uuid",
"name": "Speakers",
"is_default": false,
"device_ids": ["device_uuid_1", "device_uuid_2"],
"created_at": "2024-01-15T10:30:00Z"
}
```
### ChannelVoiceAssignment
```json
{
"profile_ids": ["profile_1", "profile_2"]
}
```
## Use Cases
### Multi-Output Setup
**Scenario:** Stream with different voice characters
1. Create "Stream" channel → OBS virtual audio
2. Create "Monitor" channel → Headphones
3. Assign "Narrator" profile → Both channels
4. Assign "Character 1" profile → Stream only
### Virtual Audio Cables
Common device IDs for virtual audio:
- VB-Audio Virtual Cable
- BlackHole (macOS)
- Soundflower (macOS)
## Frontend Integration
The frontend needs to:
1. **Enumerate devices** using Web Audio API or Tauri
2. **Display channel list** with device assignments
3. **Allow profile assignment** via drag/drop or dropdown
4. **Route playback** to correct device based on profile's channel
## Limitations
- Device IDs are OS-specific
- Hot-plugging may invalidate device IDs
- Default channel cannot be modified/deleted
- Frontend handles actual audio routing (backend just stores config)
docs/content/docs/developer/autoupdater.mdx
+84
View File
@@ -0,0 +1,84 @@
---
title: "Auto-Updater"
description: "Configure and use the Tauri auto-updater"
---
## Overview
Voicebox uses Tauri's built-in auto-updater to deliver updates to users automatically.
## Quick Reference
For detailed setup instructions, see the existing documentation:
- [AUTOUPDATER_QUICKSTART.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER_QUICKSTART.md)
- [AUTOUPDATER.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER.md)
## How It Works
The auto-updater follows a secure update process:
1. **Check for Updates** - The Voicebox app periodically checks GitHub Releases for new versions
2. **Download Update** - If a new version is found, the update package is downloaded
3. **Verify Signature** - The downloaded package is cryptographically verified using the public key
4. **Install** - After verification, the update is installed
5. **Restart** - The app restarts with the new version
## Configuration
Updates are configured in `tauri/src-tauri/tauri.conf.json`:
```json
{
"updater": {
"active": true,
"endpoints": [
"https://github.com/jamiepine/voicebox/releases/latest/download/latest.json"
],
"dialog": true,
"pubkey": "YOUR_PUBLIC_KEY"
}
}
```
## Generating Keys
```bash
# Generate signing keys
bun run generate:keys
# Keys saved to ~/.tauri/voicebox.key
```
<Warning>
Keep your private key secure! Never commit it to the repository.
</Warning>
## Release Process
1. **Bump version** using bumpversion
2. **Push tag** to trigger CI/CD
3. **GitHub Actions** builds and signs releases
4. **Users** receive update notification
## User Experience
When an update is available:
1. User sees a notification dialog
2. User clicks "Update"
3. Update downloads in background
4. App restarts with new version
## For Developers
See the full documentation files for:
- Setting up signing keys
- Configuring GitHub releases
- Testing updates locally
- Troubleshooting update failures
<Card title="View Full Docs" href="https://github.com/jamiepine/voicebox/tree/main/docs">
Access AUTOUPDATER.md and AUTOUPDATER_QUICKSTART.md in the repository
</Card>
docs/content/docs/developer/building.mdx
+270
View File
@@ -0,0 +1,270 @@
---
title: "Building"
description: "Build Voicebox for production"
---
## Overview
Voicebox uses a multi-step build process to create platform-specific installers.
## Quick Build
```bash
# Build for your current platform (automatically builds server binary first)
make build
# Or manually
bun run build
```
This automatically:
1. Builds the Python server binary (`bun run build:server`)
2. Builds the Tauri app (`cd tauri && bun run tauri build`)
## Build Process
The build process consists of two steps, but `bun run build` handles both automatically:
### 1. Server Binary Build (Automatic)
The Python backend is compiled into a standalone executable using PyInstaller. This happens automatically when you run `bun run build`.
**Platform-specific binaries:**
- macOS (Apple Silicon): `voicebox-server-aarch64-apple-darwin` (includes MLX backend)
- macOS (Intel): `voicebox-server-x86_64-apple-darwin` (PyTorch backend)
- Windows: `voicebox-server-x86_64-pc-windows-msvc.exe` (PyTorch backend)
- Linux: `voicebox-server-x86_64-unknown-linux-gnu` (PyTorch backend)
<Note>
The build script automatically detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
</Note>
**Manual build (if needed):**
```bash
bun run build:server
```
### 2. Tauri App Build (Automatic)
The Tauri app build is also handled automatically, which:
1. Builds the React frontend (Vite)
2. Compiles the Rust backend
3. Bundles the server binary as a sidecar
4. Creates platform-specific installers
**Manual build (if needed):**
```bash
cd tauri && bun run tauri build
```
### 3. Output
Installers are created in `tauri/src-tauri/target/release/bundle/`:
**macOS:**
- `dmg/` - Disk image installer
- `macos/` - App bundle
**Windows:**
- `msi/` - MSI installer
- `nsis/` - NSIS installer
**Linux:**
- `deb/` - Debian package
- `appimage/` - AppImage
## Advanced Options
### Building for Specific Platform
```bash
# Build for macOS (Apple Silicon)
bun run tauri build -- --target aarch64-apple-darwin
# Build for macOS (Intel)
bun run tauri build -- --target x86_64-apple-darwin
# Build for Windows
bun run tauri build -- --target x86_64-pc-windows-msvc
# Build for Linux
bun run tauri build -- --target x86_64-unknown-linux-gnu
```
### Using Local Qwen3-TTS
If you're developing Qwen3-TTS locally:
```bash
export QWEN_TTS_PATH=~/path/to/Qwen3-TTS
bun run build:server # Build server binary only
# or
bun run build # Build everything
```
This makes PyInstaller use your local version instead of the pip package.
### Debug Build
```bash
cd tauri
bun run tauri build --debug
```
Creates a debug build with symbols and logging.
## Build Configuration
### Tauri Config
Edit `tauri/src-tauri/tauri.conf.json`:
```json
{
"bundle": {
"identifier": "com.voicebox.app",
"icon": [
"icons/32x32.png",
"icons/128x128.png",
"icons/icon.icns",
"icons/icon.ico"
]
}
}
```
### Sidecar Configuration
The Python server is bundled as a sidecar:
```json
{
"tauri": {
"bundle": {
"externalBin": [
"binaries/voicebox-server"
]
}
}
}
```
## Code Signing
### macOS
To sign the app for distribution:
```bash
# Set signing identity
export APPLE_SIGNING_IDENTITY="Developer ID Application: Your Name"
# Build with signing
bun run tauri build
```
For notarization:
```bash
# Set credentials
export APPLE_ID="[email protected]"
export APPLE_PASSWORD="app-specific-password"
# Build and notarize
bun run tauri build
```
### Windows
For Windows code signing:
```bash
# Set certificate
export WINDOWS_CERTIFICATE_PATH="/path/to/cert.pfx"
export WINDOWS_CERTIFICATE_PASSWORD="password"
# Build with signing
bun run tauri build
```
## Release Process
The full release process is automated:
```bash
# 1. Bump version
bumpversion patch # or minor/major
# 2. Build all platforms (CI/CD handles this)
git push --tags
# 3. GitHub Actions creates releases
```
See [CONTRIBUTING.md](/development/contributing) for the full release workflow.
## Troubleshooting
<AccordionGroup>
<Accordion title="Server Binary Build Fails">
**Common issues:**
- Missing Python dependencies: `pip install -r requirements.txt`
- PyInstaller not found: `pip install pyinstaller`
- Qwen3-TTS not installed: `pip install git+https://github.com/QwenLM/Qwen3-TTS.git`
**Solution:**
```bash
cd backend
source venv/bin/activate
pip install -r requirements.txt
pip install pyinstaller
```
</Accordion>
<Accordion title="Tauri Build Fails">
**Common issues:**
- Rust not installed: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
- Server binary missing: Usually auto-built, but can run manually: `./scripts/build-server.sh`
- Node modules outdated: `bun install`
**Solution:**
```bash
# Clean and rebuild
cd tauri/src-tauri
cargo clean
cd ../..
bun run build # Automatically builds server binary first
```
</Accordion>
<Accordion title="App Won't Launch After Build">
**Check:**
- Server binary has execute permissions
- All dependencies are bundled
- Check logs in the app's data directory
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
## CI/CD
GitHub Actions automatically builds releases when tags are pushed:
```yaml
# .github/workflows/release.yml
on:
push:
tags:
- 'v*'
```
See the [repository](https://github.com/jamiepine/voicebox) for the full CI/CD configuration.
docs/content/docs/developer/contributing.mdx
+330
View File
@@ -0,0 +1,330 @@
---
title: "Contributing"
description: "How to contribute to Voicebox"
---
Thank you for your interest in contributing to Voicebox! This guide will help you get started.
## Code of Conduct
- Be respectful and inclusive
- Welcome newcomers and help them learn
- Focus on constructive feedback
- Respect different viewpoints and experiences
## Getting Started
Before you start contributing, make sure you have:
1. **Read the documentation** to understand how Voicebox works
2. **Set up your development environment** - see [Development Setup](/development/setup)
3. **Explored the codebase** to understand the project structure
4. **Checked existing issues** to see if someone else is working on something similar
## Ways to Contribute
<CardGroup cols={2}>
<Card title="Report Bugs" icon="bug">
Found a bug? Open an issue with reproduction steps
</Card>
<Card title="Request Features" icon="lightbulb">
Have an idea? Start a discussion or open an issue
</Card>
<Card title="Improve Docs" icon="book">
Fix typos, add examples, or clarify instructions
</Card>
<Card title="Write Code" icon="code">
Fix bugs, add features, or optimize performance
</Card>
</CardGroup>
## Development Workflow
### 1. Fork & Clone
```bash
# Fork the repository on GitHub
# Then clone your fork
git clone https://github.com/YOUR_USERNAME/voicebox.git
cd voicebox
```
### 2. Create a Branch
Use descriptive branch names:
```bash
# For features
git checkout -b feature/voice-effects
# For bug fixes
git checkout -b fix/audio-playback-issue
# For documentation
git checkout -b docs/api-examples
```
### 3. Make Your Changes
Follow these guidelines:
<AccordionGroup>
<Accordion title="Code Style">
**TypeScript/React:**
- Use TypeScript strict mode
- Prefer functional components with hooks
- Use named exports
- Format with Biome (runs automatically)
**Python:**
- Follow PEP 8
- Use type hints
- Use async/await for I/O
- Document functions with docstrings
**Rust:**
- Follow Rust conventions
- Use meaningful names
- Handle errors explicitly
- Run `rustfmt`
</Accordion>
<Accordion title="Commit Messages">
Write clear, descriptive commit messages:
```bash
# Good
git commit -m "Add voice profile export feature"
git commit -m "Fix audio playback stopping after 30 seconds"
# Avoid
git commit -m "Update code"
git commit -m "Fix bug"
```
Format:
- Use imperative mood ("Add feature" not "Added feature")
- Keep first line under 50 characters
- Add detailed description if needed
</Accordion>
<Accordion title="Testing">
- Test your changes manually in the app
- Ensure backend API endpoints work
- Check for TypeScript/Python errors
- Verify UI components render correctly
- Add automated tests when possible
</Accordion>
</AccordionGroup>
### 4. Push & Create PR
```bash
# Push your branch
git push origin feature/your-feature-name
# Then create a pull request on GitHub
```
## Pull Request Guidelines
When creating a pull request:
<Steps>
<Step title="Use a Clear Title">
Examples:
- "Add voice profile export functionality"
- "Fix audio playback stopping after 30 seconds"
- "Improve generation speed with caching"
</Step>
<Step title="Provide Description">
Include:
- What changes you made
- Why you made them
- How to test them
- Screenshots (for UI changes)
- Reference related issues
</Step>
<Step title="Update Documentation">
- Update relevant docs if behavior changes
- Add API documentation for new endpoints
- Update README if needed
</Step>
<Step title="Check the Checklist">
- [ ] Code follows style guidelines
- [ ] Documentation updated
- [ ] Changes tested
- [ ] No breaking changes (or documented)
- [ ] CHANGELOG.md updated
</Step>
</Steps>
## Project Structure
Understanding the codebase:
```
voicebox/
├── app/ # Shared React frontend
│ ├── src/
│ │ ├── components/ # UI components
│ │ ├── lib/ # Utilities and API client
│ │ ├── hooks/ # React hooks
│ │ └── stores/ # Zustand state stores
├── backend/ # Python FastAPI server
│ ├── main.py # API routes
│ ├── tts.py # Voice synthesis logic
│ ├── database.py # SQLite operations
│ └── models.py # Pydantic models
├── tauri/ # Desktop app wrapper
│ └── src-tauri/ # Rust backend
├── web/ # Web deployment
├── landing/ # Marketing website
└── scripts/ # Build & release scripts
```
## Areas for Contribution
### Bug Fixes
- Check [existing issues](https://github.com/jamiepine/voicebox/issues) for bugs
- Test your fix thoroughly
- Add regression tests if possible
### New Features
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
- Discuss major features in an issue first
- Keep features focused and well-scoped
### Documentation
- Improve clarity and fix typos
- Add code examples
- Create tutorials or guides
- Document API endpoints
### UI/UX Improvements
- Improve accessibility
- Enhance visual design
- Optimize performance
- Add animations/transitions
### Infrastructure
- Improve build process
- Add CI/CD improvements
- Optimize bundle size
- Add testing infrastructure
## API Development
When adding new API endpoints:
<Steps>
<Step title="Add Route">
In `backend/main.py`:
```python
@app.post("/api/new-endpoint")
async def new_endpoint(data: RequestModel) -> ResponseModel:
"""Endpoint description."""
# Implementation
return response
```
</Step>
<Step title="Create Models">
In `backend/models.py`:
```python
class RequestModel(BaseModel):
field: str
class ResponseModel(BaseModel):
result: str
```
</Step>
<Step title="Regenerate Client">
```bash
bun run generate:api
```
This updates the TypeScript client with type-safe bindings.
</Step>
<Step title="Update Docs">
The API documentation is automatically generated from the OpenAPI schema. Ensure your endpoint has proper docstrings and type hints, then regenerate the docs:
```bash
bun run generate:api
```
</Step>
</Steps>
## Testing
Currently testing is primarily manual. When adding tests:
**Backend:**
```bash
cd backend
pytest
```
**Frontend:**
```bash
bun run test
```
**E2E (future):**
```bash
bun run test:e2e
```
## Release Process
Releases are managed by maintainers using `bumpversion`:
```bash
# Bump version (patch, minor, or major)
bumpversion patch
# Push with tags
git push && git push --tags
```
GitHub Actions automatically builds and publishes releases when tags are pushed.
## Community
- **GitHub Issues:** Bug reports and feature requests
- **GitHub Discussions:** General questions and ideas
- **Discord:** Real-time chat (coming soon)
## Recognition
Contributors are recognized in:
- [CHANGELOG.md](https://github.com/jamiepine/voicebox/blob/main/CHANGELOG.md)
- GitHub contributor list
- Release notes
## License
By contributing, you agree that your contributions will be licensed under the MIT License.
## Questions?
If you have questions:
1. Check the [documentation](/overview/introduction)
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
3. Open a new issue or discussion
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
Thank you for contributing to Voicebox! 🎉
docs/content/docs/developer/history.mdx
+260
View File
@@ -0,0 +1,260 @@
---
title: "Generation History"
description: "How generation history tracking works in Voicebox"
---
## Overview
The history module tracks all generated audio, providing a searchable record of past generations. Each generation stores the text, settings, and a reference to the audio file.
## Data Model
### Generation Table
```python
class Generation(Base):
__tablename__ = "generations"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
text = Column(Text, nullable=False)
language = Column(String, default="en")
audio_path = Column(String, nullable=False)
duration = Column(Float, nullable=False)
seed = Column(Integer)
instruct = Column(Text)
created_at = Column(DateTime)
```
## File Storage
Generated audio is stored in:
```
data/
└── generations/
└── {generation_id}.wav
```
## Core Functions
### Creating a Generation Record
After TTS generates audio, a history entry is created:
```python
async def create_generation(
profile_id: str,
text: str,
language: str,
audio_path: str,
duration: float,
seed: Optional[int],
db: Session,
instruct: Optional[str] = None,
) -> GenerationResponse:
db_generation = DBGeneration(
id=str(uuid.uuid4()),
profile_id=profile_id,
text=text,
language=language,
audio_path=audio_path,
duration=duration,
seed=seed,
instruct=instruct,
created_at=datetime.utcnow(),
)
db.add(db_generation)
db.commit()
return GenerationResponse.model_validate(db_generation)
```
### Listing Generations
Supports filtering and pagination:
```python
async def list_generations(
query: HistoryQuery,
db: Session,
) -> HistoryListResponse:
# Build query with profile name join
q = db.query(
DBGeneration,
DBVoiceProfile.name.label('profile_name')
).join(
DBVoiceProfile,
DBGeneration.profile_id == DBVoiceProfile.id
)
# Apply filters
if query.profile_id:
q = q.filter(DBGeneration.profile_id == query.profile_id)
if query.search:
q = q.filter(DBGeneration.text.like(f"%{query.search}%"))
# Order and paginate
total = q.count()
q = q.order_by(DBGeneration.created_at.desc())
q = q.offset(query.offset).limit(query.limit)
return HistoryListResponse(items=results, total=total)
```
### Getting Statistics
Aggregate statistics for the dashboard:
```python
async def get_generation_stats(db: Session) -> dict:
total = db.query(func.count(DBGeneration.id)).scalar()
total_duration = db.query(func.sum(DBGeneration.duration)).scalar()
by_profile = db.query(
DBGeneration.profile_id,
func.count(DBGeneration.id).label('count')
).group_by(DBGeneration.profile_id).all()
return {
"total_generations": total,
"total_duration_seconds": total_duration,
"generations_by_profile": {
profile_id: count for profile_id, count in by_profile
},
}
```
## Deletion
Deleting a generation removes both the database record and audio file:
```python
async def delete_generation(generation_id: str, db: Session) -> bool:
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
if not generation:
return False
# Delete audio file
audio_path = Path(generation.audio_path)
if audio_path.exists():
audio_path.unlink()
# Delete database record
db.delete(generation)
db.commit()
return True
```
### Cascade Delete
When deleting a profile, all its generations are also deleted:
```python
async def delete_generations_by_profile(profile_id: str, db: Session) -> int:
generations = db.query(DBGeneration).filter_by(profile_id=profile_id).all()
for generation in generations:
Path(generation.audio_path).unlink(missing_ok=True)
db.delete(generation)
db.commit()
return len(generations)
```
## Export/Import
### Exporting a Generation
Generations can be exported as ZIP archives:
```
generation_export.zip
├── generation.json # Metadata
└── audio.wav # Audio file
```
### Importing a Generation
The import process:
1. Extract ZIP archive
2. Validate metadata and audio
3. Create new generation ID
4. Copy audio to generations directory
5. Create database record
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/history` | List generations with filters |
| GET | `/history/stats` | Get aggregate statistics |
| GET | `/history/{id}` | Get generation by ID |
| DELETE | `/history/{id}` | Delete generation |
| GET | `/history/{id}/export` | Export as ZIP |
| GET | `/history/{id}/export-audio` | Export audio only |
| POST | `/history/import` | Import from ZIP |
### Query Parameters
```
GET /history?profile_id=uuid&search=hello&limit=50&offset=0
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `profile_id` | string | null | Filter by profile |
| `search` | string | null | Search in text |
| `limit` | int | 50 | Results per page |
| `offset` | int | 0 | Pagination offset |
### Response Schema
```json
{
"items": [
{
"id": "uuid",
"profile_id": "uuid",
"profile_name": "My Voice",
"text": "Hello world",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 1.5,
"seed": 42,
"instruct": null,
"created_at": "2024-01-15T10:30:00Z"
}
],
"total": 150
}
```
## Usage in Stories
Generations can be added to stories for multi-voice narratives. The story system references generations by ID:
```python
class StoryItem(Base):
generation_id = Column(String, ForeignKey("generations.id"))
```
This allows the same generation to be reused across multiple stories without duplicating audio files.
## Storage Considerations
### Disk Usage
Each generation creates a WAV file. For a 10-second clip at 24kHz:
- ~480KB per file (mono, 16-bit)
### Cleanup Strategy
Consider implementing:
- Automatic cleanup of old generations
- Storage quota per profile
- Compression for archival
docs/content/docs/developer/meta.json
+17
View File
@@ -0,0 +1,17 @@
{
"title": "Developer",
"pages": [
"setup",
"architecture",
"contributing",
"building",
"autoupdater",
"voice-profiles",
"tts-generation",
"history",
"stories",
"transcription",
"audio-channels",
"model-management"
]
}
docs/content/docs/developer/model-management.mdx
+341
View File
@@ -0,0 +1,341 @@
---
title: "Model Management"
description: "How model downloading, loading, and status tracking works in Voicebox"
---
## Overview
Voicebox manages two types of models:
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
**ASR Models:** Whisper for transcription (tiny through large).
Models are downloaded from HuggingFace Hub on first use and cached locally.
## Available Models
### TTS Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
### Whisper Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
| base | `openai/whisper-base` | ~300MB | ~1GB |
| small | `openai/whisper-small` | ~500MB | ~2GB |
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
| large | `openai/whisper-large` | ~3GB | ~10GB |
## Model Storage
Models are cached in the HuggingFace cache directory:
```
~/.cache/huggingface/hub/
├── models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/
├── models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/
├── models--openai--whisper-base/
└── ...
```
## Progress Tracking
### Progress Manager
Tracks download progress across all models:
```python
class ProgressManager:
def __init__(self):
self._progress = {} # model_name -> progress_info
def update_progress(
self,
model_name: str,
current: int,
total: int,
filename: str,
status: str,
):
self._progress[model_name] = {
"current": current,
"total": total,
"filename": filename,
"status": status, # downloading, complete, error
"updated_at": datetime.utcnow(),
}
def get_progress(self, model_name: str) -> Optional[dict]:
return self._progress.get(model_name)
```
### HuggingFace Progress Callback
Hooks into HuggingFace's download system:
```python
class HFProgressTracker:
def __init__(self, callback):
self.callback = callback
@contextmanager
def patch_download(self):
"""Context manager to intercept HF downloads."""
original_download = hf_hub_download
def patched_download(*args, **kwargs):
# Intercept progress
result = original_download(*args, **kwargs)
self.callback(progress_info)
return result
# Apply patch
with patch('huggingface_hub.hf_hub_download', patched_download):
yield
```
### Server-Sent Events (SSE)
Progress is streamed to the frontend:
```python
@app.get("/models/progress/{model_name}")
async def get_model_progress(model_name: str):
async def event_generator():
while True:
progress = progress_manager.get_progress(model_name)
if progress:
yield f"data: {json.dumps(progress)}\n\n"
if progress and progress["status"] in ["complete", "error"]:
break
await asyncio.sleep(0.5)
return StreamingResponse(
event_generator(),
media_type="text/event-stream"
)
```
## Task Manager
Tracks active downloads and generations:
```python
class TaskManager:
def __init__(self):
self._active_downloads = {}
self._active_generations = {}
def start_download(self, model_name: str):
self._active_downloads[model_name] = {
"status": "downloading",
"started_at": datetime.utcnow(),
}
def complete_download(self, model_name: str):
if model_name in self._active_downloads:
del self._active_downloads[model_name]
def get_active_tasks(self) -> dict:
return {
"downloads": list(self._active_downloads.values()),
"generations": list(self._active_generations.values()),
}
```
## Model Status
Check which models are downloaded and loaded:
```python
@app.get("/models/status")
async def get_model_status() -> ModelStatusListResponse:
models = []
# Check TTS models
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
downloaded = is_model_downloaded(hf_id)
loaded = tts_model._current_model_size == size
models.append(ModelStatus(
model_name=f"qwen-tts-{size}",
display_name=f"Qwen3-TTS {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=loaded,
))
# Check Whisper models
for size in ["tiny", "base", "small", "medium", "large"]:
hf_id = f"openai/whisper-{size}"
downloaded = is_model_downloaded(hf_id)
models.append(ModelStatus(
model_name=f"whisper-{size}",
display_name=f"Whisper {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=False, # Whisper is loaded on-demand
))
return ModelStatusListResponse(models=models)
```
## Manual Model Operations
### Load Model
```python
@app.post("/models/load")
async def load_model(model_size: str = "1.7B"):
tts_model = get_tts_model()
await tts_model.load_model_async(model_size)
return {"status": "loaded", "model_size": model_size}
```
### Unload Model
```python
@app.post("/models/unload")
async def unload_model():
tts_model = get_tts_model()
tts_model.unload_model()
return {"status": "unloaded"}
```
### Trigger Download
```python
@app.post("/models/download")
async def trigger_model_download(request: ModelDownloadRequest):
# This triggers the download in background
# Progress is tracked via /models/progress/{model_name}
if request.model_name.startswith("qwen-tts"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_tts_model(size))
elif request.model_name.startswith("whisper"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_whisper_model(size))
return {"status": "downloading"}
```
### Delete Model
```python
@app.delete("/models/{model_name}")
async def delete_model(model_name: str):
# Find and delete from HuggingFace cache
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
for model_dir in model_dirs:
shutil.rmtree(model_dir)
return {"status": "deleted"}
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Get status of all models |
| POST | `/models/load` | Load TTS model |
| POST | `/models/unload` | Unload TTS model |
| POST | `/models/download` | Trigger model download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete downloaded model |
| GET | `/tasks/active` | Get active downloads/generations |
## Response Schemas
### ModelStatus
```json
{
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen3-TTS 1.7B",
"downloaded": true,
"size_mb": 3400,
"loaded": true
}
```
### ActiveTasksResponse
```json
{
"downloads": [
{
"model_name": "whisper-medium",
"status": "downloading",
"started_at": "2024-01-15T10:30:00Z"
}
],
"generations": [
{
"task_id": "uuid",
"profile_id": "uuid",
"text_preview": "Hello world...",
"started_at": "2024-01-15T10:30:00Z"
}
]
}
```
## Frontend Integration
### Progress Display
```typescript
// Subscribe to download progress via SSE
const eventSource = new EventSource(`/models/progress/${modelName}`);
eventSource.onmessage = (event) => {
const progress = JSON.parse(event.data);
updateProgressBar(progress.current / progress.total);
if (progress.status === 'complete') {
eventSource.close();
}
};
```
### Model Status UI
```typescript
// Fetch model status
const { data: models } = useQuery({
queryKey: ['models', 'status'],
queryFn: () => api.getModelStatus(),
});
// Display download/load buttons based on status
models.map(model => (
<ModelCard
name={model.display_name}
downloaded={model.downloaded}
loaded={model.loaded}
onDownload={() => triggerDownload(model.model_name)}
onLoad={() => loadModel(model.model_name)}
/>
));
```
## Error Handling
| Error | Cause | Solution |
|-------|-------|----------|
| Download failed | Network issue | Retry download |
| OOM on load | Model too large | Use smaller model |
| Model not found | Cache corrupted | Re-download |
| Slow download | HF rate limit | Wait and retry |
docs/content/docs/developer/setup.mdx
+239
View File
@@ -0,0 +1,239 @@
---
title: "Development Setup"
description: "Set up your local development environment for Voicebox"
---
## Prerequisites
Before you begin, ensure you have the following installed:
<CardGroup cols={3}>
<Card title="Bun" icon="package">
[Download Bun](https://bun.sh)
```bash
curl -fsSL https://bun.sh/install | bash
```
</Card>
<Card title="Python 3.11+" icon="python">
[Download Python](https://python.org)
```bash
python --version
```
</Card>
<Card title="Rust" icon="rust">
[Install Rust](https://rustup.rs)
```bash
rustc --version
```
</Card>
</CardGroup>
## Clone the Repository
```bash
git clone https://github.com/jamiepine/voicebox.git
cd voicebox
```
## Quick Setup (Recommended)
The easiest way to get started is using the Makefile:
```bash
# Setup everything
make setup
# Start development
make dev
```
<Note>
The Makefile is available on macOS and Linux. Windows users should follow the manual setup below.
</Note>
## Manual Setup
### 1. Install JavaScript Dependencies
```bash
bun install
```
This installs dependencies for:
- `app/` - Shared React frontend
- `tauri/` - Tauri desktop wrapper
- `web/` - Web deployment wrapper
### 2. Set Up Python Backend
```bash
cd backend
# Create virtual environment
python -m venv venv
# Activate virtual environment
source venv/bin/activate # macOS/Linux
# or
venv\Scripts\activate # Windows
# Install Python dependencies
pip install -r requirements.txt
# Install MLX dependencies (Apple Silicon only - for faster inference)
# On Apple Silicon, this enables native Metal acceleration
if [[ $(uname -m) == "arm64" ]]; then
pip install -r requirements-mlx.txt
fi
# Install Qwen3-TTS
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
## Running in Development
Development requires **two terminals**: one for the Python backend, one for the Tauri app.
<Tabs>
<Tab title="Terminal 1: Backend">
Start the Python server first:
```bash
cd backend
source venv/bin/activate # Activate venv
bun run dev:server
```
Or manually:
```bash
uvicorn main:app --reload --port 17493
```
Backend will be available at `http://localhost:17493`
</Tab>
<Tab title="Terminal 2: Desktop App">
Then start the Tauri app:
```bash
bun run dev
```
This will:
- Create a placeholder sidecar binary
- Start Vite dev server on port 5173
- Launch Tauri window
- Enable hot reload
</Tab>
</Tabs>
<Info>
In dev mode, the app connects to your manually-started Python server. The bundled server binary is only used in production builds.
</Info>
### Optional: Web App
```bash
bun run dev:web
```
Web app will be available at `http://localhost:5174`
## Model Downloads
Models are automatically downloaded from HuggingFace Hub on first use:
- **Whisper** (transcription): Auto-downloads on first transcription
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
<Warning>
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
</Warning>
## Project Structure
```
voicebox/
├── app/ # Shared React frontend
│ └── src/
│ ├── components/ # UI components
│ ├── lib/ # Utilities and API client
│ └── hooks/ # React hooks
├── backend/ # Python FastAPI server
│ ├── main.py # API routes
│ ├── tts.py # Voice synthesis
│ └── database.py # SQLite operations
├── tauri/ # Desktop app wrapper
│ └── src-tauri/ # Rust backend
├── web/ # Web deployment
├── landing/ # Marketing website
└── scripts/ # Build & release scripts
```
## Available Make Commands
Run `make help` to see all available commands:
```bash
make setup # Install all dependencies
make dev # Start development servers
make dev-web # Start web development server
make build # Build desktop app
make build-web # Build web app
make clean # Clean build artifacts
make test # Run tests
```
## Generate OpenAPI Client
After starting the backend server, generate the TypeScript API client:
```bash
./scripts/generate-api.sh
# or
bun run generate:api
```
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
## Next Steps
<CardGroup cols={2}>
<Card title="Architecture" icon="diagram-project" href="/development/architecture">
Understand the system architecture
</Card>
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
Read the contribution guidelines
</Card>
<Card title="Building" icon="hammer" href="/development/building">
Learn how to build production releases
</Card>
<Card title="API Reference" icon="code" href="/api-reference">
Explore the REST API
</Card>
</CardGroup>
## Troubleshooting
<AccordionGroup>
<Accordion title="Backend won't start">
- Check Python version (must be 3.11+)
- Ensure virtual environment is activated
- Verify all dependencies are installed: `pip install -r requirements.txt`
- Check if port 17493 is available
</Accordion>
<Accordion title="Tauri build fails">
- Ensure Rust is installed: `rustc --version`
- Clean the build: `cd tauri/src-tauri && cargo clean`
- Try rebuilding: `bun run dev`
</Accordion>
<Accordion title="OpenAPI client generation fails">
- Ensure backend is running: `curl http://localhost:17493/openapi.json`
- Check network connectivity
- Verify the backend is accessible at localhost:17493
</Accordion>
</AccordionGroup>
See the full [Troubleshooting Guide](/guides/troubleshooting) for more issues and solutions.
docs/content/docs/developer/stories.mdx
+320
View File
@@ -0,0 +1,320 @@
---
title: "Stories & Timeline"
description: "How the multi-voice timeline editor works in Voicebox"
---
## Overview
Stories allow users to arrange multiple voice generations on a timeline to create multi-voice narratives. The system supports tracks, trimming, splitting, and audio mixing.
## Architecture
**Story:** A container that holds story items with metadata.
**Story Item:** Links a generation to a story with timeline position, track, and trim data.
**Export:** Combines all items into a single mixed audio file.
## Data Model
### Story Table
```python
class Story(Base):
__tablename__ = "stories"
id = Column(String, primary_key=True)
name = Column(String, nullable=False)
description = Column(Text)
created_at = Column(DateTime)
updated_at = Column(DateTime)
```
### StoryItem Table
```python
class StoryItem(Base):
__tablename__ = "story_items"
id = Column(String, primary_key=True)
story_id = Column(String, ForeignKey("stories.id"))
generation_id = Column(String, ForeignKey("generations.id"))
start_time_ms = Column(Integer, default=0) # Timeline position
track = Column(Integer, default=0) # Track number
trim_start_ms = Column(Integer, default=0) # Trim from start
trim_end_ms = Column(Integer, default=0) # Trim from end
created_at = Column(DateTime)
```
## Timeline Concepts
### Start Time
`start_time_ms` defines when an item begins on the timeline:
```
Timeline (ms): 0----1000----2000----3000----4000
Item 1: [======]
Item 2: [==========]
Item 3: [====]
```
### Tracks
Multiple tracks allow overlapping audio:
```
Track 0: [Item 1] [Item 3]
Track 1: [Item 2]
```
### Trimming
Trim values cut audio from the start or end without destroying the original:
```
Original: [=========AUDIO=========]
trim_start: ^^
trim_end: ^^
Result: [=====AUDIO=====]
```
## Core Operations
### Adding Items
When adding a generation to a story:
```python
async def add_item_to_story(
story_id: str,
data: StoryItemCreate,
db: Session,
) -> StoryItemDetail:
# Calculate start time if not provided
if data.start_time_ms is None:
# Find the end of all existing items
existing_items = get_items_with_durations(story_id, db)
max_end_time_ms = max(
item.start_time_ms + int(gen.duration * 1000)
for item, gen in existing_items
)
start_time_ms = max_end_time_ms + 200 # 200ms gap
# Create the item
item = DBStoryItem(
id=str(uuid.uuid4()),
story_id=story_id,
generation_id=data.generation_id,
start_time_ms=start_time_ms,
track=data.track or 0,
)
db.add(item)
db.commit()
```
### Moving Items
Update position and/or track:
```python
async def move_story_item(
story_id: str,
item_id: str,
data: StoryItemMove,
db: Session,
) -> StoryItemDetail:
item = get_item(story_id, item_id, db)
item.start_time_ms = data.start_time_ms
item.track = data.track
db.commit()
```
### Trimming Items
Non-destructive trimming:
```python
async def trim_story_item(
story_id: str,
item_id: str,
data: StoryItemTrim,
db: Session,
) -> StoryItemDetail:
item = get_item(story_id, item_id, db)
generation = get_generation(item.generation_id, db)
# Validate trim doesn't exceed duration
max_duration_ms = int(generation.duration * 1000)
if data.trim_start_ms + data.trim_end_ms >= max_duration_ms:
return None # Invalid trim
item.trim_start_ms = data.trim_start_ms
item.trim_end_ms = data.trim_end_ms
db.commit()
```
### Splitting Items
Split one item into two at a specific time:
```python
async def split_story_item(
story_id: str,
item_id: str,
data: StoryItemSplit,
db: Session,
) -> List[StoryItemDetail]:
item = get_item(story_id, item_id, db)
generation = get_generation(item.generation_id, db)
# Calculate split point
current_trim_start = item.trim_start_ms
current_trim_end = item.trim_end_ms
original_duration_ms = int(generation.duration * 1000)
absolute_split_ms = current_trim_start + data.split_time_ms
# Update original: trim from end
item.trim_end_ms = original_duration_ms - absolute_split_ms
# Create new item: trim from start
new_item = DBStoryItem(
generation_id=item.generation_id, # Same generation
start_time_ms=item.start_time_ms + data.split_time_ms,
track=item.track,
trim_start_ms=absolute_split_ms,
trim_end_ms=current_trim_end,
)
db.add(new_item)
db.commit()
return [item, new_item]
```
### Duplicating Items
Create a copy with all properties:
```python
async def duplicate_story_item(
story_id: str,
item_id: str,
db: Session,
) -> StoryItemDetail:
original = get_item(story_id, item_id, db)
generation = get_generation(original.generation_id, db)
# Calculate effective duration for positioning
effective_duration_ms = (
int(generation.duration * 1000)
- original.trim_start_ms
- original.trim_end_ms
)
# Place copy after original with 200ms gap
new_item = DBStoryItem(
generation_id=original.generation_id,
start_time_ms=original.start_time_ms + effective_duration_ms + 200,
track=original.track,
trim_start_ms=original.trim_start_ms,
trim_end_ms=original.trim_end_ms,
)
db.add(new_item)
db.commit()
```
## Audio Export
### Mixing Algorithm
The export function mixes all items into a single audio file:
```python
async def export_story_audio(story_id: str, db: Session) -> bytes:
items = get_all_items_with_generations(story_id, db)
# Calculate total duration
max_end_time_ms = max(
data['start_time_ms'] + data['duration_ms']
for data in audio_data
)
# Create output buffer
total_samples = int((max_end_time_ms / 1000.0) * sample_rate)
final_audio = np.zeros(total_samples, dtype=np.float32)
# Mix each item at its position
for data in audio_data:
audio = data['audio']
start_sample = int((data['start_time_ms'] / 1000.0) * sample_rate)
# Apply trim
trimmed_audio = audio[trim_start_sample:len(audio) - trim_end_sample]
# Add to buffer (overlapping items sum together)
final_audio[start_sample:start_sample + len(trimmed_audio)] += trimmed_audio
# Normalize to prevent clipping
max_val = np.abs(final_audio).max()
if max_val > 1.0:
final_audio = final_audio / max_val
return audio_to_bytes(final_audio, sample_rate)
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/stories` | List all stories |
| POST | `/stories` | Create a story |
| GET | `/stories/{id}` | Get story with items |
| PUT | `/stories/{id}` | Update story metadata |
| DELETE | `/stories/{id}` | Delete story |
| POST | `/stories/{id}/items` | Add item to story |
| DELETE | `/stories/{id}/items/{item_id}` | Remove item |
| PUT | `/stories/{id}/items/{item_id}/move` | Move item |
| PUT | `/stories/{id}/items/{item_id}/trim` | Trim item |
| POST | `/stories/{id}/items/{item_id}/split` | Split item |
| POST | `/stories/{id}/items/{item_id}/duplicate` | Duplicate item |
| PUT | `/stories/{id}/items/times` | Batch update times |
| PUT | `/stories/{id}/items/reorder` | Reorder items |
| GET | `/stories/{id}/export-audio` | Export mixed audio |
## Response Schemas
### StoryItemDetail
```json
{
"id": "item_uuid",
"story_id": "story_uuid",
"generation_id": "generation_uuid",
"start_time_ms": 1500,
"track": 0,
"trim_start_ms": 200,
"trim_end_ms": 100,
"profile_id": "profile_uuid",
"profile_name": "Narrator",
"text": "Hello world",
"audio_path": "/path/to/audio.wav",
"duration": 2.5,
"created_at": "2024-01-15T10:30:00Z"
}
```
## Frontend Integration
The timeline UI needs to:
1. **Fetch story** with all items
2. **Render waveforms** for each item
3. **Handle drag/drop** to move items
4. **Handle edge drag** for trimming
5. **Sync playhead** across all tracks
6. **Export** when user clicks download
docs/content/docs/developer/transcription.mdx
+299
View File
@@ -0,0 +1,299 @@
---
title: "Transcription"
description: "How Whisper-based audio transcription works in Voicebox"
---
## Overview
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
## Architecture
The transcription system is built around the `WhisperModel` class:
**Model Loading:** Lazy loading with HuggingFace Hub download.
**Audio Processing:** Resampling and preprocessing for Whisper.
**Inference:** Running transcription with optional language hints.
## WhisperModel Class
```python
class WhisperModel:
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
```
### Model Sizes
| Size | Parameters | VRAM | Speed | Quality |
|------|------------|------|-------|---------|
| tiny | 39M | ~1GB | Fastest | Basic |
| base | 74M | ~1GB | Fast | Good |
| small | 244M | ~2GB | Medium | Better |
| medium | 769M | ~5GB | Slow | High |
| large | 1550M | ~10GB | Slowest | Best |
Default is `base` for balance of speed and quality.
## Model Loading
Models are downloaded from HuggingFace Hub:
```python
def load_model(self, model_size: Optional[str] = None):
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
# Track download progress
progress_manager = get_progress_manager()
task_manager = get_task_manager()
task_manager.start_download(f"whisper-{model_size}")
# Load processor and model
with tracker.patch_download():
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
# Mark complete
progress_manager.mark_complete(f"whisper-{model_size}")
task_manager.complete_download(f"whisper-{model_size}")
```
### Async Loading
Like TTS, loading runs in a thread pool:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self.model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Transcription
### Basic Transcription
```python
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
await self.load_model_async()
def _transcribe_sync():
# Load and resample to 16kHz (Whisper requirement)
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language hint if provided
forced_decoder_ids = None
if language:
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language,
task="transcribe",
)
# Generate
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
return await asyncio.to_thread(_transcribe_sync)
```
### Supported Languages
Whisper supports 99+ languages. Common ones in Voicebox:
| Code | Language |
|------|----------|
| en | English |
| zh | Chinese |
| ja | Japanese |
| ko | Korean |
| de | German |
| fr | French |
| ru | Russian |
| pt | Portuguese |
| es | Spanish |
| it | Italian |
### Language Detection
When no language is specified, Whisper auto-detects:
```python
# Without language hint - auto-detect
transcription = await whisper.transcribe(audio_path)
# With language hint - more accurate for short clips
transcription = await whisper.transcribe(audio_path, language="en")
```
## Transcription with Timestamps
For advanced use cases, word-level timestamps are available:
```python
async def transcribe_with_timestamps(
self,
audio_path: str,
language: Optional[str] = None,
) -> List[Dict[str, any]]:
await self.load_model_async()
def _transcribe_timestamps_sync():
audio, sr = load_audio(audio_path, sample_rate=16000)
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
return_timestamps=True,
)
# Parse timestamps
return [
{
"text": transcription,
"start": 0.0,
"end": len(audio) / sr,
}
]
return await asyncio.to_thread(_transcribe_timestamps_sync)
```
## Memory Management
### Unloading
Free memory when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Global Instance
A singleton pattern manages the model:
```python
_whisper_model: Optional[WhisperModel] = None
def get_whisper_model() -> WhisperModel:
global _whisper_model
if _whisper_model is None:
_whisper_model = WhisperModel()
return _whisper_model
```
## Audio Preprocessing
### Resampling
Whisper requires 16kHz audio:
```python
audio, sr = load_audio(audio_path, sample_rate=16000)
```
### Format Support
The `load_audio` utility handles:
- WAV
- MP3
- FLAC
- OGG
- M4A
All formats are converted to mono 16kHz.
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/transcribe` | Transcribe audio file |
### Request
Multipart form data:
```
POST /transcribe
Content-Type: multipart/form-data
file: <audio_file>
language: en (optional)
```
### Response
```json
{
"text": "Hello, this is a test transcription.",
"duration": 3.5
}
```
## Use Cases
### Reference Text for Voice Cloning
1. User records audio sample
2. Audio is sent to `/transcribe`
3. Transcription becomes `reference_text`
4. Both are added to voice profile
### Quality Tips
- Provide language hint for short audio
- Use clean audio with minimal noise
- Longer audio (>5s) improves accuracy
- Consider `small` or `medium` model for better quality
## Error Handling
Common issues:
| Error | Cause | Solution |
|-------|-------|----------|
| Model not found | First run, download failed | Retry with network |
| OOM | Model too large | Use smaller model |
| Empty result | No speech detected | Check audio has speech |
| Wrong language | Auto-detect failed | Provide language hint |
docs/content/docs/developer/tts-generation.mdx
+283
View File
@@ -0,0 +1,283 @@
---
title: "TTS Generation"
description: "How text-to-speech generation works in Voicebox"
---
## Overview
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
## Architecture
The TTS system is built around the `TTSModel` class which manages:
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
**Voice Prompts:** Converting reference audio into embeddings.
**Generation:** Synthesizing speech from text using voice prompts.
## TTSModel Class
```python
class TTSModel:
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device() # cuda, mps, or cpu
```
### Device Selection
The model automatically selects the best available device:
```python
def _get_device(self) -> str:
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return "cpu" # MPS can have issues, use CPU for stability
return "cpu"
```
## Model Loading
Models are downloaded from HuggingFace Hub on first use:
```python
def load_model(self, model_size: Optional[str] = None):
# Model IDs on HuggingFace Hub
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
# Load with progress tracking
with tracker.patch_download():
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16, # float32 on CPU
)
```
### Async Loading
Loading runs in a thread pool to avoid blocking the event loop:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self._current_model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Voice Prompt Creation
Voice prompts are created from reference audio and cached for reuse:
```python
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
await self.load_model_async()
# Check cache
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached = get_cached_voice_prompt(cache_key)
if cached:
return cached, True
# Create prompt (blocking, run in thread pool)
voice_prompt = await asyncio.to_thread(
self.model.create_voice_clone_prompt,
ref_audio=audio_path,
ref_text=reference_text,
)
# Cache the result
cache_voice_prompt(cache_key, voice_prompt)
return voice_prompt, False
```
### Combining Multiple Samples
When a profile has multiple samples, they're combined:
```python
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate and normalize
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
```
## Speech Generation
The core generation function:
```python
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
await self.load_model_async()
def _generate_sync():
# Set seed for reproducibility
if seed is not None:
torch.manual_seed(seed)
# Generate audio
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct, # Natural language delivery control
)
return wavs[0], sample_rate
# Run in thread pool
return await asyncio.to_thread(_generate_sync)
```
### Instruct Feature
The `instruct` parameter allows natural language control over speech delivery:
```python
# Examples:
instruct = "Speak slowly and clearly"
instruct = "Sound excited and enthusiastic"
instruct = "Whisper softly"
```
## Caching Strategy
Voice prompts are cached to avoid recomputation:
```python
def get_cache_key(audio_path: str, reference_text: str) -> str:
"""Generate cache key from audio hash and text."""
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
return f"{audio_hash}_{text_hash}"
```
Cache is stored in `data/cache/voice_prompts/`.
## Memory Management
### Unloading Models
Free VRAM/RAM when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
self.model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Model Switching
When switching between model sizes (1.7B ↔ 0.6B):
```python
# Unload existing model first
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
```
## Generation Flow
1. **Request** → Validate text and profile ID
2. **Profile** → Load profile samples from database
3. **Voice Prompt** → Create or retrieve cached prompt
4. **Generate** → Run TTS inference
5. **Save** → Write audio to generations directory
6. **Record** → Create history entry in database
7. **Response** → Return audio path and metadata
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/generate` | Generate speech from text |
| GET | `/audio/{id}` | Serve generated audio file |
### Request Schema
```json
{
"profile_id": "uuid",
"text": "Text to synthesize",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "Speak clearly"
}
```
### Response Schema
```json
{
"id": "generation_uuid",
"profile_id": "profile_uuid",
"text": "Text to synthesize",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 3.5,
"seed": 42,
"instruct": "Speak clearly",
"created_at": "2024-01-15T10:30:00Z"
}
```
## Performance Considerations
### GPU Acceleration
- CUDA provides fastest inference
- MPS (Apple Silicon) has stability issues, uses CPU fallback
- CPU inference is slower but always works
### Batch Size
Currently generates one utterance at a time. For long texts, consider:
- Splitting into sentences
- Sequential generation
- Concatenating results
### Memory Usage
| Model | VRAM/RAM Required |
|-------|-------------------|
| 0.6B | ~2GB |
| 1.7B | ~6GB |
docs/content/docs/developer/voice-profiles.mdx
+202
View File
@@ -0,0 +1,202 @@
---
title: "Voice Profiles"
description: "How voice profile management works in Voicebox"
---
## Overview
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
## Architecture
The voice profile system consists of three main components:
**Database Layer:** SQLite tables store profile metadata and sample references.
**File Storage:** Audio samples are stored on disk in a structured directory format.
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
## Data Model
### VoiceProfile Table
```python
class VoiceProfile(Base):
__tablename__ = "profiles"
id = Column(String, primary_key=True)
name = Column(String, unique=True, nullable=False)
description = Column(Text)
language = Column(String, default="en")
created_at = Column(DateTime)
updated_at = Column(DateTime)
```
### ProfileSample Table
```python
class ProfileSample(Base):
__tablename__ = "profile_samples"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
audio_path = Column(String, nullable=False)
reference_text = Column(Text, nullable=False)
```
## File Structure
Profiles are stored in the data directory:
```
data/
└── profiles/
└── {profile_id}/
├── {sample_id_1}.wav
├── {sample_id_2}.wav
└── ...
```
## Core Functions
### Creating a Profile
```python
async def create_profile(data: VoiceProfileCreate, db: Session) -> VoiceProfileResponse:
# 1. Create database record
db_profile = DBVoiceProfile(
id=str(uuid.uuid4()),
name=data.name,
description=data.description,
language=data.language,
)
db.add(db_profile)
db.commit()
# 2. Create profile directory
profile_dir = profiles_dir / db_profile.id
profile_dir.mkdir(parents=True, exist_ok=True)
return VoiceProfileResponse.model_validate(db_profile)
```
### Adding Samples
When a sample is added, the audio is validated and copied to the profile directory:
```python
async def add_profile_sample(
profile_id: str,
audio_path: str,
reference_text: str,
db: Session,
) -> ProfileSampleResponse:
# 1. Validate audio (duration, format, quality)
is_valid, error_msg = validate_reference_audio(audio_path)
if not is_valid:
raise ValueError(f"Invalid reference audio: {error_msg}")
# 2. Copy to profile directory
sample_id = str(uuid.uuid4())
dest_path = profile_dir / f"{sample_id}.wav"
audio, sr = load_audio(audio_path)
save_audio(audio, str(dest_path), sr)
# 3. Create database record
db_sample = DBProfileSample(
id=sample_id,
profile_id=profile_id,
audio_path=str(dest_path),
reference_text=reference_text,
)
db.add(db_sample)
db.commit()
```
### Voice Prompt Creation
When generating speech, samples are combined into a voice prompt:
```python
async def create_voice_prompt_for_profile(
profile_id: str,
db: Session,
) -> dict:
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
if len(samples) == 1:
# Single sample - use directly
voice_prompt, _ = await tts_model.create_voice_prompt(
sample.audio_path,
sample.reference_text,
)
else:
# Multiple samples - combine them
combined_audio, combined_text = await tts_model.combine_voice_prompts(
[s.audio_path for s in samples],
[s.reference_text for s in samples],
)
voice_prompt, _ = await tts_model.create_voice_prompt(
combined_audio_path,
combined_text,
)
return voice_prompt
```
## Audio Validation
Reference audio is validated before being accepted:
- **Duration:** 3-30 seconds recommended
- **Format:** WAV, MP3, FLAC, OGG supported
- **Sample Rate:** Resampled to 24kHz
- **Channels:** Converted to mono if stereo
## Export/Import
Profiles can be exported as ZIP archives for sharing:
```
profile_export.zip
├── profile.json # Metadata
├── samples/
│ ├── sample_1.wav
│ └── sample_1.json # Reference text
└── ...
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/profiles` | List all profiles |
| POST | `/profiles` | Create a profile |
| GET | `/profiles/{id}` | Get profile by ID |
| PUT | `/profiles/{id}` | Update profile |
| DELETE | `/profiles/{id}` | Delete profile |
| GET | `/profiles/{id}/samples` | Get profile samples |
| POST | `/profiles/{id}/samples` | Add sample to profile |
| PUT | `/profiles/samples/{id}` | Update sample text |
| DELETE | `/profiles/samples/{id}` | Delete sample |
| GET | `/profiles/{id}/export` | Export as ZIP |
| POST | `/profiles/import` | Import from ZIP |
## Best Practices
### Sample Quality
- Use clean audio with minimal background noise
- Ensure the reference text exactly matches what is spoken
- Multiple samples (3-5) improve voice cloning quality
### Language Matching
- Set the profile language to match the reference audio
- Supported languages: en, zh, ja, ko, de, fr, ru, pt, es, it
### Naming Conventions
- Use descriptive names that identify the voice
- Avoid special characters that may cause filesystem issues
docs/content/docs/index.mdx
+50
View File
@@ -0,0 +1,50 @@
---
title: "Voicebox Documentation"
description: "Welcome to Voicebox - the open-source voice synthesis studio"
---
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
<Frame>
<img src="/images/app-screenshot-1.webp" alt="Voicebox App Screenshot" />
</Frame>
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
- **Complete privacy** — models and voice data stay on your machine
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
- **Native performance** — built with Tauri (Rust), not Electron
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
## Key Features
<CardGroup cols={2}>
<Card title="Voice Cloning" icon="microphone">
Instant cloning from just a few seconds of audio with Qwen3-TTS
</Card>
<Card title="Stories Editor" icon="film">
Multi-track timeline for creating conversations and narratives
</Card>
<Card title="Full API" icon="code">
REST API for integrating voice synthesis into your apps
</Card>
<Card title="Local-First" icon="shield">
Everything runs on your machine - complete privacy
</Card>
</CardGroup>
## Get Started
<CardGroup cols={2}>
<Card title="Installation" icon="download" href="/docs/overview/installation">
Download and install Voicebox on your machine
</Card>
<Card title="Quick Start" icon="rocket" href="/docs/overview/quick-start">
Get up and running in 5 minutes
</Card>
</CardGroup>
docs/content/docs/meta.json
+9
View File
@@ -0,0 +1,9 @@
{
"title": "Voicebox Documentation",
"pages": [
"overview",
"api-reference",
"developer",
"plans"
]
}
docs/content/docs/overview/building-stories.mdx
+37
View File
@@ -0,0 +1,37 @@
---
title: "Building Stories"
description: "Create multi-voice narratives with the Stories Editor"
---
## Getting Started
The Stories Editor is perfect for creating podcasts, audiobooks, and multi-speaker content.
<Steps>
<Step title="Create Story">
**Stories** → **+ New Story**
</Step>
<Step title="Add Tracks">
Create tracks for each speaker
</Step>
<Step title="Add Clips">
Generate or drag audio to tracks
</Step>
<Step title="Arrange">
Position and trim clips on timeline
</Step>
<Step title="Export">
Render final audio
</Step>
</Steps>
## Use Cases
- Multi-host podcasts
- Audiobook narration with character voices
- Game dialogue scenes
- Educational content with multiple speakers
## Coming Soon
Full timeline editor documentation will be added as features are finalized.
docs/content/docs/overview/creating-voice-profiles.mdx
+296
View File
@@ -0,0 +1,296 @@
---
title: "Creating Voice Profiles"
description: "Advanced guide to creating high-quality voice profiles"
---
## Overview
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
## Quick Start
<Steps>
<Step title="Prepare Audio">
10-30 seconds of clear speech
</Step>
<Step title="Create Profile">
**Profiles** → **+ New Profile**
</Step>
<Step title="Upload Sample">
Add your audio file
</Step>
<Step title="Generate">
Use the profile to generate speech
</Step>
</Steps>
## Audio Requirements
### Ideal Sample Characteristics
<CardGroup cols={2}>
<Card title="Duration" icon="clock">
**10-30 seconds**
Too short: Poor quality
Too long: Unnecessary
</Card>
<Card title="Clarity" icon="volume">
**Clear speech**
No background noise
No music or overlapping voices
</Card>
<Card title="Quality" icon="sparkles">
**High fidelity**
44.1kHz or 48kHz sample rate
Minimal compression
</Card>
<Card title="Content" icon="microphone">
**Natural speech**
Conversational tone
Complete sentences
</Card>
</CardGroup>
### File Formats
Supported formats:
- **WAV** (recommended) - Lossless quality
- **MP3** - Acceptable, minimal compression
- **M4A** - Acceptable
- **FLAC** - Lossless alternative
<Tip>
Use WAV for best results. Avoid heavily compressed formats.
</Tip>
## Recording Tips
### Environment
<AccordionGroup>
<Accordion title="Quiet Space">
- Record in a quiet room
- Turn off fans, AC, appliances
- Close windows to reduce outside noise
- Use soft furnishings to reduce echo
</Accordion>
<Accordion title="Microphone Placement">
- 6-12 inches from mouth
- Slight angle to reduce plosives (p, b, t)
- Use a pop filter if available
- Maintain consistent distance
</Accordion>
<Accordion title="Recording Settings">
- 44.1kHz or 48kHz sample rate
- 16-bit or 24-bit depth
- Mono is fine (stereo will be converted)
- Avoid automatic gain control
</Accordion>
</AccordionGroup>
### Speaking
- **Natural pace** - Don't rush or speak too slowly
- **Clear articulation** - Pronounce words clearly
- **Consistent volume** - Maintain steady loudness
- **Normal tone** - Speak as you normally would
- **Complete sentences** - Avoid fragments or "ums"
## Multiple Samples
Adding multiple samples can significantly improve quality:
### Why Multiple Samples?
<CardGroup cols={2}>
<Card title="Robustness" icon="shield">
Model learns a more complete representation
</Card>
<Card title="Versatility" icon="palette">
Handles different speaking styles better
</Card>
<Card title="Quality" icon="star">
Reduces artifacts and improves naturalness
</Card>
<Card title="Consistency" icon="check">
More reliable across different texts
</Card>
</CardGroup>
### Sample Variety
Consider adding samples with:
1. **Different tones**
- Casual conversation
- Professional/formal
- Excited/enthusiastic
- Calm/serious
2. **Different content**
- Narratives
- Questions
- Statements
- Emotions (happy, sad, neutral)
3. **Different recording conditions**
- Studio quality
- Phone call quality (if needed)
- Room acoustics
<Warning>
All samples should be from the **same speaker**. Mixing voices will produce poor results.
</Warning>
## Processing Existing Audio
If you have existing audio (podcasts, videos, etc.):
### Extracting Clean Segments
<Steps>
<Step title="Find Clean Speech">
Look for segments with:
- Just the target speaker
- No background music
- Minimal noise
</Step>
<Step title="Use Audio Editor">
Tools like Audacity or Adobe Audition:
- Cut out clean 10-30s segments
- Remove silence at start/end
- Normalize volume if needed
</Step>
<Step title="Export as WAV">
Save as high-quality WAV file
</Step>
</Steps>
### Noise Reduction
If you have light background noise:
```
1. Use noise reduction in Audacity:
- Select noise-only section
- Get Noise Profile
- Select full audio
- Apply noise reduction (gentle settings)
2. Avoid over-processing:
- Can introduce artifacts
- May reduce voice quality
```
## Testing & Iteration
### Test Your Profile
After creating a profile:
<Steps>
<Step title="Generate Test">
Generate a simple phrase:
```
"Hello, this is a test of my voice profile."
```
</Step>
<Step title="Evaluate Quality">
Listen for:
- Natural tone
- Clear pronunciation
- Proper prosody
- Lack of artifacts
</Step>
<Step title="Iterate">
If quality is poor:
- Add more samples
- Try different source audio
- Check sample quality
</Step>
</Steps>
### Common Issues
<AccordionGroup>
<Accordion title="Robotic Voice">
**Cause**: Poor quality samples or too short
**Fix**: Use longer, higher quality samples
</Accordion>
<Accordion title="Wrong Tone">
**Cause**: Sample tone doesn't match desired output
**Fix**: Record samples in the style you want to generate
</Accordion>
<Accordion title="Artifacts/Glitches">
**Cause**: Background noise or audio issues in samples
**Fix**: Clean up samples or re-record in quieter environment
</Accordion>
</AccordionGroup>
## Advanced Tips
### Celebrity/Character Voices
For cloning public figures or characters:
1. **Legal considerations** - Ensure you have rights or it's fair use
2. **Source quality** - Find high-quality interview audio or clean clips
3. **Consistency** - Use clips where they speak similarly
4. **Multiple samples** - Very important for recognizable voices
### Accent & Dialect
The model will preserve accent and dialect:
- British English will generate British English
- Southern accent will produce Southern accent
- Regional pronunciations will be maintained
### Emotion Transfer
The emotional tone of samples affects generation:
- Energetic samples → Energetic output
- Calm samples → Calm output
- Mix samples for versatile profile
## Managing Profiles
### Organization
- **Descriptive names** - "John Smith - Professional Narrator"
- **Add descriptions** - Note recording conditions, use cases
- **Language tags** - Mark the primary language
- **Archive unused** - Keep profile list manageable
### Export/Import
- **Export** profiles to share or backup
- **Import** from colleagues or teammates
- Profiles include voice embeddings, not original audio
## Next Steps
<CardGroup cols={2}>
<Card title="Generate Speech" icon="waveform" href="/overview/generating-speech">
Use your profile to generate speech
</Card>
<Card title="Build Stories" icon="film" href="/overview/building-stories">
Create multi-voice narratives
</Card>
</CardGroup>
docs/content/docs/overview/generating-speech.mdx
+65
View File
@@ -0,0 +1,65 @@
---
title: "Generating Speech"
description: "Generate high-quality speech from text"
---
## Basic Generation
<Steps>
<Step title="Select Profile">
Choose a voice profile from the dropdown
</Step>
<Step title="Enter Text">
Type or paste your text
</Step>
<Step title="Generate">
Click **Generate** and wait a few seconds
</Step>
<Step title="Play & Export">
Preview and download the result
</Step>
</Steps>
## Text Formatting Tips
The way you format text affects the output quality.
### Punctuation
Use proper punctuation for natural pauses:
```
Good: "Hello! How are you today? I'm doing great."
Bad: "Hello how are you today Im doing great"
```
### Emphasis
Use formatting to suggest emphasis:
```
- ALL CAPS for louder/emphasized: "That was AMAZING!"
- Italics for subtle emphasis: "I *really* enjoyed that"
- Bold for strong emphasis: "This is **very** important"
```
<Note>
The model interprets these hints but results may vary.
</Note>
## Advanced Features
### Batch Generation
For long-form content, split into smaller chunks for better control and faster processing.
### Voice Caching
Voicebox caches voice prompts for faster re-generation with the same profile.
## Coming Soon
- Real-time streaming
- Word-level timing control
- Emotion and style controls
- SSML support
docs/content/docs/overview/generation-history.mdx
+88
View File
@@ -0,0 +1,88 @@
---
title: "Generation History"
description: "Track and manage all your generated audio"
---
## Overview
Voicebox keeps a complete history of all generated audio, making it easy to find, reuse, and manage your creations.
## Features
<CardGroup cols={2}>
<Card title="Full History" icon="clock">
Every generation is automatically saved
</Card>
<Card title="Search & Filter" icon="search">
Find by text, voice, or date
</Card>
<Card title="Re-generate" icon="rotate">
Regenerate any past generation with one click
</Card>
<Card title="Export" icon="download">
Download individual or batch exports
</Card>
</CardGroup>
## Viewing History
Navigate to the **History** tab to see all your generations.
Each entry shows:
- Generated text
- Voice profile used
- Timestamp
- Audio duration
- Language
## Actions
### Play
Click any generation to play it immediately.
### Re-generate
Regenerate with the same settings or modify the text/voice.
### Download
Export as WAV, MP3, or M4A.
### Delete
Remove unwanted generations to free up space.
### Add to Story
Drag generations to the Stories Editor timeline.
## Search & Filter
<Tabs>
<Tab title="By Text">
Search for specific text content
```
"Hello world"
```
</Tab>
<Tab title="By Voice">
Filter by voice profile
```
Select from dropdown
```
</Tab>
<Tab title="By Date">
Filter by date range
```
Last 7 days, Last 30 days, Custom range
```
</Tab>
</Tabs>
## Storage
History is stored locally:
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
- **Linux**: `~/.config/com.voicebox.app/data/`
<Warning>
Deleting the data directory will remove all history. Export important files first.
</Warning>
docs/content/docs/overview/installation.mdx
+119
View File
@@ -0,0 +1,119 @@
---
title: "Installation"
description: "Download and install Voicebox on macOS, Windows, or Linux"
---
## Download
Voicebox is available for macOS and Windows, with Linux builds coming soon.
<CardGroup cols={2}>
<Card title="macOS" icon="apple">
Download for Apple Silicon or Intel Macs
</Card>
<Card title="Windows" icon="windows">
Download MSI installer or Setup executable
</Card>
</CardGroup>
### macOS
<Tabs>
<Tab title="Apple Silicon">
Download: [voicebox_aarch64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_aarch64.app.tar.gz)
```bash
# Extract the archive
tar -xzf voicebox_aarch64.app.tar.gz
# Move to Applications
mv Voicebox.app /Applications/
```
</Tab>
<Tab title="Intel">
Download: [voicebox_x64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64.app.tar.gz)
```bash
# Extract the archive
tar -xzf voicebox_x64.app.tar.gz
# Move to Applications
mv Voicebox.app /Applications/
```
</Tab>
</Tabs>
### Windows
<Tabs>
<Tab title="MSI Installer">
Download: [voicebox_x64_en-US.msi](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64_en-US.msi)
Double-click the MSI file and follow the installation wizard.
</Tab>
<Tab title="Setup Executable">
Download: [voicebox_x64-setup.exe](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64-setup.exe)
Run the executable and follow the installation wizard.
</Tab>
</Tabs>
### Linux
<Note>
Linux builds are coming soon. Currently blocked by GitHub runner disk space limitations.
</Note>
## First Launch
When you launch Voicebox for the first time:
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
2. **Data Directory** — Voice profiles and generated audio are stored in:
- macOS: `~/Library/Application Support/com.voicebox.app/`
- Windows: `%APPDATA%/com.voicebox.app/`
- Linux: `~/.config/com.voicebox.app/`
3. **Backend Server** — The bundled Python server starts automatically
<Tip>
First generation will be slower due to model downloads. Subsequent runs use cached models.
</Tip>
## System Requirements
### Minimum
- **OS:** macOS 11+, Windows 10+, or Linux
- **RAM:** 8GB
- **Storage:** 5GB free space (for models and data)
- **CPU:** Modern multi-core processor
### Recommended
- **RAM:** 16GB+
- **GPU:** CUDA-capable NVIDIA GPU (for faster generation)
- **Storage:** 10GB+ free space
<Note>
CPU inference is supported but significantly slower than GPU. A CUDA-capable GPU is highly recommended for real-time workflows.
</Note>
## Verification
After installation, verify everything works:
1. Launch Voicebox
2. Check the server status indicator in the bottom-left corner (should be green)
3. Navigate to **Profiles** and create a test profile
4. Generate a short audio clip to verify the TTS engine works
<Check>
If you see a green status indicator and can generate audio, you're all set!
</Check>
## Next Steps
<Card title="Quick Start Guide" icon="rocket" href="/overview/quick-start">
Create your first voice profile and generate speech
</Card>
docs/content/docs/overview/introduction.mdx
+58
View File
@@ -0,0 +1,58 @@
---
title: "Introduction"
description: "Welcome to Voicebox - the open-source voice synthesis studio"
---
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
<Frame>
<img src="/images/app-screenshot-1.webp" alt="Voicebox App Screenshot" />
</Frame>
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
- **Complete privacy** — models and voice data stay on your machine
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
- **Native performance** — built with Tauri (Rust), not Electron
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
## Key Features
<CardGroup cols={2}>
<Card title="Voice Cloning" icon="microphone">
Instant cloning from just a few seconds of audio with Qwen3-TTS
</Card>
<Card title="Stories Editor" icon="film">
Multi-track timeline for creating conversations and narratives
</Card>
<Card title="Full API" icon="code">
REST API for integrating voice synthesis into your apps
</Card>
<Card title="Local-First" icon="shield">
Everything runs on your machine - complete privacy
</Card>
</CardGroup>
## Use Cases
- **Game Development** — Generate dynamic dialogue for characters
- **Content Creation** — Produce podcasts and video voiceovers
- **Accessibility** — Build text-to-speech tools
- **Voice Assistants** — Create custom voice interfaces
- **Production Pipelines** — Automate voiceover workflows
## Next Steps
<CardGroup cols={2}>
<Card title="Installation" icon="download" href="/overview/installation">
Download and install Voicebox on your machine
</Card>
<Card title="Quick Start" icon="rocket" href="/overview/quick-start">
Get up and running in 5 minutes
</Card>
</CardGroup>
docs/content/docs/overview/meta.json
+17
View File
@@ -0,0 +1,17 @@
{
"title": "Overview",
"pages": [
"introduction",
"installation",
"quick-start",
"voice-cloning",
"stories-editor",
"recording-transcription",
"generation-history",
"remote-mode",
"creating-voice-profiles",
"generating-speech",
"building-stories",
"troubleshooting"
]
}
docs/content/docs/overview/quick-start.mdx
+154
View File
@@ -0,0 +1,154 @@
---
title: "Quick Start"
description: "Get started with Voicebox in 5 minutes"
---
This guide will walk you through creating your first voice profile and generating speech.
## Prerequisites
Make sure you have [installed Voicebox](/overview/installation) and launched the app.
## Step 1: Create a Voice Profile
Voice profiles are the foundation of Voicebox. Each profile contains voice samples that the AI uses to clone the voice.
<Steps>
<Step title="Navigate to Profiles">
Click the **Profiles** tab in the sidebar
</Step>
<Step title="Create New Profile">
Click the **+ New Profile** button
Fill in the details:
- **Name:** A descriptive name (e.g., "John Smith")
- **Language:** Select the primary language
- **Description:** Optional notes about the voice
</Step>
<Step title="Add Voice Sample">
You have two options:
**Option A: Upload Audio**
- Click **Upload Sample**
- Select an audio file (WAV, MP3, or M4A)
- Ideal length: 10-30 seconds of clear speech
**Option B: Record Live**
- Click **Record Sample**
- Speak clearly for 10-30 seconds
- Click stop when finished
</Step>
<Step title="Save Profile">
Click **Create Profile** to save
</Step>
</Steps>
<Tip>
For best results, use clean audio with minimal background noise and consistent speaking tone.
</Tip>
## Step 2: Generate Speech
Now let's use your new voice profile to generate speech.
<Steps>
<Step title="Go to Generation">
Click the **Generate** tab in the sidebar
</Step>
<Step title="Select Voice Profile">
Choose your newly created profile from the dropdown
</Step>
<Step title="Enter Text">
Type or paste the text you want to generate:
```
Hello! This is my first voice generation with Voicebox.
```
</Step>
<Step title="Generate">
Click **Generate** and wait a few seconds
<Note>
First generation may take longer due to model initialization. Subsequent generations will be faster.
</Note>
</Step>
<Step title="Play & Download">
- Click **Play** to preview the audio
- Click **Download** to save the audio file
- The generation is also saved to your **History**
</Step>
</Steps>
## Step 3: Build a Story (Optional)
The Stories Editor lets you create multi-voice narratives with a timeline-based interface.
<Steps>
<Step title="Create New Story">
Navigate to **Stories** and click **+ New Story**
</Step>
<Step title="Add Voice Tracks">
Click **+ Add Track** to create tracks for different speakers
</Step>
<Step title="Add Audio Clips">
- Drag generated audio from your History
- Or generate new clips directly in the timeline
- Arrange clips on the timeline
</Step>
<Step title="Edit & Export">
- Trim clips by dragging edges
- Adjust timing and spacing
- Click **Export** to render the final audio
</Step>
</Steps>
## What's Next?
<CardGroup cols={2}>
<Card title="Voice Cloning Guide" icon="microphone" href="/overview/creating-voice-profiles">
Learn advanced techniques for high-quality voice cloning
</Card>
<Card title="API Integration" icon="code" href="/api-reference">
Integrate Voicebox into your own applications
</Card>
<Card title="Stories Editor" icon="film" href="/overview/stories-editor">
Master the multi-track timeline editor
</Card>
<Card title="Remote Mode" icon="server" href="/overview/remote-mode">
Connect to a GPU server for faster generation
</Card>
</CardGroup>
## Tips for Success
<AccordionGroup>
<Accordion title="Getting the Best Voice Quality">
- Use 10-30 seconds of clear, consistent speech
- Avoid background noise and echo
- Multiple samples from the same speaker improve quality
- Match the speaking style you want to generate
</Accordion>
<Accordion title="Improving Generation Speed">
- Use a CUDA-capable GPU for 5-10x faster generation
- Enable voice prompt caching for repeated generations
- Consider running the backend on a remote GPU server
</Accordion>
<Accordion title="Troubleshooting Common Issues">
- **Server won't start:** Check if port 17493 is available
- **Poor audio quality:** Try adding more voice samples
- **Slow generation:** Verify GPU acceleration is enabled
- See the full [Troubleshooting Guide](/overview/troubleshooting) for more
</Accordion>
</AccordionGroup>
docs/content/docs/overview/recording-transcription.mdx
+64
View File
@@ -0,0 +1,64 @@
---
title: "Recording & Transcription"
description: "Record audio and transcribe speech with Whisper"
---
## Recording
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
### Features
- **Microphone input** - Record from any audio input device
- **System audio capture** - Record desktop audio (macOS/Windows)
- **Waveform visualization** - See audio levels in real-time
- **Multiple formats** - Export as WAV, MP3, or M4A
### How to Record
<Steps>
<Step title="Select Input">
Choose your microphone or system audio
</Step>
<Step title="Start Recording">
Click the record button and speak clearly
</Step>
<Step title="Stop & Save">
Click stop when finished
</Step>
<Step title="Use or Export">
Use as voice sample or export to file
</Step>
</Steps>
## Transcription
Automatic speech-to-text powered by OpenAI's Whisper model.
### Features
- **High accuracy** - Industry-leading speech recognition
- **Multiple languages** - Supports 50+ languages
- **Automatic detection** - Language auto-detection
- **Timestamps** - Word-level timing information
### How to Transcribe
<Steps>
<Step title="Select Audio">
Choose a recording or upload an audio file
</Step>
<Step title="Choose Language">
Select language or use auto-detect
</Step>
<Step title="Transcribe">
Click transcribe and wait for processing
</Step>
<Step title="Review & Export">
Review text and export as needed
</Step>
</Steps>
<Tip>
Transcription is useful for creating voice samples from existing audio or generating subtitles.
</Tip>
docs/content/docs/overview/remote-mode.mdx
+138
View File
@@ -0,0 +1,138 @@
---
title: "Remote Mode"
description: "Connect to a GPU server for faster generation"
---
## Overview
Remote Mode allows you to run the Voicebox backend on a separate machine (like a GPU server) while using the desktop app on your local machine.
## Use Cases
- **No local GPU** - Use a cloud GPU or remote workstation
- **Faster generation** - Leverage powerful remote hardware
- **Shared infrastructure** - Multiple users connect to one server
- **Laptop workflows** - Keep your laptop cool and battery-efficient
## Architecture
In Remote Mode, the Voicebox desktop app (running on your local machine) communicates with the backend server (running on a remote machine) via HTTP. The local app provides only the user interface, while the remote server handles all the heavy processing including the TTS models, API endpoints, and audio generation.
## Setting Up Remote Mode
### On the Server
<Steps>
<Step title="Install Dependencies">
```bash
# Clone the repo
git clone https://github.com/jamiepine/voicebox.git
cd voicebox/backend
# Install Python dependencies
pip install -r requirements.txt
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Step>
<Step title="Start the Server">
```bash
# Allow external connections
uvicorn main:app --host 0.0.0.0 --port 17493
```
<Warning>
This exposes the server to your network. Use a firewall or VPN for security.
</Warning>
</Step>
<Step title="Open Firewall">
```bash
# Ubuntu/Debian
sudo ufw allow 17493
# Or use your cloud provider's firewall settings
```
</Step>
</Steps>
### On the Client
<Steps>
<Step title="Open Settings">
In Voicebox, go to **Settings → Server**
</Step>
<Step title="Enable Remote Mode">
Toggle **Use Remote Server**
</Step>
<Step title="Enter Server URL">
```
http://<server-ip>:17493
```
Replace `<server-ip>` with your server's IP address
</Step>
<Step title="Test Connection">
Click **Test Connection** to verify
</Step>
</Steps>
## Cloud Deployment
### AWS EC2
```bash
# Launch a GPU instance (e.g., g4dn.xlarge)
# Install dependencies
# Start server with --host 0.0.0.0
```
### Vast.ai
```bash
# Rent a GPU instance
# SSH in and clone repo
# Start server
```
### RunPod
```bash
# Deploy a pod with CUDA support
# Install Voicebox backend
# Expose port 17493
```
## Security Considerations
<Warning>
The API currently has no authentication. Only use on trusted networks or with a VPN.
</Warning>
**Best Practices:**
- Use a VPN (WireGuard, Tailscale) instead of exposing to the internet
- Run behind a reverse proxy with authentication (nginx + basic auth)
- Use HTTPS with SSL certificates
- Firewall rules to limit access to specific IPs
## Performance
Expected performance on various GPUs:
| GPU | Generation Speed |
|-----|------------------|
| RTX 4090 | ~2-3s per 10 words |
| RTX 3090 | ~3-4s per 10 words |
| RTX 3060 | ~5-7s per 10 words |
| CPU (12-core) | ~20-30s per 10 words |
<Tip>
A GPU with 8GB+ VRAM is recommended for best performance.
</Tip>
## Troubleshooting
See the [Troubleshooting Guide](/guides/troubleshooting#remote-mode-issues) for common remote mode issues.
docs/content/docs/overview/stories-editor.mdx
+64
View File
@@ -0,0 +1,64 @@
---
title: "Stories Editor"
description: "Create multi-voice narratives with a timeline-based editor"
---
## Overview
The Stories Editor is a DAW-like timeline interface for creating multi-voice narratives, podcasts, and conversations.
## Features
<CardGroup cols={2}>
<Card title="Multi-Track Timeline" icon="timeline">
Arrange multiple voice tracks in parallel
</Card>
<Card title="Inline Editing" icon="scissors">
Trim and split clips directly in the timeline
</Card>
<Card title="Auto-Playback" icon="play">
Preview with synchronized playhead
</Card>
<Card title="Voice Mixing" icon="users">
Build conversations with multiple speakers
</Card>
</CardGroup>
## Creating a Story
<Steps>
<Step title="Create New Story">
Navigate to **Stories** and click **+ New Story**
</Step>
<Step title="Add Tracks">
Create separate tracks for each voice/speaker
</Step>
<Step title="Add Clips">
- Drag from generation history
- Generate new clips inline
- Upload audio files
</Step>
<Step title="Arrange & Edit">
- Position clips on timeline
- Trim clip edges
- Adjust spacing and timing
</Step>
<Step title="Export">
Render the final mixed audio
</Step>
</Steps>
## Use Cases
- **Podcasts**: Multi-host conversations
- **Audiobooks**: Narrator + character voices
- **Game Dialogue**: Character interactions
- **Video Voiceovers**: Multiple speakers
- **Audio Drama**: Full voice casts
## Coming Soon
- Word-level editing
- Crossfades and transitions
- Audio effects (reverb, EQ)
- Real-time collaboration
docs/content/docs/overview/troubleshooting.mdx
+477
View File
@@ -0,0 +1,477 @@
---
title: "Troubleshooting"
description: "Common issues and solutions for Voicebox"
---
This guide covers common issues you might encounter when using or developing Voicebox, along with solutions.
## Installation Issues
### macOS: "App is damaged and can't be opened"
This occurs because the app isn't signed with an Apple Developer certificate.
**Solution:**
```bash
# Remove the quarantine attribute
xattr -cr /Applications/Voicebox.app
```
### Windows: SmartScreen Warning
Windows SmartScreen may warn that the app is unrecognized.
**Solution:**
- Click "More info"
- Click "Run anyway"
<Note>
This is expected for unsigned applications. We're working on code signing for future releases.
</Note>
## Server Issues
### Backend Server Won't Start
**Symptoms:**
- Red status indicator in bottom-left corner
- "Failed to connect to server" error
**Solutions:**
<AccordionGroup>
<Accordion title="Port Already in Use">
Check if port 17493 is already in use:
```bash
# macOS/Linux
lsof -i :17493
# Windows
netstat -ano | findstr :17493
```
Kill the process using the port:
```bash
# macOS/Linux
kill -9 <PID>
# Windows
taskkill /PID <PID> /F
```
</Accordion>
<Accordion title="Permission Issues">
The server binary might not have execute permissions:
```bash
# macOS/Linux
chmod +x ~/Library/Application\ Support/com.voicebox.app/backend/voicebox-server
```
</Accordion>
<Accordion title="Check Logs">
View server logs for errors:
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
### Connection Timeout
**Symptoms:**
- Long loading times
- "Connection timeout" errors
**Solution:**
- Restart the app
- Check your firewall settings
- Ensure localhost is accessible
## Generation Issues
### First Generation is Very Slow
**Symptoms:**
- First generation takes 2-5 minutes
- Progress indicator stuck at "Loading model..."
**Explanation:**
This is expected behavior. The first generation downloads the Qwen3-TTS model (~2-4GB) and initializes it.
**Solution:**
- Wait for the initial download to complete
- Subsequent generations will be much faster
- Check your internet connection
### Poor Voice Quality
**Symptoms:**
- Robotic or unnatural voice
- Missing emotion or prosody
- Pronunciation errors
**Solutions:**
<Steps>
<Step title="Improve Voice Samples">
- Use 10-30 seconds of clear audio
- Avoid background noise
- Ensure consistent speaking tone
- Add multiple samples from the same speaker
</Step>
<Step title="Match Speaking Style">
The generated voice will mimic the tone and style of your samples. If your sample is monotone, the generation will be too.
</Step>
<Step title="Adjust Text Formatting">
- Use proper punctuation
- Add commas for natural pauses
- Capitalize proper nouns
</Step>
</Steps>
### Generation Fails with "Out of Memory"
**Symptoms:**
- Generation crashes
- "CUDA out of memory" or "RuntimeError: out of memory"
**Solutions:**
<AccordionGroup>
<Accordion title="Free GPU Memory">
Close other GPU-intensive applications:
- Games
- Video editors
- Multiple browser tabs with WebGL
Then restart Voicebox.
</Accordion>
<Accordion title="Use CPU Mode">
If your GPU doesn't have enough VRAM (need 6GB+), use CPU mode:
Settings → Generation → Use CPU instead of GPU
<Warning>
CPU generation is 5-10x slower but uses system RAM instead of VRAM.
</Warning>
</Accordion>
<Accordion title="Reduce Batch Size">
For long text, split it into smaller chunks instead of generating all at once.
</Accordion>
</AccordionGroup>
## Audio Issues
### No Audio Playback
**Symptoms:**
- Generated audio won't play
- Playback button doesn't respond
**Solutions:**
- Check system audio settings
- Ensure audio output device is connected
- Try exporting and playing in a media player
### Crackling or Distorted Audio
**Symptoms:**
- Audio has static or distortion
- Clipping sounds
**Solutions:**
- Check if your input samples have distortion
- Reduce playback volume
- Re-generate with cleaner voice samples
## Development Issues
### Backend Won't Start in Dev Mode
**Symptoms:**
- `bun run dev:server` fails
- Import errors or module not found
**Solutions:**
<AccordionGroup>
<Accordion title="Python Version">
Ensure Python 3.11 or higher:
```bash
python --version
```
If not, install Python 3.11+ and recreate the virtual environment.
</Accordion>
<Accordion title="Virtual Environment">
Ensure venv is activated:
```bash
# macOS/Linux
source backend/venv/bin/activate
# Windows
backend\venv\Scripts\activate
```
You should see `(venv)` in your prompt.
</Accordion>
<Accordion title="Dependencies">
Reinstall dependencies:
```bash
cd backend
pip install -r requirements.txt
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Accordion>
</AccordionGroup>
### Tauri Build Fails
**Symptoms:**
- `bun run tauri build` fails
- Rust compilation errors
**Solutions:**
```bash
# Clean build artifacts
cd tauri/src-tauri
cargo clean
# Update Rust
rustup update
# Try building again
cd ../..
bun run tauri build
```
### OpenAPI Client Generation Fails
**Symptoms:**
- `./scripts/generate-api.sh` fails
- "Failed to fetch schema" error
**Solutions:**
<Steps>
<Step title="Ensure Backend is Running">
```bash
curl http://localhost:17493/openapi.json
```
Should return JSON. If not, start the backend.
</Step>
<Step title="Check Port">
Ensure nothing else is using port 17493
</Step>
<Step title="Regenerate Manually">
```bash
cd backend
source venv/bin/activate
uvicorn main:app --reload --port 17493
# In another terminal
./scripts/generate-api.sh
```
</Step>
</Steps>
## Database Issues
### "Database is locked" Error
**Symptoms:**
- Profile or generation operations fail
- SQLite lock errors
**Solutions:**
- Close all Voicebox instances
- Delete the lock file:
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-wal
```
### Corrupted Database
**Symptoms:**
- App crashes on launch
- Data missing or corrupted
**Solutions:**
<Warning>
This will delete all your voice profiles and generation history. Export important profiles first if possible.
</Warning>
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db
# Windows
del %APPDATA%\com.voicebox.app\data\voicebox.db
```
Restart the app to create a fresh database.
## Model Issues
### Model Download Fails
**Symptoms:**
- "Failed to download model" error
- Stuck at "Downloading..."
**Solutions:**
- Check your internet connection
- Check HuggingFace Hub status
- Try using a VPN if HuggingFace is blocked in your region
- Manually download and place in cache directory
### Wrong Model Version
**Symptoms:**
- Generation quality suddenly degraded
- Different voice output
**Solutions:**
Clear the model cache and re-download:
```bash
# macOS
rm -rf ~/.cache/huggingface/hub/models--Qwen*
# Windows
rmdir /s %USERPROFILE%\.cache\huggingface\hub\models--Qwen*
```
## Performance Issues
### Slow Generation on GPU
**Symptoms:**
- Generation slower than expected
- GPU not being utilized
**Solutions:**
<AccordionGroup>
<Accordion title="Verify CUDA Installation">
```bash
nvidia-smi
```
Should show your GPU. If not, install CUDA drivers.
</Accordion>
<Accordion title="Check GPU Selection">
If you have multiple GPUs, ensure Voicebox is using the right one.
Settings → Generation → GPU Device
</Accordion>
<Accordion title="Update GPU Drivers">
Outdated drivers can cause performance issues. Update to the latest NVIDIA drivers.
</Accordion>
</AccordionGroup>
### High Memory Usage
**Symptoms:**
- App uses excessive RAM
- System becomes sluggish
**Solutions:**
- Close unused voice profiles
- Clear generation history
- Restart the app periodically
## Remote Mode Issues
### Can't Connect to Remote Server
**Symptoms:**
- "Connection refused" error
- Remote server not found
**Solutions:**
<Steps>
<Step title="Check Server Status">
Ensure the remote server is running:
```bash
curl http://<server-ip>:17493/health
```
</Step>
<Step title="Check Firewall">
Ensure port 17493 is open on the remote server:
```bash
# Allow port on Ubuntu/Debian
sudo ufw allow 17493
```
</Step>
<Step title="Verify Network">
- Ensure both machines are on the same network (for local servers)
- Use IP address instead of hostname
- Try pinging the server: `ping <server-ip>`
</Step>
</Steps>
## Still Having Issues?
If you're still experiencing problems:
1. **Check GitHub Issues:** [github.com/jamiepine/voicebox/issues](https://github.com/jamiepine/voicebox/issues)
2. **Open a New Issue:** Provide:
- Operating system and version
- Voicebox version
- Steps to reproduce
- Error messages or logs
3. **Join Discord:** [discord.gg/voicebox](https://discord.gg/voicebox) (coming soon)
## Diagnostic Information
When reporting issues, include this information:
```bash
# Voicebox version
# Check Help → About in the app
# Operating system
uname -a # macOS/Linux
systeminfo # Windows
# Python version (for dev issues)
python --version
# GPU info (if generation issues)
nvidia-smi # NVIDIA GPUs
```
For more detailed troubleshooting, see the [TROUBLESHOOTING.md](https://github.com/jamiepine/voicebox/blob/main/docs/TROUBLESHOOTING.md) file in the repository.
docs/content/docs/overview/voice-cloning.mdx
+75
View File
@@ -0,0 +1,75 @@
---
title: "Voice Cloning"
description: "Clone any voice from just a few seconds of audio"
---
## Overview
Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning from just a few seconds of audio. The model captures prosody, emotion, and natural cadence.
## How It Works
<Steps>
<Step title="Upload or Record Sample">
Provide 10-30 seconds of clear speech from the target voice
</Step>
<Step title="Model Analysis">
Qwen3-TTS analyzes vocal characteristics, tone, and speaking patterns
</Step>
<Step title="Voice Profile Created">
The model generates a voice embedding for synthesis
</Step>
<Step title="Generate Speech">
Use the profile to generate any text in the cloned voice
</Step>
</Steps>
## Best Practices
### Sample Quality
<CardGroup cols={2}>
<Card title="Do" icon="check">
- Use 10-30 seconds of audio
- Clear, consistent speaking
- Minimal background noise
- Natural speaking pace
</Card>
<Card title="Don't" icon="xmark">
- Very short clips (< 5 seconds)
- Heavy background noise
- Music or overlapping voices
- Heavily processed audio
</Card>
</CardGroup>
### Multiple Samples
Adding multiple samples from the same speaker can improve quality:
- Different speaking styles (casual, formal)
- Different emotions (happy, serious)
- Different recording conditions
<Tip>
The model will learn a more robust representation from diverse samples.
</Tip>
## Supported Languages
Currently supported:
- English
- Chinese (Mandarin)
More languages coming soon.
## Limitations
<Warning>
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice.
</Warning>
- Quality depends on sample clarity
- Works best with consistent speaking tone
- May struggle with extreme accents or speech impediments
- Background noise reduces quality
docs/content/docs/plans/DOCKER_DEPLOYMENT.md
+761
View File
@@ -0,0 +1,761 @@
---
title: "Docker Deployment Guide"
description: "Docker deployment guide for Voicebox (In Development)"
---
**Status:** In Development for v0.2.0
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
## Overview
Docker support makes Voicebox easier to deploy, especially for:
- **Consistent Environments**: Same setup across dev/staging/prod
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
- **Server Deployments**: Run on headless Linux servers
- **Multi-User Setups**: Isolate instances per user/team
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
## Quick Start
### Using Pre-Built Images (Recommended)
```bash
# CPU-only version
docker run -p 8000:8000 -v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest
# NVIDIA GPU version
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest-cuda
# AMD GPU version (experimental)
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
-v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest-rocm
```
Then open: `http://localhost:8000`
### Using Docker Compose (Easiest)
Create `docker-compose.yml`:
```yaml
version: '3.8'
services:
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
ports:
- "8000:8000"
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
- TTS_MODE=local
- WHISPER_MODE=local
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
voicebox-data:
huggingface-cache:
```
Run:
```bash
docker compose up -d
```
## Building From Source
### Basic Dockerfile
```dockerfile
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
build-essential \
ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Copy application
COPY backend/ /app/backend/
COPY requirements.txt /app/
# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
# Create data directory
RUN mkdir -p /app/data
# Expose port
EXPOSE 8000
# Run server
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Build and run:
```bash
docker build -t voicebox .
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
```
### Multi-Stage Build (Optimized)
Smaller image size by separating build and runtime:
```dockerfile
# Dockerfile.optimized
# Stage 1: Build dependencies
FROM python:3.11-slim AS builder
WORKDIR /build
RUN apt-get update && apt-get install -y \
git build-essential && \
rm -rf /var/lib/apt/lists/*
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --target=/build/packages \
-r requirements.txt
RUN pip install --no-cache-dir --target=/build/packages \
git+https://github.com/QwenLM/Qwen3-TTS.git
# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
# Install only runtime dependencies
RUN apt-get update && apt-get install -y \
ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Copy installed packages from builder
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
# Copy application code
COPY backend/ /app/backend/
# Create data directory
RUN mkdir -p /app/data
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Build:
```bash
docker build -f Dockerfile.optimized -t voicebox:slim .
```
## GPU Support
### NVIDIA GPUs (CUDA)
**Dockerfile:**
```dockerfile
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
# Install Python
RUN apt-get update && apt-get install -y \
python3.11 python3-pip git ffmpeg && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Install PyTorch with CUDA support
COPY backend/requirements.txt .
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
# Install other dependencies
RUN pip3 install -r requirements.txt
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
COPY backend/ /app/backend/
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
**Run with GPU:**
```bash
docker run --gpus all -p 8000:8000 \
-v voicebox-data:/app/data \
voicebox:cuda
```
**Docker Compose with GPU:**
```yaml
services:
voicebox:
image: voicebox:cuda
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
```
### AMD GPUs (ROCm) - Experimental
**Dockerfile:**
```dockerfile
FROM rocm/dev-ubuntu-22.04:6.0
# Install Python
RUN apt-get update && apt-get install -y \
python3.11 python3-pip git ffmpeg && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Install PyTorch with ROCm support
COPY backend/requirements.txt .
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
# Install other dependencies
RUN pip3 install -r requirements.txt
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
# Set ROCm environment variables
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
ENV ROCM_PATH=/opt/rocm
COPY backend/ /app/backend/
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
**Run with AMD GPU:**
```bash
docker run --device=/dev/kfd --device=/dev/dri \
--group-add video --ipc=host --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-p 8000:8000 -v voicebox-data:/app/data \
voicebox:rocm
```
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
## Volume Mounts
### Essential Volumes
```bash
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
-p 8000:8000 voicebox
```
### Development Volume Mounts
For development with hot-reload:
```bash
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
-v voicebox-data:/app/data \
-e RELOAD=true \
-p 8000:8000 voicebox
```
### Custom Model Storage
Use external model directory:
```bash
docker run -v /path/to/models:/models \
-e MODELS_DIR=/models \
-v voicebox-data:/app/data \
-p 8000:8000 voicebox
```
## Environment Variables
Configure Voicebox via environment variables:
```bash
docker run -e TTS_MODE=local \
-e WHISPER_MODE=openai-api \
-e OPENAI_API_KEY=sk-... \
-e GPU_MEMORY_FRACTION=0.8 \
-e LOG_LEVEL=info \
-p 8000:8000 voicebox
```
### Available Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
| `MODELS_DIR` | `/app/models` | Directory for local models |
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
| `RELOAD` | `false` | Enable hot-reload for development |
## Complete Docker Compose Examples
### Production Deployment
```yaml
# docker-compose.prod.yml
version: '3.8'
services:
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
container_name: voicebox
restart: unless-stopped
ports:
- "8000:8000"
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- TTS_MODE=local
- WHISPER_MODE=local
- GPU_MEMORY_FRACTION=0.8
- LOG_LEVEL=info
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
volumes:
voicebox-data:
driver: local
huggingface-cache:
driver: local
```
Run:
```bash
docker compose -f docker-compose.prod.yml up -d
```
### Development Setup
```yaml
# docker-compose.dev.yml
version: '3.8'
services:
voicebox:
build:
context: .
dockerfile: Dockerfile
ports:
- "8000:8000"
volumes:
- ./backend:/app/backend:ro
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- RELOAD=true
- LOG_LEVEL=debug
- TTS_MODE=local
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
volumes:
voicebox-data:
huggingface-cache:
```
### Multi-Service Stack
Full stack with reverse proxy and monitoring:
```yaml
# docker-compose.stack.yml
version: '3.8'
services:
# Main Voicebox app
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
restart: unless-stopped
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- TTS_MODE=local
- WHISPER_MODE=local
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
# Nginx reverse proxy
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- voicebox
# Prometheus monitoring (optional)
prometheus:
image: prom/prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus
volumes:
voicebox-data:
huggingface-cache:
prometheus-data:
```
## Cloud Deployment
### AWS EC2
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
2. **Install Docker + nvidia-docker:**
```bash
# Amazon Linux 2
sudo yum install -y docker
sudo systemctl start docker
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
```
3. **Deploy:**
```bash
docker run --gpus all -d -p 80:8000 \
-v voicebox-data:/app/data \
--restart unless-stopped \
ghcr.io/jamiepine/voicebox:latest-cuda
```
### DigitalOcean
Use GPU Droplet + Docker:
```bash
# Create droplet via CLI
doctl compute droplet create voicebox \
--size gpu-h100x1-80gb \
--image ubuntu-22-04-x64 \
--region nyc3
# SSH and deploy
ssh root@<droplet-ip>
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
docker run --gpus all -d -p 80:8000 voicebox:cuda
```
### Google Cloud Run (CPU-only)
```bash
# Build and push
docker build -t gcr.io/your-project/voicebox .
docker push gcr.io/your-project/voicebox
# Deploy to Cloud Run
gcloud run deploy voicebox \
--image gcr.io/your-project/voicebox \
--platform managed \
--region us-central1 \
--memory 4Gi \
--cpu 2 \
--port 8000
```
### Fly.io
Create `fly.toml`:
```toml
app = "voicebox"
[build]
image = "ghcr.io/jamiepine/voicebox:latest"
[[services]]
http_checks = []
internal_port = 8000
protocol = "tcp"
[[services.ports]]
port = 80
handlers = ["http"]
[[services.ports]]
port = 443
handlers = ["tls", "http"]
[mounts]
source = "voicebox_data"
destination = "/app/data"
```
Deploy:
```bash
fly launch
fly deploy
```
## Troubleshooting
### GPU Not Detected
**Check NVIDIA Docker:**
```bash
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
```
If this fails, reinstall nvidia-docker2.
**Check AMD ROCm:**
```bash
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
```
### Permission Errors
Container can't write to volumes:
```bash
# Fix permissions
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
```
### Out of Memory
Reduce GPU memory usage:
```bash
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
```
Or use CPU-only:
```bash
docker run -e DEVICE=cpu voicebox
```
### Model Download Fails
Ensure HuggingFace cache is writable:
```bash
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
```
Or use host cache:
```bash
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
```
### Port Already in Use
Change host port:
```bash
docker run -p 8080:8000 voicebox # Use port 8080 instead
```
## Security Best Practices
### 1. Don't Run as Root
Create non-root user in Dockerfile:
```dockerfile
RUN useradd -m -u 1000 voicebox
USER voicebox
```
### 2. Use Secrets for API Keys
Don't put API keys in docker-compose.yml:
```bash
# Use Docker secrets
echo "sk-your-key" | docker secret create openai_key -
docker service create \
--secret openai_key \
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
voicebox
```
### 3. Network Isolation
Use internal networks for multi-container setups:
```yaml
services:
voicebox:
networks:
- internal
nginx:
networks:
- internal
- external
ports:
- "80:80"
networks:
internal:
internal: true
external:
```
### 4. Resource Limits
Prevent resource exhaustion:
```yaml
services:
voicebox:
deploy:
resources:
limits:
cpus: '4'
memory: 8G
reservations:
cpus: '2'
memory: 4G
```
## Performance Tuning
### GPU Memory Management
```bash
# Use 80% of GPU (default 90%)
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
# Allow GPU memory growth (prevents OOM)
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
```
### Model Caching
Pre-download models to volume:
```bash
# Download models first
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
voicebox python -c "
from transformers import WhisperProcessor, WhisperForConditionalGeneration
WhisperProcessor.from_pretrained('openai/whisper-base')
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
"
# Then run normally
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
```
### Multi-Worker Setup
Use uvicorn workers for better throughput:
```dockerfile
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
```
## Monitoring
### Health Checks
Built-in health endpoint:
```bash
curl http://localhost:8000/health
```
Docker health check:
```yaml
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
```
### Prometheus Metrics
Add metrics exporter:
```python
# backend/main.py
from prometheus_fastapi_instrumentator import Instrumentator
Instrumentator().instrument(app).expose(app)
```
Then scrape `/metrics` with Prometheus.
### Logs
View container logs:
```bash
docker logs -f voicebox
# Or with compose
docker compose logs -f voicebox
```
## Next Steps
- [ ] Publish official images to GitHub Container Registry
- [ ] Add Kubernetes Helm charts
- [ ] Create Docker Desktop extension
- [ ] Add automated vulnerability scanning
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
## Contributing
Help improve Docker support:
1. Test on different platforms (AMD GPU, ARM64, etc.)
2. Submit Dockerfile optimizations
3. Share deployment configurations
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
## Resources
- [Docker Documentation](https://docs.docker.com)
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
docs/content/docs/plans/EXTERNAL_PROVIDERS.md
+438
View File
@@ -0,0 +1,438 @@
---
title: "External Provider Support"
description: "External provider support for Voicebox (Planned)"
---
**Status:** Planned for v0.2.0
**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...)
## Overview
External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for:
- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server
- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI
- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc.
- **Team Sharing**: Multiple users share one GPU server running models
- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa
## Architecture
```
┌─────────────────┐ HTTP/API ┌──────────────────┐
│ Voicebox UI │ ───────────────────────> │ Your TTS Server │
│ + Backend │ │ (Qwen3-TTS on │
│ │ <─────────────────────── │ AMD/NVIDIA GPU)│
│ - Profiles │ Audio + Metadata └──────────────────┘
│ - History │
│ - Audio Edit │ HTTP/API ┌──────────────────┐
│ - UI │ ───────────────────────> │ Whisper Service │
└─────────────────┘ │ (OpenAI API or │
│ self-hosted) │
└──────────────────┘
```
**What Voicebox Still Handles:**
- Voice profile management
- Generation history
- Audio trimming/editing
- Multi-track story editor
- UI/UX layer
**What External Providers Handle:**
- Model inference (TTS generation, transcription)
- GPU allocation
- Model loading/caching
## Configuration
### Environment Variables
```bash
# TTS Provider
TTS_MODE=remote # local | remote
TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL
TTS_API_KEY=your-api-key # Optional authentication
# Whisper Provider
WHISPER_MODE=openai-api # local | openai-api | remote
WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper
OPENAI_API_KEY=sk-... # For OpenAI Whisper API
```
### Voicebox Config UI (Planned)
Settings page will include:
- Provider selection dropdowns
- URL/API key inputs
- Connection test button
- Latency/status indicators
## Hosting External Services
### Option 1: Simple FastAPI Server (Recommended)
Create a lightweight server to expose your local Qwen3-TTS model:
```python
# tts_server.py
from fastapi import FastAPI, UploadFile, File
from qwen_tts import Qwen3TTSModel
import numpy as np
import base64
app = FastAPI()
model = Qwen3TTSModel.from_pretrained(
"Qwen/Qwen3-TTS-12Hz-1.7B-Base",
device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm
)
@app.post("/v1/generate")
async def generate(
text: str,
voice_prompt: dict,
language: str = "en",
seed: int = None
):
"""Generate speech from text using voice prompt."""
audio, sample_rate = model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
)
# Return as base64 for transport
audio_bytes = audio.tobytes()
return {
"audio": base64.b64encode(audio_bytes).decode(),
"sample_rate": sample_rate,
"dtype": str(audio.dtype)
}
@app.post("/v1/create_voice_prompt")
async def create_voice_prompt(
audio: UploadFile = File(...),
reference_text: str = ""
):
"""Create voice prompt from reference audio."""
# Save uploaded audio temporarily
audio_path = f"/tmp/{audio.filename}"
with open(audio_path, "wb") as f:
f.write(await audio.read())
# Create voice prompt
voice_prompt = model.create_voice_clone_prompt(
ref_audio=audio_path,
ref_text=reference_text,
)
return {"voice_prompt": voice_prompt}
@app.get("/health")
async def health():
return {
"status": "healthy",
"model": "Qwen3-TTS-12Hz-1.7B-Base",
"device": str(model.device)
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
**Run it:**
```bash
# Install dependencies
pip install fastapi uvicorn qwen-tts torch
# For AMD GPUs, use ROCm PyTorch:
pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
# Start server
python tts_server.py
```
### Option 2: vLLM (If Supported)
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--host 0.0.0.0 \
--port 8000 \
--gpu-memory-utilization 0.9
```
### Option 3: Cloud Platforms
**Modal.com Example:**
```python
import modal
app = modal.App("qwen-tts")
image = modal.Image.debian_slim().pip_install("qwen-tts", "torch")
@app.function(gpu="A10G", image=image)
@modal.web_endpoint(method="POST")
def generate(text: str, voice_prompt: dict):
from qwen_tts import Qwen3TTSModel
model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
audio, sr = model.generate_voice_clone(text, voice_prompt)
return {"audio": audio.tolist(), "sample_rate": sr}
```
Deploy: `modal deploy tts_server.py`
Get URL: `https://yourapp--generate.modal.run`
## API Specification
External TTS providers must implement these endpoints:
### `POST /v1/generate`
Generate speech from text.
**Request:**
```json
{
"text": "Hello, this is a test.",
"voice_prompt": { /* voice prompt object */ },
"language": "en",
"seed": 12345
}
```
**Response:**
```json
{
"audio": "base64-encoded-audio-bytes",
"sample_rate": 24000,
"dtype": "float32"
}
```
### `POST /v1/create_voice_prompt`
Create a voice prompt from reference audio.
**Request:** (multipart/form-data)
- `audio`: Audio file upload
- `reference_text`: Transcript of the audio
**Response:**
```json
{
"voice_prompt": { /* voice prompt object */ }
}
```
### `GET /health`
Health check endpoint.
**Response:**
```json
{
"status": "healthy",
"model": "Qwen3-TTS-12Hz-1.7B-Base",
"device": "cuda:0"
}
```
## Whisper External Providers
### OpenAI Whisper API
Simply set:
```bash
WHISPER_MODE=openai-api
OPENAI_API_KEY=sk-...
```
Voicebox will use OpenAI's Whisper API automatically.
### Self-Hosted Whisper
Run your own Whisper server:
```python
# whisper_server.py
from fastapi import FastAPI, UploadFile, File
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
app = FastAPI()
processor = WhisperProcessor.from_pretrained("openai/whisper-base")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
@app.post("/v1/transcribe")
async def transcribe(audio: UploadFile = File(...), language: str = None):
# Load audio
audio_path = f"/tmp/{audio.filename}"
with open(audio_path, "wb") as f:
f.write(await audio.read())
audio_data, sr = librosa.load(audio_path, sr=16000)
# Process
inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt")
predicted_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
return {"text": transcription}
```
Configure Voicebox:
```bash
WHISPER_MODE=remote
WHISPER_REMOTE_URL=http://localhost:9000
```
## Use Cases
### 1. AMD GPU User with Existing Setup
**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux.
**Setup:**
1. Run `tts_server.py` on your AMD box (ROCm PyTorch)
2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000`
3. Use Voicebox UI for profiles, generation, editing
4. TTS happens on your AMD GPU
### 2. Team Deployment
**Scenario:** 5 team members, 1 GPU server.
**Setup:**
1. Deploy TTS server on shared GPU box
2. Each person runs Voicebox desktop app locally
3. All point to same `TTS_REMOTE_URL`
4. Profiles and history stay local per user
5. GPU usage is shared
### 3. Hybrid Local/Remote
**Scenario:** Fast local Whisper, heavy TTS on cloud.
**Setup:**
```bash
TTS_MODE=remote
TTS_REMOTE_URL=https://your-modal-app.modal.run
WHISPER_MODE=local # Fast transcription on your CPU
```
### 4. OpenAI Whisper + Self-Hosted TTS
**Scenario:** Use OpenAI's API for transcription, run TTS locally.
**Setup:**
```bash
TTS_MODE=local
WHISPER_MODE=openai-api
OPENAI_API_KEY=sk-...
```
## Security Considerations
### Authentication
Add API key authentication to your external server:
```python
from fastapi import Header, HTTPException
API_KEY = "your-secret-key"
async def verify_api_key(x_api_key: str = Header(...)):
if x_api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
@app.post("/v1/generate", dependencies=[Depends(verify_api_key)])
async def generate(...):
...
```
Configure Voicebox:
```bash
TTS_API_KEY=your-secret-key
```
### Network Security
- **VPN/Tailscale**: Use private network for remote servers
- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates
- **Firewall**: Restrict access to known IPs
### Rate Limiting
Protect your external server:
```python
from slowapi import Limiter
from slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter
@app.post("/v1/generate")
@limiter.limit("10/minute")
async def generate(...):
...
```
## Performance Considerations
### Latency
External providers add network latency:
- **Local network**: ~10-50ms overhead (negligible)
- **Same datacenter**: ~1-5ms overhead
- **Cross-region cloud**: 50-200ms+ overhead
For real-time applications, keep TTS server on local network or same cloud region.
### Caching
Implement response caching on external server:
```python
from functools import lru_cache
@lru_cache(maxsize=1000)
def get_cached_generation(text, voice_prompt_hash, language, seed):
return model.generate_voice_clone(text, voice_prompt)
```
### Load Balancing
For high-traffic deployments, run multiple TTS servers behind a load balancer:
```
Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1)
├──> TTS Server 2 (GPU 2)
└──> TTS Server 3 (GPU 3)
```
## Future Enhancements
- [ ] **Provider Marketplace**: Built-in directory of compatible providers
- [ ] **Automatic Fallback**: If remote fails, fallback to local
- [ ] **Cost Tracking**: Monitor API usage and costs
- [ ] **Performance Metrics**: Latency, throughput dashboards
- [ ] **Multi-Provider**: Use different providers for different voices/languages
## Contributing
If you build an external provider, please share:
1. Server implementation
2. Performance benchmarks
3. Deployment guide
Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions)
## Questions?
- **Discord**: [Join the community](https://discord.gg/...)
- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues)
- **Docs**: [Full documentation](https://voicebox.sh/docs)
docs/content/docs/plans/MLX_AUDIO.md
+399
View File
@@ -0,0 +1,399 @@
---
title: "MLX Audio Integration"
description: "MLX Audio integration for Voicebox (Validated)"
---
**Status:** Validated ✅
**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio)
## Validation Results
We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings:
| Metric | Result |
|--------|--------|
| MLX Version | 0.30.4 |
| Model Load Time | ~1s (after initial download) |
| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) |
| Test Hardware | Apple Silicon Mac |
### Model Mapping
| voicebox (PyTorch) | mlx-audio (MLX) |
|--------------------|-----------------|
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) |
### mlx-audio API
The API uses a **generator-based streaming pattern**:
```python
from mlx_audio.tts import load
model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16")
# generate() yields GenerationResult objects
for result in model.generate("Hello world"):
audio = result.audio # numpy array of samples
sample_rate = result.sample_rate # 24000
rtf = result.real_time_factor # e.g., 0.55
```
### Known Warnings (harmless)
```
You are using a model of type qwen3_tts to instantiate a model of type .
The tokenizer you are loading... with an incorrect regex pattern...
```
These warnings appear but don't affect functionality or output quality.
### Demo Script
Run `mlx-test/demo.py` to test:
```bash
cd mlx-test && source venv/bin/activate && python demo.py "Your text here"
```
## Problem
Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs.
This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads.
## Solution
Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use.
**Key wins:**
- Native GPU acceleration on Apple Silicon (no more CPU fallback)
- Streaming TTS support (faster perceived latency)
- Memory optimizations (run larger models on less RAM)
- Fixed 0.6B silence bug that we currently ship
- Same Qwen3-TTS models (zero migration cost for users)
## Architecture
### Current Stack
```
┌─────────────────────────┐
│ PyTorch + Qwen3-TTS │
│ (CPU only on macOS) │
└─────────────────────────┘
```
### Proposed Stack
```
┌─────────────────────────────────────────┐
│ Platform Detection at Runtime │
└─────────────────────────────────────────┘
│
├─── Apple Silicon (aarch64-darwin)
│ ┌─────────────────────────┐
│ │ MLX Audio Backend │
│ │ - Qwen3-TTS (mlx) │
│ │ - Whisper (mlx) │
│ │ - Streaming support │
│ └─────────────────────────┘
│
└─── Other (x86_64, Windows, Linux)
┌─────────────────────────┐
│ PyTorch Backend │
│ - Qwen3-TTS (pytorch) │
│ - Whisper (pytorch) │
│ - CUDA if available │
└─────────────────────────┘
```
## Implementation Phases
### Phase 1: Platform Detection & Dependency Management
Create a backend that switches between PyTorch and MLX based on runtime platform detection.
**New files:**
- `backend/platform.py` - Detect Apple Silicon, return backend type
- `backend/backends/__init__.py` - Backend factory pattern
- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only)
**Modified files:**
- `backend/requirements.txt` - Keep PyTorch as default
- `backend/main.py` - Import from backend factory instead of direct imports
**Platform detection logic:**
```python
def get_backend_type() -> str:
"""Detect best backend for current platform."""
if platform.system() == "Darwin" and platform.machine() == "arm64":
# Apple Silicon detected
try:
import mlx
return "mlx"
except ImportError:
return "pytorch" # Fallback if mlx not installed
return "pytorch"
```
### Phase 2: MLX Backend Implementation
Create parallel implementations of TTS and STT using mlx-audio.
**New files:**
- `backend/backends/mlx_backend.py` - MLX inference engine
- `backend/backends/pytorch_backend.py` - Refactor current code into backend
**Interface both backends must implement:**
```python
class TTSBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ...
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ...
async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ...
def unload_model(self) -> None: ...
class STTBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ...
def unload_model(self) -> None: ...
```
**MLX backend implementation notes:**
mlx-audio's `generate()` returns a generator by default (streaming is built-in):
```python
# MLX backend wrapper
from mlx_audio.tts import load
class MLXTTSBackend:
def __init__(self):
self.model = None
async def load_model(self, model_size: str) -> None:
model_map = {
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
# "0.6B": needs conversion to mlx format
}
self.model = load(model_map[model_size])
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]:
# Collect all chunks from generator
chunks = []
for result in self.model.generate(text): # TODO: add voice_prompt support
chunks.append(np.array(result.audio))
return np.concatenate(chunks), 24000
```
**MLX-specific features to expose:**
- Streaming TTS (new endpoint: `/api/generate/stream`)
- Memory-optimized model loading
- Qwen3-ASR for transcription (in addition to Whisper)
### Phase 3: API Layer Updates
Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility.
**Modified files:**
- `backend/main.py` - Add streaming endpoints
- `backend/tts.py` - Refactor to use backend abstraction
- `backend/transcribe.py` - Refactor to use backend abstraction
**New endpoints:**
```python
@app.post("/api/generate/stream")
async def generate_stream(...) -> StreamingResponse:
"""Stream TTS chunks as they're generated (MLX only)."""
backend = get_backend()
if not hasattr(backend, 'generate_streaming'):
raise HTTPException(501, "Streaming not supported on this backend")
return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav")
```
**Backward compatibility:**
- Keep all existing `/api/generate` endpoints unchanged
- PyTorch backend users see no behavior change
- MLX users automatically get faster inference, streaming is opt-in
### Phase 4: Frontend Integration
Add UI indicators for backend type and streaming progress.
**Modified files:**
- `app/src/hooks/useGenerationForm.tsx` - Add streaming support
- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle
- `app/src/lib/api.ts` - Add streaming API client
**UI additions:**
- Badge showing current backend ("MLX" or "PyTorch")
- Toggle for streaming mode (disabled if PyTorch)
- Real-time streaming playback (WaveSurfer progressive loading)
### Phase 5: Build & Distribution
Create separate installers for MLX (Apple Silicon) and PyTorch (Universal).
**Modified files:**
- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds
- `.github/workflows/release.yml` - Build both variants
**Build matrix:**
```yaml
- target: aarch64-apple-darwin
backend: mlx
installer: voicebox-macos-silicon-{version}.dmg
- target: x86_64-apple-darwin
backend: pytorch
installer: voicebox-macos-intel-{version}.dmg
- target: x86_64-pc-windows-msvc
backend: pytorch
installer: voicebox-windows-{version}.exe
```
**Installation flow:**
- Auto-detect architecture, recommend correct installer
- MLX installer includes `mlx-audio` in embedded Python
- PyTorch installer includes `torch` in embedded Python
- Both can coexist (different backend, same profile format)
### Phase 6: Testing & Validation
Ensure both backends produce compatible outputs.
**New files:**
- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio
- `backend/tests/test_streaming.py` - Streaming-specific tests
**Test scenarios:**
- Same voice prompt on both backends → similar (not identical) audio output
- Profile created on MLX → loads on PyTorch (and vice versa)
- Streaming chunks assemble into valid WAV file
- Model downloads work on both backends
- Memory usage stays within bounds
### Phase 7: Documentation
Update user-facing docs and developer guides.
**New files:**
- `docs/developer/BACKENDS.md` - Guide for adding new backends
- `docs/overview/performance.md` - Backend comparison benchmarks
**Modified files:**
- `README.md` - Note Apple Silicon acceleration
- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues
**Key docs to write:**
- Which installer to download (architecture detection)
- Performance comparison (MLX vs PyTorch on same M2 hardware)
- How streaming mode works
- How to force PyTorch on Apple Silicon (for debugging)
## Technical Decisions
### Why Dual Backend Instead of MLX-Only?
**Pros of dual backend:**
- Windows and Intel Mac users unaffected
- Easier testing (can compare outputs)
- Fallback if MLX has issues
**Cons of dual backend:**
- More code to maintain
- Two dependency trees
- Build complexity (separate installers)
**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback.
### Why Separate Installers Instead of Runtime Detection?
**Pros of separate installers:**
- Smaller bundle size (don't ship both PyTorch and MLX)
- Clearer to users which version they have
- Easier to debug (no "which backend am I running?" confusion)
- Can optimize each build for its target
**Cons:**
- More installers to build and test
- Users might download the wrong one
**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page.
### Streaming vs Batch Generation
MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be:
1. MLX-only feature (✅ chosen)
2. Implemented for both (lots of work)
3. Not exposed at all (wasted opportunity)
**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend).
## Migration Path
Nothing needs migrating, macos users will just notice a speed-boost in inference
**Data format compatibility:**
- Profiles (SQLite) → no schema changes needed
- Voice prompts (cached) → backend-agnostic (just numpy arrays)
- Audio files → unchanged
## Performance Expectations
### Measured Results (from validation)
| Metric | MLX (measured) | PyTorch CPU (estimated) |
|--------|----------------|-------------------------|
| **6s audio generation** | ~3-4s | ~10-15s |
| **Real-time factor** | 0.5-0.6x | 2-3x |
| **Model load (cached)** | ~1s | ~3-5s |
### TTS Generation (1.7B model, ~20s output)
- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time)
- **MLX (M2 Max):** ~8-12s (faster than real-time)
- **Improvement:** ~4-5x faster
### Whisper Transcription (10s audio clip)
- **PyTorch CPU:** ~5-8s
- **MLX:** ~1-2s
- **Improvement:** ~3-4x faster
### Memory Usage (1.7B model)
- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM)
- **MLX:** ~4-6GB (unified memory, better optimization)
- **Improvement:** ~40% less RAM
Full benchmarks will be in `docs/overview/performance.md` after Phase 6.
## Open Questions
- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure
- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No
- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing
- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users.
## Success Metrics
How we'll know this worked:
1. **Performance:** Apple Silicon users report generation faster than real-time
2. **Adoption:** >80% of macOS downloads are MLX build within 1 month
3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions)
4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance
## Related Work
- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly
- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API
- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data
## Next Steps
1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/`
2. Get approval on dual-backend architecture
3. Start Phase 1 (platform detection)
## Questions?
Feedback welcome in GitHub discussions or Discord.
docs/content/docs/plans/OPENAI_SUPPORT.md
+238
View File
@@ -0,0 +1,238 @@
---
title: "OpenAI API Compatibility"
description: "OpenAI API compatibility for Voicebox (Planned)"
---
**Status:** Planned for v0.2.0
**Issue:** [#10 OpenAI API compatibility](https://github.com/jamiepine/voicebox/issues/10)
## Overview
This feature exposes OpenAI-compatible endpoints from Voicebox, allowing any tool, library, or application that speaks the OpenAI Audio API to use Voicebox as a drop-in local replacement.
```mermaid
flowchart LR
subgraph clients [External Clients]
SDK[OpenAI SDK]
Curl[curl / HTTP]
Apps[Third-party Apps]
end
subgraph voicebox [Voicebox Server]
OpenAI["/v1/audio/* endpoints"]
TTS[TTSModel]
Whisper[WhisperModel]
Profiles[Voice Profiles]
end
SDK --> OpenAI
Curl --> OpenAI
Apps --> OpenAI
OpenAI --> TTS
OpenAI --> Whisper
OpenAI --> Profiles
```
## Use Cases
- **OpenAI SDK users**: `openai.audio.speech.create()` works with Voicebox
- **LLM frameworks**: LangChain, AutoGen, etc. can use Voicebox for TTS
- **Shell scripts**: `curl` commands copy-pasted from OpenAI docs work
- **Existing integrations**: Any tool expecting OpenAI's API works without code changes
## Endpoints to Implement
### 1. `POST /v1/audio/speech` (TTS)
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createSpeech
**Request:**
```json
{
"model": "tts-1",
"input": "Hello world!",
"voice": "alloy",
"response_format": "mp3",
"speed": 1.0
}
```
**Response:** Audio file (mp3, wav, opus, aac, flac, pcm)
**Voice Mapping Strategy:**
- `voice` parameter maps to Voicebox profile names (case-insensitive)
- If no match, use a configurable default profile
- Support special syntax: `voice: "profile:uuid"` for explicit profile ID
### 2. `POST /v1/audio/transcriptions` (Whisper)
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createTranscription
**Request:** (multipart/form-data)
- `file`: Audio file
- `model`: "whisper-1"
- `language`: Optional language hint
- `response_format`: json, text, srt, verbose_json, vtt
**Response:**
```json
{
"text": "Hello world!"
}
```
## Implementation Details
### New File: `backend/openai_compat.py`
Create a dedicated module with an APIRouter for OpenAI-compatible endpoints:
```python
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from typing import Literal, Optional
router = APIRouter(prefix="/v1/audio", tags=["OpenAI Compatible"])
class SpeechRequest(BaseModel):
model: str = "tts-1"
input: str
voice: str = "alloy"
response_format: Literal["mp3", "wav", "opus", "aac", "flac", "pcm"] = "mp3"
speed: float = 1.0
@router.post("/speech")
async def create_speech(request: SpeechRequest, db: Session = Depends(get_db)):
# 1. Map voice name to profile
# 2. Generate audio using existing TTSModel
# 3. Convert to requested format
# 4. Return audio stream
...
@router.post("/transcriptions")
async def create_transcription(
file: UploadFile = File(...),
model: str = Form("whisper-1"),
language: Optional[str] = Form(None),
response_format: str = Form("json"),
):
# 1. Save uploaded file
# 2. Transcribe using existing WhisperModel
# 3. Return in requested format
...
```
### Voice Profile Resolution
Add helper in [backend/profiles.py](backend/profiles.py):
```python
async def resolve_voice_for_openai(voice: str, db: Session) -> Optional[VoiceProfile]:
"""
Resolve OpenAI voice parameter to a Voicebox profile.
Priority:
1. Exact profile name match (case-insensitive)
2. Profile ID match (if voice starts with "profile:")
3. Default profile from config
4. First available profile
"""
...
```
### Audio Format Conversion
Add conversion utilities in [backend/utils/audio.py](backend/utils/audio.py):
```python
def convert_audio_format(
audio: np.ndarray,
sample_rate: int,
target_format: str, # mp3, wav, opus, aac, flac, pcm
) -> bytes:
"""Convert audio to target format using ffmpeg or pydub."""
...
```
### Configuration
Add to [backend/config.py](backend/config.py):
```python
# OpenAI API Compatibility
OPENAI_COMPAT_ENABLED = True
OPENAI_COMPAT_DEFAULT_VOICE = None # Profile ID or name for default voice
OPENAI_COMPAT_REQUIRE_AUTH = False # Require API key validation
OPENAI_COMPAT_API_KEY = None # If set, validate against this
```
### Integration with main.py
In [backend/main.py](backend/main.py), include the router:
```python
from . import openai_compat
# Add OpenAI-compatible routes
if config.OPENAI_COMPAT_ENABLED:
app.include_router(openai_compat.router)
```
## Streaming Support (Future Enhancement)
Initial implementation returns complete audio. Streaming can be added later:
```python
@router.post("/speech")
async def create_speech(request: SpeechRequest):
if request.stream:
return StreamingResponse(
generate_audio_chunks(request),
media_type=f"audio/{request.response_format}"
)
...
```
## Testing
Example usage after implementation:
```bash
# TTS with curl
curl http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "tts-1", "input": "Hello!", "voice": "MyProfile"}' \
--output speech.mp3
# With OpenAI Python SDK
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.audio.speech.create(
model="tts-1",
voice="MyProfile",
input="Hello world!"
)
response.stream_to_file("output.mp3")
# Transcription
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@audio.mp3 \
-F model="whisper-1"
```
## Security Considerations
- Optional API key validation (for shared deployments)
- Rate limiting on endpoints
- Input length limits (same as existing `/generate` endpoint)
## Dependencies
- `pydub` or `ffmpeg-python` for audio format conversion (mp3, opus, etc.)
- No changes to existing TTS/Whisper model code
docs/content/docs/plans/meta.json
+9
View File
@@ -0,0 +1,9 @@
{
"title": "Plans",
"pages": [
"DOCKER_DEPLOYMENT",
"EXTERNAL_PROVIDERS",
"MLX_AUDIO",
"OPENAI_SUPPORT"
]
}