Compare commits

...
57 Commits
Author SHA1 Message Date
Jamie Pine 1b66a528d1 Enhance README and UI Components for Performance and Features
- Updated README.md to highlight MLX backend performance improvements on Mac with Metal acceleration.
- Refined ProfileCard and ProfileForm components by optimizing imports and improving error handling for avatar uploads.
- Adjusted landing page content to better describe features, including a new multi-voice narrative editor and performance optimizations for different platforms.
- Bumped version to 0.1.11 in Cargo.lock to reflect recent changes.
2026-01-30 02:53:15 -08:00
Jamie Pine bef4092e6e Bump version: 0.1.10 → 0.1.11 2026-01-30 02:28:08 -08:00
Jamie Pine 9654f7b642 Refactor MLX and PyTorch Backend Model Loading
- Updated hidden imports in build_binary.py to replace 'mlx_audio.asr' with 'mlx_audio.stt'.
- Enhanced model loading logic in MLX and PyTorch backends to ensure proper progress tracking during model downloads.
- Improved error handling and context management for progress tracking in both backends.
- Bumped version to 0.1.10 in Cargo.lock to reflect recent changes.
2026-01-30 02:26:50 -08:00
Jamie Pine eba1244add Bump version: 0.1.9 → 0.1.10 2026-01-29 23:12:16 -08:00
Jamie Pine 94487f32a5 Enhance MLX and PyTorch Backend Integration
- Added support for MLX backend on Apple Silicon, enabling optimized performance for TTS and STT tasks.
- Implemented platform detection to dynamically select between MLX and PyTorch based on the runtime environment.
- Updated build process to include MLX-specific dependencies and configurations for macOS.
- Refactored backend code to improve model loading and inference logic, accommodating backend-specific requirements.
- Enhanced documentation to clarify backend selection and performance benefits for different platforms.
- Streamlined installation instructions and troubleshooting guidance for MLX-related issues.
2026-01-29 23:11:48 -08:00
Jamie Pine 081f45e680 ADDED MLX FOR SUPER FAST GENERATIONS ON APPLE SILICON
- Added support for MLX backend on Apple Silicon, enabling optimized performance for TTS and STT tasks.
- Updated release workflow to include MLX-specific dependencies and configurations for macOS platforms.
- Refactored backend code to dynamically select between MLX and PyTorch based on the runtime environment.
- Enhanced model loading and inference logic to accommodate backend-specific requirements, including updated model IDs and hidden imports.
- Improved health check and model status reporting to reflect the active backend type.
- Streamlined caching mechanisms to support both backend types, ensuring compatibility and performance.
2026-01-29 21:50:46 -08:00
Jamie Pine 86768288ce Enhance MLX Audio Documentation and Testing Framework
- Updated MLX_AUDIO.md to reflect validated status and included detailed validation results, model mapping, and API usage examples.
- Added a demo script (demo.py) for testing audio generation speed and functionality.
- Introduced a test script (test_tts.py) to validate MLX audio model loading and generation, ensuring robust testing for future developments.
- Created a .gitignore file in the mlx-test directory to exclude unnecessary files from version control.
2026-01-29 21:28:22 -08:00
Jamie Pine 0fd063442a Remove risks and mitigations section from MLX_AUDIO.md and update open questions with responses for clarity. This streamlines the documentation and provides clearer guidance on future considerations. 2026-01-29 21:08:28 -08:00
Jamie Pine a0c2493e98 Add MLX Audio Integration for Apple Silicon Support
- Introduced a new backend for MLX audio to enable GPU acceleration on macOS Apple Silicon, improving performance and user experience.
- Implemented platform detection to switch between MLX and PyTorch backends based on the runtime environment.
- Added new streaming capabilities for TTS and STT, enhancing real-time audio generation.
- Updated API endpoints and frontend components to support new features while maintaining backward compatibility.
- Created documentation for backend integration and performance comparisons.
2026-01-29 20:57:52 -08:00
Jamie Pine b39f48cc81 Refactor useGenerationForm to streamline model download handling
- Removed unnecessary isDownloading variable and related logic.
- Consolidated model download state reset to the finally block for improved clarity and reliability.
- Enhanced error handling by ensuring model download state is reset in case of failure.
2026-01-29 20:17:55 -08:00
Jamie Pine 4ff775bc98 Update packageManager version in package.json to [email protected] 2026-01-29 19:53:38 -08:00
Jamie Pine 6351aa75e9 Refactor StoryList component for improved readability and organization
- Reorganized import statements for clarity and consistency.
- Adjusted formatting of state declarations for better readability.
- Streamlined JSX structure for improved visual hierarchy.
- Updated dialog descriptions for consistency in presentation.
- Made minor adjustments to spacing and layout for enhanced UI consistency.
2026-01-29 19:46:27 -08:00
Jamie Pine 43873a883b Update StoryList component styles for improved UI consistency
- Changed border radius of the "No stories yet" message to rounded-2xl for a softer appearance.
- Updated story item borders to rounded-2xl to enhance visual cohesion across the component.
2026-01-29 19:34:28 -08:00
Jamie Pine 60012b81c0 Refactor ProfileForm and SampleList components for improved UI and functionality
- Updated button styles in ProfileForm for better visual consistency and user experience.
- Replaced Pencil icon with Edit in SampleList for clearer action representation.
- Introduced CircleButton component for action buttons in SampleList, enhancing UI responsiveness and clarity.
- Improved layout and hover effects for action buttons in SampleList to streamline user interactions.
2026-01-29 19:32:05 -08:00
Jamie Pine 89f3127c37 Implement avatar upload and management for voice profiles
- Added functionality to upload, delete, and retrieve avatar images for voice profiles.
- Introduced new API endpoints for avatar management, including upload and delete operations.
- Enhanced profile forms and components to support avatar image handling, including previews and error handling.
- Updated database schema to include avatar_path for profiles and added necessary migrations.
- Implemented image validation and processing utilities to ensure proper avatar uploads.
2026-01-29 19:28:42 -08:00
Jamie Pine ef3c3a7f8c Refactor ProfileForm for improved readability and maintainability
- Reorganized import statements for clarity.
- Enhanced conditional checks for restoring saved files with improved formatting.
- Streamlined draft saving logic by consolidating variable declarations.
- Updated UI components for better structure and readability in the form layout.
2026-01-29 18:56:22 -08:00
Jamie Pine 7b5e73cfa8 Add .npmrc for bun usage and update dependencies
- Created a new .npmrc file to enforce bun usage.
- Bumped version numbers for multiple packages to 0.1.9 in bun.lock.
- Added react-sound-visualizer dependency to enhance audio visualization features.
- Introduced convert:assets script in package.json for asset optimization.
- Updated CONTRIBUTING.md with instructions for converting assets to web formats.
- Added documentation files for API endpoints and developer guidelines in the docs directory.
2026-01-29 18:56:10 -08:00
Jamie Pine 462f104494 Enhance ProgressManager for thread safety and event loop integration
- Added thread-safe mechanisms to the ProgressManager for handling model download progress updates.
- Introduced a main event loop setter to ensure safe operations from background threads.
- Improved listener notification to handle updates in a thread-safe manner.
- Updated methods to ensure thread safety when accessing progress data.
2026-01-29 16:23:46 -08:00
Jamie Pine fadb57164e Update README to reflect API endpoint changes and enhance profile creation example
- Updated API endpoints from `/api/...` to `/...` for consistency.
- Modified the speech generation example to include a language parameter.
- Revised the profile creation example to use JSON format instead of form data.
2026-01-29 16:14:19 -08:00
Jamie Pine e870d65136 Add badges to README for downloads, releases, stars, and license 2026-01-29 16:09:10 -08:00
Jamie PineandGitHub 3df40278cc Merge pull request #5 from Snowy7/fix/dev-mode-sidecar
Fix dev mode sidecar and cross-platform HuggingFace cache paths
2026-01-29 16:05:00 -08:00
Jamie PineandGitHub deeef5a474 Merge pull request #12 from tomasmach/feat/makefile
feat: add Makefile for streamlined development workflow
2026-01-29 16:04:48 -08:00
Jamie Pine 236e464525 Bump version: 0.1.8 → 0.1.9 2026-01-29 15:58:31 -08:00
Jamie Pine cf3cf3f002 Enhance model download handling in useGenerationForm and ProgressManager
- Introduced a flag to track download status in useGenerationForm, ensuring proper UI updates during model downloads.
- Updated ProgressManager to only send initial progress updates if the model is actively downloading or extracting, preventing outdated status messages from being sent.
- Improved error handling and logging for better visibility into model download processes.
2026-01-29 15:57:53 -08:00
tomasmach 9d98e1e768 fix: improve Makefile robustness and update CONTRIBUTING docs
- Add exit 1 to test-backend when pytest not installed
- Add exit 1 to test-frontend when no test script configured
- Add venv dependency to db-init target
- Document Makefile usage in CONTRIBUTING.md
2026-01-30 00:51:41 +01:00
Jamie Pine 3be8980f48 Refactor ProfileForm to support draft state management and improve file handling
- Introduced functionality to save and restore form state as a draft when creating a new voice profile.
- Added helper functions for converting files to and from base64 format to facilitate file handling.
- Updated the API types to use a more flexible LanguageCode type for language parameters.
- Enhanced the UI store to manage profile form drafts, improving user experience during profile creation.
2026-01-29 15:45:14 -08:00
tomasmach 76bc070f5b docs: update CHANGELOG with Makefile feature 2026-01-30 00:44:10 +01:00
tomasmach 01838f4773 docs: add Makefile reference and setup instructions to README 2026-01-30 00:35:20 +01:00
tomasmach 39e4f9d08c fix: correct backend server port to match frontend expectations (17493) 2026-01-30 00:33:50 +01:00
Jamie Pine 341d71470c Implement auto-scroll feature in StoryTrackEditor to keep playhead centered during playback
- Added a useEffect hook to automatically scroll the timeline when the playhead moves past the halfway point of the visible area, enhancing user experience during playback.
2026-01-29 15:32:34 -08:00
Jamie Pine bb6cea24ba Refactor StoryTrackEditor to account for time ruler height during drag operations
- Introduced a constant for TIME_RULER_HEIGHT to improve code readability.
- Updated drag position calculations to subtract the time ruler height, ensuring accurate positioning of clips relative to the tracks area.
2026-01-29 15:31:23 -08:00
Jamie Pine fa7ac88abc Reset playback timing anchors in story store for fresh initialization by playback hook 2026-01-29 15:27:27 -08:00
Jamie Pine c68ddc45b1 Enhance contribution guidelines and improve FloatingGenerateBox component
- Updated CONTRIBUTING.md to include instructions for building with a local Qwen3-TTS development version, facilitating easier testing and development.
- Refactored FloatingGenerateBox component to streamline the rendering of text and instruct fields, improving code readability and maintainability.
- Added functionality to handle auto-resizing of text areas based on content changes, enhancing user experience.
- Improved event handling for keyboard interactions in StoryTrackEditor, allowing for play/pause functionality with the spacebar.
- Introduced a MiniSamplePlayer component in SampleList for better audio playback control, including play, pause, and seek features.
- Implemented sample update functionality in the backend, allowing users to edit reference text for audio samples, with appropriate error handling and user feedback.
2026-01-29 15:25:40 -08:00
tomasmach f89dc66d0c feat: add Python version fallback (3.12 > 3.13 > python3) and compatibility warning 2026-01-30 00:10:30 +01:00
tomasmach cba7d7bc23 feat: add Makefile for streamlined development workflow 2026-01-30 00:05:36 +01:00
Jamie PineandGitHub 3c89b068f3 Merge pull request #6 from jamiepine/windows-server-shutdown
Windows server shutdown
2026-01-29 03:12:40 -08:00
Jamie Pine 229841e05e Add GPU type information to health check response
- Updated the health check endpoint to include the type of GPU available (CUDA or MPS).
- Modified the HealthResponse model to accommodate the new gpu_type field, enhancing the response with detailed GPU information.
- This change improves the clarity of system capabilities for users and developers.
2026-01-29 03:12:11 -08:00
Jamie Pine 123e8215e4 Merge branch 'main' into windows-server-shutdown 2026-01-29 03:00:08 -08:00
Jamie Pine 2a3afec2ca Implement graceful shutdown for the server and enhance process management on Windows
- Added a new `/shutdown` endpoint to allow graceful server shutdown via HTTP.
- Implemented process tree management functions to handle child processes during shutdown on Windows.
- Updated the `stop_server` function to attempt graceful shutdown before forcefully terminating processes.
- Enhanced error handling and logging for shutdown operations.
2026-01-29 02:58:21 -08:00
Jamie Pine 99ddd5a0b4 Add asynchronous model download handling for TTS and Whisper models
- Implemented background tasks for downloading TTS and Whisper models to prevent blocking HTTP responses.
- Enhanced error handling during model downloads, providing users with real-time feedback on download status.
- Updated HTTP responses to indicate when models are being downloaded, improving user experience during model initialization.
2026-01-29 02:55:17 -08:00
Jamie Pine 8d730621bc Refactor model download handling to use background tasks
- Moved model download logic into a separate asynchronous function to allow non-blocking HTTP responses.
- Improved error handling by tracking download status and reporting errors without interrupting the main request flow.
- The frontend is now expected to poll the progress endpoint for download status updates.
2026-01-29 02:42:02 -08:00
Jamie Pine e23118f610 Bump version: 0.1.7 → 0.1.8 2026-01-29 02:21:01 -08:00
Jamie Pine d4bfdc0d68 Update version handling in backend and improve HuggingFace cache management
- Added __version__ variable in backend/__init__.py to centralize versioning.
- Updated main.py to use __version__ for API versioning in the FastAPI app.
- Enhanced cache directory handling by utilizing HuggingFace's constants for improved compatibility across platforms.
2026-01-29 02:20:33 -08:00
Jamie Pine 116c108906 Update screenshot asset in landing page for consistency with current design 2026-01-29 00:06:06 -08:00
Jamie Pine 2d23c8e06a Swap screenshot assets in landing page for improved visual representation
- Replaced app screenshot paths to ensure correct images are displayed.
- Adjusted alt text for screenshots to accurately reflect their content.
2026-01-29 00:06:00 -08:00
Jamie Pine 3973a59ba3 Revise README to clarify Voicebox features and benefits
- Changed section title from "Why Voicebox?" to "What is Voicebox?" for better clarity.
- Expanded description to emphasize local-first voice cloning capabilities and professional tools.
- Highlighted privacy, model flexibility, and native performance as key advantages over cloud services.
2026-01-28 23:59:47 -08:00
Jamie Pine d9aa75253a Enhance README with new features and multi-track editor details
- Added multi-sample support for higher quality cloning.
- Introduced a new Stories Editor section with features for multi-track composition, inline audio editing, auto-playback, and voice mixing.
- Updated recording section to include system audio capture for macOS and Windows.
2026-01-28 23:55:44 -08:00
Jamie Pine b22bf36565 Update README and landing page with new screenshots; bump version to 0.1.7
- Replaced existing screenshot paths in README and landing page with new assets.
- Added additional screenshots to the landing page for enhanced visual representation.
- Updated version in Cargo.lock from 0.1.6 to 0.1.7.
2026-01-28 23:51:43 -08:00
Jamie Pine 33f4ed9b44 Bump version: 0.1.6 → 0.1.7 2026-01-28 22:28:18 -08:00
Jamie Pine cc37e04221 Refactor HistoryTable and SampleList components for improved code consistency
- Cleaned up formatting in HistoryTable for better readability.
- Adjusted import statements in SampleList to maintain consistent structure.
2026-01-28 22:28:01 -08:00
Jamie Pine 2b4fbe5173 Refactor AudioPlayer and related components to support conditional auto-play functionality
- Updated AudioPlayer to auto-play only if the shouldAutoPlay flag is set, enhancing user control over playback.
- Refactored HistoryTable, SampleList, and useGenerationForm to utilize setAudioWithAutoPlay for consistent audio loading and playback behavior.
- Improved user experience by ensuring audio is only played when explicitly intended, reducing unexpected playback.
2026-01-28 22:27:37 -08:00
Snowy 423d69b7cc Use HuggingFace's built-in cache detection for cross-platform support
Replace hardcoded ~/.cache/huggingface/hub paths with
huggingface_hub.constants.HF_HUB_CACHE which correctly handles
OS-specific cache locations (Windows uses AppData, etc.)
2026-01-29 09:25:41 +03:00
Jamie Pine ea943876dc formatting 2026-01-28 22:23:27 -08:00
Jamie Pine b55d8cc567 Implement auto-activation of stories in StoryTrackEditor and improve playback state management
- Added useEffect to automatically activate the story when the editor is shown, ensuring the playhead is visible.
- Introduced setActiveStory function in storyStore to manage story activation without playback.
- Updated playback state checks to reflect the current playing status accurately.
- Enhanced UI to always display the playhead for better user experience during playback.
2026-01-28 22:22:47 -08:00
Jamie Pine 036d90dc8e Enhance story item management with trimming, splitting, and duplication features
- Updated StoryTrackEditor and StoryContent components to support trimming and splitting of story items.
- Introduced new API endpoints for trimming, splitting, and duplicating story items, enhancing item management capabilities.
- Refactored related hooks and state management to accommodate new functionalities.
- Improved data models to include trim start and end times for better audio playback control.
- Enhanced UI interactions for selecting and managing story items within the track editor.
2026-01-28 22:16:53 -08:00
Snowy c513451277 Fix dev mode to work without pre-built server binary
Previously, running `bun run dev` would fail because Tauri requires
the sidecar binary to exist at compile time, even in development mode.
This forced developers to build the full PyInstaller binary before
they could start development.

This change introduces a streamlined dev workflow:

1. Add `scripts/setup-dev-sidecar.js` - Creates minimal placeholder
   binaries that satisfy Tauri's compile-time check. Works cross-platform
   (Windows PE stub, Unix shell script).

2. Update Rust code to gracefully handle dev mode - When the sidecar
   fails to start, it checks if a manually-started server is already
   running on the expected port and connects to it instead.

3. Update npm scripts - `bun run dev` now auto-runs the setup script,
   and `dev:server` uses the correct port (17493).

4. Update CONTRIBUTING.md with clearer dev workflow documentation.

New development workflow:
  Terminal 1: bun run dev:server
  Terminal 2: bun run dev

The bundled binary is only required for production builds.
2026-01-29 09:11:41 +03:00
Jamie PineandGitHub 27ae6dfbab Merge pull request #3 from jamiepine/stories
Stories
2026-01-28 21:18:19 -08:00
124 changed files with 15291 additions and 1549 deletions
+4 -4
View File
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.1.6
current_version = 0.1.11
commit = True
tag = True
tag_name = v{new_version}
@@ -34,6 +34,6 @@ replace = "version": "{new_version}"
search = "version": "{current_version}"
replace = "version": "{new_version}"
[bumpversion:file:backend/main.py]
search = "version": "{current_version}"
replace = "version": "{new_version}"
[bumpversion:file:backend/__init__.py]
search = __version__ = "{current_version}"
replace = __version__ = "{new_version}"
+11 -1
View File
@@ -17,15 +17,19 @@ jobs:
- platform: 'macos-latest'
args: '--target aarch64-apple-darwin'
python-version: '3.12'
backend: 'mlx'
- platform: 'macos-15-intel'
args: '--target x86_64-apple-darwin'
python-version: '3.12'
backend: 'pytorch'
# - platform: 'ubuntu-22.04'
# args: ''
# python-version: '3.12'
# backend: 'pytorch'
- platform: 'windows-latest'
args: ''
python-version: '3.12'
backend: 'pytorch'
runs-on: ${{ matrix.platform }}
@@ -57,6 +61,11 @@ jobs:
pip install pyinstaller
pip install -r backend/requirements.txt
- name: Install MLX dependencies (Apple Silicon only)
if: matrix.backend == 'mlx'
run: |
pip install -r backend/requirements-mlx.txt
- name: Build Python server (Linux/macOS)
if: matrix.platform != 'windows-latest'
run: |
@@ -133,7 +142,8 @@ jobs:
See the assets below to download and install this version.
### Installation
- **macOS**: Download the `.dmg` file
- **macOS (Apple Silicon)**: Download the `aarch64.dmg` file - uses MLX for fast native inference
- **macOS (Intel)**: Download the `x64.dmg` file - uses PyTorch
- **Windows**: Download the `.msi` installer
- **Linux**: Download the `.AppImage` or `.deb` package
+2
View File
@@ -0,0 +1,2 @@
# Force bun usage
engine-strict=true
+14
View File
@@ -53,6 +53,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks
- Includes Python version detection and compatibility warnings
- Self-documenting help system with `make help`
- Colored output for better readability
- Supports parallel development server execution
### Changed
- **README** - Added Makefile reference and updated Quick Start with Makefile-based setup instructions alongside manual setup
---
## [Unreleased - Planned]
### Planned
- Real-time streaming synthesis
- Conversation mode with multiple speakers
+65 -17
View File
@@ -32,6 +32,10 @@ Thank you for your interest in contributing to Voicebox! This document provides
### Development Setup
**Using the Makefile (recommended for macOS/Linux):** Run `make setup` to install all dependencies, then `make dev` to start development servers. See `make help` for all available commands.
**Manual setup (required for Windows):**
1. **Fork and clone the repository**
```bash
git clone https://github.com/YOUR_USERNAME/voicebox.git
@@ -62,37 +66,43 @@ Thank you for your interest in contributing to Voicebox! This document provides
# Install Python dependencies
pip install -r requirements.txt
# Install MLX dependencies (Apple Silicon only - for faster inference)
# On Apple Silicon, this enables native Metal acceleration
if [[ $(uname -m) == "arm64" ]]; then
pip install -r requirements-mlx.txt
fi
# Install Qwen3-TTS (required for voice synthesis)
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
4. **Initialize database**
```bash
cd backend
python -c "from database import init_db; init_db()"
```
This creates the SQLite database at `data/voicebox.db`.
4. **Start development servers**
5. **Start development servers**
**Terminal 1: Backend server**
Development requires two terminals: one for the Python backend, one for the Tauri app.
**Terminal 1: Backend server** (start this first)
```bash
cd backend
source venv/bin/activate # Activate venv if not already active
bun run dev:server
# Or manually: uvicorn main:app --reload --port 8000
# Or manually: uvicorn main:app --reload --port 17493
```
Backend will be available at `http://localhost:8000`
Backend will be available at `http://localhost:17493`
**Terminal 2: Desktop app**
```bash
bun run dev
```
This will:
- Create a placeholder sidecar binary (for Tauri compilation)
- Start Vite dev server on port 5173
- Launch Tauri window pointing to localhost:5173
- Connect to the Python server you started in Terminal 1
- Enable hot reload
> **Note:** In dev mode, the app connects to your manually-started Python server.
> The bundled server binary is only used in production builds.
**Optional: Web app**
```bash
bun run dev:web
@@ -109,18 +119,36 @@ First-time usage will be slower due to model downloads, but subsequent runs will
### Building
**Build Python server binary:**
**Build everything (recommended):**
```bash
bun run build
```
This automatically:
1. Builds the Python server binary (`./scripts/build-server.sh`)
2. Builds the Tauri desktop app (`cd tauri && bun run tauri build`)
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`) in `tauri/src-tauri/target/release/bundle/`.
**Note:** The build process detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
**Build server binary only:**
```bash
bun run build:server
# or
./scripts/build-server.sh
```
Creates platform-specific binary in `tauri/src-tauri/binaries/`
**Build Tauri desktop app:**
**Building with local Qwen3-TTS development version:**
If you're actively developing or modifying the Qwen3-TTS library, set the `QWEN_TTS_PATH` environment variable to point to your local clone:
```bash
cd tauri
bun run tauri build
export QWEN_TTS_PATH=~/path/to/your/Qwen3-TTS
bun run build:server
```
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`)
This makes PyInstaller use your local qwen-tts version instead of the pip-installed package. Useful when testing changes to the TTS library before they're published to PyPI or when using an editable install (`pip install -e`).
**Build web app:**
```bash
@@ -137,6 +165,26 @@ After starting the backend server:
```
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
### Convert Assets to Web Formats
To optimize images and videos for the web, run:
```bash
bun run convert:assets
```
This script:
- Converts PNG → WebP (better compression, same quality)
- Converts MOV → WebM (VP9 codec, smaller file size)
- Processes files in `landing/public/` and `docs/public/`
- **Deletes original files** after successful conversion
**Requirements:** Install `webp` and `ffmpeg`:
```bash
brew install webp ffmpeg
```
> **Note:** Run this before committing new images or videos to keep the repository size small.
## Development Workflow
### 1. Create a Branch
+245
View File
@@ -0,0 +1,245 @@
# Voicebox Makefile
# Unix-only (macOS/Linux). Windows users should use WSL.
SHELL := /bin/bash
.DEFAULT_GOAL := help
# Directories
BACKEND_DIR := backend
TAURI_DIR := tauri
WEB_DIR := web
APP_DIR := app
# Python (prefer 3.12, fallback to 3.13, then python3)
PYTHON := $(shell command -v python3.12 2>/dev/null || command -v python3.13 2>/dev/null || echo python3)
VENV := $(CURDIR)/$(BACKEND_DIR)/venv
VENV_BIN := $(VENV)/bin
PIP := $(VENV_BIN)/pip
PYTHON_VENV := $(VENV_BIN)/python
# Colors for output
BLUE := \033[0;34m
GREEN := \033[0;32m
YELLOW := \033[0;33m
NC := \033[0m # No Color
.PHONY: help
help: ## Show this help message
@echo -e "$(BLUE)Voicebox$(NC) - Development Commands"
@echo ""
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
awk 'BEGIN {FS = ":.*?## "}; {printf " $(GREEN)%-20s$(NC) %s\n", $$1, $$2}'
# =============================================================================
# SETUP
# =============================================================================
.PHONY: setup setup-js setup-python setup-rust
setup: setup-js setup-python ## Full project setup (all dependencies)
@echo -e "$(GREEN)✓ Setup complete!$(NC)"
@echo -e " Run $(YELLOW)make dev$(NC) to start development servers"
setup-js: ## Install JavaScript dependencies (bun)
@echo -e "$(BLUE)Installing JavaScript dependencies...$(NC)"
bun install
setup-python: $(VENV)/bin/activate ## Set up Python virtual environment and dependencies
@echo -e "$(BLUE)Installing Python dependencies...$(NC)"
$(PIP) install --upgrade pip
$(PIP) install -r $(BACKEND_DIR)/requirements.txt
@if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \
echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \
$(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \
echo -e "$(GREEN)✓ MLX backend enabled (native Metal acceleration)$(NC)"; \
fi
$(PIP) install git+https://github.com/QwenLM/Qwen3-TTS.git
@echo -e "$(GREEN)✓ Python environment ready$(NC)"
$(VENV)/bin/activate:
@echo -e "$(BLUE)Creating Python virtual environment...$(NC)"
@PY_MINOR=$$($(PYTHON) -c "import sys; print(sys.version_info[1])"); \
if [ "$$PY_MINOR" -gt 13 ]; then \
echo -e "$(YELLOW)Warning: Python 3.$$PY_MINOR detected. ML packages may not be compatible.$(NC)"; \
echo -e "$(YELLOW)Recommended: Use Python 3.12 or 3.13 (brew install [email protected])$(NC)"; \
fi
$(PYTHON) -m venv $(VENV)
setup-rust: ## Install Rust toolchain (if not present)
@command -v rustc >/dev/null 2>&1 || curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# =============================================================================
# DEVELOPMENT
# =============================================================================
.PHONY: dev dev-backend dev-frontend dev-web kill-dev
dev: ## Start backend + desktop app (parallel)
@echo -e "$(BLUE)Starting development servers...$(NC)"
@echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)"
@trap 'kill 0' EXIT; \
$(MAKE) dev-backend & \
sleep 2 && $(MAKE) dev-frontend & \
wait
dev-backend: ## Start FastAPI backend server
@echo -e "$(BLUE)Starting backend server on http://localhost:17493$(NC)"
$(VENV_BIN)/uvicorn backend.main:app --reload --port 17493
dev-frontend: ## Start Tauri desktop app
@echo -e "$(BLUE)Starting Tauri desktop app...$(NC)"
bun run dev
dev-web: ## Start backend + web app (parallel)
@echo -e "$(BLUE)Starting web development servers...$(NC)"
@trap 'kill 0' EXIT; \
$(MAKE) dev-backend & \
sleep 2 && cd $(WEB_DIR) && bun run dev & \
wait
kill-dev: ## Kill all development processes
@echo -e "$(YELLOW)Killing development processes...$(NC)"
-pkill -f "uvicorn main:app" 2>/dev/null || true
-pkill -f "vite" 2>/dev/null || true
@echo -e "$(GREEN)✓ Processes killed$(NC)"
# =============================================================================
# BUILD
# =============================================================================
.PHONY: build build-server build-tauri build-web
build: build-server build-tauri ## Build everything (server binary + desktop app)
@echo -e "$(GREEN)✓ Build complete!$(NC)"
build-server: ## Build Python server binary
@echo -e "$(BLUE)Building server binary...$(NC)"
PATH="$(VENV_BIN):$$PATH" ./scripts/build-server.sh
build-tauri: ## Build Tauri desktop app
@echo -e "$(BLUE)Building Tauri desktop app...$(NC)"
cd $(TAURI_DIR) && bun run tauri build
build-web: ## Build web app
@echo -e "$(BLUE)Building web app...$(NC)"
cd $(WEB_DIR) && bun run build
@echo -e "$(GREEN)✓ Web build output in $(WEB_DIR)/dist/$(NC)"
# =============================================================================
# DATABASE & API
# =============================================================================
.PHONY: db-init db-reset generate-api
db-init: $(VENV)/bin/activate ## Initialize SQLite database
@echo -e "$(BLUE)Initializing database...$(NC)"
cd $(BACKEND_DIR) && $(PYTHON_VENV) -c "from database import init_db; init_db()"
@echo -e "$(GREEN)✓ Database created at $(BACKEND_DIR)/data/voicebox.db$(NC)"
db-reset: ## Reset database (delete and reinitialize)
@echo -e "$(YELLOW)Resetting database...$(NC)"
rm -f $(BACKEND_DIR)/data/voicebox.db
$(MAKE) db-init
generate-api: ## Generate TypeScript API client from OpenAPI schema
@echo -e "$(BLUE)Generating API client...$(NC)"
@echo -e "$(YELLOW)Note: Backend must be running (make dev-backend)$(NC)"
./scripts/generate-api.sh
@echo -e "$(GREEN)✓ API client generated in $(APP_DIR)/src/lib/api/$(NC)"
# =============================================================================
# CODE QUALITY
# =============================================================================
.PHONY: lint format typecheck check
lint: ## Run linter (Biome)
@echo -e "$(BLUE)Linting...$(NC)"
bun run lint
format: ## Format code (Biome)
@echo -e "$(BLUE)Formatting...$(NC)"
bun run format
typecheck: ## Run TypeScript type checking
@echo -e "$(BLUE)Type checking...$(NC)"
bun run tsc --noEmit
check: ## Run all checks (Biome lint + format + type check)
@echo -e "$(BLUE)Running all checks...$(NC)"
bun run check
@echo -e "$(GREEN)✓ All checks passed$(NC)"
# =============================================================================
# TESTING
# =============================================================================
.PHONY: test test-backend test-frontend
test: test-backend test-frontend ## Run all tests
@echo -e "$(GREEN)✓ All tests passed$(NC)"
test-backend: ## Run Python backend tests (requires pytest)
@echo -e "$(BLUE)Running backend tests...$(NC)"
@if [ -f "$(VENV_BIN)/pytest" ]; then \
cd $(BACKEND_DIR) && $(VENV_BIN)/pytest -v; \
else \
echo -e "$(YELLOW)pytest not installed. Run: $(PIP) install pytest$(NC)"; \
exit 1; \
fi
test-frontend: ## Run frontend tests (requires test script in package.json)
@echo -e "$(BLUE)Running frontend tests...$(NC)"
@if bun run test --help >/dev/null 2>&1; then \
bun run test; \
else \
echo -e "$(YELLOW)No test script configured$(NC)"; \
exit 1; \
fi
# =============================================================================
# LOGS & DEBUGGING
# =============================================================================
.PHONY: logs docs
logs: ## Tail backend logs
@echo -e "$(BLUE)Tailing logs (Ctrl+C to stop)...$(NC)"
tail -f $(BACKEND_DIR)/logs/*.log 2>/dev/null || echo "No log files found"
docs: ## Open API documentation (backend must be running)
@echo -e "$(BLUE)Opening API docs...$(NC)"
open http://localhost:17493/docs 2>/dev/null || xdg-open http://localhost:17493/docs
# =============================================================================
# CLEAN
# =============================================================================
.PHONY: clean clean-python clean-build clean-all
clean: ## Clean build artifacts
@echo -e "$(BLUE)Cleaning build artifacts...$(NC)"
rm -rf $(TAURI_DIR)/src-tauri/target/release
rm -rf $(WEB_DIR)/dist
rm -rf $(APP_DIR)/dist
@echo -e "$(GREEN)✓ Build artifacts cleaned$(NC)"
clean-python: ## Clean Python cache and virtual environment
@echo -e "$(BLUE)Cleaning Python files...$(NC)"
rm -rf $(VENV)
find $(BACKEND_DIR) -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
find $(BACKEND_DIR) -type f -name "*.pyc" -delete 2>/dev/null || true
@echo -e "$(GREEN)✓ Python environment cleaned$(NC)"
clean-build: ## Clean Rust/Tauri build cache
@echo -e "$(BLUE)Cleaning Rust build cache...$(NC)"
cd $(TAURI_DIR)/src-tauri && cargo clean
@echo -e "$(GREEN)✓ Rust cache cleaned$(NC)"
clean-all: clean clean-python clean-build ## Nuclear clean (everything)
@echo -e "$(BLUE)Cleaning node_modules...$(NC)"
rm -rf node_modules
rm -rf $(APP_DIR)/node_modules
rm -rf $(TAURI_DIR)/node_modules
rm -rf $(WEB_DIR)/node_modules
@echo -e "$(GREEN)✓ Full clean complete$(NC)"
+82 -19
View File
@@ -10,6 +10,21 @@
All running locally on your machine.
</p>
<p align="center">
<a href="https://github.com/jamiepine/voicebox/releases">
<img src="https://img.shields.io/github/downloads/jamiepine/voicebox/total?style=flat&color=blue" alt="Downloads" />
</a>
<a href="https://github.com/jamiepine/voicebox/releases/latest">
<img src="https://img.shields.io/github/v/release/jamiepine/voicebox?style=flat" alt="Release" />
</a>
<a href="https://github.com/jamiepine/voicebox/stargazers">
<img src="https://img.shields.io/github/stars/jamiepine/voicebox?style=flat" alt="Stars" />
</a>
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
</a>
</p>
<p align="center">
<a href="https://voicebox.sh">voicebox.sh</a> •
<a href="#download">Download</a> •
@@ -22,7 +37,7 @@
<p align="center">
<a href="https://voicebox.sh">
<img src=".github/assets/screenshot.webp" alt="Voicebox App Screenshot" width="800" />
<img src="landing/public/assets/app-screenshot-1.webp" alt="Voicebox App Screenshot" width="800" />
</a>
</p>
@@ -32,17 +47,30 @@
<br/>
## Why Voicebox?
<p align="center">
<img src="landing/public/assets/app-screenshot-2.webp" alt="Voicebox Screenshot 2" width="800" />
</p>
Voice AI is exploding, but most tools are either cloud-locked, expensive, or a nightmare to set up. Voicebox is different:
<p align="center">
<img src="landing/public/assets/app-screenshot-3.webp" alt="Voicebox Screenshot 3" width="800" />
</p>
- **100% Local** — Your voice data never leaves your machine
- **Lightweight** — No bloated Electron, native Tauri performance
- **Fast** — Near-instant on CUDA, optimized for Apple Silicon
- **Flexible** — Use the app, integrate the API, or both
- **Open Source** — No subscriptions, no limits, no lock-in
<br/>
Built with **Tauri** (Rust), **TypeScript**, **React**, and **Python**. Native performance meets modern DX.
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
- **Complete privacy** — models and voice data stay on your machine
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
- **Native performance** — built with Tauri (Rust), not Electron
- **Super fast on Mac** — MLX backend with native Metal acceleration for 4-5x faster inference on Apple Silicon
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
---
@@ -70,11 +98,13 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
- **Instant cloning** — Upload a sample, get a voice profile
- **High fidelity** — Natural prosody, emotion, and cadence
- **Multi-language** — English, Chinese, and more coming
- **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super fast generation
### Voice Profile Management
- **Create profiles** from audio files or record directly in-app
- **Import/Export** profiles to share or backup
- **Multi-sample support** — combine multiple samples for higher quality cloning
- **Organize** with descriptions and language tags
### Speech Generation
@@ -83,9 +113,19 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
- **Batch generation** for long-form content
- **Smart caching** — regenerate instantly with voice prompt caching
### Stories Editor
Create multi-voice narratives, podcasts, and conversations with a timeline-based editor.
- **Multi-track composition** — arrange multiple voice tracks in a single project
- **Inline audio editing** — trim and split clips directly in the timeline
- **Auto-playback** — preview stories with synchronized playhead
- **Voice mixing** — build conversations with multiple participants
### Recording & Transcription
- **In-app recording** with waveform visualization
- **System audio capture** — record desktop audio on macOS and Windows
- **Automatic transcription** powered by Whisper
- **Export recordings** in multiple formats
@@ -109,17 +149,17 @@ Voicebox exposes a full REST API, so you can integrate voice synthesis into your
```bash
# Generate speech
curl -X POST http://localhost:8000/api/generate \
curl -X POST http://localhost:8000/generate \
-H "Content-Type: application/json" \
-d '{"text": "Hello world", "profile_id": "abc123"}'
-d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}'
# List voice profiles
curl http://localhost:8000/api/profiles
curl http://localhost:8000/profiles
# Create a profile from audio
curl -X POST http://localhost:8000/api/profiles \
-F "[email protected]" \
-F "name=My Voice"
# Create a profile
curl -X POST http://localhost:8000/profiles \
-H "Content-Type: application/json" \
-d '{"name": "My Voice", "language": "en"}'
```
**Use cases:**
@@ -142,8 +182,9 @@ Full API documentation available at `http://localhost:8000/docs` when running.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| Voice Model | Qwen3-TTS |
| Transcription | Whisper |
| Voice Model | Qwen3-TTS (PyTorch or MLX) |
| Transcription | Whisper (PyTorch or MLX) |
| Inference Engine | MLX (Apple Silicon) / PyTorch (Windows/Linux/Intel) |
| Database | SQLite |
| Audio | WaveSurfer.js, librosa |
@@ -184,8 +225,26 @@ Voicebox aims to be the **one-stop shop for everything voice** — cloning, synt
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed setup and contribution guidelines.
**Using the Makefile (recommended):** Run `make help` to see all available commands for setup, development, building, and testing.
### Quick Start
**With Makefile (Unix/macOS/Linux):**
```bash
# Clone the repo
git clone https://github.com/voicebox-sh/voicebox.git
cd voicebox
# Setup everything
make setup
# Start development
make dev
```
**Manual setup (all platforms):**
```bash
# Clone the repo
git clone https://github.com/voicebox-sh/voicebox.git
@@ -201,7 +260,11 @@ cd backend && pip install -r requirements.txt && cd ..
bun run dev
```
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org). CUDA-capable GPU recommended (CPU inference supported but slower).
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org).
**Performance:**
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration for 4-5x faster inference
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU recommended, CPU supported but slower)
### Project Structure
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@voicebox/app",
"version": "0.1.6",
"version": "0.1.11",
"private": true,
"type": "module",
"scripts": {
@@ -48,6 +48,7 @@
"react": "^18.3.0",
"react-dom": "^18.3.0",
"react-hook-form": "^7.53.0",
"react-sound-visualizer": "^1.4.0",
"tailwind-merge": "^2.5.4",
"wavesurfer.js": "^7.0.0",
"zod": "^3.23.8",
+16 -8
View File
@@ -357,14 +357,22 @@ export function AudioPlayer() {
}
}
// Standard WaveSurfer auto-play
// Use a small delay to ensure audio element is fully ready
setTimeout(() => {
wavesurfer.play().catch((error) => {
debug.error('Failed to autoplay:', error);
// Don't show error for autoplay failures (browser restrictions)
});
}, 100);
// Only auto-play if shouldAutoPlay flag is set (user explicitly clicked to play)
const shouldAutoPlayNow = usePlayerStore.getState().shouldAutoPlay;
if (shouldAutoPlayNow) {
// Clear the flag first
usePlayerStore.getState().clearAutoPlayFlag();
// Use a small delay to ensure audio element is fully ready
setTimeout(() => {
wavesurfer.play().catch((error) => {
debug.error('Failed to autoplay:', error);
// Don't show error for autoplay failures (browser restrictions)
});
}, 100);
} else {
debug.log('Skipping auto-play - shouldAutoPlay is false');
}
});
// Handle play/pause
@@ -112,9 +112,6 @@ export function FloatingGenerateBox({
}
}, [selectedProfileId, profiles, setSelectedProfileId]);
// Get current form value to trigger resize when it changes
const formValue = form.watch(isInstructMode ? 'instruct' : 'text');
// Auto-resize textarea based on content (only when expanded)
useEffect(() => {
if (!isExpanded) {
@@ -196,59 +193,104 @@ export function FloatingGenerateBox({
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)}>
<div className="flex gap-2">
<motion.div className="flex-1" transition={{ duration: 0.3, ease: 'easeOut' }}>
{isInstructMode && (
<span className="text-xs text-accent font-medium mb-1 block">
Delivery instructions:
</span>
)}
<FormField
control={form.control}
name={isInstructMode ? 'instruct' : 'text'}
render={({ field }) => (
<FormItem>
<FormControl>
<motion.div
animate={{
height: isExpanded ? 'auto' : '32px',
}}
transition={{ duration: 0.15, ease: 'easeOut' }}
style={{ overflow: 'hidden' }}
>
<Textarea
{...field}
ref={(node: HTMLTextAreaElement | null) => {
// Store ref for auto-resize
textareaRef.current = node;
// Forward ref to react-hook-form
if (typeof field.ref === 'function') {
field.ref(node);
}
<motion.div
className={cn('flex-1', isExpanded && 'mr-12')}
transition={{ duration: 0.3, ease: 'easeOut' }}
>
{/* Text field - hidden when in instruct mode */}
<div style={{ display: isInstructMode ? 'none' : 'block' }}>
<FormField
control={form.control}
name="text"
render={({ field }) => (
<FormItem>
<FormControl>
<motion.div
animate={{
height: isExpanded ? 'auto' : '32px',
}}
placeholder={
isInstructMode
? 'Add delivery instructions...'
: isStoriesRoute && currentStory
transition={{ duration: 0.15, ease: 'easeOut' }}
style={{ overflow: 'hidden' }}
>
<Textarea
{...field}
ref={(node: HTMLTextAreaElement | null) => {
// Store ref for auto-resize (only for active field)
if (!isInstructMode) {
textareaRef.current = node;
}
// Forward ref to react-hook-form
if (typeof field.ref === 'function') {
field.ref(node);
}
}}
placeholder={
isStoriesRoute && currentStory
? `Generate speech for "${currentStory.name}"...`
: selectedProfile
? `Generate speech using ${selectedProfile.name}...`
: 'Select a voice profile above...'
}
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
style={{
minHeight: isExpanded ? '100px' : '32px',
maxHeight: '300px',
}
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
style={{
minHeight: isExpanded ? '100px' : '32px',
maxHeight: '300px',
}}
disabled={!selectedProfileId}
onClick={() => setIsExpanded(true)}
onFocus={() => setIsExpanded(true)}
/>
</motion.div>
</FormControl>
<FormMessage className="text-xs" />
</FormItem>
)}
/>
</div>
{/* Instruct field - hidden when in text mode */}
<div style={{ display: isInstructMode ? 'block' : 'none' }}>
<FormField
control={form.control}
name="instruct"
render={({ field }) => (
<FormItem>
<FormControl>
<motion.div
animate={{
height: isExpanded ? 'auto' : '32px',
}}
disabled={!selectedProfileId}
onClick={() => setIsExpanded(true)}
onFocus={() => setIsExpanded(true)}
/>
</motion.div>
</FormControl>
<FormMessage className="text-xs" />
</FormItem>
)}
/>
transition={{ duration: 0.15, ease: 'easeOut' }}
style={{ overflow: 'hidden' }}
>
<Textarea
{...field}
ref={(node: HTMLTextAreaElement | null) => {
// Store ref for auto-resize (only for active field)
if (isInstructMode) {
textareaRef.current = node;
}
// Forward ref to react-hook-form
if (typeof field.ref === 'function') {
field.ref(node);
}
}}
placeholder="Add delivery instructions..."
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
style={{
minHeight: isExpanded ? '100px' : '32px',
maxHeight: '300px',
}}
disabled={!selectedProfileId}
onClick={() => setIsExpanded(true)}
onFocus={() => setIsExpanded(true)}
/>
</motion.div>
</FormControl>
<FormMessage className="text-xs" />
</FormItem>
)}
/>
</div>
</motion.div>
<div className="relative shrink-0">
@@ -278,9 +320,12 @@ export function FloatingGenerateBox({
variant="ghost"
size="icon"
onClick={() => setIsInstructMode(!isInstructMode)}
className={`h-10 w-10 rounded-full bg-card border border-border hover:bg-background/50 transition-all duration-200 ${
isInstructMode ? 'text-accent' : ''
}`}
className={cn(
'h-10 w-10 rounded-full transition-all duration-200',
isInstructMode
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
: 'bg-card border border-border hover:bg-background/50',
)}
>
<MessageSquare className="h-4 w-4" />
</Button>
+8 -5
View File
@@ -53,7 +53,7 @@ export function HistoryTable() {
const exportGeneration = useExportGeneration();
const exportGenerationAudio = useExportGenerationAudio();
const importGeneration = useImportGeneration();
const setAudio = usePlayerStore((state) => state.setAudio);
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio);
const currentAudioId = usePlayerStore((state) => state.audioId);
const isPlaying = usePlayerStore((state) => state.isPlaying);
@@ -77,9 +77,9 @@ export function HistoryTable() {
if (currentAudioId === audioId) {
restartCurrentAudio();
} else {
// Otherwise, load the new audio
// Otherwise, load the new audio and auto-play it
const audioUrl = apiClient.getAudioUrl(audioId);
setAudio(audioUrl, audioId, profileId, text.substring(0, 50));
setAudioWithAutoPlay(audioUrl, audioId, profileId, text.substring(0, 50));
}
};
@@ -233,7 +233,11 @@ export function HistoryTable() {
</div>
{/* Far right - Ellipsis actions */}
<div className="w-10 shrink-0 flex justify-end">
<div
className="w-10 shrink-0 flex justify-end"
onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()}
>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
@@ -241,7 +245,6 @@ export function HistoryTable() {
size="icon"
className="h-8 w-8"
aria-label="Actions"
onClick={(e) => e.stopPropagation()}
>
<MoreHorizontal className="h-4 w-4" />
</Button>
@@ -12,11 +12,10 @@ interface ModelProgressProps {
export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
const [progress, setProgress] = useState<ModelProgressType | null>(null);
const [isSubscribed, setIsSubscribed] = useState(false);
const serverUrl = useServerStore((state) => state.serverUrl);
useEffect(() => {
if (!serverUrl || isSubscribed) return;
if (!serverUrl) return;
// Subscribe to progress updates via Server-Sent Events
const eventSource = new EventSource(`${serverUrl}/models/progress/${modelName}`);
@@ -29,7 +28,6 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
// Close connection if complete or error
if (data.status === 'complete' || data.status === 'error') {
eventSource.close();
setIsSubscribed(false);
}
} catch (error) {
console.error('Error parsing progress event:', error);
@@ -39,16 +37,12 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
eventSource.onerror = (error) => {
console.error('SSE error:', error);
eventSource.close();
setIsSubscribed(false);
};
setIsSubscribed(true);
return () => {
eventSource.close();
setIsSubscribed(false);
};
}, [serverUrl, modelName, isSubscribed]);
}, [serverUrl, modelName]);
// Don't render if no progress or if complete/error and some time has passed
if (
@@ -1,6 +1,7 @@
import { useSortable } from '@dnd-kit/sortable';
import { CSS } from '@dnd-kit/utilities';
import { GripVertical, Mic, MoreHorizontal, Play, Trash2 } from 'lucide-react';
import { useState } from 'react';
import { Button } from '@/components/ui/button';
import {
DropdownMenu,
@@ -12,6 +13,7 @@ import { Textarea } from '@/components/ui/textarea';
import type { StoryItemDetail } from '@/lib/api/types';
import { cn } from '@/lib/utils/cn';
import { useStoryStore } from '@/stores/storyStore';
import { useServerStore } from '@/stores/serverStore';
interface StoryChatItemProps {
item: StoryItemDetail;
@@ -33,6 +35,10 @@ export function StoryChatItem({
isDragging,
}: StoryChatItemProps) {
const seek = useStoryStore((state) => state.seek);
const serverUrl = useServerStore((state) => state.serverUrl);
const [avatarError, setAvatarError] = useState(false);
const avatarUrl = `${serverUrl}/profiles/${item.profile_id}/avatar`;
// Check if this item is currently playing based on timecode
const itemStartMs = item.start_time_ms;
@@ -72,10 +78,22 @@ export function StoryChatItem({
</button>
)}
{/* Voice Icon */}
{/* Voice Avatar */}
<div className="shrink-0">
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center">
<Mic className="h-5 w-5 text-muted-foreground" />
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center overflow-hidden">
{!avatarError ? (
<img
src={avatarUrl}
alt={`${item.profile_name} avatar`}
className={cn(
'h-full w-full object-cover transition-all duration-200',
!isCurrentlyPlaying && 'grayscale'
)}
onError={() => setAvatarError(true)}
/>
) : (
<Mic className="h-5 w-5 text-muted-foreground" />
)}
</div>
</div>
@@ -131,13 +131,13 @@ export function StoryContent() {
}
}, [isPlaying]);
const handleRemoveItem = (generationId: string) => {
const handleRemoveItem = (itemId: string) => {
if (!story) return;
removeItem.mutate(
{
storyId: story.id,
generationId,
itemId,
},
{
onError: (error) => {
@@ -360,7 +360,7 @@ export function StoryContent() {
item={item}
storyId={story.id}
index={index}
onRemove={() => handleRemoveItem(item.generation_id)}
onRemove={() => handleRemoveItem(item.id)}
currentTimeMs={currentTimeMs}
isPlaying={isPlaying && playbackStoryId === story.id}
/>
+25 -25
View File
@@ -1,14 +1,5 @@
import { Plus, BookOpen, MoreHorizontal, Pencil, Trash2 } from 'lucide-react';
import { useState } from 'react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import {
AlertDialog,
AlertDialogAction,
@@ -19,6 +10,15 @@ import {
AlertDialogHeader,
AlertDialogTitle,
} from '@/components/ui/alert-dialog';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import {
DropdownMenu,
DropdownMenuContent,
@@ -26,18 +26,13 @@ import {
DropdownMenuTrigger,
} from '@/components/ui/dropdown-menu';
import { Input } from '@/components/ui/input';
import { Textarea } from '@/components/ui/textarea';
import { Label } from '@/components/ui/label';
import { Textarea } from '@/components/ui/textarea';
import { useToast } from '@/components/ui/use-toast';
import {
useStories,
useCreateStory,
useUpdateStory,
useDeleteStory,
} from '@/lib/hooks/useStories';
import { useStoryStore } from '@/stores/storyStore';
import { useStories, useCreateStory, useUpdateStory, useDeleteStory } from '@/lib/hooks/useStories';
import { cn } from '@/lib/utils/cn';
import { formatDate } from '@/lib/utils/format';
import { useStoryStore } from '@/stores/storyStore';
export function StoryList() {
const { data: stories, isLoading } = useStories();
@@ -49,7 +44,11 @@ export function StoryList() {
const [createDialogOpen, setCreateDialogOpen] = useState(false);
const [editDialogOpen, setEditDialogOpen] = useState(false);
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
const [editingStory, setEditingStory] = useState<{ id: string; name: string; description?: string } | null>(null);
const [editingStory, setEditingStory] = useState<{
id: string;
name: string;
description?: string;
} | null>(null);
const [deletingStoryId, setDeletingStoryId] = useState<string | null>(null);
const [newStoryName, setNewStoryName] = useState('');
const [newStoryDescription, setNewStoryDescription] = useState('');
@@ -186,7 +185,7 @@ export function StoryList() {
{/* Story List */}
<div className="flex-1 min-h-0 overflow-y-auto space-y-2">
{storyList.length === 0 ? (
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-md text-muted-foreground">
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-2xl text-muted-foreground">
<BookOpen className="h-12 w-12 mx-auto mb-4 opacity-50" />
<p className="text-sm">No stories yet</p>
<p className="text-xs mt-2">Create your first story to get started</p>
@@ -196,7 +195,7 @@ export function StoryList() {
<div
key={story.id}
className={cn(
'h-24 p-4 border rounded-md transition-colors group flex items-center',
'h-24 p-4 border rounded-2xl transition-colors group flex items-center',
selectedStoryId === story.id && 'bg-muted border-primary',
)}
>
@@ -213,7 +212,9 @@ export function StoryList() {
</p>
)}
<div className="flex items-center gap-3 mt-2 text-xs text-muted-foreground">
<span>{story.item_count} {story.item_count === 1 ? 'item' : 'items'}</span>
<span>
{story.item_count} {story.item_count === 1 ? 'item' : 'items'}
</span>
<span>•</span>
<span>{formatDate(story.updated_at)}</span>
</div>
@@ -300,9 +301,7 @@ export function StoryList() {
<DialogContent>
<DialogHeader>
<DialogTitle>Edit Story</DialogTitle>
<DialogDescription>
Update the story name and description.
</DialogDescription>
<DialogDescription>Update the story name and description.</DialogDescription>
</DialogHeader>
<div className="space-y-4 py-4">
<div className="space-y-2">
@@ -347,7 +346,8 @@ export function StoryList() {
<AlertDialogHeader>
<AlertDialogTitle>Are you sure?</AlertDialogTitle>
<AlertDialogDescription>
This will permanently delete the story and all its items. This action cannot be undone.
This will permanently delete the story and all its items. This action cannot be
undone.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
File diff suppressed because it is too large Load Diff
@@ -1,8 +1,31 @@
import { Mic, Pause, Play, Square } from 'lucide-react';
import { memo, useEffect, useState } from 'react';
import { Visualizer } from 'react-sound-visualizer';
import { Button } from '@/components/ui/button';
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
import { formatAudioDuration } from '@/lib/utils/audio';
const MemoizedWaveform = memo(function MemoizedWaveform({
audioStream,
}: {
audioStream: MediaStream;
}) {
return (
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
{({ canvasRef }) => (
<canvas
ref={canvasRef}
width={500}
height={150}
className="w-full h-full"
/>
)}
</Visualizer>
</div>
);
});
interface AudioSampleRecordingProps {
file: File | null | undefined;
isRecording: boolean;
@@ -14,6 +37,7 @@ interface AudioSampleRecordingProps {
onPlayPause: () => void;
isPlaying: boolean;
isTranscribing?: boolean;
showWaveform?: boolean;
}
export function AudioSampleRecording({
@@ -27,29 +51,67 @@ export function AudioSampleRecording({
onPlayPause,
isPlaying,
isTranscribing = false,
showWaveform = true,
}: AudioSampleRecordingProps) {
const [audioStream, setAudioStream] = useState<MediaStream | null>(null);
// Request microphone access when component mounts
useEffect(() => {
if (!showWaveform) return;
let stream: MediaStream | null = null;
navigator.mediaDevices
.getUserMedia({ audio: true, video: false })
.then((s) => {
stream = s;
setAudioStream(s);
})
.catch((err) => {
console.warn('Could not access microphone for visualization:', err);
});
return () => {
if (stream) {
stream.getTracks().forEach((track) => {
track.stop();
});
}
};
}, [showWaveform]);
return (
<FormItem>
<FormLabel>Record Audio</FormLabel>
<FormControl>
<div className="space-y-4">
{!isRecording && !file && (
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px]">
<Button type="button" onClick={onStart} size="lg" className="flex items-center gap-2">
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
{showWaveform && audioStream && (
<MemoizedWaveform audioStream={audioStream} />
)}
<Button
type="button"
onClick={onStart}
size="lg"
className="relative z-10 flex items-center gap-2"
>
<Mic className="h-5 w-5" />
Start Recording
</Button>
<p className="text-sm text-muted-foreground text-center">
<p className="relative z-10 text-sm text-muted-foreground text-center">
Click to start recording. Maximum duration: 30 seconds.
</p>
</div>
)}
{isRecording && (
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-destructive rounded-lg bg-destructive/5 min-h-[180px]">
<div className="flex items-center gap-4">
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
{showWaveform && audioStream && (
<MemoizedWaveform audioStream={audioStream} />
)}
<div className="relative z-10 flex items-center gap-4">
<div className="flex items-center gap-2">
<div className="h-3 w-3 rounded-full bg-destructive animate-pulse" />
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
<span className="text-lg font-mono font-semibold">
{formatAudioDuration(duration)}
</span>
@@ -58,13 +120,12 @@ export function AudioSampleRecording({
<Button
type="button"
onClick={onStop}
variant="destructive"
className="flex items-center gap-2"
className="relative z-10 flex items-center gap-2 bg-accent text-accent-foreground hover:bg-accent/90"
>
<Square className="h-4 w-4" />
Stop Recording
</Button>
<p className="text-sm text-muted-foreground text-center">
<p className="relative z-10 text-sm text-muted-foreground text-center">
{formatAudioDuration(30 - duration)} remaining
</p>
</div>
@@ -1,6 +1,6 @@
import { Mic, Monitor, Pause, Play, Square } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
import { formatAudioDuration } from '@/lib/utils/audio';
interface AudioSampleSystemProps {
@@ -30,7 +30,6 @@ export function AudioSampleSystem({
}: AudioSampleSystemProps) {
return (
<FormItem>
<FormLabel>Capture System Audio</FormLabel>
<FormControl>
<div className="space-y-4">
{!isRecording && !file && (
@@ -1,7 +1,7 @@
import { Mic, Pause, Play, Upload } from 'lucide-react';
import { useRef, useState } from 'react';
import { Button } from '@/components/ui/button';
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
interface AudioSampleUploadProps {
file: File | null | undefined;
@@ -31,7 +31,6 @@ export function AudioSampleUpload({
return (
<FormItem>
<FormLabel>Audio File</FormLabel>
<FormControl>
<div className="flex flex-col gap-2">
<input
@@ -15,6 +15,7 @@ import {
import type { VoiceProfileResponse } from '@/lib/api/types';
import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
import { cn } from '@/lib/utils/cn';
import { useServerStore } from '@/stores/serverStore';
import { useUIStore } from '@/stores/uiStore';
interface ProfileCardProps {
@@ -23,15 +24,19 @@ interface ProfileCardProps {
export function ProfileCard({ profile }: ProfileCardProps) {
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
const [avatarError, setAvatarError] = useState(false);
const deleteProfile = useDeleteProfile();
const exportProfile = useExportProfile();
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
const setProfileDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
const serverUrl = useServerStore((state) => state.serverUrl);
const isSelected = selectedProfileId === profile.id;
const avatarUrl = profile.avatar_path ? `${serverUrl}/profiles/${profile.id}/avatar` : null;
const handleSelect = () => {
setSelectedProfileId(isSelected ? null : profile.id);
};
@@ -67,8 +72,20 @@ export function ProfileCard({ profile }: ProfileCardProps) {
>
<CardHeader className="p-3 pb-2">
<CardTitle className="flex items-center gap-1.5 text-base font-medium">
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0">
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden">
{avatarUrl && !avatarError ? (
<img
src={avatarUrl}
alt={`${profile.name} avatar`}
className={cn(
'h-full w-full object-cover transition-all duration-200',
!isSelected && 'grayscale',
)}
onError={() => setAvatarError(true)}
/>
) : (
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
)}
</div>
<span className="break-words">{profile.name}</span>
</CardTitle>
+482 -214
View File
@@ -1,6 +1,6 @@
import { zodResolver } from '@hookform/resolvers/zod';
import { Mic, Monitor, Upload } from 'lucide-react';
import { useEffect, useState } from 'react';
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
import { useEffect, useRef, useState } from 'react';
import { useForm } from 'react-hook-form';
import * as z from 'zod';
import { Button } from '@/components/ui/button';
@@ -36,14 +36,17 @@ import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
import {
useAddSample,
useCreateProfile,
useDeleteAvatar,
useProfile,
useUpdateProfile,
useUploadAvatar,
} from '@/lib/hooks/useProfiles';
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
import { useTranscription } from '@/lib/hooks/useTranscription';
import { isTauri } from '@/lib/tauri';
import { formatAudioDuration, getAudioDuration } from '@/lib/utils/audio';
import { useUIStore } from '@/stores/uiStore';
import { useServerStore } from '@/stores/serverStore';
import { type ProfileFormDraft, useUIStore } from '@/stores/uiStore';
import { AudioSampleRecording } from './AudioSampleRecording';
import { AudioSampleSystem } from './AudioSampleSystem';
import { AudioSampleUpload } from './AudioSampleUpload';
@@ -57,6 +60,7 @@ const baseProfileSchema = z.object({
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
sampleFile: z.instanceof(File).optional(),
referenceText: z.string().max(1000).optional(),
avatarFile: z.instanceof(File).optional(),
});
const profileSchema = baseProfileSchema.refine(
@@ -75,22 +79,51 @@ const profileSchema = baseProfileSchema.refine(
type ProfileFormValues = z.infer<typeof profileSchema>;
// Helper to convert File to base64
async function fileToBase64(file: File): Promise<string> {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = () => resolve(reader.result as string);
reader.onerror = reject;
reader.readAsDataURL(file);
});
}
// Helper to convert base64 to File
function base64ToFile(base64: string, fileName: string, fileType: string): File {
const arr = base64.split(',');
const bstr = atob(arr[1]);
let n = bstr.length;
const u8arr = new Uint8Array(n);
while (n--) {
u8arr[n] = bstr.charCodeAt(n);
}
return new File([u8arr], fileName, { type: fileType });
}
export function ProfileForm() {
const open = useUIStore((state) => state.profileDialogOpen);
const setOpen = useUIStore((state) => state.setProfileDialogOpen);
const editingProfileId = useUIStore((state) => state.editingProfileId);
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
const profileFormDraft = useUIStore((state) => state.profileFormDraft);
const setProfileFormDraft = useUIStore((state) => state.setProfileFormDraft);
const { data: editingProfile } = useProfile(editingProfileId || '');
const createProfile = useCreateProfile();
const updateProfile = useUpdateProfile();
const addSample = useAddSample();
const uploadAvatar = useUploadAvatar();
const deleteAvatar = useDeleteAvatar();
const transcribe = useTranscription();
const { toast } = useToast();
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('upload');
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
const [audioDuration, setAudioDuration] = useState<number | null>(null);
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
const avatarInputRef = useRef<HTMLInputElement>(null);
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
const isCreating = !editingProfileId;
const serverUrl = useServerStore((state) => state.serverUrl);
const form = useForm<ProfileFormValues>({
resolver: zodResolver(profileSchema),
@@ -100,10 +133,12 @@ export function ProfileForm() {
language: 'en',
sampleFile: undefined,
referenceText: '',
avatarFile: undefined,
},
});
const selectedFile = form.watch('sampleFile');
const selectedAvatarFile = form.watch('avatarFile');
// Validate audio duration when file is selected
useEffect(() => {
@@ -220,6 +255,20 @@ export function ProfileForm() {
}
}, [systemRecordingError, toast]);
// Handle avatar preview
useEffect(() => {
if (selectedAvatarFile instanceof File) {
const url = URL.createObjectURL(selectedAvatarFile);
setAvatarPreview(url);
return () => URL.revokeObjectURL(url);
} else if (editingProfile?.avatar_path) {
setAvatarPreview(`${serverUrl}/profiles/${editingProfile.id}/avatar`);
} else {
setAvatarPreview(null);
}
}, [selectedAvatarFile, editingProfile, serverUrl]);
// Restore form state from draft or editing profile
useEffect(() => {
if (editingProfile) {
form.reset({
@@ -228,18 +277,46 @@ export function ProfileForm() {
language: editingProfile.language as LanguageCode,
sampleFile: undefined,
referenceText: undefined,
avatarFile: undefined,
});
} else {
} else if (profileFormDraft && open) {
// Restore from draft when opening in create mode
form.reset({
name: profileFormDraft.name,
description: profileFormDraft.description,
language: profileFormDraft.language as LanguageCode,
referenceText: profileFormDraft.referenceText,
sampleFile: undefined,
avatarFile: undefined,
});
setSampleMode(profileFormDraft.sampleMode);
// Restore the file if we have it saved
if (
profileFormDraft.sampleFileData &&
profileFormDraft.sampleFileName &&
profileFormDraft.sampleFileType
) {
const file = base64ToFile(
profileFormDraft.sampleFileData,
profileFormDraft.sampleFileName,
profileFormDraft.sampleFileType,
);
form.setValue('sampleFile', file);
}
} else if (!open) {
// Only reset to defaults when modal is closed and no draft
form.reset({
name: '',
description: '',
language: 'en',
sampleFile: undefined,
referenceText: undefined,
avatarFile: undefined,
});
setSampleMode('upload');
setSampleMode('record');
setAvatarPreview(null);
}
}, [editingProfile, form]);
}, [editingProfile, profileFormDraft, open, form]);
async function handleTranscribe() {
const file = form.getValues('sampleFile');
@@ -281,6 +358,52 @@ export function ProfileForm() {
playPause(file);
}
function handleAvatarFileChange(e: React.ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0];
if (file) {
if (!file.type.startsWith('image/')) {
toast({
title: 'Invalid file type',
description: 'Please select an image file (PNG, JPG, or WebP)',
variant: 'destructive',
});
return;
}
if (file.size > 5 * 1024 * 1024) {
toast({
title: 'File too large',
description: 'Image must be less than 5MB',
variant: 'destructive',
});
return;
}
form.setValue('avatarFile', file);
}
}
async function handleRemoveAvatar() {
if (editingProfileId && editingProfile?.avatar_path) {
try {
await deleteAvatar.mutateAsync(editingProfileId);
toast({
title: 'Avatar removed',
description: 'Avatar image has been removed successfully.',
});
} catch (error) {
toast({
title: 'Failed to remove avatar',
description: error instanceof Error ? error.message : 'Unknown error',
variant: 'destructive',
});
}
}
form.setValue('avatarFile', undefined);
setAvatarPreview(null);
if (avatarInputRef.current) {
avatarInputRef.current.value = '';
}
}
async function onSubmit(data: ProfileFormValues) {
try {
if (editingProfileId) {
@@ -293,6 +416,24 @@ export function ProfileForm() {
language: data.language,
},
});
// Handle avatar upload/update if file changed
if (data.avatarFile) {
try {
await uploadAvatar.mutateAsync({
profileId: editingProfileId,
file: data.avatarFile,
});
} catch (avatarError) {
toast({
title: 'Avatar upload failed',
description:
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
variant: 'destructive',
});
}
}
toast({
title: 'Voice updated',
description: `"${data.name}" has been updated successfully.`,
@@ -369,6 +510,24 @@ export function ProfileForm() {
file: sampleFile,
referenceText: referenceText,
});
// Handle avatar upload if provided
if (data.avatarFile) {
try {
await uploadAvatar.mutateAsync({
profileId: profile.id,
file: data.avatarFile,
});
} catch (avatarError) {
toast({
title: 'Avatar upload failed',
description:
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
variant: 'destructive',
});
}
}
toast({
title: 'Profile created',
description: `"${data.name}" has been created with a sample.`,
@@ -383,6 +542,8 @@ export function ProfileForm() {
}
}
// Clear draft and reset form on success
setProfileFormDraft(null);
form.reset();
setEditingProfileId(null);
setOpen(false);
@@ -395,12 +556,41 @@ export function ProfileForm() {
}
}
function handleOpenChange(open: boolean) {
setOpen(open);
if (!open) {
async function handleOpenChange(newOpen: boolean) {
if (!newOpen && isCreating) {
// Save draft when closing the create modal
const values = form.getValues();
const hasContent =
values.name || values.description || values.referenceText || values.sampleFile;
if (hasContent) {
const draft: ProfileFormDraft = {
name: values.name || '',
description: values.description || '',
language: values.language || 'en',
referenceText: values.referenceText || '',
sampleMode,
};
// Save file as base64 if present
if (values.sampleFile) {
try {
draft.sampleFileName = values.sampleFile.name;
draft.sampleFileType = values.sampleFile.type;
draft.sampleFileData = await fileToBase64(values.sampleFile);
} catch {
// If file conversion fails, just don't save the file
}
}
setProfileFormDraft(draft);
}
}
setOpen(newOpen);
if (!newOpen) {
setEditingProfileId(null);
form.reset();
setSampleMode('upload');
// Don't reset form here - let the effect handle it based on draft state
if (isRecording) {
cancelRecording();
}
@@ -413,174 +603,119 @@ export function ProfileForm() {
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="max-w-4xl">
<DialogHeader>
<DialogTitle>{editingProfileId ? 'Edit Voice' : 'Create Voice Profile'}</DialogTitle>
<DialogDescription>
{editingProfileId
? 'Update your voice profile details and manage samples.'
: 'Create a new voice profile with an audio sample to clone the voice.'}
</DialogDescription>
</DialogHeader>
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)}>
<div className="grid gap-6 grid-cols-2">
{/* Left column: Profile info */}
<div className="space-y-4">
<FormField
control={form.control}
name="name"
render={({ field }) => (
<FormItem>
<FormLabel>Name</FormLabel>
<FormControl>
<Input placeholder="My Voice" {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="description"
render={({ field }) => (
<FormItem>
<FormLabel>Description (Optional)</FormLabel>
<FormControl>
<Textarea placeholder="Describe this voice..." {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="language"
render={({ field }) => (
<FormItem>
<FormLabel>Language</FormLabel>
<Select onValueChange={field.onChange} defaultValue={field.value}>
<FormControl>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
</FormControl>
<SelectContent>
{LANGUAGE_OPTIONS.map((lang) => (
<SelectItem key={lang.value} value={lang.value}>
{lang.label}
</SelectItem>
))}
</SelectContent>
</Select>
<FormMessage />
</FormItem>
)}
/>
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
<DialogHeader>
<DialogTitle className="text-2xl">
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
</DialogTitle>
<DialogDescription>
{editingProfileId
? 'Update your voice profile details and manage samples.'
: 'Create a new voice profile with an audio sample to clone the voice.'}
</DialogDescription>
{isCreating && profileFormDraft && (
<div className="flex items-center gap-2 pt-2">
<span className="text-xs text-muted-foreground">Draft restored</span>
<Button
type="button"
variant="ghost"
size="sm"
className="h-6 px-2 text-xs text-muted-foreground"
onClick={() => {
setProfileFormDraft(null);
form.reset({
name: '',
description: '',
language: 'en',
sampleFile: undefined,
referenceText: '',
});
setSampleMode('record');
}}
>
<X className="h-3 w-3 mr-1" />
Discard
</Button>
</div>
)}
</DialogHeader>
{/* Right column: Sample management */}
<div className="space-y-4 border-l pl-6">
{isCreating ? (
<>
<div>
<h3 className="text-sm font-medium mb-2">Add Sample</h3>
<p className="text-sm text-muted-foreground mb-4">
Provide an audio sample to clone the voice. You can add more samples later.
</p>
</div>
<Tabs
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsList
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
{/* Left column: Sample management */}
<div className="space-y-4 border-r pr-6">
{isCreating ? (
<>
<Tabs
className="pt-4"
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{isTauri() && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
<TabsList
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
)}
</TabsList>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null && audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{isTauri() && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
</TabsTrigger>
)}
/>
</TabsContent>
</TabsList>
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleRecording
file={selectedFile}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null &&
audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
)}
/>
</TabsContent>
{isTauri() && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
<AudioSampleRecording
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
@@ -590,55 +725,188 @@ export function ProfileForm() {
)}
/>
</TabsContent>
)}
</Tabs>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
{isTauri() && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</>
) : (
// Show sample list when editing
editingProfileId && (
<div>
<SampleList profileId={editingProfileId} />
</div>
)
)}
</div>
</div>
</TabsContent>
)}
</Tabs>
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
Cancel
</Button>
<Button
type="submit"
disabled={createProfile.isPending || updateProfile.isPending || addSample.isPending}
>
{createProfile.isPending || updateProfile.isPending || addSample.isPending
? 'Saving...'
: editingProfileId
? 'Save Changes'
: 'Create Profile'}
</Button>
</div>
</form>
</Form>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</>
) : (
// Show sample list when editing
editingProfileId && (
<div>
<SampleList profileId={editingProfileId} />
</div>
)
)}
</div>
{/* Right column: Profile info */}
<div className="space-y-4">
{/* Avatar Upload */}
<FormField
control={form.control}
name="avatarFile"
render={() => (
<FormItem>
<FormControl>
<div className="flex justify-center pt-4 pb-2">
<div className="relative group">
<div className="h-24 w-24 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden border-2 border-border">
{avatarPreview ? (
<img
src={avatarPreview}
alt="Avatar preview"
className="h-full w-full object-cover"
/>
) : (
<Mic className="h-10 w-10 text-muted-foreground" />
)}
</div>
<button
type="button"
onClick={() => avatarInputRef.current?.click()}
className="absolute inset-0 rounded-full bg-accent/60 opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center cursor-pointer"
>
<Edit2 className="h-6 w-6 text-accent-foreground" />
</button>
{(avatarPreview || editingProfile?.avatar_path) && (
<button
type="button"
onClick={handleRemoveAvatar}
disabled={deleteAvatar.isPending}
className="absolute bottom-0 right-0 h-6 w-6 rounded-full bg-background/60 backdrop-blur-sm text-muted-foreground flex items-center justify-center hover:bg-background/80 hover:text-foreground transition-colors shadow-sm border border-border/50"
>
<X className="h-3.5 w-3.5" />
</button>
)}
</div>
<input
ref={avatarInputRef}
type="file"
accept="image/png,image/jpeg,image/webp"
onChange={handleAvatarFileChange}
className="hidden"
/>
</div>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="name"
render={({ field }) => (
<FormItem>
<FormLabel>Name</FormLabel>
<FormControl>
<Input placeholder="My Voice" {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="description"
render={({ field }) => (
<FormItem>
<FormLabel>Description (Optional)</FormLabel>
<FormControl>
<Textarea placeholder="Describe this voice..." {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="language"
render={({ field }) => (
<FormItem>
<FormLabel>Language</FormLabel>
<Select onValueChange={field.onChange} defaultValue={field.value}>
<FormControl>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
</FormControl>
<SelectContent>
{LANGUAGE_OPTIONS.map((lang) => (
<SelectItem key={lang.value} value={lang.value}>
{lang.label}
</SelectItem>
))}
</SelectContent>
</Select>
<FormMessage />
</FormItem>
)}
/>
</div>
</div>
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
Cancel
</Button>
<Button
type="submit"
disabled={
createProfile.isPending || updateProfile.isPending || addSample.isPending
}
>
{createProfile.isPending || updateProfile.isPending || addSample.isPending
? 'Saving...'
: editingProfileId
? 'Save Changes'
: 'Create Profile'}
</Button>
</div>
</form>
</Form>
</div>
</DialogContent>
</Dialog>
);
+260 -52
View File
@@ -1,11 +1,133 @@
import { Plus, Trash2, Play } from 'lucide-react';
import { useState } from 'react';
import { Plus, Trash2, Play, Edit, Check, X, Volume2, Pause } from 'lucide-react';
import { useEffect, useRef, useState } from 'react';
import { Button } from '@/components/ui/button';
import { useDeleteSample, useProfileSamples } from '@/lib/hooks/useProfiles';
import { usePlayerStore } from '@/stores/playerStore';
import { CircleButton } from '@/components/ui/circle-button';
import { Textarea } from '@/components/ui/textarea';
import { Slider } from '@/components/ui/slider';
import { useToast } from '@/components/ui/use-toast';
import { apiClient } from '@/lib/api/client';
import { useDeleteSample, useProfileSamples, useUpdateSample } from '@/lib/hooks/useProfiles';
import { formatAudioDuration } from '@/lib/utils/audio';
import { cn } from '@/lib/utils/cn';
import { SampleUpload } from './SampleUpload';
interface MiniSamplePlayerProps {
audioUrl: string;
}
function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
const audioRef = useRef<HTMLAudioElement | null>(null);
const [isPlaying, setIsPlaying] = useState(false);
const [currentTime, setCurrentTime] = useState(0);
const [duration, setDuration] = useState(0);
const [isLoading, setIsLoading] = useState(true);
useEffect(() => {
const audio = new Audio(audioUrl);
audioRef.current = audio;
const handleLoadedMetadata = () => {
setDuration(audio.duration);
setIsLoading(false);
};
const handleTimeUpdate = () => {
setCurrentTime(audio.currentTime);
};
const handleEnded = () => {
setIsPlaying(false);
setCurrentTime(0);
};
const handlePlay = () => setIsPlaying(true);
const handlePause = () => setIsPlaying(false);
audio.addEventListener('loadedmetadata', handleLoadedMetadata);
audio.addEventListener('timeupdate', handleTimeUpdate);
audio.addEventListener('ended', handleEnded);
audio.addEventListener('play', handlePlay);
audio.addEventListener('pause', handlePause);
return () => {
audio.pause();
audio.removeEventListener('loadedmetadata', handleLoadedMetadata);
audio.removeEventListener('timeupdate', handleTimeUpdate);
audio.removeEventListener('ended', handleEnded);
audio.removeEventListener('play', handlePlay);
audio.removeEventListener('pause', handlePause);
audio.src = '';
};
}, [audioUrl]);
const handlePlayPause = () => {
if (!audioRef.current) return;
if (isPlaying) {
audioRef.current.pause();
} else {
audioRef.current.play();
}
};
const handleSeek = (value: number[]) => {
if (!audioRef.current || duration === 0) return;
const progress = value[0] / 100;
audioRef.current.currentTime = progress * duration;
};
const handleStop = () => {
if (audioRef.current) {
audioRef.current.pause();
audioRef.current.currentTime = 0;
}
setIsPlaying(false);
setCurrentTime(0);
};
return (
<div className="border-t bg-muted/30 px-3 py-2 mt-2">
<div className="flex items-center gap-2">
<Button
type="button"
variant="ghost"
size="icon"
className="h-7 w-7 shrink-0"
onClick={handlePlayPause}
disabled={isLoading}
>
{isPlaying ? <Pause className="h-3.5 w-3.5" /> : <Play className="h-3.5 w-3.5 ml-0.5" />}
</Button>
<div className="flex-1 min-w-0 flex items-center gap-2">
<Slider
value={duration > 0 ? [(currentTime / duration) * 100] : [0]}
onValueChange={handleSeek}
max={100}
step={0.1}
className="flex-1"
/>
<div className="flex items-center gap-1 text-xs text-muted-foreground shrink-0 min-w-[70px]">
<span className="font-mono">{formatAudioDuration(currentTime)}</span>
<span>/</span>
<span className="font-mono">{formatAudioDuration(duration)}</span>
</div>
</div>
<Button
type="button"
variant="ghost"
size="icon"
className="h-7 w-7 shrink-0"
onClick={handleStop}
title="Stop"
>
<X className="h-3.5 w-3.5" />
</Button>
</div>
</div>
);
}
interface SampleListProps {
profileId: string;
}
@@ -13,10 +135,11 @@ interface SampleListProps {
export function SampleList({ profileId }: SampleListProps) {
const { data: samples, isLoading } = useProfileSamples(profileId);
const deleteSample = useDeleteSample();
const updateSample = useUpdateSample();
const { toast } = useToast();
const [uploadOpen, setUploadOpen] = useState(false);
const setAudio = usePlayerStore((state) => state.setAudio);
const currentAudioId = usePlayerStore((state) => state.audioId);
const isPlaying = usePlayerStore((state) => state.isPlaying);
const [editingSampleId, setEditingSampleId] = useState<string | null>(null);
const [editedText, setEditedText] = useState<string>('');
const handleDelete = (sampleId: string) => {
if (confirm('Are you sure you want to delete this sample?')) {
@@ -24,9 +147,41 @@ export function SampleList({ profileId }: SampleListProps) {
}
};
const handlePlay = (referenceText: string, sampleId: string) => {
const audioUrl = apiClient.getSampleUrl(sampleId);
setAudio(audioUrl, sampleId, referenceText.substring(0, 50));
const handleStartEdit = (sampleId: string, currentText: string) => {
setEditingSampleId(sampleId);
setEditedText(currentText);
};
const handleCancelEdit = () => {
setEditingSampleId(null);
setEditedText('');
};
const handleSaveEdit = async (sampleId: string) => {
if (!editedText.trim()) {
toast({
title: 'Invalid text',
description: 'Reference text cannot be empty.',
variant: 'destructive',
});
return;
}
try {
await updateSample.mutateAsync({ sampleId, referenceText: editedText.trim() });
toast({
title: 'Sample updated',
description: 'Reference text has been updated successfully.',
});
setEditingSampleId(null);
setEditedText('');
} catch (error) {
toast({
title: 'Update failed',
description: error instanceof Error ? error.message : 'Failed to update sample',
variant: 'destructive',
});
}
};
if (isLoading) {
@@ -34,56 +189,109 @@ export function SampleList({ profileId }: SampleListProps) {
}
return (
<div className="space-y-4">
<div className="flex items-center justify-between">
<h3 className="text-lg font-semibold">Audio Samples</h3>
<Button type="button" size="sm" onClick={() => setUploadOpen(true)}>
<Plus className="mr-2 h-4 w-4" />
Add Sample
</Button>
</div>
<div className="space-y-4 pt-4">
{samples && samples.length === 0 ? (
<div className="text-sm text-muted-foreground py-4">
No samples yet. Add your first audio sample.
<div className="flex flex-col items-center justify-center py-8 text-center border border-dashed rounded-lg">
<Volume2 className="h-8 w-8 text-muted-foreground/50 mb-2" />
<p className="text-sm text-muted-foreground">No samples yet</p>
<p className="text-xs text-muted-foreground/70 mt-1">Add your first audio sample to get started</p>
</div>
) : (
<div className="space-y-2">
{samples?.map((sample) => (
<div
key={sample.id}
className="flex items-center justify-between p-3 border rounded-lg"
>
<div className="flex-1">
<p className="text-sm font-medium">{sample.reference_text}</p>
<p className="text-xs text-muted-foreground mt-1">{sample.audio_path}</p>
{samples?.map((sample, index) => {
const isEditing = editingSampleId === sample.id;
return (
<div
key={sample.id}
className={cn(
'group relative rounded-lg border bg-card transition-all duration-200',
isEditing ? 'ring-2 ring-primary/20' : 'hover:border-primary/30'
)}
>
{isEditing ? (
/* Edit Mode */
<div className="p-4 space-y-3">
<div className="flex items-center gap-2 text-xs text-muted-foreground mb-2">
<Edit className="h-3 w-3" />
<span>Editing transcription</span>
</div>
<Textarea
value={editedText}
onChange={(e) => setEditedText(e.target.value)}
className="min-h-[100px] text-sm resize-none"
placeholder="Enter reference text..."
autoFocus
/>
<div className="flex items-center justify-end gap-2 pt-1">
<Button
type="button"
size="sm"
variant="ghost"
onClick={handleCancelEdit}
disabled={updateSample.isPending}
>
<X className="h-4 w-4 mr-1" />
Cancel
</Button>
<Button
type="button"
size="sm"
onClick={() => handleSaveEdit(sample.id)}
disabled={updateSample.isPending}
>
<Check className="h-4 w-4 mr-1" />
{updateSample.isPending ? 'Saving...' : 'Save'}
</Button>
</div>
</div>
) : (
<>
{/* View Mode */}
<div className="flex items-center gap-3 p-3 h-[72px]">
{/* Text Content */}
<div className="flex-1 min-w-0 py-0.5">
<p className="text-sm font-medium line-clamp-2 leading-snug">
{sample.reference_text}
</p>
</div>
{/* Action Buttons */}
<div className="shrink-0 flex items-center gap-0.5 opacity-0 group-hover:opacity-100 transition-opacity">
<CircleButton
icon={Edit}
title="Edit transcription"
onClick={() => handleStartEdit(sample.id, sample.reference_text)}
/>
<CircleButton
icon={Trash2}
title="Delete sample"
onClick={() => handleDelete(sample.id)}
disabled={deleteSample.isPending}
/>
</div>
{/* Sample Number Badge */}
<div className="absolute top-1 right-2 text-[10px] text-muted-foreground/50 font-medium">
#{index + 1}
</div>
</div>
{/* Mini Player - Always visible */}
<MiniSamplePlayer audioUrl={apiClient.getSampleUrl(sample.id)} />
</>
)}
</div>
<div className="flex gap-2">
<Button
type="button"
variant="ghost"
size="sm"
onClick={() => handlePlay(sample.reference_text, sample.id)}
className={currentAudioId === sample.id && isPlaying ? 'text-primary' : ''}
>
<Play className="h-4 w-4 mr-1" />
Play
</Button>
<Button
type="button"
variant="ghost"
size="sm"
onClick={() => handleDelete(sample.id)}
disabled={deleteSample.isPending}
>
<Trash2 className="h-4 w-4 text-destructive" />
</Button>
</div>
</div>
))}
);
})}
</div>
)}
<Button type="button" variant="outline" className="w-full" onClick={() => setUploadOpen(true)}>
<Plus className="mr-2 h-4 w-4" />
Add Sample
</Button>
<SampleUpload profileId={profileId} open={uploadOpen} onOpenChange={setUploadOpen} />
</div>
);
+64 -5
View File
@@ -1,4 +1,5 @@
import { useServerStore } from '@/stores/serverStore';
import type { LanguageCode } from '@/lib/constants/languages';
import type {
VoiceProfileCreate,
VoiceProfileResponse,
@@ -21,6 +22,8 @@ import type {
StoryItemBatchUpdate,
StoryItemReorder,
StoryItemMove,
StoryItemTrim,
StoryItemSplit,
} from './types';
class ApiClient {
@@ -118,6 +121,16 @@ class ApiClient {
});
}
async updateProfileSample(
sampleId: string,
referenceText: string,
): Promise<ProfileSampleResponse> {
return this.request<ProfileSampleResponse>(`/profiles/samples/${sampleId}`, {
method: 'PUT',
body: JSON.stringify({ reference_text: referenceText }),
});
}
async exportProfile(profileId: string): Promise<Blob> {
const url = `${this.getBaseUrl()}/profiles/${profileId}/export`;
const response = await fetch(url);
@@ -152,6 +165,32 @@ class ApiClient {
return response.json();
}
async uploadAvatar(profileId: string, file: File): Promise<VoiceProfileResponse> {
const url = `${this.getBaseUrl()}/profiles/${profileId}/avatar`;
const formData = new FormData();
formData.append('file', file);
const response = await fetch(url, {
method: 'POST',
body: formData,
});
if (!response.ok) {
const error = await response.json().catch(() => ({
detail: response.statusText,
}));
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
}
return response.json();
}
async deleteAvatar(profileId: string): Promise<void> {
await this.request<void>(`/profiles/${profileId}/avatar`, {
method: 'DELETE',
});
}
// Generation
async generateSpeech(data: GenerationRequest): Promise<GenerationResponse> {
return this.request<GenerationResponse>('/generate', {
@@ -242,7 +281,7 @@ class ApiClient {
}
// Transcription
async transcribeAudio(file: File, language?: 'en' | 'zh'): Promise<TranscriptionResponse> {
async transcribeAudio(file: File, language?: LanguageCode): Promise<TranscriptionResponse> {
const formData = new FormData();
formData.append('file', file);
if (language) {
@@ -406,8 +445,8 @@ class ApiClient {
});
}
async removeStoryItem(storyId: string, generationId: string): Promise<void> {
await this.request<void>(`/stories/${storyId}/items/${generationId}`, {
async removeStoryItem(storyId: string, itemId: string): Promise<void> {
await this.request<void>(`/stories/${storyId}/items/${itemId}`, {
method: 'DELETE',
});
}
@@ -426,13 +465,33 @@ class ApiClient {
});
}
async moveStoryItem(storyId: string, generationId: string, data: StoryItemMove): Promise<StoryItemDetail> {
return this.request<StoryItemDetail>(`/stories/${storyId}/items/${generationId}/move`, {
async moveStoryItem(storyId: string, itemId: string, data: StoryItemMove): Promise<StoryItemDetail> {
return this.request<StoryItemDetail>(`/stories/${storyId}/items/${itemId}/move`, {
method: 'PUT',
body: JSON.stringify(data),
});
}
async trimStoryItem(storyId: string, itemId: string, data: StoryItemTrim): Promise<StoryItemDetail> {
return this.request<StoryItemDetail>(`/stories/${storyId}/items/${itemId}/trim`, {
method: 'PUT',
body: JSON.stringify(data),
});
}
async splitStoryItem(storyId: string, itemId: string, data: StoryItemSplit): Promise<StoryItemDetail[]> {
return this.request<StoryItemDetail[]>(`/stories/${storyId}/items/${itemId}/split`, {
method: 'POST',
body: JSON.stringify(data),
});
}
async duplicateStoryItem(storyId: string, itemId: string): Promise<StoryItemDetail> {
return this.request<StoryItemDetail>(`/stories/${storyId}/items/${itemId}/duplicate`, {
method: 'POST',
});
}
async exportStoryAudio(storyId: string): Promise<Blob> {
const url = `${this.getBaseUrl()}/stories/${storyId}/export-audio`;
const response = await fetch(url);
+16 -3
View File
@@ -1,9 +1,10 @@
// API Types matching backend Pydantic models
import type { LanguageCode } from '@/lib/constants/languages';
export interface VoiceProfileCreate {
name: string;
description?: string;
language: 'en' | 'zh';
language: LanguageCode;
}
export interface VoiceProfileResponse {
@@ -11,6 +12,7 @@ export interface VoiceProfileResponse {
name: string;
description?: string;
language: string;
avatar_path?: string;
created_at: string;
updated_at: string;
}
@@ -29,7 +31,7 @@ export interface ProfileSampleResponse {
export interface GenerationRequest {
profile_id: string;
text: string;
language: 'en' | 'zh';
language: LanguageCode;
seed?: number;
model_size?: '1.7B' | '0.6B';
}
@@ -62,7 +64,7 @@ export interface HistoryListResponse {
}
export interface TranscriptionRequest {
language?: 'en' | 'zh';
language?: LanguageCode;
}
export interface TranscriptionResponse {
@@ -144,6 +146,8 @@ export interface StoryItemDetail {
generation_id: string;
start_time_ms: number;
track: number;
trim_start_ms: number;
trim_end_ms: number;
created_at: string;
profile_id: string;
profile_name: string;
@@ -188,3 +192,12 @@ export interface StoryItemMove {
start_time_ms: number;
track: number;
}
export interface StoryItemTrim {
trim_start_ms: number;
trim_end_ms: number;
}
export interface StoryItemSplit {
split_time_ms: number;
}
+2 -2
View File
@@ -28,7 +28,7 @@ interface UseGenerationFormOptions {
export function useGenerationForm(options: UseGenerationFormOptions = {}) {
const { toast } = useToast();
const generation = useGeneration();
const setAudio = usePlayerStore((state) => state.setAudio);
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
const setIsGenerating = useGenerationStore((state) => state.setIsGenerating);
const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null);
const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null);
@@ -97,7 +97,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
});
const audioUrl = apiClient.getAudioUrl(result.id);
setAudio(audioUrl, result.id, selectedProfileId, data.text.substring(0, 50));
setAudioWithAutoPlay(audioUrl, result.id, selectedProfileId, data.text.substring(0, 50));
form.reset();
options.onSuccess?.(result.id);
+2 -2
View File
@@ -140,8 +140,8 @@ export function useModelDownloadToast({
}
};
eventSource.onerror = () => {
console.error('SSE error');
eventSource.onerror = (error) => {
console.error('SSE error:', error);
eventSource.close();
eventSourceRef.current = null;
+47
View File
@@ -98,6 +98,24 @@ export function useDeleteSample() {
});
}
export function useUpdateSample() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ sampleId, referenceText }: { sampleId: string; referenceText: string }) =>
apiClient.updateProfileSample(sampleId, referenceText),
onSuccess: (data) => {
queryClient.invalidateQueries({
queryKey: ['profiles', data.profile_id, 'samples'],
});
queryClient.invalidateQueries({
queryKey: ['profiles', data.profile_id],
});
queryClient.invalidateQueries({ queryKey: ['profiles'] });
},
});
}
export function useExportProfile() {
return useMutation({
mutationFn: async (profileId: string) => {
@@ -167,3 +185,32 @@ export function useImportProfile() {
},
});
}
export function useUploadAvatar() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ profileId, file }: { profileId: string; file: File }) =>
apiClient.uploadAvatar(profileId, file),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['profiles'] });
queryClient.invalidateQueries({
queryKey: ['profiles', variables.profileId],
});
},
});
}
export function useDeleteAvatar() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (profileId: string) => apiClient.deleteAvatar(profileId),
onSuccess: (_, profileId) => {
queryClient.invalidateQueries({ queryKey: ['profiles'] });
queryClient.invalidateQueries({
queryKey: ['profiles', profileId],
});
},
});
}
+44 -5
View File
@@ -1,6 +1,6 @@
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
import { apiClient } from '@/lib/api/client';
import type { StoryCreate, StoryItemCreate, StoryItemBatchUpdate, StoryItemReorder, StoryItemMove } from '@/lib/api/types';
import type { StoryCreate, StoryItemCreate, StoryItemBatchUpdate, StoryItemReorder, StoryItemMove, StoryItemTrim, StoryItemSplit } from '@/lib/api/types';
import { isTauri } from '@/lib/tauri';
export function useStories() {
@@ -70,8 +70,8 @@ export function useRemoveStoryItem() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ storyId, generationId }: { storyId: string; generationId: string }) =>
apiClient.removeStoryItem(storyId, generationId),
mutationFn: ({ storyId, itemId }: { storyId: string; itemId: string }) =>
apiClient.removeStoryItem(storyId, itemId),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', variables.storyId] });
@@ -109,8 +109,47 @@ export function useMoveStoryItem() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ storyId, generationId, data }: { storyId: string; generationId: string; data: StoryItemMove }) =>
apiClient.moveStoryItem(storyId, generationId, data),
mutationFn: ({ storyId, itemId, data }: { storyId: string; itemId: string; data: StoryItemMove }) =>
apiClient.moveStoryItem(storyId, itemId, data),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', variables.storyId] });
},
});
}
export function useTrimStoryItem() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ storyId, itemId, data }: { storyId: string; itemId: string; data: StoryItemTrim }) =>
apiClient.trimStoryItem(storyId, itemId, data),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', variables.storyId] });
},
});
}
export function useSplitStoryItem() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ storyId, itemId, data }: { storyId: string; itemId: string; data: StoryItemSplit }) =>
apiClient.splitStoryItem(storyId, itemId, data),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', variables.storyId] });
},
});
}
export function useDuplicateStoryItem() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ storyId, itemId }: { storyId: string; itemId: string }) =>
apiClient.duplicateStoryItem(storyId, itemId),
onSuccess: (_, variables) => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', variables.storyId] });
+36 -22
View File
@@ -5,6 +5,7 @@ import { useStoryStore } from '@/stores/storyStore';
interface ActiveSource {
source: AudioBufferSourceNode;
itemId: string;
generationId: string;
startTimeMs: number;
endTimeMs: number;
@@ -26,9 +27,9 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
const audioContextRef = useRef<AudioContext | null>(null);
// Master gain for volume control
const masterGainRef = useRef<GainNode | null>(null);
// Preloaded AudioBuffers by generation_id
// Preloaded AudioBuffers by generation_id (audio file is shared between split clips)
const audioBuffersRef = useRef<Map<string, AudioBuffer>>(new Map());
// Currently playing AudioBufferSourceNodes by generation_id
// Currently playing AudioBufferSourceNodes by item.id (unique per clip)
const activeSourcesRef = useRef<Map<string, ActiveSource>>(new Map());
// Animation frame for syncing visual playhead
const animationFrameRef = useRef<number | null>(null);
@@ -56,16 +57,16 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
return audioContextRef.current;
}, []);
// Stop a source
const stopSource = useCallback((generationId: string) => {
const activeSource = activeSourcesRef.current.get(generationId);
// Stop a source by item id
const stopSource = useCallback((itemId: string) => {
const activeSource = activeSourcesRef.current.get(itemId);
if (activeSource) {
try {
activeSource.source.stop();
} catch {
// Source may have already stopped
}
activeSourcesRef.current.delete(generationId);
activeSourcesRef.current.delete(itemId);
}
}, []);
@@ -123,8 +124,8 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
useEffect(() => {
return () => {
// Stop all sources
for (const [generationId] of activeSourcesRef.current) {
stopSource(generationId);
for (const [itemId] of activeSourcesRef.current) {
stopSource(itemId);
}
activeSourcesRef.current.clear();
@@ -151,7 +152,11 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
(storyTimeMs: number, itemList: StoryItemDetail[]): StoryItemDetail[] => {
return itemList.filter((item) => {
const itemStart = item.start_time_ms;
const itemEnd = item.start_time_ms + item.duration * 1000;
// Use effective duration (accounting for trims)
const trimStartMs = item.trim_start_ms || 0;
const trimEndMs = item.trim_end_ms || 0;
const effectiveDurationMs = item.duration * 1000 - trimStartMs - trimEndMs;
const itemEnd = item.start_time_ms + effectiveDurationMs;
return storyTimeMs >= itemStart && storyTimeMs < itemEnd;
});
},
@@ -185,8 +190,8 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
// Stop all sources
const stopAllSources = useCallback(() => {
console.log('[StoryPlayback] Stopping all sources');
for (const [generationId] of activeSourcesRef.current) {
stopSource(generationId);
for (const [itemId] of activeSourcesRef.current) {
stopSource(itemId);
}
activeSourcesRef.current.clear();
}, [stopSource]);
@@ -199,18 +204,18 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
// Find all items that should be playing
const shouldBePlaying = findActiveItems(storyTimeMs, itemList);
const shouldBePlayingIds = new Set(shouldBePlaying.map((item) => item.generation_id));
const shouldBePlayingIds = new Set(shouldBePlaying.map((item) => item.id));
// Stop sources that shouldn't be playing anymore
for (const [generationId] of activeSourcesRef.current) {
if (!shouldBePlayingIds.has(generationId)) {
stopSource(generationId);
for (const [itemId] of activeSourcesRef.current) {
if (!shouldBePlayingIds.has(itemId)) {
stopSource(itemId);
}
}
// Schedule new sources for items that should be playing
for (const item of shouldBePlaying) {
if (!activeSourcesRef.current.has(item.generation_id)) {
if (!activeSourcesRef.current.has(item.id)) {
const buffer = audioBuffersRef.current.get(item.generation_id);
if (!buffer) {
console.warn('[StoryPlayback] Buffer not loaded for:', item.generation_id);
@@ -219,16 +224,24 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
// Calculate when this item should start in AudioContext time
const itemStartContextTime = storyTimeToContextTime(item.start_time_ms);
const itemEndStoryTime = item.start_time_ms + item.duration * 1000;
// Calculate effective duration and trim offsets
const trimStartSec = (item.trim_start_ms || 0) / 1000;
const trimEndSec = (item.trim_end_ms || 0) / 1000;
const effectiveDuration = item.duration - trimStartSec - trimEndSec;
const itemEndStoryTime = item.start_time_ms + effectiveDuration * 1000;
// Calculate offset into the buffer (if seeking mid-way)
const offsetIntoBuffer = Math.max(0, (storyTimeMs - item.start_time_ms) / 1000);
const duration = item.duration - offsetIntoBuffer;
// Offset is relative to the trimmed start of the clip
const offsetIntoEffectiveClip = Math.max(0, (storyTimeMs - item.start_time_ms) / 1000);
const offsetIntoBuffer = trimStartSec + offsetIntoEffectiveClip;
const duration = effectiveDuration - offsetIntoEffectiveClip;
// If the item should have already started, schedule it to start immediately
const startAtContextTime = Math.max(currentContextTime, itemStartContextTime);
console.log('[StoryPlayback] Scheduling source:', {
itemId: item.id,
generationId: item.generation_id,
storyTimeMs,
itemStart: item.start_time_ms,
@@ -243,20 +256,21 @@ export function useStoryPlayback(items: StoryItemDetail[] | undefined) {
const activeSource: ActiveSource = {
source,
itemId: item.id,
generationId: item.generation_id,
startTimeMs: item.start_time_ms,
endTimeMs: itemEndStoryTime,
};
activeSourcesRef.current.set(item.generation_id, activeSource);
activeSourcesRef.current.set(item.id, activeSource);
// Schedule playback
source.start(startAtContextTime, offsetIntoBuffer, duration);
// Clean up when source ends
source.onended = () => {
console.log('[StoryPlayback] Source ended:', item.generation_id);
activeSourcesRef.current.delete(item.generation_id);
console.log('[StoryPlayback] Source ended:', item.id);
activeSourcesRef.current.delete(item.id);
};
}
}
+2 -1
View File
@@ -1,9 +1,10 @@
import { useMutation } from '@tanstack/react-query';
import { apiClient } from '@/lib/api/client';
import type { LanguageCode } from '@/lib/constants/languages';
export function useTranscription() {
return useMutation({
mutationFn: ({ file, language }: { file: File; language?: 'en' | 'zh' }) =>
mutationFn: ({ file, language }: { file: File; language?: LanguageCode }) =>
apiClient.transcribeAudio(file, language),
});
}
+31 -8
View File
@@ -5,6 +5,8 @@ interface StoryPlaybackState {
// Selection
selectedStoryId: string | null;
setSelectedStoryId: (id: string | null) => void;
selectedClipId: string | null;
setSelectedClipId: (id: string | null) => void;
// Track editor UI state
trackEditorHeight: number;
@@ -26,6 +28,7 @@ interface StoryPlaybackState {
stop: () => void;
seek: (timeMs: number) => void;
setPlaybackTiming: (contextTime: number, storyTime: number) => void; // Set timing anchors for Web Audio API
setActiveStory: (storyId: string, items: StoryItemDetail[], totalDurationMs: number) => void; // Activate story for seeking without playing
}
const DEFAULT_TRACK_EDITOR_HEIGHT = 250;
@@ -34,6 +37,8 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
// Selection
selectedStoryId: null,
setSelectedStoryId: (id) => set({ selectedStoryId: id }),
selectedClipId: null,
setSelectedClipId: (id) => set({ selectedClipId: id }),
// Track editor UI state
trackEditorHeight: DEFAULT_TRACK_EDITOR_HEIGHT,
@@ -53,14 +58,11 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
// Calculate total duration from items
const maxEndTimeMs = Math.max(
...items.map((item) => item.start_time_ms + item.duration * 1000),
0
0,
);
// Find the minimum start time (first item)
const minStartTimeMs = Math.min(
...items.map((item) => item.start_time_ms),
0
);
const minStartTimeMs = Math.min(...items.map((item) => item.start_time_ms), 0);
// If resuming the same story, keep position; otherwise start at first item
const currentState = get();
@@ -70,7 +72,11 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
console.log('[StoryStore] Play called:', {
storyId,
itemCount: items.length,
items: items.map(i => ({ id: i.generation_id, start: i.start_time_ms, duration: i.duration })),
items: items.map((i) => ({
id: i.generation_id,
start: i.start_time_ms,
duration: i.duration,
})),
maxEndTimeMs,
minStartTimeMs,
startTimeMs,
@@ -83,11 +89,14 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
playbackItems: items,
totalDurationMs: maxEndTimeMs,
currentTimeMs: startTimeMs,
// Reset timing anchors - will be set fresh by the playback hook
playbackStartContextTime: null,
playbackStartStoryTime: null,
});
},
pause: () => {
set({
set({
isPlaying: false,
// Keep timing anchors so we can resume from same position
});
@@ -108,7 +117,7 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
seek: (timeMs) => {
const state = get();
const clampedTime = Math.max(0, Math.min(timeMs, state.totalDurationMs));
set({
set({
currentTimeMs: clampedTime,
// Reset timing anchors - will be set by hook when playback resumes
playbackStartContextTime: null,
@@ -122,4 +131,18 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
playbackStartStoryTime: storyTime,
});
},
setActiveStory: (storyId, items, totalDurationMs) => {
const currentState = get();
// Only update if switching to a different story
if (currentState.playbackStoryId !== storyId) {
set({
playbackStoryId: storyId,
playbackItems: items,
totalDurationMs,
currentTimeMs: 0,
isPlaying: false,
});
}
},
}));
+20
View File
@@ -1,5 +1,18 @@
import { create } from 'zustand';
// Draft state for the create voice profile form
export interface ProfileFormDraft {
name: string;
description: string;
language: string;
referenceText: string;
sampleMode: 'upload' | 'record' | 'system';
// Note: File objects can't be persisted, so we store metadata
sampleFileName?: string;
sampleFileType?: string;
sampleFileData?: string; // Base64 encoded
}
interface UIStore {
// Sidebar
sidebarOpen: boolean;
@@ -18,6 +31,10 @@ interface UIStore {
selectedProfileId: string | null;
setSelectedProfileId: (id: string | null) => void;
// Profile form draft (for persisting create voice modal state)
profileFormDraft: ProfileFormDraft | null;
setProfileFormDraft: (draft: ProfileFormDraft | null) => void;
// Theme
theme: 'light' | 'dark';
setTheme: (theme: 'light' | 'dark') => void;
@@ -38,6 +55,9 @@ export const useUIStore = create<UIStore>((set) => ({
selectedProfileId: null,
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
profileFormDraft: null,
setProfileFormDraft: (draft) => set({ profileFormDraft: draft }),
theme: 'light',
setTheme: (theme) => {
set({ theme });
+28 -7
View File
@@ -19,8 +19,13 @@ Production-quality FastAPI backend for Qwen3-TTS voice cloning.
backend/
├── main.py # FastAPI app with all routes
├── models.py # Pydantic request/response models
├── tts.py # Qwen3-TTS inference
├── transcribe.py # Whisper ASR
├── platform_detect.py # Platform detection for backend selection
├── tts.py # TTS backend abstraction (delegates to MLX or PyTorch)
├── transcribe.py # STT backend abstraction (delegates to MLX or PyTorch)
├── backends/ # Backend implementations
│ ├── __init__.py # Backend factory and protocols
│ ├── mlx_backend.py # MLX backend (Apple Silicon)
│ └── pytorch_backend.py # PyTorch backend (Windows/Linux/Intel)
├── profiles.py # Voice profile CRUD
├── history.py # Generation history
├── studio.py # Audio editing (TODO)
@@ -31,6 +36,15 @@ backend/
└── validation.py # Input validation
```
### Backend Selection
Voicebox automatically selects the best backend based on platform:
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration (4-5x faster)
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU if available, CPU fallback)
The backend is detected at runtime via `platform_detect.py`. Both backends implement the same interface, so the API remains consistent across platforms.
## API Endpoints
### Health & Info
@@ -47,12 +61,20 @@ Health check with model status.
"status": "healthy",
"model_loaded": true,
"gpu_available": true,
"vram_used_mb": 1024.5
"gpu_type": "Metal (Apple Silicon via MLX)",
"backend_type": "mlx",
"vram_used_mb": null
}
```
**Backend Types:**
- `"mlx"` - MLX backend (Apple Silicon with Metal acceleration)
- `"pytorch"` - PyTorch backend (Windows/Linux/Intel Mac)
### Voice Profiles
**Note:** The database is automatically initialized when the server starts. No manual setup required.
#### `POST /profiles`
Create a new voice profile.
@@ -266,13 +288,12 @@ data/
pip install -r requirements.txt
```
### 2. Initialize Database
**Note:** On Apple Silicon, also install MLX dependencies for faster inference:
```bash
python -c "from database import init_db; init_db()"
pip install -r requirements-mlx.txt
```
### 3. Download Models (Automatic)
### 2. Download Models (Automatic)
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
+2
View File
@@ -1 +1,3 @@
# Backend package
__version__ = "0.1.11"
+166
View File
@@ -0,0 +1,166 @@
"""
Backend abstraction layer for TTS and STT.
Provides a unified interface for MLX and PyTorch backends.
"""
from typing import Protocol, Optional, Tuple, List
from typing_extensions import runtime_checkable
import numpy as np
from ..platform_detect import get_backend_type
@runtime_checkable
class TTSBackend(Protocol):
"""Protocol for TTS backend implementations."""
async def load_model(self, model_size: str) -> None:
"""Load TTS model."""
...
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
Returns:
Tuple of (voice_prompt_dict, was_cached)
"""
...
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""
Combine multiple voice prompts.
Returns:
Tuple of (combined_audio_array, combined_text)
"""
...
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text.
Returns:
Tuple of (audio_array, sample_rate)
"""
...
def unload_model(self) -> None:
"""Unload model to free memory."""
...
def is_loaded(self) -> bool:
"""Check if model is loaded."""
...
def _get_model_path(self, model_size: str) -> str:
"""
Get model path for a given size.
Returns:
Model path or HuggingFace Hub ID
"""
...
@runtime_checkable
class STTBackend(Protocol):
"""Protocol for STT (Speech-to-Text) backend implementations."""
async def load_model(self, model_size: str) -> None:
"""Load STT model."""
...
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
"""
Transcribe audio to text.
Returns:
Transcribed text
"""
...
def unload_model(self) -> None:
"""Unload model to free memory."""
...
def is_loaded(self) -> bool:
"""Check if model is loaded."""
...
# Global backend instances
_tts_backend: Optional[TTSBackend] = None
_stt_backend: Optional[STTBackend] = None
def get_tts_backend() -> TTSBackend:
"""
Get or create TTS backend instance based on platform.
Returns:
TTS backend instance (MLX or PyTorch)
"""
global _tts_backend
if _tts_backend is None:
backend_type = get_backend_type()
if backend_type == "mlx":
from .mlx_backend import MLXTTSBackend
_tts_backend = MLXTTSBackend()
else:
from .pytorch_backend import PyTorchTTSBackend
_tts_backend = PyTorchTTSBackend()
return _tts_backend
def get_stt_backend() -> STTBackend:
"""
Get or create STT backend instance based on platform.
Returns:
STT backend instance (MLX or PyTorch)
"""
global _stt_backend
if _stt_backend is None:
backend_type = get_backend_type()
if backend_type == "mlx":
from .mlx_backend import MLXSTTBackend
_stt_backend = MLXSTTBackend()
else:
from .pytorch_backend import PyTorchSTTBackend
_stt_backend = PyTorchSTTBackend()
return _stt_backend
def reset_backends():
"""Reset backend instances (useful for testing)."""
global _tts_backend, _stt_backend
_tts_backend = None
_stt_backend = None
+458
View File
@@ -0,0 +1,458 @@
"""
MLX backend implementation for TTS and STT using mlx-audio.
"""
from typing import Optional, List, Tuple
import asyncio
import numpy as np
from pathlib import Path
from . import TTSBackend, STTBackend
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from ..utils.audio import normalize_audio, load_audio
from ..utils.progress import get_progress_manager
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from ..utils.tasks import get_task_manager
class MLXTTSBackend:
"""MLX-based TTS backend using mlx-audio."""
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self._current_model_size = None
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
def _get_model_path(self, model_size: str) -> str:
"""
Get the MLX model path.
Args:
model_size: Model size (1.7B or 0.6B)
Returns:
HuggingFace Hub model ID for MLX
"""
# MLX model mapping
mlx_model_map = {
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
# 0.6B not yet converted to MLX format
"0.6B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16", # Fallback to 1.7B
}
if model_size not in mlx_model_map:
raise ValueError(f"Unknown model size: {model_size}")
hf_model_id = mlx_model_map[model_size]
print(f"Will download MLX model from HuggingFace Hub: {hf_model_id}")
return hf_model_id
async def load_model_async(self, model_size: Optional[str] = None):
"""
Lazy load the MLX TTS model.
Args:
model_size: Model size to load (1.7B or 0.6B)
"""
if model_size is None:
model_size = self.model_size
# If already loaded with correct size, return
if self.model is not None and self._current_model_size == model_size:
return
# Unload existing model if different size requested
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
# Run blocking load in thread pool
await asyncio.to_thread(self._load_model_sync, model_size)
# Alias for compatibility
load_model = load_model_async
def _load_model_sync(self, model_size: str):
"""Synchronous model loading."""
try:
from mlx_audio.tts import load
# Get model path
model_path = self._get_model_path(model_size)
# Set up progress tracking
progress_manager = get_progress_manager()
model_name = f"qwen-tts-{model_size}"
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(model_name)
print(f"Loading MLX TTS model {model_size}...")
# Initialize progress state
progress_manager.update_progress(
model_name=model_name,
current=0,
total=1,
filename="",
status="downloading",
)
# Set up progress callback
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Use progress tracker during download
with tracker.patch_download():
# Load MLX model (downloads automatically)
self.model = load(model_path)
self._current_model_size = model_size
self.model_size = model_size
# Mark as complete
progress_manager.mark_complete(model_name)
task_manager.complete_download(model_name)
print(f"MLX TTS model {model_size} loaded successfully")
except ImportError as e:
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
except Exception as e:
print(f"Error loading MLX TTS model: {e}")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
self.model = None
self._current_model_size = None
print("MLX TTS model unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
MLX backend stores voice prompt as a dict with audio path and text.
The actual voice prompt processing happens during generation.
Args:
audio_path: Path to reference audio file
reference_text: Transcript of reference audio
use_cache: Whether to use cached prompt if available
Returns:
Tuple of (voice_prompt_dict, was_cached)
"""
await self.load_model_async(None)
# Check cache if enabled
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached_prompt = get_cached_voice_prompt(cache_key)
if cached_prompt is not None:
# Return cached prompt (should be dict format)
if isinstance(cached_prompt, dict):
return cached_prompt, True
# MLX voice prompt format - store audio path and text
# The model will process this during generation
voice_prompt_items = {
"ref_audio": str(audio_path),
"ref_text": reference_text,
}
# Cache if enabled
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cache_voice_prompt(cache_key, voice_prompt_items)
return voice_prompt_items, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""
Combine multiple reference samples for better quality.
Args:
audio_paths: List of audio file paths
reference_texts: List of reference texts
Returns:
Tuple of (combined_audio, combined_text)
"""
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate audio
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text using voice prompt.
Args:
text: Text to synthesize
voice_prompt: Voice prompt dictionary with ref_audio and ref_text
language: Language code (en or zh) - may not be fully supported by MLX
seed: Random seed for reproducibility
instruct: Natural language instruction (may not be supported by MLX)
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model_async(None)
print(f"Generating audio for text: {text}")
def _generate_sync():
"""Run synchronous generation in thread pool."""
# MLX generate() returns a generator yielding GenerationResult objects
audio_chunks = []
sample_rate = 24000
# Set seed if provided (MLX uses numpy random)
if seed is not None:
import mlx.core as mx
np.random.seed(seed)
mx.random.seed(seed)
# Extract voice prompt info
ref_audio = voice_prompt.get("ref_audio") or voice_prompt.get("ref_audio_path")
ref_text = voice_prompt.get("ref_text", "")
# Check if model supports voice cloning via generate method
# MLX API may support ref_audio parameter directly
try:
# Try with voice cloning parameters if supported
if ref_audio:
# Check if generate accepts ref_audio parameter
import inspect
sig = inspect.signature(self.model.generate)
if "ref_audio" in sig.parameters:
# Generate with voice cloning
for result in self.model.generate(text, ref_audio=ref_audio, ref_text=ref_text):
audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate
else:
# Fallback: generate without voice cloning
for result in self.model.generate(text):
audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate
else:
# No voice prompt, generate normally
for result in self.model.generate(text):
audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate
except Exception as e:
# If voice cloning fails, try without it
print(f"Warning: Voice cloning failed, generating without voice prompt: {e}")
for result in self.model.generate(text):
audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate
# Concatenate all chunks
if audio_chunks:
audio = np.concatenate([np.asarray(chunk, dtype=np.float32) for chunk in audio_chunks])
else:
# Fallback: empty audio
audio = np.array([], dtype=np.float32)
return audio, sample_rate
# Run blocking inference in thread pool
audio, sample_rate = await asyncio.to_thread(_generate_sync)
return audio, sample_rate
class MLXSTTBackend:
"""MLX-based STT backend using mlx-audio Whisper."""
def __init__(self, model_size: str = "base"):
self.model = None
self.model_size = model_size
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
async def load_model_async(self, model_size: Optional[str] = None):
"""
Lazy load the MLX Whisper model.
Args:
model_size: Model size (tiny, base, small, medium, large)
"""
if model_size is None:
model_size = self.model_size
if self.model is not None and self.model_size == model_size:
return
# Run blocking load in thread pool
await asyncio.to_thread(self._load_model_sync, model_size)
# Alias for compatibility
load_model = load_model_async
def _load_model_sync(self, model_size: str):
"""Synchronous model loading."""
try:
# IMPORTANT: Set up progress tracking BEFORE importing mlx_audio
# This ensures tqdm is patched before any HuggingFace Hub imports
progress_manager = get_progress_manager()
progress_model_name = f"whisper-{model_size}"
# Set up progress callback and tracker
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Patch tqdm BEFORE importing mlx_audio
# This is critical because mlx_audio imports huggingface_hub which imports tqdm
print("[DEBUG] Starting tqdm patch BEFORE mlx_audio import")
tracker_context = tracker.patch_download()
tracker_context.__enter__()
print("[DEBUG] tqdm patched, now importing mlx_audio")
# NOW import mlx_audio - it will use our patched tqdm
from mlx_audio.stt import load
# MLX Whisper uses the standard OpenAI models
model_name = f"openai/whisper-{model_size}"
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(progress_model_name)
print(f"Loading MLX Whisper model {model_size}...")
# Initialize progress state
progress_manager.update_progress(
model_name=progress_model_name,
current=0,
total=1,
filename="",
status="downloading",
)
# Load the model (tqdm is already patched from above)
try:
self.model = load(model_name)
finally:
# Exit the patch context
tracker_context.__exit__(None, None, None)
self.model_size = model_size
# Mark as complete
progress_manager.mark_complete(progress_model_name)
task_manager.complete_download(progress_model_name)
print(f"MLX Whisper model {model_size} loaded successfully")
except ImportError as e:
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
progress_model_name = f"whisper-{model_size}"
progress_manager.mark_error(progress_model_name, str(e))
task_manager.error_download(progress_model_name, str(e))
raise
except Exception as e:
print(f"Error loading MLX Whisper model: {e}")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
progress_model_name = f"whisper-{model_size}"
progress_manager.mark_error(progress_model_name, str(e))
task_manager.error_download(progress_model_name, str(e))
raise
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
self.model = None
print("MLX Whisper model unloaded")
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
"""
Transcribe audio to text.
Args:
audio_path: Path to audio file
language: Optional language hint (en or zh)
Returns:
Transcribed text
"""
await self.load_model_async(None)
def _transcribe_sync():
"""Run synchronous transcription in thread pool."""
# MLX Whisper transcription using generate method
# The generate method accepts audio path directly
decode_options = {}
if language:
decode_options["language"] = language
result = self.model.generate(str(audio_path), **decode_options)
# Extract text from result
if isinstance(result, str):
return result.strip()
elif isinstance(result, dict):
return result.get("text", "").strip()
elif hasattr(result, "text"):
return result.text.strip()
else:
return str(result).strip()
# Run blocking transcription in thread pool
return await asyncio.to_thread(_transcribe_sync)
+484
View File
@@ -0,0 +1,484 @@
"""
PyTorch backend implementation for TTS and STT.
"""
from typing import Optional, List, Tuple
import asyncio
import torch
import numpy as np
from pathlib import Path
from . import TTSBackend, STTBackend
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from ..utils.audio import normalize_audio, load_audio
from ..utils.progress import get_progress_manager
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from ..utils.tasks import get_task_manager
class PyTorchTTSBackend:
"""PyTorch-based TTS backend using Qwen3-TTS."""
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device()
self._current_model_size = None
def _get_device(self) -> str:
"""Get the best available device."""
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
# MPS can have issues, use CPU for stability
return "cpu"
return "cpu"
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
def _get_model_path(self, model_size: str) -> str:
"""
Get the HuggingFace Hub model ID.
Args:
model_size: Model size (1.7B or 0.6B)
Returns:
HuggingFace Hub model ID
"""
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
if model_size not in hf_model_map:
raise ValueError(f"Unknown model size: {model_size}")
return hf_model_map[model_size]
async def load_model_async(self, model_size: Optional[str] = None):
"""
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
Args:
model_size: Model size to load (1.7B or 0.6B)
"""
if model_size is None:
model_size = self.model_size
# If already loaded with correct size, return
if self.model is not None and self._current_model_size == model_size:
return
# Unload existing model if different size requested
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
# Run blocking load in thread pool
await asyncio.to_thread(self._load_model_sync, model_size)
# Alias for compatibility
load_model = load_model_async
def _load_model_sync(self, model_size: str):
"""Synchronous model loading."""
try:
# IMPORTANT: Set up progress tracking BEFORE importing qwen_tts
# This ensures tqdm is patched before any HuggingFace Hub imports
progress_manager = get_progress_manager()
model_name = f"qwen-tts-{model_size}"
# Set up progress callback and tracker
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Patch tqdm BEFORE importing qwen_tts
tracker_context = tracker.patch_download()
tracker_context.__enter__()
# NOW import qwen_tts - it will use our patched tqdm
from qwen_tts import Qwen3TTSModel
# Get model path (local or HuggingFace Hub ID)
model_path = self._get_model_path(model_size)
print(f"Loading TTS model {model_size} on {self.device}...")
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(model_name)
# Initialize progress state to show download has started
progress_manager.update_progress(
model_name=model_name,
current=0,
total=1, # Set to 1 initially, will be updated by callback
filename="",
status="downloading",
)
# Load the model (tqdm is already patched from above)
try:
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
)
finally:
# Exit the patch context
tracker_context.__exit__(None, None, None)
# Mark as complete
progress_manager.mark_complete(model_name)
task_manager.complete_download(model_name)
self._current_model_size = model_size
self.model_size = model_size
print(f"TTS model {model_size} loaded successfully")
except ImportError as e:
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
except Exception as e:
print(f"Error loading TTS model: {e}")
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
self.model = None
self._current_model_size = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
print("TTS model unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
Args:
audio_path: Path to reference audio file
reference_text: Transcript of reference audio
use_cache: Whether to use cached prompt if available
Returns:
Tuple of (voice_prompt_dict, was_cached)
"""
await self.load_model_async(None)
# Check cache if enabled
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached_prompt = get_cached_voice_prompt(cache_key)
if cached_prompt is not None:
# Cache stores as torch.Tensor but actual prompt is dict
# Convert if needed
if isinstance(cached_prompt, dict):
return cached_prompt, True
elif isinstance(cached_prompt, torch.Tensor):
# Legacy cache format - convert to dict
# This shouldn't happen in practice, but handle it
return {"prompt": cached_prompt}, True
def _create_prompt_sync():
"""Run synchronous voice prompt creation in thread pool."""
return self.model.create_voice_clone_prompt(
ref_audio=str(audio_path),
ref_text=reference_text,
x_vector_only_mode=False,
)
# Run blocking operation in thread pool
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
# Cache if enabled
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cache_voice_prompt(cache_key, voice_prompt_items)
return voice_prompt_items, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""
Combine multiple reference samples for better quality.
Args:
audio_paths: List of audio file paths
reference_texts: List of reference texts
Returns:
Tuple of (combined_audio, combined_text)
"""
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate audio
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text using voice prompt.
Args:
text: Text to synthesize
voice_prompt: Voice prompt dictionary from create_voice_prompt
language: Language code (en or zh)
seed: Random seed for reproducibility
instruct: Natural language instruction for speech delivery control
Returns:
Tuple of (audio_array, sample_rate)
"""
# Load model
await self.load_model_async(None)
def _generate_sync():
"""Run synchronous generation in thread pool."""
# Set seed if provided
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
# Generate audio - this is the blocking operation
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct,
)
return wavs[0], sample_rate
# Run blocking inference in thread pool to avoid blocking event loop
audio, sample_rate = await asyncio.to_thread(_generate_sync)
return audio, sample_rate
class PyTorchSTTBackend:
"""PyTorch-based STT backend using Whisper."""
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
def _get_device(self) -> str:
"""Get the best available device."""
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
# MPS support for Whisper
return "cpu" # Use CPU for stability
return "cpu"
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
async def load_model_async(self, model_size: Optional[str] = None):
"""
Lazy load the Whisper model.
Args:
model_size: Model size (tiny, base, small, medium, large)
"""
print(f"[DEBUG] load_model_async called with size: {model_size}")
if model_size is None:
model_size = self.model_size
print(f"[DEBUG] Model already loaded? {self.model is not None}, current size: {self.model_size}, requested: {model_size}")
if self.model is not None and self.model_size == model_size:
print(f"[DEBUG] Early return - model already loaded")
return
print(f"[DEBUG] Calling asyncio.to_thread for _load_model_sync")
# Run blocking load in thread pool
await asyncio.to_thread(self._load_model_sync, model_size)
print(f"[DEBUG] asyncio.to_thread completed")
# Alias for compatibility
load_model = load_model_async
def _load_model_sync(self, model_size: str):
"""Synchronous model loading."""
print(f"[DEBUG] _load_model_sync called for Whisper {model_size}")
try:
# IMPORTANT: Set up progress tracking BEFORE importing transformers
# This ensures tqdm is patched before any HuggingFace Hub imports
progress_manager = get_progress_manager()
progress_model_name = f"whisper-{model_size}"
# Set up progress callback and tracker
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Patch tqdm BEFORE importing transformers
print("[DEBUG] Starting tqdm patch BEFORE transformers import")
tracker_context = tracker.patch_download()
tracker_context.__enter__()
print("[DEBUG] tqdm patched, now importing transformers")
# NOW import transformers - it will use our patched tqdm
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
print(f"[DEBUG] Model name: {model_name}")
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(progress_model_name)
print(f"[DEBUG] Task manager started download")
print(f"Loading Whisper model {model_size} on {self.device}...")
# Initialize progress state to show download has started
print(f"[DEBUG] Calling update_progress...")
progress_manager.update_progress(
model_name=progress_model_name,
current=0,
total=1, # Set to 1 initially, will be updated by callback
filename="",
status="downloading",
)
print(f"[DEBUG] update_progress called, listeners: {len(progress_manager._listeners.get(progress_model_name, []))}")
# Load models (tqdm is already patched from above)
try:
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
finally:
# Exit the patch context
tracker_context.__exit__(None, None, None)
self.model.to(self.device)
self.model_size = model_size
# Mark as complete
progress_manager.mark_complete(progress_model_name)
task_manager.complete_download(progress_model_name)
print(f"Whisper model {model_size} loaded successfully")
except Exception as e:
print(f"Error loading Whisper model: {e}")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
progress_model_name = f"whisper-{model_size}"
progress_manager.mark_error(progress_model_name, str(e))
task_manager.error_download(progress_model_name, str(e))
raise
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
print("Whisper model unloaded")
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
"""
Transcribe audio to text.
Args:
audio_path: Path to audio file
language: Optional language hint (en or zh)
Returns:
Transcribed text
"""
await self.load_model_async(None)
def _transcribe_sync():
"""Run synchronous transcription in thread pool."""
# Load audio
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language if provided
forced_decoder_ids = None
if language:
# Support all languages from frontend: en, zh, ja, ko, de, fr, ru, pt, es, it
# Whisper supports these and many more
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language,
task="transcribe",
)
# Generate transcription
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
# Run blocking transcription in thread pool
return await asyncio.to_thread(_transcribe_sync)
+38 -8
View File
@@ -4,16 +4,19 @@ PyInstaller build script for creating standalone Python server binary.
import PyInstaller.__main__
import os
import platform
from pathlib import Path
def is_apple_silicon():
"""Check if running on Apple Silicon."""
return platform.system() == "Darwin" and platform.machine() == "arm64"
def build_server():
"""Build Python server as standalone binary."""
backend_dir = Path(__file__).parent
# Check for local editable qwen_tts install
local_qwen_path = Path.home() / 'Projects' / 'voice' / 'Qwen3-TTS'
# PyInstaller arguments
args = [
'server.py', # Use server.py as entry point instead of main.py
@@ -21,12 +24,13 @@ def build_server():
'--name', 'voicebox-server',
]
# Add local qwen_tts path if it exists (for editable installs)
if local_qwen_path.exists():
args.extend(['--paths', str(local_qwen_path)])
print(f"Using local qwen_tts source from: {local_qwen_path}")
# Add local qwen_tts path if specified (for editable installs)
qwen_tts_path = os.getenv('QWEN_TTS_PATH')
if qwen_tts_path and Path(qwen_tts_path).exists():
args.extend(['--paths', str(qwen_tts_path)])
print(f"Using local qwen_tts source from: {qwen_tts_path}")
# Add hidden imports
# Add common hidden imports
args.extend([
'--hidden-import', 'backend',
'--hidden-import', 'backend.main',
@@ -37,6 +41,9 @@ def build_server():
'--hidden-import', 'backend.history',
'--hidden-import', 'backend.tts',
'--hidden-import', 'backend.transcribe',
'--hidden-import', 'backend.platform_detect',
'--hidden-import', 'backend.backends',
'--hidden-import', 'backend.backends.pytorch_backend',
'--hidden-import', 'backend.utils.audio',
'--hidden-import', 'backend.utils.cache',
'--hidden-import', 'backend.utils.progress',
@@ -61,6 +68,29 @@ def build_server():
# Fix for pkg_resources and jaraco namespace packages
'--hidden-import', 'pkg_resources.extern',
'--collect-submodules', 'jaraco',
])
# Add MLX-specific imports if building on Apple Silicon
if is_apple_silicon():
print("Building for Apple Silicon - including MLX dependencies")
args.extend([
'--hidden-import', 'backend.backends.mlx_backend',
'--hidden-import', 'mlx',
'--hidden-import', 'mlx.core',
'--hidden-import', 'mlx.nn',
'--hidden-import', 'mlx_audio',
'--hidden-import', 'mlx_audio.tts',
'--hidden-import', 'mlx_audio.stt',
'--collect-submodules', 'mlx',
'--collect-submodules', 'mlx_audio',
# Collect MLX data files including Metal shader libraries (.metallib)
'--collect-data', 'mlx',
'--collect-data', 'mlx_audio',
])
else:
print("Building for non-Apple Silicon platform - PyTorch only")
args.extend([
'--noconfirm',
'--clean',
])
+32 -1
View File
@@ -17,11 +17,12 @@ Base = declarative_base()
class VoiceProfile(Base):
"""Voice profile database model."""
__tablename__ = "profiles"
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
name = Column(String, unique=True, nullable=False)
description = Column(Text)
language = Column(String, default="en")
avatar_path = Column(String, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
@@ -71,6 +72,8 @@ class StoryItem(Base):
generation_id = Column(String, ForeignKey("generations.id"), nullable=False)
start_time_ms = Column(Integer, nullable=False, default=0) # Milliseconds from story start
track = Column(Integer, nullable=False, default=0) # Track number (0 = main track)
trim_start_ms = Column(Integer, nullable=False, default=0) # Milliseconds trimmed from start
trim_end_ms = Column(Integer, nullable=False, default=0) # Milliseconds trimmed from end
created_at = Column(DateTime, default=datetime.utcnow)
@@ -256,6 +259,34 @@ def _run_migrations(engine):
conn.execute(text("ALTER TABLE story_items ADD COLUMN track INTEGER NOT NULL DEFAULT 0"))
conn.commit()
print("Added track column to story_items")
# Migration: Add trim columns if they don't exist
# Re-check columns after potential track migration
columns = {col['name'] for col in inspector.get_columns('story_items')}
if 'trim_start_ms' not in columns:
print("Migrating story_items: adding trim_start_ms column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE story_items ADD COLUMN trim_start_ms INTEGER NOT NULL DEFAULT 0"))
conn.commit()
print("Added trim_start_ms column to story_items")
columns = {col['name'] for col in inspector.get_columns('story_items')}
if 'trim_end_ms' not in columns:
print("Migrating story_items: adding trim_end_ms column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE story_items ADD COLUMN trim_end_ms INTEGER NOT NULL DEFAULT 0"))
conn.commit()
print("Added trim_end_ms column to story_items")
# Migration: Add avatar_path to profiles table
if 'profiles' in inspector.get_table_names():
columns = {col['name'] for col in inspector.get_columns('profiles')}
if 'avatar_path' not in columns:
print("Migrating profiles: adding avatar_path column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE profiles ADD COLUMN avatar_path VARCHAR"))
conn.commit()
print("Added avatar_path column to profiles")
def get_db():
+40 -9
View File
@@ -75,6 +75,16 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
zip_buffer = io.BytesIO()
with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
# Check if profile has avatar
has_avatar = False
if profile.avatar_path:
avatar_path = Path(profile.avatar_path)
if avatar_path.exists():
has_avatar = True
# Add avatar to ZIP root with original extension
avatar_ext = avatar_path.suffix
zip_file.write(avatar_path, f"avatar{avatar_ext}")
# Create manifest.json
manifest = {
"version": "1.0",
@@ -82,30 +92,31 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
"name": profile.name,
"description": profile.description,
"language": profile.language,
}
},
"has_avatar": has_avatar,
}
zip_file.writestr("manifest.json", json.dumps(manifest, indent=2))
# Create samples.json mapping
samples_data = {}
profile_dir = _get_profiles_dir() / profile_id
for sample in samples:
# Get filename from audio_path (should be {sample_id}.wav)
audio_path = Path(sample.audio_path)
filename = audio_path.name
# Read audio file
if not audio_path.exists():
raise ValueError(f"Audio file not found: {audio_path}")
# Add to samples directory in ZIP
zip_path = f"samples/{filename}"
zip_file.write(audio_path, zip_path)
# Map filename to reference text
samples_data[filename] = sample.reference_text
zip_file.writestr("samples.json", json.dumps(samples_data, indent=2))
zip_buffer.seek(0)
@@ -168,11 +179,31 @@ async def import_profile_from_zip(file_bytes: bytes, db: Session) -> VoiceProfil
)
profile = await create_profile(profile_create, db)
# Extract and add samples
profile_dir = _get_profiles_dir() / profile.id
profile_dir.mkdir(parents=True, exist_ok=True)
# Handle avatar if present
avatar_files = [f for f in namelist if f.startswith("avatar.")]
if avatar_files:
try:
avatar_file = avatar_files[0]
# Extract to temporary file
import tempfile
with tempfile.NamedTemporaryFile(suffix=Path(avatar_file).suffix, delete=False) as tmp:
tmp.write(zip_file.read(avatar_file))
tmp_path = tmp.name
try:
from .profiles import upload_avatar
await upload_avatar(profile.id, tmp_path, db)
finally:
Path(tmp_path).unlink(missing_ok=True)
except Exception as e:
# Avatar import is optional - continue even if it fails
pass
for filename, reference_text in samples_data.items():
# Validate filename
if not filename.endswith('.wav'):
+297 -47
View File
@@ -11,6 +11,7 @@ from fastapi.staticfiles import StaticFiles
from sqlalchemy.orm import Session
from typing import List, Optional
from datetime import datetime
import asyncio
import uvicorn
import argparse
import torch
@@ -18,16 +19,20 @@ import tempfile
import io
from pathlib import Path
import uuid
import asyncio
import signal
import os
from . import database, models, profiles, history, tts, transcribe, config, export_import, channels, stories
from . import database, models, profiles, history, tts, transcribe, config, export_import, channels, stories, __version__
from .database import get_db, Generation as DBGeneration, VoiceProfile as DBVoiceProfile
from .utils.progress import get_progress_manager
from .utils.tasks import get_task_manager
from .platform_detect import get_backend_type
app = FastAPI(
title="voicebox API",
description="Production-quality Qwen3-TTS voice cloning API",
version="0.1.0",
version=__version__,
)
# CORS middleware
@@ -47,23 +52,43 @@ app.add_middleware(
@app.get("/")
async def root():
"""Root endpoint."""
return {"message": "voicebox API", "version": "0.1.6"}
return {"message": "voicebox API", "version": __version__}
@app.post("/shutdown")
async def shutdown():
"""Gracefully shutdown the server."""
async def shutdown_async():
await asyncio.sleep(0.1) # Give response time to send
os.kill(os.getpid(), signal.SIGTERM)
asyncio.create_task(shutdown_async())
return {"message": "Shutting down..."}
@app.get("/health", response_model=models.HealthResponse)
async def health():
"""Health check endpoint."""
from huggingface_hub import hf_hub_download
from huggingface_hub import hf_hub_download, constants as hf_constants
from pathlib import Path
import os
tts_model = tts.get_tts_model()
backend_type = get_backend_type()
# Check for GPU availability (CUDA or MPS)
has_cuda = torch.cuda.is_available()
has_mps = hasattr(torch.backends, 'mps') and torch.backends.mps.is_available()
gpu_available = has_cuda or has_mps
gpu_type = None
if has_cuda:
gpu_type = f"CUDA ({torch.cuda.get_device_name(0)})"
elif has_mps:
gpu_type = "MPS (Apple Silicon)"
elif backend_type == "mlx":
gpu_type = "Metal (Apple Silicon via MLX)"
vram_used = None
if has_cuda:
vram_used = torch.cuda.memory_allocated() / 1024 / 1024 # MB
@@ -90,7 +115,11 @@ async def health():
model_downloaded = None
try:
# Check if the default model (1.7B) is cached
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
# Use different model IDs based on backend
if backend_type == "mlx":
default_model_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
else:
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
# Method 1: Try scan_cache_dir if available
try:
@@ -101,15 +130,16 @@ async def health():
model_downloaded = True
break
except (ImportError, Exception):
# Method 2: Check cache directory
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
repo_cache = Path(cache_dir) / "models--" + default_model_id.replace("/", "--")
# Method 2: Check cache directory (using HuggingFace's OS-specific cache location)
cache_dir = hf_constants.HF_HUB_CACHE
repo_cache = Path(cache_dir) / ("models--" + default_model_id.replace("/", "--"))
if repo_cache.exists():
has_model_files = (
any(repo_cache.rglob("*.bin")) or
any(repo_cache.rglob("*.safetensors")) or
any(repo_cache.rglob("*.pt")) or
any(repo_cache.rglob("*.pth"))
any(repo_cache.rglob("*.pth")) or
any(repo_cache.rglob("*.npz")) # MLX models may use npz
)
model_downloaded = has_model_files
except Exception:
@@ -121,7 +151,9 @@ async def health():
model_downloaded=model_downloaded,
model_size=model_size,
gpu_available=gpu_available,
gpu_type=gpu_type,
vram_used_mb=vram_used,
backend_type=backend_type,
)
@@ -261,6 +293,74 @@ async def delete_profile_sample(
return {"message": "Sample deleted successfully"}
@app.put("/profiles/samples/{sample_id}", response_model=models.ProfileSampleResponse)
async def update_profile_sample(
sample_id: str,
data: models.ProfileSampleUpdate,
db: Session = Depends(get_db),
):
"""Update a profile sample's reference text."""
sample = await profiles.update_profile_sample(sample_id, data.reference_text, db)
if not sample:
raise HTTPException(status_code=404, detail="Sample not found")
return sample
@app.post("/profiles/{profile_id}/avatar", response_model=models.VoiceProfileResponse)
async def upload_profile_avatar(
profile_id: str,
file: UploadFile = File(...),
db: Session = Depends(get_db),
):
"""Upload or update avatar image for a profile."""
# Save uploaded file to temp location
with tempfile.NamedTemporaryFile(delete=False, suffix=Path(file.filename).suffix) as tmp:
content = await file.read()
tmp.write(content)
tmp_path = tmp.name
try:
profile = await profiles.upload_avatar(profile_id, tmp_path, db)
return profile
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
finally:
# Clean up temp file
Path(tmp_path).unlink(missing_ok=True)
@app.get("/profiles/{profile_id}/avatar")
async def get_profile_avatar(
profile_id: str,
db: Session = Depends(get_db),
):
"""Get avatar image for a profile."""
profile = await profiles.get_profile(profile_id, db)
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
if not profile.avatar_path:
raise HTTPException(status_code=404, detail="No avatar found for this profile")
avatar_path = Path(profile.avatar_path)
if not avatar_path.exists():
raise HTTPException(status_code=404, detail="Avatar file not found")
return FileResponse(avatar_path)
@app.delete("/profiles/{profile_id}/avatar")
async def delete_profile_avatar(
profile_id: str,
db: Session = Depends(get_db),
):
"""Delete avatar image for a profile."""
success = await profiles.delete_avatar(profile_id, db)
if not success:
raise HTTPException(status_code=404, detail="Profile not found or no avatar to delete")
return {"message": "Avatar deleted successfully"}
@app.get("/profiles/{profile_id}/export")
async def export_profile(
profile_id: str,
@@ -451,6 +551,36 @@ async def generate_speech(
tts_model = tts.get_tts_model()
# Load the requested model size if different from current (async to not block)
model_size = data.model_size or "1.7B"
# Check if model needs to be downloaded first
model_path = tts_model._get_model_path(model_size)
if model_path.startswith("Qwen/"):
# Model not cached - check if it exists remotely or needs download
from huggingface_hub import constants as hf_constants
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_path.replace("/", "--"))
if not repo_cache.exists():
# Start download in background
model_name = f"qwen-tts-{model_size}"
async def download_model_background():
try:
await tts_model.load_model_async(model_size)
except Exception as e:
task_manager.error_download(model_name, str(e))
task_manager.start_download(model_name)
asyncio.create_task(download_model_background())
# Return 202 Accepted with download info
raise HTTPException(
status_code=202,
detail={
"message": f"Model {model_size} is being downloaded. Please wait and try again.",
"model_name": model_name,
"downloading": True
}
)
await tts_model.load_model_async(model_size)
audio, sample_rate = await tts_model.generate(
data.text,
@@ -684,6 +814,37 @@ async def transcribe_audio(
# Transcribe
whisper_model = transcribe.get_whisper_model()
# Check if Whisper model is downloaded (uses default size "base")
model_size = whisper_model.model_size
model_name = f"openai/whisper-{model_size}"
# Check if model is cached
from huggingface_hub import constants as hf_constants
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_name.replace("/", "--"))
if not repo_cache.exists():
# Start download in background
progress_model_name = f"whisper-{model_size}"
async def download_whisper_background():
try:
await whisper_model.load_model_async(model_size)
except Exception as e:
get_task_manager().error_download(progress_model_name, str(e))
get_task_manager().start_download(progress_model_name)
asyncio.create_task(download_whisper_background())
# Return 202 Accepted
raise HTTPException(
status_code=202,
detail={
"message": f"Whisper model {model_size} is being downloaded. Please wait and try again.",
"model_name": progress_model_name,
"downloading": True
}
)
text = await whisper_model.transcribe(tmp_path, language)
return models.TranscriptionResponse(
@@ -770,14 +931,14 @@ async def add_story_item(
return item
@app.delete("/stories/{story_id}/items/{generation_id}")
@app.delete("/stories/{story_id}/items/{item_id}")
async def remove_story_item(
story_id: str,
generation_id: str,
item_id: str,
db: Session = Depends(get_db),
):
"""Remove a generation from a story."""
success = await stories.remove_item_from_story(story_id, generation_id, db)
"""Remove a story item from a story."""
success = await stories.remove_item_from_story(story_id, item_id, db)
if not success:
raise HTTPException(status_code=404, detail="Story item not found")
return {"message": "Item removed successfully"}
@@ -809,15 +970,56 @@ async def reorder_story_items(
return items
@app.put("/stories/{story_id}/items/{generation_id}/move", response_model=models.StoryItemDetail)
@app.put("/stories/{story_id}/items/{item_id}/move", response_model=models.StoryItemDetail)
async def move_story_item(
story_id: str,
generation_id: str,
item_id: str,
data: models.StoryItemMove,
db: Session = Depends(get_db),
):
"""Move a story item (update position and/or track)."""
item = await stories.move_story_item(story_id, generation_id, data, db)
item = await stories.move_story_item(story_id, item_id, data, db)
if item is None:
raise HTTPException(status_code=404, detail="Story item not found")
return item
@app.put("/stories/{story_id}/items/{item_id}/trim", response_model=models.StoryItemDetail)
async def trim_story_item(
story_id: str,
item_id: str,
data: models.StoryItemTrim,
db: Session = Depends(get_db),
):
"""Trim a story item (update trim_start_ms and trim_end_ms)."""
item = await stories.trim_story_item(story_id, item_id, data, db)
if item is None:
raise HTTPException(status_code=404, detail="Story item not found or invalid trim values")
return item
@app.post("/stories/{story_id}/items/{item_id}/split", response_model=List[models.StoryItemDetail])
async def split_story_item(
story_id: str,
item_id: str,
data: models.StoryItemSplit,
db: Session = Depends(get_db),
):
"""Split a story item at a given time, creating two clips."""
items = await stories.split_story_item(story_id, item_id, data, db)
if items is None:
raise HTTPException(status_code=404, detail="Story item not found or invalid split point")
return items
@app.post("/stories/{story_id}/items/{item_id}/duplicate", response_model=models.StoryItemDetail)
async def duplicate_story_item(
story_id: str,
item_id: str,
db: Session = Depends(get_db),
):
"""Duplicate a story item, creating a copy with all properties."""
item = await stories.duplicate_story_item(story_id, item_id, db)
if item is None:
raise HTTPException(status_code=404, detail="Story item not found")
return item
@@ -953,10 +1155,12 @@ async def get_model_progress(model_name: str):
@app.get("/models/status", response_model=models.ModelStatusListResponse)
async def get_model_status():
"""Get status of all available models."""
from huggingface_hub import hf_hub_download
from huggingface_hub import hf_hub_download, constants as hf_constants
from pathlib import Path
import os
backend_type = get_backend_type()
# Try to import scan_cache_dir (might not be available in older versions)
try:
from huggingface_hub import scan_cache_dir
@@ -968,7 +1172,7 @@ async def get_model_status():
"""Check if TTS model is loaded with specific size."""
try:
tts_model = tts.get_tts_model()
return tts_model.is_loaded() and tts_model.model_size == model_size
return tts_model.is_loaded() and getattr(tts_model, 'model_size', None) == model_size
except Exception:
return False
@@ -976,50 +1180,66 @@ async def get_model_status():
"""Check if Whisper model is loaded with specific size."""
try:
whisper_model = transcribe.get_whisper_model()
return whisper_model.is_loaded() and whisper_model.model_size == model_size
return whisper_model.is_loaded() and getattr(whisper_model, 'model_size', None) == model_size
except Exception:
return False
# Use backend-specific model IDs
if backend_type == "mlx":
tts_1_7b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
tts_0_6b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16" # Fallback to 1.7B
whisper_base_id = "mlx-community/whisper-base"
whisper_small_id = "mlx-community/whisper-small"
whisper_medium_id = "mlx-community/whisper-medium"
whisper_large_id = "mlx-community/whisper-large"
else:
tts_1_7b_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
tts_0_6b_id = "Qwen/Qwen3-TTS-12Hz-0.6B-Base"
whisper_base_id = "openai/whisper-base"
whisper_small_id = "openai/whisper-small"
whisper_medium_id = "openai/whisper-medium"
whisper_large_id = "openai/whisper-large"
model_configs = [
{
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen TTS 1.7B",
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"hf_repo_id": tts_1_7b_id,
"model_size": "1.7B",
"check_loaded": lambda: check_tts_loaded("1.7B"),
},
{
"model_name": "qwen-tts-0.6B",
"display_name": "Qwen TTS 0.6B",
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
"hf_repo_id": tts_0_6b_id,
"model_size": "0.6B",
"check_loaded": lambda: check_tts_loaded("0.6B"),
},
{
"model_name": "whisper-base",
"display_name": "Whisper Base",
"hf_repo_id": "openai/whisper-base",
"hf_repo_id": whisper_base_id,
"model_size": "base",
"check_loaded": lambda: check_whisper_loaded("base"),
},
{
"model_name": "whisper-small",
"display_name": "Whisper Small",
"hf_repo_id": "openai/whisper-small",
"hf_repo_id": whisper_small_id,
"model_size": "small",
"check_loaded": lambda: check_whisper_loaded("small"),
},
{
"model_name": "whisper-medium",
"display_name": "Whisper Medium",
"hf_repo_id": "openai/whisper-medium",
"hf_repo_id": whisper_medium_id,
"model_size": "medium",
"check_loaded": lambda: check_whisper_loaded("medium"),
},
{
"model_name": "whisper-large",
"display_name": "Whisper Large",
"hf_repo_id": "openai/whisper-large",
"hf_repo_id": whisper_large_id,
"model_size": "large",
"check_loaded": lambda: check_whisper_loaded("large"),
},
@@ -1056,19 +1276,21 @@ async def get_model_status():
pass
break
# Method 2: Fallback to checking cache directory directly
# Method 2: Fallback to checking cache directory directly (using HuggingFace's OS-specific cache location)
if not downloaded:
try:
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
repo_cache = Path(cache_dir) / "models--" + config["hf_repo_id"].replace("/", "--")
cache_dir = hf_constants.HF_HUB_CACHE
repo_cache = Path(cache_dir) / ("models--" + config["hf_repo_id"].replace("/", "--"))
if repo_cache.exists():
# Check for model files (bin, safetensors, or other common model files)
# MLX models may use .npz or .safetensors
has_model_files = (
any(repo_cache.rglob("*.bin")) or
any(repo_cache.rglob("*.safetensors")) or
any(repo_cache.rglob("*.pt")) or
any(repo_cache.rglob("*.pth")) or
any(repo_cache.rglob("*.npz")) or
any(repo_cache.rglob("model.safetensors.index.json")) or
any(repo_cache.rglob("pytorch_model.bin.index.json"))
)
@@ -1168,22 +1390,26 @@ async def trigger_model_download(request: models.ModelDownloadRequest):
config = model_configs[request.model_name]
try:
# Start tracking download
task_manager.start_download(request.model_name)
# Trigger download by loading the model (which will download if not cached)
# Run in background to avoid blocking
await asyncio.to_thread(config["load_func"])
# Mark download as complete
task_manager.complete_download(request.model_name)
return {"message": f"Model {request.model_name} download started"}
except Exception as e:
# Mark download as failed
task_manager.error_download(request.model_name, str(e))
raise HTTPException(status_code=500, detail=str(e))
async def download_in_background():
"""Download model in background without blocking the HTTP request."""
try:
# Call the load function (which may be async)
result = config["load_func"]()
# If it's a coroutine, await it
if asyncio.iscoroutine(result):
await result
task_manager.complete_download(request.model_name)
except Exception as e:
task_manager.error_download(request.model_name, str(e))
# Start tracking download
task_manager.start_download(request.model_name)
# Start download in background task (don't await)
asyncio.create_task(download_in_background())
# Return immediately - frontend should poll progress endpoint
return {"message": f"Model {request.model_name} download started"}
@app.delete("/models/{model_name}")
@@ -1191,6 +1417,7 @@ async def delete_model(model_name: str):
"""Delete a downloaded model from the HuggingFace cache."""
import shutil
import os
from huggingface_hub import constants as hf_constants
# Map model names to HuggingFace repo IDs
model_configs = {
@@ -1243,8 +1470,8 @@ async def delete_model(model_name: str):
if whisper_model.is_loaded() and whisper_model.model_size == config["model_size"]:
transcribe.unload_whisper_model()
# Find and delete the cache directory
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
# Find and delete the cache directory (using HuggingFace's OS-specific cache location)
cache_dir = hf_constants.HF_HUB_CACHE
repo_cache_dir = Path(cache_dir) / ("models--" + hf_repo_id.replace("/", "--"))
# Check if the cache directory exists
@@ -1340,10 +1567,13 @@ async def get_active_tasks():
def _get_gpu_status() -> str:
"""Get GPU availability status."""
backend_type = get_backend_type()
if torch.cuda.is_available():
return f"CUDA ({torch.cuda.get_device_name(0)})"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return "MPS (Apple Silicon)"
elif backend_type == "mlx":
return "Metal (Apple Silicon via MLX)"
return "None (CPU only)"
@@ -1353,8 +1583,28 @@ async def startup_event():
print("voicebox API starting up...")
database.init_db()
print(f"Database initialized at {database._db_path}")
backend_type = get_backend_type()
print(f"Backend: {backend_type.upper()}")
print(f"GPU available: {_get_gpu_status()}")
# Initialize progress manager with main event loop for thread-safe operations
try:
progress_manager = get_progress_manager()
progress_manager._set_main_loop(asyncio.get_running_loop())
print("Progress manager initialized with event loop")
except Exception as e:
print(f"Warning: Could not initialize progress manager event loop: {e}")
# Ensure HuggingFace cache directory exists
try:
from huggingface_hub import constants as hf_constants
cache_dir = Path(hf_constants.HF_HUB_CACHE)
cache_dir.mkdir(parents=True, exist_ok=True)
print(f"HuggingFace cache directory: {cache_dir}")
except Exception as e:
print(f"Warning: Could not create HuggingFace cache directory: {e}")
print("Model downloads may fail. Please ensure the directory exists and has write permissions.")
@app.on_event("shutdown")
async def shutdown_event():
+21
View File
@@ -20,6 +20,7 @@ class VoiceProfileResponse(BaseModel):
name: str
description: Optional[str]
language: str
avatar_path: Optional[str] = None
created_at: datetime
updated_at: datetime
@@ -32,6 +33,11 @@ class ProfileSampleCreate(BaseModel):
reference_text: str = Field(..., min_length=1, max_length=1000)
class ProfileSampleUpdate(BaseModel):
"""Request model for updating a profile sample."""
reference_text: str = Field(..., min_length=1, max_length=1000)
class ProfileSampleResponse(BaseModel):
"""Response model for profile sample."""
id: str
@@ -118,7 +124,9 @@ class HealthResponse(BaseModel):
model_downloaded: Optional[bool] = None # Whether model is cached/downloaded
model_size: Optional[str] = None # Current model size if loaded
gpu_available: bool
gpu_type: Optional[str] = None # GPU type (CUDA, MPS, or None)
vram_used_mb: Optional[float] = None
backend_type: Optional[str] = None # Backend type (mlx or pytorch)
class ModelStatus(BaseModel):
@@ -221,6 +229,8 @@ class StoryItemDetail(BaseModel):
generation_id: str
start_time_ms: int
track: int = 0
trim_start_ms: int = 0
trim_end_ms: int = 0
created_at: datetime
# Generation details
profile_id: str
@@ -277,3 +287,14 @@ class StoryItemMove(BaseModel):
"""Request model for moving a story item (position and/or track)."""
start_time_ms: int = Field(..., ge=0)
track: int = 0
class StoryItemTrim(BaseModel):
"""Request model for trimming a story item."""
trim_start_ms: int = Field(..., ge=0)
trim_end_ms: int = Field(..., ge=0)
class StoryItemSplit(BaseModel):
"""Request model for splitting a story item."""
split_time_ms: int = Field(..., ge=0) # Time within the clip to split at (relative to clip start)
+33
View File
@@ -0,0 +1,33 @@
"""
Platform detection for backend selection.
"""
import platform
from typing import Literal
def is_apple_silicon() -> bool:
"""
Check if running on Apple Silicon (arm64 macOS).
Returns:
True if on Apple Silicon, False otherwise
"""
return platform.system() == "Darwin" and platform.machine() == "arm64"
def get_backend_type() -> Literal["mlx", "pytorch"]:
"""
Detect the best backend for the current platform.
Returns:
"mlx" on Apple Silicon (if MLX is available), "pytorch" otherwise
"""
if is_apple_silicon():
try:
import mlx
return "mlx"
except ImportError:
# MLX not installed, fallback to PyTorch
return "pytorch"
return "pytorch"
+132 -8
View File
@@ -21,6 +21,7 @@ from .database import (
ProfileSample as DBProfileSample,
)
from .utils.audio import validate_reference_audio, load_audio, save_audio
from .utils.images import validate_image, process_avatar
from .tts import get_tts_model
from . import config
@@ -273,6 +274,33 @@ async def delete_profile_sample(
return True
async def update_profile_sample(
sample_id: str,
reference_text: str,
db: Session,
) -> Optional[ProfileSampleResponse]:
"""
Update a profile sample's reference text.
Args:
sample_id: Sample ID
reference_text: Updated reference text
db: Database session
Returns:
Updated sample or None if not found
"""
sample = db.query(DBProfileSample).filter_by(id=sample_id).first()
if not sample:
return None
sample.reference_text = reference_text
db.commit()
db.refresh(sample)
return ProfileSampleResponse.model_validate(sample)
async def create_voice_prompt_for_profile(
profile_id: str,
db: Session,
@@ -280,23 +308,23 @@ async def create_voice_prompt_for_profile(
) -> dict:
"""
Create a combined voice prompt from all samples in a profile.
Args:
profile_id: Profile ID
db: Database session
use_cache: Whether to use cached prompts
Returns:
Voice prompt dictionary
"""
# Get all samples for profile
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
if not samples:
raise ValueError(f"No samples found for profile {profile_id}")
tts_model = get_tts_model()
if len(samples) == 1:
# Single sample - use directly
sample = samples[0]
@@ -310,19 +338,19 @@ async def create_voice_prompt_for_profile(
# Multiple samples - combine them
audio_paths = [s.audio_path for s in samples]
reference_texts = [s.reference_text for s in samples]
# Combine audio
combined_audio, combined_text = await tts_model.combine_voice_prompts(
audio_paths,
reference_texts,
)
# Save combined audio temporarily
import tempfile
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
save_audio(combined_audio, tmp.name, 24000)
tmp_path = tmp.name
try:
# Create prompt from combined audio
voice_prompt, _ = await tts_model.create_voice_prompt(
@@ -334,3 +362,99 @@ async def create_voice_prompt_for_profile(
finally:
# Clean up temp file
Path(tmp_path).unlink(missing_ok=True)
async def upload_avatar(
profile_id: str,
image_path: str,
db: Session,
) -> VoiceProfileResponse:
"""
Upload and process avatar image for a profile.
Args:
profile_id: Profile ID
image_path: Path to uploaded image file
db: Database session
Returns:
Updated profile
"""
# Validate profile exists
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile {profile_id} not found")
# Validate image
is_valid, error_msg = validate_image(image_path)
if not is_valid:
raise ValueError(error_msg)
# Delete existing avatar if present
if profile.avatar_path:
old_avatar = Path(profile.avatar_path)
if old_avatar.exists():
old_avatar.unlink()
# Determine file extension from uploaded file
from PIL import Image
with Image.open(image_path) as img:
# Normalize JPEG variants (MPO is multi-picture format from some cameras)
img_format = img.format
if img_format in ('MPO', 'JPG'):
img_format = 'JPEG'
ext_map = {
'PNG': '.png',
'JPEG': '.jpg',
'WEBP': '.webp'
}
ext = ext_map.get(img_format, '.png')
# Save processed image to profile directory
profile_dir = _get_profiles_dir() / profile_id
profile_dir.mkdir(parents=True, exist_ok=True)
output_path = profile_dir / f"avatar{ext}"
process_avatar(image_path, str(output_path))
# Update database
profile.avatar_path = str(output_path)
profile.updated_at = datetime.utcnow()
db.commit()
db.refresh(profile)
return VoiceProfileResponse.model_validate(profile)
async def delete_avatar(
profile_id: str,
db: Session,
) -> bool:
"""
Delete avatar image for a profile.
Args:
profile_id: Profile ID
db: Database session
Returns:
True if deleted, False if not found or no avatar
"""
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile or not profile.avatar_path:
return False
# Delete avatar file
avatar_path = Path(profile.avatar_path)
if avatar_path.exists():
avatar_path.unlink()
# Update database
profile.avatar_path = None
profile.updated_at = datetime.utcnow()
db.commit()
return True
+5
View File
@@ -0,0 +1,5 @@
# MLX-specific dependencies (Apple Silicon only)
# These should only be installed on aarch64-apple-darwin platforms
mlx>=0.30.0
mlx-audio>=0.3.1
+1
View File
@@ -21,3 +21,4 @@ numpy>=1.24.0
# Utilities
python-multipart>=0.0.6
Pillow>=10.0.0
+311 -11
View File
@@ -18,6 +18,8 @@ from .models import (
StoryItemCreate,
StoryItemBatchUpdate,
StoryItemMove,
StoryItemTrim,
StoryItemSplit,
)
from .database import Story as DBStory, StoryItem as DBStoryItem, Generation as DBGeneration, VoiceProfile as DBVoiceProfile
from .utils.audio import load_audio, save_audio
@@ -129,6 +131,8 @@ async def get_story(
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=getattr(item, 'trim_start_ms', 0),
trim_end_ms=getattr(item, 'trim_end_ms', 0),
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile_name,
@@ -252,6 +256,8 @@ async def add_item_to_story(
generation_id=existing.generation_id,
start_time_ms=existing.start_time_ms,
track=existing.track,
trim_start_ms=getattr(existing, 'trim_start_ms', 0),
trim_end_ms=getattr(existing, 'trim_end_ms', 0),
created_at=existing.created_at,
profile_id=generation.profile_id,
profile_name=profile.name if profile else "Unknown",
@@ -321,6 +327,8 @@ async def add_item_to_story(
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=getattr(item, 'trim_start_ms', 0),
trim_end_ms=getattr(item, 'trim_end_ms', 0),
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile.name if profile else "Unknown",
@@ -336,7 +344,7 @@ async def add_item_to_story(
async def move_story_item(
story_id: str,
generation_id: str,
item_id: str,
data: StoryItemMove,
db: Session,
) -> Optional[StoryItemDetail]:
@@ -345,7 +353,7 @@ async def move_story_item(
Args:
story_id: Story ID
generation_id: Generation ID of the item to move
item_id: Story item ID
data: New position and track data
db: Database session
@@ -354,14 +362,14 @@ async def move_story_item(
"""
# Get the item
item = db.query(DBStoryItem).filter_by(
id=item_id,
story_id=story_id,
generation_id=generation_id
).first()
if not item:
return None
# Get the generation
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
generation = db.query(DBGeneration).filter_by(id=item.generation_id).first()
if not generation:
return None
@@ -386,6 +394,8 @@ async def move_story_item(
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=getattr(item, 'trim_start_ms', 0),
trim_end_ms=getattr(item, 'trim_end_ms', 0),
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile.name if profile else "Unknown",
@@ -401,23 +411,23 @@ async def move_story_item(
async def remove_item_from_story(
story_id: str,
generation_id: str,
item_id: str,
db: Session,
) -> bool:
"""
Remove a generation from a story.
Remove a story item from a story.
Args:
story_id: Story ID
generation_id: Generation ID to remove
item_id: Story item ID to remove
db: Database session
Returns:
True if removed, False if not found
"""
item = db.query(DBStoryItem).filter_by(
id=item_id,
story_id=story_id,
generation_id=generation_id
).first()
if not item:
return False
@@ -434,6 +444,277 @@ async def remove_item_from_story(
return True
async def trim_story_item(
story_id: str,
item_id: str,
data: StoryItemTrim,
db: Session,
) -> Optional[StoryItemDetail]:
"""
Trim a story item (update trim_start_ms and trim_end_ms).
Args:
story_id: Story ID
item_id: Story item ID
data: Trim data (trim_start_ms, trim_end_ms)
db: Database session
Returns:
Updated item detail or None if not found
"""
# Get the item
item = db.query(DBStoryItem).filter_by(
id=item_id,
story_id=story_id,
).first()
if not item:
return None
# Get the generation
generation = db.query(DBGeneration).filter_by(id=item.generation_id).first()
if not generation:
return None
# Validate trim values don't exceed duration
max_duration_ms = int(generation.duration * 1000)
if data.trim_start_ms + data.trim_end_ms >= max_duration_ms:
return None # Invalid trim - would result in zero or negative duration
# Update trim values
item.trim_start_ms = data.trim_start_ms
item.trim_end_ms = data.trim_end_ms
# Update story updated_at
story = db.query(DBStory).filter_by(id=story_id).first()
if story:
story.updated_at = datetime.utcnow()
db.commit()
db.refresh(item)
# Get profile name
profile = db.query(DBVoiceProfile).filter_by(id=generation.profile_id).first()
return StoryItemDetail(
id=item.id,
story_id=item.story_id,
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=item.trim_start_ms,
trim_end_ms=item.trim_end_ms,
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile.name if profile else "Unknown",
text=generation.text,
language=generation.language,
audio_path=generation.audio_path,
duration=generation.duration,
seed=generation.seed,
instruct=generation.instruct,
generation_created_at=generation.created_at,
)
async def split_story_item(
story_id: str,
item_id: str,
data: StoryItemSplit,
db: Session,
) -> Optional[List[StoryItemDetail]]:
"""
Split a story item at a given time, creating two clips.
Args:
story_id: Story ID
item_id: Story item ID to split
data: Split data (split_time_ms - time within clip to split at)
db: Database session
Returns:
List of two updated item details (original and new) or None if not found/invalid
"""
# Get the item
item = db.query(DBStoryItem).filter_by(
id=item_id,
story_id=story_id,
).first()
if not item:
return None
# Get the generation
generation = db.query(DBGeneration).filter_by(id=item.generation_id).first()
if not generation:
return None
# Calculate effective duration and validate split point
current_trim_start = getattr(item, 'trim_start_ms', 0)
current_trim_end = getattr(item, 'trim_end_ms', 0)
original_duration_ms = int(generation.duration * 1000)
effective_duration_ms = original_duration_ms - current_trim_start - current_trim_end
# Validate split_time_ms is within the effective duration
if data.split_time_ms <= 0 or data.split_time_ms >= effective_duration_ms:
return None # Invalid split point
# Calculate the absolute time in the original audio where we're splitting
absolute_split_ms = current_trim_start + data.split_time_ms
# Update original clip: trim from the end
item.trim_end_ms = original_duration_ms - absolute_split_ms
# Create new clip: starts after the split, trimmed from the start
new_item = DBStoryItem(
id=str(uuid.uuid4()),
story_id=story_id,
generation_id=item.generation_id, # Same generation, different trim
start_time_ms=item.start_time_ms + data.split_time_ms,
track=item.track,
trim_start_ms=absolute_split_ms,
trim_end_ms=current_trim_end,
created_at=datetime.utcnow(),
)
db.add(new_item)
# Update story updated_at
story = db.query(DBStory).filter_by(id=story_id).first()
if story:
story.updated_at = datetime.utcnow()
db.commit()
db.refresh(item)
db.refresh(new_item)
# Get profile name
profile = db.query(DBVoiceProfile).filter_by(id=generation.profile_id).first()
profile_name = profile.name if profile else "Unknown"
# Build response items
original_item_detail = StoryItemDetail(
id=item.id,
story_id=item.story_id,
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=item.trim_start_ms,
trim_end_ms=item.trim_end_ms,
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile_name,
text=generation.text,
language=generation.language,
audio_path=generation.audio_path,
duration=generation.duration,
seed=generation.seed,
instruct=generation.instruct,
generation_created_at=generation.created_at,
)
new_item_detail = StoryItemDetail(
id=new_item.id,
story_id=new_item.story_id,
generation_id=new_item.generation_id,
start_time_ms=new_item.start_time_ms,
track=new_item.track,
trim_start_ms=new_item.trim_start_ms,
trim_end_ms=new_item.trim_end_ms,
created_at=new_item.created_at,
profile_id=generation.profile_id,
profile_name=profile_name,
text=generation.text,
language=generation.language,
audio_path=generation.audio_path,
duration=generation.duration,
seed=generation.seed,
instruct=generation.instruct,
generation_created_at=generation.created_at,
)
return [original_item_detail, new_item_detail]
async def duplicate_story_item(
story_id: str,
item_id: str,
db: Session,
) -> Optional[StoryItemDetail]:
"""
Duplicate a story item, creating a copy with all properties.
Args:
story_id: Story ID
item_id: Story item ID to duplicate
db: Database session
Returns:
New item detail or None if not found
"""
# Get the original item
original_item = db.query(DBStoryItem).filter_by(
id=item_id,
story_id=story_id,
).first()
if not original_item:
return None
# Get the generation
generation = db.query(DBGeneration).filter_by(id=original_item.generation_id).first()
if not generation:
return None
# Calculate effective duration
current_trim_start = getattr(original_item, 'trim_start_ms', 0)
current_trim_end = getattr(original_item, 'trim_end_ms', 0)
original_duration_ms = int(generation.duration * 1000)
effective_duration_ms = original_duration_ms - current_trim_start - current_trim_end
# Create duplicate item - place it right after the original
new_item = DBStoryItem(
id=str(uuid.uuid4()),
story_id=story_id,
generation_id=original_item.generation_id, # Same generation as original
start_time_ms=original_item.start_time_ms + effective_duration_ms + 200, # 200ms gap
track=original_item.track,
trim_start_ms=current_trim_start,
trim_end_ms=current_trim_end,
created_at=datetime.utcnow(),
)
db.add(new_item)
# Update story updated_at
story = db.query(DBStory).filter_by(id=story_id).first()
if story:
story.updated_at = datetime.utcnow()
db.commit()
db.refresh(new_item)
# Get profile name
profile = db.query(DBVoiceProfile).filter_by(id=generation.profile_id).first()
return StoryItemDetail(
id=new_item.id,
story_id=new_item.story_id,
generation_id=new_item.generation_id,
start_time_ms=new_item.start_time_ms,
track=new_item.track,
trim_start_ms=new_item.trim_start_ms,
trim_end_ms=new_item.trim_end_ms,
created_at=new_item.created_at,
profile_id=generation.profile_id,
profile_name=profile.name if profile else "Unknown",
text=generation.text,
language=generation.language,
audio_path=generation.audio_path,
duration=generation.duration,
seed=generation.seed,
instruct=generation.instruct,
generation_created_at=generation.created_at,
)
async def update_story_item_times(
story_id: str,
data: StoryItemBatchUpdate,
@@ -538,6 +819,8 @@ async def reorder_story_items(
generation_id=item.generation_id,
start_time_ms=item.start_time_ms,
track=item.track,
trim_start_ms=getattr(item, 'trim_start_ms', 0),
trim_end_ms=getattr(item, 'trim_end_ms', 0),
created_at=item.created_at,
profile_id=generation.profile_id,
profile_name=profile_name,
@@ -602,14 +885,31 @@ async def export_story_audio(
audio, sr = load_audio(str(audio_path), sample_rate=sample_rate)
sample_rate = sr # Use actual sample rate from first file
# Get trim values
trim_start_ms = getattr(item, 'trim_start_ms', 0)
trim_end_ms = getattr(item, 'trim_end_ms', 0)
# Calculate effective duration
original_duration_ms = int(generation.duration * 1000)
effective_duration_ms = original_duration_ms - trim_start_ms - trim_end_ms
# Slice audio based on trim values
trim_start_sample = int((trim_start_ms / 1000.0) * sample_rate)
trim_end_sample = int((trim_end_ms / 1000.0) * sample_rate)
# Extract the trimmed portion
if trim_end_ms > 0:
trimmed_audio = audio[trim_start_sample:-trim_end_sample] if trim_end_sample > 0 else audio[trim_start_sample:]
else:
trimmed_audio = audio[trim_start_sample:]
# Store audio with its timecode info
start_time_ms = item.start_time_ms
duration_ms = int(generation.duration * 1000)
audio_data.append({
'audio': audio,
'audio': trimmed_audio,
'start_time_ms': start_time_ms,
'duration_ms': duration_ms,
'duration_ms': effective_duration_ms,
})
except Exception:
# Skip files that can't be loaded
+12 -264
View File
@@ -1,274 +1,22 @@
"""
Whisper ASR module for transcription.
STT (Speech-to-Text) module - delegates to backend abstraction layer.
"""
from typing import Optional, List, Dict
import asyncio
import torch
import numpy as np
from pathlib import Path
from .utils.progress import get_progress_manager
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from .utils.tasks import get_task_manager
from typing import Optional
from .backends import get_stt_backend, STTBackend
class WhisperModel:
"""Manages Whisper model loading and transcription."""
def get_whisper_model() -> STTBackend:
"""
Get STT backend instance (MLX or PyTorch based on platform).
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
def _get_device(self) -> str:
"""Get the best available device."""
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
# MPS support for Whisper
return "cpu" # Use CPU for stability
return "cpu"
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
def load_model(self, model_size: Optional[str] = None):
"""
Lazy load the Whisper model.
Args:
model_size: Model size (tiny, base, small, medium, large)
"""
if model_size is None:
model_size = self.model_size
if self.model is not None and self.model_size == model_size:
return
try:
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
# Set up progress tracking
progress_manager = get_progress_manager()
progress_model_name = f"whisper-{model_size}"
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(progress_model_name)
print(f"Loading Whisper model {model_size} on {self.device}...")
# Initialize progress state to show download has started
progress_manager.update_progress(
model_name=progress_model_name,
current=0,
total=1, # Set to 1 initially, will be updated by callback
filename="",
status="downloading",
)
# Set up progress callback
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Use progress tracker during download
with tracker.patch_download():
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
self.model_size = model_size
# Mark as complete
progress_manager.mark_complete(progress_model_name)
task_manager.complete_download(progress_model_name)
print(f"Whisper model {model_size} loaded successfully")
except Exception as e:
print(f"Error loading Whisper model: {e}")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
progress_model_name = f"whisper-{model_size}"
progress_manager.mark_error(progress_model_name, str(e))
task_manager.error_download(progress_model_name, str(e))
raise
async def load_model_async(self, model_size: Optional[str] = None):
"""
Async version of load_model that runs in thread pool.
This prevents blocking the event loop during model loading.
"""
if model_size is None:
model_size = self.model_size
# If already loaded with correct size, return immediately
if self.model is not None and self.model_size == model_size:
return
# Run the blocking load operation in a thread pool
await asyncio.to_thread(self.load_model, model_size)
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
print("Whisper model unloaded")
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
"""
Transcribe audio to text.
Args:
audio_path: Path to audio file
language: Optional language hint (en or zh)
Returns:
Transcribed text
"""
await self.load_model_async()
from .utils.audio import load_audio
def _transcribe_sync():
"""Run synchronous transcription in thread pool."""
# Load audio
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language if provided
forced_decoder_ids = None
if language:
lang_code = "en" if language == "en" else "zh"
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=lang_code,
task="transcribe",
)
# Generate transcription
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
# Run blocking transcription in thread pool
return await asyncio.to_thread(_transcribe_sync)
async def transcribe_with_timestamps(
self,
audio_path: str,
language: Optional[str] = None,
) -> List[Dict[str, any]]:
"""
Transcribe audio with word-level timestamps.
Args:
audio_path: Path to audio file
language: Optional language hint
Returns:
List of word segments with timestamps
"""
await self.load_model_async()
from .utils.audio import load_audio
def _transcribe_timestamps_sync():
"""Run synchronous transcription with timestamps in thread pool."""
# Load audio
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language if provided
forced_decoder_ids = None
if language:
lang_code = "en" if language == "en" else "zh"
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=lang_code,
task="transcribe",
)
# Generate with timestamps
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
return_timestamps=True,
)
# Parse timestamps (simplified - would need more robust parsing)
# For now, return basic transcription
# TODO: Implement proper timestamp parsing
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return [
{
"text": transcription,
"start": 0.0,
"end": len(audio) / sr,
}
]
# Run blocking transcription in thread pool
return await asyncio.to_thread(_transcribe_timestamps_sync)
# Global model instance
_whisper_model: Optional[WhisperModel] = None
def get_whisper_model() -> WhisperModel:
"""Get or create Whisper model instance."""
global _whisper_model
if _whisper_model is None:
_whisper_model = WhisperModel()
return _whisper_model
Returns:
STT backend instance
"""
return get_stt_backend()
def unload_whisper_model():
"""Unload Whisper model to free memory."""
global _whisper_model
if _whisper_model is not None:
_whisper_model.unload_model()
backend = get_stt_backend()
backend.unload_model()
+20 -355
View File
@@ -1,372 +1,37 @@
"""
TTS inference module using Qwen3-TTS.
TTS inference module - delegates to backend abstraction layer.
"""
from typing import Optional, List, Tuple
import asyncio
import torch
from typing import Optional
import numpy as np
import io
import soundfile as sf
from pathlib import Path
from .utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from .utils.audio import normalize_audio
from .utils.progress import get_progress_manager
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from .utils.tasks import get_task_manager
from . import config
from .backends import get_tts_backend, TTSBackend
class TTSModel:
"""Manages Qwen3-TTS model loading and inference."""
def get_tts_model() -> TTSBackend:
"""
Get TTS backend instance (MLX or PyTorch based on platform).
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device()
self._current_model_size = None
def _get_device(self) -> str:
"""Get the best available device."""
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
# MPS can have issues, use CPU for stability
return "cpu"
return "cpu"
def is_loaded(self) -> bool:
"""Check if model is loaded."""
return self.model is not None
def _get_model_path(self, model_size: str) -> str:
"""
Get the model path, downloading from HuggingFace Hub if needed.
Args:
model_size: Model size (1.7B or 0.6B)
Returns:
Path to model (either local or HuggingFace Hub ID)
"""
# HuggingFace Hub model IDs
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
# Local directory names (for backwards compatibility)
local_model_map = {
"1.7B": "Qwen--Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen--Qwen3-TTS-12Hz-0.6B-Base",
}
if model_size not in hf_model_map:
raise ValueError(f"Unknown model size: {model_size}")
# Check if model exists locally (backwards compatibility)
local_path = config.get_models_dir() / local_model_map[model_size]
if local_path.exists():
print(f"Found local model at {local_path}")
return str(local_path)
# Use HuggingFace Hub model ID (will auto-download)
hf_model_id = hf_model_map[model_size]
print(f"Will download model from HuggingFace Hub: {hf_model_id}")
return hf_model_id
def load_model(self, model_size: Optional[str] = None):
"""
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
The model will be automatically downloaded on first use and cached locally.
This works similar to how Whisper models are loaded.
Args:
model_size: Model size to load (1.7B or 0.6B)
"""
if model_size is None:
model_size = self.model_size
# If already loaded with correct size, return
if self.model is not None and self._current_model_size == model_size:
return
# Unload existing model if different size requested
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
try:
from qwen_tts import Qwen3TTSModel
# Get model path (local or HuggingFace Hub ID)
model_path = self._get_model_path(model_size)
# Set up progress tracking
progress_manager = get_progress_manager()
model_name = f"qwen-tts-{model_size}"
# Check if model is being downloaded from HuggingFace Hub
if model_path.startswith("Qwen/"):
print(f"Loading TTS model {model_size} on {self.device}...")
# Start tracking download task
task_manager = get_task_manager()
task_manager.start_download(model_name)
# Initialize progress state to show download has started
progress_manager.update_progress(
model_name=model_name,
current=0,
total=1, # Set to 1 initially, will be updated by callback
filename="",
status="downloading",
)
# Set up progress callback
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback)
# Use progress tracker during download
with tracker.patch_download():
# Load the model - downloads will happen automatically with progress tracking
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
)
# Mark as complete
progress_manager.mark_complete(model_name)
task_manager.complete_download(model_name)
else:
# Local model, no download needed
print(f"Loading TTS model {model_size} on {self.device}...")
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
)
self._current_model_size = model_size
self.model_size = model_size
print(f"TTS model {model_size} loaded successfully")
except ImportError as e:
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
except Exception as e:
print(f"Error loading TTS model: {e}")
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = f"qwen-tts-{model_size}"
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
async def load_model_async(self, model_size: Optional[str] = None):
"""
Async version of load_model that runs in thread pool.
This prevents blocking the event loop during model loading.
"""
if model_size is None:
model_size = self.model_size
# If already loaded with correct size, return immediately
if self.model is not None and self._current_model_size == model_size:
return
# Run the blocking load operation in a thread pool
await asyncio.to_thread(self.load_model, model_size)
def unload_model(self):
"""Unload the model to free memory."""
if self.model is not None:
del self.model
self.model = None
self._current_model_size = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
print("TTS model unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
Args:
audio_path: Path to reference audio file
reference_text: Transcript of reference audio
use_cache: Whether to use cached prompt if available
Returns:
Tuple of (voice_prompt_dict, was_cached)
"""
await self.load_model_async()
# Check cache if enabled
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached_prompt = get_cached_voice_prompt(cache_key)
if cached_prompt is not None:
return cached_prompt, True
def _create_prompt_sync():
"""Run synchronous voice prompt creation in thread pool."""
return self.model.create_voice_clone_prompt(
ref_audio=str(audio_path),
ref_text=reference_text,
x_vector_only_mode=False,
)
# Run blocking operation in thread pool
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
# Cache if enabled
if use_cache:
cache_voice_prompt(cache_key, voice_prompt_items)
return voice_prompt_items, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""
Combine multiple reference samples for better quality.
Args:
audio_paths: List of audio file paths
reference_texts: List of reference texts
Returns:
Tuple of (combined_audio, combined_text)
"""
from .utils.audio import load_audio
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate audio
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text using voice prompt.
Args:
text: Text to synthesize
voice_prompt: Voice prompt dictionary from create_voice_prompt
language: Language code (en or zh)
seed: Random seed for reproducibility
instruct: Natural language instruction for speech delivery control
Returns:
Tuple of (audio_array, sample_rate)
"""
# Load model (already handles async via to_thread if needed)
await self.load_model_async()
def _generate_sync():
"""Run synchronous generation in thread pool."""
# Set seed if provided
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
# Generate audio - this is the blocking operation
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct,
)
return wavs[0], sample_rate
# Run blocking inference in thread pool to avoid blocking event loop
audio, sample_rate = await asyncio.to_thread(_generate_sync)
return audio, sample_rate
async def generate_from_reference(
self,
text: str,
audio_path: str,
reference_text: str,
language: str = "en",
seed: Optional[int] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio directly from reference (convenience method).
Args:
text: Text to synthesize
audio_path: Path to reference audio
reference_text: Transcript of reference audio
language: Language code
seed: Random seed
Returns:
Tuple of (audio_array, sample_rate)
"""
# Create voice prompt (with caching)
voice_prompt, _ = await self.create_voice_prompt(audio_path, reference_text)
# Generate
return await self.generate(text, voice_prompt, language, seed)
# Global model instance
_tts_model: Optional[TTSModel] = None
def get_tts_model() -> TTSModel:
"""Get or create TTS model instance."""
global _tts_model
if _tts_model is None:
_tts_model = TTSModel()
return _tts_model
Returns:
TTS backend instance
"""
return get_tts_backend()
def unload_tts_model():
"""Unload TTS model to free memory."""
global _tts_model
if _tts_model is not None:
_tts_model.unload_model()
backend = get_tts_backend()
backend.unload_model()
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
"""Convert audio array to WAV bytes."""
buffer = io.BytesIO()
sf.write(buffer, audio, sample_rate, format="WAV")
buffer.seek(0)
return buffer.read()
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
+8 -8
View File
@@ -5,7 +5,7 @@ Voice prompt caching utilities.
import hashlib
import torch
from pathlib import Path
from typing import Optional
from typing import Optional, Union, Dict, Any
from .. import config
@@ -15,8 +15,8 @@ def _get_cache_dir() -> Path:
return config.get_cache_dir()
# In-memory cache
_memory_cache: dict[str, torch.Tensor] = {}
# In-memory cache - can store dict (voice prompt) or tensor (legacy)
_memory_cache: dict[str, Union[torch.Tensor, Dict[str, Any]]] = {}
def get_cache_key(audio_path: str, reference_text: str) -> str:
@@ -43,7 +43,7 @@ def get_cache_key(audio_path: str, reference_text: str) -> str:
def get_cached_voice_prompt(
cache_key: str,
) -> Optional[torch.Tensor]:
) -> Optional[Union[torch.Tensor, Dict[str, Any]]]:
"""
Get cached voice prompt if available.
@@ -51,7 +51,7 @@ def get_cached_voice_prompt(
cache_key: Cache key
Returns:
Cached voice prompt tensor or None
Cached voice prompt (dict or tensor) or None
"""
# Check in-memory cache
if cache_key in _memory_cache:
@@ -73,18 +73,18 @@ def get_cached_voice_prompt(
def cache_voice_prompt(
cache_key: str,
voice_prompt: torch.Tensor,
voice_prompt: Union[torch.Tensor, Dict[str, Any]],
) -> None:
"""
Cache voice prompt to memory and disk.
Args:
cache_key: Cache key
voice_prompt: Voice prompt tensor
voice_prompt: Voice prompt (dict or tensor)
"""
# Store in memory
_memory_cache[cache_key] = voice_prompt
# Store on disk
# Store on disk (torch.save can handle both dicts and tensors)
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
torch.save(voice_prompt, cache_file)
+16 -5
View File
@@ -29,8 +29,9 @@ class HFProgressTracker:
class TrackedTqdm(original_tqdm):
"""A tqdm subclass that reports progress to our tracker."""
def __init__(self, *args, **kwargs):
print(f"[DEBUG TrackedTqdm] __init__ called with desc: {kwargs.get('desc', '')}")
# Extract filename from desc before passing to parent
desc = kwargs.get("desc", "")
if not desc and args:
@@ -79,8 +80,9 @@ class HFProgressTracker:
}
def update(self, n=1):
print(f"[DEBUG TrackedTqdm] update called with n={n}")
result = super().update(n)
# Report progress
with tracker._lock:
if id(self) in tracker._active_tqdms:
@@ -118,11 +120,13 @@ class HFProgressTracker:
@contextmanager
def patch_download(self):
"""Context manager to patch tqdm for progress tracking."""
print("[DEBUG HFProgressTracker] patch_download called")
try:
import tqdm as tqdm_module
# Store original tqdm class
self._original_tqdm_class = tqdm_module.tqdm
print(f"[DEBUG HFProgressTracker] Original tqdm class: {self._original_tqdm_class}")
# Reset totals
with self._lock:
@@ -135,18 +139,22 @@ class HFProgressTracker:
# Create our tracked tqdm class
tracked_tqdm = self._create_tracked_tqdm_class()
print(f"[DEBUG HFProgressTracker] Created TrackedTqdm class: {tracked_tqdm}")
# Patch tqdm.tqdm
tqdm_module.tqdm = tracked_tqdm
print(f"[DEBUG HFProgressTracker] Patched tqdm.tqdm")
# Also patch tqdm.auto.tqdm if it exists (used by huggingface_hub)
self._original_tqdm_auto = None
if hasattr(tqdm_module, "auto") and hasattr(tqdm_module.auto, "tqdm"):
self._original_tqdm_auto = tqdm_module.auto.tqdm
tqdm_module.auto.tqdm = tracked_tqdm
print(f"[DEBUG HFProgressTracker] Patched tqdm.auto.tqdm")
# Patch in sys.modules to catch already-imported references
self._patched_modules = {}
patched_count = 0
for module_name in list(sys.modules.keys()):
if "huggingface" in module_name or module_name.startswith("tqdm"):
try:
@@ -159,8 +167,11 @@ class HFProgressTracker:
):
self._patched_modules[module_name] = attr
setattr(module, "tqdm", tracked_tqdm)
patched_count += 1
print(f"[DEBUG HFProgressTracker] Patched {module_name}.tqdm")
except (AttributeError, TypeError):
pass
print(f"[DEBUG HFProgressTracker] Patched {patched_count} modules in sys.modules")
yield
+114
View File
@@ -0,0 +1,114 @@
"""Image processing utilities for avatar uploads."""
from pathlib import Path
from typing import Optional, Tuple
from PIL import Image
# JPEG can be reported as 'JPEG' or 'MPO' (for multi-picture format from some cameras)
ALLOWED_FORMATS = {'PNG', 'JPEG', 'WEBP', 'MPO', 'JPG'}
MAX_SIZE = 512
MAX_FILE_SIZE = 5 * 1024 * 1024 # 5MB
def validate_image(file_path: str) -> Tuple[bool, Optional[str]]:
"""
Validate image format and file size.
Args:
file_path: Path to image file
Returns:
Tuple of (is_valid, error_message)
"""
path = Path(file_path)
# Check file size
if path.stat().st_size > MAX_FILE_SIZE:
return False, f"File size exceeds maximum of {MAX_FILE_SIZE // (1024 * 1024)}MB"
try:
with Image.open(file_path) as img:
# Verify the image can be loaded
img.load()
# Check format (normalize JPEG variants)
img_format = img.format
if img_format in ('MPO', 'JPG'):
img_format = 'JPEG'
if img_format not in {'PNG', 'JPEG', 'WEBP'}:
return False, f"Invalid format '{img_format}'. Allowed formats: PNG, JPEG, WEBP"
return True, None
except Exception as e:
return False, f"Invalid image file: {str(e)}"
def process_avatar(input_path: str, output_path: str, max_size: int = MAX_SIZE) -> None:
"""
Process avatar image: resize and optimize.
Resizes image to fit within max_size x max_size while maintaining aspect ratio.
Args:
input_path: Path to input image
output_path: Path to save processed image
max_size: Maximum width or height in pixels
"""
with Image.open(input_path) as img:
# Handle EXIF orientation for JPEG images
try:
from PIL import ExifTags
for orientation in ExifTags.TAGS.keys():
if ExifTags.TAGS[orientation] == 'Orientation':
break
exif = img._getexif()
if exif is not None:
orientation_value = exif.get(orientation)
if orientation_value == 3:
img = img.rotate(180, expand=True)
elif orientation_value == 6:
img = img.rotate(270, expand=True)
elif orientation_value == 8:
img = img.rotate(90, expand=True)
except (AttributeError, KeyError, IndexError, TypeError):
# No EXIF data or orientation tag
pass
# Convert to RGB if necessary (handles RGBA, P, CMYK, etc.)
if img.mode not in ('RGB', 'L'):
if img.mode == 'RGBA':
# Create white background for RGBA images
background = Image.new('RGB', img.size, (255, 255, 255))
background.paste(img, mask=img.split()[3]) # Use alpha channel as mask
img = background
elif img.mode == 'CMYK':
# Convert CMYK to RGB
img = img.convert('RGB')
elif img.mode == 'P':
# Convert palette mode to RGB
img = img.convert('RGB')
else:
img = img.convert('RGB')
# Calculate new size maintaining aspect ratio
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
# Determine output format from extension
output_ext = Path(output_path).suffix.lower()
format_map = {
'.png': 'PNG',
'.jpeg': 'JPEG',
'.jpg': 'JPEG',
'.webp': 'WEBP'
}
output_format = format_map.get(output_ext, 'PNG')
# Save with optimization
save_kwargs = {'optimize': True}
if output_format == 'JPEG':
save_kwargs['quality'] = 90
img.save(output_path, format=output_format, **save_kwargs)
+157 -48
View File
@@ -6,16 +6,55 @@ from typing import Optional, Callable, Dict, List
from fastapi.responses import StreamingResponse
import asyncio
import json
import threading
from datetime import datetime
class ProgressManager:
"""Manages download progress for multiple models."""
"""Manages download progress for multiple models.
Thread-safe: can be called from background threads (e.g., via asyncio.to_thread).
"""
def __init__(self):
self._progress: Dict[str, Dict] = {}
self._listeners: Dict[str, list] = {}
self._lock = threading.Lock() # Thread-safe lock for progress dict
self._main_loop: Optional[asyncio.AbstractEventLoop] = None
def _set_main_loop(self, loop: asyncio.AbstractEventLoop):
"""Set the main event loop for thread-safe operations."""
self._main_loop = loop
def _notify_listeners_threadsafe(self, model_name: str, progress_data: Dict):
"""Notify listeners in a thread-safe manner."""
import logging
logger = logging.getLogger(__name__)
if model_name not in self._listeners:
return
for queue in self._listeners[model_name]:
try:
# Check if we're in the main event loop thread
try:
running_loop = asyncio.get_running_loop()
# We're in an async context, can use put_nowait directly
queue.put_nowait(progress_data.copy())
except RuntimeError:
# Not in async context (running in background thread)
# Use call_soon_threadsafe to safely put on queue
if self._main_loop and self._main_loop.is_running():
self._main_loop.call_soon_threadsafe(
lambda q=queue, d=progress_data.copy(): q.put_nowait(d) if not q.full() else None
)
else:
logger.debug(f"No main loop available for {model_name}, skipping notification")
except asyncio.QueueFull:
logger.warning(f"Queue full for {model_name}, dropping update")
except Exception as e:
logger.warning(f"Error notifying listener for {model_name}: {e}")
def update_progress(
self,
model_name: str,
@@ -26,7 +65,9 @@ class ProgressManager:
):
"""
Update progress for a model download.
Thread-safe: can be called from background threads.
Args:
model_name: Name of the model (e.g., "qwen-tts-1.7B", "whisper-base")
current: Current bytes downloaded
@@ -34,9 +75,12 @@ class ProgressManager:
filename: Current file being downloaded
status: Status string (downloading, extracting, complete, error)
"""
import logging
logger = logging.getLogger(__name__)
progress_pct = (current / total * 100) if total > 0 else 0
self._progress[model_name] = {
progress_data = {
"model_name": model_name,
"current": current,
"total": total,
@@ -45,26 +89,43 @@ class ProgressManager:
"status": status,
"timestamp": datetime.now().isoformat(),
}
# Notify all listeners
if model_name in self._listeners:
for queue in self._listeners[model_name]:
try:
queue.put_nowait(self._progress[model_name].copy())
except asyncio.QueueFull:
pass
print(f"[DEBUG] update_progress called: {model_name}, {progress_pct:.1f}%")
# Thread-safe update of progress dict
with self._lock:
self._progress[model_name] = progress_data
# Notify all listeners (thread-safe)
listener_count = len(self._listeners.get(model_name, []))
print(f"[DEBUG] Listener count for {model_name}: {listener_count}")
print(f"[DEBUG] All listeners: {list(self._listeners.keys())}")
print(f"[DEBUG] Main loop set: {self._main_loop is not None}")
if self._main_loop:
print(f"[DEBUG] Main loop running: {self._main_loop.is_running()}")
if listener_count > 0:
logger.debug(f"Notifying {listener_count} listeners for {model_name}: {progress_pct:.1f}% ({filename})")
print(f"[DEBUG] About to notify listeners...")
self._notify_listeners_threadsafe(model_name, progress_data)
print(f"[DEBUG] Notified listeners")
else:
logger.debug(f"No listeners for {model_name}, progress update stored: {progress_pct:.1f}%")
def get_progress(self, model_name: str) -> Optional[Dict]:
"""Get current progress for a model."""
return self._progress.get(model_name)
"""Get current progress for a model. Thread-safe."""
with self._lock:
progress = self._progress.get(model_name)
return progress.copy() if progress else None
def get_all_active(self) -> List[Dict]:
"""Get all active downloads (status is 'downloading' or 'extracting')."""
"""Get all active downloads (status is 'downloading' or 'extracting'). Thread-safe."""
active = []
for model_name, progress in self._progress.items():
status = progress.get("status", "")
if status in ("downloading", "extracting"):
active.append(progress.copy())
with self._lock:
for model_name, progress in self._progress.items():
status = progress.get("status", "")
if status in ("downloading", "extracting"):
active.append(progress.copy())
return active
def create_progress_callback(self, model_name: str, filename: Optional[str] = None):
@@ -98,30 +159,57 @@ class ProgressManager:
async def subscribe(self, model_name: str):
"""
Subscribe to progress updates for a model.
Yields progress updates as Server-Sent Events.
"""
queue = asyncio.Queue(maxsize=10)
import logging
logger = logging.getLogger(__name__)
# Store the main event loop for thread-safe operations
try:
self._main_loop = asyncio.get_running_loop()
except RuntimeError:
pass
queue = asyncio.Queue(maxsize=10)
# Add to listeners
if model_name not in self._listeners:
self._listeners[model_name] = []
self._listeners[model_name].append(queue)
logger.info(f"SSE client subscribed to {model_name}, total listeners: {len(self._listeners[model_name])}")
try:
# Send initial progress if available
if model_name in self._progress:
yield f"data: {json.dumps(self._progress[model_name])}\n\n"
# Send initial progress if available and still in progress (thread-safe read)
with self._lock:
initial_progress = self._progress.get(model_name)
if initial_progress:
initial_progress = initial_progress.copy()
if initial_progress:
status = initial_progress.get('status')
# Only send initial progress if download is actually in progress
# Don't send old 'complete' or 'error' status from previous downloads
if status in ('downloading', 'extracting'):
logger.info(f"Sending initial progress for {model_name}: {status}")
yield f"data: {json.dumps(initial_progress)}\n\n"
else:
logger.info(f"Skipping initial progress for {model_name} (status: {status})")
else:
logger.info(f"No initial progress available for {model_name}")
# Stream updates
while True:
try:
# Wait for update with timeout
progress = await asyncio.wait_for(queue.get(), timeout=1.0)
logger.debug(f"Sending progress update for {model_name}: {progress.get('status')} - {progress.get('progress', 0):.1f}%")
yield f"data: {json.dumps(progress)}\n\n"
# Stop if complete or error
if progress.get("status") in ("complete", "error"):
logger.info(f"Download {progress.get('status')} for {model_name}, closing SSE connection")
break
except asyncio.TimeoutError:
# Send heartbeat
@@ -133,32 +221,53 @@ class ProgressManager:
self._listeners[model_name].remove(queue)
if not self._listeners[model_name]:
del self._listeners[model_name]
logger.info(f"SSE client unsubscribed from {model_name}, remaining listeners: {len(self._listeners.get(model_name, []))}")
def mark_complete(self, model_name: str):
"""Mark a model download as complete."""
if model_name in self._progress:
self._progress[model_name]["status"] = "complete"
self._progress[model_name]["progress"] = 100.0
# Notify listeners
if model_name in self._listeners:
for queue in self._listeners[model_name]:
try:
queue.put_nowait(self._progress[model_name].copy())
except asyncio.QueueFull:
pass
"""Mark a model download as complete. Thread-safe."""
import logging
logger = logging.getLogger(__name__)
with self._lock:
if model_name in self._progress:
self._progress[model_name]["status"] = "complete"
self._progress[model_name]["progress"] = 100.0
progress_data = self._progress[model_name].copy()
else:
logger.warning(f"Cannot mark {model_name} as complete: not found in progress")
return
logger.info(f"Marked {model_name} as complete")
# Notify listeners (thread-safe)
self._notify_listeners_threadsafe(model_name, progress_data)
def mark_error(self, model_name: str, error: str):
"""Mark a model download as failed."""
if model_name in self._progress:
self._progress[model_name]["status"] = "error"
self._progress[model_name]["error"] = error
# Notify listeners
if model_name in self._listeners:
for queue in self._listeners[model_name]:
try:
queue.put_nowait(self._progress[model_name].copy())
except asyncio.QueueFull:
pass
"""Mark a model download as failed. Thread-safe."""
import logging
logger = logging.getLogger(__name__)
with self._lock:
if model_name in self._progress:
self._progress[model_name]["status"] = "error"
self._progress[model_name]["error"] = error
progress_data = self._progress[model_name].copy()
else:
# Create new progress entry for error
progress_data = {
"model_name": model_name,
"current": 0,
"total": 0,
"progress": 0,
"filename": None,
"status": "error",
"error": error,
"timestamp": datetime.now().isoformat(),
}
self._progress[model_name] = progress_data
logger.error(f"Marked {model_name} as error: {error}")
# Notify listeners (thread-safe)
self._notify_listeners_threadsafe(model_name, progress_data)
# Global progress manager instance
+6 -2
View File
@@ -4,16 +4,20 @@ from PyInstaller.utils.hooks import collect_submodules
from PyInstaller.utils.hooks import copy_metadata
datas = []
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern']
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += collect_data_files('qwen_tts')
datas += collect_data_files('mlx')
datas += collect_data_files('mlx_audio')
datas += copy_metadata('qwen-tts')
hiddenimports += collect_submodules('qwen_tts')
hiddenimports += collect_submodules('jaraco')
hiddenimports += collect_submodules('mlx')
hiddenimports += collect_submodules('mlx_audio')
a = Analysis(
['server.py'],
pathex=['C:\\Users\\ijame\\Projects\\voice\\Qwen3-TTS'],
pathex=[],
binaries=[],
datas=datas,
hiddenimports=hiddenimports,
+9 -4
View File
@@ -13,7 +13,7 @@
},
"app": {
"name": "@voicebox/app",
"version": "0.1.5",
"version": "0.1.9",
"dependencies": {
"@dnd-kit/core": "^6.3.1",
"@dnd-kit/sortable": "^10.0.0",
@@ -50,6 +50,7 @@
"react": "^18.3.0",
"react-dom": "^18.3.0",
"react-hook-form": "^7.53.0",
"react-sound-visualizer": "^1.4.0",
"tailwind-merge": "^2.5.4",
"wavesurfer.js": "^7.0.0",
"zod": "^3.23.8",
@@ -67,7 +68,7 @@
},
"landing": {
"name": "@voicebox/landing",
"version": "0.1.5",
"version": "0.1.9",
"dependencies": {
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slot": "^1.2.4",
@@ -92,7 +93,7 @@
},
"tauri": {
"name": "@voicebox/tauri",
"version": "0.1.5",
"version": "0.1.9",
"dependencies": {
"@tauri-apps/api": "^2.0.0",
"@tauri-apps/plugin-shell": "^2.0.0",
@@ -111,7 +112,7 @@
},
"web": {
"name": "@voicebox/web",
"version": "0.1.5",
"version": "0.1.9",
"dependencies": {
"@tanstack/react-query": "^5.0.0",
"react": "^18.3.0",
@@ -970,6 +971,8 @@
"react-remove-scroll-bar": ["[email protected]", "", { "dependencies": { "react-style-singleton": "^2.2.2", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" }, "optionalPeers": ["@types/react"] }, "sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q=="],
"react-sound-visualizer": ["[email protected]", "", { "dependencies": { "sound-visualizer": "^1.2.0" }, "peerDependencies": { "react": ">= 16" } }, "sha512-Qe7tFTd1owtQ8nYrUYXg7QLt8mw7iUy86mqj/+IwmXzSw+NlhnMnAGPuisb1Lk3ncliFnM+AQbZb3C4RQN9uMQ=="],
"react-style-singleton": ["[email protected]", "", { "dependencies": { "get-nonce": "^1.0.0", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-b6jSvxvVnyptAiLjbkWLE/lOnR4lfTtDAl+eUC7RZy+QQWc6wRzIV2CE6xBuMmDxc2qIihtDCZD5NPOFl7fRBQ=="],
"read-cache": ["[email protected]", "", { "dependencies": { "pify": "^2.3.0" } }, "sha512-Owdv/Ft7IjOgm/i0xvNDZ1LrRANRfew4b2prF3OWMQLxLfu3bS8FVhCsrSCMK4lR56Y9ya+AThoTpDCTxCmpRA=="],
@@ -1004,6 +1007,8 @@
"slash": ["[email protected]", "", {}, "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q=="],
"sound-visualizer": ["[email protected]", "", {}, "sha512-2+Un0PrrBgXylnCjrVYUoRW7KEDH29h7O8/MGzeDOgFGBPb9oX/2n/RGBxJXvVv2U3KFwX5olUWeJKf0Rr5TLQ=="],
"source-map-js": ["[email protected]", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="],
+3
View File
@@ -0,0 +1,3 @@
node_modules
.mintlify
.DS_Store
+64
View File
@@ -0,0 +1,64 @@
# Voicebox Documentation
This directory contains the documentation for Voicebox, built with [Mintlify](https://mintlify.com).
## Development
### Prerequisites
Install Mintlify globally using bun:
```bash
bun add -g mintlify
```
Or use the helper script:
```bash
bun run install:mintlify
```
### Running Locally
```bash
bun run dev
```
This will start the Mintlify dev server.
The docs will be available at `http://localhost:3000`
### Structure
```
docs/
├── mint.json # Mintlify configuration
├── custom.css # Custom styles
├── overview/ # Getting started & feature docs
├── guides/ # User guides
├── api/ # API reference
├── development/ # Developer documentation
├── logo/ # Logo assets
└── public/ # Static assets
```
### Writing Docs
- Use `.mdx` files for all documentation pages
- Follow the existing structure in `mint.json` for navigation
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
## Deployment
Docs are automatically deployed when changes are pushed to the main branch.
To manually deploy:
```bash
mintlify deploy
```
## Contributing
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
+32 -4
View File
@@ -90,6 +90,26 @@ chmod +x voicebox-*.AppImage
- Slower but works without GPU
- Backend automatically falls back to CPU
### MLX "Failed to load the default metallib" error (Apple Silicon)
**Symptoms:** Generation fails with "library not found" or "metallib" errors
**Solutions:**
1. **Rebuild server binary**
```bash
bun run build:server
```
The build script should automatically include MLX Metal shader libraries.
2. **Check MLX installation**
```bash
pip install -r backend/requirements-mlx.txt
```
3. **Verify backend detection**
- Check server logs for "Backend: MLX"
- If showing "Backend: PYTORCH", MLX may not be installed correctly
### Audio playback issues
**Symptoms:** Generated audio won't play
@@ -111,19 +131,27 @@ chmod +x voicebox-*.AppImage
**Symptoms:** Generation takes >30 seconds
**Solutions:**
1. **Use GPU** (if available)
1. **Check backend type** (Apple Silicon)
- Check Settings → Server Status
- Should show "Backend: MLX" on Apple Silicon
- If showing "Backend: PYTORCH", install MLX: `pip install -r backend/requirements-mlx.txt`
- MLX provides 4-5x faster inference on Apple Silicon
2. **Use GPU** (if available)
- Check Settings → Server Status
- Should show "GPU available: true"
- Apple Silicon: Should show "Metal (Apple Silicon via MLX)"
- Windows/Linux: Should show "CUDA" if GPU available
2. **Enable caching**
3. **Enable caching**
- Voice prompts are cached automatically
- Second generation with same voice should be faster
3. **Use smaller model**
4. **Use smaller model**
- 0.6B model is faster than 1.7B
- Quality difference is minimal for most voices
4. **Check system resources**
5. **Check system resources**
- Close other CPU/GPU intensive apps
- Ensure adequate RAM (8GB+ recommended)
+55
View File
@@ -0,0 +1,55 @@
---
title: "Authentication"
description: "API authentication and security"
---
## Current Status
<Warning>
Authentication is not currently implemented in Voicebox. The API is intended for local use only.
</Warning>
## Local Usage
For local development and usage:
- API runs on `localhost:17493`
- No authentication required
- Access restricted to local machine
## Future Implementation
Authentication will be added in a future release for:
- Remote deployments
- Multi-user access
- Production environments
Planned authentication methods:
- API keys
- OAuth 2.0
- JWT tokens
## Security Best Practices
Until authentication is implemented:
<CardGroup cols={2}>
<Card title="Use VPN" icon="shield">
Use WireGuard or Tailscale for remote access
</Card>
<Card title="Reverse Proxy" icon="server">
Run behind nginx with basic auth
</Card>
<Card title="Firewall" icon="fire">
Restrict access to trusted IPs only
</Card>
<Card title="Local Only" icon="laptop">
Don't expose to public internet
</Card>
</CardGroup>
## Coming Soon
- API key management
- User accounts
- Rate limiting
- Access control
+119
View File
@@ -0,0 +1,119 @@
---
title: "Generation API"
description: "Generate speech from text"
---
## Generate Speech
```http
POST /generate
```
**Request:**
```json
{
"text": "Hello world",
"profile_id": "abc123",
"language": "en"
}
```
**Response:**
```json
{
"id": "gen123",
"text": "Hello world",
"profile_id": "abc123",
"language": "en",
"audio_url": "/audio/gen123.wav",
"duration": 2.3,
"created_at": "2024-01-29T12:00:00Z"
}
```
## List History
```http
GET /history
```
**Query Parameters:**
- `profile_id` (optional) - Filter by voice profile
- `limit` (optional) - Number of results (default: 50)
- `offset` (optional) - Pagination offset
**Response:**
```json
{
"generations": [
{
"id": "gen123",
"text": "Hello world",
"profile_id": "abc123",
"duration": 2.3,
"created_at": "2024-01-29T12:00:00Z"
}
],
"total": 100
}
```
## Get Generation
```http
GET /history/{id}
```
**Response:**
```json
{
"id": "gen123",
"text": "Hello world",
"profile_id": "abc123",
"language": "en",
"audio_url": "/audio/gen123.wav",
"duration": 2.3,
"created_at": "2024-01-29T12:00:00Z"
}
```
## Delete Generation
```http
DELETE /history/{id}
```
**Response:**
```json
{
"success": true
}
```
## TypeScript Example
```typescript
import { VoiceboxClient } from '@/lib/api'
const client = new VoiceboxClient({
baseUrl: 'http://localhost:17493'
})
// Generate speech
const generation = await client.generate({
text: 'Hello world',
profile_id: 'abc123',
language: 'en'
})
// Get audio URL
const audioUrl = generation.audio_url
// List history
const history = await client.listHistory({
profile_id: 'abc123',
limit: 20
})
```
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
+219
View File
@@ -0,0 +1,219 @@
---
title: "API Overview"
description: "Integrate voice synthesis into your applications with the Voicebox REST API"
---
## Introduction
Voicebox exposes a full REST API that allows you to integrate voice synthesis into your own applications. The API runs on `http://localhost:17493` by default.
<Card title="Interactive API Docs" icon="book" href="http://localhost:17493/docs">
When Voicebox is running, visit the auto-generated API documentation at `http://localhost:17493/docs`
</Card>
## Base URL
```
http://localhost:17493
```
For remote deployments, replace `localhost` with your server's IP or hostname.
## Authentication
<Note>
Currently, the API does not require authentication for local development. Authentication will be added in a future release for production deployments.
</Note>
## Quick Example
Here's a simple example of generating speech:
```bash
# Generate speech
curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \
-d '{
"text": "Hello world",
"profile_id": "abc123",
"language": "en"
}'
```
## API Endpoints
The Voicebox API is organized into several categories:
<CardGroup cols={2}>
<Card title="Voice Profiles" icon="user" href="/api/voice-profiles">
Create, list, update, and delete voice profiles
</Card>
<Card title="Generation" icon="waveform" href="/api/generation">
Generate speech from text using voice profiles
</Card>
<Card title="Recordings" icon="microphone" href="/api/recordings">
Record and transcribe audio
</Card>
<Card title="Stories" icon="film">
Create and manage multi-voice stories (coming soon)
</Card>
</CardGroup>
## Core Endpoints
### Voice Profiles
```http
GET /profiles # List all profiles
POST /profiles # Create a new profile
GET /profiles/{id} # Get profile details
PUT /profiles/{id} # Update a profile
DELETE /profiles/{id} # Delete a profile
POST /profiles/{id}/samples # Add voice sample
```
### Generation
```http
POST /generate # Generate speech
GET /history # List generation history
GET /history/{id} # Get generation details
DELETE /history/{id} # Delete from history
```
### Recordings
```http
POST /recordings # Start recording
POST /recordings/stop # Stop recording
POST /transcribe # Transcribe audio
```
## Response Format
All API responses follow a consistent JSON format:
```json
{
"success": true,
"data": {
// Response data
},
"error": null
}
```
Error responses:
```json
{
"success": false,
"data": null,
"error": {
"message": "Error description",
"code": "ERROR_CODE"
}
}
```
## Data Models
### Voice Profile
```json
{
"id": "abc123",
"name": "John Smith",
"language": "en",
"description": "Professional narrator voice",
"created_at": "2024-01-29T12:00:00Z",
"samples": [
{
"id": "sample123",
"audio_path": "/path/to/sample.wav",
"duration": 15.5
}
]
}
```
### Generation
```json
{
"id": "gen123",
"text": "Hello world",
"profile_id": "abc123",
"language": "en",
"audio_path": "/path/to/output.wav",
"duration": 2.3,
"created_at": "2024-01-29T12:00:00Z"
}
```
## TypeScript Client
Voicebox provides an auto-generated TypeScript client with full type safety:
```typescript
import { VoiceboxClient } from '@/lib/api'
const client = new VoiceboxClient({
baseUrl: 'http://localhost:17493'
})
// Create a profile
const profile = await client.createProfile({
name: 'John Smith',
language: 'en'
})
// Generate speech
const generation = await client.generate({
text: 'Hello world',
profile_id: profile.id,
language: 'en'
})
```
The client is automatically generated from the OpenAPI schema. See [Development Setup](/development/setup#generate-openapi-client) for details.
## Rate Limiting
<Info>
Currently, there are no rate limits for local usage. Rate limiting will be added in a future release for production deployments.
</Info>
## WebSocket Support
<Note>
Real-time streaming generation via WebSockets is planned for a future release.
</Note>
## Use Cases
<CardGroup cols={2}>
<Card title="Game Development" icon="gamepad">
Generate dynamic dialogue for NPCs and characters
</Card>
<Card title="Content Creation" icon="video">
Automate voiceovers for videos and podcasts
</Card>
<Card title="Accessibility" icon="universal-access">
Build text-to-speech tools for visually impaired users
</Card>
<Card title="Voice Assistants" icon="robot">
Create custom voice interfaces
</Card>
</CardGroup>
## Next Steps
<CardGroup cols={2}>
<Card title="Voice Profiles API" icon="user" href="/api/voice-profiles">
Learn how to manage voice profiles
</Card>
<Card title="Generation API" icon="waveform" href="/api/generation">
Generate speech from text
</Card>
</CardGroup>
+95
View File
@@ -0,0 +1,95 @@
---
title: "Recordings API"
description: "Record and transcribe audio"
---
## Start Recording
```http
POST /recordings/start
```
**Request:**
```json
{
"source": "microphone"
}
```
**Response:**
```json
{
"recording_id": "rec123",
"status": "recording"
}
```
## Stop Recording
```http
POST /recordings/stop
```
**Request:**
```json
{
"recording_id": "rec123"
}
```
**Response:**
```json
{
"recording_id": "rec123",
"audio_url": "/audio/rec123.wav",
"duration": 15.5
}
```
## Transcribe Audio
```http
POST /transcribe
```
**Request:** (multipart/form-data)
```
audio: <file>
language: "en" (optional)
```
**Response:**
```json
{
"text": "Transcribed speech text here",
"language": "en",
"duration": 15.5,
"confidence": 0.95
}
```
## TypeScript Example
```typescript
import { VoiceboxClient } from '@/lib/api'
const client = new VoiceboxClient({
baseUrl: 'http://localhost:17493'
})
// Start recording
const recording = await client.startRecording({
source: 'microphone'
})
// ... record audio ...
// Stop recording
const result = await client.stopRecording(recording.id)
// Transcribe
const transcription = await client.transcribe(audioFile, 'en')
console.log(transcription.text)
```
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
+149
View File
@@ -0,0 +1,149 @@
---
title: "Voice Profiles API"
description: "Manage voice profiles programmatically"
---
## Endpoints
### List Profiles
```http
GET /profiles
```
**Response:**
```json
{
"profiles": [
{
"id": "abc123",
"name": "John Smith",
"language": "en",
"description": "Professional narrator",
"created_at": "2024-01-29T12:00:00Z",
"sample_count": 2
}
]
}
```
### Get Profile
```http
GET /profiles/{id}
```
**Response:**
```json
{
"id": "abc123",
"name": "John Smith",
"language": "en",
"description": "Professional narrator",
"created_at": "2024-01-29T12:00:00Z",
"samples": [
{
"id": "sample123",
"duration": 15.5,
"created_at": "2024-01-29T12:00:00Z"
}
]
}
```
### Create Profile
```http
POST /profiles
```
**Request:**
```json
{
"name": "John Smith",
"language": "en",
"description": "Professional narrator"
}
```
**Response:**
```json
{
"id": "abc123",
"name": "John Smith",
"language": "en",
"description": "Professional narrator",
"created_at": "2024-01-29T12:00:00Z"
}
```
### Update Profile
```http
PUT /profiles/{id}
```
**Request:**
```json
{
"name": "Updated Name",
"description": "Updated description"
}
```
### Delete Profile
```http
DELETE /profiles/{id}
```
**Response:**
```json
{
"success": true
}
```
### Add Voice Sample
```http
POST /profiles/{id}/samples
```
**Request:** (multipart/form-data)
```
audio: <file>
```
**Response:**
```json
{
"sample_id": "sample123",
"duration": 15.5
}
```
## TypeScript Example
```typescript
import { VoiceboxClient } from '@/lib/api'
const client = new VoiceboxClient({
baseUrl: 'http://localhost:17493'
})
// Create profile
const profile = await client.createProfile({
name: 'John Smith',
language: 'en',
description: 'Professional narrator'
})
// Add sample
await client.addSample(profile.id, audioFile)
// List all profiles
const profiles = await client.listProfiles()
```
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
+1831
View File
File diff suppressed because it is too large Load Diff
+15
View File
@@ -0,0 +1,15 @@
/* Anchor hover styles */
.nav-anchor:hover {
@apply text-[#BF9E40];
}
/* Icon wrapper on hover */
.nav-anchor:hover div {
background: #BF9E40 !important;
filter: brightness(1) !important;
}
/* Icon SVG on hover */
.nav-anchor:hover svg {
@apply bg-white !important;
}
+206
View File
@@ -0,0 +1,206 @@
---
title: "Architecture"
description: "Understanding Voicebox's technical architecture"
---
## System Overview
Voicebox uses a client-server architecture with a React frontend and Python backend. The desktop app is built with Tauri and contains two main layers:
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
These two layers communicate via HTTP, with the frontend making API requests to the backend.
## Frontend Architecture
### Tech Stack
- **Framework**: React 18 with TypeScript
- **State Management**: Zustand stores
- **Data Fetching**: React Query (TanStack Query)
- **Styling**: Tailwind CSS
- **Audio**: WaveSurfer.js
- **Desktop**: Tauri (Rust)
### Component Structure
```
app/src/
├── components/ # React components
│ ├── profiles/ # Voice profile UI
│ ├── generation/ # Speech generation UI
│ ├── stories/ # Timeline editor
│ └── shared/ # Reusable components
├── lib/ # Utilities
│ ├── api/ # Generated API client
│ └── utils/ # Helper functions
├── hooks/ # React hooks
└── stores/ # Zustand state stores
```
### State Management
```typescript
// Example: Profile store
const useProfileStore = create((set) => ({
profiles: [],
selectedProfile: null,
setProfiles: (profiles) => set({ profiles }),
selectProfile: (id) => set({ selectedProfile: id })
}))
```
## Backend Architecture
### Tech Stack
- **Framework**: FastAPI (Python 3.11+)
- **TTS Model**: Qwen3-TTS
- **Transcription**: Whisper
- **Database**: SQLite
- **Audio**: librosa, soundfile
### API Structure
```python
# main.py - API routes
@app.post("/generate")
async def generate_speech(request: GenerateRequest):
# 1. Validate request
# 2. Load voice profile
# 3. Generate audio with TTS
# 4. Save to database
# 5. Return response
```
### Data Model
The database uses three main tables:
**Profile Table:** Stores voice profiles with fields for id, name, and language.
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
## Desktop App (Tauri)
### Rust Backend
```rust
// Sidecar process management
// File system access
// Native integrations
```
### Responsibilities
- Launch Python backend as sidecar process
- Native file dialogs
- System tray integration
- Auto-updates
- OS-specific features
## Build Process
### Development
```bash
# Frontend (Vite dev server)
cd app && bun run dev
# Backend (manual start)
cd backend && uvicorn main:app --reload
# Desktop app (connects to manual backend)
bun run dev
```
### Production
```bash
# Build everything (server binary + Tauri app)
bun run build
# Or build separately:
# 1. Build server binary (PyInstaller)
bun run build:server
# 2. Build Tauri app (includes server)
cd tauri && bun run tauri build
```
## Data Flow
### Generation Flow
When a user generates speech, the data flows through the following stages:
1. **User Input** - User enters text in a React component
2. **State Update** - Text is stored in Zustand state
3. **API Request** - React Query mutation triggers an API call via fetch
4. **Backend Processing** - FastAPI endpoint receives the request
5. **TTS Generation** - Qwen3-TTS model generates the audio
6. **Storage** - Audio file is saved to disk and a database record is created
7. **Response** - Backend returns the audio URL
8. **Cache Update** - React Query updates its cache with the response
9. **UI Update** - Component re-renders with new data
10. **Playback** - User can play the generated audio
## Performance Considerations
### Frontend
- **Code splitting** - Lazy load routes
- **Memoization** - React.memo for heavy components
- **Virtual scrolling** - For large lists
- **Debouncing** - Search and input handling
### Backend
- **Async operations** - All I/O is async
- **Model caching** - Keep TTS model in memory
- **Voice prompt caching** - Reuse embeddings
- **Connection pooling** - Database connections
## Security
### Current
- Local-only by default
- No authentication (localhost trust)
- File system sandboxing via Tauri
### Planned
- API key authentication
- User accounts
- Rate limiting
- HTTPS support
## Deployment Modes
### Local Mode
- Backend runs as sidecar
- All data stays on device
- No network required
### Remote Mode
- Backend on separate machine
- Frontend connects via HTTP
- Shared infrastructure possible
## Next Steps
<CardGroup cols={2}>
<Card title="Development Setup" icon="code" href="/development/setup">
Set up your dev environment
</Card>
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
Contribute to Voicebox
</Card>
</CardGroup>
+310
View File
@@ -0,0 +1,310 @@
---
title: "Audio Channels"
description: "How audio output routing works in Voicebox"
---
## Overview
Audio channels allow routing voice output to different audio devices. This is useful for multi-output setups where different voices should play through different speakers or applications.
## Architecture
**Channel:** A named audio bus that can be assigned to output devices.
**Device Mapping:** Links channels to OS audio device identifiers.
**Profile Mapping:** Links voice profiles to channels (many-to-many).
## Data Model
### AudioChannel Table
```python
class AudioChannel(Base):
__tablename__ = "audio_channels"
id = Column(String, primary_key=True)
name = Column(String, nullable=False)
is_default = Column(Boolean, default=False)
created_at = Column(DateTime)
```
### ChannelDeviceMapping Table
```python
class ChannelDeviceMapping(Base):
__tablename__ = "channel_device_mappings"
id = Column(String, primary_key=True)
channel_id = Column(String, ForeignKey("audio_channels.id"))
device_id = Column(String) # OS device identifier
```
### ProfileChannelMapping Table
```python
class ProfileChannelMapping(Base):
__tablename__ = "profile_channel_mappings"
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
```
## Default Channel
A default channel is created on database initialization:
```python
def init_db():
# Create default channel if it doesn't exist
default_channel = db.query(AudioChannel).filter(
AudioChannel.is_default == True
).first()
if not default_channel:
default_channel = AudioChannel(
id=str(uuid.uuid4()),
name="Default",
is_default=True
)
db.add(default_channel)
# Assign all existing profiles to default channel
profiles = db.query(VoiceProfile).all()
for profile in profiles:
mapping = ProfileChannelMapping(
profile_id=profile.id,
channel_id=default_channel.id
)
db.add(mapping)
```
## Core Operations
### Creating a Channel
```python
async def create_channel(
data: AudioChannelCreate,
db: Session,
) -> AudioChannelResponse:
# Check name uniqueness
existing = db.query(DBAudioChannel).filter_by(name=data.name).first()
if existing:
raise ValueError(f"Channel with name '{data.name}' already exists")
# Create channel
channel = DBAudioChannel(
id=str(uuid.uuid4()),
name=data.name,
is_default=False,
)
db.add(channel)
# Add device mappings
for device_id in data.device_ids:
mapping = DBChannelDeviceMapping(
id=str(uuid.uuid4()),
channel_id=channel.id,
device_id=device_id,
)
db.add(mapping)
db.commit()
```
### Updating a Channel
```python
async def update_channel(
channel_id: str,
data: AudioChannelUpdate,
db: Session,
) -> AudioChannelResponse:
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
# Cannot modify default channel
if channel.is_default:
raise ValueError("Cannot modify the default channel")
# Update name
if data.name is not None:
channel.name = data.name
# Update device mappings
if data.device_ids is not None:
# Delete existing
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
# Add new
for device_id in data.device_ids:
mapping = DBChannelDeviceMapping(
channel_id=channel.id,
device_id=device_id,
)
db.add(mapping)
db.commit()
```
### Deleting a Channel
```python
async def delete_channel(channel_id: str, db: Session) -> bool:
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
# Cannot delete default channel
if channel.is_default:
raise ValueError("Cannot delete the default channel")
# Delete device mappings
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
# Delete profile-channel mappings
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
# Delete channel
db.delete(channel)
db.commit()
```
## Voice Assignment
### Assigning Voices to Channel
```python
async def set_channel_voices(
channel_id: str,
data: ChannelVoiceAssignment,
db: Session,
) -> None:
# Verify channel exists
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
if not channel:
raise ValueError(f"Channel {channel_id} not found")
# Verify all profiles exist
for profile_id in data.profile_ids:
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile {profile_id} not found")
# Delete existing mappings
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
# Add new mappings
for profile_id in data.profile_ids:
mapping = DBProfileChannelMapping(
profile_id=profile_id,
channel_id=channel_id,
)
db.add(mapping)
db.commit()
```
### Assigning Channels to Voice
```python
async def set_profile_channels(
profile_id: str,
data: ProfileChannelAssignment,
db: Session,
) -> None:
# Verify profile exists
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile {profile_id} not found")
# Delete existing mappings
db.query(DBProfileChannelMapping).filter_by(profile_id=profile_id).delete()
# Add new mappings
for channel_id in data.channel_ids:
mapping = DBProfileChannelMapping(
profile_id=profile_id,
channel_id=channel_id,
)
db.add(mapping)
db.commit()
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/channels` | List all channels |
| POST | `/channels` | Create a channel |
| GET | `/channels/{id}` | Get channel by ID |
| PUT | `/channels/{id}` | Update channel |
| DELETE | `/channels/{id}` | Delete channel |
| GET | `/channels/{id}/voices` | Get assigned voices |
| PUT | `/channels/{id}/voices` | Set assigned voices |
| GET | `/profiles/{id}/channels` | Get profile's channels |
| PUT | `/profiles/{id}/channels` | Set profile's channels |
## Request/Response Schemas
### AudioChannelCreate
```json
{
"name": "Speakers",
"device_ids": ["device_uuid_1", "device_uuid_2"]
}
```
### AudioChannelResponse
```json
{
"id": "channel_uuid",
"name": "Speakers",
"is_default": false,
"device_ids": ["device_uuid_1", "device_uuid_2"],
"created_at": "2024-01-15T10:30:00Z"
}
```
### ChannelVoiceAssignment
```json
{
"profile_ids": ["profile_1", "profile_2"]
}
```
## Use Cases
### Multi-Output Setup
**Scenario:** Stream with different voice characters
1. Create "Stream" channel → OBS virtual audio
2. Create "Monitor" channel → Headphones
3. Assign "Narrator" profile → Both channels
4. Assign "Character 1" profile → Stream only
### Virtual Audio Cables
Common device IDs for virtual audio:
- VB-Audio Virtual Cable
- BlackHole (macOS)
- Soundflower (macOS)
## Frontend Integration
The frontend needs to:
1. **Enumerate devices** using Web Audio API or Tauri
2. **Display channel list** with device assignments
3. **Allow profile assignment** via drag/drop or dropdown
4. **Route playback** to correct device based on profile's channel
## Limitations
- Device IDs are OS-specific
- Hot-plugging may invalidate device IDs
- Default channel cannot be modified/deleted
- Frontend handles actual audio routing (backend just stores config)
+84
View File
@@ -0,0 +1,84 @@
---
title: "Auto-Updater"
description: "Configure and use the Tauri auto-updater"
---
## Overview
Voicebox uses Tauri's built-in auto-updater to deliver updates to users automatically.
## Quick Reference
For detailed setup instructions, see the existing documentation:
- [AUTOUPDATER_QUICKSTART.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER_QUICKSTART.md)
- [AUTOUPDATER.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER.md)
## How It Works
The auto-updater follows a secure update process:
1. **Check for Updates** - The Voicebox app periodically checks GitHub Releases for new versions
2. **Download Update** - If a new version is found, the update package is downloaded
3. **Verify Signature** - The downloaded package is cryptographically verified using the public key
4. **Install** - After verification, the update is installed
5. **Restart** - The app restarts with the new version
## Configuration
Updates are configured in `tauri/src-tauri/tauri.conf.json`:
```json
{
"updater": {
"active": true,
"endpoints": [
"https://github.com/jamiepine/voicebox/releases/latest/download/latest.json"
],
"dialog": true,
"pubkey": "YOUR_PUBLIC_KEY"
}
}
```
## Generating Keys
```bash
# Generate signing keys
bun run generate:keys
# Keys saved to ~/.tauri/voicebox.key
```
<Warning>
Keep your private key secure! Never commit it to the repository.
</Warning>
## Release Process
1. **Bump version** using bumpversion
2. **Push tag** to trigger CI/CD
3. **GitHub Actions** builds and signs releases
4. **Users** receive update notification
## User Experience
When an update is available:
1. User sees a notification dialog
2. User clicks "Update"
3. Update downloads in background
4. App restarts with new version
## For Developers
See the full documentation files for:
- Setting up signing keys
- Configuring GitHub releases
- Testing updates locally
- Troubleshooting update failures
<Card title="View Full Docs" href="https://github.com/jamiepine/voicebox/tree/main/docs">
Access AUTOUPDATER.md and AUTOUPDATER_QUICKSTART.md in the repository
</Card>
+270
View File
@@ -0,0 +1,270 @@
---
title: "Building"
description: "Build Voicebox for production"
---
## Overview
Voicebox uses a multi-step build process to create platform-specific installers.
## Quick Build
```bash
# Build for your current platform (automatically builds server binary first)
make build
# Or manually
bun run build
```
This automatically:
1. Builds the Python server binary (`bun run build:server`)
2. Builds the Tauri app (`cd tauri && bun run tauri build`)
## Build Process
The build process consists of two steps, but `bun run build` handles both automatically:
### 1. Server Binary Build (Automatic)
The Python backend is compiled into a standalone executable using PyInstaller. This happens automatically when you run `bun run build`.
**Platform-specific binaries:**
- macOS (Apple Silicon): `voicebox-server-aarch64-apple-darwin` (includes MLX backend)
- macOS (Intel): `voicebox-server-x86_64-apple-darwin` (PyTorch backend)
- Windows: `voicebox-server-x86_64-pc-windows-msvc.exe` (PyTorch backend)
- Linux: `voicebox-server-x86_64-unknown-linux-gnu` (PyTorch backend)
<Note>
The build script automatically detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
</Note>
**Manual build (if needed):**
```bash
bun run build:server
```
### 2. Tauri App Build (Automatic)
The Tauri app build is also handled automatically, which:
1. Builds the React frontend (Vite)
2. Compiles the Rust backend
3. Bundles the server binary as a sidecar
4. Creates platform-specific installers
**Manual build (if needed):**
```bash
cd tauri && bun run tauri build
```
### 3. Output
Installers are created in `tauri/src-tauri/target/release/bundle/`:
**macOS:**
- `dmg/` - Disk image installer
- `macos/` - App bundle
**Windows:**
- `msi/` - MSI installer
- `nsis/` - NSIS installer
**Linux:**
- `deb/` - Debian package
- `appimage/` - AppImage
## Advanced Options
### Building for Specific Platform
```bash
# Build for macOS (Apple Silicon)
bun run tauri build -- --target aarch64-apple-darwin
# Build for macOS (Intel)
bun run tauri build -- --target x86_64-apple-darwin
# Build for Windows
bun run tauri build -- --target x86_64-pc-windows-msvc
# Build for Linux
bun run tauri build -- --target x86_64-unknown-linux-gnu
```
### Using Local Qwen3-TTS
If you're developing Qwen3-TTS locally:
```bash
export QWEN_TTS_PATH=~/path/to/Qwen3-TTS
bun run build:server # Build server binary only
# or
bun run build # Build everything
```
This makes PyInstaller use your local version instead of the pip package.
### Debug Build
```bash
cd tauri
bun run tauri build --debug
```
Creates a debug build with symbols and logging.
## Build Configuration
### Tauri Config
Edit `tauri/src-tauri/tauri.conf.json`:
```json
{
"bundle": {
"identifier": "com.voicebox.app",
"icon": [
"icons/32x32.png",
"icons/128x128.png",
"icons/icon.icns",
"icons/icon.ico"
]
}
}
```
### Sidecar Configuration
The Python server is bundled as a sidecar:
```json
{
"tauri": {
"bundle": {
"externalBin": [
"binaries/voicebox-server"
]
}
}
}
```
## Code Signing
### macOS
To sign the app for distribution:
```bash
# Set signing identity
export APPLE_SIGNING_IDENTITY="Developer ID Application: Your Name"
# Build with signing
bun run tauri build
```
For notarization:
```bash
# Set credentials
export APPLE_ID="[email protected]"
export APPLE_PASSWORD="app-specific-password"
# Build and notarize
bun run tauri build
```
### Windows
For Windows code signing:
```bash
# Set certificate
export WINDOWS_CERTIFICATE_PATH="/path/to/cert.pfx"
export WINDOWS_CERTIFICATE_PASSWORD="password"
# Build with signing
bun run tauri build
```
## Release Process
The full release process is automated:
```bash
# 1. Bump version
bumpversion patch # or minor/major
# 2. Build all platforms (CI/CD handles this)
git push --tags
# 3. GitHub Actions creates releases
```
See [CONTRIBUTING.md](/development/contributing) for the full release workflow.
## Troubleshooting
<AccordionGroup>
<Accordion title="Server Binary Build Fails">
**Common issues:**
- Missing Python dependencies: `pip install -r requirements.txt`
- PyInstaller not found: `pip install pyinstaller`
- Qwen3-TTS not installed: `pip install git+https://github.com/QwenLM/Qwen3-TTS.git`
**Solution:**
```bash
cd backend
source venv/bin/activate
pip install -r requirements.txt
pip install pyinstaller
```
</Accordion>
<Accordion title="Tauri Build Fails">
**Common issues:**
- Rust not installed: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
- Server binary missing: Usually auto-built, but can run manually: `./scripts/build-server.sh`
- Node modules outdated: `bun install`
**Solution:**
```bash
# Clean and rebuild
cd tauri/src-tauri
cargo clean
cd ../..
bun run build # Automatically builds server binary first
```
</Accordion>
<Accordion title="App Won't Launch After Build">
**Check:**
- Server binary has execute permissions
- All dependencies are bundled
- Check logs in the app's data directory
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
## CI/CD
GitHub Actions automatically builds releases when tags are pushed:
```yaml
# .github/workflows/release.yml
on:
push:
tags:
- 'v*'
```
See the [repository](https://github.com/jamiepine/voicebox) for the full CI/CD configuration.
+326
View File
@@ -0,0 +1,326 @@
---
title: "Contributing"
description: "How to contribute to Voicebox"
---
Thank you for your interest in contributing to Voicebox! This guide will help you get started.
## Code of Conduct
- Be respectful and inclusive
- Welcome newcomers and help them learn
- Focus on constructive feedback
- Respect different viewpoints and experiences
## Getting Started
Before you start contributing, make sure you have:
1. **Read the documentation** to understand how Voicebox works
2. **Set up your development environment** - see [Development Setup](/development/setup)
3. **Explored the codebase** to understand the project structure
4. **Checked existing issues** to see if someone else is working on something similar
## Ways to Contribute
<CardGroup cols={2}>
<Card title="Report Bugs" icon="bug">
Found a bug? Open an issue with reproduction steps
</Card>
<Card title="Request Features" icon="lightbulb">
Have an idea? Start a discussion or open an issue
</Card>
<Card title="Improve Docs" icon="book">
Fix typos, add examples, or clarify instructions
</Card>
<Card title="Write Code" icon="code">
Fix bugs, add features, or optimize performance
</Card>
</CardGroup>
## Development Workflow
### 1. Fork & Clone
```bash
# Fork the repository on GitHub
# Then clone your fork
git clone https://github.com/YOUR_USERNAME/voicebox.git
cd voicebox
```
### 2. Create a Branch
Use descriptive branch names:
```bash
# For features
git checkout -b feature/voice-effects
# For bug fixes
git checkout -b fix/audio-playback-issue
# For documentation
git checkout -b docs/api-examples
```
### 3. Make Your Changes
Follow these guidelines:
<AccordionGroup>
<Accordion title="Code Style">
**TypeScript/React:**
- Use TypeScript strict mode
- Prefer functional components with hooks
- Use named exports
- Format with Biome (runs automatically)
**Python:**
- Follow PEP 8
- Use type hints
- Use async/await for I/O
- Document functions with docstrings
**Rust:**
- Follow Rust conventions
- Use meaningful names
- Handle errors explicitly
- Run `rustfmt`
</Accordion>
<Accordion title="Commit Messages">
Write clear, descriptive commit messages:
```bash
# Good
git commit -m "Add voice profile export feature"
git commit -m "Fix audio playback stopping after 30 seconds"
# Avoid
git commit -m "Update code"
git commit -m "Fix bug"
```
Format:
- Use imperative mood ("Add feature" not "Added feature")
- Keep first line under 50 characters
- Add detailed description if needed
</Accordion>
<Accordion title="Testing">
- Test your changes manually in the app
- Ensure backend API endpoints work
- Check for TypeScript/Python errors
- Verify UI components render correctly
- Add automated tests when possible
</Accordion>
</AccordionGroup>
### 4. Push & Create PR
```bash
# Push your branch
git push origin feature/your-feature-name
# Then create a pull request on GitHub
```
## Pull Request Guidelines
When creating a pull request:
<Steps>
<Step title="Use a Clear Title">
Examples:
- "Add voice profile export functionality"
- "Fix audio playback stopping after 30 seconds"
- "Improve generation speed with caching"
</Step>
<Step title="Provide Description">
Include:
- What changes you made
- Why you made them
- How to test them
- Screenshots (for UI changes)
- Reference related issues
</Step>
<Step title="Update Documentation">
- Update relevant docs if behavior changes
- Add API documentation for new endpoints
- Update README if needed
</Step>
<Step title="Check the Checklist">
- [ ] Code follows style guidelines
- [ ] Documentation updated
- [ ] Changes tested
- [ ] No breaking changes (or documented)
- [ ] CHANGELOG.md updated
</Step>
</Steps>
## Project Structure
Understanding the codebase:
```
voicebox/
├── app/ # Shared React frontend
│ ├── src/
│ │ ├── components/ # UI components
│ │ ├── lib/ # Utilities and API client
│ │ ├── hooks/ # React hooks
│ │ └── stores/ # Zustand state stores
├── backend/ # Python FastAPI server
│ ├── main.py # API routes
│ ├── tts.py # Voice synthesis logic
│ ├── database.py # SQLite operations
│ └── models.py # Pydantic models
├── tauri/ # Desktop app wrapper
│ └── src-tauri/ # Rust backend
├── web/ # Web deployment
├── landing/ # Marketing website
└── scripts/ # Build & release scripts
```
## Areas for Contribution
### Bug Fixes
- Check [existing issues](https://github.com/jamiepine/voicebox/issues) for bugs
- Test your fix thoroughly
- Add regression tests if possible
### New Features
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
- Discuss major features in an issue first
- Keep features focused and well-scoped
### Documentation
- Improve clarity and fix typos
- Add code examples
- Create tutorials or guides
- Document API endpoints
### UI/UX Improvements
- Improve accessibility
- Enhance visual design
- Optimize performance
- Add animations/transitions
### Infrastructure
- Improve build process
- Add CI/CD improvements
- Optimize bundle size
- Add testing infrastructure
## API Development
When adding new API endpoints:
<Steps>
<Step title="Add Route">
In `backend/main.py`:
```python
@app.post("/api/new-endpoint")
async def new_endpoint(data: RequestModel) -> ResponseModel:
"""Endpoint description."""
# Implementation
return response
```
</Step>
<Step title="Create Models">
In `backend/models.py`:
```python
class RequestModel(BaseModel):
field: str
class ResponseModel(BaseModel):
result: str
```
</Step>
<Step title="Regenerate Client">
```bash
bun run generate:api
```
This updates the TypeScript client with type-safe bindings.
</Step>
<Step title="Update Docs">
Add documentation in `/docs/api/`
</Step>
</Steps>
## Testing
Currently testing is primarily manual. When adding tests:
**Backend:**
```bash
cd backend
pytest
```
**Frontend:**
```bash
bun run test
```
**E2E (future):**
```bash
bun run test:e2e
```
## Release Process
Releases are managed by maintainers using `bumpversion`:
```bash
# Bump version (patch, minor, or major)
bumpversion patch
# Push with tags
git push && git push --tags
```
GitHub Actions automatically builds and publishes releases when tags are pushed.
## Community
- **GitHub Issues:** Bug reports and feature requests
- **GitHub Discussions:** General questions and ideas
- **Discord:** Real-time chat (coming soon)
## Recognition
Contributors are recognized in:
- [CHANGELOG.md](https://github.com/jamiepine/voicebox/blob/main/CHANGELOG.md)
- GitHub contributor list
- Release notes
## License
By contributing, you agree that your contributions will be licensed under the MIT License.
## Questions?
If you have questions:
1. Check the [documentation](/overview/introduction)
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
3. Open a new issue or discussion
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
Thank you for contributing to Voicebox! 🎉
+260
View File
@@ -0,0 +1,260 @@
---
title: "Generation History"
description: "How generation history tracking works in Voicebox"
---
## Overview
The history module tracks all generated audio, providing a searchable record of past generations. Each generation stores the text, settings, and a reference to the audio file.
## Data Model
### Generation Table
```python
class Generation(Base):
__tablename__ = "generations"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
text = Column(Text, nullable=False)
language = Column(String, default="en")
audio_path = Column(String, nullable=False)
duration = Column(Float, nullable=False)
seed = Column(Integer)
instruct = Column(Text)
created_at = Column(DateTime)
```
## File Storage
Generated audio is stored in:
```
data/
└── generations/
└── {generation_id}.wav
```
## Core Functions
### Creating a Generation Record
After TTS generates audio, a history entry is created:
```python
async def create_generation(
profile_id: str,
text: str,
language: str,
audio_path: str,
duration: float,
seed: Optional[int],
db: Session,
instruct: Optional[str] = None,
) -> GenerationResponse:
db_generation = DBGeneration(
id=str(uuid.uuid4()),
profile_id=profile_id,
text=text,
language=language,
audio_path=audio_path,
duration=duration,
seed=seed,
instruct=instruct,
created_at=datetime.utcnow(),
)
db.add(db_generation)
db.commit()
return GenerationResponse.model_validate(db_generation)
```
### Listing Generations
Supports filtering and pagination:
```python
async def list_generations(
query: HistoryQuery,
db: Session,
) -> HistoryListResponse:
# Build query with profile name join
q = db.query(
DBGeneration,
DBVoiceProfile.name.label('profile_name')
).join(
DBVoiceProfile,
DBGeneration.profile_id == DBVoiceProfile.id
)
# Apply filters
if query.profile_id:
q = q.filter(DBGeneration.profile_id == query.profile_id)
if query.search:
q = q.filter(DBGeneration.text.like(f"%{query.search}%"))
# Order and paginate
total = q.count()
q = q.order_by(DBGeneration.created_at.desc())
q = q.offset(query.offset).limit(query.limit)
return HistoryListResponse(items=results, total=total)
```
### Getting Statistics
Aggregate statistics for the dashboard:
```python
async def get_generation_stats(db: Session) -> dict:
total = db.query(func.count(DBGeneration.id)).scalar()
total_duration = db.query(func.sum(DBGeneration.duration)).scalar()
by_profile = db.query(
DBGeneration.profile_id,
func.count(DBGeneration.id).label('count')
).group_by(DBGeneration.profile_id).all()
return {
"total_generations": total,
"total_duration_seconds": total_duration,
"generations_by_profile": {
profile_id: count for profile_id, count in by_profile
},
}
```
## Deletion
Deleting a generation removes both the database record and audio file:
```python
async def delete_generation(generation_id: str, db: Session) -> bool:
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
if not generation:
return False
# Delete audio file
audio_path = Path(generation.audio_path)
if audio_path.exists():
audio_path.unlink()
# Delete database record
db.delete(generation)
db.commit()
return True
```
### Cascade Delete
When deleting a profile, all its generations are also deleted:
```python
async def delete_generations_by_profile(profile_id: str, db: Session) -> int:
generations = db.query(DBGeneration).filter_by(profile_id=profile_id).all()
for generation in generations:
Path(generation.audio_path).unlink(missing_ok=True)
db.delete(generation)
db.commit()
return len(generations)
```
## Export/Import
### Exporting a Generation
Generations can be exported as ZIP archives:
```
generation_export.zip
├── generation.json # Metadata
└── audio.wav # Audio file
```
### Importing a Generation
The import process:
1. Extract ZIP archive
2. Validate metadata and audio
3. Create new generation ID
4. Copy audio to generations directory
5. Create database record
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/history` | List generations with filters |
| GET | `/history/stats` | Get aggregate statistics |
| GET | `/history/{id}` | Get generation by ID |
| DELETE | `/history/{id}` | Delete generation |
| GET | `/history/{id}/export` | Export as ZIP |
| GET | `/history/{id}/export-audio` | Export audio only |
| POST | `/history/import` | Import from ZIP |
### Query Parameters
```
GET /history?profile_id=uuid&search=hello&limit=50&offset=0
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `profile_id` | string | null | Filter by profile |
| `search` | string | null | Search in text |
| `limit` | int | 50 | Results per page |
| `offset` | int | 0 | Pagination offset |
### Response Schema
```json
{
"items": [
{
"id": "uuid",
"profile_id": "uuid",
"profile_name": "My Voice",
"text": "Hello world",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 1.5,
"seed": 42,
"instruct": null,
"created_at": "2024-01-15T10:30:00Z"
}
],
"total": 150
}
```
## Usage in Stories
Generations can be added to stories for multi-voice narratives. The story system references generations by ID:
```python
class StoryItem(Base):
generation_id = Column(String, ForeignKey("generations.id"))
```
This allows the same generation to be reused across multiple stories without duplicating audio files.
## Storage Considerations
### Disk Usage
Each generation creates a WAV file. For a 10-second clip at 24kHz:
- ~480KB per file (mono, 16-bit)
### Cleanup Strategy
Consider implementing:
- Automatic cleanup of old generations
- Storage quota per profile
- Compression for archival
+341
View File
@@ -0,0 +1,341 @@
---
title: "Model Management"
description: "How model downloading, loading, and status tracking works in Voicebox"
---
## Overview
Voicebox manages two types of models:
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
**ASR Models:** Whisper for transcription (tiny through large).
Models are downloaded from HuggingFace Hub on first use and cached locally.
## Available Models
### TTS Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
### Whisper Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
| base | `openai/whisper-base` | ~300MB | ~1GB |
| small | `openai/whisper-small` | ~500MB | ~2GB |
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
| large | `openai/whisper-large` | ~3GB | ~10GB |
## Model Storage
Models are cached in the HuggingFace cache directory:
```
~/.cache/huggingface/hub/
├── models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/
├── models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/
├── models--openai--whisper-base/
└── ...
```
## Progress Tracking
### Progress Manager
Tracks download progress across all models:
```python
class ProgressManager:
def __init__(self):
self._progress = {} # model_name -> progress_info
def update_progress(
self,
model_name: str,
current: int,
total: int,
filename: str,
status: str,
):
self._progress[model_name] = {
"current": current,
"total": total,
"filename": filename,
"status": status, # downloading, complete, error
"updated_at": datetime.utcnow(),
}
def get_progress(self, model_name: str) -> Optional[dict]:
return self._progress.get(model_name)
```
### HuggingFace Progress Callback
Hooks into HuggingFace's download system:
```python
class HFProgressTracker:
def __init__(self, callback):
self.callback = callback
@contextmanager
def patch_download(self):
"""Context manager to intercept HF downloads."""
original_download = hf_hub_download
def patched_download(*args, **kwargs):
# Intercept progress
result = original_download(*args, **kwargs)
self.callback(progress_info)
return result
# Apply patch
with patch('huggingface_hub.hf_hub_download', patched_download):
yield
```
### Server-Sent Events (SSE)
Progress is streamed to the frontend:
```python
@app.get("/models/progress/{model_name}")
async def get_model_progress(model_name: str):
async def event_generator():
while True:
progress = progress_manager.get_progress(model_name)
if progress:
yield f"data: {json.dumps(progress)}\n\n"
if progress and progress["status"] in ["complete", "error"]:
break
await asyncio.sleep(0.5)
return StreamingResponse(
event_generator(),
media_type="text/event-stream"
)
```
## Task Manager
Tracks active downloads and generations:
```python
class TaskManager:
def __init__(self):
self._active_downloads = {}
self._active_generations = {}
def start_download(self, model_name: str):
self._active_downloads[model_name] = {
"status": "downloading",
"started_at": datetime.utcnow(),
}
def complete_download(self, model_name: str):
if model_name in self._active_downloads:
del self._active_downloads[model_name]
def get_active_tasks(self) -> dict:
return {
"downloads": list(self._active_downloads.values()),
"generations": list(self._active_generations.values()),
}
```
## Model Status
Check which models are downloaded and loaded:
```python
@app.get("/models/status")
async def get_model_status() -> ModelStatusListResponse:
models = []
# Check TTS models
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
downloaded = is_model_downloaded(hf_id)
loaded = tts_model._current_model_size == size
models.append(ModelStatus(
model_name=f"qwen-tts-{size}",
display_name=f"Qwen3-TTS {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=loaded,
))
# Check Whisper models
for size in ["tiny", "base", "small", "medium", "large"]:
hf_id = f"openai/whisper-{size}"
downloaded = is_model_downloaded(hf_id)
models.append(ModelStatus(
model_name=f"whisper-{size}",
display_name=f"Whisper {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=False, # Whisper is loaded on-demand
))
return ModelStatusListResponse(models=models)
```
## Manual Model Operations
### Load Model
```python
@app.post("/models/load")
async def load_model(model_size: str = "1.7B"):
tts_model = get_tts_model()
await tts_model.load_model_async(model_size)
return {"status": "loaded", "model_size": model_size}
```
### Unload Model
```python
@app.post("/models/unload")
async def unload_model():
tts_model = get_tts_model()
tts_model.unload_model()
return {"status": "unloaded"}
```
### Trigger Download
```python
@app.post("/models/download")
async def trigger_model_download(request: ModelDownloadRequest):
# This triggers the download in background
# Progress is tracked via /models/progress/{model_name}
if request.model_name.startswith("qwen-tts"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_tts_model(size))
elif request.model_name.startswith("whisper"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_whisper_model(size))
return {"status": "downloading"}
```
### Delete Model
```python
@app.delete("/models/{model_name}")
async def delete_model(model_name: str):
# Find and delete from HuggingFace cache
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
for model_dir in model_dirs:
shutil.rmtree(model_dir)
return {"status": "deleted"}
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Get status of all models |
| POST | `/models/load` | Load TTS model |
| POST | `/models/unload` | Unload TTS model |
| POST | `/models/download` | Trigger model download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete downloaded model |
| GET | `/tasks/active` | Get active downloads/generations |
## Response Schemas
### ModelStatus
```json
{
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen3-TTS 1.7B",
"downloaded": true,
"size_mb": 3400,
"loaded": true
}
```
### ActiveTasksResponse
```json
{
"downloads": [
{
"model_name": "whisper-medium",
"status": "downloading",
"started_at": "2024-01-15T10:30:00Z"
}
],
"generations": [
{
"task_id": "uuid",
"profile_id": "uuid",
"text_preview": "Hello world...",
"started_at": "2024-01-15T10:30:00Z"
}
]
}
```
## Frontend Integration
### Progress Display
```typescript
// Subscribe to download progress via SSE
const eventSource = new EventSource(`/models/progress/${modelName}`);
eventSource.onmessage = (event) => {
const progress = JSON.parse(event.data);
updateProgressBar(progress.current / progress.total);
if (progress.status === 'complete') {
eventSource.close();
}
};
```
### Model Status UI
```typescript
// Fetch model status
const { data: models } = useQuery({
queryKey: ['models', 'status'],
queryFn: () => api.getModelStatus(),
});
// Display download/load buttons based on status
models.map(model => (
<ModelCard
name={model.display_name}
downloaded={model.downloaded}
loaded={model.loaded}
onDownload={() => triggerDownload(model.model_name)}
onLoad={() => loadModel(model.model_name)}
/>
));
```
## Error Handling
| Error | Cause | Solution |
|-------|-------|----------|
| Download failed | Network issue | Retry download |
| OOM on load | Model too large | Use smaller model |
| Model not found | Cache corrupted | Re-download |
| Slow download | HF rate limit | Wait and retry |
+239
View File
@@ -0,0 +1,239 @@
---
title: "Development Setup"
description: "Set up your local development environment for Voicebox"
---
## Prerequisites
Before you begin, ensure you have the following installed:
<CardGroup cols={3}>
<Card title="Bun" icon="package">
[Download Bun](https://bun.sh)
```bash
curl -fsSL https://bun.sh/install | bash
```
</Card>
<Card title="Python 3.11+" icon="python">
[Download Python](https://python.org)
```bash
python --version
```
</Card>
<Card title="Rust" icon="rust">
[Install Rust](https://rustup.rs)
```bash
rustc --version
```
</Card>
</CardGroup>
## Clone the Repository
```bash
git clone https://github.com/jamiepine/voicebox.git
cd voicebox
```
## Quick Setup (Recommended)
The easiest way to get started is using the Makefile:
```bash
# Setup everything
make setup
# Start development
make dev
```
<Note>
The Makefile is available on macOS and Linux. Windows users should follow the manual setup below.
</Note>
## Manual Setup
### 1. Install JavaScript Dependencies
```bash
bun install
```
This installs dependencies for:
- `app/` - Shared React frontend
- `tauri/` - Tauri desktop wrapper
- `web/` - Web deployment wrapper
### 2. Set Up Python Backend
```bash
cd backend
# Create virtual environment
python -m venv venv
# Activate virtual environment
source venv/bin/activate # macOS/Linux
# or
venv\Scripts\activate # Windows
# Install Python dependencies
pip install -r requirements.txt
# Install MLX dependencies (Apple Silicon only - for faster inference)
# On Apple Silicon, this enables native Metal acceleration
if [[ $(uname -m) == "arm64" ]]; then
pip install -r requirements-mlx.txt
fi
# Install Qwen3-TTS
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
## Running in Development
Development requires **two terminals**: one for the Python backend, one for the Tauri app.
<Tabs>
<Tab title="Terminal 1: Backend">
Start the Python server first:
```bash
cd backend
source venv/bin/activate # Activate venv
bun run dev:server
```
Or manually:
```bash
uvicorn main:app --reload --port 17493
```
Backend will be available at `http://localhost:17493`
</Tab>
<Tab title="Terminal 2: Desktop App">
Then start the Tauri app:
```bash
bun run dev
```
This will:
- Create a placeholder sidecar binary
- Start Vite dev server on port 5173
- Launch Tauri window
- Enable hot reload
</Tab>
</Tabs>
<Info>
In dev mode, the app connects to your manually-started Python server. The bundled server binary is only used in production builds.
</Info>
### Optional: Web App
```bash
bun run dev:web
```
Web app will be available at `http://localhost:5174`
## Model Downloads
Models are automatically downloaded from HuggingFace Hub on first use:
- **Whisper** (transcription): Auto-downloads on first transcription
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
<Warning>
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
</Warning>
## Project Structure
```
voicebox/
├── app/ # Shared React frontend
│ └── src/
│ ├── components/ # UI components
│ ├── lib/ # Utilities and API client
│ └── hooks/ # React hooks
├── backend/ # Python FastAPI server
│ ├── main.py # API routes
│ ├── tts.py # Voice synthesis
│ └── database.py # SQLite operations
├── tauri/ # Desktop app wrapper
│ └── src-tauri/ # Rust backend
├── web/ # Web deployment
├── landing/ # Marketing website
└── scripts/ # Build & release scripts
```
## Available Make Commands
Run `make help` to see all available commands:
```bash
make setup # Install all dependencies
make dev # Start development servers
make dev-web # Start web development server
make build # Build desktop app
make build-web # Build web app
make clean # Clean build artifacts
make test # Run tests
```
## Generate OpenAPI Client
After starting the backend server, generate the TypeScript API client:
```bash
./scripts/generate-api.sh
# or
bun run generate:api
```
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
## Next Steps
<CardGroup cols={2}>
<Card title="Architecture" icon="diagram-project" href="/development/architecture">
Understand the system architecture
</Card>
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
Read the contribution guidelines
</Card>
<Card title="Building" icon="hammer" href="/development/building">
Learn how to build production releases
</Card>
<Card title="API Reference" icon="code" href="/api/overview">
Explore the REST API
</Card>
</CardGroup>
## Troubleshooting
<AccordionGroup>
<Accordion title="Backend won't start">
- Check Python version (must be 3.11+)
- Ensure virtual environment is activated
- Verify all dependencies are installed: `pip install -r requirements.txt`
- Check if port 17493 is available
</Accordion>
<Accordion title="Tauri build fails">
- Ensure Rust is installed: `rustc --version`
- Clean the build: `cd tauri/src-tauri && cargo clean`
- Try rebuilding: `bun run dev`
</Accordion>
<Accordion title="OpenAPI client generation fails">
- Ensure backend is running: `curl http://localhost:17493/openapi.json`
- Check network connectivity
- Verify the backend is accessible at localhost:17493
</Accordion>
</AccordionGroup>
See the full [Troubleshooting Guide](/guides/troubleshooting) for more issues and solutions.
+320
View File
@@ -0,0 +1,320 @@
---
title: "Stories & Timeline"
description: "How the multi-voice timeline editor works in Voicebox"
---
## Overview
Stories allow users to arrange multiple voice generations on a timeline to create multi-voice narratives. The system supports tracks, trimming, splitting, and audio mixing.
## Architecture
**Story:** A container that holds story items with metadata.
**Story Item:** Links a generation to a story with timeline position, track, and trim data.
**Export:** Combines all items into a single mixed audio file.
## Data Model
### Story Table
```python
class Story(Base):
__tablename__ = "stories"
id = Column(String, primary_key=True)
name = Column(String, nullable=False)
description = Column(Text)
created_at = Column(DateTime)
updated_at = Column(DateTime)
```
### StoryItem Table
```python
class StoryItem(Base):
__tablename__ = "story_items"
id = Column(String, primary_key=True)
story_id = Column(String, ForeignKey("stories.id"))
generation_id = Column(String, ForeignKey("generations.id"))
start_time_ms = Column(Integer, default=0) # Timeline position
track = Column(Integer, default=0) # Track number
trim_start_ms = Column(Integer, default=0) # Trim from start
trim_end_ms = Column(Integer, default=0) # Trim from end
created_at = Column(DateTime)
```
## Timeline Concepts
### Start Time
`start_time_ms` defines when an item begins on the timeline:
```
Timeline (ms): 0----1000----2000----3000----4000
Item 1: [======]
Item 2: [==========]
Item 3: [====]
```
### Tracks
Multiple tracks allow overlapping audio:
```
Track 0: [Item 1] [Item 3]
Track 1: [Item 2]
```
### Trimming
Trim values cut audio from the start or end without destroying the original:
```
Original: [=========AUDIO=========]
trim_start: ^^
trim_end: ^^
Result: [=====AUDIO=====]
```
## Core Operations
### Adding Items
When adding a generation to a story:
```python
async def add_item_to_story(
story_id: str,
data: StoryItemCreate,
db: Session,
) -> StoryItemDetail:
# Calculate start time if not provided
if data.start_time_ms is None:
# Find the end of all existing items
existing_items = get_items_with_durations(story_id, db)
max_end_time_ms = max(
item.start_time_ms + int(gen.duration * 1000)
for item, gen in existing_items
)
start_time_ms = max_end_time_ms + 200 # 200ms gap
# Create the item
item = DBStoryItem(
id=str(uuid.uuid4()),
story_id=story_id,
generation_id=data.generation_id,
start_time_ms=start_time_ms,
track=data.track or 0,
)
db.add(item)
db.commit()
```
### Moving Items
Update position and/or track:
```python
async def move_story_item(
story_id: str,
item_id: str,
data: StoryItemMove,
db: Session,
) -> StoryItemDetail:
item = get_item(story_id, item_id, db)
item.start_time_ms = data.start_time_ms
item.track = data.track
db.commit()
```
### Trimming Items
Non-destructive trimming:
```python
async def trim_story_item(
story_id: str,
item_id: str,
data: StoryItemTrim,
db: Session,
) -> StoryItemDetail:
item = get_item(story_id, item_id, db)
generation = get_generation(item.generation_id, db)
# Validate trim doesn't exceed duration
max_duration_ms = int(generation.duration * 1000)
if data.trim_start_ms + data.trim_end_ms >= max_duration_ms:
return None # Invalid trim
item.trim_start_ms = data.trim_start_ms
item.trim_end_ms = data.trim_end_ms
db.commit()
```
### Splitting Items
Split one item into two at a specific time:
```python
async def split_story_item(
story_id: str,
item_id: str,
data: StoryItemSplit,
db: Session,
) -> List[StoryItemDetail]:
item = get_item(story_id, item_id, db)
generation = get_generation(item.generation_id, db)
# Calculate split point
current_trim_start = item.trim_start_ms
current_trim_end = item.trim_end_ms
original_duration_ms = int(generation.duration * 1000)
absolute_split_ms = current_trim_start + data.split_time_ms
# Update original: trim from end
item.trim_end_ms = original_duration_ms - absolute_split_ms
# Create new item: trim from start
new_item = DBStoryItem(
generation_id=item.generation_id, # Same generation
start_time_ms=item.start_time_ms + data.split_time_ms,
track=item.track,
trim_start_ms=absolute_split_ms,
trim_end_ms=current_trim_end,
)
db.add(new_item)
db.commit()
return [item, new_item]
```
### Duplicating Items
Create a copy with all properties:
```python
async def duplicate_story_item(
story_id: str,
item_id: str,
db: Session,
) -> StoryItemDetail:
original = get_item(story_id, item_id, db)
generation = get_generation(original.generation_id, db)
# Calculate effective duration for positioning
effective_duration_ms = (
int(generation.duration * 1000)
- original.trim_start_ms
- original.trim_end_ms
)
# Place copy after original with 200ms gap
new_item = DBStoryItem(
generation_id=original.generation_id,
start_time_ms=original.start_time_ms + effective_duration_ms + 200,
track=original.track,
trim_start_ms=original.trim_start_ms,
trim_end_ms=original.trim_end_ms,
)
db.add(new_item)
db.commit()
```
## Audio Export
### Mixing Algorithm
The export function mixes all items into a single audio file:
```python
async def export_story_audio(story_id: str, db: Session) -> bytes:
items = get_all_items_with_generations(story_id, db)
# Calculate total duration
max_end_time_ms = max(
data['start_time_ms'] + data['duration_ms']
for data in audio_data
)
# Create output buffer
total_samples = int((max_end_time_ms / 1000.0) * sample_rate)
final_audio = np.zeros(total_samples, dtype=np.float32)
# Mix each item at its position
for data in audio_data:
audio = data['audio']
start_sample = int((data['start_time_ms'] / 1000.0) * sample_rate)
# Apply trim
trimmed_audio = audio[trim_start_sample:len(audio) - trim_end_sample]
# Add to buffer (overlapping items sum together)
final_audio[start_sample:start_sample + len(trimmed_audio)] += trimmed_audio
# Normalize to prevent clipping
max_val = np.abs(final_audio).max()
if max_val > 1.0:
final_audio = final_audio / max_val
return audio_to_bytes(final_audio, sample_rate)
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/stories` | List all stories |
| POST | `/stories` | Create a story |
| GET | `/stories/{id}` | Get story with items |
| PUT | `/stories/{id}` | Update story metadata |
| DELETE | `/stories/{id}` | Delete story |
| POST | `/stories/{id}/items` | Add item to story |
| DELETE | `/stories/{id}/items/{item_id}` | Remove item |
| PUT | `/stories/{id}/items/{item_id}/move` | Move item |
| PUT | `/stories/{id}/items/{item_id}/trim` | Trim item |
| POST | `/stories/{id}/items/{item_id}/split` | Split item |
| POST | `/stories/{id}/items/{item_id}/duplicate` | Duplicate item |
| PUT | `/stories/{id}/items/times` | Batch update times |
| PUT | `/stories/{id}/items/reorder` | Reorder items |
| GET | `/stories/{id}/export-audio` | Export mixed audio |
## Response Schemas
### StoryItemDetail
```json
{
"id": "item_uuid",
"story_id": "story_uuid",
"generation_id": "generation_uuid",
"start_time_ms": 1500,
"track": 0,
"trim_start_ms": 200,
"trim_end_ms": 100,
"profile_id": "profile_uuid",
"profile_name": "Narrator",
"text": "Hello world",
"audio_path": "/path/to/audio.wav",
"duration": 2.5,
"created_at": "2024-01-15T10:30:00Z"
}
```
## Frontend Integration
The timeline UI needs to:
1. **Fetch story** with all items
2. **Render waveforms** for each item
3. **Handle drag/drop** to move items
4. **Handle edge drag** for trimming
5. **Sync playhead** across all tracks
6. **Export** when user clicks download
+299
View File
@@ -0,0 +1,299 @@
---
title: "Transcription"
description: "How Whisper-based audio transcription works in Voicebox"
---
## Overview
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
## Architecture
The transcription system is built around the `WhisperModel` class:
**Model Loading:** Lazy loading with HuggingFace Hub download.
**Audio Processing:** Resampling and preprocessing for Whisper.
**Inference:** Running transcription with optional language hints.
## WhisperModel Class
```python
class WhisperModel:
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
```
### Model Sizes
| Size | Parameters | VRAM | Speed | Quality |
|------|------------|------|-------|---------|
| tiny | 39M | ~1GB | Fastest | Basic |
| base | 74M | ~1GB | Fast | Good |
| small | 244M | ~2GB | Medium | Better |
| medium | 769M | ~5GB | Slow | High |
| large | 1550M | ~10GB | Slowest | Best |
Default is `base` for balance of speed and quality.
## Model Loading
Models are downloaded from HuggingFace Hub:
```python
def load_model(self, model_size: Optional[str] = None):
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
# Track download progress
progress_manager = get_progress_manager()
task_manager = get_task_manager()
task_manager.start_download(f"whisper-{model_size}")
# Load processor and model
with tracker.patch_download():
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
# Mark complete
progress_manager.mark_complete(f"whisper-{model_size}")
task_manager.complete_download(f"whisper-{model_size}")
```
### Async Loading
Like TTS, loading runs in a thread pool:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self.model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Transcription
### Basic Transcription
```python
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
await self.load_model_async()
def _transcribe_sync():
# Load and resample to 16kHz (Whisper requirement)
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language hint if provided
forced_decoder_ids = None
if language:
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language,
task="transcribe",
)
# Generate
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
return await asyncio.to_thread(_transcribe_sync)
```
### Supported Languages
Whisper supports 99+ languages. Common ones in Voicebox:
| Code | Language |
|------|----------|
| en | English |
| zh | Chinese |
| ja | Japanese |
| ko | Korean |
| de | German |
| fr | French |
| ru | Russian |
| pt | Portuguese |
| es | Spanish |
| it | Italian |
### Language Detection
When no language is specified, Whisper auto-detects:
```python
# Without language hint - auto-detect
transcription = await whisper.transcribe(audio_path)
# With language hint - more accurate for short clips
transcription = await whisper.transcribe(audio_path, language="en")
```
## Transcription with Timestamps
For advanced use cases, word-level timestamps are available:
```python
async def transcribe_with_timestamps(
self,
audio_path: str,
language: Optional[str] = None,
) -> List[Dict[str, any]]:
await self.load_model_async()
def _transcribe_timestamps_sync():
audio, sr = load_audio(audio_path, sample_rate=16000)
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
return_timestamps=True,
)
# Parse timestamps
return [
{
"text": transcription,
"start": 0.0,
"end": len(audio) / sr,
}
]
return await asyncio.to_thread(_transcribe_timestamps_sync)
```
## Memory Management
### Unloading
Free memory when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Global Instance
A singleton pattern manages the model:
```python
_whisper_model: Optional[WhisperModel] = None
def get_whisper_model() -> WhisperModel:
global _whisper_model
if _whisper_model is None:
_whisper_model = WhisperModel()
return _whisper_model
```
## Audio Preprocessing
### Resampling
Whisper requires 16kHz audio:
```python
audio, sr = load_audio(audio_path, sample_rate=16000)
```
### Format Support
The `load_audio` utility handles:
- WAV
- MP3
- FLAC
- OGG
- M4A
All formats are converted to mono 16kHz.
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/transcribe` | Transcribe audio file |
### Request
Multipart form data:
```
POST /transcribe
Content-Type: multipart/form-data
file: <audio_file>
language: en (optional)
```
### Response
```json
{
"text": "Hello, this is a test transcription.",
"duration": 3.5
}
```
## Use Cases
### Reference Text for Voice Cloning
1. User records audio sample
2. Audio is sent to `/transcribe`
3. Transcription becomes `reference_text`
4. Both are added to voice profile
### Quality Tips
- Provide language hint for short audio
- Use clean audio with minimal noise
- Longer audio (>5s) improves accuracy
- Consider `small` or `medium` model for better quality
## Error Handling
Common issues:
| Error | Cause | Solution |
|-------|-------|----------|
| Model not found | First run, download failed | Retry with network |
| OOM | Model too large | Use smaller model |
| Empty result | No speech detected | Check audio has speech |
| Wrong language | Auto-detect failed | Provide language hint |
+283
View File
@@ -0,0 +1,283 @@
---
title: "TTS Generation"
description: "How text-to-speech generation works in Voicebox"
---
## Overview
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
## Architecture
The TTS system is built around the `TTSModel` class which manages:
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
**Voice Prompts:** Converting reference audio into embeddings.
**Generation:** Synthesizing speech from text using voice prompts.
## TTSModel Class
```python
class TTSModel:
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device() # cuda, mps, or cpu
```
### Device Selection
The model automatically selects the best available device:
```python
def _get_device(self) -> str:
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return "cpu" # MPS can have issues, use CPU for stability
return "cpu"
```
## Model Loading
Models are downloaded from HuggingFace Hub on first use:
```python
def load_model(self, model_size: Optional[str] = None):
# Model IDs on HuggingFace Hub
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
# Load with progress tracking
with tracker.patch_download():
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16, # float32 on CPU
)
```
### Async Loading
Loading runs in a thread pool to avoid blocking the event loop:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self._current_model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Voice Prompt Creation
Voice prompts are created from reference audio and cached for reuse:
```python
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
await self.load_model_async()
# Check cache
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached = get_cached_voice_prompt(cache_key)
if cached:
return cached, True
# Create prompt (blocking, run in thread pool)
voice_prompt = await asyncio.to_thread(
self.model.create_voice_clone_prompt,
ref_audio=audio_path,
ref_text=reference_text,
)
# Cache the result
cache_voice_prompt(cache_key, voice_prompt)
return voice_prompt, False
```
### Combining Multiple Samples
When a profile has multiple samples, they're combined:
```python
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate and normalize
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
```
## Speech Generation
The core generation function:
```python
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
await self.load_model_async()
def _generate_sync():
# Set seed for reproducibility
if seed is not None:
torch.manual_seed(seed)
# Generate audio
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct, # Natural language delivery control
)
return wavs[0], sample_rate
# Run in thread pool
return await asyncio.to_thread(_generate_sync)
```
### Instruct Feature
The `instruct` parameter allows natural language control over speech delivery:
```python
# Examples:
instruct = "Speak slowly and clearly"
instruct = "Sound excited and enthusiastic"
instruct = "Whisper softly"
```
## Caching Strategy
Voice prompts are cached to avoid recomputation:
```python
def get_cache_key(audio_path: str, reference_text: str) -> str:
"""Generate cache key from audio hash and text."""
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
return f"{audio_hash}_{text_hash}"
```
Cache is stored in `data/cache/voice_prompts/`.
## Memory Management
### Unloading Models
Free VRAM/RAM when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
self.model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Model Switching
When switching between model sizes (1.7B ↔ 0.6B):
```python
# Unload existing model first
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
```
## Generation Flow
1. **Request** → Validate text and profile ID
2. **Profile** → Load profile samples from database
3. **Voice Prompt** → Create or retrieve cached prompt
4. **Generate** → Run TTS inference
5. **Save** → Write audio to generations directory
6. **Record** → Create history entry in database
7. **Response** → Return audio path and metadata
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/generate` | Generate speech from text |
| GET | `/audio/{id}` | Serve generated audio file |
### Request Schema
```json
{
"profile_id": "uuid",
"text": "Text to synthesize",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "Speak clearly"
}
```
### Response Schema
```json
{
"id": "generation_uuid",
"profile_id": "profile_uuid",
"text": "Text to synthesize",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 3.5,
"seed": 42,
"instruct": "Speak clearly",
"created_at": "2024-01-15T10:30:00Z"
}
```
## Performance Considerations
### GPU Acceleration
- CUDA provides fastest inference
- MPS (Apple Silicon) has stability issues, uses CPU fallback
- CPU inference is slower but always works
### Batch Size
Currently generates one utterance at a time. For long texts, consider:
- Splitting into sentences
- Sequential generation
- Concatenating results
### Memory Usage
| Model | VRAM/RAM Required |
|-------|-------------------|
| 0.6B | ~2GB |
| 1.7B | ~6GB |
+202
View File
@@ -0,0 +1,202 @@
---
title: "Voice Profiles"
description: "How voice profile management works in Voicebox"
---
## Overview
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
## Architecture
The voice profile system consists of three main components:
**Database Layer:** SQLite tables store profile metadata and sample references.
**File Storage:** Audio samples are stored on disk in a structured directory format.
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
## Data Model
### VoiceProfile Table
```python
class VoiceProfile(Base):
__tablename__ = "profiles"
id = Column(String, primary_key=True)
name = Column(String, unique=True, nullable=False)
description = Column(Text)
language = Column(String, default="en")
created_at = Column(DateTime)
updated_at = Column(DateTime)
```
### ProfileSample Table
```python
class ProfileSample(Base):
__tablename__ = "profile_samples"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
audio_path = Column(String, nullable=False)
reference_text = Column(Text, nullable=False)
```
## File Structure
Profiles are stored in the data directory:
```
data/
└── profiles/
└── {profile_id}/
├── {sample_id_1}.wav
├── {sample_id_2}.wav
└── ...
```
## Core Functions
### Creating a Profile
```python
async def create_profile(data: VoiceProfileCreate, db: Session) -> VoiceProfileResponse:
# 1. Create database record
db_profile = DBVoiceProfile(
id=str(uuid.uuid4()),
name=data.name,
description=data.description,
language=data.language,
)
db.add(db_profile)
db.commit()
# 2. Create profile directory
profile_dir = profiles_dir / db_profile.id
profile_dir.mkdir(parents=True, exist_ok=True)
return VoiceProfileResponse.model_validate(db_profile)
```
### Adding Samples
When a sample is added, the audio is validated and copied to the profile directory:
```python
async def add_profile_sample(
profile_id: str,
audio_path: str,
reference_text: str,
db: Session,
) -> ProfileSampleResponse:
# 1. Validate audio (duration, format, quality)
is_valid, error_msg = validate_reference_audio(audio_path)
if not is_valid:
raise ValueError(f"Invalid reference audio: {error_msg}")
# 2. Copy to profile directory
sample_id = str(uuid.uuid4())
dest_path = profile_dir / f"{sample_id}.wav"
audio, sr = load_audio(audio_path)
save_audio(audio, str(dest_path), sr)
# 3. Create database record
db_sample = DBProfileSample(
id=sample_id,
profile_id=profile_id,
audio_path=str(dest_path),
reference_text=reference_text,
)
db.add(db_sample)
db.commit()
```
### Voice Prompt Creation
When generating speech, samples are combined into a voice prompt:
```python
async def create_voice_prompt_for_profile(
profile_id: str,
db: Session,
) -> dict:
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
if len(samples) == 1:
# Single sample - use directly
voice_prompt, _ = await tts_model.create_voice_prompt(
sample.audio_path,
sample.reference_text,
)
else:
# Multiple samples - combine them
combined_audio, combined_text = await tts_model.combine_voice_prompts(
[s.audio_path for s in samples],
[s.reference_text for s in samples],
)
voice_prompt, _ = await tts_model.create_voice_prompt(
combined_audio_path,
combined_text,
)
return voice_prompt
```
## Audio Validation
Reference audio is validated before being accepted:
- **Duration:** 3-30 seconds recommended
- **Format:** WAV, MP3, FLAC, OGG supported
- **Sample Rate:** Resampled to 24kHz
- **Channels:** Converted to mono if stereo
## Export/Import
Profiles can be exported as ZIP archives for sharing:
```
profile_export.zip
├── profile.json # Metadata
├── samples/
│ ├── sample_1.wav
│ └── sample_1.json # Reference text
└── ...
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/profiles` | List all profiles |
| POST | `/profiles` | Create a profile |
| GET | `/profiles/{id}` | Get profile by ID |
| PUT | `/profiles/{id}` | Update profile |
| DELETE | `/profiles/{id}` | Delete profile |
| GET | `/profiles/{id}/samples` | Get profile samples |
| POST | `/profiles/{id}/samples` | Add sample to profile |
| PUT | `/profiles/samples/{id}` | Update sample text |
| DELETE | `/profiles/samples/{id}` | Delete sample |
| GET | `/profiles/{id}/export` | Export as ZIP |
| POST | `/profiles/import` | Import from ZIP |
## Best Practices
### Sample Quality
- Use clean audio with minimal background noise
- Ensure the reference text exactly matches what is spoken
- Multiple samples (3-5) improve voice cloning quality
### Language Matching
- Set the profile language to match the reference audio
- Supported languages: en, zh, ja, ko, de, fr, ru, pt, es, it
### Naming Conventions
- Use descriptive names that identify the voice
- Avoid special characters that may cause filesystem issues
Binary file not shown.

After

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 129 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

+102
View File
@@ -0,0 +1,102 @@
{
"$schema": "https://mintlify.com/schema.json",
"name": "Voicebox",
"logo": {
"light": "/logo/icon-light.png",
"dark": "/logo/icon-dark.png"
},
"favicon": "/favicon.png",
"colors": {
"primary": "#BF9E40",
"light": "#D4B560",
"dark": "#A68A35"
},
"styles": {
"css": ["/custom.css"]
},
"anchors": [
{
"name": "Overview",
"icon": "book-open",
"url": "overview"
},
{
"name": "API",
"icon": "code",
"url": "api"
},
{
"name": "Developer",
"icon": "book-open-cover",
"url": "developer"
},
{
"name": "GitHub",
"icon": "github",
"url": "https://github.com/jamiepine/voicebox"
}
],
"navigation": [
{
"group": "Getting Started",
"icon": "rocket",
"pages": ["overview/introduction", "overview/installation", "overview/quick-start"]
},
{
"group": "Features",
"icon": "sparkles",
"pages": [
"overview/voice-cloning",
"overview/stories-editor",
"overview/recording-transcription",
"overview/generation-history",
"overview/remote-mode"
]
},
{
"group": "User Guides",
"icon": "compass",
"pages": [
"overview/creating-voice-profiles",
"overview/generating-speech",
"overview/building-stories",
"overview/troubleshooting"
]
},
{
"group": "Development",
"icon": "wrench",
"pages": [
"developer/setup",
"developer/architecture",
"developer/contributing",
"developer/building",
"developer/autoupdater"
]
},
{
"group": "API Reference",
"icon": "code",
"pages": [
"api/overview",
"api/authentication",
"api/voice-profiles",
"api/generation",
"api/recordings"
]
},
{
"group": "Architecture",
"icon": "book-open-cover",
"pages": [
"developer/voice-profiles",
"developer/tts-generation",
"developer/history",
"developer/stories",
"developer/transcription",
"developer/audio-channels",
"developer/model-management"
]
}
]
}
+11
View File
@@ -0,0 +1,11 @@
[phases.setup]
nixPkgs = ["nodejs_20", "bun"]
[phases.install]
cmds = ["bun install"]
[phases.build]
cmds = ["bun run build"]
[start]
cmd = "bun run start"
+37
View File
@@ -0,0 +1,37 @@
---
title: "Building Stories"
description: "Create multi-voice narratives with the Stories Editor"
---
## Getting Started
The Stories Editor is perfect for creating podcasts, audiobooks, and multi-speaker content.
<Steps>
<Step title="Create Story">
**Stories** → **+ New Story**
</Step>
<Step title="Add Tracks">
Create tracks for each speaker
</Step>
<Step title="Add Clips">
Generate or drag audio to tracks
</Step>
<Step title="Arrange">
Position and trim clips on timeline
</Step>
<Step title="Export">
Render final audio
</Step>
</Steps>
## Use Cases
- Multi-host podcasts
- Audiobook narration with character voices
- Game dialogue scenes
- Educational content with multiple speakers
## Coming Soon
Full timeline editor documentation will be added as features are finalized.
+296
View File
@@ -0,0 +1,296 @@
---
title: "Creating Voice Profiles"
description: "Advanced guide to creating high-quality voice profiles"
---
## Overview
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
## Quick Start
<Steps>
<Step title="Prepare Audio">
10-30 seconds of clear speech
</Step>
<Step title="Create Profile">
**Profiles** → **+ New Profile**
</Step>
<Step title="Upload Sample">
Add your audio file
</Step>
<Step title="Generate">
Use the profile to generate speech
</Step>
</Steps>
## Audio Requirements
### Ideal Sample Characteristics
<CardGroup cols={2}>
<Card title="Duration" icon="clock">
**10-30 seconds**
Too short: Poor quality
Too long: Unnecessary
</Card>
<Card title="Clarity" icon="volume">
**Clear speech**
No background noise
No music or overlapping voices
</Card>
<Card title="Quality" icon="sparkles">
**High fidelity**
44.1kHz or 48kHz sample rate
Minimal compression
</Card>
<Card title="Content" icon="microphone">
**Natural speech**
Conversational tone
Complete sentences
</Card>
</CardGroup>
### File Formats
Supported formats:
- **WAV** (recommended) - Lossless quality
- **MP3** - Acceptable, minimal compression
- **M4A** - Acceptable
- **FLAC** - Lossless alternative
<Tip>
Use WAV for best results. Avoid heavily compressed formats.
</Tip>
## Recording Tips
### Environment
<AccordionGroup>
<Accordion title="Quiet Space">
- Record in a quiet room
- Turn off fans, AC, appliances
- Close windows to reduce outside noise
- Use soft furnishings to reduce echo
</Accordion>
<Accordion title="Microphone Placement">
- 6-12 inches from mouth
- Slight angle to reduce plosives (p, b, t)
- Use a pop filter if available
- Maintain consistent distance
</Accordion>
<Accordion title="Recording Settings">
- 44.1kHz or 48kHz sample rate
- 16-bit or 24-bit depth
- Mono is fine (stereo will be converted)
- Avoid automatic gain control
</Accordion>
</AccordionGroup>
### Speaking
- **Natural pace** - Don't rush or speak too slowly
- **Clear articulation** - Pronounce words clearly
- **Consistent volume** - Maintain steady loudness
- **Normal tone** - Speak as you normally would
- **Complete sentences** - Avoid fragments or "ums"
## Multiple Samples
Adding multiple samples can significantly improve quality:
### Why Multiple Samples?
<CardGroup cols={2}>
<Card title="Robustness" icon="shield">
Model learns a more complete representation
</Card>
<Card title="Versatility" icon="palette">
Handles different speaking styles better
</Card>
<Card title="Quality" icon="star">
Reduces artifacts and improves naturalness
</Card>
<Card title="Consistency" icon="check">
More reliable across different texts
</Card>
</CardGroup>
### Sample Variety
Consider adding samples with:
1. **Different tones**
- Casual conversation
- Professional/formal
- Excited/enthusiastic
- Calm/serious
2. **Different content**
- Narratives
- Questions
- Statements
- Emotions (happy, sad, neutral)
3. **Different recording conditions**
- Studio quality
- Phone call quality (if needed)
- Room acoustics
<Warning>
All samples should be from the **same speaker**. Mixing voices will produce poor results.
</Warning>
## Processing Existing Audio
If you have existing audio (podcasts, videos, etc.):
### Extracting Clean Segments
<Steps>
<Step title="Find Clean Speech">
Look for segments with:
- Just the target speaker
- No background music
- Minimal noise
</Step>
<Step title="Use Audio Editor">
Tools like Audacity or Adobe Audition:
- Cut out clean 10-30s segments
- Remove silence at start/end
- Normalize volume if needed
</Step>
<Step title="Export as WAV">
Save as high-quality WAV file
</Step>
</Steps>
### Noise Reduction
If you have light background noise:
```
1. Use noise reduction in Audacity:
- Select noise-only section
- Get Noise Profile
- Select full audio
- Apply noise reduction (gentle settings)
2. Avoid over-processing:
- Can introduce artifacts
- May reduce voice quality
```
## Testing & Iteration
### Test Your Profile
After creating a profile:
<Steps>
<Step title="Generate Test">
Generate a simple phrase:
```
"Hello, this is a test of my voice profile."
```
</Step>
<Step title="Evaluate Quality">
Listen for:
- Natural tone
- Clear pronunciation
- Proper prosody
- Lack of artifacts
</Step>
<Step title="Iterate">
If quality is poor:
- Add more samples
- Try different source audio
- Check sample quality
</Step>
</Steps>
### Common Issues
<AccordionGroup>
<Accordion title="Robotic Voice">
**Cause**: Poor quality samples or too short
**Fix**: Use longer, higher quality samples
</Accordion>
<Accordion title="Wrong Tone">
**Cause**: Sample tone doesn't match desired output
**Fix**: Record samples in the style you want to generate
</Accordion>
<Accordion title="Artifacts/Glitches">
**Cause**: Background noise or audio issues in samples
**Fix**: Clean up samples or re-record in quieter environment
</Accordion>
</AccordionGroup>
## Advanced Tips
### Celebrity/Character Voices
For cloning public figures or characters:
1. **Legal considerations** - Ensure you have rights or it's fair use
2. **Source quality** - Find high-quality interview audio or clean clips
3. **Consistency** - Use clips where they speak similarly
4. **Multiple samples** - Very important for recognizable voices
### Accent & Dialect
The model will preserve accent and dialect:
- British English will generate British English
- Southern accent will produce Southern accent
- Regional pronunciations will be maintained
### Emotion Transfer
The emotional tone of samples affects generation:
- Energetic samples → Energetic output
- Calm samples → Calm output
- Mix samples for versatile profile
## Managing Profiles
### Organization
- **Descriptive names** - "John Smith - Professional Narrator"
- **Add descriptions** - Note recording conditions, use cases
- **Language tags** - Mark the primary language
- **Archive unused** - Keep profile list manageable
### Export/Import
- **Export** profiles to share or backup
- **Import** from colleagues or teammates
- Profiles include voice embeddings, not original audio
## Next Steps
<CardGroup cols={2}>
<Card title="Generate Speech" icon="waveform" href="/guides/generating-speech">
Use your profile to generate speech
</Card>
<Card title="Build Stories" icon="film" href="/guides/building-stories">
Create multi-voice narratives
</Card>
</CardGroup>
+65
View File
@@ -0,0 +1,65 @@
---
title: "Generating Speech"
description: "Generate high-quality speech from text"
---
## Basic Generation
<Steps>
<Step title="Select Profile">
Choose a voice profile from the dropdown
</Step>
<Step title="Enter Text">
Type or paste your text
</Step>
<Step title="Generate">
Click **Generate** and wait a few seconds
</Step>
<Step title="Play & Export">
Preview and download the result
</Step>
</Steps>
## Text Formatting Tips
The way you format text affects the output quality.
### Punctuation
Use proper punctuation for natural pauses:
```
Good: "Hello! How are you today? I'm doing great."
Bad: "Hello how are you today Im doing great"
```
### Emphasis
Use formatting to suggest emphasis:
```
- ALL CAPS for louder/emphasized: "That was AMAZING!"
- Italics for subtle emphasis: "I *really* enjoyed that"
- Bold for strong emphasis: "This is **very** important"
```
<Note>
The model interprets these hints but results may vary.
</Note>
## Advanced Features
### Batch Generation
For long-form content, split into smaller chunks for better control and faster processing.
### Voice Caching
Voicebox caches voice prompts for faster re-generation with the same profile.
## Coming Soon
- Real-time streaming
- Word-level timing control
- Emotion and style controls
- SSML support
+88
View File
@@ -0,0 +1,88 @@
---
title: "Generation History"
description: "Track and manage all your generated audio"
---
## Overview
Voicebox keeps a complete history of all generated audio, making it easy to find, reuse, and manage your creations.
## Features
<CardGroup cols={2}>
<Card title="Full History" icon="clock">
Every generation is automatically saved
</Card>
<Card title="Search & Filter" icon="search">
Find by text, voice, or date
</Card>
<Card title="Re-generate" icon="rotate">
Regenerate any past generation with one click
</Card>
<Card title="Export" icon="download">
Download individual or batch exports
</Card>
</CardGroup>
## Viewing History
Navigate to the **History** tab to see all your generations.
Each entry shows:
- Generated text
- Voice profile used
- Timestamp
- Audio duration
- Language
## Actions
### Play
Click any generation to play it immediately.
### Re-generate
Regenerate with the same settings or modify the text/voice.
### Download
Export as WAV, MP3, or M4A.
### Delete
Remove unwanted generations to free up space.
### Add to Story
Drag generations to the Stories Editor timeline.
## Search & Filter
<Tabs>
<Tab title="By Text">
Search for specific text content
```
"Hello world"
```
</Tab>
<Tab title="By Voice">
Filter by voice profile
```
Select from dropdown
```
</Tab>
<Tab title="By Date">
Filter by date range
```
Last 7 days, Last 30 days, Custom range
```
</Tab>
</Tabs>
## Storage
History is stored locally:
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
- **Linux**: `~/.config/com.voicebox.app/data/`
<Warning>
Deleting the data directory will remove all history. Export important files first.
</Warning>
+119
View File
@@ -0,0 +1,119 @@
---
title: "Installation"
description: "Download and install Voicebox on macOS, Windows, or Linux"
---
## Download
Voicebox is available for macOS and Windows, with Linux builds coming soon.
<CardGroup cols={2}>
<Card title="macOS" icon="apple">
Download for Apple Silicon or Intel Macs
</Card>
<Card title="Windows" icon="windows">
Download MSI installer or Setup executable
</Card>
</CardGroup>
### macOS
<Tabs>
<Tab title="Apple Silicon">
Download: [voicebox_aarch64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_aarch64.app.tar.gz)
```bash
# Extract the archive
tar -xzf voicebox_aarch64.app.tar.gz
# Move to Applications
mv Voicebox.app /Applications/
```
</Tab>
<Tab title="Intel">
Download: [voicebox_x64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64.app.tar.gz)
```bash
# Extract the archive
tar -xzf voicebox_x64.app.tar.gz
# Move to Applications
mv Voicebox.app /Applications/
```
</Tab>
</Tabs>
### Windows
<Tabs>
<Tab title="MSI Installer">
Download: [voicebox_x64_en-US.msi](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64_en-US.msi)
Double-click the MSI file and follow the installation wizard.
</Tab>
<Tab title="Setup Executable">
Download: [voicebox_x64-setup.exe](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64-setup.exe)
Run the executable and follow the installation wizard.
</Tab>
</Tabs>
### Linux
<Note>
Linux builds are coming soon. Currently blocked by GitHub runner disk space limitations.
</Note>
## First Launch
When you launch Voicebox for the first time:
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
2. **Data Directory** — Voice profiles and generated audio are stored in:
- macOS: `~/Library/Application Support/com.voicebox.app/`
- Windows: `%APPDATA%/com.voicebox.app/`
- Linux: `~/.config/com.voicebox.app/`
3. **Backend Server** — The bundled Python server starts automatically
<Tip>
First generation will be slower due to model downloads. Subsequent runs use cached models.
</Tip>
## System Requirements
### Minimum
- **OS:** macOS 11+, Windows 10+, or Linux
- **RAM:** 8GB
- **Storage:** 5GB free space (for models and data)
- **CPU:** Modern multi-core processor
### Recommended
- **RAM:** 16GB+
- **GPU:** CUDA-capable NVIDIA GPU (for faster generation)
- **Storage:** 10GB+ free space
<Note>
CPU inference is supported but significantly slower than GPU. A CUDA-capable GPU is highly recommended for real-time workflows.
</Note>
## Verification
After installation, verify everything works:
1. Launch Voicebox
2. Check the server status indicator in the bottom-left corner (should be green)
3. Navigate to **Profiles** and create a test profile
4. Generate a short audio clip to verify the TTS engine works
<Check>
If you see a green status indicator and can generate audio, you're all set!
</Check>
## Next Steps
<Card title="Quick Start Guide" icon="rocket" href="/overview/quick-start">
Create your first voice profile and generate speech
</Card>
+58
View File
@@ -0,0 +1,58 @@
---
title: "Introduction"
description: "Welcome to Voicebox - the open-source voice synthesis studio"
---
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
<Frame>
<img src="/images/app-screenshot-1.webp" alt="Voicebox App Screenshot" />
</Frame>
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
- **Complete privacy** — models and voice data stay on your machine
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
- **Native performance** — built with Tauri (Rust), not Electron
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
## Key Features
<CardGroup cols={2}>
<Card title="Voice Cloning" icon="microphone">
Instant cloning from just a few seconds of audio with Qwen3-TTS
</Card>
<Card title="Stories Editor" icon="film">
Multi-track timeline for creating conversations and narratives
</Card>
<Card title="Full API" icon="code">
REST API for integrating voice synthesis into your apps
</Card>
<Card title="Local-First" icon="shield">
Everything runs on your machine - complete privacy
</Card>
</CardGroup>
## Use Cases
- **Game Development** — Generate dynamic dialogue for characters
- **Content Creation** — Produce podcasts and video voiceovers
- **Accessibility** — Build text-to-speech tools
- **Voice Assistants** — Create custom voice interfaces
- **Production Pipelines** — Automate voiceover workflows
## Next Steps
<CardGroup cols={2}>
<Card title="Installation" icon="download" href="/overview/installation">
Download and install Voicebox on your machine
</Card>
<Card title="Quick Start" icon="rocket" href="/overview/quick-start">
Get up and running in 5 minutes
</Card>
</CardGroup>
+154
View File
@@ -0,0 +1,154 @@
---
title: "Quick Start"
description: "Get started with Voicebox in 5 minutes"
---
This guide will walk you through creating your first voice profile and generating speech.
## Prerequisites
Make sure you have [installed Voicebox](/overview/installation) and launched the app.
## Step 1: Create a Voice Profile
Voice profiles are the foundation of Voicebox. Each profile contains voice samples that the AI uses to clone the voice.
<Steps>
<Step title="Navigate to Profiles">
Click the **Profiles** tab in the sidebar
</Step>
<Step title="Create New Profile">
Click the **+ New Profile** button
Fill in the details:
- **Name:** A descriptive name (e.g., "John Smith")
- **Language:** Select the primary language
- **Description:** Optional notes about the voice
</Step>
<Step title="Add Voice Sample">
You have two options:
**Option A: Upload Audio**
- Click **Upload Sample**
- Select an audio file (WAV, MP3, or M4A)
- Ideal length: 10-30 seconds of clear speech
**Option B: Record Live**
- Click **Record Sample**
- Speak clearly for 10-30 seconds
- Click stop when finished
</Step>
<Step title="Save Profile">
Click **Create Profile** to save
</Step>
</Steps>
<Tip>
For best results, use clean audio with minimal background noise and consistent speaking tone.
</Tip>
## Step 2: Generate Speech
Now let's use your new voice profile to generate speech.
<Steps>
<Step title="Go to Generation">
Click the **Generate** tab in the sidebar
</Step>
<Step title="Select Voice Profile">
Choose your newly created profile from the dropdown
</Step>
<Step title="Enter Text">
Type or paste the text you want to generate:
```
Hello! This is my first voice generation with Voicebox.
```
</Step>
<Step title="Generate">
Click **Generate** and wait a few seconds
<Note>
First generation may take longer due to model initialization. Subsequent generations will be faster.
</Note>
</Step>
<Step title="Play & Download">
- Click **Play** to preview the audio
- Click **Download** to save the audio file
- The generation is also saved to your **History**
</Step>
</Steps>
## Step 3: Build a Story (Optional)
The Stories Editor lets you create multi-voice narratives with a timeline-based interface.
<Steps>
<Step title="Create New Story">
Navigate to **Stories** and click **+ New Story**
</Step>
<Step title="Add Voice Tracks">
Click **+ Add Track** to create tracks for different speakers
</Step>
<Step title="Add Audio Clips">
- Drag generated audio from your History
- Or generate new clips directly in the timeline
- Arrange clips on the timeline
</Step>
<Step title="Edit & Export">
- Trim clips by dragging edges
- Adjust timing and spacing
- Click **Export** to render the final audio
</Step>
</Steps>
## What's Next?
<CardGroup cols={2}>
<Card title="Voice Cloning Guide" icon="microphone" href="/guides/creating-voice-profiles">
Learn advanced techniques for high-quality voice cloning
</Card>
<Card title="API Integration" icon="code" href="/api/overview">
Integrate Voicebox into your own applications
</Card>
<Card title="Stories Editor" icon="film" href="/overview/stories-editor">
Master the multi-track timeline editor
</Card>
<Card title="Remote Mode" icon="server" href="/overview/remote-mode">
Connect to a GPU server for faster generation
</Card>
</CardGroup>
## Tips for Success
<AccordionGroup>
<Accordion title="Getting the Best Voice Quality">
- Use 10-30 seconds of clear, consistent speech
- Avoid background noise and echo
- Multiple samples from the same speaker improve quality
- Match the speaking style you want to generate
</Accordion>
<Accordion title="Improving Generation Speed">
- Use a CUDA-capable GPU for 5-10x faster generation
- Enable voice prompt caching for repeated generations
- Consider running the backend on a remote GPU server
</Accordion>
<Accordion title="Troubleshooting Common Issues">
- **Server won't start:** Check if port 17493 is available
- **Poor audio quality:** Try adding more voice samples
- **Slow generation:** Verify GPU acceleration is enabled
- See the full [Troubleshooting Guide](/guides/troubleshooting) for more
</Accordion>
</AccordionGroup>
+64
View File
@@ -0,0 +1,64 @@
---
title: "Recording & Transcription"
description: "Record audio and transcribe speech with Whisper"
---
## Recording
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
### Features
- **Microphone input** - Record from any audio input device
- **System audio capture** - Record desktop audio (macOS/Windows)
- **Waveform visualization** - See audio levels in real-time
- **Multiple formats** - Export as WAV, MP3, or M4A
### How to Record
<Steps>
<Step title="Select Input">
Choose your microphone or system audio
</Step>
<Step title="Start Recording">
Click the record button and speak clearly
</Step>
<Step title="Stop & Save">
Click stop when finished
</Step>
<Step title="Use or Export">
Use as voice sample or export to file
</Step>
</Steps>
## Transcription
Automatic speech-to-text powered by OpenAI's Whisper model.
### Features
- **High accuracy** - Industry-leading speech recognition
- **Multiple languages** - Supports 50+ languages
- **Automatic detection** - Language auto-detection
- **Timestamps** - Word-level timing information
### How to Transcribe
<Steps>
<Step title="Select Audio">
Choose a recording or upload an audio file
</Step>
<Step title="Choose Language">
Select language or use auto-detect
</Step>
<Step title="Transcribe">
Click transcribe and wait for processing
</Step>
<Step title="Review & Export">
Review text and export as needed
</Step>
</Steps>
<Tip>
Transcription is useful for creating voice samples from existing audio or generating subtitles.
</Tip>
+138
View File
@@ -0,0 +1,138 @@
---
title: "Remote Mode"
description: "Connect to a GPU server for faster generation"
---
## Overview
Remote Mode allows you to run the Voicebox backend on a separate machine (like a GPU server) while using the desktop app on your local machine.
## Use Cases
- **No local GPU** - Use a cloud GPU or remote workstation
- **Faster generation** - Leverage powerful remote hardware
- **Shared infrastructure** - Multiple users connect to one server
- **Laptop workflows** - Keep your laptop cool and battery-efficient
## Architecture
In Remote Mode, the Voicebox desktop app (running on your local machine) communicates with the backend server (running on a remote machine) via HTTP. The local app provides only the user interface, while the remote server handles all the heavy processing including the TTS models, API endpoints, and audio generation.
## Setting Up Remote Mode
### On the Server
<Steps>
<Step title="Install Dependencies">
```bash
# Clone the repo
git clone https://github.com/jamiepine/voicebox.git
cd voicebox/backend
# Install Python dependencies
pip install -r requirements.txt
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Step>
<Step title="Start the Server">
```bash
# Allow external connections
uvicorn main:app --host 0.0.0.0 --port 17493
```
<Warning>
This exposes the server to your network. Use a firewall or VPN for security.
</Warning>
</Step>
<Step title="Open Firewall">
```bash
# Ubuntu/Debian
sudo ufw allow 17493
# Or use your cloud provider's firewall settings
```
</Step>
</Steps>
### On the Client
<Steps>
<Step title="Open Settings">
In Voicebox, go to **Settings → Server**
</Step>
<Step title="Enable Remote Mode">
Toggle **Use Remote Server**
</Step>
<Step title="Enter Server URL">
```
http://<server-ip>:17493
```
Replace `<server-ip>` with your server's IP address
</Step>
<Step title="Test Connection">
Click **Test Connection** to verify
</Step>
</Steps>
## Cloud Deployment
### AWS EC2
```bash
# Launch a GPU instance (e.g., g4dn.xlarge)
# Install dependencies
# Start server with --host 0.0.0.0
```
### Vast.ai
```bash
# Rent a GPU instance
# SSH in and clone repo
# Start server
```
### RunPod
```bash
# Deploy a pod with CUDA support
# Install Voicebox backend
# Expose port 17493
```
## Security Considerations
<Warning>
The API currently has no authentication. Only use on trusted networks or with a VPN.
</Warning>
**Best Practices:**
- Use a VPN (WireGuard, Tailscale) instead of exposing to the internet
- Run behind a reverse proxy with authentication (nginx + basic auth)
- Use HTTPS with SSL certificates
- Firewall rules to limit access to specific IPs
## Performance
Expected performance on various GPUs:
| GPU | Generation Speed |
|-----|------------------|
| RTX 4090 | ~2-3s per 10 words |
| RTX 3090 | ~3-4s per 10 words |
| RTX 3060 | ~5-7s per 10 words |
| CPU (12-core) | ~20-30s per 10 words |
<Tip>
A GPU with 8GB+ VRAM is recommended for best performance.
</Tip>
## Troubleshooting
See the [Troubleshooting Guide](/guides/troubleshooting#remote-mode-issues) for common remote mode issues.
+64
View File
@@ -0,0 +1,64 @@
---
title: "Stories Editor"
description: "Create multi-voice narratives with a timeline-based editor"
---
## Overview
The Stories Editor is a DAW-like timeline interface for creating multi-voice narratives, podcasts, and conversations.
## Features
<CardGroup cols={2}>
<Card title="Multi-Track Timeline" icon="timeline">
Arrange multiple voice tracks in parallel
</Card>
<Card title="Inline Editing" icon="scissors">
Trim and split clips directly in the timeline
</Card>
<Card title="Auto-Playback" icon="play">
Preview with synchronized playhead
</Card>
<Card title="Voice Mixing" icon="users">
Build conversations with multiple speakers
</Card>
</CardGroup>
## Creating a Story
<Steps>
<Step title="Create New Story">
Navigate to **Stories** and click **+ New Story**
</Step>
<Step title="Add Tracks">
Create separate tracks for each voice/speaker
</Step>
<Step title="Add Clips">
- Drag from generation history
- Generate new clips inline
- Upload audio files
</Step>
<Step title="Arrange & Edit">
- Position clips on timeline
- Trim clip edges
- Adjust spacing and timing
</Step>
<Step title="Export">
Render the final mixed audio
</Step>
</Steps>
## Use Cases
- **Podcasts**: Multi-host conversations
- **Audiobooks**: Narrator + character voices
- **Game Dialogue**: Character interactions
- **Video Voiceovers**: Multiple speakers
- **Audio Drama**: Full voice casts
## Coming Soon
- Word-level editing
- Crossfades and transitions
- Audio effects (reverb, EQ)
- Real-time collaboration
+477
View File
@@ -0,0 +1,477 @@
---
title: "Troubleshooting"
description: "Common issues and solutions for Voicebox"
---
This guide covers common issues you might encounter when using or developing Voicebox, along with solutions.
## Installation Issues
### macOS: "App is damaged and can't be opened"
This occurs because the app isn't signed with an Apple Developer certificate.
**Solution:**
```bash
# Remove the quarantine attribute
xattr -cr /Applications/Voicebox.app
```
### Windows: SmartScreen Warning
Windows SmartScreen may warn that the app is unrecognized.
**Solution:**
- Click "More info"
- Click "Run anyway"
<Note>
This is expected for unsigned applications. We're working on code signing for future releases.
</Note>
## Server Issues
### Backend Server Won't Start
**Symptoms:**
- Red status indicator in bottom-left corner
- "Failed to connect to server" error
**Solutions:**
<AccordionGroup>
<Accordion title="Port Already in Use">
Check if port 17493 is already in use:
```bash
# macOS/Linux
lsof -i :17493
# Windows
netstat -ano | findstr :17493
```
Kill the process using the port:
```bash
# macOS/Linux
kill -9 <PID>
# Windows
taskkill /PID <PID> /F
```
</Accordion>
<Accordion title="Permission Issues">
The server binary might not have execute permissions:
```bash
# macOS/Linux
chmod +x ~/Library/Application\ Support/com.voicebox.app/backend/voicebox-server
```
</Accordion>
<Accordion title="Check Logs">
View server logs for errors:
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
### Connection Timeout
**Symptoms:**
- Long loading times
- "Connection timeout" errors
**Solution:**
- Restart the app
- Check your firewall settings
- Ensure localhost is accessible
## Generation Issues
### First Generation is Very Slow
**Symptoms:**
- First generation takes 2-5 minutes
- Progress indicator stuck at "Loading model..."
**Explanation:**
This is expected behavior. The first generation downloads the Qwen3-TTS model (~2-4GB) and initializes it.
**Solution:**
- Wait for the initial download to complete
- Subsequent generations will be much faster
- Check your internet connection
### Poor Voice Quality
**Symptoms:**
- Robotic or unnatural voice
- Missing emotion or prosody
- Pronunciation errors
**Solutions:**
<Steps>
<Step title="Improve Voice Samples">
- Use 10-30 seconds of clear audio
- Avoid background noise
- Ensure consistent speaking tone
- Add multiple samples from the same speaker
</Step>
<Step title="Match Speaking Style">
The generated voice will mimic the tone and style of your samples. If your sample is monotone, the generation will be too.
</Step>
<Step title="Adjust Text Formatting">
- Use proper punctuation
- Add commas for natural pauses
- Capitalize proper nouns
</Step>
</Steps>
### Generation Fails with "Out of Memory"
**Symptoms:**
- Generation crashes
- "CUDA out of memory" or "RuntimeError: out of memory"
**Solutions:**
<AccordionGroup>
<Accordion title="Free GPU Memory">
Close other GPU-intensive applications:
- Games
- Video editors
- Multiple browser tabs with WebGL
Then restart Voicebox.
</Accordion>
<Accordion title="Use CPU Mode">
If your GPU doesn't have enough VRAM (need 6GB+), use CPU mode:
Settings → Generation → Use CPU instead of GPU
<Warning>
CPU generation is 5-10x slower but uses system RAM instead of VRAM.
</Warning>
</Accordion>
<Accordion title="Reduce Batch Size">
For long text, split it into smaller chunks instead of generating all at once.
</Accordion>
</AccordionGroup>
## Audio Issues
### No Audio Playback
**Symptoms:**
- Generated audio won't play
- Playback button doesn't respond
**Solutions:**
- Check system audio settings
- Ensure audio output device is connected
- Try exporting and playing in a media player
### Crackling or Distorted Audio
**Symptoms:**
- Audio has static or distortion
- Clipping sounds
**Solutions:**
- Check if your input samples have distortion
- Reduce playback volume
- Re-generate with cleaner voice samples
## Development Issues
### Backend Won't Start in Dev Mode
**Symptoms:**
- `bun run dev:server` fails
- Import errors or module not found
**Solutions:**
<AccordionGroup>
<Accordion title="Python Version">
Ensure Python 3.11 or higher:
```bash
python --version
```
If not, install Python 3.11+ and recreate the virtual environment.
</Accordion>
<Accordion title="Virtual Environment">
Ensure venv is activated:
```bash
# macOS/Linux
source backend/venv/bin/activate
# Windows
backend\venv\Scripts\activate
```
You should see `(venv)` in your prompt.
</Accordion>
<Accordion title="Dependencies">
Reinstall dependencies:
```bash
cd backend
pip install -r requirements.txt
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Accordion>
</AccordionGroup>
### Tauri Build Fails
**Symptoms:**
- `bun run tauri build` fails
- Rust compilation errors
**Solutions:**
```bash
# Clean build artifacts
cd tauri/src-tauri
cargo clean
# Update Rust
rustup update
# Try building again
cd ../..
bun run tauri build
```
### OpenAPI Client Generation Fails
**Symptoms:**
- `./scripts/generate-api.sh` fails
- "Failed to fetch schema" error
**Solutions:**
<Steps>
<Step title="Ensure Backend is Running">
```bash
curl http://localhost:17493/openapi.json
```
Should return JSON. If not, start the backend.
</Step>
<Step title="Check Port">
Ensure nothing else is using port 17493
</Step>
<Step title="Regenerate Manually">
```bash
cd backend
source venv/bin/activate
uvicorn main:app --reload --port 17493
# In another terminal
./scripts/generate-api.sh
```
</Step>
</Steps>
## Database Issues
### "Database is locked" Error
**Symptoms:**
- Profile or generation operations fail
- SQLite lock errors
**Solutions:**
- Close all Voicebox instances
- Delete the lock file:
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-wal
```
### Corrupted Database
**Symptoms:**
- App crashes on launch
- Data missing or corrupted
**Solutions:**
<Warning>
This will delete all your voice profiles and generation history. Export important profiles first if possible.
</Warning>
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db
# Windows
del %APPDATA%\com.voicebox.app\data\voicebox.db
```
Restart the app to create a fresh database.
## Model Issues
### Model Download Fails
**Symptoms:**
- "Failed to download model" error
- Stuck at "Downloading..."
**Solutions:**
- Check your internet connection
- Check HuggingFace Hub status
- Try using a VPN if HuggingFace is blocked in your region
- Manually download and place in cache directory
### Wrong Model Version
**Symptoms:**
- Generation quality suddenly degraded
- Different voice output
**Solutions:**
Clear the model cache and re-download:
```bash
# macOS
rm -rf ~/.cache/huggingface/hub/models--Qwen*
# Windows
rmdir /s %USERPROFILE%\.cache\huggingface\hub\models--Qwen*
```
## Performance Issues
### Slow Generation on GPU
**Symptoms:**
- Generation slower than expected
- GPU not being utilized
**Solutions:**
<AccordionGroup>
<Accordion title="Verify CUDA Installation">
```bash
nvidia-smi
```
Should show your GPU. If not, install CUDA drivers.
</Accordion>
<Accordion title="Check GPU Selection">
If you have multiple GPUs, ensure Voicebox is using the right one.
Settings → Generation → GPU Device
</Accordion>
<Accordion title="Update GPU Drivers">
Outdated drivers can cause performance issues. Update to the latest NVIDIA drivers.
</Accordion>
</AccordionGroup>
### High Memory Usage
**Symptoms:**
- App uses excessive RAM
- System becomes sluggish
**Solutions:**
- Close unused voice profiles
- Clear generation history
- Restart the app periodically
## Remote Mode Issues
### Can't Connect to Remote Server
**Symptoms:**
- "Connection refused" error
- Remote server not found
**Solutions:**
<Steps>
<Step title="Check Server Status">
Ensure the remote server is running:
```bash
curl http://<server-ip>:17493/health
```
</Step>
<Step title="Check Firewall">
Ensure port 17493 is open on the remote server:
```bash
# Allow port on Ubuntu/Debian
sudo ufw allow 17493
```
</Step>
<Step title="Verify Network">
- Ensure both machines are on the same network (for local servers)
- Use IP address instead of hostname
- Try pinging the server: `ping <server-ip>`
</Step>
</Steps>
## Still Having Issues?
If you're still experiencing problems:
1. **Check GitHub Issues:** [github.com/jamiepine/voicebox/issues](https://github.com/jamiepine/voicebox/issues)
2. **Open a New Issue:** Provide:
- Operating system and version
- Voicebox version
- Steps to reproduce
- Error messages or logs
3. **Join Discord:** [discord.gg/voicebox](https://discord.gg/voicebox) (coming soon)
## Diagnostic Information
When reporting issues, include this information:
```bash
# Voicebox version
# Check Help → About in the app
# Operating system
uname -a # macOS/Linux
systeminfo # Windows
# Python version (for dev issues)
python --version
# GPU info (if generation issues)
nvidia-smi # NVIDIA GPUs
```
For more detailed troubleshooting, see the [TROUBLESHOOTING.md](https://github.com/jamiepine/voicebox/blob/main/docs/TROUBLESHOOTING.md) file in the repository.
+75
View File
@@ -0,0 +1,75 @@
---
title: "Voice Cloning"
description: "Clone any voice from just a few seconds of audio"
---
## Overview
Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning from just a few seconds of audio. The model captures prosody, emotion, and natural cadence.
## How It Works
<Steps>
<Step title="Upload or Record Sample">
Provide 10-30 seconds of clear speech from the target voice
</Step>
<Step title="Model Analysis">
Qwen3-TTS analyzes vocal characteristics, tone, and speaking patterns
</Step>
<Step title="Voice Profile Created">
The model generates a voice embedding for synthesis
</Step>
<Step title="Generate Speech">
Use the profile to generate any text in the cloned voice
</Step>
</Steps>
## Best Practices
### Sample Quality
<CardGroup cols={2}>
<Card title="Do" icon="check">
- Use 10-30 seconds of audio
- Clear, consistent speaking
- Minimal background noise
- Natural speaking pace
</Card>
<Card title="Don't" icon="xmark">
- Very short clips (< 5 seconds)
- Heavy background noise
- Music or overlapping voices
- Heavily processed audio
</Card>
</CardGroup>
### Multiple Samples
Adding multiple samples from the same speaker can improve quality:
- Different speaking styles (casual, formal)
- Different emotions (happy, serious)
- Different recording conditions
<Tip>
The model will learn a more robust representation from diverse samples.
</Tip>
## Supported Languages
Currently supported:
- English
- Chinese (Mandarin)
More languages coming soon.
## Limitations
<Warning>
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice.
</Warning>
- Quality depends on sample clarity
- Works best with consistent speaking tone
- May struggle with extreme accents or speech impediments
- Background noise reduces quality
+17
View File
@@ -0,0 +1,17 @@
{
"name": "voicebox-docs",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "mintlify dev",
"build": "mintlify build",
"start": "mintlify serve",
"install:mintlify": "bun add -g mintlify"
},
"devDependencies": {
"mintlify": "latest"
},
"engines": {
"bun": ">=1.0.0"
}
}
+758
View File
@@ -0,0 +1,758 @@
# Docker Deployment Guide
**Status:** In Development for v0.2.0
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
## Overview
Docker support makes Voicebox easier to deploy, especially for:
- **Consistent Environments**: Same setup across dev/staging/prod
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
- **Server Deployments**: Run on headless Linux servers
- **Multi-User Setups**: Isolate instances per user/team
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
## Quick Start
### Using Pre-Built Images (Recommended)
```bash
# CPU-only version
docker run -p 8000:8000 -v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest
# NVIDIA GPU version
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest-cuda
# AMD GPU version (experimental)
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
-v voicebox-data:/app/data \
ghcr.io/jamiepine/voicebox:latest-rocm
```
Then open: `http://localhost:8000`
### Using Docker Compose (Easiest)
Create `docker-compose.yml`:
```yaml
version: '3.8'
services:
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
ports:
- "8000:8000"
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
- TTS_MODE=local
- WHISPER_MODE=local
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
voicebox-data:
huggingface-cache:
```
Run:
```bash
docker compose up -d
```
## Building From Source
### Basic Dockerfile
```dockerfile
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
build-essential \
ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Copy application
COPY backend/ /app/backend/
COPY requirements.txt /app/
# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
# Create data directory
RUN mkdir -p /app/data
# Expose port
EXPOSE 8000
# Run server
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Build and run:
```bash
docker build -t voicebox .
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
```
### Multi-Stage Build (Optimized)
Smaller image size by separating build and runtime:
```dockerfile
# Dockerfile.optimized
# Stage 1: Build dependencies
FROM python:3.11-slim AS builder
WORKDIR /build
RUN apt-get update && apt-get install -y \
git build-essential && \
rm -rf /var/lib/apt/lists/*
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --target=/build/packages \
-r requirements.txt
RUN pip install --no-cache-dir --target=/build/packages \
git+https://github.com/QwenLM/Qwen3-TTS.git
# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
# Install only runtime dependencies
RUN apt-get update && apt-get install -y \
ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Copy installed packages from builder
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
# Copy application code
COPY backend/ /app/backend/
# Create data directory
RUN mkdir -p /app/data
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Build:
```bash
docker build -f Dockerfile.optimized -t voicebox:slim .
```
## GPU Support
### NVIDIA GPUs (CUDA)
**Dockerfile:**
```dockerfile
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
# Install Python
RUN apt-get update && apt-get install -y \
python3.11 python3-pip git ffmpeg && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Install PyTorch with CUDA support
COPY backend/requirements.txt .
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
# Install other dependencies
RUN pip3 install -r requirements.txt
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
COPY backend/ /app/backend/
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
**Run with GPU:**
```bash
docker run --gpus all -p 8000:8000 \
-v voicebox-data:/app/data \
voicebox:cuda
```
**Docker Compose with GPU:**
```yaml
services:
voicebox:
image: voicebox:cuda
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
```
### AMD GPUs (ROCm) - Experimental
**Dockerfile:**
```dockerfile
FROM rocm/dev-ubuntu-22.04:6.0
# Install Python
RUN apt-get update && apt-get install -y \
python3.11 python3-pip git ffmpeg && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Install PyTorch with ROCm support
COPY backend/requirements.txt .
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
# Install other dependencies
RUN pip3 install -r requirements.txt
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
# Set ROCm environment variables
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
ENV ROCM_PATH=/opt/rocm
COPY backend/ /app/backend/
EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
**Run with AMD GPU:**
```bash
docker run --device=/dev/kfd --device=/dev/dri \
--group-add video --ipc=host --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-p 8000:8000 -v voicebox-data:/app/data \
voicebox:rocm
```
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
## Volume Mounts
### Essential Volumes
```bash
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
-p 8000:8000 voicebox
```
### Development Volume Mounts
For development with hot-reload:
```bash
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
-v voicebox-data:/app/data \
-e RELOAD=true \
-p 8000:8000 voicebox
```
### Custom Model Storage
Use external model directory:
```bash
docker run -v /path/to/models:/models \
-e MODELS_DIR=/models \
-v voicebox-data:/app/data \
-p 8000:8000 voicebox
```
## Environment Variables
Configure Voicebox via environment variables:
```bash
docker run -e TTS_MODE=local \
-e WHISPER_MODE=openai-api \
-e OPENAI_API_KEY=sk-... \
-e GPU_MEMORY_FRACTION=0.8 \
-e LOG_LEVEL=info \
-p 8000:8000 voicebox
```
### Available Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
| `MODELS_DIR` | `/app/models` | Directory for local models |
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
| `RELOAD` | `false` | Enable hot-reload for development |
## Complete Docker Compose Examples
### Production Deployment
```yaml
# docker-compose.prod.yml
version: '3.8'
services:
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
container_name: voicebox
restart: unless-stopped
ports:
- "8000:8000"
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- TTS_MODE=local
- WHISPER_MODE=local
- GPU_MEMORY_FRACTION=0.8
- LOG_LEVEL=info
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
volumes:
voicebox-data:
driver: local
huggingface-cache:
driver: local
```
Run:
```bash
docker compose -f docker-compose.prod.yml up -d
```
### Development Setup
```yaml
# docker-compose.dev.yml
version: '3.8'
services:
voicebox:
build:
context: .
dockerfile: Dockerfile
ports:
- "8000:8000"
volumes:
- ./backend:/app/backend:ro
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- RELOAD=true
- LOG_LEVEL=debug
- TTS_MODE=local
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
volumes:
voicebox-data:
huggingface-cache:
```
### Multi-Service Stack
Full stack with reverse proxy and monitoring:
```yaml
# docker-compose.stack.yml
version: '3.8'
services:
# Main Voicebox app
voicebox:
image: ghcr.io/jamiepine/voicebox:latest-cuda
restart: unless-stopped
volumes:
- voicebox-data:/app/data
- huggingface-cache:/root/.cache/huggingface
environment:
- TTS_MODE=local
- WHISPER_MODE=local
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
# Nginx reverse proxy
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- voicebox
# Prometheus monitoring (optional)
prometheus:
image: prom/prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus
volumes:
voicebox-data:
huggingface-cache:
prometheus-data:
```
## Cloud Deployment
### AWS EC2
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
2. **Install Docker + nvidia-docker:**
```bash
# Amazon Linux 2
sudo yum install -y docker
sudo systemctl start docker
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
```
3. **Deploy:**
```bash
docker run --gpus all -d -p 80:8000 \
-v voicebox-data:/app/data \
--restart unless-stopped \
ghcr.io/jamiepine/voicebox:latest-cuda
```
### DigitalOcean
Use GPU Droplet + Docker:
```bash
# Create droplet via CLI
doctl compute droplet create voicebox \
--size gpu-h100x1-80gb \
--image ubuntu-22-04-x64 \
--region nyc3
# SSH and deploy
ssh root@<droplet-ip>
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
docker run --gpus all -d -p 80:8000 voicebox:cuda
```
### Google Cloud Run (CPU-only)
```bash
# Build and push
docker build -t gcr.io/your-project/voicebox .
docker push gcr.io/your-project/voicebox
# Deploy to Cloud Run
gcloud run deploy voicebox \
--image gcr.io/your-project/voicebox \
--platform managed \
--region us-central1 \
--memory 4Gi \
--cpu 2 \
--port 8000
```
### Fly.io
Create `fly.toml`:
```toml
app = "voicebox"
[build]
image = "ghcr.io/jamiepine/voicebox:latest"
[[services]]
http_checks = []
internal_port = 8000
protocol = "tcp"
[[services.ports]]
port = 80
handlers = ["http"]
[[services.ports]]
port = 443
handlers = ["tls", "http"]
[mounts]
source = "voicebox_data"
destination = "/app/data"
```
Deploy:
```bash
fly launch
fly deploy
```
## Troubleshooting
### GPU Not Detected
**Check NVIDIA Docker:**
```bash
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
```
If this fails, reinstall nvidia-docker2.
**Check AMD ROCm:**
```bash
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
```
### Permission Errors
Container can't write to volumes:
```bash
# Fix permissions
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
```
### Out of Memory
Reduce GPU memory usage:
```bash
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
```
Or use CPU-only:
```bash
docker run -e DEVICE=cpu voicebox
```
### Model Download Fails
Ensure HuggingFace cache is writable:
```bash
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
```
Or use host cache:
```bash
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
```
### Port Already in Use
Change host port:
```bash
docker run -p 8080:8000 voicebox # Use port 8080 instead
```
## Security Best Practices
### 1. Don't Run as Root
Create non-root user in Dockerfile:
```dockerfile
RUN useradd -m -u 1000 voicebox
USER voicebox
```
### 2. Use Secrets for API Keys
Don't put API keys in docker-compose.yml:
```bash
# Use Docker secrets
echo "sk-your-key" | docker secret create openai_key -
docker service create \
--secret openai_key \
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
voicebox
```
### 3. Network Isolation
Use internal networks for multi-container setups:
```yaml
services:
voicebox:
networks:
- internal
nginx:
networks:
- internal
- external
ports:
- "80:80"
networks:
internal:
internal: true
external:
```
### 4. Resource Limits
Prevent resource exhaustion:
```yaml
services:
voicebox:
deploy:
resources:
limits:
cpus: '4'
memory: 8G
reservations:
cpus: '2'
memory: 4G
```
## Performance Tuning
### GPU Memory Management
```bash
# Use 80% of GPU (default 90%)
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
# Allow GPU memory growth (prevents OOM)
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
```
### Model Caching
Pre-download models to volume:
```bash
# Download models first
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
voicebox python -c "
from transformers import WhisperProcessor, WhisperForConditionalGeneration
WhisperProcessor.from_pretrained('openai/whisper-base')
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
"
# Then run normally
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
```
### Multi-Worker Setup
Use uvicorn workers for better throughput:
```dockerfile
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
```
## Monitoring
### Health Checks
Built-in health endpoint:
```bash
curl http://localhost:8000/health
```
Docker health check:
```yaml
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
```
### Prometheus Metrics
Add metrics exporter:
```python
# backend/main.py
from prometheus_fastapi_instrumentator import Instrumentator
Instrumentator().instrument(app).expose(app)
```
Then scrape `/metrics` with Prometheus.
### Logs
View container logs:
```bash
docker logs -f voicebox
# Or with compose
docker compose logs -f voicebox
```
## Next Steps
- [ ] Publish official images to GitHub Container Registry
- [ ] Add Kubernetes Helm charts
- [ ] Create Docker Desktop extension
- [ ] Add automated vulnerability scanning
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
## Contributing
Help improve Docker support:
1. Test on different platforms (AMD GPU, ARM64, etc.)
2. Submit Dockerfile optimizations
3. Share deployment configurations
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
## Resources
- [Docker Documentation](https://docs.docker.com)
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
+435
View File
@@ -0,0 +1,435 @@
# External Provider Support
**Status:** Planned for v0.2.0
**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...)
## Overview
External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for:
- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server
- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI
- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc.
- **Team Sharing**: Multiple users share one GPU server running models
- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa
## Architecture
```
┌─────────────────┐ HTTP/API ┌──────────────────┐
│ Voicebox UI │ ───────────────────────> │ Your TTS Server │
│ + Backend │ │ (Qwen3-TTS on │
│ │ <─────────────────────── │ AMD/NVIDIA GPU)│
│ - Profiles │ Audio + Metadata └──────────────────┘
│ - History │
│ - Audio Edit │ HTTP/API ┌──────────────────┐
│ - UI │ ───────────────────────> │ Whisper Service │
└─────────────────┘ │ (OpenAI API or │
│ self-hosted) │
└──────────────────┘
```
**What Voicebox Still Handles:**
- Voice profile management
- Generation history
- Audio trimming/editing
- Multi-track story editor
- UI/UX layer
**What External Providers Handle:**
- Model inference (TTS generation, transcription)
- GPU allocation
- Model loading/caching
## Configuration
### Environment Variables
```bash
# TTS Provider
TTS_MODE=remote # local | remote
TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL
TTS_API_KEY=your-api-key # Optional authentication
# Whisper Provider
WHISPER_MODE=openai-api # local | openai-api | remote
WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper
OPENAI_API_KEY=sk-... # For OpenAI Whisper API
```
### Voicebox Config UI (Planned)
Settings page will include:
- Provider selection dropdowns
- URL/API key inputs
- Connection test button
- Latency/status indicators
## Hosting External Services
### Option 1: Simple FastAPI Server (Recommended)
Create a lightweight server to expose your local Qwen3-TTS model:
```python
# tts_server.py
from fastapi import FastAPI, UploadFile, File
from qwen_tts import Qwen3TTSModel
import numpy as np
import base64
app = FastAPI()
model = Qwen3TTSModel.from_pretrained(
"Qwen/Qwen3-TTS-12Hz-1.7B-Base",
device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm
)
@app.post("/v1/generate")
async def generate(
text: str,
voice_prompt: dict,
language: str = "en",
seed: int = None
):
"""Generate speech from text using voice prompt."""
audio, sample_rate = model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
)
# Return as base64 for transport
audio_bytes = audio.tobytes()
return {
"audio": base64.b64encode(audio_bytes).decode(),
"sample_rate": sample_rate,
"dtype": str(audio.dtype)
}
@app.post("/v1/create_voice_prompt")
async def create_voice_prompt(
audio: UploadFile = File(...),
reference_text: str = ""
):
"""Create voice prompt from reference audio."""
# Save uploaded audio temporarily
audio_path = f"/tmp/{audio.filename}"
with open(audio_path, "wb") as f:
f.write(await audio.read())
# Create voice prompt
voice_prompt = model.create_voice_clone_prompt(
ref_audio=audio_path,
ref_text=reference_text,
)
return {"voice_prompt": voice_prompt}
@app.get("/health")
async def health():
return {
"status": "healthy",
"model": "Qwen3-TTS-12Hz-1.7B-Base",
"device": str(model.device)
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
**Run it:**
```bash
# Install dependencies
pip install fastapi uvicorn qwen-tts torch
# For AMD GPUs, use ROCm PyTorch:
pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
# Start server
python tts_server.py
```
### Option 2: vLLM (If Supported)
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--host 0.0.0.0 \
--port 8000 \
--gpu-memory-utilization 0.9
```
### Option 3: Cloud Platforms
**Modal.com Example:**
```python
import modal
app = modal.App("qwen-tts")
image = modal.Image.debian_slim().pip_install("qwen-tts", "torch")
@app.function(gpu="A10G", image=image)
@modal.web_endpoint(method="POST")
def generate(text: str, voice_prompt: dict):
from qwen_tts import Qwen3TTSModel
model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
audio, sr = model.generate_voice_clone(text, voice_prompt)
return {"audio": audio.tolist(), "sample_rate": sr}
```
Deploy: `modal deploy tts_server.py`
Get URL: `https://yourapp--generate.modal.run`
## API Specification
External TTS providers must implement these endpoints:
### `POST /v1/generate`
Generate speech from text.
**Request:**
```json
{
"text": "Hello, this is a test.",
"voice_prompt": { /* voice prompt object */ },
"language": "en",
"seed": 12345
}
```
**Response:**
```json
{
"audio": "base64-encoded-audio-bytes",
"sample_rate": 24000,
"dtype": "float32"
}
```
### `POST /v1/create_voice_prompt`
Create a voice prompt from reference audio.
**Request:** (multipart/form-data)
- `audio`: Audio file upload
- `reference_text`: Transcript of the audio
**Response:**
```json
{
"voice_prompt": { /* voice prompt object */ }
}
```
### `GET /health`
Health check endpoint.
**Response:**
```json
{
"status": "healthy",
"model": "Qwen3-TTS-12Hz-1.7B-Base",
"device": "cuda:0"
}
```
## Whisper External Providers
### OpenAI Whisper API
Simply set:
```bash
WHISPER_MODE=openai-api
OPENAI_API_KEY=sk-...
```
Voicebox will use OpenAI's Whisper API automatically.
### Self-Hosted Whisper
Run your own Whisper server:
```python
# whisper_server.py
from fastapi import FastAPI, UploadFile, File
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
app = FastAPI()
processor = WhisperProcessor.from_pretrained("openai/whisper-base")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
@app.post("/v1/transcribe")
async def transcribe(audio: UploadFile = File(...), language: str = None):
# Load audio
audio_path = f"/tmp/{audio.filename}"
with open(audio_path, "wb") as f:
f.write(await audio.read())
audio_data, sr = librosa.load(audio_path, sr=16000)
# Process
inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt")
predicted_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
return {"text": transcription}
```
Configure Voicebox:
```bash
WHISPER_MODE=remote
WHISPER_REMOTE_URL=http://localhost:9000
```
## Use Cases
### 1. AMD GPU User with Existing Setup
**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux.
**Setup:**
1. Run `tts_server.py` on your AMD box (ROCm PyTorch)
2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000`
3. Use Voicebox UI for profiles, generation, editing
4. TTS happens on your AMD GPU
### 2. Team Deployment
**Scenario:** 5 team members, 1 GPU server.
**Setup:**
1. Deploy TTS server on shared GPU box
2. Each person runs Voicebox desktop app locally
3. All point to same `TTS_REMOTE_URL`
4. Profiles and history stay local per user
5. GPU usage is shared
### 3. Hybrid Local/Remote
**Scenario:** Fast local Whisper, heavy TTS on cloud.
**Setup:**
```bash
TTS_MODE=remote
TTS_REMOTE_URL=https://your-modal-app.modal.run
WHISPER_MODE=local # Fast transcription on your CPU
```
### 4. OpenAI Whisper + Self-Hosted TTS
**Scenario:** Use OpenAI's API for transcription, run TTS locally.
**Setup:**
```bash
TTS_MODE=local
WHISPER_MODE=openai-api
OPENAI_API_KEY=sk-...
```
## Security Considerations
### Authentication
Add API key authentication to your external server:
```python
from fastapi import Header, HTTPException
API_KEY = "your-secret-key"
async def verify_api_key(x_api_key: str = Header(...)):
if x_api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
@app.post("/v1/generate", dependencies=[Depends(verify_api_key)])
async def generate(...):
...
```
Configure Voicebox:
```bash
TTS_API_KEY=your-secret-key
```
### Network Security
- **VPN/Tailscale**: Use private network for remote servers
- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates
- **Firewall**: Restrict access to known IPs
### Rate Limiting
Protect your external server:
```python
from slowapi import Limiter
from slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter
@app.post("/v1/generate")
@limiter.limit("10/minute")
async def generate(...):
...
```
## Performance Considerations
### Latency
External providers add network latency:
- **Local network**: ~10-50ms overhead (negligible)
- **Same datacenter**: ~1-5ms overhead
- **Cross-region cloud**: 50-200ms+ overhead
For real-time applications, keep TTS server on local network or same cloud region.
### Caching
Implement response caching on external server:
```python
from functools import lru_cache
@lru_cache(maxsize=1000)
def get_cached_generation(text, voice_prompt_hash, language, seed):
return model.generate_voice_clone(text, voice_prompt)
```
### Load Balancing
For high-traffic deployments, run multiple TTS servers behind a load balancer:
```
Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1)
├──> TTS Server 2 (GPU 2)
└──> TTS Server 3 (GPU 3)
```
## Future Enhancements
- [ ] **Provider Marketplace**: Built-in directory of compatible providers
- [ ] **Automatic Fallback**: If remote fails, fallback to local
- [ ] **Cost Tracking**: Monitor API usage and costs
- [ ] **Performance Metrics**: Latency, throughput dashboards
- [ ] **Multi-Provider**: Use different providers for different voices/languages
## Contributing
If you build an external provider, please share:
1. Server implementation
2. Performance benchmarks
3. Deployment guide
Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions)
## Questions?
- **Discord**: [Join the community](https://discord.gg/...)
- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues)
- **Docs**: [Full documentation](https://voicebox.sh/docs)
+396
View File
@@ -0,0 +1,396 @@
# MLX Audio Integration
**Status:** Validated ✅
**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio)
## Validation Results
We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings:
| Metric | Result |
|--------|--------|
| MLX Version | 0.30.4 |
| Model Load Time | ~1s (after initial download) |
| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) |
| Test Hardware | Apple Silicon Mac |
### Model Mapping
| voicebox (PyTorch) | mlx-audio (MLX) |
|--------------------|-----------------|
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) |
### mlx-audio API
The API uses a **generator-based streaming pattern**:
```python
from mlx_audio.tts import load
model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16")
# generate() yields GenerationResult objects
for result in model.generate("Hello world"):
audio = result.audio # numpy array of samples
sample_rate = result.sample_rate # 24000
rtf = result.real_time_factor # e.g., 0.55
```
### Known Warnings (harmless)
```
You are using a model of type qwen3_tts to instantiate a model of type .
The tokenizer you are loading... with an incorrect regex pattern...
```
These warnings appear but don't affect functionality or output quality.
### Demo Script
Run `mlx-test/demo.py` to test:
```bash
cd mlx-test && source venv/bin/activate && python demo.py "Your text here"
```
## Problem
Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs.
This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads.
## Solution
Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use.
**Key wins:**
- Native GPU acceleration on Apple Silicon (no more CPU fallback)
- Streaming TTS support (faster perceived latency)
- Memory optimizations (run larger models on less RAM)
- Fixed 0.6B silence bug that we currently ship
- Same Qwen3-TTS models (zero migration cost for users)
## Architecture
### Current Stack
```
┌─────────────────────────┐
│ PyTorch + Qwen3-TTS │
│ (CPU only on macOS) │
└─────────────────────────┘
```
### Proposed Stack
```
┌─────────────────────────────────────────┐
│ Platform Detection at Runtime │
└─────────────────────────────────────────┘
│
├─── Apple Silicon (aarch64-darwin)
│ ┌─────────────────────────┐
│ │ MLX Audio Backend │
│ │ - Qwen3-TTS (mlx) │
│ │ - Whisper (mlx) │
│ │ - Streaming support │
│ └─────────────────────────┘
│
└─── Other (x86_64, Windows, Linux)
┌─────────────────────────┐
│ PyTorch Backend │
│ - Qwen3-TTS (pytorch) │
│ - Whisper (pytorch) │
│ - CUDA if available │
└─────────────────────────┘
```
## Implementation Phases
### Phase 1: Platform Detection & Dependency Management
Create a backend that switches between PyTorch and MLX based on runtime platform detection.
**New files:**
- `backend/platform.py` - Detect Apple Silicon, return backend type
- `backend/backends/__init__.py` - Backend factory pattern
- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only)
**Modified files:**
- `backend/requirements.txt` - Keep PyTorch as default
- `backend/main.py` - Import from backend factory instead of direct imports
**Platform detection logic:**
```python
def get_backend_type() -> str:
"""Detect best backend for current platform."""
if platform.system() == "Darwin" and platform.machine() == "arm64":
# Apple Silicon detected
try:
import mlx
return "mlx"
except ImportError:
return "pytorch" # Fallback if mlx not installed
return "pytorch"
```
### Phase 2: MLX Backend Implementation
Create parallel implementations of TTS and STT using mlx-audio.
**New files:**
- `backend/backends/mlx_backend.py` - MLX inference engine
- `backend/backends/pytorch_backend.py` - Refactor current code into backend
**Interface both backends must implement:**
```python
class TTSBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ...
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ...
async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ...
def unload_model(self) -> None: ...
class STTBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ...
def unload_model(self) -> None: ...
```
**MLX backend implementation notes:**
mlx-audio's `generate()` returns a generator by default (streaming is built-in):
```python
# MLX backend wrapper
from mlx_audio.tts import load
class MLXTTSBackend:
def __init__(self):
self.model = None
async def load_model(self, model_size: str) -> None:
model_map = {
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
# "0.6B": needs conversion to mlx format
}
self.model = load(model_map[model_size])
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]:
# Collect all chunks from generator
chunks = []
for result in self.model.generate(text): # TODO: add voice_prompt support
chunks.append(np.array(result.audio))
return np.concatenate(chunks), 24000
```
**MLX-specific features to expose:**
- Streaming TTS (new endpoint: `/api/generate/stream`)
- Memory-optimized model loading
- Qwen3-ASR for transcription (in addition to Whisper)
### Phase 3: API Layer Updates
Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility.
**Modified files:**
- `backend/main.py` - Add streaming endpoints
- `backend/tts.py` - Refactor to use backend abstraction
- `backend/transcribe.py` - Refactor to use backend abstraction
**New endpoints:**
```python
@app.post("/api/generate/stream")
async def generate_stream(...) -> StreamingResponse:
"""Stream TTS chunks as they're generated (MLX only)."""
backend = get_backend()
if not hasattr(backend, 'generate_streaming'):
raise HTTPException(501, "Streaming not supported on this backend")
return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav")
```
**Backward compatibility:**
- Keep all existing `/api/generate` endpoints unchanged
- PyTorch backend users see no behavior change
- MLX users automatically get faster inference, streaming is opt-in
### Phase 4: Frontend Integration
Add UI indicators for backend type and streaming progress.
**Modified files:**
- `app/src/hooks/useGenerationForm.tsx` - Add streaming support
- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle
- `app/src/lib/api.ts` - Add streaming API client
**UI additions:**
- Badge showing current backend ("MLX" or "PyTorch")
- Toggle for streaming mode (disabled if PyTorch)
- Real-time streaming playback (WaveSurfer progressive loading)
### Phase 5: Build & Distribution
Create separate installers for MLX (Apple Silicon) and PyTorch (Universal).
**Modified files:**
- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds
- `.github/workflows/release.yml` - Build both variants
**Build matrix:**
```yaml
- target: aarch64-apple-darwin
backend: mlx
installer: voicebox-macos-silicon-{version}.dmg
- target: x86_64-apple-darwin
backend: pytorch
installer: voicebox-macos-intel-{version}.dmg
- target: x86_64-pc-windows-msvc
backend: pytorch
installer: voicebox-windows-{version}.exe
```
**Installation flow:**
- Auto-detect architecture, recommend correct installer
- MLX installer includes `mlx-audio` in embedded Python
- PyTorch installer includes `torch` in embedded Python
- Both can coexist (different backend, same profile format)
### Phase 6: Testing & Validation
Ensure both backends produce compatible outputs.
**New files:**
- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio
- `backend/tests/test_streaming.py` - Streaming-specific tests
**Test scenarios:**
- Same voice prompt on both backends → similar (not identical) audio output
- Profile created on MLX → loads on PyTorch (and vice versa)
- Streaming chunks assemble into valid WAV file
- Model downloads work on both backends
- Memory usage stays within bounds
### Phase 7: Documentation
Update user-facing docs and developer guides.
**New files:**
- `docs/developer/BACKENDS.md` - Guide for adding new backends
- `docs/overview/performance.md` - Backend comparison benchmarks
**Modified files:**
- `README.md` - Note Apple Silicon acceleration
- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues
**Key docs to write:**
- Which installer to download (architecture detection)
- Performance comparison (MLX vs PyTorch on same M2 hardware)
- How streaming mode works
- How to force PyTorch on Apple Silicon (for debugging)
## Technical Decisions
### Why Dual Backend Instead of MLX-Only?
**Pros of dual backend:**
- Windows and Intel Mac users unaffected
- Easier testing (can compare outputs)
- Fallback if MLX has issues
**Cons of dual backend:**
- More code to maintain
- Two dependency trees
- Build complexity (separate installers)
**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback.
### Why Separate Installers Instead of Runtime Detection?
**Pros of separate installers:**
- Smaller bundle size (don't ship both PyTorch and MLX)
- Clearer to users which version they have
- Easier to debug (no "which backend am I running?" confusion)
- Can optimize each build for its target
**Cons:**
- More installers to build and test
- Users might download the wrong one
**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page.
### Streaming vs Batch Generation
MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be:
1. MLX-only feature (✅ chosen)
2. Implemented for both (lots of work)
3. Not exposed at all (wasted opportunity)
**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend).
## Migration Path
Nothing needs migrating, macos users will just notice a speed-boost in inference
**Data format compatibility:**
- Profiles (SQLite) → no schema changes needed
- Voice prompts (cached) → backend-agnostic (just numpy arrays)
- Audio files → unchanged
## Performance Expectations
### Measured Results (from validation)
| Metric | MLX (measured) | PyTorch CPU (estimated) |
|--------|----------------|-------------------------|
| **6s audio generation** | ~3-4s | ~10-15s |
| **Real-time factor** | 0.5-0.6x | 2-3x |
| **Model load (cached)** | ~1s | ~3-5s |
### TTS Generation (1.7B model, ~20s output)
- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time)
- **MLX (M2 Max):** ~8-12s (faster than real-time)
- **Improvement:** ~4-5x faster
### Whisper Transcription (10s audio clip)
- **PyTorch CPU:** ~5-8s
- **MLX:** ~1-2s
- **Improvement:** ~3-4x faster
### Memory Usage (1.7B model)
- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM)
- **MLX:** ~4-6GB (unified memory, better optimization)
- **Improvement:** ~40% less RAM
Full benchmarks will be in `docs/overview/performance.md` after Phase 6.
## Open Questions
- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure
- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No
- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing
- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users.
## Success Metrics
How we'll know this worked:
1. **Performance:** Apple Silicon users report generation faster than real-time
2. **Adoption:** >80% of macOS downloads are MLX build within 1 month
3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions)
4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance
## Related Work
- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly
- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API
- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data
## Next Steps
1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/`
2. Get approval on dual-backend architecture
3. Start Phase 1 (platform detection)
## Questions?
Feedback welcome in GitHub discussions or Discord.

Some files were not shown because too many files have changed in this diff Show More