Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1b66a528d1 | ||
|
|
bef4092e6e | ||
|
|
9654f7b642 | ||
|
|
eba1244add | ||
|
|
94487f32a5 | ||
|
|
081f45e680 | ||
|
|
86768288ce | ||
|
|
0fd063442a | ||
|
|
a0c2493e98 | ||
|
|
b39f48cc81 | ||
|
|
4ff775bc98 | ||
|
|
6351aa75e9 | ||
|
|
43873a883b | ||
|
|
60012b81c0 | ||
|
|
89f3127c37 | ||
|
|
ef3c3a7f8c | ||
|
|
7b5e73cfa8 | ||
|
|
462f104494 | ||
|
|
fadb57164e | ||
|
|
e870d65136 | ||
|
|
3df40278cc | ||
|
|
deeef5a474 | ||
|
|
236e464525 | ||
|
|
cf3cf3f002 | ||
|
|
9d98e1e768 | ||
|
|
3be8980f48 | ||
|
|
76bc070f5b | ||
|
|
01838f4773 | ||
|
|
39e4f9d08c | ||
|
|
341d71470c | ||
|
|
bb6cea24ba | ||
|
|
fa7ac88abc | ||
|
|
c68ddc45b1 | ||
|
|
f89dc66d0c | ||
|
|
cba7d7bc23 | ||
|
|
3c89b068f3 | ||
|
|
423d69b7cc | ||
|
|
c513451277 |
@@ -1,5 +1,5 @@
|
||||
[bumpversion]
|
||||
current_version = 0.1.8
|
||||
current_version = 0.1.11
|
||||
commit = True
|
||||
tag = True
|
||||
tag_name = v{new_version}
|
||||
|
||||
@@ -17,15 +17,19 @@ jobs:
|
||||
- platform: 'macos-latest'
|
||||
args: '--target aarch64-apple-darwin'
|
||||
python-version: '3.12'
|
||||
backend: 'mlx'
|
||||
- platform: 'macos-15-intel'
|
||||
args: '--target x86_64-apple-darwin'
|
||||
python-version: '3.12'
|
||||
backend: 'pytorch'
|
||||
# - platform: 'ubuntu-22.04'
|
||||
# args: ''
|
||||
# python-version: '3.12'
|
||||
# backend: 'pytorch'
|
||||
- platform: 'windows-latest'
|
||||
args: ''
|
||||
python-version: '3.12'
|
||||
backend: 'pytorch'
|
||||
|
||||
runs-on: ${{ matrix.platform }}
|
||||
|
||||
@@ -57,6 +61,11 @@ jobs:
|
||||
pip install pyinstaller
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
- name: Install MLX dependencies (Apple Silicon only)
|
||||
if: matrix.backend == 'mlx'
|
||||
run: |
|
||||
pip install -r backend/requirements-mlx.txt
|
||||
|
||||
- name: Build Python server (Linux/macOS)
|
||||
if: matrix.platform != 'windows-latest'
|
||||
run: |
|
||||
@@ -133,7 +142,8 @@ jobs:
|
||||
See the assets below to download and install this version.
|
||||
|
||||
### Installation
|
||||
- **macOS**: Download the `.dmg` file
|
||||
- **macOS (Apple Silicon)**: Download the `aarch64.dmg` file - uses MLX for fast native inference
|
||||
- **macOS (Intel)**: Download the `x64.dmg` file - uses PyTorch
|
||||
- **Windows**: Download the `.msi` installer
|
||||
- **Linux**: Download the `.AppImage` or `.deb` package
|
||||
|
||||
|
||||
@@ -53,6 +53,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks
|
||||
- Includes Python version detection and compatibility warnings
|
||||
- Self-documenting help system with `make help`
|
||||
- Colored output for better readability
|
||||
- Supports parallel development server execution
|
||||
|
||||
### Changed
|
||||
- **README** - Added Makefile reference and updated Quick Start with Makefile-based setup instructions alongside manual setup
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased - Planned]
|
||||
|
||||
### Planned
|
||||
- Real-time streaming synthesis
|
||||
- Conversation mode with multiple speakers
|
||||
|
||||
@@ -32,6 +32,10 @@ Thank you for your interest in contributing to Voicebox! This document provides
|
||||
|
||||
### Development Setup
|
||||
|
||||
**Using the Makefile (recommended for macOS/Linux):** Run `make setup` to install all dependencies, then `make dev` to start development servers. See `make help` for all available commands.
|
||||
|
||||
**Manual setup (required for Windows):**
|
||||
|
||||
1. **Fork and clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/YOUR_USERNAME/voicebox.git
|
||||
@@ -62,37 +66,43 @@ Thank you for your interest in contributing to Voicebox! This document provides
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Install MLX dependencies (Apple Silicon only - for faster inference)
|
||||
# On Apple Silicon, this enables native Metal acceleration
|
||||
if [[ $(uname -m) == "arm64" ]]; then
|
||||
pip install -r requirements-mlx.txt
|
||||
fi
|
||||
|
||||
# Install Qwen3-TTS (required for voice synthesis)
|
||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
```
|
||||
|
||||
4. **Initialize database**
|
||||
```bash
|
||||
cd backend
|
||||
python -c "from database import init_db; init_db()"
|
||||
```
|
||||
This creates the SQLite database at `data/voicebox.db`.
|
||||
4. **Start development servers**
|
||||
|
||||
5. **Start development servers**
|
||||
|
||||
**Terminal 1: Backend server**
|
||||
Development requires two terminals: one for the Python backend, one for the Tauri app.
|
||||
|
||||
**Terminal 1: Backend server** (start this first)
|
||||
```bash
|
||||
cd backend
|
||||
source venv/bin/activate # Activate venv if not already active
|
||||
bun run dev:server
|
||||
# Or manually: uvicorn main:app --reload --port 8000
|
||||
# Or manually: uvicorn main:app --reload --port 17493
|
||||
```
|
||||
Backend will be available at `http://localhost:8000`
|
||||
|
||||
Backend will be available at `http://localhost:17493`
|
||||
|
||||
**Terminal 2: Desktop app**
|
||||
```bash
|
||||
bun run dev
|
||||
```
|
||||
This will:
|
||||
- Create a placeholder sidecar binary (for Tauri compilation)
|
||||
- Start Vite dev server on port 5173
|
||||
- Launch Tauri window pointing to localhost:5173
|
||||
- Connect to the Python server you started in Terminal 1
|
||||
- Enable hot reload
|
||||
|
||||
> **Note:** In dev mode, the app connects to your manually-started Python server.
|
||||
> The bundled server binary is only used in production builds.
|
||||
|
||||
**Optional: Web app**
|
||||
```bash
|
||||
bun run dev:web
|
||||
@@ -109,18 +119,36 @@ First-time usage will be slower due to model downloads, but subsequent runs will
|
||||
|
||||
### Building
|
||||
|
||||
**Build Python server binary:**
|
||||
**Build everything (recommended):**
|
||||
```bash
|
||||
bun run build
|
||||
```
|
||||
This automatically:
|
||||
1. Builds the Python server binary (`./scripts/build-server.sh`)
|
||||
2. Builds the Tauri desktop app (`cd tauri && bun run tauri build`)
|
||||
|
||||
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`) in `tauri/src-tauri/target/release/bundle/`.
|
||||
|
||||
**Note:** The build process detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
|
||||
|
||||
**Build server binary only:**
|
||||
```bash
|
||||
bun run build:server
|
||||
# or
|
||||
./scripts/build-server.sh
|
||||
```
|
||||
Creates platform-specific binary in `tauri/src-tauri/binaries/`
|
||||
|
||||
**Build Tauri desktop app:**
|
||||
**Building with local Qwen3-TTS development version:**
|
||||
|
||||
If you're actively developing or modifying the Qwen3-TTS library, set the `QWEN_TTS_PATH` environment variable to point to your local clone:
|
||||
|
||||
```bash
|
||||
cd tauri
|
||||
bun run tauri build
|
||||
export QWEN_TTS_PATH=~/path/to/your/Qwen3-TTS
|
||||
bun run build:server
|
||||
```
|
||||
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`)
|
||||
|
||||
This makes PyInstaller use your local qwen-tts version instead of the pip-installed package. Useful when testing changes to the TTS library before they're published to PyPI or when using an editable install (`pip install -e`).
|
||||
|
||||
**Build web app:**
|
||||
```bash
|
||||
@@ -137,6 +165,26 @@ After starting the backend server:
|
||||
```
|
||||
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
||||
|
||||
### Convert Assets to Web Formats
|
||||
|
||||
To optimize images and videos for the web, run:
|
||||
```bash
|
||||
bun run convert:assets
|
||||
```
|
||||
|
||||
This script:
|
||||
- Converts PNG → WebP (better compression, same quality)
|
||||
- Converts MOV → WebM (VP9 codec, smaller file size)
|
||||
- Processes files in `landing/public/` and `docs/public/`
|
||||
- **Deletes original files** after successful conversion
|
||||
|
||||
**Requirements:** Install `webp` and `ffmpeg`:
|
||||
```bash
|
||||
brew install webp ffmpeg
|
||||
```
|
||||
|
||||
> **Note:** Run this before committing new images or videos to keep the repository size small.
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### 1. Create a Branch
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
# Voicebox Makefile
|
||||
# Unix-only (macOS/Linux). Windows users should use WSL.
|
||||
|
||||
SHELL := /bin/bash
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
# Directories
|
||||
BACKEND_DIR := backend
|
||||
TAURI_DIR := tauri
|
||||
WEB_DIR := web
|
||||
APP_DIR := app
|
||||
|
||||
# Python (prefer 3.12, fallback to 3.13, then python3)
|
||||
PYTHON := $(shell command -v python3.12 2>/dev/null || command -v python3.13 2>/dev/null || echo python3)
|
||||
VENV := $(CURDIR)/$(BACKEND_DIR)/venv
|
||||
VENV_BIN := $(VENV)/bin
|
||||
PIP := $(VENV_BIN)/pip
|
||||
PYTHON_VENV := $(VENV_BIN)/python
|
||||
|
||||
# Colors for output
|
||||
BLUE := \033[0;34m
|
||||
GREEN := \033[0;32m
|
||||
YELLOW := \033[0;33m
|
||||
NC := \033[0m # No Color
|
||||
|
||||
.PHONY: help
|
||||
help: ## Show this help message
|
||||
@echo -e "$(BLUE)Voicebox$(NC) - Development Commands"
|
||||
@echo ""
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
|
||||
awk 'BEGIN {FS = ":.*?## "}; {printf " $(GREEN)%-20s$(NC) %s\n", $$1, $$2}'
|
||||
|
||||
# =============================================================================
|
||||
# SETUP
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: setup setup-js setup-python setup-rust
|
||||
|
||||
setup: setup-js setup-python ## Full project setup (all dependencies)
|
||||
@echo -e "$(GREEN)✓ Setup complete!$(NC)"
|
||||
@echo -e " Run $(YELLOW)make dev$(NC) to start development servers"
|
||||
|
||||
setup-js: ## Install JavaScript dependencies (bun)
|
||||
@echo -e "$(BLUE)Installing JavaScript dependencies...$(NC)"
|
||||
bun install
|
||||
|
||||
setup-python: $(VENV)/bin/activate ## Set up Python virtual environment and dependencies
|
||||
@echo -e "$(BLUE)Installing Python dependencies...$(NC)"
|
||||
$(PIP) install --upgrade pip
|
||||
$(PIP) install -r $(BACKEND_DIR)/requirements.txt
|
||||
@if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \
|
||||
echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \
|
||||
$(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \
|
||||
echo -e "$(GREEN)✓ MLX backend enabled (native Metal acceleration)$(NC)"; \
|
||||
fi
|
||||
$(PIP) install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
@echo -e "$(GREEN)✓ Python environment ready$(NC)"
|
||||
|
||||
$(VENV)/bin/activate:
|
||||
@echo -e "$(BLUE)Creating Python virtual environment...$(NC)"
|
||||
@PY_MINOR=$$($(PYTHON) -c "import sys; print(sys.version_info[1])"); \
|
||||
if [ "$$PY_MINOR" -gt 13 ]; then \
|
||||
echo -e "$(YELLOW)Warning: Python 3.$$PY_MINOR detected. ML packages may not be compatible.$(NC)"; \
|
||||
echo -e "$(YELLOW)Recommended: Use Python 3.12 or 3.13 (brew install [email protected])$(NC)"; \
|
||||
fi
|
||||
$(PYTHON) -m venv $(VENV)
|
||||
|
||||
setup-rust: ## Install Rust toolchain (if not present)
|
||||
@command -v rustc >/dev/null 2>&1 || curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
|
||||
# =============================================================================
|
||||
# DEVELOPMENT
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: dev dev-backend dev-frontend dev-web kill-dev
|
||||
|
||||
dev: ## Start backend + desktop app (parallel)
|
||||
@echo -e "$(BLUE)Starting development servers...$(NC)"
|
||||
@echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)"
|
||||
@trap 'kill 0' EXIT; \
|
||||
$(MAKE) dev-backend & \
|
||||
sleep 2 && $(MAKE) dev-frontend & \
|
||||
wait
|
||||
|
||||
dev-backend: ## Start FastAPI backend server
|
||||
@echo -e "$(BLUE)Starting backend server on http://localhost:17493$(NC)"
|
||||
$(VENV_BIN)/uvicorn backend.main:app --reload --port 17493
|
||||
|
||||
dev-frontend: ## Start Tauri desktop app
|
||||
@echo -e "$(BLUE)Starting Tauri desktop app...$(NC)"
|
||||
bun run dev
|
||||
|
||||
dev-web: ## Start backend + web app (parallel)
|
||||
@echo -e "$(BLUE)Starting web development servers...$(NC)"
|
||||
@trap 'kill 0' EXIT; \
|
||||
$(MAKE) dev-backend & \
|
||||
sleep 2 && cd $(WEB_DIR) && bun run dev & \
|
||||
wait
|
||||
|
||||
kill-dev: ## Kill all development processes
|
||||
@echo -e "$(YELLOW)Killing development processes...$(NC)"
|
||||
-pkill -f "uvicorn main:app" 2>/dev/null || true
|
||||
-pkill -f "vite" 2>/dev/null || true
|
||||
@echo -e "$(GREEN)✓ Processes killed$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# BUILD
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: build build-server build-tauri build-web
|
||||
|
||||
build: build-server build-tauri ## Build everything (server binary + desktop app)
|
||||
@echo -e "$(GREEN)✓ Build complete!$(NC)"
|
||||
|
||||
build-server: ## Build Python server binary
|
||||
@echo -e "$(BLUE)Building server binary...$(NC)"
|
||||
PATH="$(VENV_BIN):$$PATH" ./scripts/build-server.sh
|
||||
|
||||
build-tauri: ## Build Tauri desktop app
|
||||
@echo -e "$(BLUE)Building Tauri desktop app...$(NC)"
|
||||
cd $(TAURI_DIR) && bun run tauri build
|
||||
|
||||
build-web: ## Build web app
|
||||
@echo -e "$(BLUE)Building web app...$(NC)"
|
||||
cd $(WEB_DIR) && bun run build
|
||||
@echo -e "$(GREEN)✓ Web build output in $(WEB_DIR)/dist/$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# DATABASE & API
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: db-init db-reset generate-api
|
||||
|
||||
db-init: $(VENV)/bin/activate ## Initialize SQLite database
|
||||
@echo -e "$(BLUE)Initializing database...$(NC)"
|
||||
cd $(BACKEND_DIR) && $(PYTHON_VENV) -c "from database import init_db; init_db()"
|
||||
@echo -e "$(GREEN)✓ Database created at $(BACKEND_DIR)/data/voicebox.db$(NC)"
|
||||
|
||||
db-reset: ## Reset database (delete and reinitialize)
|
||||
@echo -e "$(YELLOW)Resetting database...$(NC)"
|
||||
rm -f $(BACKEND_DIR)/data/voicebox.db
|
||||
$(MAKE) db-init
|
||||
|
||||
generate-api: ## Generate TypeScript API client from OpenAPI schema
|
||||
@echo -e "$(BLUE)Generating API client...$(NC)"
|
||||
@echo -e "$(YELLOW)Note: Backend must be running (make dev-backend)$(NC)"
|
||||
./scripts/generate-api.sh
|
||||
@echo -e "$(GREEN)✓ API client generated in $(APP_DIR)/src/lib/api/$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# CODE QUALITY
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: lint format typecheck check
|
||||
|
||||
lint: ## Run linter (Biome)
|
||||
@echo -e "$(BLUE)Linting...$(NC)"
|
||||
bun run lint
|
||||
|
||||
format: ## Format code (Biome)
|
||||
@echo -e "$(BLUE)Formatting...$(NC)"
|
||||
bun run format
|
||||
|
||||
typecheck: ## Run TypeScript type checking
|
||||
@echo -e "$(BLUE)Type checking...$(NC)"
|
||||
bun run tsc --noEmit
|
||||
|
||||
check: ## Run all checks (Biome lint + format + type check)
|
||||
@echo -e "$(BLUE)Running all checks...$(NC)"
|
||||
bun run check
|
||||
@echo -e "$(GREEN)✓ All checks passed$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# TESTING
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: test test-backend test-frontend
|
||||
|
||||
test: test-backend test-frontend ## Run all tests
|
||||
@echo -e "$(GREEN)✓ All tests passed$(NC)"
|
||||
|
||||
test-backend: ## Run Python backend tests (requires pytest)
|
||||
@echo -e "$(BLUE)Running backend tests...$(NC)"
|
||||
@if [ -f "$(VENV_BIN)/pytest" ]; then \
|
||||
cd $(BACKEND_DIR) && $(VENV_BIN)/pytest -v; \
|
||||
else \
|
||||
echo -e "$(YELLOW)pytest not installed. Run: $(PIP) install pytest$(NC)"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
test-frontend: ## Run frontend tests (requires test script in package.json)
|
||||
@echo -e "$(BLUE)Running frontend tests...$(NC)"
|
||||
@if bun run test --help >/dev/null 2>&1; then \
|
||||
bun run test; \
|
||||
else \
|
||||
echo -e "$(YELLOW)No test script configured$(NC)"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# LOGS & DEBUGGING
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: logs docs
|
||||
|
||||
logs: ## Tail backend logs
|
||||
@echo -e "$(BLUE)Tailing logs (Ctrl+C to stop)...$(NC)"
|
||||
tail -f $(BACKEND_DIR)/logs/*.log 2>/dev/null || echo "No log files found"
|
||||
|
||||
docs: ## Open API documentation (backend must be running)
|
||||
@echo -e "$(BLUE)Opening API docs...$(NC)"
|
||||
open http://localhost:17493/docs 2>/dev/null || xdg-open http://localhost:17493/docs
|
||||
|
||||
# =============================================================================
|
||||
# CLEAN
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: clean clean-python clean-build clean-all
|
||||
|
||||
clean: ## Clean build artifacts
|
||||
@echo -e "$(BLUE)Cleaning build artifacts...$(NC)"
|
||||
rm -rf $(TAURI_DIR)/src-tauri/target/release
|
||||
rm -rf $(WEB_DIR)/dist
|
||||
rm -rf $(APP_DIR)/dist
|
||||
@echo -e "$(GREEN)✓ Build artifacts cleaned$(NC)"
|
||||
|
||||
clean-python: ## Clean Python cache and virtual environment
|
||||
@echo -e "$(BLUE)Cleaning Python files...$(NC)"
|
||||
rm -rf $(VENV)
|
||||
find $(BACKEND_DIR) -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find $(BACKEND_DIR) -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
@echo -e "$(GREEN)✓ Python environment cleaned$(NC)"
|
||||
|
||||
clean-build: ## Clean Rust/Tauri build cache
|
||||
@echo -e "$(BLUE)Cleaning Rust build cache...$(NC)"
|
||||
cd $(TAURI_DIR)/src-tauri && cargo clean
|
||||
@echo -e "$(GREEN)✓ Rust cache cleaned$(NC)"
|
||||
|
||||
clean-all: clean clean-python clean-build ## Nuclear clean (everything)
|
||||
@echo -e "$(BLUE)Cleaning node_modules...$(NC)"
|
||||
rm -rf node_modules
|
||||
rm -rf $(APP_DIR)/node_modules
|
||||
rm -rf $(TAURI_DIR)/node_modules
|
||||
rm -rf $(WEB_DIR)/node_modules
|
||||
@echo -e "$(GREEN)✓ Full clean complete$(NC)"
|
||||
@@ -10,6 +10,21 @@
|
||||
All running locally on your machine.
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/jamiepine/voicebox/releases">
|
||||
<img src="https://img.shields.io/github/downloads/jamiepine/voicebox/total?style=flat&color=blue" alt="Downloads" />
|
||||
</a>
|
||||
<a href="https://github.com/jamiepine/voicebox/releases/latest">
|
||||
<img src="https://img.shields.io/github/v/release/jamiepine/voicebox?style=flat" alt="Release" />
|
||||
</a>
|
||||
<a href="https://github.com/jamiepine/voicebox/stargazers">
|
||||
<img src="https://img.shields.io/github/stars/jamiepine/voicebox?style=flat" alt="Stars" />
|
||||
</a>
|
||||
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
|
||||
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://voicebox.sh">voicebox.sh</a> •
|
||||
<a href="#download">Download</a> •
|
||||
@@ -53,6 +68,7 @@ Unlike cloud services that lock your voice data behind subscriptions, Voicebox g
|
||||
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
|
||||
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
|
||||
- **Native performance** — built with Tauri (Rust), not Electron
|
||||
- **Super fast on Mac** — MLX backend with native Metal acceleration for 4-5x faster inference on Apple Silicon
|
||||
|
||||
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
|
||||
|
||||
@@ -82,6 +98,7 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
|
||||
- **Instant cloning** — Upload a sample, get a voice profile
|
||||
- **High fidelity** — Natural prosody, emotion, and cadence
|
||||
- **Multi-language** — English, Chinese, and more coming
|
||||
- **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super fast generation
|
||||
|
||||
### Voice Profile Management
|
||||
|
||||
@@ -132,17 +149,17 @@ Voicebox exposes a full REST API, so you can integrate voice synthesis into your
|
||||
|
||||
```bash
|
||||
# Generate speech
|
||||
curl -X POST http://localhost:8000/api/generate \
|
||||
curl -X POST http://localhost:8000/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Hello world", "profile_id": "abc123"}'
|
||||
-d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}'
|
||||
|
||||
# List voice profiles
|
||||
curl http://localhost:8000/api/profiles
|
||||
curl http://localhost:8000/profiles
|
||||
|
||||
# Create a profile from audio
|
||||
curl -X POST http://localhost:8000/api/profiles \
|
||||
-F "[email protected]" \
|
||||
-F "name=My Voice"
|
||||
# Create a profile
|
||||
curl -X POST http://localhost:8000/profiles \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "My Voice", "language": "en"}'
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
@@ -165,8 +182,9 @@ Full API documentation available at `http://localhost:8000/docs` when running.
|
||||
| Frontend | React, TypeScript, Tailwind CSS |
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| Voice Model | Qwen3-TTS |
|
||||
| Transcription | Whisper |
|
||||
| Voice Model | Qwen3-TTS (PyTorch or MLX) |
|
||||
| Transcription | Whisper (PyTorch or MLX) |
|
||||
| Inference Engine | MLX (Apple Silicon) / PyTorch (Windows/Linux/Intel) |
|
||||
| Database | SQLite |
|
||||
| Audio | WaveSurfer.js, librosa |
|
||||
|
||||
@@ -207,8 +225,26 @@ Voicebox aims to be the **one-stop shop for everything voice** — cloning, synt
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed setup and contribution guidelines.
|
||||
|
||||
**Using the Makefile (recommended):** Run `make help` to see all available commands for setup, development, building, and testing.
|
||||
|
||||
### Quick Start
|
||||
|
||||
**With Makefile (Unix/macOS/Linux):**
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/voicebox-sh/voicebox.git
|
||||
cd voicebox
|
||||
|
||||
# Setup everything
|
||||
make setup
|
||||
|
||||
# Start development
|
||||
make dev
|
||||
```
|
||||
|
||||
**Manual setup (all platforms):**
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/voicebox-sh/voicebox.git
|
||||
@@ -224,7 +260,11 @@ cd backend && pip install -r requirements.txt && cd ..
|
||||
bun run dev
|
||||
```
|
||||
|
||||
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org). CUDA-capable GPU recommended (CPU inference supported but slower).
|
||||
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org).
|
||||
|
||||
**Performance:**
|
||||
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration for 4-5x faster inference
|
||||
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU recommended, CPU supported but slower)
|
||||
|
||||
### Project Structure
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@voicebox/app",
|
||||
"version": "0.1.8",
|
||||
"version": "0.1.11",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
@@ -48,6 +48,7 @@
|
||||
"react": "^18.3.0",
|
||||
"react-dom": "^18.3.0",
|
||||
"react-hook-form": "^7.53.0",
|
||||
"react-sound-visualizer": "^1.4.0",
|
||||
"tailwind-merge": "^2.5.4",
|
||||
"wavesurfer.js": "^7.0.0",
|
||||
"zod": "^3.23.8",
|
||||
|
||||
@@ -112,9 +112,6 @@ export function FloatingGenerateBox({
|
||||
}
|
||||
}, [selectedProfileId, profiles, setSelectedProfileId]);
|
||||
|
||||
// Get current form value to trigger resize when it changes
|
||||
const formValue = form.watch(isInstructMode ? 'instruct' : 'text');
|
||||
|
||||
// Auto-resize textarea based on content (only when expanded)
|
||||
useEffect(() => {
|
||||
if (!isExpanded) {
|
||||
@@ -196,59 +193,104 @@ export function FloatingGenerateBox({
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)}>
|
||||
<div className="flex gap-2">
|
||||
<motion.div className="flex-1" transition={{ duration: 0.3, ease: 'easeOut' }}>
|
||||
{isInstructMode && (
|
||||
<span className="text-xs text-accent font-medium mb-1 block">
|
||||
Delivery instructions:
|
||||
</span>
|
||||
)}
|
||||
<FormField
|
||||
control={form.control}
|
||||
name={isInstructMode ? 'instruct' : 'text'}
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
<Textarea
|
||||
{...field}
|
||||
ref={(node: HTMLTextAreaElement | null) => {
|
||||
// Store ref for auto-resize
|
||||
textareaRef.current = node;
|
||||
// Forward ref to react-hook-form
|
||||
if (typeof field.ref === 'function') {
|
||||
field.ref(node);
|
||||
}
|
||||
<motion.div
|
||||
className={cn('flex-1', isExpanded && 'mr-12')}
|
||||
transition={{ duration: 0.3, ease: 'easeOut' }}
|
||||
>
|
||||
{/* Text field - hidden when in instruct mode */}
|
||||
<div style={{ display: isInstructMode ? 'none' : 'block' }}>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="text"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
placeholder={
|
||||
isInstructMode
|
||||
? 'Add delivery instructions...'
|
||||
: isStoriesRoute && currentStory
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
<Textarea
|
||||
{...field}
|
||||
ref={(node: HTMLTextAreaElement | null) => {
|
||||
// Store ref for auto-resize (only for active field)
|
||||
if (!isInstructMode) {
|
||||
textareaRef.current = node;
|
||||
}
|
||||
// Forward ref to react-hook-form
|
||||
if (typeof field.ref === 'function') {
|
||||
field.ref(node);
|
||||
}
|
||||
}}
|
||||
placeholder={
|
||||
isStoriesRoute && currentStory
|
||||
? `Generate speech for "${currentStory.name}"...`
|
||||
: selectedProfile
|
||||
? `Generate speech using ${selectedProfile.name}...`
|
||||
: 'Select a voice profile above...'
|
||||
}
|
||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
}
|
||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
{/* Instruct field - hidden when in text mode */}
|
||||
<div style={{ display: isInstructMode ? 'block' : 'none' }}>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
<Textarea
|
||||
{...field}
|
||||
ref={(node: HTMLTextAreaElement | null) => {
|
||||
// Store ref for auto-resize (only for active field)
|
||||
if (isInstructMode) {
|
||||
textareaRef.current = node;
|
||||
}
|
||||
// Forward ref to react-hook-form
|
||||
if (typeof field.ref === 'function') {
|
||||
field.ref(node);
|
||||
}
|
||||
}}
|
||||
placeholder="Add delivery instructions..."
|
||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
</motion.div>
|
||||
|
||||
<div className="relative shrink-0">
|
||||
@@ -278,9 +320,12 @@ export function FloatingGenerateBox({
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => setIsInstructMode(!isInstructMode)}
|
||||
className={`h-10 w-10 rounded-full bg-card border border-border hover:bg-background/50 transition-all duration-200 ${
|
||||
isInstructMode ? 'text-accent' : ''
|
||||
}`}
|
||||
className={cn(
|
||||
'h-10 w-10 rounded-full transition-all duration-200',
|
||||
isInstructMode
|
||||
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
|
||||
: 'bg-card border border-border hover:bg-background/50',
|
||||
)}
|
||||
>
|
||||
<MessageSquare className="h-4 w-4" />
|
||||
</Button>
|
||||
|
||||
@@ -12,11 +12,10 @@ interface ModelProgressProps {
|
||||
|
||||
export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
||||
const [progress, setProgress] = useState<ModelProgressType | null>(null);
|
||||
const [isSubscribed, setIsSubscribed] = useState(false);
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
|
||||
useEffect(() => {
|
||||
if (!serverUrl || isSubscribed) return;
|
||||
if (!serverUrl) return;
|
||||
|
||||
// Subscribe to progress updates via Server-Sent Events
|
||||
const eventSource = new EventSource(`${serverUrl}/models/progress/${modelName}`);
|
||||
@@ -29,7 +28,6 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
||||
// Close connection if complete or error
|
||||
if (data.status === 'complete' || data.status === 'error') {
|
||||
eventSource.close();
|
||||
setIsSubscribed(false);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error parsing progress event:', error);
|
||||
@@ -39,16 +37,12 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
||||
eventSource.onerror = (error) => {
|
||||
console.error('SSE error:', error);
|
||||
eventSource.close();
|
||||
setIsSubscribed(false);
|
||||
};
|
||||
|
||||
setIsSubscribed(true);
|
||||
|
||||
return () => {
|
||||
eventSource.close();
|
||||
setIsSubscribed(false);
|
||||
};
|
||||
}, [serverUrl, modelName, isSubscribed]);
|
||||
}, [serverUrl, modelName]);
|
||||
|
||||
// Don't render if no progress or if complete/error and some time has passed
|
||||
if (
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { useSortable } from '@dnd-kit/sortable';
|
||||
import { CSS } from '@dnd-kit/utilities';
|
||||
import { GripVertical, Mic, MoreHorizontal, Play, Trash2 } from 'lucide-react';
|
||||
import { useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import {
|
||||
DropdownMenu,
|
||||
@@ -12,6 +13,7 @@ import { Textarea } from '@/components/ui/textarea';
|
||||
import type { StoryItemDetail } from '@/lib/api/types';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { useStoryStore } from '@/stores/storyStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
|
||||
interface StoryChatItemProps {
|
||||
item: StoryItemDetail;
|
||||
@@ -33,6 +35,10 @@ export function StoryChatItem({
|
||||
isDragging,
|
||||
}: StoryChatItemProps) {
|
||||
const seek = useStoryStore((state) => state.seek);
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
const [avatarError, setAvatarError] = useState(false);
|
||||
|
||||
const avatarUrl = `${serverUrl}/profiles/${item.profile_id}/avatar`;
|
||||
|
||||
// Check if this item is currently playing based on timecode
|
||||
const itemStartMs = item.start_time_ms;
|
||||
@@ -72,10 +78,22 @@ export function StoryChatItem({
|
||||
</button>
|
||||
)}
|
||||
|
||||
{/* Voice Icon */}
|
||||
{/* Voice Avatar */}
|
||||
<div className="shrink-0">
|
||||
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center">
|
||||
<Mic className="h-5 w-5 text-muted-foreground" />
|
||||
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center overflow-hidden">
|
||||
{!avatarError ? (
|
||||
<img
|
||||
src={avatarUrl}
|
||||
alt={`${item.profile_name} avatar`}
|
||||
className={cn(
|
||||
'h-full w-full object-cover transition-all duration-200',
|
||||
!isCurrentlyPlaying && 'grayscale'
|
||||
)}
|
||||
onError={() => setAvatarError(true)}
|
||||
/>
|
||||
) : (
|
||||
<Mic className="h-5 w-5 text-muted-foreground" />
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -1,14 +1,5 @@
|
||||
import { Plus, BookOpen, MoreHorizontal, Pencil, Trash2 } from 'lucide-react';
|
||||
import { useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from '@/components/ui/dialog';
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
@@ -19,6 +10,15 @@ import {
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from '@/components/ui/alert-dialog';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from '@/components/ui/dialog';
|
||||
import {
|
||||
DropdownMenu,
|
||||
DropdownMenuContent,
|
||||
@@ -26,18 +26,13 @@ import {
|
||||
DropdownMenuTrigger,
|
||||
} from '@/components/ui/dropdown-menu';
|
||||
import { Input } from '@/components/ui/input';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { Label } from '@/components/ui/label';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { useToast } from '@/components/ui/use-toast';
|
||||
import {
|
||||
useStories,
|
||||
useCreateStory,
|
||||
useUpdateStory,
|
||||
useDeleteStory,
|
||||
} from '@/lib/hooks/useStories';
|
||||
import { useStoryStore } from '@/stores/storyStore';
|
||||
import { useStories, useCreateStory, useUpdateStory, useDeleteStory } from '@/lib/hooks/useStories';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { formatDate } from '@/lib/utils/format';
|
||||
import { useStoryStore } from '@/stores/storyStore';
|
||||
|
||||
export function StoryList() {
|
||||
const { data: stories, isLoading } = useStories();
|
||||
@@ -49,7 +44,11 @@ export function StoryList() {
|
||||
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
||||
const [editDialogOpen, setEditDialogOpen] = useState(false);
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||
const [editingStory, setEditingStory] = useState<{ id: string; name: string; description?: string } | null>(null);
|
||||
const [editingStory, setEditingStory] = useState<{
|
||||
id: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
} | null>(null);
|
||||
const [deletingStoryId, setDeletingStoryId] = useState<string | null>(null);
|
||||
const [newStoryName, setNewStoryName] = useState('');
|
||||
const [newStoryDescription, setNewStoryDescription] = useState('');
|
||||
@@ -186,7 +185,7 @@ export function StoryList() {
|
||||
{/* Story List */}
|
||||
<div className="flex-1 min-h-0 overflow-y-auto space-y-2">
|
||||
{storyList.length === 0 ? (
|
||||
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-md text-muted-foreground">
|
||||
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-2xl text-muted-foreground">
|
||||
<BookOpen className="h-12 w-12 mx-auto mb-4 opacity-50" />
|
||||
<p className="text-sm">No stories yet</p>
|
||||
<p className="text-xs mt-2">Create your first story to get started</p>
|
||||
@@ -196,7 +195,7 @@ export function StoryList() {
|
||||
<div
|
||||
key={story.id}
|
||||
className={cn(
|
||||
'h-24 p-4 border rounded-md transition-colors group flex items-center',
|
||||
'h-24 p-4 border rounded-2xl transition-colors group flex items-center',
|
||||
selectedStoryId === story.id && 'bg-muted border-primary',
|
||||
)}
|
||||
>
|
||||
@@ -213,7 +212,9 @@ export function StoryList() {
|
||||
</p>
|
||||
)}
|
||||
<div className="flex items-center gap-3 mt-2 text-xs text-muted-foreground">
|
||||
<span>{story.item_count} {story.item_count === 1 ? 'item' : 'items'}</span>
|
||||
<span>
|
||||
{story.item_count} {story.item_count === 1 ? 'item' : 'items'}
|
||||
</span>
|
||||
<span>•</span>
|
||||
<span>{formatDate(story.updated_at)}</span>
|
||||
</div>
|
||||
@@ -300,9 +301,7 @@ export function StoryList() {
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Edit Story</DialogTitle>
|
||||
<DialogDescription>
|
||||
Update the story name and description.
|
||||
</DialogDescription>
|
||||
<DialogDescription>Update the story name and description.</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="space-y-4 py-4">
|
||||
<div className="space-y-2">
|
||||
@@ -347,7 +346,8 @@ export function StoryList() {
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>Are you sure?</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
This will permanently delete the story and all its items. This action cannot be undone.
|
||||
This will permanently delete the story and all its items. This action cannot be
|
||||
undone.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
|
||||
@@ -111,6 +111,7 @@ interface StoryTrackEditorProps {
|
||||
}
|
||||
|
||||
const TRACK_HEIGHT = 48;
|
||||
const TIME_RULER_HEIGHT = 24; // h-6 = 1.5rem = 24px
|
||||
const MIN_PIXELS_PER_SECOND = 10;
|
||||
const MAX_PIXELS_PER_SECOND = 200;
|
||||
const DEFAULT_PIXELS_PER_SECOND = 50;
|
||||
@@ -539,7 +540,10 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (e.key === 'Escape') {
|
||||
if (e.key === ' ') {
|
||||
e.preventDefault();
|
||||
handlePlayPause();
|
||||
} else if (e.key === 'Escape') {
|
||||
setSelectedClipId(null);
|
||||
} else if (e.key === 's' || e.key === 'S') {
|
||||
if (selectedClipId) {
|
||||
@@ -561,7 +565,14 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
|
||||
window.addEventListener('keydown', handleKeyDown);
|
||||
return () => window.removeEventListener('keydown', handleKeyDown);
|
||||
}, [selectedClipId, handleSplit, handleDuplicate, handleDelete, setSelectedClipId]);
|
||||
}, [
|
||||
selectedClipId,
|
||||
handleSplit,
|
||||
handleDuplicate,
|
||||
handleDelete,
|
||||
setSelectedClipId,
|
||||
handlePlayPause,
|
||||
]);
|
||||
|
||||
// Add global mouse listeners for trimming
|
||||
useEffect(() => {
|
||||
@@ -586,7 +597,8 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
});
|
||||
setDragPosition({
|
||||
x: rect.left - tracksRef.current.getBoundingClientRect().left + tracksRef.current.scrollLeft,
|
||||
y: rect.top - tracksRef.current.getBoundingClientRect().top,
|
||||
// Subtract ruler height since clips are positioned relative to tracks area, not the scrollable container
|
||||
y: rect.top - tracksRef.current.getBoundingClientRect().top - TIME_RULER_HEIGHT,
|
||||
});
|
||||
setDraggingItem(item.id);
|
||||
};
|
||||
@@ -597,7 +609,8 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
|
||||
const rect = tracksRef.current.getBoundingClientRect();
|
||||
const x = e.clientX - rect.left + tracksRef.current.scrollLeft - dragOffset.x;
|
||||
const y = e.clientY - rect.top - dragOffset.y;
|
||||
// Subtract ruler height since clips are positioned relative to tracks area
|
||||
const y = e.clientY - rect.top - dragOffset.y - TIME_RULER_HEIGHT;
|
||||
|
||||
setDragPosition({ x: Math.max(0, x), y });
|
||||
},
|
||||
@@ -673,6 +686,22 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
// Playhead position
|
||||
const playheadLeft = msToPixels(currentTimeMs);
|
||||
|
||||
// Auto-scroll timeline to follow playhead during playback
|
||||
useEffect(() => {
|
||||
if (!isCurrentlyPlaying || !tracksRef.current) return;
|
||||
|
||||
const container = tracksRef.current;
|
||||
const containerWidth = container.clientWidth;
|
||||
const scrollLeft = container.scrollLeft;
|
||||
const halfwayPoint = scrollLeft + containerWidth / 2;
|
||||
|
||||
// If playhead is past the halfway point, scroll to keep it centered
|
||||
if (playheadLeft > halfwayPoint) {
|
||||
const targetScroll = playheadLeft - containerWidth / 2;
|
||||
container.scrollLeft = targetScroll;
|
||||
}
|
||||
}, [isCurrentlyPlaying, playheadLeft]);
|
||||
|
||||
// Calculate tracks area height
|
||||
const tracksAreaHeight = tracks.length * TRACK_HEIGHT;
|
||||
const timelineContainerHeight = editorHeight - 40; // Subtract toolbar height
|
||||
@@ -701,7 +730,13 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
<div className="flex items-center justify-between px-3 py-2 border-b bg-muted/30 mt-2">
|
||||
{/* Play controls - left side */}
|
||||
<div className="flex items-center gap-2">
|
||||
<Button variant="ghost" size="icon" className="h-7 w-7" onClick={handlePlayPause}>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-7 w-7"
|
||||
onClick={handlePlayPause}
|
||||
title="Play/Pause (Space)"
|
||||
>
|
||||
{isCurrentlyPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
|
||||
</Button>
|
||||
<Button
|
||||
|
||||
@@ -1,8 +1,31 @@
|
||||
import { Mic, Pause, Play, Square } from 'lucide-react';
|
||||
import { memo, useEffect, useState } from 'react';
|
||||
import { Visualizer } from 'react-sound-visualizer';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
||||
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||
|
||||
const MemoizedWaveform = memo(function MemoizedWaveform({
|
||||
audioStream,
|
||||
}: {
|
||||
audioStream: MediaStream;
|
||||
}) {
|
||||
return (
|
||||
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
|
||||
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
|
||||
{({ canvasRef }) => (
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
width={500}
|
||||
height={150}
|
||||
className="w-full h-full"
|
||||
/>
|
||||
)}
|
||||
</Visualizer>
|
||||
</div>
|
||||
);
|
||||
});
|
||||
|
||||
interface AudioSampleRecordingProps {
|
||||
file: File | null | undefined;
|
||||
isRecording: boolean;
|
||||
@@ -14,6 +37,7 @@ interface AudioSampleRecordingProps {
|
||||
onPlayPause: () => void;
|
||||
isPlaying: boolean;
|
||||
isTranscribing?: boolean;
|
||||
showWaveform?: boolean;
|
||||
}
|
||||
|
||||
export function AudioSampleRecording({
|
||||
@@ -27,29 +51,67 @@ export function AudioSampleRecording({
|
||||
onPlayPause,
|
||||
isPlaying,
|
||||
isTranscribing = false,
|
||||
showWaveform = true,
|
||||
}: AudioSampleRecordingProps) {
|
||||
const [audioStream, setAudioStream] = useState<MediaStream | null>(null);
|
||||
|
||||
// Request microphone access when component mounts
|
||||
useEffect(() => {
|
||||
if (!showWaveform) return;
|
||||
|
||||
let stream: MediaStream | null = null;
|
||||
|
||||
navigator.mediaDevices
|
||||
.getUserMedia({ audio: true, video: false })
|
||||
.then((s) => {
|
||||
stream = s;
|
||||
setAudioStream(s);
|
||||
})
|
||||
.catch((err) => {
|
||||
console.warn('Could not access microphone for visualization:', err);
|
||||
});
|
||||
|
||||
return () => {
|
||||
if (stream) {
|
||||
stream.getTracks().forEach((track) => {
|
||||
track.stop();
|
||||
});
|
||||
}
|
||||
};
|
||||
}, [showWaveform]);
|
||||
|
||||
return (
|
||||
<FormItem>
|
||||
<FormLabel>Record Audio</FormLabel>
|
||||
<FormControl>
|
||||
<div className="space-y-4">
|
||||
{!isRecording && !file && (
|
||||
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px]">
|
||||
<Button type="button" onClick={onStart} size="lg" className="flex items-center gap-2">
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
<Button
|
||||
type="button"
|
||||
onClick={onStart}
|
||||
size="lg"
|
||||
className="relative z-10 flex items-center gap-2"
|
||||
>
|
||||
<Mic className="h-5 w-5" />
|
||||
Start Recording
|
||||
</Button>
|
||||
<p className="text-sm text-muted-foreground text-center">
|
||||
<p className="relative z-10 text-sm text-muted-foreground text-center">
|
||||
Click to start recording. Maximum duration: 30 seconds.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{isRecording && (
|
||||
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-destructive rounded-lg bg-destructive/5 min-h-[180px]">
|
||||
<div className="flex items-center gap-4">
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
<div className="relative z-10 flex items-center gap-4">
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="h-3 w-3 rounded-full bg-destructive animate-pulse" />
|
||||
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
|
||||
<span className="text-lg font-mono font-semibold">
|
||||
{formatAudioDuration(duration)}
|
||||
</span>
|
||||
@@ -58,13 +120,12 @@ export function AudioSampleRecording({
|
||||
<Button
|
||||
type="button"
|
||||
onClick={onStop}
|
||||
variant="destructive"
|
||||
className="flex items-center gap-2"
|
||||
className="relative z-10 flex items-center gap-2 bg-accent text-accent-foreground hover:bg-accent/90"
|
||||
>
|
||||
<Square className="h-4 w-4" />
|
||||
Stop Recording
|
||||
</Button>
|
||||
<p className="text-sm text-muted-foreground text-center">
|
||||
<p className="relative z-10 text-sm text-muted-foreground text-center">
|
||||
{formatAudioDuration(30 - duration)} remaining
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { Mic, Monitor, Pause, Play, Square } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
||||
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||
|
||||
interface AudioSampleSystemProps {
|
||||
@@ -30,7 +30,6 @@ export function AudioSampleSystem({
|
||||
}: AudioSampleSystemProps) {
|
||||
return (
|
||||
<FormItem>
|
||||
<FormLabel>Capture System Audio</FormLabel>
|
||||
<FormControl>
|
||||
<div className="space-y-4">
|
||||
{!isRecording && !file && (
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { Mic, Pause, Play, Upload } from 'lucide-react';
|
||||
import { useRef, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
||||
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||
|
||||
interface AudioSampleUploadProps {
|
||||
file: File | null | undefined;
|
||||
@@ -31,7 +31,6 @@ export function AudioSampleUpload({
|
||||
|
||||
return (
|
||||
<FormItem>
|
||||
<FormLabel>Audio File</FormLabel>
|
||||
<FormControl>
|
||||
<div className="flex flex-col gap-2">
|
||||
<input
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
import type { VoiceProfileResponse } from '@/lib/api/types';
|
||||
import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
|
||||
interface ProfileCardProps {
|
||||
@@ -23,15 +24,19 @@ interface ProfileCardProps {
|
||||
|
||||
export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||
const [avatarError, setAvatarError] = useState(false);
|
||||
const deleteProfile = useDeleteProfile();
|
||||
const exportProfile = useExportProfile();
|
||||
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
||||
const setProfileDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
|
||||
const isSelected = selectedProfileId === profile.id;
|
||||
|
||||
const avatarUrl = profile.avatar_path ? `${serverUrl}/profiles/${profile.id}/avatar` : null;
|
||||
|
||||
const handleSelect = () => {
|
||||
setSelectedProfileId(isSelected ? null : profile.id);
|
||||
};
|
||||
@@ -67,8 +72,20 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
>
|
||||
<CardHeader className="p-3 pb-2">
|
||||
<CardTitle className="flex items-center gap-1.5 text-base font-medium">
|
||||
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0">
|
||||
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
|
||||
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden">
|
||||
{avatarUrl && !avatarError ? (
|
||||
<img
|
||||
src={avatarUrl}
|
||||
alt={`${profile.name} avatar`}
|
||||
className={cn(
|
||||
'h-full w-full object-cover transition-all duration-200',
|
||||
!isSelected && 'grayscale',
|
||||
)}
|
||||
onError={() => setAvatarError(true)}
|
||||
/>
|
||||
) : (
|
||||
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
|
||||
)}
|
||||
</div>
|
||||
<span className="break-words">{profile.name}</span>
|
||||
</CardTitle>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { zodResolver } from '@hookform/resolvers/zod';
|
||||
import { Mic, Monitor, Upload } from 'lucide-react';
|
||||
import { useEffect, useState } from 'react';
|
||||
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { useForm } from 'react-hook-form';
|
||||
import * as z from 'zod';
|
||||
import { Button } from '@/components/ui/button';
|
||||
@@ -36,14 +36,17 @@ import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
|
||||
import {
|
||||
useAddSample,
|
||||
useCreateProfile,
|
||||
useDeleteAvatar,
|
||||
useProfile,
|
||||
useUpdateProfile,
|
||||
useUploadAvatar,
|
||||
} from '@/lib/hooks/useProfiles';
|
||||
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
|
||||
import { useTranscription } from '@/lib/hooks/useTranscription';
|
||||
import { isTauri } from '@/lib/tauri';
|
||||
import { formatAudioDuration, getAudioDuration } from '@/lib/utils/audio';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { type ProfileFormDraft, useUIStore } from '@/stores/uiStore';
|
||||
import { AudioSampleRecording } from './AudioSampleRecording';
|
||||
import { AudioSampleSystem } from './AudioSampleSystem';
|
||||
import { AudioSampleUpload } from './AudioSampleUpload';
|
||||
@@ -57,6 +60,7 @@ const baseProfileSchema = z.object({
|
||||
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
||||
sampleFile: z.instanceof(File).optional(),
|
||||
referenceText: z.string().max(1000).optional(),
|
||||
avatarFile: z.instanceof(File).optional(),
|
||||
});
|
||||
|
||||
const profileSchema = baseProfileSchema.refine(
|
||||
@@ -75,22 +79,51 @@ const profileSchema = baseProfileSchema.refine(
|
||||
|
||||
type ProfileFormValues = z.infer<typeof profileSchema>;
|
||||
|
||||
// Helper to convert File to base64
|
||||
async function fileToBase64(file: File): Promise<string> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => resolve(reader.result as string);
|
||||
reader.onerror = reject;
|
||||
reader.readAsDataURL(file);
|
||||
});
|
||||
}
|
||||
|
||||
// Helper to convert base64 to File
|
||||
function base64ToFile(base64: string, fileName: string, fileType: string): File {
|
||||
const arr = base64.split(',');
|
||||
const bstr = atob(arr[1]);
|
||||
let n = bstr.length;
|
||||
const u8arr = new Uint8Array(n);
|
||||
while (n--) {
|
||||
u8arr[n] = bstr.charCodeAt(n);
|
||||
}
|
||||
return new File([u8arr], fileName, { type: fileType });
|
||||
}
|
||||
|
||||
export function ProfileForm() {
|
||||
const open = useUIStore((state) => state.profileDialogOpen);
|
||||
const setOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||
const editingProfileId = useUIStore((state) => state.editingProfileId);
|
||||
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
||||
const profileFormDraft = useUIStore((state) => state.profileFormDraft);
|
||||
const setProfileFormDraft = useUIStore((state) => state.setProfileFormDraft);
|
||||
const { data: editingProfile } = useProfile(editingProfileId || '');
|
||||
const createProfile = useCreateProfile();
|
||||
const updateProfile = useUpdateProfile();
|
||||
const addSample = useAddSample();
|
||||
const uploadAvatar = useUploadAvatar();
|
||||
const deleteAvatar = useDeleteAvatar();
|
||||
const transcribe = useTranscription();
|
||||
const { toast } = useToast();
|
||||
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('upload');
|
||||
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
|
||||
const [audioDuration, setAudioDuration] = useState<number | null>(null);
|
||||
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
|
||||
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
|
||||
const avatarInputRef = useRef<HTMLInputElement>(null);
|
||||
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
|
||||
const isCreating = !editingProfileId;
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
|
||||
const form = useForm<ProfileFormValues>({
|
||||
resolver: zodResolver(profileSchema),
|
||||
@@ -100,10 +133,12 @@ export function ProfileForm() {
|
||||
language: 'en',
|
||||
sampleFile: undefined,
|
||||
referenceText: '',
|
||||
avatarFile: undefined,
|
||||
},
|
||||
});
|
||||
|
||||
const selectedFile = form.watch('sampleFile');
|
||||
const selectedAvatarFile = form.watch('avatarFile');
|
||||
|
||||
// Validate audio duration when file is selected
|
||||
useEffect(() => {
|
||||
@@ -220,6 +255,20 @@ export function ProfileForm() {
|
||||
}
|
||||
}, [systemRecordingError, toast]);
|
||||
|
||||
// Handle avatar preview
|
||||
useEffect(() => {
|
||||
if (selectedAvatarFile instanceof File) {
|
||||
const url = URL.createObjectURL(selectedAvatarFile);
|
||||
setAvatarPreview(url);
|
||||
return () => URL.revokeObjectURL(url);
|
||||
} else if (editingProfile?.avatar_path) {
|
||||
setAvatarPreview(`${serverUrl}/profiles/${editingProfile.id}/avatar`);
|
||||
} else {
|
||||
setAvatarPreview(null);
|
||||
}
|
||||
}, [selectedAvatarFile, editingProfile, serverUrl]);
|
||||
|
||||
// Restore form state from draft or editing profile
|
||||
useEffect(() => {
|
||||
if (editingProfile) {
|
||||
form.reset({
|
||||
@@ -228,18 +277,46 @@ export function ProfileForm() {
|
||||
language: editingProfile.language as LanguageCode,
|
||||
sampleFile: undefined,
|
||||
referenceText: undefined,
|
||||
avatarFile: undefined,
|
||||
});
|
||||
} else {
|
||||
} else if (profileFormDraft && open) {
|
||||
// Restore from draft when opening in create mode
|
||||
form.reset({
|
||||
name: profileFormDraft.name,
|
||||
description: profileFormDraft.description,
|
||||
language: profileFormDraft.language as LanguageCode,
|
||||
referenceText: profileFormDraft.referenceText,
|
||||
sampleFile: undefined,
|
||||
avatarFile: undefined,
|
||||
});
|
||||
setSampleMode(profileFormDraft.sampleMode);
|
||||
// Restore the file if we have it saved
|
||||
if (
|
||||
profileFormDraft.sampleFileData &&
|
||||
profileFormDraft.sampleFileName &&
|
||||
profileFormDraft.sampleFileType
|
||||
) {
|
||||
const file = base64ToFile(
|
||||
profileFormDraft.sampleFileData,
|
||||
profileFormDraft.sampleFileName,
|
||||
profileFormDraft.sampleFileType,
|
||||
);
|
||||
form.setValue('sampleFile', file);
|
||||
}
|
||||
} else if (!open) {
|
||||
// Only reset to defaults when modal is closed and no draft
|
||||
form.reset({
|
||||
name: '',
|
||||
description: '',
|
||||
language: 'en',
|
||||
sampleFile: undefined,
|
||||
referenceText: undefined,
|
||||
avatarFile: undefined,
|
||||
});
|
||||
setSampleMode('upload');
|
||||
setSampleMode('record');
|
||||
setAvatarPreview(null);
|
||||
}
|
||||
}, [editingProfile, form]);
|
||||
}, [editingProfile, profileFormDraft, open, form]);
|
||||
|
||||
async function handleTranscribe() {
|
||||
const file = form.getValues('sampleFile');
|
||||
@@ -281,6 +358,52 @@ export function ProfileForm() {
|
||||
playPause(file);
|
||||
}
|
||||
|
||||
function handleAvatarFileChange(e: React.ChangeEvent<HTMLInputElement>) {
|
||||
const file = e.target.files?.[0];
|
||||
if (file) {
|
||||
if (!file.type.startsWith('image/')) {
|
||||
toast({
|
||||
title: 'Invalid file type',
|
||||
description: 'Please select an image file (PNG, JPG, or WebP)',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (file.size > 5 * 1024 * 1024) {
|
||||
toast({
|
||||
title: 'File too large',
|
||||
description: 'Image must be less than 5MB',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
}
|
||||
form.setValue('avatarFile', file);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleRemoveAvatar() {
|
||||
if (editingProfileId && editingProfile?.avatar_path) {
|
||||
try {
|
||||
await deleteAvatar.mutateAsync(editingProfileId);
|
||||
toast({
|
||||
title: 'Avatar removed',
|
||||
description: 'Avatar image has been removed successfully.',
|
||||
});
|
||||
} catch (error) {
|
||||
toast({
|
||||
title: 'Failed to remove avatar',
|
||||
description: error instanceof Error ? error.message : 'Unknown error',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
}
|
||||
form.setValue('avatarFile', undefined);
|
||||
setAvatarPreview(null);
|
||||
if (avatarInputRef.current) {
|
||||
avatarInputRef.current.value = '';
|
||||
}
|
||||
}
|
||||
|
||||
async function onSubmit(data: ProfileFormValues) {
|
||||
try {
|
||||
if (editingProfileId) {
|
||||
@@ -293,6 +416,24 @@ export function ProfileForm() {
|
||||
language: data.language,
|
||||
},
|
||||
});
|
||||
|
||||
// Handle avatar upload/update if file changed
|
||||
if (data.avatarFile) {
|
||||
try {
|
||||
await uploadAvatar.mutateAsync({
|
||||
profileId: editingProfileId,
|
||||
file: data.avatarFile,
|
||||
});
|
||||
} catch (avatarError) {
|
||||
toast({
|
||||
title: 'Avatar upload failed',
|
||||
description:
|
||||
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
toast({
|
||||
title: 'Voice updated',
|
||||
description: `"${data.name}" has been updated successfully.`,
|
||||
@@ -369,6 +510,24 @@ export function ProfileForm() {
|
||||
file: sampleFile,
|
||||
referenceText: referenceText,
|
||||
});
|
||||
|
||||
// Handle avatar upload if provided
|
||||
if (data.avatarFile) {
|
||||
try {
|
||||
await uploadAvatar.mutateAsync({
|
||||
profileId: profile.id,
|
||||
file: data.avatarFile,
|
||||
});
|
||||
} catch (avatarError) {
|
||||
toast({
|
||||
title: 'Avatar upload failed',
|
||||
description:
|
||||
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
toast({
|
||||
title: 'Profile created',
|
||||
description: `"${data.name}" has been created with a sample.`,
|
||||
@@ -383,6 +542,8 @@ export function ProfileForm() {
|
||||
}
|
||||
}
|
||||
|
||||
// Clear draft and reset form on success
|
||||
setProfileFormDraft(null);
|
||||
form.reset();
|
||||
setEditingProfileId(null);
|
||||
setOpen(false);
|
||||
@@ -395,12 +556,41 @@ export function ProfileForm() {
|
||||
}
|
||||
}
|
||||
|
||||
function handleOpenChange(open: boolean) {
|
||||
setOpen(open);
|
||||
if (!open) {
|
||||
async function handleOpenChange(newOpen: boolean) {
|
||||
if (!newOpen && isCreating) {
|
||||
// Save draft when closing the create modal
|
||||
const values = form.getValues();
|
||||
const hasContent =
|
||||
values.name || values.description || values.referenceText || values.sampleFile;
|
||||
|
||||
if (hasContent) {
|
||||
const draft: ProfileFormDraft = {
|
||||
name: values.name || '',
|
||||
description: values.description || '',
|
||||
language: values.language || 'en',
|
||||
referenceText: values.referenceText || '',
|
||||
sampleMode,
|
||||
};
|
||||
|
||||
// Save file as base64 if present
|
||||
if (values.sampleFile) {
|
||||
try {
|
||||
draft.sampleFileName = values.sampleFile.name;
|
||||
draft.sampleFileType = values.sampleFile.type;
|
||||
draft.sampleFileData = await fileToBase64(values.sampleFile);
|
||||
} catch {
|
||||
// If file conversion fails, just don't save the file
|
||||
}
|
||||
}
|
||||
|
||||
setProfileFormDraft(draft);
|
||||
}
|
||||
}
|
||||
|
||||
setOpen(newOpen);
|
||||
if (!newOpen) {
|
||||
setEditingProfileId(null);
|
||||
form.reset();
|
||||
setSampleMode('upload');
|
||||
// Don't reset form here - let the effect handle it based on draft state
|
||||
if (isRecording) {
|
||||
cancelRecording();
|
||||
}
|
||||
@@ -413,174 +603,119 @@ export function ProfileForm() {
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={handleOpenChange}>
|
||||
<DialogContent className="max-w-4xl">
|
||||
<DialogHeader>
|
||||
<DialogTitle>{editingProfileId ? 'Edit Voice' : 'Create Voice Profile'}</DialogTitle>
|
||||
<DialogDescription>
|
||||
{editingProfileId
|
||||
? 'Update your voice profile details and manage samples.'
|
||||
: 'Create a new voice profile with an audio sample to clone the voice.'}
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)}>
|
||||
<div className="grid gap-6 grid-cols-2">
|
||||
{/* Left column: Profile info */}
|
||||
<div className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="name"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Name</FormLabel>
|
||||
<FormControl>
|
||||
<Input placeholder="My Voice" {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="description"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Description (Optional)</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea placeholder="Describe this voice..." {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="language"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Language</FormLabel>
|
||||
<Select onValueChange={field.onChange} defaultValue={field.value}>
|
||||
<FormControl>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
{LANGUAGE_OPTIONS.map((lang) => (
|
||||
<SelectItem key={lang.value} value={lang.value}>
|
||||
{lang.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
|
||||
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
|
||||
<DialogHeader>
|
||||
<DialogTitle className="text-2xl">
|
||||
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
|
||||
</DialogTitle>
|
||||
<DialogDescription>
|
||||
{editingProfileId
|
||||
? 'Update your voice profile details and manage samples.'
|
||||
: 'Create a new voice profile with an audio sample to clone the voice.'}
|
||||
</DialogDescription>
|
||||
{isCreating && profileFormDraft && (
|
||||
<div className="flex items-center gap-2 pt-2">
|
||||
<span className="text-xs text-muted-foreground">Draft restored</span>
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="h-6 px-2 text-xs text-muted-foreground"
|
||||
onClick={() => {
|
||||
setProfileFormDraft(null);
|
||||
form.reset({
|
||||
name: '',
|
||||
description: '',
|
||||
language: 'en',
|
||||
sampleFile: undefined,
|
||||
referenceText: '',
|
||||
});
|
||||
setSampleMode('record');
|
||||
}}
|
||||
>
|
||||
<X className="h-3 w-3 mr-1" />
|
||||
Discard
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</DialogHeader>
|
||||
|
||||
{/* Right column: Sample management */}
|
||||
<div className="space-y-4 border-l pl-6">
|
||||
{isCreating ? (
|
||||
<>
|
||||
<div>
|
||||
<h3 className="text-sm font-medium mb-2">Add Sample</h3>
|
||||
<p className="text-sm text-muted-foreground mb-4">
|
||||
Provide an audio sample to clone the voice. You can add more samples later.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<Tabs
|
||||
value={sampleMode}
|
||||
onValueChange={(v) => {
|
||||
const newMode = v as 'upload' | 'record' | 'system';
|
||||
// Cancel any active recordings when switching modes
|
||||
if (isRecording && newMode !== 'record') {
|
||||
cancelRecording();
|
||||
}
|
||||
if (isSystemRecording && newMode !== 'system') {
|
||||
cancelSystemRecording();
|
||||
}
|
||||
setSampleMode(newMode);
|
||||
}}
|
||||
>
|
||||
<TabsList
|
||||
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
|
||||
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
|
||||
{/* Left column: Sample management */}
|
||||
<div className="space-y-4 border-r pr-6">
|
||||
{isCreating ? (
|
||||
<>
|
||||
<Tabs
|
||||
className="pt-4"
|
||||
value={sampleMode}
|
||||
onValueChange={(v) => {
|
||||
const newMode = v as 'upload' | 'record' | 'system';
|
||||
// Cancel any active recordings when switching modes
|
||||
if (isRecording && newMode !== 'record') {
|
||||
cancelRecording();
|
||||
}
|
||||
if (isSystemRecording && newMode !== 'system') {
|
||||
cancelSystemRecording();
|
||||
}
|
||||
setSampleMode(newMode);
|
||||
}}
|
||||
>
|
||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||
<Upload className="h-4 w-4 shrink-0" />
|
||||
Upload
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="record" className="flex items-center gap-2">
|
||||
<Mic className="h-4 w-4 shrink-0" />
|
||||
Record
|
||||
</TabsTrigger>
|
||||
{isTauri() && isSystemAudioSupported && (
|
||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||
<Monitor className="h-4 w-4 shrink-0" />
|
||||
System Audio
|
||||
<TabsList
|
||||
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||
>
|
||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||
<Upload className="h-4 w-4 shrink-0" />
|
||||
Upload
|
||||
</TabsTrigger>
|
||||
)}
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="upload" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={({ field: { onChange, name } }) => (
|
||||
<AudioSampleUpload
|
||||
file={selectedFile}
|
||||
onFileChange={onChange}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isValidating={isValidatingAudio}
|
||||
isTranscribing={transcribe.isPending}
|
||||
isDisabled={
|
||||
audioDuration !== null && audioDuration > MAX_AUDIO_DURATION_SECONDS
|
||||
}
|
||||
fieldName={name}
|
||||
/>
|
||||
<TabsTrigger value="record" className="flex items-center gap-2">
|
||||
<Mic className="h-4 w-4 shrink-0" />
|
||||
Record
|
||||
</TabsTrigger>
|
||||
{isTauri() && isSystemAudioSupported && (
|
||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||
<Monitor className="h-4 w-4 shrink-0" />
|
||||
System Audio
|
||||
</TabsTrigger>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="record" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleRecording
|
||||
file={selectedFile}
|
||||
isRecording={isRecording}
|
||||
duration={duration}
|
||||
onStart={startRecording}
|
||||
onStop={stopRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
<TabsContent value="upload" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={({ field: { onChange, name } }) => (
|
||||
<AudioSampleUpload
|
||||
file={selectedFile}
|
||||
onFileChange={onChange}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isValidating={isValidatingAudio}
|
||||
isTranscribing={transcribe.isPending}
|
||||
isDisabled={
|
||||
audioDuration !== null &&
|
||||
audioDuration > MAX_AUDIO_DURATION_SECONDS
|
||||
}
|
||||
fieldName={name}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
|
||||
{isTauri() && isSystemAudioSupported && (
|
||||
<TabsContent value="system" className="space-y-4">
|
||||
<TabsContent value="record" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleSystem
|
||||
<AudioSampleRecording
|
||||
file={selectedFile}
|
||||
isRecording={isSystemRecording}
|
||||
duration={systemDuration}
|
||||
onStart={startSystemRecording}
|
||||
onStop={stopSystemRecording}
|
||||
isRecording={isRecording}
|
||||
duration={duration}
|
||||
onStart={startRecording}
|
||||
onStop={stopRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
@@ -590,55 +725,188 @@ export function ProfileForm() {
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
)}
|
||||
</Tabs>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="referenceText"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Reference Text</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea
|
||||
placeholder="Enter the exact text spoken in the audio..."
|
||||
className="min-h-[100px]"
|
||||
{...field}
|
||||
{isTauri() && isSystemAudioSupported && (
|
||||
<TabsContent value="system" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleSystem
|
||||
file={selectedFile}
|
||||
isRecording={isSystemRecording}
|
||||
duration={systemDuration}
|
||||
onStart={startSystemRecording}
|
||||
onStop={stopSystemRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</>
|
||||
) : (
|
||||
// Show sample list when editing
|
||||
editingProfileId && (
|
||||
<div>
|
||||
<SampleList profileId={editingProfileId} />
|
||||
</div>
|
||||
)
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</TabsContent>
|
||||
)}
|
||||
</Tabs>
|
||||
|
||||
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
|
||||
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
type="submit"
|
||||
disabled={createProfile.isPending || updateProfile.isPending || addSample.isPending}
|
||||
>
|
||||
{createProfile.isPending || updateProfile.isPending || addSample.isPending
|
||||
? 'Saving...'
|
||||
: editingProfileId
|
||||
? 'Save Changes'
|
||||
: 'Create Profile'}
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
</Form>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="referenceText"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Reference Text</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea
|
||||
placeholder="Enter the exact text spoken in the audio..."
|
||||
className="min-h-[100px]"
|
||||
{...field}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</>
|
||||
) : (
|
||||
// Show sample list when editing
|
||||
editingProfileId && (
|
||||
<div>
|
||||
<SampleList profileId={editingProfileId} />
|
||||
</div>
|
||||
)
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Right column: Profile info */}
|
||||
<div className="space-y-4">
|
||||
{/* Avatar Upload */}
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="avatarFile"
|
||||
render={() => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<div className="flex justify-center pt-4 pb-2">
|
||||
<div className="relative group">
|
||||
<div className="h-24 w-24 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden border-2 border-border">
|
||||
{avatarPreview ? (
|
||||
<img
|
||||
src={avatarPreview}
|
||||
alt="Avatar preview"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
) : (
|
||||
<Mic className="h-10 w-10 text-muted-foreground" />
|
||||
)}
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => avatarInputRef.current?.click()}
|
||||
className="absolute inset-0 rounded-full bg-accent/60 opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center cursor-pointer"
|
||||
>
|
||||
<Edit2 className="h-6 w-6 text-accent-foreground" />
|
||||
</button>
|
||||
{(avatarPreview || editingProfile?.avatar_path) && (
|
||||
<button
|
||||
type="button"
|
||||
onClick={handleRemoveAvatar}
|
||||
disabled={deleteAvatar.isPending}
|
||||
className="absolute bottom-0 right-0 h-6 w-6 rounded-full bg-background/60 backdrop-blur-sm text-muted-foreground flex items-center justify-center hover:bg-background/80 hover:text-foreground transition-colors shadow-sm border border-border/50"
|
||||
>
|
||||
<X className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<input
|
||||
ref={avatarInputRef}
|
||||
type="file"
|
||||
accept="image/png,image/jpeg,image/webp"
|
||||
onChange={handleAvatarFileChange}
|
||||
className="hidden"
|
||||
/>
|
||||
</div>
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="name"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Name</FormLabel>
|
||||
<FormControl>
|
||||
<Input placeholder="My Voice" {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="description"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Description (Optional)</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea placeholder="Describe this voice..." {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="language"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Language</FormLabel>
|
||||
<Select onValueChange={field.onChange} defaultValue={field.value}>
|
||||
<FormControl>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
{LANGUAGE_OPTIONS.map((lang) => (
|
||||
<SelectItem key={lang.value} value={lang.value}>
|
||||
{lang.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
|
||||
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
type="submit"
|
||||
disabled={
|
||||
createProfile.isPending || updateProfile.isPending || addSample.isPending
|
||||
}
|
||||
>
|
||||
{createProfile.isPending || updateProfile.isPending || addSample.isPending
|
||||
? 'Saving...'
|
||||
: editingProfileId
|
||||
? 'Save Changes'
|
||||
: 'Create Profile'}
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
</Form>
|
||||
</div>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
|
||||
@@ -1,11 +1,133 @@
|
||||
import { Plus, Trash2, Play } from 'lucide-react';
|
||||
import { useState } from 'react';
|
||||
import { Plus, Trash2, Play, Edit, Check, X, Volume2, Pause } from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { CircleButton } from '@/components/ui/circle-button';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { Slider } from '@/components/ui/slider';
|
||||
import { useToast } from '@/components/ui/use-toast';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import { useDeleteSample, useProfileSamples } from '@/lib/hooks/useProfiles';
|
||||
import { usePlayerStore } from '@/stores/playerStore';
|
||||
import { useDeleteSample, useProfileSamples, useUpdateSample } from '@/lib/hooks/useProfiles';
|
||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { SampleUpload } from './SampleUpload';
|
||||
|
||||
interface MiniSamplePlayerProps {
|
||||
audioUrl: string;
|
||||
}
|
||||
|
||||
function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null);
|
||||
const [isPlaying, setIsPlaying] = useState(false);
|
||||
const [currentTime, setCurrentTime] = useState(0);
|
||||
const [duration, setDuration] = useState(0);
|
||||
const [isLoading, setIsLoading] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
const audio = new Audio(audioUrl);
|
||||
audioRef.current = audio;
|
||||
|
||||
const handleLoadedMetadata = () => {
|
||||
setDuration(audio.duration);
|
||||
setIsLoading(false);
|
||||
};
|
||||
|
||||
const handleTimeUpdate = () => {
|
||||
setCurrentTime(audio.currentTime);
|
||||
};
|
||||
|
||||
const handleEnded = () => {
|
||||
setIsPlaying(false);
|
||||
setCurrentTime(0);
|
||||
};
|
||||
|
||||
const handlePlay = () => setIsPlaying(true);
|
||||
const handlePause = () => setIsPlaying(false);
|
||||
|
||||
audio.addEventListener('loadedmetadata', handleLoadedMetadata);
|
||||
audio.addEventListener('timeupdate', handleTimeUpdate);
|
||||
audio.addEventListener('ended', handleEnded);
|
||||
audio.addEventListener('play', handlePlay);
|
||||
audio.addEventListener('pause', handlePause);
|
||||
|
||||
return () => {
|
||||
audio.pause();
|
||||
audio.removeEventListener('loadedmetadata', handleLoadedMetadata);
|
||||
audio.removeEventListener('timeupdate', handleTimeUpdate);
|
||||
audio.removeEventListener('ended', handleEnded);
|
||||
audio.removeEventListener('play', handlePlay);
|
||||
audio.removeEventListener('pause', handlePause);
|
||||
audio.src = '';
|
||||
};
|
||||
}, [audioUrl]);
|
||||
|
||||
const handlePlayPause = () => {
|
||||
if (!audioRef.current) return;
|
||||
if (isPlaying) {
|
||||
audioRef.current.pause();
|
||||
} else {
|
||||
audioRef.current.play();
|
||||
}
|
||||
};
|
||||
|
||||
const handleSeek = (value: number[]) => {
|
||||
if (!audioRef.current || duration === 0) return;
|
||||
const progress = value[0] / 100;
|
||||
audioRef.current.currentTime = progress * duration;
|
||||
};
|
||||
|
||||
const handleStop = () => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause();
|
||||
audioRef.current.currentTime = 0;
|
||||
}
|
||||
setIsPlaying(false);
|
||||
setCurrentTime(0);
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="border-t bg-muted/30 px-3 py-2 mt-2">
|
||||
<div className="flex items-center gap-2">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-7 w-7 shrink-0"
|
||||
onClick={handlePlayPause}
|
||||
disabled={isLoading}
|
||||
>
|
||||
{isPlaying ? <Pause className="h-3.5 w-3.5" /> : <Play className="h-3.5 w-3.5 ml-0.5" />}
|
||||
</Button>
|
||||
|
||||
<div className="flex-1 min-w-0 flex items-center gap-2">
|
||||
<Slider
|
||||
value={duration > 0 ? [(currentTime / duration) * 100] : [0]}
|
||||
onValueChange={handleSeek}
|
||||
max={100}
|
||||
step={0.1}
|
||||
className="flex-1"
|
||||
/>
|
||||
<div className="flex items-center gap-1 text-xs text-muted-foreground shrink-0 min-w-[70px]">
|
||||
<span className="font-mono">{formatAudioDuration(currentTime)}</span>
|
||||
<span>/</span>
|
||||
<span className="font-mono">{formatAudioDuration(duration)}</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-7 w-7 shrink-0"
|
||||
onClick={handleStop}
|
||||
title="Stop"
|
||||
>
|
||||
<X className="h-3.5 w-3.5" />
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
interface SampleListProps {
|
||||
profileId: string;
|
||||
}
|
||||
@@ -13,10 +135,11 @@ interface SampleListProps {
|
||||
export function SampleList({ profileId }: SampleListProps) {
|
||||
const { data: samples, isLoading } = useProfileSamples(profileId);
|
||||
const deleteSample = useDeleteSample();
|
||||
const updateSample = useUpdateSample();
|
||||
const { toast } = useToast();
|
||||
const [uploadOpen, setUploadOpen] = useState(false);
|
||||
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
|
||||
const currentAudioId = usePlayerStore((state) => state.audioId);
|
||||
const isPlaying = usePlayerStore((state) => state.isPlaying);
|
||||
const [editingSampleId, setEditingSampleId] = useState<string | null>(null);
|
||||
const [editedText, setEditedText] = useState<string>('');
|
||||
|
||||
const handleDelete = (sampleId: string) => {
|
||||
if (confirm('Are you sure you want to delete this sample?')) {
|
||||
@@ -24,9 +147,41 @@ export function SampleList({ profileId }: SampleListProps) {
|
||||
}
|
||||
};
|
||||
|
||||
const handlePlay = (referenceText: string, sampleId: string) => {
|
||||
const audioUrl = apiClient.getSampleUrl(sampleId);
|
||||
setAudioWithAutoPlay(audioUrl, sampleId, null, referenceText.substring(0, 50));
|
||||
const handleStartEdit = (sampleId: string, currentText: string) => {
|
||||
setEditingSampleId(sampleId);
|
||||
setEditedText(currentText);
|
||||
};
|
||||
|
||||
const handleCancelEdit = () => {
|
||||
setEditingSampleId(null);
|
||||
setEditedText('');
|
||||
};
|
||||
|
||||
const handleSaveEdit = async (sampleId: string) => {
|
||||
if (!editedText.trim()) {
|
||||
toast({
|
||||
title: 'Invalid text',
|
||||
description: 'Reference text cannot be empty.',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await updateSample.mutateAsync({ sampleId, referenceText: editedText.trim() });
|
||||
toast({
|
||||
title: 'Sample updated',
|
||||
description: 'Reference text has been updated successfully.',
|
||||
});
|
||||
setEditingSampleId(null);
|
||||
setEditedText('');
|
||||
} catch (error) {
|
||||
toast({
|
||||
title: 'Update failed',
|
||||
description: error instanceof Error ? error.message : 'Failed to update sample',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
if (isLoading) {
|
||||
@@ -34,56 +189,109 @@ export function SampleList({ profileId }: SampleListProps) {
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<div className="flex items-center justify-between">
|
||||
<h3 className="text-lg font-semibold">Audio Samples</h3>
|
||||
<Button type="button" size="sm" onClick={() => setUploadOpen(true)}>
|
||||
<Plus className="mr-2 h-4 w-4" />
|
||||
Add Sample
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<div className="space-y-4 pt-4">
|
||||
{samples && samples.length === 0 ? (
|
||||
<div className="text-sm text-muted-foreground py-4">
|
||||
No samples yet. Add your first audio sample.
|
||||
<div className="flex flex-col items-center justify-center py-8 text-center border border-dashed rounded-lg">
|
||||
<Volume2 className="h-8 w-8 text-muted-foreground/50 mb-2" />
|
||||
<p className="text-sm text-muted-foreground">No samples yet</p>
|
||||
<p className="text-xs text-muted-foreground/70 mt-1">Add your first audio sample to get started</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-2">
|
||||
{samples?.map((sample) => (
|
||||
<div
|
||||
key={sample.id}
|
||||
className="flex items-center justify-between p-3 border rounded-lg"
|
||||
>
|
||||
<div className="flex-1">
|
||||
<p className="text-sm font-medium">{sample.reference_text}</p>
|
||||
<p className="text-xs text-muted-foreground mt-1">{sample.audio_path}</p>
|
||||
{samples?.map((sample, index) => {
|
||||
const isEditing = editingSampleId === sample.id;
|
||||
|
||||
return (
|
||||
<div
|
||||
key={sample.id}
|
||||
className={cn(
|
||||
'group relative rounded-lg border bg-card transition-all duration-200',
|
||||
isEditing ? 'ring-2 ring-primary/20' : 'hover:border-primary/30'
|
||||
)}
|
||||
>
|
||||
{isEditing ? (
|
||||
/* Edit Mode */
|
||||
<div className="p-4 space-y-3">
|
||||
<div className="flex items-center gap-2 text-xs text-muted-foreground mb-2">
|
||||
<Edit className="h-3 w-3" />
|
||||
<span>Editing transcription</span>
|
||||
</div>
|
||||
<Textarea
|
||||
value={editedText}
|
||||
onChange={(e) => setEditedText(e.target.value)}
|
||||
className="min-h-[100px] text-sm resize-none"
|
||||
placeholder="Enter reference text..."
|
||||
autoFocus
|
||||
/>
|
||||
<div className="flex items-center justify-end gap-2 pt-1">
|
||||
<Button
|
||||
type="button"
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={handleCancelEdit}
|
||||
disabled={updateSample.isPending}
|
||||
>
|
||||
<X className="h-4 w-4 mr-1" />
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
size="sm"
|
||||
onClick={() => handleSaveEdit(sample.id)}
|
||||
disabled={updateSample.isPending}
|
||||
>
|
||||
<Check className="h-4 w-4 mr-1" />
|
||||
{updateSample.isPending ? 'Saving...' : 'Save'}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{/* View Mode */}
|
||||
<div className="flex items-center gap-3 p-3 h-[72px]">
|
||||
{/* Text Content */}
|
||||
<div className="flex-1 min-w-0 py-0.5">
|
||||
<p className="text-sm font-medium line-clamp-2 leading-snug">
|
||||
{sample.reference_text}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Action Buttons */}
|
||||
<div className="shrink-0 flex items-center gap-0.5 opacity-0 group-hover:opacity-100 transition-opacity">
|
||||
<CircleButton
|
||||
icon={Edit}
|
||||
title="Edit transcription"
|
||||
onClick={() => handleStartEdit(sample.id, sample.reference_text)}
|
||||
/>
|
||||
<CircleButton
|
||||
icon={Trash2}
|
||||
title="Delete sample"
|
||||
onClick={() => handleDelete(sample.id)}
|
||||
disabled={deleteSample.isPending}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Sample Number Badge */}
|
||||
<div className="absolute top-1 right-2 text-[10px] text-muted-foreground/50 font-medium">
|
||||
#{index + 1}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Mini Player - Always visible */}
|
||||
<MiniSamplePlayer audioUrl={apiClient.getSampleUrl(sample.id)} />
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => handlePlay(sample.reference_text, sample.id)}
|
||||
className={currentAudioId === sample.id && isPlaying ? 'text-primary' : ''}
|
||||
>
|
||||
<Play className="h-4 w-4 mr-1" />
|
||||
Play
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => handleDelete(sample.id)}
|
||||
disabled={deleteSample.isPending}
|
||||
>
|
||||
<Trash2 className="h-4 w-4 text-destructive" />
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<Button type="button" variant="outline" className="w-full" onClick={() => setUploadOpen(true)}>
|
||||
<Plus className="mr-2 h-4 w-4" />
|
||||
Add Sample
|
||||
</Button>
|
||||
|
||||
<SampleUpload profileId={profileId} open={uploadOpen} onOpenChange={setUploadOpen} />
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import type { LanguageCode } from '@/lib/constants/languages';
|
||||
import type {
|
||||
VoiceProfileCreate,
|
||||
VoiceProfileResponse,
|
||||
@@ -120,6 +121,16 @@ class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async updateProfileSample(
|
||||
sampleId: string,
|
||||
referenceText: string,
|
||||
): Promise<ProfileSampleResponse> {
|
||||
return this.request<ProfileSampleResponse>(`/profiles/samples/${sampleId}`, {
|
||||
method: 'PUT',
|
||||
body: JSON.stringify({ reference_text: referenceText }),
|
||||
});
|
||||
}
|
||||
|
||||
async exportProfile(profileId: string): Promise<Blob> {
|
||||
const url = `${this.getBaseUrl()}/profiles/${profileId}/export`;
|
||||
const response = await fetch(url);
|
||||
@@ -154,6 +165,32 @@ class ApiClient {
|
||||
return response.json();
|
||||
}
|
||||
|
||||
async uploadAvatar(profileId: string, file: File): Promise<VoiceProfileResponse> {
|
||||
const url = `${this.getBaseUrl()}/profiles/${profileId}/avatar`;
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
body: formData,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
}
|
||||
|
||||
async deleteAvatar(profileId: string): Promise<void> {
|
||||
await this.request<void>(`/profiles/${profileId}/avatar`, {
|
||||
method: 'DELETE',
|
||||
});
|
||||
}
|
||||
|
||||
// Generation
|
||||
async generateSpeech(data: GenerationRequest): Promise<GenerationResponse> {
|
||||
return this.request<GenerationResponse>('/generate', {
|
||||
@@ -244,7 +281,7 @@ class ApiClient {
|
||||
}
|
||||
|
||||
// Transcription
|
||||
async transcribeAudio(file: File, language?: 'en' | 'zh'): Promise<TranscriptionResponse> {
|
||||
async transcribeAudio(file: File, language?: LanguageCode): Promise<TranscriptionResponse> {
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
if (language) {
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
// API Types matching backend Pydantic models
|
||||
import type { LanguageCode } from '@/lib/constants/languages';
|
||||
|
||||
export interface VoiceProfileCreate {
|
||||
name: string;
|
||||
description?: string;
|
||||
language: 'en' | 'zh';
|
||||
language: LanguageCode;
|
||||
}
|
||||
|
||||
export interface VoiceProfileResponse {
|
||||
@@ -11,6 +12,7 @@ export interface VoiceProfileResponse {
|
||||
name: string;
|
||||
description?: string;
|
||||
language: string;
|
||||
avatar_path?: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
@@ -29,7 +31,7 @@ export interface ProfileSampleResponse {
|
||||
export interface GenerationRequest {
|
||||
profile_id: string;
|
||||
text: string;
|
||||
language: 'en' | 'zh';
|
||||
language: LanguageCode;
|
||||
seed?: number;
|
||||
model_size?: '1.7B' | '0.6B';
|
||||
}
|
||||
@@ -62,7 +64,7 @@ export interface HistoryListResponse {
|
||||
}
|
||||
|
||||
export interface TranscriptionRequest {
|
||||
language?: 'en' | 'zh';
|
||||
language?: LanguageCode;
|
||||
}
|
||||
|
||||
export interface TranscriptionResponse {
|
||||
|
||||
@@ -140,8 +140,8 @@ export function useModelDownloadToast({
|
||||
}
|
||||
};
|
||||
|
||||
eventSource.onerror = () => {
|
||||
console.error('SSE error');
|
||||
eventSource.onerror = (error) => {
|
||||
console.error('SSE error:', error);
|
||||
eventSource.close();
|
||||
eventSourceRef.current = null;
|
||||
|
||||
|
||||
@@ -98,6 +98,24 @@ export function useDeleteSample() {
|
||||
});
|
||||
}
|
||||
|
||||
export function useUpdateSample() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
return useMutation({
|
||||
mutationFn: ({ sampleId, referenceText }: { sampleId: string; referenceText: string }) =>
|
||||
apiClient.updateProfileSample(sampleId, referenceText),
|
||||
onSuccess: (data) => {
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ['profiles', data.profile_id, 'samples'],
|
||||
});
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ['profiles', data.profile_id],
|
||||
});
|
||||
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useExportProfile() {
|
||||
return useMutation({
|
||||
mutationFn: async (profileId: string) => {
|
||||
@@ -167,3 +185,32 @@ export function useImportProfile() {
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useUploadAvatar() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
return useMutation({
|
||||
mutationFn: ({ profileId, file }: { profileId: string; file: File }) =>
|
||||
apiClient.uploadAvatar(profileId, file),
|
||||
onSuccess: (_, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ['profiles', variables.profileId],
|
||||
});
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useDeleteAvatar() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
return useMutation({
|
||||
mutationFn: (profileId: string) => apiClient.deleteAvatar(profileId),
|
||||
onSuccess: (_, profileId) => {
|
||||
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ['profiles', profileId],
|
||||
});
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
import { useMutation } from '@tanstack/react-query';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { LanguageCode } from '@/lib/constants/languages';
|
||||
|
||||
export function useTranscription() {
|
||||
return useMutation({
|
||||
mutationFn: ({ file, language }: { file: File; language?: 'en' | 'zh' }) =>
|
||||
mutationFn: ({ file, language }: { file: File; language?: LanguageCode }) =>
|
||||
apiClient.transcribeAudio(file, language),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -89,6 +89,9 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
|
||||
playbackItems: items,
|
||||
totalDurationMs: maxEndTimeMs,
|
||||
currentTimeMs: startTimeMs,
|
||||
// Reset timing anchors - will be set fresh by the playback hook
|
||||
playbackStartContextTime: null,
|
||||
playbackStartStoryTime: null,
|
||||
});
|
||||
},
|
||||
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
import { create } from 'zustand';
|
||||
|
||||
// Draft state for the create voice profile form
|
||||
export interface ProfileFormDraft {
|
||||
name: string;
|
||||
description: string;
|
||||
language: string;
|
||||
referenceText: string;
|
||||
sampleMode: 'upload' | 'record' | 'system';
|
||||
// Note: File objects can't be persisted, so we store metadata
|
||||
sampleFileName?: string;
|
||||
sampleFileType?: string;
|
||||
sampleFileData?: string; // Base64 encoded
|
||||
}
|
||||
|
||||
interface UIStore {
|
||||
// Sidebar
|
||||
sidebarOpen: boolean;
|
||||
@@ -18,6 +31,10 @@ interface UIStore {
|
||||
selectedProfileId: string | null;
|
||||
setSelectedProfileId: (id: string | null) => void;
|
||||
|
||||
// Profile form draft (for persisting create voice modal state)
|
||||
profileFormDraft: ProfileFormDraft | null;
|
||||
setProfileFormDraft: (draft: ProfileFormDraft | null) => void;
|
||||
|
||||
// Theme
|
||||
theme: 'light' | 'dark';
|
||||
setTheme: (theme: 'light' | 'dark') => void;
|
||||
@@ -38,6 +55,9 @@ export const useUIStore = create<UIStore>((set) => ({
|
||||
selectedProfileId: null,
|
||||
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
|
||||
|
||||
profileFormDraft: null,
|
||||
setProfileFormDraft: (draft) => set({ profileFormDraft: draft }),
|
||||
|
||||
theme: 'light',
|
||||
setTheme: (theme) => {
|
||||
set({ theme });
|
||||
|
||||
@@ -19,8 +19,13 @@ Production-quality FastAPI backend for Qwen3-TTS voice cloning.
|
||||
backend/
|
||||
├── main.py # FastAPI app with all routes
|
||||
├── models.py # Pydantic request/response models
|
||||
├── tts.py # Qwen3-TTS inference
|
||||
├── transcribe.py # Whisper ASR
|
||||
├── platform_detect.py # Platform detection for backend selection
|
||||
├── tts.py # TTS backend abstraction (delegates to MLX or PyTorch)
|
||||
├── transcribe.py # STT backend abstraction (delegates to MLX or PyTorch)
|
||||
├── backends/ # Backend implementations
|
||||
│ ├── __init__.py # Backend factory and protocols
|
||||
│ ├── mlx_backend.py # MLX backend (Apple Silicon)
|
||||
│ └── pytorch_backend.py # PyTorch backend (Windows/Linux/Intel)
|
||||
├── profiles.py # Voice profile CRUD
|
||||
├── history.py # Generation history
|
||||
├── studio.py # Audio editing (TODO)
|
||||
@@ -31,6 +36,15 @@ backend/
|
||||
└── validation.py # Input validation
|
||||
```
|
||||
|
||||
### Backend Selection
|
||||
|
||||
Voicebox automatically selects the best backend based on platform:
|
||||
|
||||
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration (4-5x faster)
|
||||
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU if available, CPU fallback)
|
||||
|
||||
The backend is detected at runtime via `platform_detect.py`. Both backends implement the same interface, so the API remains consistent across platforms.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Health & Info
|
||||
@@ -47,12 +61,20 @@ Health check with model status.
|
||||
"status": "healthy",
|
||||
"model_loaded": true,
|
||||
"gpu_available": true,
|
||||
"vram_used_mb": 1024.5
|
||||
"gpu_type": "Metal (Apple Silicon via MLX)",
|
||||
"backend_type": "mlx",
|
||||
"vram_used_mb": null
|
||||
}
|
||||
```
|
||||
|
||||
**Backend Types:**
|
||||
- `"mlx"` - MLX backend (Apple Silicon with Metal acceleration)
|
||||
- `"pytorch"` - PyTorch backend (Windows/Linux/Intel Mac)
|
||||
|
||||
### Voice Profiles
|
||||
|
||||
**Note:** The database is automatically initialized when the server starts. No manual setup required.
|
||||
|
||||
#### `POST /profiles`
|
||||
Create a new voice profile.
|
||||
|
||||
@@ -266,13 +288,12 @@ data/
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Initialize Database
|
||||
|
||||
**Note:** On Apple Silicon, also install MLX dependencies for faster inference:
|
||||
```bash
|
||||
python -c "from database import init_db; init_db()"
|
||||
pip install -r requirements-mlx.txt
|
||||
```
|
||||
|
||||
### 3. Download Models (Automatic)
|
||||
### 2. Download Models (Automatic)
|
||||
|
||||
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
# Backend package
|
||||
|
||||
__version__ = "0.1.8"
|
||||
__version__ = "0.1.11"
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
"""
|
||||
Backend abstraction layer for TTS and STT.
|
||||
|
||||
Provides a unified interface for MLX and PyTorch backends.
|
||||
"""
|
||||
|
||||
from typing import Protocol, Optional, Tuple, List
|
||||
from typing_extensions import runtime_checkable
|
||||
import numpy as np
|
||||
|
||||
from ..platform_detect import get_backend_type
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class TTSBackend(Protocol):
|
||||
"""Protocol for TTS backend implementations."""
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
"""Load TTS model."""
|
||||
...
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
...
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple voice prompts.
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio_array, combined_text)
|
||||
"""
|
||||
...
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text.
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
...
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
...
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
...
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get model path for a given size.
|
||||
|
||||
Returns:
|
||||
Model path or HuggingFace Hub ID
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class STTBackend(Protocol):
|
||||
"""Protocol for STT (Speech-to-Text) backend implementations."""
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
"""Load STT model."""
|
||||
...
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
...
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
...
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
...
|
||||
|
||||
|
||||
# Global backend instances
|
||||
_tts_backend: Optional[TTSBackend] = None
|
||||
_stt_backend: Optional[STTBackend] = None
|
||||
|
||||
|
||||
def get_tts_backend() -> TTSBackend:
|
||||
"""
|
||||
Get or create TTS backend instance based on platform.
|
||||
|
||||
Returns:
|
||||
TTS backend instance (MLX or PyTorch)
|
||||
"""
|
||||
global _tts_backend
|
||||
|
||||
if _tts_backend is None:
|
||||
backend_type = get_backend_type()
|
||||
|
||||
if backend_type == "mlx":
|
||||
from .mlx_backend import MLXTTSBackend
|
||||
_tts_backend = MLXTTSBackend()
|
||||
else:
|
||||
from .pytorch_backend import PyTorchTTSBackend
|
||||
_tts_backend = PyTorchTTSBackend()
|
||||
|
||||
return _tts_backend
|
||||
|
||||
|
||||
def get_stt_backend() -> STTBackend:
|
||||
"""
|
||||
Get or create STT backend instance based on platform.
|
||||
|
||||
Returns:
|
||||
STT backend instance (MLX or PyTorch)
|
||||
"""
|
||||
global _stt_backend
|
||||
|
||||
if _stt_backend is None:
|
||||
backend_type = get_backend_type()
|
||||
|
||||
if backend_type == "mlx":
|
||||
from .mlx_backend import MLXSTTBackend
|
||||
_stt_backend = MLXSTTBackend()
|
||||
else:
|
||||
from .pytorch_backend import PyTorchSTTBackend
|
||||
_stt_backend = PyTorchSTTBackend()
|
||||
|
||||
return _stt_backend
|
||||
|
||||
|
||||
def reset_backends():
|
||||
"""Reset backend instances (useful for testing)."""
|
||||
global _tts_backend, _stt_backend
|
||||
_tts_backend = None
|
||||
_stt_backend = None
|
||||
@@ -0,0 +1,458 @@
|
||||
"""
|
||||
MLX backend implementation for TTS and STT using mlx-audio.
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
from . import TTSBackend, STTBackend
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
|
||||
class MLXTTSBackend:
|
||||
"""MLX-based TTS backend using mlx-audio."""
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self._current_model_size = None
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get the MLX model path.
|
||||
|
||||
Args:
|
||||
model_size: Model size (1.7B or 0.6B)
|
||||
|
||||
Returns:
|
||||
HuggingFace Hub model ID for MLX
|
||||
"""
|
||||
# MLX model mapping
|
||||
mlx_model_map = {
|
||||
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
|
||||
# 0.6B not yet converted to MLX format
|
||||
"0.6B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16", # Fallback to 1.7B
|
||||
}
|
||||
|
||||
if model_size not in mlx_model_map:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
|
||||
hf_model_id = mlx_model_map[model_size]
|
||||
print(f"Will download MLX model from HuggingFace Hub: {hf_model_id}")
|
||||
|
||||
return hf_model_id
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the MLX TTS model.
|
||||
|
||||
Args:
|
||||
model_size: Model size to load (1.7B or 0.6B)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
# If already loaded with correct size, return
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
# Unload existing model if different size requested
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
from mlx_audio.tts import load
|
||||
|
||||
# Get model path
|
||||
model_path = self._get_model_path(model_size)
|
||||
|
||||
# Set up progress tracking
|
||||
progress_manager = get_progress_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(model_name)
|
||||
|
||||
print(f"Loading MLX TTS model {model_size}...")
|
||||
|
||||
# Initialize progress state
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=1,
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Set up progress callback
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Use progress tracker during download
|
||||
with tracker.patch_download():
|
||||
# Load MLX model (downloads automatically)
|
||||
self.model = load(model_path)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
print(f"MLX TTS model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading MLX TTS model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
print("MLX TTS model unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
MLX backend stores voice prompt as a dict with audio path and text.
|
||||
The actual voice prompt processing happens during generation.
|
||||
|
||||
Args:
|
||||
audio_path: Path to reference audio file
|
||||
reference_text: Transcript of reference audio
|
||||
use_cache: Whether to use cached prompt if available
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
# Check cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cached_prompt = get_cached_voice_prompt(cache_key)
|
||||
if cached_prompt is not None:
|
||||
# Return cached prompt (should be dict format)
|
||||
if isinstance(cached_prompt, dict):
|
||||
return cached_prompt, True
|
||||
|
||||
# MLX voice prompt format - store audio path and text
|
||||
# The model will process this during generation
|
||||
voice_prompt_items = {
|
||||
"ref_audio": str(audio_path),
|
||||
"ref_text": reference_text,
|
||||
}
|
||||
|
||||
# Cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||
|
||||
return voice_prompt_items, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples for better quality.
|
||||
|
||||
Args:
|
||||
audio_paths: List of audio file paths
|
||||
reference_texts: List of reference texts
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio, combined_text)
|
||||
"""
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate audio
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text using voice prompt.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Voice prompt dictionary with ref_audio and ref_text
|
||||
language: Language code (en or zh) - may not be fully supported by MLX
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Natural language instruction (may not be supported by MLX)
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
print(f"Generating audio for text: {text}")
|
||||
|
||||
def _generate_sync():
|
||||
"""Run synchronous generation in thread pool."""
|
||||
# MLX generate() returns a generator yielding GenerationResult objects
|
||||
audio_chunks = []
|
||||
sample_rate = 24000
|
||||
|
||||
# Set seed if provided (MLX uses numpy random)
|
||||
if seed is not None:
|
||||
import mlx.core as mx
|
||||
np.random.seed(seed)
|
||||
mx.random.seed(seed)
|
||||
|
||||
# Extract voice prompt info
|
||||
ref_audio = voice_prompt.get("ref_audio") or voice_prompt.get("ref_audio_path")
|
||||
ref_text = voice_prompt.get("ref_text", "")
|
||||
|
||||
# Check if model supports voice cloning via generate method
|
||||
# MLX API may support ref_audio parameter directly
|
||||
try:
|
||||
# Try with voice cloning parameters if supported
|
||||
if ref_audio:
|
||||
# Check if generate accepts ref_audio parameter
|
||||
import inspect
|
||||
sig = inspect.signature(self.model.generate)
|
||||
if "ref_audio" in sig.parameters:
|
||||
# Generate with voice cloning
|
||||
for result in self.model.generate(text, ref_audio=ref_audio, ref_text=ref_text):
|
||||
audio_chunks.append(np.array(result.audio))
|
||||
sample_rate = result.sample_rate
|
||||
else:
|
||||
# Fallback: generate without voice cloning
|
||||
for result in self.model.generate(text):
|
||||
audio_chunks.append(np.array(result.audio))
|
||||
sample_rate = result.sample_rate
|
||||
else:
|
||||
# No voice prompt, generate normally
|
||||
for result in self.model.generate(text):
|
||||
audio_chunks.append(np.array(result.audio))
|
||||
sample_rate = result.sample_rate
|
||||
except Exception as e:
|
||||
# If voice cloning fails, try without it
|
||||
print(f"Warning: Voice cloning failed, generating without voice prompt: {e}")
|
||||
for result in self.model.generate(text):
|
||||
audio_chunks.append(np.array(result.audio))
|
||||
sample_rate = result.sample_rate
|
||||
|
||||
# Concatenate all chunks
|
||||
if audio_chunks:
|
||||
audio = np.concatenate([np.asarray(chunk, dtype=np.float32) for chunk in audio_chunks])
|
||||
else:
|
||||
# Fallback: empty audio
|
||||
audio = np.array([], dtype=np.float32)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
# Run blocking inference in thread pool
|
||||
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
class MLXSTTBackend:
|
||||
"""MLX-based STT backend using mlx-audio Whisper."""
|
||||
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the MLX Whisper model.
|
||||
|
||||
Args:
|
||||
model_size: Model size (tiny, base, small, medium, large)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
# IMPORTANT: Set up progress tracking BEFORE importing mlx_audio
|
||||
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||
progress_manager = get_progress_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
|
||||
# Set up progress callback and tracker
|
||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Patch tqdm BEFORE importing mlx_audio
|
||||
# This is critical because mlx_audio imports huggingface_hub which imports tqdm
|
||||
print("[DEBUG] Starting tqdm patch BEFORE mlx_audio import")
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
print("[DEBUG] tqdm patched, now importing mlx_audio")
|
||||
|
||||
# NOW import mlx_audio - it will use our patched tqdm
|
||||
from mlx_audio.stt import load
|
||||
|
||||
# MLX Whisper uses the standard OpenAI models
|
||||
model_name = f"openai/whisper-{model_size}"
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(progress_model_name)
|
||||
|
||||
print(f"Loading MLX Whisper model {model_size}...")
|
||||
|
||||
# Initialize progress state
|
||||
progress_manager.update_progress(
|
||||
model_name=progress_model_name,
|
||||
current=0,
|
||||
total=1,
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Load the model (tqdm is already patched from above)
|
||||
try:
|
||||
self.model = load(model_name)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
self.model_size = model_size
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(progress_model_name)
|
||||
task_manager.complete_download(progress_model_name)
|
||||
|
||||
print(f"MLX Whisper model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading MLX Whisper model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
print("MLX Whisper model unloaded")
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint (en or zh)
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
def _transcribe_sync():
|
||||
"""Run synchronous transcription in thread pool."""
|
||||
# MLX Whisper transcription using generate method
|
||||
# The generate method accepts audio path directly
|
||||
decode_options = {}
|
||||
if language:
|
||||
decode_options["language"] = language
|
||||
|
||||
result = self.model.generate(str(audio_path), **decode_options)
|
||||
|
||||
# Extract text from result
|
||||
if isinstance(result, str):
|
||||
return result.strip()
|
||||
elif isinstance(result, dict):
|
||||
return result.get("text", "").strip()
|
||||
elif hasattr(result, "text"):
|
||||
return result.text.strip()
|
||||
else:
|
||||
return str(result).strip()
|
||||
|
||||
# Run blocking transcription in thread pool
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
@@ -0,0 +1,484 @@
|
||||
"""
|
||||
PyTorch backend implementation for TTS and STT.
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import torch
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
from . import TTSBackend, STTBackend
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
|
||||
class PyTorchTTSBackend:
|
||||
"""PyTorch-based TTS backend using Qwen3-TTS."""
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
self._current_model_size = None
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
# MPS can have issues, use CPU for stability
|
||||
return "cpu"
|
||||
return "cpu"
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get the HuggingFace Hub model ID.
|
||||
|
||||
Args:
|
||||
model_size: Model size (1.7B or 0.6B)
|
||||
|
||||
Returns:
|
||||
HuggingFace Hub model ID
|
||||
"""
|
||||
hf_model_map = {
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
if model_size not in hf_model_map:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
|
||||
return hf_model_map[model_size]
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
|
||||
|
||||
Args:
|
||||
model_size: Model size to load (1.7B or 0.6B)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
# If already loaded with correct size, return
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
# Unload existing model if different size requested
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
# IMPORTANT: Set up progress tracking BEFORE importing qwen_tts
|
||||
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||
progress_manager = get_progress_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
|
||||
# Set up progress callback and tracker
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Patch tqdm BEFORE importing qwen_tts
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
# NOW import qwen_tts - it will use our patched tqdm
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
|
||||
# Get model path (local or HuggingFace Hub ID)
|
||||
model_path = self._get_model_path(model_size)
|
||||
|
||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(model_name)
|
||||
|
||||
# Initialize progress state to show download has started
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=1, # Set to 1 initially, will be updated by callback
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Load the model (tqdm is already patched from above)
|
||||
try:
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
||||
)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"TTS model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading TTS model: {e}")
|
||||
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("TTS model unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
Args:
|
||||
audio_path: Path to reference audio file
|
||||
reference_text: Transcript of reference audio
|
||||
use_cache: Whether to use cached prompt if available
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
# Check cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cached_prompt = get_cached_voice_prompt(cache_key)
|
||||
if cached_prompt is not None:
|
||||
# Cache stores as torch.Tensor but actual prompt is dict
|
||||
# Convert if needed
|
||||
if isinstance(cached_prompt, dict):
|
||||
return cached_prompt, True
|
||||
elif isinstance(cached_prompt, torch.Tensor):
|
||||
# Legacy cache format - convert to dict
|
||||
# This shouldn't happen in practice, but handle it
|
||||
return {"prompt": cached_prompt}, True
|
||||
|
||||
def _create_prompt_sync():
|
||||
"""Run synchronous voice prompt creation in thread pool."""
|
||||
return self.model.create_voice_clone_prompt(
|
||||
ref_audio=str(audio_path),
|
||||
ref_text=reference_text,
|
||||
x_vector_only_mode=False,
|
||||
)
|
||||
|
||||
# Run blocking operation in thread pool
|
||||
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
|
||||
|
||||
# Cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||
|
||||
return voice_prompt_items, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples for better quality.
|
||||
|
||||
Args:
|
||||
audio_paths: List of audio file paths
|
||||
reference_texts: List of reference texts
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio, combined_text)
|
||||
"""
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate audio
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text using voice prompt.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Voice prompt dictionary from create_voice_prompt
|
||||
language: Language code (en or zh)
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Natural language instruction for speech delivery control
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
# Load model
|
||||
await self.load_model_async(None)
|
||||
|
||||
def _generate_sync():
|
||||
"""Run synchronous generation in thread pool."""
|
||||
# Set seed if provided
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
|
||||
# Generate audio - this is the blocking operation
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
instruct=instruct,
|
||||
)
|
||||
return wavs[0], sample_rate
|
||||
|
||||
# Run blocking inference in thread pool to avoid blocking event loop
|
||||
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
class PyTorchSTTBackend:
|
||||
"""PyTorch-based STT backend using Whisper."""
|
||||
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
# MPS support for Whisper
|
||||
return "cpu" # Use CPU for stability
|
||||
return "cpu"
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the Whisper model.
|
||||
|
||||
Args:
|
||||
model_size: Model size (tiny, base, small, medium, large)
|
||||
"""
|
||||
print(f"[DEBUG] load_model_async called with size: {model_size}")
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
print(f"[DEBUG] Model already loaded? {self.model is not None}, current size: {self.model_size}, requested: {model_size}")
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
print(f"[DEBUG] Early return - model already loaded")
|
||||
return
|
||||
|
||||
print(f"[DEBUG] Calling asyncio.to_thread for _load_model_sync")
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
print(f"[DEBUG] asyncio.to_thread completed")
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
print(f"[DEBUG] _load_model_sync called for Whisper {model_size}")
|
||||
try:
|
||||
# IMPORTANT: Set up progress tracking BEFORE importing transformers
|
||||
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||
progress_manager = get_progress_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
|
||||
# Set up progress callback and tracker
|
||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Patch tqdm BEFORE importing transformers
|
||||
print("[DEBUG] Starting tqdm patch BEFORE transformers import")
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
print("[DEBUG] tqdm patched, now importing transformers")
|
||||
|
||||
# NOW import transformers - it will use our patched tqdm
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
|
||||
model_name = f"openai/whisper-{model_size}"
|
||||
print(f"[DEBUG] Model name: {model_name}")
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(progress_model_name)
|
||||
print(f"[DEBUG] Task manager started download")
|
||||
|
||||
print(f"Loading Whisper model {model_size} on {self.device}...")
|
||||
|
||||
# Initialize progress state to show download has started
|
||||
print(f"[DEBUG] Calling update_progress...")
|
||||
progress_manager.update_progress(
|
||||
model_name=progress_model_name,
|
||||
current=0,
|
||||
total=1, # Set to 1 initially, will be updated by callback
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
print(f"[DEBUG] update_progress called, listeners: {len(progress_manager._listeners.get(progress_model_name, []))}")
|
||||
|
||||
# Load models (tqdm is already patched from above)
|
||||
try:
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
self.model.to(self.device)
|
||||
self.model_size = model_size
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(progress_model_name)
|
||||
task_manager.complete_download(progress_model_name)
|
||||
|
||||
print(f"Whisper model {model_size} loaded successfully")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error loading Whisper model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
del self.processor
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("Whisper model unloaded")
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint (en or zh)
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
def _transcribe_sync():
|
||||
"""Run synchronous transcription in thread pool."""
|
||||
# Load audio
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
sampling_rate=16000,
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
# Set language if provided
|
||||
forced_decoder_ids = None
|
||||
if language:
|
||||
# Support all languages from frontend: en, zh, ja, ko, de, fr, ru, pt, es, it
|
||||
# Whisper supports these and many more
|
||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||
language=language,
|
||||
task="transcribe",
|
||||
)
|
||||
|
||||
# Generate transcription
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
forced_decoder_ids=forced_decoder_ids,
|
||||
)
|
||||
|
||||
# Decode
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
return transcription.strip()
|
||||
|
||||
# Run blocking transcription in thread pool
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
@@ -4,16 +4,19 @@ PyInstaller build script for creating standalone Python server binary.
|
||||
|
||||
import PyInstaller.__main__
|
||||
import os
|
||||
import platform
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def is_apple_silicon():
|
||||
"""Check if running on Apple Silicon."""
|
||||
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||
|
||||
|
||||
def build_server():
|
||||
"""Build Python server as standalone binary."""
|
||||
backend_dir = Path(__file__).parent
|
||||
|
||||
# Check for local editable qwen_tts install
|
||||
local_qwen_path = Path.home() / 'Projects' / 'voice' / 'Qwen3-TTS'
|
||||
|
||||
# PyInstaller arguments
|
||||
args = [
|
||||
'server.py', # Use server.py as entry point instead of main.py
|
||||
@@ -21,12 +24,13 @@ def build_server():
|
||||
'--name', 'voicebox-server',
|
||||
]
|
||||
|
||||
# Add local qwen_tts path if it exists (for editable installs)
|
||||
if local_qwen_path.exists():
|
||||
args.extend(['--paths', str(local_qwen_path)])
|
||||
print(f"Using local qwen_tts source from: {local_qwen_path}")
|
||||
# Add local qwen_tts path if specified (for editable installs)
|
||||
qwen_tts_path = os.getenv('QWEN_TTS_PATH')
|
||||
if qwen_tts_path and Path(qwen_tts_path).exists():
|
||||
args.extend(['--paths', str(qwen_tts_path)])
|
||||
print(f"Using local qwen_tts source from: {qwen_tts_path}")
|
||||
|
||||
# Add hidden imports
|
||||
# Add common hidden imports
|
||||
args.extend([
|
||||
'--hidden-import', 'backend',
|
||||
'--hidden-import', 'backend.main',
|
||||
@@ -37,6 +41,9 @@ def build_server():
|
||||
'--hidden-import', 'backend.history',
|
||||
'--hidden-import', 'backend.tts',
|
||||
'--hidden-import', 'backend.transcribe',
|
||||
'--hidden-import', 'backend.platform_detect',
|
||||
'--hidden-import', 'backend.backends',
|
||||
'--hidden-import', 'backend.backends.pytorch_backend',
|
||||
'--hidden-import', 'backend.utils.audio',
|
||||
'--hidden-import', 'backend.utils.cache',
|
||||
'--hidden-import', 'backend.utils.progress',
|
||||
@@ -61,6 +68,29 @@ def build_server():
|
||||
# Fix for pkg_resources and jaraco namespace packages
|
||||
'--hidden-import', 'pkg_resources.extern',
|
||||
'--collect-submodules', 'jaraco',
|
||||
])
|
||||
|
||||
# Add MLX-specific imports if building on Apple Silicon
|
||||
if is_apple_silicon():
|
||||
print("Building for Apple Silicon - including MLX dependencies")
|
||||
args.extend([
|
||||
'--hidden-import', 'backend.backends.mlx_backend',
|
||||
'--hidden-import', 'mlx',
|
||||
'--hidden-import', 'mlx.core',
|
||||
'--hidden-import', 'mlx.nn',
|
||||
'--hidden-import', 'mlx_audio',
|
||||
'--hidden-import', 'mlx_audio.tts',
|
||||
'--hidden-import', 'mlx_audio.stt',
|
||||
'--collect-submodules', 'mlx',
|
||||
'--collect-submodules', 'mlx_audio',
|
||||
# Collect MLX data files including Metal shader libraries (.metallib)
|
||||
'--collect-data', 'mlx',
|
||||
'--collect-data', 'mlx_audio',
|
||||
])
|
||||
else:
|
||||
print("Building for non-Apple Silicon platform - PyTorch only")
|
||||
|
||||
args.extend([
|
||||
'--noconfirm',
|
||||
'--clean',
|
||||
])
|
||||
|
||||
@@ -17,11 +17,12 @@ Base = declarative_base()
|
||||
class VoiceProfile(Base):
|
||||
"""Voice profile database model."""
|
||||
__tablename__ = "profiles"
|
||||
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text)
|
||||
language = Column(String, default="en")
|
||||
avatar_path = Column(String, nullable=True)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
@@ -277,6 +278,16 @@ def _run_migrations(engine):
|
||||
conn.commit()
|
||||
print("Added trim_end_ms column to story_items")
|
||||
|
||||
# Migration: Add avatar_path to profiles table
|
||||
if 'profiles' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('profiles')}
|
||||
if 'avatar_path' not in columns:
|
||||
print("Migrating profiles: adding avatar_path column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE profiles ADD COLUMN avatar_path VARCHAR"))
|
||||
conn.commit()
|
||||
print("Added avatar_path column to profiles")
|
||||
|
||||
|
||||
def get_db():
|
||||
"""Get database session (generator for dependency injection)."""
|
||||
|
||||
@@ -75,6 +75,16 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
|
||||
zip_buffer = io.BytesIO()
|
||||
|
||||
with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
|
||||
# Check if profile has avatar
|
||||
has_avatar = False
|
||||
if profile.avatar_path:
|
||||
avatar_path = Path(profile.avatar_path)
|
||||
if avatar_path.exists():
|
||||
has_avatar = True
|
||||
# Add avatar to ZIP root with original extension
|
||||
avatar_ext = avatar_path.suffix
|
||||
zip_file.write(avatar_path, f"avatar{avatar_ext}")
|
||||
|
||||
# Create manifest.json
|
||||
manifest = {
|
||||
"version": "1.0",
|
||||
@@ -82,30 +92,31 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
|
||||
"name": profile.name,
|
||||
"description": profile.description,
|
||||
"language": profile.language,
|
||||
}
|
||||
},
|
||||
"has_avatar": has_avatar,
|
||||
}
|
||||
zip_file.writestr("manifest.json", json.dumps(manifest, indent=2))
|
||||
|
||||
|
||||
# Create samples.json mapping
|
||||
samples_data = {}
|
||||
profile_dir = _get_profiles_dir() / profile_id
|
||||
|
||||
|
||||
for sample in samples:
|
||||
# Get filename from audio_path (should be {sample_id}.wav)
|
||||
audio_path = Path(sample.audio_path)
|
||||
filename = audio_path.name
|
||||
|
||||
|
||||
# Read audio file
|
||||
if not audio_path.exists():
|
||||
raise ValueError(f"Audio file not found: {audio_path}")
|
||||
|
||||
|
||||
# Add to samples directory in ZIP
|
||||
zip_path = f"samples/{filename}"
|
||||
zip_file.write(audio_path, zip_path)
|
||||
|
||||
|
||||
# Map filename to reference text
|
||||
samples_data[filename] = sample.reference_text
|
||||
|
||||
|
||||
zip_file.writestr("samples.json", json.dumps(samples_data, indent=2))
|
||||
|
||||
zip_buffer.seek(0)
|
||||
@@ -168,11 +179,31 @@ async def import_profile_from_zip(file_bytes: bytes, db: Session) -> VoiceProfil
|
||||
)
|
||||
|
||||
profile = await create_profile(profile_create, db)
|
||||
|
||||
|
||||
# Extract and add samples
|
||||
profile_dir = _get_profiles_dir() / profile.id
|
||||
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
# Handle avatar if present
|
||||
avatar_files = [f for f in namelist if f.startswith("avatar.")]
|
||||
if avatar_files:
|
||||
try:
|
||||
avatar_file = avatar_files[0]
|
||||
# Extract to temporary file
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(suffix=Path(avatar_file).suffix, delete=False) as tmp:
|
||||
tmp.write(zip_file.read(avatar_file))
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
from .profiles import upload_avatar
|
||||
await upload_avatar(profile.id, tmp_path, db)
|
||||
finally:
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
except Exception as e:
|
||||
# Avatar import is optional - continue even if it fails
|
||||
pass
|
||||
|
||||
for filename, reference_text in samples_data.items():
|
||||
# Validate filename
|
||||
if not filename.endswith('.wav'):
|
||||
|
||||
@@ -27,6 +27,7 @@ from . import database, models, profiles, history, tts, transcribe, config, expo
|
||||
from .database import get_db, Generation as DBGeneration, VoiceProfile as DBVoiceProfile
|
||||
from .utils.progress import get_progress_manager
|
||||
from .utils.tasks import get_task_manager
|
||||
from .platform_detect import get_backend_type
|
||||
|
||||
app = FastAPI(
|
||||
title="voicebox API",
|
||||
@@ -73,6 +74,7 @@ async def health():
|
||||
import os
|
||||
|
||||
tts_model = tts.get_tts_model()
|
||||
backend_type = get_backend_type()
|
||||
|
||||
# Check for GPU availability (CUDA or MPS)
|
||||
has_cuda = torch.cuda.is_available()
|
||||
@@ -84,6 +86,8 @@ async def health():
|
||||
gpu_type = f"CUDA ({torch.cuda.get_device_name(0)})"
|
||||
elif has_mps:
|
||||
gpu_type = "MPS (Apple Silicon)"
|
||||
elif backend_type == "mlx":
|
||||
gpu_type = "Metal (Apple Silicon via MLX)"
|
||||
|
||||
vram_used = None
|
||||
if has_cuda:
|
||||
@@ -111,7 +115,11 @@ async def health():
|
||||
model_downloaded = None
|
||||
try:
|
||||
# Check if the default model (1.7B) is cached
|
||||
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
# Use different model IDs based on backend
|
||||
if backend_type == "mlx":
|
||||
default_model_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
|
||||
else:
|
||||
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
|
||||
# Method 1: Try scan_cache_dir if available
|
||||
try:
|
||||
@@ -124,13 +132,14 @@ async def health():
|
||||
except (ImportError, Exception):
|
||||
# Method 2: Check cache directory (using HuggingFace's OS-specific cache location)
|
||||
cache_dir = hf_constants.HF_HUB_CACHE
|
||||
repo_cache = Path(cache_dir) / "models--" + default_model_id.replace("/", "--")
|
||||
repo_cache = Path(cache_dir) / ("models--" + default_model_id.replace("/", "--"))
|
||||
if repo_cache.exists():
|
||||
has_model_files = (
|
||||
any(repo_cache.rglob("*.bin")) or
|
||||
any(repo_cache.rglob("*.safetensors")) or
|
||||
any(repo_cache.rglob("*.pt")) or
|
||||
any(repo_cache.rglob("*.pth"))
|
||||
any(repo_cache.rglob("*.pth")) or
|
||||
any(repo_cache.rglob("*.npz")) # MLX models may use npz
|
||||
)
|
||||
model_downloaded = has_model_files
|
||||
except Exception:
|
||||
@@ -144,6 +153,7 @@ async def health():
|
||||
gpu_available=gpu_available,
|
||||
gpu_type=gpu_type,
|
||||
vram_used_mb=vram_used,
|
||||
backend_type=backend_type,
|
||||
)
|
||||
|
||||
|
||||
@@ -283,6 +293,74 @@ async def delete_profile_sample(
|
||||
return {"message": "Sample deleted successfully"}
|
||||
|
||||
|
||||
@app.put("/profiles/samples/{sample_id}", response_model=models.ProfileSampleResponse)
|
||||
async def update_profile_sample(
|
||||
sample_id: str,
|
||||
data: models.ProfileSampleUpdate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Update a profile sample's reference text."""
|
||||
sample = await profiles.update_profile_sample(sample_id, data.reference_text, db)
|
||||
if not sample:
|
||||
raise HTTPException(status_code=404, detail="Sample not found")
|
||||
return sample
|
||||
|
||||
|
||||
@app.post("/profiles/{profile_id}/avatar", response_model=models.VoiceProfileResponse)
|
||||
async def upload_profile_avatar(
|
||||
profile_id: str,
|
||||
file: UploadFile = File(...),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Upload or update avatar image for a profile."""
|
||||
# Save uploaded file to temp location
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=Path(file.filename).suffix) as tmp:
|
||||
content = await file.read()
|
||||
tmp.write(content)
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
profile = await profiles.upload_avatar(profile_id, tmp_path, db)
|
||||
return profile
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
finally:
|
||||
# Clean up temp file
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
|
||||
|
||||
@app.get("/profiles/{profile_id}/avatar")
|
||||
async def get_profile_avatar(
|
||||
profile_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get avatar image for a profile."""
|
||||
profile = await profiles.get_profile(profile_id, db)
|
||||
if not profile:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
|
||||
if not profile.avatar_path:
|
||||
raise HTTPException(status_code=404, detail="No avatar found for this profile")
|
||||
|
||||
avatar_path = Path(profile.avatar_path)
|
||||
if not avatar_path.exists():
|
||||
raise HTTPException(status_code=404, detail="Avatar file not found")
|
||||
|
||||
return FileResponse(avatar_path)
|
||||
|
||||
|
||||
@app.delete("/profiles/{profile_id}/avatar")
|
||||
async def delete_profile_avatar(
|
||||
profile_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Delete avatar image for a profile."""
|
||||
success = await profiles.delete_avatar(profile_id, db)
|
||||
if not success:
|
||||
raise HTTPException(status_code=404, detail="Profile not found or no avatar to delete")
|
||||
return {"message": "Avatar deleted successfully"}
|
||||
|
||||
|
||||
@app.get("/profiles/{profile_id}/export")
|
||||
async def export_profile(
|
||||
profile_id: str,
|
||||
@@ -1081,6 +1159,8 @@ async def get_model_status():
|
||||
from pathlib import Path
|
||||
import os
|
||||
|
||||
backend_type = get_backend_type()
|
||||
|
||||
# Try to import scan_cache_dir (might not be available in older versions)
|
||||
try:
|
||||
from huggingface_hub import scan_cache_dir
|
||||
@@ -1092,7 +1172,7 @@ async def get_model_status():
|
||||
"""Check if TTS model is loaded with specific size."""
|
||||
try:
|
||||
tts_model = tts.get_tts_model()
|
||||
return tts_model.is_loaded() and tts_model.model_size == model_size
|
||||
return tts_model.is_loaded() and getattr(tts_model, 'model_size', None) == model_size
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
@@ -1100,50 +1180,66 @@ async def get_model_status():
|
||||
"""Check if Whisper model is loaded with specific size."""
|
||||
try:
|
||||
whisper_model = transcribe.get_whisper_model()
|
||||
return whisper_model.is_loaded() and whisper_model.model_size == model_size
|
||||
return whisper_model.is_loaded() and getattr(whisper_model, 'model_size', None) == model_size
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
# Use backend-specific model IDs
|
||||
if backend_type == "mlx":
|
||||
tts_1_7b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
|
||||
tts_0_6b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16" # Fallback to 1.7B
|
||||
whisper_base_id = "mlx-community/whisper-base"
|
||||
whisper_small_id = "mlx-community/whisper-small"
|
||||
whisper_medium_id = "mlx-community/whisper-medium"
|
||||
whisper_large_id = "mlx-community/whisper-large"
|
||||
else:
|
||||
tts_1_7b_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
tts_0_6b_id = "Qwen/Qwen3-TTS-12Hz-0.6B-Base"
|
||||
whisper_base_id = "openai/whisper-base"
|
||||
whisper_small_id = "openai/whisper-small"
|
||||
whisper_medium_id = "openai/whisper-medium"
|
||||
whisper_large_id = "openai/whisper-large"
|
||||
|
||||
model_configs = [
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen TTS 1.7B",
|
||||
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"hf_repo_id": tts_1_7b_id,
|
||||
"model_size": "1.7B",
|
||||
"check_loaded": lambda: check_tts_loaded("1.7B"),
|
||||
},
|
||||
{
|
||||
"model_name": "qwen-tts-0.6B",
|
||||
"display_name": "Qwen TTS 0.6B",
|
||||
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
"hf_repo_id": tts_0_6b_id,
|
||||
"model_size": "0.6B",
|
||||
"check_loaded": lambda: check_tts_loaded("0.6B"),
|
||||
},
|
||||
{
|
||||
"model_name": "whisper-base",
|
||||
"display_name": "Whisper Base",
|
||||
"hf_repo_id": "openai/whisper-base",
|
||||
"hf_repo_id": whisper_base_id,
|
||||
"model_size": "base",
|
||||
"check_loaded": lambda: check_whisper_loaded("base"),
|
||||
},
|
||||
{
|
||||
"model_name": "whisper-small",
|
||||
"display_name": "Whisper Small",
|
||||
"hf_repo_id": "openai/whisper-small",
|
||||
"hf_repo_id": whisper_small_id,
|
||||
"model_size": "small",
|
||||
"check_loaded": lambda: check_whisper_loaded("small"),
|
||||
},
|
||||
{
|
||||
"model_name": "whisper-medium",
|
||||
"display_name": "Whisper Medium",
|
||||
"hf_repo_id": "openai/whisper-medium",
|
||||
"hf_repo_id": whisper_medium_id,
|
||||
"model_size": "medium",
|
||||
"check_loaded": lambda: check_whisper_loaded("medium"),
|
||||
},
|
||||
{
|
||||
"model_name": "whisper-large",
|
||||
"display_name": "Whisper Large",
|
||||
"hf_repo_id": "openai/whisper-large",
|
||||
"hf_repo_id": whisper_large_id,
|
||||
"model_size": "large",
|
||||
"check_loaded": lambda: check_whisper_loaded("large"),
|
||||
},
|
||||
@@ -1184,15 +1280,17 @@ async def get_model_status():
|
||||
if not downloaded:
|
||||
try:
|
||||
cache_dir = hf_constants.HF_HUB_CACHE
|
||||
repo_cache = Path(cache_dir) / "models--" + config["hf_repo_id"].replace("/", "--")
|
||||
repo_cache = Path(cache_dir) / ("models--" + config["hf_repo_id"].replace("/", "--"))
|
||||
|
||||
if repo_cache.exists():
|
||||
# Check for model files (bin, safetensors, or other common model files)
|
||||
# MLX models may use .npz or .safetensors
|
||||
has_model_files = (
|
||||
any(repo_cache.rglob("*.bin")) or
|
||||
any(repo_cache.rglob("*.safetensors")) or
|
||||
any(repo_cache.rglob("*.pt")) or
|
||||
any(repo_cache.rglob("*.pth")) or
|
||||
any(repo_cache.rglob("*.npz")) or
|
||||
any(repo_cache.rglob("model.safetensors.index.json")) or
|
||||
any(repo_cache.rglob("pytorch_model.bin.index.json"))
|
||||
)
|
||||
@@ -1295,7 +1393,11 @@ async def trigger_model_download(request: models.ModelDownloadRequest):
|
||||
async def download_in_background():
|
||||
"""Download model in background without blocking the HTTP request."""
|
||||
try:
|
||||
await asyncio.to_thread(config["load_func"])
|
||||
# Call the load function (which may be async)
|
||||
result = config["load_func"]()
|
||||
# If it's a coroutine, await it
|
||||
if asyncio.iscoroutine(result):
|
||||
await result
|
||||
task_manager.complete_download(request.model_name)
|
||||
except Exception as e:
|
||||
task_manager.error_download(request.model_name, str(e))
|
||||
@@ -1465,10 +1567,13 @@ async def get_active_tasks():
|
||||
|
||||
def _get_gpu_status() -> str:
|
||||
"""Get GPU availability status."""
|
||||
backend_type = get_backend_type()
|
||||
if torch.cuda.is_available():
|
||||
return f"CUDA ({torch.cuda.get_device_name(0)})"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
return "MPS (Apple Silicon)"
|
||||
elif backend_type == "mlx":
|
||||
return "Metal (Apple Silicon via MLX)"
|
||||
return "None (CPU only)"
|
||||
|
||||
|
||||
@@ -1478,8 +1583,18 @@ async def startup_event():
|
||||
print("voicebox API starting up...")
|
||||
database.init_db()
|
||||
print(f"Database initialized at {database._db_path}")
|
||||
backend_type = get_backend_type()
|
||||
print(f"Backend: {backend_type.upper()}")
|
||||
print(f"GPU available: {_get_gpu_status()}")
|
||||
|
||||
# Initialize progress manager with main event loop for thread-safe operations
|
||||
try:
|
||||
progress_manager = get_progress_manager()
|
||||
progress_manager._set_main_loop(asyncio.get_running_loop())
|
||||
print("Progress manager initialized with event loop")
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not initialize progress manager event loop: {e}")
|
||||
|
||||
# Ensure HuggingFace cache directory exists
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
@@ -20,6 +20,7 @@ class VoiceProfileResponse(BaseModel):
|
||||
name: str
|
||||
description: Optional[str]
|
||||
language: str
|
||||
avatar_path: Optional[str] = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
@@ -32,6 +33,11 @@ class ProfileSampleCreate(BaseModel):
|
||||
reference_text: str = Field(..., min_length=1, max_length=1000)
|
||||
|
||||
|
||||
class ProfileSampleUpdate(BaseModel):
|
||||
"""Request model for updating a profile sample."""
|
||||
reference_text: str = Field(..., min_length=1, max_length=1000)
|
||||
|
||||
|
||||
class ProfileSampleResponse(BaseModel):
|
||||
"""Response model for profile sample."""
|
||||
id: str
|
||||
@@ -120,6 +126,7 @@ class HealthResponse(BaseModel):
|
||||
gpu_available: bool
|
||||
gpu_type: Optional[str] = None # GPU type (CUDA, MPS, or None)
|
||||
vram_used_mb: Optional[float] = None
|
||||
backend_type: Optional[str] = None # Backend type (mlx or pytorch)
|
||||
|
||||
|
||||
class ModelStatus(BaseModel):
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
"""
|
||||
Platform detection for backend selection.
|
||||
"""
|
||||
|
||||
import platform
|
||||
from typing import Literal
|
||||
|
||||
|
||||
def is_apple_silicon() -> bool:
|
||||
"""
|
||||
Check if running on Apple Silicon (arm64 macOS).
|
||||
|
||||
Returns:
|
||||
True if on Apple Silicon, False otherwise
|
||||
"""
|
||||
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||
|
||||
|
||||
def get_backend_type() -> Literal["mlx", "pytorch"]:
|
||||
"""
|
||||
Detect the best backend for the current platform.
|
||||
|
||||
Returns:
|
||||
"mlx" on Apple Silicon (if MLX is available), "pytorch" otherwise
|
||||
"""
|
||||
if is_apple_silicon():
|
||||
try:
|
||||
import mlx
|
||||
return "mlx"
|
||||
except ImportError:
|
||||
# MLX not installed, fallback to PyTorch
|
||||
return "pytorch"
|
||||
return "pytorch"
|
||||
@@ -21,6 +21,7 @@ from .database import (
|
||||
ProfileSample as DBProfileSample,
|
||||
)
|
||||
from .utils.audio import validate_reference_audio, load_audio, save_audio
|
||||
from .utils.images import validate_image, process_avatar
|
||||
from .tts import get_tts_model
|
||||
from . import config
|
||||
|
||||
@@ -273,6 +274,33 @@ async def delete_profile_sample(
|
||||
return True
|
||||
|
||||
|
||||
async def update_profile_sample(
|
||||
sample_id: str,
|
||||
reference_text: str,
|
||||
db: Session,
|
||||
) -> Optional[ProfileSampleResponse]:
|
||||
"""
|
||||
Update a profile sample's reference text.
|
||||
|
||||
Args:
|
||||
sample_id: Sample ID
|
||||
reference_text: Updated reference text
|
||||
db: Database session
|
||||
|
||||
Returns:
|
||||
Updated sample or None if not found
|
||||
"""
|
||||
sample = db.query(DBProfileSample).filter_by(id=sample_id).first()
|
||||
if not sample:
|
||||
return None
|
||||
|
||||
sample.reference_text = reference_text
|
||||
db.commit()
|
||||
db.refresh(sample)
|
||||
|
||||
return ProfileSampleResponse.model_validate(sample)
|
||||
|
||||
|
||||
async def create_voice_prompt_for_profile(
|
||||
profile_id: str,
|
||||
db: Session,
|
||||
@@ -280,23 +308,23 @@ async def create_voice_prompt_for_profile(
|
||||
) -> dict:
|
||||
"""
|
||||
Create a combined voice prompt from all samples in a profile.
|
||||
|
||||
|
||||
Args:
|
||||
profile_id: Profile ID
|
||||
db: Database session
|
||||
use_cache: Whether to use cached prompts
|
||||
|
||||
|
||||
Returns:
|
||||
Voice prompt dictionary
|
||||
"""
|
||||
# Get all samples for profile
|
||||
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
|
||||
|
||||
|
||||
if not samples:
|
||||
raise ValueError(f"No samples found for profile {profile_id}")
|
||||
|
||||
|
||||
tts_model = get_tts_model()
|
||||
|
||||
|
||||
if len(samples) == 1:
|
||||
# Single sample - use directly
|
||||
sample = samples[0]
|
||||
@@ -310,19 +338,19 @@ async def create_voice_prompt_for_profile(
|
||||
# Multiple samples - combine them
|
||||
audio_paths = [s.audio_path for s in samples]
|
||||
reference_texts = [s.reference_text for s in samples]
|
||||
|
||||
|
||||
# Combine audio
|
||||
combined_audio, combined_text = await tts_model.combine_voice_prompts(
|
||||
audio_paths,
|
||||
reference_texts,
|
||||
)
|
||||
|
||||
|
||||
# Save combined audio temporarily
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
|
||||
save_audio(combined_audio, tmp.name, 24000)
|
||||
tmp_path = tmp.name
|
||||
|
||||
|
||||
try:
|
||||
# Create prompt from combined audio
|
||||
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||
@@ -334,3 +362,99 @@ async def create_voice_prompt_for_profile(
|
||||
finally:
|
||||
# Clean up temp file
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
|
||||
|
||||
async def upload_avatar(
|
||||
profile_id: str,
|
||||
image_path: str,
|
||||
db: Session,
|
||||
) -> VoiceProfileResponse:
|
||||
"""
|
||||
Upload and process avatar image for a profile.
|
||||
|
||||
Args:
|
||||
profile_id: Profile ID
|
||||
image_path: Path to uploaded image file
|
||||
db: Database session
|
||||
|
||||
Returns:
|
||||
Updated profile
|
||||
"""
|
||||
# Validate profile exists
|
||||
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||
if not profile:
|
||||
raise ValueError(f"Profile {profile_id} not found")
|
||||
|
||||
# Validate image
|
||||
is_valid, error_msg = validate_image(image_path)
|
||||
if not is_valid:
|
||||
raise ValueError(error_msg)
|
||||
|
||||
# Delete existing avatar if present
|
||||
if profile.avatar_path:
|
||||
old_avatar = Path(profile.avatar_path)
|
||||
if old_avatar.exists():
|
||||
old_avatar.unlink()
|
||||
|
||||
# Determine file extension from uploaded file
|
||||
from PIL import Image
|
||||
with Image.open(image_path) as img:
|
||||
# Normalize JPEG variants (MPO is multi-picture format from some cameras)
|
||||
img_format = img.format
|
||||
if img_format in ('MPO', 'JPG'):
|
||||
img_format = 'JPEG'
|
||||
|
||||
ext_map = {
|
||||
'PNG': '.png',
|
||||
'JPEG': '.jpg',
|
||||
'WEBP': '.webp'
|
||||
}
|
||||
ext = ext_map.get(img_format, '.png')
|
||||
|
||||
# Save processed image to profile directory
|
||||
profile_dir = _get_profiles_dir() / profile_id
|
||||
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = profile_dir / f"avatar{ext}"
|
||||
|
||||
process_avatar(image_path, str(output_path))
|
||||
|
||||
# Update database
|
||||
profile.avatar_path = str(output_path)
|
||||
profile.updated_at = datetime.utcnow()
|
||||
|
||||
db.commit()
|
||||
db.refresh(profile)
|
||||
|
||||
return VoiceProfileResponse.model_validate(profile)
|
||||
|
||||
|
||||
async def delete_avatar(
|
||||
profile_id: str,
|
||||
db: Session,
|
||||
) -> bool:
|
||||
"""
|
||||
Delete avatar image for a profile.
|
||||
|
||||
Args:
|
||||
profile_id: Profile ID
|
||||
db: Database session
|
||||
|
||||
Returns:
|
||||
True if deleted, False if not found or no avatar
|
||||
"""
|
||||
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||
if not profile or not profile.avatar_path:
|
||||
return False
|
||||
|
||||
# Delete avatar file
|
||||
avatar_path = Path(profile.avatar_path)
|
||||
if avatar_path.exists():
|
||||
avatar_path.unlink()
|
||||
|
||||
# Update database
|
||||
profile.avatar_path = None
|
||||
profile.updated_at = datetime.utcnow()
|
||||
|
||||
db.commit()
|
||||
|
||||
return True
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# MLX-specific dependencies (Apple Silicon only)
|
||||
# These should only be installed on aarch64-apple-darwin platforms
|
||||
|
||||
mlx>=0.30.0
|
||||
mlx-audio>=0.3.1
|
||||
@@ -21,3 +21,4 @@ numpy>=1.24.0
|
||||
|
||||
# Utilities
|
||||
python-multipart>=0.0.6
|
||||
Pillow>=10.0.0
|
||||
|
||||
@@ -1,274 +1,22 @@
|
||||
"""
|
||||
Whisper ASR module for transcription.
|
||||
STT (Speech-to-Text) module - delegates to backend abstraction layer.
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Dict
|
||||
import asyncio
|
||||
import torch
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
from .utils.progress import get_progress_manager
|
||||
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from .utils.tasks import get_task_manager
|
||||
from typing import Optional
|
||||
from .backends import get_stt_backend, STTBackend
|
||||
|
||||
|
||||
class WhisperModel:
|
||||
"""Manages Whisper model loading and transcription."""
|
||||
def get_whisper_model() -> STTBackend:
|
||||
"""
|
||||
Get STT backend instance (MLX or PyTorch based on platform).
|
||||
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
# MPS support for Whisper
|
||||
return "cpu" # Use CPU for stability
|
||||
return "cpu"
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the Whisper model.
|
||||
|
||||
Args:
|
||||
model_size: Model size (tiny, base, small, medium, large)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
|
||||
try:
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
|
||||
model_name = f"openai/whisper-{model_size}"
|
||||
|
||||
# Set up progress tracking
|
||||
progress_manager = get_progress_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(progress_model_name)
|
||||
|
||||
print(f"Loading Whisper model {model_size} on {self.device}...")
|
||||
|
||||
# Initialize progress state to show download has started
|
||||
progress_manager.update_progress(
|
||||
model_name=progress_model_name,
|
||||
current=0,
|
||||
total=1, # Set to 1 initially, will be updated by callback
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Set up progress callback
|
||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Use progress tracker during download
|
||||
with tracker.patch_download():
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
|
||||
self.model.to(self.device)
|
||||
self.model_size = model_size
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(progress_model_name)
|
||||
task_manager.complete_download(progress_model_name)
|
||||
|
||||
print(f"Whisper model {model_size} loaded successfully")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error loading Whisper model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Async version of load_model that runs in thread pool.
|
||||
|
||||
This prevents blocking the event loop during model loading.
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
# If already loaded with correct size, return immediately
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
|
||||
# Run the blocking load operation in a thread pool
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
del self.processor
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("Whisper model unloaded")
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint (en or zh)
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
await self.load_model_async()
|
||||
|
||||
from .utils.audio import load_audio
|
||||
|
||||
def _transcribe_sync():
|
||||
"""Run synchronous transcription in thread pool."""
|
||||
# Load audio
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
sampling_rate=16000,
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
# Set language if provided
|
||||
forced_decoder_ids = None
|
||||
if language:
|
||||
lang_code = "en" if language == "en" else "zh"
|
||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||
language=lang_code,
|
||||
task="transcribe",
|
||||
)
|
||||
|
||||
# Generate transcription
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
forced_decoder_ids=forced_decoder_ids,
|
||||
)
|
||||
|
||||
# Decode
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
return transcription.strip()
|
||||
|
||||
# Run blocking transcription in thread pool
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
|
||||
async def transcribe_with_timestamps(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> List[Dict[str, any]]:
|
||||
"""
|
||||
Transcribe audio with word-level timestamps.
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint
|
||||
|
||||
Returns:
|
||||
List of word segments with timestamps
|
||||
"""
|
||||
await self.load_model_async()
|
||||
|
||||
from .utils.audio import load_audio
|
||||
|
||||
def _transcribe_timestamps_sync():
|
||||
"""Run synchronous transcription with timestamps in thread pool."""
|
||||
# Load audio
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
sampling_rate=16000,
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
# Set language if provided
|
||||
forced_decoder_ids = None
|
||||
if language:
|
||||
lang_code = "en" if language == "en" else "zh"
|
||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||
language=lang_code,
|
||||
task="transcribe",
|
||||
)
|
||||
|
||||
# Generate with timestamps
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
forced_decoder_ids=forced_decoder_ids,
|
||||
return_timestamps=True,
|
||||
)
|
||||
|
||||
# Parse timestamps (simplified - would need more robust parsing)
|
||||
# For now, return basic transcription
|
||||
# TODO: Implement proper timestamp parsing
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
return [
|
||||
{
|
||||
"text": transcription,
|
||||
"start": 0.0,
|
||||
"end": len(audio) / sr,
|
||||
}
|
||||
]
|
||||
|
||||
# Run blocking transcription in thread pool
|
||||
return await asyncio.to_thread(_transcribe_timestamps_sync)
|
||||
|
||||
|
||||
# Global model instance
|
||||
_whisper_model: Optional[WhisperModel] = None
|
||||
|
||||
|
||||
def get_whisper_model() -> WhisperModel:
|
||||
"""Get or create Whisper model instance."""
|
||||
global _whisper_model
|
||||
if _whisper_model is None:
|
||||
_whisper_model = WhisperModel()
|
||||
return _whisper_model
|
||||
Returns:
|
||||
STT backend instance
|
||||
"""
|
||||
return get_stt_backend()
|
||||
|
||||
|
||||
def unload_whisper_model():
|
||||
"""Unload Whisper model to free memory."""
|
||||
global _whisper_model
|
||||
if _whisper_model is not None:
|
||||
_whisper_model.unload_model()
|
||||
backend = get_stt_backend()
|
||||
backend.unload_model()
|
||||
|
||||
@@ -1,372 +1,37 @@
|
||||
"""
|
||||
TTS inference module using Qwen3-TTS.
|
||||
TTS inference module - delegates to backend abstraction layer.
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import torch
|
||||
from typing import Optional
|
||||
import numpy as np
|
||||
import io
|
||||
import soundfile as sf
|
||||
from pathlib import Path
|
||||
|
||||
from .utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from .utils.audio import normalize_audio
|
||||
from .utils.progress import get_progress_manager
|
||||
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from .utils.tasks import get_task_manager
|
||||
from . import config
|
||||
from .backends import get_tts_backend, TTSBackend
|
||||
|
||||
|
||||
class TTSModel:
|
||||
"""Manages Qwen3-TTS model loading and inference."""
|
||||
def get_tts_model() -> TTSBackend:
|
||||
"""
|
||||
Get TTS backend instance (MLX or PyTorch based on platform).
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
self._current_model_size = None
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
# MPS can have issues, use CPU for stability
|
||||
return "cpu"
|
||||
return "cpu"
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get the model path, downloading from HuggingFace Hub if needed.
|
||||
|
||||
Args:
|
||||
model_size: Model size (1.7B or 0.6B)
|
||||
|
||||
Returns:
|
||||
Path to model (either local or HuggingFace Hub ID)
|
||||
"""
|
||||
# HuggingFace Hub model IDs
|
||||
hf_model_map = {
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
# Local directory names (for backwards compatibility)
|
||||
local_model_map = {
|
||||
"1.7B": "Qwen--Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen--Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
if model_size not in hf_model_map:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
|
||||
# Check if model exists locally (backwards compatibility)
|
||||
local_path = config.get_models_dir() / local_model_map[model_size]
|
||||
if local_path.exists():
|
||||
print(f"Found local model at {local_path}")
|
||||
return str(local_path)
|
||||
|
||||
# Use HuggingFace Hub model ID (will auto-download)
|
||||
hf_model_id = hf_model_map[model_size]
|
||||
print(f"Will download model from HuggingFace Hub: {hf_model_id}")
|
||||
|
||||
return hf_model_id
|
||||
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
|
||||
|
||||
The model will be automatically downloaded on first use and cached locally.
|
||||
This works similar to how Whisper models are loaded.
|
||||
|
||||
Args:
|
||||
model_size: Model size to load (1.7B or 0.6B)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
# If already loaded with correct size, return
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
# Unload existing model if different size requested
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
try:
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
|
||||
# Get model path (local or HuggingFace Hub ID)
|
||||
model_path = self._get_model_path(model_size)
|
||||
|
||||
# Set up progress tracking
|
||||
progress_manager = get_progress_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
|
||||
# Check if model is being downloaded from HuggingFace Hub
|
||||
if model_path.startswith("Qwen/"):
|
||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
||||
|
||||
# Start tracking download task
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(model_name)
|
||||
|
||||
# Initialize progress state to show download has started
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=1, # Set to 1 initially, will be updated by callback
|
||||
filename="",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Set up progress callback
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback)
|
||||
|
||||
# Use progress tracker during download
|
||||
with tracker.patch_download():
|
||||
# Load the model - downloads will happen automatically with progress tracking
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
||||
)
|
||||
|
||||
# Mark as complete
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
else:
|
||||
# Local model, no download needed
|
||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
||||
)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"TTS model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading TTS model: {e}")
|
||||
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Async version of load_model that runs in thread pool.
|
||||
|
||||
This prevents blocking the event loop during model loading.
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
# If already loaded with correct size, return immediately
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
# Run the blocking load operation in a thread pool
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("TTS model unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
Args:
|
||||
audio_path: Path to reference audio file
|
||||
reference_text: Transcript of reference audio
|
||||
use_cache: Whether to use cached prompt if available
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
await self.load_model_async()
|
||||
|
||||
# Check cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cached_prompt = get_cached_voice_prompt(cache_key)
|
||||
if cached_prompt is not None:
|
||||
return cached_prompt, True
|
||||
|
||||
def _create_prompt_sync():
|
||||
"""Run synchronous voice prompt creation in thread pool."""
|
||||
return self.model.create_voice_clone_prompt(
|
||||
ref_audio=str(audio_path),
|
||||
ref_text=reference_text,
|
||||
x_vector_only_mode=False,
|
||||
)
|
||||
|
||||
# Run blocking operation in thread pool
|
||||
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
|
||||
|
||||
# Cache if enabled
|
||||
if use_cache:
|
||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||
|
||||
return voice_prompt_items, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples for better quality.
|
||||
|
||||
Args:
|
||||
audio_paths: List of audio file paths
|
||||
reference_texts: List of reference texts
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio, combined_text)
|
||||
"""
|
||||
from .utils.audio import load_audio
|
||||
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate audio
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text using voice prompt.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Voice prompt dictionary from create_voice_prompt
|
||||
language: Language code (en or zh)
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Natural language instruction for speech delivery control
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
# Load model (already handles async via to_thread if needed)
|
||||
await self.load_model_async()
|
||||
|
||||
def _generate_sync():
|
||||
"""Run synchronous generation in thread pool."""
|
||||
# Set seed if provided
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
|
||||
# Generate audio - this is the blocking operation
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
instruct=instruct,
|
||||
)
|
||||
return wavs[0], sample_rate
|
||||
|
||||
# Run blocking inference in thread pool to avoid blocking event loop
|
||||
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
async def generate_from_reference(
|
||||
self,
|
||||
text: str,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio directly from reference (convenience method).
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
audio_path: Path to reference audio
|
||||
reference_text: Transcript of reference audio
|
||||
language: Language code
|
||||
seed: Random seed
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
# Create voice prompt (with caching)
|
||||
voice_prompt, _ = await self.create_voice_prompt(audio_path, reference_text)
|
||||
|
||||
# Generate
|
||||
return await self.generate(text, voice_prompt, language, seed)
|
||||
|
||||
|
||||
# Global model instance
|
||||
_tts_model: Optional[TTSModel] = None
|
||||
|
||||
|
||||
def get_tts_model() -> TTSModel:
|
||||
"""Get or create TTS model instance."""
|
||||
global _tts_model
|
||||
if _tts_model is None:
|
||||
_tts_model = TTSModel()
|
||||
return _tts_model
|
||||
Returns:
|
||||
TTS backend instance
|
||||
"""
|
||||
return get_tts_backend()
|
||||
|
||||
|
||||
def unload_tts_model():
|
||||
"""Unload TTS model to free memory."""
|
||||
global _tts_model
|
||||
if _tts_model is not None:
|
||||
_tts_model.unload_model()
|
||||
backend = get_tts_backend()
|
||||
backend.unload_model()
|
||||
|
||||
|
||||
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
|
||||
"""Convert audio array to WAV bytes."""
|
||||
buffer = io.BytesIO()
|
||||
sf.write(buffer, audio, sample_rate, format="WAV")
|
||||
buffer.seek(0)
|
||||
return buffer.read()
|
||||
|
||||
|
||||
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
|
||||
|
||||
@@ -5,7 +5,7 @@ Voice prompt caching utilities.
|
||||
import hashlib
|
||||
import torch
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from typing import Optional, Union, Dict, Any
|
||||
|
||||
from .. import config
|
||||
|
||||
@@ -15,8 +15,8 @@ def _get_cache_dir() -> Path:
|
||||
return config.get_cache_dir()
|
||||
|
||||
|
||||
# In-memory cache
|
||||
_memory_cache: dict[str, torch.Tensor] = {}
|
||||
# In-memory cache - can store dict (voice prompt) or tensor (legacy)
|
||||
_memory_cache: dict[str, Union[torch.Tensor, Dict[str, Any]]] = {}
|
||||
|
||||
|
||||
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||
@@ -43,7 +43,7 @@ def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||
|
||||
def get_cached_voice_prompt(
|
||||
cache_key: str,
|
||||
) -> Optional[torch.Tensor]:
|
||||
) -> Optional[Union[torch.Tensor, Dict[str, Any]]]:
|
||||
"""
|
||||
Get cached voice prompt if available.
|
||||
|
||||
@@ -51,7 +51,7 @@ def get_cached_voice_prompt(
|
||||
cache_key: Cache key
|
||||
|
||||
Returns:
|
||||
Cached voice prompt tensor or None
|
||||
Cached voice prompt (dict or tensor) or None
|
||||
"""
|
||||
# Check in-memory cache
|
||||
if cache_key in _memory_cache:
|
||||
@@ -73,18 +73,18 @@ def get_cached_voice_prompt(
|
||||
|
||||
def cache_voice_prompt(
|
||||
cache_key: str,
|
||||
voice_prompt: torch.Tensor,
|
||||
voice_prompt: Union[torch.Tensor, Dict[str, Any]],
|
||||
) -> None:
|
||||
"""
|
||||
Cache voice prompt to memory and disk.
|
||||
|
||||
Args:
|
||||
cache_key: Cache key
|
||||
voice_prompt: Voice prompt tensor
|
||||
voice_prompt: Voice prompt (dict or tensor)
|
||||
"""
|
||||
# Store in memory
|
||||
_memory_cache[cache_key] = voice_prompt
|
||||
|
||||
# Store on disk
|
||||
# Store on disk (torch.save can handle both dicts and tensors)
|
||||
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
|
||||
torch.save(voice_prompt, cache_file)
|
||||
|
||||
@@ -29,8 +29,9 @@ class HFProgressTracker:
|
||||
|
||||
class TrackedTqdm(original_tqdm):
|
||||
"""A tqdm subclass that reports progress to our tracker."""
|
||||
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
print(f"[DEBUG TrackedTqdm] __init__ called with desc: {kwargs.get('desc', '')}")
|
||||
# Extract filename from desc before passing to parent
|
||||
desc = kwargs.get("desc", "")
|
||||
if not desc and args:
|
||||
@@ -79,8 +80,9 @@ class HFProgressTracker:
|
||||
}
|
||||
|
||||
def update(self, n=1):
|
||||
print(f"[DEBUG TrackedTqdm] update called with n={n}")
|
||||
result = super().update(n)
|
||||
|
||||
|
||||
# Report progress
|
||||
with tracker._lock:
|
||||
if id(self) in tracker._active_tqdms:
|
||||
@@ -118,11 +120,13 @@ class HFProgressTracker:
|
||||
@contextmanager
|
||||
def patch_download(self):
|
||||
"""Context manager to patch tqdm for progress tracking."""
|
||||
print("[DEBUG HFProgressTracker] patch_download called")
|
||||
try:
|
||||
import tqdm as tqdm_module
|
||||
|
||||
|
||||
# Store original tqdm class
|
||||
self._original_tqdm_class = tqdm_module.tqdm
|
||||
print(f"[DEBUG HFProgressTracker] Original tqdm class: {self._original_tqdm_class}")
|
||||
|
||||
# Reset totals
|
||||
with self._lock:
|
||||
@@ -135,18 +139,22 @@ class HFProgressTracker:
|
||||
|
||||
# Create our tracked tqdm class
|
||||
tracked_tqdm = self._create_tracked_tqdm_class()
|
||||
|
||||
print(f"[DEBUG HFProgressTracker] Created TrackedTqdm class: {tracked_tqdm}")
|
||||
|
||||
# Patch tqdm.tqdm
|
||||
tqdm_module.tqdm = tracked_tqdm
|
||||
|
||||
print(f"[DEBUG HFProgressTracker] Patched tqdm.tqdm")
|
||||
|
||||
# Also patch tqdm.auto.tqdm if it exists (used by huggingface_hub)
|
||||
self._original_tqdm_auto = None
|
||||
if hasattr(tqdm_module, "auto") and hasattr(tqdm_module.auto, "tqdm"):
|
||||
self._original_tqdm_auto = tqdm_module.auto.tqdm
|
||||
tqdm_module.auto.tqdm = tracked_tqdm
|
||||
print(f"[DEBUG HFProgressTracker] Patched tqdm.auto.tqdm")
|
||||
|
||||
# Patch in sys.modules to catch already-imported references
|
||||
self._patched_modules = {}
|
||||
patched_count = 0
|
||||
for module_name in list(sys.modules.keys()):
|
||||
if "huggingface" in module_name or module_name.startswith("tqdm"):
|
||||
try:
|
||||
@@ -159,8 +167,11 @@ class HFProgressTracker:
|
||||
):
|
||||
self._patched_modules[module_name] = attr
|
||||
setattr(module, "tqdm", tracked_tqdm)
|
||||
patched_count += 1
|
||||
print(f"[DEBUG HFProgressTracker] Patched {module_name}.tqdm")
|
||||
except (AttributeError, TypeError):
|
||||
pass
|
||||
print(f"[DEBUG HFProgressTracker] Patched {patched_count} modules in sys.modules")
|
||||
|
||||
yield
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Image processing utilities for avatar uploads."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
from PIL import Image
|
||||
|
||||
# JPEG can be reported as 'JPEG' or 'MPO' (for multi-picture format from some cameras)
|
||||
ALLOWED_FORMATS = {'PNG', 'JPEG', 'WEBP', 'MPO', 'JPG'}
|
||||
MAX_SIZE = 512
|
||||
MAX_FILE_SIZE = 5 * 1024 * 1024 # 5MB
|
||||
|
||||
|
||||
def validate_image(file_path: str) -> Tuple[bool, Optional[str]]:
|
||||
"""
|
||||
Validate image format and file size.
|
||||
|
||||
Args:
|
||||
file_path: Path to image file
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_message)
|
||||
"""
|
||||
path = Path(file_path)
|
||||
|
||||
# Check file size
|
||||
if path.stat().st_size > MAX_FILE_SIZE:
|
||||
return False, f"File size exceeds maximum of {MAX_FILE_SIZE // (1024 * 1024)}MB"
|
||||
|
||||
try:
|
||||
with Image.open(file_path) as img:
|
||||
# Verify the image can be loaded
|
||||
img.load()
|
||||
|
||||
# Check format (normalize JPEG variants)
|
||||
img_format = img.format
|
||||
if img_format in ('MPO', 'JPG'):
|
||||
img_format = 'JPEG'
|
||||
|
||||
if img_format not in {'PNG', 'JPEG', 'WEBP'}:
|
||||
return False, f"Invalid format '{img_format}'. Allowed formats: PNG, JPEG, WEBP"
|
||||
|
||||
return True, None
|
||||
except Exception as e:
|
||||
return False, f"Invalid image file: {str(e)}"
|
||||
|
||||
|
||||
def process_avatar(input_path: str, output_path: str, max_size: int = MAX_SIZE) -> None:
|
||||
"""
|
||||
Process avatar image: resize and optimize.
|
||||
|
||||
Resizes image to fit within max_size x max_size while maintaining aspect ratio.
|
||||
|
||||
Args:
|
||||
input_path: Path to input image
|
||||
output_path: Path to save processed image
|
||||
max_size: Maximum width or height in pixels
|
||||
"""
|
||||
with Image.open(input_path) as img:
|
||||
# Handle EXIF orientation for JPEG images
|
||||
try:
|
||||
from PIL import ExifTags
|
||||
for orientation in ExifTags.TAGS.keys():
|
||||
if ExifTags.TAGS[orientation] == 'Orientation':
|
||||
break
|
||||
exif = img._getexif()
|
||||
if exif is not None:
|
||||
orientation_value = exif.get(orientation)
|
||||
if orientation_value == 3:
|
||||
img = img.rotate(180, expand=True)
|
||||
elif orientation_value == 6:
|
||||
img = img.rotate(270, expand=True)
|
||||
elif orientation_value == 8:
|
||||
img = img.rotate(90, expand=True)
|
||||
except (AttributeError, KeyError, IndexError, TypeError):
|
||||
# No EXIF data or orientation tag
|
||||
pass
|
||||
|
||||
# Convert to RGB if necessary (handles RGBA, P, CMYK, etc.)
|
||||
if img.mode not in ('RGB', 'L'):
|
||||
if img.mode == 'RGBA':
|
||||
# Create white background for RGBA images
|
||||
background = Image.new('RGB', img.size, (255, 255, 255))
|
||||
background.paste(img, mask=img.split()[3]) # Use alpha channel as mask
|
||||
img = background
|
||||
elif img.mode == 'CMYK':
|
||||
# Convert CMYK to RGB
|
||||
img = img.convert('RGB')
|
||||
elif img.mode == 'P':
|
||||
# Convert palette mode to RGB
|
||||
img = img.convert('RGB')
|
||||
else:
|
||||
img = img.convert('RGB')
|
||||
|
||||
# Calculate new size maintaining aspect ratio
|
||||
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
|
||||
|
||||
# Determine output format from extension
|
||||
output_ext = Path(output_path).suffix.lower()
|
||||
|
||||
format_map = {
|
||||
'.png': 'PNG',
|
||||
'.jpeg': 'JPEG',
|
||||
'.jpg': 'JPEG',
|
||||
'.webp': 'WEBP'
|
||||
}
|
||||
|
||||
output_format = format_map.get(output_ext, 'PNG')
|
||||
|
||||
# Save with optimization
|
||||
save_kwargs = {'optimize': True}
|
||||
if output_format == 'JPEG':
|
||||
save_kwargs['quality'] = 90
|
||||
|
||||
img.save(output_path, format=output_format, **save_kwargs)
|
||||
@@ -6,16 +6,55 @@ from typing import Optional, Callable, Dict, List
|
||||
from fastapi.responses import StreamingResponse
|
||||
import asyncio
|
||||
import json
|
||||
import threading
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
class ProgressManager:
|
||||
"""Manages download progress for multiple models."""
|
||||
"""Manages download progress for multiple models.
|
||||
|
||||
Thread-safe: can be called from background threads (e.g., via asyncio.to_thread).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._progress: Dict[str, Dict] = {}
|
||||
self._listeners: Dict[str, list] = {}
|
||||
self._lock = threading.Lock() # Thread-safe lock for progress dict
|
||||
self._main_loop: Optional[asyncio.AbstractEventLoop] = None
|
||||
|
||||
def _set_main_loop(self, loop: asyncio.AbstractEventLoop):
|
||||
"""Set the main event loop for thread-safe operations."""
|
||||
self._main_loop = loop
|
||||
|
||||
def _notify_listeners_threadsafe(self, model_name: str, progress_data: Dict):
|
||||
"""Notify listeners in a thread-safe manner."""
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if model_name not in self._listeners:
|
||||
return
|
||||
|
||||
for queue in self._listeners[model_name]:
|
||||
try:
|
||||
# Check if we're in the main event loop thread
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
# We're in an async context, can use put_nowait directly
|
||||
queue.put_nowait(progress_data.copy())
|
||||
except RuntimeError:
|
||||
# Not in async context (running in background thread)
|
||||
# Use call_soon_threadsafe to safely put on queue
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
self._main_loop.call_soon_threadsafe(
|
||||
lambda q=queue, d=progress_data.copy(): q.put_nowait(d) if not q.full() else None
|
||||
)
|
||||
else:
|
||||
logger.debug(f"No main loop available for {model_name}, skipping notification")
|
||||
except asyncio.QueueFull:
|
||||
logger.warning(f"Queue full for {model_name}, dropping update")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error notifying listener for {model_name}: {e}")
|
||||
|
||||
def update_progress(
|
||||
self,
|
||||
model_name: str,
|
||||
@@ -26,7 +65,9 @@ class ProgressManager:
|
||||
):
|
||||
"""
|
||||
Update progress for a model download.
|
||||
|
||||
|
||||
Thread-safe: can be called from background threads.
|
||||
|
||||
Args:
|
||||
model_name: Name of the model (e.g., "qwen-tts-1.7B", "whisper-base")
|
||||
current: Current bytes downloaded
|
||||
@@ -34,9 +75,12 @@ class ProgressManager:
|
||||
filename: Current file being downloaded
|
||||
status: Status string (downloading, extracting, complete, error)
|
||||
"""
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
progress_pct = (current / total * 100) if total > 0 else 0
|
||||
|
||||
self._progress[model_name] = {
|
||||
|
||||
progress_data = {
|
||||
"model_name": model_name,
|
||||
"current": current,
|
||||
"total": total,
|
||||
@@ -45,26 +89,43 @@ class ProgressManager:
|
||||
"status": status,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
}
|
||||
|
||||
# Notify all listeners
|
||||
if model_name in self._listeners:
|
||||
for queue in self._listeners[model_name]:
|
||||
try:
|
||||
queue.put_nowait(self._progress[model_name].copy())
|
||||
except asyncio.QueueFull:
|
||||
pass
|
||||
|
||||
print(f"[DEBUG] update_progress called: {model_name}, {progress_pct:.1f}%")
|
||||
|
||||
# Thread-safe update of progress dict
|
||||
with self._lock:
|
||||
self._progress[model_name] = progress_data
|
||||
|
||||
# Notify all listeners (thread-safe)
|
||||
listener_count = len(self._listeners.get(model_name, []))
|
||||
print(f"[DEBUG] Listener count for {model_name}: {listener_count}")
|
||||
print(f"[DEBUG] All listeners: {list(self._listeners.keys())}")
|
||||
print(f"[DEBUG] Main loop set: {self._main_loop is not None}")
|
||||
if self._main_loop:
|
||||
print(f"[DEBUG] Main loop running: {self._main_loop.is_running()}")
|
||||
|
||||
if listener_count > 0:
|
||||
logger.debug(f"Notifying {listener_count} listeners for {model_name}: {progress_pct:.1f}% ({filename})")
|
||||
print(f"[DEBUG] About to notify listeners...")
|
||||
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||
print(f"[DEBUG] Notified listeners")
|
||||
else:
|
||||
logger.debug(f"No listeners for {model_name}, progress update stored: {progress_pct:.1f}%")
|
||||
|
||||
def get_progress(self, model_name: str) -> Optional[Dict]:
|
||||
"""Get current progress for a model."""
|
||||
return self._progress.get(model_name)
|
||||
"""Get current progress for a model. Thread-safe."""
|
||||
with self._lock:
|
||||
progress = self._progress.get(model_name)
|
||||
return progress.copy() if progress else None
|
||||
|
||||
def get_all_active(self) -> List[Dict]:
|
||||
"""Get all active downloads (status is 'downloading' or 'extracting')."""
|
||||
"""Get all active downloads (status is 'downloading' or 'extracting'). Thread-safe."""
|
||||
active = []
|
||||
for model_name, progress in self._progress.items():
|
||||
status = progress.get("status", "")
|
||||
if status in ("downloading", "extracting"):
|
||||
active.append(progress.copy())
|
||||
with self._lock:
|
||||
for model_name, progress in self._progress.items():
|
||||
status = progress.get("status", "")
|
||||
if status in ("downloading", "extracting"):
|
||||
active.append(progress.copy())
|
||||
return active
|
||||
|
||||
def create_progress_callback(self, model_name: str, filename: Optional[str] = None):
|
||||
@@ -98,30 +159,57 @@ class ProgressManager:
|
||||
async def subscribe(self, model_name: str):
|
||||
"""
|
||||
Subscribe to progress updates for a model.
|
||||
|
||||
|
||||
Yields progress updates as Server-Sent Events.
|
||||
"""
|
||||
queue = asyncio.Queue(maxsize=10)
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Store the main event loop for thread-safe operations
|
||||
try:
|
||||
self._main_loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
queue = asyncio.Queue(maxsize=10)
|
||||
|
||||
# Add to listeners
|
||||
if model_name not in self._listeners:
|
||||
self._listeners[model_name] = []
|
||||
self._listeners[model_name].append(queue)
|
||||
|
||||
|
||||
logger.info(f"SSE client subscribed to {model_name}, total listeners: {len(self._listeners[model_name])}")
|
||||
|
||||
try:
|
||||
# Send initial progress if available
|
||||
if model_name in self._progress:
|
||||
yield f"data: {json.dumps(self._progress[model_name])}\n\n"
|
||||
# Send initial progress if available and still in progress (thread-safe read)
|
||||
with self._lock:
|
||||
initial_progress = self._progress.get(model_name)
|
||||
if initial_progress:
|
||||
initial_progress = initial_progress.copy()
|
||||
|
||||
if initial_progress:
|
||||
status = initial_progress.get('status')
|
||||
# Only send initial progress if download is actually in progress
|
||||
# Don't send old 'complete' or 'error' status from previous downloads
|
||||
if status in ('downloading', 'extracting'):
|
||||
logger.info(f"Sending initial progress for {model_name}: {status}")
|
||||
yield f"data: {json.dumps(initial_progress)}\n\n"
|
||||
else:
|
||||
logger.info(f"Skipping initial progress for {model_name} (status: {status})")
|
||||
else:
|
||||
logger.info(f"No initial progress available for {model_name}")
|
||||
|
||||
# Stream updates
|
||||
while True:
|
||||
try:
|
||||
# Wait for update with timeout
|
||||
progress = await asyncio.wait_for(queue.get(), timeout=1.0)
|
||||
logger.debug(f"Sending progress update for {model_name}: {progress.get('status')} - {progress.get('progress', 0):.1f}%")
|
||||
yield f"data: {json.dumps(progress)}\n\n"
|
||||
|
||||
|
||||
# Stop if complete or error
|
||||
if progress.get("status") in ("complete", "error"):
|
||||
logger.info(f"Download {progress.get('status')} for {model_name}, closing SSE connection")
|
||||
break
|
||||
except asyncio.TimeoutError:
|
||||
# Send heartbeat
|
||||
@@ -133,32 +221,53 @@ class ProgressManager:
|
||||
self._listeners[model_name].remove(queue)
|
||||
if not self._listeners[model_name]:
|
||||
del self._listeners[model_name]
|
||||
logger.info(f"SSE client unsubscribed from {model_name}, remaining listeners: {len(self._listeners.get(model_name, []))}")
|
||||
|
||||
def mark_complete(self, model_name: str):
|
||||
"""Mark a model download as complete."""
|
||||
if model_name in self._progress:
|
||||
self._progress[model_name]["status"] = "complete"
|
||||
self._progress[model_name]["progress"] = 100.0
|
||||
# Notify listeners
|
||||
if model_name in self._listeners:
|
||||
for queue in self._listeners[model_name]:
|
||||
try:
|
||||
queue.put_nowait(self._progress[model_name].copy())
|
||||
except asyncio.QueueFull:
|
||||
pass
|
||||
"""Mark a model download as complete. Thread-safe."""
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
with self._lock:
|
||||
if model_name in self._progress:
|
||||
self._progress[model_name]["status"] = "complete"
|
||||
self._progress[model_name]["progress"] = 100.0
|
||||
progress_data = self._progress[model_name].copy()
|
||||
else:
|
||||
logger.warning(f"Cannot mark {model_name} as complete: not found in progress")
|
||||
return
|
||||
|
||||
logger.info(f"Marked {model_name} as complete")
|
||||
# Notify listeners (thread-safe)
|
||||
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||
|
||||
def mark_error(self, model_name: str, error: str):
|
||||
"""Mark a model download as failed."""
|
||||
if model_name in self._progress:
|
||||
self._progress[model_name]["status"] = "error"
|
||||
self._progress[model_name]["error"] = error
|
||||
# Notify listeners
|
||||
if model_name in self._listeners:
|
||||
for queue in self._listeners[model_name]:
|
||||
try:
|
||||
queue.put_nowait(self._progress[model_name].copy())
|
||||
except asyncio.QueueFull:
|
||||
pass
|
||||
"""Mark a model download as failed. Thread-safe."""
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
with self._lock:
|
||||
if model_name in self._progress:
|
||||
self._progress[model_name]["status"] = "error"
|
||||
self._progress[model_name]["error"] = error
|
||||
progress_data = self._progress[model_name].copy()
|
||||
else:
|
||||
# Create new progress entry for error
|
||||
progress_data = {
|
||||
"model_name": model_name,
|
||||
"current": 0,
|
||||
"total": 0,
|
||||
"progress": 0,
|
||||
"filename": None,
|
||||
"status": "error",
|
||||
"error": error,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
}
|
||||
self._progress[model_name] = progress_data
|
||||
|
||||
logger.error(f"Marked {model_name} as error: {error}")
|
||||
# Notify listeners (thread-safe)
|
||||
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||
|
||||
|
||||
# Global progress manager instance
|
||||
|
||||
@@ -4,16 +4,20 @@ from PyInstaller.utils.hooks import collect_submodules
|
||||
from PyInstaller.utils.hooks import copy_metadata
|
||||
|
||||
datas = []
|
||||
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern']
|
||||
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
|
||||
datas += collect_data_files('qwen_tts')
|
||||
datas += collect_data_files('mlx')
|
||||
datas += collect_data_files('mlx_audio')
|
||||
datas += copy_metadata('qwen-tts')
|
||||
hiddenimports += collect_submodules('qwen_tts')
|
||||
hiddenimports += collect_submodules('jaraco')
|
||||
hiddenimports += collect_submodules('mlx')
|
||||
hiddenimports += collect_submodules('mlx_audio')
|
||||
|
||||
|
||||
a = Analysis(
|
||||
['server.py'],
|
||||
pathex=['C:\\Users\\ijame\\Projects\\voice\\Qwen3-TTS'],
|
||||
pathex=[],
|
||||
binaries=[],
|
||||
datas=datas,
|
||||
hiddenimports=hiddenimports,
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
},
|
||||
"app": {
|
||||
"name": "@voicebox/app",
|
||||
"version": "0.1.5",
|
||||
"version": "0.1.9",
|
||||
"dependencies": {
|
||||
"@dnd-kit/core": "^6.3.1",
|
||||
"@dnd-kit/sortable": "^10.0.0",
|
||||
@@ -50,6 +50,7 @@
|
||||
"react": "^18.3.0",
|
||||
"react-dom": "^18.3.0",
|
||||
"react-hook-form": "^7.53.0",
|
||||
"react-sound-visualizer": "^1.4.0",
|
||||
"tailwind-merge": "^2.5.4",
|
||||
"wavesurfer.js": "^7.0.0",
|
||||
"zod": "^3.23.8",
|
||||
@@ -67,7 +68,7 @@
|
||||
},
|
||||
"landing": {
|
||||
"name": "@voicebox/landing",
|
||||
"version": "0.1.5",
|
||||
"version": "0.1.9",
|
||||
"dependencies": {
|
||||
"@radix-ui/react-separator": "^1.1.8",
|
||||
"@radix-ui/react-slot": "^1.2.4",
|
||||
@@ -92,7 +93,7 @@
|
||||
},
|
||||
"tauri": {
|
||||
"name": "@voicebox/tauri",
|
||||
"version": "0.1.5",
|
||||
"version": "0.1.9",
|
||||
"dependencies": {
|
||||
"@tauri-apps/api": "^2.0.0",
|
||||
"@tauri-apps/plugin-shell": "^2.0.0",
|
||||
@@ -111,7 +112,7 @@
|
||||
},
|
||||
"web": {
|
||||
"name": "@voicebox/web",
|
||||
"version": "0.1.5",
|
||||
"version": "0.1.9",
|
||||
"dependencies": {
|
||||
"@tanstack/react-query": "^5.0.0",
|
||||
"react": "^18.3.0",
|
||||
@@ -970,6 +971,8 @@
|
||||
|
||||
"react-remove-scroll-bar": ["[email protected]", "", { "dependencies": { "react-style-singleton": "^2.2.2", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" }, "optionalPeers": ["@types/react"] }, "sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q=="],
|
||||
|
||||
"react-sound-visualizer": ["[email protected]", "", { "dependencies": { "sound-visualizer": "^1.2.0" }, "peerDependencies": { "react": ">= 16" } }, "sha512-Qe7tFTd1owtQ8nYrUYXg7QLt8mw7iUy86mqj/+IwmXzSw+NlhnMnAGPuisb1Lk3ncliFnM+AQbZb3C4RQN9uMQ=="],
|
||||
|
||||
"react-style-singleton": ["[email protected]", "", { "dependencies": { "get-nonce": "^1.0.0", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-b6jSvxvVnyptAiLjbkWLE/lOnR4lfTtDAl+eUC7RZy+QQWc6wRzIV2CE6xBuMmDxc2qIihtDCZD5NPOFl7fRBQ=="],
|
||||
|
||||
"read-cache": ["[email protected]", "", { "dependencies": { "pify": "^2.3.0" } }, "sha512-Owdv/Ft7IjOgm/i0xvNDZ1LrRANRfew4b2prF3OWMQLxLfu3bS8FVhCsrSCMK4lR56Y9ya+AThoTpDCTxCmpRA=="],
|
||||
@@ -1004,6 +1007,8 @@
|
||||
|
||||
"slash": ["[email protected]", "", {}, "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q=="],
|
||||
|
||||
"sound-visualizer": ["[email protected]", "", {}, "sha512-2+Un0PrrBgXylnCjrVYUoRW7KEDH29h7O8/MGzeDOgFGBPb9oX/2n/RGBxJXvVv2U3KFwX5olUWeJKf0Rr5TLQ=="],
|
||||
|
||||
"source-map-js": ["[email protected]", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
|
||||
|
||||
"strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="],
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
node_modules
|
||||
.mintlify
|
||||
.DS_Store
|
||||
@@ -0,0 +1,64 @@
|
||||
# Voicebox Documentation
|
||||
|
||||
This directory contains the documentation for Voicebox, built with [Mintlify](https://mintlify.com).
|
||||
|
||||
## Development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Install Mintlify globally using bun:
|
||||
|
||||
```bash
|
||||
bun add -g mintlify
|
||||
```
|
||||
|
||||
Or use the helper script:
|
||||
|
||||
```bash
|
||||
bun run install:mintlify
|
||||
```
|
||||
|
||||
### Running Locally
|
||||
|
||||
```bash
|
||||
bun run dev
|
||||
```
|
||||
|
||||
This will start the Mintlify dev server.
|
||||
|
||||
The docs will be available at `http://localhost:3000`
|
||||
|
||||
### Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── mint.json # Mintlify configuration
|
||||
├── custom.css # Custom styles
|
||||
├── overview/ # Getting started & feature docs
|
||||
├── guides/ # User guides
|
||||
├── api/ # API reference
|
||||
├── development/ # Developer documentation
|
||||
├── logo/ # Logo assets
|
||||
└── public/ # Static assets
|
||||
```
|
||||
|
||||
### Writing Docs
|
||||
|
||||
- Use `.mdx` files for all documentation pages
|
||||
- Follow the existing structure in `mint.json` for navigation
|
||||
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
|
||||
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
|
||||
|
||||
## Deployment
|
||||
|
||||
Docs are automatically deployed when changes are pushed to the main branch.
|
||||
|
||||
To manually deploy:
|
||||
|
||||
```bash
|
||||
mintlify deploy
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
|
||||
@@ -90,6 +90,26 @@ chmod +x voicebox-*.AppImage
|
||||
- Slower but works without GPU
|
||||
- Backend automatically falls back to CPU
|
||||
|
||||
### MLX "Failed to load the default metallib" error (Apple Silicon)
|
||||
|
||||
**Symptoms:** Generation fails with "library not found" or "metallib" errors
|
||||
|
||||
**Solutions:**
|
||||
1. **Rebuild server binary**
|
||||
```bash
|
||||
bun run build:server
|
||||
```
|
||||
The build script should automatically include MLX Metal shader libraries.
|
||||
|
||||
2. **Check MLX installation**
|
||||
```bash
|
||||
pip install -r backend/requirements-mlx.txt
|
||||
```
|
||||
|
||||
3. **Verify backend detection**
|
||||
- Check server logs for "Backend: MLX"
|
||||
- If showing "Backend: PYTORCH", MLX may not be installed correctly
|
||||
|
||||
### Audio playback issues
|
||||
|
||||
**Symptoms:** Generated audio won't play
|
||||
@@ -111,19 +131,27 @@ chmod +x voicebox-*.AppImage
|
||||
**Symptoms:** Generation takes >30 seconds
|
||||
|
||||
**Solutions:**
|
||||
1. **Use GPU** (if available)
|
||||
1. **Check backend type** (Apple Silicon)
|
||||
- Check Settings → Server Status
|
||||
- Should show "Backend: MLX" on Apple Silicon
|
||||
- If showing "Backend: PYTORCH", install MLX: `pip install -r backend/requirements-mlx.txt`
|
||||
- MLX provides 4-5x faster inference on Apple Silicon
|
||||
|
||||
2. **Use GPU** (if available)
|
||||
- Check Settings → Server Status
|
||||
- Should show "GPU available: true"
|
||||
- Apple Silicon: Should show "Metal (Apple Silicon via MLX)"
|
||||
- Windows/Linux: Should show "CUDA" if GPU available
|
||||
|
||||
2. **Enable caching**
|
||||
3. **Enable caching**
|
||||
- Voice prompts are cached automatically
|
||||
- Second generation with same voice should be faster
|
||||
|
||||
3. **Use smaller model**
|
||||
4. **Use smaller model**
|
||||
- 0.6B model is faster than 1.7B
|
||||
- Quality difference is minimal for most voices
|
||||
|
||||
4. **Check system resources**
|
||||
5. **Check system resources**
|
||||
- Close other CPU/GPU intensive apps
|
||||
- Ensure adequate RAM (8GB+ recommended)
|
||||
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: "Authentication"
|
||||
description: "API authentication and security"
|
||||
---
|
||||
|
||||
## Current Status
|
||||
|
||||
<Warning>
|
||||
Authentication is not currently implemented in Voicebox. The API is intended for local use only.
|
||||
</Warning>
|
||||
|
||||
## Local Usage
|
||||
|
||||
For local development and usage:
|
||||
- API runs on `localhost:17493`
|
||||
- No authentication required
|
||||
- Access restricted to local machine
|
||||
|
||||
## Future Implementation
|
||||
|
||||
Authentication will be added in a future release for:
|
||||
- Remote deployments
|
||||
- Multi-user access
|
||||
- Production environments
|
||||
|
||||
Planned authentication methods:
|
||||
- API keys
|
||||
- OAuth 2.0
|
||||
- JWT tokens
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
Until authentication is implemented:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Use VPN" icon="shield">
|
||||
Use WireGuard or Tailscale for remote access
|
||||
</Card>
|
||||
<Card title="Reverse Proxy" icon="server">
|
||||
Run behind nginx with basic auth
|
||||
</Card>
|
||||
<Card title="Firewall" icon="fire">
|
||||
Restrict access to trusted IPs only
|
||||
</Card>
|
||||
<Card title="Local Only" icon="laptop">
|
||||
Don't expose to public internet
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Coming Soon
|
||||
|
||||
- API key management
|
||||
- User accounts
|
||||
- Rate limiting
|
||||
- Access control
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "Generation API"
|
||||
description: "Generate speech from text"
|
||||
---
|
||||
|
||||
## Generate Speech
|
||||
|
||||
```http
|
||||
POST /generate
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"language": "en"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "gen123",
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"language": "en",
|
||||
"audio_url": "/audio/gen123.wav",
|
||||
"duration": 2.3,
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## List History
|
||||
|
||||
```http
|
||||
GET /history
|
||||
```
|
||||
|
||||
**Query Parameters:**
|
||||
- `profile_id` (optional) - Filter by voice profile
|
||||
- `limit` (optional) - Number of results (default: 50)
|
||||
- `offset` (optional) - Pagination offset
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"generations": [
|
||||
{
|
||||
"id": "gen123",
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"duration": 2.3,
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
],
|
||||
"total": 100
|
||||
}
|
||||
```
|
||||
|
||||
## Get Generation
|
||||
|
||||
```http
|
||||
GET /history/{id}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "gen123",
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"language": "en",
|
||||
"audio_url": "/audio/gen123.wav",
|
||||
"duration": 2.3,
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Delete Generation
|
||||
|
||||
```http
|
||||
DELETE /history/{id}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
## TypeScript Example
|
||||
|
||||
```typescript
|
||||
import { VoiceboxClient } from '@/lib/api'
|
||||
|
||||
const client = new VoiceboxClient({
|
||||
baseUrl: 'http://localhost:17493'
|
||||
})
|
||||
|
||||
// Generate speech
|
||||
const generation = await client.generate({
|
||||
text: 'Hello world',
|
||||
profile_id: 'abc123',
|
||||
language: 'en'
|
||||
})
|
||||
|
||||
// Get audio URL
|
||||
const audioUrl = generation.audio_url
|
||||
|
||||
// List history
|
||||
const history = await client.listHistory({
|
||||
profile_id: 'abc123',
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
title: "API Overview"
|
||||
description: "Integrate voice synthesis into your applications with the Voicebox REST API"
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
Voicebox exposes a full REST API that allows you to integrate voice synthesis into your own applications. The API runs on `http://localhost:17493` by default.
|
||||
|
||||
<Card title="Interactive API Docs" icon="book" href="http://localhost:17493/docs">
|
||||
When Voicebox is running, visit the auto-generated API documentation at `http://localhost:17493/docs`
|
||||
</Card>
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
http://localhost:17493
|
||||
```
|
||||
|
||||
For remote deployments, replace `localhost` with your server's IP or hostname.
|
||||
|
||||
## Authentication
|
||||
|
||||
<Note>
|
||||
Currently, the API does not require authentication for local development. Authentication will be added in a future release for production deployments.
|
||||
</Note>
|
||||
|
||||
## Quick Example
|
||||
|
||||
Here's a simple example of generating speech:
|
||||
|
||||
```bash
|
||||
# Generate speech
|
||||
curl -X POST http://localhost:17493/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"language": "en"
|
||||
}'
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
The Voicebox API is organized into several categories:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Voice Profiles" icon="user" href="/api/voice-profiles">
|
||||
Create, list, update, and delete voice profiles
|
||||
</Card>
|
||||
<Card title="Generation" icon="waveform" href="/api/generation">
|
||||
Generate speech from text using voice profiles
|
||||
</Card>
|
||||
<Card title="Recordings" icon="microphone" href="/api/recordings">
|
||||
Record and transcribe audio
|
||||
</Card>
|
||||
<Card title="Stories" icon="film">
|
||||
Create and manage multi-voice stories (coming soon)
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Core Endpoints
|
||||
|
||||
### Voice Profiles
|
||||
|
||||
```http
|
||||
GET /profiles # List all profiles
|
||||
POST /profiles # Create a new profile
|
||||
GET /profiles/{id} # Get profile details
|
||||
PUT /profiles/{id} # Update a profile
|
||||
DELETE /profiles/{id} # Delete a profile
|
||||
POST /profiles/{id}/samples # Add voice sample
|
||||
```
|
||||
|
||||
### Generation
|
||||
|
||||
```http
|
||||
POST /generate # Generate speech
|
||||
GET /history # List generation history
|
||||
GET /history/{id} # Get generation details
|
||||
DELETE /history/{id} # Delete from history
|
||||
```
|
||||
|
||||
### Recordings
|
||||
|
||||
```http
|
||||
POST /recordings # Start recording
|
||||
POST /recordings/stop # Stop recording
|
||||
POST /transcribe # Transcribe audio
|
||||
```
|
||||
|
||||
## Response Format
|
||||
|
||||
All API responses follow a consistent JSON format:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
// Response data
|
||||
},
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
Error responses:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"data": null,
|
||||
"error": {
|
||||
"message": "Error description",
|
||||
"code": "ERROR_CODE"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Data Models
|
||||
|
||||
### Voice Profile
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "abc123",
|
||||
"name": "John Smith",
|
||||
"language": "en",
|
||||
"description": "Professional narrator voice",
|
||||
"created_at": "2024-01-29T12:00:00Z",
|
||||
"samples": [
|
||||
{
|
||||
"id": "sample123",
|
||||
"audio_path": "/path/to/sample.wav",
|
||||
"duration": 15.5
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Generation
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "gen123",
|
||||
"text": "Hello world",
|
||||
"profile_id": "abc123",
|
||||
"language": "en",
|
||||
"audio_path": "/path/to/output.wav",
|
||||
"duration": 2.3,
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## TypeScript Client
|
||||
|
||||
Voicebox provides an auto-generated TypeScript client with full type safety:
|
||||
|
||||
```typescript
|
||||
import { VoiceboxClient } from '@/lib/api'
|
||||
|
||||
const client = new VoiceboxClient({
|
||||
baseUrl: 'http://localhost:17493'
|
||||
})
|
||||
|
||||
// Create a profile
|
||||
const profile = await client.createProfile({
|
||||
name: 'John Smith',
|
||||
language: 'en'
|
||||
})
|
||||
|
||||
// Generate speech
|
||||
const generation = await client.generate({
|
||||
text: 'Hello world',
|
||||
profile_id: profile.id,
|
||||
language: 'en'
|
||||
})
|
||||
```
|
||||
|
||||
The client is automatically generated from the OpenAPI schema. See [Development Setup](/development/setup#generate-openapi-client) for details.
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
<Info>
|
||||
Currently, there are no rate limits for local usage. Rate limiting will be added in a future release for production deployments.
|
||||
</Info>
|
||||
|
||||
## WebSocket Support
|
||||
|
||||
<Note>
|
||||
Real-time streaming generation via WebSockets is planned for a future release.
|
||||
</Note>
|
||||
|
||||
## Use Cases
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Game Development" icon="gamepad">
|
||||
Generate dynamic dialogue for NPCs and characters
|
||||
</Card>
|
||||
<Card title="Content Creation" icon="video">
|
||||
Automate voiceovers for videos and podcasts
|
||||
</Card>
|
||||
<Card title="Accessibility" icon="universal-access">
|
||||
Build text-to-speech tools for visually impaired users
|
||||
</Card>
|
||||
<Card title="Voice Assistants" icon="robot">
|
||||
Create custom voice interfaces
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Voice Profiles API" icon="user" href="/api/voice-profiles">
|
||||
Learn how to manage voice profiles
|
||||
</Card>
|
||||
<Card title="Generation API" icon="waveform" href="/api/generation">
|
||||
Generate speech from text
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
title: "Recordings API"
|
||||
description: "Record and transcribe audio"
|
||||
---
|
||||
|
||||
## Start Recording
|
||||
|
||||
```http
|
||||
POST /recordings/start
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"source": "microphone"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"recording_id": "rec123",
|
||||
"status": "recording"
|
||||
}
|
||||
```
|
||||
|
||||
## Stop Recording
|
||||
|
||||
```http
|
||||
POST /recordings/stop
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"recording_id": "rec123"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"recording_id": "rec123",
|
||||
"audio_url": "/audio/rec123.wav",
|
||||
"duration": 15.5
|
||||
}
|
||||
```
|
||||
|
||||
## Transcribe Audio
|
||||
|
||||
```http
|
||||
POST /transcribe
|
||||
```
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
```
|
||||
audio: <file>
|
||||
language: "en" (optional)
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"text": "Transcribed speech text here",
|
||||
"language": "en",
|
||||
"duration": 15.5,
|
||||
"confidence": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
## TypeScript Example
|
||||
|
||||
```typescript
|
||||
import { VoiceboxClient } from '@/lib/api'
|
||||
|
||||
const client = new VoiceboxClient({
|
||||
baseUrl: 'http://localhost:17493'
|
||||
})
|
||||
|
||||
// Start recording
|
||||
const recording = await client.startRecording({
|
||||
source: 'microphone'
|
||||
})
|
||||
|
||||
// ... record audio ...
|
||||
|
||||
// Stop recording
|
||||
const result = await client.stopRecording(recording.id)
|
||||
|
||||
// Transcribe
|
||||
const transcription = await client.transcribe(audioFile, 'en')
|
||||
console.log(transcription.text)
|
||||
```
|
||||
|
||||
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
title: "Voice Profiles API"
|
||||
description: "Manage voice profiles programmatically"
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### List Profiles
|
||||
|
||||
```http
|
||||
GET /profiles
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"profiles": [
|
||||
{
|
||||
"id": "abc123",
|
||||
"name": "John Smith",
|
||||
"language": "en",
|
||||
"description": "Professional narrator",
|
||||
"created_at": "2024-01-29T12:00:00Z",
|
||||
"sample_count": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Get Profile
|
||||
|
||||
```http
|
||||
GET /profiles/{id}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "abc123",
|
||||
"name": "John Smith",
|
||||
"language": "en",
|
||||
"description": "Professional narrator",
|
||||
"created_at": "2024-01-29T12:00:00Z",
|
||||
"samples": [
|
||||
{
|
||||
"id": "sample123",
|
||||
"duration": 15.5,
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Create Profile
|
||||
|
||||
```http
|
||||
POST /profiles
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "John Smith",
|
||||
"language": "en",
|
||||
"description": "Professional narrator"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "abc123",
|
||||
"name": "John Smith",
|
||||
"language": "en",
|
||||
"description": "Professional narrator",
|
||||
"created_at": "2024-01-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Update Profile
|
||||
|
||||
```http
|
||||
PUT /profiles/{id}
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "Updated Name",
|
||||
"description": "Updated description"
|
||||
}
|
||||
```
|
||||
|
||||
### Delete Profile
|
||||
|
||||
```http
|
||||
DELETE /profiles/{id}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
### Add Voice Sample
|
||||
|
||||
```http
|
||||
POST /profiles/{id}/samples
|
||||
```
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
```
|
||||
audio: <file>
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"sample_id": "sample123",
|
||||
"duration": 15.5
|
||||
}
|
||||
```
|
||||
|
||||
## TypeScript Example
|
||||
|
||||
```typescript
|
||||
import { VoiceboxClient } from '@/lib/api'
|
||||
|
||||
const client = new VoiceboxClient({
|
||||
baseUrl: 'http://localhost:17493'
|
||||
})
|
||||
|
||||
// Create profile
|
||||
const profile = await client.createProfile({
|
||||
name: 'John Smith',
|
||||
language: 'en',
|
||||
description: 'Professional narrator'
|
||||
})
|
||||
|
||||
// Add sample
|
||||
await client.addSample(profile.id, audioFile)
|
||||
|
||||
// List all profiles
|
||||
const profiles = await client.listProfiles()
|
||||
```
|
||||
|
||||
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||
@@ -0,0 +1,15 @@
|
||||
/* Anchor hover styles */
|
||||
.nav-anchor:hover {
|
||||
@apply text-[#BF9E40];
|
||||
}
|
||||
|
||||
/* Icon wrapper on hover */
|
||||
.nav-anchor:hover div {
|
||||
background: #BF9E40 !important;
|
||||
filter: brightness(1) !important;
|
||||
}
|
||||
|
||||
/* Icon SVG on hover */
|
||||
.nav-anchor:hover svg {
|
||||
@apply bg-white !important;
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
---
|
||||
title: "Architecture"
|
||||
description: "Understanding Voicebox's technical architecture"
|
||||
---
|
||||
|
||||
## System Overview
|
||||
|
||||
Voicebox uses a client-server architecture with a React frontend and Python backend. The desktop app is built with Tauri and contains two main layers:
|
||||
|
||||
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
|
||||
|
||||
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
|
||||
|
||||
These two layers communicate via HTTP, with the frontend making API requests to the backend.
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
### Tech Stack
|
||||
|
||||
- **Framework**: React 18 with TypeScript
|
||||
- **State Management**: Zustand stores
|
||||
- **Data Fetching**: React Query (TanStack Query)
|
||||
- **Styling**: Tailwind CSS
|
||||
- **Audio**: WaveSurfer.js
|
||||
- **Desktop**: Tauri (Rust)
|
||||
|
||||
### Component Structure
|
||||
|
||||
```
|
||||
app/src/
|
||||
├── components/ # React components
|
||||
│ ├── profiles/ # Voice profile UI
|
||||
│ ├── generation/ # Speech generation UI
|
||||
│ ├── stories/ # Timeline editor
|
||||
│ └── shared/ # Reusable components
|
||||
├── lib/ # Utilities
|
||||
│ ├── api/ # Generated API client
|
||||
│ └── utils/ # Helper functions
|
||||
├── hooks/ # React hooks
|
||||
└── stores/ # Zustand state stores
|
||||
```
|
||||
|
||||
### State Management
|
||||
|
||||
```typescript
|
||||
// Example: Profile store
|
||||
const useProfileStore = create((set) => ({
|
||||
profiles: [],
|
||||
selectedProfile: null,
|
||||
setProfiles: (profiles) => set({ profiles }),
|
||||
selectProfile: (id) => set({ selectedProfile: id })
|
||||
}))
|
||||
```
|
||||
|
||||
## Backend Architecture
|
||||
|
||||
### Tech Stack
|
||||
|
||||
- **Framework**: FastAPI (Python 3.11+)
|
||||
- **TTS Model**: Qwen3-TTS
|
||||
- **Transcription**: Whisper
|
||||
- **Database**: SQLite
|
||||
- **Audio**: librosa, soundfile
|
||||
|
||||
### API Structure
|
||||
|
||||
```python
|
||||
# main.py - API routes
|
||||
@app.post("/generate")
|
||||
async def generate_speech(request: GenerateRequest):
|
||||
# 1. Validate request
|
||||
# 2. Load voice profile
|
||||
# 3. Generate audio with TTS
|
||||
# 4. Save to database
|
||||
# 5. Return response
|
||||
```
|
||||
|
||||
### Data Model
|
||||
|
||||
The database uses three main tables:
|
||||
|
||||
**Profile Table:** Stores voice profiles with fields for id, name, and language.
|
||||
|
||||
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
|
||||
|
||||
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
|
||||
|
||||
## Desktop App (Tauri)
|
||||
|
||||
### Rust Backend
|
||||
|
||||
```rust
|
||||
// Sidecar process management
|
||||
// File system access
|
||||
// Native integrations
|
||||
```
|
||||
|
||||
### Responsibilities
|
||||
|
||||
- Launch Python backend as sidecar process
|
||||
- Native file dialogs
|
||||
- System tray integration
|
||||
- Auto-updates
|
||||
- OS-specific features
|
||||
|
||||
## Build Process
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Frontend (Vite dev server)
|
||||
cd app && bun run dev
|
||||
|
||||
# Backend (manual start)
|
||||
cd backend && uvicorn main:app --reload
|
||||
|
||||
# Desktop app (connects to manual backend)
|
||||
bun run dev
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build everything (server binary + Tauri app)
|
||||
bun run build
|
||||
|
||||
# Or build separately:
|
||||
# 1. Build server binary (PyInstaller)
|
||||
bun run build:server
|
||||
|
||||
# 2. Build Tauri app (includes server)
|
||||
cd tauri && bun run tauri build
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Generation Flow
|
||||
|
||||
When a user generates speech, the data flows through the following stages:
|
||||
|
||||
1. **User Input** - User enters text in a React component
|
||||
2. **State Update** - Text is stored in Zustand state
|
||||
3. **API Request** - React Query mutation triggers an API call via fetch
|
||||
4. **Backend Processing** - FastAPI endpoint receives the request
|
||||
5. **TTS Generation** - Qwen3-TTS model generates the audio
|
||||
6. **Storage** - Audio file is saved to disk and a database record is created
|
||||
7. **Response** - Backend returns the audio URL
|
||||
8. **Cache Update** - React Query updates its cache with the response
|
||||
9. **UI Update** - Component re-renders with new data
|
||||
10. **Playback** - User can play the generated audio
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Code splitting** - Lazy load routes
|
||||
- **Memoization** - React.memo for heavy components
|
||||
- **Virtual scrolling** - For large lists
|
||||
- **Debouncing** - Search and input handling
|
||||
|
||||
### Backend
|
||||
|
||||
- **Async operations** - All I/O is async
|
||||
- **Model caching** - Keep TTS model in memory
|
||||
- **Voice prompt caching** - Reuse embeddings
|
||||
- **Connection pooling** - Database connections
|
||||
|
||||
## Security
|
||||
|
||||
### Current
|
||||
|
||||
- Local-only by default
|
||||
- No authentication (localhost trust)
|
||||
- File system sandboxing via Tauri
|
||||
|
||||
### Planned
|
||||
|
||||
- API key authentication
|
||||
- User accounts
|
||||
- Rate limiting
|
||||
- HTTPS support
|
||||
|
||||
## Deployment Modes
|
||||
|
||||
### Local Mode
|
||||
|
||||
- Backend runs as sidecar
|
||||
- All data stays on device
|
||||
- No network required
|
||||
|
||||
### Remote Mode
|
||||
|
||||
- Backend on separate machine
|
||||
- Frontend connects via HTTP
|
||||
- Shared infrastructure possible
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Development Setup" icon="code" href="/development/setup">
|
||||
Set up your dev environment
|
||||
</Card>
|
||||
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
|
||||
Contribute to Voicebox
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,310 @@
|
||||
---
|
||||
title: "Audio Channels"
|
||||
description: "How audio output routing works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Audio channels allow routing voice output to different audio devices. This is useful for multi-output setups where different voices should play through different speakers or applications.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Channel:** A named audio bus that can be assigned to output devices.
|
||||
|
||||
**Device Mapping:** Links channels to OS audio device identifiers.
|
||||
|
||||
**Profile Mapping:** Links voice profiles to channels (many-to-many).
|
||||
|
||||
## Data Model
|
||||
|
||||
### AudioChannel Table
|
||||
|
||||
```python
|
||||
class AudioChannel(Base):
|
||||
__tablename__ = "audio_channels"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
name = Column(String, nullable=False)
|
||||
is_default = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime)
|
||||
```
|
||||
|
||||
### ChannelDeviceMapping Table
|
||||
|
||||
```python
|
||||
class ChannelDeviceMapping(Base):
|
||||
__tablename__ = "channel_device_mappings"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"))
|
||||
device_id = Column(String) # OS device identifier
|
||||
```
|
||||
|
||||
### ProfileChannelMapping Table
|
||||
|
||||
```python
|
||||
class ProfileChannelMapping(Base):
|
||||
__tablename__ = "profile_channel_mappings"
|
||||
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
|
||||
```
|
||||
|
||||
## Default Channel
|
||||
|
||||
A default channel is created on database initialization:
|
||||
|
||||
```python
|
||||
def init_db():
|
||||
# Create default channel if it doesn't exist
|
||||
default_channel = db.query(AudioChannel).filter(
|
||||
AudioChannel.is_default == True
|
||||
).first()
|
||||
|
||||
if not default_channel:
|
||||
default_channel = AudioChannel(
|
||||
id=str(uuid.uuid4()),
|
||||
name="Default",
|
||||
is_default=True
|
||||
)
|
||||
db.add(default_channel)
|
||||
|
||||
# Assign all existing profiles to default channel
|
||||
profiles = db.query(VoiceProfile).all()
|
||||
for profile in profiles:
|
||||
mapping = ProfileChannelMapping(
|
||||
profile_id=profile.id,
|
||||
channel_id=default_channel.id
|
||||
)
|
||||
db.add(mapping)
|
||||
```
|
||||
|
||||
## Core Operations
|
||||
|
||||
### Creating a Channel
|
||||
|
||||
```python
|
||||
async def create_channel(
|
||||
data: AudioChannelCreate,
|
||||
db: Session,
|
||||
) -> AudioChannelResponse:
|
||||
# Check name uniqueness
|
||||
existing = db.query(DBAudioChannel).filter_by(name=data.name).first()
|
||||
if existing:
|
||||
raise ValueError(f"Channel with name '{data.name}' already exists")
|
||||
|
||||
# Create channel
|
||||
channel = DBAudioChannel(
|
||||
id=str(uuid.uuid4()),
|
||||
name=data.name,
|
||||
is_default=False,
|
||||
)
|
||||
db.add(channel)
|
||||
|
||||
# Add device mappings
|
||||
for device_id in data.device_ids:
|
||||
mapping = DBChannelDeviceMapping(
|
||||
id=str(uuid.uuid4()),
|
||||
channel_id=channel.id,
|
||||
device_id=device_id,
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Updating a Channel
|
||||
|
||||
```python
|
||||
async def update_channel(
|
||||
channel_id: str,
|
||||
data: AudioChannelUpdate,
|
||||
db: Session,
|
||||
) -> AudioChannelResponse:
|
||||
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||
|
||||
# Cannot modify default channel
|
||||
if channel.is_default:
|
||||
raise ValueError("Cannot modify the default channel")
|
||||
|
||||
# Update name
|
||||
if data.name is not None:
|
||||
channel.name = data.name
|
||||
|
||||
# Update device mappings
|
||||
if data.device_ids is not None:
|
||||
# Delete existing
|
||||
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
|
||||
|
||||
# Add new
|
||||
for device_id in data.device_ids:
|
||||
mapping = DBChannelDeviceMapping(
|
||||
channel_id=channel.id,
|
||||
device_id=device_id,
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Deleting a Channel
|
||||
|
||||
```python
|
||||
async def delete_channel(channel_id: str, db: Session) -> bool:
|
||||
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||
|
||||
# Cannot delete default channel
|
||||
if channel.is_default:
|
||||
raise ValueError("Cannot delete the default channel")
|
||||
|
||||
# Delete device mappings
|
||||
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
|
||||
|
||||
# Delete profile-channel mappings
|
||||
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
|
||||
|
||||
# Delete channel
|
||||
db.delete(channel)
|
||||
db.commit()
|
||||
```
|
||||
|
||||
## Voice Assignment
|
||||
|
||||
### Assigning Voices to Channel
|
||||
|
||||
```python
|
||||
async def set_channel_voices(
|
||||
channel_id: str,
|
||||
data: ChannelVoiceAssignment,
|
||||
db: Session,
|
||||
) -> None:
|
||||
# Verify channel exists
|
||||
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||
if not channel:
|
||||
raise ValueError(f"Channel {channel_id} not found")
|
||||
|
||||
# Verify all profiles exist
|
||||
for profile_id in data.profile_ids:
|
||||
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||
if not profile:
|
||||
raise ValueError(f"Profile {profile_id} not found")
|
||||
|
||||
# Delete existing mappings
|
||||
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
|
||||
|
||||
# Add new mappings
|
||||
for profile_id in data.profile_ids:
|
||||
mapping = DBProfileChannelMapping(
|
||||
profile_id=profile_id,
|
||||
channel_id=channel_id,
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Assigning Channels to Voice
|
||||
|
||||
```python
|
||||
async def set_profile_channels(
|
||||
profile_id: str,
|
||||
data: ProfileChannelAssignment,
|
||||
db: Session,
|
||||
) -> None:
|
||||
# Verify profile exists
|
||||
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||
if not profile:
|
||||
raise ValueError(f"Profile {profile_id} not found")
|
||||
|
||||
# Delete existing mappings
|
||||
db.query(DBProfileChannelMapping).filter_by(profile_id=profile_id).delete()
|
||||
|
||||
# Add new mappings
|
||||
for channel_id in data.channel_ids:
|
||||
mapping = DBProfileChannelMapping(
|
||||
profile_id=profile_id,
|
||||
channel_id=channel_id,
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/channels` | List all channels |
|
||||
| POST | `/channels` | Create a channel |
|
||||
| GET | `/channels/{id}` | Get channel by ID |
|
||||
| PUT | `/channels/{id}` | Update channel |
|
||||
| DELETE | `/channels/{id}` | Delete channel |
|
||||
| GET | `/channels/{id}/voices` | Get assigned voices |
|
||||
| PUT | `/channels/{id}/voices` | Set assigned voices |
|
||||
| GET | `/profiles/{id}/channels` | Get profile's channels |
|
||||
| PUT | `/profiles/{id}/channels` | Set profile's channels |
|
||||
|
||||
## Request/Response Schemas
|
||||
|
||||
### AudioChannelCreate
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Speakers",
|
||||
"device_ids": ["device_uuid_1", "device_uuid_2"]
|
||||
}
|
||||
```
|
||||
|
||||
### AudioChannelResponse
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "channel_uuid",
|
||||
"name": "Speakers",
|
||||
"is_default": false,
|
||||
"device_ids": ["device_uuid_1", "device_uuid_2"],
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### ChannelVoiceAssignment
|
||||
|
||||
```json
|
||||
{
|
||||
"profile_ids": ["profile_1", "profile_2"]
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Multi-Output Setup
|
||||
|
||||
**Scenario:** Stream with different voice characters
|
||||
|
||||
1. Create "Stream" channel → OBS virtual audio
|
||||
2. Create "Monitor" channel → Headphones
|
||||
3. Assign "Narrator" profile → Both channels
|
||||
4. Assign "Character 1" profile → Stream only
|
||||
|
||||
### Virtual Audio Cables
|
||||
|
||||
Common device IDs for virtual audio:
|
||||
- VB-Audio Virtual Cable
|
||||
- BlackHole (macOS)
|
||||
- Soundflower (macOS)
|
||||
|
||||
## Frontend Integration
|
||||
|
||||
The frontend needs to:
|
||||
|
||||
1. **Enumerate devices** using Web Audio API or Tauri
|
||||
2. **Display channel list** with device assignments
|
||||
3. **Allow profile assignment** via drag/drop or dropdown
|
||||
4. **Route playback** to correct device based on profile's channel
|
||||
|
||||
## Limitations
|
||||
|
||||
- Device IDs are OS-specific
|
||||
- Hot-plugging may invalidate device IDs
|
||||
- Default channel cannot be modified/deleted
|
||||
- Frontend handles actual audio routing (backend just stores config)
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: "Auto-Updater"
|
||||
description: "Configure and use the Tauri auto-updater"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses Tauri's built-in auto-updater to deliver updates to users automatically.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
For detailed setup instructions, see the existing documentation:
|
||||
|
||||
- [AUTOUPDATER_QUICKSTART.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER_QUICKSTART.md)
|
||||
- [AUTOUPDATER.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER.md)
|
||||
|
||||
## How It Works
|
||||
|
||||
The auto-updater follows a secure update process:
|
||||
|
||||
1. **Check for Updates** - The Voicebox app periodically checks GitHub Releases for new versions
|
||||
2. **Download Update** - If a new version is found, the update package is downloaded
|
||||
3. **Verify Signature** - The downloaded package is cryptographically verified using the public key
|
||||
4. **Install** - After verification, the update is installed
|
||||
5. **Restart** - The app restarts with the new version
|
||||
|
||||
## Configuration
|
||||
|
||||
Updates are configured in `tauri/src-tauri/tauri.conf.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"updater": {
|
||||
"active": true,
|
||||
"endpoints": [
|
||||
"https://github.com/jamiepine/voicebox/releases/latest/download/latest.json"
|
||||
],
|
||||
"dialog": true,
|
||||
"pubkey": "YOUR_PUBLIC_KEY"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Generating Keys
|
||||
|
||||
```bash
|
||||
# Generate signing keys
|
||||
bun run generate:keys
|
||||
|
||||
# Keys saved to ~/.tauri/voicebox.key
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Keep your private key secure! Never commit it to the repository.
|
||||
</Warning>
|
||||
|
||||
## Release Process
|
||||
|
||||
1. **Bump version** using bumpversion
|
||||
2. **Push tag** to trigger CI/CD
|
||||
3. **GitHub Actions** builds and signs releases
|
||||
4. **Users** receive update notification
|
||||
|
||||
## User Experience
|
||||
|
||||
When an update is available:
|
||||
|
||||
1. User sees a notification dialog
|
||||
2. User clicks "Update"
|
||||
3. Update downloads in background
|
||||
4. App restarts with new version
|
||||
|
||||
## For Developers
|
||||
|
||||
See the full documentation files for:
|
||||
|
||||
- Setting up signing keys
|
||||
- Configuring GitHub releases
|
||||
- Testing updates locally
|
||||
- Troubleshooting update failures
|
||||
|
||||
<Card title="View Full Docs" href="https://github.com/jamiepine/voicebox/tree/main/docs">
|
||||
Access AUTOUPDATER.md and AUTOUPDATER_QUICKSTART.md in the repository
|
||||
</Card>
|
||||
@@ -0,0 +1,270 @@
|
||||
---
|
||||
title: "Building"
|
||||
description: "Build Voicebox for production"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses a multi-step build process to create platform-specific installers.
|
||||
|
||||
## Quick Build
|
||||
|
||||
```bash
|
||||
# Build for your current platform (automatically builds server binary first)
|
||||
make build
|
||||
|
||||
# Or manually
|
||||
bun run build
|
||||
```
|
||||
|
||||
This automatically:
|
||||
1. Builds the Python server binary (`bun run build:server`)
|
||||
2. Builds the Tauri app (`cd tauri && bun run tauri build`)
|
||||
|
||||
## Build Process
|
||||
|
||||
The build process consists of two steps, but `bun run build` handles both automatically:
|
||||
|
||||
### 1. Server Binary Build (Automatic)
|
||||
|
||||
The Python backend is compiled into a standalone executable using PyInstaller. This happens automatically when you run `bun run build`.
|
||||
|
||||
**Platform-specific binaries:**
|
||||
- macOS (Apple Silicon): `voicebox-server-aarch64-apple-darwin` (includes MLX backend)
|
||||
- macOS (Intel): `voicebox-server-x86_64-apple-darwin` (PyTorch backend)
|
||||
- Windows: `voicebox-server-x86_64-pc-windows-msvc.exe` (PyTorch backend)
|
||||
- Linux: `voicebox-server-x86_64-unknown-linux-gnu` (PyTorch backend)
|
||||
|
||||
<Note>
|
||||
The build script automatically detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
|
||||
</Note>
|
||||
|
||||
**Manual build (if needed):**
|
||||
```bash
|
||||
bun run build:server
|
||||
```
|
||||
|
||||
### 2. Tauri App Build (Automatic)
|
||||
|
||||
The Tauri app build is also handled automatically, which:
|
||||
1. Builds the React frontend (Vite)
|
||||
2. Compiles the Rust backend
|
||||
3. Bundles the server binary as a sidecar
|
||||
4. Creates platform-specific installers
|
||||
|
||||
**Manual build (if needed):**
|
||||
```bash
|
||||
cd tauri && bun run tauri build
|
||||
```
|
||||
|
||||
### 3. Output
|
||||
|
||||
Installers are created in `tauri/src-tauri/target/release/bundle/`:
|
||||
|
||||
**macOS:**
|
||||
- `dmg/` - Disk image installer
|
||||
- `macos/` - App bundle
|
||||
|
||||
**Windows:**
|
||||
- `msi/` - MSI installer
|
||||
- `nsis/` - NSIS installer
|
||||
|
||||
**Linux:**
|
||||
- `deb/` - Debian package
|
||||
- `appimage/` - AppImage
|
||||
|
||||
## Advanced Options
|
||||
|
||||
### Building for Specific Platform
|
||||
|
||||
```bash
|
||||
# Build for macOS (Apple Silicon)
|
||||
bun run tauri build -- --target aarch64-apple-darwin
|
||||
|
||||
# Build for macOS (Intel)
|
||||
bun run tauri build -- --target x86_64-apple-darwin
|
||||
|
||||
# Build for Windows
|
||||
bun run tauri build -- --target x86_64-pc-windows-msvc
|
||||
|
||||
# Build for Linux
|
||||
bun run tauri build -- --target x86_64-unknown-linux-gnu
|
||||
```
|
||||
|
||||
### Using Local Qwen3-TTS
|
||||
|
||||
If you're developing Qwen3-TTS locally:
|
||||
|
||||
```bash
|
||||
export QWEN_TTS_PATH=~/path/to/Qwen3-TTS
|
||||
bun run build:server # Build server binary only
|
||||
# or
|
||||
bun run build # Build everything
|
||||
```
|
||||
|
||||
This makes PyInstaller use your local version instead of the pip package.
|
||||
|
||||
### Debug Build
|
||||
|
||||
```bash
|
||||
cd tauri
|
||||
bun run tauri build --debug
|
||||
```
|
||||
|
||||
Creates a debug build with symbols and logging.
|
||||
|
||||
## Build Configuration
|
||||
|
||||
### Tauri Config
|
||||
|
||||
Edit `tauri/src-tauri/tauri.conf.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle": {
|
||||
"identifier": "com.voicebox.app",
|
||||
"icon": [
|
||||
"icons/32x32.png",
|
||||
"icons/128x128.png",
|
||||
"icons/icon.icns",
|
||||
"icons/icon.ico"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Sidecar Configuration
|
||||
|
||||
The Python server is bundled as a sidecar:
|
||||
|
||||
```json
|
||||
{
|
||||
"tauri": {
|
||||
"bundle": {
|
||||
"externalBin": [
|
||||
"binaries/voicebox-server"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Code Signing
|
||||
|
||||
### macOS
|
||||
|
||||
To sign the app for distribution:
|
||||
|
||||
```bash
|
||||
# Set signing identity
|
||||
export APPLE_SIGNING_IDENTITY="Developer ID Application: Your Name"
|
||||
|
||||
# Build with signing
|
||||
bun run tauri build
|
||||
```
|
||||
|
||||
For notarization:
|
||||
|
||||
```bash
|
||||
# Set credentials
|
||||
export APPLE_ID="[email protected]"
|
||||
export APPLE_PASSWORD="app-specific-password"
|
||||
|
||||
# Build and notarize
|
||||
bun run tauri build
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
For Windows code signing:
|
||||
|
||||
```bash
|
||||
# Set certificate
|
||||
export WINDOWS_CERTIFICATE_PATH="/path/to/cert.pfx"
|
||||
export WINDOWS_CERTIFICATE_PASSWORD="password"
|
||||
|
||||
# Build with signing
|
||||
bun run tauri build
|
||||
```
|
||||
|
||||
## Release Process
|
||||
|
||||
The full release process is automated:
|
||||
|
||||
```bash
|
||||
# 1. Bump version
|
||||
bumpversion patch # or minor/major
|
||||
|
||||
# 2. Build all platforms (CI/CD handles this)
|
||||
git push --tags
|
||||
|
||||
# 3. GitHub Actions creates releases
|
||||
```
|
||||
|
||||
See [CONTRIBUTING.md](/development/contributing) for the full release workflow.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Server Binary Build Fails">
|
||||
**Common issues:**
|
||||
- Missing Python dependencies: `pip install -r requirements.txt`
|
||||
- PyInstaller not found: `pip install pyinstaller`
|
||||
- Qwen3-TTS not installed: `pip install git+https://github.com/QwenLM/Qwen3-TTS.git`
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
cd backend
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
pip install pyinstaller
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Tauri Build Fails">
|
||||
**Common issues:**
|
||||
- Rust not installed: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
|
||||
- Server binary missing: Usually auto-built, but can run manually: `./scripts/build-server.sh`
|
||||
- Node modules outdated: `bun install`
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Clean and rebuild
|
||||
cd tauri/src-tauri
|
||||
cargo clean
|
||||
cd ../..
|
||||
bun run build # Automatically builds server binary first
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="App Won't Launch After Build">
|
||||
**Check:**
|
||||
- Server binary has execute permissions
|
||||
- All dependencies are bundled
|
||||
- Check logs in the app's data directory
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
```bash
|
||||
type %APPDATA%\com.voicebox.app\logs\server.log
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## CI/CD
|
||||
|
||||
GitHub Actions automatically builds releases when tags are pushed:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/release.yml
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
```
|
||||
|
||||
See the [repository](https://github.com/jamiepine/voicebox) for the full CI/CD configuration.
|
||||
@@ -0,0 +1,326 @@
|
||||
---
|
||||
title: "Contributing"
|
||||
description: "How to contribute to Voicebox"
|
||||
---
|
||||
|
||||
Thank you for your interest in contributing to Voicebox! This guide will help you get started.
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
- Be respectful and inclusive
|
||||
- Welcome newcomers and help them learn
|
||||
- Focus on constructive feedback
|
||||
- Respect different viewpoints and experiences
|
||||
|
||||
## Getting Started
|
||||
|
||||
Before you start contributing, make sure you have:
|
||||
|
||||
1. **Read the documentation** to understand how Voicebox works
|
||||
2. **Set up your development environment** - see [Development Setup](/development/setup)
|
||||
3. **Explored the codebase** to understand the project structure
|
||||
4. **Checked existing issues** to see if someone else is working on something similar
|
||||
|
||||
## Ways to Contribute
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Report Bugs" icon="bug">
|
||||
Found a bug? Open an issue with reproduction steps
|
||||
</Card>
|
||||
<Card title="Request Features" icon="lightbulb">
|
||||
Have an idea? Start a discussion or open an issue
|
||||
</Card>
|
||||
<Card title="Improve Docs" icon="book">
|
||||
Fix typos, add examples, or clarify instructions
|
||||
</Card>
|
||||
<Card title="Write Code" icon="code">
|
||||
Fix bugs, add features, or optimize performance
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### 1. Fork & Clone
|
||||
|
||||
```bash
|
||||
# Fork the repository on GitHub
|
||||
# Then clone your fork
|
||||
git clone https://github.com/YOUR_USERNAME/voicebox.git
|
||||
cd voicebox
|
||||
```
|
||||
|
||||
### 2. Create a Branch
|
||||
|
||||
Use descriptive branch names:
|
||||
|
||||
```bash
|
||||
# For features
|
||||
git checkout -b feature/voice-effects
|
||||
|
||||
# For bug fixes
|
||||
git checkout -b fix/audio-playback-issue
|
||||
|
||||
# For documentation
|
||||
git checkout -b docs/api-examples
|
||||
```
|
||||
|
||||
### 3. Make Your Changes
|
||||
|
||||
Follow these guidelines:
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Code Style">
|
||||
**TypeScript/React:**
|
||||
- Use TypeScript strict mode
|
||||
- Prefer functional components with hooks
|
||||
- Use named exports
|
||||
- Format with Biome (runs automatically)
|
||||
|
||||
**Python:**
|
||||
- Follow PEP 8
|
||||
- Use type hints
|
||||
- Use async/await for I/O
|
||||
- Document functions with docstrings
|
||||
|
||||
**Rust:**
|
||||
- Follow Rust conventions
|
||||
- Use meaningful names
|
||||
- Handle errors explicitly
|
||||
- Run `rustfmt`
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Commit Messages">
|
||||
Write clear, descriptive commit messages:
|
||||
|
||||
```bash
|
||||
# Good
|
||||
git commit -m "Add voice profile export feature"
|
||||
git commit -m "Fix audio playback stopping after 30 seconds"
|
||||
|
||||
# Avoid
|
||||
git commit -m "Update code"
|
||||
git commit -m "Fix bug"
|
||||
```
|
||||
|
||||
Format:
|
||||
- Use imperative mood ("Add feature" not "Added feature")
|
||||
- Keep first line under 50 characters
|
||||
- Add detailed description if needed
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Testing">
|
||||
- Test your changes manually in the app
|
||||
- Ensure backend API endpoints work
|
||||
- Check for TypeScript/Python errors
|
||||
- Verify UI components render correctly
|
||||
- Add automated tests when possible
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### 4. Push & Create PR
|
||||
|
||||
```bash
|
||||
# Push your branch
|
||||
git push origin feature/your-feature-name
|
||||
|
||||
# Then create a pull request on GitHub
|
||||
```
|
||||
|
||||
## Pull Request Guidelines
|
||||
|
||||
When creating a pull request:
|
||||
|
||||
<Steps>
|
||||
<Step title="Use a Clear Title">
|
||||
Examples:
|
||||
- "Add voice profile export functionality"
|
||||
- "Fix audio playback stopping after 30 seconds"
|
||||
- "Improve generation speed with caching"
|
||||
</Step>
|
||||
|
||||
<Step title="Provide Description">
|
||||
Include:
|
||||
- What changes you made
|
||||
- Why you made them
|
||||
- How to test them
|
||||
- Screenshots (for UI changes)
|
||||
- Reference related issues
|
||||
</Step>
|
||||
|
||||
<Step title="Update Documentation">
|
||||
- Update relevant docs if behavior changes
|
||||
- Add API documentation for new endpoints
|
||||
- Update README if needed
|
||||
</Step>
|
||||
|
||||
<Step title="Check the Checklist">
|
||||
- [ ] Code follows style guidelines
|
||||
- [ ] Documentation updated
|
||||
- [ ] Changes tested
|
||||
- [ ] No breaking changes (or documented)
|
||||
- [ ] CHANGELOG.md updated
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Project Structure
|
||||
|
||||
Understanding the codebase:
|
||||
|
||||
```
|
||||
voicebox/
|
||||
├── app/ # Shared React frontend
|
||||
│ ├── src/
|
||||
│ │ ├── components/ # UI components
|
||||
│ │ ├── lib/ # Utilities and API client
|
||||
│ │ ├── hooks/ # React hooks
|
||||
│ │ └── stores/ # Zustand state stores
|
||||
├── backend/ # Python FastAPI server
|
||||
│ ├── main.py # API routes
|
||||
│ ├── tts.py # Voice synthesis logic
|
||||
│ ├── database.py # SQLite operations
|
||||
│ └── models.py # Pydantic models
|
||||
├── tauri/ # Desktop app wrapper
|
||||
│ └── src-tauri/ # Rust backend
|
||||
├── web/ # Web deployment
|
||||
├── landing/ # Marketing website
|
||||
└── scripts/ # Build & release scripts
|
||||
```
|
||||
|
||||
## Areas for Contribution
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Check [existing issues](https://github.com/jamiepine/voicebox/issues) for bugs
|
||||
- Test your fix thoroughly
|
||||
- Add regression tests if possible
|
||||
|
||||
### New Features
|
||||
|
||||
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
|
||||
- Discuss major features in an issue first
|
||||
- Keep features focused and well-scoped
|
||||
|
||||
### Documentation
|
||||
|
||||
- Improve clarity and fix typos
|
||||
- Add code examples
|
||||
- Create tutorials or guides
|
||||
- Document API endpoints
|
||||
|
||||
### UI/UX Improvements
|
||||
|
||||
- Improve accessibility
|
||||
- Enhance visual design
|
||||
- Optimize performance
|
||||
- Add animations/transitions
|
||||
|
||||
### Infrastructure
|
||||
|
||||
- Improve build process
|
||||
- Add CI/CD improvements
|
||||
- Optimize bundle size
|
||||
- Add testing infrastructure
|
||||
|
||||
## API Development
|
||||
|
||||
When adding new API endpoints:
|
||||
|
||||
<Steps>
|
||||
<Step title="Add Route">
|
||||
In `backend/main.py`:
|
||||
|
||||
```python
|
||||
@app.post("/api/new-endpoint")
|
||||
async def new_endpoint(data: RequestModel) -> ResponseModel:
|
||||
"""Endpoint description."""
|
||||
# Implementation
|
||||
return response
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Create Models">
|
||||
In `backend/models.py`:
|
||||
|
||||
```python
|
||||
class RequestModel(BaseModel):
|
||||
field: str
|
||||
|
||||
class ResponseModel(BaseModel):
|
||||
result: str
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Regenerate Client">
|
||||
```bash
|
||||
bun run generate:api
|
||||
```
|
||||
|
||||
This updates the TypeScript client with type-safe bindings.
|
||||
</Step>
|
||||
|
||||
<Step title="Update Docs">
|
||||
Add documentation in `/docs/api/`
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Testing
|
||||
|
||||
Currently testing is primarily manual. When adding tests:
|
||||
|
||||
**Backend:**
|
||||
```bash
|
||||
cd backend
|
||||
pytest
|
||||
```
|
||||
|
||||
**Frontend:**
|
||||
```bash
|
||||
bun run test
|
||||
```
|
||||
|
||||
**E2E (future):**
|
||||
```bash
|
||||
bun run test:e2e
|
||||
```
|
||||
|
||||
## Release Process
|
||||
|
||||
Releases are managed by maintainers using `bumpversion`:
|
||||
|
||||
```bash
|
||||
# Bump version (patch, minor, or major)
|
||||
bumpversion patch
|
||||
|
||||
# Push with tags
|
||||
git push && git push --tags
|
||||
```
|
||||
|
||||
GitHub Actions automatically builds and publishes releases when tags are pushed.
|
||||
|
||||
## Community
|
||||
|
||||
- **GitHub Issues:** Bug reports and feature requests
|
||||
- **GitHub Discussions:** General questions and ideas
|
||||
- **Discord:** Real-time chat (coming soon)
|
||||
|
||||
## Recognition
|
||||
|
||||
Contributors are recognized in:
|
||||
- [CHANGELOG.md](https://github.com/jamiepine/voicebox/blob/main/CHANGELOG.md)
|
||||
- GitHub contributor list
|
||||
- Release notes
|
||||
|
||||
## License
|
||||
|
||||
By contributing, you agree that your contributions will be licensed under the MIT License.
|
||||
|
||||
## Questions?
|
||||
|
||||
If you have questions:
|
||||
|
||||
1. Check the [documentation](/overview/introduction)
|
||||
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
|
||||
3. Open a new issue or discussion
|
||||
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
|
||||
|
||||
Thank you for contributing to Voicebox! 🎉
|
||||
@@ -0,0 +1,260 @@
|
||||
---
|
||||
title: "Generation History"
|
||||
description: "How generation history tracking works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The history module tracks all generated audio, providing a searchable record of past generations. Each generation stores the text, settings, and a reference to the audio file.
|
||||
|
||||
## Data Model
|
||||
|
||||
### Generation Table
|
||||
|
||||
```python
|
||||
class Generation(Base):
|
||||
__tablename__ = "generations"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||
text = Column(Text, nullable=False)
|
||||
language = Column(String, default="en")
|
||||
audio_path = Column(String, nullable=False)
|
||||
duration = Column(Float, nullable=False)
|
||||
seed = Column(Integer)
|
||||
instruct = Column(Text)
|
||||
created_at = Column(DateTime)
|
||||
```
|
||||
|
||||
## File Storage
|
||||
|
||||
Generated audio is stored in:
|
||||
|
||||
```
|
||||
data/
|
||||
└── generations/
|
||||
└── {generation_id}.wav
|
||||
```
|
||||
|
||||
## Core Functions
|
||||
|
||||
### Creating a Generation Record
|
||||
|
||||
After TTS generates audio, a history entry is created:
|
||||
|
||||
```python
|
||||
async def create_generation(
|
||||
profile_id: str,
|
||||
text: str,
|
||||
language: str,
|
||||
audio_path: str,
|
||||
duration: float,
|
||||
seed: Optional[int],
|
||||
db: Session,
|
||||
instruct: Optional[str] = None,
|
||||
) -> GenerationResponse:
|
||||
db_generation = DBGeneration(
|
||||
id=str(uuid.uuid4()),
|
||||
profile_id=profile_id,
|
||||
text=text,
|
||||
language=language,
|
||||
audio_path=audio_path,
|
||||
duration=duration,
|
||||
seed=seed,
|
||||
instruct=instruct,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
db.add(db_generation)
|
||||
db.commit()
|
||||
|
||||
return GenerationResponse.model_validate(db_generation)
|
||||
```
|
||||
|
||||
### Listing Generations
|
||||
|
||||
Supports filtering and pagination:
|
||||
|
||||
```python
|
||||
async def list_generations(
|
||||
query: HistoryQuery,
|
||||
db: Session,
|
||||
) -> HistoryListResponse:
|
||||
# Build query with profile name join
|
||||
q = db.query(
|
||||
DBGeneration,
|
||||
DBVoiceProfile.name.label('profile_name')
|
||||
).join(
|
||||
DBVoiceProfile,
|
||||
DBGeneration.profile_id == DBVoiceProfile.id
|
||||
)
|
||||
|
||||
# Apply filters
|
||||
if query.profile_id:
|
||||
q = q.filter(DBGeneration.profile_id == query.profile_id)
|
||||
|
||||
if query.search:
|
||||
q = q.filter(DBGeneration.text.like(f"%{query.search}%"))
|
||||
|
||||
# Order and paginate
|
||||
total = q.count()
|
||||
q = q.order_by(DBGeneration.created_at.desc())
|
||||
q = q.offset(query.offset).limit(query.limit)
|
||||
|
||||
return HistoryListResponse(items=results, total=total)
|
||||
```
|
||||
|
||||
### Getting Statistics
|
||||
|
||||
Aggregate statistics for the dashboard:
|
||||
|
||||
```python
|
||||
async def get_generation_stats(db: Session) -> dict:
|
||||
total = db.query(func.count(DBGeneration.id)).scalar()
|
||||
total_duration = db.query(func.sum(DBGeneration.duration)).scalar()
|
||||
|
||||
by_profile = db.query(
|
||||
DBGeneration.profile_id,
|
||||
func.count(DBGeneration.id).label('count')
|
||||
).group_by(DBGeneration.profile_id).all()
|
||||
|
||||
return {
|
||||
"total_generations": total,
|
||||
"total_duration_seconds": total_duration,
|
||||
"generations_by_profile": {
|
||||
profile_id: count for profile_id, count in by_profile
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Deletion
|
||||
|
||||
Deleting a generation removes both the database record and audio file:
|
||||
|
||||
```python
|
||||
async def delete_generation(generation_id: str, db: Session) -> bool:
|
||||
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
|
||||
if not generation:
|
||||
return False
|
||||
|
||||
# Delete audio file
|
||||
audio_path = Path(generation.audio_path)
|
||||
if audio_path.exists():
|
||||
audio_path.unlink()
|
||||
|
||||
# Delete database record
|
||||
db.delete(generation)
|
||||
db.commit()
|
||||
|
||||
return True
|
||||
```
|
||||
|
||||
### Cascade Delete
|
||||
|
||||
When deleting a profile, all its generations are also deleted:
|
||||
|
||||
```python
|
||||
async def delete_generations_by_profile(profile_id: str, db: Session) -> int:
|
||||
generations = db.query(DBGeneration).filter_by(profile_id=profile_id).all()
|
||||
|
||||
for generation in generations:
|
||||
Path(generation.audio_path).unlink(missing_ok=True)
|
||||
db.delete(generation)
|
||||
|
||||
db.commit()
|
||||
return len(generations)
|
||||
```
|
||||
|
||||
## Export/Import
|
||||
|
||||
### Exporting a Generation
|
||||
|
||||
Generations can be exported as ZIP archives:
|
||||
|
||||
```
|
||||
generation_export.zip
|
||||
├── generation.json # Metadata
|
||||
└── audio.wav # Audio file
|
||||
```
|
||||
|
||||
### Importing a Generation
|
||||
|
||||
The import process:
|
||||
|
||||
1. Extract ZIP archive
|
||||
2. Validate metadata and audio
|
||||
3. Create new generation ID
|
||||
4. Copy audio to generations directory
|
||||
5. Create database record
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/history` | List generations with filters |
|
||||
| GET | `/history/stats` | Get aggregate statistics |
|
||||
| GET | `/history/{id}` | Get generation by ID |
|
||||
| DELETE | `/history/{id}` | Delete generation |
|
||||
| GET | `/history/{id}/export` | Export as ZIP |
|
||||
| GET | `/history/{id}/export-audio` | Export audio only |
|
||||
| POST | `/history/import` | Import from ZIP |
|
||||
|
||||
### Query Parameters
|
||||
|
||||
```
|
||||
GET /history?profile_id=uuid&search=hello&limit=50&offset=0
|
||||
```
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `profile_id` | string | null | Filter by profile |
|
||||
| `search` | string | null | Search in text |
|
||||
| `limit` | int | 50 | Results per page |
|
||||
| `offset` | int | 0 | Pagination offset |
|
||||
|
||||
### Response Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "uuid",
|
||||
"profile_id": "uuid",
|
||||
"profile_name": "My Voice",
|
||||
"text": "Hello world",
|
||||
"language": "en",
|
||||
"audio_path": "/path/to/audio.wav",
|
||||
"duration": 1.5,
|
||||
"seed": 42,
|
||||
"instruct": null,
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"total": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Usage in Stories
|
||||
|
||||
Generations can be added to stories for multi-voice narratives. The story system references generations by ID:
|
||||
|
||||
```python
|
||||
class StoryItem(Base):
|
||||
generation_id = Column(String, ForeignKey("generations.id"))
|
||||
```
|
||||
|
||||
This allows the same generation to be reused across multiple stories without duplicating audio files.
|
||||
|
||||
## Storage Considerations
|
||||
|
||||
### Disk Usage
|
||||
|
||||
Each generation creates a WAV file. For a 10-second clip at 24kHz:
|
||||
- ~480KB per file (mono, 16-bit)
|
||||
|
||||
### Cleanup Strategy
|
||||
|
||||
Consider implementing:
|
||||
- Automatic cleanup of old generations
|
||||
- Storage quota per profile
|
||||
- Compression for archival
|
||||
@@ -0,0 +1,341 @@
|
||||
---
|
||||
title: "Model Management"
|
||||
description: "How model downloading, loading, and status tracking works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox manages two types of models:
|
||||
|
||||
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
|
||||
|
||||
**ASR Models:** Whisper for transcription (tiny through large).
|
||||
|
||||
Models are downloaded from HuggingFace Hub on first use and cached locally.
|
||||
|
||||
## Available Models
|
||||
|
||||
### TTS Models
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
|
||||
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
|
||||
|
||||
### Whisper Models
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
|
||||
| base | `openai/whisper-base` | ~300MB | ~1GB |
|
||||
| small | `openai/whisper-small` | ~500MB | ~2GB |
|
||||
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
|
||||
| large | `openai/whisper-large` | ~3GB | ~10GB |
|
||||
|
||||
## Model Storage
|
||||
|
||||
Models are cached in the HuggingFace cache directory:
|
||||
|
||||
```
|
||||
~/.cache/huggingface/hub/
|
||||
├── models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/
|
||||
├── models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/
|
||||
├── models--openai--whisper-base/
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
### Progress Manager
|
||||
|
||||
Tracks download progress across all models:
|
||||
|
||||
```python
|
||||
class ProgressManager:
|
||||
def __init__(self):
|
||||
self._progress = {} # model_name -> progress_info
|
||||
|
||||
def update_progress(
|
||||
self,
|
||||
model_name: str,
|
||||
current: int,
|
||||
total: int,
|
||||
filename: str,
|
||||
status: str,
|
||||
):
|
||||
self._progress[model_name] = {
|
||||
"current": current,
|
||||
"total": total,
|
||||
"filename": filename,
|
||||
"status": status, # downloading, complete, error
|
||||
"updated_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def get_progress(self, model_name: str) -> Optional[dict]:
|
||||
return self._progress.get(model_name)
|
||||
```
|
||||
|
||||
### HuggingFace Progress Callback
|
||||
|
||||
Hooks into HuggingFace's download system:
|
||||
|
||||
```python
|
||||
class HFProgressTracker:
|
||||
def __init__(self, callback):
|
||||
self.callback = callback
|
||||
|
||||
@contextmanager
|
||||
def patch_download(self):
|
||||
"""Context manager to intercept HF downloads."""
|
||||
original_download = hf_hub_download
|
||||
|
||||
def patched_download(*args, **kwargs):
|
||||
# Intercept progress
|
||||
result = original_download(*args, **kwargs)
|
||||
self.callback(progress_info)
|
||||
return result
|
||||
|
||||
# Apply patch
|
||||
with patch('huggingface_hub.hf_hub_download', patched_download):
|
||||
yield
|
||||
```
|
||||
|
||||
### Server-Sent Events (SSE)
|
||||
|
||||
Progress is streamed to the frontend:
|
||||
|
||||
```python
|
||||
@app.get("/models/progress/{model_name}")
|
||||
async def get_model_progress(model_name: str):
|
||||
async def event_generator():
|
||||
while True:
|
||||
progress = progress_manager.get_progress(model_name)
|
||||
if progress:
|
||||
yield f"data: {json.dumps(progress)}\n\n"
|
||||
|
||||
if progress and progress["status"] in ["complete", "error"]:
|
||||
break
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream"
|
||||
)
|
||||
```
|
||||
|
||||
## Task Manager
|
||||
|
||||
Tracks active downloads and generations:
|
||||
|
||||
```python
|
||||
class TaskManager:
|
||||
def __init__(self):
|
||||
self._active_downloads = {}
|
||||
self._active_generations = {}
|
||||
|
||||
def start_download(self, model_name: str):
|
||||
self._active_downloads[model_name] = {
|
||||
"status": "downloading",
|
||||
"started_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def complete_download(self, model_name: str):
|
||||
if model_name in self._active_downloads:
|
||||
del self._active_downloads[model_name]
|
||||
|
||||
def get_active_tasks(self) -> dict:
|
||||
return {
|
||||
"downloads": list(self._active_downloads.values()),
|
||||
"generations": list(self._active_generations.values()),
|
||||
}
|
||||
```
|
||||
|
||||
## Model Status
|
||||
|
||||
Check which models are downloaded and loaded:
|
||||
|
||||
```python
|
||||
@app.get("/models/status")
|
||||
async def get_model_status() -> ModelStatusListResponse:
|
||||
models = []
|
||||
|
||||
# Check TTS models
|
||||
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
loaded = tts_model._current_model_size == size
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"qwen-tts-{size}",
|
||||
display_name=f"Qwen3-TTS {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=loaded,
|
||||
))
|
||||
|
||||
# Check Whisper models
|
||||
for size in ["tiny", "base", "small", "medium", "large"]:
|
||||
hf_id = f"openai/whisper-{size}"
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"whisper-{size}",
|
||||
display_name=f"Whisper {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=False, # Whisper is loaded on-demand
|
||||
))
|
||||
|
||||
return ModelStatusListResponse(models=models)
|
||||
```
|
||||
|
||||
## Manual Model Operations
|
||||
|
||||
### Load Model
|
||||
|
||||
```python
|
||||
@app.post("/models/load")
|
||||
async def load_model(model_size: str = "1.7B"):
|
||||
tts_model = get_tts_model()
|
||||
await tts_model.load_model_async(model_size)
|
||||
return {"status": "loaded", "model_size": model_size}
|
||||
```
|
||||
|
||||
### Unload Model
|
||||
|
||||
```python
|
||||
@app.post("/models/unload")
|
||||
async def unload_model():
|
||||
tts_model = get_tts_model()
|
||||
tts_model.unload_model()
|
||||
return {"status": "unloaded"}
|
||||
```
|
||||
|
||||
### Trigger Download
|
||||
|
||||
```python
|
||||
@app.post("/models/download")
|
||||
async def trigger_model_download(request: ModelDownloadRequest):
|
||||
# This triggers the download in background
|
||||
# Progress is tracked via /models/progress/{model_name}
|
||||
|
||||
if request.model_name.startswith("qwen-tts"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_tts_model(size))
|
||||
elif request.model_name.startswith("whisper"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_whisper_model(size))
|
||||
|
||||
return {"status": "downloading"}
|
||||
```
|
||||
|
||||
### Delete Model
|
||||
|
||||
```python
|
||||
@app.delete("/models/{model_name}")
|
||||
async def delete_model(model_name: str):
|
||||
# Find and delete from HuggingFace cache
|
||||
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
|
||||
|
||||
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
|
||||
for model_dir in model_dirs:
|
||||
shutil.rmtree(model_dir)
|
||||
|
||||
return {"status": "deleted"}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/models/status` | Get status of all models |
|
||||
| POST | `/models/load` | Load TTS model |
|
||||
| POST | `/models/unload` | Unload TTS model |
|
||||
| POST | `/models/download` | Trigger model download |
|
||||
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||
| DELETE | `/models/{name}` | Delete downloaded model |
|
||||
| GET | `/tasks/active` | Get active downloads/generations |
|
||||
|
||||
## Response Schemas
|
||||
|
||||
### ModelStatus
|
||||
|
||||
```json
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen3-TTS 1.7B",
|
||||
"downloaded": true,
|
||||
"size_mb": 3400,
|
||||
"loaded": true
|
||||
}
|
||||
```
|
||||
|
||||
### ActiveTasksResponse
|
||||
|
||||
```json
|
||||
{
|
||||
"downloads": [
|
||||
{
|
||||
"model_name": "whisper-medium",
|
||||
"status": "downloading",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"generations": [
|
||||
{
|
||||
"task_id": "uuid",
|
||||
"profile_id": "uuid",
|
||||
"text_preview": "Hello world...",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend Integration
|
||||
|
||||
### Progress Display
|
||||
|
||||
```typescript
|
||||
// Subscribe to download progress via SSE
|
||||
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
const progress = JSON.parse(event.data);
|
||||
updateProgressBar(progress.current / progress.total);
|
||||
|
||||
if (progress.status === 'complete') {
|
||||
eventSource.close();
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Model Status UI
|
||||
|
||||
```typescript
|
||||
// Fetch model status
|
||||
const { data: models } = useQuery({
|
||||
queryKey: ['models', 'status'],
|
||||
queryFn: () => api.getModelStatus(),
|
||||
});
|
||||
|
||||
// Display download/load buttons based on status
|
||||
models.map(model => (
|
||||
<ModelCard
|
||||
name={model.display_name}
|
||||
downloaded={model.downloaded}
|
||||
loaded={model.loaded}
|
||||
onDownload={() => triggerDownload(model.model_name)}
|
||||
onLoad={() => loadModel(model.model_name)}
|
||||
/>
|
||||
));
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Download failed | Network issue | Retry download |
|
||||
| OOM on load | Model too large | Use smaller model |
|
||||
| Model not found | Cache corrupted | Re-download |
|
||||
| Slow download | HF rate limit | Wait and retry |
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: "Development Setup"
|
||||
description: "Set up your local development environment for Voicebox"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before you begin, ensure you have the following installed:
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Bun" icon="package">
|
||||
[Download Bun](https://bun.sh)
|
||||
```bash
|
||||
curl -fsSL https://bun.sh/install | bash
|
||||
```
|
||||
</Card>
|
||||
<Card title="Python 3.11+" icon="python">
|
||||
[Download Python](https://python.org)
|
||||
```bash
|
||||
python --version
|
||||
```
|
||||
</Card>
|
||||
<Card title="Rust" icon="rust">
|
||||
[Install Rust](https://rustup.rs)
|
||||
```bash
|
||||
rustc --version
|
||||
```
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/jamiepine/voicebox.git
|
||||
cd voicebox
|
||||
```
|
||||
|
||||
## Quick Setup (Recommended)
|
||||
|
||||
The easiest way to get started is using the Makefile:
|
||||
|
||||
```bash
|
||||
# Setup everything
|
||||
make setup
|
||||
|
||||
# Start development
|
||||
make dev
|
||||
```
|
||||
|
||||
<Note>
|
||||
The Makefile is available on macOS and Linux. Windows users should follow the manual setup below.
|
||||
</Note>
|
||||
|
||||
## Manual Setup
|
||||
|
||||
### 1. Install JavaScript Dependencies
|
||||
|
||||
```bash
|
||||
bun install
|
||||
```
|
||||
|
||||
This installs dependencies for:
|
||||
- `app/` - Shared React frontend
|
||||
- `tauri/` - Tauri desktop wrapper
|
||||
- `web/` - Web deployment wrapper
|
||||
|
||||
### 2. Set Up Python Backend
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# Create virtual environment
|
||||
python -m venv venv
|
||||
|
||||
# Activate virtual environment
|
||||
source venv/bin/activate # macOS/Linux
|
||||
# or
|
||||
venv\Scripts\activate # Windows
|
||||
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Install MLX dependencies (Apple Silicon only - for faster inference)
|
||||
# On Apple Silicon, this enables native Metal acceleration
|
||||
if [[ $(uname -m) == "arm64" ]]; then
|
||||
pip install -r requirements-mlx.txt
|
||||
fi
|
||||
|
||||
# Install Qwen3-TTS
|
||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
```
|
||||
|
||||
## Running in Development
|
||||
|
||||
Development requires **two terminals**: one for the Python backend, one for the Tauri app.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Terminal 1: Backend">
|
||||
Start the Python server first:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
source venv/bin/activate # Activate venv
|
||||
bun run dev:server
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
uvicorn main:app --reload --port 17493
|
||||
```
|
||||
|
||||
Backend will be available at `http://localhost:17493`
|
||||
</Tab>
|
||||
|
||||
<Tab title="Terminal 2: Desktop App">
|
||||
Then start the Tauri app:
|
||||
|
||||
```bash
|
||||
bun run dev
|
||||
```
|
||||
|
||||
This will:
|
||||
- Create a placeholder sidecar binary
|
||||
- Start Vite dev server on port 5173
|
||||
- Launch Tauri window
|
||||
- Enable hot reload
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Info>
|
||||
In dev mode, the app connects to your manually-started Python server. The bundled server binary is only used in production builds.
|
||||
</Info>
|
||||
|
||||
### Optional: Web App
|
||||
|
||||
```bash
|
||||
bun run dev:web
|
||||
```
|
||||
|
||||
Web app will be available at `http://localhost:5174`
|
||||
|
||||
## Model Downloads
|
||||
|
||||
Models are automatically downloaded from HuggingFace Hub on first use:
|
||||
|
||||
- **Whisper** (transcription): Auto-downloads on first transcription
|
||||
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
|
||||
|
||||
<Warning>
|
||||
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
|
||||
</Warning>
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
voicebox/
|
||||
├── app/ # Shared React frontend
|
||||
│ └── src/
|
||||
│ ├── components/ # UI components
|
||||
│ ├── lib/ # Utilities and API client
|
||||
│ └── hooks/ # React hooks
|
||||
├── backend/ # Python FastAPI server
|
||||
│ ├── main.py # API routes
|
||||
│ ├── tts.py # Voice synthesis
|
||||
│ └── database.py # SQLite operations
|
||||
├── tauri/ # Desktop app wrapper
|
||||
│ └── src-tauri/ # Rust backend
|
||||
├── web/ # Web deployment
|
||||
├── landing/ # Marketing website
|
||||
└── scripts/ # Build & release scripts
|
||||
```
|
||||
|
||||
## Available Make Commands
|
||||
|
||||
Run `make help` to see all available commands:
|
||||
|
||||
```bash
|
||||
make setup # Install all dependencies
|
||||
make dev # Start development servers
|
||||
make dev-web # Start web development server
|
||||
make build # Build desktop app
|
||||
make build-web # Build web app
|
||||
make clean # Clean build artifacts
|
||||
make test # Run tests
|
||||
```
|
||||
|
||||
## Generate OpenAPI Client
|
||||
|
||||
After starting the backend server, generate the TypeScript API client:
|
||||
|
||||
```bash
|
||||
./scripts/generate-api.sh
|
||||
# or
|
||||
bun run generate:api
|
||||
```
|
||||
|
||||
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Architecture" icon="diagram-project" href="/development/architecture">
|
||||
Understand the system architecture
|
||||
</Card>
|
||||
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
|
||||
Read the contribution guidelines
|
||||
</Card>
|
||||
<Card title="Building" icon="hammer" href="/development/building">
|
||||
Learn how to build production releases
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="/api/overview">
|
||||
Explore the REST API
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Backend won't start">
|
||||
- Check Python version (must be 3.11+)
|
||||
- Ensure virtual environment is activated
|
||||
- Verify all dependencies are installed: `pip install -r requirements.txt`
|
||||
- Check if port 17493 is available
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Tauri build fails">
|
||||
- Ensure Rust is installed: `rustc --version`
|
||||
- Clean the build: `cd tauri/src-tauri && cargo clean`
|
||||
- Try rebuilding: `bun run dev`
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="OpenAPI client generation fails">
|
||||
- Ensure backend is running: `curl http://localhost:17493/openapi.json`
|
||||
- Check network connectivity
|
||||
- Verify the backend is accessible at localhost:17493
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
See the full [Troubleshooting Guide](/guides/troubleshooting) for more issues and solutions.
|
||||
@@ -0,0 +1,320 @@
|
||||
---
|
||||
title: "Stories & Timeline"
|
||||
description: "How the multi-voice timeline editor works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Stories allow users to arrange multiple voice generations on a timeline to create multi-voice narratives. The system supports tracks, trimming, splitting, and audio mixing.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Story:** A container that holds story items with metadata.
|
||||
|
||||
**Story Item:** Links a generation to a story with timeline position, track, and trim data.
|
||||
|
||||
**Export:** Combines all items into a single mixed audio file.
|
||||
|
||||
## Data Model
|
||||
|
||||
### Story Table
|
||||
|
||||
```python
|
||||
class Story(Base):
|
||||
__tablename__ = "stories"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
name = Column(String, nullable=False)
|
||||
description = Column(Text)
|
||||
created_at = Column(DateTime)
|
||||
updated_at = Column(DateTime)
|
||||
```
|
||||
|
||||
### StoryItem Table
|
||||
|
||||
```python
|
||||
class StoryItem(Base):
|
||||
__tablename__ = "story_items"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
story_id = Column(String, ForeignKey("stories.id"))
|
||||
generation_id = Column(String, ForeignKey("generations.id"))
|
||||
start_time_ms = Column(Integer, default=0) # Timeline position
|
||||
track = Column(Integer, default=0) # Track number
|
||||
trim_start_ms = Column(Integer, default=0) # Trim from start
|
||||
trim_end_ms = Column(Integer, default=0) # Trim from end
|
||||
created_at = Column(DateTime)
|
||||
```
|
||||
|
||||
## Timeline Concepts
|
||||
|
||||
### Start Time
|
||||
|
||||
`start_time_ms` defines when an item begins on the timeline:
|
||||
|
||||
```
|
||||
Timeline (ms): 0----1000----2000----3000----4000
|
||||
Item 1: [======]
|
||||
Item 2: [==========]
|
||||
Item 3: [====]
|
||||
```
|
||||
|
||||
### Tracks
|
||||
|
||||
Multiple tracks allow overlapping audio:
|
||||
|
||||
```
|
||||
Track 0: [Item 1] [Item 3]
|
||||
Track 1: [Item 2]
|
||||
```
|
||||
|
||||
### Trimming
|
||||
|
||||
Trim values cut audio from the start or end without destroying the original:
|
||||
|
||||
```
|
||||
Original: [=========AUDIO=========]
|
||||
trim_start: ^^
|
||||
trim_end: ^^
|
||||
Result: [=====AUDIO=====]
|
||||
```
|
||||
|
||||
## Core Operations
|
||||
|
||||
### Adding Items
|
||||
|
||||
When adding a generation to a story:
|
||||
|
||||
```python
|
||||
async def add_item_to_story(
|
||||
story_id: str,
|
||||
data: StoryItemCreate,
|
||||
db: Session,
|
||||
) -> StoryItemDetail:
|
||||
# Calculate start time if not provided
|
||||
if data.start_time_ms is None:
|
||||
# Find the end of all existing items
|
||||
existing_items = get_items_with_durations(story_id, db)
|
||||
max_end_time_ms = max(
|
||||
item.start_time_ms + int(gen.duration * 1000)
|
||||
for item, gen in existing_items
|
||||
)
|
||||
start_time_ms = max_end_time_ms + 200 # 200ms gap
|
||||
|
||||
# Create the item
|
||||
item = DBStoryItem(
|
||||
id=str(uuid.uuid4()),
|
||||
story_id=story_id,
|
||||
generation_id=data.generation_id,
|
||||
start_time_ms=start_time_ms,
|
||||
track=data.track or 0,
|
||||
)
|
||||
db.add(item)
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Moving Items
|
||||
|
||||
Update position and/or track:
|
||||
|
||||
```python
|
||||
async def move_story_item(
|
||||
story_id: str,
|
||||
item_id: str,
|
||||
data: StoryItemMove,
|
||||
db: Session,
|
||||
) -> StoryItemDetail:
|
||||
item = get_item(story_id, item_id, db)
|
||||
|
||||
item.start_time_ms = data.start_time_ms
|
||||
item.track = data.track
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Trimming Items
|
||||
|
||||
Non-destructive trimming:
|
||||
|
||||
```python
|
||||
async def trim_story_item(
|
||||
story_id: str,
|
||||
item_id: str,
|
||||
data: StoryItemTrim,
|
||||
db: Session,
|
||||
) -> StoryItemDetail:
|
||||
item = get_item(story_id, item_id, db)
|
||||
generation = get_generation(item.generation_id, db)
|
||||
|
||||
# Validate trim doesn't exceed duration
|
||||
max_duration_ms = int(generation.duration * 1000)
|
||||
if data.trim_start_ms + data.trim_end_ms >= max_duration_ms:
|
||||
return None # Invalid trim
|
||||
|
||||
item.trim_start_ms = data.trim_start_ms
|
||||
item.trim_end_ms = data.trim_end_ms
|
||||
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Splitting Items
|
||||
|
||||
Split one item into two at a specific time:
|
||||
|
||||
```python
|
||||
async def split_story_item(
|
||||
story_id: str,
|
||||
item_id: str,
|
||||
data: StoryItemSplit,
|
||||
db: Session,
|
||||
) -> List[StoryItemDetail]:
|
||||
item = get_item(story_id, item_id, db)
|
||||
generation = get_generation(item.generation_id, db)
|
||||
|
||||
# Calculate split point
|
||||
current_trim_start = item.trim_start_ms
|
||||
current_trim_end = item.trim_end_ms
|
||||
original_duration_ms = int(generation.duration * 1000)
|
||||
absolute_split_ms = current_trim_start + data.split_time_ms
|
||||
|
||||
# Update original: trim from end
|
||||
item.trim_end_ms = original_duration_ms - absolute_split_ms
|
||||
|
||||
# Create new item: trim from start
|
||||
new_item = DBStoryItem(
|
||||
generation_id=item.generation_id, # Same generation
|
||||
start_time_ms=item.start_time_ms + data.split_time_ms,
|
||||
track=item.track,
|
||||
trim_start_ms=absolute_split_ms,
|
||||
trim_end_ms=current_trim_end,
|
||||
)
|
||||
|
||||
db.add(new_item)
|
||||
db.commit()
|
||||
|
||||
return [item, new_item]
|
||||
```
|
||||
|
||||
### Duplicating Items
|
||||
|
||||
Create a copy with all properties:
|
||||
|
||||
```python
|
||||
async def duplicate_story_item(
|
||||
story_id: str,
|
||||
item_id: str,
|
||||
db: Session,
|
||||
) -> StoryItemDetail:
|
||||
original = get_item(story_id, item_id, db)
|
||||
generation = get_generation(original.generation_id, db)
|
||||
|
||||
# Calculate effective duration for positioning
|
||||
effective_duration_ms = (
|
||||
int(generation.duration * 1000)
|
||||
- original.trim_start_ms
|
||||
- original.trim_end_ms
|
||||
)
|
||||
|
||||
# Place copy after original with 200ms gap
|
||||
new_item = DBStoryItem(
|
||||
generation_id=original.generation_id,
|
||||
start_time_ms=original.start_time_ms + effective_duration_ms + 200,
|
||||
track=original.track,
|
||||
trim_start_ms=original.trim_start_ms,
|
||||
trim_end_ms=original.trim_end_ms,
|
||||
)
|
||||
|
||||
db.add(new_item)
|
||||
db.commit()
|
||||
```
|
||||
|
||||
## Audio Export
|
||||
|
||||
### Mixing Algorithm
|
||||
|
||||
The export function mixes all items into a single audio file:
|
||||
|
||||
```python
|
||||
async def export_story_audio(story_id: str, db: Session) -> bytes:
|
||||
items = get_all_items_with_generations(story_id, db)
|
||||
|
||||
# Calculate total duration
|
||||
max_end_time_ms = max(
|
||||
data['start_time_ms'] + data['duration_ms']
|
||||
for data in audio_data
|
||||
)
|
||||
|
||||
# Create output buffer
|
||||
total_samples = int((max_end_time_ms / 1000.0) * sample_rate)
|
||||
final_audio = np.zeros(total_samples, dtype=np.float32)
|
||||
|
||||
# Mix each item at its position
|
||||
for data in audio_data:
|
||||
audio = data['audio']
|
||||
start_sample = int((data['start_time_ms'] / 1000.0) * sample_rate)
|
||||
|
||||
# Apply trim
|
||||
trimmed_audio = audio[trim_start_sample:len(audio) - trim_end_sample]
|
||||
|
||||
# Add to buffer (overlapping items sum together)
|
||||
final_audio[start_sample:start_sample + len(trimmed_audio)] += trimmed_audio
|
||||
|
||||
# Normalize to prevent clipping
|
||||
max_val = np.abs(final_audio).max()
|
||||
if max_val > 1.0:
|
||||
final_audio = final_audio / max_val
|
||||
|
||||
return audio_to_bytes(final_audio, sample_rate)
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/stories` | List all stories |
|
||||
| POST | `/stories` | Create a story |
|
||||
| GET | `/stories/{id}` | Get story with items |
|
||||
| PUT | `/stories/{id}` | Update story metadata |
|
||||
| DELETE | `/stories/{id}` | Delete story |
|
||||
| POST | `/stories/{id}/items` | Add item to story |
|
||||
| DELETE | `/stories/{id}/items/{item_id}` | Remove item |
|
||||
| PUT | `/stories/{id}/items/{item_id}/move` | Move item |
|
||||
| PUT | `/stories/{id}/items/{item_id}/trim` | Trim item |
|
||||
| POST | `/stories/{id}/items/{item_id}/split` | Split item |
|
||||
| POST | `/stories/{id}/items/{item_id}/duplicate` | Duplicate item |
|
||||
| PUT | `/stories/{id}/items/times` | Batch update times |
|
||||
| PUT | `/stories/{id}/items/reorder` | Reorder items |
|
||||
| GET | `/stories/{id}/export-audio` | Export mixed audio |
|
||||
|
||||
## Response Schemas
|
||||
|
||||
### StoryItemDetail
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "item_uuid",
|
||||
"story_id": "story_uuid",
|
||||
"generation_id": "generation_uuid",
|
||||
"start_time_ms": 1500,
|
||||
"track": 0,
|
||||
"trim_start_ms": 200,
|
||||
"trim_end_ms": 100,
|
||||
"profile_id": "profile_uuid",
|
||||
"profile_name": "Narrator",
|
||||
"text": "Hello world",
|
||||
"audio_path": "/path/to/audio.wav",
|
||||
"duration": 2.5,
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend Integration
|
||||
|
||||
The timeline UI needs to:
|
||||
|
||||
1. **Fetch story** with all items
|
||||
2. **Render waveforms** for each item
|
||||
3. **Handle drag/drop** to move items
|
||||
4. **Handle edge drag** for trimming
|
||||
5. **Sync playhead** across all tracks
|
||||
6. **Export** when user clicks download
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: "Transcription"
|
||||
description: "How Whisper-based audio transcription works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
|
||||
|
||||
## Architecture
|
||||
|
||||
The transcription system is built around the `WhisperModel` class:
|
||||
|
||||
**Model Loading:** Lazy loading with HuggingFace Hub download.
|
||||
|
||||
**Audio Processing:** Resampling and preprocessing for Whisper.
|
||||
|
||||
**Inference:** Running transcription with optional language hints.
|
||||
|
||||
## WhisperModel Class
|
||||
|
||||
```python
|
||||
class WhisperModel:
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
```
|
||||
|
||||
### Model Sizes
|
||||
|
||||
| Size | Parameters | VRAM | Speed | Quality |
|
||||
|------|------------|------|-------|---------|
|
||||
| tiny | 39M | ~1GB | Fastest | Basic |
|
||||
| base | 74M | ~1GB | Fast | Good |
|
||||
| small | 244M | ~2GB | Medium | Better |
|
||||
| medium | 769M | ~5GB | Slow | High |
|
||||
| large | 1550M | ~10GB | Slowest | Best |
|
||||
|
||||
Default is `base` for balance of speed and quality.
|
||||
|
||||
## Model Loading
|
||||
|
||||
Models are downloaded from HuggingFace Hub:
|
||||
|
||||
```python
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
|
||||
model_name = f"openai/whisper-{model_size}"
|
||||
|
||||
# Track download progress
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(f"whisper-{model_size}")
|
||||
|
||||
# Load processor and model
|
||||
with tracker.patch_download():
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
|
||||
self.model.to(self.device)
|
||||
|
||||
# Mark complete
|
||||
progress_manager.mark_complete(f"whisper-{model_size}")
|
||||
task_manager.complete_download(f"whisper-{model_size}")
|
||||
```
|
||||
|
||||
### Async Loading
|
||||
|
||||
Like TTS, loading runs in a thread pool:
|
||||
|
||||
```python
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
```
|
||||
|
||||
## Transcription
|
||||
|
||||
### Basic Transcription
|
||||
|
||||
```python
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
await self.load_model_async()
|
||||
|
||||
def _transcribe_sync():
|
||||
# Load and resample to 16kHz (Whisper requirement)
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
sampling_rate=16000,
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
# Set language hint if provided
|
||||
forced_decoder_ids = None
|
||||
if language:
|
||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||
language=language,
|
||||
task="transcribe",
|
||||
)
|
||||
|
||||
# Generate
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
forced_decoder_ids=forced_decoder_ids,
|
||||
)
|
||||
|
||||
# Decode
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
return transcription.strip()
|
||||
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
```
|
||||
|
||||
### Supported Languages
|
||||
|
||||
Whisper supports 99+ languages. Common ones in Voicebox:
|
||||
|
||||
| Code | Language |
|
||||
|------|----------|
|
||||
| en | English |
|
||||
| zh | Chinese |
|
||||
| ja | Japanese |
|
||||
| ko | Korean |
|
||||
| de | German |
|
||||
| fr | French |
|
||||
| ru | Russian |
|
||||
| pt | Portuguese |
|
||||
| es | Spanish |
|
||||
| it | Italian |
|
||||
|
||||
### Language Detection
|
||||
|
||||
When no language is specified, Whisper auto-detects:
|
||||
|
||||
```python
|
||||
# Without language hint - auto-detect
|
||||
transcription = await whisper.transcribe(audio_path)
|
||||
|
||||
# With language hint - more accurate for short clips
|
||||
transcription = await whisper.transcribe(audio_path, language="en")
|
||||
```
|
||||
|
||||
## Transcription with Timestamps
|
||||
|
||||
For advanced use cases, word-level timestamps are available:
|
||||
|
||||
```python
|
||||
async def transcribe_with_timestamps(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> List[Dict[str, any]]:
|
||||
await self.load_model_async()
|
||||
|
||||
def _transcribe_timestamps_sync():
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
return_timestamps=True,
|
||||
)
|
||||
|
||||
# Parse timestamps
|
||||
return [
|
||||
{
|
||||
"text": transcription,
|
||||
"start": 0.0,
|
||||
"end": len(audio) / sr,
|
||||
}
|
||||
]
|
||||
|
||||
return await asyncio.to_thread(_transcribe_timestamps_sync)
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
### Unloading
|
||||
|
||||
Free memory when not needed:
|
||||
|
||||
```python
|
||||
def unload_model(self):
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
del self.processor
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
```
|
||||
|
||||
### Global Instance
|
||||
|
||||
A singleton pattern manages the model:
|
||||
|
||||
```python
|
||||
_whisper_model: Optional[WhisperModel] = None
|
||||
|
||||
def get_whisper_model() -> WhisperModel:
|
||||
global _whisper_model
|
||||
if _whisper_model is None:
|
||||
_whisper_model = WhisperModel()
|
||||
return _whisper_model
|
||||
```
|
||||
|
||||
## Audio Preprocessing
|
||||
|
||||
### Resampling
|
||||
|
||||
Whisper requires 16kHz audio:
|
||||
|
||||
```python
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
```
|
||||
|
||||
### Format Support
|
||||
|
||||
The `load_audio` utility handles:
|
||||
- WAV
|
||||
- MP3
|
||||
- FLAC
|
||||
- OGG
|
||||
- M4A
|
||||
|
||||
All formats are converted to mono 16kHz.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| POST | `/transcribe` | Transcribe audio file |
|
||||
|
||||
### Request
|
||||
|
||||
Multipart form data:
|
||||
|
||||
```
|
||||
POST /transcribe
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <audio_file>
|
||||
language: en (optional)
|
||||
```
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello, this is a test transcription.",
|
||||
"duration": 3.5
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Reference Text for Voice Cloning
|
||||
|
||||
1. User records audio sample
|
||||
2. Audio is sent to `/transcribe`
|
||||
3. Transcription becomes `reference_text`
|
||||
4. Both are added to voice profile
|
||||
|
||||
### Quality Tips
|
||||
|
||||
- Provide language hint for short audio
|
||||
- Use clean audio with minimal noise
|
||||
- Longer audio (>5s) improves accuracy
|
||||
- Consider `small` or `medium` model for better quality
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common issues:
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Model not found | First run, download failed | Retry with network |
|
||||
| OOM | Model too large | Use smaller model |
|
||||
| Empty result | No speech detected | Check audio has speech |
|
||||
| Wrong language | Auto-detect failed | Provide language hint |
|
||||
@@ -0,0 +1,283 @@
|
||||
---
|
||||
title: "TTS Generation"
|
||||
description: "How text-to-speech generation works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
|
||||
|
||||
## Architecture
|
||||
|
||||
The TTS system is built around the `TTSModel` class which manages:
|
||||
|
||||
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
|
||||
|
||||
**Voice Prompts:** Converting reference audio into embeddings.
|
||||
|
||||
**Generation:** Synthesizing speech from text using voice prompts.
|
||||
|
||||
## TTSModel Class
|
||||
|
||||
```python
|
||||
class TTSModel:
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device() # cuda, mps, or cpu
|
||||
```
|
||||
|
||||
### Device Selection
|
||||
|
||||
The model automatically selects the best available device:
|
||||
|
||||
```python
|
||||
def _get_device(self) -> str:
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
return "cpu" # MPS can have issues, use CPU for stability
|
||||
return "cpu"
|
||||
```
|
||||
|
||||
## Model Loading
|
||||
|
||||
Models are downloaded from HuggingFace Hub on first use:
|
||||
|
||||
```python
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
# Model IDs on HuggingFace Hub
|
||||
hf_model_map = {
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
# Load with progress tracking
|
||||
with tracker.patch_download():
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16, # float32 on CPU
|
||||
)
|
||||
```
|
||||
|
||||
### Async Loading
|
||||
|
||||
Loading runs in a thread pool to avoid blocking the event loop:
|
||||
|
||||
```python
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
```
|
||||
|
||||
## Voice Prompt Creation
|
||||
|
||||
Voice prompts are created from reference audio and cached for reuse:
|
||||
|
||||
```python
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
await self.load_model_async()
|
||||
|
||||
# Check cache
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cached = get_cached_voice_prompt(cache_key)
|
||||
if cached:
|
||||
return cached, True
|
||||
|
||||
# Create prompt (blocking, run in thread pool)
|
||||
voice_prompt = await asyncio.to_thread(
|
||||
self.model.create_voice_clone_prompt,
|
||||
ref_audio=audio_path,
|
||||
ref_text=reference_text,
|
||||
)
|
||||
|
||||
# Cache the result
|
||||
cache_voice_prompt(cache_key, voice_prompt)
|
||||
return voice_prompt, False
|
||||
```
|
||||
|
||||
### Combining Multiple Samples
|
||||
|
||||
When a profile has multiple samples, they're combined:
|
||||
|
||||
```python
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate and normalize
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
```
|
||||
|
||||
## Speech Generation
|
||||
|
||||
The core generation function:
|
||||
|
||||
```python
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
await self.load_model_async()
|
||||
|
||||
def _generate_sync():
|
||||
# Set seed for reproducibility
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
|
||||
# Generate audio
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
instruct=instruct, # Natural language delivery control
|
||||
)
|
||||
return wavs[0], sample_rate
|
||||
|
||||
# Run in thread pool
|
||||
return await asyncio.to_thread(_generate_sync)
|
||||
```
|
||||
|
||||
### Instruct Feature
|
||||
|
||||
The `instruct` parameter allows natural language control over speech delivery:
|
||||
|
||||
```python
|
||||
# Examples:
|
||||
instruct = "Speak slowly and clearly"
|
||||
instruct = "Sound excited and enthusiastic"
|
||||
instruct = "Whisper softly"
|
||||
```
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
Voice prompts are cached to avoid recomputation:
|
||||
|
||||
```python
|
||||
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||
"""Generate cache key from audio hash and text."""
|
||||
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
|
||||
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
|
||||
return f"{audio_hash}_{text_hash}"
|
||||
```
|
||||
|
||||
Cache is stored in `data/cache/voice_prompts/`.
|
||||
|
||||
## Memory Management
|
||||
|
||||
### Unloading Models
|
||||
|
||||
Free VRAM/RAM when not needed:
|
||||
|
||||
```python
|
||||
def unload_model(self):
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
```
|
||||
|
||||
### Model Switching
|
||||
|
||||
When switching between model sizes (1.7B ↔ 0.6B):
|
||||
|
||||
```python
|
||||
# Unload existing model first
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
```
|
||||
|
||||
## Generation Flow
|
||||
|
||||
1. **Request** → Validate text and profile ID
|
||||
2. **Profile** → Load profile samples from database
|
||||
3. **Voice Prompt** → Create or retrieve cached prompt
|
||||
4. **Generate** → Run TTS inference
|
||||
5. **Save** → Write audio to generations directory
|
||||
6. **Record** → Create history entry in database
|
||||
7. **Response** → Return audio path and metadata
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| POST | `/generate` | Generate speech from text |
|
||||
| GET | `/audio/{id}` | Serve generated audio file |
|
||||
|
||||
### Request Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"profile_id": "uuid",
|
||||
"text": "Text to synthesize",
|
||||
"language": "en",
|
||||
"seed": 42,
|
||||
"model_size": "1.7B",
|
||||
"instruct": "Speak clearly"
|
||||
}
|
||||
```
|
||||
|
||||
### Response Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "generation_uuid",
|
||||
"profile_id": "profile_uuid",
|
||||
"text": "Text to synthesize",
|
||||
"language": "en",
|
||||
"audio_path": "/path/to/audio.wav",
|
||||
"duration": 3.5,
|
||||
"seed": 42,
|
||||
"instruct": "Speak clearly",
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### GPU Acceleration
|
||||
|
||||
- CUDA provides fastest inference
|
||||
- MPS (Apple Silicon) has stability issues, uses CPU fallback
|
||||
- CPU inference is slower but always works
|
||||
|
||||
### Batch Size
|
||||
|
||||
Currently generates one utterance at a time. For long texts, consider:
|
||||
- Splitting into sentences
|
||||
- Sequential generation
|
||||
- Concatenating results
|
||||
|
||||
### Memory Usage
|
||||
|
||||
| Model | VRAM/RAM Required |
|
||||
|-------|-------------------|
|
||||
| 0.6B | ~2GB |
|
||||
| 1.7B | ~6GB |
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: "Voice Profiles"
|
||||
description: "How voice profile management works in Voicebox"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
|
||||
|
||||
## Architecture
|
||||
|
||||
The voice profile system consists of three main components:
|
||||
|
||||
**Database Layer:** SQLite tables store profile metadata and sample references.
|
||||
|
||||
**File Storage:** Audio samples are stored on disk in a structured directory format.
|
||||
|
||||
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
|
||||
|
||||
## Data Model
|
||||
|
||||
### VoiceProfile Table
|
||||
|
||||
```python
|
||||
class VoiceProfile(Base):
|
||||
__tablename__ = "profiles"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text)
|
||||
language = Column(String, default="en")
|
||||
created_at = Column(DateTime)
|
||||
updated_at = Column(DateTime)
|
||||
```
|
||||
|
||||
### ProfileSample Table
|
||||
|
||||
```python
|
||||
class ProfileSample(Base):
|
||||
__tablename__ = "profile_samples"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||
audio_path = Column(String, nullable=False)
|
||||
reference_text = Column(Text, nullable=False)
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
Profiles are stored in the data directory:
|
||||
|
||||
```
|
||||
data/
|
||||
└── profiles/
|
||||
└── {profile_id}/
|
||||
├── {sample_id_1}.wav
|
||||
├── {sample_id_2}.wav
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Core Functions
|
||||
|
||||
### Creating a Profile
|
||||
|
||||
```python
|
||||
async def create_profile(data: VoiceProfileCreate, db: Session) -> VoiceProfileResponse:
|
||||
# 1. Create database record
|
||||
db_profile = DBVoiceProfile(
|
||||
id=str(uuid.uuid4()),
|
||||
name=data.name,
|
||||
description=data.description,
|
||||
language=data.language,
|
||||
)
|
||||
db.add(db_profile)
|
||||
db.commit()
|
||||
|
||||
# 2. Create profile directory
|
||||
profile_dir = profiles_dir / db_profile.id
|
||||
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
return VoiceProfileResponse.model_validate(db_profile)
|
||||
```
|
||||
|
||||
### Adding Samples
|
||||
|
||||
When a sample is added, the audio is validated and copied to the profile directory:
|
||||
|
||||
```python
|
||||
async def add_profile_sample(
|
||||
profile_id: str,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
db: Session,
|
||||
) -> ProfileSampleResponse:
|
||||
# 1. Validate audio (duration, format, quality)
|
||||
is_valid, error_msg = validate_reference_audio(audio_path)
|
||||
if not is_valid:
|
||||
raise ValueError(f"Invalid reference audio: {error_msg}")
|
||||
|
||||
# 2. Copy to profile directory
|
||||
sample_id = str(uuid.uuid4())
|
||||
dest_path = profile_dir / f"{sample_id}.wav"
|
||||
audio, sr = load_audio(audio_path)
|
||||
save_audio(audio, str(dest_path), sr)
|
||||
|
||||
# 3. Create database record
|
||||
db_sample = DBProfileSample(
|
||||
id=sample_id,
|
||||
profile_id=profile_id,
|
||||
audio_path=str(dest_path),
|
||||
reference_text=reference_text,
|
||||
)
|
||||
db.add(db_sample)
|
||||
db.commit()
|
||||
```
|
||||
|
||||
### Voice Prompt Creation
|
||||
|
||||
When generating speech, samples are combined into a voice prompt:
|
||||
|
||||
```python
|
||||
async def create_voice_prompt_for_profile(
|
||||
profile_id: str,
|
||||
db: Session,
|
||||
) -> dict:
|
||||
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
|
||||
|
||||
if len(samples) == 1:
|
||||
# Single sample - use directly
|
||||
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||
sample.audio_path,
|
||||
sample.reference_text,
|
||||
)
|
||||
else:
|
||||
# Multiple samples - combine them
|
||||
combined_audio, combined_text = await tts_model.combine_voice_prompts(
|
||||
[s.audio_path for s in samples],
|
||||
[s.reference_text for s in samples],
|
||||
)
|
||||
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||
combined_audio_path,
|
||||
combined_text,
|
||||
)
|
||||
|
||||
return voice_prompt
|
||||
```
|
||||
|
||||
## Audio Validation
|
||||
|
||||
Reference audio is validated before being accepted:
|
||||
|
||||
- **Duration:** 3-30 seconds recommended
|
||||
- **Format:** WAV, MP3, FLAC, OGG supported
|
||||
- **Sample Rate:** Resampled to 24kHz
|
||||
- **Channels:** Converted to mono if stereo
|
||||
|
||||
## Export/Import
|
||||
|
||||
Profiles can be exported as ZIP archives for sharing:
|
||||
|
||||
```
|
||||
profile_export.zip
|
||||
├── profile.json # Metadata
|
||||
├── samples/
|
||||
│ ├── sample_1.wav
|
||||
│ └── sample_1.json # Reference text
|
||||
└── ...
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/profiles` | List all profiles |
|
||||
| POST | `/profiles` | Create a profile |
|
||||
| GET | `/profiles/{id}` | Get profile by ID |
|
||||
| PUT | `/profiles/{id}` | Update profile |
|
||||
| DELETE | `/profiles/{id}` | Delete profile |
|
||||
| GET | `/profiles/{id}/samples` | Get profile samples |
|
||||
| POST | `/profiles/{id}/samples` | Add sample to profile |
|
||||
| PUT | `/profiles/samples/{id}` | Update sample text |
|
||||
| DELETE | `/profiles/samples/{id}` | Delete sample |
|
||||
| GET | `/profiles/{id}/export` | Export as ZIP |
|
||||
| POST | `/profiles/import` | Import from ZIP |
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Sample Quality
|
||||
|
||||
- Use clean audio with minimal background noise
|
||||
- Ensure the reference text exactly matches what is spoken
|
||||
- Multiple samples (3-5) improve voice cloning quality
|
||||
|
||||
### Language Matching
|
||||
|
||||
- Set the profile language to match the reference audio
|
||||
- Supported languages: en, zh, ja, ko, de, fr, ru, pt, es, it
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
- Use descriptive names that identify the voice
|
||||
- Avoid special characters that may cause filesystem issues
|
||||
|
After Width: | Height: | Size: 134 KiB |
|
After Width: | Height: | Size: 129 KiB |
|
After Width: | Height: | Size: 108 KiB |
|
After Width: | Height: | Size: 10 KiB |
|
After Width: | Height: | Size: 10 KiB |
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"$schema": "https://mintlify.com/schema.json",
|
||||
"name": "Voicebox",
|
||||
"logo": {
|
||||
"light": "/logo/icon-light.png",
|
||||
"dark": "/logo/icon-dark.png"
|
||||
},
|
||||
"favicon": "/favicon.png",
|
||||
"colors": {
|
||||
"primary": "#BF9E40",
|
||||
"light": "#D4B560",
|
||||
"dark": "#A68A35"
|
||||
},
|
||||
"styles": {
|
||||
"css": ["/custom.css"]
|
||||
},
|
||||
"anchors": [
|
||||
{
|
||||
"name": "Overview",
|
||||
"icon": "book-open",
|
||||
"url": "overview"
|
||||
},
|
||||
{
|
||||
"name": "API",
|
||||
"icon": "code",
|
||||
"url": "api"
|
||||
},
|
||||
{
|
||||
"name": "Developer",
|
||||
"icon": "book-open-cover",
|
||||
"url": "developer"
|
||||
},
|
||||
{
|
||||
"name": "GitHub",
|
||||
"icon": "github",
|
||||
"url": "https://github.com/jamiepine/voicebox"
|
||||
}
|
||||
],
|
||||
"navigation": [
|
||||
{
|
||||
"group": "Getting Started",
|
||||
"icon": "rocket",
|
||||
"pages": ["overview/introduction", "overview/installation", "overview/quick-start"]
|
||||
},
|
||||
{
|
||||
"group": "Features",
|
||||
"icon": "sparkles",
|
||||
"pages": [
|
||||
"overview/voice-cloning",
|
||||
"overview/stories-editor",
|
||||
"overview/recording-transcription",
|
||||
"overview/generation-history",
|
||||
"overview/remote-mode"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "User Guides",
|
||||
"icon": "compass",
|
||||
"pages": [
|
||||
"overview/creating-voice-profiles",
|
||||
"overview/generating-speech",
|
||||
"overview/building-stories",
|
||||
"overview/troubleshooting"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Development",
|
||||
"icon": "wrench",
|
||||
"pages": [
|
||||
"developer/setup",
|
||||
"developer/architecture",
|
||||
"developer/contributing",
|
||||
"developer/building",
|
||||
"developer/autoupdater"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "API Reference",
|
||||
"icon": "code",
|
||||
"pages": [
|
||||
"api/overview",
|
||||
"api/authentication",
|
||||
"api/voice-profiles",
|
||||
"api/generation",
|
||||
"api/recordings"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Architecture",
|
||||
"icon": "book-open-cover",
|
||||
"pages": [
|
||||
"developer/voice-profiles",
|
||||
"developer/tts-generation",
|
||||
"developer/history",
|
||||
"developer/stories",
|
||||
"developer/transcription",
|
||||
"developer/audio-channels",
|
||||
"developer/model-management"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
[phases.setup]
|
||||
nixPkgs = ["nodejs_20", "bun"]
|
||||
|
||||
[phases.install]
|
||||
cmds = ["bun install"]
|
||||
|
||||
[phases.build]
|
||||
cmds = ["bun run build"]
|
||||
|
||||
[start]
|
||||
cmd = "bun run start"
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: "Building Stories"
|
||||
description: "Create multi-voice narratives with the Stories Editor"
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
The Stories Editor is perfect for creating podcasts, audiobooks, and multi-speaker content.
|
||||
|
||||
<Steps>
|
||||
<Step title="Create Story">
|
||||
**Stories** → **+ New Story**
|
||||
</Step>
|
||||
<Step title="Add Tracks">
|
||||
Create tracks for each speaker
|
||||
</Step>
|
||||
<Step title="Add Clips">
|
||||
Generate or drag audio to tracks
|
||||
</Step>
|
||||
<Step title="Arrange">
|
||||
Position and trim clips on timeline
|
||||
</Step>
|
||||
<Step title="Export">
|
||||
Render final audio
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Use Cases
|
||||
|
||||
- Multi-host podcasts
|
||||
- Audiobook narration with character voices
|
||||
- Game dialogue scenes
|
||||
- Educational content with multiple speakers
|
||||
|
||||
## Coming Soon
|
||||
|
||||
Full timeline editor documentation will be added as features are finalized.
|
||||
@@ -0,0 +1,296 @@
|
||||
---
|
||||
title: "Creating Voice Profiles"
|
||||
description: "Advanced guide to creating high-quality voice profiles"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
|
||||
|
||||
## Quick Start
|
||||
|
||||
<Steps>
|
||||
<Step title="Prepare Audio">
|
||||
10-30 seconds of clear speech
|
||||
</Step>
|
||||
<Step title="Create Profile">
|
||||
**Profiles** → **+ New Profile**
|
||||
</Step>
|
||||
<Step title="Upload Sample">
|
||||
Add your audio file
|
||||
</Step>
|
||||
<Step title="Generate">
|
||||
Use the profile to generate speech
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Audio Requirements
|
||||
|
||||
### Ideal Sample Characteristics
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Duration" icon="clock">
|
||||
**10-30 seconds**
|
||||
|
||||
Too short: Poor quality
|
||||
Too long: Unnecessary
|
||||
</Card>
|
||||
<Card title="Clarity" icon="volume">
|
||||
**Clear speech**
|
||||
|
||||
No background noise
|
||||
No music or overlapping voices
|
||||
</Card>
|
||||
<Card title="Quality" icon="sparkles">
|
||||
**High fidelity**
|
||||
|
||||
44.1kHz or 48kHz sample rate
|
||||
Minimal compression
|
||||
</Card>
|
||||
<Card title="Content" icon="microphone">
|
||||
**Natural speech**
|
||||
|
||||
Conversational tone
|
||||
Complete sentences
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### File Formats
|
||||
|
||||
Supported formats:
|
||||
- **WAV** (recommended) - Lossless quality
|
||||
- **MP3** - Acceptable, minimal compression
|
||||
- **M4A** - Acceptable
|
||||
- **FLAC** - Lossless alternative
|
||||
|
||||
<Tip>
|
||||
Use WAV for best results. Avoid heavily compressed formats.
|
||||
</Tip>
|
||||
|
||||
## Recording Tips
|
||||
|
||||
### Environment
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Quiet Space">
|
||||
- Record in a quiet room
|
||||
- Turn off fans, AC, appliances
|
||||
- Close windows to reduce outside noise
|
||||
- Use soft furnishings to reduce echo
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Microphone Placement">
|
||||
- 6-12 inches from mouth
|
||||
- Slight angle to reduce plosives (p, b, t)
|
||||
- Use a pop filter if available
|
||||
- Maintain consistent distance
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Recording Settings">
|
||||
- 44.1kHz or 48kHz sample rate
|
||||
- 16-bit or 24-bit depth
|
||||
- Mono is fine (stereo will be converted)
|
||||
- Avoid automatic gain control
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### Speaking
|
||||
|
||||
- **Natural pace** - Don't rush or speak too slowly
|
||||
- **Clear articulation** - Pronounce words clearly
|
||||
- **Consistent volume** - Maintain steady loudness
|
||||
- **Normal tone** - Speak as you normally would
|
||||
- **Complete sentences** - Avoid fragments or "ums"
|
||||
|
||||
## Multiple Samples
|
||||
|
||||
Adding multiple samples can significantly improve quality:
|
||||
|
||||
### Why Multiple Samples?
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Robustness" icon="shield">
|
||||
Model learns a more complete representation
|
||||
</Card>
|
||||
<Card title="Versatility" icon="palette">
|
||||
Handles different speaking styles better
|
||||
</Card>
|
||||
<Card title="Quality" icon="star">
|
||||
Reduces artifacts and improves naturalness
|
||||
</Card>
|
||||
<Card title="Consistency" icon="check">
|
||||
More reliable across different texts
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### Sample Variety
|
||||
|
||||
Consider adding samples with:
|
||||
|
||||
1. **Different tones**
|
||||
- Casual conversation
|
||||
- Professional/formal
|
||||
- Excited/enthusiastic
|
||||
- Calm/serious
|
||||
|
||||
2. **Different content**
|
||||
- Narratives
|
||||
- Questions
|
||||
- Statements
|
||||
- Emotions (happy, sad, neutral)
|
||||
|
||||
3. **Different recording conditions**
|
||||
- Studio quality
|
||||
- Phone call quality (if needed)
|
||||
- Room acoustics
|
||||
|
||||
<Warning>
|
||||
All samples should be from the **same speaker**. Mixing voices will produce poor results.
|
||||
</Warning>
|
||||
|
||||
## Processing Existing Audio
|
||||
|
||||
If you have existing audio (podcasts, videos, etc.):
|
||||
|
||||
### Extracting Clean Segments
|
||||
|
||||
<Steps>
|
||||
<Step title="Find Clean Speech">
|
||||
Look for segments with:
|
||||
- Just the target speaker
|
||||
- No background music
|
||||
- Minimal noise
|
||||
</Step>
|
||||
|
||||
<Step title="Use Audio Editor">
|
||||
Tools like Audacity or Adobe Audition:
|
||||
- Cut out clean 10-30s segments
|
||||
- Remove silence at start/end
|
||||
- Normalize volume if needed
|
||||
</Step>
|
||||
|
||||
<Step title="Export as WAV">
|
||||
Save as high-quality WAV file
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Noise Reduction
|
||||
|
||||
If you have light background noise:
|
||||
|
||||
```
|
||||
1. Use noise reduction in Audacity:
|
||||
- Select noise-only section
|
||||
- Get Noise Profile
|
||||
- Select full audio
|
||||
- Apply noise reduction (gentle settings)
|
||||
|
||||
2. Avoid over-processing:
|
||||
- Can introduce artifacts
|
||||
- May reduce voice quality
|
||||
```
|
||||
|
||||
## Testing & Iteration
|
||||
|
||||
### Test Your Profile
|
||||
|
||||
After creating a profile:
|
||||
|
||||
<Steps>
|
||||
<Step title="Generate Test">
|
||||
Generate a simple phrase:
|
||||
```
|
||||
"Hello, this is a test of my voice profile."
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Evaluate Quality">
|
||||
Listen for:
|
||||
- Natural tone
|
||||
- Clear pronunciation
|
||||
- Proper prosody
|
||||
- Lack of artifacts
|
||||
</Step>
|
||||
|
||||
<Step title="Iterate">
|
||||
If quality is poor:
|
||||
- Add more samples
|
||||
- Try different source audio
|
||||
- Check sample quality
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Common Issues
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Robotic Voice">
|
||||
**Cause**: Poor quality samples or too short
|
||||
|
||||
**Fix**: Use longer, higher quality samples
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Wrong Tone">
|
||||
**Cause**: Sample tone doesn't match desired output
|
||||
|
||||
**Fix**: Record samples in the style you want to generate
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Artifacts/Glitches">
|
||||
**Cause**: Background noise or audio issues in samples
|
||||
|
||||
**Fix**: Clean up samples or re-record in quieter environment
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Advanced Tips
|
||||
|
||||
### Celebrity/Character Voices
|
||||
|
||||
For cloning public figures or characters:
|
||||
|
||||
1. **Legal considerations** - Ensure you have rights or it's fair use
|
||||
2. **Source quality** - Find high-quality interview audio or clean clips
|
||||
3. **Consistency** - Use clips where they speak similarly
|
||||
4. **Multiple samples** - Very important for recognizable voices
|
||||
|
||||
### Accent & Dialect
|
||||
|
||||
The model will preserve accent and dialect:
|
||||
|
||||
- British English will generate British English
|
||||
- Southern accent will produce Southern accent
|
||||
- Regional pronunciations will be maintained
|
||||
|
||||
### Emotion Transfer
|
||||
|
||||
The emotional tone of samples affects generation:
|
||||
|
||||
- Energetic samples → Energetic output
|
||||
- Calm samples → Calm output
|
||||
- Mix samples for versatile profile
|
||||
|
||||
## Managing Profiles
|
||||
|
||||
### Organization
|
||||
|
||||
- **Descriptive names** - "John Smith - Professional Narrator"
|
||||
- **Add descriptions** - Note recording conditions, use cases
|
||||
- **Language tags** - Mark the primary language
|
||||
- **Archive unused** - Keep profile list manageable
|
||||
|
||||
### Export/Import
|
||||
|
||||
- **Export** profiles to share or backup
|
||||
- **Import** from colleagues or teammates
|
||||
- Profiles include voice embeddings, not original audio
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Generate Speech" icon="waveform" href="/guides/generating-speech">
|
||||
Use your profile to generate speech
|
||||
</Card>
|
||||
<Card title="Build Stories" icon="film" href="/guides/building-stories">
|
||||
Create multi-voice narratives
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "Generating Speech"
|
||||
description: "Generate high-quality speech from text"
|
||||
---
|
||||
|
||||
## Basic Generation
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Profile">
|
||||
Choose a voice profile from the dropdown
|
||||
</Step>
|
||||
<Step title="Enter Text">
|
||||
Type or paste your text
|
||||
</Step>
|
||||
<Step title="Generate">
|
||||
Click **Generate** and wait a few seconds
|
||||
</Step>
|
||||
<Step title="Play & Export">
|
||||
Preview and download the result
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Text Formatting Tips
|
||||
|
||||
The way you format text affects the output quality.
|
||||
|
||||
### Punctuation
|
||||
|
||||
Use proper punctuation for natural pauses:
|
||||
|
||||
```
|
||||
Good: "Hello! How are you today? I'm doing great."
|
||||
Bad: "Hello how are you today Im doing great"
|
||||
```
|
||||
|
||||
### Emphasis
|
||||
|
||||
Use formatting to suggest emphasis:
|
||||
|
||||
```
|
||||
- ALL CAPS for louder/emphasized: "That was AMAZING!"
|
||||
- Italics for subtle emphasis: "I *really* enjoyed that"
|
||||
- Bold for strong emphasis: "This is **very** important"
|
||||
```
|
||||
|
||||
<Note>
|
||||
The model interprets these hints but results may vary.
|
||||
</Note>
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Batch Generation
|
||||
|
||||
For long-form content, split into smaller chunks for better control and faster processing.
|
||||
|
||||
### Voice Caching
|
||||
|
||||
Voicebox caches voice prompts for faster re-generation with the same profile.
|
||||
|
||||
## Coming Soon
|
||||
|
||||
- Real-time streaming
|
||||
- Word-level timing control
|
||||
- Emotion and style controls
|
||||
- SSML support
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: "Generation History"
|
||||
description: "Track and manage all your generated audio"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox keeps a complete history of all generated audio, making it easy to find, reuse, and manage your creations.
|
||||
|
||||
## Features
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Full History" icon="clock">
|
||||
Every generation is automatically saved
|
||||
</Card>
|
||||
<Card title="Search & Filter" icon="search">
|
||||
Find by text, voice, or date
|
||||
</Card>
|
||||
<Card title="Re-generate" icon="rotate">
|
||||
Regenerate any past generation with one click
|
||||
</Card>
|
||||
<Card title="Export" icon="download">
|
||||
Download individual or batch exports
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Viewing History
|
||||
|
||||
Navigate to the **History** tab to see all your generations.
|
||||
|
||||
Each entry shows:
|
||||
- Generated text
|
||||
- Voice profile used
|
||||
- Timestamp
|
||||
- Audio duration
|
||||
- Language
|
||||
|
||||
## Actions
|
||||
|
||||
### Play
|
||||
Click any generation to play it immediately.
|
||||
|
||||
### Re-generate
|
||||
Regenerate with the same settings or modify the text/voice.
|
||||
|
||||
### Download
|
||||
Export as WAV, MP3, or M4A.
|
||||
|
||||
### Delete
|
||||
Remove unwanted generations to free up space.
|
||||
|
||||
### Add to Story
|
||||
Drag generations to the Stories Editor timeline.
|
||||
|
||||
## Search & Filter
|
||||
|
||||
<Tabs>
|
||||
<Tab title="By Text">
|
||||
Search for specific text content
|
||||
```
|
||||
"Hello world"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="By Voice">
|
||||
Filter by voice profile
|
||||
```
|
||||
Select from dropdown
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="By Date">
|
||||
Filter by date range
|
||||
```
|
||||
Last 7 days, Last 30 days, Custom range
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Storage
|
||||
|
||||
History is stored locally:
|
||||
|
||||
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
|
||||
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
|
||||
- **Linux**: `~/.config/com.voicebox.app/data/`
|
||||
|
||||
<Warning>
|
||||
Deleting the data directory will remove all history. Export important files first.
|
||||
</Warning>
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "Installation"
|
||||
description: "Download and install Voicebox on macOS, Windows, or Linux"
|
||||
---
|
||||
|
||||
## Download
|
||||
|
||||
Voicebox is available for macOS and Windows, with Linux builds coming soon.
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="macOS" icon="apple">
|
||||
Download for Apple Silicon or Intel Macs
|
||||
</Card>
|
||||
<Card title="Windows" icon="windows">
|
||||
Download MSI installer or Setup executable
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### macOS
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Apple Silicon">
|
||||
Download: [voicebox_aarch64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_aarch64.app.tar.gz)
|
||||
|
||||
```bash
|
||||
# Extract the archive
|
||||
tar -xzf voicebox_aarch64.app.tar.gz
|
||||
|
||||
# Move to Applications
|
||||
mv Voicebox.app /Applications/
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Intel">
|
||||
Download: [voicebox_x64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64.app.tar.gz)
|
||||
|
||||
```bash
|
||||
# Extract the archive
|
||||
tar -xzf voicebox_x64.app.tar.gz
|
||||
|
||||
# Move to Applications
|
||||
mv Voicebox.app /Applications/
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Windows
|
||||
|
||||
<Tabs>
|
||||
<Tab title="MSI Installer">
|
||||
Download: [voicebox_x64_en-US.msi](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64_en-US.msi)
|
||||
|
||||
Double-click the MSI file and follow the installation wizard.
|
||||
</Tab>
|
||||
<Tab title="Setup Executable">
|
||||
Download: [voicebox_x64-setup.exe](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64-setup.exe)
|
||||
|
||||
Run the executable and follow the installation wizard.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Linux
|
||||
|
||||
<Note>
|
||||
Linux builds are coming soon. Currently blocked by GitHub runner disk space limitations.
|
||||
</Note>
|
||||
|
||||
## First Launch
|
||||
|
||||
When you launch Voicebox for the first time:
|
||||
|
||||
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
|
||||
2. **Data Directory** — Voice profiles and generated audio are stored in:
|
||||
- macOS: `~/Library/Application Support/com.voicebox.app/`
|
||||
- Windows: `%APPDATA%/com.voicebox.app/`
|
||||
- Linux: `~/.config/com.voicebox.app/`
|
||||
|
||||
3. **Backend Server** — The bundled Python server starts automatically
|
||||
|
||||
<Tip>
|
||||
First generation will be slower due to model downloads. Subsequent runs use cached models.
|
||||
</Tip>
|
||||
|
||||
## System Requirements
|
||||
|
||||
### Minimum
|
||||
|
||||
- **OS:** macOS 11+, Windows 10+, or Linux
|
||||
- **RAM:** 8GB
|
||||
- **Storage:** 5GB free space (for models and data)
|
||||
- **CPU:** Modern multi-core processor
|
||||
|
||||
### Recommended
|
||||
|
||||
- **RAM:** 16GB+
|
||||
- **GPU:** CUDA-capable NVIDIA GPU (for faster generation)
|
||||
- **Storage:** 10GB+ free space
|
||||
|
||||
<Note>
|
||||
CPU inference is supported but significantly slower than GPU. A CUDA-capable GPU is highly recommended for real-time workflows.
|
||||
</Note>
|
||||
|
||||
## Verification
|
||||
|
||||
After installation, verify everything works:
|
||||
|
||||
1. Launch Voicebox
|
||||
2. Check the server status indicator in the bottom-left corner (should be green)
|
||||
3. Navigate to **Profiles** and create a test profile
|
||||
4. Generate a short audio clip to verify the TTS engine works
|
||||
|
||||
<Check>
|
||||
If you see a green status indicator and can generate audio, you're all set!
|
||||
</Check>
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Card title="Quick Start Guide" icon="rocket" href="/overview/quick-start">
|
||||
Create your first voice profile and generate speech
|
||||
</Card>
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: "Introduction"
|
||||
description: "Welcome to Voicebox - the open-source voice synthesis studio"
|
||||
---
|
||||
|
||||
## What is Voicebox?
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/app-screenshot-1.webp" alt="Voicebox App Screenshot" />
|
||||
</Frame>
|
||||
|
||||
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
|
||||
|
||||
- **Complete privacy** — models and voice data stay on your machine
|
||||
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
|
||||
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
|
||||
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
|
||||
- **Native performance** — built with Tauri (Rust), not Electron
|
||||
|
||||
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
|
||||
|
||||
## Key Features
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Voice Cloning" icon="microphone">
|
||||
Instant cloning from just a few seconds of audio with Qwen3-TTS
|
||||
</Card>
|
||||
<Card title="Stories Editor" icon="film">
|
||||
Multi-track timeline for creating conversations and narratives
|
||||
</Card>
|
||||
<Card title="Full API" icon="code">
|
||||
REST API for integrating voice synthesis into your apps
|
||||
</Card>
|
||||
<Card title="Local-First" icon="shield">
|
||||
Everything runs on your machine - complete privacy
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Game Development** — Generate dynamic dialogue for characters
|
||||
- **Content Creation** — Produce podcasts and video voiceovers
|
||||
- **Accessibility** — Build text-to-speech tools
|
||||
- **Voice Assistants** — Create custom voice interfaces
|
||||
- **Production Pipelines** — Automate voiceover workflows
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Installation" icon="download" href="/overview/installation">
|
||||
Download and install Voicebox on your machine
|
||||
</Card>
|
||||
<Card title="Quick Start" icon="rocket" href="/overview/quick-start">
|
||||
Get up and running in 5 minutes
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "Quick Start"
|
||||
description: "Get started with Voicebox in 5 minutes"
|
||||
---
|
||||
|
||||
This guide will walk you through creating your first voice profile and generating speech.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Make sure you have [installed Voicebox](/overview/installation) and launched the app.
|
||||
|
||||
## Step 1: Create a Voice Profile
|
||||
|
||||
Voice profiles are the foundation of Voicebox. Each profile contains voice samples that the AI uses to clone the voice.
|
||||
|
||||
<Steps>
|
||||
<Step title="Navigate to Profiles">
|
||||
Click the **Profiles** tab in the sidebar
|
||||
</Step>
|
||||
|
||||
<Step title="Create New Profile">
|
||||
Click the **+ New Profile** button
|
||||
|
||||
Fill in the details:
|
||||
- **Name:** A descriptive name (e.g., "John Smith")
|
||||
- **Language:** Select the primary language
|
||||
- **Description:** Optional notes about the voice
|
||||
</Step>
|
||||
|
||||
<Step title="Add Voice Sample">
|
||||
You have two options:
|
||||
|
||||
**Option A: Upload Audio**
|
||||
- Click **Upload Sample**
|
||||
- Select an audio file (WAV, MP3, or M4A)
|
||||
- Ideal length: 10-30 seconds of clear speech
|
||||
|
||||
**Option B: Record Live**
|
||||
- Click **Record Sample**
|
||||
- Speak clearly for 10-30 seconds
|
||||
- Click stop when finished
|
||||
</Step>
|
||||
|
||||
<Step title="Save Profile">
|
||||
Click **Create Profile** to save
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
<Tip>
|
||||
For best results, use clean audio with minimal background noise and consistent speaking tone.
|
||||
</Tip>
|
||||
|
||||
## Step 2: Generate Speech
|
||||
|
||||
Now let's use your new voice profile to generate speech.
|
||||
|
||||
<Steps>
|
||||
<Step title="Go to Generation">
|
||||
Click the **Generate** tab in the sidebar
|
||||
</Step>
|
||||
|
||||
<Step title="Select Voice Profile">
|
||||
Choose your newly created profile from the dropdown
|
||||
</Step>
|
||||
|
||||
<Step title="Enter Text">
|
||||
Type or paste the text you want to generate:
|
||||
|
||||
```
|
||||
Hello! This is my first voice generation with Voicebox.
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Generate">
|
||||
Click **Generate** and wait a few seconds
|
||||
|
||||
<Note>
|
||||
First generation may take longer due to model initialization. Subsequent generations will be faster.
|
||||
</Note>
|
||||
</Step>
|
||||
|
||||
<Step title="Play & Download">
|
||||
- Click **Play** to preview the audio
|
||||
- Click **Download** to save the audio file
|
||||
- The generation is also saved to your **History**
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Step 3: Build a Story (Optional)
|
||||
|
||||
The Stories Editor lets you create multi-voice narratives with a timeline-based interface.
|
||||
|
||||
<Steps>
|
||||
<Step title="Create New Story">
|
||||
Navigate to **Stories** and click **+ New Story**
|
||||
</Step>
|
||||
|
||||
<Step title="Add Voice Tracks">
|
||||
Click **+ Add Track** to create tracks for different speakers
|
||||
</Step>
|
||||
|
||||
<Step title="Add Audio Clips">
|
||||
- Drag generated audio from your History
|
||||
- Or generate new clips directly in the timeline
|
||||
- Arrange clips on the timeline
|
||||
</Step>
|
||||
|
||||
<Step title="Edit & Export">
|
||||
- Trim clips by dragging edges
|
||||
- Adjust timing and spacing
|
||||
- Click **Export** to render the final audio
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## What's Next?
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Voice Cloning Guide" icon="microphone" href="/guides/creating-voice-profiles">
|
||||
Learn advanced techniques for high-quality voice cloning
|
||||
</Card>
|
||||
<Card title="API Integration" icon="code" href="/api/overview">
|
||||
Integrate Voicebox into your own applications
|
||||
</Card>
|
||||
<Card title="Stories Editor" icon="film" href="/overview/stories-editor">
|
||||
Master the multi-track timeline editor
|
||||
</Card>
|
||||
<Card title="Remote Mode" icon="server" href="/overview/remote-mode">
|
||||
Connect to a GPU server for faster generation
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Tips for Success
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Getting the Best Voice Quality">
|
||||
- Use 10-30 seconds of clear, consistent speech
|
||||
- Avoid background noise and echo
|
||||
- Multiple samples from the same speaker improve quality
|
||||
- Match the speaking style you want to generate
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Improving Generation Speed">
|
||||
- Use a CUDA-capable GPU for 5-10x faster generation
|
||||
- Enable voice prompt caching for repeated generations
|
||||
- Consider running the backend on a remote GPU server
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Troubleshooting Common Issues">
|
||||
- **Server won't start:** Check if port 17493 is available
|
||||
- **Poor audio quality:** Try adding more voice samples
|
||||
- **Slow generation:** Verify GPU acceleration is enabled
|
||||
- See the full [Troubleshooting Guide](/guides/troubleshooting) for more
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: "Recording & Transcription"
|
||||
description: "Record audio and transcribe speech with Whisper"
|
||||
---
|
||||
|
||||
## Recording
|
||||
|
||||
Voicebox includes built-in recording capabilities for creating voice samples and capturing audio.
|
||||
|
||||
### Features
|
||||
|
||||
- **Microphone input** - Record from any audio input device
|
||||
- **System audio capture** - Record desktop audio (macOS/Windows)
|
||||
- **Waveform visualization** - See audio levels in real-time
|
||||
- **Multiple formats** - Export as WAV, MP3, or M4A
|
||||
|
||||
### How to Record
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Input">
|
||||
Choose your microphone or system audio
|
||||
</Step>
|
||||
<Step title="Start Recording">
|
||||
Click the record button and speak clearly
|
||||
</Step>
|
||||
<Step title="Stop & Save">
|
||||
Click stop when finished
|
||||
</Step>
|
||||
<Step title="Use or Export">
|
||||
Use as voice sample or export to file
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Transcription
|
||||
|
||||
Automatic speech-to-text powered by OpenAI's Whisper model.
|
||||
|
||||
### Features
|
||||
|
||||
- **High accuracy** - Industry-leading speech recognition
|
||||
- **Multiple languages** - Supports 50+ languages
|
||||
- **Automatic detection** - Language auto-detection
|
||||
- **Timestamps** - Word-level timing information
|
||||
|
||||
### How to Transcribe
|
||||
|
||||
<Steps>
|
||||
<Step title="Select Audio">
|
||||
Choose a recording or upload an audio file
|
||||
</Step>
|
||||
<Step title="Choose Language">
|
||||
Select language or use auto-detect
|
||||
</Step>
|
||||
<Step title="Transcribe">
|
||||
Click transcribe and wait for processing
|
||||
</Step>
|
||||
<Step title="Review & Export">
|
||||
Review text and export as needed
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
<Tip>
|
||||
Transcription is useful for creating voice samples from existing audio or generating subtitles.
|
||||
</Tip>
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: "Remote Mode"
|
||||
description: "Connect to a GPU server for faster generation"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Remote Mode allows you to run the Voicebox backend on a separate machine (like a GPU server) while using the desktop app on your local machine.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **No local GPU** - Use a cloud GPU or remote workstation
|
||||
- **Faster generation** - Leverage powerful remote hardware
|
||||
- **Shared infrastructure** - Multiple users connect to one server
|
||||
- **Laptop workflows** - Keep your laptop cool and battery-efficient
|
||||
|
||||
## Architecture
|
||||
|
||||
In Remote Mode, the Voicebox desktop app (running on your local machine) communicates with the backend server (running on a remote machine) via HTTP. The local app provides only the user interface, while the remote server handles all the heavy processing including the TTS models, API endpoints, and audio generation.
|
||||
|
||||
## Setting Up Remote Mode
|
||||
|
||||
### On the Server
|
||||
|
||||
<Steps>
|
||||
<Step title="Install Dependencies">
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/jamiepine/voicebox.git
|
||||
cd voicebox/backend
|
||||
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Start the Server">
|
||||
```bash
|
||||
# Allow external connections
|
||||
uvicorn main:app --host 0.0.0.0 --port 17493
|
||||
```
|
||||
|
||||
<Warning>
|
||||
This exposes the server to your network. Use a firewall or VPN for security.
|
||||
</Warning>
|
||||
</Step>
|
||||
|
||||
<Step title="Open Firewall">
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo ufw allow 17493
|
||||
|
||||
# Or use your cloud provider's firewall settings
|
||||
```
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### On the Client
|
||||
|
||||
<Steps>
|
||||
<Step title="Open Settings">
|
||||
In Voicebox, go to **Settings → Server**
|
||||
</Step>
|
||||
|
||||
<Step title="Enable Remote Mode">
|
||||
Toggle **Use Remote Server**
|
||||
</Step>
|
||||
|
||||
<Step title="Enter Server URL">
|
||||
```
|
||||
http://<server-ip>:17493
|
||||
```
|
||||
|
||||
Replace `<server-ip>` with your server's IP address
|
||||
</Step>
|
||||
|
||||
<Step title="Test Connection">
|
||||
Click **Test Connection** to verify
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
```bash
|
||||
# Launch a GPU instance (e.g., g4dn.xlarge)
|
||||
# Install dependencies
|
||||
# Start server with --host 0.0.0.0
|
||||
```
|
||||
|
||||
### Vast.ai
|
||||
|
||||
```bash
|
||||
# Rent a GPU instance
|
||||
# SSH in and clone repo
|
||||
# Start server
|
||||
```
|
||||
|
||||
### RunPod
|
||||
|
||||
```bash
|
||||
# Deploy a pod with CUDA support
|
||||
# Install Voicebox backend
|
||||
# Expose port 17493
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
<Warning>
|
||||
The API currently has no authentication. Only use on trusted networks or with a VPN.
|
||||
</Warning>
|
||||
|
||||
**Best Practices:**
|
||||
- Use a VPN (WireGuard, Tailscale) instead of exposing to the internet
|
||||
- Run behind a reverse proxy with authentication (nginx + basic auth)
|
||||
- Use HTTPS with SSL certificates
|
||||
- Firewall rules to limit access to specific IPs
|
||||
|
||||
## Performance
|
||||
|
||||
Expected performance on various GPUs:
|
||||
|
||||
| GPU | Generation Speed |
|
||||
|-----|------------------|
|
||||
| RTX 4090 | ~2-3s per 10 words |
|
||||
| RTX 3090 | ~3-4s per 10 words |
|
||||
| RTX 3060 | ~5-7s per 10 words |
|
||||
| CPU (12-core) | ~20-30s per 10 words |
|
||||
|
||||
<Tip>
|
||||
A GPU with 8GB+ VRAM is recommended for best performance.
|
||||
</Tip>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See the [Troubleshooting Guide](/guides/troubleshooting#remote-mode-issues) for common remote mode issues.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: "Stories Editor"
|
||||
description: "Create multi-voice narratives with a timeline-based editor"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Stories Editor is a DAW-like timeline interface for creating multi-voice narratives, podcasts, and conversations.
|
||||
|
||||
## Features
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Multi-Track Timeline" icon="timeline">
|
||||
Arrange multiple voice tracks in parallel
|
||||
</Card>
|
||||
<Card title="Inline Editing" icon="scissors">
|
||||
Trim and split clips directly in the timeline
|
||||
</Card>
|
||||
<Card title="Auto-Playback" icon="play">
|
||||
Preview with synchronized playhead
|
||||
</Card>
|
||||
<Card title="Voice Mixing" icon="users">
|
||||
Build conversations with multiple speakers
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Creating a Story
|
||||
|
||||
<Steps>
|
||||
<Step title="Create New Story">
|
||||
Navigate to **Stories** and click **+ New Story**
|
||||
</Step>
|
||||
<Step title="Add Tracks">
|
||||
Create separate tracks for each voice/speaker
|
||||
</Step>
|
||||
<Step title="Add Clips">
|
||||
- Drag from generation history
|
||||
- Generate new clips inline
|
||||
- Upload audio files
|
||||
</Step>
|
||||
<Step title="Arrange & Edit">
|
||||
- Position clips on timeline
|
||||
- Trim clip edges
|
||||
- Adjust spacing and timing
|
||||
</Step>
|
||||
<Step title="Export">
|
||||
Render the final mixed audio
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Podcasts**: Multi-host conversations
|
||||
- **Audiobooks**: Narrator + character voices
|
||||
- **Game Dialogue**: Character interactions
|
||||
- **Video Voiceovers**: Multiple speakers
|
||||
- **Audio Drama**: Full voice casts
|
||||
|
||||
## Coming Soon
|
||||
|
||||
- Word-level editing
|
||||
- Crossfades and transitions
|
||||
- Audio effects (reverb, EQ)
|
||||
- Real-time collaboration
|
||||
@@ -0,0 +1,477 @@
|
||||
---
|
||||
title: "Troubleshooting"
|
||||
description: "Common issues and solutions for Voicebox"
|
||||
---
|
||||
|
||||
This guide covers common issues you might encounter when using or developing Voicebox, along with solutions.
|
||||
|
||||
## Installation Issues
|
||||
|
||||
### macOS: "App is damaged and can't be opened"
|
||||
|
||||
This occurs because the app isn't signed with an Apple Developer certificate.
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Remove the quarantine attribute
|
||||
xattr -cr /Applications/Voicebox.app
|
||||
```
|
||||
|
||||
### Windows: SmartScreen Warning
|
||||
|
||||
Windows SmartScreen may warn that the app is unrecognized.
|
||||
|
||||
**Solution:**
|
||||
- Click "More info"
|
||||
- Click "Run anyway"
|
||||
|
||||
<Note>
|
||||
This is expected for unsigned applications. We're working on code signing for future releases.
|
||||
</Note>
|
||||
|
||||
## Server Issues
|
||||
|
||||
### Backend Server Won't Start
|
||||
|
||||
**Symptoms:**
|
||||
- Red status indicator in bottom-left corner
|
||||
- "Failed to connect to server" error
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Port Already in Use">
|
||||
Check if port 17493 is already in use:
|
||||
|
||||
```bash
|
||||
# macOS/Linux
|
||||
lsof -i :17493
|
||||
|
||||
# Windows
|
||||
netstat -ano | findstr :17493
|
||||
```
|
||||
|
||||
Kill the process using the port:
|
||||
```bash
|
||||
# macOS/Linux
|
||||
kill -9 <PID>
|
||||
|
||||
# Windows
|
||||
taskkill /PID <PID> /F
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Permission Issues">
|
||||
The server binary might not have execute permissions:
|
||||
|
||||
```bash
|
||||
# macOS/Linux
|
||||
chmod +x ~/Library/Application\ Support/com.voicebox.app/backend/voicebox-server
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Check Logs">
|
||||
View server logs for errors:
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
```bash
|
||||
type %APPDATA%\com.voicebox.app\logs\server.log
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### Connection Timeout
|
||||
|
||||
**Symptoms:**
|
||||
- Long loading times
|
||||
- "Connection timeout" errors
|
||||
|
||||
**Solution:**
|
||||
- Restart the app
|
||||
- Check your firewall settings
|
||||
- Ensure localhost is accessible
|
||||
|
||||
## Generation Issues
|
||||
|
||||
### First Generation is Very Slow
|
||||
|
||||
**Symptoms:**
|
||||
- First generation takes 2-5 minutes
|
||||
- Progress indicator stuck at "Loading model..."
|
||||
|
||||
**Explanation:**
|
||||
This is expected behavior. The first generation downloads the Qwen3-TTS model (~2-4GB) and initializes it.
|
||||
|
||||
**Solution:**
|
||||
- Wait for the initial download to complete
|
||||
- Subsequent generations will be much faster
|
||||
- Check your internet connection
|
||||
|
||||
### Poor Voice Quality
|
||||
|
||||
**Symptoms:**
|
||||
- Robotic or unnatural voice
|
||||
- Missing emotion or prosody
|
||||
- Pronunciation errors
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<Steps>
|
||||
<Step title="Improve Voice Samples">
|
||||
- Use 10-30 seconds of clear audio
|
||||
- Avoid background noise
|
||||
- Ensure consistent speaking tone
|
||||
- Add multiple samples from the same speaker
|
||||
</Step>
|
||||
|
||||
<Step title="Match Speaking Style">
|
||||
The generated voice will mimic the tone and style of your samples. If your sample is monotone, the generation will be too.
|
||||
</Step>
|
||||
|
||||
<Step title="Adjust Text Formatting">
|
||||
- Use proper punctuation
|
||||
- Add commas for natural pauses
|
||||
- Capitalize proper nouns
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Generation Fails with "Out of Memory"
|
||||
|
||||
**Symptoms:**
|
||||
- Generation crashes
|
||||
- "CUDA out of memory" or "RuntimeError: out of memory"
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Free GPU Memory">
|
||||
Close other GPU-intensive applications:
|
||||
- Games
|
||||
- Video editors
|
||||
- Multiple browser tabs with WebGL
|
||||
|
||||
Then restart Voicebox.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Use CPU Mode">
|
||||
If your GPU doesn't have enough VRAM (need 6GB+), use CPU mode:
|
||||
|
||||
Settings → Generation → Use CPU instead of GPU
|
||||
|
||||
<Warning>
|
||||
CPU generation is 5-10x slower but uses system RAM instead of VRAM.
|
||||
</Warning>
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Reduce Batch Size">
|
||||
For long text, split it into smaller chunks instead of generating all at once.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Audio Issues
|
||||
|
||||
### No Audio Playback
|
||||
|
||||
**Symptoms:**
|
||||
- Generated audio won't play
|
||||
- Playback button doesn't respond
|
||||
|
||||
**Solutions:**
|
||||
- Check system audio settings
|
||||
- Ensure audio output device is connected
|
||||
- Try exporting and playing in a media player
|
||||
|
||||
### Crackling or Distorted Audio
|
||||
|
||||
**Symptoms:**
|
||||
- Audio has static or distortion
|
||||
- Clipping sounds
|
||||
|
||||
**Solutions:**
|
||||
- Check if your input samples have distortion
|
||||
- Reduce playback volume
|
||||
- Re-generate with cleaner voice samples
|
||||
|
||||
## Development Issues
|
||||
|
||||
### Backend Won't Start in Dev Mode
|
||||
|
||||
**Symptoms:**
|
||||
- `bun run dev:server` fails
|
||||
- Import errors or module not found
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Python Version">
|
||||
Ensure Python 3.11 or higher:
|
||||
|
||||
```bash
|
||||
python --version
|
||||
```
|
||||
|
||||
If not, install Python 3.11+ and recreate the virtual environment.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Virtual Environment">
|
||||
Ensure venv is activated:
|
||||
|
||||
```bash
|
||||
# macOS/Linux
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Windows
|
||||
backend\venv\Scripts\activate
|
||||
```
|
||||
|
||||
You should see `(venv)` in your prompt.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Dependencies">
|
||||
Reinstall dependencies:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
pip install -r requirements.txt
|
||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### Tauri Build Fails
|
||||
|
||||
**Symptoms:**
|
||||
- `bun run tauri build` fails
|
||||
- Rust compilation errors
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Clean build artifacts
|
||||
cd tauri/src-tauri
|
||||
cargo clean
|
||||
|
||||
# Update Rust
|
||||
rustup update
|
||||
|
||||
# Try building again
|
||||
cd ../..
|
||||
bun run tauri build
|
||||
```
|
||||
|
||||
### OpenAPI Client Generation Fails
|
||||
|
||||
**Symptoms:**
|
||||
- `./scripts/generate-api.sh` fails
|
||||
- "Failed to fetch schema" error
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<Steps>
|
||||
<Step title="Ensure Backend is Running">
|
||||
```bash
|
||||
curl http://localhost:17493/openapi.json
|
||||
```
|
||||
|
||||
Should return JSON. If not, start the backend.
|
||||
</Step>
|
||||
|
||||
<Step title="Check Port">
|
||||
Ensure nothing else is using port 17493
|
||||
</Step>
|
||||
|
||||
<Step title="Regenerate Manually">
|
||||
```bash
|
||||
cd backend
|
||||
source venv/bin/activate
|
||||
uvicorn main:app --reload --port 17493
|
||||
|
||||
# In another terminal
|
||||
./scripts/generate-api.sh
|
||||
```
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Database Issues
|
||||
|
||||
### "Database is locked" Error
|
||||
|
||||
**Symptoms:**
|
||||
- Profile or generation operations fail
|
||||
- SQLite lock errors
|
||||
|
||||
**Solutions:**
|
||||
- Close all Voicebox instances
|
||||
- Delete the lock file:
|
||||
```bash
|
||||
# macOS
|
||||
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-shm
|
||||
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-wal
|
||||
```
|
||||
|
||||
### Corrupted Database
|
||||
|
||||
**Symptoms:**
|
||||
- App crashes on launch
|
||||
- Data missing or corrupted
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<Warning>
|
||||
This will delete all your voice profiles and generation history. Export important profiles first if possible.
|
||||
</Warning>
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db
|
||||
|
||||
# Windows
|
||||
del %APPDATA%\com.voicebox.app\data\voicebox.db
|
||||
```
|
||||
|
||||
Restart the app to create a fresh database.
|
||||
|
||||
## Model Issues
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
**Symptoms:**
|
||||
- "Failed to download model" error
|
||||
- Stuck at "Downloading..."
|
||||
|
||||
**Solutions:**
|
||||
- Check your internet connection
|
||||
- Check HuggingFace Hub status
|
||||
- Try using a VPN if HuggingFace is blocked in your region
|
||||
- Manually download and place in cache directory
|
||||
|
||||
### Wrong Model Version
|
||||
|
||||
**Symptoms:**
|
||||
- Generation quality suddenly degraded
|
||||
- Different voice output
|
||||
|
||||
**Solutions:**
|
||||
Clear the model cache and re-download:
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
rm -rf ~/.cache/huggingface/hub/models--Qwen*
|
||||
|
||||
# Windows
|
||||
rmdir /s %USERPROFILE%\.cache\huggingface\hub\models--Qwen*
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Slow Generation on GPU
|
||||
|
||||
**Symptoms:**
|
||||
- Generation slower than expected
|
||||
- GPU not being utilized
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Verify CUDA Installation">
|
||||
```bash
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
Should show your GPU. If not, install CUDA drivers.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Check GPU Selection">
|
||||
If you have multiple GPUs, ensure Voicebox is using the right one.
|
||||
|
||||
Settings → Generation → GPU Device
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Update GPU Drivers">
|
||||
Outdated drivers can cause performance issues. Update to the latest NVIDIA drivers.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
**Symptoms:**
|
||||
- App uses excessive RAM
|
||||
- System becomes sluggish
|
||||
|
||||
**Solutions:**
|
||||
- Close unused voice profiles
|
||||
- Clear generation history
|
||||
- Restart the app periodically
|
||||
|
||||
## Remote Mode Issues
|
||||
|
||||
### Can't Connect to Remote Server
|
||||
|
||||
**Symptoms:**
|
||||
- "Connection refused" error
|
||||
- Remote server not found
|
||||
|
||||
**Solutions:**
|
||||
|
||||
<Steps>
|
||||
<Step title="Check Server Status">
|
||||
Ensure the remote server is running:
|
||||
|
||||
```bash
|
||||
curl http://<server-ip>:17493/health
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Check Firewall">
|
||||
Ensure port 17493 is open on the remote server:
|
||||
|
||||
```bash
|
||||
# Allow port on Ubuntu/Debian
|
||||
sudo ufw allow 17493
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Verify Network">
|
||||
- Ensure both machines are on the same network (for local servers)
|
||||
- Use IP address instead of hostname
|
||||
- Try pinging the server: `ping <server-ip>`
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Still Having Issues?
|
||||
|
||||
If you're still experiencing problems:
|
||||
|
||||
1. **Check GitHub Issues:** [github.com/jamiepine/voicebox/issues](https://github.com/jamiepine/voicebox/issues)
|
||||
2. **Open a New Issue:** Provide:
|
||||
- Operating system and version
|
||||
- Voicebox version
|
||||
- Steps to reproduce
|
||||
- Error messages or logs
|
||||
3. **Join Discord:** [discord.gg/voicebox](https://discord.gg/voicebox) (coming soon)
|
||||
|
||||
## Diagnostic Information
|
||||
|
||||
When reporting issues, include this information:
|
||||
|
||||
```bash
|
||||
# Voicebox version
|
||||
# Check Help → About in the app
|
||||
|
||||
# Operating system
|
||||
uname -a # macOS/Linux
|
||||
systeminfo # Windows
|
||||
|
||||
# Python version (for dev issues)
|
||||
python --version
|
||||
|
||||
# GPU info (if generation issues)
|
||||
nvidia-smi # NVIDIA GPUs
|
||||
```
|
||||
|
||||
For more detailed troubleshooting, see the [TROUBLESHOOTING.md](https://github.com/jamiepine/voicebox/blob/main/docs/TROUBLESHOOTING.md) file in the repository.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: "Voice Cloning"
|
||||
description: "Clone any voice from just a few seconds of audio"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning from just a few seconds of audio. The model captures prosody, emotion, and natural cadence.
|
||||
|
||||
## How It Works
|
||||
|
||||
<Steps>
|
||||
<Step title="Upload or Record Sample">
|
||||
Provide 10-30 seconds of clear speech from the target voice
|
||||
</Step>
|
||||
<Step title="Model Analysis">
|
||||
Qwen3-TTS analyzes vocal characteristics, tone, and speaking patterns
|
||||
</Step>
|
||||
<Step title="Voice Profile Created">
|
||||
The model generates a voice embedding for synthesis
|
||||
</Step>
|
||||
<Step title="Generate Speech">
|
||||
Use the profile to generate any text in the cloned voice
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Sample Quality
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Do" icon="check">
|
||||
- Use 10-30 seconds of audio
|
||||
- Clear, consistent speaking
|
||||
- Minimal background noise
|
||||
- Natural speaking pace
|
||||
</Card>
|
||||
<Card title="Don't" icon="xmark">
|
||||
- Very short clips (< 5 seconds)
|
||||
- Heavy background noise
|
||||
- Music or overlapping voices
|
||||
- Heavily processed audio
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### Multiple Samples
|
||||
|
||||
Adding multiple samples from the same speaker can improve quality:
|
||||
|
||||
- Different speaking styles (casual, formal)
|
||||
- Different emotions (happy, serious)
|
||||
- Different recording conditions
|
||||
|
||||
<Tip>
|
||||
The model will learn a more robust representation from diverse samples.
|
||||
</Tip>
|
||||
|
||||
## Supported Languages
|
||||
|
||||
Currently supported:
|
||||
- English
|
||||
- Chinese (Mandarin)
|
||||
|
||||
More languages coming soon.
|
||||
|
||||
## Limitations
|
||||
|
||||
<Warning>
|
||||
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice.
|
||||
</Warning>
|
||||
|
||||
- Quality depends on sample clarity
|
||||
- Works best with consistent speaking tone
|
||||
- May struggle with extreme accents or speech impediments
|
||||
- Background noise reduces quality
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "voicebox-docs",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "mintlify dev",
|
||||
"build": "mintlify build",
|
||||
"start": "mintlify serve",
|
||||
"install:mintlify": "bun add -g mintlify"
|
||||
},
|
||||
"devDependencies": {
|
||||
"mintlify": "latest"
|
||||
},
|
||||
"engines": {
|
||||
"bun": ">=1.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,758 @@
|
||||
# Docker Deployment Guide
|
||||
|
||||
**Status:** In Development for v0.2.0
|
||||
**Requested By:** Reddit community ([thread](https://reddit.com/r/LocalLLaMA/...))
|
||||
|
||||
## Overview
|
||||
|
||||
Docker support makes Voicebox easier to deploy, especially for:
|
||||
|
||||
- **Consistent Environments**: Same setup across dev/staging/prod
|
||||
- **GPU Passthrough**: Easy NVIDIA/AMD GPU access
|
||||
- **Server Deployments**: Run on headless Linux servers
|
||||
- **Multi-User Setups**: Isolate instances per user/team
|
||||
- **Cloud Platforms**: Deploy to AWS, GCP, Azure, DigitalOcean
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Pre-Built Images (Recommended)
|
||||
|
||||
```bash
|
||||
# CPU-only version
|
||||
docker run -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest
|
||||
|
||||
# NVIDIA GPU version
|
||||
docker run --gpus all -p 8000:8000 -v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
|
||||
# AMD GPU version (experimental)
|
||||
docker run --device=/dev/kfd --device=/dev/dri -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
ghcr.io/jamiepine/voicebox:latest-rocm
|
||||
```
|
||||
|
||||
Then open: `http://localhost:8000`
|
||||
|
||||
### Using Docker Compose (Easiest)
|
||||
|
||||
Create `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- GPU_MEMORY_FRACTION=0.8 # Use 80% of GPU memory
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Building From Source
|
||||
|
||||
### Basic Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git \
|
||||
build-essential \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy application
|
||||
COPY backend/ /app/backend/
|
||||
COPY requirements.txt /app/
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN pip install --no-cache-dir git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8000
|
||||
|
||||
# Run server
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build and run:
|
||||
```bash
|
||||
docker build -t voicebox .
|
||||
docker run -p 8000:8000 -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Multi-Stage Build (Optimized)
|
||||
|
||||
Smaller image size by separating build and runtime:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.optimized
|
||||
# Stage 1: Build dependencies
|
||||
FROM python:3.11-slim AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
git build-essential && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
-r requirements.txt
|
||||
|
||||
RUN pip install --no-cache-dir --target=/build/packages \
|
||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Stage 2: Runtime
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install only runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy installed packages from builder
|
||||
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages/
|
||||
|
||||
# Copy application code
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
Build:
|
||||
```bash
|
||||
docker build -f Dockerfile.optimized -t voicebox:slim .
|
||||
```
|
||||
|
||||
## GPU Support
|
||||
|
||||
### NVIDIA GPUs (CUDA)
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with CUDA support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with GPU:**
|
||||
```bash
|
||||
docker run --gpus all -p 8000:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
voicebox:cuda
|
||||
```
|
||||
|
||||
**Docker Compose with GPU:**
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
image: voicebox:cuda
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
### AMD GPUs (ROCm) - Experimental
|
||||
|
||||
**Dockerfile:**
|
||||
```dockerfile
|
||||
FROM rocm/dev-ubuntu-22.04:6.0
|
||||
|
||||
# Install Python
|
||||
RUN apt-get update && apt-get install -y \
|
||||
python3.11 python3-pip git ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyTorch with ROCm support
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
|
||||
|
||||
# Install other dependencies
|
||||
RUN pip3 install -r requirements.txt
|
||||
RUN pip3 install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# Set ROCm environment variables
|
||||
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
**Run with AMD GPU:**
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video --ipc=host --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-p 8000:8000 -v voicebox-data:/app/data \
|
||||
voicebox:rocm
|
||||
```
|
||||
|
||||
**Note:** ROCm support varies by GPU model. Works best on Linux. See [AMD ROCm docs](https://rocm.docs.amd.com) for compatibility.
|
||||
|
||||
## Volume Mounts
|
||||
|
||||
### Essential Volumes
|
||||
|
||||
```bash
|
||||
docker run -v voicebox-data:/app/data \ # Profiles, generations, history
|
||||
-v huggingface-cache:/root/.cache/huggingface \ # Downloaded models
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Development Volume Mounts
|
||||
|
||||
For development with hot-reload:
|
||||
|
||||
```bash
|
||||
docker run -v $(pwd)/backend:/app/backend \ # Live code changes
|
||||
-v voicebox-data:/app/data \
|
||||
-e RELOAD=true \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Custom Model Storage
|
||||
|
||||
Use external model directory:
|
||||
|
||||
```bash
|
||||
docker run -v /path/to/models:/models \
|
||||
-e MODELS_DIR=/models \
|
||||
-v voicebox-data:/app/data \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Configure Voicebox via environment variables:
|
||||
|
||||
```bash
|
||||
docker run -e TTS_MODE=local \
|
||||
-e WHISPER_MODE=openai-api \
|
||||
-e OPENAI_API_KEY=sk-... \
|
||||
-e GPU_MEMORY_FRACTION=0.8 \
|
||||
-e LOG_LEVEL=info \
|
||||
-p 8000:8000 voicebox
|
||||
```
|
||||
|
||||
### Available Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `TTS_MODE` | `local` | TTS provider: `local`, `remote` |
|
||||
| `TTS_REMOTE_URL` | - | URL for remote TTS server |
|
||||
| `WHISPER_MODE` | `local` | Whisper provider: `local`, `openai-api`, `remote` |
|
||||
| `WHISPER_REMOTE_URL` | - | URL for remote Whisper server |
|
||||
| `OPENAI_API_KEY` | - | OpenAI API key (if using OpenAI Whisper) |
|
||||
| `GPU_MEMORY_FRACTION` | `0.9` | Fraction of GPU memory to use (0.0-1.0) |
|
||||
| `DATA_DIR` | `/app/data` | Directory for profiles/generations |
|
||||
| `MODELS_DIR` | `/app/models` | Directory for local models |
|
||||
| `LOG_LEVEL` | `info` | Logging level: `debug`, `info`, `warning`, `error` |
|
||||
| `RELOAD` | `false` | Enable hot-reload for development |
|
||||
|
||||
## Complete Docker Compose Examples
|
||||
|
||||
### Production Deployment
|
||||
|
||||
```yaml
|
||||
# docker-compose.prod.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
container_name: voicebox
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
- GPU_MEMORY_FRACTION=0.8
|
||||
- LOG_LEVEL=info
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
driver: local
|
||||
huggingface-cache:
|
||||
driver: local
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### Development Setup
|
||||
|
||||
```yaml
|
||||
# docker-compose.dev.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
voicebox:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./backend:/app/backend:ro
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- RELOAD=true
|
||||
- LOG_LEVEL=debug
|
||||
- TTS_MODE=local
|
||||
command: uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
```
|
||||
|
||||
### Multi-Service Stack
|
||||
|
||||
Full stack with reverse proxy and monitoring:
|
||||
|
||||
```yaml
|
||||
# docker-compose.stack.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# Main Voicebox app
|
||||
voicebox:
|
||||
image: ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- voicebox-data:/app/data
|
||||
- huggingface-cache:/root/.cache/huggingface
|
||||
environment:
|
||||
- TTS_MODE=local
|
||||
- WHISPER_MODE=local
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
|
||||
# Nginx reverse proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
depends_on:
|
||||
- voicebox
|
||||
|
||||
# Prometheus monitoring (optional)
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
- prometheus-data:/prometheus
|
||||
|
||||
volumes:
|
||||
voicebox-data:
|
||||
huggingface-cache:
|
||||
prometheus-data:
|
||||
```
|
||||
|
||||
## Cloud Deployment
|
||||
|
||||
### AWS EC2
|
||||
|
||||
1. **Launch GPU Instance** (g4dn.xlarge or p3.2xlarge)
|
||||
2. **Install Docker + nvidia-docker:**
|
||||
```bash
|
||||
# Amazon Linux 2
|
||||
sudo yum install -y docker
|
||||
sudo systemctl start docker
|
||||
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
|
||||
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
|
||||
sudo apt-get update && sudo apt-get install -y nvidia-docker2
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
3. **Deploy:**
|
||||
```bash
|
||||
docker run --gpus all -d -p 80:8000 \
|
||||
-v voicebox-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/jamiepine/voicebox:latest-cuda
|
||||
```
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
Use GPU Droplet + Docker:
|
||||
|
||||
```bash
|
||||
# Create droplet via CLI
|
||||
doctl compute droplet create voicebox \
|
||||
--size gpu-h100x1-80gb \
|
||||
--image ubuntu-22-04-x64 \
|
||||
--region nyc3
|
||||
|
||||
# SSH and deploy
|
||||
ssh root@<droplet-ip>
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
docker run --gpus all -d -p 80:8000 voicebox:cuda
|
||||
```
|
||||
|
||||
### Google Cloud Run (CPU-only)
|
||||
|
||||
```bash
|
||||
# Build and push
|
||||
docker build -t gcr.io/your-project/voicebox .
|
||||
docker push gcr.io/your-project/voicebox
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy voicebox \
|
||||
--image gcr.io/your-project/voicebox \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--memory 4Gi \
|
||||
--cpu 2 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Fly.io
|
||||
|
||||
Create `fly.toml`:
|
||||
```toml
|
||||
app = "voicebox"
|
||||
|
||||
[build]
|
||||
image = "ghcr.io/jamiepine/voicebox:latest"
|
||||
|
||||
[[services]]
|
||||
http_checks = []
|
||||
internal_port = 8000
|
||||
protocol = "tcp"
|
||||
|
||||
[[services.ports]]
|
||||
port = 80
|
||||
handlers = ["http"]
|
||||
|
||||
[[services.ports]]
|
||||
port = 443
|
||||
handlers = ["tls", "http"]
|
||||
|
||||
[mounts]
|
||||
source = "voicebox_data"
|
||||
destination = "/app/data"
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
fly launch
|
||||
fly deploy
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### GPU Not Detected
|
||||
|
||||
**Check NVIDIA Docker:**
|
||||
```bash
|
||||
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
If this fails, reinstall nvidia-docker2.
|
||||
|
||||
**Check AMD ROCm:**
|
||||
```bash
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri rocm/dev-ubuntu-22.04:6.0 rocminfo
|
||||
```
|
||||
|
||||
### Permission Errors
|
||||
|
||||
Container can't write to volumes:
|
||||
```bash
|
||||
# Fix permissions
|
||||
docker run --user $(id -u):$(id -g) -v $(pwd)/data:/app/data voicebox
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
Reduce GPU memory usage:
|
||||
```bash
|
||||
docker run -e GPU_MEMORY_FRACTION=0.5 voicebox
|
||||
```
|
||||
|
||||
Or use CPU-only:
|
||||
```bash
|
||||
docker run -e DEVICE=cpu voicebox
|
||||
```
|
||||
|
||||
### Model Download Fails
|
||||
|
||||
Ensure HuggingFace cache is writable:
|
||||
```bash
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
Or use host cache:
|
||||
```bash
|
||||
docker run -v ~/.cache/huggingface:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
Change host port:
|
||||
```bash
|
||||
docker run -p 8080:8000 voicebox # Use port 8080 instead
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Don't Run as Root
|
||||
|
||||
Create non-root user in Dockerfile:
|
||||
```dockerfile
|
||||
RUN useradd -m -u 1000 voicebox
|
||||
USER voicebox
|
||||
```
|
||||
|
||||
### 2. Use Secrets for API Keys
|
||||
|
||||
Don't put API keys in docker-compose.yml:
|
||||
|
||||
```bash
|
||||
# Use Docker secrets
|
||||
echo "sk-your-key" | docker secret create openai_key -
|
||||
|
||||
docker service create \
|
||||
--secret openai_key \
|
||||
-e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
|
||||
voicebox
|
||||
```
|
||||
|
||||
### 3. Network Isolation
|
||||
|
||||
Use internal networks for multi-container setups:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
networks:
|
||||
- internal
|
||||
nginx:
|
||||
networks:
|
||||
- internal
|
||||
- external
|
||||
ports:
|
||||
- "80:80"
|
||||
|
||||
networks:
|
||||
internal:
|
||||
internal: true
|
||||
external:
|
||||
```
|
||||
|
||||
### 4. Resource Limits
|
||||
|
||||
Prevent resource exhaustion:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
voicebox:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '4'
|
||||
memory: 8G
|
||||
reservations:
|
||||
cpus: '2'
|
||||
memory: 4G
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### GPU Memory Management
|
||||
|
||||
```bash
|
||||
# Use 80% of GPU (default 90%)
|
||||
docker run -e GPU_MEMORY_FRACTION=0.8 voicebox
|
||||
|
||||
# Allow GPU memory growth (prevents OOM)
|
||||
docker run -e TF_FORCE_GPU_ALLOW_GROWTH=true voicebox
|
||||
```
|
||||
|
||||
### Model Caching
|
||||
|
||||
Pre-download models to volume:
|
||||
|
||||
```bash
|
||||
# Download models first
|
||||
docker run --rm -v huggingface-cache:/root/.cache/huggingface \
|
||||
voicebox python -c "
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
WhisperProcessor.from_pretrained('openai/whisper-base')
|
||||
WhisperForConditionalGeneration.from_pretrained('openai/whisper-base')
|
||||
"
|
||||
|
||||
# Then run normally
|
||||
docker run -v huggingface-cache:/root/.cache/huggingface voicebox
|
||||
```
|
||||
|
||||
### Multi-Worker Setup
|
||||
|
||||
Use uvicorn workers for better throughput:
|
||||
|
||||
```dockerfile
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
Built-in health endpoint:
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
Docker health check:
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Add metrics exporter:
|
||||
```python
|
||||
# backend/main.py
|
||||
from prometheus_fastapi_instrumentator import Instrumentator
|
||||
|
||||
Instrumentator().instrument(app).expose(app)
|
||||
```
|
||||
|
||||
Then scrape `/metrics` with Prometheus.
|
||||
|
||||
### Logs
|
||||
|
||||
View container logs:
|
||||
```bash
|
||||
docker logs -f voicebox
|
||||
|
||||
# Or with compose
|
||||
docker compose logs -f voicebox
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] Publish official images to GitHub Container Registry
|
||||
- [ ] Add Kubernetes Helm charts
|
||||
- [ ] Create Docker Desktop extension
|
||||
- [ ] Add automated vulnerability scanning
|
||||
- [ ] Support ARM64 builds for Raspberry Pi / Apple Silicon
|
||||
|
||||
## Contributing
|
||||
|
||||
Help improve Docker support:
|
||||
1. Test on different platforms (AMD GPU, ARM64, etc.)
|
||||
2. Submit Dockerfile optimizations
|
||||
3. Share deployment configurations
|
||||
4. Report issues: [GitHub Issues](https://github.com/jamiepine/voicebox/issues)
|
||||
|
||||
## Resources
|
||||
|
||||
- [Docker Documentation](https://docs.docker.com)
|
||||
- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker)
|
||||
- [AMD ROCm Docker](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html)
|
||||
- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/)
|
||||
@@ -0,0 +1,435 @@
|
||||
# External Provider Support
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
**Discussion:** [Reddit Thread](https://reddit.com/r/LocalLLaMA/...)
|
||||
|
||||
## Overview
|
||||
|
||||
External provider support allows you to connect Voicebox to remotely-hosted TTS and Whisper services instead of running models locally. This is useful for:
|
||||
|
||||
- **Existing GPU Infrastructure**: You already have Qwen3-TTS running on a GPU server
|
||||
- **AMD GPU Users**: Run models on your AMD hardware, use Voicebox as the UI
|
||||
- **Cloud Deployments**: Host models on Modal, Replicate, RunPod, etc.
|
||||
- **Team Sharing**: Multiple users share one GPU server running models
|
||||
- **Mixed Deployments**: Local Whisper + remote TTS, or vice versa
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ HTTP/API ┌──────────────────┐
|
||||
│ Voicebox UI │ ───────────────────────> │ Your TTS Server │
|
||||
│ + Backend │ │ (Qwen3-TTS on │
|
||||
│ │ <─────────────────────── │ AMD/NVIDIA GPU)│
|
||||
│ - Profiles │ Audio + Metadata └──────────────────┘
|
||||
│ - History │
|
||||
│ - Audio Edit │ HTTP/API ┌──────────────────┐
|
||||
│ - UI │ ───────────────────────> │ Whisper Service │
|
||||
└─────────────────┘ │ (OpenAI API or │
|
||||
│ self-hosted) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**What Voicebox Still Handles:**
|
||||
- Voice profile management
|
||||
- Generation history
|
||||
- Audio trimming/editing
|
||||
- Multi-track story editor
|
||||
- UI/UX layer
|
||||
|
||||
**What External Providers Handle:**
|
||||
- Model inference (TTS generation, transcription)
|
||||
- GPU allocation
|
||||
- Model loading/caching
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# TTS Provider
|
||||
TTS_MODE=remote # local | remote
|
||||
TTS_REMOTE_URL=http://192.168.1.100:8000 # Your TTS server URL
|
||||
TTS_API_KEY=your-api-key # Optional authentication
|
||||
|
||||
# Whisper Provider
|
||||
WHISPER_MODE=openai-api # local | openai-api | remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000 # For self-hosted Whisper
|
||||
OPENAI_API_KEY=sk-... # For OpenAI Whisper API
|
||||
```
|
||||
|
||||
### Voicebox Config UI (Planned)
|
||||
|
||||
Settings page will include:
|
||||
- Provider selection dropdowns
|
||||
- URL/API key inputs
|
||||
- Connection test button
|
||||
- Latency/status indicators
|
||||
|
||||
## Hosting External Services
|
||||
|
||||
### Option 1: Simple FastAPI Server (Recommended)
|
||||
|
||||
Create a lightweight server to expose your local Qwen3-TTS model:
|
||||
|
||||
```python
|
||||
# tts_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
import numpy as np
|
||||
import base64
|
||||
|
||||
app = FastAPI()
|
||||
model = Qwen3TTSModel.from_pretrained(
|
||||
"Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
device_map="cuda" # or "cpu" for AMD ROCm: use torch+rocm
|
||||
)
|
||||
|
||||
@app.post("/v1/generate")
|
||||
async def generate(
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: int = None
|
||||
):
|
||||
"""Generate speech from text using voice prompt."""
|
||||
audio, sample_rate = model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
)
|
||||
|
||||
# Return as base64 for transport
|
||||
audio_bytes = audio.tobytes()
|
||||
return {
|
||||
"audio": base64.b64encode(audio_bytes).decode(),
|
||||
"sample_rate": sample_rate,
|
||||
"dtype": str(audio.dtype)
|
||||
}
|
||||
|
||||
@app.post("/v1/create_voice_prompt")
|
||||
async def create_voice_prompt(
|
||||
audio: UploadFile = File(...),
|
||||
reference_text: str = ""
|
||||
):
|
||||
"""Create voice prompt from reference audio."""
|
||||
# Save uploaded audio temporarily
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
# Create voice prompt
|
||||
voice_prompt = model.create_voice_clone_prompt(
|
||||
ref_audio=audio_path,
|
||||
ref_text=reference_text,
|
||||
)
|
||||
|
||||
return {"voice_prompt": voice_prompt}
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": str(model.device)
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
```
|
||||
|
||||
**Run it:**
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install fastapi uvicorn qwen-tts torch
|
||||
|
||||
# For AMD GPUs, use ROCm PyTorch:
|
||||
pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
|
||||
|
||||
# Start server
|
||||
python tts_server.py
|
||||
```
|
||||
|
||||
### Option 2: vLLM (If Supported)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
### Option 3: Cloud Platforms
|
||||
|
||||
**Modal.com Example:**
|
||||
```python
|
||||
import modal
|
||||
|
||||
app = modal.App("qwen-tts")
|
||||
image = modal.Image.debian_slim().pip_install("qwen-tts", "torch")
|
||||
|
||||
@app.function(gpu="A10G", image=image)
|
||||
@modal.web_endpoint(method="POST")
|
||||
def generate(text: str, voice_prompt: dict):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
|
||||
audio, sr = model.generate_voice_clone(text, voice_prompt)
|
||||
return {"audio": audio.tolist(), "sample_rate": sr}
|
||||
```
|
||||
|
||||
Deploy: `modal deploy tts_server.py`
|
||||
Get URL: `https://yourapp--generate.modal.run`
|
||||
|
||||
## API Specification
|
||||
|
||||
External TTS providers must implement these endpoints:
|
||||
|
||||
### `POST /v1/generate`
|
||||
|
||||
Generate speech from text.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"text": "Hello, this is a test.",
|
||||
"voice_prompt": { /* voice prompt object */ },
|
||||
"language": "en",
|
||||
"seed": 12345
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"audio": "base64-encoded-audio-bytes",
|
||||
"sample_rate": 24000,
|
||||
"dtype": "float32"
|
||||
}
|
||||
```
|
||||
|
||||
### `POST /v1/create_voice_prompt`
|
||||
|
||||
Create a voice prompt from reference audio.
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
- `audio`: Audio file upload
|
||||
- `reference_text`: Transcript of the audio
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"voice_prompt": { /* voice prompt object */ }
|
||||
}
|
||||
```
|
||||
|
||||
### `GET /health`
|
||||
|
||||
Health check endpoint.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"model": "Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"device": "cuda:0"
|
||||
}
|
||||
```
|
||||
|
||||
## Whisper External Providers
|
||||
|
||||
### OpenAI Whisper API
|
||||
|
||||
Simply set:
|
||||
```bash
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
Voicebox will use OpenAI's Whisper API automatically.
|
||||
|
||||
### Self-Hosted Whisper
|
||||
|
||||
Run your own Whisper server:
|
||||
|
||||
```python
|
||||
# whisper_server.py
|
||||
from fastapi import FastAPI, UploadFile, File
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
import librosa
|
||||
|
||||
app = FastAPI()
|
||||
processor = WhisperProcessor.from_pretrained("openai/whisper-base")
|
||||
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
|
||||
|
||||
@app.post("/v1/transcribe")
|
||||
async def transcribe(audio: UploadFile = File(...), language: str = None):
|
||||
# Load audio
|
||||
audio_path = f"/tmp/{audio.filename}"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await audio.read())
|
||||
|
||||
audio_data, sr = librosa.load(audio_path, sr=16000)
|
||||
|
||||
# Process
|
||||
inputs = processor(audio_data, sampling_rate=16000, return_tensors="pt")
|
||||
predicted_ids = model.generate(inputs["input_features"])
|
||||
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
|
||||
|
||||
return {"text": transcription}
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
```bash
|
||||
WHISPER_MODE=remote
|
||||
WHISPER_REMOTE_URL=http://localhost:9000
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. AMD GPU User with Existing Setup
|
||||
|
||||
**Scenario:** You have a Radeon 7900 XTX running Qwen3-TTS on Linux.
|
||||
|
||||
**Setup:**
|
||||
1. Run `tts_server.py` on your AMD box (ROCm PyTorch)
|
||||
2. Configure Voicebox: `TTS_MODE=remote`, `TTS_REMOTE_URL=http://amd-box:8000`
|
||||
3. Use Voicebox UI for profiles, generation, editing
|
||||
4. TTS happens on your AMD GPU
|
||||
|
||||
### 2. Team Deployment
|
||||
|
||||
**Scenario:** 5 team members, 1 GPU server.
|
||||
|
||||
**Setup:**
|
||||
1. Deploy TTS server on shared GPU box
|
||||
2. Each person runs Voicebox desktop app locally
|
||||
3. All point to same `TTS_REMOTE_URL`
|
||||
4. Profiles and history stay local per user
|
||||
5. GPU usage is shared
|
||||
|
||||
### 3. Hybrid Local/Remote
|
||||
|
||||
**Scenario:** Fast local Whisper, heavy TTS on cloud.
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
TTS_MODE=remote
|
||||
TTS_REMOTE_URL=https://your-modal-app.modal.run
|
||||
|
||||
WHISPER_MODE=local # Fast transcription on your CPU
|
||||
```
|
||||
|
||||
### 4. OpenAI Whisper + Self-Hosted TTS
|
||||
|
||||
**Scenario:** Use OpenAI's API for transcription, run TTS locally.
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
TTS_MODE=local
|
||||
|
||||
WHISPER_MODE=openai-api
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Authentication
|
||||
|
||||
Add API key authentication to your external server:
|
||||
|
||||
```python
|
||||
from fastapi import Header, HTTPException
|
||||
|
||||
API_KEY = "your-secret-key"
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(...)):
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
|
||||
@app.post("/v1/generate", dependencies=[Depends(verify_api_key)])
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
Configure Voicebox:
|
||||
```bash
|
||||
TTS_API_KEY=your-secret-key
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- **VPN/Tailscale**: Use private network for remote servers
|
||||
- **HTTPS**: Use reverse proxy (nginx/Caddy) with SSL certificates
|
||||
- **Firewall**: Restrict access to known IPs
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Protect your external server:
|
||||
|
||||
```python
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
|
||||
@app.post("/v1/generate")
|
||||
@limiter.limit("10/minute")
|
||||
async def generate(...):
|
||||
...
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Latency
|
||||
|
||||
External providers add network latency:
|
||||
- **Local network**: ~10-50ms overhead (negligible)
|
||||
- **Same datacenter**: ~1-5ms overhead
|
||||
- **Cross-region cloud**: 50-200ms+ overhead
|
||||
|
||||
For real-time applications, keep TTS server on local network or same cloud region.
|
||||
|
||||
### Caching
|
||||
|
||||
Implement response caching on external server:
|
||||
|
||||
```python
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=1000)
|
||||
def get_cached_generation(text, voice_prompt_hash, language, seed):
|
||||
return model.generate_voice_clone(text, voice_prompt)
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
For high-traffic deployments, run multiple TTS servers behind a load balancer:
|
||||
|
||||
```
|
||||
Voicebox ──> Load Balancer ──> TTS Server 1 (GPU 1)
|
||||
├──> TTS Server 2 (GPU 2)
|
||||
└──> TTS Server 3 (GPU 3)
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] **Provider Marketplace**: Built-in directory of compatible providers
|
||||
- [ ] **Automatic Fallback**: If remote fails, fallback to local
|
||||
- [ ] **Cost Tracking**: Monitor API usage and costs
|
||||
- [ ] **Performance Metrics**: Latency, throughput dashboards
|
||||
- [ ] **Multi-Provider**: Use different providers for different voices/languages
|
||||
|
||||
## Contributing
|
||||
|
||||
If you build an external provider, please share:
|
||||
1. Server implementation
|
||||
2. Performance benchmarks
|
||||
3. Deployment guide
|
||||
|
||||
Submit to: [GitHub Discussions](https://github.com/jamiepine/voicebox/discussions)
|
||||
|
||||
## Questions?
|
||||
|
||||
- **Discord**: [Join the community](https://discord.gg/...)
|
||||
- **GitHub**: [Open an issue](https://github.com/jamiepine/voicebox/issues)
|
||||
- **Docs**: [Full documentation](https://voicebox.sh/docs)
|
||||
@@ -0,0 +1,396 @@
|
||||
# MLX Audio Integration
|
||||
|
||||
**Status:** Validated ✅
|
||||
**Context:** [mlx-audio v0.3.1 release](https://github.com/Blaizzy/mlx-audio)
|
||||
|
||||
## Validation Results
|
||||
|
||||
We validated mlx-audio in an isolated environment (`mlx-test/`). Key findings:
|
||||
|
||||
| Metric | Result |
|
||||
|--------|--------|
|
||||
| MLX Version | 0.30.4 |
|
||||
| Model Load Time | ~1s (after initial download) |
|
||||
| Generation RTF | **0.5-0.6x** (1.7-2x faster than real-time) |
|
||||
| Test Hardware | Apple Silicon Mac |
|
||||
|
||||
### Model Mapping
|
||||
|
||||
| voicebox (PyTorch) | mlx-audio (MLX) |
|
||||
|--------------------|-----------------|
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` |
|
||||
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | (not yet converted) |
|
||||
|
||||
### mlx-audio API
|
||||
|
||||
The API uses a **generator-based streaming pattern**:
|
||||
|
||||
```python
|
||||
from mlx_audio.tts import load
|
||||
|
||||
model = load("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16")
|
||||
|
||||
# generate() yields GenerationResult objects
|
||||
for result in model.generate("Hello world"):
|
||||
audio = result.audio # numpy array of samples
|
||||
sample_rate = result.sample_rate # 24000
|
||||
rtf = result.real_time_factor # e.g., 0.55
|
||||
```
|
||||
|
||||
### Known Warnings (harmless)
|
||||
|
||||
```
|
||||
You are using a model of type qwen3_tts to instantiate a model of type .
|
||||
The tokenizer you are loading... with an incorrect regex pattern...
|
||||
```
|
||||
|
||||
These warnings appear but don't affect functionality or output quality.
|
||||
|
||||
### Demo Script
|
||||
|
||||
Run `mlx-test/demo.py` to test:
|
||||
```bash
|
||||
cd mlx-test && source venv/bin/activate && python demo.py "Your text here"
|
||||
```
|
||||
|
||||
## Problem
|
||||
|
||||
Apple Silicon users are stuck on CPU inference while Windows and Linux users get CUDA acceleration. The current PyTorch MPS backend has stability issues (lines 34-36 in `backend/tts.py` and `backend/transcribe.py`), forcing a CPU fallback that makes voicebox significantly slower on M1/M2/M3 Macs.
|
||||
|
||||
This creates a poor experience for a large portion of users who bought Apple Silicon specifically for ML workloads.
|
||||
|
||||
## Solution
|
||||
|
||||
Integrate [mlx-audio](https://github.com/Blaizzy/mlx-audio) as the inference engine for macOS Apple Silicon builds. MLX is Apple's native ML framework, optimized for Metal and the unified memory architecture. It's fast, stable, and already supports the same Qwen3-TTS models we use.
|
||||
|
||||
**Key wins:**
|
||||
- Native GPU acceleration on Apple Silicon (no more CPU fallback)
|
||||
- Streaming TTS support (faster perceived latency)
|
||||
- Memory optimizations (run larger models on less RAM)
|
||||
- Fixed 0.6B silence bug that we currently ship
|
||||
- Same Qwen3-TTS models (zero migration cost for users)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Current Stack
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch + Qwen3-TTS │
|
||||
│ (CPU only on macOS) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### Proposed Stack
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Platform Detection at Runtime │
|
||||
└─────────────────────────────────────────┘
|
||||
│
|
||||
├─── Apple Silicon (aarch64-darwin)
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ MLX Audio Backend │
|
||||
│ │ - Qwen3-TTS (mlx) │
|
||||
│ │ - Whisper (mlx) │
|
||||
│ │ - Streaming support │
|
||||
│ └─────────────────────────┘
|
||||
│
|
||||
└─── Other (x86_64, Windows, Linux)
|
||||
┌─────────────────────────┐
|
||||
│ PyTorch Backend │
|
||||
│ - Qwen3-TTS (pytorch) │
|
||||
│ - Whisper (pytorch) │
|
||||
│ - CUDA if available │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Platform Detection & Dependency Management
|
||||
|
||||
Create a backend that switches between PyTorch and MLX based on runtime platform detection.
|
||||
|
||||
**New files:**
|
||||
- `backend/platform.py` - Detect Apple Silicon, return backend type
|
||||
- `backend/backends/__init__.py` - Backend factory pattern
|
||||
- `backend/requirements-mlx.txt` - MLX-specific deps (macOS only)
|
||||
|
||||
**Modified files:**
|
||||
- `backend/requirements.txt` - Keep PyTorch as default
|
||||
- `backend/main.py` - Import from backend factory instead of direct imports
|
||||
|
||||
**Platform detection logic:**
|
||||
```python
|
||||
def get_backend_type() -> str:
|
||||
"""Detect best backend for current platform."""
|
||||
if platform.system() == "Darwin" and platform.machine() == "arm64":
|
||||
# Apple Silicon detected
|
||||
try:
|
||||
import mlx
|
||||
return "mlx"
|
||||
except ImportError:
|
||||
return "pytorch" # Fallback if mlx not installed
|
||||
return "pytorch"
|
||||
```
|
||||
|
||||
### Phase 2: MLX Backend Implementation
|
||||
|
||||
Create parallel implementations of TTS and STT using mlx-audio.
|
||||
|
||||
**New files:**
|
||||
- `backend/backends/mlx_backend.py` - MLX inference engine
|
||||
- `backend/backends/pytorch_backend.py` - Refactor current code into backend
|
||||
|
||||
**Interface both backends must implement:**
|
||||
```python
|
||||
class TTSBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def create_voice_prompt(self, audio_path: str, reference_text: str) -> dict: ...
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]: ...
|
||||
async def generate_streaming(self, text: str, voice_prompt: dict, **kwargs) -> AsyncIterator[bytes]: ...
|
||||
def unload_model(self) -> None: ...
|
||||
|
||||
class STTBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def transcribe(self, audio_path: str, language: Optional[str]) -> str: ...
|
||||
def unload_model(self) -> None: ...
|
||||
```
|
||||
|
||||
**MLX backend implementation notes:**
|
||||
|
||||
mlx-audio's `generate()` returns a generator by default (streaming is built-in):
|
||||
|
||||
```python
|
||||
# MLX backend wrapper
|
||||
from mlx_audio.tts import load
|
||||
|
||||
class MLXTTSBackend:
|
||||
def __init__(self):
|
||||
self.model = None
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
model_map = {
|
||||
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
|
||||
# "0.6B": needs conversion to mlx format
|
||||
}
|
||||
self.model = load(model_map[model_size])
|
||||
|
||||
async def generate(self, text: str, voice_prompt: dict, **kwargs) -> Tuple[np.ndarray, int]:
|
||||
# Collect all chunks from generator
|
||||
chunks = []
|
||||
for result in self.model.generate(text): # TODO: add voice_prompt support
|
||||
chunks.append(np.array(result.audio))
|
||||
return np.concatenate(chunks), 24000
|
||||
```
|
||||
|
||||
**MLX-specific features to expose:**
|
||||
- Streaming TTS (new endpoint: `/api/generate/stream`)
|
||||
- Memory-optimized model loading
|
||||
- Qwen3-ASR for transcription (in addition to Whisper)
|
||||
|
||||
### Phase 3: API Layer Updates
|
||||
|
||||
Update FastAPI endpoints to support new streaming capabilities and maintain backward compatibility.
|
||||
|
||||
**Modified files:**
|
||||
- `backend/main.py` - Add streaming endpoints
|
||||
- `backend/tts.py` - Refactor to use backend abstraction
|
||||
- `backend/transcribe.py` - Refactor to use backend abstraction
|
||||
|
||||
**New endpoints:**
|
||||
```python
|
||||
@app.post("/api/generate/stream")
|
||||
async def generate_stream(...) -> StreamingResponse:
|
||||
"""Stream TTS chunks as they're generated (MLX only)."""
|
||||
backend = get_backend()
|
||||
if not hasattr(backend, 'generate_streaming'):
|
||||
raise HTTPException(501, "Streaming not supported on this backend")
|
||||
return StreamingResponse(backend.generate_streaming(...), media_type="audio/wav")
|
||||
```
|
||||
|
||||
**Backward compatibility:**
|
||||
- Keep all existing `/api/generate` endpoints unchanged
|
||||
- PyTorch backend users see no behavior change
|
||||
- MLX users automatically get faster inference, streaming is opt-in
|
||||
|
||||
### Phase 4: Frontend Integration
|
||||
|
||||
Add UI indicators for backend type and streaming progress.
|
||||
|
||||
**Modified files:**
|
||||
- `app/src/hooks/useGenerationForm.tsx` - Add streaming support
|
||||
- `app/src/components/GenerationForm.tsx` - Show backend badge, streaming toggle
|
||||
- `app/src/lib/api.ts` - Add streaming API client
|
||||
|
||||
**UI additions:**
|
||||
- Badge showing current backend ("MLX" or "PyTorch")
|
||||
- Toggle for streaming mode (disabled if PyTorch)
|
||||
- Real-time streaming playback (WaveSurfer progressive loading)
|
||||
|
||||
### Phase 5: Build & Distribution
|
||||
|
||||
Create separate installers for MLX (Apple Silicon) and PyTorch (Universal).
|
||||
|
||||
**Modified files:**
|
||||
- `tauri/src-tauri/tauri.conf.json` - Add target-specific builds
|
||||
- `.github/workflows/release.yml` - Build both variants
|
||||
|
||||
**Build matrix:**
|
||||
```yaml
|
||||
- target: aarch64-apple-darwin
|
||||
backend: mlx
|
||||
installer: voicebox-macos-silicon-{version}.dmg
|
||||
|
||||
- target: x86_64-apple-darwin
|
||||
backend: pytorch
|
||||
installer: voicebox-macos-intel-{version}.dmg
|
||||
|
||||
- target: x86_64-pc-windows-msvc
|
||||
backend: pytorch
|
||||
installer: voicebox-windows-{version}.exe
|
||||
```
|
||||
|
||||
**Installation flow:**
|
||||
- Auto-detect architecture, recommend correct installer
|
||||
- MLX installer includes `mlx-audio` in embedded Python
|
||||
- PyTorch installer includes `torch` in embedded Python
|
||||
- Both can coexist (different backend, same profile format)
|
||||
|
||||
### Phase 6: Testing & Validation
|
||||
|
||||
Ensure both backends produce compatible outputs.
|
||||
|
||||
**New files:**
|
||||
- `backend/tests/test_backend_parity.py` - Verify both backends produce similar audio
|
||||
- `backend/tests/test_streaming.py` - Streaming-specific tests
|
||||
|
||||
**Test scenarios:**
|
||||
- Same voice prompt on both backends → similar (not identical) audio output
|
||||
- Profile created on MLX → loads on PyTorch (and vice versa)
|
||||
- Streaming chunks assemble into valid WAV file
|
||||
- Model downloads work on both backends
|
||||
- Memory usage stays within bounds
|
||||
|
||||
### Phase 7: Documentation
|
||||
|
||||
Update user-facing docs and developer guides.
|
||||
|
||||
**New files:**
|
||||
- `docs/developer/BACKENDS.md` - Guide for adding new backends
|
||||
- `docs/overview/performance.md` - Backend comparison benchmarks
|
||||
|
||||
**Modified files:**
|
||||
- `README.md` - Note Apple Silicon acceleration
|
||||
- `docs/TROUBLESHOOTING.md` - Add MLX-specific issues
|
||||
|
||||
**Key docs to write:**
|
||||
- Which installer to download (architecture detection)
|
||||
- Performance comparison (MLX vs PyTorch on same M2 hardware)
|
||||
- How streaming mode works
|
||||
- How to force PyTorch on Apple Silicon (for debugging)
|
||||
|
||||
## Technical Decisions
|
||||
|
||||
### Why Dual Backend Instead of MLX-Only?
|
||||
|
||||
**Pros of dual backend:**
|
||||
- Windows and Intel Mac users unaffected
|
||||
- Easier testing (can compare outputs)
|
||||
- Fallback if MLX has issues
|
||||
|
||||
**Cons of dual backend:**
|
||||
- More code to maintain
|
||||
- Two dependency trees
|
||||
- Build complexity (separate installers)
|
||||
|
||||
**Decision:** Dual backend. The maintenance cost is worth it to avoid breaking existing users and to have a fallback.
|
||||
|
||||
### Why Separate Installers Instead of Runtime Detection?
|
||||
|
||||
**Pros of separate installers:**
|
||||
- Smaller bundle size (don't ship both PyTorch and MLX)
|
||||
- Clearer to users which version they have
|
||||
- Easier to debug (no "which backend am I running?" confusion)
|
||||
- Can optimize each build for its target
|
||||
|
||||
**Cons:**
|
||||
- More installers to build and test
|
||||
- Users might download the wrong one
|
||||
|
||||
**Decision:** Separate installers. Bundle size matters (PyTorch + MLX would be huge), and we can auto-detect architecture on the download page.
|
||||
|
||||
### Streaming vs Batch Generation
|
||||
|
||||
MLX supports streaming, PyTorch doesn't (without significant work). Should streaming be:
|
||||
1. MLX-only feature (✅ chosen)
|
||||
2. Implemented for both (lots of work)
|
||||
3. Not exposed at all (wasted opportunity)
|
||||
|
||||
**Decision:** MLX-only. Expose as opt-in feature with graceful degradation (button disabled on PyTorch backend).
|
||||
|
||||
## Migration Path
|
||||
|
||||
Nothing needs migrating, macos users will just notice a speed-boost in inference
|
||||
|
||||
**Data format compatibility:**
|
||||
- Profiles (SQLite) → no schema changes needed
|
||||
- Voice prompts (cached) → backend-agnostic (just numpy arrays)
|
||||
- Audio files → unchanged
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
### Measured Results (from validation)
|
||||
|
||||
| Metric | MLX (measured) | PyTorch CPU (estimated) |
|
||||
|--------|----------------|-------------------------|
|
||||
| **6s audio generation** | ~3-4s | ~10-15s |
|
||||
| **Real-time factor** | 0.5-0.6x | 2-3x |
|
||||
| **Model load (cached)** | ~1s | ~3-5s |
|
||||
|
||||
### TTS Generation (1.7B model, ~20s output)
|
||||
- **PyTorch CPU (M2 Max):** ~45-60s (slower than real-time)
|
||||
- **MLX (M2 Max):** ~8-12s (faster than real-time)
|
||||
- **Improvement:** ~4-5x faster
|
||||
|
||||
### Whisper Transcription (10s audio clip)
|
||||
- **PyTorch CPU:** ~5-8s
|
||||
- **MLX:** ~1-2s
|
||||
- **Improvement:** ~3-4x faster
|
||||
|
||||
### Memory Usage (1.7B model)
|
||||
- **PyTorch:** ~8-10GB (no GPU offload, so CPU RAM)
|
||||
- **MLX:** ~4-6GB (unified memory, better optimization)
|
||||
- **Improvement:** ~40% less RAM
|
||||
|
||||
Full benchmarks will be in `docs/overview/performance.md` after Phase 6.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Should we support Qwen3-ASR (MLX-only) in addition to Whisper?** Adds another model option but increases complexity. Probably phase 8+. - Sure
|
||||
- **Should we backport streaming to PyTorch?** Would require chunking and callback-based generation. Probably not worth it given mlx-audio already has it. - No
|
||||
- **What's the auto-update UX for migrating PyTorch→MLX users?** Needs design. Don't want to force reinstall, but also want to make upgrade obvious. - it just updates, users see nothing
|
||||
- **Do we expose backend selection in settings or hide it?** Leaning toward auto-detect only, with env var override for power users.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
How we'll know this worked:
|
||||
|
||||
1. **Performance:** Apple Silicon users report generation faster than real-time
|
||||
2. **Adoption:** >80% of macOS downloads are MLX build within 1 month
|
||||
3. **Stability:** <5% increase in bug reports (backend abstraction doesn't introduce regressions)
|
||||
4. **Feedback:** Positive sentiment in Discord/GitHub about macOS performance
|
||||
|
||||
## Related Work
|
||||
|
||||
- [PyTorch MPS tracking issue](https://github.com/pytorch/pytorch/issues/77764) - Why we can't use MPS directly
|
||||
- [mlx-audio server implementation](https://github.com/Blaizzy/mlx-audio/blob/main/examples/server.py) - Reference for streaming API
|
||||
- [MLX Whisper benchmarks](https://github.com/ml-explore/mlx-examples/tree/main/whisper) - Performance data
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ~~Validate mlx-audio can load Qwen3-TTS models (quick test)~~ ✅ Done - see `mlx-test/`
|
||||
2. Get approval on dual-backend architecture
|
||||
3. Start Phase 1 (platform detection)
|
||||
|
||||
## Questions?
|
||||
|
||||
Feedback welcome in GitHub discussions or Discord.
|
||||
@@ -0,0 +1,235 @@
|
||||
# OpenAI API Compatibility
|
||||
|
||||
**Status:** Planned for v0.2.0
|
||||
|
||||
**Issue:** [#10 OpenAI API compatibility](https://github.com/jamiepine/voicebox/issues/10)
|
||||
|
||||
## Overview
|
||||
|
||||
This feature exposes OpenAI-compatible endpoints from Voicebox, allowing any tool, library, or application that speaks the OpenAI Audio API to use Voicebox as a drop-in local replacement.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph clients [External Clients]
|
||||
SDK[OpenAI SDK]
|
||||
Curl[curl / HTTP]
|
||||
Apps[Third-party Apps]
|
||||
end
|
||||
|
||||
subgraph voicebox [Voicebox Server]
|
||||
OpenAI["/v1/audio/* endpoints"]
|
||||
TTS[TTSModel]
|
||||
Whisper[WhisperModel]
|
||||
Profiles[Voice Profiles]
|
||||
end
|
||||
|
||||
SDK --> OpenAI
|
||||
Curl --> OpenAI
|
||||
Apps --> OpenAI
|
||||
OpenAI --> TTS
|
||||
OpenAI --> Whisper
|
||||
OpenAI --> Profiles
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **OpenAI SDK users**: `openai.audio.speech.create()` works with Voicebox
|
||||
- **LLM frameworks**: LangChain, AutoGen, etc. can use Voicebox for TTS
|
||||
- **Shell scripts**: `curl` commands copy-pasted from OpenAI docs work
|
||||
- **Existing integrations**: Any tool expecting OpenAI's API works without code changes
|
||||
|
||||
## Endpoints to Implement
|
||||
|
||||
### 1. `POST /v1/audio/speech` (TTS)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createSpeech
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "tts-1",
|
||||
"input": "Hello world!",
|
||||
"voice": "alloy",
|
||||
"response_format": "mp3",
|
||||
"speed": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
**Response:** Audio file (mp3, wav, opus, aac, flac, pcm)
|
||||
|
||||
**Voice Mapping Strategy:**
|
||||
|
||||
- `voice` parameter maps to Voicebox profile names (case-insensitive)
|
||||
- If no match, use a configurable default profile
|
||||
- Support special syntax: `voice: "profile:uuid"` for explicit profile ID
|
||||
|
||||
### 2. `POST /v1/audio/transcriptions` (Whisper)
|
||||
|
||||
OpenAI spec: https://platform.openai.com/docs/api-reference/audio/createTranscription
|
||||
|
||||
**Request:** (multipart/form-data)
|
||||
|
||||
- `file`: Audio file
|
||||
- `model`: "whisper-1"
|
||||
- `language`: Optional language hint
|
||||
- `response_format`: json, text, srt, verbose_json, vtt
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Hello world!"
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### New File: `backend/openai_compat.py`
|
||||
|
||||
Create a dedicated module with an APIRouter for OpenAI-compatible endpoints:
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Literal, Optional
|
||||
|
||||
router = APIRouter(prefix="/v1/audio", tags=["OpenAI Compatible"])
|
||||
|
||||
class SpeechRequest(BaseModel):
|
||||
model: str = "tts-1"
|
||||
input: str
|
||||
voice: str = "alloy"
|
||||
response_format: Literal["mp3", "wav", "opus", "aac", "flac", "pcm"] = "mp3"
|
||||
speed: float = 1.0
|
||||
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest, db: Session = Depends(get_db)):
|
||||
# 1. Map voice name to profile
|
||||
# 2. Generate audio using existing TTSModel
|
||||
# 3. Convert to requested format
|
||||
# 4. Return audio stream
|
||||
...
|
||||
|
||||
@router.post("/transcriptions")
|
||||
async def create_transcription(
|
||||
file: UploadFile = File(...),
|
||||
model: str = Form("whisper-1"),
|
||||
language: Optional[str] = Form(None),
|
||||
response_format: str = Form("json"),
|
||||
):
|
||||
# 1. Save uploaded file
|
||||
# 2. Transcribe using existing WhisperModel
|
||||
# 3. Return in requested format
|
||||
...
|
||||
```
|
||||
|
||||
### Voice Profile Resolution
|
||||
|
||||
Add helper in [backend/profiles.py](backend/profiles.py):
|
||||
|
||||
```python
|
||||
async def resolve_voice_for_openai(voice: str, db: Session) -> Optional[VoiceProfile]:
|
||||
"""
|
||||
Resolve OpenAI voice parameter to a Voicebox profile.
|
||||
|
||||
Priority:
|
||||
1. Exact profile name match (case-insensitive)
|
||||
2. Profile ID match (if voice starts with "profile:")
|
||||
3. Default profile from config
|
||||
4. First available profile
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### Audio Format Conversion
|
||||
|
||||
Add conversion utilities in [backend/utils/audio.py](backend/utils/audio.py):
|
||||
|
||||
```python
|
||||
def convert_audio_format(
|
||||
audio: np.ndarray,
|
||||
sample_rate: int,
|
||||
target_format: str, # mp3, wav, opus, aac, flac, pcm
|
||||
) -> bytes:
|
||||
"""Convert audio to target format using ffmpeg or pydub."""
|
||||
...
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to [backend/config.py](backend/config.py):
|
||||
|
||||
```python
|
||||
# OpenAI API Compatibility
|
||||
OPENAI_COMPAT_ENABLED = True
|
||||
OPENAI_COMPAT_DEFAULT_VOICE = None # Profile ID or name for default voice
|
||||
OPENAI_COMPAT_REQUIRE_AUTH = False # Require API key validation
|
||||
OPENAI_COMPAT_API_KEY = None # If set, validate against this
|
||||
```
|
||||
|
||||
### Integration with main.py
|
||||
|
||||
In [backend/main.py](backend/main.py), include the router:
|
||||
|
||||
```python
|
||||
from . import openai_compat
|
||||
|
||||
# Add OpenAI-compatible routes
|
||||
if config.OPENAI_COMPAT_ENABLED:
|
||||
app.include_router(openai_compat.router)
|
||||
```
|
||||
|
||||
## Streaming Support (Future Enhancement)
|
||||
|
||||
Initial implementation returns complete audio. Streaming can be added later:
|
||||
|
||||
```python
|
||||
@router.post("/speech")
|
||||
async def create_speech(request: SpeechRequest):
|
||||
if request.stream:
|
||||
return StreamingResponse(
|
||||
generate_audio_chunks(request),
|
||||
media_type=f"audio/{request.response_format}"
|
||||
)
|
||||
...
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Example usage after implementation:
|
||||
|
||||
```bash
|
||||
# TTS with curl
|
||||
curl http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Hello!", "voice": "MyProfile"}' \
|
||||
--output speech.mp3
|
||||
|
||||
# With OpenAI Python SDK
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.audio.speech.create(
|
||||
model="tts-1",
|
||||
voice="MyProfile",
|
||||
input="Hello world!"
|
||||
)
|
||||
response.stream_to_file("output.mp3")
|
||||
|
||||
# Transcription
|
||||
curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F file=@audio.mp3 \
|
||||
-F model="whisper-1"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Optional API key validation (for shared deployments)
|
||||
- Rate limiting on endpoints
|
||||
- Input length limits (same as existing `/generate` endpoint)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `pydub` or `ffmpeg-python` for audio format conversion (mp3, opus, etc.)
|
||||
- No changes to existing TTS/Whisper model code
|
||||
|
After Width: | Height: | Size: 178 KiB |
|
After Width: | Height: | Size: 157 KiB |
|
After Width: | Height: | Size: 114 KiB |
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@voicebox/landing",
|
||||
"version": "0.1.8",
|
||||
"version": "0.1.11",
|
||||
"description": "Landing page for voicebox.sh",
|
||||
"scripts": {
|
||||
"dev": "bun --bun next dev --turbo",
|
||||
|
||||
|
Before Width: | Height: | Size: 134 KiB After Width: | Height: | Size: 178 KiB |
|
Before Width: | Height: | Size: 129 KiB After Width: | Height: | Size: 157 KiB |