mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-27 14:15:16 -07:00
Compare commits
59
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e5f4606a6c | ||
|
|
146ef5aaeb | ||
|
|
971604d14f | ||
|
|
953e6ec7d8 | ||
|
|
d3c65fc6c2 | ||
|
|
b6e772c6ac | ||
|
|
a6b070201b | ||
|
|
30352e2419 | ||
|
|
bfa38b36b7 | ||
|
|
1b66a528d1 | ||
|
|
bef4092e6e | ||
|
|
9654f7b642 | ||
|
|
eba1244add | ||
|
|
94487f32a5 | ||
|
|
081f45e680 | ||
|
|
86768288ce | ||
|
|
0fd063442a | ||
|
|
a0c2493e98 | ||
|
|
b39f48cc81 | ||
|
|
4ff775bc98 | ||
|
|
6351aa75e9 | ||
|
|
43873a883b | ||
|
|
60012b81c0 | ||
|
|
89f3127c37 | ||
|
|
ef3c3a7f8c | ||
|
|
7b5e73cfa8 | ||
|
|
462f104494 | ||
|
|
fadb57164e | ||
|
|
e870d65136 | ||
|
|
3df40278cc | ||
|
|
deeef5a474 | ||
|
|
236e464525 | ||
|
|
cf3cf3f002 | ||
|
|
9d98e1e768 | ||
|
|
3be8980f48 | ||
|
|
76bc070f5b | ||
|
|
01838f4773 | ||
|
|
39e4f9d08c | ||
|
|
341d71470c | ||
|
|
bb6cea24ba | ||
|
|
fa7ac88abc | ||
|
|
c68ddc45b1 | ||
|
|
f89dc66d0c | ||
|
|
cba7d7bc23 | ||
|
|
3c89b068f3 | ||
|
|
229841e05e | ||
|
|
123e8215e4 | ||
|
|
2a3afec2ca | ||
|
|
99ddd5a0b4 | ||
|
|
8d730621bc | ||
|
|
e23118f610 | ||
|
|
d4bfdc0d68 | ||
|
|
116c108906 | ||
|
|
2d23c8e06a | ||
|
|
3973a59ba3 | ||
|
|
d9aa75253a | ||
|
|
b22bf36565 | ||
|
|
423d69b7cc | ||
|
|
c513451277 |
+4
-4
@@ -1,5 +1,5 @@
|
|||||||
[bumpversion]
|
[bumpversion]
|
||||||
current_version = 0.1.7
|
current_version = 0.1.11
|
||||||
commit = True
|
commit = True
|
||||||
tag = True
|
tag = True
|
||||||
tag_name = v{new_version}
|
tag_name = v{new_version}
|
||||||
@@ -34,6 +34,6 @@ replace = "version": "{new_version}"
|
|||||||
search = "version": "{current_version}"
|
search = "version": "{current_version}"
|
||||||
replace = "version": "{new_version}"
|
replace = "version": "{new_version}"
|
||||||
|
|
||||||
[bumpversion:file:backend/main.py]
|
[bumpversion:file:backend/__init__.py]
|
||||||
search = "version": "{current_version}"
|
search = __version__ = "{current_version}"
|
||||||
replace = "version": "{new_version}"
|
replace = __version__ = "{new_version}"
|
||||||
|
|||||||
@@ -17,15 +17,19 @@ jobs:
|
|||||||
- platform: 'macos-latest'
|
- platform: 'macos-latest'
|
||||||
args: '--target aarch64-apple-darwin'
|
args: '--target aarch64-apple-darwin'
|
||||||
python-version: '3.12'
|
python-version: '3.12'
|
||||||
|
backend: 'mlx'
|
||||||
- platform: 'macos-15-intel'
|
- platform: 'macos-15-intel'
|
||||||
args: '--target x86_64-apple-darwin'
|
args: '--target x86_64-apple-darwin'
|
||||||
python-version: '3.12'
|
python-version: '3.12'
|
||||||
|
backend: 'pytorch'
|
||||||
# - platform: 'ubuntu-22.04'
|
# - platform: 'ubuntu-22.04'
|
||||||
# args: ''
|
# args: ''
|
||||||
# python-version: '3.12'
|
# python-version: '3.12'
|
||||||
|
# backend: 'pytorch'
|
||||||
- platform: 'windows-latest'
|
- platform: 'windows-latest'
|
||||||
args: ''
|
args: ''
|
||||||
python-version: '3.12'
|
python-version: '3.12'
|
||||||
|
backend: 'pytorch'
|
||||||
|
|
||||||
runs-on: ${{ matrix.platform }}
|
runs-on: ${{ matrix.platform }}
|
||||||
|
|
||||||
@@ -57,6 +61,11 @@ jobs:
|
|||||||
pip install pyinstaller
|
pip install pyinstaller
|
||||||
pip install -r backend/requirements.txt
|
pip install -r backend/requirements.txt
|
||||||
|
|
||||||
|
- name: Install MLX dependencies (Apple Silicon only)
|
||||||
|
if: matrix.backend == 'mlx'
|
||||||
|
run: |
|
||||||
|
pip install -r backend/requirements-mlx.txt
|
||||||
|
|
||||||
- name: Build Python server (Linux/macOS)
|
- name: Build Python server (Linux/macOS)
|
||||||
if: matrix.platform != 'windows-latest'
|
if: matrix.platform != 'windows-latest'
|
||||||
run: |
|
run: |
|
||||||
@@ -133,7 +142,8 @@ jobs:
|
|||||||
See the assets below to download and install this version.
|
See the assets below to download and install this version.
|
||||||
|
|
||||||
### Installation
|
### Installation
|
||||||
- **macOS**: Download the `.dmg` file
|
- **macOS (Apple Silicon)**: Download the `aarch64.dmg` file - uses MLX for fast native inference
|
||||||
|
- **macOS (Intel)**: Download the `x64.dmg` file - uses PyTorch
|
||||||
- **Windows**: Download the `.msi` installer
|
- **Windows**: Download the `.msi` installer
|
||||||
- **Linux**: Download the `.AppImage` or `.deb` package
|
- **Linux**: Download the `.AppImage` or `.deb` package
|
||||||
|
|
||||||
|
|||||||
@@ -53,6 +53,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks
|
||||||
|
- Includes Python version detection and compatibility warnings
|
||||||
|
- Self-documenting help system with `make help`
|
||||||
|
- Colored output for better readability
|
||||||
|
- Supports parallel development server execution
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **README** - Added Makefile reference and updated Quick Start with Makefile-based setup instructions alongside manual setup
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [Unreleased - Planned]
|
||||||
|
|
||||||
### Planned
|
### Planned
|
||||||
- Real-time streaming synthesis
|
- Real-time streaming synthesis
|
||||||
- Conversation mode with multiple speakers
|
- Conversation mode with multiple speakers
|
||||||
|
|||||||
+65
-17
@@ -32,6 +32,10 @@ Thank you for your interest in contributing to Voicebox! This document provides
|
|||||||
|
|
||||||
### Development Setup
|
### Development Setup
|
||||||
|
|
||||||
|
**Using the Makefile (recommended for macOS/Linux):** Run `make setup` to install all dependencies, then `make dev` to start development servers. See `make help` for all available commands.
|
||||||
|
|
||||||
|
**Manual setup (required for Windows):**
|
||||||
|
|
||||||
1. **Fork and clone the repository**
|
1. **Fork and clone the repository**
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/YOUR_USERNAME/voicebox.git
|
git clone https://github.com/YOUR_USERNAME/voicebox.git
|
||||||
@@ -62,37 +66,43 @@ Thank you for your interest in contributing to Voicebox! This document provides
|
|||||||
# Install Python dependencies
|
# Install Python dependencies
|
||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# Install MLX dependencies (Apple Silicon only - for faster inference)
|
||||||
|
# On Apple Silicon, this enables native Metal acceleration
|
||||||
|
if [[ $(uname -m) == "arm64" ]]; then
|
||||||
|
pip install -r requirements-mlx.txt
|
||||||
|
fi
|
||||||
|
|
||||||
# Install Qwen3-TTS (required for voice synthesis)
|
# Install Qwen3-TTS (required for voice synthesis)
|
||||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||||
```
|
```
|
||||||
|
|
||||||
4. **Initialize database**
|
4. **Start development servers**
|
||||||
```bash
|
|
||||||
cd backend
|
|
||||||
python -c "from database import init_db; init_db()"
|
|
||||||
```
|
|
||||||
This creates the SQLite database at `data/voicebox.db`.
|
|
||||||
|
|
||||||
5. **Start development servers**
|
Development requires two terminals: one for the Python backend, one for the Tauri app.
|
||||||
|
|
||||||
**Terminal 1: Backend server**
|
**Terminal 1: Backend server** (start this first)
|
||||||
```bash
|
```bash
|
||||||
cd backend
|
cd backend
|
||||||
source venv/bin/activate # Activate venv if not already active
|
source venv/bin/activate # Activate venv if not already active
|
||||||
bun run dev:server
|
bun run dev:server
|
||||||
# Or manually: uvicorn main:app --reload --port 8000
|
# Or manually: uvicorn main:app --reload --port 17493
|
||||||
```
|
```
|
||||||
Backend will be available at `http://localhost:8000`
|
Backend will be available at `http://localhost:17493`
|
||||||
|
|
||||||
**Terminal 2: Desktop app**
|
**Terminal 2: Desktop app**
|
||||||
```bash
|
```bash
|
||||||
bun run dev
|
bun run dev
|
||||||
```
|
```
|
||||||
This will:
|
This will:
|
||||||
|
- Create a placeholder sidecar binary (for Tauri compilation)
|
||||||
- Start Vite dev server on port 5173
|
- Start Vite dev server on port 5173
|
||||||
- Launch Tauri window pointing to localhost:5173
|
- Launch Tauri window pointing to localhost:5173
|
||||||
|
- Connect to the Python server you started in Terminal 1
|
||||||
- Enable hot reload
|
- Enable hot reload
|
||||||
|
|
||||||
|
> **Note:** In dev mode, the app connects to your manually-started Python server.
|
||||||
|
> The bundled server binary is only used in production builds.
|
||||||
|
|
||||||
**Optional: Web app**
|
**Optional: Web app**
|
||||||
```bash
|
```bash
|
||||||
bun run dev:web
|
bun run dev:web
|
||||||
@@ -109,18 +119,36 @@ First-time usage will be slower due to model downloads, but subsequent runs will
|
|||||||
|
|
||||||
### Building
|
### Building
|
||||||
|
|
||||||
**Build Python server binary:**
|
**Build everything (recommended):**
|
||||||
```bash
|
```bash
|
||||||
|
bun run build
|
||||||
|
```
|
||||||
|
This automatically:
|
||||||
|
1. Builds the Python server binary (`./scripts/build-server.sh`)
|
||||||
|
2. Builds the Tauri desktop app (`cd tauri && bun run tauri build`)
|
||||||
|
|
||||||
|
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`) in `tauri/src-tauri/target/release/bundle/`.
|
||||||
|
|
||||||
|
**Note:** The build process detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
|
||||||
|
|
||||||
|
**Build server binary only:**
|
||||||
|
```bash
|
||||||
|
bun run build:server
|
||||||
|
# or
|
||||||
./scripts/build-server.sh
|
./scripts/build-server.sh
|
||||||
```
|
```
|
||||||
Creates platform-specific binary in `tauri/src-tauri/binaries/`
|
Creates platform-specific binary in `tauri/src-tauri/binaries/`
|
||||||
|
|
||||||
**Build Tauri desktop app:**
|
**Building with local Qwen3-TTS development version:**
|
||||||
|
|
||||||
|
If you're actively developing or modifying the Qwen3-TTS library, set the `QWEN_TTS_PATH` environment variable to point to your local clone:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd tauri
|
export QWEN_TTS_PATH=~/path/to/your/Qwen3-TTS
|
||||||
bun run tauri build
|
bun run build:server
|
||||||
```
|
```
|
||||||
Creates platform-specific installers (`.dmg`, `.msi`, `.AppImage`)
|
|
||||||
|
This makes PyInstaller use your local qwen-tts version instead of the pip-installed package. Useful when testing changes to the TTS library before they're published to PyPI or when using an editable install (`pip install -e`).
|
||||||
|
|
||||||
**Build web app:**
|
**Build web app:**
|
||||||
```bash
|
```bash
|
||||||
@@ -137,6 +165,26 @@ After starting the backend server:
|
|||||||
```
|
```
|
||||||
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
||||||
|
|
||||||
|
### Convert Assets to Web Formats
|
||||||
|
|
||||||
|
To optimize images and videos for the web, run:
|
||||||
|
```bash
|
||||||
|
bun run convert:assets
|
||||||
|
```
|
||||||
|
|
||||||
|
This script:
|
||||||
|
- Converts PNG → WebP (better compression, same quality)
|
||||||
|
- Converts MOV → WebM (VP9 codec, smaller file size)
|
||||||
|
- Processes files in `landing/public/` and `docs/public/`
|
||||||
|
- **Deletes original files** after successful conversion
|
||||||
|
|
||||||
|
**Requirements:** Install `webp` and `ffmpeg`:
|
||||||
|
```bash
|
||||||
|
brew install webp ffmpeg
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Note:** Run this before committing new images or videos to keep the repository size small.
|
||||||
|
|
||||||
## Development Workflow
|
## Development Workflow
|
||||||
|
|
||||||
### 1. Create a Branch
|
### 1. Create a Branch
|
||||||
|
|||||||
@@ -0,0 +1,245 @@
|
|||||||
|
# Voicebox Makefile
|
||||||
|
# Unix-only (macOS/Linux). Windows users should use WSL.
|
||||||
|
|
||||||
|
SHELL := /bin/bash
|
||||||
|
.DEFAULT_GOAL := help
|
||||||
|
|
||||||
|
# Directories
|
||||||
|
BACKEND_DIR := backend
|
||||||
|
TAURI_DIR := tauri
|
||||||
|
WEB_DIR := web
|
||||||
|
APP_DIR := app
|
||||||
|
|
||||||
|
# Python (prefer 3.12, fallback to 3.13, then python3)
|
||||||
|
PYTHON := $(shell command -v python3.12 2>/dev/null || command -v python3.13 2>/dev/null || echo python3)
|
||||||
|
VENV := $(CURDIR)/$(BACKEND_DIR)/venv
|
||||||
|
VENV_BIN := $(VENV)/bin
|
||||||
|
PIP := $(VENV_BIN)/pip
|
||||||
|
PYTHON_VENV := $(VENV_BIN)/python
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
BLUE := \033[0;34m
|
||||||
|
GREEN := \033[0;32m
|
||||||
|
YELLOW := \033[0;33m
|
||||||
|
NC := \033[0m # No Color
|
||||||
|
|
||||||
|
.PHONY: help
|
||||||
|
help: ## Show this help message
|
||||||
|
@echo -e "$(BLUE)Voicebox$(NC) - Development Commands"
|
||||||
|
@echo ""
|
||||||
|
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
|
||||||
|
awk 'BEGIN {FS = ":.*?## "}; {printf " $(GREEN)%-20s$(NC) %s\n", $$1, $$2}'
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# SETUP
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: setup setup-js setup-python setup-rust
|
||||||
|
|
||||||
|
setup: setup-js setup-python ## Full project setup (all dependencies)
|
||||||
|
@echo -e "$(GREEN)✓ Setup complete!$(NC)"
|
||||||
|
@echo -e " Run $(YELLOW)make dev$(NC) to start development servers"
|
||||||
|
|
||||||
|
setup-js: ## Install JavaScript dependencies (bun)
|
||||||
|
@echo -e "$(BLUE)Installing JavaScript dependencies...$(NC)"
|
||||||
|
bun install
|
||||||
|
|
||||||
|
setup-python: $(VENV)/bin/activate ## Set up Python virtual environment and dependencies
|
||||||
|
@echo -e "$(BLUE)Installing Python dependencies...$(NC)"
|
||||||
|
$(PIP) install --upgrade pip
|
||||||
|
$(PIP) install -r $(BACKEND_DIR)/requirements.txt
|
||||||
|
@if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \
|
||||||
|
echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \
|
||||||
|
$(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \
|
||||||
|
echo -e "$(GREEN)✓ MLX backend enabled (native Metal acceleration)$(NC)"; \
|
||||||
|
fi
|
||||||
|
$(PIP) install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||||
|
@echo -e "$(GREEN)✓ Python environment ready$(NC)"
|
||||||
|
|
||||||
|
$(VENV)/bin/activate:
|
||||||
|
@echo -e "$(BLUE)Creating Python virtual environment...$(NC)"
|
||||||
|
@PY_MINOR=$$($(PYTHON) -c "import sys; print(sys.version_info[1])"); \
|
||||||
|
if [ "$$PY_MINOR" -gt 13 ]; then \
|
||||||
|
echo -e "$(YELLOW)Warning: Python 3.$$PY_MINOR detected. ML packages may not be compatible.$(NC)"; \
|
||||||
|
echo -e "$(YELLOW)Recommended: Use Python 3.12 or 3.13 (brew install [email protected])$(NC)"; \
|
||||||
|
fi
|
||||||
|
$(PYTHON) -m venv $(VENV)
|
||||||
|
|
||||||
|
setup-rust: ## Install Rust toolchain (if not present)
|
||||||
|
@command -v rustc >/dev/null 2>&1 || curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# DEVELOPMENT
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: dev dev-backend dev-frontend dev-web kill-dev
|
||||||
|
|
||||||
|
dev: ## Start backend + desktop app (parallel)
|
||||||
|
@echo -e "$(BLUE)Starting development servers...$(NC)"
|
||||||
|
@echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)"
|
||||||
|
@trap 'kill 0' EXIT; \
|
||||||
|
$(MAKE) dev-backend & \
|
||||||
|
sleep 2 && $(MAKE) dev-frontend & \
|
||||||
|
wait
|
||||||
|
|
||||||
|
dev-backend: ## Start FastAPI backend server
|
||||||
|
@echo -e "$(BLUE)Starting backend server on http://localhost:17493$(NC)"
|
||||||
|
$(VENV_BIN)/uvicorn backend.main:app --reload --port 17493
|
||||||
|
|
||||||
|
dev-frontend: ## Start Tauri desktop app
|
||||||
|
@echo -e "$(BLUE)Starting Tauri desktop app...$(NC)"
|
||||||
|
bun run dev
|
||||||
|
|
||||||
|
dev-web: ## Start backend + web app (parallel)
|
||||||
|
@echo -e "$(BLUE)Starting web development servers...$(NC)"
|
||||||
|
@trap 'kill 0' EXIT; \
|
||||||
|
$(MAKE) dev-backend & \
|
||||||
|
sleep 2 && cd $(WEB_DIR) && bun run dev & \
|
||||||
|
wait
|
||||||
|
|
||||||
|
kill-dev: ## Kill all development processes
|
||||||
|
@echo -e "$(YELLOW)Killing development processes...$(NC)"
|
||||||
|
-pkill -f "uvicorn main:app" 2>/dev/null || true
|
||||||
|
-pkill -f "vite" 2>/dev/null || true
|
||||||
|
@echo -e "$(GREEN)✓ Processes killed$(NC)"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# BUILD
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: build build-server build-tauri build-web
|
||||||
|
|
||||||
|
build: build-server build-tauri ## Build everything (server binary + desktop app)
|
||||||
|
@echo -e "$(GREEN)✓ Build complete!$(NC)"
|
||||||
|
|
||||||
|
build-server: ## Build Python server binary
|
||||||
|
@echo -e "$(BLUE)Building server binary...$(NC)"
|
||||||
|
PATH="$(VENV_BIN):$$PATH" ./scripts/build-server.sh
|
||||||
|
|
||||||
|
build-tauri: ## Build Tauri desktop app
|
||||||
|
@echo -e "$(BLUE)Building Tauri desktop app...$(NC)"
|
||||||
|
cd $(TAURI_DIR) && bun run tauri build
|
||||||
|
|
||||||
|
build-web: ## Build web app
|
||||||
|
@echo -e "$(BLUE)Building web app...$(NC)"
|
||||||
|
cd $(WEB_DIR) && bun run build
|
||||||
|
@echo -e "$(GREEN)✓ Web build output in $(WEB_DIR)/dist/$(NC)"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# DATABASE & API
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: db-init db-reset generate-api
|
||||||
|
|
||||||
|
db-init: $(VENV)/bin/activate ## Initialize SQLite database
|
||||||
|
@echo -e "$(BLUE)Initializing database...$(NC)"
|
||||||
|
cd $(BACKEND_DIR) && $(PYTHON_VENV) -c "from database import init_db; init_db()"
|
||||||
|
@echo -e "$(GREEN)✓ Database created at $(BACKEND_DIR)/data/voicebox.db$(NC)"
|
||||||
|
|
||||||
|
db-reset: ## Reset database (delete and reinitialize)
|
||||||
|
@echo -e "$(YELLOW)Resetting database...$(NC)"
|
||||||
|
rm -f $(BACKEND_DIR)/data/voicebox.db
|
||||||
|
$(MAKE) db-init
|
||||||
|
|
||||||
|
generate-api: ## Generate TypeScript API client from OpenAPI schema
|
||||||
|
@echo -e "$(BLUE)Generating API client...$(NC)"
|
||||||
|
@echo -e "$(YELLOW)Note: Backend must be running (make dev-backend)$(NC)"
|
||||||
|
./scripts/generate-api.sh
|
||||||
|
@echo -e "$(GREEN)✓ API client generated in $(APP_DIR)/src/lib/api/$(NC)"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# CODE QUALITY
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: lint format typecheck check
|
||||||
|
|
||||||
|
lint: ## Run linter (Biome)
|
||||||
|
@echo -e "$(BLUE)Linting...$(NC)"
|
||||||
|
bun run lint
|
||||||
|
|
||||||
|
format: ## Format code (Biome)
|
||||||
|
@echo -e "$(BLUE)Formatting...$(NC)"
|
||||||
|
bun run format
|
||||||
|
|
||||||
|
typecheck: ## Run TypeScript type checking
|
||||||
|
@echo -e "$(BLUE)Type checking...$(NC)"
|
||||||
|
bun run tsc --noEmit
|
||||||
|
|
||||||
|
check: ## Run all checks (Biome lint + format + type check)
|
||||||
|
@echo -e "$(BLUE)Running all checks...$(NC)"
|
||||||
|
bun run check
|
||||||
|
@echo -e "$(GREEN)✓ All checks passed$(NC)"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# TESTING
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: test test-backend test-frontend
|
||||||
|
|
||||||
|
test: test-backend test-frontend ## Run all tests
|
||||||
|
@echo -e "$(GREEN)✓ All tests passed$(NC)"
|
||||||
|
|
||||||
|
test-backend: ## Run Python backend tests (requires pytest)
|
||||||
|
@echo -e "$(BLUE)Running backend tests...$(NC)"
|
||||||
|
@if [ -f "$(VENV_BIN)/pytest" ]; then \
|
||||||
|
cd $(BACKEND_DIR) && $(VENV_BIN)/pytest -v; \
|
||||||
|
else \
|
||||||
|
echo -e "$(YELLOW)pytest not installed. Run: $(PIP) install pytest$(NC)"; \
|
||||||
|
exit 1; \
|
||||||
|
fi
|
||||||
|
|
||||||
|
test-frontend: ## Run frontend tests (requires test script in package.json)
|
||||||
|
@echo -e "$(BLUE)Running frontend tests...$(NC)"
|
||||||
|
@if bun run test --help >/dev/null 2>&1; then \
|
||||||
|
bun run test; \
|
||||||
|
else \
|
||||||
|
echo -e "$(YELLOW)No test script configured$(NC)"; \
|
||||||
|
exit 1; \
|
||||||
|
fi
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# LOGS & DEBUGGING
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: logs docs
|
||||||
|
|
||||||
|
logs: ## Tail backend logs
|
||||||
|
@echo -e "$(BLUE)Tailing logs (Ctrl+C to stop)...$(NC)"
|
||||||
|
tail -f $(BACKEND_DIR)/logs/*.log 2>/dev/null || echo "No log files found"
|
||||||
|
|
||||||
|
docs: ## Open API documentation (backend must be running)
|
||||||
|
@echo -e "$(BLUE)Opening API docs...$(NC)"
|
||||||
|
open http://localhost:17493/docs 2>/dev/null || xdg-open http://localhost:17493/docs
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# CLEAN
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
.PHONY: clean clean-python clean-build clean-all
|
||||||
|
|
||||||
|
clean: ## Clean build artifacts
|
||||||
|
@echo -e "$(BLUE)Cleaning build artifacts...$(NC)"
|
||||||
|
rm -rf $(TAURI_DIR)/src-tauri/target/release
|
||||||
|
rm -rf $(WEB_DIR)/dist
|
||||||
|
rm -rf $(APP_DIR)/dist
|
||||||
|
@echo -e "$(GREEN)✓ Build artifacts cleaned$(NC)"
|
||||||
|
|
||||||
|
clean-python: ## Clean Python cache and virtual environment
|
||||||
|
@echo -e "$(BLUE)Cleaning Python files...$(NC)"
|
||||||
|
rm -rf $(VENV)
|
||||||
|
find $(BACKEND_DIR) -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
find $(BACKEND_DIR) -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||||
|
@echo -e "$(GREEN)✓ Python environment cleaned$(NC)"
|
||||||
|
|
||||||
|
clean-build: ## Clean Rust/Tauri build cache
|
||||||
|
@echo -e "$(BLUE)Cleaning Rust build cache...$(NC)"
|
||||||
|
cd $(TAURI_DIR)/src-tauri && cargo clean
|
||||||
|
@echo -e "$(GREEN)✓ Rust cache cleaned$(NC)"
|
||||||
|
|
||||||
|
clean-all: clean clean-python clean-build ## Nuclear clean (everything)
|
||||||
|
@echo -e "$(BLUE)Cleaning node_modules...$(NC)"
|
||||||
|
rm -rf node_modules
|
||||||
|
rm -rf $(APP_DIR)/node_modules
|
||||||
|
rm -rf $(TAURI_DIR)/node_modules
|
||||||
|
rm -rf $(WEB_DIR)/node_modules
|
||||||
|
@echo -e "$(GREEN)✓ Full clean complete$(NC)"
|
||||||
@@ -10,6 +10,21 @@
|
|||||||
All running locally on your machine.
|
All running locally on your machine.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="https://github.com/jamiepine/voicebox/releases">
|
||||||
|
<img src="https://img.shields.io/github/downloads/jamiepine/voicebox/total?style=flat&color=blue" alt="Downloads" />
|
||||||
|
</a>
|
||||||
|
<a href="https://github.com/jamiepine/voicebox/releases/latest">
|
||||||
|
<img src="https://img.shields.io/github/v/release/jamiepine/voicebox?style=flat" alt="Release" />
|
||||||
|
</a>
|
||||||
|
<a href="https://github.com/jamiepine/voicebox/stargazers">
|
||||||
|
<img src="https://img.shields.io/github/stars/jamiepine/voicebox?style=flat" alt="Stars" />
|
||||||
|
</a>
|
||||||
|
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
|
||||||
|
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
|
||||||
|
</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://voicebox.sh">voicebox.sh</a> •
|
<a href="https://voicebox.sh">voicebox.sh</a> •
|
||||||
<a href="#download">Download</a> •
|
<a href="#download">Download</a> •
|
||||||
@@ -22,7 +37,7 @@
|
|||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://voicebox.sh">
|
<a href="https://voicebox.sh">
|
||||||
<img src=".github/assets/screenshot.webp" alt="Voicebox App Screenshot" width="800" />
|
<img src="landing/public/assets/app-screenshot-1.webp" alt="Voicebox App Screenshot" width="800" />
|
||||||
</a>
|
</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
@@ -32,17 +47,30 @@
|
|||||||
|
|
||||||
<br/>
|
<br/>
|
||||||
|
|
||||||
## Why Voicebox?
|
<p align="center">
|
||||||
|
<img src="landing/public/assets/app-screenshot-2.webp" alt="Voicebox Screenshot 2" width="800" />
|
||||||
|
</p>
|
||||||
|
|
||||||
Voice AI is exploding, but most tools are either cloud-locked, expensive, or a nightmare to set up. Voicebox is different:
|
<p align="center">
|
||||||
|
<img src="landing/public/assets/app-screenshot-3.webp" alt="Voicebox Screenshot 3" width="800" />
|
||||||
|
</p>
|
||||||
|
|
||||||
- **100% Local** — Your voice data never leaves your machine
|
<br/>
|
||||||
- **Lightweight** — No bloated Electron, native Tauri performance
|
|
||||||
- **Fast** — Near-instant on CUDA, optimized for Apple Silicon
|
|
||||||
- **Flexible** — Use the app, integrate the API, or both
|
|
||||||
- **Open Source** — No subscriptions, no limits, no lock-in
|
|
||||||
|
|
||||||
Built with **Tauri** (Rust), **TypeScript**, **React**, and **Python**. Native performance meets modern DX.
|
## What is Voicebox?
|
||||||
|
|
||||||
|
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as the **Ollama for voice** — download models, clone voices, and generate speech entirely on your machine.
|
||||||
|
|
||||||
|
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
|
||||||
|
|
||||||
|
- **Complete privacy** — models and voice data stay on your machine
|
||||||
|
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
|
||||||
|
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
|
||||||
|
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
|
||||||
|
- **Native performance** — built with Tauri (Rust), not Electron
|
||||||
|
- **Super fast on Mac** — MLX backend with native Metal acceleration for 4-5x faster inference on Apple Silicon
|
||||||
|
|
||||||
|
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -70,11 +98,13 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
|
|||||||
- **Instant cloning** — Upload a sample, get a voice profile
|
- **Instant cloning** — Upload a sample, get a voice profile
|
||||||
- **High fidelity** — Natural prosody, emotion, and cadence
|
- **High fidelity** — Natural prosody, emotion, and cadence
|
||||||
- **Multi-language** — English, Chinese, and more coming
|
- **Multi-language** — English, Chinese, and more coming
|
||||||
|
- **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super fast generation
|
||||||
|
|
||||||
### Voice Profile Management
|
### Voice Profile Management
|
||||||
|
|
||||||
- **Create profiles** from audio files or record directly in-app
|
- **Create profiles** from audio files or record directly in-app
|
||||||
- **Import/Export** profiles to share or backup
|
- **Import/Export** profiles to share or backup
|
||||||
|
- **Multi-sample support** — combine multiple samples for higher quality cloning
|
||||||
- **Organize** with descriptions and language tags
|
- **Organize** with descriptions and language tags
|
||||||
|
|
||||||
### Speech Generation
|
### Speech Generation
|
||||||
@@ -83,9 +113,19 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
|
|||||||
- **Batch generation** for long-form content
|
- **Batch generation** for long-form content
|
||||||
- **Smart caching** — regenerate instantly with voice prompt caching
|
- **Smart caching** — regenerate instantly with voice prompt caching
|
||||||
|
|
||||||
|
### Stories Editor
|
||||||
|
|
||||||
|
Create multi-voice narratives, podcasts, and conversations with a timeline-based editor.
|
||||||
|
|
||||||
|
- **Multi-track composition** — arrange multiple voice tracks in a single project
|
||||||
|
- **Inline audio editing** — trim and split clips directly in the timeline
|
||||||
|
- **Auto-playback** — preview stories with synchronized playhead
|
||||||
|
- **Voice mixing** — build conversations with multiple participants
|
||||||
|
|
||||||
### Recording & Transcription
|
### Recording & Transcription
|
||||||
|
|
||||||
- **In-app recording** with waveform visualization
|
- **In-app recording** with waveform visualization
|
||||||
|
- **System audio capture** — record desktop audio on macOS and Windows
|
||||||
- **Automatic transcription** powered by Whisper
|
- **Automatic transcription** powered by Whisper
|
||||||
- **Export recordings** in multiple formats
|
- **Export recordings** in multiple formats
|
||||||
|
|
||||||
@@ -109,17 +149,17 @@ Voicebox exposes a full REST API, so you can integrate voice synthesis into your
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Generate speech
|
# Generate speech
|
||||||
curl -X POST http://localhost:8000/api/generate \
|
curl -X POST http://localhost:8000/generate \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"text": "Hello world", "profile_id": "abc123"}'
|
-d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}'
|
||||||
|
|
||||||
# List voice profiles
|
# List voice profiles
|
||||||
curl http://localhost:8000/api/profiles
|
curl http://localhost:8000/profiles
|
||||||
|
|
||||||
# Create a profile from audio
|
# Create a profile
|
||||||
curl -X POST http://localhost:8000/api/profiles \
|
curl -X POST http://localhost:8000/profiles \
|
||||||
-F "[email protected]" \
|
-H "Content-Type: application/json" \
|
||||||
-F "name=My Voice"
|
-d '{"name": "My Voice", "language": "en"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
**Use cases:**
|
**Use cases:**
|
||||||
@@ -142,8 +182,9 @@ Full API documentation available at `http://localhost:8000/docs` when running.
|
|||||||
| Frontend | React, TypeScript, Tailwind CSS |
|
| Frontend | React, TypeScript, Tailwind CSS |
|
||||||
| State | Zustand, React Query |
|
| State | Zustand, React Query |
|
||||||
| Backend | FastAPI (Python) |
|
| Backend | FastAPI (Python) |
|
||||||
| Voice Model | Qwen3-TTS |
|
| Voice Model | Qwen3-TTS (PyTorch or MLX) |
|
||||||
| Transcription | Whisper |
|
| Transcription | Whisper (PyTorch or MLX) |
|
||||||
|
| Inference Engine | MLX (Apple Silicon) / PyTorch (Windows/Linux/Intel) |
|
||||||
| Database | SQLite |
|
| Database | SQLite |
|
||||||
| Audio | WaveSurfer.js, librosa |
|
| Audio | WaveSurfer.js, librosa |
|
||||||
|
|
||||||
@@ -184,8 +225,26 @@ Voicebox aims to be the **one-stop shop for everything voice** — cloning, synt
|
|||||||
|
|
||||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed setup and contribution guidelines.
|
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed setup and contribution guidelines.
|
||||||
|
|
||||||
|
**Using the Makefile (recommended):** Run `make help` to see all available commands for setup, development, building, and testing.
|
||||||
|
|
||||||
### Quick Start
|
### Quick Start
|
||||||
|
|
||||||
|
**With Makefile (Unix/macOS/Linux):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repo
|
||||||
|
git clone https://github.com/voicebox-sh/voicebox.git
|
||||||
|
cd voicebox
|
||||||
|
|
||||||
|
# Setup everything
|
||||||
|
make setup
|
||||||
|
|
||||||
|
# Start development
|
||||||
|
make dev
|
||||||
|
```
|
||||||
|
|
||||||
|
**Manual setup (all platforms):**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clone the repo
|
# Clone the repo
|
||||||
git clone https://github.com/voicebox-sh/voicebox.git
|
git clone https://github.com/voicebox-sh/voicebox.git
|
||||||
@@ -201,7 +260,11 @@ cd backend && pip install -r requirements.txt && cd ..
|
|||||||
bun run dev
|
bun run dev
|
||||||
```
|
```
|
||||||
|
|
||||||
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org). CUDA-capable GPU recommended (CPU inference supported but slower).
|
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org).
|
||||||
|
|
||||||
|
**Performance:**
|
||||||
|
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration for 4-5x faster inference
|
||||||
|
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU recommended, CPU supported but slower)
|
||||||
|
|
||||||
### Project Structure
|
### Project Structure
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@voicebox/app",
|
"name": "@voicebox/app",
|
||||||
"version": "0.1.7",
|
"version": "0.1.11",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
@@ -48,6 +48,7 @@
|
|||||||
"react": "^18.3.0",
|
"react": "^18.3.0",
|
||||||
"react-dom": "^18.3.0",
|
"react-dom": "^18.3.0",
|
||||||
"react-hook-form": "^7.53.0",
|
"react-hook-form": "^7.53.0",
|
||||||
|
"react-sound-visualizer": "^1.4.0",
|
||||||
"tailwind-merge": "^2.5.4",
|
"tailwind-merge": "^2.5.4",
|
||||||
"wavesurfer.js": "^7.0.0",
|
"wavesurfer.js": "^7.0.0",
|
||||||
"zod": "^3.23.8",
|
"zod": "^3.23.8",
|
||||||
|
|||||||
+23
-16
@@ -4,15 +4,10 @@ import voiceboxLogo from '@/assets/voicebox-logo.png';
|
|||||||
import ShinyText from '@/components/ShinyText';
|
import ShinyText from '@/components/ShinyText';
|
||||||
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
||||||
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||||
import {
|
|
||||||
isTauri,
|
|
||||||
setKeepServerRunning,
|
|
||||||
setupWindowCloseHandler,
|
|
||||||
startServer,
|
|
||||||
} from '@/lib/tauri';
|
|
||||||
import { cn } from '@/lib/utils/cn';
|
import { cn } from '@/lib/utils/cn';
|
||||||
import { router } from '@/router';
|
import { router } from '@/router';
|
||||||
import { useServerStore } from '@/stores/serverStore';
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
const LOADING_MESSAGES = [
|
const LOADING_MESSAGES = [
|
||||||
'Warming up tensors...',
|
'Warming up tensors...',
|
||||||
@@ -38,29 +33,38 @@ const LOADING_MESSAGES = [
|
|||||||
];
|
];
|
||||||
|
|
||||||
function App() {
|
function App() {
|
||||||
|
const platform = usePlatform();
|
||||||
const [serverReady, setServerReady] = useState(false);
|
const [serverReady, setServerReady] = useState(false);
|
||||||
const [loadingMessageIndex, setLoadingMessageIndex] = useState(0);
|
const [loadingMessageIndex, setLoadingMessageIndex] = useState(0);
|
||||||
const serverStartingRef = useRef(false);
|
const serverStartingRef = useRef(false);
|
||||||
|
|
||||||
// Sync stored setting to Rust on startup
|
// Sync stored setting to Rust on startup
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (isTauri()) {
|
if (platform.metadata.isTauri) {
|
||||||
const keepRunning = useServerStore.getState().keepServerRunningOnClose;
|
const keepRunning = useServerStore.getState().keepServerRunningOnClose;
|
||||||
setKeepServerRunning(keepRunning).catch((error) => {
|
platform.lifecycle.setKeepServerRunning(keepRunning).catch((error) => {
|
||||||
console.error('Failed to sync initial setting to Rust:', error);
|
console.error('Failed to sync initial setting to Rust:', error);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}, []);
|
}, [platform]);
|
||||||
|
|
||||||
|
// Setup lifecycle callbacks
|
||||||
|
useEffect(() => {
|
||||||
|
platform.lifecycle.onServerReady = () => {
|
||||||
|
setServerReady(true);
|
||||||
|
};
|
||||||
|
}, [platform]);
|
||||||
|
|
||||||
// Setup window close handler and auto-start server when running in Tauri (production only)
|
// Setup window close handler and auto-start server when running in Tauri (production only)
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!isTauri()) {
|
if (!platform.metadata.isTauri) {
|
||||||
|
setServerReady(true); // Web assumes server is running
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Setup window close handler to check setting and stop server if needed
|
// Setup window close handler to check setting and stop server if needed
|
||||||
// This works in both dev and prod, but will only stop server if it was started by the app
|
// This works in both dev and prod, but will only stop server if it was started by the app
|
||||||
setupWindowCloseHandler().catch((error) => {
|
platform.lifecycle.setupWindowCloseHandler().catch((error) => {
|
||||||
console.error('Failed to setup window close handler:', error);
|
console.error('Failed to setup window close handler:', error);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -83,18 +87,21 @@ function App() {
|
|||||||
serverStartingRef.current = true;
|
serverStartingRef.current = true;
|
||||||
console.log('Production mode: Starting bundled server...');
|
console.log('Production mode: Starting bundled server...');
|
||||||
|
|
||||||
startServer(false)
|
platform.lifecycle
|
||||||
|
.startServer(false)
|
||||||
.then((serverUrl) => {
|
.then((serverUrl) => {
|
||||||
console.log('Server is ready at:', serverUrl);
|
console.log('Server is ready at:', serverUrl);
|
||||||
// Update the server URL in the store with the dynamically assigned port
|
// Update the server URL in the store with the dynamically assigned port
|
||||||
useServerStore.getState().setServerUrl(serverUrl);
|
useServerStore.getState().setServerUrl(serverUrl);
|
||||||
setServerReady(true);
|
setServerReady(true);
|
||||||
// Mark that we started the server (so we know to stop it on close)
|
// Mark that we started the server (so we know to stop it on close)
|
||||||
|
// @ts-expect-error - adding property to window
|
||||||
window.__voiceboxServerStartedByApp = true;
|
window.__voiceboxServerStartedByApp = true;
|
||||||
})
|
})
|
||||||
.catch((error) => {
|
.catch((error) => {
|
||||||
console.error('Failed to auto-start server:', error);
|
console.error('Failed to auto-start server:', error);
|
||||||
serverStartingRef.current = false;
|
serverStartingRef.current = false;
|
||||||
|
// @ts-expect-error - adding property to window
|
||||||
window.__voiceboxServerStartedByApp = false;
|
window.__voiceboxServerStartedByApp = false;
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -104,11 +111,11 @@ function App() {
|
|||||||
// Window close event handles server shutdown based on setting
|
// Window close event handles server shutdown based on setting
|
||||||
serverStartingRef.current = false;
|
serverStartingRef.current = false;
|
||||||
};
|
};
|
||||||
}, []);
|
}, [platform]);
|
||||||
|
|
||||||
// Cycle through loading messages every 3 seconds
|
// Cycle through loading messages every 3 seconds
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!isTauri() || serverReady) {
|
if (!platform.metadata.isTauri || serverReady) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -117,10 +124,10 @@ function App() {
|
|||||||
}, 3000);
|
}, 3000);
|
||||||
|
|
||||||
return () => clearInterval(interval);
|
return () => clearInterval(interval);
|
||||||
}, [serverReady]);
|
}, [serverReady, platform.metadata.isTauri]);
|
||||||
|
|
||||||
// Show loading screen while server is starting in Tauri
|
// Show loading screen while server is starting in Tauri
|
||||||
if (isTauri() && !serverReady) {
|
if (platform.metadata.isTauri && !serverReady) {
|
||||||
return (
|
return (
|
||||||
<div
|
<div
|
||||||
className={cn(
|
className={cn(
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
import { useQuery } from '@tanstack/react-query';
|
import { useQuery } from '@tanstack/react-query';
|
||||||
import { invoke } from '@tauri-apps/api/core';
|
|
||||||
import { Pause, Play, Repeat, Volume2, VolumeX, X } from 'lucide-react';
|
import { Pause, Play, Repeat, Volume2, VolumeX, X } from 'lucide-react';
|
||||||
import { useEffect, useMemo, useRef, useState } from 'react';
|
import { useEffect, useMemo, useRef, useState } from 'react';
|
||||||
import WaveSurfer from 'wavesurfer.js';
|
import WaveSurfer from 'wavesurfer.js';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import { Slider } from '@/components/ui/slider';
|
import { Slider } from '@/components/ui/slider';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import { isTauri } from '@/lib/tauri';
|
|
||||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||||
import { debug } from '@/lib/utils/debug';
|
import { debug } from '@/lib/utils/debug';
|
||||||
import { usePlayerStore } from '@/stores/playerStore';
|
import { usePlayerStore } from '@/stores/playerStore';
|
||||||
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function AudioPlayer() {
|
export function AudioPlayer() {
|
||||||
|
const platform = usePlatform();
|
||||||
const {
|
const {
|
||||||
audioUrl,
|
audioUrl,
|
||||||
audioId,
|
audioId,
|
||||||
@@ -39,7 +39,7 @@ export function AudioPlayer() {
|
|||||||
if (!profileId) return { channel_ids: [] };
|
if (!profileId) return { channel_ids: [] };
|
||||||
return apiClient.getProfileChannels(profileId);
|
return apiClient.getProfileChannels(profileId);
|
||||||
},
|
},
|
||||||
enabled: !!profileId && isTauri(),
|
enabled: !!profileId && platform.metadata.isTauri,
|
||||||
});
|
});
|
||||||
|
|
||||||
const { data: channels } = useQuery({
|
const { data: channels } = useQuery({
|
||||||
@@ -50,7 +50,7 @@ export function AudioPlayer() {
|
|||||||
|
|
||||||
// Determine if we should use native playback
|
// Determine if we should use native playback
|
||||||
const useNativePlayback = useMemo(() => {
|
const useNativePlayback = useMemo(() => {
|
||||||
if (!isTauri() || !profileChannels || !channels) {
|
if (!platform.metadata.isTauri || !profileChannels || !channels) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -195,7 +195,7 @@ export function AudioPlayer() {
|
|||||||
let runtimeProfileChannels = null;
|
let runtimeProfileChannels = null;
|
||||||
let runtimeChannels = null;
|
let runtimeChannels = null;
|
||||||
|
|
||||||
if (isTauri() && currentProfileId) {
|
if (platform.metadata.isTauri && currentProfileId) {
|
||||||
try {
|
try {
|
||||||
runtimeProfileChannels = await apiClient.getProfileChannels(currentProfileId);
|
runtimeProfileChannels = await apiClient.getProfileChannels(currentProfileId);
|
||||||
debug.log('Runtime profileChannels:', runtimeProfileChannels);
|
debug.log('Runtime profileChannels:', runtimeProfileChannels);
|
||||||
@@ -210,7 +210,7 @@ export function AudioPlayer() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
debug.log('Auto-play check:', {
|
debug.log('Auto-play check:', {
|
||||||
isTauri: isTauri(),
|
isTauri: platform.metadata.isTauri,
|
||||||
currentAudioUrl,
|
currentAudioUrl,
|
||||||
currentProfileId,
|
currentProfileId,
|
||||||
hasProfileChannels: !!runtimeProfileChannels,
|
hasProfileChannels: !!runtimeProfileChannels,
|
||||||
@@ -218,7 +218,7 @@ export function AudioPlayer() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (
|
if (
|
||||||
isTauri() &&
|
platform.metadata.isTauri &&
|
||||||
currentAudioUrl &&
|
currentAudioUrl &&
|
||||||
currentProfileId &&
|
currentProfileId &&
|
||||||
runtimeProfileChannels &&
|
runtimeProfileChannels &&
|
||||||
@@ -229,7 +229,7 @@ export function AudioPlayer() {
|
|||||||
// Stop any existing native playback first
|
// Stop any existing native playback first
|
||||||
if (isUsingNativePlaybackRef.current) {
|
if (isUsingNativePlaybackRef.current) {
|
||||||
try {
|
try {
|
||||||
await invoke('stop_audio_playback');
|
platform.audio.stopPlayback();
|
||||||
debug.log('Stopped existing native playback before starting new one');
|
debug.log('Stopped existing native playback before starting new one');
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
debug.error('Failed to stop existing playback:', error);
|
debug.error('Failed to stop existing playback:', error);
|
||||||
@@ -279,11 +279,8 @@ export function AudioPlayer() {
|
|||||||
// Play via native audio
|
// Play via native audio
|
||||||
debug.log('Invoking play_audio_to_devices...');
|
debug.log('Invoking play_audio_to_devices...');
|
||||||
try {
|
try {
|
||||||
const result = await invoke('play_audio_to_devices', {
|
await platform.audio.playToDevices(audioData, deviceIds);
|
||||||
audioData: Array.from(audioData),
|
debug.log('play_audio_to_devices completed successfully');
|
||||||
deviceIds: deviceIds,
|
|
||||||
});
|
|
||||||
debug.log('play_audio_to_devices completed successfully, result:', result);
|
|
||||||
|
|
||||||
// Mark that we're using native playback
|
// Mark that we're using native playback
|
||||||
isUsingNativePlaybackRef.current = true;
|
isUsingNativePlaybackRef.current = true;
|
||||||
@@ -516,15 +513,13 @@ export function AudioPlayer() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Stop native playback if it was active
|
// Stop native playback if it was active
|
||||||
if (isUsingNativePlaybackRef.current && isTauri()) {
|
if (isUsingNativePlaybackRef.current && platform.metadata.isTauri) {
|
||||||
(async () => {
|
try {
|
||||||
try {
|
platform.audio.stopPlayback();
|
||||||
await invoke('stop_audio_playback');
|
debug.log('Stopped native audio playback');
|
||||||
debug.log('Stopped native audio playback');
|
} catch (error) {
|
||||||
} catch (error) {
|
debug.error('Failed to stop native playback:', error);
|
||||||
debug.error('Failed to stop native playback:', error);
|
}
|
||||||
}
|
|
||||||
})();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Reset native playback flag when loading new audio
|
// Reset native playback flag when loading new audio
|
||||||
@@ -711,7 +706,7 @@ export function AudioPlayer() {
|
|||||||
if (isPlaying) {
|
if (isPlaying) {
|
||||||
// Pause: stop native playback and pause WaveSurfer visualization
|
// Pause: stop native playback and pause WaveSurfer visualization
|
||||||
try {
|
try {
|
||||||
await invoke('stop_audio_playback');
|
platform.audio.stopPlayback();
|
||||||
debug.log('Stopped native audio playback');
|
debug.log('Stopped native audio playback');
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
debug.error('Failed to stop native playback:', error);
|
debug.error('Failed to stop native playback:', error);
|
||||||
@@ -724,7 +719,7 @@ export function AudioPlayer() {
|
|||||||
try {
|
try {
|
||||||
// Stop any existing native playback first
|
// Stop any existing native playback first
|
||||||
try {
|
try {
|
||||||
await invoke('stop_audio_playback');
|
platform.audio.stopPlayback();
|
||||||
} catch (_error) {
|
} catch (_error) {
|
||||||
// Ignore errors when stopping (might not be playing)
|
// Ignore errors when stopping (might not be playing)
|
||||||
debug.log('No existing playback to stop');
|
debug.log('No existing playback to stop');
|
||||||
@@ -742,10 +737,7 @@ export function AudioPlayer() {
|
|||||||
const audioData = new Uint8Array(await response.arrayBuffer());
|
const audioData = new Uint8Array(await response.arrayBuffer());
|
||||||
|
|
||||||
// Play via native audio
|
// Play via native audio
|
||||||
await invoke('play_audio_to_devices', {
|
await platform.audio.playToDevices(audioData, deviceIds);
|
||||||
audioData: Array.from(audioData),
|
|
||||||
deviceIds: deviceIds,
|
|
||||||
});
|
|
||||||
|
|
||||||
// Mark that we're using native playback
|
// Mark that we're using native playback
|
||||||
isUsingNativePlaybackRef.current = true;
|
isUsingNativePlaybackRef.current = true;
|
||||||
@@ -806,10 +798,12 @@ export function AudioPlayer() {
|
|||||||
|
|
||||||
const handleClose = () => {
|
const handleClose = () => {
|
||||||
// Stop any native playback
|
// Stop any native playback
|
||||||
if (isUsingNativePlaybackRef.current && isTauri()) {
|
if (isUsingNativePlaybackRef.current && platform.metadata.isTauri) {
|
||||||
invoke('stop_audio_playback').catch((error) => {
|
try {
|
||||||
|
platform.audio.stopPlayback();
|
||||||
|
} catch (error) {
|
||||||
debug.error('Failed to stop native playback:', error);
|
debug.error('Failed to stop native playback:', error);
|
||||||
});
|
}
|
||||||
}
|
}
|
||||||
// Stop WaveSurfer
|
// Stop WaveSurfer
|
||||||
if (wavesurferRef.current) {
|
if (wavesurferRef.current) {
|
||||||
|
|||||||
@@ -1,5 +1,4 @@
|
|||||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
||||||
import { invoke } from '@tauri-apps/api/core';
|
|
||||||
import { Check, CheckCircle2, Edit, Plus, Speaker, Trash2 } from 'lucide-react';
|
import { Check, CheckCircle2, Edit, Plus, Speaker, Trash2 } from 'lucide-react';
|
||||||
import { useState } from 'react';
|
import { useState } from 'react';
|
||||||
import { Badge } from '@/components/ui/badge';
|
import { Badge } from '@/components/ui/badge';
|
||||||
@@ -23,9 +22,9 @@ import {
|
|||||||
} from '@/components/ui/select';
|
} from '@/components/ui/select';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||||
import { isTauri } from '@/lib/tauri';
|
|
||||||
import { cn } from '@/lib/utils/cn';
|
import { cn } from '@/lib/utils/cn';
|
||||||
import { usePlayerStore } from '@/stores/playerStore';
|
import { usePlayerStore } from '@/stores/playerStore';
|
||||||
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
interface AudioDevice {
|
interface AudioDevice {
|
||||||
id: string;
|
id: string;
|
||||||
@@ -34,6 +33,7 @@ interface AudioDevice {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function AudioTab() {
|
export function AudioTab() {
|
||||||
|
const platform = usePlatform();
|
||||||
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
||||||
const [editingChannel, setEditingChannel] = useState<string | null>(null);
|
const [editingChannel, setEditingChannel] = useState<string | null>(null);
|
||||||
const [selectedChannelId, setSelectedChannelId] = useState<string | null>(null);
|
const [selectedChannelId, setSelectedChannelId] = useState<string | null>(null);
|
||||||
@@ -49,18 +49,17 @@ export function AudioTab() {
|
|||||||
const { data: devices, isLoading: devicesLoading } = useQuery({
|
const { data: devices, isLoading: devicesLoading } = useQuery({
|
||||||
queryKey: ['audio-devices'],
|
queryKey: ['audio-devices'],
|
||||||
queryFn: async () => {
|
queryFn: async () => {
|
||||||
if (!isTauri()) {
|
if (!platform.metadata.isTauri) {
|
||||||
return [];
|
return [];
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
const result = await invoke<AudioDevice[]>('list_audio_output_devices');
|
return await platform.audio.listOutputDevices();
|
||||||
return result;
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Failed to list audio devices:', error);
|
console.error('Failed to list audio devices:', error);
|
||||||
return [];
|
return [];
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
enabled: isTauri(),
|
enabled: platform.metadata.isTauri,
|
||||||
});
|
});
|
||||||
|
|
||||||
const { data: profiles } = useQuery({
|
const { data: profiles } = useQuery({
|
||||||
@@ -342,7 +341,7 @@ export function AudioTab() {
|
|||||||
<div className="flex flex-col items-center justify-center py-12 border-2 border-dashed border-muted rounded-md">
|
<div className="flex flex-col items-center justify-center py-12 border-2 border-dashed border-muted rounded-md">
|
||||||
<CheckCircle2 className="h-12 w-12 text-muted-foreground mb-4" />
|
<CheckCircle2 className="h-12 w-12 text-muted-foreground mb-4" />
|
||||||
<p className="text-muted-foreground text-center">
|
<p className="text-muted-foreground text-center">
|
||||||
{isTauri() ? 'No audio devices found' : 'Audio device selection requires Tauri'}
|
{platform.metadata.isTauri ? 'No audio devices found' : 'Audio device selection requires Tauri'}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -112,9 +112,6 @@ export function FloatingGenerateBox({
|
|||||||
}
|
}
|
||||||
}, [selectedProfileId, profiles, setSelectedProfileId]);
|
}, [selectedProfileId, profiles, setSelectedProfileId]);
|
||||||
|
|
||||||
// Get current form value to trigger resize when it changes
|
|
||||||
const formValue = form.watch(isInstructMode ? 'instruct' : 'text');
|
|
||||||
|
|
||||||
// Auto-resize textarea based on content (only when expanded)
|
// Auto-resize textarea based on content (only when expanded)
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!isExpanded) {
|
if (!isExpanded) {
|
||||||
@@ -196,59 +193,104 @@ export function FloatingGenerateBox({
|
|||||||
<Form {...form}>
|
<Form {...form}>
|
||||||
<form onSubmit={form.handleSubmit(onSubmit)}>
|
<form onSubmit={form.handleSubmit(onSubmit)}>
|
||||||
<div className="flex gap-2">
|
<div className="flex gap-2">
|
||||||
<motion.div className="flex-1" transition={{ duration: 0.3, ease: 'easeOut' }}>
|
<motion.div
|
||||||
{isInstructMode && (
|
className={cn('flex-1', isExpanded && 'mr-12')}
|
||||||
<span className="text-xs text-accent font-medium mb-1 block">
|
transition={{ duration: 0.3, ease: 'easeOut' }}
|
||||||
Delivery instructions:
|
>
|
||||||
</span>
|
{/* Text field - hidden when in instruct mode */}
|
||||||
)}
|
<div style={{ display: isInstructMode ? 'none' : 'block' }}>
|
||||||
<FormField
|
<FormField
|
||||||
control={form.control}
|
control={form.control}
|
||||||
name={isInstructMode ? 'instruct' : 'text'}
|
name="text"
|
||||||
render={({ field }) => (
|
render={({ field }) => (
|
||||||
<FormItem>
|
<FormItem>
|
||||||
<FormControl>
|
<FormControl>
|
||||||
<motion.div
|
<motion.div
|
||||||
animate={{
|
animate={{
|
||||||
height: isExpanded ? 'auto' : '32px',
|
height: isExpanded ? 'auto' : '32px',
|
||||||
}}
|
|
||||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
|
||||||
style={{ overflow: 'hidden' }}
|
|
||||||
>
|
|
||||||
<Textarea
|
|
||||||
{...field}
|
|
||||||
ref={(node: HTMLTextAreaElement | null) => {
|
|
||||||
// Store ref for auto-resize
|
|
||||||
textareaRef.current = node;
|
|
||||||
// Forward ref to react-hook-form
|
|
||||||
if (typeof field.ref === 'function') {
|
|
||||||
field.ref(node);
|
|
||||||
}
|
|
||||||
}}
|
}}
|
||||||
placeholder={
|
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||||
isInstructMode
|
style={{ overflow: 'hidden' }}
|
||||||
? 'Add delivery instructions...'
|
>
|
||||||
: isStoriesRoute && currentStory
|
<Textarea
|
||||||
|
{...field}
|
||||||
|
ref={(node: HTMLTextAreaElement | null) => {
|
||||||
|
// Store ref for auto-resize (only for active field)
|
||||||
|
if (!isInstructMode) {
|
||||||
|
textareaRef.current = node;
|
||||||
|
}
|
||||||
|
// Forward ref to react-hook-form
|
||||||
|
if (typeof field.ref === 'function') {
|
||||||
|
field.ref(node);
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
placeholder={
|
||||||
|
isStoriesRoute && currentStory
|
||||||
? `Generate speech for "${currentStory.name}"...`
|
? `Generate speech for "${currentStory.name}"...`
|
||||||
: selectedProfile
|
: selectedProfile
|
||||||
? `Generate speech using ${selectedProfile.name}...`
|
? `Generate speech using ${selectedProfile.name}...`
|
||||||
: 'Select a voice profile above...'
|
: 'Select a voice profile above...'
|
||||||
}
|
}
|
||||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||||
style={{
|
style={{
|
||||||
minHeight: isExpanded ? '100px' : '32px',
|
minHeight: isExpanded ? '100px' : '32px',
|
||||||
maxHeight: '300px',
|
maxHeight: '300px',
|
||||||
|
}}
|
||||||
|
disabled={!selectedProfileId}
|
||||||
|
onClick={() => setIsExpanded(true)}
|
||||||
|
onFocus={() => setIsExpanded(true)}
|
||||||
|
/>
|
||||||
|
</motion.div>
|
||||||
|
</FormControl>
|
||||||
|
<FormMessage className="text-xs" />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
{/* Instruct field - hidden when in text mode */}
|
||||||
|
<div style={{ display: isInstructMode ? 'block' : 'none' }}>
|
||||||
|
<FormField
|
||||||
|
control={form.control}
|
||||||
|
name="instruct"
|
||||||
|
render={({ field }) => (
|
||||||
|
<FormItem>
|
||||||
|
<FormControl>
|
||||||
|
<motion.div
|
||||||
|
animate={{
|
||||||
|
height: isExpanded ? 'auto' : '32px',
|
||||||
}}
|
}}
|
||||||
disabled={!selectedProfileId}
|
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||||
onClick={() => setIsExpanded(true)}
|
style={{ overflow: 'hidden' }}
|
||||||
onFocus={() => setIsExpanded(true)}
|
>
|
||||||
/>
|
<Textarea
|
||||||
</motion.div>
|
{...field}
|
||||||
</FormControl>
|
ref={(node: HTMLTextAreaElement | null) => {
|
||||||
<FormMessage className="text-xs" />
|
// Store ref for auto-resize (only for active field)
|
||||||
</FormItem>
|
if (isInstructMode) {
|
||||||
)}
|
textareaRef.current = node;
|
||||||
/>
|
}
|
||||||
|
// Forward ref to react-hook-form
|
||||||
|
if (typeof field.ref === 'function') {
|
||||||
|
field.ref(node);
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
placeholder="Add delivery instructions..."
|
||||||
|
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||||
|
style={{
|
||||||
|
minHeight: isExpanded ? '100px' : '32px',
|
||||||
|
maxHeight: '300px',
|
||||||
|
}}
|
||||||
|
disabled={!selectedProfileId}
|
||||||
|
onClick={() => setIsExpanded(true)}
|
||||||
|
onFocus={() => setIsExpanded(true)}
|
||||||
|
/>
|
||||||
|
</motion.div>
|
||||||
|
</FormControl>
|
||||||
|
<FormMessage className="text-xs" />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
</motion.div>
|
</motion.div>
|
||||||
|
|
||||||
<div className="relative shrink-0">
|
<div className="relative shrink-0">
|
||||||
@@ -278,9 +320,12 @@ export function FloatingGenerateBox({
|
|||||||
variant="ghost"
|
variant="ghost"
|
||||||
size="icon"
|
size="icon"
|
||||||
onClick={() => setIsInstructMode(!isInstructMode)}
|
onClick={() => setIsInstructMode(!isInstructMode)}
|
||||||
className={`h-10 w-10 rounded-full bg-card border border-border hover:bg-background/50 transition-all duration-200 ${
|
className={cn(
|
||||||
isInstructMode ? 'text-accent' : ''
|
'h-10 w-10 rounded-full transition-all duration-200',
|
||||||
}`}
|
isInstructMode
|
||||||
|
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
|
||||||
|
: 'bg-card border border-border hover:bg-background/50',
|
||||||
|
)}
|
||||||
>
|
>
|
||||||
<MessageSquare className="h-4 w-4" />
|
<MessageSquare className="h-4 w-4" />
|
||||||
</Button>
|
</Button>
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { AudioWaveform, Download, FileArchive, MoreHorizontal, Play, Trash2 } from 'lucide-react';
|
import { AudioWaveform, Download, FileArchive, Loader2, MoreHorizontal, Play, Trash2 } from 'lucide-react';
|
||||||
import { useEffect, useRef, useState } from 'react';
|
import { useEffect, useRef, useState } from 'react';
|
||||||
|
import type { HistoryResponse } from '@/lib/api/types';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import {
|
import {
|
||||||
Dialog,
|
Dialog,
|
||||||
@@ -33,18 +34,23 @@ import { usePlayerStore } from '@/stores/playerStore';
|
|||||||
// OLD TABLE-BASED COMPONENT - REMOVED (can be found in git history)
|
// OLD TABLE-BASED COMPONENT - REMOVED (can be found in git history)
|
||||||
// This is the new alternate history view with fixed height rows
|
// This is the new alternate history view with fixed height rows
|
||||||
|
|
||||||
// NEW ALTERNATE HISTORY VIEW - FIXED HEIGHT ROWS
|
// NEW ALTERNATE HISTORY VIEW - FIXED HEIGHT ROWS WITH INFINITE SCROLL
|
||||||
export function HistoryTable() {
|
export function HistoryTable() {
|
||||||
const [page, _setPage] = useState(0);
|
const [page, setPage] = useState(0);
|
||||||
|
const [allHistory, setAllHistory] = useState<HistoryResponse[]>([]);
|
||||||
|
const [total, setTotal] = useState(0);
|
||||||
const [isScrolled, setIsScrolled] = useState(false);
|
const [isScrolled, setIsScrolled] = useState(false);
|
||||||
const scrollRef = useRef<HTMLDivElement>(null);
|
const scrollRef = useRef<HTMLDivElement>(null);
|
||||||
|
const loadMoreRef = useRef<HTMLDivElement>(null);
|
||||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||||
const [importDialogOpen, setImportDialogOpen] = useState(false);
|
const [importDialogOpen, setImportDialogOpen] = useState(false);
|
||||||
const [selectedFile, setSelectedFile] = useState<File | null>(null);
|
const [selectedFile, setSelectedFile] = useState<File | null>(null);
|
||||||
|
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||||
|
const [generationToDelete, setGenerationToDelete] = useState<{ id: string; name: string } | null>(null);
|
||||||
const limit = 20;
|
const limit = 20;
|
||||||
const { toast } = useToast();
|
const { toast } = useToast();
|
||||||
|
|
||||||
const { data: historyData, isLoading } = useHistory({
|
const { data: historyData, isLoading, isFetching } = useHistory({
|
||||||
limit,
|
limit,
|
||||||
offset: page * limit,
|
offset: page * limit,
|
||||||
});
|
});
|
||||||
@@ -60,6 +66,56 @@ export function HistoryTable() {
|
|||||||
const audioUrl = usePlayerStore((state) => state.audioUrl);
|
const audioUrl = usePlayerStore((state) => state.audioUrl);
|
||||||
const isPlayerVisible = !!audioUrl;
|
const isPlayerVisible = !!audioUrl;
|
||||||
|
|
||||||
|
// Update accumulated history when new data arrives
|
||||||
|
useEffect(() => {
|
||||||
|
if (historyData?.items) {
|
||||||
|
setTotal(historyData.total);
|
||||||
|
if (page === 0) {
|
||||||
|
// Reset to first page
|
||||||
|
setAllHistory(historyData.items);
|
||||||
|
} else {
|
||||||
|
// Append new items, avoiding duplicates
|
||||||
|
setAllHistory((prev) => {
|
||||||
|
const existingIds = new Set(prev.map((item) => item.id));
|
||||||
|
const newItems = historyData.items.filter((item) => !existingIds.has(item.id));
|
||||||
|
return [...prev, ...newItems];
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}, [historyData, page]);
|
||||||
|
|
||||||
|
// Reset to page 0 when deletions or imports occur
|
||||||
|
useEffect(() => {
|
||||||
|
if (deleteGeneration.isSuccess || importGeneration.isSuccess) {
|
||||||
|
setPage(0);
|
||||||
|
setAllHistory([]);
|
||||||
|
}
|
||||||
|
}, [deleteGeneration.isSuccess, importGeneration.isSuccess]);
|
||||||
|
|
||||||
|
// Intersection Observer for infinite scroll
|
||||||
|
useEffect(() => {
|
||||||
|
const loadMoreEl = loadMoreRef.current;
|
||||||
|
if (!loadMoreEl) return;
|
||||||
|
|
||||||
|
const observer = new IntersectionObserver(
|
||||||
|
(entries) => {
|
||||||
|
const target = entries[0];
|
||||||
|
if (target.isIntersecting && !isFetching && allHistory.length < total) {
|
||||||
|
setPage((prev) => prev + 1);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
root: scrollRef.current,
|
||||||
|
rootMargin: '100px',
|
||||||
|
threshold: 0.1,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
observer.observe(loadMoreEl);
|
||||||
|
return () => observer.disconnect();
|
||||||
|
}, [isFetching, allHistory.length, total]);
|
||||||
|
|
||||||
|
// Track scroll position for gradient effect
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const scrollEl = scrollRef.current;
|
const scrollEl = scrollRef.current;
|
||||||
if (!scrollEl) return;
|
if (!scrollEl) return;
|
||||||
@@ -113,24 +169,16 @@ export function HistoryTable() {
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
const _handleImportClick = () => {
|
const handleDeleteClick = (generationId: string, profileName: string) => {
|
||||||
file_handleImportClickk.click();
|
setGenerationToDelete({ id: generationId, name: profileName });
|
||||||
|
setDeleteDialogOpen(true);
|
||||||
};
|
};
|
||||||
|
|
||||||
const _handleFileChange = (_e: React.ChangeEvent<HTMLInputElement>) => {
|
const handleDeleteConfirm = () => {
|
||||||
cons_handleFileChangeet.files?.[0];
|
if (generationToDelete) {
|
||||||
if (file) {
|
deleteGeneration.mutate(generationToDelete.id);
|
||||||
// Validate file extension
|
setDeleteDialogOpen(false);
|
||||||
if (!file.name.endsWith('.voicebox.zip')) {
|
setGenerationToDelete(null);
|
||||||
toast({
|
|
||||||
title: 'Invalid file type',
|
|
||||||
description: 'Please select a valid .voicebox.zip file',
|
|
||||||
variant: 'destructive',
|
|
||||||
});
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
setSelectedFile(file);
|
|
||||||
setImportDialogOpen(true);
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -159,13 +207,16 @@ export function HistoryTable() {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
if (isLoading) {
|
if (isLoading && page === 0) {
|
||||||
return null;
|
return (
|
||||||
|
<div className="flex items-center justify-center h-full">
|
||||||
|
<Loader2 className="h-8 w-8 animate-spin text-muted-foreground" />
|
||||||
|
</div>
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const history = historyData?.items || [];
|
const history = allHistory;
|
||||||
const total = historyData?.total || 0;
|
const hasMore = allHistory.length < total;
|
||||||
const _hasMore = history.length === limit && (page + 1) * limit < total;
|
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="flex flex-col h-full min-h-0 relative">
|
<div className="flex flex-col h-full min-h-0 relative">
|
||||||
@@ -271,7 +322,7 @@ export function HistoryTable() {
|
|||||||
Export Package
|
Export Package
|
||||||
</DropdownMenuItem>
|
</DropdownMenuItem>
|
||||||
<DropdownMenuItem
|
<DropdownMenuItem
|
||||||
onClick={() => deleteGeneration.mutate(gen.id)}
|
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||||
disabled={deleteGeneration.isPending}
|
disabled={deleteGeneration.isPending}
|
||||||
className="text-destructive focus:text-destructive"
|
className="text-destructive focus:text-destructive"
|
||||||
>
|
>
|
||||||
@@ -284,10 +335,53 @@ export function HistoryTable() {
|
|||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
})}
|
})}
|
||||||
|
|
||||||
|
{/* Load more trigger element */}
|
||||||
|
{hasMore && (
|
||||||
|
<div ref={loadMoreRef} className="flex items-center justify-center py-4">
|
||||||
|
{isFetching && <Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* End of list indicator */}
|
||||||
|
{!hasMore && history.length > 0 && (
|
||||||
|
<div className="text-center py-4 text-xs text-muted-foreground">
|
||||||
|
You've reached the end
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
<Dialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
|
||||||
|
<DialogContent>
|
||||||
|
<DialogHeader>
|
||||||
|
<DialogTitle>Delete Generation</DialogTitle>
|
||||||
|
<DialogDescription>
|
||||||
|
Are you sure you want to delete this generation from "{generationToDelete?.name}"? This action cannot be undone.
|
||||||
|
</DialogDescription>
|
||||||
|
</DialogHeader>
|
||||||
|
<DialogFooter>
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
onClick={() => {
|
||||||
|
setDeleteDialogOpen(false);
|
||||||
|
setGenerationToDelete(null);
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Cancel
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
variant="destructive"
|
||||||
|
onClick={handleDeleteConfirm}
|
||||||
|
disabled={deleteGeneration.isPending}
|
||||||
|
>
|
||||||
|
{deleteGeneration.isPending ? 'Deleting...' : 'Delete'}
|
||||||
|
</Button>
|
||||||
|
</DialogFooter>
|
||||||
|
</DialogContent>
|
||||||
|
</Dialog>
|
||||||
|
|
||||||
<Dialog open={importDialogOpen} onOpenChange={setImportDialogOpen}>
|
<Dialog open={importDialogOpen} onOpenChange={setImportDialogOpen}>
|
||||||
<DialogContent>
|
<DialogContent>
|
||||||
<DialogHeader>
|
<DialogHeader>
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ import { Input } from '@/components/ui/input';
|
|||||||
import { Checkbox } from '@/components/ui/checkbox';
|
import { Checkbox } from '@/components/ui/checkbox';
|
||||||
import { useToast } from '@/components/ui/use-toast';
|
import { useToast } from '@/components/ui/use-toast';
|
||||||
import { useServerStore } from '@/stores/serverStore';
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
import { setKeepServerRunning } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
const connectionSchema = z.object({
|
const connectionSchema = z.object({
|
||||||
serverUrl: z.string().url('Please enter a valid URL'),
|
serverUrl: z.string().url('Please enter a valid URL'),
|
||||||
@@ -26,6 +26,7 @@ const connectionSchema = z.object({
|
|||||||
type ConnectionFormValues = z.infer<typeof connectionSchema>;
|
type ConnectionFormValues = z.infer<typeof connectionSchema>;
|
||||||
|
|
||||||
export function ConnectionForm() {
|
export function ConnectionForm() {
|
||||||
|
const platform = usePlatform();
|
||||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||||
const setServerUrl = useServerStore((state) => state.setServerUrl);
|
const setServerUrl = useServerStore((state) => state.setServerUrl);
|
||||||
const keepServerRunningOnClose = useServerStore((state) => state.keepServerRunningOnClose);
|
const keepServerRunningOnClose = useServerStore((state) => state.keepServerRunningOnClose);
|
||||||
@@ -89,7 +90,7 @@ export function ConnectionForm() {
|
|||||||
checked={keepServerRunningOnClose}
|
checked={keepServerRunningOnClose}
|
||||||
onCheckedChange={(checked: boolean) => {
|
onCheckedChange={(checked: boolean) => {
|
||||||
setKeepServerRunningOnClose(checked);
|
setKeepServerRunningOnClose(checked);
|
||||||
setKeepServerRunning(checked).catch((error) => {
|
platform.lifecycle.setKeepServerRunning(checked).catch((error) => {
|
||||||
console.error('Failed to sync setting to Rust:', error);
|
console.error('Failed to sync setting to Rust:', error);
|
||||||
});
|
});
|
||||||
toast({
|
toast({
|
||||||
|
|||||||
@@ -12,11 +12,10 @@ interface ModelProgressProps {
|
|||||||
|
|
||||||
export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
||||||
const [progress, setProgress] = useState<ModelProgressType | null>(null);
|
const [progress, setProgress] = useState<ModelProgressType | null>(null);
|
||||||
const [isSubscribed, setIsSubscribed] = useState(false);
|
|
||||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!serverUrl || isSubscribed) return;
|
if (!serverUrl) return;
|
||||||
|
|
||||||
// Subscribe to progress updates via Server-Sent Events
|
// Subscribe to progress updates via Server-Sent Events
|
||||||
const eventSource = new EventSource(`${serverUrl}/models/progress/${modelName}`);
|
const eventSource = new EventSource(`${serverUrl}/models/progress/${modelName}`);
|
||||||
@@ -29,7 +28,6 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
|||||||
// Close connection if complete or error
|
// Close connection if complete or error
|
||||||
if (data.status === 'complete' || data.status === 'error') {
|
if (data.status === 'complete' || data.status === 'error') {
|
||||||
eventSource.close();
|
eventSource.close();
|
||||||
setIsSubscribed(false);
|
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Error parsing progress event:', error);
|
console.error('Error parsing progress event:', error);
|
||||||
@@ -39,16 +37,12 @@ export function ModelProgress({ modelName, displayName }: ModelProgressProps) {
|
|||||||
eventSource.onerror = (error) => {
|
eventSource.onerror = (error) => {
|
||||||
console.error('SSE error:', error);
|
console.error('SSE error:', error);
|
||||||
eventSource.close();
|
eventSource.close();
|
||||||
setIsSubscribed(false);
|
|
||||||
};
|
};
|
||||||
|
|
||||||
setIsSubscribed(true);
|
|
||||||
|
|
||||||
return () => {
|
return () => {
|
||||||
eventSource.close();
|
eventSource.close();
|
||||||
setIsSubscribed(false);
|
|
||||||
};
|
};
|
||||||
}, [serverUrl, modelName, isSubscribed]);
|
}, [serverUrl, modelName]);
|
||||||
|
|
||||||
// Don't render if no progress or if complete/error and some time has passed
|
// Don't render if no progress or if complete/error and some time has passed
|
||||||
if (
|
if (
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
import { getVersion } from '@tauri-apps/api/app';
|
|
||||||
import { AlertCircle, Download, RefreshCw } from 'lucide-react';
|
import { AlertCircle, Download, RefreshCw } from 'lucide-react';
|
||||||
import { useEffect, useState } from 'react';
|
import { useEffect, useState } from 'react';
|
||||||
import { Badge } from '@/components/ui/badge';
|
import { Badge } from '@/components/ui/badge';
|
||||||
@@ -6,16 +5,18 @@ import { Button } from '@/components/ui/button';
|
|||||||
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
|
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
|
||||||
import { Progress } from '@/components/ui/progress';
|
import { Progress } from '@/components/ui/progress';
|
||||||
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
|
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
|
||||||
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function UpdateStatus() {
|
export function UpdateStatus() {
|
||||||
|
const platform = usePlatform();
|
||||||
const { status, checkForUpdates, downloadAndInstall, restartAndInstall } = useAutoUpdater(false);
|
const { status, checkForUpdates, downloadAndInstall, restartAndInstall } = useAutoUpdater(false);
|
||||||
const [currentVersion, setCurrentVersion] = useState<string>('');
|
const [currentVersion, setCurrentVersion] = useState<string>('');
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
getVersion()
|
platform.metadata.getVersion()
|
||||||
.then(setCurrentVersion)
|
.then(setCurrentVersion)
|
||||||
.catch(() => setCurrentVersion('0.1.0'));
|
.catch(() => setCurrentVersion('0.1.0'));
|
||||||
}, []);
|
}, [platform]);
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<Card>
|
<Card>
|
||||||
|
|||||||
@@ -1,16 +1,17 @@
|
|||||||
import { ConnectionForm } from '@/components/ServerSettings/ConnectionForm';
|
import { ConnectionForm } from '@/components/ServerSettings/ConnectionForm';
|
||||||
import { ServerStatus } from '@/components/ServerSettings/ServerStatus';
|
import { ServerStatus } from '@/components/ServerSettings/ServerStatus';
|
||||||
import { UpdateStatus } from '@/components/ServerSettings/UpdateStatus';
|
import { UpdateStatus } from '@/components/ServerSettings/UpdateStatus';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function ServerTab() {
|
export function ServerTab() {
|
||||||
|
const platform = usePlatform();
|
||||||
return (
|
return (
|
||||||
<div className="space-y-4 overflow-y-auto flex flex-col">
|
<div className="space-y-4 overflow-y-auto flex flex-col">
|
||||||
<div className="grid gap-4 md:grid-cols-2">
|
<div className="grid gap-4 md:grid-cols-2">
|
||||||
<ConnectionForm />
|
<ConnectionForm />
|
||||||
<ServerStatus />
|
<ServerStatus />
|
||||||
</div>
|
</div>
|
||||||
{isTauri() && <UpdateStatus />}
|
{platform.metadata.isTauri && <UpdateStatus />}
|
||||||
<div className="py-8 text-center text-sm text-muted-foreground">
|
<div className="py-8 text-center text-sm text-muted-foreground">
|
||||||
Created by{' '}
|
Created by{' '}
|
||||||
<a
|
<a
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import { useSortable } from '@dnd-kit/sortable';
|
import { useSortable } from '@dnd-kit/sortable';
|
||||||
import { CSS } from '@dnd-kit/utilities';
|
import { CSS } from '@dnd-kit/utilities';
|
||||||
import { GripVertical, Mic, MoreHorizontal, Play, Trash2 } from 'lucide-react';
|
import { GripVertical, Mic, MoreHorizontal, Play, Trash2 } from 'lucide-react';
|
||||||
|
import { useState } from 'react';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import {
|
import {
|
||||||
DropdownMenu,
|
DropdownMenu,
|
||||||
@@ -12,6 +13,7 @@ import { Textarea } from '@/components/ui/textarea';
|
|||||||
import type { StoryItemDetail } from '@/lib/api/types';
|
import type { StoryItemDetail } from '@/lib/api/types';
|
||||||
import { cn } from '@/lib/utils/cn';
|
import { cn } from '@/lib/utils/cn';
|
||||||
import { useStoryStore } from '@/stores/storyStore';
|
import { useStoryStore } from '@/stores/storyStore';
|
||||||
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
|
|
||||||
interface StoryChatItemProps {
|
interface StoryChatItemProps {
|
||||||
item: StoryItemDetail;
|
item: StoryItemDetail;
|
||||||
@@ -33,6 +35,10 @@ export function StoryChatItem({
|
|||||||
isDragging,
|
isDragging,
|
||||||
}: StoryChatItemProps) {
|
}: StoryChatItemProps) {
|
||||||
const seek = useStoryStore((state) => state.seek);
|
const seek = useStoryStore((state) => state.seek);
|
||||||
|
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||||
|
const [avatarError, setAvatarError] = useState(false);
|
||||||
|
|
||||||
|
const avatarUrl = `${serverUrl}/profiles/${item.profile_id}/avatar`;
|
||||||
|
|
||||||
// Check if this item is currently playing based on timecode
|
// Check if this item is currently playing based on timecode
|
||||||
const itemStartMs = item.start_time_ms;
|
const itemStartMs = item.start_time_ms;
|
||||||
@@ -72,10 +78,22 @@ export function StoryChatItem({
|
|||||||
</button>
|
</button>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Voice Icon */}
|
{/* Voice Avatar */}
|
||||||
<div className="shrink-0">
|
<div className="shrink-0">
|
||||||
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center">
|
<div className="h-10 w-10 rounded-full bg-muted flex items-center justify-center overflow-hidden">
|
||||||
<Mic className="h-5 w-5 text-muted-foreground" />
|
{!avatarError ? (
|
||||||
|
<img
|
||||||
|
src={avatarUrl}
|
||||||
|
alt={`${item.profile_name} avatar`}
|
||||||
|
className={cn(
|
||||||
|
'h-full w-full object-cover transition-all duration-200',
|
||||||
|
!isCurrentlyPlaying && 'grayscale'
|
||||||
|
)}
|
||||||
|
onError={() => setAvatarError(true)}
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<Mic className="h-5 w-5 text-muted-foreground" />
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
@@ -1,14 +1,5 @@
|
|||||||
import { Plus, BookOpen, MoreHorizontal, Pencil, Trash2 } from 'lucide-react';
|
import { Plus, BookOpen, MoreHorizontal, Pencil, Trash2 } from 'lucide-react';
|
||||||
import { useState } from 'react';
|
import { useState } from 'react';
|
||||||
import { Button } from '@/components/ui/button';
|
|
||||||
import {
|
|
||||||
Dialog,
|
|
||||||
DialogContent,
|
|
||||||
DialogDescription,
|
|
||||||
DialogFooter,
|
|
||||||
DialogHeader,
|
|
||||||
DialogTitle,
|
|
||||||
} from '@/components/ui/dialog';
|
|
||||||
import {
|
import {
|
||||||
AlertDialog,
|
AlertDialog,
|
||||||
AlertDialogAction,
|
AlertDialogAction,
|
||||||
@@ -19,6 +10,15 @@ import {
|
|||||||
AlertDialogHeader,
|
AlertDialogHeader,
|
||||||
AlertDialogTitle,
|
AlertDialogTitle,
|
||||||
} from '@/components/ui/alert-dialog';
|
} from '@/components/ui/alert-dialog';
|
||||||
|
import { Button } from '@/components/ui/button';
|
||||||
|
import {
|
||||||
|
Dialog,
|
||||||
|
DialogContent,
|
||||||
|
DialogDescription,
|
||||||
|
DialogFooter,
|
||||||
|
DialogHeader,
|
||||||
|
DialogTitle,
|
||||||
|
} from '@/components/ui/dialog';
|
||||||
import {
|
import {
|
||||||
DropdownMenu,
|
DropdownMenu,
|
||||||
DropdownMenuContent,
|
DropdownMenuContent,
|
||||||
@@ -26,18 +26,13 @@ import {
|
|||||||
DropdownMenuTrigger,
|
DropdownMenuTrigger,
|
||||||
} from '@/components/ui/dropdown-menu';
|
} from '@/components/ui/dropdown-menu';
|
||||||
import { Input } from '@/components/ui/input';
|
import { Input } from '@/components/ui/input';
|
||||||
import { Textarea } from '@/components/ui/textarea';
|
|
||||||
import { Label } from '@/components/ui/label';
|
import { Label } from '@/components/ui/label';
|
||||||
|
import { Textarea } from '@/components/ui/textarea';
|
||||||
import { useToast } from '@/components/ui/use-toast';
|
import { useToast } from '@/components/ui/use-toast';
|
||||||
import {
|
import { useStories, useCreateStory, useUpdateStory, useDeleteStory } from '@/lib/hooks/useStories';
|
||||||
useStories,
|
|
||||||
useCreateStory,
|
|
||||||
useUpdateStory,
|
|
||||||
useDeleteStory,
|
|
||||||
} from '@/lib/hooks/useStories';
|
|
||||||
import { useStoryStore } from '@/stores/storyStore';
|
|
||||||
import { cn } from '@/lib/utils/cn';
|
import { cn } from '@/lib/utils/cn';
|
||||||
import { formatDate } from '@/lib/utils/format';
|
import { formatDate } from '@/lib/utils/format';
|
||||||
|
import { useStoryStore } from '@/stores/storyStore';
|
||||||
|
|
||||||
export function StoryList() {
|
export function StoryList() {
|
||||||
const { data: stories, isLoading } = useStories();
|
const { data: stories, isLoading } = useStories();
|
||||||
@@ -49,7 +44,11 @@ export function StoryList() {
|
|||||||
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
const [createDialogOpen, setCreateDialogOpen] = useState(false);
|
||||||
const [editDialogOpen, setEditDialogOpen] = useState(false);
|
const [editDialogOpen, setEditDialogOpen] = useState(false);
|
||||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||||
const [editingStory, setEditingStory] = useState<{ id: string; name: string; description?: string } | null>(null);
|
const [editingStory, setEditingStory] = useState<{
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
description?: string;
|
||||||
|
} | null>(null);
|
||||||
const [deletingStoryId, setDeletingStoryId] = useState<string | null>(null);
|
const [deletingStoryId, setDeletingStoryId] = useState<string | null>(null);
|
||||||
const [newStoryName, setNewStoryName] = useState('');
|
const [newStoryName, setNewStoryName] = useState('');
|
||||||
const [newStoryDescription, setNewStoryDescription] = useState('');
|
const [newStoryDescription, setNewStoryDescription] = useState('');
|
||||||
@@ -186,7 +185,7 @@ export function StoryList() {
|
|||||||
{/* Story List */}
|
{/* Story List */}
|
||||||
<div className="flex-1 min-h-0 overflow-y-auto space-y-2">
|
<div className="flex-1 min-h-0 overflow-y-auto space-y-2">
|
||||||
{storyList.length === 0 ? (
|
{storyList.length === 0 ? (
|
||||||
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-md text-muted-foreground">
|
<div className="text-center py-12 px-5 border-2 border-dashed border-muted rounded-2xl text-muted-foreground">
|
||||||
<BookOpen className="h-12 w-12 mx-auto mb-4 opacity-50" />
|
<BookOpen className="h-12 w-12 mx-auto mb-4 opacity-50" />
|
||||||
<p className="text-sm">No stories yet</p>
|
<p className="text-sm">No stories yet</p>
|
||||||
<p className="text-xs mt-2">Create your first story to get started</p>
|
<p className="text-xs mt-2">Create your first story to get started</p>
|
||||||
@@ -196,7 +195,7 @@ export function StoryList() {
|
|||||||
<div
|
<div
|
||||||
key={story.id}
|
key={story.id}
|
||||||
className={cn(
|
className={cn(
|
||||||
'h-24 p-4 border rounded-md transition-colors group flex items-center',
|
'h-24 p-4 border rounded-2xl transition-colors group flex items-center',
|
||||||
selectedStoryId === story.id && 'bg-muted border-primary',
|
selectedStoryId === story.id && 'bg-muted border-primary',
|
||||||
)}
|
)}
|
||||||
>
|
>
|
||||||
@@ -213,7 +212,9 @@ export function StoryList() {
|
|||||||
</p>
|
</p>
|
||||||
)}
|
)}
|
||||||
<div className="flex items-center gap-3 mt-2 text-xs text-muted-foreground">
|
<div className="flex items-center gap-3 mt-2 text-xs text-muted-foreground">
|
||||||
<span>{story.item_count} {story.item_count === 1 ? 'item' : 'items'}</span>
|
<span>
|
||||||
|
{story.item_count} {story.item_count === 1 ? 'item' : 'items'}
|
||||||
|
</span>
|
||||||
<span>•</span>
|
<span>•</span>
|
||||||
<span>{formatDate(story.updated_at)}</span>
|
<span>{formatDate(story.updated_at)}</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -300,9 +301,7 @@ export function StoryList() {
|
|||||||
<DialogContent>
|
<DialogContent>
|
||||||
<DialogHeader>
|
<DialogHeader>
|
||||||
<DialogTitle>Edit Story</DialogTitle>
|
<DialogTitle>Edit Story</DialogTitle>
|
||||||
<DialogDescription>
|
<DialogDescription>Update the story name and description.</DialogDescription>
|
||||||
Update the story name and description.
|
|
||||||
</DialogDescription>
|
|
||||||
</DialogHeader>
|
</DialogHeader>
|
||||||
<div className="space-y-4 py-4">
|
<div className="space-y-4 py-4">
|
||||||
<div className="space-y-2">
|
<div className="space-y-2">
|
||||||
@@ -347,7 +346,8 @@ export function StoryList() {
|
|||||||
<AlertDialogHeader>
|
<AlertDialogHeader>
|
||||||
<AlertDialogTitle>Are you sure?</AlertDialogTitle>
|
<AlertDialogTitle>Are you sure?</AlertDialogTitle>
|
||||||
<AlertDialogDescription>
|
<AlertDialogDescription>
|
||||||
This will permanently delete the story and all its items. This action cannot be undone.
|
This will permanently delete the story and all its items. This action cannot be
|
||||||
|
undone.
|
||||||
</AlertDialogDescription>
|
</AlertDialogDescription>
|
||||||
</AlertDialogHeader>
|
</AlertDialogHeader>
|
||||||
<AlertDialogFooter>
|
<AlertDialogFooter>
|
||||||
|
|||||||
@@ -111,6 +111,7 @@ interface StoryTrackEditorProps {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const TRACK_HEIGHT = 48;
|
const TRACK_HEIGHT = 48;
|
||||||
|
const TIME_RULER_HEIGHT = 24; // h-6 = 1.5rem = 24px
|
||||||
const MIN_PIXELS_PER_SECOND = 10;
|
const MIN_PIXELS_PER_SECOND = 10;
|
||||||
const MAX_PIXELS_PER_SECOND = 200;
|
const MAX_PIXELS_PER_SECOND = 200;
|
||||||
const DEFAULT_PIXELS_PER_SECOND = 50;
|
const DEFAULT_PIXELS_PER_SECOND = 50;
|
||||||
@@ -539,7 +540,10 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (e.key === 'Escape') {
|
if (e.key === ' ') {
|
||||||
|
e.preventDefault();
|
||||||
|
handlePlayPause();
|
||||||
|
} else if (e.key === 'Escape') {
|
||||||
setSelectedClipId(null);
|
setSelectedClipId(null);
|
||||||
} else if (e.key === 's' || e.key === 'S') {
|
} else if (e.key === 's' || e.key === 'S') {
|
||||||
if (selectedClipId) {
|
if (selectedClipId) {
|
||||||
@@ -561,7 +565,14 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
|
|
||||||
window.addEventListener('keydown', handleKeyDown);
|
window.addEventListener('keydown', handleKeyDown);
|
||||||
return () => window.removeEventListener('keydown', handleKeyDown);
|
return () => window.removeEventListener('keydown', handleKeyDown);
|
||||||
}, [selectedClipId, handleSplit, handleDuplicate, handleDelete, setSelectedClipId]);
|
}, [
|
||||||
|
selectedClipId,
|
||||||
|
handleSplit,
|
||||||
|
handleDuplicate,
|
||||||
|
handleDelete,
|
||||||
|
setSelectedClipId,
|
||||||
|
handlePlayPause,
|
||||||
|
]);
|
||||||
|
|
||||||
// Add global mouse listeners for trimming
|
// Add global mouse listeners for trimming
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -586,7 +597,8 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
});
|
});
|
||||||
setDragPosition({
|
setDragPosition({
|
||||||
x: rect.left - tracksRef.current.getBoundingClientRect().left + tracksRef.current.scrollLeft,
|
x: rect.left - tracksRef.current.getBoundingClientRect().left + tracksRef.current.scrollLeft,
|
||||||
y: rect.top - tracksRef.current.getBoundingClientRect().top,
|
// Subtract ruler height since clips are positioned relative to tracks area, not the scrollable container
|
||||||
|
y: rect.top - tracksRef.current.getBoundingClientRect().top - TIME_RULER_HEIGHT,
|
||||||
});
|
});
|
||||||
setDraggingItem(item.id);
|
setDraggingItem(item.id);
|
||||||
};
|
};
|
||||||
@@ -597,7 +609,8 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
|
|
||||||
const rect = tracksRef.current.getBoundingClientRect();
|
const rect = tracksRef.current.getBoundingClientRect();
|
||||||
const x = e.clientX - rect.left + tracksRef.current.scrollLeft - dragOffset.x;
|
const x = e.clientX - rect.left + tracksRef.current.scrollLeft - dragOffset.x;
|
||||||
const y = e.clientY - rect.top - dragOffset.y;
|
// Subtract ruler height since clips are positioned relative to tracks area
|
||||||
|
const y = e.clientY - rect.top - dragOffset.y - TIME_RULER_HEIGHT;
|
||||||
|
|
||||||
setDragPosition({ x: Math.max(0, x), y });
|
setDragPosition({ x: Math.max(0, x), y });
|
||||||
},
|
},
|
||||||
@@ -673,6 +686,22 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
// Playhead position
|
// Playhead position
|
||||||
const playheadLeft = msToPixels(currentTimeMs);
|
const playheadLeft = msToPixels(currentTimeMs);
|
||||||
|
|
||||||
|
// Auto-scroll timeline to follow playhead during playback
|
||||||
|
useEffect(() => {
|
||||||
|
if (!isCurrentlyPlaying || !tracksRef.current) return;
|
||||||
|
|
||||||
|
const container = tracksRef.current;
|
||||||
|
const containerWidth = container.clientWidth;
|
||||||
|
const scrollLeft = container.scrollLeft;
|
||||||
|
const halfwayPoint = scrollLeft + containerWidth / 2;
|
||||||
|
|
||||||
|
// If playhead is past the halfway point, scroll to keep it centered
|
||||||
|
if (playheadLeft > halfwayPoint) {
|
||||||
|
const targetScroll = playheadLeft - containerWidth / 2;
|
||||||
|
container.scrollLeft = targetScroll;
|
||||||
|
}
|
||||||
|
}, [isCurrentlyPlaying, playheadLeft]);
|
||||||
|
|
||||||
// Calculate tracks area height
|
// Calculate tracks area height
|
||||||
const tracksAreaHeight = tracks.length * TRACK_HEIGHT;
|
const tracksAreaHeight = tracks.length * TRACK_HEIGHT;
|
||||||
const timelineContainerHeight = editorHeight - 40; // Subtract toolbar height
|
const timelineContainerHeight = editorHeight - 40; // Subtract toolbar height
|
||||||
@@ -701,7 +730,13 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
|||||||
<div className="flex items-center justify-between px-3 py-2 border-b bg-muted/30 mt-2">
|
<div className="flex items-center justify-between px-3 py-2 border-b bg-muted/30 mt-2">
|
||||||
{/* Play controls - left side */}
|
{/* Play controls - left side */}
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex items-center gap-2">
|
||||||
<Button variant="ghost" size="icon" className="h-7 w-7" onClick={handlePlayPause}>
|
<Button
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-7 w-7"
|
||||||
|
onClick={handlePlayPause}
|
||||||
|
title="Play/Pause (Space)"
|
||||||
|
>
|
||||||
{isCurrentlyPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
|
{isCurrentlyPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
|
||||||
</Button>
|
</Button>
|
||||||
<Button
|
<Button
|
||||||
|
|||||||
@@ -1,8 +1,31 @@
|
|||||||
import { Mic, Pause, Play, Square } from 'lucide-react';
|
import { Mic, Pause, Play, Square } from 'lucide-react';
|
||||||
|
import { memo, useEffect, useState } from 'react';
|
||||||
|
import { Visualizer } from 'react-sound-visualizer';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||||
|
|
||||||
|
const MemoizedWaveform = memo(function MemoizedWaveform({
|
||||||
|
audioStream,
|
||||||
|
}: {
|
||||||
|
audioStream: MediaStream;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
|
||||||
|
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
|
||||||
|
{({ canvasRef }) => (
|
||||||
|
<canvas
|
||||||
|
ref={canvasRef}
|
||||||
|
width={500}
|
||||||
|
height={150}
|
||||||
|
className="w-full h-full"
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</Visualizer>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
interface AudioSampleRecordingProps {
|
interface AudioSampleRecordingProps {
|
||||||
file: File | null | undefined;
|
file: File | null | undefined;
|
||||||
isRecording: boolean;
|
isRecording: boolean;
|
||||||
@@ -14,6 +37,7 @@ interface AudioSampleRecordingProps {
|
|||||||
onPlayPause: () => void;
|
onPlayPause: () => void;
|
||||||
isPlaying: boolean;
|
isPlaying: boolean;
|
||||||
isTranscribing?: boolean;
|
isTranscribing?: boolean;
|
||||||
|
showWaveform?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function AudioSampleRecording({
|
export function AudioSampleRecording({
|
||||||
@@ -27,29 +51,67 @@ export function AudioSampleRecording({
|
|||||||
onPlayPause,
|
onPlayPause,
|
||||||
isPlaying,
|
isPlaying,
|
||||||
isTranscribing = false,
|
isTranscribing = false,
|
||||||
|
showWaveform = true,
|
||||||
}: AudioSampleRecordingProps) {
|
}: AudioSampleRecordingProps) {
|
||||||
|
const [audioStream, setAudioStream] = useState<MediaStream | null>(null);
|
||||||
|
|
||||||
|
// Request microphone access when component mounts
|
||||||
|
useEffect(() => {
|
||||||
|
if (!showWaveform) return;
|
||||||
|
|
||||||
|
let stream: MediaStream | null = null;
|
||||||
|
|
||||||
|
navigator.mediaDevices
|
||||||
|
.getUserMedia({ audio: true, video: false })
|
||||||
|
.then((s) => {
|
||||||
|
stream = s;
|
||||||
|
setAudioStream(s);
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
console.warn('Could not access microphone for visualization:', err);
|
||||||
|
});
|
||||||
|
|
||||||
|
return () => {
|
||||||
|
if (stream) {
|
||||||
|
stream.getTracks().forEach((track) => {
|
||||||
|
track.stop();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}, [showWaveform]);
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<FormItem>
|
<FormItem>
|
||||||
<FormLabel>Record Audio</FormLabel>
|
|
||||||
<FormControl>
|
<FormControl>
|
||||||
<div className="space-y-4">
|
<div className="space-y-4">
|
||||||
{!isRecording && !file && (
|
{!isRecording && !file && (
|
||||||
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px]">
|
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
|
||||||
<Button type="button" onClick={onStart} size="lg" className="flex items-center gap-2">
|
{showWaveform && audioStream && (
|
||||||
|
<MemoizedWaveform audioStream={audioStream} />
|
||||||
|
)}
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
onClick={onStart}
|
||||||
|
size="lg"
|
||||||
|
className="relative z-10 flex items-center gap-2"
|
||||||
|
>
|
||||||
<Mic className="h-5 w-5" />
|
<Mic className="h-5 w-5" />
|
||||||
Start Recording
|
Start Recording
|
||||||
</Button>
|
</Button>
|
||||||
<p className="text-sm text-muted-foreground text-center">
|
<p className="relative z-10 text-sm text-muted-foreground text-center">
|
||||||
Click to start recording. Maximum duration: 30 seconds.
|
Click to start recording. Maximum duration: 30 seconds.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{isRecording && (
|
{isRecording && (
|
||||||
<div className="flex flex-col items-center justify-center gap-4 p-4 border-2 border-destructive rounded-lg bg-destructive/5 min-h-[180px]">
|
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
|
||||||
<div className="flex items-center gap-4">
|
{showWaveform && audioStream && (
|
||||||
|
<MemoizedWaveform audioStream={audioStream} />
|
||||||
|
)}
|
||||||
|
<div className="relative z-10 flex items-center gap-4">
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex items-center gap-2">
|
||||||
<div className="h-3 w-3 rounded-full bg-destructive animate-pulse" />
|
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
|
||||||
<span className="text-lg font-mono font-semibold">
|
<span className="text-lg font-mono font-semibold">
|
||||||
{formatAudioDuration(duration)}
|
{formatAudioDuration(duration)}
|
||||||
</span>
|
</span>
|
||||||
@@ -58,13 +120,12 @@ export function AudioSampleRecording({
|
|||||||
<Button
|
<Button
|
||||||
type="button"
|
type="button"
|
||||||
onClick={onStop}
|
onClick={onStop}
|
||||||
variant="destructive"
|
className="relative z-10 flex items-center gap-2 bg-accent text-accent-foreground hover:bg-accent/90"
|
||||||
className="flex items-center gap-2"
|
|
||||||
>
|
>
|
||||||
<Square className="h-4 w-4" />
|
<Square className="h-4 w-4" />
|
||||||
Stop Recording
|
Stop Recording
|
||||||
</Button>
|
</Button>
|
||||||
<p className="text-sm text-muted-foreground text-center">
|
<p className="relative z-10 text-sm text-muted-foreground text-center">
|
||||||
{formatAudioDuration(30 - duration)} remaining
|
{formatAudioDuration(30 - duration)} remaining
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import { Mic, Monitor, Pause, Play, Square } from 'lucide-react';
|
import { Mic, Monitor, Pause, Play, Square } from 'lucide-react';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||||
import { formatAudioDuration } from '@/lib/utils/audio';
|
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||||
|
|
||||||
interface AudioSampleSystemProps {
|
interface AudioSampleSystemProps {
|
||||||
@@ -30,7 +30,6 @@ export function AudioSampleSystem({
|
|||||||
}: AudioSampleSystemProps) {
|
}: AudioSampleSystemProps) {
|
||||||
return (
|
return (
|
||||||
<FormItem>
|
<FormItem>
|
||||||
<FormLabel>Capture System Audio</FormLabel>
|
|
||||||
<FormControl>
|
<FormControl>
|
||||||
<div className="space-y-4">
|
<div className="space-y-4">
|
||||||
{!isRecording && !file && (
|
{!isRecording && !file && (
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { Mic, Pause, Play, Upload } from 'lucide-react';
|
import { Mic, Pause, Play, Upload } from 'lucide-react';
|
||||||
import { useRef, useState } from 'react';
|
import { useRef, useState } from 'react';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import { FormControl, FormItem, FormLabel, FormMessage } from '@/components/ui/form';
|
import { FormControl, FormItem, FormMessage } from '@/components/ui/form';
|
||||||
|
|
||||||
interface AudioSampleUploadProps {
|
interface AudioSampleUploadProps {
|
||||||
file: File | null | undefined;
|
file: File | null | undefined;
|
||||||
@@ -31,7 +31,6 @@ export function AudioSampleUpload({
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<FormItem>
|
<FormItem>
|
||||||
<FormLabel>Audio File</FormLabel>
|
|
||||||
<FormControl>
|
<FormControl>
|
||||||
<div className="flex flex-col gap-2">
|
<div className="flex flex-col gap-2">
|
||||||
<input
|
<input
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import {
|
|||||||
import type { VoiceProfileResponse } from '@/lib/api/types';
|
import type { VoiceProfileResponse } from '@/lib/api/types';
|
||||||
import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
|
import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
|
||||||
import { cn } from '@/lib/utils/cn';
|
import { cn } from '@/lib/utils/cn';
|
||||||
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
import { useUIStore } from '@/stores/uiStore';
|
import { useUIStore } from '@/stores/uiStore';
|
||||||
|
|
||||||
interface ProfileCardProps {
|
interface ProfileCardProps {
|
||||||
@@ -23,15 +24,19 @@ interface ProfileCardProps {
|
|||||||
|
|
||||||
export function ProfileCard({ profile }: ProfileCardProps) {
|
export function ProfileCard({ profile }: ProfileCardProps) {
|
||||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||||
|
const [avatarError, setAvatarError] = useState(false);
|
||||||
const deleteProfile = useDeleteProfile();
|
const deleteProfile = useDeleteProfile();
|
||||||
const exportProfile = useExportProfile();
|
const exportProfile = useExportProfile();
|
||||||
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
||||||
const setProfileDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
|
const setProfileDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||||
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
|
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
|
||||||
|
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||||
|
|
||||||
const isSelected = selectedProfileId === profile.id;
|
const isSelected = selectedProfileId === profile.id;
|
||||||
|
|
||||||
|
const avatarUrl = profile.avatar_path ? `${serverUrl}/profiles/${profile.id}/avatar` : null;
|
||||||
|
|
||||||
const handleSelect = () => {
|
const handleSelect = () => {
|
||||||
setSelectedProfileId(isSelected ? null : profile.id);
|
setSelectedProfileId(isSelected ? null : profile.id);
|
||||||
};
|
};
|
||||||
@@ -67,8 +72,20 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
|||||||
>
|
>
|
||||||
<CardHeader className="p-3 pb-2">
|
<CardHeader className="p-3 pb-2">
|
||||||
<CardTitle className="flex items-center gap-1.5 text-base font-medium">
|
<CardTitle className="flex items-center gap-1.5 text-base font-medium">
|
||||||
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0">
|
<div className="h-6 w-6 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden">
|
||||||
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
|
{avatarUrl && !avatarError ? (
|
||||||
|
<img
|
||||||
|
src={avatarUrl}
|
||||||
|
alt={`${profile.name} avatar`}
|
||||||
|
className={cn(
|
||||||
|
'h-full w-full object-cover transition-all duration-200',
|
||||||
|
!isSelected && 'grayscale',
|
||||||
|
)}
|
||||||
|
onError={() => setAvatarError(true)}
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<Mic className="h-3.5 w-3.5 text-muted-foreground" />
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
<span className="break-words">{profile.name}</span>
|
<span className="break-words">{profile.name}</span>
|
||||||
</CardTitle>
|
</CardTitle>
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import { zodResolver } from '@hookform/resolvers/zod';
|
import { zodResolver } from '@hookform/resolvers/zod';
|
||||||
import { Mic, Monitor, Upload } from 'lucide-react';
|
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
|
||||||
import { useEffect, useState } from 'react';
|
import { useEffect, useRef, useState } from 'react';
|
||||||
import { useForm } from 'react-hook-form';
|
import { useForm } from 'react-hook-form';
|
||||||
import * as z from 'zod';
|
import * as z from 'zod';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
@@ -36,14 +36,17 @@ import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
|
|||||||
import {
|
import {
|
||||||
useAddSample,
|
useAddSample,
|
||||||
useCreateProfile,
|
useCreateProfile,
|
||||||
|
useDeleteAvatar,
|
||||||
useProfile,
|
useProfile,
|
||||||
useUpdateProfile,
|
useUpdateProfile,
|
||||||
|
useUploadAvatar,
|
||||||
} from '@/lib/hooks/useProfiles';
|
} from '@/lib/hooks/useProfiles';
|
||||||
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
|
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
|
||||||
import { useTranscription } from '@/lib/hooks/useTranscription';
|
import { useTranscription } from '@/lib/hooks/useTranscription';
|
||||||
import { isTauri } from '@/lib/tauri';
|
|
||||||
import { formatAudioDuration, getAudioDuration } from '@/lib/utils/audio';
|
import { formatAudioDuration, getAudioDuration } from '@/lib/utils/audio';
|
||||||
import { useUIStore } from '@/stores/uiStore';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
|
import { type ProfileFormDraft, useUIStore } from '@/stores/uiStore';
|
||||||
import { AudioSampleRecording } from './AudioSampleRecording';
|
import { AudioSampleRecording } from './AudioSampleRecording';
|
||||||
import { AudioSampleSystem } from './AudioSampleSystem';
|
import { AudioSampleSystem } from './AudioSampleSystem';
|
||||||
import { AudioSampleUpload } from './AudioSampleUpload';
|
import { AudioSampleUpload } from './AudioSampleUpload';
|
||||||
@@ -57,6 +60,7 @@ const baseProfileSchema = z.object({
|
|||||||
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
||||||
sampleFile: z.instanceof(File).optional(),
|
sampleFile: z.instanceof(File).optional(),
|
||||||
referenceText: z.string().max(1000).optional(),
|
referenceText: z.string().max(1000).optional(),
|
||||||
|
avatarFile: z.instanceof(File).optional(),
|
||||||
});
|
});
|
||||||
|
|
||||||
const profileSchema = baseProfileSchema.refine(
|
const profileSchema = baseProfileSchema.refine(
|
||||||
@@ -75,22 +79,52 @@ const profileSchema = baseProfileSchema.refine(
|
|||||||
|
|
||||||
type ProfileFormValues = z.infer<typeof profileSchema>;
|
type ProfileFormValues = z.infer<typeof profileSchema>;
|
||||||
|
|
||||||
|
// Helper to convert File to base64
|
||||||
|
async function fileToBase64(file: File): Promise<string> {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const reader = new FileReader();
|
||||||
|
reader.onload = () => resolve(reader.result as string);
|
||||||
|
reader.onerror = reject;
|
||||||
|
reader.readAsDataURL(file);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper to convert base64 to File
|
||||||
|
function base64ToFile(base64: string, fileName: string, fileType: string): File {
|
||||||
|
const arr = base64.split(',');
|
||||||
|
const bstr = atob(arr[1]);
|
||||||
|
let n = bstr.length;
|
||||||
|
const u8arr = new Uint8Array(n);
|
||||||
|
while (n--) {
|
||||||
|
u8arr[n] = bstr.charCodeAt(n);
|
||||||
|
}
|
||||||
|
return new File([u8arr], fileName, { type: fileType });
|
||||||
|
}
|
||||||
|
|
||||||
export function ProfileForm() {
|
export function ProfileForm() {
|
||||||
|
const platform = usePlatform();
|
||||||
const open = useUIStore((state) => state.profileDialogOpen);
|
const open = useUIStore((state) => state.profileDialogOpen);
|
||||||
const setOpen = useUIStore((state) => state.setProfileDialogOpen);
|
const setOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||||
const editingProfileId = useUIStore((state) => state.editingProfileId);
|
const editingProfileId = useUIStore((state) => state.editingProfileId);
|
||||||
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
const setEditingProfileId = useUIStore((state) => state.setEditingProfileId);
|
||||||
|
const profileFormDraft = useUIStore((state) => state.profileFormDraft);
|
||||||
|
const setProfileFormDraft = useUIStore((state) => state.setProfileFormDraft);
|
||||||
const { data: editingProfile } = useProfile(editingProfileId || '');
|
const { data: editingProfile } = useProfile(editingProfileId || '');
|
||||||
const createProfile = useCreateProfile();
|
const createProfile = useCreateProfile();
|
||||||
const updateProfile = useUpdateProfile();
|
const updateProfile = useUpdateProfile();
|
||||||
const addSample = useAddSample();
|
const addSample = useAddSample();
|
||||||
|
const uploadAvatar = useUploadAvatar();
|
||||||
|
const deleteAvatar = useDeleteAvatar();
|
||||||
const transcribe = useTranscription();
|
const transcribe = useTranscription();
|
||||||
const { toast } = useToast();
|
const { toast } = useToast();
|
||||||
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('upload');
|
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
|
||||||
const [audioDuration, setAudioDuration] = useState<number | null>(null);
|
const [audioDuration, setAudioDuration] = useState<number | null>(null);
|
||||||
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
|
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
|
||||||
|
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
|
||||||
|
const avatarInputRef = useRef<HTMLInputElement>(null);
|
||||||
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
|
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
|
||||||
const isCreating = !editingProfileId;
|
const isCreating = !editingProfileId;
|
||||||
|
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||||
|
|
||||||
const form = useForm<ProfileFormValues>({
|
const form = useForm<ProfileFormValues>({
|
||||||
resolver: zodResolver(profileSchema),
|
resolver: zodResolver(profileSchema),
|
||||||
@@ -100,10 +134,12 @@ export function ProfileForm() {
|
|||||||
language: 'en',
|
language: 'en',
|
||||||
sampleFile: undefined,
|
sampleFile: undefined,
|
||||||
referenceText: '',
|
referenceText: '',
|
||||||
|
avatarFile: undefined,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
const selectedFile = form.watch('sampleFile');
|
const selectedFile = form.watch('sampleFile');
|
||||||
|
const selectedAvatarFile = form.watch('avatarFile');
|
||||||
|
|
||||||
// Validate audio duration when file is selected
|
// Validate audio duration when file is selected
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -220,6 +256,20 @@ export function ProfileForm() {
|
|||||||
}
|
}
|
||||||
}, [systemRecordingError, toast]);
|
}, [systemRecordingError, toast]);
|
||||||
|
|
||||||
|
// Handle avatar preview
|
||||||
|
useEffect(() => {
|
||||||
|
if (selectedAvatarFile instanceof File) {
|
||||||
|
const url = URL.createObjectURL(selectedAvatarFile);
|
||||||
|
setAvatarPreview(url);
|
||||||
|
return () => URL.revokeObjectURL(url);
|
||||||
|
} else if (editingProfile?.avatar_path) {
|
||||||
|
setAvatarPreview(`${serverUrl}/profiles/${editingProfile.id}/avatar`);
|
||||||
|
} else {
|
||||||
|
setAvatarPreview(null);
|
||||||
|
}
|
||||||
|
}, [selectedAvatarFile, editingProfile, serverUrl]);
|
||||||
|
|
||||||
|
// Restore form state from draft or editing profile
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (editingProfile) {
|
if (editingProfile) {
|
||||||
form.reset({
|
form.reset({
|
||||||
@@ -228,18 +278,46 @@ export function ProfileForm() {
|
|||||||
language: editingProfile.language as LanguageCode,
|
language: editingProfile.language as LanguageCode,
|
||||||
sampleFile: undefined,
|
sampleFile: undefined,
|
||||||
referenceText: undefined,
|
referenceText: undefined,
|
||||||
|
avatarFile: undefined,
|
||||||
});
|
});
|
||||||
} else {
|
} else if (profileFormDraft && open) {
|
||||||
|
// Restore from draft when opening in create mode
|
||||||
|
form.reset({
|
||||||
|
name: profileFormDraft.name,
|
||||||
|
description: profileFormDraft.description,
|
||||||
|
language: profileFormDraft.language as LanguageCode,
|
||||||
|
referenceText: profileFormDraft.referenceText,
|
||||||
|
sampleFile: undefined,
|
||||||
|
avatarFile: undefined,
|
||||||
|
});
|
||||||
|
setSampleMode(profileFormDraft.sampleMode);
|
||||||
|
// Restore the file if we have it saved
|
||||||
|
if (
|
||||||
|
profileFormDraft.sampleFileData &&
|
||||||
|
profileFormDraft.sampleFileName &&
|
||||||
|
profileFormDraft.sampleFileType
|
||||||
|
) {
|
||||||
|
const file = base64ToFile(
|
||||||
|
profileFormDraft.sampleFileData,
|
||||||
|
profileFormDraft.sampleFileName,
|
||||||
|
profileFormDraft.sampleFileType,
|
||||||
|
);
|
||||||
|
form.setValue('sampleFile', file);
|
||||||
|
}
|
||||||
|
} else if (!open) {
|
||||||
|
// Only reset to defaults when modal is closed and no draft
|
||||||
form.reset({
|
form.reset({
|
||||||
name: '',
|
name: '',
|
||||||
description: '',
|
description: '',
|
||||||
language: 'en',
|
language: 'en',
|
||||||
sampleFile: undefined,
|
sampleFile: undefined,
|
||||||
referenceText: undefined,
|
referenceText: undefined,
|
||||||
|
avatarFile: undefined,
|
||||||
});
|
});
|
||||||
setSampleMode('upload');
|
setSampleMode('record');
|
||||||
|
setAvatarPreview(null);
|
||||||
}
|
}
|
||||||
}, [editingProfile, form]);
|
}, [editingProfile, profileFormDraft, open, form]);
|
||||||
|
|
||||||
async function handleTranscribe() {
|
async function handleTranscribe() {
|
||||||
const file = form.getValues('sampleFile');
|
const file = form.getValues('sampleFile');
|
||||||
@@ -281,6 +359,52 @@ export function ProfileForm() {
|
|||||||
playPause(file);
|
playPause(file);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function handleAvatarFileChange(e: React.ChangeEvent<HTMLInputElement>) {
|
||||||
|
const file = e.target.files?.[0];
|
||||||
|
if (file) {
|
||||||
|
if (!file.type.startsWith('image/')) {
|
||||||
|
toast({
|
||||||
|
title: 'Invalid file type',
|
||||||
|
description: 'Please select an image file (PNG, JPG, or WebP)',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (file.size > 5 * 1024 * 1024) {
|
||||||
|
toast({
|
||||||
|
title: 'File too large',
|
||||||
|
description: 'Image must be less than 5MB',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
form.setValue('avatarFile', file);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleRemoveAvatar() {
|
||||||
|
if (editingProfileId && editingProfile?.avatar_path) {
|
||||||
|
try {
|
||||||
|
await deleteAvatar.mutateAsync(editingProfileId);
|
||||||
|
toast({
|
||||||
|
title: 'Avatar removed',
|
||||||
|
description: 'Avatar image has been removed successfully.',
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
toast({
|
||||||
|
title: 'Failed to remove avatar',
|
||||||
|
description: error instanceof Error ? error.message : 'Unknown error',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
form.setValue('avatarFile', undefined);
|
||||||
|
setAvatarPreview(null);
|
||||||
|
if (avatarInputRef.current) {
|
||||||
|
avatarInputRef.current.value = '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function onSubmit(data: ProfileFormValues) {
|
async function onSubmit(data: ProfileFormValues) {
|
||||||
try {
|
try {
|
||||||
if (editingProfileId) {
|
if (editingProfileId) {
|
||||||
@@ -293,6 +417,24 @@ export function ProfileForm() {
|
|||||||
language: data.language,
|
language: data.language,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Handle avatar upload/update if file changed
|
||||||
|
if (data.avatarFile) {
|
||||||
|
try {
|
||||||
|
await uploadAvatar.mutateAsync({
|
||||||
|
profileId: editingProfileId,
|
||||||
|
file: data.avatarFile,
|
||||||
|
});
|
||||||
|
} catch (avatarError) {
|
||||||
|
toast({
|
||||||
|
title: 'Avatar upload failed',
|
||||||
|
description:
|
||||||
|
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
toast({
|
toast({
|
||||||
title: 'Voice updated',
|
title: 'Voice updated',
|
||||||
description: `"${data.name}" has been updated successfully.`,
|
description: `"${data.name}" has been updated successfully.`,
|
||||||
@@ -369,6 +511,24 @@ export function ProfileForm() {
|
|||||||
file: sampleFile,
|
file: sampleFile,
|
||||||
referenceText: referenceText,
|
referenceText: referenceText,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Handle avatar upload if provided
|
||||||
|
if (data.avatarFile) {
|
||||||
|
try {
|
||||||
|
await uploadAvatar.mutateAsync({
|
||||||
|
profileId: profile.id,
|
||||||
|
file: data.avatarFile,
|
||||||
|
});
|
||||||
|
} catch (avatarError) {
|
||||||
|
toast({
|
||||||
|
title: 'Avatar upload failed',
|
||||||
|
description:
|
||||||
|
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
toast({
|
toast({
|
||||||
title: 'Profile created',
|
title: 'Profile created',
|
||||||
description: `"${data.name}" has been created with a sample.`,
|
description: `"${data.name}" has been created with a sample.`,
|
||||||
@@ -383,6 +543,8 @@ export function ProfileForm() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Clear draft and reset form on success
|
||||||
|
setProfileFormDraft(null);
|
||||||
form.reset();
|
form.reset();
|
||||||
setEditingProfileId(null);
|
setEditingProfileId(null);
|
||||||
setOpen(false);
|
setOpen(false);
|
||||||
@@ -395,12 +557,41 @@ export function ProfileForm() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function handleOpenChange(open: boolean) {
|
async function handleOpenChange(newOpen: boolean) {
|
||||||
setOpen(open);
|
if (!newOpen && isCreating) {
|
||||||
if (!open) {
|
// Save draft when closing the create modal
|
||||||
|
const values = form.getValues();
|
||||||
|
const hasContent =
|
||||||
|
values.name || values.description || values.referenceText || values.sampleFile;
|
||||||
|
|
||||||
|
if (hasContent) {
|
||||||
|
const draft: ProfileFormDraft = {
|
||||||
|
name: values.name || '',
|
||||||
|
description: values.description || '',
|
||||||
|
language: values.language || 'en',
|
||||||
|
referenceText: values.referenceText || '',
|
||||||
|
sampleMode,
|
||||||
|
};
|
||||||
|
|
||||||
|
// Save file as base64 if present
|
||||||
|
if (values.sampleFile) {
|
||||||
|
try {
|
||||||
|
draft.sampleFileName = values.sampleFile.name;
|
||||||
|
draft.sampleFileType = values.sampleFile.type;
|
||||||
|
draft.sampleFileData = await fileToBase64(values.sampleFile);
|
||||||
|
} catch {
|
||||||
|
// If file conversion fails, just don't save the file
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
setProfileFormDraft(draft);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
setOpen(newOpen);
|
||||||
|
if (!newOpen) {
|
||||||
setEditingProfileId(null);
|
setEditingProfileId(null);
|
||||||
form.reset();
|
// Don't reset form here - let the effect handle it based on draft state
|
||||||
setSampleMode('upload');
|
|
||||||
if (isRecording) {
|
if (isRecording) {
|
||||||
cancelRecording();
|
cancelRecording();
|
||||||
}
|
}
|
||||||
@@ -413,174 +604,119 @@ export function ProfileForm() {
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<Dialog open={open} onOpenChange={handleOpenChange}>
|
<Dialog open={open} onOpenChange={handleOpenChange}>
|
||||||
<DialogContent className="max-w-4xl">
|
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
|
||||||
<DialogHeader>
|
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
|
||||||
<DialogTitle>{editingProfileId ? 'Edit Voice' : 'Create Voice Profile'}</DialogTitle>
|
<DialogHeader>
|
||||||
<DialogDescription>
|
<DialogTitle className="text-2xl">
|
||||||
{editingProfileId
|
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
|
||||||
? 'Update your voice profile details and manage samples.'
|
</DialogTitle>
|
||||||
: 'Create a new voice profile with an audio sample to clone the voice.'}
|
<DialogDescription>
|
||||||
</DialogDescription>
|
{editingProfileId
|
||||||
</DialogHeader>
|
? 'Update your voice profile details and manage samples.'
|
||||||
|
: 'Create a new voice profile with an audio sample to clone the voice.'}
|
||||||
<Form {...form}>
|
</DialogDescription>
|
||||||
<form onSubmit={form.handleSubmit(onSubmit)}>
|
{isCreating && profileFormDraft && (
|
||||||
<div className="grid gap-6 grid-cols-2">
|
<div className="flex items-center gap-2 pt-2">
|
||||||
{/* Left column: Profile info */}
|
<span className="text-xs text-muted-foreground">Draft restored</span>
|
||||||
<div className="space-y-4">
|
<Button
|
||||||
<FormField
|
type="button"
|
||||||
control={form.control}
|
variant="ghost"
|
||||||
name="name"
|
size="sm"
|
||||||
render={({ field }) => (
|
className="h-6 px-2 text-xs text-muted-foreground"
|
||||||
<FormItem>
|
onClick={() => {
|
||||||
<FormLabel>Name</FormLabel>
|
setProfileFormDraft(null);
|
||||||
<FormControl>
|
form.reset({
|
||||||
<Input placeholder="My Voice" {...field} />
|
name: '',
|
||||||
</FormControl>
|
description: '',
|
||||||
<FormMessage />
|
language: 'en',
|
||||||
</FormItem>
|
sampleFile: undefined,
|
||||||
)}
|
referenceText: '',
|
||||||
/>
|
});
|
||||||
|
setSampleMode('record');
|
||||||
<FormField
|
}}
|
||||||
control={form.control}
|
>
|
||||||
name="description"
|
<X className="h-3 w-3 mr-1" />
|
||||||
render={({ field }) => (
|
Discard
|
||||||
<FormItem>
|
</Button>
|
||||||
<FormLabel>Description (Optional)</FormLabel>
|
|
||||||
<FormControl>
|
|
||||||
<Textarea placeholder="Describe this voice..." {...field} />
|
|
||||||
</FormControl>
|
|
||||||
<FormMessage />
|
|
||||||
</FormItem>
|
|
||||||
)}
|
|
||||||
/>
|
|
||||||
|
|
||||||
<FormField
|
|
||||||
control={form.control}
|
|
||||||
name="language"
|
|
||||||
render={({ field }) => (
|
|
||||||
<FormItem>
|
|
||||||
<FormLabel>Language</FormLabel>
|
|
||||||
<Select onValueChange={field.onChange} defaultValue={field.value}>
|
|
||||||
<FormControl>
|
|
||||||
<SelectTrigger>
|
|
||||||
<SelectValue />
|
|
||||||
</SelectTrigger>
|
|
||||||
</FormControl>
|
|
||||||
<SelectContent>
|
|
||||||
{LANGUAGE_OPTIONS.map((lang) => (
|
|
||||||
<SelectItem key={lang.value} value={lang.value}>
|
|
||||||
{lang.label}
|
|
||||||
</SelectItem>
|
|
||||||
))}
|
|
||||||
</SelectContent>
|
|
||||||
</Select>
|
|
||||||
<FormMessage />
|
|
||||||
</FormItem>
|
|
||||||
)}
|
|
||||||
/>
|
|
||||||
</div>
|
</div>
|
||||||
|
)}
|
||||||
|
</DialogHeader>
|
||||||
|
|
||||||
{/* Right column: Sample management */}
|
<Form {...form}>
|
||||||
<div className="space-y-4 border-l pl-6">
|
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
|
||||||
{isCreating ? (
|
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
|
||||||
<>
|
{/* Left column: Sample management */}
|
||||||
<div>
|
<div className="space-y-4 border-r pr-6">
|
||||||
<h3 className="text-sm font-medium mb-2">Add Sample</h3>
|
{isCreating ? (
|
||||||
<p className="text-sm text-muted-foreground mb-4">
|
<>
|
||||||
Provide an audio sample to clone the voice. You can add more samples later.
|
<Tabs
|
||||||
</p>
|
className="pt-4"
|
||||||
</div>
|
value={sampleMode}
|
||||||
|
onValueChange={(v) => {
|
||||||
<Tabs
|
const newMode = v as 'upload' | 'record' | 'system';
|
||||||
value={sampleMode}
|
// Cancel any active recordings when switching modes
|
||||||
onValueChange={(v) => {
|
if (isRecording && newMode !== 'record') {
|
||||||
const newMode = v as 'upload' | 'record' | 'system';
|
cancelRecording();
|
||||||
// Cancel any active recordings when switching modes
|
}
|
||||||
if (isRecording && newMode !== 'record') {
|
if (isSystemRecording && newMode !== 'system') {
|
||||||
cancelRecording();
|
cancelSystemRecording();
|
||||||
}
|
}
|
||||||
if (isSystemRecording && newMode !== 'system') {
|
setSampleMode(newMode);
|
||||||
cancelSystemRecording();
|
}}
|
||||||
}
|
|
||||||
setSampleMode(newMode);
|
|
||||||
}}
|
|
||||||
>
|
|
||||||
<TabsList
|
|
||||||
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
|
||||||
>
|
>
|
||||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
<TabsList
|
||||||
<Upload className="h-4 w-4 shrink-0" />
|
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||||
Upload
|
>
|
||||||
</TabsTrigger>
|
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||||
<TabsTrigger value="record" className="flex items-center gap-2">
|
<Upload className="h-4 w-4 shrink-0" />
|
||||||
<Mic className="h-4 w-4 shrink-0" />
|
Upload
|
||||||
Record
|
|
||||||
</TabsTrigger>
|
|
||||||
{isTauri() && isSystemAudioSupported && (
|
|
||||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
|
||||||
<Monitor className="h-4 w-4 shrink-0" />
|
|
||||||
System Audio
|
|
||||||
</TabsTrigger>
|
</TabsTrigger>
|
||||||
)}
|
<TabsTrigger value="record" className="flex items-center gap-2">
|
||||||
</TabsList>
|
<Mic className="h-4 w-4 shrink-0" />
|
||||||
|
Record
|
||||||
<TabsContent value="upload" className="space-y-4">
|
</TabsTrigger>
|
||||||
<FormField
|
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||||
control={form.control}
|
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||||
name="sampleFile"
|
<Monitor className="h-4 w-4 shrink-0" />
|
||||||
render={({ field: { onChange, name } }) => (
|
System Audio
|
||||||
<AudioSampleUpload
|
</TabsTrigger>
|
||||||
file={selectedFile}
|
|
||||||
onFileChange={onChange}
|
|
||||||
onTranscribe={handleTranscribe}
|
|
||||||
onPlayPause={handlePlayPause}
|
|
||||||
isPlaying={isPlaying}
|
|
||||||
isValidating={isValidatingAudio}
|
|
||||||
isTranscribing={transcribe.isPending}
|
|
||||||
isDisabled={
|
|
||||||
audioDuration !== null && audioDuration > MAX_AUDIO_DURATION_SECONDS
|
|
||||||
}
|
|
||||||
fieldName={name}
|
|
||||||
/>
|
|
||||||
)}
|
)}
|
||||||
/>
|
</TabsList>
|
||||||
</TabsContent>
|
|
||||||
|
|
||||||
<TabsContent value="record" className="space-y-4">
|
<TabsContent value="upload" className="space-y-4">
|
||||||
<FormField
|
<FormField
|
||||||
control={form.control}
|
control={form.control}
|
||||||
name="sampleFile"
|
name="sampleFile"
|
||||||
render={() => (
|
render={({ field: { onChange, name } }) => (
|
||||||
<AudioSampleRecording
|
<AudioSampleUpload
|
||||||
file={selectedFile}
|
file={selectedFile}
|
||||||
isRecording={isRecording}
|
onFileChange={onChange}
|
||||||
duration={duration}
|
onTranscribe={handleTranscribe}
|
||||||
onStart={startRecording}
|
onPlayPause={handlePlayPause}
|
||||||
onStop={stopRecording}
|
isPlaying={isPlaying}
|
||||||
onCancel={handleCancelRecording}
|
isValidating={isValidatingAudio}
|
||||||
onTranscribe={handleTranscribe}
|
isTranscribing={transcribe.isPending}
|
||||||
onPlayPause={handlePlayPause}
|
isDisabled={
|
||||||
isPlaying={isPlaying}
|
audioDuration !== null &&
|
||||||
isTranscribing={transcribe.isPending}
|
audioDuration > MAX_AUDIO_DURATION_SECONDS
|
||||||
/>
|
}
|
||||||
)}
|
fieldName={name}
|
||||||
/>
|
/>
|
||||||
</TabsContent>
|
)}
|
||||||
|
/>
|
||||||
|
</TabsContent>
|
||||||
|
|
||||||
{isTauri() && isSystemAudioSupported && (
|
<TabsContent value="record" className="space-y-4">
|
||||||
<TabsContent value="system" className="space-y-4">
|
|
||||||
<FormField
|
<FormField
|
||||||
control={form.control}
|
control={form.control}
|
||||||
name="sampleFile"
|
name="sampleFile"
|
||||||
render={() => (
|
render={() => (
|
||||||
<AudioSampleSystem
|
<AudioSampleRecording
|
||||||
file={selectedFile}
|
file={selectedFile}
|
||||||
isRecording={isSystemRecording}
|
isRecording={isRecording}
|
||||||
duration={systemDuration}
|
duration={duration}
|
||||||
onStart={startSystemRecording}
|
onStart={startRecording}
|
||||||
onStop={stopSystemRecording}
|
onStop={stopRecording}
|
||||||
onCancel={handleCancelRecording}
|
onCancel={handleCancelRecording}
|
||||||
onTranscribe={handleTranscribe}
|
onTranscribe={handleTranscribe}
|
||||||
onPlayPause={handlePlayPause}
|
onPlayPause={handlePlayPause}
|
||||||
@@ -590,55 +726,188 @@ export function ProfileForm() {
|
|||||||
)}
|
)}
|
||||||
/>
|
/>
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
)}
|
|
||||||
</Tabs>
|
|
||||||
|
|
||||||
<FormField
|
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||||
control={form.control}
|
<TabsContent value="system" className="space-y-4">
|
||||||
name="referenceText"
|
<FormField
|
||||||
render={({ field }) => (
|
control={form.control}
|
||||||
<FormItem>
|
name="sampleFile"
|
||||||
<FormLabel>Reference Text</FormLabel>
|
render={() => (
|
||||||
<FormControl>
|
<AudioSampleSystem
|
||||||
<Textarea
|
file={selectedFile}
|
||||||
placeholder="Enter the exact text spoken in the audio..."
|
isRecording={isSystemRecording}
|
||||||
className="min-h-[100px]"
|
duration={systemDuration}
|
||||||
{...field}
|
onStart={startSystemRecording}
|
||||||
|
onStop={stopSystemRecording}
|
||||||
|
onCancel={handleCancelRecording}
|
||||||
|
onTranscribe={handleTranscribe}
|
||||||
|
onPlayPause={handlePlayPause}
|
||||||
|
isPlaying={isPlaying}
|
||||||
|
isTranscribing={transcribe.isPending}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
/>
|
/>
|
||||||
</FormControl>
|
</TabsContent>
|
||||||
<FormMessage />
|
)}
|
||||||
</FormItem>
|
</Tabs>
|
||||||
)}
|
|
||||||
/>
|
|
||||||
</>
|
|
||||||
) : (
|
|
||||||
// Show sample list when editing
|
|
||||||
editingProfileId && (
|
|
||||||
<div>
|
|
||||||
<SampleList profileId={editingProfileId} />
|
|
||||||
</div>
|
|
||||||
)
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
|
<FormField
|
||||||
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
|
control={form.control}
|
||||||
Cancel
|
name="referenceText"
|
||||||
</Button>
|
render={({ field }) => (
|
||||||
<Button
|
<FormItem>
|
||||||
type="submit"
|
<FormLabel>Reference Text</FormLabel>
|
||||||
disabled={createProfile.isPending || updateProfile.isPending || addSample.isPending}
|
<FormControl>
|
||||||
>
|
<Textarea
|
||||||
{createProfile.isPending || updateProfile.isPending || addSample.isPending
|
placeholder="Enter the exact text spoken in the audio..."
|
||||||
? 'Saving...'
|
className="min-h-[100px]"
|
||||||
: editingProfileId
|
{...field}
|
||||||
? 'Save Changes'
|
/>
|
||||||
: 'Create Profile'}
|
</FormControl>
|
||||||
</Button>
|
<FormMessage />
|
||||||
</div>
|
</FormItem>
|
||||||
</form>
|
)}
|
||||||
</Form>
|
/>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
// Show sample list when editing
|
||||||
|
editingProfileId && (
|
||||||
|
<div>
|
||||||
|
<SampleList profileId={editingProfileId} />
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Right column: Profile info */}
|
||||||
|
<div className="space-y-4">
|
||||||
|
{/* Avatar Upload */}
|
||||||
|
<FormField
|
||||||
|
control={form.control}
|
||||||
|
name="avatarFile"
|
||||||
|
render={() => (
|
||||||
|
<FormItem>
|
||||||
|
<FormControl>
|
||||||
|
<div className="flex justify-center pt-4 pb-2">
|
||||||
|
<div className="relative group">
|
||||||
|
<div className="h-24 w-24 rounded-full bg-muted flex items-center justify-center shrink-0 overflow-hidden border-2 border-border">
|
||||||
|
{avatarPreview ? (
|
||||||
|
<img
|
||||||
|
src={avatarPreview}
|
||||||
|
alt="Avatar preview"
|
||||||
|
className="h-full w-full object-cover"
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<Mic className="h-10 w-10 text-muted-foreground" />
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => avatarInputRef.current?.click()}
|
||||||
|
className="absolute inset-0 rounded-full bg-accent/60 opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center cursor-pointer"
|
||||||
|
>
|
||||||
|
<Edit2 className="h-6 w-6 text-accent-foreground" />
|
||||||
|
</button>
|
||||||
|
{(avatarPreview || editingProfile?.avatar_path) && (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={handleRemoveAvatar}
|
||||||
|
disabled={deleteAvatar.isPending}
|
||||||
|
className="absolute bottom-0 right-0 h-6 w-6 rounded-full bg-background/60 backdrop-blur-sm text-muted-foreground flex items-center justify-center hover:bg-background/80 hover:text-foreground transition-colors shadow-sm border border-border/50"
|
||||||
|
>
|
||||||
|
<X className="h-3.5 w-3.5" />
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<input
|
||||||
|
ref={avatarInputRef}
|
||||||
|
type="file"
|
||||||
|
accept="image/png,image/jpeg,image/webp"
|
||||||
|
onChange={handleAvatarFileChange}
|
||||||
|
className="hidden"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</FormControl>
|
||||||
|
<FormMessage />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
|
||||||
|
<FormField
|
||||||
|
control={form.control}
|
||||||
|
name="name"
|
||||||
|
render={({ field }) => (
|
||||||
|
<FormItem>
|
||||||
|
<FormLabel>Name</FormLabel>
|
||||||
|
<FormControl>
|
||||||
|
<Input placeholder="My Voice" {...field} />
|
||||||
|
</FormControl>
|
||||||
|
<FormMessage />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
|
||||||
|
<FormField
|
||||||
|
control={form.control}
|
||||||
|
name="description"
|
||||||
|
render={({ field }) => (
|
||||||
|
<FormItem>
|
||||||
|
<FormLabel>Description (Optional)</FormLabel>
|
||||||
|
<FormControl>
|
||||||
|
<Textarea placeholder="Describe this voice..." {...field} />
|
||||||
|
</FormControl>
|
||||||
|
<FormMessage />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
|
||||||
|
<FormField
|
||||||
|
control={form.control}
|
||||||
|
name="language"
|
||||||
|
render={({ field }) => (
|
||||||
|
<FormItem>
|
||||||
|
<FormLabel>Language</FormLabel>
|
||||||
|
<Select onValueChange={field.onChange} defaultValue={field.value}>
|
||||||
|
<FormControl>
|
||||||
|
<SelectTrigger>
|
||||||
|
<SelectValue />
|
||||||
|
</SelectTrigger>
|
||||||
|
</FormControl>
|
||||||
|
<SelectContent>
|
||||||
|
{LANGUAGE_OPTIONS.map((lang) => (
|
||||||
|
<SelectItem key={lang.value} value={lang.value}>
|
||||||
|
{lang.label}
|
||||||
|
</SelectItem>
|
||||||
|
))}
|
||||||
|
</SelectContent>
|
||||||
|
</Select>
|
||||||
|
<FormMessage />
|
||||||
|
</FormItem>
|
||||||
|
)}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex gap-2 justify-end mt-6 pt-4 border-t">
|
||||||
|
<Button type="button" variant="outline" onClick={() => handleOpenChange(false)}>
|
||||||
|
Cancel
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
type="submit"
|
||||||
|
disabled={
|
||||||
|
createProfile.isPending || updateProfile.isPending || addSample.isPending
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{createProfile.isPending || updateProfile.isPending || addSample.isPending
|
||||||
|
? 'Saving...'
|
||||||
|
: editingProfileId
|
||||||
|
? 'Save Changes'
|
||||||
|
: 'Create Profile'}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</Form>
|
||||||
|
</div>
|
||||||
</DialogContent>
|
</DialogContent>
|
||||||
</Dialog>
|
</Dialog>
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -1,11 +1,141 @@
|
|||||||
import { Plus, Trash2, Play } from 'lucide-react';
|
import { Check, Edit, Pause, Play, Plus, Trash2, Volume2, X } from 'lucide-react';
|
||||||
import { useState } from 'react';
|
import { useEffect, useRef, useState } from 'react';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
|
import { CircleButton } from '@/components/ui/circle-button';
|
||||||
|
import {
|
||||||
|
Dialog,
|
||||||
|
DialogContent,
|
||||||
|
DialogDescription,
|
||||||
|
DialogFooter,
|
||||||
|
DialogHeader,
|
||||||
|
DialogTitle,
|
||||||
|
} from '@/components/ui/dialog';
|
||||||
|
import { Slider } from '@/components/ui/slider';
|
||||||
|
import { Textarea } from '@/components/ui/textarea';
|
||||||
|
import { useToast } from '@/components/ui/use-toast';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import { useDeleteSample, useProfileSamples } from '@/lib/hooks/useProfiles';
|
import { useDeleteSample, useProfileSamples, useUpdateSample } from '@/lib/hooks/useProfiles';
|
||||||
import { usePlayerStore } from '@/stores/playerStore';
|
import { formatAudioDuration } from '@/lib/utils/audio';
|
||||||
|
import { cn } from '@/lib/utils/cn';
|
||||||
import { SampleUpload } from './SampleUpload';
|
import { SampleUpload } from './SampleUpload';
|
||||||
|
|
||||||
|
interface MiniSamplePlayerProps {
|
||||||
|
audioUrl: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
|
||||||
|
const audioRef = useRef<HTMLAudioElement | null>(null);
|
||||||
|
const [isPlaying, setIsPlaying] = useState(false);
|
||||||
|
const [currentTime, setCurrentTime] = useState(0);
|
||||||
|
const [duration, setDuration] = useState(0);
|
||||||
|
const [isLoading, setIsLoading] = useState(true);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
const audio = new Audio(audioUrl);
|
||||||
|
audioRef.current = audio;
|
||||||
|
|
||||||
|
const handleLoadedMetadata = () => {
|
||||||
|
setDuration(audio.duration);
|
||||||
|
setIsLoading(false);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleTimeUpdate = () => {
|
||||||
|
setCurrentTime(audio.currentTime);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleEnded = () => {
|
||||||
|
setIsPlaying(false);
|
||||||
|
setCurrentTime(0);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handlePlay = () => setIsPlaying(true);
|
||||||
|
const handlePause = () => setIsPlaying(false);
|
||||||
|
|
||||||
|
audio.addEventListener('loadedmetadata', handleLoadedMetadata);
|
||||||
|
audio.addEventListener('timeupdate', handleTimeUpdate);
|
||||||
|
audio.addEventListener('ended', handleEnded);
|
||||||
|
audio.addEventListener('play', handlePlay);
|
||||||
|
audio.addEventListener('pause', handlePause);
|
||||||
|
|
||||||
|
return () => {
|
||||||
|
audio.pause();
|
||||||
|
audio.removeEventListener('loadedmetadata', handleLoadedMetadata);
|
||||||
|
audio.removeEventListener('timeupdate', handleTimeUpdate);
|
||||||
|
audio.removeEventListener('ended', handleEnded);
|
||||||
|
audio.removeEventListener('play', handlePlay);
|
||||||
|
audio.removeEventListener('pause', handlePause);
|
||||||
|
audio.src = '';
|
||||||
|
};
|
||||||
|
}, [audioUrl]);
|
||||||
|
|
||||||
|
const handlePlayPause = () => {
|
||||||
|
if (!audioRef.current) return;
|
||||||
|
if (isPlaying) {
|
||||||
|
audioRef.current.pause();
|
||||||
|
} else {
|
||||||
|
audioRef.current.play();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleSeek = (value: number[]) => {
|
||||||
|
if (!audioRef.current || duration === 0) return;
|
||||||
|
const progress = value[0] / 100;
|
||||||
|
audioRef.current.currentTime = progress * duration;
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleStop = () => {
|
||||||
|
if (audioRef.current) {
|
||||||
|
audioRef.current.pause();
|
||||||
|
audioRef.current.currentTime = 0;
|
||||||
|
}
|
||||||
|
setIsPlaying(false);
|
||||||
|
setCurrentTime(0);
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="border-t bg-muted/30 px-3 py-2 mt-2">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-7 w-7 shrink-0"
|
||||||
|
onClick={handlePlayPause}
|
||||||
|
disabled={isLoading}
|
||||||
|
>
|
||||||
|
{isPlaying ? <Pause className="h-3.5 w-3.5" /> : <Play className="h-3.5 w-3.5 ml-0.5" />}
|
||||||
|
</Button>
|
||||||
|
|
||||||
|
<div className="flex-1 min-w-0 flex items-center gap-2">
|
||||||
|
<Slider
|
||||||
|
value={duration > 0 ? [(currentTime / duration) * 100] : [0]}
|
||||||
|
onValueChange={handleSeek}
|
||||||
|
max={100}
|
||||||
|
step={0.1}
|
||||||
|
className="flex-1"
|
||||||
|
/>
|
||||||
|
<div className="flex items-center gap-1 text-xs text-muted-foreground shrink-0 min-w-[70px]">
|
||||||
|
<span className="font-mono">{formatAudioDuration(currentTime)}</span>
|
||||||
|
<span>/</span>
|
||||||
|
<span className="font-mono">{formatAudioDuration(duration)}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-7 w-7 shrink-0"
|
||||||
|
onClick={handleStop}
|
||||||
|
title="Stop"
|
||||||
|
>
|
||||||
|
<X className="h-3.5 w-3.5" />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
interface SampleListProps {
|
interface SampleListProps {
|
||||||
profileId: string;
|
profileId: string;
|
||||||
}
|
}
|
||||||
@@ -13,20 +143,62 @@ interface SampleListProps {
|
|||||||
export function SampleList({ profileId }: SampleListProps) {
|
export function SampleList({ profileId }: SampleListProps) {
|
||||||
const { data: samples, isLoading } = useProfileSamples(profileId);
|
const { data: samples, isLoading } = useProfileSamples(profileId);
|
||||||
const deleteSample = useDeleteSample();
|
const deleteSample = useDeleteSample();
|
||||||
|
const updateSample = useUpdateSample();
|
||||||
|
const { toast } = useToast();
|
||||||
const [uploadOpen, setUploadOpen] = useState(false);
|
const [uploadOpen, setUploadOpen] = useState(false);
|
||||||
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
|
const [editingSampleId, setEditingSampleId] = useState<string | null>(null);
|
||||||
const currentAudioId = usePlayerStore((state) => state.audioId);
|
const [editedText, setEditedText] = useState<string>('');
|
||||||
const isPlaying = usePlayerStore((state) => state.isPlaying);
|
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||||
|
const [sampleToDelete, setSampleToDelete] = useState<string | null>(null);
|
||||||
|
|
||||||
const handleDelete = (sampleId: string) => {
|
const handleDeleteClick = (sampleId: string) => {
|
||||||
if (confirm('Are you sure you want to delete this sample?')) {
|
setSampleToDelete(sampleId);
|
||||||
deleteSample.mutate(sampleId);
|
setDeleteDialogOpen(true);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleDeleteConfirm = () => {
|
||||||
|
if (sampleToDelete) {
|
||||||
|
deleteSample.mutate(sampleToDelete);
|
||||||
|
setDeleteDialogOpen(false);
|
||||||
|
setSampleToDelete(null);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const handlePlay = (referenceText: string, sampleId: string) => {
|
const handleStartEdit = (sampleId: string, currentText: string) => {
|
||||||
const audioUrl = apiClient.getSampleUrl(sampleId);
|
setEditingSampleId(sampleId);
|
||||||
setAudioWithAutoPlay(audioUrl, sampleId, null, referenceText.substring(0, 50));
|
setEditedText(currentText);
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleCancelEdit = () => {
|
||||||
|
setEditingSampleId(null);
|
||||||
|
setEditedText('');
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleSaveEdit = async (sampleId: string) => {
|
||||||
|
if (!editedText.trim()) {
|
||||||
|
toast({
|
||||||
|
title: 'Invalid text',
|
||||||
|
description: 'Reference text cannot be empty.',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await updateSample.mutateAsync({ sampleId, referenceText: editedText.trim() });
|
||||||
|
toast({
|
||||||
|
title: 'Sample updated',
|
||||||
|
description: 'Reference text has been updated successfully.',
|
||||||
|
});
|
||||||
|
setEditingSampleId(null);
|
||||||
|
setEditedText('');
|
||||||
|
} catch (error) {
|
||||||
|
toast({
|
||||||
|
title: 'Update failed',
|
||||||
|
description: error instanceof Error ? error.message : 'Failed to update sample',
|
||||||
|
variant: 'destructive',
|
||||||
|
});
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
if (isLoading) {
|
if (isLoading) {
|
||||||
@@ -34,57 +206,152 @@ export function SampleList({ profileId }: SampleListProps) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="space-y-4">
|
<div className="space-y-4 pt-4">
|
||||||
<div className="flex items-center justify-between">
|
|
||||||
<h3 className="text-lg font-semibold">Audio Samples</h3>
|
|
||||||
<Button type="button" size="sm" onClick={() => setUploadOpen(true)}>
|
|
||||||
<Plus className="mr-2 h-4 w-4" />
|
|
||||||
Add Sample
|
|
||||||
</Button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{samples && samples.length === 0 ? (
|
{samples && samples.length === 0 ? (
|
||||||
<div className="text-sm text-muted-foreground py-4">
|
<div className="flex flex-col items-center justify-center py-8 text-center border border-dashed rounded-lg">
|
||||||
No samples yet. Add your first audio sample.
|
<Volume2 className="h-8 w-8 text-muted-foreground/50 mb-2" />
|
||||||
|
<p className="text-sm text-muted-foreground">No samples yet</p>
|
||||||
|
<p className="text-xs text-muted-foreground/70 mt-1">
|
||||||
|
Add your first audio sample to get started
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
) : (
|
) : (
|
||||||
<div className="space-y-2">
|
<div className="space-y-2">
|
||||||
{samples?.map((sample) => (
|
{samples?.map((sample, index) => {
|
||||||
<div
|
const isEditing = editingSampleId === sample.id;
|
||||||
key={sample.id}
|
|
||||||
className="flex items-center justify-between p-3 border rounded-lg"
|
return (
|
||||||
>
|
<div
|
||||||
<div className="flex-1">
|
key={sample.id}
|
||||||
<p className="text-sm font-medium">{sample.reference_text}</p>
|
className={cn(
|
||||||
<p className="text-xs text-muted-foreground mt-1">{sample.audio_path}</p>
|
'group relative rounded-lg border bg-card transition-all duration-200',
|
||||||
|
isEditing ? 'ring-2 ring-primary/20' : 'hover:border-primary/30',
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
{isEditing ? (
|
||||||
|
/* Edit Mode */
|
||||||
|
<div className="p-4 space-y-3">
|
||||||
|
<div className="flex items-center gap-2 text-xs text-muted-foreground mb-2">
|
||||||
|
<Edit className="h-3 w-3" />
|
||||||
|
<span>Editing transcription</span>
|
||||||
|
</div>
|
||||||
|
<Textarea
|
||||||
|
value={editedText}
|
||||||
|
onChange={(e) => setEditedText(e.target.value)}
|
||||||
|
className="min-h-[100px] text-sm resize-none"
|
||||||
|
placeholder="Enter reference text..."
|
||||||
|
autoFocus
|
||||||
|
/>
|
||||||
|
<div className="flex items-center justify-end gap-2 pt-1">
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
size="sm"
|
||||||
|
variant="ghost"
|
||||||
|
onClick={handleCancelEdit}
|
||||||
|
disabled={updateSample.isPending}
|
||||||
|
>
|
||||||
|
<X className="h-4 w-4 mr-1" />
|
||||||
|
Cancel
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
size="sm"
|
||||||
|
onClick={() => handleSaveEdit(sample.id)}
|
||||||
|
disabled={updateSample.isPending}
|
||||||
|
>
|
||||||
|
<Check className="h-4 w-4 mr-1" />
|
||||||
|
{updateSample.isPending ? 'Saving...' : 'Save'}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
{/* View Mode */}
|
||||||
|
<div className="flex items-center gap-3 p-3 h-[72px]">
|
||||||
|
{/* Text Content */}
|
||||||
|
<div className="flex-1 min-w-0 py-0.5">
|
||||||
|
<p className="text-sm font-medium line-clamp-2 leading-snug">
|
||||||
|
{sample.reference_text}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Action Buttons */}
|
||||||
|
<div className="shrink-0 flex items-center gap-0.5 opacity-0 group-hover:opacity-100 transition-opacity">
|
||||||
|
<CircleButton
|
||||||
|
icon={Edit}
|
||||||
|
title="Edit transcription"
|
||||||
|
onClick={() => handleStartEdit(sample.id, sample.reference_text)}
|
||||||
|
/>
|
||||||
|
<CircleButton
|
||||||
|
icon={Trash2}
|
||||||
|
title="Delete sample"
|
||||||
|
onClick={() => handleDeleteClick(sample.id)}
|
||||||
|
disabled={deleteSample.isPending}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Sample Number Badge */}
|
||||||
|
<div className="absolute top-1 right-2 text-[10px] text-muted-foreground/50 font-medium">
|
||||||
|
#{index + 1}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Mini Player - Always visible */}
|
||||||
|
<MiniSamplePlayer audioUrl={apiClient.getSampleUrl(sample.id)} />
|
||||||
|
</>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
<div className="flex gap-2">
|
);
|
||||||
<Button
|
})}
|
||||||
type="button"
|
|
||||||
variant="ghost"
|
|
||||||
size="sm"
|
|
||||||
onClick={() => handlePlay(sample.reference_text, sample.id)}
|
|
||||||
className={currentAudioId === sample.id && isPlaying ? 'text-primary' : ''}
|
|
||||||
>
|
|
||||||
<Play className="h-4 w-4 mr-1" />
|
|
||||||
Play
|
|
||||||
</Button>
|
|
||||||
<Button
|
|
||||||
type="button"
|
|
||||||
variant="ghost"
|
|
||||||
size="sm"
|
|
||||||
onClick={() => handleDelete(sample.id)}
|
|
||||||
disabled={deleteSample.isPending}
|
|
||||||
>
|
|
||||||
<Trash2 className="h-4 w-4 text-destructive" />
|
|
||||||
</Button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
))}
|
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="outline"
|
||||||
|
className="w-full"
|
||||||
|
onClick={() => setUploadOpen(true)}
|
||||||
|
>
|
||||||
|
<Plus className="mr-2 h-4 w-4" />
|
||||||
|
Add Sample
|
||||||
|
</Button>
|
||||||
|
|
||||||
|
<p className="text-xs text-muted-foreground text-center px-2">
|
||||||
|
Note: A single 30-second sample is the sweet spot. Quality may decrease with multiple
|
||||||
|
samples. In a future update samples might be interchangeable and tagged for varying styles
|
||||||
|
of the same voice.
|
||||||
|
</p>
|
||||||
|
|
||||||
<SampleUpload profileId={profileId} open={uploadOpen} onOpenChange={setUploadOpen} />
|
<SampleUpload profileId={profileId} open={uploadOpen} onOpenChange={setUploadOpen} />
|
||||||
|
|
||||||
|
<Dialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
|
||||||
|
<DialogContent>
|
||||||
|
<DialogHeader>
|
||||||
|
<DialogTitle>Delete Sample</DialogTitle>
|
||||||
|
<DialogDescription>
|
||||||
|
Are you sure you want to delete this audio sample? This action cannot be undone.
|
||||||
|
</DialogDescription>
|
||||||
|
</DialogHeader>
|
||||||
|
<DialogFooter>
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
onClick={() => {
|
||||||
|
setDeleteDialogOpen(false);
|
||||||
|
setSampleToDelete(null);
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Cancel
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
variant="destructive"
|
||||||
|
onClick={handleDeleteConfirm}
|
||||||
|
disabled={deleteSample.isPending}
|
||||||
|
>
|
||||||
|
{deleteSample.isPending ? 'Deleting...' : 'Delete'}
|
||||||
|
</Button>
|
||||||
|
</DialogFooter>
|
||||||
|
</DialogContent>
|
||||||
|
</Dialog>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
|
|||||||
import { useAddSample, useProfile } from '@/lib/hooks/useProfiles';
|
import { useAddSample, useProfile } from '@/lib/hooks/useProfiles';
|
||||||
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
|
import { useSystemAudioCapture } from '@/lib/hooks/useSystemAudioCapture';
|
||||||
import { useTranscription } from '@/lib/hooks/useTranscription';
|
import { useTranscription } from '@/lib/hooks/useTranscription';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
import { AudioSampleRecording } from './AudioSampleRecording';
|
import { AudioSampleRecording } from './AudioSampleRecording';
|
||||||
import { AudioSampleSystem } from './AudioSampleSystem';
|
import { AudioSampleSystem } from './AudioSampleSystem';
|
||||||
import { AudioSampleUpload } from './AudioSampleUpload';
|
import { AudioSampleUpload } from './AudioSampleUpload';
|
||||||
@@ -49,6 +49,7 @@ interface SampleUploadProps {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function SampleUpload({ profileId, open, onOpenChange }: SampleUploadProps) {
|
export function SampleUpload({ profileId, open, onOpenChange }: SampleUploadProps) {
|
||||||
|
const platform = usePlatform();
|
||||||
const addSample = useAddSample();
|
const addSample = useAddSample();
|
||||||
const transcribe = useTranscription();
|
const transcribe = useTranscription();
|
||||||
const { data: profile } = useProfile(profileId);
|
const { data: profile } = useProfile(profileId);
|
||||||
@@ -232,7 +233,7 @@ export function SampleUpload({ profileId, open, onOpenChange }: SampleUploadProp
|
|||||||
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
|
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
|
||||||
<Tabs value={mode} onValueChange={(v) => setMode(v as 'upload' | 'record' | 'system')}>
|
<Tabs value={mode} onValueChange={(v) => setMode(v as 'upload' | 'record' | 'system')}>
|
||||||
<TabsList
|
<TabsList
|
||||||
className={`grid w-full ${isTauri() && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||||
>
|
>
|
||||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||||
<Upload className="h-4 w-4 shrink-0" />
|
<Upload className="h-4 w-4 shrink-0" />
|
||||||
@@ -242,7 +243,7 @@ export function SampleUpload({ profileId, open, onOpenChange }: SampleUploadProp
|
|||||||
<Mic className="h-4 w-4 shrink-0" />
|
<Mic className="h-4 w-4 shrink-0" />
|
||||||
Record
|
Record
|
||||||
</TabsTrigger>
|
</TabsTrigger>
|
||||||
{isTauri() && isSystemAudioSupported && (
|
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||||
<Monitor className="h-4 w-4 shrink-0" />
|
<Monitor className="h-4 w-4 shrink-0" />
|
||||||
System Audio
|
System Audio
|
||||||
@@ -289,7 +290,7 @@ export function SampleUpload({ profileId, open, onOpenChange }: SampleUploadProp
|
|||||||
/>
|
/>
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
|
|
||||||
{isTauri() && isSystemAudioSupported && (
|
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||||
<TabsContent value="system" className="space-y-4">
|
<TabsContent value="system" className="space-y-4">
|
||||||
<FormField
|
<FormField
|
||||||
control={form.control}
|
control={form.control}
|
||||||
|
|||||||
@@ -6,10 +6,11 @@ export interface CircleButtonProps extends React.ButtonHTMLAttributes<HTMLButton
|
|||||||
}
|
}
|
||||||
|
|
||||||
const CircleButton = React.forwardRef<HTMLButtonElement, CircleButtonProps>(
|
const CircleButton = React.forwardRef<HTMLButtonElement, CircleButtonProps>(
|
||||||
({ className, icon: Icon, ...props }, ref) => {
|
({ className, icon: Icon, type = 'button', ...props }, ref) => {
|
||||||
return (
|
return (
|
||||||
<button
|
<button
|
||||||
ref={ref}
|
ref={ref}
|
||||||
|
type={type}
|
||||||
className={cn(
|
className={cn(
|
||||||
'h-7 w-7 rounded-full flex items-center justify-center flex-shrink-0',
|
'h-7 w-7 rounded-full flex items-center justify-center flex-shrink-0',
|
||||||
'hover:bg-muted transition-colors',
|
'hover:bg-muted transition-colors',
|
||||||
|
|||||||
+25
-156
@@ -1,172 +1,41 @@
|
|||||||
import { relaunch } from '@tauri-apps/plugin-process';
|
|
||||||
import { check, type Update } from '@tauri-apps/plugin-updater';
|
|
||||||
import { useCallback, useEffect, useState } from 'react';
|
import { useCallback, useEffect, useState } from 'react';
|
||||||
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
import type { UpdateStatus } from '@/platform/types';
|
||||||
|
|
||||||
export interface UpdateStatus {
|
// Re-export UpdateStatus for backwards compatibility
|
||||||
checking: boolean;
|
export type { UpdateStatus };
|
||||||
available: boolean;
|
|
||||||
version?: string;
|
|
||||||
downloading: boolean;
|
|
||||||
installing: boolean;
|
|
||||||
readyToInstall: boolean;
|
|
||||||
error?: string;
|
|
||||||
downloadProgress?: number; // 0-100 percentage
|
|
||||||
downloadedBytes?: number;
|
|
||||||
totalBytes?: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if we're on Windows (NSIS installer handles restart automatically)
|
|
||||||
const isWindows = () => {
|
|
||||||
return navigator.userAgent.includes('Windows');
|
|
||||||
};
|
|
||||||
|
|
||||||
const isTauri = () => {
|
|
||||||
return '__TAURI_INTERNALS__' in window;
|
|
||||||
};
|
|
||||||
|
|
||||||
export function useAutoUpdater(checkOnMount = false) {
|
export function useAutoUpdater(checkOnMount = false) {
|
||||||
const [status, setStatus] = useState<UpdateStatus>({
|
const platform = usePlatform();
|
||||||
checking: false,
|
const [status, setStatus] = useState<UpdateStatus>(
|
||||||
available: false,
|
platform.updater.getStatus(),
|
||||||
downloading: false,
|
);
|
||||||
installing: false,
|
|
||||||
readyToInstall: false,
|
|
||||||
});
|
|
||||||
|
|
||||||
const [update, setUpdate] = useState<Update | null>(null);
|
// Subscribe to updater status changes
|
||||||
|
useEffect(() => {
|
||||||
|
const unsubscribe = platform.updater.subscribe((newStatus) => {
|
||||||
|
setStatus(newStatus);
|
||||||
|
});
|
||||||
|
return unsubscribe;
|
||||||
|
}, [platform]);
|
||||||
|
|
||||||
const checkForUpdates = useCallback(async () => {
|
const checkForUpdates = useCallback(async () => {
|
||||||
if (!isTauri()) {
|
await platform.updater.checkForUpdates();
|
||||||
return;
|
}, [platform]);
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
const downloadAndInstall = useCallback(async () => {
|
||||||
setStatus((prev) => ({ ...prev, checking: true, error: undefined }));
|
await platform.updater.downloadAndInstall();
|
||||||
|
}, [platform]);
|
||||||
|
|
||||||
const foundUpdate = await check();
|
const restartAndInstall = useCallback(async () => {
|
||||||
|
await platform.updater.restartAndInstall();
|
||||||
if (foundUpdate?.available) {
|
}, [platform]);
|
||||||
setUpdate(foundUpdate);
|
|
||||||
setStatus({
|
|
||||||
checking: false,
|
|
||||||
available: true,
|
|
||||||
version: foundUpdate.version,
|
|
||||||
downloading: false,
|
|
||||||
installing: false,
|
|
||||||
readyToInstall: false,
|
|
||||||
});
|
|
||||||
} else {
|
|
||||||
setStatus({
|
|
||||||
checking: false,
|
|
||||||
available: false,
|
|
||||||
downloading: false,
|
|
||||||
installing: false,
|
|
||||||
readyToInstall: false,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
setStatus({
|
|
||||||
checking: false,
|
|
||||||
available: false,
|
|
||||||
downloading: false,
|
|
||||||
installing: false,
|
|
||||||
readyToInstall: false,
|
|
||||||
error: error instanceof Error ? error.message : 'Failed to check for updates',
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}, []);
|
|
||||||
|
|
||||||
// Download the update (but don't install yet)
|
|
||||||
const downloadAndInstall = async () => {
|
|
||||||
if (!update || !isTauri()) return;
|
|
||||||
|
|
||||||
try {
|
|
||||||
setStatus((prev) => ({ ...prev, downloading: true, error: undefined }));
|
|
||||||
|
|
||||||
let downloadedBytes = 0;
|
|
||||||
let totalBytes = 0;
|
|
||||||
|
|
||||||
// Just download the update
|
|
||||||
await update.download((event) => {
|
|
||||||
switch (event.event) {
|
|
||||||
case 'Started':
|
|
||||||
totalBytes = event.data.contentLength || 0;
|
|
||||||
downloadedBytes = 0;
|
|
||||||
setStatus((prev) => ({
|
|
||||||
...prev,
|
|
||||||
downloading: true,
|
|
||||||
totalBytes,
|
|
||||||
downloadedBytes: 0,
|
|
||||||
downloadProgress: 0,
|
|
||||||
}));
|
|
||||||
break;
|
|
||||||
case 'Progress': {
|
|
||||||
downloadedBytes += event.data.chunkLength;
|
|
||||||
const progress =
|
|
||||||
totalBytes > 0 ? Math.round((downloadedBytes / totalBytes) * 100) : undefined;
|
|
||||||
setStatus((prev) => ({
|
|
||||||
...prev,
|
|
||||||
downloadedBytes,
|
|
||||||
downloadProgress: progress,
|
|
||||||
}));
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case 'Finished':
|
|
||||||
setStatus((prev) => ({
|
|
||||||
...prev,
|
|
||||||
downloading: false,
|
|
||||||
readyToInstall: true,
|
|
||||||
downloadProgress: 100,
|
|
||||||
}));
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
});
|
|
||||||
} catch (error) {
|
|
||||||
setStatus((prev) => ({
|
|
||||||
...prev,
|
|
||||||
downloading: false,
|
|
||||||
installing: false,
|
|
||||||
readyToInstall: false,
|
|
||||||
downloadProgress: undefined,
|
|
||||||
downloadedBytes: undefined,
|
|
||||||
totalBytes: undefined,
|
|
||||||
error: error instanceof Error ? error.message : 'Failed to download update',
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Install the downloaded update and restart the app
|
|
||||||
const restartAndInstall = async () => {
|
|
||||||
if (!update || !isTauri()) return;
|
|
||||||
|
|
||||||
try {
|
|
||||||
setStatus((prev) => ({ ...prev, installing: true, error: undefined }));
|
|
||||||
|
|
||||||
// Install the update
|
|
||||||
await update.install();
|
|
||||||
|
|
||||||
// On Windows with NSIS, the installer handles the restart automatically.
|
|
||||||
// The process will be killed by the NSIS installer, so we won't reach here.
|
|
||||||
// On macOS/Linux, we need to manually relaunch.
|
|
||||||
if (!isWindows()) {
|
|
||||||
await relaunch();
|
|
||||||
}
|
|
||||||
// If we're on Windows and somehow still running, the NSIS installer
|
|
||||||
// should have already handled everything. Just wait for the process to end.
|
|
||||||
} catch (error) {
|
|
||||||
setStatus((prev) => ({
|
|
||||||
...prev,
|
|
||||||
installing: false,
|
|
||||||
error: error instanceof Error ? error.message : 'Failed to install update',
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (checkOnMount && isTauri()) {
|
if (checkOnMount && platform.metadata.isTauri) {
|
||||||
checkForUpdates();
|
checkForUpdates();
|
||||||
}
|
}
|
||||||
}, [checkOnMount, checkForUpdates]);
|
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
status,
|
status,
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { useServerStore } from '@/stores/serverStore';
|
import { useServerStore } from '@/stores/serverStore';
|
||||||
|
import type { LanguageCode } from '@/lib/constants/languages';
|
||||||
import type {
|
import type {
|
||||||
VoiceProfileCreate,
|
VoiceProfileCreate,
|
||||||
VoiceProfileResponse,
|
VoiceProfileResponse,
|
||||||
@@ -120,6 +121,16 @@ class ApiClient {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async updateProfileSample(
|
||||||
|
sampleId: string,
|
||||||
|
referenceText: string,
|
||||||
|
): Promise<ProfileSampleResponse> {
|
||||||
|
return this.request<ProfileSampleResponse>(`/profiles/samples/${sampleId}`, {
|
||||||
|
method: 'PUT',
|
||||||
|
body: JSON.stringify({ reference_text: referenceText }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
async exportProfile(profileId: string): Promise<Blob> {
|
async exportProfile(profileId: string): Promise<Blob> {
|
||||||
const url = `${this.getBaseUrl()}/profiles/${profileId}/export`;
|
const url = `${this.getBaseUrl()}/profiles/${profileId}/export`;
|
||||||
const response = await fetch(url);
|
const response = await fetch(url);
|
||||||
@@ -154,6 +165,32 @@ class ApiClient {
|
|||||||
return response.json();
|
return response.json();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async uploadAvatar(profileId: string, file: File): Promise<VoiceProfileResponse> {
|
||||||
|
const url = `${this.getBaseUrl()}/profiles/${profileId}/avatar`;
|
||||||
|
const formData = new FormData();
|
||||||
|
formData.append('file', file);
|
||||||
|
|
||||||
|
const response = await fetch(url, {
|
||||||
|
method: 'POST',
|
||||||
|
body: formData,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const error = await response.json().catch(() => ({
|
||||||
|
detail: response.statusText,
|
||||||
|
}));
|
||||||
|
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
async deleteAvatar(profileId: string): Promise<void> {
|
||||||
|
await this.request<void>(`/profiles/${profileId}/avatar`, {
|
||||||
|
method: 'DELETE',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
// Generation
|
// Generation
|
||||||
async generateSpeech(data: GenerationRequest): Promise<GenerationResponse> {
|
async generateSpeech(data: GenerationRequest): Promise<GenerationResponse> {
|
||||||
return this.request<GenerationResponse>('/generate', {
|
return this.request<GenerationResponse>('/generate', {
|
||||||
@@ -244,7 +281,7 @@ class ApiClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Transcription
|
// Transcription
|
||||||
async transcribeAudio(file: File, language?: 'en' | 'zh'): Promise<TranscriptionResponse> {
|
async transcribeAudio(file: File, language?: LanguageCode): Promise<TranscriptionResponse> {
|
||||||
const formData = new FormData();
|
const formData = new FormData();
|
||||||
formData.append('file', file);
|
formData.append('file', file);
|
||||||
if (language) {
|
if (language) {
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
// API Types matching backend Pydantic models
|
// API Types matching backend Pydantic models
|
||||||
|
import type { LanguageCode } from '@/lib/constants/languages';
|
||||||
|
|
||||||
export interface VoiceProfileCreate {
|
export interface VoiceProfileCreate {
|
||||||
name: string;
|
name: string;
|
||||||
description?: string;
|
description?: string;
|
||||||
language: 'en' | 'zh';
|
language: LanguageCode;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface VoiceProfileResponse {
|
export interface VoiceProfileResponse {
|
||||||
@@ -11,6 +12,7 @@ export interface VoiceProfileResponse {
|
|||||||
name: string;
|
name: string;
|
||||||
description?: string;
|
description?: string;
|
||||||
language: string;
|
language: string;
|
||||||
|
avatar_path?: string;
|
||||||
created_at: string;
|
created_at: string;
|
||||||
updated_at: string;
|
updated_at: string;
|
||||||
}
|
}
|
||||||
@@ -29,7 +31,7 @@ export interface ProfileSampleResponse {
|
|||||||
export interface GenerationRequest {
|
export interface GenerationRequest {
|
||||||
profile_id: string;
|
profile_id: string;
|
||||||
text: string;
|
text: string;
|
||||||
language: 'en' | 'zh';
|
language: LanguageCode;
|
||||||
seed?: number;
|
seed?: number;
|
||||||
model_size?: '1.7B' | '0.6B';
|
model_size?: '1.7B' | '0.6B';
|
||||||
}
|
}
|
||||||
@@ -62,7 +64,7 @@ export interface HistoryListResponse {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export interface TranscriptionRequest {
|
export interface TranscriptionRequest {
|
||||||
language?: 'en' | 'zh';
|
language?: LanguageCode;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface TranscriptionResponse {
|
export interface TranscriptionResponse {
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
import { convertToWav } from '@/lib/utils/audio';
|
import { convertToWav } from '@/lib/utils/audio';
|
||||||
|
|
||||||
interface UseAudioRecordingOptions {
|
interface UseAudioRecordingOptions {
|
||||||
@@ -11,6 +11,7 @@ export function useAudioRecording({
|
|||||||
maxDurationSeconds = 29,
|
maxDurationSeconds = 29,
|
||||||
onRecordingComplete,
|
onRecordingComplete,
|
||||||
}: UseAudioRecordingOptions = {}) {
|
}: UseAudioRecordingOptions = {}) {
|
||||||
|
const platform = usePlatform();
|
||||||
const [isRecording, setIsRecording] = useState(false);
|
const [isRecording, setIsRecording] = useState(false);
|
||||||
const [duration, setDuration] = useState(0);
|
const [duration, setDuration] = useState(0);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
@@ -40,15 +41,14 @@ export function useAudioRecording({
|
|||||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||||
|
|
||||||
if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
|
if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
|
||||||
const isTauriEnv = isTauri();
|
|
||||||
console.error('MediaDevices check:', {
|
console.error('MediaDevices check:', {
|
||||||
hasNavigator: typeof navigator !== 'undefined',
|
hasNavigator: typeof navigator !== 'undefined',
|
||||||
hasMediaDevices: !!navigator?.mediaDevices,
|
hasMediaDevices: !!navigator?.mediaDevices,
|
||||||
hasGetUserMedia: !!navigator?.mediaDevices?.getUserMedia,
|
hasGetUserMedia: !!navigator?.mediaDevices?.getUserMedia,
|
||||||
isTauri: isTauriEnv,
|
isTauri: platform.metadata.isTauri,
|
||||||
});
|
});
|
||||||
|
|
||||||
const errorMsg = isTauriEnv
|
const errorMsg = platform.metadata.isTauri
|
||||||
? 'Microphone access is not available. Please ensure:\n1. The app has microphone permissions in System Settings (macOS: System Settings > Privacy & Security > Microphone)\n2. You restart the app after granting permissions\n3. You are using Tauri v2 with a webview that supports getUserMedia'
|
? 'Microphone access is not available. Please ensure:\n1. The app has microphone permissions in System Settings (macOS: System Settings > Privacy & Security > Microphone)\n2. You restart the app after granting permissions\n3. You are using Tauri v2 with a webview that supports getUserMedia'
|
||||||
: 'Microphone access is not available. Please ensure you are using a secure context (HTTPS or localhost) and that your browser has microphone permissions enabled.';
|
: 'Microphone access is not available. Please ensure you are using a secure context (HTTPS or localhost) and that your browser has microphone permissions enabled.';
|
||||||
setError(errorMsg);
|
setError(errorMsg);
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import type { HistoryQuery } from '@/lib/api/types';
|
import type { HistoryQuery } from '@/lib/api/types';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function useHistory(query?: HistoryQuery) {
|
export function useHistory(query?: HistoryQuery) {
|
||||||
return useQuery({
|
return useQuery({
|
||||||
@@ -30,116 +30,52 @@ export function useDeleteGeneration() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function useExportGeneration() {
|
export function useExportGeneration() {
|
||||||
|
const platform = usePlatform();
|
||||||
|
|
||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: async ({ generationId, text }: { generationId: string; text: string }) => {
|
mutationFn: async ({ generationId, text }: { generationId: string; text: string }) => {
|
||||||
const blob = await apiClient.exportGeneration(generationId);
|
const blob = await apiClient.exportGeneration(generationId);
|
||||||
|
|
||||||
// Create safe filename from text
|
// Create safe filename from text
|
||||||
const safeText = text.substring(0, 30).replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
const safeText = text
|
||||||
|
.substring(0, 30)
|
||||||
|
.replace(/[^a-z0-9]/gi, '-')
|
||||||
|
.toLowerCase();
|
||||||
const filename = `generation-${safeText}.voicebox.zip`;
|
const filename = `generation-${safeText}.voicebox.zip`;
|
||||||
|
|
||||||
if (isTauri()) {
|
await platform.filesystem.saveFile(filename, blob, [
|
||||||
// Use Tauri's native save dialog
|
{
|
||||||
try {
|
name: 'Voicebox Generation',
|
||||||
const { save } = await import('@tauri-apps/plugin-dialog');
|
extensions: ['zip'],
|
||||||
const filePath = await save({
|
},
|
||||||
defaultPath: filename,
|
]);
|
||||||
filters: [
|
|
||||||
{
|
|
||||||
name: 'Voicebox Generation',
|
|
||||||
extensions: ['voicebox.zip', 'zip'],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
if (filePath) {
|
|
||||||
// Write file using Tauri's filesystem API
|
|
||||||
const { writeBinaryFile } = await import('@tauri-apps/plugin-fs');
|
|
||||||
const arrayBuffer = await blob.arrayBuffer();
|
|
||||||
await writeBinaryFile(filePath, new Uint8Array(arrayBuffer));
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to use Tauri dialog, falling back to browser download:', error);
|
|
||||||
// Fall back to browser download if Tauri dialog fails
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Browser: trigger download
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
|
|
||||||
return blob;
|
return blob;
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
export function useExportGenerationAudio() {
|
export function useExportGenerationAudio() {
|
||||||
|
const platform = usePlatform();
|
||||||
|
|
||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: async ({ generationId, text }: { generationId: string; text: string }) => {
|
mutationFn: async ({ generationId, text }: { generationId: string; text: string }) => {
|
||||||
const blob = await apiClient.exportGenerationAudio(generationId);
|
const blob = await apiClient.exportGenerationAudio(generationId);
|
||||||
|
|
||||||
// Create safe filename from text
|
// Create safe filename from text
|
||||||
const safeText = text.substring(0, 30).replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
const safeText = text
|
||||||
|
.substring(0, 30)
|
||||||
|
.replace(/[^a-z0-9]/gi, '-')
|
||||||
|
.toLowerCase();
|
||||||
const filename = `${safeText}.wav`;
|
const filename = `${safeText}.wav`;
|
||||||
|
|
||||||
if (isTauri()) {
|
await platform.filesystem.saveFile(filename, blob, [
|
||||||
// Use Tauri's native save dialog
|
{
|
||||||
try {
|
name: 'Audio File',
|
||||||
const { save } = await import('@tauri-apps/plugin-dialog');
|
extensions: ['wav'],
|
||||||
const filePath = await save({
|
},
|
||||||
defaultPath: filename,
|
]);
|
||||||
filters: [
|
|
||||||
{
|
|
||||||
name: 'Audio File',
|
|
||||||
extensions: ['wav'],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
if (filePath) {
|
|
||||||
// Write file using Tauri's filesystem API
|
|
||||||
const { writeBinaryFile } = await import('@tauri-apps/plugin-fs');
|
|
||||||
const arrayBuffer = await blob.arrayBuffer();
|
|
||||||
await writeBinaryFile(filePath, new Uint8Array(arrayBuffer));
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to use Tauri dialog, falling back to browser download:', error);
|
|
||||||
// Fall back to browser download if Tauri dialog fails
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Browser: trigger download
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
|
|
||||||
return blob;
|
return blob;
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -140,8 +140,8 @@ export function useModelDownloadToast({
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
eventSource.onerror = () => {
|
eventSource.onerror = (error) => {
|
||||||
console.error('SSE error');
|
console.error('SSE error:', error);
|
||||||
eventSource.close();
|
eventSource.close();
|
||||||
eventSourceRef.current = null;
|
eventSourceRef.current = null;
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import type { VoiceProfileCreate } from '@/lib/api/types';
|
import type { VoiceProfileCreate } from '@/lib/api/types';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function useProfiles() {
|
export function useProfiles() {
|
||||||
return useQuery({
|
return useQuery({
|
||||||
@@ -98,60 +98,43 @@ export function useDeleteSample() {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function useUpdateSample() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: ({ sampleId, referenceText }: { sampleId: string; referenceText: string }) =>
|
||||||
|
apiClient.updateProfileSample(sampleId, referenceText),
|
||||||
|
onSuccess: (data) => {
|
||||||
|
queryClient.invalidateQueries({
|
||||||
|
queryKey: ['profiles', data.profile_id, 'samples'],
|
||||||
|
});
|
||||||
|
queryClient.invalidateQueries({
|
||||||
|
queryKey: ['profiles', data.profile_id],
|
||||||
|
});
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
export function useExportProfile() {
|
export function useExportProfile() {
|
||||||
|
const platform = usePlatform();
|
||||||
|
|
||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: async (profileId: string) => {
|
mutationFn: async (profileId: string) => {
|
||||||
const blob = await apiClient.exportProfile(profileId);
|
const blob = await apiClient.exportProfile(profileId);
|
||||||
|
|
||||||
// Get profile name for filename
|
// Get profile name for filename
|
||||||
const profile = await apiClient.getProfile(profileId);
|
const profile = await apiClient.getProfile(profileId);
|
||||||
const safeName = profile.name.replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
const safeName = profile.name.replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
||||||
const filename = `profile-${safeName}.voicebox.zip`;
|
const filename = `profile-${safeName}.voicebox.zip`;
|
||||||
|
|
||||||
if (isTauri()) {
|
await platform.filesystem.saveFile(filename, blob, [
|
||||||
// Use Tauri's native save dialog
|
{
|
||||||
try {
|
name: 'Voicebox Profile',
|
||||||
const { save } = await import('@tauri-apps/plugin-dialog');
|
extensions: ['zip'],
|
||||||
const filePath = await save({
|
},
|
||||||
defaultPath: filename,
|
]);
|
||||||
filters: [
|
|
||||||
{
|
|
||||||
name: 'Voicebox Profile',
|
|
||||||
extensions: ['voicebox.zip', 'zip'],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
if (filePath) {
|
|
||||||
// Write file using Tauri's filesystem API
|
|
||||||
const { writeBinaryFile } = await import('@tauri-apps/plugin-fs');
|
|
||||||
const arrayBuffer = await blob.arrayBuffer();
|
|
||||||
await writeBinaryFile(filePath, new Uint8Array(arrayBuffer));
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to use Tauri dialog, falling back to browser download:', error);
|
|
||||||
// Fall back to browser download if Tauri dialog fails
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Browser: trigger download
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
|
|
||||||
return blob;
|
return blob;
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
@@ -167,3 +150,32 @@ export function useImportProfile() {
|
|||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function useUploadAvatar() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: ({ profileId, file }: { profileId: string; file: File }) =>
|
||||||
|
apiClient.uploadAvatar(profileId, file),
|
||||||
|
onSuccess: (_, variables) => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||||
|
queryClient.invalidateQueries({
|
||||||
|
queryKey: ['profiles', variables.profileId],
|
||||||
|
});
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useDeleteAvatar() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: (profileId: string) => apiClient.deleteAvatar(profileId),
|
||||||
|
onSuccess: (_, profileId) => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['profiles'] });
|
||||||
|
queryClient.invalidateQueries({
|
||||||
|
queryKey: ['profiles', profileId],
|
||||||
|
});
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
import type { StoryCreate, StoryItemCreate, StoryItemBatchUpdate, StoryItemReorder, StoryItemMove, StoryItemTrim, StoryItemSplit } from '@/lib/api/types';
|
import type { StoryCreate, StoryItemCreate, StoryItemBatchUpdate, StoryItemReorder, StoryItemMove, StoryItemTrim, StoryItemSplit } from '@/lib/api/types';
|
||||||
import { isTauri } from '@/lib/tauri';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
|
|
||||||
export function useStories() {
|
export function useStories() {
|
||||||
return useQuery({
|
return useQuery({
|
||||||
@@ -158,6 +158,8 @@ export function useDuplicateStoryItem() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function useExportStoryAudio() {
|
export function useExportStoryAudio() {
|
||||||
|
const platform = usePlatform();
|
||||||
|
|
||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: async ({ storyId, storyName }: { storyId: string; storyName: string }) => {
|
mutationFn: async ({ storyId, storyName }: { storyId: string; storyName: string }) => {
|
||||||
const blob = await apiClient.exportStoryAudio(storyId);
|
const blob = await apiClient.exportStoryAudio(storyId);
|
||||||
@@ -166,49 +168,12 @@ export function useExportStoryAudio() {
|
|||||||
const safeName = storyName.substring(0, 50).replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
const safeName = storyName.substring(0, 50).replace(/[^a-z0-9]/gi, '-').toLowerCase();
|
||||||
const filename = `${safeName || 'story'}.wav`;
|
const filename = `${safeName || 'story'}.wav`;
|
||||||
|
|
||||||
if (isTauri()) {
|
await platform.filesystem.saveFile(filename, blob, [
|
||||||
// Use Tauri's native save dialog
|
{
|
||||||
try {
|
name: 'Audio File',
|
||||||
const { save } = await import('@tauri-apps/plugin-dialog');
|
extensions: ['wav'],
|
||||||
const filePath = await save({
|
},
|
||||||
defaultPath: filename,
|
]);
|
||||||
filters: [
|
|
||||||
{
|
|
||||||
name: 'Audio File',
|
|
||||||
extensions: ['wav'],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
if (filePath) {
|
|
||||||
// Write file using Tauri's filesystem API
|
|
||||||
const { writeBinaryFile } = await import('@tauri-apps/plugin-fs');
|
|
||||||
const arrayBuffer = await blob.arrayBuffer();
|
|
||||||
await writeBinaryFile(filePath, new Uint8Array(arrayBuffer));
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to use Tauri dialog, falling back to browser download:', error);
|
|
||||||
// Fall back to browser download if Tauri dialog fails
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Browser: trigger download
|
|
||||||
const url = window.URL.createObjectURL(blob);
|
|
||||||
const a = document.createElement('a');
|
|
||||||
a.href = url;
|
|
||||||
a.download = filename;
|
|
||||||
document.body.appendChild(a);
|
|
||||||
a.click();
|
|
||||||
window.URL.revokeObjectURL(url);
|
|
||||||
document.body.removeChild(a);
|
|
||||||
}
|
|
||||||
|
|
||||||
return blob;
|
return blob;
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
import { useState, useRef, useCallback, useEffect } from 'react';
|
import { useState, useRef, useCallback, useEffect } from 'react';
|
||||||
import { invoke } from '@tauri-apps/api/core';
|
import { usePlatform } from '@/platform/PlatformContext';
|
||||||
import { isTauri } from '@/lib/tauri';
|
|
||||||
|
|
||||||
interface UseSystemAudioCaptureOptions {
|
interface UseSystemAudioCaptureOptions {
|
||||||
maxDurationSeconds?: number;
|
maxDurationSeconds?: number;
|
||||||
@@ -15,6 +14,7 @@ export function useSystemAudioCapture({
|
|||||||
maxDurationSeconds = 29,
|
maxDurationSeconds = 29,
|
||||||
onRecordingComplete,
|
onRecordingComplete,
|
||||||
}: UseSystemAudioCaptureOptions = {}) {
|
}: UseSystemAudioCaptureOptions = {}) {
|
||||||
|
const platform = usePlatform();
|
||||||
const [isRecording, setIsRecording] = useState(false);
|
const [isRecording, setIsRecording] = useState(false);
|
||||||
const [duration, setDuration] = useState(0);
|
const [duration, setDuration] = useState(0);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
@@ -26,22 +26,12 @@ export function useSystemAudioCapture({
|
|||||||
|
|
||||||
// Check if system audio capture is supported
|
// Check if system audio capture is supported
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!isTauri()) {
|
const supported = platform.audio.isSystemAudioSupported();
|
||||||
setIsSupported(false);
|
setIsSupported(supported);
|
||||||
return;
|
}, [platform]);
|
||||||
}
|
|
||||||
|
|
||||||
invoke<boolean>('is_system_audio_supported')
|
|
||||||
.then((supported) => {
|
|
||||||
setIsSupported(supported);
|
|
||||||
})
|
|
||||||
.catch(() => {
|
|
||||||
setIsSupported(false);
|
|
||||||
});
|
|
||||||
}, []);
|
|
||||||
|
|
||||||
const startRecording = useCallback(async () => {
|
const startRecording = useCallback(async () => {
|
||||||
if (!isTauri()) {
|
if (!platform.metadata.isTauri) {
|
||||||
const errorMsg = 'System audio capture is only available in the desktop app.';
|
const errorMsg = 'System audio capture is only available in the desktop app.';
|
||||||
setError(errorMsg);
|
setError(errorMsg);
|
||||||
return;
|
return;
|
||||||
@@ -58,9 +48,7 @@ export function useSystemAudioCapture({
|
|||||||
setDuration(0);
|
setDuration(0);
|
||||||
|
|
||||||
// Start native capture
|
// Start native capture
|
||||||
await invoke('start_system_audio_capture', {
|
await platform.audio.startSystemAudioCapture(maxDurationSeconds);
|
||||||
maxDurationSecs: maxDurationSeconds,
|
|
||||||
});
|
|
||||||
|
|
||||||
setIsRecording(true);
|
setIsRecording(true);
|
||||||
isRecordingRef.current = true;
|
isRecordingRef.current = true;
|
||||||
@@ -86,10 +74,10 @@ export function useSystemAudioCapture({
|
|||||||
setError(errorMessage);
|
setError(errorMessage);
|
||||||
setIsRecording(false);
|
setIsRecording(false);
|
||||||
}
|
}
|
||||||
}, [maxDurationSeconds, isSupported]);
|
}, [maxDurationSeconds, isSupported, platform]);
|
||||||
|
|
||||||
const stopRecording = useCallback(async () => {
|
const stopRecording = useCallback(async () => {
|
||||||
if (!isRecording || !isTauri()) {
|
if (!isRecording || !platform.metadata.isTauri) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,17 +90,9 @@ export function useSystemAudioCapture({
|
|||||||
timerRef.current = null;
|
timerRef.current = null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop capture and get base64 WAV data
|
// Stop capture and get Blob
|
||||||
const base64Data = await invoke<string>('stop_system_audio_capture');
|
const blob = await platform.audio.stopSystemAudioCapture();
|
||||||
|
|
||||||
// Convert base64 to Blob
|
|
||||||
const binaryString = atob(base64Data);
|
|
||||||
const bytes = new Uint8Array(binaryString.length);
|
|
||||||
for (let i = 0; i < binaryString.length; i++) {
|
|
||||||
bytes[i] = binaryString.charCodeAt(i);
|
|
||||||
}
|
|
||||||
|
|
||||||
const blob = new Blob([bytes], { type: 'audio/wav' });
|
|
||||||
// Pass the actual recorded duration
|
// Pass the actual recorded duration
|
||||||
const recordedDuration = startTimeRef.current
|
const recordedDuration = startTimeRef.current
|
||||||
? (Date.now() - startTimeRef.current) / 1000
|
? (Date.now() - startTimeRef.current) / 1000
|
||||||
@@ -125,7 +105,7 @@ export function useSystemAudioCapture({
|
|||||||
: 'Failed to stop system audio capture.';
|
: 'Failed to stop system audio capture.';
|
||||||
setError(errorMessage);
|
setError(errorMessage);
|
||||||
}
|
}
|
||||||
}, [isRecording, onRecordingComplete]);
|
}, [isRecording, onRecordingComplete, platform]);
|
||||||
|
|
||||||
// Store stopRecording in ref for use in timer
|
// Store stopRecording in ref for use in timer
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -155,15 +135,15 @@ export function useSystemAudioCapture({
|
|||||||
timerRef.current = null;
|
timerRef.current = null;
|
||||||
}
|
}
|
||||||
// Cancel recording on unmount if still recording
|
// Cancel recording on unmount if still recording
|
||||||
if (isRecordingRef.current && isTauri()) {
|
if (isRecordingRef.current && platform.metadata.isTauri) {
|
||||||
// Call stop directly without the callback to avoid stale closure
|
// Call stop directly without the callback to avoid stale closure
|
||||||
invoke('stop_system_audio_capture').catch((err) => {
|
platform.audio.stopSystemAudioCapture().catch((err) => {
|
||||||
console.error('Error stopping audio capture on unmount:', err);
|
console.error('Error stopping audio capture on unmount:', err);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
// biome-ignore lint/correctness/useExhaustiveDependencies: Only run on unmount
|
// biome-ignore lint/correctness/useExhaustiveDependencies: Only run on unmount
|
||||||
}, []);
|
}, [platform]);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
isRecording,
|
isRecording,
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
import { useMutation } from '@tanstack/react-query';
|
import { useMutation } from '@tanstack/react-query';
|
||||||
import { apiClient } from '@/lib/api/client';
|
import { apiClient } from '@/lib/api/client';
|
||||||
|
import type { LanguageCode } from '@/lib/constants/languages';
|
||||||
|
|
||||||
export function useTranscription() {
|
export function useTranscription() {
|
||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: ({ file, language }: { file: File; language?: 'en' | 'zh' }) =>
|
mutationFn: ({ file, language }: { file: File; language?: LanguageCode }) =>
|
||||||
apiClient.transcribeAudio(file, language),
|
apiClient.transcribeAudio(file, language),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,108 +0,0 @@
|
|||||||
/**
|
|
||||||
* Tauri integration utilities
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { invoke } from '@tauri-apps/api/core';
|
|
||||||
import { listen, emit } from '@tauri-apps/api/event';
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if running in Tauri environment
|
|
||||||
*/
|
|
||||||
export function isTauri(): boolean {
|
|
||||||
return '__TAURI_INTERNALS__' in window;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if running on macOS
|
|
||||||
*/
|
|
||||||
export function isMacOS(): boolean {
|
|
||||||
return navigator.platform.toLowerCase().includes('mac');
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Start the bundled Python server (Tauri only)
|
|
||||||
*/
|
|
||||||
export async function startServer(remote = false): Promise<string> {
|
|
||||||
if (!isTauri()) {
|
|
||||||
throw new Error('Not running in Tauri environment');
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
const result = await invoke<string>('start_server', { remote });
|
|
||||||
console.log('Server started:', result);
|
|
||||||
return result;
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to start server:', error);
|
|
||||||
throw error;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Stop the bundled Python server (Tauri only)
|
|
||||||
*/
|
|
||||||
export async function stopServer(): Promise<void> {
|
|
||||||
if (!isTauri()) {
|
|
||||||
throw new Error('Not running in Tauri environment');
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
await invoke('stop_server');
|
|
||||||
console.log('Server stopped');
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to stop server:', error);
|
|
||||||
throw error;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Set whether the server should keep running when the app closes (Tauri only)
|
|
||||||
*/
|
|
||||||
export async function setKeepServerRunning(keepRunning: boolean): Promise<void> {
|
|
||||||
if (!isTauri()) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
await invoke('set_keep_server_running', { keepRunning });
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to set keep server running setting:', error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Setup window close handler to check setting and stop server if needed
|
|
||||||
*/
|
|
||||||
export async function setupWindowCloseHandler(): Promise<void> {
|
|
||||||
if (!isTauri()) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
// Listen for window close request from Rust
|
|
||||||
await listen<null>('window-close-requested', async () => {
|
|
||||||
// Import store here to avoid circular dependency
|
|
||||||
const { useServerStore } = await import('@/stores/serverStore');
|
|
||||||
const keepRunning = useServerStore.getState().keepServerRunningOnClose;
|
|
||||||
|
|
||||||
// Check if server was started by this app instance
|
|
||||||
// In dev mode, serverStartedByApp will be false, so we won't try to stop a separately-run server
|
|
||||||
// We need to access the module-level variable - this is a bit hacky but works
|
|
||||||
// @ts-expect-error - accessing module-level variable from another module
|
|
||||||
const serverStartedByApp = window.__voiceboxServerStartedByApp ?? false;
|
|
||||||
|
|
||||||
if (!keepRunning && serverStartedByApp) {
|
|
||||||
// Stop server before closing (only if we started it)
|
|
||||||
try {
|
|
||||||
await stopServer();
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to stop server on close:', error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Emit event back to Rust to allow close
|
|
||||||
await emit('window-close-allowed');
|
|
||||||
});
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to setup window close handler:', error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
import { createContext, useContext, type ReactNode } from 'react';
|
||||||
|
import type { Platform } from './types';
|
||||||
|
|
||||||
|
const PlatformContext = createContext<Platform | null>(null);
|
||||||
|
|
||||||
|
export interface PlatformProviderProps {
|
||||||
|
platform: Platform;
|
||||||
|
children: ReactNode;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function PlatformProvider({ platform, children }: PlatformProviderProps) {
|
||||||
|
return (
|
||||||
|
<PlatformContext.Provider value={platform}>
|
||||||
|
{children}
|
||||||
|
</PlatformContext.Provider>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function usePlatform(): Platform {
|
||||||
|
const platform = useContext(PlatformContext);
|
||||||
|
if (!platform) {
|
||||||
|
throw new Error('usePlatform must be used within PlatformProvider');
|
||||||
|
}
|
||||||
|
return platform;
|
||||||
|
}
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
/**
|
||||||
|
* Platform abstraction types
|
||||||
|
* These interfaces define the contract that platform implementations must fulfill
|
||||||
|
*/
|
||||||
|
|
||||||
|
export interface FileFilter {
|
||||||
|
name: string;
|
||||||
|
extensions: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlatformFilesystem {
|
||||||
|
saveFile(filename: string, blob: Blob, filters?: FileFilter[]): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface UpdateStatus {
|
||||||
|
checking: boolean;
|
||||||
|
available: boolean;
|
||||||
|
version?: string;
|
||||||
|
downloading: boolean;
|
||||||
|
installing: boolean;
|
||||||
|
readyToInstall: boolean;
|
||||||
|
error?: string;
|
||||||
|
downloadProgress?: number; // 0-100 percentage
|
||||||
|
downloadedBytes?: number;
|
||||||
|
totalBytes?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlatformUpdater {
|
||||||
|
checkForUpdates(): Promise<void>;
|
||||||
|
downloadAndInstall(): Promise<void>;
|
||||||
|
restartAndInstall(): Promise<void>;
|
||||||
|
getStatus(): UpdateStatus;
|
||||||
|
subscribe(callback: (status: UpdateStatus) => void): () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AudioDevice {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
is_default: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlatformAudio {
|
||||||
|
isSystemAudioSupported(): boolean;
|
||||||
|
startSystemAudioCapture(maxDurationSecs: number): Promise<void>;
|
||||||
|
stopSystemAudioCapture(): Promise<Blob>;
|
||||||
|
listOutputDevices(): Promise<AudioDevice[]>;
|
||||||
|
playToDevices(audioData: Uint8Array, deviceIds: string[]): Promise<void>;
|
||||||
|
stopPlayback(): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlatformLifecycle {
|
||||||
|
startServer(remote?: boolean): Promise<string>;
|
||||||
|
stopServer(): Promise<void>;
|
||||||
|
setKeepServerRunning(keep: boolean): Promise<void>;
|
||||||
|
setupWindowCloseHandler(): Promise<void>;
|
||||||
|
onServerReady?: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlatformMetadata {
|
||||||
|
getVersion(): Promise<string>;
|
||||||
|
isTauri: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Platform {
|
||||||
|
filesystem: PlatformFilesystem;
|
||||||
|
updater: PlatformUpdater;
|
||||||
|
audio: PlatformAudio;
|
||||||
|
lifecycle: PlatformLifecycle;
|
||||||
|
metadata: PlatformMetadata;
|
||||||
|
}
|
||||||
+2
-1
@@ -10,7 +10,8 @@ import { Toaster } from '@/components/ui/toaster';
|
|||||||
import { VoicesTab } from '@/components/VoicesTab/VoicesTab';
|
import { VoicesTab } from '@/components/VoicesTab/VoicesTab';
|
||||||
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
|
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
|
||||||
import { MODEL_DISPLAY_NAMES, useRestoreActiveTasks } from '@/lib/hooks/useRestoreActiveTasks';
|
import { MODEL_DISPLAY_NAMES, useRestoreActiveTasks } from '@/lib/hooks/useRestoreActiveTasks';
|
||||||
import { isMacOS } from '@/lib/tauri';
|
// Simple platform check that works in both web and Tauri
|
||||||
|
const isMacOS = () => navigator.platform.toLowerCase().includes('mac');
|
||||||
|
|
||||||
// Root layout component
|
// Root layout component
|
||||||
function RootLayout() {
|
function RootLayout() {
|
||||||
|
|||||||
@@ -89,6 +89,9 @@ export const useStoryStore = create<StoryPlaybackState>((set, get) => ({
|
|||||||
playbackItems: items,
|
playbackItems: items,
|
||||||
totalDurationMs: maxEndTimeMs,
|
totalDurationMs: maxEndTimeMs,
|
||||||
currentTimeMs: startTimeMs,
|
currentTimeMs: startTimeMs,
|
||||||
|
// Reset timing anchors - will be set fresh by the playback hook
|
||||||
|
playbackStartContextTime: null,
|
||||||
|
playbackStartStoryTime: null,
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,18 @@
|
|||||||
import { create } from 'zustand';
|
import { create } from 'zustand';
|
||||||
|
|
||||||
|
// Draft state for the create voice profile form
|
||||||
|
export interface ProfileFormDraft {
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
language: string;
|
||||||
|
referenceText: string;
|
||||||
|
sampleMode: 'upload' | 'record' | 'system';
|
||||||
|
// Note: File objects can't be persisted, so we store metadata
|
||||||
|
sampleFileName?: string;
|
||||||
|
sampleFileType?: string;
|
||||||
|
sampleFileData?: string; // Base64 encoded
|
||||||
|
}
|
||||||
|
|
||||||
interface UIStore {
|
interface UIStore {
|
||||||
// Sidebar
|
// Sidebar
|
||||||
sidebarOpen: boolean;
|
sidebarOpen: boolean;
|
||||||
@@ -18,6 +31,10 @@ interface UIStore {
|
|||||||
selectedProfileId: string | null;
|
selectedProfileId: string | null;
|
||||||
setSelectedProfileId: (id: string | null) => void;
|
setSelectedProfileId: (id: string | null) => void;
|
||||||
|
|
||||||
|
// Profile form draft (for persisting create voice modal state)
|
||||||
|
profileFormDraft: ProfileFormDraft | null;
|
||||||
|
setProfileFormDraft: (draft: ProfileFormDraft | null) => void;
|
||||||
|
|
||||||
// Theme
|
// Theme
|
||||||
theme: 'light' | 'dark';
|
theme: 'light' | 'dark';
|
||||||
setTheme: (theme: 'light' | 'dark') => void;
|
setTheme: (theme: 'light' | 'dark') => void;
|
||||||
@@ -38,6 +55,9 @@ export const useUIStore = create<UIStore>((set) => ({
|
|||||||
selectedProfileId: null,
|
selectedProfileId: null,
|
||||||
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
|
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
|
||||||
|
|
||||||
|
profileFormDraft: null,
|
||||||
|
setProfileFormDraft: (draft) => set({ profileFormDraft: draft }),
|
||||||
|
|
||||||
theme: 'light',
|
theme: 'light',
|
||||||
setTheme: (theme) => {
|
setTheme: (theme) => {
|
||||||
set({ theme });
|
set({ theme });
|
||||||
|
|||||||
+28
-7
@@ -19,8 +19,13 @@ Production-quality FastAPI backend for Qwen3-TTS voice cloning.
|
|||||||
backend/
|
backend/
|
||||||
├── main.py # FastAPI app with all routes
|
├── main.py # FastAPI app with all routes
|
||||||
├── models.py # Pydantic request/response models
|
├── models.py # Pydantic request/response models
|
||||||
├── tts.py # Qwen3-TTS inference
|
├── platform_detect.py # Platform detection for backend selection
|
||||||
├── transcribe.py # Whisper ASR
|
├── tts.py # TTS backend abstraction (delegates to MLX or PyTorch)
|
||||||
|
├── transcribe.py # STT backend abstraction (delegates to MLX or PyTorch)
|
||||||
|
├── backends/ # Backend implementations
|
||||||
|
│ ├── __init__.py # Backend factory and protocols
|
||||||
|
│ ├── mlx_backend.py # MLX backend (Apple Silicon)
|
||||||
|
│ └── pytorch_backend.py # PyTorch backend (Windows/Linux/Intel)
|
||||||
├── profiles.py # Voice profile CRUD
|
├── profiles.py # Voice profile CRUD
|
||||||
├── history.py # Generation history
|
├── history.py # Generation history
|
||||||
├── studio.py # Audio editing (TODO)
|
├── studio.py # Audio editing (TODO)
|
||||||
@@ -31,6 +36,15 @@ backend/
|
|||||||
└── validation.py # Input validation
|
└── validation.py # Input validation
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Backend Selection
|
||||||
|
|
||||||
|
Voicebox automatically selects the best backend based on platform:
|
||||||
|
|
||||||
|
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration (4-5x faster)
|
||||||
|
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU if available, CPU fallback)
|
||||||
|
|
||||||
|
The backend is detected at runtime via `platform_detect.py`. Both backends implement the same interface, so the API remains consistent across platforms.
|
||||||
|
|
||||||
## API Endpoints
|
## API Endpoints
|
||||||
|
|
||||||
### Health & Info
|
### Health & Info
|
||||||
@@ -47,12 +61,20 @@ Health check with model status.
|
|||||||
"status": "healthy",
|
"status": "healthy",
|
||||||
"model_loaded": true,
|
"model_loaded": true,
|
||||||
"gpu_available": true,
|
"gpu_available": true,
|
||||||
"vram_used_mb": 1024.5
|
"gpu_type": "Metal (Apple Silicon via MLX)",
|
||||||
|
"backend_type": "mlx",
|
||||||
|
"vram_used_mb": null
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Backend Types:**
|
||||||
|
- `"mlx"` - MLX backend (Apple Silicon with Metal acceleration)
|
||||||
|
- `"pytorch"` - PyTorch backend (Windows/Linux/Intel Mac)
|
||||||
|
|
||||||
### Voice Profiles
|
### Voice Profiles
|
||||||
|
|
||||||
|
**Note:** The database is automatically initialized when the server starts. No manual setup required.
|
||||||
|
|
||||||
#### `POST /profiles`
|
#### `POST /profiles`
|
||||||
Create a new voice profile.
|
Create a new voice profile.
|
||||||
|
|
||||||
@@ -266,13 +288,12 @@ data/
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Initialize Database
|
**Note:** On Apple Silicon, also install MLX dependencies for faster inference:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -c "from database import init_db; init_db()"
|
pip install -r requirements-mlx.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Download Models (Automatic)
|
### 2. Download Models (Automatic)
|
||||||
|
|
||||||
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
|
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
|
||||||
|
|
||||||
|
|||||||
@@ -1 +1,3 @@
|
|||||||
# Backend package
|
# Backend package
|
||||||
|
|
||||||
|
__version__ = "0.1.11"
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
"""
|
||||||
|
Backend abstraction layer for TTS and STT.
|
||||||
|
|
||||||
|
Provides a unified interface for MLX and PyTorch backends.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Protocol, Optional, Tuple, List
|
||||||
|
from typing_extensions import runtime_checkable
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from ..platform_detect import get_backend_type
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class TTSBackend(Protocol):
|
||||||
|
"""Protocol for TTS backend implementations."""
|
||||||
|
|
||||||
|
async def load_model(self, model_size: str) -> None:
|
||||||
|
"""Load TTS model."""
|
||||||
|
...
|
||||||
|
|
||||||
|
async def create_voice_prompt(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
use_cache: bool = True,
|
||||||
|
) -> Tuple[dict, bool]:
|
||||||
|
"""
|
||||||
|
Create voice prompt from reference audio.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (voice_prompt_dict, was_cached)
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
async def combine_voice_prompts(
|
||||||
|
self,
|
||||||
|
audio_paths: List[str],
|
||||||
|
reference_texts: List[str],
|
||||||
|
) -> Tuple[np.ndarray, str]:
|
||||||
|
"""
|
||||||
|
Combine multiple voice prompts.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (combined_audio_array, combined_text)
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice_prompt: dict,
|
||||||
|
language: str = "en",
|
||||||
|
seed: Optional[int] = None,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> Tuple[np.ndarray, int]:
|
||||||
|
"""
|
||||||
|
Generate audio from text.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (audio_array, sample_rate)
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
def unload_model(self) -> None:
|
||||||
|
"""Unload model to free memory."""
|
||||||
|
...
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
...
|
||||||
|
|
||||||
|
def _get_model_path(self, model_size: str) -> str:
|
||||||
|
"""
|
||||||
|
Get model path for a given size.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Model path or HuggingFace Hub ID
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class STTBackend(Protocol):
|
||||||
|
"""Protocol for STT (Speech-to-Text) backend implementations."""
|
||||||
|
|
||||||
|
async def load_model(self, model_size: str) -> None:
|
||||||
|
"""Load STT model."""
|
||||||
|
...
|
||||||
|
|
||||||
|
async def transcribe(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
language: Optional[str] = None,
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Transcribe audio to text.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Transcribed text
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
def unload_model(self) -> None:
|
||||||
|
"""Unload model to free memory."""
|
||||||
|
...
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
|
# Global backend instances
|
||||||
|
_tts_backend: Optional[TTSBackend] = None
|
||||||
|
_stt_backend: Optional[STTBackend] = None
|
||||||
|
|
||||||
|
|
||||||
|
def get_tts_backend() -> TTSBackend:
|
||||||
|
"""
|
||||||
|
Get or create TTS backend instance based on platform.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
TTS backend instance (MLX or PyTorch)
|
||||||
|
"""
|
||||||
|
global _tts_backend
|
||||||
|
|
||||||
|
if _tts_backend is None:
|
||||||
|
backend_type = get_backend_type()
|
||||||
|
|
||||||
|
if backend_type == "mlx":
|
||||||
|
from .mlx_backend import MLXTTSBackend
|
||||||
|
_tts_backend = MLXTTSBackend()
|
||||||
|
else:
|
||||||
|
from .pytorch_backend import PyTorchTTSBackend
|
||||||
|
_tts_backend = PyTorchTTSBackend()
|
||||||
|
|
||||||
|
return _tts_backend
|
||||||
|
|
||||||
|
|
||||||
|
def get_stt_backend() -> STTBackend:
|
||||||
|
"""
|
||||||
|
Get or create STT backend instance based on platform.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
STT backend instance (MLX or PyTorch)
|
||||||
|
"""
|
||||||
|
global _stt_backend
|
||||||
|
|
||||||
|
if _stt_backend is None:
|
||||||
|
backend_type = get_backend_type()
|
||||||
|
|
||||||
|
if backend_type == "mlx":
|
||||||
|
from .mlx_backend import MLXSTTBackend
|
||||||
|
_stt_backend = MLXSTTBackend()
|
||||||
|
else:
|
||||||
|
from .pytorch_backend import PyTorchSTTBackend
|
||||||
|
_stt_backend = PyTorchSTTBackend()
|
||||||
|
|
||||||
|
return _stt_backend
|
||||||
|
|
||||||
|
|
||||||
|
def reset_backends():
|
||||||
|
"""Reset backend instances (useful for testing)."""
|
||||||
|
global _tts_backend, _stt_backend
|
||||||
|
_tts_backend = None
|
||||||
|
_stt_backend = None
|
||||||
@@ -0,0 +1,471 @@
|
|||||||
|
"""
|
||||||
|
MLX backend implementation for TTS and STT using mlx-audio.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Optional, List, Tuple
|
||||||
|
import asyncio
|
||||||
|
import numpy as np
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from . import TTSBackend, STTBackend
|
||||||
|
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||||
|
from ..utils.audio import normalize_audio, load_audio
|
||||||
|
from ..utils.progress import get_progress_manager
|
||||||
|
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||||
|
from ..utils.tasks import get_task_manager
|
||||||
|
|
||||||
|
|
||||||
|
class MLXTTSBackend:
|
||||||
|
"""MLX-based TTS backend using mlx-audio."""
|
||||||
|
|
||||||
|
def __init__(self, model_size: str = "1.7B"):
|
||||||
|
self.model = None
|
||||||
|
self.model_size = model_size
|
||||||
|
self._current_model_size = None
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
return self.model is not None
|
||||||
|
|
||||||
|
def _get_model_path(self, model_size: str) -> str:
|
||||||
|
"""
|
||||||
|
Get the MLX model path.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size (1.7B or 0.6B)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
HuggingFace Hub model ID for MLX
|
||||||
|
"""
|
||||||
|
# MLX model mapping
|
||||||
|
mlx_model_map = {
|
||||||
|
"1.7B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16",
|
||||||
|
# 0.6B not yet converted to MLX format
|
||||||
|
"0.6B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16", # Fallback to 1.7B
|
||||||
|
}
|
||||||
|
|
||||||
|
if model_size not in mlx_model_map:
|
||||||
|
raise ValueError(f"Unknown model size: {model_size}")
|
||||||
|
|
||||||
|
hf_model_id = mlx_model_map[model_size]
|
||||||
|
print(f"Will download MLX model from HuggingFace Hub: {hf_model_id}")
|
||||||
|
|
||||||
|
return hf_model_id
|
||||||
|
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
"""
|
||||||
|
Lazy load the MLX TTS model.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size to load (1.7B or 0.6B)
|
||||||
|
"""
|
||||||
|
if model_size is None:
|
||||||
|
model_size = self.model_size
|
||||||
|
|
||||||
|
# If already loaded with correct size, return
|
||||||
|
if self.model is not None and self._current_model_size == model_size:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Unload existing model if different size requested
|
||||||
|
if self.model is not None and self._current_model_size != model_size:
|
||||||
|
self.unload_model()
|
||||||
|
|
||||||
|
# Run blocking load in thread pool
|
||||||
|
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||||
|
|
||||||
|
# Alias for compatibility
|
||||||
|
load_model = load_model_async
|
||||||
|
|
||||||
|
def _load_model_sync(self, model_size: str):
|
||||||
|
"""Synchronous model loading."""
|
||||||
|
try:
|
||||||
|
from mlx_audio.tts import load
|
||||||
|
|
||||||
|
# Get model path
|
||||||
|
model_path = self._get_model_path(model_size)
|
||||||
|
|
||||||
|
# Set up progress tracking
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
|
||||||
|
# Start tracking download task
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
task_manager.start_download(model_name)
|
||||||
|
|
||||||
|
print(f"Loading MLX TTS model {model_size}...")
|
||||||
|
|
||||||
|
# Initialize progress state
|
||||||
|
progress_manager.update_progress(
|
||||||
|
model_name=model_name,
|
||||||
|
current=0,
|
||||||
|
total=1,
|
||||||
|
filename="",
|
||||||
|
status="downloading",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Set up progress callback
|
||||||
|
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||||
|
tracker = HFProgressTracker(progress_callback)
|
||||||
|
|
||||||
|
# Use progress tracker during download
|
||||||
|
with tracker.patch_download():
|
||||||
|
# Load MLX model (downloads automatically)
|
||||||
|
self.model = load(model_path)
|
||||||
|
|
||||||
|
self._current_model_size = model_size
|
||||||
|
self.model_size = model_size
|
||||||
|
|
||||||
|
# Mark as complete
|
||||||
|
progress_manager.mark_complete(model_name)
|
||||||
|
task_manager.complete_download(model_name)
|
||||||
|
|
||||||
|
print(f"MLX TTS model {model_size} loaded successfully")
|
||||||
|
|
||||||
|
except ImportError as e:
|
||||||
|
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
progress_manager.mark_error(model_name, str(e))
|
||||||
|
task_manager.error_download(model_name, str(e))
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error loading MLX TTS model: {e}")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
progress_manager.mark_error(model_name, str(e))
|
||||||
|
task_manager.error_download(model_name, str(e))
|
||||||
|
raise
|
||||||
|
|
||||||
|
def unload_model(self):
|
||||||
|
"""Unload the model to free memory."""
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
self.model = None
|
||||||
|
self._current_model_size = None
|
||||||
|
print("MLX TTS model unloaded")
|
||||||
|
|
||||||
|
async def create_voice_prompt(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
use_cache: bool = True,
|
||||||
|
) -> Tuple[dict, bool]:
|
||||||
|
"""
|
||||||
|
Create voice prompt from reference audio.
|
||||||
|
|
||||||
|
MLX backend stores voice prompt as a dict with audio path and text.
|
||||||
|
The actual voice prompt processing happens during generation.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_path: Path to reference audio file
|
||||||
|
reference_text: Transcript of reference audio
|
||||||
|
use_cache: Whether to use cached prompt if available
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (voice_prompt_dict, was_cached)
|
||||||
|
"""
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
# Check cache if enabled
|
||||||
|
if use_cache:
|
||||||
|
cache_key = get_cache_key(audio_path, reference_text)
|
||||||
|
cached_prompt = get_cached_voice_prompt(cache_key)
|
||||||
|
if cached_prompt is not None:
|
||||||
|
# Return cached prompt (should be dict format)
|
||||||
|
if isinstance(cached_prompt, dict):
|
||||||
|
# Validate that the cached audio file still exists
|
||||||
|
cached_audio_path = cached_prompt.get("ref_audio") or cached_prompt.get("ref_audio_path")
|
||||||
|
if cached_audio_path and Path(cached_audio_path).exists():
|
||||||
|
return cached_prompt, True
|
||||||
|
else:
|
||||||
|
# Cached file no longer exists, invalidate cache
|
||||||
|
print(f"Cached audio file not found: {cached_audio_path}, regenerating prompt")
|
||||||
|
|
||||||
|
# MLX voice prompt format - store audio path and text
|
||||||
|
# The model will process this during generation
|
||||||
|
voice_prompt_items = {
|
||||||
|
"ref_audio": str(audio_path),
|
||||||
|
"ref_text": reference_text,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache if enabled
|
||||||
|
if use_cache:
|
||||||
|
cache_key = get_cache_key(audio_path, reference_text)
|
||||||
|
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||||
|
|
||||||
|
return voice_prompt_items, False
|
||||||
|
|
||||||
|
async def combine_voice_prompts(
|
||||||
|
self,
|
||||||
|
audio_paths: List[str],
|
||||||
|
reference_texts: List[str],
|
||||||
|
) -> Tuple[np.ndarray, str]:
|
||||||
|
"""
|
||||||
|
Combine multiple reference samples for better quality.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_paths: List of audio file paths
|
||||||
|
reference_texts: List of reference texts
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (combined_audio, combined_text)
|
||||||
|
"""
|
||||||
|
combined_audio = []
|
||||||
|
|
||||||
|
for audio_path in audio_paths:
|
||||||
|
audio, sr = load_audio(audio_path)
|
||||||
|
audio = normalize_audio(audio)
|
||||||
|
combined_audio.append(audio)
|
||||||
|
|
||||||
|
# Concatenate audio
|
||||||
|
mixed = np.concatenate(combined_audio)
|
||||||
|
mixed = normalize_audio(mixed)
|
||||||
|
|
||||||
|
# Combine texts
|
||||||
|
combined_text = " ".join(reference_texts)
|
||||||
|
|
||||||
|
return mixed, combined_text
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice_prompt: dict,
|
||||||
|
language: str = "en",
|
||||||
|
seed: Optional[int] = None,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> Tuple[np.ndarray, int]:
|
||||||
|
"""
|
||||||
|
Generate audio from text using voice prompt.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to synthesize
|
||||||
|
voice_prompt: Voice prompt dictionary with ref_audio and ref_text
|
||||||
|
language: Language code (en or zh) - may not be fully supported by MLX
|
||||||
|
seed: Random seed for reproducibility
|
||||||
|
instruct: Natural language instruction (may not be supported by MLX)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (audio_array, sample_rate)
|
||||||
|
"""
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
print(f"Generating audio for text: {text}")
|
||||||
|
|
||||||
|
def _generate_sync():
|
||||||
|
"""Run synchronous generation in thread pool."""
|
||||||
|
# MLX generate() returns a generator yielding GenerationResult objects
|
||||||
|
audio_chunks = []
|
||||||
|
sample_rate = 24000
|
||||||
|
|
||||||
|
# Set seed if provided (MLX uses numpy random)
|
||||||
|
if seed is not None:
|
||||||
|
import mlx.core as mx
|
||||||
|
np.random.seed(seed)
|
||||||
|
mx.random.seed(seed)
|
||||||
|
|
||||||
|
# Extract voice prompt info
|
||||||
|
ref_audio = voice_prompt.get("ref_audio") or voice_prompt.get("ref_audio_path")
|
||||||
|
ref_text = voice_prompt.get("ref_text", "")
|
||||||
|
|
||||||
|
# Validate that the audio file exists
|
||||||
|
if ref_audio and not Path(ref_audio).exists():
|
||||||
|
print(f"Warning: Audio file not found: {ref_audio}")
|
||||||
|
print("This may be due to a cached voice prompt referencing a deleted temp file.")
|
||||||
|
print("Regenerating without voice prompt.")
|
||||||
|
ref_audio = None
|
||||||
|
|
||||||
|
# Check if model supports voice cloning via generate method
|
||||||
|
# MLX API may support ref_audio parameter directly
|
||||||
|
try:
|
||||||
|
# Try with voice cloning parameters if supported
|
||||||
|
if ref_audio:
|
||||||
|
# Check if generate accepts ref_audio parameter
|
||||||
|
import inspect
|
||||||
|
sig = inspect.signature(self.model.generate)
|
||||||
|
if "ref_audio" in sig.parameters:
|
||||||
|
# Generate with voice cloning
|
||||||
|
for result in self.model.generate(text, ref_audio=ref_audio, ref_text=ref_text):
|
||||||
|
audio_chunks.append(np.array(result.audio))
|
||||||
|
sample_rate = result.sample_rate
|
||||||
|
else:
|
||||||
|
# Fallback: generate without voice cloning
|
||||||
|
for result in self.model.generate(text):
|
||||||
|
audio_chunks.append(np.array(result.audio))
|
||||||
|
sample_rate = result.sample_rate
|
||||||
|
else:
|
||||||
|
# No voice prompt, generate normally
|
||||||
|
for result in self.model.generate(text):
|
||||||
|
audio_chunks.append(np.array(result.audio))
|
||||||
|
sample_rate = result.sample_rate
|
||||||
|
except Exception as e:
|
||||||
|
# If voice cloning fails, try without it
|
||||||
|
print(f"Warning: Voice cloning failed, generating without voice prompt: {e}")
|
||||||
|
for result in self.model.generate(text):
|
||||||
|
audio_chunks.append(np.array(result.audio))
|
||||||
|
sample_rate = result.sample_rate
|
||||||
|
|
||||||
|
# Concatenate all chunks
|
||||||
|
if audio_chunks:
|
||||||
|
audio = np.concatenate([np.asarray(chunk, dtype=np.float32) for chunk in audio_chunks])
|
||||||
|
else:
|
||||||
|
# Fallback: empty audio
|
||||||
|
audio = np.array([], dtype=np.float32)
|
||||||
|
|
||||||
|
return audio, sample_rate
|
||||||
|
|
||||||
|
# Run blocking inference in thread pool
|
||||||
|
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||||
|
|
||||||
|
return audio, sample_rate
|
||||||
|
|
||||||
|
|
||||||
|
class MLXSTTBackend:
|
||||||
|
"""MLX-based STT backend using mlx-audio Whisper."""
|
||||||
|
|
||||||
|
def __init__(self, model_size: str = "base"):
|
||||||
|
self.model = None
|
||||||
|
self.model_size = model_size
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
return self.model is not None
|
||||||
|
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
"""
|
||||||
|
Lazy load the MLX Whisper model.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size (tiny, base, small, medium, large)
|
||||||
|
"""
|
||||||
|
if model_size is None:
|
||||||
|
model_size = self.model_size
|
||||||
|
|
||||||
|
if self.model is not None and self.model_size == model_size:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Run blocking load in thread pool
|
||||||
|
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||||
|
|
||||||
|
# Alias for compatibility
|
||||||
|
load_model = load_model_async
|
||||||
|
|
||||||
|
def _load_model_sync(self, model_size: str):
|
||||||
|
"""Synchronous model loading."""
|
||||||
|
try:
|
||||||
|
# IMPORTANT: Set up progress tracking BEFORE importing mlx_audio
|
||||||
|
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
|
||||||
|
# Set up progress callback and tracker
|
||||||
|
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||||
|
tracker = HFProgressTracker(progress_callback)
|
||||||
|
|
||||||
|
# Patch tqdm BEFORE importing mlx_audio
|
||||||
|
# This is critical because mlx_audio imports huggingface_hub which imports tqdm
|
||||||
|
print("[DEBUG] Starting tqdm patch BEFORE mlx_audio import")
|
||||||
|
tracker_context = tracker.patch_download()
|
||||||
|
tracker_context.__enter__()
|
||||||
|
print("[DEBUG] tqdm patched, now importing mlx_audio")
|
||||||
|
|
||||||
|
# NOW import mlx_audio - it will use our patched tqdm
|
||||||
|
from mlx_audio.stt import load
|
||||||
|
|
||||||
|
# MLX Whisper uses the standard OpenAI models
|
||||||
|
model_name = f"openai/whisper-{model_size}"
|
||||||
|
|
||||||
|
# Start tracking download task
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
task_manager.start_download(progress_model_name)
|
||||||
|
|
||||||
|
print(f"Loading MLX Whisper model {model_size}...")
|
||||||
|
|
||||||
|
# Initialize progress state
|
||||||
|
progress_manager.update_progress(
|
||||||
|
model_name=progress_model_name,
|
||||||
|
current=0,
|
||||||
|
total=1,
|
||||||
|
filename="",
|
||||||
|
status="downloading",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Load the model (tqdm is already patched from above)
|
||||||
|
try:
|
||||||
|
self.model = load(model_name)
|
||||||
|
finally:
|
||||||
|
# Exit the patch context
|
||||||
|
tracker_context.__exit__(None, None, None)
|
||||||
|
|
||||||
|
self.model_size = model_size
|
||||||
|
|
||||||
|
# Mark as complete
|
||||||
|
progress_manager.mark_complete(progress_model_name)
|
||||||
|
task_manager.complete_download(progress_model_name)
|
||||||
|
|
||||||
|
print(f"MLX Whisper model {model_size} loaded successfully")
|
||||||
|
|
||||||
|
except ImportError as e:
|
||||||
|
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
progress_manager.mark_error(progress_model_name, str(e))
|
||||||
|
task_manager.error_download(progress_model_name, str(e))
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error loading MLX Whisper model: {e}")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
progress_manager.mark_error(progress_model_name, str(e))
|
||||||
|
task_manager.error_download(progress_model_name, str(e))
|
||||||
|
raise
|
||||||
|
|
||||||
|
def unload_model(self):
|
||||||
|
"""Unload the model to free memory."""
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
self.model = None
|
||||||
|
print("MLX Whisper model unloaded")
|
||||||
|
|
||||||
|
async def transcribe(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
language: Optional[str] = None,
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Transcribe audio to text.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_path: Path to audio file
|
||||||
|
language: Optional language hint (en or zh)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Transcribed text
|
||||||
|
"""
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
def _transcribe_sync():
|
||||||
|
"""Run synchronous transcription in thread pool."""
|
||||||
|
# MLX Whisper transcription using generate method
|
||||||
|
# The generate method accepts audio path directly
|
||||||
|
decode_options = {}
|
||||||
|
if language:
|
||||||
|
decode_options["language"] = language
|
||||||
|
|
||||||
|
result = self.model.generate(str(audio_path), **decode_options)
|
||||||
|
|
||||||
|
# Extract text from result
|
||||||
|
if isinstance(result, str):
|
||||||
|
return result.strip()
|
||||||
|
elif isinstance(result, dict):
|
||||||
|
return result.get("text", "").strip()
|
||||||
|
elif hasattr(result, "text"):
|
||||||
|
return result.text.strip()
|
||||||
|
else:
|
||||||
|
return str(result).strip()
|
||||||
|
|
||||||
|
# Run blocking transcription in thread pool
|
||||||
|
return await asyncio.to_thread(_transcribe_sync)
|
||||||
@@ -0,0 +1,486 @@
|
|||||||
|
"""
|
||||||
|
PyTorch backend implementation for TTS and STT.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Optional, List, Tuple
|
||||||
|
import asyncio
|
||||||
|
import torch
|
||||||
|
import numpy as np
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from . import TTSBackend, STTBackend
|
||||||
|
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||||
|
from ..utils.audio import normalize_audio, load_audio
|
||||||
|
from ..utils.progress import get_progress_manager
|
||||||
|
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||||
|
from ..utils.tasks import get_task_manager
|
||||||
|
|
||||||
|
|
||||||
|
class PyTorchTTSBackend:
|
||||||
|
"""PyTorch-based TTS backend using Qwen3-TTS."""
|
||||||
|
|
||||||
|
def __init__(self, model_size: str = "1.7B"):
|
||||||
|
self.model = None
|
||||||
|
self.model_size = model_size
|
||||||
|
self.device = self._get_device()
|
||||||
|
self._current_model_size = None
|
||||||
|
|
||||||
|
def _get_device(self) -> str:
|
||||||
|
"""Get the best available device."""
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
return "cuda"
|
||||||
|
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||||
|
# MPS can have issues, use CPU for stability
|
||||||
|
return "cpu"
|
||||||
|
return "cpu"
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
return self.model is not None
|
||||||
|
|
||||||
|
def _get_model_path(self, model_size: str) -> str:
|
||||||
|
"""
|
||||||
|
Get the HuggingFace Hub model ID.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size (1.7B or 0.6B)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
HuggingFace Hub model ID
|
||||||
|
"""
|
||||||
|
hf_model_map = {
|
||||||
|
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||||
|
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||||
|
}
|
||||||
|
|
||||||
|
if model_size not in hf_model_map:
|
||||||
|
raise ValueError(f"Unknown model size: {model_size}")
|
||||||
|
|
||||||
|
return hf_model_map[model_size]
|
||||||
|
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
"""
|
||||||
|
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size to load (1.7B or 0.6B)
|
||||||
|
"""
|
||||||
|
if model_size is None:
|
||||||
|
model_size = self.model_size
|
||||||
|
|
||||||
|
# If already loaded with correct size, return
|
||||||
|
if self.model is not None and self._current_model_size == model_size:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Unload existing model if different size requested
|
||||||
|
if self.model is not None and self._current_model_size != model_size:
|
||||||
|
self.unload_model()
|
||||||
|
|
||||||
|
# Run blocking load in thread pool
|
||||||
|
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||||
|
|
||||||
|
# Alias for compatibility
|
||||||
|
load_model = load_model_async
|
||||||
|
|
||||||
|
def _load_model_sync(self, model_size: str):
|
||||||
|
"""Synchronous model loading."""
|
||||||
|
try:
|
||||||
|
# IMPORTANT: Set up progress tracking BEFORE importing qwen_tts
|
||||||
|
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
|
||||||
|
# Set up progress callback and tracker
|
||||||
|
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||||
|
tracker = HFProgressTracker(progress_callback)
|
||||||
|
|
||||||
|
# Patch tqdm BEFORE importing qwen_tts
|
||||||
|
tracker_context = tracker.patch_download()
|
||||||
|
tracker_context.__enter__()
|
||||||
|
|
||||||
|
# NOW import qwen_tts - it will use our patched tqdm
|
||||||
|
from qwen_tts import Qwen3TTSModel
|
||||||
|
|
||||||
|
# Get model path (local or HuggingFace Hub ID)
|
||||||
|
model_path = self._get_model_path(model_size)
|
||||||
|
|
||||||
|
print(f"Loading TTS model {model_size} on {self.device}...")
|
||||||
|
|
||||||
|
# Start tracking download task
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
task_manager.start_download(model_name)
|
||||||
|
|
||||||
|
# Initialize progress state to show download has started
|
||||||
|
progress_manager.update_progress(
|
||||||
|
model_name=model_name,
|
||||||
|
current=0,
|
||||||
|
total=1, # Set to 1 initially, will be updated by callback
|
||||||
|
filename="",
|
||||||
|
status="downloading",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Load the model (tqdm is already patched from above)
|
||||||
|
try:
|
||||||
|
self.model = Qwen3TTSModel.from_pretrained(
|
||||||
|
model_path,
|
||||||
|
device_map=self.device,
|
||||||
|
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
# Exit the patch context
|
||||||
|
tracker_context.__exit__(None, None, None)
|
||||||
|
|
||||||
|
# Mark as complete
|
||||||
|
progress_manager.mark_complete(model_name)
|
||||||
|
task_manager.complete_download(model_name)
|
||||||
|
|
||||||
|
self._current_model_size = model_size
|
||||||
|
self.model_size = model_size
|
||||||
|
|
||||||
|
print(f"TTS model {model_size} loaded successfully")
|
||||||
|
|
||||||
|
except ImportError as e:
|
||||||
|
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
progress_manager.mark_error(model_name, str(e))
|
||||||
|
task_manager.error_download(model_name, str(e))
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error loading TTS model: {e}")
|
||||||
|
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
progress_manager.mark_error(model_name, str(e))
|
||||||
|
task_manager.error_download(model_name, str(e))
|
||||||
|
raise
|
||||||
|
|
||||||
|
def unload_model(self):
|
||||||
|
"""Unload the model to free memory."""
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
self.model = None
|
||||||
|
self._current_model_size = None
|
||||||
|
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
|
||||||
|
print("TTS model unloaded")
|
||||||
|
|
||||||
|
async def create_voice_prompt(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
use_cache: bool = True,
|
||||||
|
) -> Tuple[dict, bool]:
|
||||||
|
"""
|
||||||
|
Create voice prompt from reference audio.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_path: Path to reference audio file
|
||||||
|
reference_text: Transcript of reference audio
|
||||||
|
use_cache: Whether to use cached prompt if available
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (voice_prompt_dict, was_cached)
|
||||||
|
"""
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
# Check cache if enabled
|
||||||
|
if use_cache:
|
||||||
|
cache_key = get_cache_key(audio_path, reference_text)
|
||||||
|
cached_prompt = get_cached_voice_prompt(cache_key)
|
||||||
|
if cached_prompt is not None:
|
||||||
|
# Cache stores as torch.Tensor but actual prompt is dict
|
||||||
|
# Convert if needed
|
||||||
|
if isinstance(cached_prompt, dict):
|
||||||
|
# For PyTorch backend, the dict should contain tensors, not file paths
|
||||||
|
# So we can safely return it
|
||||||
|
return cached_prompt, True
|
||||||
|
elif isinstance(cached_prompt, torch.Tensor):
|
||||||
|
# Legacy cache format - convert to dict
|
||||||
|
# This shouldn't happen in practice, but handle it
|
||||||
|
return {"prompt": cached_prompt}, True
|
||||||
|
|
||||||
|
def _create_prompt_sync():
|
||||||
|
"""Run synchronous voice prompt creation in thread pool."""
|
||||||
|
return self.model.create_voice_clone_prompt(
|
||||||
|
ref_audio=str(audio_path),
|
||||||
|
ref_text=reference_text,
|
||||||
|
x_vector_only_mode=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Run blocking operation in thread pool
|
||||||
|
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
|
||||||
|
|
||||||
|
# Cache if enabled
|
||||||
|
if use_cache:
|
||||||
|
cache_key = get_cache_key(audio_path, reference_text)
|
||||||
|
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||||
|
|
||||||
|
return voice_prompt_items, False
|
||||||
|
|
||||||
|
async def combine_voice_prompts(
|
||||||
|
self,
|
||||||
|
audio_paths: List[str],
|
||||||
|
reference_texts: List[str],
|
||||||
|
) -> Tuple[np.ndarray, str]:
|
||||||
|
"""
|
||||||
|
Combine multiple reference samples for better quality.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_paths: List of audio file paths
|
||||||
|
reference_texts: List of reference texts
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (combined_audio, combined_text)
|
||||||
|
"""
|
||||||
|
combined_audio = []
|
||||||
|
|
||||||
|
for audio_path in audio_paths:
|
||||||
|
audio, sr = load_audio(audio_path)
|
||||||
|
audio = normalize_audio(audio)
|
||||||
|
combined_audio.append(audio)
|
||||||
|
|
||||||
|
# Concatenate audio
|
||||||
|
mixed = np.concatenate(combined_audio)
|
||||||
|
mixed = normalize_audio(mixed)
|
||||||
|
|
||||||
|
# Combine texts
|
||||||
|
combined_text = " ".join(reference_texts)
|
||||||
|
|
||||||
|
return mixed, combined_text
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice_prompt: dict,
|
||||||
|
language: str = "en",
|
||||||
|
seed: Optional[int] = None,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> Tuple[np.ndarray, int]:
|
||||||
|
"""
|
||||||
|
Generate audio from text using voice prompt.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to synthesize
|
||||||
|
voice_prompt: Voice prompt dictionary from create_voice_prompt
|
||||||
|
language: Language code (en or zh)
|
||||||
|
seed: Random seed for reproducibility
|
||||||
|
instruct: Natural language instruction for speech delivery control
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (audio_array, sample_rate)
|
||||||
|
"""
|
||||||
|
# Load model
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
def _generate_sync():
|
||||||
|
"""Run synchronous generation in thread pool."""
|
||||||
|
# Set seed if provided
|
||||||
|
if seed is not None:
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.manual_seed(seed)
|
||||||
|
|
||||||
|
# Generate audio - this is the blocking operation
|
||||||
|
wavs, sample_rate = self.model.generate_voice_clone(
|
||||||
|
text=text,
|
||||||
|
voice_clone_prompt=voice_prompt,
|
||||||
|
instruct=instruct,
|
||||||
|
)
|
||||||
|
return wavs[0], sample_rate
|
||||||
|
|
||||||
|
# Run blocking inference in thread pool to avoid blocking event loop
|
||||||
|
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||||
|
|
||||||
|
return audio, sample_rate
|
||||||
|
|
||||||
|
|
||||||
|
class PyTorchSTTBackend:
|
||||||
|
"""PyTorch-based STT backend using Whisper."""
|
||||||
|
|
||||||
|
def __init__(self, model_size: str = "base"):
|
||||||
|
self.model = None
|
||||||
|
self.processor = None
|
||||||
|
self.model_size = model_size
|
||||||
|
self.device = self._get_device()
|
||||||
|
|
||||||
|
def _get_device(self) -> str:
|
||||||
|
"""Get the best available device."""
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
return "cuda"
|
||||||
|
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||||
|
# MPS support for Whisper
|
||||||
|
return "cpu" # Use CPU for stability
|
||||||
|
return "cpu"
|
||||||
|
|
||||||
|
def is_loaded(self) -> bool:
|
||||||
|
"""Check if model is loaded."""
|
||||||
|
return self.model is not None
|
||||||
|
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
"""
|
||||||
|
Lazy load the Whisper model.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
model_size: Model size (tiny, base, small, medium, large)
|
||||||
|
"""
|
||||||
|
print(f"[DEBUG] load_model_async called with size: {model_size}")
|
||||||
|
if model_size is None:
|
||||||
|
model_size = self.model_size
|
||||||
|
|
||||||
|
print(f"[DEBUG] Model already loaded? {self.model is not None}, current size: {self.model_size}, requested: {model_size}")
|
||||||
|
if self.model is not None and self.model_size == model_size:
|
||||||
|
print(f"[DEBUG] Early return - model already loaded")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"[DEBUG] Calling asyncio.to_thread for _load_model_sync")
|
||||||
|
# Run blocking load in thread pool
|
||||||
|
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||||
|
print(f"[DEBUG] asyncio.to_thread completed")
|
||||||
|
|
||||||
|
# Alias for compatibility
|
||||||
|
load_model = load_model_async
|
||||||
|
|
||||||
|
def _load_model_sync(self, model_size: str):
|
||||||
|
"""Synchronous model loading."""
|
||||||
|
print(f"[DEBUG] _load_model_sync called for Whisper {model_size}")
|
||||||
|
try:
|
||||||
|
# IMPORTANT: Set up progress tracking BEFORE importing transformers
|
||||||
|
# This ensures tqdm is patched before any HuggingFace Hub imports
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
|
||||||
|
# Set up progress callback and tracker
|
||||||
|
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||||
|
tracker = HFProgressTracker(progress_callback)
|
||||||
|
|
||||||
|
# Patch tqdm BEFORE importing transformers
|
||||||
|
print("[DEBUG] Starting tqdm patch BEFORE transformers import")
|
||||||
|
tracker_context = tracker.patch_download()
|
||||||
|
tracker_context.__enter__()
|
||||||
|
print("[DEBUG] tqdm patched, now importing transformers")
|
||||||
|
|
||||||
|
# NOW import transformers - it will use our patched tqdm
|
||||||
|
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||||
|
|
||||||
|
model_name = f"openai/whisper-{model_size}"
|
||||||
|
print(f"[DEBUG] Model name: {model_name}")
|
||||||
|
|
||||||
|
# Start tracking download task
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
task_manager.start_download(progress_model_name)
|
||||||
|
print(f"[DEBUG] Task manager started download")
|
||||||
|
|
||||||
|
print(f"Loading Whisper model {model_size} on {self.device}...")
|
||||||
|
|
||||||
|
# Initialize progress state to show download has started
|
||||||
|
print(f"[DEBUG] Calling update_progress...")
|
||||||
|
progress_manager.update_progress(
|
||||||
|
model_name=progress_model_name,
|
||||||
|
current=0,
|
||||||
|
total=1, # Set to 1 initially, will be updated by callback
|
||||||
|
filename="",
|
||||||
|
status="downloading",
|
||||||
|
)
|
||||||
|
print(f"[DEBUG] update_progress called, listeners: {len(progress_manager._listeners.get(progress_model_name, []))}")
|
||||||
|
|
||||||
|
# Load models (tqdm is already patched from above)
|
||||||
|
try:
|
||||||
|
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||||
|
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||||
|
finally:
|
||||||
|
# Exit the patch context
|
||||||
|
tracker_context.__exit__(None, None, None)
|
||||||
|
|
||||||
|
self.model.to(self.device)
|
||||||
|
self.model_size = model_size
|
||||||
|
|
||||||
|
# Mark as complete
|
||||||
|
progress_manager.mark_complete(progress_model_name)
|
||||||
|
task_manager.complete_download(progress_model_name)
|
||||||
|
|
||||||
|
print(f"Whisper model {model_size} loaded successfully")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error loading Whisper model: {e}")
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
progress_manager.mark_error(progress_model_name, str(e))
|
||||||
|
task_manager.error_download(progress_model_name, str(e))
|
||||||
|
raise
|
||||||
|
|
||||||
|
def unload_model(self):
|
||||||
|
"""Unload the model to free memory."""
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
del self.processor
|
||||||
|
self.model = None
|
||||||
|
self.processor = None
|
||||||
|
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
|
||||||
|
print("Whisper model unloaded")
|
||||||
|
|
||||||
|
async def transcribe(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
language: Optional[str] = None,
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Transcribe audio to text.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
audio_path: Path to audio file
|
||||||
|
language: Optional language hint (en or zh)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Transcribed text
|
||||||
|
"""
|
||||||
|
await self.load_model_async(None)
|
||||||
|
|
||||||
|
def _transcribe_sync():
|
||||||
|
"""Run synchronous transcription in thread pool."""
|
||||||
|
# Load audio
|
||||||
|
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||||
|
|
||||||
|
# Process audio
|
||||||
|
inputs = self.processor(
|
||||||
|
audio,
|
||||||
|
sampling_rate=16000,
|
||||||
|
return_tensors="pt",
|
||||||
|
)
|
||||||
|
inputs = inputs.to(self.device)
|
||||||
|
|
||||||
|
# Set language if provided
|
||||||
|
forced_decoder_ids = None
|
||||||
|
if language:
|
||||||
|
# Support all languages from frontend: en, zh, ja, ko, de, fr, ru, pt, es, it
|
||||||
|
# Whisper supports these and many more
|
||||||
|
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||||
|
language=language,
|
||||||
|
task="transcribe",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Generate transcription
|
||||||
|
with torch.no_grad():
|
||||||
|
predicted_ids = self.model.generate(
|
||||||
|
inputs["input_features"],
|
||||||
|
forced_decoder_ids=forced_decoder_ids,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Decode
|
||||||
|
transcription = self.processor.batch_decode(
|
||||||
|
predicted_ids,
|
||||||
|
skip_special_tokens=True,
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
return transcription.strip()
|
||||||
|
|
||||||
|
# Run blocking transcription in thread pool
|
||||||
|
return await asyncio.to_thread(_transcribe_sync)
|
||||||
+38
-8
@@ -4,16 +4,19 @@ PyInstaller build script for creating standalone Python server binary.
|
|||||||
|
|
||||||
import PyInstaller.__main__
|
import PyInstaller.__main__
|
||||||
import os
|
import os
|
||||||
|
import platform
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def is_apple_silicon():
|
||||||
|
"""Check if running on Apple Silicon."""
|
||||||
|
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||||
|
|
||||||
|
|
||||||
def build_server():
|
def build_server():
|
||||||
"""Build Python server as standalone binary."""
|
"""Build Python server as standalone binary."""
|
||||||
backend_dir = Path(__file__).parent
|
backend_dir = Path(__file__).parent
|
||||||
|
|
||||||
# Check for local editable qwen_tts install
|
|
||||||
local_qwen_path = Path.home() / 'Projects' / 'voice' / 'Qwen3-TTS'
|
|
||||||
|
|
||||||
# PyInstaller arguments
|
# PyInstaller arguments
|
||||||
args = [
|
args = [
|
||||||
'server.py', # Use server.py as entry point instead of main.py
|
'server.py', # Use server.py as entry point instead of main.py
|
||||||
@@ -21,12 +24,13 @@ def build_server():
|
|||||||
'--name', 'voicebox-server',
|
'--name', 'voicebox-server',
|
||||||
]
|
]
|
||||||
|
|
||||||
# Add local qwen_tts path if it exists (for editable installs)
|
# Add local qwen_tts path if specified (for editable installs)
|
||||||
if local_qwen_path.exists():
|
qwen_tts_path = os.getenv('QWEN_TTS_PATH')
|
||||||
args.extend(['--paths', str(local_qwen_path)])
|
if qwen_tts_path and Path(qwen_tts_path).exists():
|
||||||
print(f"Using local qwen_tts source from: {local_qwen_path}")
|
args.extend(['--paths', str(qwen_tts_path)])
|
||||||
|
print(f"Using local qwen_tts source from: {qwen_tts_path}")
|
||||||
|
|
||||||
# Add hidden imports
|
# Add common hidden imports
|
||||||
args.extend([
|
args.extend([
|
||||||
'--hidden-import', 'backend',
|
'--hidden-import', 'backend',
|
||||||
'--hidden-import', 'backend.main',
|
'--hidden-import', 'backend.main',
|
||||||
@@ -37,6 +41,9 @@ def build_server():
|
|||||||
'--hidden-import', 'backend.history',
|
'--hidden-import', 'backend.history',
|
||||||
'--hidden-import', 'backend.tts',
|
'--hidden-import', 'backend.tts',
|
||||||
'--hidden-import', 'backend.transcribe',
|
'--hidden-import', 'backend.transcribe',
|
||||||
|
'--hidden-import', 'backend.platform_detect',
|
||||||
|
'--hidden-import', 'backend.backends',
|
||||||
|
'--hidden-import', 'backend.backends.pytorch_backend',
|
||||||
'--hidden-import', 'backend.utils.audio',
|
'--hidden-import', 'backend.utils.audio',
|
||||||
'--hidden-import', 'backend.utils.cache',
|
'--hidden-import', 'backend.utils.cache',
|
||||||
'--hidden-import', 'backend.utils.progress',
|
'--hidden-import', 'backend.utils.progress',
|
||||||
@@ -61,6 +68,29 @@ def build_server():
|
|||||||
# Fix for pkg_resources and jaraco namespace packages
|
# Fix for pkg_resources and jaraco namespace packages
|
||||||
'--hidden-import', 'pkg_resources.extern',
|
'--hidden-import', 'pkg_resources.extern',
|
||||||
'--collect-submodules', 'jaraco',
|
'--collect-submodules', 'jaraco',
|
||||||
|
])
|
||||||
|
|
||||||
|
# Add MLX-specific imports if building on Apple Silicon
|
||||||
|
if is_apple_silicon():
|
||||||
|
print("Building for Apple Silicon - including MLX dependencies")
|
||||||
|
args.extend([
|
||||||
|
'--hidden-import', 'backend.backends.mlx_backend',
|
||||||
|
'--hidden-import', 'mlx',
|
||||||
|
'--hidden-import', 'mlx.core',
|
||||||
|
'--hidden-import', 'mlx.nn',
|
||||||
|
'--hidden-import', 'mlx_audio',
|
||||||
|
'--hidden-import', 'mlx_audio.tts',
|
||||||
|
'--hidden-import', 'mlx_audio.stt',
|
||||||
|
'--collect-submodules', 'mlx',
|
||||||
|
'--collect-submodules', 'mlx_audio',
|
||||||
|
# Collect MLX data files including Metal shader libraries (.metallib)
|
||||||
|
'--collect-data', 'mlx',
|
||||||
|
'--collect-data', 'mlx_audio',
|
||||||
|
])
|
||||||
|
else:
|
||||||
|
print("Building for non-Apple Silicon platform - PyTorch only")
|
||||||
|
|
||||||
|
args.extend([
|
||||||
'--noconfirm',
|
'--noconfirm',
|
||||||
'--clean',
|
'--clean',
|
||||||
])
|
])
|
||||||
|
|||||||
+12
-1
@@ -17,11 +17,12 @@ Base = declarative_base()
|
|||||||
class VoiceProfile(Base):
|
class VoiceProfile(Base):
|
||||||
"""Voice profile database model."""
|
"""Voice profile database model."""
|
||||||
__tablename__ = "profiles"
|
__tablename__ = "profiles"
|
||||||
|
|
||||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||||
name = Column(String, unique=True, nullable=False)
|
name = Column(String, unique=True, nullable=False)
|
||||||
description = Column(Text)
|
description = Column(Text)
|
||||||
language = Column(String, default="en")
|
language = Column(String, default="en")
|
||||||
|
avatar_path = Column(String, nullable=True)
|
||||||
created_at = Column(DateTime, default=datetime.utcnow)
|
created_at = Column(DateTime, default=datetime.utcnow)
|
||||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||||
|
|
||||||
@@ -277,6 +278,16 @@ def _run_migrations(engine):
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
print("Added trim_end_ms column to story_items")
|
print("Added trim_end_ms column to story_items")
|
||||||
|
|
||||||
|
# Migration: Add avatar_path to profiles table
|
||||||
|
if 'profiles' in inspector.get_table_names():
|
||||||
|
columns = {col['name'] for col in inspector.get_columns('profiles')}
|
||||||
|
if 'avatar_path' not in columns:
|
||||||
|
print("Migrating profiles: adding avatar_path column")
|
||||||
|
with engine.connect() as conn:
|
||||||
|
conn.execute(text("ALTER TABLE profiles ADD COLUMN avatar_path VARCHAR"))
|
||||||
|
conn.commit()
|
||||||
|
print("Added avatar_path column to profiles")
|
||||||
|
|
||||||
|
|
||||||
def get_db():
|
def get_db():
|
||||||
"""Get database session (generator for dependency injection)."""
|
"""Get database session (generator for dependency injection)."""
|
||||||
|
|||||||
@@ -75,6 +75,16 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
|
|||||||
zip_buffer = io.BytesIO()
|
zip_buffer = io.BytesIO()
|
||||||
|
|
||||||
with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
|
with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
|
||||||
|
# Check if profile has avatar
|
||||||
|
has_avatar = False
|
||||||
|
if profile.avatar_path:
|
||||||
|
avatar_path = Path(profile.avatar_path)
|
||||||
|
if avatar_path.exists():
|
||||||
|
has_avatar = True
|
||||||
|
# Add avatar to ZIP root with original extension
|
||||||
|
avatar_ext = avatar_path.suffix
|
||||||
|
zip_file.write(avatar_path, f"avatar{avatar_ext}")
|
||||||
|
|
||||||
# Create manifest.json
|
# Create manifest.json
|
||||||
manifest = {
|
manifest = {
|
||||||
"version": "1.0",
|
"version": "1.0",
|
||||||
@@ -82,30 +92,31 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
|
|||||||
"name": profile.name,
|
"name": profile.name,
|
||||||
"description": profile.description,
|
"description": profile.description,
|
||||||
"language": profile.language,
|
"language": profile.language,
|
||||||
}
|
},
|
||||||
|
"has_avatar": has_avatar,
|
||||||
}
|
}
|
||||||
zip_file.writestr("manifest.json", json.dumps(manifest, indent=2))
|
zip_file.writestr("manifest.json", json.dumps(manifest, indent=2))
|
||||||
|
|
||||||
# Create samples.json mapping
|
# Create samples.json mapping
|
||||||
samples_data = {}
|
samples_data = {}
|
||||||
profile_dir = _get_profiles_dir() / profile_id
|
profile_dir = _get_profiles_dir() / profile_id
|
||||||
|
|
||||||
for sample in samples:
|
for sample in samples:
|
||||||
# Get filename from audio_path (should be {sample_id}.wav)
|
# Get filename from audio_path (should be {sample_id}.wav)
|
||||||
audio_path = Path(sample.audio_path)
|
audio_path = Path(sample.audio_path)
|
||||||
filename = audio_path.name
|
filename = audio_path.name
|
||||||
|
|
||||||
# Read audio file
|
# Read audio file
|
||||||
if not audio_path.exists():
|
if not audio_path.exists():
|
||||||
raise ValueError(f"Audio file not found: {audio_path}")
|
raise ValueError(f"Audio file not found: {audio_path}")
|
||||||
|
|
||||||
# Add to samples directory in ZIP
|
# Add to samples directory in ZIP
|
||||||
zip_path = f"samples/{filename}"
|
zip_path = f"samples/{filename}"
|
||||||
zip_file.write(audio_path, zip_path)
|
zip_file.write(audio_path, zip_path)
|
||||||
|
|
||||||
# Map filename to reference text
|
# Map filename to reference text
|
||||||
samples_data[filename] = sample.reference_text
|
samples_data[filename] = sample.reference_text
|
||||||
|
|
||||||
zip_file.writestr("samples.json", json.dumps(samples_data, indent=2))
|
zip_file.writestr("samples.json", json.dumps(samples_data, indent=2))
|
||||||
|
|
||||||
zip_buffer.seek(0)
|
zip_buffer.seek(0)
|
||||||
@@ -168,11 +179,31 @@ async def import_profile_from_zip(file_bytes: bytes, db: Session) -> VoiceProfil
|
|||||||
)
|
)
|
||||||
|
|
||||||
profile = await create_profile(profile_create, db)
|
profile = await create_profile(profile_create, db)
|
||||||
|
|
||||||
# Extract and add samples
|
# Extract and add samples
|
||||||
profile_dir = _get_profiles_dir() / profile.id
|
profile_dir = _get_profiles_dir() / profile.id
|
||||||
profile_dir.mkdir(parents=True, exist_ok=True)
|
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Handle avatar if present
|
||||||
|
avatar_files = [f for f in namelist if f.startswith("avatar.")]
|
||||||
|
if avatar_files:
|
||||||
|
try:
|
||||||
|
avatar_file = avatar_files[0]
|
||||||
|
# Extract to temporary file
|
||||||
|
import tempfile
|
||||||
|
with tempfile.NamedTemporaryFile(suffix=Path(avatar_file).suffix, delete=False) as tmp:
|
||||||
|
tmp.write(zip_file.read(avatar_file))
|
||||||
|
tmp_path = tmp.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .profiles import upload_avatar
|
||||||
|
await upload_avatar(profile.id, tmp_path, db)
|
||||||
|
finally:
|
||||||
|
Path(tmp_path).unlink(missing_ok=True)
|
||||||
|
except Exception as e:
|
||||||
|
# Avatar import is optional - continue even if it fails
|
||||||
|
pass
|
||||||
|
|
||||||
for filename, reference_text in samples_data.items():
|
for filename, reference_text in samples_data.items():
|
||||||
# Validate filename
|
# Validate filename
|
||||||
if not filename.endswith('.wav'):
|
if not filename.endswith('.wav'):
|
||||||
|
|||||||
+263
-40
@@ -11,6 +11,7 @@ from fastapi.staticfiles import StaticFiles
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
from typing import List, Optional
|
from typing import List, Optional
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
import asyncio
|
||||||
import uvicorn
|
import uvicorn
|
||||||
import argparse
|
import argparse
|
||||||
import torch
|
import torch
|
||||||
@@ -18,16 +19,21 @@ import tempfile
|
|||||||
import io
|
import io
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import uuid
|
import uuid
|
||||||
|
import asyncio
|
||||||
|
import signal
|
||||||
|
import os
|
||||||
|
|
||||||
from . import database, models, profiles, history, tts, transcribe, config, export_import, channels, stories
|
from . import database, models, profiles, history, tts, transcribe, config, export_import, channels, stories, __version__
|
||||||
from .database import get_db, Generation as DBGeneration, VoiceProfile as DBVoiceProfile
|
from .database import get_db, Generation as DBGeneration, VoiceProfile as DBVoiceProfile
|
||||||
from .utils.progress import get_progress_manager
|
from .utils.progress import get_progress_manager
|
||||||
from .utils.tasks import get_task_manager
|
from .utils.tasks import get_task_manager
|
||||||
|
from .utils.cache import clear_voice_prompt_cache
|
||||||
|
from .platform_detect import get_backend_type
|
||||||
|
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="voicebox API",
|
title="voicebox API",
|
||||||
description="Production-quality Qwen3-TTS voice cloning API",
|
description="Production-quality Qwen3-TTS voice cloning API",
|
||||||
version="0.1.0",
|
version=__version__,
|
||||||
)
|
)
|
||||||
|
|
||||||
# CORS middleware
|
# CORS middleware
|
||||||
@@ -47,23 +53,43 @@ app.add_middleware(
|
|||||||
@app.get("/")
|
@app.get("/")
|
||||||
async def root():
|
async def root():
|
||||||
"""Root endpoint."""
|
"""Root endpoint."""
|
||||||
return {"message": "voicebox API", "version": "0.1.7"}
|
return {"message": "voicebox API", "version": __version__}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/shutdown")
|
||||||
|
async def shutdown():
|
||||||
|
"""Gracefully shutdown the server."""
|
||||||
|
async def shutdown_async():
|
||||||
|
await asyncio.sleep(0.1) # Give response time to send
|
||||||
|
os.kill(os.getpid(), signal.SIGTERM)
|
||||||
|
|
||||||
|
asyncio.create_task(shutdown_async())
|
||||||
|
return {"message": "Shutting down..."}
|
||||||
|
|
||||||
|
|
||||||
@app.get("/health", response_model=models.HealthResponse)
|
@app.get("/health", response_model=models.HealthResponse)
|
||||||
async def health():
|
async def health():
|
||||||
"""Health check endpoint."""
|
"""Health check endpoint."""
|
||||||
from huggingface_hub import hf_hub_download
|
from huggingface_hub import hf_hub_download, constants as hf_constants
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import os
|
import os
|
||||||
|
|
||||||
tts_model = tts.get_tts_model()
|
tts_model = tts.get_tts_model()
|
||||||
|
backend_type = get_backend_type()
|
||||||
|
|
||||||
# Check for GPU availability (CUDA or MPS)
|
# Check for GPU availability (CUDA or MPS)
|
||||||
has_cuda = torch.cuda.is_available()
|
has_cuda = torch.cuda.is_available()
|
||||||
has_mps = hasattr(torch.backends, 'mps') and torch.backends.mps.is_available()
|
has_mps = hasattr(torch.backends, 'mps') and torch.backends.mps.is_available()
|
||||||
gpu_available = has_cuda or has_mps
|
gpu_available = has_cuda or has_mps
|
||||||
|
|
||||||
|
gpu_type = None
|
||||||
|
if has_cuda:
|
||||||
|
gpu_type = f"CUDA ({torch.cuda.get_device_name(0)})"
|
||||||
|
elif has_mps:
|
||||||
|
gpu_type = "MPS (Apple Silicon)"
|
||||||
|
elif backend_type == "mlx":
|
||||||
|
gpu_type = "Metal (Apple Silicon via MLX)"
|
||||||
|
|
||||||
vram_used = None
|
vram_used = None
|
||||||
if has_cuda:
|
if has_cuda:
|
||||||
vram_used = torch.cuda.memory_allocated() / 1024 / 1024 # MB
|
vram_used = torch.cuda.memory_allocated() / 1024 / 1024 # MB
|
||||||
@@ -90,7 +116,11 @@ async def health():
|
|||||||
model_downloaded = None
|
model_downloaded = None
|
||||||
try:
|
try:
|
||||||
# Check if the default model (1.7B) is cached
|
# Check if the default model (1.7B) is cached
|
||||||
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
# Use different model IDs based on backend
|
||||||
|
if backend_type == "mlx":
|
||||||
|
default_model_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
|
||||||
|
else:
|
||||||
|
default_model_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||||
|
|
||||||
# Method 1: Try scan_cache_dir if available
|
# Method 1: Try scan_cache_dir if available
|
||||||
try:
|
try:
|
||||||
@@ -101,15 +131,16 @@ async def health():
|
|||||||
model_downloaded = True
|
model_downloaded = True
|
||||||
break
|
break
|
||||||
except (ImportError, Exception):
|
except (ImportError, Exception):
|
||||||
# Method 2: Check cache directory
|
# Method 2: Check cache directory (using HuggingFace's OS-specific cache location)
|
||||||
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
|
cache_dir = hf_constants.HF_HUB_CACHE
|
||||||
repo_cache = Path(cache_dir) / "models--" + default_model_id.replace("/", "--")
|
repo_cache = Path(cache_dir) / ("models--" + default_model_id.replace("/", "--"))
|
||||||
if repo_cache.exists():
|
if repo_cache.exists():
|
||||||
has_model_files = (
|
has_model_files = (
|
||||||
any(repo_cache.rglob("*.bin")) or
|
any(repo_cache.rglob("*.bin")) or
|
||||||
any(repo_cache.rglob("*.safetensors")) or
|
any(repo_cache.rglob("*.safetensors")) or
|
||||||
any(repo_cache.rglob("*.pt")) or
|
any(repo_cache.rglob("*.pt")) or
|
||||||
any(repo_cache.rglob("*.pth"))
|
any(repo_cache.rglob("*.pth")) or
|
||||||
|
any(repo_cache.rglob("*.npz")) # MLX models may use npz
|
||||||
)
|
)
|
||||||
model_downloaded = has_model_files
|
model_downloaded = has_model_files
|
||||||
except Exception:
|
except Exception:
|
||||||
@@ -121,7 +152,9 @@ async def health():
|
|||||||
model_downloaded=model_downloaded,
|
model_downloaded=model_downloaded,
|
||||||
model_size=model_size,
|
model_size=model_size,
|
||||||
gpu_available=gpu_available,
|
gpu_available=gpu_available,
|
||||||
|
gpu_type=gpu_type,
|
||||||
vram_used_mb=vram_used,
|
vram_used_mb=vram_used,
|
||||||
|
backend_type=backend_type,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -261,6 +294,74 @@ async def delete_profile_sample(
|
|||||||
return {"message": "Sample deleted successfully"}
|
return {"message": "Sample deleted successfully"}
|
||||||
|
|
||||||
|
|
||||||
|
@app.put("/profiles/samples/{sample_id}", response_model=models.ProfileSampleResponse)
|
||||||
|
async def update_profile_sample(
|
||||||
|
sample_id: str,
|
||||||
|
data: models.ProfileSampleUpdate,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
"""Update a profile sample's reference text."""
|
||||||
|
sample = await profiles.update_profile_sample(sample_id, data.reference_text, db)
|
||||||
|
if not sample:
|
||||||
|
raise HTTPException(status_code=404, detail="Sample not found")
|
||||||
|
return sample
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{profile_id}/avatar", response_model=models.VoiceProfileResponse)
|
||||||
|
async def upload_profile_avatar(
|
||||||
|
profile_id: str,
|
||||||
|
file: UploadFile = File(...),
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
"""Upload or update avatar image for a profile."""
|
||||||
|
# Save uploaded file to temp location
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix=Path(file.filename).suffix) as tmp:
|
||||||
|
content = await file.read()
|
||||||
|
tmp.write(content)
|
||||||
|
tmp_path = tmp.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
profile = await profiles.upload_avatar(profile_id, tmp_path, db)
|
||||||
|
return profile
|
||||||
|
except ValueError as e:
|
||||||
|
raise HTTPException(status_code=400, detail=str(e))
|
||||||
|
finally:
|
||||||
|
# Clean up temp file
|
||||||
|
Path(tmp_path).unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/profiles/{profile_id}/avatar")
|
||||||
|
async def get_profile_avatar(
|
||||||
|
profile_id: str,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
"""Get avatar image for a profile."""
|
||||||
|
profile = await profiles.get_profile(profile_id, db)
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
|
||||||
|
if not profile.avatar_path:
|
||||||
|
raise HTTPException(status_code=404, detail="No avatar found for this profile")
|
||||||
|
|
||||||
|
avatar_path = Path(profile.avatar_path)
|
||||||
|
if not avatar_path.exists():
|
||||||
|
raise HTTPException(status_code=404, detail="Avatar file not found")
|
||||||
|
|
||||||
|
return FileResponse(avatar_path)
|
||||||
|
|
||||||
|
|
||||||
|
@app.delete("/profiles/{profile_id}/avatar")
|
||||||
|
async def delete_profile_avatar(
|
||||||
|
profile_id: str,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
"""Delete avatar image for a profile."""
|
||||||
|
success = await profiles.delete_avatar(profile_id, db)
|
||||||
|
if not success:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found or no avatar to delete")
|
||||||
|
return {"message": "Avatar deleted successfully"}
|
||||||
|
|
||||||
|
|
||||||
@app.get("/profiles/{profile_id}/export")
|
@app.get("/profiles/{profile_id}/export")
|
||||||
async def export_profile(
|
async def export_profile(
|
||||||
profile_id: str,
|
profile_id: str,
|
||||||
@@ -451,6 +552,36 @@ async def generate_speech(
|
|||||||
tts_model = tts.get_tts_model()
|
tts_model = tts.get_tts_model()
|
||||||
# Load the requested model size if different from current (async to not block)
|
# Load the requested model size if different from current (async to not block)
|
||||||
model_size = data.model_size or "1.7B"
|
model_size = data.model_size or "1.7B"
|
||||||
|
|
||||||
|
# Check if model needs to be downloaded first
|
||||||
|
model_path = tts_model._get_model_path(model_size)
|
||||||
|
if model_path.startswith("Qwen/"):
|
||||||
|
# Model not cached - check if it exists remotely or needs download
|
||||||
|
from huggingface_hub import constants as hf_constants
|
||||||
|
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_path.replace("/", "--"))
|
||||||
|
if not repo_cache.exists():
|
||||||
|
# Start download in background
|
||||||
|
model_name = f"qwen-tts-{model_size}"
|
||||||
|
|
||||||
|
async def download_model_background():
|
||||||
|
try:
|
||||||
|
await tts_model.load_model_async(model_size)
|
||||||
|
except Exception as e:
|
||||||
|
task_manager.error_download(model_name, str(e))
|
||||||
|
|
||||||
|
task_manager.start_download(model_name)
|
||||||
|
asyncio.create_task(download_model_background())
|
||||||
|
|
||||||
|
# Return 202 Accepted with download info
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=202,
|
||||||
|
detail={
|
||||||
|
"message": f"Model {model_size} is being downloaded. Please wait and try again.",
|
||||||
|
"model_name": model_name,
|
||||||
|
"downloading": True
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
await tts_model.load_model_async(model_size)
|
await tts_model.load_model_async(model_size)
|
||||||
audio, sample_rate = await tts_model.generate(
|
audio, sample_rate = await tts_model.generate(
|
||||||
data.text,
|
data.text,
|
||||||
@@ -684,6 +815,37 @@ async def transcribe_audio(
|
|||||||
|
|
||||||
# Transcribe
|
# Transcribe
|
||||||
whisper_model = transcribe.get_whisper_model()
|
whisper_model = transcribe.get_whisper_model()
|
||||||
|
|
||||||
|
# Check if Whisper model is downloaded (uses default size "base")
|
||||||
|
model_size = whisper_model.model_size
|
||||||
|
model_name = f"openai/whisper-{model_size}"
|
||||||
|
|
||||||
|
# Check if model is cached
|
||||||
|
from huggingface_hub import constants as hf_constants
|
||||||
|
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_name.replace("/", "--"))
|
||||||
|
if not repo_cache.exists():
|
||||||
|
# Start download in background
|
||||||
|
progress_model_name = f"whisper-{model_size}"
|
||||||
|
|
||||||
|
async def download_whisper_background():
|
||||||
|
try:
|
||||||
|
await whisper_model.load_model_async(model_size)
|
||||||
|
except Exception as e:
|
||||||
|
get_task_manager().error_download(progress_model_name, str(e))
|
||||||
|
|
||||||
|
get_task_manager().start_download(progress_model_name)
|
||||||
|
asyncio.create_task(download_whisper_background())
|
||||||
|
|
||||||
|
# Return 202 Accepted
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=202,
|
||||||
|
detail={
|
||||||
|
"message": f"Whisper model {model_size} is being downloaded. Please wait and try again.",
|
||||||
|
"model_name": progress_model_name,
|
||||||
|
"downloading": True
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
text = await whisper_model.transcribe(tmp_path, language)
|
text = await whisper_model.transcribe(tmp_path, language)
|
||||||
|
|
||||||
return models.TranscriptionResponse(
|
return models.TranscriptionResponse(
|
||||||
@@ -994,10 +1156,12 @@ async def get_model_progress(model_name: str):
|
|||||||
@app.get("/models/status", response_model=models.ModelStatusListResponse)
|
@app.get("/models/status", response_model=models.ModelStatusListResponse)
|
||||||
async def get_model_status():
|
async def get_model_status():
|
||||||
"""Get status of all available models."""
|
"""Get status of all available models."""
|
||||||
from huggingface_hub import hf_hub_download
|
from huggingface_hub import hf_hub_download, constants as hf_constants
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import os
|
import os
|
||||||
|
|
||||||
|
backend_type = get_backend_type()
|
||||||
|
|
||||||
# Try to import scan_cache_dir (might not be available in older versions)
|
# Try to import scan_cache_dir (might not be available in older versions)
|
||||||
try:
|
try:
|
||||||
from huggingface_hub import scan_cache_dir
|
from huggingface_hub import scan_cache_dir
|
||||||
@@ -1009,7 +1173,7 @@ async def get_model_status():
|
|||||||
"""Check if TTS model is loaded with specific size."""
|
"""Check if TTS model is loaded with specific size."""
|
||||||
try:
|
try:
|
||||||
tts_model = tts.get_tts_model()
|
tts_model = tts.get_tts_model()
|
||||||
return tts_model.is_loaded() and tts_model.model_size == model_size
|
return tts_model.is_loaded() and getattr(tts_model, 'model_size', None) == model_size
|
||||||
except Exception:
|
except Exception:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
@@ -1017,50 +1181,66 @@ async def get_model_status():
|
|||||||
"""Check if Whisper model is loaded with specific size."""
|
"""Check if Whisper model is loaded with specific size."""
|
||||||
try:
|
try:
|
||||||
whisper_model = transcribe.get_whisper_model()
|
whisper_model = transcribe.get_whisper_model()
|
||||||
return whisper_model.is_loaded() and whisper_model.model_size == model_size
|
return whisper_model.is_loaded() and getattr(whisper_model, 'model_size', None) == model_size
|
||||||
except Exception:
|
except Exception:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
# Use backend-specific model IDs
|
||||||
|
if backend_type == "mlx":
|
||||||
|
tts_1_7b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
|
||||||
|
tts_0_6b_id = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16" # Fallback to 1.7B
|
||||||
|
whisper_base_id = "mlx-community/whisper-base"
|
||||||
|
whisper_small_id = "mlx-community/whisper-small"
|
||||||
|
whisper_medium_id = "mlx-community/whisper-medium"
|
||||||
|
whisper_large_id = "mlx-community/whisper-large"
|
||||||
|
else:
|
||||||
|
tts_1_7b_id = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||||
|
tts_0_6b_id = "Qwen/Qwen3-TTS-12Hz-0.6B-Base"
|
||||||
|
whisper_base_id = "openai/whisper-base"
|
||||||
|
whisper_small_id = "openai/whisper-small"
|
||||||
|
whisper_medium_id = "openai/whisper-medium"
|
||||||
|
whisper_large_id = "openai/whisper-large"
|
||||||
|
|
||||||
model_configs = [
|
model_configs = [
|
||||||
{
|
{
|
||||||
"model_name": "qwen-tts-1.7B",
|
"model_name": "qwen-tts-1.7B",
|
||||||
"display_name": "Qwen TTS 1.7B",
|
"display_name": "Qwen TTS 1.7B",
|
||||||
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
"hf_repo_id": tts_1_7b_id,
|
||||||
"model_size": "1.7B",
|
"model_size": "1.7B",
|
||||||
"check_loaded": lambda: check_tts_loaded("1.7B"),
|
"check_loaded": lambda: check_tts_loaded("1.7B"),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"model_name": "qwen-tts-0.6B",
|
"model_name": "qwen-tts-0.6B",
|
||||||
"display_name": "Qwen TTS 0.6B",
|
"display_name": "Qwen TTS 0.6B",
|
||||||
"hf_repo_id": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
"hf_repo_id": tts_0_6b_id,
|
||||||
"model_size": "0.6B",
|
"model_size": "0.6B",
|
||||||
"check_loaded": lambda: check_tts_loaded("0.6B"),
|
"check_loaded": lambda: check_tts_loaded("0.6B"),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"model_name": "whisper-base",
|
"model_name": "whisper-base",
|
||||||
"display_name": "Whisper Base",
|
"display_name": "Whisper Base",
|
||||||
"hf_repo_id": "openai/whisper-base",
|
"hf_repo_id": whisper_base_id,
|
||||||
"model_size": "base",
|
"model_size": "base",
|
||||||
"check_loaded": lambda: check_whisper_loaded("base"),
|
"check_loaded": lambda: check_whisper_loaded("base"),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"model_name": "whisper-small",
|
"model_name": "whisper-small",
|
||||||
"display_name": "Whisper Small",
|
"display_name": "Whisper Small",
|
||||||
"hf_repo_id": "openai/whisper-small",
|
"hf_repo_id": whisper_small_id,
|
||||||
"model_size": "small",
|
"model_size": "small",
|
||||||
"check_loaded": lambda: check_whisper_loaded("small"),
|
"check_loaded": lambda: check_whisper_loaded("small"),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"model_name": "whisper-medium",
|
"model_name": "whisper-medium",
|
||||||
"display_name": "Whisper Medium",
|
"display_name": "Whisper Medium",
|
||||||
"hf_repo_id": "openai/whisper-medium",
|
"hf_repo_id": whisper_medium_id,
|
||||||
"model_size": "medium",
|
"model_size": "medium",
|
||||||
"check_loaded": lambda: check_whisper_loaded("medium"),
|
"check_loaded": lambda: check_whisper_loaded("medium"),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"model_name": "whisper-large",
|
"model_name": "whisper-large",
|
||||||
"display_name": "Whisper Large",
|
"display_name": "Whisper Large",
|
||||||
"hf_repo_id": "openai/whisper-large",
|
"hf_repo_id": whisper_large_id,
|
||||||
"model_size": "large",
|
"model_size": "large",
|
||||||
"check_loaded": lambda: check_whisper_loaded("large"),
|
"check_loaded": lambda: check_whisper_loaded("large"),
|
||||||
},
|
},
|
||||||
@@ -1097,19 +1277,21 @@ async def get_model_status():
|
|||||||
pass
|
pass
|
||||||
break
|
break
|
||||||
|
|
||||||
# Method 2: Fallback to checking cache directory directly
|
# Method 2: Fallback to checking cache directory directly (using HuggingFace's OS-specific cache location)
|
||||||
if not downloaded:
|
if not downloaded:
|
||||||
try:
|
try:
|
||||||
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
|
cache_dir = hf_constants.HF_HUB_CACHE
|
||||||
repo_cache = Path(cache_dir) / "models--" + config["hf_repo_id"].replace("/", "--")
|
repo_cache = Path(cache_dir) / ("models--" + config["hf_repo_id"].replace("/", "--"))
|
||||||
|
|
||||||
if repo_cache.exists():
|
if repo_cache.exists():
|
||||||
# Check for model files (bin, safetensors, or other common model files)
|
# Check for model files (bin, safetensors, or other common model files)
|
||||||
|
# MLX models may use .npz or .safetensors
|
||||||
has_model_files = (
|
has_model_files = (
|
||||||
any(repo_cache.rglob("*.bin")) or
|
any(repo_cache.rglob("*.bin")) or
|
||||||
any(repo_cache.rglob("*.safetensors")) or
|
any(repo_cache.rglob("*.safetensors")) or
|
||||||
any(repo_cache.rglob("*.pt")) or
|
any(repo_cache.rglob("*.pt")) or
|
||||||
any(repo_cache.rglob("*.pth")) or
|
any(repo_cache.rglob("*.pth")) or
|
||||||
|
any(repo_cache.rglob("*.npz")) or
|
||||||
any(repo_cache.rglob("model.safetensors.index.json")) or
|
any(repo_cache.rglob("model.safetensors.index.json")) or
|
||||||
any(repo_cache.rglob("pytorch_model.bin.index.json"))
|
any(repo_cache.rglob("pytorch_model.bin.index.json"))
|
||||||
)
|
)
|
||||||
@@ -1209,22 +1391,26 @@ async def trigger_model_download(request: models.ModelDownloadRequest):
|
|||||||
|
|
||||||
config = model_configs[request.model_name]
|
config = model_configs[request.model_name]
|
||||||
|
|
||||||
try:
|
async def download_in_background():
|
||||||
# Start tracking download
|
"""Download model in background without blocking the HTTP request."""
|
||||||
task_manager.start_download(request.model_name)
|
try:
|
||||||
|
# Call the load function (which may be async)
|
||||||
# Trigger download by loading the model (which will download if not cached)
|
result = config["load_func"]()
|
||||||
# Run in background to avoid blocking
|
# If it's a coroutine, await it
|
||||||
await asyncio.to_thread(config["load_func"])
|
if asyncio.iscoroutine(result):
|
||||||
|
await result
|
||||||
# Mark download as complete
|
task_manager.complete_download(request.model_name)
|
||||||
task_manager.complete_download(request.model_name)
|
except Exception as e:
|
||||||
|
task_manager.error_download(request.model_name, str(e))
|
||||||
return {"message": f"Model {request.model_name} download started"}
|
|
||||||
except Exception as e:
|
# Start tracking download
|
||||||
# Mark download as failed
|
task_manager.start_download(request.model_name)
|
||||||
task_manager.error_download(request.model_name, str(e))
|
|
||||||
raise HTTPException(status_code=500, detail=str(e))
|
# Start download in background task (don't await)
|
||||||
|
asyncio.create_task(download_in_background())
|
||||||
|
|
||||||
|
# Return immediately - frontend should poll progress endpoint
|
||||||
|
return {"message": f"Model {request.model_name} download started"}
|
||||||
|
|
||||||
|
|
||||||
@app.delete("/models/{model_name}")
|
@app.delete("/models/{model_name}")
|
||||||
@@ -1232,6 +1418,7 @@ async def delete_model(model_name: str):
|
|||||||
"""Delete a downloaded model from the HuggingFace cache."""
|
"""Delete a downloaded model from the HuggingFace cache."""
|
||||||
import shutil
|
import shutil
|
||||||
import os
|
import os
|
||||||
|
from huggingface_hub import constants as hf_constants
|
||||||
|
|
||||||
# Map model names to HuggingFace repo IDs
|
# Map model names to HuggingFace repo IDs
|
||||||
model_configs = {
|
model_configs = {
|
||||||
@@ -1284,8 +1471,8 @@ async def delete_model(model_name: str):
|
|||||||
if whisper_model.is_loaded() and whisper_model.model_size == config["model_size"]:
|
if whisper_model.is_loaded() and whisper_model.model_size == config["model_size"]:
|
||||||
transcribe.unload_whisper_model()
|
transcribe.unload_whisper_model()
|
||||||
|
|
||||||
# Find and delete the cache directory
|
# Find and delete the cache directory (using HuggingFace's OS-specific cache location)
|
||||||
cache_dir = os.path.expanduser("~/.cache/huggingface/hub")
|
cache_dir = hf_constants.HF_HUB_CACHE
|
||||||
repo_cache_dir = Path(cache_dir) / ("models--" + hf_repo_id.replace("/", "--"))
|
repo_cache_dir = Path(cache_dir) / ("models--" + hf_repo_id.replace("/", "--"))
|
||||||
|
|
||||||
# Check if the cache directory exists
|
# Check if the cache directory exists
|
||||||
@@ -1309,6 +1496,19 @@ async def delete_model(model_name: str):
|
|||||||
raise HTTPException(status_code=500, detail=f"Failed to delete model: {str(e)}")
|
raise HTTPException(status_code=500, detail=f"Failed to delete model: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/cache/clear")
|
||||||
|
async def clear_cache():
|
||||||
|
"""Clear all voice prompt caches (memory and disk)."""
|
||||||
|
try:
|
||||||
|
deleted_count = clear_voice_prompt_cache()
|
||||||
|
return {
|
||||||
|
"message": f"Voice prompt cache cleared successfully",
|
||||||
|
"files_deleted": deleted_count,
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
raise HTTPException(status_code=500, detail=f"Failed to clear cache: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
# ============================================
|
# ============================================
|
||||||
# TASK MANAGEMENT
|
# TASK MANAGEMENT
|
||||||
# ============================================
|
# ============================================
|
||||||
@@ -1381,10 +1581,13 @@ async def get_active_tasks():
|
|||||||
|
|
||||||
def _get_gpu_status() -> str:
|
def _get_gpu_status() -> str:
|
||||||
"""Get GPU availability status."""
|
"""Get GPU availability status."""
|
||||||
|
backend_type = get_backend_type()
|
||||||
if torch.cuda.is_available():
|
if torch.cuda.is_available():
|
||||||
return f"CUDA ({torch.cuda.get_device_name(0)})"
|
return f"CUDA ({torch.cuda.get_device_name(0)})"
|
||||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||||
return "MPS (Apple Silicon)"
|
return "MPS (Apple Silicon)"
|
||||||
|
elif backend_type == "mlx":
|
||||||
|
return "Metal (Apple Silicon via MLX)"
|
||||||
return "None (CPU only)"
|
return "None (CPU only)"
|
||||||
|
|
||||||
|
|
||||||
@@ -1394,8 +1597,28 @@ async def startup_event():
|
|||||||
print("voicebox API starting up...")
|
print("voicebox API starting up...")
|
||||||
database.init_db()
|
database.init_db()
|
||||||
print(f"Database initialized at {database._db_path}")
|
print(f"Database initialized at {database._db_path}")
|
||||||
|
backend_type = get_backend_type()
|
||||||
|
print(f"Backend: {backend_type.upper()}")
|
||||||
print(f"GPU available: {_get_gpu_status()}")
|
print(f"GPU available: {_get_gpu_status()}")
|
||||||
|
|
||||||
|
# Initialize progress manager with main event loop for thread-safe operations
|
||||||
|
try:
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
progress_manager._set_main_loop(asyncio.get_running_loop())
|
||||||
|
print("Progress manager initialized with event loop")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Warning: Could not initialize progress manager event loop: {e}")
|
||||||
|
|
||||||
|
# Ensure HuggingFace cache directory exists
|
||||||
|
try:
|
||||||
|
from huggingface_hub import constants as hf_constants
|
||||||
|
cache_dir = Path(hf_constants.HF_HUB_CACHE)
|
||||||
|
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
print(f"HuggingFace cache directory: {cache_dir}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Warning: Could not create HuggingFace cache directory: {e}")
|
||||||
|
print("Model downloads may fail. Please ensure the directory exists and has write permissions.")
|
||||||
|
|
||||||
|
|
||||||
@app.on_event("shutdown")
|
@app.on_event("shutdown")
|
||||||
async def shutdown_event():
|
async def shutdown_event():
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ class VoiceProfileResponse(BaseModel):
|
|||||||
name: str
|
name: str
|
||||||
description: Optional[str]
|
description: Optional[str]
|
||||||
language: str
|
language: str
|
||||||
|
avatar_path: Optional[str] = None
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
updated_at: datetime
|
updated_at: datetime
|
||||||
|
|
||||||
@@ -32,6 +33,11 @@ class ProfileSampleCreate(BaseModel):
|
|||||||
reference_text: str = Field(..., min_length=1, max_length=1000)
|
reference_text: str = Field(..., min_length=1, max_length=1000)
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileSampleUpdate(BaseModel):
|
||||||
|
"""Request model for updating a profile sample."""
|
||||||
|
reference_text: str = Field(..., min_length=1, max_length=1000)
|
||||||
|
|
||||||
|
|
||||||
class ProfileSampleResponse(BaseModel):
|
class ProfileSampleResponse(BaseModel):
|
||||||
"""Response model for profile sample."""
|
"""Response model for profile sample."""
|
||||||
id: str
|
id: str
|
||||||
@@ -118,7 +124,9 @@ class HealthResponse(BaseModel):
|
|||||||
model_downloaded: Optional[bool] = None # Whether model is cached/downloaded
|
model_downloaded: Optional[bool] = None # Whether model is cached/downloaded
|
||||||
model_size: Optional[str] = None # Current model size if loaded
|
model_size: Optional[str] = None # Current model size if loaded
|
||||||
gpu_available: bool
|
gpu_available: bool
|
||||||
|
gpu_type: Optional[str] = None # GPU type (CUDA, MPS, or None)
|
||||||
vram_used_mb: Optional[float] = None
|
vram_used_mb: Optional[float] = None
|
||||||
|
backend_type: Optional[str] = None # Backend type (mlx or pytorch)
|
||||||
|
|
||||||
|
|
||||||
class ModelStatus(BaseModel):
|
class ModelStatus(BaseModel):
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
"""
|
||||||
|
Platform detection for backend selection.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import platform
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
|
||||||
|
def is_apple_silicon() -> bool:
|
||||||
|
"""
|
||||||
|
Check if running on Apple Silicon (arm64 macOS).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if on Apple Silicon, False otherwise
|
||||||
|
"""
|
||||||
|
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||||
|
|
||||||
|
|
||||||
|
def get_backend_type() -> Literal["mlx", "pytorch"]:
|
||||||
|
"""
|
||||||
|
Detect the best backend for the current platform.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
"mlx" on Apple Silicon (if MLX is available), "pytorch" otherwise
|
||||||
|
"""
|
||||||
|
if is_apple_silicon():
|
||||||
|
try:
|
||||||
|
import mlx
|
||||||
|
return "mlx"
|
||||||
|
except ImportError:
|
||||||
|
# MLX not installed, fallback to PyTorch
|
||||||
|
return "pytorch"
|
||||||
|
return "pytorch"
|
||||||
+172
-22
@@ -21,6 +21,8 @@ from .database import (
|
|||||||
ProfileSample as DBProfileSample,
|
ProfileSample as DBProfileSample,
|
||||||
)
|
)
|
||||||
from .utils.audio import validate_reference_audio, load_audio, save_audio
|
from .utils.audio import validate_reference_audio, load_audio, save_audio
|
||||||
|
from .utils.images import validate_image, process_avatar
|
||||||
|
from .utils.cache import _get_cache_dir, clear_profile_cache
|
||||||
from .tts import get_tts_model
|
from .tts import get_tts_model
|
||||||
from . import config
|
from . import config
|
||||||
|
|
||||||
@@ -119,6 +121,10 @@ async def add_profile_sample(
|
|||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(db_sample)
|
db.refresh(db_sample)
|
||||||
|
|
||||||
|
# Invalidate combined audio cache for this profile
|
||||||
|
# Since a new sample was added, any cached combined audio is now stale
|
||||||
|
clear_profile_cache(profile_id)
|
||||||
|
|
||||||
return ProfileSampleResponse.model_validate(db_sample)
|
return ProfileSampleResponse.model_validate(db_sample)
|
||||||
|
|
||||||
|
|
||||||
@@ -240,6 +246,9 @@ async def delete_profile(
|
|||||||
if profile_dir.exists():
|
if profile_dir.exists():
|
||||||
shutil.rmtree(profile_dir)
|
shutil.rmtree(profile_dir)
|
||||||
|
|
||||||
|
# Clean up combined audio cache files for this profile
|
||||||
|
clear_profile_cache(profile_id)
|
||||||
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
@@ -261,6 +270,9 @@ async def delete_profile_sample(
|
|||||||
if not sample:
|
if not sample:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
# Store profile_id before deleting
|
||||||
|
profile_id = sample.profile_id
|
||||||
|
|
||||||
# Delete audio file
|
# Delete audio file
|
||||||
audio_path = Path(sample.audio_path)
|
audio_path = Path(sample.audio_path)
|
||||||
if audio_path.exists():
|
if audio_path.exists():
|
||||||
@@ -270,9 +282,47 @@ async def delete_profile_sample(
|
|||||||
db.delete(sample)
|
db.delete(sample)
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
# Invalidate combined audio cache for this profile
|
||||||
|
# Since the sample set changed, any cached combined audio is now stale
|
||||||
|
clear_profile_cache(profile_id)
|
||||||
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
async def update_profile_sample(
|
||||||
|
sample_id: str,
|
||||||
|
reference_text: str,
|
||||||
|
db: Session,
|
||||||
|
) -> Optional[ProfileSampleResponse]:
|
||||||
|
"""
|
||||||
|
Update a profile sample's reference text.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
sample_id: Sample ID
|
||||||
|
reference_text: Updated reference text
|
||||||
|
db: Database session
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Updated sample or None if not found
|
||||||
|
"""
|
||||||
|
sample = db.query(DBProfileSample).filter_by(id=sample_id).first()
|
||||||
|
if not sample:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Store profile_id before updating
|
||||||
|
profile_id = sample.profile_id
|
||||||
|
|
||||||
|
sample.reference_text = reference_text
|
||||||
|
db.commit()
|
||||||
|
db.refresh(sample)
|
||||||
|
|
||||||
|
# Invalidate combined audio cache for this profile
|
||||||
|
# Since the reference text changed, cache keys and combined text are now stale
|
||||||
|
clear_profile_cache(profile_id)
|
||||||
|
|
||||||
|
return ProfileSampleResponse.model_validate(sample)
|
||||||
|
|
||||||
|
|
||||||
async def create_voice_prompt_for_profile(
|
async def create_voice_prompt_for_profile(
|
||||||
profile_id: str,
|
profile_id: str,
|
||||||
db: Session,
|
db: Session,
|
||||||
@@ -280,23 +330,23 @@ async def create_voice_prompt_for_profile(
|
|||||||
) -> dict:
|
) -> dict:
|
||||||
"""
|
"""
|
||||||
Create a combined voice prompt from all samples in a profile.
|
Create a combined voice prompt from all samples in a profile.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
profile_id: Profile ID
|
profile_id: Profile ID
|
||||||
db: Database session
|
db: Database session
|
||||||
use_cache: Whether to use cached prompts
|
use_cache: Whether to use cached prompts
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Voice prompt dictionary
|
Voice prompt dictionary
|
||||||
"""
|
"""
|
||||||
# Get all samples for profile
|
# Get all samples for profile
|
||||||
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
|
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
|
||||||
|
|
||||||
if not samples:
|
if not samples:
|
||||||
raise ValueError(f"No samples found for profile {profile_id}")
|
raise ValueError(f"No samples found for profile {profile_id}")
|
||||||
|
|
||||||
tts_model = get_tts_model()
|
tts_model = get_tts_model()
|
||||||
|
|
||||||
if len(samples) == 1:
|
if len(samples) == 1:
|
||||||
# Single sample - use directly
|
# Single sample - use directly
|
||||||
sample = samples[0]
|
sample = samples[0]
|
||||||
@@ -310,27 +360,127 @@ async def create_voice_prompt_for_profile(
|
|||||||
# Multiple samples - combine them
|
# Multiple samples - combine them
|
||||||
audio_paths = [s.audio_path for s in samples]
|
audio_paths = [s.audio_path for s in samples]
|
||||||
reference_texts = [s.reference_text for s in samples]
|
reference_texts = [s.reference_text for s in samples]
|
||||||
|
|
||||||
# Combine audio
|
# Combine audio
|
||||||
combined_audio, combined_text = await tts_model.combine_voice_prompts(
|
combined_audio, combined_text = await tts_model.combine_voice_prompts(
|
||||||
audio_paths,
|
audio_paths,
|
||||||
reference_texts,
|
reference_texts,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Save combined audio to cache directory (persistent)
|
||||||
|
# Create a hash of sample IDs to identify this specific combination
|
||||||
|
import hashlib
|
||||||
|
sample_ids_str = "-".join(sorted([s.id for s in samples]))
|
||||||
|
combination_hash = hashlib.md5(sample_ids_str.encode()).hexdigest()[:12]
|
||||||
|
|
||||||
# Save combined audio temporarily
|
# Store in cache directory
|
||||||
import tempfile
|
cache_dir = _get_cache_dir()
|
||||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
|
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||||
save_audio(combined_audio, tmp.name, 24000)
|
combined_path = cache_dir / f"combined_{profile_id}_{combination_hash}.wav"
|
||||||
tmp_path = tmp.name
|
|
||||||
|
|
||||||
try:
|
# Save combined audio
|
||||||
# Create prompt from combined audio
|
save_audio(combined_audio, str(combined_path), 24000)
|
||||||
voice_prompt, _ = await tts_model.create_voice_prompt(
|
|
||||||
tmp_path,
|
# Create prompt from combined audio
|
||||||
combined_text,
|
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||||
use_cache=use_cache,
|
str(combined_path),
|
||||||
)
|
combined_text,
|
||||||
return voice_prompt
|
use_cache=use_cache,
|
||||||
finally:
|
)
|
||||||
# Clean up temp file
|
return voice_prompt
|
||||||
Path(tmp_path).unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
async def upload_avatar(
|
||||||
|
profile_id: str,
|
||||||
|
image_path: str,
|
||||||
|
db: Session,
|
||||||
|
) -> VoiceProfileResponse:
|
||||||
|
"""
|
||||||
|
Upload and process avatar image for a profile.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
profile_id: Profile ID
|
||||||
|
image_path: Path to uploaded image file
|
||||||
|
db: Database session
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Updated profile
|
||||||
|
"""
|
||||||
|
# Validate profile exists
|
||||||
|
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise ValueError(f"Profile {profile_id} not found")
|
||||||
|
|
||||||
|
# Validate image
|
||||||
|
is_valid, error_msg = validate_image(image_path)
|
||||||
|
if not is_valid:
|
||||||
|
raise ValueError(error_msg)
|
||||||
|
|
||||||
|
# Delete existing avatar if present
|
||||||
|
if profile.avatar_path:
|
||||||
|
old_avatar = Path(profile.avatar_path)
|
||||||
|
if old_avatar.exists():
|
||||||
|
old_avatar.unlink()
|
||||||
|
|
||||||
|
# Determine file extension from uploaded file
|
||||||
|
from PIL import Image
|
||||||
|
with Image.open(image_path) as img:
|
||||||
|
# Normalize JPEG variants (MPO is multi-picture format from some cameras)
|
||||||
|
img_format = img.format
|
||||||
|
if img_format in ('MPO', 'JPG'):
|
||||||
|
img_format = 'JPEG'
|
||||||
|
|
||||||
|
ext_map = {
|
||||||
|
'PNG': '.png',
|
||||||
|
'JPEG': '.jpg',
|
||||||
|
'WEBP': '.webp'
|
||||||
|
}
|
||||||
|
ext = ext_map.get(img_format, '.png')
|
||||||
|
|
||||||
|
# Save processed image to profile directory
|
||||||
|
profile_dir = _get_profiles_dir() / profile_id
|
||||||
|
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
output_path = profile_dir / f"avatar{ext}"
|
||||||
|
|
||||||
|
process_avatar(image_path, str(output_path))
|
||||||
|
|
||||||
|
# Update database
|
||||||
|
profile.avatar_path = str(output_path)
|
||||||
|
profile.updated_at = datetime.utcnow()
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
db.refresh(profile)
|
||||||
|
|
||||||
|
return VoiceProfileResponse.model_validate(profile)
|
||||||
|
|
||||||
|
|
||||||
|
async def delete_avatar(
|
||||||
|
profile_id: str,
|
||||||
|
db: Session,
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Delete avatar image for a profile.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
profile_id: Profile ID
|
||||||
|
db: Database session
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if deleted, False if not found or no avatar
|
||||||
|
"""
|
||||||
|
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||||
|
if not profile or not profile.avatar_path:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Delete avatar file
|
||||||
|
avatar_path = Path(profile.avatar_path)
|
||||||
|
if avatar_path.exists():
|
||||||
|
avatar_path.unlink()
|
||||||
|
|
||||||
|
# Update database
|
||||||
|
profile.avatar_path = None
|
||||||
|
profile.updated_at = datetime.utcnow()
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
return True
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# MLX-specific dependencies (Apple Silicon only)
|
||||||
|
# These should only be installed on aarch64-apple-darwin platforms
|
||||||
|
|
||||||
|
mlx>=0.30.0
|
||||||
|
mlx-audio>=0.3.1
|
||||||
@@ -21,3 +21,4 @@ numpy>=1.24.0
|
|||||||
|
|
||||||
# Utilities
|
# Utilities
|
||||||
python-multipart>=0.0.6
|
python-multipart>=0.0.6
|
||||||
|
Pillow>=10.0.0
|
||||||
|
|||||||
+12
-264
@@ -1,274 +1,22 @@
|
|||||||
"""
|
"""
|
||||||
Whisper ASR module for transcription.
|
STT (Speech-to-Text) module - delegates to backend abstraction layer.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Optional, List, Dict
|
from typing import Optional
|
||||||
import asyncio
|
from .backends import get_stt_backend, STTBackend
|
||||||
import torch
|
|
||||||
import numpy as np
|
|
||||||
from pathlib import Path
|
|
||||||
from .utils.progress import get_progress_manager
|
|
||||||
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
|
||||||
from .utils.tasks import get_task_manager
|
|
||||||
|
|
||||||
|
|
||||||
class WhisperModel:
|
def get_whisper_model() -> STTBackend:
|
||||||
"""Manages Whisper model loading and transcription."""
|
"""
|
||||||
|
Get STT backend instance (MLX or PyTorch based on platform).
|
||||||
|
|
||||||
def __init__(self, model_size: str = "base"):
|
Returns:
|
||||||
self.model = None
|
STT backend instance
|
||||||
self.processor = None
|
"""
|
||||||
self.model_size = model_size
|
return get_stt_backend()
|
||||||
self.device = self._get_device()
|
|
||||||
|
|
||||||
def _get_device(self) -> str:
|
|
||||||
"""Get the best available device."""
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
return "cuda"
|
|
||||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
|
||||||
# MPS support for Whisper
|
|
||||||
return "cpu" # Use CPU for stability
|
|
||||||
return "cpu"
|
|
||||||
|
|
||||||
def is_loaded(self) -> bool:
|
|
||||||
"""Check if model is loaded."""
|
|
||||||
return self.model is not None
|
|
||||||
|
|
||||||
def load_model(self, model_size: Optional[str] = None):
|
|
||||||
"""
|
|
||||||
Lazy load the Whisper model.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
model_size: Model size (tiny, base, small, medium, large)
|
|
||||||
"""
|
|
||||||
if model_size is None:
|
|
||||||
model_size = self.model_size
|
|
||||||
|
|
||||||
if self.model is not None and self.model_size == model_size:
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
|
||||||
|
|
||||||
model_name = f"openai/whisper-{model_size}"
|
|
||||||
|
|
||||||
# Set up progress tracking
|
|
||||||
progress_manager = get_progress_manager()
|
|
||||||
progress_model_name = f"whisper-{model_size}"
|
|
||||||
|
|
||||||
# Start tracking download task
|
|
||||||
task_manager = get_task_manager()
|
|
||||||
task_manager.start_download(progress_model_name)
|
|
||||||
|
|
||||||
print(f"Loading Whisper model {model_size} on {self.device}...")
|
|
||||||
|
|
||||||
# Initialize progress state to show download has started
|
|
||||||
progress_manager.update_progress(
|
|
||||||
model_name=progress_model_name,
|
|
||||||
current=0,
|
|
||||||
total=1, # Set to 1 initially, will be updated by callback
|
|
||||||
filename="",
|
|
||||||
status="downloading",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Set up progress callback
|
|
||||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
|
||||||
tracker = HFProgressTracker(progress_callback)
|
|
||||||
|
|
||||||
# Use progress tracker during download
|
|
||||||
with tracker.patch_download():
|
|
||||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
|
||||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
|
||||||
|
|
||||||
self.model.to(self.device)
|
|
||||||
self.model_size = model_size
|
|
||||||
|
|
||||||
# Mark as complete
|
|
||||||
progress_manager.mark_complete(progress_model_name)
|
|
||||||
task_manager.complete_download(progress_model_name)
|
|
||||||
|
|
||||||
print(f"Whisper model {model_size} loaded successfully")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error loading Whisper model: {e}")
|
|
||||||
progress_manager = get_progress_manager()
|
|
||||||
task_manager = get_task_manager()
|
|
||||||
progress_model_name = f"whisper-{model_size}"
|
|
||||||
progress_manager.mark_error(progress_model_name, str(e))
|
|
||||||
task_manager.error_download(progress_model_name, str(e))
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def load_model_async(self, model_size: Optional[str] = None):
|
|
||||||
"""
|
|
||||||
Async version of load_model that runs in thread pool.
|
|
||||||
|
|
||||||
This prevents blocking the event loop during model loading.
|
|
||||||
"""
|
|
||||||
if model_size is None:
|
|
||||||
model_size = self.model_size
|
|
||||||
|
|
||||||
# If already loaded with correct size, return immediately
|
|
||||||
if self.model is not None and self.model_size == model_size:
|
|
||||||
return
|
|
||||||
|
|
||||||
# Run the blocking load operation in a thread pool
|
|
||||||
await asyncio.to_thread(self.load_model, model_size)
|
|
||||||
|
|
||||||
def unload_model(self):
|
|
||||||
"""Unload the model to free memory."""
|
|
||||||
if self.model is not None:
|
|
||||||
del self.model
|
|
||||||
del self.processor
|
|
||||||
self.model = None
|
|
||||||
self.processor = None
|
|
||||||
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
torch.cuda.empty_cache()
|
|
||||||
|
|
||||||
print("Whisper model unloaded")
|
|
||||||
|
|
||||||
async def transcribe(
|
|
||||||
self,
|
|
||||||
audio_path: str,
|
|
||||||
language: Optional[str] = None,
|
|
||||||
) -> str:
|
|
||||||
"""
|
|
||||||
Transcribe audio to text.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio_path: Path to audio file
|
|
||||||
language: Optional language hint (en or zh)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Transcribed text
|
|
||||||
"""
|
|
||||||
await self.load_model_async()
|
|
||||||
|
|
||||||
from .utils.audio import load_audio
|
|
||||||
|
|
||||||
def _transcribe_sync():
|
|
||||||
"""Run synchronous transcription in thread pool."""
|
|
||||||
# Load audio
|
|
||||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
|
||||||
|
|
||||||
# Process audio
|
|
||||||
inputs = self.processor(
|
|
||||||
audio,
|
|
||||||
sampling_rate=16000,
|
|
||||||
return_tensors="pt",
|
|
||||||
)
|
|
||||||
inputs = inputs.to(self.device)
|
|
||||||
|
|
||||||
# Set language if provided
|
|
||||||
forced_decoder_ids = None
|
|
||||||
if language:
|
|
||||||
lang_code = "en" if language == "en" else "zh"
|
|
||||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
|
||||||
language=lang_code,
|
|
||||||
task="transcribe",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Generate transcription
|
|
||||||
with torch.no_grad():
|
|
||||||
predicted_ids = self.model.generate(
|
|
||||||
inputs["input_features"],
|
|
||||||
forced_decoder_ids=forced_decoder_ids,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Decode
|
|
||||||
transcription = self.processor.batch_decode(
|
|
||||||
predicted_ids,
|
|
||||||
skip_special_tokens=True,
|
|
||||||
)[0]
|
|
||||||
|
|
||||||
return transcription.strip()
|
|
||||||
|
|
||||||
# Run blocking transcription in thread pool
|
|
||||||
return await asyncio.to_thread(_transcribe_sync)
|
|
||||||
|
|
||||||
async def transcribe_with_timestamps(
|
|
||||||
self,
|
|
||||||
audio_path: str,
|
|
||||||
language: Optional[str] = None,
|
|
||||||
) -> List[Dict[str, any]]:
|
|
||||||
"""
|
|
||||||
Transcribe audio with word-level timestamps.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio_path: Path to audio file
|
|
||||||
language: Optional language hint
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of word segments with timestamps
|
|
||||||
"""
|
|
||||||
await self.load_model_async()
|
|
||||||
|
|
||||||
from .utils.audio import load_audio
|
|
||||||
|
|
||||||
def _transcribe_timestamps_sync():
|
|
||||||
"""Run synchronous transcription with timestamps in thread pool."""
|
|
||||||
# Load audio
|
|
||||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
|
||||||
|
|
||||||
# Process audio
|
|
||||||
inputs = self.processor(
|
|
||||||
audio,
|
|
||||||
sampling_rate=16000,
|
|
||||||
return_tensors="pt",
|
|
||||||
)
|
|
||||||
inputs = inputs.to(self.device)
|
|
||||||
|
|
||||||
# Set language if provided
|
|
||||||
forced_decoder_ids = None
|
|
||||||
if language:
|
|
||||||
lang_code = "en" if language == "en" else "zh"
|
|
||||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
|
||||||
language=lang_code,
|
|
||||||
task="transcribe",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Generate with timestamps
|
|
||||||
with torch.no_grad():
|
|
||||||
predicted_ids = self.model.generate(
|
|
||||||
inputs["input_features"],
|
|
||||||
forced_decoder_ids=forced_decoder_ids,
|
|
||||||
return_timestamps=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse timestamps (simplified - would need more robust parsing)
|
|
||||||
# For now, return basic transcription
|
|
||||||
# TODO: Implement proper timestamp parsing
|
|
||||||
transcription = self.processor.batch_decode(
|
|
||||||
predicted_ids,
|
|
||||||
skip_special_tokens=True,
|
|
||||||
)[0]
|
|
||||||
|
|
||||||
return [
|
|
||||||
{
|
|
||||||
"text": transcription,
|
|
||||||
"start": 0.0,
|
|
||||||
"end": len(audio) / sr,
|
|
||||||
}
|
|
||||||
]
|
|
||||||
|
|
||||||
# Run blocking transcription in thread pool
|
|
||||||
return await asyncio.to_thread(_transcribe_timestamps_sync)
|
|
||||||
|
|
||||||
|
|
||||||
# Global model instance
|
|
||||||
_whisper_model: Optional[WhisperModel] = None
|
|
||||||
|
|
||||||
|
|
||||||
def get_whisper_model() -> WhisperModel:
|
|
||||||
"""Get or create Whisper model instance."""
|
|
||||||
global _whisper_model
|
|
||||||
if _whisper_model is None:
|
|
||||||
_whisper_model = WhisperModel()
|
|
||||||
return _whisper_model
|
|
||||||
|
|
||||||
|
|
||||||
def unload_whisper_model():
|
def unload_whisper_model():
|
||||||
"""Unload Whisper model to free memory."""
|
"""Unload Whisper model to free memory."""
|
||||||
global _whisper_model
|
backend = get_stt_backend()
|
||||||
if _whisper_model is not None:
|
backend.unload_model()
|
||||||
_whisper_model.unload_model()
|
|
||||||
|
|||||||
+20
-355
@@ -1,372 +1,37 @@
|
|||||||
"""
|
"""
|
||||||
TTS inference module using Qwen3-TTS.
|
TTS inference module - delegates to backend abstraction layer.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Optional, List, Tuple
|
from typing import Optional
|
||||||
import asyncio
|
|
||||||
import torch
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import io
|
import io
|
||||||
import soundfile as sf
|
import soundfile as sf
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from .utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
from .backends import get_tts_backend, TTSBackend
|
||||||
from .utils.audio import normalize_audio
|
|
||||||
from .utils.progress import get_progress_manager
|
|
||||||
from .utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
|
||||||
from .utils.tasks import get_task_manager
|
|
||||||
from . import config
|
|
||||||
|
|
||||||
|
|
||||||
class TTSModel:
|
def get_tts_model() -> TTSBackend:
|
||||||
"""Manages Qwen3-TTS model loading and inference."""
|
"""
|
||||||
|
Get TTS backend instance (MLX or PyTorch based on platform).
|
||||||
|
|
||||||
def __init__(self, model_size: str = "1.7B"):
|
Returns:
|
||||||
self.model = None
|
TTS backend instance
|
||||||
self.model_size = model_size
|
"""
|
||||||
self.device = self._get_device()
|
return get_tts_backend()
|
||||||
self._current_model_size = None
|
|
||||||
|
|
||||||
def _get_device(self) -> str:
|
|
||||||
"""Get the best available device."""
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
return "cuda"
|
|
||||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
|
||||||
# MPS can have issues, use CPU for stability
|
|
||||||
return "cpu"
|
|
||||||
return "cpu"
|
|
||||||
|
|
||||||
def is_loaded(self) -> bool:
|
|
||||||
"""Check if model is loaded."""
|
|
||||||
return self.model is not None
|
|
||||||
|
|
||||||
def _get_model_path(self, model_size: str) -> str:
|
|
||||||
"""
|
|
||||||
Get the model path, downloading from HuggingFace Hub if needed.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
model_size: Model size (1.7B or 0.6B)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Path to model (either local or HuggingFace Hub ID)
|
|
||||||
"""
|
|
||||||
# HuggingFace Hub model IDs
|
|
||||||
hf_model_map = {
|
|
||||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
|
||||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
|
||||||
}
|
|
||||||
|
|
||||||
# Local directory names (for backwards compatibility)
|
|
||||||
local_model_map = {
|
|
||||||
"1.7B": "Qwen--Qwen3-TTS-12Hz-1.7B-Base",
|
|
||||||
"0.6B": "Qwen--Qwen3-TTS-12Hz-0.6B-Base",
|
|
||||||
}
|
|
||||||
|
|
||||||
if model_size not in hf_model_map:
|
|
||||||
raise ValueError(f"Unknown model size: {model_size}")
|
|
||||||
|
|
||||||
# Check if model exists locally (backwards compatibility)
|
|
||||||
local_path = config.get_models_dir() / local_model_map[model_size]
|
|
||||||
if local_path.exists():
|
|
||||||
print(f"Found local model at {local_path}")
|
|
||||||
return str(local_path)
|
|
||||||
|
|
||||||
# Use HuggingFace Hub model ID (will auto-download)
|
|
||||||
hf_model_id = hf_model_map[model_size]
|
|
||||||
print(f"Will download model from HuggingFace Hub: {hf_model_id}")
|
|
||||||
|
|
||||||
return hf_model_id
|
|
||||||
|
|
||||||
def load_model(self, model_size: Optional[str] = None):
|
|
||||||
"""
|
|
||||||
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
|
|
||||||
|
|
||||||
The model will be automatically downloaded on first use and cached locally.
|
|
||||||
This works similar to how Whisper models are loaded.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
model_size: Model size to load (1.7B or 0.6B)
|
|
||||||
"""
|
|
||||||
if model_size is None:
|
|
||||||
model_size = self.model_size
|
|
||||||
|
|
||||||
# If already loaded with correct size, return
|
|
||||||
if self.model is not None and self._current_model_size == model_size:
|
|
||||||
return
|
|
||||||
|
|
||||||
# Unload existing model if different size requested
|
|
||||||
if self.model is not None and self._current_model_size != model_size:
|
|
||||||
self.unload_model()
|
|
||||||
|
|
||||||
try:
|
|
||||||
from qwen_tts import Qwen3TTSModel
|
|
||||||
|
|
||||||
# Get model path (local or HuggingFace Hub ID)
|
|
||||||
model_path = self._get_model_path(model_size)
|
|
||||||
|
|
||||||
# Set up progress tracking
|
|
||||||
progress_manager = get_progress_manager()
|
|
||||||
model_name = f"qwen-tts-{model_size}"
|
|
||||||
|
|
||||||
# Check if model is being downloaded from HuggingFace Hub
|
|
||||||
if model_path.startswith("Qwen/"):
|
|
||||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
|
||||||
|
|
||||||
# Start tracking download task
|
|
||||||
task_manager = get_task_manager()
|
|
||||||
task_manager.start_download(model_name)
|
|
||||||
|
|
||||||
# Initialize progress state to show download has started
|
|
||||||
progress_manager.update_progress(
|
|
||||||
model_name=model_name,
|
|
||||||
current=0,
|
|
||||||
total=1, # Set to 1 initially, will be updated by callback
|
|
||||||
filename="",
|
|
||||||
status="downloading",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Set up progress callback
|
|
||||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
|
||||||
tracker = HFProgressTracker(progress_callback)
|
|
||||||
|
|
||||||
# Use progress tracker during download
|
|
||||||
with tracker.patch_download():
|
|
||||||
# Load the model - downloads will happen automatically with progress tracking
|
|
||||||
self.model = Qwen3TTSModel.from_pretrained(
|
|
||||||
model_path,
|
|
||||||
device_map=self.device,
|
|
||||||
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Mark as complete
|
|
||||||
progress_manager.mark_complete(model_name)
|
|
||||||
task_manager.complete_download(model_name)
|
|
||||||
else:
|
|
||||||
# Local model, no download needed
|
|
||||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
|
||||||
self.model = Qwen3TTSModel.from_pretrained(
|
|
||||||
model_path,
|
|
||||||
device_map=self.device,
|
|
||||||
torch_dtype=torch.float32 if self.device == "cpu" else torch.bfloat16,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._current_model_size = model_size
|
|
||||||
self.model_size = model_size
|
|
||||||
|
|
||||||
print(f"TTS model {model_size} loaded successfully")
|
|
||||||
|
|
||||||
except ImportError as e:
|
|
||||||
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
|
|
||||||
progress_manager = get_progress_manager()
|
|
||||||
task_manager = get_task_manager()
|
|
||||||
model_name = f"qwen-tts-{model_size}"
|
|
||||||
progress_manager.mark_error(model_name, str(e))
|
|
||||||
task_manager.error_download(model_name, str(e))
|
|
||||||
raise
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error loading TTS model: {e}")
|
|
||||||
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
|
|
||||||
progress_manager = get_progress_manager()
|
|
||||||
task_manager = get_task_manager()
|
|
||||||
model_name = f"qwen-tts-{model_size}"
|
|
||||||
progress_manager.mark_error(model_name, str(e))
|
|
||||||
task_manager.error_download(model_name, str(e))
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def load_model_async(self, model_size: Optional[str] = None):
|
|
||||||
"""
|
|
||||||
Async version of load_model that runs in thread pool.
|
|
||||||
|
|
||||||
This prevents blocking the event loop during model loading.
|
|
||||||
"""
|
|
||||||
if model_size is None:
|
|
||||||
model_size = self.model_size
|
|
||||||
|
|
||||||
# If already loaded with correct size, return immediately
|
|
||||||
if self.model is not None and self._current_model_size == model_size:
|
|
||||||
return
|
|
||||||
|
|
||||||
# Run the blocking load operation in a thread pool
|
|
||||||
await asyncio.to_thread(self.load_model, model_size)
|
|
||||||
|
|
||||||
def unload_model(self):
|
|
||||||
"""Unload the model to free memory."""
|
|
||||||
if self.model is not None:
|
|
||||||
del self.model
|
|
||||||
self.model = None
|
|
||||||
self._current_model_size = None
|
|
||||||
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
torch.cuda.empty_cache()
|
|
||||||
|
|
||||||
print("TTS model unloaded")
|
|
||||||
|
|
||||||
async def create_voice_prompt(
|
|
||||||
self,
|
|
||||||
audio_path: str,
|
|
||||||
reference_text: str,
|
|
||||||
use_cache: bool = True,
|
|
||||||
) -> Tuple[dict, bool]:
|
|
||||||
"""
|
|
||||||
Create voice prompt from reference audio.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio_path: Path to reference audio file
|
|
||||||
reference_text: Transcript of reference audio
|
|
||||||
use_cache: Whether to use cached prompt if available
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Tuple of (voice_prompt_dict, was_cached)
|
|
||||||
"""
|
|
||||||
await self.load_model_async()
|
|
||||||
|
|
||||||
# Check cache if enabled
|
|
||||||
if use_cache:
|
|
||||||
cache_key = get_cache_key(audio_path, reference_text)
|
|
||||||
cached_prompt = get_cached_voice_prompt(cache_key)
|
|
||||||
if cached_prompt is not None:
|
|
||||||
return cached_prompt, True
|
|
||||||
|
|
||||||
def _create_prompt_sync():
|
|
||||||
"""Run synchronous voice prompt creation in thread pool."""
|
|
||||||
return self.model.create_voice_clone_prompt(
|
|
||||||
ref_audio=str(audio_path),
|
|
||||||
ref_text=reference_text,
|
|
||||||
x_vector_only_mode=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Run blocking operation in thread pool
|
|
||||||
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
|
|
||||||
|
|
||||||
# Cache if enabled
|
|
||||||
if use_cache:
|
|
||||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
|
||||||
|
|
||||||
return voice_prompt_items, False
|
|
||||||
|
|
||||||
async def combine_voice_prompts(
|
|
||||||
self,
|
|
||||||
audio_paths: List[str],
|
|
||||||
reference_texts: List[str],
|
|
||||||
) -> Tuple[np.ndarray, str]:
|
|
||||||
"""
|
|
||||||
Combine multiple reference samples for better quality.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio_paths: List of audio file paths
|
|
||||||
reference_texts: List of reference texts
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Tuple of (combined_audio, combined_text)
|
|
||||||
"""
|
|
||||||
from .utils.audio import load_audio
|
|
||||||
|
|
||||||
combined_audio = []
|
|
||||||
|
|
||||||
for audio_path in audio_paths:
|
|
||||||
audio, sr = load_audio(audio_path)
|
|
||||||
audio = normalize_audio(audio)
|
|
||||||
combined_audio.append(audio)
|
|
||||||
|
|
||||||
# Concatenate audio
|
|
||||||
mixed = np.concatenate(combined_audio)
|
|
||||||
mixed = normalize_audio(mixed)
|
|
||||||
|
|
||||||
# Combine texts
|
|
||||||
combined_text = " ".join(reference_texts)
|
|
||||||
|
|
||||||
return mixed, combined_text
|
|
||||||
|
|
||||||
async def generate(
|
|
||||||
self,
|
|
||||||
text: str,
|
|
||||||
voice_prompt: dict,
|
|
||||||
language: str = "en",
|
|
||||||
seed: Optional[int] = None,
|
|
||||||
instruct: Optional[str] = None,
|
|
||||||
) -> Tuple[np.ndarray, int]:
|
|
||||||
"""
|
|
||||||
Generate audio from text using voice prompt.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
text: Text to synthesize
|
|
||||||
voice_prompt: Voice prompt dictionary from create_voice_prompt
|
|
||||||
language: Language code (en or zh)
|
|
||||||
seed: Random seed for reproducibility
|
|
||||||
instruct: Natural language instruction for speech delivery control
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Tuple of (audio_array, sample_rate)
|
|
||||||
"""
|
|
||||||
# Load model (already handles async via to_thread if needed)
|
|
||||||
await self.load_model_async()
|
|
||||||
|
|
||||||
def _generate_sync():
|
|
||||||
"""Run synchronous generation in thread pool."""
|
|
||||||
# Set seed if provided
|
|
||||||
if seed is not None:
|
|
||||||
torch.manual_seed(seed)
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
torch.cuda.manual_seed(seed)
|
|
||||||
|
|
||||||
# Generate audio - this is the blocking operation
|
|
||||||
wavs, sample_rate = self.model.generate_voice_clone(
|
|
||||||
text=text,
|
|
||||||
voice_clone_prompt=voice_prompt,
|
|
||||||
instruct=instruct,
|
|
||||||
)
|
|
||||||
return wavs[0], sample_rate
|
|
||||||
|
|
||||||
# Run blocking inference in thread pool to avoid blocking event loop
|
|
||||||
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
|
||||||
|
|
||||||
return audio, sample_rate
|
|
||||||
|
|
||||||
async def generate_from_reference(
|
|
||||||
self,
|
|
||||||
text: str,
|
|
||||||
audio_path: str,
|
|
||||||
reference_text: str,
|
|
||||||
language: str = "en",
|
|
||||||
seed: Optional[int] = None,
|
|
||||||
) -> Tuple[np.ndarray, int]:
|
|
||||||
"""
|
|
||||||
Generate audio directly from reference (convenience method).
|
|
||||||
|
|
||||||
Args:
|
|
||||||
text: Text to synthesize
|
|
||||||
audio_path: Path to reference audio
|
|
||||||
reference_text: Transcript of reference audio
|
|
||||||
language: Language code
|
|
||||||
seed: Random seed
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Tuple of (audio_array, sample_rate)
|
|
||||||
"""
|
|
||||||
# Create voice prompt (with caching)
|
|
||||||
voice_prompt, _ = await self.create_voice_prompt(audio_path, reference_text)
|
|
||||||
|
|
||||||
# Generate
|
|
||||||
return await self.generate(text, voice_prompt, language, seed)
|
|
||||||
|
|
||||||
|
|
||||||
# Global model instance
|
|
||||||
_tts_model: Optional[TTSModel] = None
|
|
||||||
|
|
||||||
|
|
||||||
def get_tts_model() -> TTSModel:
|
|
||||||
"""Get or create TTS model instance."""
|
|
||||||
global _tts_model
|
|
||||||
if _tts_model is None:
|
|
||||||
_tts_model = TTSModel()
|
|
||||||
return _tts_model
|
|
||||||
|
|
||||||
|
|
||||||
def unload_tts_model():
|
def unload_tts_model():
|
||||||
"""Unload TTS model to free memory."""
|
"""Unload TTS model to free memory."""
|
||||||
global _tts_model
|
backend = get_tts_backend()
|
||||||
if _tts_model is not None:
|
backend.unload_model()
|
||||||
_tts_model.unload_model()
|
|
||||||
|
|
||||||
|
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
|
||||||
|
"""Convert audio array to WAV bytes."""
|
||||||
|
buffer = io.BytesIO()
|
||||||
|
sf.write(buffer, audio, sample_rate, format="WAV")
|
||||||
|
buffer.seek(0)
|
||||||
|
return buffer.read()
|
||||||
|
|
||||||
|
|
||||||
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
|
def audio_to_wav_bytes(audio: np.ndarray, sample_rate: int) -> bytes:
|
||||||
|
|||||||
+68
-8
@@ -5,7 +5,7 @@ Voice prompt caching utilities.
|
|||||||
import hashlib
|
import hashlib
|
||||||
import torch
|
import torch
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional
|
from typing import Optional, Union, Dict, Any
|
||||||
|
|
||||||
from .. import config
|
from .. import config
|
||||||
|
|
||||||
@@ -15,8 +15,8 @@ def _get_cache_dir() -> Path:
|
|||||||
return config.get_cache_dir()
|
return config.get_cache_dir()
|
||||||
|
|
||||||
|
|
||||||
# In-memory cache
|
# In-memory cache - can store dict (voice prompt) or tensor (legacy)
|
||||||
_memory_cache: dict[str, torch.Tensor] = {}
|
_memory_cache: dict[str, Union[torch.Tensor, Dict[str, Any]]] = {}
|
||||||
|
|
||||||
|
|
||||||
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||||
@@ -43,7 +43,7 @@ def get_cache_key(audio_path: str, reference_text: str) -> str:
|
|||||||
|
|
||||||
def get_cached_voice_prompt(
|
def get_cached_voice_prompt(
|
||||||
cache_key: str,
|
cache_key: str,
|
||||||
) -> Optional[torch.Tensor]:
|
) -> Optional[Union[torch.Tensor, Dict[str, Any]]]:
|
||||||
"""
|
"""
|
||||||
Get cached voice prompt if available.
|
Get cached voice prompt if available.
|
||||||
|
|
||||||
@@ -51,7 +51,7 @@ def get_cached_voice_prompt(
|
|||||||
cache_key: Cache key
|
cache_key: Cache key
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Cached voice prompt tensor or None
|
Cached voice prompt (dict or tensor) or None
|
||||||
"""
|
"""
|
||||||
# Check in-memory cache
|
# Check in-memory cache
|
||||||
if cache_key in _memory_cache:
|
if cache_key in _memory_cache:
|
||||||
@@ -73,18 +73,78 @@ def get_cached_voice_prompt(
|
|||||||
|
|
||||||
def cache_voice_prompt(
|
def cache_voice_prompt(
|
||||||
cache_key: str,
|
cache_key: str,
|
||||||
voice_prompt: torch.Tensor,
|
voice_prompt: Union[torch.Tensor, Dict[str, Any]],
|
||||||
) -> None:
|
) -> None:
|
||||||
"""
|
"""
|
||||||
Cache voice prompt to memory and disk.
|
Cache voice prompt to memory and disk.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
cache_key: Cache key
|
cache_key: Cache key
|
||||||
voice_prompt: Voice prompt tensor
|
voice_prompt: Voice prompt (dict or tensor)
|
||||||
"""
|
"""
|
||||||
# Store in memory
|
# Store in memory
|
||||||
_memory_cache[cache_key] = voice_prompt
|
_memory_cache[cache_key] = voice_prompt
|
||||||
|
|
||||||
# Store on disk
|
# Store on disk (torch.save can handle both dicts and tensors)
|
||||||
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
|
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
|
||||||
torch.save(voice_prompt, cache_file)
|
torch.save(voice_prompt, cache_file)
|
||||||
|
|
||||||
|
|
||||||
|
def clear_voice_prompt_cache() -> int:
|
||||||
|
"""
|
||||||
|
Clear all voice prompt caches (memory and disk).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Number of cache files deleted
|
||||||
|
"""
|
||||||
|
# Clear memory cache
|
||||||
|
_memory_cache.clear()
|
||||||
|
|
||||||
|
# Clear disk cache
|
||||||
|
cache_dir = _get_cache_dir()
|
||||||
|
deleted_count = 0
|
||||||
|
|
||||||
|
if cache_dir.exists():
|
||||||
|
# Delete prompt cache files
|
||||||
|
for cache_file in cache_dir.glob("*.prompt"):
|
||||||
|
try:
|
||||||
|
cache_file.unlink()
|
||||||
|
deleted_count += 1
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Failed to delete cache file {cache_file}: {e}")
|
||||||
|
|
||||||
|
# Delete combined audio files
|
||||||
|
for audio_file in cache_dir.glob("combined_*.wav"):
|
||||||
|
try:
|
||||||
|
audio_file.unlink()
|
||||||
|
deleted_count += 1
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Failed to delete combined audio file {audio_file}: {e}")
|
||||||
|
|
||||||
|
return deleted_count
|
||||||
|
|
||||||
|
|
||||||
|
def clear_profile_cache(profile_id: str) -> int:
|
||||||
|
"""
|
||||||
|
Clear cache files for a specific profile.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
profile_id: Profile ID
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Number of cache files deleted
|
||||||
|
"""
|
||||||
|
cache_dir = _get_cache_dir()
|
||||||
|
deleted_count = 0
|
||||||
|
|
||||||
|
if cache_dir.exists():
|
||||||
|
# Delete combined audio files for this profile
|
||||||
|
pattern = f"combined_{profile_id}_*.wav"
|
||||||
|
for audio_file in cache_dir.glob(pattern):
|
||||||
|
try:
|
||||||
|
audio_file.unlink()
|
||||||
|
deleted_count += 1
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Failed to delete combined audio file {audio_file}: {e}")
|
||||||
|
|
||||||
|
return deleted_count
|
||||||
|
|||||||
@@ -29,8 +29,9 @@ class HFProgressTracker:
|
|||||||
|
|
||||||
class TrackedTqdm(original_tqdm):
|
class TrackedTqdm(original_tqdm):
|
||||||
"""A tqdm subclass that reports progress to our tracker."""
|
"""A tqdm subclass that reports progress to our tracker."""
|
||||||
|
|
||||||
def __init__(self, *args, **kwargs):
|
def __init__(self, *args, **kwargs):
|
||||||
|
print(f"[DEBUG TrackedTqdm] __init__ called with desc: {kwargs.get('desc', '')}")
|
||||||
# Extract filename from desc before passing to parent
|
# Extract filename from desc before passing to parent
|
||||||
desc = kwargs.get("desc", "")
|
desc = kwargs.get("desc", "")
|
||||||
if not desc and args:
|
if not desc and args:
|
||||||
@@ -79,8 +80,9 @@ class HFProgressTracker:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def update(self, n=1):
|
def update(self, n=1):
|
||||||
|
print(f"[DEBUG TrackedTqdm] update called with n={n}")
|
||||||
result = super().update(n)
|
result = super().update(n)
|
||||||
|
|
||||||
# Report progress
|
# Report progress
|
||||||
with tracker._lock:
|
with tracker._lock:
|
||||||
if id(self) in tracker._active_tqdms:
|
if id(self) in tracker._active_tqdms:
|
||||||
@@ -118,11 +120,13 @@ class HFProgressTracker:
|
|||||||
@contextmanager
|
@contextmanager
|
||||||
def patch_download(self):
|
def patch_download(self):
|
||||||
"""Context manager to patch tqdm for progress tracking."""
|
"""Context manager to patch tqdm for progress tracking."""
|
||||||
|
print("[DEBUG HFProgressTracker] patch_download called")
|
||||||
try:
|
try:
|
||||||
import tqdm as tqdm_module
|
import tqdm as tqdm_module
|
||||||
|
|
||||||
# Store original tqdm class
|
# Store original tqdm class
|
||||||
self._original_tqdm_class = tqdm_module.tqdm
|
self._original_tqdm_class = tqdm_module.tqdm
|
||||||
|
print(f"[DEBUG HFProgressTracker] Original tqdm class: {self._original_tqdm_class}")
|
||||||
|
|
||||||
# Reset totals
|
# Reset totals
|
||||||
with self._lock:
|
with self._lock:
|
||||||
@@ -135,18 +139,22 @@ class HFProgressTracker:
|
|||||||
|
|
||||||
# Create our tracked tqdm class
|
# Create our tracked tqdm class
|
||||||
tracked_tqdm = self._create_tracked_tqdm_class()
|
tracked_tqdm = self._create_tracked_tqdm_class()
|
||||||
|
print(f"[DEBUG HFProgressTracker] Created TrackedTqdm class: {tracked_tqdm}")
|
||||||
|
|
||||||
# Patch tqdm.tqdm
|
# Patch tqdm.tqdm
|
||||||
tqdm_module.tqdm = tracked_tqdm
|
tqdm_module.tqdm = tracked_tqdm
|
||||||
|
print(f"[DEBUG HFProgressTracker] Patched tqdm.tqdm")
|
||||||
|
|
||||||
# Also patch tqdm.auto.tqdm if it exists (used by huggingface_hub)
|
# Also patch tqdm.auto.tqdm if it exists (used by huggingface_hub)
|
||||||
self._original_tqdm_auto = None
|
self._original_tqdm_auto = None
|
||||||
if hasattr(tqdm_module, "auto") and hasattr(tqdm_module.auto, "tqdm"):
|
if hasattr(tqdm_module, "auto") and hasattr(tqdm_module.auto, "tqdm"):
|
||||||
self._original_tqdm_auto = tqdm_module.auto.tqdm
|
self._original_tqdm_auto = tqdm_module.auto.tqdm
|
||||||
tqdm_module.auto.tqdm = tracked_tqdm
|
tqdm_module.auto.tqdm = tracked_tqdm
|
||||||
|
print(f"[DEBUG HFProgressTracker] Patched tqdm.auto.tqdm")
|
||||||
|
|
||||||
# Patch in sys.modules to catch already-imported references
|
# Patch in sys.modules to catch already-imported references
|
||||||
self._patched_modules = {}
|
self._patched_modules = {}
|
||||||
|
patched_count = 0
|
||||||
for module_name in list(sys.modules.keys()):
|
for module_name in list(sys.modules.keys()):
|
||||||
if "huggingface" in module_name or module_name.startswith("tqdm"):
|
if "huggingface" in module_name or module_name.startswith("tqdm"):
|
||||||
try:
|
try:
|
||||||
@@ -159,8 +167,11 @@ class HFProgressTracker:
|
|||||||
):
|
):
|
||||||
self._patched_modules[module_name] = attr
|
self._patched_modules[module_name] = attr
|
||||||
setattr(module, "tqdm", tracked_tqdm)
|
setattr(module, "tqdm", tracked_tqdm)
|
||||||
|
patched_count += 1
|
||||||
|
print(f"[DEBUG HFProgressTracker] Patched {module_name}.tqdm")
|
||||||
except (AttributeError, TypeError):
|
except (AttributeError, TypeError):
|
||||||
pass
|
pass
|
||||||
|
print(f"[DEBUG HFProgressTracker] Patched {patched_count} modules in sys.modules")
|
||||||
|
|
||||||
yield
|
yield
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,114 @@
|
|||||||
|
"""Image processing utilities for avatar uploads."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional, Tuple
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
# JPEG can be reported as 'JPEG' or 'MPO' (for multi-picture format from some cameras)
|
||||||
|
ALLOWED_FORMATS = {'PNG', 'JPEG', 'WEBP', 'MPO', 'JPG'}
|
||||||
|
MAX_SIZE = 512
|
||||||
|
MAX_FILE_SIZE = 5 * 1024 * 1024 # 5MB
|
||||||
|
|
||||||
|
|
||||||
|
def validate_image(file_path: str) -> Tuple[bool, Optional[str]]:
|
||||||
|
"""
|
||||||
|
Validate image format and file size.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
file_path: Path to image file
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (is_valid, error_message)
|
||||||
|
"""
|
||||||
|
path = Path(file_path)
|
||||||
|
|
||||||
|
# Check file size
|
||||||
|
if path.stat().st_size > MAX_FILE_SIZE:
|
||||||
|
return False, f"File size exceeds maximum of {MAX_FILE_SIZE // (1024 * 1024)}MB"
|
||||||
|
|
||||||
|
try:
|
||||||
|
with Image.open(file_path) as img:
|
||||||
|
# Verify the image can be loaded
|
||||||
|
img.load()
|
||||||
|
|
||||||
|
# Check format (normalize JPEG variants)
|
||||||
|
img_format = img.format
|
||||||
|
if img_format in ('MPO', 'JPG'):
|
||||||
|
img_format = 'JPEG'
|
||||||
|
|
||||||
|
if img_format not in {'PNG', 'JPEG', 'WEBP'}:
|
||||||
|
return False, f"Invalid format '{img_format}'. Allowed formats: PNG, JPEG, WEBP"
|
||||||
|
|
||||||
|
return True, None
|
||||||
|
except Exception as e:
|
||||||
|
return False, f"Invalid image file: {str(e)}"
|
||||||
|
|
||||||
|
|
||||||
|
def process_avatar(input_path: str, output_path: str, max_size: int = MAX_SIZE) -> None:
|
||||||
|
"""
|
||||||
|
Process avatar image: resize and optimize.
|
||||||
|
|
||||||
|
Resizes image to fit within max_size x max_size while maintaining aspect ratio.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
input_path: Path to input image
|
||||||
|
output_path: Path to save processed image
|
||||||
|
max_size: Maximum width or height in pixels
|
||||||
|
"""
|
||||||
|
with Image.open(input_path) as img:
|
||||||
|
# Handle EXIF orientation for JPEG images
|
||||||
|
try:
|
||||||
|
from PIL import ExifTags
|
||||||
|
for orientation in ExifTags.TAGS.keys():
|
||||||
|
if ExifTags.TAGS[orientation] == 'Orientation':
|
||||||
|
break
|
||||||
|
exif = img._getexif()
|
||||||
|
if exif is not None:
|
||||||
|
orientation_value = exif.get(orientation)
|
||||||
|
if orientation_value == 3:
|
||||||
|
img = img.rotate(180, expand=True)
|
||||||
|
elif orientation_value == 6:
|
||||||
|
img = img.rotate(270, expand=True)
|
||||||
|
elif orientation_value == 8:
|
||||||
|
img = img.rotate(90, expand=True)
|
||||||
|
except (AttributeError, KeyError, IndexError, TypeError):
|
||||||
|
# No EXIF data or orientation tag
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Convert to RGB if necessary (handles RGBA, P, CMYK, etc.)
|
||||||
|
if img.mode not in ('RGB', 'L'):
|
||||||
|
if img.mode == 'RGBA':
|
||||||
|
# Create white background for RGBA images
|
||||||
|
background = Image.new('RGB', img.size, (255, 255, 255))
|
||||||
|
background.paste(img, mask=img.split()[3]) # Use alpha channel as mask
|
||||||
|
img = background
|
||||||
|
elif img.mode == 'CMYK':
|
||||||
|
# Convert CMYK to RGB
|
||||||
|
img = img.convert('RGB')
|
||||||
|
elif img.mode == 'P':
|
||||||
|
# Convert palette mode to RGB
|
||||||
|
img = img.convert('RGB')
|
||||||
|
else:
|
||||||
|
img = img.convert('RGB')
|
||||||
|
|
||||||
|
# Calculate new size maintaining aspect ratio
|
||||||
|
img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
|
||||||
|
|
||||||
|
# Determine output format from extension
|
||||||
|
output_ext = Path(output_path).suffix.lower()
|
||||||
|
|
||||||
|
format_map = {
|
||||||
|
'.png': 'PNG',
|
||||||
|
'.jpeg': 'JPEG',
|
||||||
|
'.jpg': 'JPEG',
|
||||||
|
'.webp': 'WEBP'
|
||||||
|
}
|
||||||
|
|
||||||
|
output_format = format_map.get(output_ext, 'PNG')
|
||||||
|
|
||||||
|
# Save with optimization
|
||||||
|
save_kwargs = {'optimize': True}
|
||||||
|
if output_format == 'JPEG':
|
||||||
|
save_kwargs['quality'] = 90
|
||||||
|
|
||||||
|
img.save(output_path, format=output_format, **save_kwargs)
|
||||||
+157
-48
@@ -6,16 +6,55 @@ from typing import Optional, Callable, Dict, List
|
|||||||
from fastapi.responses import StreamingResponse
|
from fastapi.responses import StreamingResponse
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
|
import threading
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
|
|
||||||
class ProgressManager:
|
class ProgressManager:
|
||||||
"""Manages download progress for multiple models."""
|
"""Manages download progress for multiple models.
|
||||||
|
|
||||||
|
Thread-safe: can be called from background threads (e.g., via asyncio.to_thread).
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self._progress: Dict[str, Dict] = {}
|
self._progress: Dict[str, Dict] = {}
|
||||||
self._listeners: Dict[str, list] = {}
|
self._listeners: Dict[str, list] = {}
|
||||||
|
self._lock = threading.Lock() # Thread-safe lock for progress dict
|
||||||
|
self._main_loop: Optional[asyncio.AbstractEventLoop] = None
|
||||||
|
|
||||||
|
def _set_main_loop(self, loop: asyncio.AbstractEventLoop):
|
||||||
|
"""Set the main event loop for thread-safe operations."""
|
||||||
|
self._main_loop = loop
|
||||||
|
|
||||||
|
def _notify_listeners_threadsafe(self, model_name: str, progress_data: Dict):
|
||||||
|
"""Notify listeners in a thread-safe manner."""
|
||||||
|
import logging
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
if model_name not in self._listeners:
|
||||||
|
return
|
||||||
|
|
||||||
|
for queue in self._listeners[model_name]:
|
||||||
|
try:
|
||||||
|
# Check if we're in the main event loop thread
|
||||||
|
try:
|
||||||
|
running_loop = asyncio.get_running_loop()
|
||||||
|
# We're in an async context, can use put_nowait directly
|
||||||
|
queue.put_nowait(progress_data.copy())
|
||||||
|
except RuntimeError:
|
||||||
|
# Not in async context (running in background thread)
|
||||||
|
# Use call_soon_threadsafe to safely put on queue
|
||||||
|
if self._main_loop and self._main_loop.is_running():
|
||||||
|
self._main_loop.call_soon_threadsafe(
|
||||||
|
lambda q=queue, d=progress_data.copy(): q.put_nowait(d) if not q.full() else None
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.debug(f"No main loop available for {model_name}, skipping notification")
|
||||||
|
except asyncio.QueueFull:
|
||||||
|
logger.warning(f"Queue full for {model_name}, dropping update")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error notifying listener for {model_name}: {e}")
|
||||||
|
|
||||||
def update_progress(
|
def update_progress(
|
||||||
self,
|
self,
|
||||||
model_name: str,
|
model_name: str,
|
||||||
@@ -26,7 +65,9 @@ class ProgressManager:
|
|||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
Update progress for a model download.
|
Update progress for a model download.
|
||||||
|
|
||||||
|
Thread-safe: can be called from background threads.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
model_name: Name of the model (e.g., "qwen-tts-1.7B", "whisper-base")
|
model_name: Name of the model (e.g., "qwen-tts-1.7B", "whisper-base")
|
||||||
current: Current bytes downloaded
|
current: Current bytes downloaded
|
||||||
@@ -34,9 +75,12 @@ class ProgressManager:
|
|||||||
filename: Current file being downloaded
|
filename: Current file being downloaded
|
||||||
status: Status string (downloading, extracting, complete, error)
|
status: Status string (downloading, extracting, complete, error)
|
||||||
"""
|
"""
|
||||||
|
import logging
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
progress_pct = (current / total * 100) if total > 0 else 0
|
progress_pct = (current / total * 100) if total > 0 else 0
|
||||||
|
|
||||||
self._progress[model_name] = {
|
progress_data = {
|
||||||
"model_name": model_name,
|
"model_name": model_name,
|
||||||
"current": current,
|
"current": current,
|
||||||
"total": total,
|
"total": total,
|
||||||
@@ -45,26 +89,43 @@ class ProgressManager:
|
|||||||
"status": status,
|
"status": status,
|
||||||
"timestamp": datetime.now().isoformat(),
|
"timestamp": datetime.now().isoformat(),
|
||||||
}
|
}
|
||||||
|
|
||||||
# Notify all listeners
|
print(f"[DEBUG] update_progress called: {model_name}, {progress_pct:.1f}%")
|
||||||
if model_name in self._listeners:
|
|
||||||
for queue in self._listeners[model_name]:
|
# Thread-safe update of progress dict
|
||||||
try:
|
with self._lock:
|
||||||
queue.put_nowait(self._progress[model_name].copy())
|
self._progress[model_name] = progress_data
|
||||||
except asyncio.QueueFull:
|
|
||||||
pass
|
# Notify all listeners (thread-safe)
|
||||||
|
listener_count = len(self._listeners.get(model_name, []))
|
||||||
|
print(f"[DEBUG] Listener count for {model_name}: {listener_count}")
|
||||||
|
print(f"[DEBUG] All listeners: {list(self._listeners.keys())}")
|
||||||
|
print(f"[DEBUG] Main loop set: {self._main_loop is not None}")
|
||||||
|
if self._main_loop:
|
||||||
|
print(f"[DEBUG] Main loop running: {self._main_loop.is_running()}")
|
||||||
|
|
||||||
|
if listener_count > 0:
|
||||||
|
logger.debug(f"Notifying {listener_count} listeners for {model_name}: {progress_pct:.1f}% ({filename})")
|
||||||
|
print(f"[DEBUG] About to notify listeners...")
|
||||||
|
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||||
|
print(f"[DEBUG] Notified listeners")
|
||||||
|
else:
|
||||||
|
logger.debug(f"No listeners for {model_name}, progress update stored: {progress_pct:.1f}%")
|
||||||
|
|
||||||
def get_progress(self, model_name: str) -> Optional[Dict]:
|
def get_progress(self, model_name: str) -> Optional[Dict]:
|
||||||
"""Get current progress for a model."""
|
"""Get current progress for a model. Thread-safe."""
|
||||||
return self._progress.get(model_name)
|
with self._lock:
|
||||||
|
progress = self._progress.get(model_name)
|
||||||
|
return progress.copy() if progress else None
|
||||||
|
|
||||||
def get_all_active(self) -> List[Dict]:
|
def get_all_active(self) -> List[Dict]:
|
||||||
"""Get all active downloads (status is 'downloading' or 'extracting')."""
|
"""Get all active downloads (status is 'downloading' or 'extracting'). Thread-safe."""
|
||||||
active = []
|
active = []
|
||||||
for model_name, progress in self._progress.items():
|
with self._lock:
|
||||||
status = progress.get("status", "")
|
for model_name, progress in self._progress.items():
|
||||||
if status in ("downloading", "extracting"):
|
status = progress.get("status", "")
|
||||||
active.append(progress.copy())
|
if status in ("downloading", "extracting"):
|
||||||
|
active.append(progress.copy())
|
||||||
return active
|
return active
|
||||||
|
|
||||||
def create_progress_callback(self, model_name: str, filename: Optional[str] = None):
|
def create_progress_callback(self, model_name: str, filename: Optional[str] = None):
|
||||||
@@ -98,30 +159,57 @@ class ProgressManager:
|
|||||||
async def subscribe(self, model_name: str):
|
async def subscribe(self, model_name: str):
|
||||||
"""
|
"""
|
||||||
Subscribe to progress updates for a model.
|
Subscribe to progress updates for a model.
|
||||||
|
|
||||||
Yields progress updates as Server-Sent Events.
|
Yields progress updates as Server-Sent Events.
|
||||||
"""
|
"""
|
||||||
queue = asyncio.Queue(maxsize=10)
|
import logging
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Store the main event loop for thread-safe operations
|
||||||
|
try:
|
||||||
|
self._main_loop = asyncio.get_running_loop()
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
queue = asyncio.Queue(maxsize=10)
|
||||||
|
|
||||||
# Add to listeners
|
# Add to listeners
|
||||||
if model_name not in self._listeners:
|
if model_name not in self._listeners:
|
||||||
self._listeners[model_name] = []
|
self._listeners[model_name] = []
|
||||||
self._listeners[model_name].append(queue)
|
self._listeners[model_name].append(queue)
|
||||||
|
|
||||||
|
logger.info(f"SSE client subscribed to {model_name}, total listeners: {len(self._listeners[model_name])}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Send initial progress if available
|
# Send initial progress if available and still in progress (thread-safe read)
|
||||||
if model_name in self._progress:
|
with self._lock:
|
||||||
yield f"data: {json.dumps(self._progress[model_name])}\n\n"
|
initial_progress = self._progress.get(model_name)
|
||||||
|
if initial_progress:
|
||||||
|
initial_progress = initial_progress.copy()
|
||||||
|
|
||||||
|
if initial_progress:
|
||||||
|
status = initial_progress.get('status')
|
||||||
|
# Only send initial progress if download is actually in progress
|
||||||
|
# Don't send old 'complete' or 'error' status from previous downloads
|
||||||
|
if status in ('downloading', 'extracting'):
|
||||||
|
logger.info(f"Sending initial progress for {model_name}: {status}")
|
||||||
|
yield f"data: {json.dumps(initial_progress)}\n\n"
|
||||||
|
else:
|
||||||
|
logger.info(f"Skipping initial progress for {model_name} (status: {status})")
|
||||||
|
else:
|
||||||
|
logger.info(f"No initial progress available for {model_name}")
|
||||||
|
|
||||||
# Stream updates
|
# Stream updates
|
||||||
while True:
|
while True:
|
||||||
try:
|
try:
|
||||||
# Wait for update with timeout
|
# Wait for update with timeout
|
||||||
progress = await asyncio.wait_for(queue.get(), timeout=1.0)
|
progress = await asyncio.wait_for(queue.get(), timeout=1.0)
|
||||||
|
logger.debug(f"Sending progress update for {model_name}: {progress.get('status')} - {progress.get('progress', 0):.1f}%")
|
||||||
yield f"data: {json.dumps(progress)}\n\n"
|
yield f"data: {json.dumps(progress)}\n\n"
|
||||||
|
|
||||||
# Stop if complete or error
|
# Stop if complete or error
|
||||||
if progress.get("status") in ("complete", "error"):
|
if progress.get("status") in ("complete", "error"):
|
||||||
|
logger.info(f"Download {progress.get('status')} for {model_name}, closing SSE connection")
|
||||||
break
|
break
|
||||||
except asyncio.TimeoutError:
|
except asyncio.TimeoutError:
|
||||||
# Send heartbeat
|
# Send heartbeat
|
||||||
@@ -133,32 +221,53 @@ class ProgressManager:
|
|||||||
self._listeners[model_name].remove(queue)
|
self._listeners[model_name].remove(queue)
|
||||||
if not self._listeners[model_name]:
|
if not self._listeners[model_name]:
|
||||||
del self._listeners[model_name]
|
del self._listeners[model_name]
|
||||||
|
logger.info(f"SSE client unsubscribed from {model_name}, remaining listeners: {len(self._listeners.get(model_name, []))}")
|
||||||
|
|
||||||
def mark_complete(self, model_name: str):
|
def mark_complete(self, model_name: str):
|
||||||
"""Mark a model download as complete."""
|
"""Mark a model download as complete. Thread-safe."""
|
||||||
if model_name in self._progress:
|
import logging
|
||||||
self._progress[model_name]["status"] = "complete"
|
logger = logging.getLogger(__name__)
|
||||||
self._progress[model_name]["progress"] = 100.0
|
|
||||||
# Notify listeners
|
with self._lock:
|
||||||
if model_name in self._listeners:
|
if model_name in self._progress:
|
||||||
for queue in self._listeners[model_name]:
|
self._progress[model_name]["status"] = "complete"
|
||||||
try:
|
self._progress[model_name]["progress"] = 100.0
|
||||||
queue.put_nowait(self._progress[model_name].copy())
|
progress_data = self._progress[model_name].copy()
|
||||||
except asyncio.QueueFull:
|
else:
|
||||||
pass
|
logger.warning(f"Cannot mark {model_name} as complete: not found in progress")
|
||||||
|
return
|
||||||
|
|
||||||
|
logger.info(f"Marked {model_name} as complete")
|
||||||
|
# Notify listeners (thread-safe)
|
||||||
|
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||||
|
|
||||||
def mark_error(self, model_name: str, error: str):
|
def mark_error(self, model_name: str, error: str):
|
||||||
"""Mark a model download as failed."""
|
"""Mark a model download as failed. Thread-safe."""
|
||||||
if model_name in self._progress:
|
import logging
|
||||||
self._progress[model_name]["status"] = "error"
|
logger = logging.getLogger(__name__)
|
||||||
self._progress[model_name]["error"] = error
|
|
||||||
# Notify listeners
|
with self._lock:
|
||||||
if model_name in self._listeners:
|
if model_name in self._progress:
|
||||||
for queue in self._listeners[model_name]:
|
self._progress[model_name]["status"] = "error"
|
||||||
try:
|
self._progress[model_name]["error"] = error
|
||||||
queue.put_nowait(self._progress[model_name].copy())
|
progress_data = self._progress[model_name].copy()
|
||||||
except asyncio.QueueFull:
|
else:
|
||||||
pass
|
# Create new progress entry for error
|
||||||
|
progress_data = {
|
||||||
|
"model_name": model_name,
|
||||||
|
"current": 0,
|
||||||
|
"total": 0,
|
||||||
|
"progress": 0,
|
||||||
|
"filename": None,
|
||||||
|
"status": "error",
|
||||||
|
"error": error,
|
||||||
|
"timestamp": datetime.now().isoformat(),
|
||||||
|
}
|
||||||
|
self._progress[model_name] = progress_data
|
||||||
|
|
||||||
|
logger.error(f"Marked {model_name} as error: {error}")
|
||||||
|
# Notify listeners (thread-safe)
|
||||||
|
self._notify_listeners_threadsafe(model_name, progress_data)
|
||||||
|
|
||||||
|
|
||||||
# Global progress manager instance
|
# Global progress manager instance
|
||||||
|
|||||||
@@ -4,16 +4,20 @@ from PyInstaller.utils.hooks import collect_submodules
|
|||||||
from PyInstaller.utils.hooks import copy_metadata
|
from PyInstaller.utils.hooks import copy_metadata
|
||||||
|
|
||||||
datas = []
|
datas = []
|
||||||
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern']
|
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.profiles', 'backend.history', 'backend.tts', 'backend.transcribe', 'backend.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.utils.validation', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'librosa', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'pkg_resources.extern', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
|
||||||
datas += collect_data_files('qwen_tts')
|
datas += collect_data_files('qwen_tts')
|
||||||
|
datas += collect_data_files('mlx')
|
||||||
|
datas += collect_data_files('mlx_audio')
|
||||||
datas += copy_metadata('qwen-tts')
|
datas += copy_metadata('qwen-tts')
|
||||||
hiddenimports += collect_submodules('qwen_tts')
|
hiddenimports += collect_submodules('qwen_tts')
|
||||||
hiddenimports += collect_submodules('jaraco')
|
hiddenimports += collect_submodules('jaraco')
|
||||||
|
hiddenimports += collect_submodules('mlx')
|
||||||
|
hiddenimports += collect_submodules('mlx_audio')
|
||||||
|
|
||||||
|
|
||||||
a = Analysis(
|
a = Analysis(
|
||||||
['server.py'],
|
['server.py'],
|
||||||
pathex=['C:\\Users\\ijame\\Projects\\voice\\Qwen3-TTS'],
|
pathex=[],
|
||||||
binaries=[],
|
binaries=[],
|
||||||
datas=datas,
|
datas=datas,
|
||||||
hiddenimports=hiddenimports,
|
hiddenimports=hiddenimports,
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
},
|
},
|
||||||
"app": {
|
"app": {
|
||||||
"name": "@voicebox/app",
|
"name": "@voicebox/app",
|
||||||
"version": "0.1.5",
|
"version": "0.1.9",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@dnd-kit/core": "^6.3.1",
|
"@dnd-kit/core": "^6.3.1",
|
||||||
"@dnd-kit/sortable": "^10.0.0",
|
"@dnd-kit/sortable": "^10.0.0",
|
||||||
@@ -50,6 +50,7 @@
|
|||||||
"react": "^18.3.0",
|
"react": "^18.3.0",
|
||||||
"react-dom": "^18.3.0",
|
"react-dom": "^18.3.0",
|
||||||
"react-hook-form": "^7.53.0",
|
"react-hook-form": "^7.53.0",
|
||||||
|
"react-sound-visualizer": "^1.4.0",
|
||||||
"tailwind-merge": "^2.5.4",
|
"tailwind-merge": "^2.5.4",
|
||||||
"wavesurfer.js": "^7.0.0",
|
"wavesurfer.js": "^7.0.0",
|
||||||
"zod": "^3.23.8",
|
"zod": "^3.23.8",
|
||||||
@@ -67,7 +68,7 @@
|
|||||||
},
|
},
|
||||||
"landing": {
|
"landing": {
|
||||||
"name": "@voicebox/landing",
|
"name": "@voicebox/landing",
|
||||||
"version": "0.1.5",
|
"version": "0.1.9",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@radix-ui/react-separator": "^1.1.8",
|
"@radix-ui/react-separator": "^1.1.8",
|
||||||
"@radix-ui/react-slot": "^1.2.4",
|
"@radix-ui/react-slot": "^1.2.4",
|
||||||
@@ -92,7 +93,7 @@
|
|||||||
},
|
},
|
||||||
"tauri": {
|
"tauri": {
|
||||||
"name": "@voicebox/tauri",
|
"name": "@voicebox/tauri",
|
||||||
"version": "0.1.5",
|
"version": "0.1.9",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@tauri-apps/api": "^2.0.0",
|
"@tauri-apps/api": "^2.0.0",
|
||||||
"@tauri-apps/plugin-shell": "^2.0.0",
|
"@tauri-apps/plugin-shell": "^2.0.0",
|
||||||
@@ -111,7 +112,7 @@
|
|||||||
},
|
},
|
||||||
"web": {
|
"web": {
|
||||||
"name": "@voicebox/web",
|
"name": "@voicebox/web",
|
||||||
"version": "0.1.5",
|
"version": "0.1.9",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@tanstack/react-query": "^5.0.0",
|
"@tanstack/react-query": "^5.0.0",
|
||||||
"react": "^18.3.0",
|
"react": "^18.3.0",
|
||||||
@@ -970,6 +971,8 @@
|
|||||||
|
|
||||||
"react-remove-scroll-bar": ["[email protected]", "", { "dependencies": { "react-style-singleton": "^2.2.2", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" }, "optionalPeers": ["@types/react"] }, "sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q=="],
|
"react-remove-scroll-bar": ["[email protected]", "", { "dependencies": { "react-style-singleton": "^2.2.2", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" }, "optionalPeers": ["@types/react"] }, "sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q=="],
|
||||||
|
|
||||||
|
"react-sound-visualizer": ["[email protected]", "", { "dependencies": { "sound-visualizer": "^1.2.0" }, "peerDependencies": { "react": ">= 16" } }, "sha512-Qe7tFTd1owtQ8nYrUYXg7QLt8mw7iUy86mqj/+IwmXzSw+NlhnMnAGPuisb1Lk3ncliFnM+AQbZb3C4RQN9uMQ=="],
|
||||||
|
|
||||||
"react-style-singleton": ["[email protected]", "", { "dependencies": { "get-nonce": "^1.0.0", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-b6jSvxvVnyptAiLjbkWLE/lOnR4lfTtDAl+eUC7RZy+QQWc6wRzIV2CE6xBuMmDxc2qIihtDCZD5NPOFl7fRBQ=="],
|
"react-style-singleton": ["[email protected]", "", { "dependencies": { "get-nonce": "^1.0.0", "tslib": "^2.0.0" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-b6jSvxvVnyptAiLjbkWLE/lOnR4lfTtDAl+eUC7RZy+QQWc6wRzIV2CE6xBuMmDxc2qIihtDCZD5NPOFl7fRBQ=="],
|
||||||
|
|
||||||
"read-cache": ["[email protected]", "", { "dependencies": { "pify": "^2.3.0" } }, "sha512-Owdv/Ft7IjOgm/i0xvNDZ1LrRANRfew4b2prF3OWMQLxLfu3bS8FVhCsrSCMK4lR56Y9ya+AThoTpDCTxCmpRA=="],
|
"read-cache": ["[email protected]", "", { "dependencies": { "pify": "^2.3.0" } }, "sha512-Owdv/Ft7IjOgm/i0xvNDZ1LrRANRfew4b2prF3OWMQLxLfu3bS8FVhCsrSCMK4lR56Y9ya+AThoTpDCTxCmpRA=="],
|
||||||
@@ -1004,6 +1007,8 @@
|
|||||||
|
|
||||||
"slash": ["[email protected]", "", {}, "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q=="],
|
"slash": ["[email protected]", "", {}, "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q=="],
|
||||||
|
|
||||||
|
"sound-visualizer": ["[email protected]", "", {}, "sha512-2+Un0PrrBgXylnCjrVYUoRW7KEDH29h7O8/MGzeDOgFGBPb9oX/2n/RGBxJXvVv2U3KFwX5olUWeJKf0Rr5TLQ=="],
|
||||||
|
|
||||||
"source-map-js": ["[email protected]", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
|
"source-map-js": ["[email protected]", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
|
||||||
|
|
||||||
"strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="],
|
"strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="],
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
node_modules
|
||||||
|
.mintlify
|
||||||
|
.DS_Store
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Voicebox Documentation
|
||||||
|
|
||||||
|
This directory contains the documentation for Voicebox, built with [Mintlify](https://mintlify.com).
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
Install Mintlify globally using bun:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun add -g mintlify
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use the helper script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run install:mintlify
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running Locally
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
This will start the Mintlify dev server.
|
||||||
|
|
||||||
|
The docs will be available at `http://localhost:3000`
|
||||||
|
|
||||||
|
### Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
docs/
|
||||||
|
├── mint.json # Mintlify configuration
|
||||||
|
├── custom.css # Custom styles
|
||||||
|
├── overview/ # Getting started & feature docs
|
||||||
|
├── guides/ # User guides
|
||||||
|
├── api/ # API reference
|
||||||
|
├── development/ # Developer documentation
|
||||||
|
├── logo/ # Logo assets
|
||||||
|
└── public/ # Static assets
|
||||||
|
```
|
||||||
|
|
||||||
|
### Writing Docs
|
||||||
|
|
||||||
|
- Use `.mdx` files for all documentation pages
|
||||||
|
- Follow the existing structure in `mint.json` for navigation
|
||||||
|
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
|
||||||
|
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
Docs are automatically deployed when changes are pushed to the main branch.
|
||||||
|
|
||||||
|
To manually deploy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mintlify deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
|
||||||
+32
-4
@@ -90,6 +90,26 @@ chmod +x voicebox-*.AppImage
|
|||||||
- Slower but works without GPU
|
- Slower but works without GPU
|
||||||
- Backend automatically falls back to CPU
|
- Backend automatically falls back to CPU
|
||||||
|
|
||||||
|
### MLX "Failed to load the default metallib" error (Apple Silicon)
|
||||||
|
|
||||||
|
**Symptoms:** Generation fails with "library not found" or "metallib" errors
|
||||||
|
|
||||||
|
**Solutions:**
|
||||||
|
1. **Rebuild server binary**
|
||||||
|
```bash
|
||||||
|
bun run build:server
|
||||||
|
```
|
||||||
|
The build script should automatically include MLX Metal shader libraries.
|
||||||
|
|
||||||
|
2. **Check MLX installation**
|
||||||
|
```bash
|
||||||
|
pip install -r backend/requirements-mlx.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Verify backend detection**
|
||||||
|
- Check server logs for "Backend: MLX"
|
||||||
|
- If showing "Backend: PYTORCH", MLX may not be installed correctly
|
||||||
|
|
||||||
### Audio playback issues
|
### Audio playback issues
|
||||||
|
|
||||||
**Symptoms:** Generated audio won't play
|
**Symptoms:** Generated audio won't play
|
||||||
@@ -111,19 +131,27 @@ chmod +x voicebox-*.AppImage
|
|||||||
**Symptoms:** Generation takes >30 seconds
|
**Symptoms:** Generation takes >30 seconds
|
||||||
|
|
||||||
**Solutions:**
|
**Solutions:**
|
||||||
1. **Use GPU** (if available)
|
1. **Check backend type** (Apple Silicon)
|
||||||
|
- Check Settings → Server Status
|
||||||
|
- Should show "Backend: MLX" on Apple Silicon
|
||||||
|
- If showing "Backend: PYTORCH", install MLX: `pip install -r backend/requirements-mlx.txt`
|
||||||
|
- MLX provides 4-5x faster inference on Apple Silicon
|
||||||
|
|
||||||
|
2. **Use GPU** (if available)
|
||||||
- Check Settings → Server Status
|
- Check Settings → Server Status
|
||||||
- Should show "GPU available: true"
|
- Should show "GPU available: true"
|
||||||
|
- Apple Silicon: Should show "Metal (Apple Silicon via MLX)"
|
||||||
|
- Windows/Linux: Should show "CUDA" if GPU available
|
||||||
|
|
||||||
2. **Enable caching**
|
3. **Enable caching**
|
||||||
- Voice prompts are cached automatically
|
- Voice prompts are cached automatically
|
||||||
- Second generation with same voice should be faster
|
- Second generation with same voice should be faster
|
||||||
|
|
||||||
3. **Use smaller model**
|
4. **Use smaller model**
|
||||||
- 0.6B model is faster than 1.7B
|
- 0.6B model is faster than 1.7B
|
||||||
- Quality difference is minimal for most voices
|
- Quality difference is minimal for most voices
|
||||||
|
|
||||||
4. **Check system resources**
|
5. **Check system resources**
|
||||||
- Close other CPU/GPU intensive apps
|
- Close other CPU/GPU intensive apps
|
||||||
- Ensure adequate RAM (8GB+ recommended)
|
- Ensure adequate RAM (8GB+ recommended)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
title: "Authentication"
|
||||||
|
description: "API authentication and security"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
Authentication is not currently implemented in Voicebox. The API is intended for local use only.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
## Local Usage
|
||||||
|
|
||||||
|
For local development and usage:
|
||||||
|
- API runs on `localhost:17493`
|
||||||
|
- No authentication required
|
||||||
|
- Access restricted to local machine
|
||||||
|
|
||||||
|
## Future Implementation
|
||||||
|
|
||||||
|
Authentication will be added in a future release for:
|
||||||
|
- Remote deployments
|
||||||
|
- Multi-user access
|
||||||
|
- Production environments
|
||||||
|
|
||||||
|
Planned authentication methods:
|
||||||
|
- API keys
|
||||||
|
- OAuth 2.0
|
||||||
|
- JWT tokens
|
||||||
|
|
||||||
|
## Security Best Practices
|
||||||
|
|
||||||
|
Until authentication is implemented:
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Use VPN" icon="shield">
|
||||||
|
Use WireGuard or Tailscale for remote access
|
||||||
|
</Card>
|
||||||
|
<Card title="Reverse Proxy" icon="server">
|
||||||
|
Run behind nginx with basic auth
|
||||||
|
</Card>
|
||||||
|
<Card title="Firewall" icon="fire">
|
||||||
|
Restrict access to trusted IPs only
|
||||||
|
</Card>
|
||||||
|
<Card title="Local Only" icon="laptop">
|
||||||
|
Don't expose to public internet
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Coming Soon
|
||||||
|
|
||||||
|
- API key management
|
||||||
|
- User accounts
|
||||||
|
- Rate limiting
|
||||||
|
- Access control
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
title: "Generation API"
|
||||||
|
description: "Generate speech from text"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Generate Speech
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /generate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"language": "en"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "gen123",
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"language": "en",
|
||||||
|
"audio_url": "/audio/gen123.wav",
|
||||||
|
"duration": 2.3,
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## List History
|
||||||
|
|
||||||
|
```http
|
||||||
|
GET /history
|
||||||
|
```
|
||||||
|
|
||||||
|
**Query Parameters:**
|
||||||
|
- `profile_id` (optional) - Filter by voice profile
|
||||||
|
- `limit` (optional) - Number of results (default: 50)
|
||||||
|
- `offset` (optional) - Pagination offset
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"generations": [
|
||||||
|
{
|
||||||
|
"id": "gen123",
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"duration": 2.3,
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total": 100
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Get Generation
|
||||||
|
|
||||||
|
```http
|
||||||
|
GET /history/{id}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "gen123",
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"language": "en",
|
||||||
|
"audio_url": "/audio/gen123.wav",
|
||||||
|
"duration": 2.3,
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Delete Generation
|
||||||
|
|
||||||
|
```http
|
||||||
|
DELETE /history/{id}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## TypeScript Example
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { VoiceboxClient } from '@/lib/api'
|
||||||
|
|
||||||
|
const client = new VoiceboxClient({
|
||||||
|
baseUrl: 'http://localhost:17493'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Generate speech
|
||||||
|
const generation = await client.generate({
|
||||||
|
text: 'Hello world',
|
||||||
|
profile_id: 'abc123',
|
||||||
|
language: 'en'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Get audio URL
|
||||||
|
const audioUrl = generation.audio_url
|
||||||
|
|
||||||
|
// List history
|
||||||
|
const history = await client.listHistory({
|
||||||
|
profile_id: 'abc123',
|
||||||
|
limit: 20
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||||
@@ -0,0 +1,219 @@
|
|||||||
|
---
|
||||||
|
title: "API Overview"
|
||||||
|
description: "Integrate voice synthesis into your applications with the Voicebox REST API"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
Voicebox exposes a full REST API that allows you to integrate voice synthesis into your own applications. The API runs on `http://localhost:17493` by default.
|
||||||
|
|
||||||
|
<Card title="Interactive API Docs" icon="book" href="http://localhost:17493/docs">
|
||||||
|
When Voicebox is running, visit the auto-generated API documentation at `http://localhost:17493/docs`
|
||||||
|
</Card>
|
||||||
|
|
||||||
|
## Base URL
|
||||||
|
|
||||||
|
```
|
||||||
|
http://localhost:17493
|
||||||
|
```
|
||||||
|
|
||||||
|
For remote deployments, replace `localhost` with your server's IP or hostname.
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Currently, the API does not require authentication for local development. Authentication will be added in a future release for production deployments.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Quick Example
|
||||||
|
|
||||||
|
Here's a simple example of generating speech:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate speech
|
||||||
|
curl -X POST http://localhost:17493/generate \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"language": "en"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
The Voicebox API is organized into several categories:
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Voice Profiles" icon="user" href="/api/voice-profiles">
|
||||||
|
Create, list, update, and delete voice profiles
|
||||||
|
</Card>
|
||||||
|
<Card title="Generation" icon="waveform" href="/api/generation">
|
||||||
|
Generate speech from text using voice profiles
|
||||||
|
</Card>
|
||||||
|
<Card title="Recordings" icon="microphone" href="/api/recordings">
|
||||||
|
Record and transcribe audio
|
||||||
|
</Card>
|
||||||
|
<Card title="Stories" icon="film">
|
||||||
|
Create and manage multi-voice stories (coming soon)
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Core Endpoints
|
||||||
|
|
||||||
|
### Voice Profiles
|
||||||
|
|
||||||
|
```http
|
||||||
|
GET /profiles # List all profiles
|
||||||
|
POST /profiles # Create a new profile
|
||||||
|
GET /profiles/{id} # Get profile details
|
||||||
|
PUT /profiles/{id} # Update a profile
|
||||||
|
DELETE /profiles/{id} # Delete a profile
|
||||||
|
POST /profiles/{id}/samples # Add voice sample
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generation
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /generate # Generate speech
|
||||||
|
GET /history # List generation history
|
||||||
|
GET /history/{id} # Get generation details
|
||||||
|
DELETE /history/{id} # Delete from history
|
||||||
|
```
|
||||||
|
|
||||||
|
### Recordings
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /recordings # Start recording
|
||||||
|
POST /recordings/stop # Stop recording
|
||||||
|
POST /transcribe # Transcribe audio
|
||||||
|
```
|
||||||
|
|
||||||
|
## Response Format
|
||||||
|
|
||||||
|
All API responses follow a consistent JSON format:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"data": {
|
||||||
|
// Response data
|
||||||
|
},
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Error responses:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": false,
|
||||||
|
"data": null,
|
||||||
|
"error": {
|
||||||
|
"message": "Error description",
|
||||||
|
"code": "ERROR_CODE"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Models
|
||||||
|
|
||||||
|
### Voice Profile
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "abc123",
|
||||||
|
"name": "John Smith",
|
||||||
|
"language": "en",
|
||||||
|
"description": "Professional narrator voice",
|
||||||
|
"created_at": "2024-01-29T12:00:00Z",
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"id": "sample123",
|
||||||
|
"audio_path": "/path/to/sample.wav",
|
||||||
|
"duration": 15.5
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generation
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "gen123",
|
||||||
|
"text": "Hello world",
|
||||||
|
"profile_id": "abc123",
|
||||||
|
"language": "en",
|
||||||
|
"audio_path": "/path/to/output.wav",
|
||||||
|
"duration": 2.3,
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## TypeScript Client
|
||||||
|
|
||||||
|
Voicebox provides an auto-generated TypeScript client with full type safety:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { VoiceboxClient } from '@/lib/api'
|
||||||
|
|
||||||
|
const client = new VoiceboxClient({
|
||||||
|
baseUrl: 'http://localhost:17493'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Create a profile
|
||||||
|
const profile = await client.createProfile({
|
||||||
|
name: 'John Smith',
|
||||||
|
language: 'en'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Generate speech
|
||||||
|
const generation = await client.generate({
|
||||||
|
text: 'Hello world',
|
||||||
|
profile_id: profile.id,
|
||||||
|
language: 'en'
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
The client is automatically generated from the OpenAPI schema. See [Development Setup](/development/setup#generate-openapi-client) for details.
|
||||||
|
|
||||||
|
## Rate Limiting
|
||||||
|
|
||||||
|
<Info>
|
||||||
|
Currently, there are no rate limits for local usage. Rate limiting will be added in a future release for production deployments.
|
||||||
|
</Info>
|
||||||
|
|
||||||
|
## WebSocket Support
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Real-time streaming generation via WebSockets is planned for a future release.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Game Development" icon="gamepad">
|
||||||
|
Generate dynamic dialogue for NPCs and characters
|
||||||
|
</Card>
|
||||||
|
<Card title="Content Creation" icon="video">
|
||||||
|
Automate voiceovers for videos and podcasts
|
||||||
|
</Card>
|
||||||
|
<Card title="Accessibility" icon="universal-access">
|
||||||
|
Build text-to-speech tools for visually impaired users
|
||||||
|
</Card>
|
||||||
|
<Card title="Voice Assistants" icon="robot">
|
||||||
|
Create custom voice interfaces
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Voice Profiles API" icon="user" href="/api/voice-profiles">
|
||||||
|
Learn how to manage voice profiles
|
||||||
|
</Card>
|
||||||
|
<Card title="Generation API" icon="waveform" href="/api/generation">
|
||||||
|
Generate speech from text
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
---
|
||||||
|
title: "Recordings API"
|
||||||
|
description: "Record and transcribe audio"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Start Recording
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /recordings/start
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source": "microphone"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"recording_id": "rec123",
|
||||||
|
"status": "recording"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stop Recording
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /recordings/stop
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"recording_id": "rec123"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"recording_id": "rec123",
|
||||||
|
"audio_url": "/audio/rec123.wav",
|
||||||
|
"duration": 15.5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Transcribe Audio
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /transcribe
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:** (multipart/form-data)
|
||||||
|
```
|
||||||
|
audio: <file>
|
||||||
|
language: "en" (optional)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"text": "Transcribed speech text here",
|
||||||
|
"language": "en",
|
||||||
|
"duration": 15.5,
|
||||||
|
"confidence": 0.95
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## TypeScript Example
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { VoiceboxClient } from '@/lib/api'
|
||||||
|
|
||||||
|
const client = new VoiceboxClient({
|
||||||
|
baseUrl: 'http://localhost:17493'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Start recording
|
||||||
|
const recording = await client.startRecording({
|
||||||
|
source: 'microphone'
|
||||||
|
})
|
||||||
|
|
||||||
|
// ... record audio ...
|
||||||
|
|
||||||
|
// Stop recording
|
||||||
|
const result = await client.stopRecording(recording.id)
|
||||||
|
|
||||||
|
// Transcribe
|
||||||
|
const transcription = await client.transcribe(audioFile, 'en')
|
||||||
|
console.log(transcription.text)
|
||||||
|
```
|
||||||
|
|
||||||
|
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
---
|
||||||
|
title: "Voice Profiles API"
|
||||||
|
description: "Manage voice profiles programmatically"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
### List Profiles
|
||||||
|
|
||||||
|
```http
|
||||||
|
GET /profiles
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"profiles": [
|
||||||
|
{
|
||||||
|
"id": "abc123",
|
||||||
|
"name": "John Smith",
|
||||||
|
"language": "en",
|
||||||
|
"description": "Professional narrator",
|
||||||
|
"created_at": "2024-01-29T12:00:00Z",
|
||||||
|
"sample_count": 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Profile
|
||||||
|
|
||||||
|
```http
|
||||||
|
GET /profiles/{id}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "abc123",
|
||||||
|
"name": "John Smith",
|
||||||
|
"language": "en",
|
||||||
|
"description": "Professional narrator",
|
||||||
|
"created_at": "2024-01-29T12:00:00Z",
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"id": "sample123",
|
||||||
|
"duration": 15.5,
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create Profile
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /profiles
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "John Smith",
|
||||||
|
"language": "en",
|
||||||
|
"description": "Professional narrator"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "abc123",
|
||||||
|
"name": "John Smith",
|
||||||
|
"language": "en",
|
||||||
|
"description": "Professional narrator",
|
||||||
|
"created_at": "2024-01-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Update Profile
|
||||||
|
|
||||||
|
```http
|
||||||
|
PUT /profiles/{id}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "Updated Name",
|
||||||
|
"description": "Updated description"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete Profile
|
||||||
|
|
||||||
|
```http
|
||||||
|
DELETE /profiles/{id}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Add Voice Sample
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /profiles/{id}/samples
|
||||||
|
```
|
||||||
|
|
||||||
|
**Request:** (multipart/form-data)
|
||||||
|
```
|
||||||
|
audio: <file>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sample_id": "sample123",
|
||||||
|
"duration": 15.5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## TypeScript Example
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { VoiceboxClient } from '@/lib/api'
|
||||||
|
|
||||||
|
const client = new VoiceboxClient({
|
||||||
|
baseUrl: 'http://localhost:17493'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Create profile
|
||||||
|
const profile = await client.createProfile({
|
||||||
|
name: 'John Smith',
|
||||||
|
language: 'en',
|
||||||
|
description: 'Professional narrator'
|
||||||
|
})
|
||||||
|
|
||||||
|
// Add sample
|
||||||
|
await client.addSample(profile.id, audioFile)
|
||||||
|
|
||||||
|
// List all profiles
|
||||||
|
const profiles = await client.listProfiles()
|
||||||
|
```
|
||||||
|
|
||||||
|
For full API documentation, visit `http://localhost:17493/docs` when the server is running.
|
||||||
+1831
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,15 @@
|
|||||||
|
/* Anchor hover styles */
|
||||||
|
.nav-anchor:hover {
|
||||||
|
@apply text-[#BF9E40];
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Icon wrapper on hover */
|
||||||
|
.nav-anchor:hover div {
|
||||||
|
background: #BF9E40 !important;
|
||||||
|
filter: brightness(1) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Icon SVG on hover */
|
||||||
|
.nav-anchor:hover svg {
|
||||||
|
@apply bg-white !important;
|
||||||
|
}
|
||||||
@@ -0,0 +1,206 @@
|
|||||||
|
---
|
||||||
|
title: "Architecture"
|
||||||
|
description: "Understanding Voicebox's technical architecture"
|
||||||
|
---
|
||||||
|
|
||||||
|
## System Overview
|
||||||
|
|
||||||
|
Voicebox uses a client-server architecture with a React frontend and Python backend. The desktop app is built with Tauri and contains two main layers:
|
||||||
|
|
||||||
|
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
|
||||||
|
|
||||||
|
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
|
||||||
|
|
||||||
|
These two layers communicate via HTTP, with the frontend making API requests to the backend.
|
||||||
|
|
||||||
|
## Frontend Architecture
|
||||||
|
|
||||||
|
### Tech Stack
|
||||||
|
|
||||||
|
- **Framework**: React 18 with TypeScript
|
||||||
|
- **State Management**: Zustand stores
|
||||||
|
- **Data Fetching**: React Query (TanStack Query)
|
||||||
|
- **Styling**: Tailwind CSS
|
||||||
|
- **Audio**: WaveSurfer.js
|
||||||
|
- **Desktop**: Tauri (Rust)
|
||||||
|
|
||||||
|
### Component Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
app/src/
|
||||||
|
├── components/ # React components
|
||||||
|
│ ├── profiles/ # Voice profile UI
|
||||||
|
│ ├── generation/ # Speech generation UI
|
||||||
|
│ ├── stories/ # Timeline editor
|
||||||
|
│ └── shared/ # Reusable components
|
||||||
|
├── lib/ # Utilities
|
||||||
|
│ ├── api/ # Generated API client
|
||||||
|
│ └── utils/ # Helper functions
|
||||||
|
├── hooks/ # React hooks
|
||||||
|
└── stores/ # Zustand state stores
|
||||||
|
```
|
||||||
|
|
||||||
|
### State Management
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Example: Profile store
|
||||||
|
const useProfileStore = create((set) => ({
|
||||||
|
profiles: [],
|
||||||
|
selectedProfile: null,
|
||||||
|
setProfiles: (profiles) => set({ profiles }),
|
||||||
|
selectProfile: (id) => set({ selectedProfile: id })
|
||||||
|
}))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backend Architecture
|
||||||
|
|
||||||
|
### Tech Stack
|
||||||
|
|
||||||
|
- **Framework**: FastAPI (Python 3.11+)
|
||||||
|
- **TTS Model**: Qwen3-TTS
|
||||||
|
- **Transcription**: Whisper
|
||||||
|
- **Database**: SQLite
|
||||||
|
- **Audio**: librosa, soundfile
|
||||||
|
|
||||||
|
### API Structure
|
||||||
|
|
||||||
|
```python
|
||||||
|
# main.py - API routes
|
||||||
|
@app.post("/generate")
|
||||||
|
async def generate_speech(request: GenerateRequest):
|
||||||
|
# 1. Validate request
|
||||||
|
# 2. Load voice profile
|
||||||
|
# 3. Generate audio with TTS
|
||||||
|
# 4. Save to database
|
||||||
|
# 5. Return response
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Model
|
||||||
|
|
||||||
|
The database uses three main tables:
|
||||||
|
|
||||||
|
**Profile Table:** Stores voice profiles with fields for id, name, and language.
|
||||||
|
|
||||||
|
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
|
||||||
|
|
||||||
|
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
|
||||||
|
|
||||||
|
## Desktop App (Tauri)
|
||||||
|
|
||||||
|
### Rust Backend
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Sidecar process management
|
||||||
|
// File system access
|
||||||
|
// Native integrations
|
||||||
|
```
|
||||||
|
|
||||||
|
### Responsibilities
|
||||||
|
|
||||||
|
- Launch Python backend as sidecar process
|
||||||
|
- Native file dialogs
|
||||||
|
- System tray integration
|
||||||
|
- Auto-updates
|
||||||
|
- OS-specific features
|
||||||
|
|
||||||
|
## Build Process
|
||||||
|
|
||||||
|
### Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Frontend (Vite dev server)
|
||||||
|
cd app && bun run dev
|
||||||
|
|
||||||
|
# Backend (manual start)
|
||||||
|
cd backend && uvicorn main:app --reload
|
||||||
|
|
||||||
|
# Desktop app (connects to manual backend)
|
||||||
|
bun run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
### Production
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build everything (server binary + Tauri app)
|
||||||
|
bun run build
|
||||||
|
|
||||||
|
# Or build separately:
|
||||||
|
# 1. Build server binary (PyInstaller)
|
||||||
|
bun run build:server
|
||||||
|
|
||||||
|
# 2. Build Tauri app (includes server)
|
||||||
|
cd tauri && bun run tauri build
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
### Generation Flow
|
||||||
|
|
||||||
|
When a user generates speech, the data flows through the following stages:
|
||||||
|
|
||||||
|
1. **User Input** - User enters text in a React component
|
||||||
|
2. **State Update** - Text is stored in Zustand state
|
||||||
|
3. **API Request** - React Query mutation triggers an API call via fetch
|
||||||
|
4. **Backend Processing** - FastAPI endpoint receives the request
|
||||||
|
5. **TTS Generation** - Qwen3-TTS model generates the audio
|
||||||
|
6. **Storage** - Audio file is saved to disk and a database record is created
|
||||||
|
7. **Response** - Backend returns the audio URL
|
||||||
|
8. **Cache Update** - React Query updates its cache with the response
|
||||||
|
9. **UI Update** - Component re-renders with new data
|
||||||
|
10. **Playback** - User can play the generated audio
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
|
||||||
|
- **Code splitting** - Lazy load routes
|
||||||
|
- **Memoization** - React.memo for heavy components
|
||||||
|
- **Virtual scrolling** - For large lists
|
||||||
|
- **Debouncing** - Search and input handling
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
- **Async operations** - All I/O is async
|
||||||
|
- **Model caching** - Keep TTS model in memory
|
||||||
|
- **Voice prompt caching** - Reuse embeddings
|
||||||
|
- **Connection pooling** - Database connections
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
### Current
|
||||||
|
|
||||||
|
- Local-only by default
|
||||||
|
- No authentication (localhost trust)
|
||||||
|
- File system sandboxing via Tauri
|
||||||
|
|
||||||
|
### Planned
|
||||||
|
|
||||||
|
- API key authentication
|
||||||
|
- User accounts
|
||||||
|
- Rate limiting
|
||||||
|
- HTTPS support
|
||||||
|
|
||||||
|
## Deployment Modes
|
||||||
|
|
||||||
|
### Local Mode
|
||||||
|
|
||||||
|
- Backend runs as sidecar
|
||||||
|
- All data stays on device
|
||||||
|
- No network required
|
||||||
|
|
||||||
|
### Remote Mode
|
||||||
|
|
||||||
|
- Backend on separate machine
|
||||||
|
- Frontend connects via HTTP
|
||||||
|
- Shared infrastructure possible
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Development Setup" icon="code" href="/development/setup">
|
||||||
|
Set up your dev environment
|
||||||
|
</Card>
|
||||||
|
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
|
||||||
|
Contribute to Voicebox
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
@@ -0,0 +1,310 @@
|
|||||||
|
---
|
||||||
|
title: "Audio Channels"
|
||||||
|
description: "How audio output routing works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Audio channels allow routing voice output to different audio devices. This is useful for multi-output setups where different voices should play through different speakers or applications.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
**Channel:** A named audio bus that can be assigned to output devices.
|
||||||
|
|
||||||
|
**Device Mapping:** Links channels to OS audio device identifiers.
|
||||||
|
|
||||||
|
**Profile Mapping:** Links voice profiles to channels (many-to-many).
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### AudioChannel Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class AudioChannel(Base):
|
||||||
|
__tablename__ = "audio_channels"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
name = Column(String, nullable=False)
|
||||||
|
is_default = Column(Boolean, default=False)
|
||||||
|
created_at = Column(DateTime)
|
||||||
|
```
|
||||||
|
|
||||||
|
### ChannelDeviceMapping Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ChannelDeviceMapping(Base):
|
||||||
|
__tablename__ = "channel_device_mappings"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
channel_id = Column(String, ForeignKey("audio_channels.id"))
|
||||||
|
device_id = Column(String) # OS device identifier
|
||||||
|
```
|
||||||
|
|
||||||
|
### ProfileChannelMapping Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ProfileChannelMapping(Base):
|
||||||
|
__tablename__ = "profile_channel_mappings"
|
||||||
|
|
||||||
|
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
|
||||||
|
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Default Channel
|
||||||
|
|
||||||
|
A default channel is created on database initialization:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def init_db():
|
||||||
|
# Create default channel if it doesn't exist
|
||||||
|
default_channel = db.query(AudioChannel).filter(
|
||||||
|
AudioChannel.is_default == True
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not default_channel:
|
||||||
|
default_channel = AudioChannel(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
name="Default",
|
||||||
|
is_default=True
|
||||||
|
)
|
||||||
|
db.add(default_channel)
|
||||||
|
|
||||||
|
# Assign all existing profiles to default channel
|
||||||
|
profiles = db.query(VoiceProfile).all()
|
||||||
|
for profile in profiles:
|
||||||
|
mapping = ProfileChannelMapping(
|
||||||
|
profile_id=profile.id,
|
||||||
|
channel_id=default_channel.id
|
||||||
|
)
|
||||||
|
db.add(mapping)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Operations
|
||||||
|
|
||||||
|
### Creating a Channel
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_channel(
|
||||||
|
data: AudioChannelCreate,
|
||||||
|
db: Session,
|
||||||
|
) -> AudioChannelResponse:
|
||||||
|
# Check name uniqueness
|
||||||
|
existing = db.query(DBAudioChannel).filter_by(name=data.name).first()
|
||||||
|
if existing:
|
||||||
|
raise ValueError(f"Channel with name '{data.name}' already exists")
|
||||||
|
|
||||||
|
# Create channel
|
||||||
|
channel = DBAudioChannel(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
name=data.name,
|
||||||
|
is_default=False,
|
||||||
|
)
|
||||||
|
db.add(channel)
|
||||||
|
|
||||||
|
# Add device mappings
|
||||||
|
for device_id in data.device_ids:
|
||||||
|
mapping = DBChannelDeviceMapping(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
channel_id=channel.id,
|
||||||
|
device_id=device_id,
|
||||||
|
)
|
||||||
|
db.add(mapping)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Updating a Channel
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def update_channel(
|
||||||
|
channel_id: str,
|
||||||
|
data: AudioChannelUpdate,
|
||||||
|
db: Session,
|
||||||
|
) -> AudioChannelResponse:
|
||||||
|
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||||
|
|
||||||
|
# Cannot modify default channel
|
||||||
|
if channel.is_default:
|
||||||
|
raise ValueError("Cannot modify the default channel")
|
||||||
|
|
||||||
|
# Update name
|
||||||
|
if data.name is not None:
|
||||||
|
channel.name = data.name
|
||||||
|
|
||||||
|
# Update device mappings
|
||||||
|
if data.device_ids is not None:
|
||||||
|
# Delete existing
|
||||||
|
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
|
||||||
|
|
||||||
|
# Add new
|
||||||
|
for device_id in data.device_ids:
|
||||||
|
mapping = DBChannelDeviceMapping(
|
||||||
|
channel_id=channel.id,
|
||||||
|
device_id=device_id,
|
||||||
|
)
|
||||||
|
db.add(mapping)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deleting a Channel
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def delete_channel(channel_id: str, db: Session) -> bool:
|
||||||
|
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||||
|
|
||||||
|
# Cannot delete default channel
|
||||||
|
if channel.is_default:
|
||||||
|
raise ValueError("Cannot delete the default channel")
|
||||||
|
|
||||||
|
# Delete device mappings
|
||||||
|
db.query(DBChannelDeviceMapping).filter_by(channel_id=channel_id).delete()
|
||||||
|
|
||||||
|
# Delete profile-channel mappings
|
||||||
|
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
|
||||||
|
|
||||||
|
# Delete channel
|
||||||
|
db.delete(channel)
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Voice Assignment
|
||||||
|
|
||||||
|
### Assigning Voices to Channel
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def set_channel_voices(
|
||||||
|
channel_id: str,
|
||||||
|
data: ChannelVoiceAssignment,
|
||||||
|
db: Session,
|
||||||
|
) -> None:
|
||||||
|
# Verify channel exists
|
||||||
|
channel = db.query(DBAudioChannel).filter_by(id=channel_id).first()
|
||||||
|
if not channel:
|
||||||
|
raise ValueError(f"Channel {channel_id} not found")
|
||||||
|
|
||||||
|
# Verify all profiles exist
|
||||||
|
for profile_id in data.profile_ids:
|
||||||
|
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise ValueError(f"Profile {profile_id} not found")
|
||||||
|
|
||||||
|
# Delete existing mappings
|
||||||
|
db.query(DBProfileChannelMapping).filter_by(channel_id=channel_id).delete()
|
||||||
|
|
||||||
|
# Add new mappings
|
||||||
|
for profile_id in data.profile_ids:
|
||||||
|
mapping = DBProfileChannelMapping(
|
||||||
|
profile_id=profile_id,
|
||||||
|
channel_id=channel_id,
|
||||||
|
)
|
||||||
|
db.add(mapping)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Assigning Channels to Voice
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def set_profile_channels(
|
||||||
|
profile_id: str,
|
||||||
|
data: ProfileChannelAssignment,
|
||||||
|
db: Session,
|
||||||
|
) -> None:
|
||||||
|
# Verify profile exists
|
||||||
|
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise ValueError(f"Profile {profile_id} not found")
|
||||||
|
|
||||||
|
# Delete existing mappings
|
||||||
|
db.query(DBProfileChannelMapping).filter_by(profile_id=profile_id).delete()
|
||||||
|
|
||||||
|
# Add new mappings
|
||||||
|
for channel_id in data.channel_ids:
|
||||||
|
mapping = DBProfileChannelMapping(
|
||||||
|
profile_id=profile_id,
|
||||||
|
channel_id=channel_id,
|
||||||
|
)
|
||||||
|
db.add(mapping)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| GET | `/channels` | List all channels |
|
||||||
|
| POST | `/channels` | Create a channel |
|
||||||
|
| GET | `/channels/{id}` | Get channel by ID |
|
||||||
|
| PUT | `/channels/{id}` | Update channel |
|
||||||
|
| DELETE | `/channels/{id}` | Delete channel |
|
||||||
|
| GET | `/channels/{id}/voices` | Get assigned voices |
|
||||||
|
| PUT | `/channels/{id}/voices` | Set assigned voices |
|
||||||
|
| GET | `/profiles/{id}/channels` | Get profile's channels |
|
||||||
|
| PUT | `/profiles/{id}/channels` | Set profile's channels |
|
||||||
|
|
||||||
|
## Request/Response Schemas
|
||||||
|
|
||||||
|
### AudioChannelCreate
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "Speakers",
|
||||||
|
"device_ids": ["device_uuid_1", "device_uuid_2"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### AudioChannelResponse
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "channel_uuid",
|
||||||
|
"name": "Speakers",
|
||||||
|
"is_default": false,
|
||||||
|
"device_ids": ["device_uuid_1", "device_uuid_2"],
|
||||||
|
"created_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### ChannelVoiceAssignment
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"profile_ids": ["profile_1", "profile_2"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Multi-Output Setup
|
||||||
|
|
||||||
|
**Scenario:** Stream with different voice characters
|
||||||
|
|
||||||
|
1. Create "Stream" channel → OBS virtual audio
|
||||||
|
2. Create "Monitor" channel → Headphones
|
||||||
|
3. Assign "Narrator" profile → Both channels
|
||||||
|
4. Assign "Character 1" profile → Stream only
|
||||||
|
|
||||||
|
### Virtual Audio Cables
|
||||||
|
|
||||||
|
Common device IDs for virtual audio:
|
||||||
|
- VB-Audio Virtual Cable
|
||||||
|
- BlackHole (macOS)
|
||||||
|
- Soundflower (macOS)
|
||||||
|
|
||||||
|
## Frontend Integration
|
||||||
|
|
||||||
|
The frontend needs to:
|
||||||
|
|
||||||
|
1. **Enumerate devices** using Web Audio API or Tauri
|
||||||
|
2. **Display channel list** with device assignments
|
||||||
|
3. **Allow profile assignment** via drag/drop or dropdown
|
||||||
|
4. **Route playback** to correct device based on profile's channel
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- Device IDs are OS-specific
|
||||||
|
- Hot-plugging may invalidate device IDs
|
||||||
|
- Default channel cannot be modified/deleted
|
||||||
|
- Frontend handles actual audio routing (backend just stores config)
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
title: "Auto-Updater"
|
||||||
|
description: "Configure and use the Tauri auto-updater"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox uses Tauri's built-in auto-updater to deliver updates to users automatically.
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
For detailed setup instructions, see the existing documentation:
|
||||||
|
|
||||||
|
- [AUTOUPDATER_QUICKSTART.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER_QUICKSTART.md)
|
||||||
|
- [AUTOUPDATER.md](https://github.com/jamiepine/voicebox/blob/main/docs/AUTOUPDATER.md)
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
The auto-updater follows a secure update process:
|
||||||
|
|
||||||
|
1. **Check for Updates** - The Voicebox app periodically checks GitHub Releases for new versions
|
||||||
|
2. **Download Update** - If a new version is found, the update package is downloaded
|
||||||
|
3. **Verify Signature** - The downloaded package is cryptographically verified using the public key
|
||||||
|
4. **Install** - After verification, the update is installed
|
||||||
|
5. **Restart** - The app restarts with the new version
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Updates are configured in `tauri/src-tauri/tauri.conf.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"updater": {
|
||||||
|
"active": true,
|
||||||
|
"endpoints": [
|
||||||
|
"https://github.com/jamiepine/voicebox/releases/latest/download/latest.json"
|
||||||
|
],
|
||||||
|
"dialog": true,
|
||||||
|
"pubkey": "YOUR_PUBLIC_KEY"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generating Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate signing keys
|
||||||
|
bun run generate:keys
|
||||||
|
|
||||||
|
# Keys saved to ~/.tauri/voicebox.key
|
||||||
|
```
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
Keep your private key secure! Never commit it to the repository.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
## Release Process
|
||||||
|
|
||||||
|
1. **Bump version** using bumpversion
|
||||||
|
2. **Push tag** to trigger CI/CD
|
||||||
|
3. **GitHub Actions** builds and signs releases
|
||||||
|
4. **Users** receive update notification
|
||||||
|
|
||||||
|
## User Experience
|
||||||
|
|
||||||
|
When an update is available:
|
||||||
|
|
||||||
|
1. User sees a notification dialog
|
||||||
|
2. User clicks "Update"
|
||||||
|
3. Update downloads in background
|
||||||
|
4. App restarts with new version
|
||||||
|
|
||||||
|
## For Developers
|
||||||
|
|
||||||
|
See the full documentation files for:
|
||||||
|
|
||||||
|
- Setting up signing keys
|
||||||
|
- Configuring GitHub releases
|
||||||
|
- Testing updates locally
|
||||||
|
- Troubleshooting update failures
|
||||||
|
|
||||||
|
<Card title="View Full Docs" href="https://github.com/jamiepine/voicebox/tree/main/docs">
|
||||||
|
Access AUTOUPDATER.md and AUTOUPDATER_QUICKSTART.md in the repository
|
||||||
|
</Card>
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
---
|
||||||
|
title: "Building"
|
||||||
|
description: "Build Voicebox for production"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox uses a multi-step build process to create platform-specific installers.
|
||||||
|
|
||||||
|
## Quick Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build for your current platform (automatically builds server binary first)
|
||||||
|
make build
|
||||||
|
|
||||||
|
# Or manually
|
||||||
|
bun run build
|
||||||
|
```
|
||||||
|
|
||||||
|
This automatically:
|
||||||
|
1. Builds the Python server binary (`bun run build:server`)
|
||||||
|
2. Builds the Tauri app (`cd tauri && bun run tauri build`)
|
||||||
|
|
||||||
|
## Build Process
|
||||||
|
|
||||||
|
The build process consists of two steps, but `bun run build` handles both automatically:
|
||||||
|
|
||||||
|
### 1. Server Binary Build (Automatic)
|
||||||
|
|
||||||
|
The Python backend is compiled into a standalone executable using PyInstaller. This happens automatically when you run `bun run build`.
|
||||||
|
|
||||||
|
**Platform-specific binaries:**
|
||||||
|
- macOS (Apple Silicon): `voicebox-server-aarch64-apple-darwin` (includes MLX backend)
|
||||||
|
- macOS (Intel): `voicebox-server-x86_64-apple-darwin` (PyTorch backend)
|
||||||
|
- Windows: `voicebox-server-x86_64-pc-windows-msvc.exe` (PyTorch backend)
|
||||||
|
- Linux: `voicebox-server-x86_64-unknown-linux-gnu` (PyTorch backend)
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
The build script automatically detects your platform and includes the appropriate backend (MLX for Apple Silicon, PyTorch for others).
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
**Manual build (if needed):**
|
||||||
|
```bash
|
||||||
|
bun run build:server
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Tauri App Build (Automatic)
|
||||||
|
|
||||||
|
The Tauri app build is also handled automatically, which:
|
||||||
|
1. Builds the React frontend (Vite)
|
||||||
|
2. Compiles the Rust backend
|
||||||
|
3. Bundles the server binary as a sidecar
|
||||||
|
4. Creates platform-specific installers
|
||||||
|
|
||||||
|
**Manual build (if needed):**
|
||||||
|
```bash
|
||||||
|
cd tauri && bun run tauri build
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Output
|
||||||
|
|
||||||
|
Installers are created in `tauri/src-tauri/target/release/bundle/`:
|
||||||
|
|
||||||
|
**macOS:**
|
||||||
|
- `dmg/` - Disk image installer
|
||||||
|
- `macos/` - App bundle
|
||||||
|
|
||||||
|
**Windows:**
|
||||||
|
- `msi/` - MSI installer
|
||||||
|
- `nsis/` - NSIS installer
|
||||||
|
|
||||||
|
**Linux:**
|
||||||
|
- `deb/` - Debian package
|
||||||
|
- `appimage/` - AppImage
|
||||||
|
|
||||||
|
## Advanced Options
|
||||||
|
|
||||||
|
### Building for Specific Platform
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build for macOS (Apple Silicon)
|
||||||
|
bun run tauri build -- --target aarch64-apple-darwin
|
||||||
|
|
||||||
|
# Build for macOS (Intel)
|
||||||
|
bun run tauri build -- --target x86_64-apple-darwin
|
||||||
|
|
||||||
|
# Build for Windows
|
||||||
|
bun run tauri build -- --target x86_64-pc-windows-msvc
|
||||||
|
|
||||||
|
# Build for Linux
|
||||||
|
bun run tauri build -- --target x86_64-unknown-linux-gnu
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Local Qwen3-TTS
|
||||||
|
|
||||||
|
If you're developing Qwen3-TTS locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export QWEN_TTS_PATH=~/path/to/Qwen3-TTS
|
||||||
|
bun run build:server # Build server binary only
|
||||||
|
# or
|
||||||
|
bun run build # Build everything
|
||||||
|
```
|
||||||
|
|
||||||
|
This makes PyInstaller use your local version instead of the pip package.
|
||||||
|
|
||||||
|
### Debug Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd tauri
|
||||||
|
bun run tauri build --debug
|
||||||
|
```
|
||||||
|
|
||||||
|
Creates a debug build with symbols and logging.
|
||||||
|
|
||||||
|
## Build Configuration
|
||||||
|
|
||||||
|
### Tauri Config
|
||||||
|
|
||||||
|
Edit `tauri/src-tauri/tauri.conf.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"bundle": {
|
||||||
|
"identifier": "com.voicebox.app",
|
||||||
|
"icon": [
|
||||||
|
"icons/32x32.png",
|
||||||
|
"icons/128x128.png",
|
||||||
|
"icons/icon.icns",
|
||||||
|
"icons/icon.ico"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sidecar Configuration
|
||||||
|
|
||||||
|
The Python server is bundled as a sidecar:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"tauri": {
|
||||||
|
"bundle": {
|
||||||
|
"externalBin": [
|
||||||
|
"binaries/voicebox-server"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code Signing
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
|
||||||
|
To sign the app for distribution:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set signing identity
|
||||||
|
export APPLE_SIGNING_IDENTITY="Developer ID Application: Your Name"
|
||||||
|
|
||||||
|
# Build with signing
|
||||||
|
bun run tauri build
|
||||||
|
```
|
||||||
|
|
||||||
|
For notarization:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set credentials
|
||||||
|
export APPLE_ID="[email protected]"
|
||||||
|
export APPLE_PASSWORD="app-specific-password"
|
||||||
|
|
||||||
|
# Build and notarize
|
||||||
|
bun run tauri build
|
||||||
|
```
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
|
||||||
|
For Windows code signing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set certificate
|
||||||
|
export WINDOWS_CERTIFICATE_PATH="/path/to/cert.pfx"
|
||||||
|
export WINDOWS_CERTIFICATE_PASSWORD="password"
|
||||||
|
|
||||||
|
# Build with signing
|
||||||
|
bun run tauri build
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release Process
|
||||||
|
|
||||||
|
The full release process is automated:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Bump version
|
||||||
|
bumpversion patch # or minor/major
|
||||||
|
|
||||||
|
# 2. Build all platforms (CI/CD handles this)
|
||||||
|
git push --tags
|
||||||
|
|
||||||
|
# 3. GitHub Actions creates releases
|
||||||
|
```
|
||||||
|
|
||||||
|
See [CONTRIBUTING.md](/development/contributing) for the full release workflow.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
<AccordionGroup>
|
||||||
|
<Accordion title="Server Binary Build Fails">
|
||||||
|
**Common issues:**
|
||||||
|
- Missing Python dependencies: `pip install -r requirements.txt`
|
||||||
|
- PyInstaller not found: `pip install pyinstaller`
|
||||||
|
- Qwen3-TTS not installed: `pip install git+https://github.com/QwenLM/Qwen3-TTS.git`
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
pip install pyinstaller
|
||||||
|
```
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Tauri Build Fails">
|
||||||
|
**Common issues:**
|
||||||
|
- Rust not installed: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
|
||||||
|
- Server binary missing: Usually auto-built, but can run manually: `./scripts/build-server.sh`
|
||||||
|
- Node modules outdated: `bun install`
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```bash
|
||||||
|
# Clean and rebuild
|
||||||
|
cd tauri/src-tauri
|
||||||
|
cargo clean
|
||||||
|
cd ../..
|
||||||
|
bun run build # Automatically builds server binary first
|
||||||
|
```
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="App Won't Launch After Build">
|
||||||
|
**Check:**
|
||||||
|
- Server binary has execute permissions
|
||||||
|
- All dependencies are bundled
|
||||||
|
- Check logs in the app's data directory
|
||||||
|
|
||||||
|
**macOS:**
|
||||||
|
```bash
|
||||||
|
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
|
||||||
|
```
|
||||||
|
|
||||||
|
**Windows:**
|
||||||
|
```bash
|
||||||
|
type %APPDATA%\com.voicebox.app\logs\server.log
|
||||||
|
```
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
|
## CI/CD
|
||||||
|
|
||||||
|
GitHub Actions automatically builds releases when tags are pushed:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .github/workflows/release.yml
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- 'v*'
|
||||||
|
```
|
||||||
|
|
||||||
|
See the [repository](https://github.com/jamiepine/voicebox) for the full CI/CD configuration.
|
||||||
@@ -0,0 +1,326 @@
|
|||||||
|
---
|
||||||
|
title: "Contributing"
|
||||||
|
description: "How to contribute to Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
Thank you for your interest in contributing to Voicebox! This guide will help you get started.
|
||||||
|
|
||||||
|
## Code of Conduct
|
||||||
|
|
||||||
|
- Be respectful and inclusive
|
||||||
|
- Welcome newcomers and help them learn
|
||||||
|
- Focus on constructive feedback
|
||||||
|
- Respect different viewpoints and experiences
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
Before you start contributing, make sure you have:
|
||||||
|
|
||||||
|
1. **Read the documentation** to understand how Voicebox works
|
||||||
|
2. **Set up your development environment** - see [Development Setup](/development/setup)
|
||||||
|
3. **Explored the codebase** to understand the project structure
|
||||||
|
4. **Checked existing issues** to see if someone else is working on something similar
|
||||||
|
|
||||||
|
## Ways to Contribute
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Report Bugs" icon="bug">
|
||||||
|
Found a bug? Open an issue with reproduction steps
|
||||||
|
</Card>
|
||||||
|
<Card title="Request Features" icon="lightbulb">
|
||||||
|
Have an idea? Start a discussion or open an issue
|
||||||
|
</Card>
|
||||||
|
<Card title="Improve Docs" icon="book">
|
||||||
|
Fix typos, add examples, or clarify instructions
|
||||||
|
</Card>
|
||||||
|
<Card title="Write Code" icon="code">
|
||||||
|
Fix bugs, add features, or optimize performance
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Development Workflow
|
||||||
|
|
||||||
|
### 1. Fork & Clone
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fork the repository on GitHub
|
||||||
|
# Then clone your fork
|
||||||
|
git clone https://github.com/YOUR_USERNAME/voicebox.git
|
||||||
|
cd voicebox
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Create a Branch
|
||||||
|
|
||||||
|
Use descriptive branch names:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For features
|
||||||
|
git checkout -b feature/voice-effects
|
||||||
|
|
||||||
|
# For bug fixes
|
||||||
|
git checkout -b fix/audio-playback-issue
|
||||||
|
|
||||||
|
# For documentation
|
||||||
|
git checkout -b docs/api-examples
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Make Your Changes
|
||||||
|
|
||||||
|
Follow these guidelines:
|
||||||
|
|
||||||
|
<AccordionGroup>
|
||||||
|
<Accordion title="Code Style">
|
||||||
|
**TypeScript/React:**
|
||||||
|
- Use TypeScript strict mode
|
||||||
|
- Prefer functional components with hooks
|
||||||
|
- Use named exports
|
||||||
|
- Format with Biome (runs automatically)
|
||||||
|
|
||||||
|
**Python:**
|
||||||
|
- Follow PEP 8
|
||||||
|
- Use type hints
|
||||||
|
- Use async/await for I/O
|
||||||
|
- Document functions with docstrings
|
||||||
|
|
||||||
|
**Rust:**
|
||||||
|
- Follow Rust conventions
|
||||||
|
- Use meaningful names
|
||||||
|
- Handle errors explicitly
|
||||||
|
- Run `rustfmt`
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Commit Messages">
|
||||||
|
Write clear, descriptive commit messages:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Good
|
||||||
|
git commit -m "Add voice profile export feature"
|
||||||
|
git commit -m "Fix audio playback stopping after 30 seconds"
|
||||||
|
|
||||||
|
# Avoid
|
||||||
|
git commit -m "Update code"
|
||||||
|
git commit -m "Fix bug"
|
||||||
|
```
|
||||||
|
|
||||||
|
Format:
|
||||||
|
- Use imperative mood ("Add feature" not "Added feature")
|
||||||
|
- Keep first line under 50 characters
|
||||||
|
- Add detailed description if needed
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Testing">
|
||||||
|
- Test your changes manually in the app
|
||||||
|
- Ensure backend API endpoints work
|
||||||
|
- Check for TypeScript/Python errors
|
||||||
|
- Verify UI components render correctly
|
||||||
|
- Add automated tests when possible
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
|
### 4. Push & Create PR
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Push your branch
|
||||||
|
git push origin feature/your-feature-name
|
||||||
|
|
||||||
|
# Then create a pull request on GitHub
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pull Request Guidelines
|
||||||
|
|
||||||
|
When creating a pull request:
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Use a Clear Title">
|
||||||
|
Examples:
|
||||||
|
- "Add voice profile export functionality"
|
||||||
|
- "Fix audio playback stopping after 30 seconds"
|
||||||
|
- "Improve generation speed with caching"
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Provide Description">
|
||||||
|
Include:
|
||||||
|
- What changes you made
|
||||||
|
- Why you made them
|
||||||
|
- How to test them
|
||||||
|
- Screenshots (for UI changes)
|
||||||
|
- Reference related issues
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Update Documentation">
|
||||||
|
- Update relevant docs if behavior changes
|
||||||
|
- Add API documentation for new endpoints
|
||||||
|
- Update README if needed
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Check the Checklist">
|
||||||
|
- [ ] Code follows style guidelines
|
||||||
|
- [ ] Documentation updated
|
||||||
|
- [ ] Changes tested
|
||||||
|
- [ ] No breaking changes (or documented)
|
||||||
|
- [ ] CHANGELOG.md updated
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
Understanding the codebase:
|
||||||
|
|
||||||
|
```
|
||||||
|
voicebox/
|
||||||
|
├── app/ # Shared React frontend
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── components/ # UI components
|
||||||
|
│ │ ├── lib/ # Utilities and API client
|
||||||
|
│ │ ├── hooks/ # React hooks
|
||||||
|
│ │ └── stores/ # Zustand state stores
|
||||||
|
├── backend/ # Python FastAPI server
|
||||||
|
│ ├── main.py # API routes
|
||||||
|
│ ├── tts.py # Voice synthesis logic
|
||||||
|
│ ├── database.py # SQLite operations
|
||||||
|
│ └── models.py # Pydantic models
|
||||||
|
├── tauri/ # Desktop app wrapper
|
||||||
|
│ └── src-tauri/ # Rust backend
|
||||||
|
├── web/ # Web deployment
|
||||||
|
├── landing/ # Marketing website
|
||||||
|
└── scripts/ # Build & release scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Areas for Contribution
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
- Check [existing issues](https://github.com/jamiepine/voicebox/issues) for bugs
|
||||||
|
- Test your fix thoroughly
|
||||||
|
- Add regression tests if possible
|
||||||
|
|
||||||
|
### New Features
|
||||||
|
|
||||||
|
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
|
||||||
|
- Discuss major features in an issue first
|
||||||
|
- Keep features focused and well-scoped
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
- Improve clarity and fix typos
|
||||||
|
- Add code examples
|
||||||
|
- Create tutorials or guides
|
||||||
|
- Document API endpoints
|
||||||
|
|
||||||
|
### UI/UX Improvements
|
||||||
|
|
||||||
|
- Improve accessibility
|
||||||
|
- Enhance visual design
|
||||||
|
- Optimize performance
|
||||||
|
- Add animations/transitions
|
||||||
|
|
||||||
|
### Infrastructure
|
||||||
|
|
||||||
|
- Improve build process
|
||||||
|
- Add CI/CD improvements
|
||||||
|
- Optimize bundle size
|
||||||
|
- Add testing infrastructure
|
||||||
|
|
||||||
|
## API Development
|
||||||
|
|
||||||
|
When adding new API endpoints:
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Add Route">
|
||||||
|
In `backend/main.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.post("/api/new-endpoint")
|
||||||
|
async def new_endpoint(data: RequestModel) -> ResponseModel:
|
||||||
|
"""Endpoint description."""
|
||||||
|
# Implementation
|
||||||
|
return response
|
||||||
|
```
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Create Models">
|
||||||
|
In `backend/models.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class RequestModel(BaseModel):
|
||||||
|
field: str
|
||||||
|
|
||||||
|
class ResponseModel(BaseModel):
|
||||||
|
result: str
|
||||||
|
```
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Regenerate Client">
|
||||||
|
```bash
|
||||||
|
bun run generate:api
|
||||||
|
```
|
||||||
|
|
||||||
|
This updates the TypeScript client with type-safe bindings.
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Update Docs">
|
||||||
|
Add documentation in `/docs/api/`
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Currently testing is primarily manual. When adding tests:
|
||||||
|
|
||||||
|
**Backend:**
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Frontend:**
|
||||||
|
```bash
|
||||||
|
bun run test
|
||||||
|
```
|
||||||
|
|
||||||
|
**E2E (future):**
|
||||||
|
```bash
|
||||||
|
bun run test:e2e
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release Process
|
||||||
|
|
||||||
|
Releases are managed by maintainers using `bumpversion`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Bump version (patch, minor, or major)
|
||||||
|
bumpversion patch
|
||||||
|
|
||||||
|
# Push with tags
|
||||||
|
git push && git push --tags
|
||||||
|
```
|
||||||
|
|
||||||
|
GitHub Actions automatically builds and publishes releases when tags are pushed.
|
||||||
|
|
||||||
|
## Community
|
||||||
|
|
||||||
|
- **GitHub Issues:** Bug reports and feature requests
|
||||||
|
- **GitHub Discussions:** General questions and ideas
|
||||||
|
- **Discord:** Real-time chat (coming soon)
|
||||||
|
|
||||||
|
## Recognition
|
||||||
|
|
||||||
|
Contributors are recognized in:
|
||||||
|
- [CHANGELOG.md](https://github.com/jamiepine/voicebox/blob/main/CHANGELOG.md)
|
||||||
|
- GitHub contributor list
|
||||||
|
- Release notes
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
By contributing, you agree that your contributions will be licensed under the MIT License.
|
||||||
|
|
||||||
|
## Questions?
|
||||||
|
|
||||||
|
If you have questions:
|
||||||
|
|
||||||
|
1. Check the [documentation](/overview/introduction)
|
||||||
|
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
|
||||||
|
3. Open a new issue or discussion
|
||||||
|
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
|
||||||
|
|
||||||
|
Thank you for contributing to Voicebox! 🎉
|
||||||
@@ -0,0 +1,260 @@
|
|||||||
|
---
|
||||||
|
title: "Generation History"
|
||||||
|
description: "How generation history tracking works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The history module tracks all generated audio, providing a searchable record of past generations. Each generation stores the text, settings, and a reference to the audio file.
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### Generation Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Generation(Base):
|
||||||
|
__tablename__ = "generations"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||||
|
text = Column(Text, nullable=False)
|
||||||
|
language = Column(String, default="en")
|
||||||
|
audio_path = Column(String, nullable=False)
|
||||||
|
duration = Column(Float, nullable=False)
|
||||||
|
seed = Column(Integer)
|
||||||
|
instruct = Column(Text)
|
||||||
|
created_at = Column(DateTime)
|
||||||
|
```
|
||||||
|
|
||||||
|
## File Storage
|
||||||
|
|
||||||
|
Generated audio is stored in:
|
||||||
|
|
||||||
|
```
|
||||||
|
data/
|
||||||
|
└── generations/
|
||||||
|
└── {generation_id}.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Functions
|
||||||
|
|
||||||
|
### Creating a Generation Record
|
||||||
|
|
||||||
|
After TTS generates audio, a history entry is created:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_generation(
|
||||||
|
profile_id: str,
|
||||||
|
text: str,
|
||||||
|
language: str,
|
||||||
|
audio_path: str,
|
||||||
|
duration: float,
|
||||||
|
seed: Optional[int],
|
||||||
|
db: Session,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> GenerationResponse:
|
||||||
|
db_generation = DBGeneration(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
profile_id=profile_id,
|
||||||
|
text=text,
|
||||||
|
language=language,
|
||||||
|
audio_path=audio_path,
|
||||||
|
duration=duration,
|
||||||
|
seed=seed,
|
||||||
|
instruct=instruct,
|
||||||
|
created_at=datetime.utcnow(),
|
||||||
|
)
|
||||||
|
|
||||||
|
db.add(db_generation)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
return GenerationResponse.model_validate(db_generation)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Listing Generations
|
||||||
|
|
||||||
|
Supports filtering and pagination:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def list_generations(
|
||||||
|
query: HistoryQuery,
|
||||||
|
db: Session,
|
||||||
|
) -> HistoryListResponse:
|
||||||
|
# Build query with profile name join
|
||||||
|
q = db.query(
|
||||||
|
DBGeneration,
|
||||||
|
DBVoiceProfile.name.label('profile_name')
|
||||||
|
).join(
|
||||||
|
DBVoiceProfile,
|
||||||
|
DBGeneration.profile_id == DBVoiceProfile.id
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply filters
|
||||||
|
if query.profile_id:
|
||||||
|
q = q.filter(DBGeneration.profile_id == query.profile_id)
|
||||||
|
|
||||||
|
if query.search:
|
||||||
|
q = q.filter(DBGeneration.text.like(f"%{query.search}%"))
|
||||||
|
|
||||||
|
# Order and paginate
|
||||||
|
total = q.count()
|
||||||
|
q = q.order_by(DBGeneration.created_at.desc())
|
||||||
|
q = q.offset(query.offset).limit(query.limit)
|
||||||
|
|
||||||
|
return HistoryListResponse(items=results, total=total)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Getting Statistics
|
||||||
|
|
||||||
|
Aggregate statistics for the dashboard:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def get_generation_stats(db: Session) -> dict:
|
||||||
|
total = db.query(func.count(DBGeneration.id)).scalar()
|
||||||
|
total_duration = db.query(func.sum(DBGeneration.duration)).scalar()
|
||||||
|
|
||||||
|
by_profile = db.query(
|
||||||
|
DBGeneration.profile_id,
|
||||||
|
func.count(DBGeneration.id).label('count')
|
||||||
|
).group_by(DBGeneration.profile_id).all()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"total_generations": total,
|
||||||
|
"total_duration_seconds": total_duration,
|
||||||
|
"generations_by_profile": {
|
||||||
|
profile_id: count for profile_id, count in by_profile
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deletion
|
||||||
|
|
||||||
|
Deleting a generation removes both the database record and audio file:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def delete_generation(generation_id: str, db: Session) -> bool:
|
||||||
|
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
|
||||||
|
if not generation:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Delete audio file
|
||||||
|
audio_path = Path(generation.audio_path)
|
||||||
|
if audio_path.exists():
|
||||||
|
audio_path.unlink()
|
||||||
|
|
||||||
|
# Delete database record
|
||||||
|
db.delete(generation)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
return True
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cascade Delete
|
||||||
|
|
||||||
|
When deleting a profile, all its generations are also deleted:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def delete_generations_by_profile(profile_id: str, db: Session) -> int:
|
||||||
|
generations = db.query(DBGeneration).filter_by(profile_id=profile_id).all()
|
||||||
|
|
||||||
|
for generation in generations:
|
||||||
|
Path(generation.audio_path).unlink(missing_ok=True)
|
||||||
|
db.delete(generation)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
return len(generations)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Export/Import
|
||||||
|
|
||||||
|
### Exporting a Generation
|
||||||
|
|
||||||
|
Generations can be exported as ZIP archives:
|
||||||
|
|
||||||
|
```
|
||||||
|
generation_export.zip
|
||||||
|
├── generation.json # Metadata
|
||||||
|
└── audio.wav # Audio file
|
||||||
|
```
|
||||||
|
|
||||||
|
### Importing a Generation
|
||||||
|
|
||||||
|
The import process:
|
||||||
|
|
||||||
|
1. Extract ZIP archive
|
||||||
|
2. Validate metadata and audio
|
||||||
|
3. Create new generation ID
|
||||||
|
4. Copy audio to generations directory
|
||||||
|
5. Create database record
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| GET | `/history` | List generations with filters |
|
||||||
|
| GET | `/history/stats` | Get aggregate statistics |
|
||||||
|
| GET | `/history/{id}` | Get generation by ID |
|
||||||
|
| DELETE | `/history/{id}` | Delete generation |
|
||||||
|
| GET | `/history/{id}/export` | Export as ZIP |
|
||||||
|
| GET | `/history/{id}/export-audio` | Export audio only |
|
||||||
|
| POST | `/history/import` | Import from ZIP |
|
||||||
|
|
||||||
|
### Query Parameters
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /history?profile_id=uuid&search=hello&limit=50&offset=0
|
||||||
|
```
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|-----------|------|---------|-------------|
|
||||||
|
| `profile_id` | string | null | Filter by profile |
|
||||||
|
| `search` | string | null | Search in text |
|
||||||
|
| `limit` | int | 50 | Results per page |
|
||||||
|
| `offset` | int | 0 | Pagination offset |
|
||||||
|
|
||||||
|
### Response Schema
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"id": "uuid",
|
||||||
|
"profile_id": "uuid",
|
||||||
|
"profile_name": "My Voice",
|
||||||
|
"text": "Hello world",
|
||||||
|
"language": "en",
|
||||||
|
"audio_path": "/path/to/audio.wav",
|
||||||
|
"duration": 1.5,
|
||||||
|
"seed": 42,
|
||||||
|
"instruct": null,
|
||||||
|
"created_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total": 150
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage in Stories
|
||||||
|
|
||||||
|
Generations can be added to stories for multi-voice narratives. The story system references generations by ID:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class StoryItem(Base):
|
||||||
|
generation_id = Column(String, ForeignKey("generations.id"))
|
||||||
|
```
|
||||||
|
|
||||||
|
This allows the same generation to be reused across multiple stories without duplicating audio files.
|
||||||
|
|
||||||
|
## Storage Considerations
|
||||||
|
|
||||||
|
### Disk Usage
|
||||||
|
|
||||||
|
Each generation creates a WAV file. For a 10-second clip at 24kHz:
|
||||||
|
- ~480KB per file (mono, 16-bit)
|
||||||
|
|
||||||
|
### Cleanup Strategy
|
||||||
|
|
||||||
|
Consider implementing:
|
||||||
|
- Automatic cleanup of old generations
|
||||||
|
- Storage quota per profile
|
||||||
|
- Compression for archival
|
||||||
@@ -0,0 +1,341 @@
|
|||||||
|
---
|
||||||
|
title: "Model Management"
|
||||||
|
description: "How model downloading, loading, and status tracking works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox manages two types of models:
|
||||||
|
|
||||||
|
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
|
||||||
|
|
||||||
|
**ASR Models:** Whisper for transcription (tiny through large).
|
||||||
|
|
||||||
|
Models are downloaded from HuggingFace Hub on first use and cached locally.
|
||||||
|
|
||||||
|
## Available Models
|
||||||
|
|
||||||
|
### TTS Models
|
||||||
|
|
||||||
|
| Model | HuggingFace ID | Size | VRAM |
|
||||||
|
|-------|----------------|------|------|
|
||||||
|
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
|
||||||
|
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
|
||||||
|
|
||||||
|
### Whisper Models
|
||||||
|
|
||||||
|
| Model | HuggingFace ID | Size | VRAM |
|
||||||
|
|-------|----------------|------|------|
|
||||||
|
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
|
||||||
|
| base | `openai/whisper-base` | ~300MB | ~1GB |
|
||||||
|
| small | `openai/whisper-small` | ~500MB | ~2GB |
|
||||||
|
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
|
||||||
|
| large | `openai/whisper-large` | ~3GB | ~10GB |
|
||||||
|
|
||||||
|
## Model Storage
|
||||||
|
|
||||||
|
Models are cached in the HuggingFace cache directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.cache/huggingface/hub/
|
||||||
|
├── models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/
|
||||||
|
├── models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/
|
||||||
|
├── models--openai--whisper-base/
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Progress Tracking
|
||||||
|
|
||||||
|
### Progress Manager
|
||||||
|
|
||||||
|
Tracks download progress across all models:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ProgressManager:
|
||||||
|
def __init__(self):
|
||||||
|
self._progress = {} # model_name -> progress_info
|
||||||
|
|
||||||
|
def update_progress(
|
||||||
|
self,
|
||||||
|
model_name: str,
|
||||||
|
current: int,
|
||||||
|
total: int,
|
||||||
|
filename: str,
|
||||||
|
status: str,
|
||||||
|
):
|
||||||
|
self._progress[model_name] = {
|
||||||
|
"current": current,
|
||||||
|
"total": total,
|
||||||
|
"filename": filename,
|
||||||
|
"status": status, # downloading, complete, error
|
||||||
|
"updated_at": datetime.utcnow(),
|
||||||
|
}
|
||||||
|
|
||||||
|
def get_progress(self, model_name: str) -> Optional[dict]:
|
||||||
|
return self._progress.get(model_name)
|
||||||
|
```
|
||||||
|
|
||||||
|
### HuggingFace Progress Callback
|
||||||
|
|
||||||
|
Hooks into HuggingFace's download system:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class HFProgressTracker:
|
||||||
|
def __init__(self, callback):
|
||||||
|
self.callback = callback
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def patch_download(self):
|
||||||
|
"""Context manager to intercept HF downloads."""
|
||||||
|
original_download = hf_hub_download
|
||||||
|
|
||||||
|
def patched_download(*args, **kwargs):
|
||||||
|
# Intercept progress
|
||||||
|
result = original_download(*args, **kwargs)
|
||||||
|
self.callback(progress_info)
|
||||||
|
return result
|
||||||
|
|
||||||
|
# Apply patch
|
||||||
|
with patch('huggingface_hub.hf_hub_download', patched_download):
|
||||||
|
yield
|
||||||
|
```
|
||||||
|
|
||||||
|
### Server-Sent Events (SSE)
|
||||||
|
|
||||||
|
Progress is streamed to the frontend:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.get("/models/progress/{model_name}")
|
||||||
|
async def get_model_progress(model_name: str):
|
||||||
|
async def event_generator():
|
||||||
|
while True:
|
||||||
|
progress = progress_manager.get_progress(model_name)
|
||||||
|
if progress:
|
||||||
|
yield f"data: {json.dumps(progress)}\n\n"
|
||||||
|
|
||||||
|
if progress and progress["status"] in ["complete", "error"]:
|
||||||
|
break
|
||||||
|
|
||||||
|
await asyncio.sleep(0.5)
|
||||||
|
|
||||||
|
return StreamingResponse(
|
||||||
|
event_generator(),
|
||||||
|
media_type="text/event-stream"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Task Manager
|
||||||
|
|
||||||
|
Tracks active downloads and generations:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class TaskManager:
|
||||||
|
def __init__(self):
|
||||||
|
self._active_downloads = {}
|
||||||
|
self._active_generations = {}
|
||||||
|
|
||||||
|
def start_download(self, model_name: str):
|
||||||
|
self._active_downloads[model_name] = {
|
||||||
|
"status": "downloading",
|
||||||
|
"started_at": datetime.utcnow(),
|
||||||
|
}
|
||||||
|
|
||||||
|
def complete_download(self, model_name: str):
|
||||||
|
if model_name in self._active_downloads:
|
||||||
|
del self._active_downloads[model_name]
|
||||||
|
|
||||||
|
def get_active_tasks(self) -> dict:
|
||||||
|
return {
|
||||||
|
"downloads": list(self._active_downloads.values()),
|
||||||
|
"generations": list(self._active_generations.values()),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Model Status
|
||||||
|
|
||||||
|
Check which models are downloaded and loaded:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.get("/models/status")
|
||||||
|
async def get_model_status() -> ModelStatusListResponse:
|
||||||
|
models = []
|
||||||
|
|
||||||
|
# Check TTS models
|
||||||
|
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
|
||||||
|
downloaded = is_model_downloaded(hf_id)
|
||||||
|
loaded = tts_model._current_model_size == size
|
||||||
|
|
||||||
|
models.append(ModelStatus(
|
||||||
|
model_name=f"qwen-tts-{size}",
|
||||||
|
display_name=f"Qwen3-TTS {size}",
|
||||||
|
downloaded=downloaded,
|
||||||
|
size_mb=get_model_size_mb(hf_id),
|
||||||
|
loaded=loaded,
|
||||||
|
))
|
||||||
|
|
||||||
|
# Check Whisper models
|
||||||
|
for size in ["tiny", "base", "small", "medium", "large"]:
|
||||||
|
hf_id = f"openai/whisper-{size}"
|
||||||
|
downloaded = is_model_downloaded(hf_id)
|
||||||
|
|
||||||
|
models.append(ModelStatus(
|
||||||
|
model_name=f"whisper-{size}",
|
||||||
|
display_name=f"Whisper {size}",
|
||||||
|
downloaded=downloaded,
|
||||||
|
size_mb=get_model_size_mb(hf_id),
|
||||||
|
loaded=False, # Whisper is loaded on-demand
|
||||||
|
))
|
||||||
|
|
||||||
|
return ModelStatusListResponse(models=models)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual Model Operations
|
||||||
|
|
||||||
|
### Load Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.post("/models/load")
|
||||||
|
async def load_model(model_size: str = "1.7B"):
|
||||||
|
tts_model = get_tts_model()
|
||||||
|
await tts_model.load_model_async(model_size)
|
||||||
|
return {"status": "loaded", "model_size": model_size}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unload Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.post("/models/unload")
|
||||||
|
async def unload_model():
|
||||||
|
tts_model = get_tts_model()
|
||||||
|
tts_model.unload_model()
|
||||||
|
return {"status": "unloaded"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Trigger Download
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.post("/models/download")
|
||||||
|
async def trigger_model_download(request: ModelDownloadRequest):
|
||||||
|
# This triggers the download in background
|
||||||
|
# Progress is tracked via /models/progress/{model_name}
|
||||||
|
|
||||||
|
if request.model_name.startswith("qwen-tts"):
|
||||||
|
size = request.model_name.split("-")[-1]
|
||||||
|
asyncio.create_task(download_tts_model(size))
|
||||||
|
elif request.model_name.startswith("whisper"):
|
||||||
|
size = request.model_name.split("-")[-1]
|
||||||
|
asyncio.create_task(download_whisper_model(size))
|
||||||
|
|
||||||
|
return {"status": "downloading"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.delete("/models/{model_name}")
|
||||||
|
async def delete_model(model_name: str):
|
||||||
|
# Find and delete from HuggingFace cache
|
||||||
|
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
|
||||||
|
|
||||||
|
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
|
||||||
|
for model_dir in model_dirs:
|
||||||
|
shutil.rmtree(model_dir)
|
||||||
|
|
||||||
|
return {"status": "deleted"}
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| GET | `/models/status` | Get status of all models |
|
||||||
|
| POST | `/models/load` | Load TTS model |
|
||||||
|
| POST | `/models/unload` | Unload TTS model |
|
||||||
|
| POST | `/models/download` | Trigger model download |
|
||||||
|
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||||
|
| DELETE | `/models/{name}` | Delete downloaded model |
|
||||||
|
| GET | `/tasks/active` | Get active downloads/generations |
|
||||||
|
|
||||||
|
## Response Schemas
|
||||||
|
|
||||||
|
### ModelStatus
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"model_name": "qwen-tts-1.7B",
|
||||||
|
"display_name": "Qwen3-TTS 1.7B",
|
||||||
|
"downloaded": true,
|
||||||
|
"size_mb": 3400,
|
||||||
|
"loaded": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### ActiveTasksResponse
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"downloads": [
|
||||||
|
{
|
||||||
|
"model_name": "whisper-medium",
|
||||||
|
"status": "downloading",
|
||||||
|
"started_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"generations": [
|
||||||
|
{
|
||||||
|
"task_id": "uuid",
|
||||||
|
"profile_id": "uuid",
|
||||||
|
"text_preview": "Hello world...",
|
||||||
|
"started_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Frontend Integration
|
||||||
|
|
||||||
|
### Progress Display
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Subscribe to download progress via SSE
|
||||||
|
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||||
|
|
||||||
|
eventSource.onmessage = (event) => {
|
||||||
|
const progress = JSON.parse(event.data);
|
||||||
|
updateProgressBar(progress.current / progress.total);
|
||||||
|
|
||||||
|
if (progress.status === 'complete') {
|
||||||
|
eventSource.close();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### Model Status UI
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Fetch model status
|
||||||
|
const { data: models } = useQuery({
|
||||||
|
queryKey: ['models', 'status'],
|
||||||
|
queryFn: () => api.getModelStatus(),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Display download/load buttons based on status
|
||||||
|
models.map(model => (
|
||||||
|
<ModelCard
|
||||||
|
name={model.display_name}
|
||||||
|
downloaded={model.downloaded}
|
||||||
|
loaded={model.loaded}
|
||||||
|
onDownload={() => triggerDownload(model.model_name)}
|
||||||
|
onLoad={() => loadModel(model.model_name)}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Error | Cause | Solution |
|
||||||
|
|-------|-------|----------|
|
||||||
|
| Download failed | Network issue | Retry download |
|
||||||
|
| OOM on load | Model too large | Use smaller model |
|
||||||
|
| Model not found | Cache corrupted | Re-download |
|
||||||
|
| Slow download | HF rate limit | Wait and retry |
|
||||||
@@ -0,0 +1,239 @@
|
|||||||
|
---
|
||||||
|
title: "Development Setup"
|
||||||
|
description: "Set up your local development environment for Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before you begin, ensure you have the following installed:
|
||||||
|
|
||||||
|
<CardGroup cols={3}>
|
||||||
|
<Card title="Bun" icon="package">
|
||||||
|
[Download Bun](https://bun.sh)
|
||||||
|
```bash
|
||||||
|
curl -fsSL https://bun.sh/install | bash
|
||||||
|
```
|
||||||
|
</Card>
|
||||||
|
<Card title="Python 3.11+" icon="python">
|
||||||
|
[Download Python](https://python.org)
|
||||||
|
```bash
|
||||||
|
python --version
|
||||||
|
```
|
||||||
|
</Card>
|
||||||
|
<Card title="Rust" icon="rust">
|
||||||
|
[Install Rust](https://rustup.rs)
|
||||||
|
```bash
|
||||||
|
rustc --version
|
||||||
|
```
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Clone the Repository
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/jamiepine/voicebox.git
|
||||||
|
cd voicebox
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Setup (Recommended)
|
||||||
|
|
||||||
|
The easiest way to get started is using the Makefile:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Setup everything
|
||||||
|
make setup
|
||||||
|
|
||||||
|
# Start development
|
||||||
|
make dev
|
||||||
|
```
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
The Makefile is available on macOS and Linux. Windows users should follow the manual setup below.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Manual Setup
|
||||||
|
|
||||||
|
### 1. Install JavaScript Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun install
|
||||||
|
```
|
||||||
|
|
||||||
|
This installs dependencies for:
|
||||||
|
- `app/` - Shared React frontend
|
||||||
|
- `tauri/` - Tauri desktop wrapper
|
||||||
|
- `web/` - Web deployment wrapper
|
||||||
|
|
||||||
|
### 2. Set Up Python Backend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
|
||||||
|
# Create virtual environment
|
||||||
|
python -m venv venv
|
||||||
|
|
||||||
|
# Activate virtual environment
|
||||||
|
source venv/bin/activate # macOS/Linux
|
||||||
|
# or
|
||||||
|
venv\Scripts\activate # Windows
|
||||||
|
|
||||||
|
# Install Python dependencies
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# Install MLX dependencies (Apple Silicon only - for faster inference)
|
||||||
|
# On Apple Silicon, this enables native Metal acceleration
|
||||||
|
if [[ $(uname -m) == "arm64" ]]; then
|
||||||
|
pip install -r requirements-mlx.txt
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Install Qwen3-TTS
|
||||||
|
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||||
|
```
|
||||||
|
|
||||||
|
## Running in Development
|
||||||
|
|
||||||
|
Development requires **two terminals**: one for the Python backend, one for the Tauri app.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<Tab title="Terminal 1: Backend">
|
||||||
|
Start the Python server first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
source venv/bin/activate # Activate venv
|
||||||
|
bun run dev:server
|
||||||
|
```
|
||||||
|
|
||||||
|
Or manually:
|
||||||
|
```bash
|
||||||
|
uvicorn main:app --reload --port 17493
|
||||||
|
```
|
||||||
|
|
||||||
|
Backend will be available at `http://localhost:17493`
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab title="Terminal 2: Desktop App">
|
||||||
|
Then start the Tauri app:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
This will:
|
||||||
|
- Create a placeholder sidecar binary
|
||||||
|
- Start Vite dev server on port 5173
|
||||||
|
- Launch Tauri window
|
||||||
|
- Enable hot reload
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
<Info>
|
||||||
|
In dev mode, the app connects to your manually-started Python server. The bundled server binary is only used in production builds.
|
||||||
|
</Info>
|
||||||
|
|
||||||
|
### Optional: Web App
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run dev:web
|
||||||
|
```
|
||||||
|
|
||||||
|
Web app will be available at `http://localhost:5174`
|
||||||
|
|
||||||
|
## Model Downloads
|
||||||
|
|
||||||
|
Models are automatically downloaded from HuggingFace Hub on first use:
|
||||||
|
|
||||||
|
- **Whisper** (transcription): Auto-downloads on first transcription
|
||||||
|
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
voicebox/
|
||||||
|
├── app/ # Shared React frontend
|
||||||
|
│ └── src/
|
||||||
|
│ ├── components/ # UI components
|
||||||
|
│ ├── lib/ # Utilities and API client
|
||||||
|
│ └── hooks/ # React hooks
|
||||||
|
├── backend/ # Python FastAPI server
|
||||||
|
│ ├── main.py # API routes
|
||||||
|
│ ├── tts.py # Voice synthesis
|
||||||
|
│ └── database.py # SQLite operations
|
||||||
|
├── tauri/ # Desktop app wrapper
|
||||||
|
│ └── src-tauri/ # Rust backend
|
||||||
|
├── web/ # Web deployment
|
||||||
|
├── landing/ # Marketing website
|
||||||
|
└── scripts/ # Build & release scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Available Make Commands
|
||||||
|
|
||||||
|
Run `make help` to see all available commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make setup # Install all dependencies
|
||||||
|
make dev # Start development servers
|
||||||
|
make dev-web # Start web development server
|
||||||
|
make build # Build desktop app
|
||||||
|
make build-web # Build web app
|
||||||
|
make clean # Clean build artifacts
|
||||||
|
make test # Run tests
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generate OpenAPI Client
|
||||||
|
|
||||||
|
After starting the backend server, generate the TypeScript API client:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/generate-api.sh
|
||||||
|
# or
|
||||||
|
bun run generate:api
|
||||||
|
```
|
||||||
|
|
||||||
|
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Architecture" icon="diagram-project" href="/development/architecture">
|
||||||
|
Understand the system architecture
|
||||||
|
</Card>
|
||||||
|
<Card title="Contributing" icon="code-pull-request" href="/development/contributing">
|
||||||
|
Read the contribution guidelines
|
||||||
|
</Card>
|
||||||
|
<Card title="Building" icon="hammer" href="/development/building">
|
||||||
|
Learn how to build production releases
|
||||||
|
</Card>
|
||||||
|
<Card title="API Reference" icon="code" href="/api/overview">
|
||||||
|
Explore the REST API
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
<AccordionGroup>
|
||||||
|
<Accordion title="Backend won't start">
|
||||||
|
- Check Python version (must be 3.11+)
|
||||||
|
- Ensure virtual environment is activated
|
||||||
|
- Verify all dependencies are installed: `pip install -r requirements.txt`
|
||||||
|
- Check if port 17493 is available
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Tauri build fails">
|
||||||
|
- Ensure Rust is installed: `rustc --version`
|
||||||
|
- Clean the build: `cd tauri/src-tauri && cargo clean`
|
||||||
|
- Try rebuilding: `bun run dev`
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="OpenAPI client generation fails">
|
||||||
|
- Ensure backend is running: `curl http://localhost:17493/openapi.json`
|
||||||
|
- Check network connectivity
|
||||||
|
- Verify the backend is accessible at localhost:17493
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
|
See the full [Troubleshooting Guide](/guides/troubleshooting) for more issues and solutions.
|
||||||
@@ -0,0 +1,320 @@
|
|||||||
|
---
|
||||||
|
title: "Stories & Timeline"
|
||||||
|
description: "How the multi-voice timeline editor works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Stories allow users to arrange multiple voice generations on a timeline to create multi-voice narratives. The system supports tracks, trimming, splitting, and audio mixing.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
**Story:** A container that holds story items with metadata.
|
||||||
|
|
||||||
|
**Story Item:** Links a generation to a story with timeline position, track, and trim data.
|
||||||
|
|
||||||
|
**Export:** Combines all items into a single mixed audio file.
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### Story Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Story(Base):
|
||||||
|
__tablename__ = "stories"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
name = Column(String, nullable=False)
|
||||||
|
description = Column(Text)
|
||||||
|
created_at = Column(DateTime)
|
||||||
|
updated_at = Column(DateTime)
|
||||||
|
```
|
||||||
|
|
||||||
|
### StoryItem Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class StoryItem(Base):
|
||||||
|
__tablename__ = "story_items"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
story_id = Column(String, ForeignKey("stories.id"))
|
||||||
|
generation_id = Column(String, ForeignKey("generations.id"))
|
||||||
|
start_time_ms = Column(Integer, default=0) # Timeline position
|
||||||
|
track = Column(Integer, default=0) # Track number
|
||||||
|
trim_start_ms = Column(Integer, default=0) # Trim from start
|
||||||
|
trim_end_ms = Column(Integer, default=0) # Trim from end
|
||||||
|
created_at = Column(DateTime)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Timeline Concepts
|
||||||
|
|
||||||
|
### Start Time
|
||||||
|
|
||||||
|
`start_time_ms` defines when an item begins on the timeline:
|
||||||
|
|
||||||
|
```
|
||||||
|
Timeline (ms): 0----1000----2000----3000----4000
|
||||||
|
Item 1: [======]
|
||||||
|
Item 2: [==========]
|
||||||
|
Item 3: [====]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tracks
|
||||||
|
|
||||||
|
Multiple tracks allow overlapping audio:
|
||||||
|
|
||||||
|
```
|
||||||
|
Track 0: [Item 1] [Item 3]
|
||||||
|
Track 1: [Item 2]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Trimming
|
||||||
|
|
||||||
|
Trim values cut audio from the start or end without destroying the original:
|
||||||
|
|
||||||
|
```
|
||||||
|
Original: [=========AUDIO=========]
|
||||||
|
trim_start: ^^
|
||||||
|
trim_end: ^^
|
||||||
|
Result: [=====AUDIO=====]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Operations
|
||||||
|
|
||||||
|
### Adding Items
|
||||||
|
|
||||||
|
When adding a generation to a story:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def add_item_to_story(
|
||||||
|
story_id: str,
|
||||||
|
data: StoryItemCreate,
|
||||||
|
db: Session,
|
||||||
|
) -> StoryItemDetail:
|
||||||
|
# Calculate start time if not provided
|
||||||
|
if data.start_time_ms is None:
|
||||||
|
# Find the end of all existing items
|
||||||
|
existing_items = get_items_with_durations(story_id, db)
|
||||||
|
max_end_time_ms = max(
|
||||||
|
item.start_time_ms + int(gen.duration * 1000)
|
||||||
|
for item, gen in existing_items
|
||||||
|
)
|
||||||
|
start_time_ms = max_end_time_ms + 200 # 200ms gap
|
||||||
|
|
||||||
|
# Create the item
|
||||||
|
item = DBStoryItem(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
story_id=story_id,
|
||||||
|
generation_id=data.generation_id,
|
||||||
|
start_time_ms=start_time_ms,
|
||||||
|
track=data.track or 0,
|
||||||
|
)
|
||||||
|
db.add(item)
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Moving Items
|
||||||
|
|
||||||
|
Update position and/or track:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def move_story_item(
|
||||||
|
story_id: str,
|
||||||
|
item_id: str,
|
||||||
|
data: StoryItemMove,
|
||||||
|
db: Session,
|
||||||
|
) -> StoryItemDetail:
|
||||||
|
item = get_item(story_id, item_id, db)
|
||||||
|
|
||||||
|
item.start_time_ms = data.start_time_ms
|
||||||
|
item.track = data.track
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Trimming Items
|
||||||
|
|
||||||
|
Non-destructive trimming:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def trim_story_item(
|
||||||
|
story_id: str,
|
||||||
|
item_id: str,
|
||||||
|
data: StoryItemTrim,
|
||||||
|
db: Session,
|
||||||
|
) -> StoryItemDetail:
|
||||||
|
item = get_item(story_id, item_id, db)
|
||||||
|
generation = get_generation(item.generation_id, db)
|
||||||
|
|
||||||
|
# Validate trim doesn't exceed duration
|
||||||
|
max_duration_ms = int(generation.duration * 1000)
|
||||||
|
if data.trim_start_ms + data.trim_end_ms >= max_duration_ms:
|
||||||
|
return None # Invalid trim
|
||||||
|
|
||||||
|
item.trim_start_ms = data.trim_start_ms
|
||||||
|
item.trim_end_ms = data.trim_end_ms
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Splitting Items
|
||||||
|
|
||||||
|
Split one item into two at a specific time:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def split_story_item(
|
||||||
|
story_id: str,
|
||||||
|
item_id: str,
|
||||||
|
data: StoryItemSplit,
|
||||||
|
db: Session,
|
||||||
|
) -> List[StoryItemDetail]:
|
||||||
|
item = get_item(story_id, item_id, db)
|
||||||
|
generation = get_generation(item.generation_id, db)
|
||||||
|
|
||||||
|
# Calculate split point
|
||||||
|
current_trim_start = item.trim_start_ms
|
||||||
|
current_trim_end = item.trim_end_ms
|
||||||
|
original_duration_ms = int(generation.duration * 1000)
|
||||||
|
absolute_split_ms = current_trim_start + data.split_time_ms
|
||||||
|
|
||||||
|
# Update original: trim from end
|
||||||
|
item.trim_end_ms = original_duration_ms - absolute_split_ms
|
||||||
|
|
||||||
|
# Create new item: trim from start
|
||||||
|
new_item = DBStoryItem(
|
||||||
|
generation_id=item.generation_id, # Same generation
|
||||||
|
start_time_ms=item.start_time_ms + data.split_time_ms,
|
||||||
|
track=item.track,
|
||||||
|
trim_start_ms=absolute_split_ms,
|
||||||
|
trim_end_ms=current_trim_end,
|
||||||
|
)
|
||||||
|
|
||||||
|
db.add(new_item)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
return [item, new_item]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Duplicating Items
|
||||||
|
|
||||||
|
Create a copy with all properties:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def duplicate_story_item(
|
||||||
|
story_id: str,
|
||||||
|
item_id: str,
|
||||||
|
db: Session,
|
||||||
|
) -> StoryItemDetail:
|
||||||
|
original = get_item(story_id, item_id, db)
|
||||||
|
generation = get_generation(original.generation_id, db)
|
||||||
|
|
||||||
|
# Calculate effective duration for positioning
|
||||||
|
effective_duration_ms = (
|
||||||
|
int(generation.duration * 1000)
|
||||||
|
- original.trim_start_ms
|
||||||
|
- original.trim_end_ms
|
||||||
|
)
|
||||||
|
|
||||||
|
# Place copy after original with 200ms gap
|
||||||
|
new_item = DBStoryItem(
|
||||||
|
generation_id=original.generation_id,
|
||||||
|
start_time_ms=original.start_time_ms + effective_duration_ms + 200,
|
||||||
|
track=original.track,
|
||||||
|
trim_start_ms=original.trim_start_ms,
|
||||||
|
trim_end_ms=original.trim_end_ms,
|
||||||
|
)
|
||||||
|
|
||||||
|
db.add(new_item)
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audio Export
|
||||||
|
|
||||||
|
### Mixing Algorithm
|
||||||
|
|
||||||
|
The export function mixes all items into a single audio file:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def export_story_audio(story_id: str, db: Session) -> bytes:
|
||||||
|
items = get_all_items_with_generations(story_id, db)
|
||||||
|
|
||||||
|
# Calculate total duration
|
||||||
|
max_end_time_ms = max(
|
||||||
|
data['start_time_ms'] + data['duration_ms']
|
||||||
|
for data in audio_data
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create output buffer
|
||||||
|
total_samples = int((max_end_time_ms / 1000.0) * sample_rate)
|
||||||
|
final_audio = np.zeros(total_samples, dtype=np.float32)
|
||||||
|
|
||||||
|
# Mix each item at its position
|
||||||
|
for data in audio_data:
|
||||||
|
audio = data['audio']
|
||||||
|
start_sample = int((data['start_time_ms'] / 1000.0) * sample_rate)
|
||||||
|
|
||||||
|
# Apply trim
|
||||||
|
trimmed_audio = audio[trim_start_sample:len(audio) - trim_end_sample]
|
||||||
|
|
||||||
|
# Add to buffer (overlapping items sum together)
|
||||||
|
final_audio[start_sample:start_sample + len(trimmed_audio)] += trimmed_audio
|
||||||
|
|
||||||
|
# Normalize to prevent clipping
|
||||||
|
max_val = np.abs(final_audio).max()
|
||||||
|
if max_val > 1.0:
|
||||||
|
final_audio = final_audio / max_val
|
||||||
|
|
||||||
|
return audio_to_bytes(final_audio, sample_rate)
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| GET | `/stories` | List all stories |
|
||||||
|
| POST | `/stories` | Create a story |
|
||||||
|
| GET | `/stories/{id}` | Get story with items |
|
||||||
|
| PUT | `/stories/{id}` | Update story metadata |
|
||||||
|
| DELETE | `/stories/{id}` | Delete story |
|
||||||
|
| POST | `/stories/{id}/items` | Add item to story |
|
||||||
|
| DELETE | `/stories/{id}/items/{item_id}` | Remove item |
|
||||||
|
| PUT | `/stories/{id}/items/{item_id}/move` | Move item |
|
||||||
|
| PUT | `/stories/{id}/items/{item_id}/trim` | Trim item |
|
||||||
|
| POST | `/stories/{id}/items/{item_id}/split` | Split item |
|
||||||
|
| POST | `/stories/{id}/items/{item_id}/duplicate` | Duplicate item |
|
||||||
|
| PUT | `/stories/{id}/items/times` | Batch update times |
|
||||||
|
| PUT | `/stories/{id}/items/reorder` | Reorder items |
|
||||||
|
| GET | `/stories/{id}/export-audio` | Export mixed audio |
|
||||||
|
|
||||||
|
## Response Schemas
|
||||||
|
|
||||||
|
### StoryItemDetail
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "item_uuid",
|
||||||
|
"story_id": "story_uuid",
|
||||||
|
"generation_id": "generation_uuid",
|
||||||
|
"start_time_ms": 1500,
|
||||||
|
"track": 0,
|
||||||
|
"trim_start_ms": 200,
|
||||||
|
"trim_end_ms": 100,
|
||||||
|
"profile_id": "profile_uuid",
|
||||||
|
"profile_name": "Narrator",
|
||||||
|
"text": "Hello world",
|
||||||
|
"audio_path": "/path/to/audio.wav",
|
||||||
|
"duration": 2.5,
|
||||||
|
"created_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Frontend Integration
|
||||||
|
|
||||||
|
The timeline UI needs to:
|
||||||
|
|
||||||
|
1. **Fetch story** with all items
|
||||||
|
2. **Render waveforms** for each item
|
||||||
|
3. **Handle drag/drop** to move items
|
||||||
|
4. **Handle edge drag** for trimming
|
||||||
|
5. **Sync playhead** across all tracks
|
||||||
|
6. **Export** when user clicks download
|
||||||
@@ -0,0 +1,299 @@
|
|||||||
|
---
|
||||||
|
title: "Transcription"
|
||||||
|
description: "How Whisper-based audio transcription works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The transcription system is built around the `WhisperModel` class:
|
||||||
|
|
||||||
|
**Model Loading:** Lazy loading with HuggingFace Hub download.
|
||||||
|
|
||||||
|
**Audio Processing:** Resampling and preprocessing for Whisper.
|
||||||
|
|
||||||
|
**Inference:** Running transcription with optional language hints.
|
||||||
|
|
||||||
|
## WhisperModel Class
|
||||||
|
|
||||||
|
```python
|
||||||
|
class WhisperModel:
|
||||||
|
def __init__(self, model_size: str = "base"):
|
||||||
|
self.model = None
|
||||||
|
self.processor = None
|
||||||
|
self.model_size = model_size
|
||||||
|
self.device = self._get_device()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Model Sizes
|
||||||
|
|
||||||
|
| Size | Parameters | VRAM | Speed | Quality |
|
||||||
|
|------|------------|------|-------|---------|
|
||||||
|
| tiny | 39M | ~1GB | Fastest | Basic |
|
||||||
|
| base | 74M | ~1GB | Fast | Good |
|
||||||
|
| small | 244M | ~2GB | Medium | Better |
|
||||||
|
| medium | 769M | ~5GB | Slow | High |
|
||||||
|
| large | 1550M | ~10GB | Slowest | Best |
|
||||||
|
|
||||||
|
Default is `base` for balance of speed and quality.
|
||||||
|
|
||||||
|
## Model Loading
|
||||||
|
|
||||||
|
Models are downloaded from HuggingFace Hub:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def load_model(self, model_size: Optional[str] = None):
|
||||||
|
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||||
|
|
||||||
|
model_name = f"openai/whisper-{model_size}"
|
||||||
|
|
||||||
|
# Track download progress
|
||||||
|
progress_manager = get_progress_manager()
|
||||||
|
task_manager = get_task_manager()
|
||||||
|
task_manager.start_download(f"whisper-{model_size}")
|
||||||
|
|
||||||
|
# Load processor and model
|
||||||
|
with tracker.patch_download():
|
||||||
|
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||||
|
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||||
|
|
||||||
|
self.model.to(self.device)
|
||||||
|
|
||||||
|
# Mark complete
|
||||||
|
progress_manager.mark_complete(f"whisper-{model_size}")
|
||||||
|
task_manager.complete_download(f"whisper-{model_size}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Async Loading
|
||||||
|
|
||||||
|
Like TTS, loading runs in a thread pool:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
if self.model is not None and self.model_size == model_size:
|
||||||
|
return
|
||||||
|
await asyncio.to_thread(self.load_model, model_size)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Transcription
|
||||||
|
|
||||||
|
### Basic Transcription
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def transcribe(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
language: Optional[str] = None,
|
||||||
|
) -> str:
|
||||||
|
await self.load_model_async()
|
||||||
|
|
||||||
|
def _transcribe_sync():
|
||||||
|
# Load and resample to 16kHz (Whisper requirement)
|
||||||
|
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||||
|
|
||||||
|
# Process audio
|
||||||
|
inputs = self.processor(
|
||||||
|
audio,
|
||||||
|
sampling_rate=16000,
|
||||||
|
return_tensors="pt",
|
||||||
|
)
|
||||||
|
inputs = inputs.to(self.device)
|
||||||
|
|
||||||
|
# Set language hint if provided
|
||||||
|
forced_decoder_ids = None
|
||||||
|
if language:
|
||||||
|
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||||
|
language=language,
|
||||||
|
task="transcribe",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Generate
|
||||||
|
with torch.no_grad():
|
||||||
|
predicted_ids = self.model.generate(
|
||||||
|
inputs["input_features"],
|
||||||
|
forced_decoder_ids=forced_decoder_ids,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Decode
|
||||||
|
transcription = self.processor.batch_decode(
|
||||||
|
predicted_ids,
|
||||||
|
skip_special_tokens=True,
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
return transcription.strip()
|
||||||
|
|
||||||
|
return await asyncio.to_thread(_transcribe_sync)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Supported Languages
|
||||||
|
|
||||||
|
Whisper supports 99+ languages. Common ones in Voicebox:
|
||||||
|
|
||||||
|
| Code | Language |
|
||||||
|
|------|----------|
|
||||||
|
| en | English |
|
||||||
|
| zh | Chinese |
|
||||||
|
| ja | Japanese |
|
||||||
|
| ko | Korean |
|
||||||
|
| de | German |
|
||||||
|
| fr | French |
|
||||||
|
| ru | Russian |
|
||||||
|
| pt | Portuguese |
|
||||||
|
| es | Spanish |
|
||||||
|
| it | Italian |
|
||||||
|
|
||||||
|
### Language Detection
|
||||||
|
|
||||||
|
When no language is specified, Whisper auto-detects:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Without language hint - auto-detect
|
||||||
|
transcription = await whisper.transcribe(audio_path)
|
||||||
|
|
||||||
|
# With language hint - more accurate for short clips
|
||||||
|
transcription = await whisper.transcribe(audio_path, language="en")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Transcription with Timestamps
|
||||||
|
|
||||||
|
For advanced use cases, word-level timestamps are available:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def transcribe_with_timestamps(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
language: Optional[str] = None,
|
||||||
|
) -> List[Dict[str, any]]:
|
||||||
|
await self.load_model_async()
|
||||||
|
|
||||||
|
def _transcribe_timestamps_sync():
|
||||||
|
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||||
|
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
predicted_ids = self.model.generate(
|
||||||
|
inputs["input_features"],
|
||||||
|
return_timestamps=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Parse timestamps
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"text": transcription,
|
||||||
|
"start": 0.0,
|
||||||
|
"end": len(audio) / sr,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
return await asyncio.to_thread(_transcribe_timestamps_sync)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Memory Management
|
||||||
|
|
||||||
|
### Unloading
|
||||||
|
|
||||||
|
Free memory when not needed:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def unload_model(self):
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
del self.processor
|
||||||
|
self.model = None
|
||||||
|
self.processor = None
|
||||||
|
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Global Instance
|
||||||
|
|
||||||
|
A singleton pattern manages the model:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_whisper_model: Optional[WhisperModel] = None
|
||||||
|
|
||||||
|
def get_whisper_model() -> WhisperModel:
|
||||||
|
global _whisper_model
|
||||||
|
if _whisper_model is None:
|
||||||
|
_whisper_model = WhisperModel()
|
||||||
|
return _whisper_model
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audio Preprocessing
|
||||||
|
|
||||||
|
### Resampling
|
||||||
|
|
||||||
|
Whisper requires 16kHz audio:
|
||||||
|
|
||||||
|
```python
|
||||||
|
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Format Support
|
||||||
|
|
||||||
|
The `load_audio` utility handles:
|
||||||
|
- WAV
|
||||||
|
- MP3
|
||||||
|
- FLAC
|
||||||
|
- OGG
|
||||||
|
- M4A
|
||||||
|
|
||||||
|
All formats are converted to mono 16kHz.
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| POST | `/transcribe` | Transcribe audio file |
|
||||||
|
|
||||||
|
### Request
|
||||||
|
|
||||||
|
Multipart form data:
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /transcribe
|
||||||
|
Content-Type: multipart/form-data
|
||||||
|
|
||||||
|
file: <audio_file>
|
||||||
|
language: en (optional)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Response
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"text": "Hello, this is a test transcription.",
|
||||||
|
"duration": 3.5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Reference Text for Voice Cloning
|
||||||
|
|
||||||
|
1. User records audio sample
|
||||||
|
2. Audio is sent to `/transcribe`
|
||||||
|
3. Transcription becomes `reference_text`
|
||||||
|
4. Both are added to voice profile
|
||||||
|
|
||||||
|
### Quality Tips
|
||||||
|
|
||||||
|
- Provide language hint for short audio
|
||||||
|
- Use clean audio with minimal noise
|
||||||
|
- Longer audio (>5s) improves accuracy
|
||||||
|
- Consider `small` or `medium` model for better quality
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
Common issues:
|
||||||
|
|
||||||
|
| Error | Cause | Solution |
|
||||||
|
|-------|-------|----------|
|
||||||
|
| Model not found | First run, download failed | Retry with network |
|
||||||
|
| OOM | Model too large | Use smaller model |
|
||||||
|
| Empty result | No speech detected | Check audio has speech |
|
||||||
|
| Wrong language | Auto-detect failed | Provide language hint |
|
||||||
@@ -0,0 +1,283 @@
|
|||||||
|
---
|
||||||
|
title: "TTS Generation"
|
||||||
|
description: "How text-to-speech generation works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The TTS system is built around the `TTSModel` class which manages:
|
||||||
|
|
||||||
|
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
|
||||||
|
|
||||||
|
**Voice Prompts:** Converting reference audio into embeddings.
|
||||||
|
|
||||||
|
**Generation:** Synthesizing speech from text using voice prompts.
|
||||||
|
|
||||||
|
## TTSModel Class
|
||||||
|
|
||||||
|
```python
|
||||||
|
class TTSModel:
|
||||||
|
def __init__(self, model_size: str = "1.7B"):
|
||||||
|
self.model = None
|
||||||
|
self.model_size = model_size
|
||||||
|
self.device = self._get_device() # cuda, mps, or cpu
|
||||||
|
```
|
||||||
|
|
||||||
|
### Device Selection
|
||||||
|
|
||||||
|
The model automatically selects the best available device:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _get_device(self) -> str:
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
return "cuda"
|
||||||
|
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||||
|
return "cpu" # MPS can have issues, use CPU for stability
|
||||||
|
return "cpu"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Model Loading
|
||||||
|
|
||||||
|
Models are downloaded from HuggingFace Hub on first use:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def load_model(self, model_size: Optional[str] = None):
|
||||||
|
# Model IDs on HuggingFace Hub
|
||||||
|
hf_model_map = {
|
||||||
|
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||||
|
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Load with progress tracking
|
||||||
|
with tracker.patch_download():
|
||||||
|
self.model = Qwen3TTSModel.from_pretrained(
|
||||||
|
model_path,
|
||||||
|
device_map=self.device,
|
||||||
|
torch_dtype=torch.bfloat16, # float32 on CPU
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Async Loading
|
||||||
|
|
||||||
|
Loading runs in a thread pool to avoid blocking the event loop:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def load_model_async(self, model_size: Optional[str] = None):
|
||||||
|
if self.model is not None and self._current_model_size == model_size:
|
||||||
|
return
|
||||||
|
await asyncio.to_thread(self.load_model, model_size)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Voice Prompt Creation
|
||||||
|
|
||||||
|
Voice prompts are created from reference audio and cached for reuse:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_voice_prompt(
|
||||||
|
self,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
use_cache: bool = True,
|
||||||
|
) -> Tuple[dict, bool]:
|
||||||
|
await self.load_model_async()
|
||||||
|
|
||||||
|
# Check cache
|
||||||
|
if use_cache:
|
||||||
|
cache_key = get_cache_key(audio_path, reference_text)
|
||||||
|
cached = get_cached_voice_prompt(cache_key)
|
||||||
|
if cached:
|
||||||
|
return cached, True
|
||||||
|
|
||||||
|
# Create prompt (blocking, run in thread pool)
|
||||||
|
voice_prompt = await asyncio.to_thread(
|
||||||
|
self.model.create_voice_clone_prompt,
|
||||||
|
ref_audio=audio_path,
|
||||||
|
ref_text=reference_text,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Cache the result
|
||||||
|
cache_voice_prompt(cache_key, voice_prompt)
|
||||||
|
return voice_prompt, False
|
||||||
|
```
|
||||||
|
|
||||||
|
### Combining Multiple Samples
|
||||||
|
|
||||||
|
When a profile has multiple samples, they're combined:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def combine_voice_prompts(
|
||||||
|
self,
|
||||||
|
audio_paths: List[str],
|
||||||
|
reference_texts: List[str],
|
||||||
|
) -> Tuple[np.ndarray, str]:
|
||||||
|
combined_audio = []
|
||||||
|
|
||||||
|
for audio_path in audio_paths:
|
||||||
|
audio, sr = load_audio(audio_path)
|
||||||
|
audio = normalize_audio(audio)
|
||||||
|
combined_audio.append(audio)
|
||||||
|
|
||||||
|
# Concatenate and normalize
|
||||||
|
mixed = np.concatenate(combined_audio)
|
||||||
|
mixed = normalize_audio(mixed)
|
||||||
|
|
||||||
|
# Combine texts
|
||||||
|
combined_text = " ".join(reference_texts)
|
||||||
|
|
||||||
|
return mixed, combined_text
|
||||||
|
```
|
||||||
|
|
||||||
|
## Speech Generation
|
||||||
|
|
||||||
|
The core generation function:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def generate(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
voice_prompt: dict,
|
||||||
|
language: str = "en",
|
||||||
|
seed: Optional[int] = None,
|
||||||
|
instruct: Optional[str] = None,
|
||||||
|
) -> Tuple[np.ndarray, int]:
|
||||||
|
await self.load_model_async()
|
||||||
|
|
||||||
|
def _generate_sync():
|
||||||
|
# Set seed for reproducibility
|
||||||
|
if seed is not None:
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
|
||||||
|
# Generate audio
|
||||||
|
wavs, sample_rate = self.model.generate_voice_clone(
|
||||||
|
text=text,
|
||||||
|
voice_clone_prompt=voice_prompt,
|
||||||
|
instruct=instruct, # Natural language delivery control
|
||||||
|
)
|
||||||
|
return wavs[0], sample_rate
|
||||||
|
|
||||||
|
# Run in thread pool
|
||||||
|
return await asyncio.to_thread(_generate_sync)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Instruct Feature
|
||||||
|
|
||||||
|
The `instruct` parameter allows natural language control over speech delivery:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Examples:
|
||||||
|
instruct = "Speak slowly and clearly"
|
||||||
|
instruct = "Sound excited and enthusiastic"
|
||||||
|
instruct = "Whisper softly"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Caching Strategy
|
||||||
|
|
||||||
|
Voice prompts are cached to avoid recomputation:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||||
|
"""Generate cache key from audio hash and text."""
|
||||||
|
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
|
||||||
|
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
|
||||||
|
return f"{audio_hash}_{text_hash}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Cache is stored in `data/cache/voice_prompts/`.
|
||||||
|
|
||||||
|
## Memory Management
|
||||||
|
|
||||||
|
### Unloading Models
|
||||||
|
|
||||||
|
Free VRAM/RAM when not needed:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def unload_model(self):
|
||||||
|
if self.model is not None:
|
||||||
|
del self.model
|
||||||
|
self.model = None
|
||||||
|
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Model Switching
|
||||||
|
|
||||||
|
When switching between model sizes (1.7B ↔ 0.6B):
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Unload existing model first
|
||||||
|
if self.model is not None and self._current_model_size != model_size:
|
||||||
|
self.unload_model()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generation Flow
|
||||||
|
|
||||||
|
1. **Request** → Validate text and profile ID
|
||||||
|
2. **Profile** → Load profile samples from database
|
||||||
|
3. **Voice Prompt** → Create or retrieve cached prompt
|
||||||
|
4. **Generate** → Run TTS inference
|
||||||
|
5. **Save** → Write audio to generations directory
|
||||||
|
6. **Record** → Create history entry in database
|
||||||
|
7. **Response** → Return audio path and metadata
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| POST | `/generate` | Generate speech from text |
|
||||||
|
| GET | `/audio/{id}` | Serve generated audio file |
|
||||||
|
|
||||||
|
### Request Schema
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"profile_id": "uuid",
|
||||||
|
"text": "Text to synthesize",
|
||||||
|
"language": "en",
|
||||||
|
"seed": 42,
|
||||||
|
"model_size": "1.7B",
|
||||||
|
"instruct": "Speak clearly"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Response Schema
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "generation_uuid",
|
||||||
|
"profile_id": "profile_uuid",
|
||||||
|
"text": "Text to synthesize",
|
||||||
|
"language": "en",
|
||||||
|
"audio_path": "/path/to/audio.wav",
|
||||||
|
"duration": 3.5,
|
||||||
|
"seed": 42,
|
||||||
|
"instruct": "Speak clearly",
|
||||||
|
"created_at": "2024-01-15T10:30:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### GPU Acceleration
|
||||||
|
|
||||||
|
- CUDA provides fastest inference
|
||||||
|
- MPS (Apple Silicon) has stability issues, uses CPU fallback
|
||||||
|
- CPU inference is slower but always works
|
||||||
|
|
||||||
|
### Batch Size
|
||||||
|
|
||||||
|
Currently generates one utterance at a time. For long texts, consider:
|
||||||
|
- Splitting into sentences
|
||||||
|
- Sequential generation
|
||||||
|
- Concatenating results
|
||||||
|
|
||||||
|
### Memory Usage
|
||||||
|
|
||||||
|
| Model | VRAM/RAM Required |
|
||||||
|
|-------|-------------------|
|
||||||
|
| 0.6B | ~2GB |
|
||||||
|
| 1.7B | ~6GB |
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
---
|
||||||
|
title: "Voice Profiles"
|
||||||
|
description: "How voice profile management works in Voicebox"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The voice profile system consists of three main components:
|
||||||
|
|
||||||
|
**Database Layer:** SQLite tables store profile metadata and sample references.
|
||||||
|
|
||||||
|
**File Storage:** Audio samples are stored on disk in a structured directory format.
|
||||||
|
|
||||||
|
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### VoiceProfile Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class VoiceProfile(Base):
|
||||||
|
__tablename__ = "profiles"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
name = Column(String, unique=True, nullable=False)
|
||||||
|
description = Column(Text)
|
||||||
|
language = Column(String, default="en")
|
||||||
|
created_at = Column(DateTime)
|
||||||
|
updated_at = Column(DateTime)
|
||||||
|
```
|
||||||
|
|
||||||
|
### ProfileSample Table
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ProfileSample(Base):
|
||||||
|
__tablename__ = "profile_samples"
|
||||||
|
|
||||||
|
id = Column(String, primary_key=True)
|
||||||
|
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||||
|
audio_path = Column(String, nullable=False)
|
||||||
|
reference_text = Column(Text, nullable=False)
|
||||||
|
```
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
Profiles are stored in the data directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
data/
|
||||||
|
└── profiles/
|
||||||
|
└── {profile_id}/
|
||||||
|
├── {sample_id_1}.wav
|
||||||
|
├── {sample_id_2}.wav
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Functions
|
||||||
|
|
||||||
|
### Creating a Profile
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_profile(data: VoiceProfileCreate, db: Session) -> VoiceProfileResponse:
|
||||||
|
# 1. Create database record
|
||||||
|
db_profile = DBVoiceProfile(
|
||||||
|
id=str(uuid.uuid4()),
|
||||||
|
name=data.name,
|
||||||
|
description=data.description,
|
||||||
|
language=data.language,
|
||||||
|
)
|
||||||
|
db.add(db_profile)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
# 2. Create profile directory
|
||||||
|
profile_dir = profiles_dir / db_profile.id
|
||||||
|
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
return VoiceProfileResponse.model_validate(db_profile)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding Samples
|
||||||
|
|
||||||
|
When a sample is added, the audio is validated and copied to the profile directory:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def add_profile_sample(
|
||||||
|
profile_id: str,
|
||||||
|
audio_path: str,
|
||||||
|
reference_text: str,
|
||||||
|
db: Session,
|
||||||
|
) -> ProfileSampleResponse:
|
||||||
|
# 1. Validate audio (duration, format, quality)
|
||||||
|
is_valid, error_msg = validate_reference_audio(audio_path)
|
||||||
|
if not is_valid:
|
||||||
|
raise ValueError(f"Invalid reference audio: {error_msg}")
|
||||||
|
|
||||||
|
# 2. Copy to profile directory
|
||||||
|
sample_id = str(uuid.uuid4())
|
||||||
|
dest_path = profile_dir / f"{sample_id}.wav"
|
||||||
|
audio, sr = load_audio(audio_path)
|
||||||
|
save_audio(audio, str(dest_path), sr)
|
||||||
|
|
||||||
|
# 3. Create database record
|
||||||
|
db_sample = DBProfileSample(
|
||||||
|
id=sample_id,
|
||||||
|
profile_id=profile_id,
|
||||||
|
audio_path=str(dest_path),
|
||||||
|
reference_text=reference_text,
|
||||||
|
)
|
||||||
|
db.add(db_sample)
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Voice Prompt Creation
|
||||||
|
|
||||||
|
When generating speech, samples are combined into a voice prompt:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_voice_prompt_for_profile(
|
||||||
|
profile_id: str,
|
||||||
|
db: Session,
|
||||||
|
) -> dict:
|
||||||
|
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
|
||||||
|
|
||||||
|
if len(samples) == 1:
|
||||||
|
# Single sample - use directly
|
||||||
|
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||||
|
sample.audio_path,
|
||||||
|
sample.reference_text,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Multiple samples - combine them
|
||||||
|
combined_audio, combined_text = await tts_model.combine_voice_prompts(
|
||||||
|
[s.audio_path for s in samples],
|
||||||
|
[s.reference_text for s in samples],
|
||||||
|
)
|
||||||
|
voice_prompt, _ = await tts_model.create_voice_prompt(
|
||||||
|
combined_audio_path,
|
||||||
|
combined_text,
|
||||||
|
)
|
||||||
|
|
||||||
|
return voice_prompt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audio Validation
|
||||||
|
|
||||||
|
Reference audio is validated before being accepted:
|
||||||
|
|
||||||
|
- **Duration:** 3-30 seconds recommended
|
||||||
|
- **Format:** WAV, MP3, FLAC, OGG supported
|
||||||
|
- **Sample Rate:** Resampled to 24kHz
|
||||||
|
- **Channels:** Converted to mono if stereo
|
||||||
|
|
||||||
|
## Export/Import
|
||||||
|
|
||||||
|
Profiles can be exported as ZIP archives for sharing:
|
||||||
|
|
||||||
|
```
|
||||||
|
profile_export.zip
|
||||||
|
├── profile.json # Metadata
|
||||||
|
├── samples/
|
||||||
|
│ ├── sample_1.wav
|
||||||
|
│ └── sample_1.json # Reference text
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| GET | `/profiles` | List all profiles |
|
||||||
|
| POST | `/profiles` | Create a profile |
|
||||||
|
| GET | `/profiles/{id}` | Get profile by ID |
|
||||||
|
| PUT | `/profiles/{id}` | Update profile |
|
||||||
|
| DELETE | `/profiles/{id}` | Delete profile |
|
||||||
|
| GET | `/profiles/{id}/samples` | Get profile samples |
|
||||||
|
| POST | `/profiles/{id}/samples` | Add sample to profile |
|
||||||
|
| PUT | `/profiles/samples/{id}` | Update sample text |
|
||||||
|
| DELETE | `/profiles/samples/{id}` | Delete sample |
|
||||||
|
| GET | `/profiles/{id}/export` | Export as ZIP |
|
||||||
|
| POST | `/profiles/import` | Import from ZIP |
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### Sample Quality
|
||||||
|
|
||||||
|
- Use clean audio with minimal background noise
|
||||||
|
- Ensure the reference text exactly matches what is spoken
|
||||||
|
- Multiple samples (3-5) improve voice cloning quality
|
||||||
|
|
||||||
|
### Language Matching
|
||||||
|
|
||||||
|
- Set the profile language to match the reference audio
|
||||||
|
- Supported languages: en, zh, ja, ko, de, fr, ru, pt, es, it
|
||||||
|
|
||||||
|
### Naming Conventions
|
||||||
|
|
||||||
|
- Use descriptive names that identify the voice
|
||||||
|
- Avoid special characters that may cause filesystem issues
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 129 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
+102
@@ -0,0 +1,102 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://mintlify.com/schema.json",
|
||||||
|
"name": "Voicebox",
|
||||||
|
"logo": {
|
||||||
|
"light": "/logo/icon-light.png",
|
||||||
|
"dark": "/logo/icon-dark.png"
|
||||||
|
},
|
||||||
|
"favicon": "/favicon.png",
|
||||||
|
"colors": {
|
||||||
|
"primary": "#BF9E40",
|
||||||
|
"light": "#D4B560",
|
||||||
|
"dark": "#A68A35"
|
||||||
|
},
|
||||||
|
"styles": {
|
||||||
|
"css": ["/custom.css"]
|
||||||
|
},
|
||||||
|
"anchors": [
|
||||||
|
{
|
||||||
|
"name": "Overview",
|
||||||
|
"icon": "book-open",
|
||||||
|
"url": "overview"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "API",
|
||||||
|
"icon": "code",
|
||||||
|
"url": "api"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Developer",
|
||||||
|
"icon": "book-open-cover",
|
||||||
|
"url": "developer"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "GitHub",
|
||||||
|
"icon": "github",
|
||||||
|
"url": "https://github.com/jamiepine/voicebox"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"navigation": [
|
||||||
|
{
|
||||||
|
"group": "Getting Started",
|
||||||
|
"icon": "rocket",
|
||||||
|
"pages": ["overview/introduction", "overview/installation", "overview/quick-start"]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"group": "Features",
|
||||||
|
"icon": "sparkles",
|
||||||
|
"pages": [
|
||||||
|
"overview/voice-cloning",
|
||||||
|
"overview/stories-editor",
|
||||||
|
"overview/recording-transcription",
|
||||||
|
"overview/generation-history",
|
||||||
|
"overview/remote-mode"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"group": "User Guides",
|
||||||
|
"icon": "compass",
|
||||||
|
"pages": [
|
||||||
|
"overview/creating-voice-profiles",
|
||||||
|
"overview/generating-speech",
|
||||||
|
"overview/building-stories",
|
||||||
|
"overview/troubleshooting"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"group": "Development",
|
||||||
|
"icon": "wrench",
|
||||||
|
"pages": [
|
||||||
|
"developer/setup",
|
||||||
|
"developer/architecture",
|
||||||
|
"developer/contributing",
|
||||||
|
"developer/building",
|
||||||
|
"developer/autoupdater"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"group": "API Reference",
|
||||||
|
"icon": "code",
|
||||||
|
"pages": [
|
||||||
|
"api/overview",
|
||||||
|
"api/authentication",
|
||||||
|
"api/voice-profiles",
|
||||||
|
"api/generation",
|
||||||
|
"api/recordings"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"group": "Architecture",
|
||||||
|
"icon": "book-open-cover",
|
||||||
|
"pages": [
|
||||||
|
"developer/voice-profiles",
|
||||||
|
"developer/tts-generation",
|
||||||
|
"developer/history",
|
||||||
|
"developer/stories",
|
||||||
|
"developer/transcription",
|
||||||
|
"developer/audio-channels",
|
||||||
|
"developer/model-management"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[phases.setup]
|
||||||
|
nixPkgs = ["nodejs_20", "bun"]
|
||||||
|
|
||||||
|
[phases.install]
|
||||||
|
cmds = ["bun install"]
|
||||||
|
|
||||||
|
[phases.build]
|
||||||
|
cmds = ["bun run build"]
|
||||||
|
|
||||||
|
[start]
|
||||||
|
cmd = "bun run start"
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
---
|
||||||
|
title: "Building Stories"
|
||||||
|
description: "Create multi-voice narratives with the Stories Editor"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
The Stories Editor is perfect for creating podcasts, audiobooks, and multi-speaker content.
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Create Story">
|
||||||
|
**Stories** → **+ New Story**
|
||||||
|
</Step>
|
||||||
|
<Step title="Add Tracks">
|
||||||
|
Create tracks for each speaker
|
||||||
|
</Step>
|
||||||
|
<Step title="Add Clips">
|
||||||
|
Generate or drag audio to tracks
|
||||||
|
</Step>
|
||||||
|
<Step title="Arrange">
|
||||||
|
Position and trim clips on timeline
|
||||||
|
</Step>
|
||||||
|
<Step title="Export">
|
||||||
|
Render final audio
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
- Multi-host podcasts
|
||||||
|
- Audiobook narration with character voices
|
||||||
|
- Game dialogue scenes
|
||||||
|
- Educational content with multiple speakers
|
||||||
|
|
||||||
|
## Coming Soon
|
||||||
|
|
||||||
|
Full timeline editor documentation will be added as features are finalized.
|
||||||
@@ -0,0 +1,296 @@
|
|||||||
|
---
|
||||||
|
title: "Creating Voice Profiles"
|
||||||
|
description: "Advanced guide to creating high-quality voice profiles"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Prepare Audio">
|
||||||
|
10-30 seconds of clear speech
|
||||||
|
</Step>
|
||||||
|
<Step title="Create Profile">
|
||||||
|
**Profiles** → **+ New Profile**
|
||||||
|
</Step>
|
||||||
|
<Step title="Upload Sample">
|
||||||
|
Add your audio file
|
||||||
|
</Step>
|
||||||
|
<Step title="Generate">
|
||||||
|
Use the profile to generate speech
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Audio Requirements
|
||||||
|
|
||||||
|
### Ideal Sample Characteristics
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Duration" icon="clock">
|
||||||
|
**10-30 seconds**
|
||||||
|
|
||||||
|
Too short: Poor quality
|
||||||
|
Too long: Unnecessary
|
||||||
|
</Card>
|
||||||
|
<Card title="Clarity" icon="volume">
|
||||||
|
**Clear speech**
|
||||||
|
|
||||||
|
No background noise
|
||||||
|
No music or overlapping voices
|
||||||
|
</Card>
|
||||||
|
<Card title="Quality" icon="sparkles">
|
||||||
|
**High fidelity**
|
||||||
|
|
||||||
|
44.1kHz or 48kHz sample rate
|
||||||
|
Minimal compression
|
||||||
|
</Card>
|
||||||
|
<Card title="Content" icon="microphone">
|
||||||
|
**Natural speech**
|
||||||
|
|
||||||
|
Conversational tone
|
||||||
|
Complete sentences
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
### File Formats
|
||||||
|
|
||||||
|
Supported formats:
|
||||||
|
- **WAV** (recommended) - Lossless quality
|
||||||
|
- **MP3** - Acceptable, minimal compression
|
||||||
|
- **M4A** - Acceptable
|
||||||
|
- **FLAC** - Lossless alternative
|
||||||
|
|
||||||
|
<Tip>
|
||||||
|
Use WAV for best results. Avoid heavily compressed formats.
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
## Recording Tips
|
||||||
|
|
||||||
|
### Environment
|
||||||
|
|
||||||
|
<AccordionGroup>
|
||||||
|
<Accordion title="Quiet Space">
|
||||||
|
- Record in a quiet room
|
||||||
|
- Turn off fans, AC, appliances
|
||||||
|
- Close windows to reduce outside noise
|
||||||
|
- Use soft furnishings to reduce echo
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Microphone Placement">
|
||||||
|
- 6-12 inches from mouth
|
||||||
|
- Slight angle to reduce plosives (p, b, t)
|
||||||
|
- Use a pop filter if available
|
||||||
|
- Maintain consistent distance
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Recording Settings">
|
||||||
|
- 44.1kHz or 48kHz sample rate
|
||||||
|
- 16-bit or 24-bit depth
|
||||||
|
- Mono is fine (stereo will be converted)
|
||||||
|
- Avoid automatic gain control
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
|
### Speaking
|
||||||
|
|
||||||
|
- **Natural pace** - Don't rush or speak too slowly
|
||||||
|
- **Clear articulation** - Pronounce words clearly
|
||||||
|
- **Consistent volume** - Maintain steady loudness
|
||||||
|
- **Normal tone** - Speak as you normally would
|
||||||
|
- **Complete sentences** - Avoid fragments or "ums"
|
||||||
|
|
||||||
|
## Multiple Samples
|
||||||
|
|
||||||
|
Adding multiple samples can significantly improve quality:
|
||||||
|
|
||||||
|
### Why Multiple Samples?
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Robustness" icon="shield">
|
||||||
|
Model learns a more complete representation
|
||||||
|
</Card>
|
||||||
|
<Card title="Versatility" icon="palette">
|
||||||
|
Handles different speaking styles better
|
||||||
|
</Card>
|
||||||
|
<Card title="Quality" icon="star">
|
||||||
|
Reduces artifacts and improves naturalness
|
||||||
|
</Card>
|
||||||
|
<Card title="Consistency" icon="check">
|
||||||
|
More reliable across different texts
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
### Sample Variety
|
||||||
|
|
||||||
|
Consider adding samples with:
|
||||||
|
|
||||||
|
1. **Different tones**
|
||||||
|
- Casual conversation
|
||||||
|
- Professional/formal
|
||||||
|
- Excited/enthusiastic
|
||||||
|
- Calm/serious
|
||||||
|
|
||||||
|
2. **Different content**
|
||||||
|
- Narratives
|
||||||
|
- Questions
|
||||||
|
- Statements
|
||||||
|
- Emotions (happy, sad, neutral)
|
||||||
|
|
||||||
|
3. **Different recording conditions**
|
||||||
|
- Studio quality
|
||||||
|
- Phone call quality (if needed)
|
||||||
|
- Room acoustics
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
All samples should be from the **same speaker**. Mixing voices will produce poor results.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
## Processing Existing Audio
|
||||||
|
|
||||||
|
If you have existing audio (podcasts, videos, etc.):
|
||||||
|
|
||||||
|
### Extracting Clean Segments
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Find Clean Speech">
|
||||||
|
Look for segments with:
|
||||||
|
- Just the target speaker
|
||||||
|
- No background music
|
||||||
|
- Minimal noise
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Use Audio Editor">
|
||||||
|
Tools like Audacity or Adobe Audition:
|
||||||
|
- Cut out clean 10-30s segments
|
||||||
|
- Remove silence at start/end
|
||||||
|
- Normalize volume if needed
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Export as WAV">
|
||||||
|
Save as high-quality WAV file
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
### Noise Reduction
|
||||||
|
|
||||||
|
If you have light background noise:
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Use noise reduction in Audacity:
|
||||||
|
- Select noise-only section
|
||||||
|
- Get Noise Profile
|
||||||
|
- Select full audio
|
||||||
|
- Apply noise reduction (gentle settings)
|
||||||
|
|
||||||
|
2. Avoid over-processing:
|
||||||
|
- Can introduce artifacts
|
||||||
|
- May reduce voice quality
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Iteration
|
||||||
|
|
||||||
|
### Test Your Profile
|
||||||
|
|
||||||
|
After creating a profile:
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Generate Test">
|
||||||
|
Generate a simple phrase:
|
||||||
|
```
|
||||||
|
"Hello, this is a test of my voice profile."
|
||||||
|
```
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Evaluate Quality">
|
||||||
|
Listen for:
|
||||||
|
- Natural tone
|
||||||
|
- Clear pronunciation
|
||||||
|
- Proper prosody
|
||||||
|
- Lack of artifacts
|
||||||
|
</Step>
|
||||||
|
|
||||||
|
<Step title="Iterate">
|
||||||
|
If quality is poor:
|
||||||
|
- Add more samples
|
||||||
|
- Try different source audio
|
||||||
|
- Check sample quality
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
<AccordionGroup>
|
||||||
|
<Accordion title="Robotic Voice">
|
||||||
|
**Cause**: Poor quality samples or too short
|
||||||
|
|
||||||
|
**Fix**: Use longer, higher quality samples
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Wrong Tone">
|
||||||
|
**Cause**: Sample tone doesn't match desired output
|
||||||
|
|
||||||
|
**Fix**: Record samples in the style you want to generate
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Artifacts/Glitches">
|
||||||
|
**Cause**: Background noise or audio issues in samples
|
||||||
|
|
||||||
|
**Fix**: Clean up samples or re-record in quieter environment
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
|
## Advanced Tips
|
||||||
|
|
||||||
|
### Celebrity/Character Voices
|
||||||
|
|
||||||
|
For cloning public figures or characters:
|
||||||
|
|
||||||
|
1. **Legal considerations** - Ensure you have rights or it's fair use
|
||||||
|
2. **Source quality** - Find high-quality interview audio or clean clips
|
||||||
|
3. **Consistency** - Use clips where they speak similarly
|
||||||
|
4. **Multiple samples** - Very important for recognizable voices
|
||||||
|
|
||||||
|
### Accent & Dialect
|
||||||
|
|
||||||
|
The model will preserve accent and dialect:
|
||||||
|
|
||||||
|
- British English will generate British English
|
||||||
|
- Southern accent will produce Southern accent
|
||||||
|
- Regional pronunciations will be maintained
|
||||||
|
|
||||||
|
### Emotion Transfer
|
||||||
|
|
||||||
|
The emotional tone of samples affects generation:
|
||||||
|
|
||||||
|
- Energetic samples → Energetic output
|
||||||
|
- Calm samples → Calm output
|
||||||
|
- Mix samples for versatile profile
|
||||||
|
|
||||||
|
## Managing Profiles
|
||||||
|
|
||||||
|
### Organization
|
||||||
|
|
||||||
|
- **Descriptive names** - "John Smith - Professional Narrator"
|
||||||
|
- **Add descriptions** - Note recording conditions, use cases
|
||||||
|
- **Language tags** - Mark the primary language
|
||||||
|
- **Archive unused** - Keep profile list manageable
|
||||||
|
|
||||||
|
### Export/Import
|
||||||
|
|
||||||
|
- **Export** profiles to share or backup
|
||||||
|
- **Import** from colleagues or teammates
|
||||||
|
- Profiles include voice embeddings, not original audio
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Generate Speech" icon="waveform" href="/guides/generating-speech">
|
||||||
|
Use your profile to generate speech
|
||||||
|
</Card>
|
||||||
|
<Card title="Build Stories" icon="film" href="/guides/building-stories">
|
||||||
|
Create multi-voice narratives
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
---
|
||||||
|
title: "Generating Speech"
|
||||||
|
description: "Generate high-quality speech from text"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Basic Generation
|
||||||
|
|
||||||
|
<Steps>
|
||||||
|
<Step title="Select Profile">
|
||||||
|
Choose a voice profile from the dropdown
|
||||||
|
</Step>
|
||||||
|
<Step title="Enter Text">
|
||||||
|
Type or paste your text
|
||||||
|
</Step>
|
||||||
|
<Step title="Generate">
|
||||||
|
Click **Generate** and wait a few seconds
|
||||||
|
</Step>
|
||||||
|
<Step title="Play & Export">
|
||||||
|
Preview and download the result
|
||||||
|
</Step>
|
||||||
|
</Steps>
|
||||||
|
|
||||||
|
## Text Formatting Tips
|
||||||
|
|
||||||
|
The way you format text affects the output quality.
|
||||||
|
|
||||||
|
### Punctuation
|
||||||
|
|
||||||
|
Use proper punctuation for natural pauses:
|
||||||
|
|
||||||
|
```
|
||||||
|
Good: "Hello! How are you today? I'm doing great."
|
||||||
|
Bad: "Hello how are you today Im doing great"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Emphasis
|
||||||
|
|
||||||
|
Use formatting to suggest emphasis:
|
||||||
|
|
||||||
|
```
|
||||||
|
- ALL CAPS for louder/emphasized: "That was AMAZING!"
|
||||||
|
- Italics for subtle emphasis: "I *really* enjoyed that"
|
||||||
|
- Bold for strong emphasis: "This is **very** important"
|
||||||
|
```
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
The model interprets these hints but results may vary.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Advanced Features
|
||||||
|
|
||||||
|
### Batch Generation
|
||||||
|
|
||||||
|
For long-form content, split into smaller chunks for better control and faster processing.
|
||||||
|
|
||||||
|
### Voice Caching
|
||||||
|
|
||||||
|
Voicebox caches voice prompts for faster re-generation with the same profile.
|
||||||
|
|
||||||
|
## Coming Soon
|
||||||
|
|
||||||
|
- Real-time streaming
|
||||||
|
- Word-level timing control
|
||||||
|
- Emotion and style controls
|
||||||
|
- SSML support
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
---
|
||||||
|
title: "Generation History"
|
||||||
|
description: "Track and manage all your generated audio"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Voicebox keeps a complete history of all generated audio, making it easy to find, reuse, and manage your creations.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="Full History" icon="clock">
|
||||||
|
Every generation is automatically saved
|
||||||
|
</Card>
|
||||||
|
<Card title="Search & Filter" icon="search">
|
||||||
|
Find by text, voice, or date
|
||||||
|
</Card>
|
||||||
|
<Card title="Re-generate" icon="rotate">
|
||||||
|
Regenerate any past generation with one click
|
||||||
|
</Card>
|
||||||
|
<Card title="Export" icon="download">
|
||||||
|
Download individual or batch exports
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
## Viewing History
|
||||||
|
|
||||||
|
Navigate to the **History** tab to see all your generations.
|
||||||
|
|
||||||
|
Each entry shows:
|
||||||
|
- Generated text
|
||||||
|
- Voice profile used
|
||||||
|
- Timestamp
|
||||||
|
- Audio duration
|
||||||
|
- Language
|
||||||
|
|
||||||
|
## Actions
|
||||||
|
|
||||||
|
### Play
|
||||||
|
Click any generation to play it immediately.
|
||||||
|
|
||||||
|
### Re-generate
|
||||||
|
Regenerate with the same settings or modify the text/voice.
|
||||||
|
|
||||||
|
### Download
|
||||||
|
Export as WAV, MP3, or M4A.
|
||||||
|
|
||||||
|
### Delete
|
||||||
|
Remove unwanted generations to free up space.
|
||||||
|
|
||||||
|
### Add to Story
|
||||||
|
Drag generations to the Stories Editor timeline.
|
||||||
|
|
||||||
|
## Search & Filter
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<Tab title="By Text">
|
||||||
|
Search for specific text content
|
||||||
|
```
|
||||||
|
"Hello world"
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="By Voice">
|
||||||
|
Filter by voice profile
|
||||||
|
```
|
||||||
|
Select from dropdown
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="By Date">
|
||||||
|
Filter by date range
|
||||||
|
```
|
||||||
|
Last 7 days, Last 30 days, Custom range
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
History is stored locally:
|
||||||
|
|
||||||
|
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
|
||||||
|
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
|
||||||
|
- **Linux**: `~/.config/com.voicebox.app/data/`
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
Deleting the data directory will remove all history. Export important files first.
|
||||||
|
</Warning>
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
title: "Installation"
|
||||||
|
description: "Download and install Voicebox on macOS, Windows, or Linux"
|
||||||
|
---
|
||||||
|
|
||||||
|
## Download
|
||||||
|
|
||||||
|
Voicebox is available for macOS and Windows, with Linux builds coming soon.
|
||||||
|
|
||||||
|
<CardGroup cols={2}>
|
||||||
|
<Card title="macOS" icon="apple">
|
||||||
|
Download for Apple Silicon or Intel Macs
|
||||||
|
</Card>
|
||||||
|
<Card title="Windows" icon="windows">
|
||||||
|
Download MSI installer or Setup executable
|
||||||
|
</Card>
|
||||||
|
</CardGroup>
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<Tab title="Apple Silicon">
|
||||||
|
Download: [voicebox_aarch64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_aarch64.app.tar.gz)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Extract the archive
|
||||||
|
tar -xzf voicebox_aarch64.app.tar.gz
|
||||||
|
|
||||||
|
# Move to Applications
|
||||||
|
mv Voicebox.app /Applications/
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="Intel">
|
||||||
|
Download: [voicebox_x64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64.app.tar.gz)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Extract the archive
|
||||||
|
tar -xzf voicebox_x64.app.tar.gz
|
||||||
|
|
||||||
|
# Move to Applications
|
||||||
|
mv Voicebox.app /Applications/
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<Tab title="MSI Installer">
|
||||||
|
Download: [voicebox_x64_en-US.msi](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64_en-US.msi)
|
||||||
|
|
||||||
|
Double-click the MSI file and follow the installation wizard.
|
||||||
|
</Tab>
|
||||||
|
<Tab title="Setup Executable">
|
||||||
|
Download: [voicebox_x64-setup.exe](https://github.com/jamiepine/voicebox/releases/latest/download/voicebox_x64-setup.exe)
|
||||||
|
|
||||||
|
Run the executable and follow the installation wizard.
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
### Linux
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Linux builds are coming soon. Currently blocked by GitHub runner disk space limitations.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## First Launch
|
||||||
|
|
||||||
|
When you launch Voicebox for the first time:
|
||||||
|
|
||||||
|
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
|
||||||
|
2. **Data Directory** — Voice profiles and generated audio are stored in:
|
||||||
|
- macOS: `~/Library/Application Support/com.voicebox.app/`
|
||||||
|
- Windows: `%APPDATA%/com.voicebox.app/`
|
||||||
|
- Linux: `~/.config/com.voicebox.app/`
|
||||||
|
|
||||||
|
3. **Backend Server** — The bundled Python server starts automatically
|
||||||
|
|
||||||
|
<Tip>
|
||||||
|
First generation will be slower due to model downloads. Subsequent runs use cached models.
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
## System Requirements
|
||||||
|
|
||||||
|
### Minimum
|
||||||
|
|
||||||
|
- **OS:** macOS 11+, Windows 10+, or Linux
|
||||||
|
- **RAM:** 8GB
|
||||||
|
- **Storage:** 5GB free space (for models and data)
|
||||||
|
- **CPU:** Modern multi-core processor
|
||||||
|
|
||||||
|
### Recommended
|
||||||
|
|
||||||
|
- **RAM:** 16GB+
|
||||||
|
- **GPU:** CUDA-capable NVIDIA GPU (for faster generation)
|
||||||
|
- **Storage:** 10GB+ free space
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
CPU inference is supported but significantly slower than GPU. A CUDA-capable GPU is highly recommended for real-time workflows.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
After installation, verify everything works:
|
||||||
|
|
||||||
|
1. Launch Voicebox
|
||||||
|
2. Check the server status indicator in the bottom-left corner (should be green)
|
||||||
|
3. Navigate to **Profiles** and create a test profile
|
||||||
|
4. Generate a short audio clip to verify the TTS engine works
|
||||||
|
|
||||||
|
<Check>
|
||||||
|
If you see a green status indicator and can generate audio, you're all set!
|
||||||
|
</Check>
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
<Card title="Quick Start Guide" icon="rocket" href="/overview/quick-start">
|
||||||
|
Create your first voice profile and generate speech
|
||||||
|
</Card>
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user