James PineandClaude Opus 4.7
4ffbc03d15
Review feedback on the preprocessor: 1. ``trim_top_db=30`` was labelled "conservative" in the docstring but is actually *more* aggressive than librosa's default of 60. Normal speech dynamic range sits around 30 dB, so 30 dB would eat quiet trailing syllables and soft consonants. Raise the default to 40 dB — below normal speech dynamic range but still catching obvious edge silence — and fix the docstring. 2. Unconditional 100 ms edge padding ran even when ``librosa.effects.trim`` removed nothing. For a well-recorded 29.9 s upload that path would push the waveform past the 30 s ceiling and trigger a spurious "too long" rejection. Only pad when trimming actually shortened the audio, and cap the pad so the output never exceeds the input length. Adds a regression test for the net-neutral length behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Voicebox Backend
FastAPI server powering voice cloning, speech generation, and audio processing. Runs locally as a Tauri sidecar or standalone via python -m backend.main.
Running
# Via justfile (recommended)
just dev:server
# Standalone
python -m backend.main --host 127.0.0.1 --port 17493
# With custom data directory
python -m backend.main --data-dir /path/to/data
The server auto-initializes the SQLite database on first startup. Models are downloaded from HuggingFace on first use.
Architecture
backend/
app.py # FastAPI app factory, CORS, lifecycle events
main.py # Entry point (imports app, runs uvicorn)
config.py # Data directory paths and configuration
models.py # Pydantic request/response schemas
server.py # Tauri sidecar launcher, parent-pid watchdog
routes/ # Thin HTTP handlers — validation, delegation, response formatting
services/ # Business logic, CRUD, orchestration
backends/ # TTS/STT engine implementations (MLX, PyTorch, etc.)
database/ # ORM models, session management, migrations, seed data
utils/ # Shared utilities (audio, effects, caching, progress tracking)
Request flow
HTTP request
-> routes/ (validate input, parse params)
-> services/ (business logic, database queries, orchestration)
-> backends/ (TTS/STT inference)
-> utils/ (audio processing, effects, caching)
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in services/.
Key modules
services/generation.py -- Single run_generation() function that handles all three generation modes (generate, retry, regenerate). Manages model loading, voice prompt creation, chunked inference, normalization, effects, and version persistence.
services/task_queue.py -- Serial generation queue. Ensures only one GPU inference runs at a time. Background tasks are tracked to prevent garbage collection.
backends/__init__.py -- Protocol definitions (TTSBackend, STTBackend), model config registry, and factory functions. Adding a new engine means implementing the protocol and registering a config entry.
backends/base.py -- Shared utilities used across all engine implementations: HuggingFace cache checks, device detection, voice prompt combination, progress tracking.
database/ -- SQLAlchemy ORM models with a re-exporting __init__.py for backward compatibility. Migrations run automatically on startup.
Backend selection
The server detects the best inference backend at startup:
| Platform | Backend | Acceleration |
|---|---|---|
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
| Linux (AMD) | PyTorch | ROCm |
| Intel Arc | PyTorch | IPEX / XPU |
| Windows (any GPU) | PyTorch | DirectML |
| Any | PyTorch | CPU fallback |
Detection is handled by utils/platform_detect.py. Both backends implement the same TTSBackend protocol, so the API layer is engine-agnostic.
API
90 endpoints organized by domain. Full interactive documentation available at http://localhost:17493/docs when the server is running.
| Domain | Prefix | Description |
|---|---|---|
| Health | /, /health |
Server status, GPU info, filesystem checks |
| Profiles | /profiles |
Voice profile CRUD, samples, avatars, import/export |
| Channels | /channels |
Audio channel management and voice assignment |
| Generation | /generate |
TTS generation, retry, regenerate, status SSE |
| History | /history |
Generation history, search, favorites, export |
| Transcription | /transcribe |
Whisper-based audio-to-text |
| Stories | /stories |
Multi-track timeline editor, audio export |
| Effects | /effects |
Effect presets, preview, version management |
| Audio | /audio, /samples |
Audio file serving |
| Models | /models |
Load, unload, download, migrate, status |
| Tasks | /tasks, /cache |
Active task tracking, cache management |
| CUDA | /backend/cuda-* |
CUDA binary download and management |
Quick examples
# Generate speech
curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \
-d '{"text": "Hello world", "profile_id": "...", "language": "en"}'
# List profiles
curl http://localhost:17493/profiles
# Stream generation status (SSE)
curl http://localhost:17493/generate/{id}/status
Data directory
{data_dir}/
voicebox.db # SQLite database
profiles/{id}/ # Voice samples per profile
generations/ # Generated audio files
cache/ # Voice prompt cache (memory + disk)
backends/ # Downloaded CUDA binary (if applicable)
Default location is the OS-specific app data directory. Override with --data-dir or the VOICEBOX_DATA_DIR environment variable.
Code quality
Linting and formatting are enforced by ruff, configured in pyproject.toml. See STYLE_GUIDE.md for conventions.
just check-python # lint + format check
just fix-python # auto-fix lint issues + reformat
just test # run pytest
Dependencies
Runtime dependencies are in requirements.txt. macOS-only MLX dependencies are in requirements-mlx.txt. Dev tools (ruff, pytest) are installed automatically by just setup-python.