e61c85365b
The PyTorch Whisper transcribe() called the HF processor without truncation=False and model.generate() without return_timestamps=True. With those defaults, WhisperFeatureExtractor silently truncates inputs to 30 s (Whisper's native receptive field), so any dictation longer than ~30 s lost its tail. Setting truncation=False + padding="longest" + return_attention_mask=True on the processor, then forwarding the attention mask plus return_timestamps=True to generate(), flips HF Whisper into long-form mode: autoregressive decoding over rolling 30 s windows. Verified by round-tripping a 56.6 s Kokoro TTS sample through /transcribe — full text returned including content past the 30 s mark; previously the transcript was cut off roughly halfway through. MLX backend (mlx_backend.py) is intentionally unchanged: mlx_audio.stt's generate() already implements rolling-window long-form transcription with condition_on_previous_text in the upstream library, so it does not have the same bug. The HF-only kwargs added here would also break the MLX call signature. Co-Authored-By: Claude Opus 4.7 <[email protected]>
Voicebox Backend
FastAPI server powering voice cloning, speech generation, and audio processing. Runs locally as a Tauri sidecar or standalone via python -m backend.main.
Running
# Via justfile (recommended)
just dev:server
# Standalone
python -m backend.main --host 127.0.0.1 --port 17493
# With custom data directory
python -m backend.main --data-dir /path/to/data
The server auto-initializes the SQLite database on first startup. Models are downloaded from HuggingFace on first use.
Architecture
backend/
app.py # FastAPI app factory, CORS, lifecycle events
main.py # Entry point (imports app, runs uvicorn)
config.py # Data directory paths and configuration
models.py # Pydantic request/response schemas
server.py # Tauri sidecar launcher, parent-pid watchdog
routes/ # Thin HTTP handlers — validation, delegation, response formatting
services/ # Business logic, CRUD, orchestration
backends/ # TTS/STT engine implementations (MLX, PyTorch, etc.)
database/ # ORM models, session management, migrations, seed data
utils/ # Shared utilities (audio, effects, caching, progress tracking)
Request flow
HTTP request
-> routes/ (validate input, parse params)
-> services/ (business logic, database queries, orchestration)
-> backends/ (TTS/STT inference)
-> utils/ (audio processing, effects, caching)
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in services/.
Key modules
services/generation.py -- Single run_generation() function that handles all three generation modes (generate, retry, regenerate). Manages model loading, voice prompt creation, chunked inference, normalization, effects, and version persistence.
services/task_queue.py -- Serial generation queue. Ensures only one GPU inference runs at a time. Background tasks are tracked to prevent garbage collection.
backends/__init__.py -- Protocol definitions (TTSBackend, STTBackend), model config registry, and factory functions. Adding a new engine means implementing the protocol and registering a config entry.
backends/base.py -- Shared utilities used across all engine implementations: HuggingFace cache checks, device detection, voice prompt combination, progress tracking.
database/ -- SQLAlchemy ORM models with a re-exporting __init__.py for backward compatibility. Migrations run automatically on startup.
Backend selection
The server detects the best inference backend at startup:
| Platform | Backend | Acceleration |
|---|---|---|
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
| Linux (AMD) | PyTorch | ROCm |
| Intel Arc | PyTorch | IPEX / XPU |
| Windows (any GPU) | PyTorch | DirectML |
| Any | PyTorch | CPU fallback |
Detection is handled by utils/platform_detect.py. Both backends implement the same TTSBackend protocol, so the API layer is engine-agnostic.
API
90 endpoints organized by domain. Full interactive documentation available at http://localhost:17493/docs when the server is running.
| Domain | Prefix | Description |
|---|---|---|
| Health | /, /health |
Server status, GPU info, filesystem checks |
| Profiles | /profiles |
Voice profile CRUD, samples, avatars, import/export |
| Channels | /channels |
Audio channel management and voice assignment |
| Generation | /generate |
TTS generation, retry, regenerate, status SSE |
| History | /history |
Generation history, search, favorites, export |
| Transcription | /transcribe |
Whisper-based audio-to-text |
| Stories | /stories |
Multi-track timeline editor, audio export |
| Effects | /effects |
Effect presets, preview, version management |
| Audio | /audio, /samples |
Audio file serving |
| Models | /models |
Load, unload, download, migrate, status |
| Tasks | /tasks, /cache |
Active task tracking, cache management |
| CUDA | /backend/cuda-* |
CUDA binary download and management |
Quick examples
# Generate speech
curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \
-d '{"text": "Hello world", "profile_id": "...", "language": "en"}'
# List profiles
curl http://localhost:17493/profiles
# Stream generation status (SSE)
curl http://localhost:17493/generate/{id}/status
Data directory
{data_dir}/
voicebox.db # SQLite database
profiles/{id}/ # Voice samples per profile
generations/ # Generated audio files
cache/ # Voice prompt cache (memory + disk)
backends/ # Downloaded CUDA binary (if applicable)
Default location is the OS-specific app data directory. Override with --data-dir or the VOICEBOX_DATA_DIR environment variable.
Code quality
Linting and formatting are enforced by ruff, configured in pyproject.toml. See STYLE_GUIDE.md for conventions.
just check-python # lint + format check
just fix-python # auto-fix lint issues + reformat
just test # run pytest
Dependencies
Runtime dependencies are in requirements.txt. macOS-only MLX dependencies are in requirements-mlx.txt. Dev tools (ruff, pytest) are installed automatically by just setup-python.