Compare commits

..
Author SHA1 Message Date
James PineandClaude Opus 4.6 da05acdee5 fix(paths): strip legacy "data/" prefix when resolving stored paths
0.3.0 sometimes stored relative media paths with the data-dir name baked in
(e.g. "data/profiles/<uuid>/sample.wav"). resolve_storage_path joined those
directly with _data_dir, producing "<data_dir>/data/profiles/..." — a
spurious double nest that breaks file reads after upgrading to 0.4.0.

The 0.4.0 startup migration didn't catch it because resolve_storage_path
produced the buggy double-nested path, to_storage_path saw "data" at the
first (legitimate) index, and the normalized value matched the stored value
so the row was skipped.

Strip any leading "data/" component before joining. This unblocks runtime
reads and lets _normalize_storage_paths rewrite the affected rows on next
startup — no manual migration needed.

Fixes "No such file or directory: '<data_dir>/data/profiles/...'" and
associated 404s on GET /audio/<id> after upgrading from 0.3.0 to 0.4.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 20:18:31 -07:00
James PineandClaude Opus 4.6 67bf8e906a docs/landing: update for 0.4.0 — new engines, GPU docs, donate button, voice docs restructure
Docs:
- Add gpu-acceleration.mdx (all 9 platform/GPU combos, CUDA backend swap, Blackwell, XPU, troubleshooting)
- Add preset-voices.mdx (Kokoro 50 voices, Qwen CustomVoice 9 voices, instruct mode docs)
- Restructure voice-cloning.mdx to cover all 5 cloning engines with comparison table
- Restructure creating-voice-profiles.mdx around cloned vs preset workflows
- Update voice-profiles.mdx schema with voice_type discriminator, preset/design columns

Landing:
- Add 3 new engine cards (Qwen CustomVoice, HumeAI TADA, Kokoro) to Multi-Engine section
- Add Donate button (Buy Me a Coffee) to navbar and footer
- Add DONATE_URL constant

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 19:51:43 -07:00
James Pine 625e1ba549 Bump version: 0.3.1 → 0.4.0 2026-04-16 03:16:06 -07:00
James PineandClaude Opus 4.6 cfe6770639 style: apply biome format to drifted authored source
Catches format drift that accumulated across 13 authored files in
app/src and docs/. Auto-generated artifacts (tauri/src-tauri/gen,
docs/openapi.json, docs/cli.json, app/src/lib/api) were left alone
since the build regenerates them on each run — baking their formatted
state into git just causes churn next build.

No behavioral changes. Trailing commas, line wrapping, and indentation
only.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 03:13:45 -07:00
James PineandClaude Opus 4.6 00452b51a8 style(changelog): make entry version headings bigger in settings UI
The in-app changelog viewer rendered each entry's version number at the
same size as body text (text-sm font-medium), so visually there was no
clear anchor for where one release's notes ended and the next began.

Bump the version heading to text-xl font-semibold tracking-tight and
widen the bottom margin so each release reads as a proper section
header.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 03:12:09 -07:00
James PineandClaude Opus 4.6 106aec46a8 feat(generate): restore instruct toggle in floating generate box (Qwen CustomVoice)
Before 0.4 every engine was a cloning model, so the instruct UI in the
floating generate box applied the same way everywhere. Commit 3187344
hid the instruct toggle because the mix of new engines landing in 0.4
made it unclear which ones honored the kwarg. With Qwen CustomVoice
now shipping as the only engine actually tuned for instruct-style
control, bring the button back — conditionally, and only for that
engine.

Changes:
  • FloatingGenerateBox: SlidersHorizontal toggle button appears left
    of Generate when the box is expanded AND engine is
    qwen_custom_voice. Clicking it reveals an additive instruct
    textarea below the main text field (not a modal swap like the old
    version). State persists across engine switches so the toggle
    remembers its last position.
  • GenerationForm: narrow the instruct FormField's conditional from
    `qwen || qwen_custom_voice` to just `qwen_custom_voice`.
  • useGenerationForm: narrow supportsInstruct for the same reason.
    Base Qwen3-TTS accepts the kwarg but the model itself doesn't honor
    it — only CustomVoice was trained for instruction-based style
    control.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 03:09:55 -07:00
James PineandClaude Opus 4.6 c9e5c5d9a7 fix: clean up scroll effect timers and fix disabled+selected card toggle
- Add cleanup for requestAnimationFrame and setTimeout in scroll effect
  to prevent stale DOM writes on unmount or rapid selection changes
- Fix disabled+selected card click: bounce the selection to re-trigger
  the engine auto-switch instead of deselecting

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 02:56:45 -07:00
James PineandClaude Opus 4.6 48cd1f369a feat: gray out unsupported profiles instead of filtering, auto-switch engine on selection
- Show all voice profiles with unsupported ones grayed out (opacity) instead of hidden
- Clicking a grayed-out profile selects it and auto-switches the engine to a compatible one
- Sort supported profiles first, with info tip about compatibility at the bottom
- Scroll to selected profile after engine/sort changes with safe margin
- Fix engine desync on tab navigation by initializing form engine from store

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 02:56:45 -07:00
James PineandClaude Opus 4.6 2bfe400457 feat(skills): add triage-prs skill for pre-release PR speedruns
Immortalizes the workflow used to clear the open-PR backlog before
0.4.0: classify every open PR into merge / candidate / supersede /
defer tiers, write a working triage doc, then run the merge loop —
rebasing where needed, merging in batches, applying post-merge
follow-ups, and closing superseded PRs with credit.

Captures the gotchas that matter most:
  • Never review a stale branch via `git diff main..HEAD` — it shows
    every intermediate main commit as a deletion and makes a 3-line
    PR look like a 700-line revert
  • Always rebase before squash-merging; GitHub's squash computes
    diff(PR-head, merge-base), so a stale branch will revert
    in-between work
  • Route-ordering, weak-framework linking, dependency floors, !Send
    audio types, and why PyTorch nightly isn't shippable

Paired with draft-release-notes and release-bump: triage → draft →
bump.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 02:22:57 -07:00
0aa19a9994 feat(history): add "Clear failed" button to wipe failed generations (#412)
When the model wasn't loaded, the app was closed mid-run, or a
generation otherwise errored out, the resulting "Failed" rows
accumulate in history and there was no way to remove them in bulk —
individual delete was the only option.

Adds a header row above the history list (only rendered when at
least one failed generation is present) with a "Clear failed" button
that opens a confirmation dialog, then calls a new
DELETE /history/failed endpoint which sweeps all status='failed'
rows (plus their version files / audio files on disk).

Closes jamiepine/voicebox#410

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-04-16 02:12:30 -07:00
73170d0e92 feat(health): warn when GPU arch isn't supported by PyTorch build
Applies the compatibility-checker portion of #367. Adds a
check_cuda_compatibility() helper that compares the current device's
compute capability against torch.cuda._get_arch_list() and returns a
human-readable warning if the PyTorch build doesn't support it.

Wired into three places:
  • HealthResponse gains a gpu_compatibility_warning field so clients
    can surface the issue in the UI
  • Startup logs the warning as WARN level
  • _get_gpu_status() appends "[UNSUPPORTED - see logs]" to the GPU
    label shown in settings

Skipped #367's other half — the switch from stable to nightly cu128
wheels across release.yml, build_binary.py, and justfile. That's
redundant with #401's TORCH_CUDA_ARCH_LIST=...12.0+PTX approach and
would introduce non-deterministic builds from shifting nightly
releases.

Co-Authored-By: nyzxor <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 02:11:23 -07:00
James PineandClaude Opus 4.6 0317626677 fix(qwen): unify HF cache dir to avoid split cache on Windows
Applies the cache_dir portion of #218. On Windows local setups, model
assets can split between .hf-cache/hub and .hf-cache/transformers when
Qwen3TTSModel.from_pretrained doesn't explicitly pin the cache root —
speech_tokenizer and preprocessor_config.json then fail to resolve
during load, causing 500s at generation time.

Routes both HF Hub and Transformers through hf_constants.HF_HUB_CACHE.

Skipped the torch_dtype= → dtype= rename from #218: transformers 4.36
(our minimum) doesn't accept the dtype alias, only 4.46+. Once we bump
the minimum we can make that change.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 02:08:59 -07:00
a5d5c780c2 fix: avoid ScreenCaptureKit launch crash on macOS 11 (#424)
Co-authored-by: txhno <[email protected]>
2026-04-16 01:58:29 -07:00
James PineandClaude Opus 4.6 7184a25e44 fix(watchdog): clear stale .keep-running sentinel on startup
Follow-up to #402. The sentinel is only removed inside the grace-period
"sentinel found" branch. When the HTTP /watchdog/disable request wins
the race (normal case on macOS/Linux, occasional on Windows), the
_watchdog_disabled=True check returns first and the sentinel is left on
disk indefinitely.

If a later session spawns a fresh server and the user exits without
"keep running", the new watchdog would find that stale sentinel during
its grace period and keep the server alive against user intent.

Wipe any pre-existing sentinel when the watchdog starts so only signals
written during this session's lifetime can influence grace-period
decisions.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 01:57:05 -07:00
479bc7fc5e fix: reliably keep server alive after GUI close on Windows (#402)
The HTTP /watchdog/disable request races with process exit on Windows,
causing the watchdog to kill the server before the request arrives.

Added a .keep-running sentinel file as a reliable fallback:
- Tauri writes the file to data_dir before sending the HTTP request
- The watchdog checks for it during the grace period after detecting
  parent death
- The file is removed after being read to avoid stale state

This approach works regardless of HTTP timing because file writes
complete synchronously before the Tauri process exits.

Fixes #372

Co-authored-by: Matt Van Horn <[email protected]>
2026-04-16 01:56:30 -07:00
Cocoon-BreakandGitHub 3e7727d1d2 fix: keep cpal Stream alive until playback completes (Closes #404) (#405)
The cpal Stream was created and play() called but then immediately
dropped when play_to_device() returned. When a cpal Stream is dropped,
audio output stops immediately. This caused silent playback.

Fix: add a spin-wait loop that holds the Stream in scope until all
samples have been consumed (or stop_flag is set).
2026-04-16 01:54:31 -07:00
JunghwanandGitHub be7c0cec12 fix: add asyncio.Lock to prevent concurrent CUDA downloads (#428)
* fix: add asyncio.Lock to prevent concurrent CUDA downloads

The startup auto-update task and the manual download endpoint can both
invoke download_cuda_binary() concurrently. Without mutual exclusion,
both coroutines write to the same temp file path, corrupting the
download. The progress-manager status check is a TOCTOU race because
the status is not set until after several synchronous checks complete.

Add a module-level asyncio.Lock acquired at the top of
download_cuda_binary() so only one download can proceed at a time.

* fix: fast-reject duplicate CUDA download when lock is held

Address CodeRabbit review feedback: check _download_lock.locked()
before awaiting the lock so concurrent callers return immediately
instead of queueing behind the first download. This prevents the
route handler from returning "started" to multiple callers when only
one download actually proceeds.
2026-04-16 01:51:21 -07:00
c9d8142a78 feat: add Blackwell GPU (sm_120) CUDA support (#401)
Set TORCH_CUDA_ARCH_LIST in the CUDA build step to include 12.0+PTX
for forward compatibility with Blackwell GPUs (RTX 5070 Ti, 5080, etc).

Pre-built PyTorch cu128 wheels only ship native kernels for sm_80/86/89/90.
Without this, Blackwell GPU users get "no kernel image is available for
execution on the device" at runtime.

Fixes #386
Related: #395, #396, #399, #400

Co-authored-by: Matt Van Horn <[email protected]>
2026-04-16 01:51:18 -07:00
13ba5f1aa6 fix: prevent intermittent clip splitting failures (#403)
Two changes to address the race condition causing "Failed to split clip":

Backend (stories.py): Added with_for_update() to the item query in
split_story_item so concurrent requests for the same clip are
serialized via a row lock instead of racing.

Frontend (StoryTrackEditor.tsx): Guard handleSplit with
splitItem.isPending to prevent rapid double-clicks from firing
multiple mutations before the first completes.

Fixes #366

Co-authored-by: Matt Van Horn <[email protected]>
2026-04-16 01:51:15 -07:00
9a3c307c75 fix(history): populate status/error/engine fields from DB row (#394)
* fix(history): populate status/error/engine/model_size/is_favorited from DB

GET /history/{generation_id} was constructing HistoryResponse without
passing status, error, engine, model_size, or is_favorited from the
DB row. Since HistoryResponse.status defaults to "completed" in the
Pydantic model (models.py:141), this endpoint returned
status="completed" for every generation regardless of the actual DB
state — including jobs still in "loading_model" or "generating", and
even "failed" jobs.

This breaks any client polling /history/{id} for job completion:
the API lies about the status, so the only trustworthy success
signal becomes `audio_path` being non-empty. All other fields left
at their model defaults were similarly masked.

Fix: pass all fields through from the DB row, matching the pattern
used elsewhere in the codebase. The DB model (Generation in
database/models.py) already has all these columns.

* fix(history): apply NULL fallbacks to match list endpoint

Align the defensive mappings with services/history.py:206-223 so
both the single-item and list history endpoints handle legacy rows
with NULL status/engine/is_favorited identically. Without this,
HistoryResponse's non-Optional str/bool fields would raise a
pydantic ValidationError (500) on any row where these columns are
NULL — possible from direct SQL updates or past migrations.

Addresses review feedback on PR #394.

---------

Co-authored-by: malletfils <[email protected]>
2026-04-16 01:51:12 -07:00
JunghwanandGitHub 1da16cfc57 fix: harden voice prompt cache loading and SPA path guard (#429)
Two small safety improvements:

1. Voice prompt cache (cache.py): add weights_only=True to torch.load()
   so cached .prompt files are loaded using the safe unpickler instead of
   the unrestricted pickle deserializer. This follows the PyTorch 2.6+
   best practice of opting in to safe loading for all torch.load() calls.

2. SPA catch-all (app.py): replace str.startswith() path guard with
   Path.is_relative_to(). The string prefix check passes for sibling
   paths like /app/frontend_evil/ that share the /app/frontend prefix.
   is_relative_to() correctly tests directory containment.
2026-04-16 01:49:19 -07:00
Luis SambranoandGitHub a1807be04d fix(deps): relax torch requirement for macOS x86_64 compatibility (#416) 2026-04-16 01:49:16 -07:00
Khaled SolimanandGitHub fdba18e9ee fix: resolve ModuleNotFoundError by using relative import for utils (#384) 2026-04-16 01:49:13 -07:00
MaxandGitHub 07a845cece Add NUMBA_CACHE_DIR environment variable (#425) 2026-04-16 01:49:10 -07:00
James PineandClaude Opus 4.6 615d604ceb fix(numpy-compat): raise on unknown dtype + add fp16/complex
Follow-up to #361. The original fallback silently mapped unknown numpy
dtypes to torch.float32, which would reinterpret the memcpy'd bytes in
the wrong dtype and corrupt data (e.g. fp16 tensors from some TTS
engines) rather than erroring loudly.

- Hoist dtype_map out of the inner function so it's built once
- Add float16, complex64, complex128 mappings
- Raise TypeError on unknown dtype instead of silent float32 fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 01:47:22 -07:00
a383ff6863 fix: torch.from_numpy crash with numpy 2.x in frozen binary (#361)
torch is compiled against numpy 1.x. numpy 2.x changed the ABI version
returned by PyArray_GetNDArrayCVersion() (0x01000009 → 0x02000000), so
torch's is_numpy_available() always returns False and torch.from_numpy()
raises RuntimeError. This causes TTS generation to fail with:

  ValueError: Unable to create tensor, you should probably activate
  padding with 'padding=True'

Two fixes:

1. Pin numpy<2.0 in requirements.txt so new builds bundle a compatible
   numpy version. (The existing comment already flagged this intention
   but the upper bound was never added.)

2. Add a PyInstaller runtime hook (pyi_rth_numpy_compat.py) that installs
   a ctypes memmove fallback for torch.from_numpy() at startup. Runtime
   hooks run after FrozenImporter is registered so frozen torch is
   importable. The fallback catches RuntimeError from the C-level ABI
   check and copies the numpy array into a new tensor via raw memory copy,
   bypassing the check entirely. This is a belt-and-suspenders fix that
   works regardless of the bundled numpy version.

Co-authored-by: aimaaaimaa <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
2026-04-16 01:46:40 -07:00
Jamie PineandGitHub 75abbb02c3 Merge pull request #344 from pandego/fix/308-docker-compose-startup
fix: include changelog in docker web build
2026-03-26 23:06:47 -07:00
Jamie PineandGitHub b49f14a814 Merge pull request #319 from jamiepine/fix/startup-and-server-switch
fix: GUI startup with external server + data refresh on server switch
2026-03-26 23:06:29 -07:00
Jamie PineandGitHub 05686efbfd Merge pull request #345 from ArfianID/fix/backend-import-error
Fix: "Failed to Save" preset error by resolving backend import path resolution
2026-03-22 09:45:25 -07:00
Arfian e2c03fef9a Fix: move lazy imports to top-level and use absolute paths to resolve ModuleNotFoundError in production 2026-03-22 22:29:25 +07:00
pandego 4347eaed4c fix: include changelog in docker web build 2026-03-22 09:32:32 +01:00
James Pine 60aac279ce fix: address PR #319 review feedback — health validation, error handling, queryClient decoupling
- Rust: Replace fragile body.contains("status") with proper JSON
  deserialization validating status=="healthy", model_loaded (bool),
  and gpu_available (bool) to prevent misidentifying non-Voicebox services

- Frontend: Validate health response has Voicebox-specific fields before
  marking server as ready during fallback polling

- Frontend: Discriminate port-in-use errors (poll for external server) from
  real startup failures (missing sidecar, signing issues) — surface errors
  immediately with a startupError state and Retry button in the UI

- Frontend: Set explicit startup-error state when 2-minute polling timeout
  expires so the loading screen shows actionable feedback instead of hanging

- Architecture: Extract QueryClient to standalone side-effect-free module
  (lib/queryClient.ts) to decouple serverStore from React bootstrap entrypoint
2026-03-21 10:24:15 -07:00
James Pine 8b1c7552be Merge remote-tracking branch 'origin/fix/startup-and-server-switch' into pr-319 2026-03-21 10:18:45 -07:00
Jamie PineandGitHub 9a955a77d2 Merge pull request #320 from jamiepine/feat/intel-xpu-support
feat: Intel Arc (XPU) GPU support
2026-03-21 08:39:51 -07:00
Jamie PineandGitHub c18591c0c3 Merge pull request #318 from jamiepine/fix/offline-model-loading
fix: force offline mode when loading cached models (Qwen TTS & Whisper)
2026-03-21 08:38:08 -07:00
Jamie PineandGitHub ea3469f2dc Merge pull request #332 from nicoschtein/patch-1
Fix links in Get Started section of index.mdx
2026-03-21 08:37:00 -07:00
James Pine b108bb1cb1 fix: store media paths relative to data dir 2026-03-20 15:06:07 -07:00
Nicolas SchteinschraberandGitHub 8b796bc6b4 Fix links in Get Started section of index.mdx
Updated links in the Get Started section for correct paths.
2026-03-20 13:58:53 -03:00
James Pine e6f419cd70 fix: show all engines in floating generator 2026-03-19 19:52:08 -07:00
James Pine 72c13fd3fc fix: enforce preset profile engine compatibility 2026-03-19 19:51:53 -07:00
James Pine 4e0c731db8 feat: add Qwen CustomVoice preset engine 2026-03-19 19:48:50 -07:00
James Pine d70b878b71 fix: tighten kokoro profile handling 2026-03-19 19:32:49 -07:00
Jamie PineandGitHub a71011741d Merge pull request #325 from jamiepine/feat/kokoro-engine
feat: Kokoro 82M TTS engine + voice profile type system
2026-03-19 19:21:15 -07:00
James Pine d6f48ace3e Mirror the regular /generate endpoint behavior more closely 2026-03-19 16:11:38 -07:00
James Pine 0fc2192204 fix: resolve relative paths using configured data dir, not CWD 2026-03-19 10:37:11 -07:00
James Pine 9e726ad048 fix: remove engine dropdown filtering — profile grid handles it 2026-03-19 10:14:33 -07:00
James Pine 3584283d84 feat: Kokoro 82M TTS engine + voice profile type system
Add Kokoro-82M as a new TTS engine — 82M params, CPU realtime, 8 languages,
Apache 2.0. Unlike cloning engines, Kokoro uses pre-built voice styles, which
required a new profile type system to support non-cloning engines cleanly.

Kokoro engine:
- New kokoro_backend.py implementing TTSBackend protocol
- 50 built-in voices across en/es/fr/hi/it/pt/ja/zh
- KPipeline API with language-aware G2P routing via misaki
- PyInstaller bundling for misaki, language_tags, espeakng_loader, en_core_web_sm

Voice profile type system:
- New voice_type column: 'cloned' | 'preset' | 'designed' (future)
- Preset profiles store engine + voice ID instead of audio samples
- default_engine field on profiles — auto-selects engine on profile pick
- Create Voice dialog: toggle between 'Clone from audio' and 'Built-in voice'
- Edit dialog shows preset voice info instead of sample list for preset profiles
- Engine selector locks to preset engine when preset profile is selected
- Profile grid filters by engine — shows Kokoro voices when Kokoro selected
- Custom empty state when no preset profiles exist for selected engine

Bug fixes:
- Fix relative audio paths in DB causing 404s in production builds
- config.set_data_dir() now resolves to absolute paths
- Startup migration converts existing relative paths to absolute

Also updates PROJECT_STATUS.md and tts-engines.mdx developer guide.
2026-03-19 10:09:48 -07:00
Jamie PineandGitHub e4def9365f Merge pull request #321 from liorshahverdi/fix/delete-failed-generations-292
fix/allows deletion of failed generations 292
2026-03-19 09:36:55 -07:00
James Pine 707046237c fix: complete Intel XPU support — device-aware seeding, GPU status reporting, and setup detection
Address CodeRabbit review feedback and user-reported GPU acceleration failure:

- Use shared manual_seed() in chatterbox, chatterbox_turbo, and luxtts
  backends so XPU (and future accelerators) get proper device seeding
- Add XPU branch to _get_gpu_status() so startup log reports Intel Arc
  GPUs instead of 'None (CPU only)'
- Add XPU VRAM reporting and correct backend_variant fallback in the
  /health endpoint
- Switch justfile GPU detection from Get-WmiObject to Get-CimInstance,
  simplify the Arc regex to match 'Arc' (not 'Intel.*Arc'), log
  detected GPUs, and print manual install instructions on miss

Resolves the root cause where IPEX was silently not installed due to
WMI detection failure, causing CPU-only fallback on Intel Arc systems.
2026-03-18 17:01:12 -07:00
Lior Shahverdi 12ed2d51ce Adds a trash icon button alongside the existing retry button for
failed generations, giving users a way to clean up failed entries
  without having to retry them first.
2026-03-18 15:44:37 -04:00
James Pine 83ebababe7 feat: add Intel Arc (XPU) GPU support across all backends
Auto-detect Intel Arc GPUs during Windows setup and install PyTorch
with XPU support + intel-extension-for-pytorch. Enable allow_xpu=True
on all TTS backends (Chatterbox, Chatterbox Turbo, Hume TADA, LuxTTS)
that previously only supported CUDA. Add shared empty_device_cache()
and manual_seed() helpers in base.py to handle XPU memory management
and reproducible seeding alongside CUDA.
2026-03-18 11:24:51 -07:00
James Pine eb5869e59f fix: GUI startup with external server + data refresh on server switch
Two fixes for issue #312:

1. GUI stuck on loading screen when backend is already running externally
   (e.g. via python/uvicorn/Docker):

   - Rust: add HTTP health check fallback when the process on the port
     doesn't have 'voicebox' in its name. If /health responds with a
     valid Voicebox response, reuse the server instead of erroring.
   - Frontend: when startServer() fails, fall back to polling the
     health endpoint every 2s instead of permanently blocking.

2. No data refresh when switching server URLs in settings:

   - serverStore.setServerUrl() now invalidates all React Query caches
     when the URL actually changes, so profiles/history/models/stories
     are re-fetched from the new server.
   - Export queryClient from main.tsx for store-level cache invalidation.

Fixes #312
2026-03-18 10:59:59 -07:00
James Pine 2e95b7c5d8 fix: force offline mode when loading cached models (Qwen TTS & Whisper)
Qwen TTS and Whisper Base make network calls to HuggingFace even when
model weights are fully cached locally, because from_pretrained()
defaults to local_files_only=False. This causes failures for offline
users.

Add a reusable force_offline_if_cached() context manager that sets
HF_HUB_OFFLINE=1 during model loading when is_model_cached() is True.
Applied to all four affected load paths:

- PyTorchTTSBackend (Qwen TTS)
- PyTorchSTTBackend (Whisper)
- MLXTTSBackend (refactored from inline implementation)
- MLXSTTBackend (previously unprotected)

Closes #82
2026-03-18 10:31:30 -07:00
Jamie PineandGitHub ffc1b54812 Merge pull request #316 from jamiepine/fix/cuda-cu128-upgrade
Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI
2026-03-18 07:58:12 -07:00
James Pine fc5ed1ff40 upgrade CUDA backend from cu126 to cu128 and fix GPU settings UI
Upgrade CUDA toolkit from 12.6 (cu126) to 12.8 (cu128) for proper
RTX 50-series (Blackwell) GPU support. Users with RTX 5070/5080/5090
were reporting CUDA detection failures with cu126.

Also fix the GPU Acceleration settings panel where the 'Switch to CPU
Backend' button was unreachable — it was inside a conditional block
that required !isCurrentlyCuda, making it impossible to switch back
to CPU once running on CUDA.

Closes #315
2026-03-18 07:47:39 -07:00
Jamie PineandGitHub c9f38dd496 Merge pull request #305 from jamiepine/fix/qwen-tts-pyinstaller-source-files
fix: bundle qwen_tts source files in PyInstaller build
2026-03-17 09:24:42 -07:00
James Pine 58b19e4e9f fix: bundle qwen_tts source files in PyInstaller build
Replace --collect-submodules + --collect-data with --collect-all for
qwen_tts. The qwen_tts runtime expects physical .py source files
(e.g. modeling_qwen3_tts.py) under _MEIPASS, which only --collect-all
provides. This is the same pattern used for inflect/typeguard.

Fixes #212
2026-03-17 09:23:30 -07:00
Jamie PineandGitHub 0245c31dba Merge pull request #298 from jamiepine/feat/cuda-libs-addon
feat: split CUDA backend into independently versioned server + libs archives
2026-03-17 09:17:31 -07:00
Jamie Pine 81864e831a fix: always clean up temp archive on failure and fix justfile data dir path
- Wrap download/verify/extract in try/finally so .download-*.tmp is
  always deleted, even on mid-download or extraction failures
- Fix justfile build-server-cuda to use sh.voicebox.app (production path)
2026-03-17 09:15:23 -07:00
Jamie Pine 7bd72ea9f7 Bump version: 0.3.0 → 0.3.1 2026-03-17 07:50:12 -07:00
Jamie Pine f96eae2567 fix: address PR review feedback from CodeRabbit
- Upgrade softprops/action-gh-release@v1 to @v2 (Node 16 EOL)
- Fail-fast on checksum fetch failure instead of extracting unverified archives
- Abort packaging if no NVIDIA files found (prevents empty cuda-libs archive)
- Fix nvidia/ path detection bug (list membership vs substring check)
- Fix justfile Copy-Item nesting (copy contents, not the directory itself)
2026-03-17 06:53:54 -07:00
Jamie Pine 7d53699c96 fix: update build-server-cuda to copy onedir folder instead of single exe 2026-03-17 06:12:30 -07:00
Jamie Pine 28e91ce2c1 chore: add .spec and nul to .gitignore 2026-03-17 04:58:13 -07:00
Jamie Pine 88be097b62 fix: update package_cuda.py for PyInstaller 6.18 layout and remove split_binary.py
- Fix is_nvidia_file() to match NVIDIA DLLs in _internal/torch/lib/
  (PyInstaller 6.18 + torch 2.10 no longer uses nvidia/ subdirectories)
- Remove deprecated split_binary.py (both archives are under 2GB)
- Update torch_compat range to >=2.6.0,<2.11.0
- Update build docs for new dual-archive packaging flow
2026-03-17 04:57:05 -07:00
James Pine 564d787927 feat: split CUDA backend into independently versioned server + libs archives
Switch CUDA builds from PyInstaller --onefile to --onedir and split the
output into two separately versioned archives:

1. Server core (~200-400MB) — versioned with the app, redownloaded on
   every app update
2. CUDA libs (~2GB) — versioned independently (cu126-v1), only
   redownloaded when the CUDA toolkit or torch version changes

This eliminates the ~2.4GB full redownload on every version bump.
After initial setup, most app updates only need ~200-400MB.

Closes #297
2026-03-17 04:04:17 -07:00
James Pine 2c1ee94891 docs: add TADA learnings to TTS engine guide and CUDA libs addon plan
Enrich tts-engines.mdx with patterns discovered during TADA integration:
- Phase 0.2: new greps for @torch.jit.script, torchaudio.load, gated repos
- Phase 3.4: model naming inconsistency warning
- Phase 5.2: TADA shim failure added to lessons table
- Phase 6: four new workaround sections (gated repos, torchcodec,
  torch.jit.script, toxic dependency shim pattern)
- Checklist: four new items matching the new scan patterns
- Remove TADA from upcoming engines (now shipped)

Add CUDA_LIBS_ADDON.md exploring --onedir split to avoid 2.4GB
redownloads on every version bump.
2026-03-17 03:53:40 -07:00
Jamie PineandGitHub e789c937ad Merge pull request #296 from jamiepine/feat/add-tada-tts-engine
Add HumeAI TADA TTS engine (1B English + 3B Multilingual)
2026-03-17 03:47:47 -07:00
James Pine 273483ffcf fix TorchScript error in frozen builds and update docs for TADA
Remove @torch.jit.script from the DAC shim's snake() function —
TorchScript calls inspect.getsource() which fails in PyInstaller
binaries (no .py source files).

Update all user-facing docs: 4 → 5 TTS engines, add TADA row to
every engine comparison table, mark TADA as Shipped in the upcoming
engines list, update architecture diagrams and tech stack tables.
2026-03-17 03:28:58 -07:00
James Pine 5774a168a9 fix TADA 3B model name: tada-3b -> tada-3b-ml 2026-03-17 03:17:53 -07:00
James Pine 6bf40bd2d0 fix tokenizer patch corrupting AutoTokenizer for other engines
Replace the monkey-patch on AutoTokenizer.from_pretrained (which broke
the classmethod descriptor and caused 'Tokenizer not loaded' errors
when loading Qwen after TADA) with two targeted config patches:
- Set AlignerConfig.tokenizer_name to the local ungated tokenizer path
- Pre-load TadaConfig, inject tokenizer_name, pass config= to from_pretrained

No global state is modified; other engines are unaffected.
2026-03-17 03:15:57 -07:00
James Pine 12cda2e090 fix torchcodec error by using soundfile instead of torchaudio.load
torchaudio 2.10+ switched its default audio loading backend to
torchcodec, which isn't installed. Replace torchaudio.load() with
soundfile.read() in create_voice_prompt(). TADA's internal use of
torchaudio.functional.resample() is unaffected (pure PyTorch math,
no torchcodec dependency).
2026-03-17 02:25:05 -07:00
James Pine 7a90290a76 fix gated Llama tokenizer error by redirecting to ungated mirror
TADA hardcodes 'meta-llama/Llama-3.2-1B' as its tokenizer source in
both the Aligner and TadaForCausalLM.from_pretrained(). That repo is
gated and requires accepting Meta's license on HuggingFace.

Monkey-patch AutoTokenizer.from_pretrained during model loading to
redirect Llama tokenizer requests to 'unsloth/Llama-3.2-1B', an
ungated mirror with identical tokenizer files. The patch is scoped
to model loading only and restored immediately after.
2026-03-17 02:22:26 -07:00
James Pine b02ce8e2f3 replace descript-audio-codec with lightweight DAC shim
The real descript-audio-codec package pulls in descript-audiotools,
which transitively requires onnx, tensorboard, protobuf, matplotlib,
pystoi, and other heavy dependencies. onnx fails to build from source
on macOS due to CMake version incompatibility.

TADA only uses Snake1d (a 7-line PyTorch module) from DAC. This commit
adds a shim in backend/utils/dac_shim.py that registers fake dac.*
modules in sys.modules with just the Snake1d class, completely
eliminating the DAC/audiotools dependency chain.
2026-03-17 02:16:33 -07:00
James Pine 4e7772a21d add HumeAI TADA TTS engine (1B English + 3B Multilingual)
Integrates HumeAI's TADA (Text-Acoustic Dual Alignment) speech-language
model as a new TTS engine. TADA uses a novel 1:1 token-audio alignment
that produces coherent speech over long sequences (700s+).

Two model variants:
- tada-1b: English-only, ~4GB, built on Llama 3.2 1B
- tada-3b-ml: 10 languages, ~8GB, built on Llama 3.2 3B

Backend uses the Encoder for voice prompt encoding with caching, and
TadaForCausalLM with flow-matching diffusion for generation. Supports
bf16 inference on CUDA, forces CPU on macOS (MPS compatibility).

Installed with --no-deps due to torch>=2.7 pin conflict; descript-audio-codec
and torchaudio added as explicit sub-dependencies.
2026-03-17 01:55:15 -07:00
James Pine 51fb320b8c readme 2026-03-17 01:24:48 -07:00
James Pine 8ac202aa58 docs for adding new engines 2026-03-17 01:13:30 -07:00
Jamie Pine ac68052945 create stub resource files when actool output is missing
Older Xcode versions don't produce Assets.car from .icon assets.
Fall back to empty stubs for all platforms so the bundler succeeds.
2026-03-17 01:07:54 -07:00
Jamie Pine e601fd2ca4 generate icon assets at build time instead of tracking them
build.rs now generates voicebox.icns via sips + iconutil alongside the
existing actool Assets.car compilation. On non-macOS, empty stub files
are created so Tauri's resource bundler doesn't fail on missing paths.
2026-03-17 00:51:12 -07:00
Jamie Pine 7b25e0ba0b Bump version: 0.2.3 → 0.3.0 2026-03-17 00:25:56 -07:00
Jamie PineandGitHub a6817cd082 Merge pull request #295 from jamiepine/fix/misc-bugs
fix: batch of bug fixes from issue tracker
2026-03-17 00:08:17 -07:00
118 changed files with 6629 additions and 1156 deletions
+120
View File
@@ -0,0 +1,120 @@
---
name: add-tts-engine
description: Use this skill to add a new TTS engine to Voicebox. It walks through dependency research, backend implementation, frontend wiring, PyInstaller bundling, and frozen-build testing. Always start with Phase 0 (dependency audit) before writing any code.
---
# Add TTS Engine
## Goal
Integrate a new text-to-speech engine into Voicebox end-to-end: dependency research, backend protocol implementation, frontend UI wiring, PyInstaller bundling, and frozen-build verification. The user should only need to test the final build locally.
## Reference Doc
The full phased guide lives at `docs/content/docs/developer/tts-engines.mdx`. **Read this file in its entirety before starting.** It contains:
- Phase 0: Dependency research (mandatory before writing code)
- Phase 1: Backend implementation (`TTSBackend` protocol)
- Phase 2: Route and service integration (usually zero changes)
- Phase 3: Frontend integration (5 files)
- Phase 4: Dependencies (`requirements.txt`, justfile, CI, Docker)
- Phase 5: PyInstaller bundling (`build_binary.py` + `server.py`)
- Phase 6: Common upstream workarounds
- Implementation checklist (gate between phases)
## Workflow
### 1. Read the guide
```bash
# Read the full TTS engines doc
cat docs/content/docs/developer/tts-engines.mdx
```
Internalize all phases, especially Phase 0 and Phase 5. The v0.2.3 release was three patch releases because Phase 0 was skipped.
### 2. Dependency research (Phase 0)
Clone the model library into a temporary directory and audit it. Do NOT skip this.
```bash
mkdir /tmp/engine-research && cd /tmp/engine-research
git clone <model-library-url>
```
Run the grep searches from Phase 0.2 in the guide against the cloned source and its transitive dependencies. Produce a written dependency audit covering:
1. PyPI vs non-PyPI packages
2. PyInstaller directives needed (`--collect-all`, `--copy-metadata`, `--hidden-import`)
3. Runtime data files that must be bundled
4. Native library paths that need env var overrides in frozen builds
5. Monkey-patches needed (`torch.load`, float64, MPS, HF token)
6. Sample rate
7. Model download method (`from_pretrained` vs `snapshot_download` + `from_local`)
Test model loading and generation on CPU in the throwaway venv before proceeding.
### 3. Implement (Phases 1–4)
Follow the guide's phases in order. Key files to modify:
**Backend (Phase 1):**
- Create `backend/backends/<engine>_backend.py`
- Register in `backend/backends/__init__.py` (ModelConfig + TTS_ENGINES + factory)
- Update regex in `backend/models.py`
**Frontend (Phase 3):**
- `app/src/lib/api/types.ts` — engine union type
- `app/src/lib/constants/languages.ts` — ENGINE_LANGUAGES
- `app/src/components/Generation/EngineModelSelector.tsx` — ENGINE_OPTIONS, ENGINE_DESCRIPTIONS
- `app/src/lib/hooks/useGenerationForm.ts` — Zod schema, model-name mapping
- `app/src/components/ServerSettings/ModelManagement.tsx` — MODEL_DESCRIPTIONS
**Dependencies (Phase 4):**
- `backend/requirements.txt`
- `justfile` (setup-python, setup-python-release targets)
- `.github/workflows/release.yml`
- `Dockerfile` (if applicable)
### 4. PyInstaller bundling (Phase 5)
Register the engine in `backend/build_binary.py`:
- `--hidden-import` for the backend module and model package
- `--collect-all` for packages using `inspect.getsource`, shipping data files, or native libraries
- `--copy-metadata` for packages using `importlib.metadata`
If the engine has native data paths, add `os.environ.setdefault()` in `backend/server.py` inside the `if getattr(sys, 'frozen', False):` block.
### 5. Verify in dev mode
```bash
just dev
```
Test the full chain: model download → load → generate → voice cloning.
### 6. Use the checklist
Walk through the Implementation Checklist at the bottom of `tts-engines.mdx`. Every item must be checked before handing the build to the user.
## Key Lessons (from v0.2.3)
These are the most common failure modes. Phase 0 research catches all of them:
| Pattern | Symptom in Frozen Build | Fix |
|---------|------------------------|-----|
| `@typechecked` / `inspect.getsource()` | "could not get source code" | `--collect-all <package>` |
| Package ships pretrained model files | `FileNotFoundError` for `.pth.tar`, `.yaml` | `--collect-all <package>` |
| C library with hardcoded system paths | `FileNotFoundError` for `/usr/share/...` | `--collect-all` + env var in `server.py` |
| `importlib.metadata.version()` | "No package metadata found" | `--copy-metadata <package>` |
| `torch.load` without `map_location` | CUDA device not available on CPU build | Monkey-patch `torch.load` |
| `torch.from_numpy` on float64 data | dtype mismatch RuntimeError | Cast to `.float()` |
| `token=True` in HF download calls | Auth failure without stored HF token | Use `snapshot_download(token=None)` + `from_local()` |
## Notes
- The route and service layers have zero per-engine dispatch points. `main.py` requires zero changes.
- The model config registry in `backends/__init__.py` handles all dispatch automatically.
- Use `get_torch_device()` and `model_load_progress()` from `backends/base.py` — don't reimplement device detection or progress tracking.
- Always test with a **clean HuggingFace cache** (no pre-downloaded models from dev).
- Do NOT push or create a release. Hand the build to the user for local testing.
+299
View File
@@ -0,0 +1,299 @@
---
name: triage-prs
description: Use this skill to triage the open PR queue before a release. Classifies every open PR into must-merge, candidate, superseded, or deferred; writes a working triage doc; and runs the merge loop end-to-end. Designed for the pre-release "PR speedrun" pass where a solo maintainer wants to clear the inbound backlog in a single session.
---
# Triage PRs
## Goal
Turn a backlog of open PRs into a shipped set of merges in a single focused session. Produce a tracked, resumable plan (`<VERSION>_PR_TRIAGE.md`), then work it — rebasing where needed, merging in isolation-safe batches, applying post-merge follow-ups, and closing superseded or partially-applicable PRs with credit to their authors.
This skill pairs with `draft-release-notes` and `release-bump`: triage first, then draft notes against the new main, then cut the release.
## When to use
- Before a minor or major release when 10+ open PRs have accumulated
- When you want to unblock merging without losing the narrative of what's landing
- When you know you can't personally review every PR deeply, but need to land the critical subset fast
## Prerequisites
- `gh` CLI authenticated against the repo
- A dedicated worktree for PR review (avoid contaminating `main` with checkouts of contributor branches)
- Clarity on the target version — the triage doc is named after it (e.g. `0.4.0_PR_TRIAGE.md`)
## Workflow
### 1. Set up an isolated PR-review worktree
```bash
git worktree list # check for stale ones first
git worktree prune
git worktree add ../voicebox-pr-review -b pr-review-<VERSION> main
```
Keep the main worktree for release-prep work (changelog drafts, direct-to-main follow-ups). Keep the review worktree for `gh pr checkout` — each checkout moves HEAD to a contributor branch, which you don't want to do in the main worktree.
### 2. Gather metadata for every open PR
```bash
gh pr list --state open --limit 50 --json \
number,title,author,isDraft,mergeable,mergeStateStatus,files,additions,deletions,reviewDecision,statusCheckRollup,maintainerCanModify \
--jq '.[] | {num: .number, title, author: .author.login, mergeable, state: .mergeStateStatus, canModify: .maintainerCanModify, changes: "+\(.additions)/-\(.deletions)", files: [.files[].path]}'
```
You want, for each PR:
- Size (`+additions/-deletions`)
- Mergeable state (`CLEAN`, `UNSTABLE`, `DIRTY` = conflicts, `UNKNOWN` = GitHub still computing)
- Whether maintainer edits are allowed on the branch (needed later if you rebase for the author)
- File paths touched (helps spot overlaps between PRs)
`UNKNOWN` is common right after a push to main — just try the merge and see.
### 3. Classify into tiers
Sort each PR into exactly one bucket:
**Tier 1 — Merge:** small, mergeable, fixes a real bug, clean CI, low review cost. One-liners, dependency relaxations, targeted safety hardening. These are the easy wins.
**Tier 2 — Candidate, review:** medium size (50-200 lines), touches more surface area, looks sound but needs a closer read. New user-facing features that fit the product direction.
**Supersede:** the fix or feature is already covered by something merged. Close with a comment pointing to the superseding PR. Check carefully — "similar title" isn't proof; compare the actual diffs.
**Defer to next release:** big features, dirty conflicts, draft PRs, anything touching the release pipeline in ways that would introduce risk. Don't merge these in a speedrun — they need dedicated focus.
### 4. Write the triage doc
Create `<VERSION>_PR_TRIAGE.md` in the PR-review worktree root. Structure:
```markdown
# <Repo> <VERSION> — PR Triage
Working doc for tracking which open PRs land in <VERSION>. Delete after release cut.
Last updated: <DATE>
## Progress
**Tier 1: 0 / N merged**
**Tier 2: 0 / M handled**
**Supersede triage: pending**
---
## Merge for <VERSION> — critical bug fixes
| PR | Status | Size | What it fixes | Why must-have |
|---|---|---|---|---|
| [#123](url) | [ ] | +5/-0 | ... | ... |
## Strong candidate — needs a quick review
| PR | Status | Size | Summary |
|---|---|---|---|
## Close as superseded
| PR | Status | Reason |
|---|---|---|
## Defer to <NEXT_VERSION>
- [#xxx](url) ... — reason
---
## Order of attack
1. Close superseded PRs (one-liner comments)
2. Merge tier-1 in dependency-free batches — check file paths don't overlap
3. Review tier-2 individually
4. Rerun `draft-release-notes` to pick up everything
5. Run `release-bump`
```
The **Progress** header is the most important part — it's your scoreboard and lets you resume cleanly if the session gets interrupted.
### 5. Work the loop — per PR
For each PR in the tier-1 / tier-2 list:
**a. Checkout in the review worktree:**
```bash
cd ../voicebox-pr-review
git checkout pr-review-<VERSION> # reset to neutral base
gh pr checkout <N>
```
**b. Read the *actual* commit, not `main..HEAD`:**
```bash
git show HEAD # the PR's actual changes
git show --stat HEAD # files touched + line counts
```
**Do NOT review via `git diff main..HEAD`** if the PR branch is older than main. That diff includes *every commit that landed on main after the PR was forked* as `-` (deletion) lines. A 3-line PR can look like a 700-line revert. This is the single easiest way to misjudge a PR.
**c. Evaluate concerns:** correctness, scope, interaction with already-merged work, version compatibility (e.g. can't use an API that requires a dependency version we don't yet pin).
**d. Rebase if the branch is behind main:**
```bash
git fetch origin main
git rebase origin/main
```
This is **essential** before squash-merging. GitHub's squash computes `diff(PR-head, merge-base)` — on a stale branch, that diff includes reverting every in-between commit. Rebasing moves the merge-base forward so the squash is clean.
**e. If maintainer edits are allowed, push the rebase back to the contributor's fork:**
```bash
git remote add <author> https://github.com/<author>/<repo>.git
git fetch <author> <branch> # get their ref first
git push <author> HEAD:<branch> --force-with-lease
```
This keeps GitHub's PR UI in sync with the rebased state and makes the merge clean from the GitHub side.
**f. Merge:**
```bash
gh pr merge <N> --squash
```
**g. Update the triage doc** — flip the checkbox to `✅ merged <sha>` (use the short SHA from `gh pr view <N> --json mergeCommit --jq '.mergeCommit.oid[0:7]'`). Update the Progress header.
### 6. Batch tiny fixes
PRs with ≤5 line changes, clean CI, non-overlapping file paths, and obviously-correct intent (e.g. one-line dependency relax, env var add, import path fix) can be merged in a single loop without the review-per-PR ceremony:
```bash
for pr in 425 384 416 429; do
echo "=== Merging PR $pr ==="
gh pr merge $pr --squash
done
```
Verify afterward that each landed cleanly:
```bash
for pr in 425 384 416 429; do
gh pr view $pr --json state,mergeCommit --jq "{pr: $pr, state, sha: .mergeCommit.oid[0:7]}"
done
```
### 7. Post-merge follow-ups
Sometimes a PR is worth merging despite a known minor issue (e.g. incomplete dtype map, stale sentinel cleanup). Don't block the merge; apply the follow-up as a normal branch + PR right after:
```bash
cd <main-worktree>
git pull --ff-only origin main
git checkout -b fix/<short-name>
# edit...
git commit -m "fix(<area>): <one-liner>"
git push -u origin fix/<short-name>
gh pr create --title "..." --body "Follow-up to #<N>. ..."
```
Record both SHAs in the triage doc (`✅ merged <pr-sha> + follow-up <pr>`).
**Direct-to-main exception:** only under an explicit, scoped policy (e.g. "release speedrun"). Don't default to it.
### 8. Supersede: close with a credit-pointing comment
```bash
gh pr close <N> --comment "Closing — superseded by merged #<M> which landed <brief description>. Thanks!"
```
Check the diffs first — "similar title" is not enough. If the PR is *partially* superseded (the diagnosis is right but only half the changes are still needed), do a partial-apply instead.
### 9. Partial-apply pattern
When a PR has both valuable and questionable changes bundled:
```bash
cd <main-worktree>
git pull --ff-only origin main
# Cherry-pick specific files from the PR branch
git checkout <pr-commit-sha> -- <file1> <file2>
# Review the staged changes, adjust as needed
git diff --cached
# Apply any surgical edits to files you don't want to bulk-replace
# (e.g. the PR's file predates a recent main commit you need to preserve)
# Commit with a trailer crediting the original author
git commit -m "$(cat <<'EOF'
<subject>
<body explaining what was kept vs dropped>
Co-Authored-By: <author> <[email protected]>
EOF
)"
git push ... # branch + PR, unless under the direct-to-main exception
```
Then close the PR with a comment explaining what was applied and what was dropped, referencing the commit SHA.
### 10. Keep the doc current
Every merge, every close, every follow-up → update `<VERSION>_PR_TRIAGE.md`. The doc is your session log. If you're interrupted and resume tomorrow, the doc is the only source of truth for "where am I."
### 11. When triage is done
- Every PR in the doc has a terminal status (✅ merged / ✅ closed / deferred)
- Progress header shows N/N for each tier
- Next skill to run is `draft-release-notes` (to regenerate `[Unreleased]` against the new main), then `release-bump`
You can delete the triage doc after the release ships, or keep it in version history as a record.
## Gotchas
- **`main..HEAD` on a stale branch lies.** It shows everything main gained since the branch split as deletions. Always review via `git show HEAD` for the PR's actual commit.
- **Squash-merging an unrebased branch reverts in-between work.** The squash computes `diff(PR-head, merge-base)`. Rebase moves the merge-base forward.
- **`mergeable=UNKNOWN`** is transient — GitHub is recomputing after a push. Just try the merge.
- **Route ordering matters (FastAPI and similar):** `DELETE /history/failed` must be registered *before* `DELETE /history/{id}`, or the parameterized path will consume `"failed"` as an ID.
- **Apple's `-weak_framework` overrides `-framework`** for the same framework, regardless of order — use it via `cargo:rustc-link-arg=-Wl,-weak_framework,Name` when a dependency hard-links something optional.
- **Dependency version floors constrain what you can apply.** Before accepting a kwarg rename like `torch_dtype=` → `dtype=`, check the min-version pin supports it. Sometimes the right move is to cherry-pick half the PR.
- **`cpal::Stream` and similar `!Send` audio types** can't cross `await` points or `spawn_blocking`. Sometimes a "not-ideal but correct" sync wait is the best available fix; flag but don't block.
- **PyTorch nightly builds are not shippable for releases** — non-deterministic, can regress between runs. If a PR suggests switching to nightly to fix a GPU issue, prefer `TORCH_CUDA_ARCH_LIST=...+PTX` or wait for stable support instead.
## Canonical commands reference
```bash
# Bulk PR metadata
gh pr list --state open --limit 50 --json number,title,author,mergeable,mergeStateStatus,additions,deletions,maintainerCanModify,files
# Detailed single-PR view
gh pr view <N> --json body,author,headRefName,baseRefName,mergeable,maintainerCanModify,files,statusCheckRollup
# The actual commit, not the branch-vs-main diff
git show HEAD
git show --stat HEAD
gh pr diff <N>
# Rebase contributor branch onto current main
git fetch origin main && git rebase origin/main
# Push rebase back to contributor fork (maintainerCanModify=true required)
git remote add <author> https://github.com/<author>/<repo>.git
git fetch <author> <branch>
git push <author> HEAD:<branch> --force-with-lease
# Merge
gh pr merge <N> --squash
# Confirm merge SHA for triage doc
gh pr view <N> --json state,mergeCommit --jq '{state, sha: .mergeCommit.oid[0:7]}'
# Close superseded
gh pr close <N> --comment "Closing — superseded by merged #<M>. Thanks!"
```
## Notes
- **Never review a stale branch via `main..HEAD`.** This is the single most important line in this skill.
- **The triage doc is the session state.** Lose the doc, lose the session. Update it after every action.
- **Credit contributors even on partial-applies.** Use `Co-Authored-By:` trailers and close comments that link to the applied commit.
- **Don't let perfect be the enemy of shipped.** A fix that goes from "broken" to "works with a minor known issue" is a strict improvement. Flag the issue, file a follow-up, merge the fix.
+1 -1
View File
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.2.3
current_version = 0.4.0
commit = True
tag = True
tag_name = v{new_version}
-1
View File
@@ -38,7 +38,6 @@ biome.json
.bumpversion.cfg
.npmrc
Makefile
CHANGELOG.md
CONTRIBUTING.md
SECURITY.md
LICENSE
+27 -15
View File
@@ -62,6 +62,7 @@ jobs:
pip install pyinstaller
pip install -r backend/requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
- name: Install MLX dependencies (Apple Silicon only)
if: matrix.backend == 'mlx'
@@ -188,43 +189,54 @@ jobs:
pip install pyinstaller
pip install -r backend/requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
- name: Install PyTorch with CUDA 12.6
- name: Install PyTorch with CUDA 12.8
run: |
pip install torch --index-url https://download.pytorch.org/whl/cu126 --force-reinstall --no-deps
pip install torchaudio --index-url https://download.pytorch.org/whl/cu126 --force-reinstall --no-deps
pip install torch --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
pip install torchaudio --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
- name: Verify CUDA support in torch
run: |
python -c "import torch; print(f'CUDA available in build: {torch.cuda.is_available()}'); print(f'CUDA version: {torch.version.cuda}')"
- name: Build CUDA server binary
- name: Build CUDA server binary (onedir)
shell: bash
working-directory: backend
env:
# Include Blackwell (sm_120) via PTX forward compatibility.
# Pre-built PyTorch cu128 wheels ship native kernels for sm_80/86/89/90
# but not sm_120. Setting this env var causes torch.utils.cpp_extension
# (and any JIT-compiled kernels) to target Blackwell GPUs as well.
TORCH_CUDA_ARCH_LIST: "8.0;8.6;8.9;9.0;12.0+PTX"
run: python build_binary.py --cuda
- name: Split binary for GitHub Releases
- name: Package into server core + CUDA libs archives
shell: bash
run: |
python scripts/split_binary.py \
backend/dist/voicebox-server-cuda.exe \
--output release-assets/
python scripts/package_cuda.py \
backend/dist/voicebox-server-cuda/ \
--output release-assets/ \
--cuda-libs-version cu128-v1 \
--torch-compat ">=2.7.0,<2.11.0"
- name: Upload split parts to GitHub Release
- name: Upload archives to GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v1
uses: softprops/action-gh-release@v2
with:
files: |
release-assets/voicebox-server-cuda.part*.exe
release-assets/voicebox-server-cuda.sha256
release-assets/voicebox-server-cuda.manifest
release-assets/voicebox-server-cuda.tar.gz
release-assets/voicebox-server-cuda.tar.gz.sha256
release-assets/cuda-libs-cu128-v1.tar.gz
release-assets/cuda-libs-cu128-v1.tar.gz.sha256
release-assets/cuda-libs.json
draft: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload binary as workflow artifact
- name: Upload onedir as workflow artifact
uses: actions/upload-artifact@v4
with:
name: voicebox-server-cuda-windows
path: backend/dist/voicebox-server-cuda.exe
path: backend/dist/voicebox-server-cuda/
retention-days: 7
+7
View File
@@ -51,6 +51,13 @@ app/openapi.json
tauri/src-tauri/binaries/*
tauri/src-tauri/gen/Assets.car
tauri/src-tauri/gen/voicebox.icns
tauri/src-tauri/gen/partial.plist
# PyInstaller
*.spec
# Windows artifacts
nul
# Temporary
tmp/
+143 -3
View File
@@ -7,9 +7,136 @@
## [Unreleased]
This release rewrites the backend into a modular architecture, migrates the documentation site to Fumadocs, and ships a batch of bug fixes and UI polish across the stack.
## [0.4.0] - 2026-04-16
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, model loading status is now visible in the UI, effects presets get a dropdown, and several race conditions and accessibility gaps are closed.
The biggest Voicebox release yet. Three new TTS engines bring the lineup to **seven** — HumeAI TADA, Kokoro 82M, and Qwen CustomVoice join Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo. GPU support broadens to Intel Arc (XPU) and NVIDIA Blackwell (RTX 50-series), with runtime diagnostics that warn when your PyTorch build doesn't match your GPU. The CUDA backend is now split into independently versioned server and library archives, so upgrading no longer redownloads 4 GB of PyTorch/CUDA DLLs.
This release also marks a big community moment: **13 new contributors** shipped fixes and features in 0.4.0. Thirty-plus bug fixes target the most-reported issues in the tracker — numpy 2.x TTS crashes, Windows background-server reliability, macOS 11 launch failures, audio playback silence, Stories clip-splitting races, history status staleness, and more.
### New TTS Engines
#### HumeAI TADA — Expressive English & Multilingual ([#296](https://github.com/jamiepine/voicebox/pull/296))
- Added `tada-1b` (English) and `tada-3b-ml` (multilingual) backends
- Replaced `descript-audio-codec` with a lightweight DAC shim to cut dependencies
- Switched audio decoding to `soundfile` to sidestep `torchcodec` bundling issues
- Redirected gated Llama tokenizer lookups to an ungated mirror so model loading works out of the box
- Fixed tokenizer patch that was corrupting `AutoTokenizer` for other engines
- Fixed TorchScript error in frozen builds
#### Kokoro 82M — Fast Lightweight TTS ([#325](https://github.com/jamiepine/voicebox/pull/325))
- Added Kokoro 82M engine with a new voice profile type system that distinguishes preset voices from cloned profiles
- Profile grid now handles engine compatibility directly — removed redundant dropdown filtering
- Tightened Kokoro profile handling so preset voices can't be edited like cloned profiles
#### Qwen CustomVoice ([#328](https://github.com/jamiepine/voicebox/pull/328))
- Added `qwen-custom-voice` preset engine backed by Qwen3-TTS
- Enforced preset/profile engine compatibility across the generation flow
- Floating generator now shows all engines instead of silently filtering
### Voice Profile UX
Until 0.4, every engine in Voicebox was a cloning model, so every voice profile was usable with every engine and the profile grid just showed them all. Introducing Kokoro and Qwen CustomVoice — which work from preset voices rather than cloned samples — broke that assumption for the first time. An early cut on `main` filtered the grid by the selected engine, which left users running pre-release builds thinking their cloned voices had vanished whenever they switched to a preset-only engine.
This release ships the resolution before it ever reaches a tagged version:
- **Grey-out instead of filter** — all profiles are always visible; unsupported ones render dimmed with a compatibility hint at the bottom of the grid
- **Auto-switch on selection** — clicking a greyed-out profile selects it AND switches the engine to a compatible one, instead of silently doing nothing
- **Instruct toggle restored for Qwen CustomVoice** — the floating generate box now reveals a delivery-instructions input (tone, emotion, pace) when CustomVoice is selected. Hidden across the board while the new multi-engine lineup was stabilizing because most engines don't honor the kwarg; now conditionally exposed only for the one engine that was actually trained for instruction-based style control
- Supported profiles sort first; the grid scrolls the selected profile into view after engine/sort changes
- Fixed engine desync on tab navigation — the form now initializes its engine from the store
- Fixed the disabled-and-selected card click edge case by bouncing selection to re-trigger the auto-switch
- Cleaned up scroll effect timers (requestAnimationFrame + setTimeout) to prevent stale DOM writes on unmount or rapid selection changes
### GPU & Platform
#### Intel Arc (XPU) Support ([#320](https://github.com/jamiepine/voicebox/pull/320))
- First-class Intel Arc support across all PyTorch-based backends
- Device-aware seeding, XPU detection in the GPU status panel, and setup flow detection
- Reports correct device name and VRAM in settings
#### Blackwell / RTX 50-series Support ([#316](https://github.com/jamiepine/voicebox/pull/316), [#401](https://github.com/jamiepine/voicebox/pull/401))
- Upgraded the CUDA backend from cu126 → cu128 for RTX 50-series support
- Added `sm_120+PTX` to the CUDA build via `TORCH_CUDA_ARCH_LIST` for forward-compatibility with Blackwell architectures (closes 5 open reports: #386, #395, #396, #399, #400)
- GPU settings UI fixes around install/uninstall state
#### GPU Compatibility Diagnostics ([#367](https://github.com/jamiepine/voicebox/pull/367), adapted)
- New `check_cuda_compatibility()` compares the current device's compute capability against the bundled PyTorch's architecture list
- Health endpoint exposes a `gpu_compatibility_warning` field so the UI can surface mismatches
- Startup logs a `WARN` when the installed PyTorch build doesn't support the detected GPU
- GPU status label shows `[UNSUPPORTED - see logs]` — no more silent "no kernel image" failures
#### Split CUDA Backend ([#298](https://github.com/jamiepine/voicebox/pull/298))
- CUDA backend now ships as two independently versioned archives: a small server binary and a large libs archive (the ~4 GB of PyTorch/CUDA DLLs)
- Upgrading Voicebox no longer redownloads the libs archive when only the server binary changed
- Added `asyncio.Lock` around `download_cuda_binary()` so auto-update and manual download can't race on the same temp file ([#428](https://github.com/jamiepine/voicebox/pull/428))
- Updated `package_cuda.py` for PyInstaller 6.18 onedir layout
- Temp archives are always cleaned up on failure, even when the install aborts mid-extract
### Bug Fixes
#### Critical: TTS Generation
- **numpy 2.x `torch.from_numpy` crash** ([#361](https://github.com/jamiepine/voicebox/pull/361)) — torch compiled against numpy 1.x ABI fails silently when paired with numpy 2.x, causing `RuntimeError: Numpy is not available` / `Unable to create tensor` on every TTS request in bundled macOS Intel / Rosetta builds. Pinned `numpy<2.0` in requirements and added a PyInstaller runtime hook with a `ctypes.memmove` fallback as belt-and-suspenders. Hardened afterward to raise on unknown dtypes instead of silently reinterpreting bytes as float32.
#### Platform Reliability
- **Windows background server** ([#402](https://github.com/jamiepine/voicebox/pull/402)) — "keep server running after close" now actually keeps the server running. The HTTP `/watchdog/disable` request could lose the race against process exit on Windows; added a `.keep-running` sentinel file as a synchronous fallback, with stale-sentinel cleanup on startup to avoid orphan server processes
- **macOS 11 launch crash** ([#424](https://github.com/jamiepine/voicebox/pull/424)) — weak-linked ScreenCaptureKit so the app can launch on macOS < 12.3 instead of crashing at dyld resolution. Gated system audio capture behind a real `sw_vers` version check so unsupported systems cleanly advertise "not available" rather than crashing at runtime
- **macOS Intel (x86_64) setup** ([#416](https://github.com/jamiepine/voicebox/pull/416)) — relaxed `torch>=2.7.0` → `torch>=2.2.0`. PyTorch dropped pre-built x86_64 wheels after 2.2.2, so Intel Mac devs could no longer `pip install`. Now resolves to the latest compatible torch per platform
- **Offline model loading** ([#318](https://github.com/jamiepine/voicebox/pull/318)) — Qwen TTS and Whisper force offline mode when loading cached models, so startup works without network access
- **GUI startup with external server** ([#319](https://github.com/jamiepine/voicebox/pull/319)) — fixed GUI launch when pointed at a remote/external server, and added data refresh on server switch; hardened health validation and error handling
- **Qwen3-TTS cache split on Windows** (adapted from [#218](https://github.com/jamiepine/voicebox/pull/218)) — route `Qwen3TTSModel.from_pretrained` through `hf_constants.HF_HUB_CACHE` so the speech tokenizer and `preprocessor_config.json` resolve from a single cache root
- **Qwen3-TTS bundling** ([#305](https://github.com/jamiepine/voicebox/pull/305)) — bundle `qwen_tts` source files in the PyInstaller build to fix `inspect.getsource` errors in frozen builds
- **Backend import paths** ([#345](https://github.com/jamiepine/voicebox/pull/345)) — moved lazy imports to top-level with absolute paths to resolve the "Failed to Save" preset error caused by `ModuleNotFoundError` in production builds
- **Effects service import** ([#384](https://github.com/jamiepine/voicebox/pull/384)) — fixed `ModuleNotFoundError` on preset create/update by switching to relative imports (#349)
#### Audio & Playback
- **cpal stream silent playback** ([#405](https://github.com/jamiepine/voicebox/pull/405)) — `cpal::Stream` was dropped on function return immediately after `play()`, causing every playback to fall silent. Now holds the stream until either the buffer drains or the stop flag fires (#404)
#### Stories & History
- **Clip-splitting race** ([#403](https://github.com/jamiepine/voicebox/pull/403)) — rapid double-clicks on split could race through `split_story_item` with inconsistent state. Added `with_for_update()` row locking on the backend and an `isPending` guard on the frontend (#366)
- **History `status` staleness** ([#394](https://github.com/jamiepine/voicebox/pull/394)) — `GET /history/{id}` was hardcoding `status="completed"` regardless of the DB row, breaking any client polling for job completion. Now returns `status`, `error`, `engine`, `model_size`, and `is_favorited` from the actual row
- **"Clear failed" bulk button** ([#412](https://github.com/jamiepine/voicebox/pull/412)) — new `DELETE /history/failed` endpoint and a header strip showing `"N failed generations"` with a Clear button, complementing the per-row trash icon added in #321 (#410)
- **Delete failed generations** ([#321](https://github.com/jamiepine/voicebox/pull/321)) — added a trash icon next to the retry button so failed entries can be cleaned up without having to retry first
#### Security & Safety
- **Voice prompt cache hardening** ([#429](https://github.com/jamiepine/voicebox/pull/429)) — `torch.load(weights_only=True)` on cached voice prompts per PyTorch 2.6 recommendation; replaced string-based SPA path guard with `Path.is_relative_to()` for more robust path-traversal protection
#### Infrastructure & Docker
- **Docker web build** ([#344](https://github.com/jamiepine/voicebox/pull/344)) — include `CHANGELOG.md` in the Docker web build so the in-app changelog page works in Docker deployments
- **Docker numba cache** ([#425](https://github.com/jamiepine/voicebox/pull/425)) — set `NUMBA_CACHE_DIR` in docker-compose so numba can write its JIT cache in container runtime (#308)
- **Relative media paths** ([#332](https://github.com/jamiepine/voicebox/pull/332)) — media paths now stored relative to the configured data dir rather than resolved against CWD, so the data directory is portable between installs
### Developer Tooling
- New `triage-prs` agent skill — encodes the end-to-end PR-speedrun workflow (classification → triage doc → rebase → squash-merge → follow-ups) so future release cycles can reproduce it
- Rewrote the TTS engine guide with the patterns learned from adding TADA and Kokoro
- Added the API refactor plan and CUDA libs addon design doc
- Fixed broken links in the Get Started section ([#332](https://github.com/jamiepine/voicebox/pull/332))
### New Contributors
Huge thank you to everyone who contributed their first PR to Voicebox in this release:
[@liorshahverdi](https://github.com/liorshahverdi), [@nicoschtein](https://github.com/nicoschtein), [@ArfianID](https://github.com/ArfianID), [@aimaaaimaa](https://github.com/aimaaaimaa), [@maxmcoding](https://github.com/maxmcoding), [@Khalodddd](https://github.com/Khalodddd), [@LuisSambrano](https://github.com/LuisSambrano), [@shaun0927](https://github.com/shaun0927), [@malletfils](https://github.com/malletfils), [@mvanhorn](https://github.com/mvanhorn), [@kuishou68](https://github.com/kuishou68), [@txhno](https://github.com/txhno), [@MukundaKatta](https://github.com/MukundaKatta)
## [0.3.0] - 2026-03-17
This release rewrites the backend into a modular architecture, overhauls the settings UI into routed sub-pages, fixes audio player freezing, migrates documentation to Fumadocs, and ships a batch of bug fixes targeting the most-reported issues from the tracker.
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, settings have been split into dedicated routed pages with server logs, a changelog viewer, and an about page. The audio player no longer freezes mid-playback, and model loading status is now visible in the UI. Seven user-reported bugs have been fixed, including server crashes during sample uploads, generation list staleness, cryptic error messages, and CUDA support for RTX 50-series GPUs.
### Settings Overhaul ([#294](https://github.com/jamiepine/voicebox/pull/294))
- Split settings into routed sub-tabs: General, Generation, GPU, Logs, Changelog, About
- Added live server log viewer with auto-scroll
- Added in-app changelog page that parses `CHANGELOG.md` at build time
- Added About page with version info, license, and generation folder quick-open
- Extracted reusable `SettingRow` component for consistent setting layouts
### Audio Player Fix ([#293](https://github.com/jamiepine/voicebox/pull/293))
- Fixed audio player freezing during playback
- Improved playback UX with better state management and listener cleanup
- Fixed restart race condition during regeneration
- Added stable keys for audio element re-rendering
- Improved accessibility across player controls
### Backend Refactor ([#285](https://github.com/jamiepine/voicebox/pull/285))
- Extracted all routes from `main.py` into 13 domain routers under `backend/routes/` — `main.py` dropped from ~3,100 lines to ~10
@@ -40,6 +167,17 @@ The backend's 3,000-line monolith `main.py` has been decomposed into domain rout
- Softened select focus indicator opacity
- Addressed 4 critical and 12 major issues from CodeRabbit review
### Bug Fixes ([#295](https://github.com/jamiepine/voicebox/pull/295))
- Fixed sample uploads crashing the server — audio decoding now runs in a thread pool instead of blocking the async event loop ([#278](https://github.com/jamiepine/voicebox/issues/278))
- Fixed generation list not updating when a generation completes — switched to `refetchQueries` for reliable cache busting, added SSE error fallback, and page reset on completion ([#231](https://github.com/jamiepine/voicebox/issues/231))
- Fixed error toasts showing `[object Object]` instead of the actual error message ([#290](https://github.com/jamiepine/voicebox/issues/290))
- Added Whisper model selection (`base`, `small`, `medium`, `large`, `turbo`) and expanded language support to the `/transcribe` endpoint ([#233](https://github.com/jamiepine/voicebox/issues/233))
- Upgraded CUDA backend build from cu121 to cu126 for RTX 50-series (Blackwell) GPU support ([#289](https://github.com/jamiepine/voicebox/issues/289))
- Handled client disconnects in SSE and streaming endpoints to suppress `[Errno 32] Broken Pipe` errors ([#248](https://github.com/jamiepine/voicebox/issues/248))
- Fixed Docker build failure from pip hash mismatch on Qwen3-TTS dependencies ([#286](https://github.com/jamiepine/voicebox/issues/286))
- Added 50 MB upload size limit with chunked reads to prevent unbounded memory allocation on sample uploads
- Eliminated redundant double audio decode in sample processing pipeline
### Platform Fixes
- Replaced `netstat` with `TcpStream` + PowerShell for Windows port detection ([#277](https://github.com/jamiepine/voicebox/pull/277))
- Fixed Docker frontend build and cleaned up Docker docs
@@ -417,7 +555,9 @@ The first public release of Voicebox — an open-source voice synthesis studio p
Tauri v2, React, TypeScript, Tailwind CSS, FastAPI, Qwen3-TTS, Whisper, SQLite
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.2.3...HEAD
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/jamiepine/voicebox/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/jamiepine/voicebox/compare/v0.2.3...v0.3.0
[0.2.3]: https://github.com/jamiepine/voicebox/compare/v0.2.2...v0.2.3
[0.2.2]: https://github.com/jamiepine/voicebox/compare/v0.2.1...v0.2.2
[0.2.1]: https://github.com/jamiepine/voicebox/compare/v0.1.13...v0.2.1
+3 -1
View File
@@ -9,7 +9,7 @@ FROM oven/bun:1 AS frontend
WORKDIR /build
# Copy workspace config and frontend source
COPY package.json bun.lock ./
COPY package.json bun.lock CHANGELOG.md ./
COPY app/ ./app/
COPY web/ ./web/
@@ -35,6 +35,8 @@ RUN pip install --no-cache-dir --upgrade pip
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
RUN pip install --no-cache-dir --prefix=/install --no-deps chatterbox-tts
RUN pip install --no-cache-dir --prefix=/install --no-deps hume-tada
RUN pip install --no-cache-dir --prefix=/install \
git+https://github.com/QwenLM/Qwen3-TTS.git
+12 -5
View File
@@ -59,10 +59,10 @@
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** — models and voice data stay on your machine
- **4 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -93,7 +93,7 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
### Multi-Engine Voice Cloning
Four TTS engines with different strengths, switchable per-generation:
Five TTS engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
@@ -101,6 +101,7 @@ Four TTS engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
### Emotions & Paralinguistic Tags
@@ -230,7 +231,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
@@ -245,7 +246,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| ----------------------- | ---------------------------------------------- |
| **Real-time Streaming** | Stream audio as it generates, word by word |
| **Voice Design** | Create new voices from text descriptions |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **More Models** | XTTS, Bark, and other open-source voice models |
| **Plugin Architecture** | Extend with custom models and effects |
| **Mobile Companion** | Control Voicebox from your phone |
@@ -276,6 +277,12 @@ just build # Build CPU server binary + Tauri app
just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app
```
### Adding New Voice Models
The multi-engine architecture makes adding new TTS engines straightforward. A [step-by-step guide](docs/content/docs/developer/tts-engines.mdx) covers the full process: dependency research, backend protocol implementation, frontend wiring, and PyInstaller bundling.
The guide is optimized for AI coding agents. An [agent skill](.agents/skills/add-tts-engine/SKILL.md) can pick up a model name and handle the entire integration autonomously — you just test the build locally.
### Project Structure
```
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@voicebox/app",
"version": "0.2.3",
"version": "0.4.0",
"private": true,
"type": "module",
"scripts": {
+98 -9
View File
@@ -4,6 +4,8 @@ import voiceboxLogo from '@/assets/voicebox-logo.png';
import ShinyText from '@/components/ShinyText';
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
import { apiClient } from '@/lib/api/client';
import type { HealthResponse } from '@/lib/api/types';
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import { cn } from '@/lib/utils/cn';
import { usePlatform } from '@/platform/PlatformContext';
@@ -11,6 +13,33 @@ import { router } from '@/router';
import { useLogStore } from '@/stores/logStore';
import { useServerStore } from '@/stores/serverStore';
/**
* Validate that a health response has the expected Voicebox-specific shape.
* Prevents misidentifying an unrelated service on the same port.
*/
function isVoiceboxHealthResponse(health: HealthResponse): boolean {
return (
health?.status === 'healthy' &&
typeof health.model_loaded === 'boolean' &&
typeof health.gpu_available === 'boolean'
);
}
/**
* Check whether a startup error indicates the port is occupied by an external
* server (which we should try to reuse via health-check polling) vs. a real
* failure (missing sidecar, signing issue, etc.) that should surface immediately.
*/
function isPortInUseError(error: unknown): boolean {
const msg = error instanceof Error ? error.message : String(error);
return (
msg.includes('already in use') ||
msg.includes('port') ||
msg.includes('EADDRINUSE') ||
msg.includes('address already in use')
);
}
const LOADING_MESSAGES = [
'Warming up tensors...',
'Calibrating synthesizer engine...',
@@ -37,6 +66,7 @@ const LOADING_MESSAGES = [
function App() {
const platform = usePlatform();
const [serverReady, setServerReady] = useState(false);
const [startupError, setStartupError] = useState<string | null>(null);
const [loadingMessageIndex, setLoadingMessageIndex] = useState(0);
const serverStartingRef = useRef(false);
@@ -122,6 +152,46 @@ function App() {
serverStartingRef.current = false;
// @ts-expect-error - adding property to window
window.__voiceboxServerStartedByApp = false;
// Only fall back to health-check polling when the error indicates the
// port is occupied (likely an external server). For real failures
// (missing sidecar, signing issues, etc.) surface the error immediately.
if (!isPortInUseError(error)) {
const msg = error instanceof Error ? error.message : String(error);
console.error('Real startup failure — not polling:', msg);
setStartupError(msg);
return;
}
// Fall back to polling: the server may already be running externally
// (e.g. started via python/uvicorn/Docker). Poll the health endpoint
// until it responds with a valid Voicebox payload, then transition to
// the main UI.
console.log('Falling back to health-check polling...');
const pollInterval = setInterval(async () => {
try {
const health = await apiClient.getHealth();
if (!isVoiceboxHealthResponse(health)) {
console.log('Health response is not from a Voicebox server, keep polling...');
return;
}
console.log('External Voicebox server detected via health check');
clearInterval(pollInterval);
setServerReady(true);
} catch {
// Server not ready yet, keep polling
}
}, 2000);
// Stop polling after 2 minutes and surface the failure
setTimeout(() => {
clearInterval(pollInterval);
serverStartingRef.current = false;
setStartupError(
'Could not connect to a Voicebox server within 2 minutes. ' +
'Please check that the server is running and try again.',
);
}, 120_000);
});
// Cleanup: stop server on actual unmount (not StrictMode remount)
@@ -168,15 +238,34 @@ function App() {
className="w-48 h-48 object-contain animate-fade-in-scale relative z-10"
/>
</div>
<div className="animate-fade-in-delayed">
<ShinyText
text={LOADING_MESSAGES[loadingMessageIndex]}
className="text-lg font-medium text-muted-foreground"
speed={2}
color="hsl(var(--muted-foreground))"
shineColor="hsl(var(--foreground))"
/>
</div>
{startupError ? (
<div className="animate-fade-in-delayed max-w-md mx-auto space-y-3">
<p className="text-lg font-medium text-destructive">Server startup failed</p>
<p className="text-sm text-muted-foreground">{startupError}</p>
<button
type="button"
className="mt-2 px-4 py-2 text-sm rounded-md bg-primary text-primary-foreground hover:bg-primary/90 transition-colors"
onClick={() => {
setStartupError(null);
serverStartingRef.current = false;
// Trigger a re-mount of the effect by toggling state
window.location.reload();
}}
>
Retry
</button>
</div>
) : (
<div className="animate-fade-in-delayed">
<ShinyText
text={LOADING_MESSAGES[loadingMessageIndex]}
className="text-lg font-medium text-muted-foreground"
speed={2}
color="hsl(var(--muted-foreground))"
shineColor="hsl(var(--foreground))"
/>
</div>
)}
</div>
</div>
);
+5 -3
View File
@@ -14,15 +14,17 @@ interface AppFrameProps {
export function AppFrame({ children }: AppFrameProps) {
const routerState = useRouterState();
const isStoriesRoute = routerState.location.pathname === '/stories';
const selectedStoryId = useStoryStore((state) => state.selectedStoryId);
const { data: story } = useStory(selectedStoryId);
// Show track editor when on stories route with a selected story that has items
const showTrackEditor = isStoriesRoute && selectedStoryId && story && story.items.length > 0;
return (
<div className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}>
<div
className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}
>
<TitleBarDragRegion />
{children}
{showTrackEditor ? (
+16 -16
View File
@@ -1,20 +1,20 @@
import {EffectsDetail} from "./EffectsDetail";
import {EffectsList} from "./EffectsList";
import { EffectsDetail } from './EffectsDetail';
import { EffectsList } from './EffectsList';
export function EffectsTab() {
return (
<div className="flex flex-col h-full min-h-0 overflow-hidden">
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
{/* Left - Presets list */}
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
<EffectsList />
</div>
return (
<div className="flex flex-col h-full min-h-0 overflow-hidden">
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
{/* Left - Presets list */}
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
<EffectsList />
</div>
{/* Right - Detail / editor */}
<div className="flex-1 min-h-0 flex flex-col">
<EffectsDetail />
</div>
</div>
</div>
);
{/* Right - Detail / editor */}
<div className="flex-1 min-h-0 flex flex-col">
<EffectsDetail />
</div>
</div>
</div>
);
}
@@ -1,3 +1,4 @@
import { useEffect } from 'react';
import type { UseFormReturn } from 'react-hook-form';
import { FormControl } from '@/components/ui/form';
import {
@@ -7,6 +8,7 @@ import {
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import type { VoiceProfileResponse } from '@/lib/api/types';
import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
import type { GenerationFormValues } from '@/lib/hooks/useGenerationForm';
@@ -15,30 +17,57 @@ import type { GenerationFormValues } from '@/lib/hooks/useGenerationForm';
* Adding a new engine means adding one entry here.
*/
const ENGINE_OPTIONS = [
{ value: 'qwen:1.7B', label: 'Qwen3-TTS 1.7B' },
{ value: 'qwen:0.6B', label: 'Qwen3-TTS 0.6B' },
{ value: 'luxtts', label: 'LuxTTS' },
{ value: 'chatterbox', label: 'Chatterbox' },
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
{ value: 'qwen:1.7B', label: 'Qwen3-TTS 1.7B', engine: 'qwen' },
{ value: 'qwen:0.6B', label: 'Qwen3-TTS 0.6B', engine: 'qwen' },
{ value: 'qwen_custom_voice:1.7B', label: 'Qwen CustomVoice 1.7B', engine: 'qwen_custom_voice' },
{ value: 'qwen_custom_voice:0.6B', label: 'Qwen CustomVoice 0.6B', engine: 'qwen_custom_voice' },
{ value: 'luxtts', label: 'LuxTTS', engine: 'luxtts' },
{ value: 'chatterbox', label: 'Chatterbox', engine: 'chatterbox' },
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo', engine: 'chatterbox_turbo' },
{ value: 'tada:1B', label: 'TADA 1B', engine: 'tada' },
{ value: 'tada:3B', label: 'TADA 3B Multilingual', engine: 'tada' },
{ value: 'kokoro', label: 'Kokoro 82M', engine: 'kokoro' },
] as const;
const ENGINE_DESCRIPTIONS: Record<string, string> = {
qwen: 'Multi-language, two sizes',
qwen_custom_voice: '9 preset voices, instruct control',
luxtts: 'Fast, English-focused',
chatterbox: '23 languages, incl. Hebrew',
chatterbox_turbo: 'English, [laugh] [cough] tags',
tada: 'HumeAI, 700s+ coherent audio',
kokoro: '82M params, CPU realtime, 8 langs',
};
/** Engines that only support English and should force language to 'en' on select. */
const ENGLISH_ONLY_ENGINES = new Set(['luxtts', 'chatterbox_turbo']);
/** Engines that support cloned (reference audio) profiles. */
const CLONING_ENGINES = new Set(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada']);
function getAvailableOptions(selectedProfile?: VoiceProfileResponse | null) {
if (!selectedProfile) return ENGINE_OPTIONS;
return ENGINE_OPTIONS.filter((opt) => isProfileCompatibleWithEngine(selectedProfile, opt.engine));
}
function getSelectValue(engine: string, modelSize?: string): string {
if (engine === 'qwen') return `qwen:${modelSize || '1.7B'}`;
if (engine === 'qwen_custom_voice') return `qwen_custom_voice:${modelSize || '1.7B'}`;
if (engine === 'tada') return `tada:${modelSize || '1B'}`;
return engine;
}
function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: string) {
if (value.startsWith('qwen:')) {
export function applyEngineSelection(form: UseFormReturn<GenerationFormValues>, value: string) {
if (value.startsWith('qwen_custom_voice:')) {
const [, modelSize] = value.split(':');
form.setValue('engine', 'qwen_custom_voice');
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
const currentLang = form.getValues('language');
const available = getLanguageOptionsForEngine('qwen_custom_voice');
if (!available.some((l) => l.value === currentLang)) {
form.setValue('language', available[0]?.value ?? 'en');
}
} else if (value.startsWith('qwen:')) {
const [, modelSize] = value.split(':');
form.setValue('engine', 'qwen');
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
@@ -48,6 +77,20 @@ function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: st
if (!available.some((l) => l.value === currentLang)) {
form.setValue('language', available[0]?.value ?? 'en');
}
} else if (value.startsWith('tada:')) {
const [, modelSize] = value.split(':');
form.setValue('engine', 'tada');
form.setValue('modelSize', modelSize as '1B' | '3B');
// TADA 1B is English-only; 3B is multilingual
if (modelSize === '1B') {
form.setValue('language', 'en');
} else {
const currentLang = form.getValues('language');
const available = getLanguageOptionsForEngine('tada');
if (!available.some((l) => l.value === currentLang)) {
form.setValue('language', available[0]?.value ?? 'en');
}
}
} else {
form.setValue('engine', value as GenerationFormValues['engine']);
form.setValue('modelSize', undefined as unknown as '1.7B' | '0.6B');
@@ -67,12 +110,22 @@ function handleEngineChange(form: UseFormReturn<GenerationFormValues>, value: st
interface EngineModelSelectorProps {
form: UseFormReturn<GenerationFormValues>;
compact?: boolean;
selectedProfile?: VoiceProfileResponse | null;
}
export function EngineModelSelector({ form, compact }: EngineModelSelectorProps) {
export function EngineModelSelector({ form, compact, selectedProfile }: EngineModelSelectorProps) {
const engine = form.watch('engine') || 'qwen';
const modelSize = form.watch('modelSize');
const selectValue = getSelectValue(engine, modelSize);
const availableOptions = getAvailableOptions(selectedProfile);
const currentEngineAvailable = availableOptions.some((opt) => opt.value === selectValue);
useEffect(() => {
if (!currentEngineAvailable && availableOptions.length > 0) {
applyEngineSelection(form, availableOptions[0].value);
}
}, [availableOptions, currentEngineAvailable, form]);
const itemClass = compact ? 'text-xs text-muted-foreground' : undefined;
const triggerClass = compact
@@ -80,14 +133,14 @@ export function EngineModelSelector({ form, compact }: EngineModelSelectorProps)
: undefined;
return (
<Select value={selectValue} onValueChange={(v) => handleEngineChange(form, v)}>
<Select value={selectValue} onValueChange={(v) => applyEngineSelection(form, v)}>
<FormControl>
<SelectTrigger className={triggerClass}>
<SelectValue />
</SelectTrigger>
</FormControl>
<SelectContent>
{ENGINE_OPTIONS.map((opt) => (
{availableOptions.map((opt) => (
<SelectItem key={opt.value} value={opt.value} className={itemClass}>
{opt.label}
</SelectItem>
@@ -101,3 +154,17 @@ export function EngineModelSelector({ form, compact }: EngineModelSelectorProps)
export function getEngineDescription(engine: string): string {
return ENGINE_DESCRIPTIONS[engine] ?? '';
}
/**
* Check if a profile is compatible with the currently selected engine.
* Useful for UI hints.
*/
export function isProfileCompatibleWithEngine(
profile: VoiceProfileResponse,
engine: string,
): boolean {
const voiceType = profile.voice_type || 'cloned';
if (voiceType === 'preset') return profile.preset_engine === engine;
if (voiceType === 'cloned') return CLONING_ENGINES.has(engine);
return true; // designed — future
}
@@ -1,7 +1,7 @@
import { useQuery } from '@tanstack/react-query';
import { useMatchRoute } from '@tanstack/react-router';
import { AnimatePresence, motion } from 'framer-motion';
import { Loader2, Sparkles } from 'lucide-react';
import { Loader2, SlidersHorizontal, Sparkles } from 'lucide-react';
import { useEffect, useRef, useState } from 'react';
import { Button } from '@/components/ui/button';
import { Form, FormControl, FormField, FormItem, FormMessage } from '@/components/ui/form';
@@ -36,9 +36,11 @@ export function FloatingGenerateBox({
}: FloatingGenerateBoxProps) {
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
const setSelectedEngine = useUIStore((state) => state.setSelectedEngine);
const { data: selectedProfile } = useProfile(selectedProfileId || '');
const { data: profiles } = useProfiles();
const [isExpanded, setIsExpanded] = useState(false);
const [isInstructExpanded, setIsInstructExpanded] = useState(false);
const [selectedPresetId, setSelectedPresetId] = useState<string | null>(null);
const containerRef = useRef<HTMLDivElement>(null);
const textareaRef = useRef<HTMLTextAreaElement | null>(null);
@@ -67,7 +69,12 @@ export function FloatingGenerateBox({
}
},
getEffectsChain: () => {
if (!selectedPresetId || !effectPresets) return undefined;
if (!selectedPresetId) return undefined;
// Profile's own effects chain (no matching preset)
if (selectedPresetId === '_profile') {
return selectedProfile?.effects_chain ?? undefined;
}
if (!effectPresets) return undefined;
const preset = effectPresets.find((p) => p.id === selectedPresetId);
return preset?.effects_chain;
},
@@ -110,12 +117,63 @@ export function FloatingGenerateBox({
}
}, [selectedProfileId, profiles, setSelectedProfileId]);
// Sync generation form language with selected profile's language
// Sync engine selection to global store so ProfileList can filter
const watchedEngine = form.watch('engine');
useEffect(() => {
if (watchedEngine) {
setSelectedEngine(watchedEngine);
}
}, [watchedEngine, setSelectedEngine]);
// Sync generation form language, engine, and effects with selected profile
type EngineValue =
| 'qwen'
| 'luxtts'
| 'chatterbox'
| 'chatterbox_turbo'
| 'tada'
| 'kokoro'
| 'qwen_custom_voice';
useEffect(() => {
if (selectedProfile?.language) {
form.setValue('language', selectedProfile.language as LanguageCode);
}
}, [selectedProfile, form]);
// Auto-switch engine to match the profile
const engine = selectedProfile?.default_engine ?? selectedProfile?.preset_engine;
if (engine) {
form.setValue('engine', engine as EngineValue);
} else if (selectedProfile && selectedProfile.voice_type !== 'preset') {
// Cloned/designed profile with no default — ensure a compatible (non-preset) engine
const currentEngine = form.getValues('engine');
const presetEngines = new Set(['kokoro', 'qwen_custom_voice']);
if (presetEngines.has(currentEngine)) {
form.setValue('engine', 'qwen');
}
}
// Pre-fill effects from profile defaults
if (
selectedProfile?.effects_chain &&
selectedProfile.effects_chain.length > 0 &&
effectPresets
) {
// Try to match against a known preset
const profileChainJson = JSON.stringify(selectedProfile.effects_chain);
const matchingPreset = effectPresets.find(
(p) => JSON.stringify(p.effects_chain) === profileChainJson,
);
if (matchingPreset) {
setSelectedPresetId(matchingPreset.id);
} else {
// No matching preset — use special value to pass profile chain directly
setSelectedPresetId('_profile');
}
} else if (
selectedProfile &&
(!selectedProfile.effects_chain || selectedProfile.effects_chain.length === 0)
) {
setSelectedPresetId(null);
}
}, [selectedProfile, effectPresets, form]);
// Auto-resize textarea based on content (only when expanded)
useEffect(() => {
@@ -296,9 +354,80 @@ export function FloatingGenerateBox({
: 'Generate speech'}
</span>
</div>
{/* Instruct toggle — only for Qwen CustomVoice, which actually honors the kwarg */}
<AnimatePresence>
{isExpanded && form.watch('engine') === 'qwen_custom_voice' && (
<motion.div
initial={{ opacity: 0, scale: 0.8 }}
animate={{ opacity: 1, scale: 1 }}
exit={{ opacity: 0, scale: 0.8 }}
transition={{ duration: 0.2 }}
className="absolute top-0 right-[calc(100%+0.5rem)]"
>
<div className="group relative">
<Button
type="button"
variant="ghost"
size="icon"
onClick={() => setIsInstructExpanded((prev) => !prev)}
className={cn(
'h-10 w-10 rounded-full transition-all duration-200',
isInstructExpanded
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
: 'bg-card border border-border hover:bg-background/50',
)}
aria-label={
isInstructExpanded
? 'Hide delivery instructions'
: 'Show delivery instructions'
}
aria-pressed={isInstructExpanded}
>
<SlidersHorizontal className="h-4 w-4" />
</Button>
<span className="pointer-events-none absolute bottom-full left-1/2 -translate-x-1/2 mb-2 whitespace-nowrap rounded-md bg-popover px-3 py-1.5 text-xs text-popover-foreground border border-border opacity-0 transition-opacity group-hover:opacity-100 z-[9999]">
Delivery instructions (tone, emotion, pace)
</span>
</div>
</motion.div>
)}
</AnimatePresence>
</div>
</div>
{/* Additive instruct textarea — shown below main text when toggle is on and engine supports it */}
<AnimatePresence>
{isInstructExpanded && form.watch('engine') === 'qwen_custom_voice' && (
<motion.div
initial={{ opacity: 0, height: 0 }}
animate={{ opacity: 1, height: 'auto' }}
exit={{ opacity: 0, height: 0 }}
transition={{ duration: 0.2, ease: 'easeOut' }}
className="overflow-hidden"
>
<FormField
control={form.control}
name="instruct"
render={({ field }) => (
<FormItem className="mt-2">
<FormControl>
<Textarea
{...field}
placeholder="Delivery instructions — e.g. Speak slowly with warmth, Authoritative and clear..."
className="resize-none bg-transparent border border-accent/20 focus-visible:ring-1 focus-visible:ring-accent/40 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full px-3 py-2"
style={{ minHeight: '60px', maxHeight: '160px' }}
maxLength={500}
/>
</FormControl>
<FormMessage className="text-xs" />
</FormItem>
)}
/>
</motion.div>
)}
</AnimatePresence>
<AnimatePresence>
<motion.div
initial={{ height: 0, opacity: 0 }}
@@ -375,6 +504,12 @@ export function FloatingGenerateBox({
<SelectItem value="none" className="text-xs">
No effects
</SelectItem>
{selectedProfile?.effects_chain &&
selectedProfile.effects_chain.length > 0 && (
<SelectItem value="_profile" className="text-xs">
Profile default
</SelectItem>
)}
{effectPresets?.map((preset) => (
<SelectItem key={preset.id} value={preset.id} className="text-xs">
{preset.name}
@@ -1,4 +1,5 @@
import { Loader2, Mic } from 'lucide-react';
import { useEffect } from 'react';
import { Button } from '@/components/ui/button';
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
import {
@@ -19,19 +20,45 @@ import {
SelectValue,
} from '@/components/ui/select';
import { Textarea } from '@/components/ui/textarea';
import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
import { getLanguageOptionsForEngine, type LanguageCode } from '@/lib/constants/languages';
import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
import { useProfile } from '@/lib/hooks/useProfiles';
import { useUIStore } from '@/stores/uiStore';
import { EngineModelSelector, getEngineDescription } from './EngineModelSelector';
import {
applyEngineSelection,
EngineModelSelector,
getEngineDescription,
} from './EngineModelSelector';
import { ParalinguisticInput } from './ParalinguisticInput';
function getEngineSelectValue(engine: string): string {
if (engine === 'qwen') return 'qwen:1.7B';
if (engine === 'qwen_custom_voice') return 'qwen_custom_voice:1.7B';
if (engine === 'tada') return 'tada:1B';
return engine;
}
export function GenerationForm() {
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
const { data: selectedProfile } = useProfile(selectedProfileId || '');
const { form, handleSubmit, isPending } = useGenerationForm();
useEffect(() => {
if (!selectedProfile) {
return;
}
if (selectedProfile.language) {
form.setValue('language', selectedProfile.language as LanguageCode);
}
const preferredEngine = selectedProfile.default_engine || selectedProfile.preset_engine;
if (preferredEngine) {
applyEngineSelection(form, getEngineSelectValue(preferredEngine));
}
}, [form, selectedProfile]);
async function onSubmit(data: Parameters<typeof handleSubmit>[0]) {
await handleSubmit(data, selectedProfileId);
}
@@ -91,7 +118,7 @@ export function GenerationForm() {
)}
/>
{form.watch('engine') === 'qwen' && (
{form.watch('engine') === 'qwen_custom_voice' && (
<FormField
control={form.control}
name="instruct"
@@ -118,7 +145,7 @@ export function GenerationForm() {
<div className="grid gap-4 md:grid-cols-3">
<FormItem>
<FormLabel>Model</FormLabel>
<EngineModelSelector form={form} />
<EngineModelSelector form={form} selectedProfile={selectedProfile} />
<FormDescription>
{getEngineDescription(form.watch('engine') || 'qwen')}
</FormDescription>
+89 -11
View File
@@ -45,6 +45,7 @@ import { apiClient } from '@/lib/api/client';
import type { EffectConfig, GenerationVersionResponse, HistoryResponse } from '@/lib/api/types';
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import {
useClearFailedGenerations,
useDeleteGeneration,
useExportGeneration,
useExportGenerationAudio,
@@ -124,6 +125,8 @@ export function HistoryTable() {
});
const deleteGeneration = useDeleteGeneration();
const clearFailed = useClearFailedGenerations();
const [clearFailedDialogOpen, setClearFailedDialogOpen] = useState(false);
const exportGeneration = useExportGeneration();
const exportGenerationAudio = useExportGenerationAudio();
const importGeneration = useImportGeneration();
@@ -157,11 +160,11 @@ export function HistoryTable() {
const pendingCount = useGenerationStore((state) => state.pendingGenerationIds.size);
const prevPendingCountRef = useRef(pendingCount);
useEffect(() => {
if (deleteGeneration.isSuccess || importGeneration.isSuccess) {
if (deleteGeneration.isSuccess || importGeneration.isSuccess || clearFailed.isSuccess) {
setPage(0);
setAllHistory([]);
}
}, [deleteGeneration.isSuccess, importGeneration.isSuccess]);
}, [deleteGeneration.isSuccess, importGeneration.isSuccess, clearFailed.isSuccess]);
useEffect(() => {
// A generation finished (pending count decreased) — scroll back to show it
@@ -415,6 +418,27 @@ export function HistoryTable() {
const history = allHistory;
const hasMore = allHistory.length < total;
const failedCount = history.filter((g) => g.status === 'failed').length;
const handleClearFailedConfirm = () => {
clearFailed.mutate(undefined, {
onSuccess: (data) => {
setClearFailedDialogOpen(false);
toast({
title: 'Cleared failed generations',
description: `${data.deleted} failed ${data.deleted === 1 ? 'generation' : 'generations'} removed.`,
});
},
onError: (error) => {
setClearFailedDialogOpen(false);
toast({
title: 'Failed to clear',
description: error instanceof Error ? error.message : 'Unknown error',
variant: 'destructive',
});
},
});
};
return (
<div className="flex flex-col h-full min-h-0 relative">
@@ -424,6 +448,23 @@ export function HistoryTable() {
</div>
) : (
<>
{failedCount > 0 && (
<div className="flex items-center justify-between px-1 pb-2">
<span className="text-xs text-muted-foreground">
{failedCount} failed {failedCount === 1 ? 'generation' : 'generations'}
</span>
<Button
variant="ghost"
size="sm"
className="h-7 text-xs text-muted-foreground hover:text-destructive"
onClick={() => setClearFailedDialogOpen(true)}
disabled={clearFailed.isPending}
>
<Trash2 className="h-3 w-3 mr-1.5" />
{clearFailed.isPending ? 'Clearing...' : 'Clear failed'}
</Button>
</div>
)}
{isScrolled && (
<div className="absolute top-0 left-0 right-0 h-16 bg-gradient-to-b from-background to-transparent z-10 pointer-events-none" />
)}
@@ -569,15 +610,27 @@ export function HistoryTable() {
)}
{isFailed ? (
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Retry generation"
onClick={() => handleRetry(gen.id)}
>
<RotateCcw className="h-2 w-2" />
</Button>
<>
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Retry generation"
onClick={() => handleRetry(gen.id)}
>
<RotateCcw className="h-2 w-2" />
</Button>
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Delete generation"
disabled={deleteGeneration.isPending}
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
>
<Trash2 className="h-2 w-2" />
</Button>
</>
) : (
<>
<DropdownMenu>
@@ -747,6 +800,31 @@ export function HistoryTable() {
</DialogContent>
</Dialog>
<Dialog open={clearFailedDialogOpen} onOpenChange={setClearFailedDialogOpen}>
<DialogContent>
<DialogHeader>
<DialogTitle>Clear failed generations</DialogTitle>
<DialogDescription>
This will permanently delete {failedCount} failed{' '}
{failedCount === 1 ? 'generation' : 'generations'} from your history. This cannot be
undone.
</DialogDescription>
</DialogHeader>
<DialogFooter>
<Button variant="outline" onClick={() => setClearFailedDialogOpen(false)}>
Cancel
</Button>
<Button
variant="destructive"
onClick={handleClearFailedConfirm}
disabled={clearFailed.isPending}
>
{clearFailed.isPending ? 'Clearing...' : 'Clear all'}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
<Dialog open={importDialogOpen} onOpenChange={setImportDialogOpen}>
<DialogContent>
<DialogHeader>
@@ -243,7 +243,40 @@ export function GpuAcceleration() {
{/* Native GPU detected - no CUDA download needed */}
{/* CUDA download section - only show when no GPU is active (native or CUDA) */}
{/* Currently running CUDA - show switch back to CPU */}
{isCurrentlyCuda && platform.metadata.isTauri && (
<>
{restartPhase !== 'idle' ? (
<div className="flex items-center gap-2 p-3 rounded-lg bg-primary/5 border">
<Loader2 className="h-4 w-4 animate-spin" />
<span className="text-sm">
{restartPhase === 'stopping' && 'Stopping server...'}
{restartPhase === 'waiting' && 'Restarting server...'}
{restartPhase === 'ready' && 'Server restarted successfully!'}
</span>
</div>
) : (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
re-download later).
</p>
<Button onClick={handleSwitchToCpu} variant="outline" className="w-full" size="sm">
<RotateCw className="h-4 w-4 mr-2" />
Switch to CPU Backend
</Button>
</div>
)}
{error && (
<div className="flex items-center gap-2 text-sm text-destructive">
<AlertCircle className="h-4 w-4 shrink-0" />
<span>{error}</span>
</div>
)}
</>
)}
{/* CUDA download/manage section - show when no native GPU and not currently running CUDA */}
{!hasNativeGpu && !isCurrentlyCuda && (
<>
{/* Download progress (manual download or auto-update) */}
@@ -315,7 +348,7 @@ export function GpuAcceleration() {
)}
{/* Downloaded but not active - show switch button */}
{cudaAvailable && !isCurrentlyCuda && platform.metadata.isTauri && (
{cudaAvailable && platform.metadata.isTauri && (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
CUDA backend is downloaded and ready. Restart the server to enable GPU
@@ -328,27 +361,8 @@ export function GpuAcceleration() {
</div>
)}
{/* Currently active - show switch back to CPU */}
{isCurrentlyCuda && platform.metadata.isTauri && (
<div className="space-y-3">
<p className="text-sm text-muted-foreground">
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
re-download later).
</p>
<Button
onClick={handleSwitchToCpu}
variant="outline"
className="w-full"
size="sm"
>
<RotateCw className="h-4 w-4 mr-2" />
Switch to CPU Backend
</Button>
</div>
)}
{/* Delete option when downloaded (and not active) */}
{cudaAvailable && !isCurrentlyCuda && (
{cudaAvailable && (
<Button
onClick={handleDelete}
variant="ghost"
@@ -62,6 +62,16 @@ const MODEL_DESCRIPTIONS: Record<string, string> = {
'Production-grade open source TTS by Resemble AI. Supports 23 languages with voice cloning and emotion exaggeration control.',
'chatterbox-turbo':
'Streamlined 350M parameter TTS by Resemble AI. High-quality English speech with less compute and VRAM than larger models.',
'tada-1b':
'HumeAI TADA 1B — English speech-language model built on Llama 3.2 1B. Generates 700s+ of coherent audio with synchronized text-acoustic alignment.',
'tada-3b-ml':
'HumeAI TADA 3B Multilingual — built on Llama 3.2 3B. Supports 10 languages with high-fidelity voice cloning via text-acoustic dual alignment.',
kokoro:
'Kokoro 82M by hexgrad. Tiny 82M-parameter TTS that runs at CPU realtime. Supports 8 languages with pre-built voice styles. Apache 2.0 licensed.',
'qwen-custom-voice-1.7B':
'Qwen3-TTS CustomVoice 1.7B by Alibaba. 9 premium preset voices with instruct-based style control for tone, emotion, and prosody. Supports 10 languages.',
'qwen-custom-voice-0.6B':
'Qwen3-TTS CustomVoice 0.6B by Alibaba. Lightweight version with the same 9 preset voices and instruct control. Faster inference for lower-end hardware.',
'whisper-base':
'Smallest Whisper model (74M parameters). Fast transcription with moderate accuracy.',
'whisper-small':
@@ -390,8 +400,11 @@ export function ModelManagement() {
modelStatus?.models.filter(
(m) =>
m.model_name.startsWith('qwen-tts') ||
m.model_name.startsWith('qwen-custom-voice') ||
m.model_name.startsWith('luxtts') ||
m.model_name.startsWith('chatterbox'),
m.model_name.startsWith('chatterbox') ||
m.model_name.startsWith('tada') ||
m.model_name.startsWith('kokoro'),
) ?? [];
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
@@ -12,7 +12,11 @@ interface ModelProgressProps {
isDownloading?: boolean;
}
export function ModelProgress({ modelName, displayName, isDownloading = false }: ModelProgressProps) {
export function ModelProgress({
modelName,
displayName,
isDownloading = false,
}: ModelProgressProps) {
const [progress, setProgress] = useState<ModelProgressType | null>(null);
const serverUrl = useServerStore((state) => state.serverUrl);
@@ -182,8 +182,8 @@ function ChangelogEntryCard({ entry }: { entry: ChangelogEntry }) {
return (
<div className="border-b border-border/50 pb-6">
<div className="flex items-baseline gap-3 mb-1">
<h3 className="text-sm font-medium">{entry.version}</h3>
<div className="flex items-baseline gap-3 mb-3">
<h3 className="text-xl font-semibold tracking-tight">{entry.version}</h3>
{entry.date && <span className="text-xs text-muted-foreground">{entry.date}</span>}
{entry.version === 'Unreleased' && <Badge variant="outline">dev</Badge>}
</div>
+12 -16
View File
@@ -87,7 +87,7 @@ export function StoryChatItem({
alt={`${item.profile_name} avatar`}
className={cn(
'h-full w-full object-cover transition-all duration-200',
!isCurrentlyPlaying && 'grayscale'
!isCurrentlyPlaying && 'grayscale',
)}
onError={() => setAvatarError(true)}
/>
@@ -127,7 +127,10 @@ export function StoryChatItem({
<Play className="mr-2 h-4 w-4" />
Play from here
</DropdownMenuItem>
<DropdownMenuItem onClick={onRemove} className="text-destructive focus:text-destructive">
<DropdownMenuItem
onClick={onRemove}
className="text-destructive focus:text-destructive"
>
<Trash2 className="mr-2 h-4 w-4" />
Remove from Story
</DropdownMenuItem>
@@ -139,15 +142,12 @@ export function StoryChatItem({
}
// Sortable wrapper component
export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>) {
const {
attributes,
listeners,
setNodeRef,
transform,
transition,
isDragging,
} = useSortable({ id: props.item.generation_id });
export function SortableStoryChatItem(
props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>,
) {
const { attributes, listeners, setNodeRef, transform, transition, isDragging } = useSortable({
id: props.item.generation_id,
});
const style = {
transform: CSS.Transform.toString(transform),
@@ -156,11 +156,7 @@ export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandl
return (
<div ref={setNodeRef} style={style} {...attributes}>
<StoryChatItem
{...props}
dragHandleProps={listeners}
isDragging={isDragging}
/>
<StoryChatItem {...props} dragHandleProps={listeners} isDragging={isDragging} />
</div>
);
}
@@ -500,7 +500,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
}, [trimmingItem, trimSide, tempTrimValues, storyId, trimItem, toast]);
const handleSplit = useCallback(() => {
if (!selectedClipId) return;
if (!selectedClipId || splitItem.isPending) return;
const item = items.find((i) => i.id === selectedClipId);
if (!item) return;
@@ -14,12 +14,7 @@ const MemoizedWaveform = memo(function MemoizedWaveform({
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
{({ canvasRef }) => (
<canvas
ref={canvasRef}
width={500}
height={150}
className="w-full h-full"
/>
<canvas ref={canvasRef} width={500} height={150} className="w-full h-full" />
)}
</Visualizer>
</div>
@@ -87,9 +82,7 @@ export function AudioSampleRecording({
<div className="space-y-4">
{!isRecording && !file && (
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
{showWaveform && audioStream && (
<MemoizedWaveform audioStream={audioStream} />
)}
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
<Button
type="button"
onClick={onStart}
@@ -107,9 +100,7 @@ export function AudioSampleRecording({
{isRecording && (
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
{showWaveform && audioStream && (
<MemoizedWaveform audioStream={audioStream} />
)}
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
<div className="relative z-10 flex items-center gap-4">
<div className="flex items-center gap-2">
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
@@ -17,11 +17,18 @@ import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
import { cn } from '@/lib/utils/cn';
import { useUIStore } from '@/stores/uiStore';
/** Human-readable display names for preset engine badges. */
const ENGINE_DISPLAY_NAMES: Record<string, string> = {
kokoro: 'Kokoro',
qwen_custom_voice: 'CustomVoice',
};
interface ProfileCardProps {
profile: VoiceProfileResponse;
disabled?: boolean;
}
export function ProfileCard({ profile }: ProfileCardProps) {
export function ProfileCard({ profile, disabled }: ProfileCardProps) {
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
const deleteProfile = useDeleteProfile();
@@ -34,6 +41,12 @@ export function ProfileCard({ profile }: ProfileCardProps) {
const isSelected = selectedProfileId === profile.id;
const handleSelect = () => {
// If disabled but already selected, bounce the selection to re-trigger engine auto-switch
if (disabled && isSelected) {
setSelectedProfileId(null);
setTimeout(() => setSelectedProfileId(profile.id), 0);
return;
}
setSelectedProfileId(isSelected ? null : profile.id);
};
@@ -74,8 +87,9 @@ export function ProfileCard({ profile }: ProfileCardProps) {
<>
<Card
className={cn(
'cursor-pointer hover:shadow-md transition-all flex flex-col h-[162px]',
isSelected && 'ring-2 ring-accent shadow-md',
'cursor-pointer transition-all flex flex-col h-[162px]',
disabled ? 'opacity-40 hover:opacity-60' : 'hover:shadow-md',
isSelected && !disabled && 'ring-2 ring-accent shadow-md',
)}
onClick={handleSelect}
tabIndex={0}
@@ -97,6 +111,16 @@ export function ProfileCard({ profile }: ProfileCardProps) {
<Badge variant="outline" className="text-xs h-5 px-1.5 text-muted-foreground">
{profile.language}
</Badge>
{profile.voice_type === 'preset' && (
<Badge variant="secondary" className="text-xs h-5 px-1.5">
{ENGINE_DISPLAY_NAMES[profile.preset_engine ?? ''] ?? profile.preset_engine}
</Badge>
)}
{profile.voice_type === 'designed' && (
<Badge variant="secondary" className="text-xs h-5 px-1.5">
designed
</Badge>
)}
{profile.effects_chain && profile.effects_chain.length > 0 && (
<Sparkles className="h-3.5 w-3.5 text-accent fill-accent" />
)}
+416 -131
View File
@@ -1,9 +1,11 @@
import { zodResolver } from '@hookform/resolvers/zod';
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
import { useQuery } from '@tanstack/react-query';
import { Edit2, Mic, Monitor, Music, Upload, X } from 'lucide-react';
import { useEffect, useRef, useState } from 'react';
import { useForm } from 'react-hook-form';
import * as z from 'zod';
import { EffectsChainEditor } from '@/components/Effects/EffectsChainEditor';
import { Badge } from '@/components/ui/badge';
import { Button } from '@/components/ui/button';
import {
Dialog,
@@ -15,6 +17,7 @@ import {
import {
Form,
FormControl,
FormDescription,
FormField,
FormItem,
FormLabel,
@@ -32,7 +35,7 @@ import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
import { Textarea } from '@/components/ui/textarea';
import { useToast } from '@/components/ui/use-toast';
import { apiClient } from '@/lib/api/client';
import type { EffectConfig } from '@/lib/api/types';
import type { EffectConfig, PresetVoice, VoiceType } from '@/lib/api/types';
import { LANGUAGE_CODES, LANGUAGE_OPTIONS, type LanguageCode } from '@/lib/constants/languages';
import { useAudioPlayer } from '@/lib/hooks/useAudioPlayer';
import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
@@ -40,6 +43,7 @@ import {
useAddSample,
useCreateProfile,
useDeleteAvatar,
useDeleteProfile,
useProfile,
useUpdateProfile,
useUploadAvatar,
@@ -56,6 +60,16 @@ import { AudioSampleUpload } from './AudioSampleUpload';
import { SampleList } from './SampleList';
const MAX_AUDIO_DURATION_SECONDS = 30;
const PRESET_ONLY_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
const DEFAULT_ENGINE_OPTIONS = [
{ value: 'qwen', label: 'Qwen3-TTS' },
{ value: 'qwen_custom_voice', label: 'Qwen CustomVoice' },
{ value: 'luxtts', label: 'LuxTTS' },
{ value: 'chatterbox', label: 'Chatterbox' },
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
{ value: 'tada', label: 'TADA' },
{ value: 'kokoro', label: 'Kokoro 82M' },
] as const;
const baseProfileSchema = z.object({
name: z.string().min(1, 'Name is required').max(100),
@@ -116,20 +130,25 @@ export function ProfileForm() {
const createProfile = useCreateProfile();
const updateProfile = useUpdateProfile();
const addSample = useAddSample();
const deleteProfile = useDeleteProfile();
const uploadAvatar = useUploadAvatar();
const deleteAvatar = useDeleteAvatar();
const transcribe = useTranscription();
const { toast } = useToast();
const [voiceSource, setVoiceSource] = useState<'clone' | 'builtin'>('clone');
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
const [audioDuration, setAudioDuration] = useState<number | null>(null);
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
const [selectedPresetEngine, setSelectedPresetEngine] = useState<string>('kokoro');
const [selectedPresetVoiceId, setSelectedPresetVoiceId] = useState<string>('');
const avatarInputRef = useRef<HTMLInputElement>(null);
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
const isCreating = !editingProfileId;
const serverUrl = useServerStore((state) => state.serverUrl);
const [profileEffectsChain, setProfileEffectsChain] = useState<EffectConfig[]>([]);
const [effectsDirty, setEffectsDirty] = useState(false);
const [defaultEngine, setDefaultEngine] = useState<string>('');
const form = useForm<ProfileFormValues>({
resolver: zodResolver(profileSchema),
@@ -239,6 +258,26 @@ export function ProfileForm() {
},
});
// Fetch available preset voices for the selected engine
const presetEngineToQuery = isCreating
? selectedPresetEngine
: (editingProfile?.preset_engine ?? '');
const { data: presetVoicesData } = useQuery({
queryKey: ['presetVoices', presetEngineToQuery],
queryFn: () => apiClient.listPresetVoices(presetEngineToQuery),
enabled:
!!presetEngineToQuery &&
((voiceSource === 'builtin' && isCreating) ||
(!isCreating && editingProfile?.voice_type === 'preset')),
});
const presetVoices = presetVoicesData?.voices ?? [];
const isSampleBasedProfile = isCreating
? voiceSource === 'clone'
: editingProfile?.voice_type !== 'preset';
const availableDefaultEngines = DEFAULT_ENGINE_OPTIONS.filter(
(option) => !isSampleBasedProfile || !PRESET_ONLY_ENGINES.has(option.value),
);
// Show recording errors
useEffect(() => {
if (recordingError) {
@@ -287,6 +326,7 @@ export function ProfileForm() {
});
setProfileEffectsChain(editingProfile.effects_chain ?? []);
setEffectsDirty(false);
setDefaultEngine(editingProfile.default_engine ?? '');
} else if (profileFormDraft && open) {
// Restore from draft when opening in create mode
form.reset({
@@ -326,6 +366,24 @@ export function ProfileForm() {
}
}, [editingProfile, profileFormDraft, open, form]);
useEffect(() => {
if (
defaultEngine &&
!availableDefaultEngines.some((option) => option.value === defaultEngine)
) {
setDefaultEngine('');
}
}, [availableDefaultEngines, defaultEngine]);
useEffect(() => {
if (!selectedPresetVoiceId) {
return;
}
if (!presetVoices.some((voice: PresetVoice) => voice.voice_id === selectedPresetVoiceId)) {
setSelectedPresetVoiceId('');
}
}, [presetVoices, selectedPresetVoiceId]);
async function handleTranscribe() {
const file = form.getValues('sampleFile');
if (!file) {
@@ -415,13 +473,14 @@ export function ProfileForm() {
async function onSubmit(data: ProfileFormValues) {
try {
if (editingProfileId) {
// Editing: just update profile
// Editing: update profile
await updateProfile.mutateAsync({
profileId: editingProfileId,
data: {
name: data.name,
description: data.description,
language: data.language,
default_engine: defaultEngine || undefined,
},
});
@@ -464,8 +523,50 @@ export function ProfileForm() {
title: 'Voice updated',
description: `"${data.name}" has been updated successfully.`,
});
} else if (voiceSource === 'builtin') {
// Creating preset profile from built-in voice
if (!selectedPresetVoiceId) {
toast({
title: 'No voice selected',
description: 'Please select a built-in voice.',
variant: 'destructive',
});
return;
}
const profile = await createProfile.mutateAsync({
name: data.name,
description: data.description,
language: data.language,
voice_type: 'preset' as VoiceType,
preset_engine: selectedPresetEngine,
preset_voice_id: selectedPresetVoiceId,
default_engine: selectedPresetEngine,
});
// Handle avatar upload if provided
if (data.avatarFile) {
try {
await uploadAvatar.mutateAsync({
profileId: profile.id,
file: data.avatarFile,
});
} catch (avatarError) {
toast({
title: 'Avatar upload failed',
description:
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
variant: 'destructive',
});
}
}
toast({
title: 'Profile created',
description: `"${data.name}" has been created with a built-in voice.`,
});
} else {
// Creating: require sample file and reference text
// Creating cloned profile: require sample file and reference text
const sampleFile = form.getValues('sampleFile');
const referenceText = form.getValues('referenceText');
@@ -528,6 +629,7 @@ export function ProfileForm() {
name: data.name,
description: data.description,
language: data.language,
default_engine: defaultEngine || undefined,
});
// Convert non-WAV uploads to WAV so the backend can always use soundfile.
@@ -572,12 +674,32 @@ export function ProfileForm() {
description: `"${data.name}" has been created with a sample.`,
});
} catch (sampleError) {
// Profile was created but sample failed - still show error
let rollbackSucceeded = false;
try {
await deleteProfile.mutateAsync(profile.id);
rollbackSucceeded = true;
} catch (rollbackError) {
toast({
title: 'Rollback failed',
description:
rollbackError instanceof Error
? rollbackError.message
: 'Created profile could not be removed after sample upload failure.',
variant: 'destructive',
});
}
toast({
title: 'Failed to add sample',
description: `Profile "${data.name}" was created, but failed to add sample: ${sampleError instanceof Error ? sampleError.message : 'Unknown error'}`,
description:
sampleError instanceof Error
? `${sampleError.message}${rollbackSucceeded ? ' The profile was rolled back.' : ''}`
: rollbackSucceeded
? 'Failed to add sample. The profile was rolled back.'
: 'Failed to add sample.',
variant: 'destructive',
});
return;
}
}
@@ -642,16 +764,16 @@ export function ProfileForm() {
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-hidden">
<div className="max-w-5xl h-[85vh] mx-auto my-auto w-full flex flex-col overflow-hidden">
<DialogHeader>
<DialogTitle className="text-2xl">
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
{editingProfileId ? 'Edit Voice' : 'Create Voice'}
</DialogTitle>
<DialogDescription>
{editingProfileId
? 'Update your voice profile details and manage samples.'
: 'Create a new voice profile with an audio sample to clone the voice.'}
: 'Create a new voice profile from an audio sample or a built-in voice.'}
</DialogDescription>
{isCreating && profileFormDraft && (
<div className="flex items-center gap-2 pt-2">
@@ -682,143 +804,276 @@ export function ProfileForm() {
<Form {...form}>
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
<div className="grid gap-6 grid-cols-2 flex-1 min-h-0 overflow-hidden">
{/* Left column: Sample management */}
<div className="space-y-4 border-r pr-6">
<div className="space-y-4 border-r pr-6 overflow-y-auto min-h-0">
{isCreating ? (
<>
<Tabs
className="pt-4"
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsList
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
</TabsTrigger>
)}
</TabsList>
{/* Voice source selector */}
<div className="flex pt-4 pb-2">
<div className="inline-flex rounded-lg border border-border p-0.5 bg-muted/50">
<button
type="button"
onClick={() => setVoiceSource('clone')}
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
voiceSource === 'clone'
? 'bg-accent text-accent-foreground shadow-sm'
: 'text-muted-foreground hover:text-foreground'
}`}
>
<Mic className="h-3.5 w-3.5" />
Clone from audio
</button>
<button
type="button"
onClick={() => setVoiceSource('builtin')}
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
voiceSource === 'builtin'
? 'bg-accent text-accent-foreground shadow-sm'
: 'text-muted-foreground hover:text-foreground'
}`}
>
<Music className="h-3.5 w-3.5" />
Built-in voice
</button>
</div>
</div>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null &&
audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
)}
/>
</TabsContent>
{voiceSource === 'builtin' ? (
<div className="space-y-4">
<FormDescription>
Choose a pre-built voice. These don't require an audio sample.
</FormDescription>
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleRecording
file={selectedFile}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
)}
</Tabs>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
{/* Engine selector */}
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
/>
</FormControl>
<FormMessage />
<FormLabel>Engine</FormLabel>
<Select
value={selectedPresetEngine}
onValueChange={setSelectedPresetEngine}
>
<FormControl>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value="kokoro">Kokoro 82M</SelectItem>
<SelectItem value="qwen_custom_voice">Qwen CustomVoice</SelectItem>
</SelectContent>
</Select>
</FormItem>
)}
/>
{/* Voice picker */}
<FormItem>
<FormLabel>Voice</FormLabel>
<div className="grid grid-cols-2 gap-1.5 max-h-[340px] overflow-y-auto pr-1">
{presetVoices.map((voice: PresetVoice) => (
<button
key={voice.voice_id}
type="button"
onClick={() => {
setSelectedPresetVoiceId(voice.voice_id);
// Auto-set language from voice
if (voice.language) {
form.setValue('language', voice.language as LanguageCode);
}
}}
className={`text-left px-3 py-2 rounded-md border text-sm transition-colors ${
selectedPresetVoiceId === voice.voice_id
? 'border-accent bg-accent/10 text-accent-foreground'
: 'border-border hover:bg-muted'
}`}
>
<div className="font-medium">{voice.name}</div>
<div className="flex gap-1.5 mt-0.5">
<Badge variant="outline" className="text-[10px] h-4 px-1">
{voice.gender}
</Badge>
<Badge variant="outline" className="text-[10px] h-4 px-1">
{voice.language}
</Badge>
</div>
</button>
))}
</div>
</FormItem>
</div>
) : (
<>
<Tabs
className="pt-0"
value={sampleMode}
onValueChange={(v) => {
const newMode = v as 'upload' | 'record' | 'system';
// Cancel any active recordings when switching modes
if (isRecording && newMode !== 'record') {
cancelRecording();
}
if (isSystemRecording && newMode !== 'system') {
cancelSystemRecording();
}
setSampleMode(newMode);
}}
>
<TabsList
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
>
<TabsTrigger value="upload" className="flex items-center gap-2">
<Upload className="h-4 w-4 shrink-0" />
Upload
</TabsTrigger>
<TabsTrigger value="record" className="flex items-center gap-2">
<Mic className="h-4 w-4 shrink-0" />
Record
</TabsTrigger>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsTrigger value="system" className="flex items-center gap-2">
<Monitor className="h-4 w-4 shrink-0" />
System Audio
</TabsTrigger>
)}
</TabsList>
<TabsContent value="upload" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={({ field: { onChange, name } }) => (
<AudioSampleUpload
file={selectedFile}
onFileChange={onChange}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isValidating={isValidatingAudio}
isTranscribing={transcribe.isPending}
isDisabled={
audioDuration !== null &&
audioDuration > MAX_AUDIO_DURATION_SECONDS
}
fieldName={name}
/>
)}
/>
</TabsContent>
<TabsContent value="record" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleRecording
file={selectedFile}
isRecording={isRecording}
duration={duration}
onStart={startRecording}
onStop={stopRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
{platform.metadata.isTauri && isSystemAudioSupported && (
<TabsContent value="system" className="space-y-4">
<FormField
control={form.control}
name="sampleFile"
render={() => (
<AudioSampleSystem
file={selectedFile}
isRecording={isSystemRecording}
duration={systemDuration}
onStart={startSystemRecording}
onStop={stopSystemRecording}
onCancel={handleCancelRecording}
onTranscribe={handleTranscribe}
onPlayPause={handlePlayPause}
isPlaying={isPlaying}
isTranscribing={transcribe.isPending}
/>
)}
/>
</TabsContent>
)}
</Tabs>
<FormField
control={form.control}
name="referenceText"
render={({ field }) => (
<FormItem>
<FormLabel>Reference Text</FormLabel>
<FormControl>
<Textarea
placeholder="Enter the exact text spoken in the audio..."
className="min-h-[100px]"
{...field}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</>
)}
</>
) : (
// Show sample list when editing
editingProfileId && (
// Editing mode
editingProfileId &&
editingProfile &&
(editingProfile.voice_type === 'preset' ? (
<div className="space-y-4 pt-4">
<div className="rounded-lg border border-border p-4 space-y-3">
<div className="text-sm font-medium text-muted-foreground">
Built-in Voice
</div>
<div className="flex items-center gap-3">
<div className="text-lg font-semibold">
{presetVoices.find(
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
)?.name ?? editingProfile.preset_voice_id}
</div>
<Badge variant="secondary" className="text-xs">
{editingProfile.preset_engine}
</Badge>
</div>
{(() => {
const voice = presetVoices.find(
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
);
return voice ? (
<div className="flex gap-1.5">
<Badge variant="outline" className="text-xs">
{voice.gender}
</Badge>
<Badge variant="outline" className="text-xs">
{voice.language}
</Badge>
</div>
) : null;
})()}
</div>
<p className="text-xs text-muted-foreground">
This profile uses a built-in voice. The voice cannot be changed after
creation.
</p>
</div>
) : (
<div>
<SampleList profileId={editingProfileId} />
</div>
)
))
)}
</div>
{/* Right column: Profile info */}
<div className="space-y-4">
<div className="space-y-4 overflow-y-auto min-h-0">
{/* Avatar Upload */}
<FormField
control={form.control}
@@ -924,6 +1179,36 @@ export function ProfileForm() {
)}
/>
<FormItem>
<FormLabel>Default Engine</FormLabel>
<Select
value={defaultEngine || '_none'}
onValueChange={(v) => {
setDefaultEngine(v === '_none' ? '' : v);
}}
disabled={
voiceSource === 'builtin' || editingProfile?.voice_type === 'preset'
}
>
<FormControl>
<SelectTrigger>
<SelectValue placeholder="No preference" />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value="_none">No preference</SelectItem>
{availableDefaultEngines.map((option) => (
<SelectItem key={option.value} value={option.value}>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Auto-selects this engine when the profile is chosen.
</p>
</FormItem>
{editingProfileId && (
<div className="space-y-2">
<FormLabel>Default Effects</FormLabel>
@@ -1,4 +1,5 @@
import { Mic, Sparkles } from 'lucide-react';
import { Info, Mic, Sparkles } from 'lucide-react';
import { useEffect, useRef } from 'react';
import { Button } from '@/components/ui/button';
import { Card, CardContent } from '@/components/ui/card';
import { useProfiles } from '@/lib/hooks/useProfiles';
@@ -6,9 +7,36 @@ import { useUIStore } from '@/stores/uiStore';
import { ProfileCard } from './ProfileCard';
import { ProfileForm } from './ProfileForm';
/** Engines that use preset (built-in) voices instead of cloned profiles. */
const PRESET_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
export function ProfileList() {
const { data: profiles, isLoading, error } = useProfiles();
const setDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
const selectedEngine = useUIStore((state) => state.selectedEngine);
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
const cardRefs = useRef<Map<string, HTMLDivElement>>(new Map());
// Scroll to the selected profile after engine/sort changes
useEffect(() => {
if (!selectedProfileId) return;
let timeoutId: ReturnType<typeof setTimeout> | null = null;
const rafId = requestAnimationFrame(() => {
const el = cardRefs.current.get(selectedProfileId);
if (!el) return;
// Temporarily apply scroll-margin so it doesn't land flush at the top
el.style.scrollMarginTop = '180px';
el.scrollIntoView({ behavior: 'smooth', block: 'nearest', inline: 'nearest' });
timeoutId = setTimeout(() => {
el.style.scrollMarginTop = '';
}, 500);
});
return () => {
cancelAnimationFrame(rafId);
if (timeoutId) clearTimeout(timeoutId);
};
}, [selectedProfileId, selectedEngine]);
if (isLoading) {
return null;
@@ -23,6 +51,20 @@ export function ProfileList() {
}
const allProfiles = profiles || [];
const isPresetEngine = PRESET_ENGINES.has(selectedEngine);
/** Whether a profile is supported by the currently selected engine. */
const isSupported = (p: (typeof allProfiles)[number]) =>
isPresetEngine
? p.voice_type === 'preset' && p.preset_engine === selectedEngine
: p.voice_type !== 'preset';
// Sort so supported profiles come first
const sortedProfiles = [...allProfiles].sort(
(a, b) => (isSupported(a) ? 0 : 1) - (isSupported(b) ? 0 : 1),
);
const hasUnsupported = sortedProfiles.some((p) => !isSupported(p));
return (
<div className="flex flex-col">
@@ -42,11 +84,24 @@ export function ProfileList() {
</Card>
) : (
<div className="flex gap-4 overflow-x-auto p-1 pb-1 lg:grid lg:grid-cols-3 lg:auto-rows-auto lg:overflow-x-visible lg:pb-[150px]">
{allProfiles.map((profile) => (
<div key={profile.id} className="shrink-0 w-[200px] lg:w-auto lg:shrink">
<ProfileCard profile={profile} />
{sortedProfiles.map((profile) => (
<div
key={profile.id}
className="shrink-0 w-[200px] lg:w-auto lg:shrink"
ref={(el) => {
if (el) cardRefs.current.set(profile.id, el);
else cardRefs.current.delete(profile.id);
}}
>
<ProfileCard profile={profile} disabled={!isSupported(profile)} />
</div>
))}
{hasUnsupported && (
<div className="col-span-full flex items-center gap-2 text-xs text-muted-foreground py-2">
<Info className="h-3.5 w-3.5 shrink-0" />
<span>Only supported voice profiles can be selected for the current model.</span>
</div>
)}
</div>
)}
</div>
+1 -1
View File
@@ -111,4 +111,4 @@ export {
AlertDialogDescription,
AlertDialogAction,
AlertDialogCancel,
};
};
+11
View File
@@ -17,6 +17,7 @@ import type {
HistoryResponse,
ModelDownloadRequest,
ModelStatusListResponse,
PresetVoice,
ProfileSampleResponse,
StoryCreate,
StoryDetailResponse,
@@ -97,6 +98,10 @@ class ApiClient {
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`);
}
async listPresetVoices(engine: string): Promise<{ engine: string; voices: PresetVoice[] }> {
return this.request<{ engine: string; voices: PresetVoice[] }>(`/profiles/presets/${engine}`);
}
async updateProfile(profileId: string, data: VoiceProfileCreate): Promise<VoiceProfileResponse> {
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`, {
method: 'PUT',
@@ -265,6 +270,12 @@ class ApiClient {
});
}
async clearFailedGenerations(): Promise<{ deleted: number }> {
return this.request<{ deleted: number }>(`/history/failed`, {
method: 'DELETE',
});
}
async exportGeneration(generationId: string): Promise<Blob> {
const url = `${this.getBaseUrl()}/history/${generationId}/export`;
const response = await fetch(url);
+28 -2
View File
@@ -1,10 +1,17 @@
// API Types matching backend Pydantic models
import type { LanguageCode } from '@/lib/constants/languages';
export type VoiceType = 'cloned' | 'preset' | 'designed';
export interface VoiceProfileCreate {
name: string;
description?: string;
language: LanguageCode;
voice_type?: VoiceType;
preset_engine?: string;
preset_voice_id?: string;
design_prompt?: string;
default_engine?: string;
}
export interface VoiceProfileResponse {
@@ -14,12 +21,24 @@ export interface VoiceProfileResponse {
language: string;
avatar_path?: string;
effects_chain?: EffectConfig[];
voice_type: VoiceType;
preset_engine?: string;
preset_voice_id?: string;
design_prompt?: string;
default_engine?: string;
generation_count: number;
sample_count: number;
created_at: string;
updated_at: string;
}
export interface PresetVoice {
voice_id: string;
name: string;
gender: 'male' | 'female';
language: string;
}
export interface ProfileSampleCreate {
reference_text: string;
}
@@ -42,8 +61,15 @@ export interface GenerationRequest {
text: string;
language: LanguageCode;
seed?: number;
model_size?: '1.7B' | '0.6B';
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo';
model_size?: '1.7B' | '0.6B' | '1B' | '3B';
engine?:
| 'qwen'
| 'qwen_custom_voice'
| 'luxtts'
| 'chatterbox'
| 'chatterbox_turbo'
| 'tada'
| 'kokoro';
instruct?: string;
max_chunk_chars?: number;
crossfade_ms?: number;
+4
View File
@@ -5,6 +5,7 @@
* LuxTTS is English-only.
* Chatterbox Multilingual supports 23 languages.
* Chatterbox Turbo is English-only.
* Kokoro supports 8 languages.
*/
/** All languages that any engine supports. */
@@ -66,6 +67,9 @@ export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
'zh',
],
chatterbox_turbo: ['en'],
tada: ['en', 'ar', 'zh', 'de', 'es', 'fr', 'it', 'ja', 'pl', 'pt'],
kokoro: ['en', 'es', 'fr', 'hi', 'it', 'pt', 'ja', 'zh'],
qwen_custom_voice: ['zh', 'en', 'ja', 'ko', 'de', 'fr', 'ru', 'pt', 'es', 'it'],
} as const;
/** Helper: get language options for a given engine. */
+44 -10
View File
@@ -10,14 +10,25 @@ import { useGeneration } from '@/lib/hooks/useGeneration';
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
import { useGenerationStore } from '@/stores/generationStore';
import { useServerStore } from '@/stores/serverStore';
import { useUIStore } from '@/stores/uiStore';
const generationSchema = z.object({
text: z.string().min(1, '').max(50000),
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
seed: z.number().int().optional(),
modelSize: z.enum(['1.7B', '0.6B']).optional(),
modelSize: z.enum(['1.7B', '0.6B', '1B', '3B']).optional(),
instruct: z.string().max(500).optional(),
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo']).optional(),
engine: z
.enum([
'qwen',
'qwen_custom_voice',
'luxtts',
'chatterbox',
'chatterbox_turbo',
'tada',
'kokoro',
])
.optional(),
});
export type GenerationFormValues = z.infer<typeof generationSchema>;
@@ -35,6 +46,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
const maxChunkChars = useServerStore((state) => state.maxChunkChars);
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
const selectedEngine = useUIStore((state) => state.selectedEngine);
const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null);
const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null);
@@ -52,7 +64,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
seed: undefined,
modelSize: '1.7B',
instruct: '',
engine: 'qwen',
engine: (selectedEngine as GenerationFormValues['engine']) || 'qwen',
...options.defaultValues,
},
});
@@ -79,7 +91,15 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
? 'chatterbox-tts'
: engine === 'chatterbox_turbo'
? 'chatterbox-turbo'
: `qwen-tts-${data.modelSize}`;
: engine === 'tada'
? data.modelSize === '3B'
? 'tada-3b-ml'
: 'tada-1b'
: engine === 'kokoro'
? 'kokoro'
: engine === 'qwen_custom_voice'
? `qwen-custom-voice-${data.modelSize}`
: `qwen-tts-${data.modelSize}`;
const displayName =
engine === 'luxtts'
? 'LuxTTS'
@@ -87,9 +107,19 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
? 'Chatterbox TTS'
: engine === 'chatterbox_turbo'
? 'Chatterbox Turbo'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
: engine === 'tada'
? data.modelSize === '3B'
? 'TADA 3B Multilingual'
: 'TADA 1B'
: engine === 'kokoro'
? 'Kokoro 82M'
: engine === 'qwen_custom_voice'
? data.modelSize === '1.7B'
? 'Qwen CustomVoice 1.7B'
: 'Qwen CustomVoice 0.6B'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
// Check if model needs downloading
try {
@@ -104,7 +134,11 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
console.error('Failed to check model status:', error);
}
const isQwen = engine === 'qwen';
const hasModelSizes =
engine === 'qwen' || engine === 'qwen_custom_voice' || engine === 'tada';
// Only Qwen CustomVoice actually honors the instruct kwarg at model level.
// Base Qwen3-TTS accepts the kwarg but ignores it.
const supportsInstruct = engine === 'qwen_custom_voice';
const effectsChain = options.getEffectsChain?.();
// This now returns immediately with status="generating"
const result = await generation.mutateAsync({
@@ -112,9 +146,9 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
text: data.text,
language: data.language,
seed: data.seed,
model_size: isQwen ? data.modelSize : undefined,
model_size: hasModelSizes ? data.modelSize : undefined,
engine,
instruct: isQwen ? data.instruct || undefined : undefined,
instruct: supportsInstruct ? data.instruct || undefined : undefined,
max_chunk_chars: maxChunkChars,
crossfade_ms: crossfadeMs,
normalize: normalizeAudio,
+11
View File
@@ -29,6 +29,17 @@ export function useDeleteGeneration() {
});
}
export function useClearFailedGenerations() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: () => apiClient.clearFailedGenerations(),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ['history'] });
},
});
}
export function useExportGeneration() {
const platform = usePlatform();
+2 -1
View File
@@ -131,7 +131,8 @@ export function useModelDownloadToast({
)}
</div>
),
duration: progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
duration:
progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
});
// Close connection and dismiss toast on completion or error
+18 -2
View File
@@ -26,8 +26,24 @@ export function useSystemAudioCapture({
// Check if system audio capture is supported
useEffect(() => {
const supported = platform.audio.isSystemAudioSupported();
setIsSupported(supported);
let isActive = true;
void platform.audio
.isSystemAudioSupported()
.then((supported) => {
if (isActive) {
setIsSupported(supported);
}
})
.catch(() => {
if (isActive) {
setIsSupported(false);
}
});
return () => {
isActive = false;
};
}, [platform]);
const startRecording = useCallback(async () => {
+19
View File
@@ -0,0 +1,19 @@
import { QueryClient } from '@tanstack/react-query';
/**
* Shared QueryClient instance used across the app.
*
* Extracted into its own side-effect-free module so it can be imported from
* both the React bootstrap (main.tsx) and non-React code (stores, utilities)
* without pulling in ReactDOM or other bootstrap side effects.
*/
export const queryClient = new QueryClient({
defaultOptions: {
queries: {
staleTime: 1000 * 60 * 5, // 5 minutes
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
retry: 1,
refetchOnWindowFocus: false,
},
},
});
+2 -12
View File
@@ -1,20 +1,10 @@
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { QueryClientProvider } from '@tanstack/react-query';
// import { ReactQueryDevtools } from '@tanstack/react-query-devtools';
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import './index.css';
const queryClient = new QueryClient({
defaultOptions: {
queries: {
staleTime: 1000 * 60 * 5, // 5 minutes
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
retry: 1,
refetchOnWindowFocus: false,
},
},
});
import { queryClient } from './lib/queryClient';
ReactDOM.createRoot(document.getElementById('root')!).render(
<React.StrictMode>
+1 -5
View File
@@ -9,11 +9,7 @@ export interface PlatformProviderProps {
}
export function PlatformProvider({ platform, children }: PlatformProviderProps) {
return (
<PlatformContext.Provider value={platform}>
{children}
</PlatformContext.Provider>
);
return <PlatformContext.Provider value={platform}>{children}</PlatformContext.Provider>;
}
export function usePlatform(): Platform {
+1 -1
View File
@@ -42,7 +42,7 @@ export interface AudioDevice {
}
export interface PlatformAudio {
isSystemAudioSupported(): boolean;
isSystemAudioSupported(): Promise<boolean>;
startSystemAudioCapture(maxDurationSecs: number): Promise<void>;
stopSystemAudioCapture(): Promise<Blob>;
listOutputDevices(): Promise<AudioDevice[]>;
+17 -2
View File
@@ -1,5 +1,6 @@
import { create } from 'zustand';
import { persist } from 'zustand/middleware';
import { queryClient } from '@/lib/queryClient';
interface ServerStore {
serverUrl: string;
@@ -30,11 +31,25 @@ interface ServerStore {
setCustomModelsDir: (dir: string | null) => void;
}
/**
* Invalidate all React Query caches so stale data from the previous
* server is not shown. Called when the server URL changes.
*/
function invalidateAllServerData() {
queryClient.invalidateQueries();
}
export const useServerStore = create<ServerStore>()(
persist(
(set) => ({
(set, get) => ({
serverUrl: 'http://127.0.0.1:17493',
setServerUrl: (url) => set({ serverUrl: url }),
setServerUrl: (url) => {
const prev = get().serverUrl;
set({ serverUrl: url });
if (url !== prev) {
invalidateAllServerData();
}
},
isConnected: false,
setIsConnected: (connected) => set({ isConnected: connected }),
+7
View File
@@ -31,6 +31,10 @@ interface UIStore {
selectedProfileId: string | null;
setSelectedProfileId: (id: string | null) => void;
// Currently selected engine (synced from generation form)
selectedEngine: string;
setSelectedEngine: (engine: string) => void;
// Selected voice in Voices tab inspector
selectedVoiceId: string | null;
setSelectedVoiceId: (id: string | null) => void;
@@ -59,6 +63,9 @@ export const useUIStore = create<UIStore>((set) => ({
selectedProfileId: null,
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
selectedEngine: 'qwen',
setSelectedEngine: (engine) => set({ selectedEngine: engine }),
selectedVoiceId: null,
setSelectedVoiceId: (id) => set({ selectedVoiceId: id }),
+1 -1
View File
@@ -1,3 +1,3 @@
# Backend package
__version__ = "0.2.3"
__version__ = "0.4.0"
+31 -3
View File
@@ -135,7 +135,7 @@ def _mount_frontend(application: FastAPI) -> None:
async def serve_spa(full_path: str):
file_path = (frontend_dir / full_path).resolve()
# Guard against path traversal — only serve files inside frontend_dir
if full_path and file_path.is_file() and str(file_path).startswith(str(frontend_dir)):
if full_path and file_path.is_file() and file_path.is_relative_to(frontend_dir):
return FileResponse(file_path)
return FileResponse(frontend_dir / "index.html", media_type="text/html")
@@ -146,15 +146,36 @@ def _get_gpu_status() -> str:
"""Return a human-readable string describing GPU availability."""
backend_type = get_backend_type()
if torch.cuda.is_available():
from .backends.base import check_cuda_compatibility
device_name = torch.cuda.get_device_name(0)
compatible, _warning = check_cuda_compatibility()
is_rocm = hasattr(torch.version, "hip") and torch.version.hip is not None
if is_rocm:
return f"ROCm ({device_name})"
return f"CUDA ({device_name})"
label = f"ROCm ({device_name})"
else:
label = f"CUDA ({device_name})"
if not compatible:
label += " [UNSUPPORTED - see logs]"
return label
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
return "MPS (Apple Silicon)"
elif backend_type == "mlx":
return "Metal (Apple Silicon via MLX)"
# Intel XPU (Arc / Data Center) via IPEX
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
try:
xpu_name = torch.xpu.get_device_name(0)
except Exception:
xpu_name = "Intel GPU"
return f"XPU ({xpu_name})"
except ImportError:
pass
return "None (CPU only)"
@@ -216,6 +237,13 @@ def _register_lifecycle(application: FastAPI) -> None:
logger.info("Backend: %s", backend_type.upper())
logger.info("GPU: %s", _get_gpu_status())
# Warn if GPU architecture is not supported by this PyTorch build
from .backends.base import check_cuda_compatibility
_compatible, _cuda_warning = check_cuda_compatibility()
if not _compatible:
logger.warning("GPU COMPATIBILITY: %s", _cuda_warning)
from .services.cuda import check_and_update_cuda_binary
create_background_task(check_and_update_cuda_binary())
+90 -5
View File
@@ -163,9 +163,12 @@ _stt_backend: Optional[STTBackend] = None
# The factory function uses this for the if/elif chain; the model configs live on the backend classes.
TTS_ENGINES = {
"qwen": "Qwen TTS",
"qwen_custom_voice": "Qwen CustomVoice",
"luxtts": "LuxTTS",
"chatterbox": "Chatterbox TTS",
"chatterbox_turbo": "Chatterbox Turbo",
"tada": "TADA",
"kokoro": "Kokoro",
}
@@ -203,6 +206,32 @@ def _get_qwen_model_configs() -> list[ModelConfig]:
]
def _get_qwen_custom_voice_configs() -> list[ModelConfig]:
"""Return Qwen CustomVoice model configs."""
return [
ModelConfig(
model_name="qwen-custom-voice-1.7B",
display_name="Qwen CustomVoice 1.7B",
engine="qwen_custom_voice",
hf_repo_id="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
model_size="1.7B",
size_mb=3500,
supports_instruct=True,
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
),
ModelConfig(
model_name="qwen-custom-voice-0.6B",
display_name="Qwen CustomVoice 0.6B",
engine="qwen_custom_voice",
hf_repo_id="Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
model_size="0.6B",
size_mb=1200,
supports_instruct=True,
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
),
]
def _get_non_qwen_tts_configs() -> list[ModelConfig]:
"""Return model configs for non-Qwen TTS engines.
@@ -259,6 +288,32 @@ def _get_non_qwen_tts_configs() -> list[ModelConfig]:
needs_trim=True,
languages=["en"],
),
ModelConfig(
model_name="tada-1b",
display_name="TADA 1B (English)",
engine="tada",
hf_repo_id="HumeAI/tada-1b",
model_size="1B",
size_mb=4000,
languages=["en"],
),
ModelConfig(
model_name="tada-3b-ml",
display_name="TADA 3B Multilingual",
engine="tada",
hf_repo_id="HumeAI/tada-3b-ml",
model_size="3B",
size_mb=8000,
languages=["en", "ar", "zh", "de", "es", "fr", "it", "ja", "pl", "pt"],
),
ModelConfig(
model_name="kokoro",
display_name="Kokoro 82M",
engine="kokoro",
hf_repo_id="hexgrad/Kokoro-82M",
size_mb=350,
languages=["en", "es", "fr", "hi", "it", "pt", "ja", "zh"],
),
]
@@ -305,12 +360,12 @@ def _get_whisper_configs() -> list[ModelConfig]:
def get_all_model_configs() -> list[ModelConfig]:
"""Return the full list of model configs (TTS + STT)."""
return _get_qwen_model_configs() + _get_non_qwen_tts_configs() + _get_whisper_configs()
return _get_qwen_model_configs() + _get_qwen_custom_voice_configs() + _get_non_qwen_tts_configs() + _get_whisper_configs()
def get_tts_model_configs() -> list[ModelConfig]:
"""Return only TTS model configs."""
return _get_qwen_model_configs() + _get_non_qwen_tts_configs()
return _get_qwen_model_configs() + _get_qwen_custom_voice_configs() + _get_non_qwen_tts_configs()
# Lookup helpers — these replace the if/elif chains in main.py
@@ -339,10 +394,12 @@ def engine_has_model_sizes(engine: str) -> bool:
async def load_engine_model(engine: str, model_size: str = "default") -> None:
"""Load a model for the given engine, handling the Qwen model_size special case."""
"""Load a model for the given engine, handling engines with multiple model sizes."""
backend = get_tts_backend_for_engine(engine)
if engine == "qwen":
if engine in ("qwen", "qwen_custom_voice"):
await backend.load_model_async(model_size)
elif engine == "tada":
await backend.load_model(model_size)
else:
await backend.load_model()
@@ -358,7 +415,7 @@ async def ensure_model_cached_or_raise(engine: str, model_size: str = "default")
cfg = c
break
if engine == "qwen":
if engine in ("qwen", "qwen_custom_voice", "tada"):
if not backend._is_model_cached(model_size):
raise HTTPException(
status_code=400,
@@ -393,6 +450,14 @@ def unload_model_by_config(config: ModelConfig) -> bool:
return True
return False
if config.engine == "qwen_custom_voice":
backend = get_tts_backend_for_engine(config.engine)
loaded_size = getattr(backend, "_current_model_size", None) or getattr(backend, "model_size", None)
if backend.is_loaded() and loaded_size == config.model_size:
backend.unload_model()
return True
return False
# All other TTS engines
backend = get_tts_backend_for_engine(config.engine)
if backend.is_loaded():
@@ -416,6 +481,11 @@ def check_model_loaded(config: ModelConfig) -> bool:
loaded_size = getattr(tts_model, "_current_model_size", None) or getattr(tts_model, "model_size", None)
return tts_model.is_loaded() and loaded_size == config.model_size
if config.engine == "qwen_custom_voice":
backend = get_tts_backend_for_engine(config.engine)
loaded_size = getattr(backend, "_current_model_size", None) or getattr(backend, "model_size", None)
return backend.is_loaded() and loaded_size == config.model_size
backend = get_tts_backend_for_engine(config.engine)
return backend.is_loaded()
except Exception:
@@ -433,6 +503,9 @@ def get_model_load_func(config: ModelConfig):
if config.engine == "qwen":
return lambda: tts.get_tts_model().load_model(config.model_size)
if config.engine == "qwen_custom_voice":
return lambda: get_tts_backend_for_engine(config.engine).load_model(config.model_size)
return lambda: get_tts_backend_for_engine(config.engine).load_model()
@@ -490,6 +563,18 @@ def get_tts_backend_for_engine(engine: str) -> TTSBackend:
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
backend = ChatterboxTurboTTSBackend()
elif engine == "tada":
from .hume_backend import HumeTadaBackend
backend = HumeTadaBackend()
elif engine == "kokoro":
from .kokoro_backend import KokoroTTSBackend
backend = KokoroTTSBackend()
elif engine == "qwen_custom_voice":
from .qwen_custom_voice_backend import QwenCustomVoiceBackend
backend = QwenCustomVoiceBackend()
else:
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
+69
View File
@@ -126,6 +126,75 @@ def get_torch_device(
return "cpu"
def check_cuda_compatibility() -> tuple[bool, str | None]:
"""Check if the installed PyTorch supports the current GPU's compute capability.
Returns:
(compatible, warning_message) — compatible is True if OK or no CUDA GPU,
warning_message is a human-readable string if there's a problem.
"""
import torch
if not torch.cuda.is_available():
return True, None
major, minor = torch.cuda.get_device_capability(0)
capability = f"{major}.{minor}"
device_name = torch.cuda.get_device_name(0)
sm_tag = f"sm_{major}{minor}"
# torch.cuda._get_arch_list() returns the SM architectures this build
# was compiled for (e.g. ["sm_50", "sm_60", ..., "sm_90"]).
try:
arch_list = torch.cuda._get_arch_list()
if arch_list:
# Check for both sm_XX and compute_XX (JIT-compiled) entries
compute_tag = f"compute_{major}{minor}"
if sm_tag not in arch_list and compute_tag not in arch_list:
return False, (
f"{device_name} (compute capability {capability} / {sm_tag}) "
f"is not supported by this PyTorch build. "
f"Supported architectures: {', '.join(arch_list)}. "
f"Install PyTorch nightly (cu128) for newer GPU support: "
f"pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128"
)
except AttributeError:
pass
return True, None
def empty_device_cache(device: str) -> None:
"""
Free cached memory on the given device (CUDA or XPU).
Backends should call this after unloading models so VRAM is returned
to the OS.
"""
import torch
if device == "cuda" and torch.cuda.is_available():
torch.cuda.empty_cache()
elif device == "xpu" and hasattr(torch, "xpu"):
torch.xpu.empty_cache()
def manual_seed(seed: int, device: str) -> None:
"""
Set the random seed on both CPU and the active accelerator.
Covers CUDA and Intel XPU so that generation is reproducible
regardless of which GPU backend is in use.
"""
import torch
torch.manual_seed(seed)
if device == "cuda" and torch.cuda.is_available():
torch.cuda.manual_seed(seed)
elif device == "xpu" and hasattr(torch, "xpu"):
torch.xpu.manual_seed(seed)
async def combine_voice_prompts(
audio_paths: List[str],
reference_texts: List[str],
+6 -10
View File
@@ -18,6 +18,8 @@ from . import TTSBackend
from .base import (
is_model_cached,
get_torch_device,
empty_device_cache,
manual_seed,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
patch_chatterbox_f32,
@@ -48,7 +50,7 @@ class ChatterboxTTSBackend:
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
return get_torch_device(force_cpu_on_mac=True)
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
def is_loaded(self) -> bool:
return self.model is not None
@@ -117,10 +119,7 @@ class ChatterboxTTSBackend:
del self.model
self.model = None
self._device = None
if device == "cuda":
import torch
torch.cuda.empty_cache()
empty_device_cache(device)
logger.info("Chatterbox unloaded")
async def create_voice_prompt(
@@ -200,7 +199,7 @@ class ChatterboxTTSBackend:
import torch
if seed is not None:
torch.manual_seed(seed)
manual_seed(seed, self._device)
logger.info(f"[Chatterbox] Generating: lang={language}")
@@ -220,10 +219,7 @@ class ChatterboxTTSBackend:
else:
audio = np.asarray(wav, dtype=np.float32)
sample_rate = (
getattr(self.model, "sr", None)
or getattr(self.model, "sample_rate", 24000)
)
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
return audio, sample_rate
+6 -10
View File
@@ -18,6 +18,8 @@ from . import TTSBackend
from .base import (
is_model_cached,
get_torch_device,
empty_device_cache,
manual_seed,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
patch_chatterbox_f32,
@@ -48,7 +50,7 @@ class ChatterboxTurboTTSBackend:
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
return get_torch_device(force_cpu_on_mac=True)
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
def is_loaded(self) -> bool:
return self.model is not None
@@ -116,10 +118,7 @@ class ChatterboxTurboTTSBackend:
del self.model
self.model = None
self._device = None
if device == "cuda":
import torch
torch.cuda.empty_cache()
empty_device_cache(device)
logger.info("Chatterbox Turbo unloaded")
async def create_voice_prompt(
@@ -181,7 +180,7 @@ class ChatterboxTurboTTSBackend:
import torch
if seed is not None:
torch.manual_seed(seed)
manual_seed(seed, self._device)
logger.info("[Chatterbox Turbo] Generating (English)")
@@ -200,10 +199,7 @@ class ChatterboxTurboTTSBackend:
else:
audio = np.asarray(wav, dtype=np.float32)
sample_rate = (
getattr(self.model, "sr", None)
or getattr(self.model, "sample_rate", 24000)
)
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
return audio, sample_rate
+346
View File
@@ -0,0 +1,346 @@
"""
HumeAI TADA TTS backend implementation.
Wraps HumeAI's TADA (Text-Acoustic Dual Alignment) model for
high-quality voice cloning. Two model variants:
- tada-1b: English-only, ~2B params (Llama 3.2 1B base)
- tada-3b-ml: Multilingual, ~4B params (Llama 3.2 3B base)
Both use a shared encoder/codec (HumeAI/tada-codec). The encoder
produces 1:1 aligned token embeddings from reference audio, and the
causal LM generates speech via flow-matching diffusion.
24kHz output, bf16 inference on CUDA, fp32 on CPU.
"""
import asyncio
import logging
import threading
from typing import ClassVar, List, Optional, Tuple
import numpy as np
from . import TTSBackend
from .base import (
is_model_cached,
get_torch_device,
empty_device_cache,
manual_seed,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
logger = logging.getLogger(__name__)
# HuggingFace repos
TADA_CODEC_REPO = "HumeAI/tada-codec"
TADA_1B_REPO = "HumeAI/tada-1b"
TADA_3B_ML_REPO = "HumeAI/tada-3b-ml"
TADA_MODEL_REPOS = {
"1B": TADA_1B_REPO,
"3B": TADA_3B_ML_REPO,
}
# Key weight files for cache detection
_TADA_MODEL_WEIGHT_FILES = [
"model.safetensors",
]
_TADA_CODEC_WEIGHT_FILES = [
"encoder/model.safetensors",
]
class HumeTadaBackend:
"""HumeAI TADA TTS backend for high-quality voice cloning."""
_load_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self):
self.model = None
self.encoder = None
self.model_size = "1B" # default to 1B
self._device = None
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
# Force CPU on macOS — MPS has issues with flow matching
# and large vocab lm_head (>65536 output channels)
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
def is_loaded(self) -> bool:
return self.model is not None
def _get_model_path(self, model_size: str = "1B") -> str:
return TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
def _is_model_cached(self, model_size: str = "1B") -> bool:
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
model_cached = is_model_cached(repo, required_files=_TADA_MODEL_WEIGHT_FILES)
codec_cached = is_model_cached(TADA_CODEC_REPO, required_files=_TADA_CODEC_WEIGHT_FILES)
return model_cached and codec_cached
async def load_model(self, model_size: str = "1B") -> None:
"""Load the TADA model and encoder."""
if self.model is not None and self.model_size == model_size:
return
async with self._model_load_lock:
if self.model is not None and self.model_size == model_size:
return
# Unload existing model if switching sizes
if self.model is not None:
self.unload_model()
self.model_size = model_size
await asyncio.to_thread(self._load_model_sync, model_size)
def _load_model_sync(self, model_size: str = "1B"):
"""Synchronous model loading with progress tracking."""
model_name = f"tada-{model_size.lower()}"
is_cached = self._is_model_cached(model_size)
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
with model_load_progress(model_name, is_cached):
# Install DAC shim before importing tada — tada's encoder/decoder
# import dac.nn.layers.Snake1d which requires the descript-audio-codec
# package. The real package pulls in onnx/tensorboard/matplotlib via
# descript-audiotools, so we use a lightweight shim instead.
from ..utils.dac_shim import install_dac_shim
install_dac_shim()
import torch
from huggingface_hub import snapshot_download
device = self._get_device()
self._device = device
logger.info(f"Loading HumeAI TADA {model_size} on {device}...")
# Download codec (encoder + decoder) if not cached
logger.info("Downloading TADA codec...")
snapshot_download(
repo_id=TADA_CODEC_REPO,
token=None,
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin"],
)
# Download model weights if not cached
logger.info(f"Downloading TADA {model_size} model...")
snapshot_download(
repo_id=repo,
token=None,
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin", "*.model"],
)
# TADA hardcodes "meta-llama/Llama-3.2-1B" as the tokenizer
# source in its Aligner and TadaForCausalLM.from_pretrained().
# That repo is gated (requires Meta license acceptance).
# Download the tokenizer from an ungated mirror and get its
# local cache path so we can point TADA at it directly.
logger.info("Downloading Llama tokenizer (ungated mirror)...")
tokenizer_path = snapshot_download(
repo_id="unsloth/Llama-3.2-1B",
token=None,
allow_patterns=["tokenizer*", "special_tokens*"],
)
# Determine dtype — use bf16 on CUDA/XPU for ~50% memory savings
if device == "cuda" and torch.cuda.is_bf16_supported():
model_dtype = torch.bfloat16
elif device == "xpu":
# Intel Arc (Alchemist+) supports bf16 natively
model_dtype = torch.bfloat16
else:
model_dtype = torch.float32
# Patch the Aligner config class to use the local tokenizer
# path instead of the gated "meta-llama/Llama-3.2-1B" default.
# This avoids monkey-patching AutoTokenizer.from_pretrained
# which corrupts the classmethod descriptor for other engines.
from tada.modules.aligner import AlignerConfig
AlignerConfig.tokenizer_name = tokenizer_path
# Load encoder (only needed for voice prompt encoding)
from tada.modules.encoder import Encoder
logger.info("Loading TADA encoder...")
self.encoder = Encoder.from_pretrained(TADA_CODEC_REPO, subfolder="encoder").to(device)
self.encoder.eval()
# Load the causal LM (includes decoder for wav generation).
# TadaForCausalLM.from_pretrained() calls
# getattr(config, "tokenizer_name", "meta-llama/Llama-3.2-1B")
# which hits the gated repo. Pre-load the config from HF,
# inject the local tokenizer path, then pass it in.
from tada.modules.tada import TadaForCausalLM, TadaConfig
logger.info(f"Loading TADA {model_size} model...")
config = TadaConfig.from_pretrained(repo)
config.tokenizer_name = tokenizer_path
self.model = TadaForCausalLM.from_pretrained(repo, config=config, torch_dtype=model_dtype).to(device)
self.model.eval()
logger.info(f"HumeAI TADA {model_size} loaded successfully on {device}")
def unload_model(self) -> None:
"""Unload model and encoder to free memory."""
if self.model is not None:
del self.model
self.model = None
if self.encoder is not None:
del self.encoder
self.encoder = None
device = self._device
self._device = None
if device:
empty_device_cache(device)
logger.info("HumeAI TADA unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio using TADA's encoder.
TADA's encoder performs forced alignment between audio and text tokens,
producing an EncoderOutput with 1:1 token-audio alignment. If no
reference_text is provided, the encoder uses built-in ASR (English only).
We serialize the EncoderOutput to a dict for caching.
"""
await self.load_model(self.model_size)
cache_key = ("tada_" + get_cache_key(audio_path, reference_text)) if use_cache else None
if cache_key:
cached = get_cached_voice_prompt(cache_key)
if cached is not None and isinstance(cached, dict):
return cached, True
def _encode_sync():
import torch
import soundfile as sf
device = self._device
# Load audio with soundfile (torchaudio 2.10+ requires torchcodec)
audio_np, sr = sf.read(str(audio_path), dtype="float32")
audio = torch.from_numpy(audio_np).float()
if audio.ndim == 1:
audio = audio.unsqueeze(0) # (samples,) -> (1, samples)
else:
audio = audio.T # (samples, channels) -> (channels, samples)
audio = audio.to(device)
# Encode with forced alignment
text_arg = [reference_text] if reference_text else None
prompt = self.encoder(audio, text=text_arg, sample_rate=sr)
# Serialize EncoderOutput to a dict of CPU tensors for caching
prompt_dict = {}
for field_name in prompt.__dataclass_fields__:
val = getattr(prompt, field_name)
if isinstance(val, torch.Tensor):
prompt_dict[field_name] = val.detach().cpu()
elif isinstance(val, list):
prompt_dict[field_name] = val
elif isinstance(val, (int, float)):
prompt_dict[field_name] = val
else:
prompt_dict[field_name] = val
return prompt_dict
encoded = await asyncio.to_thread(_encode_sync)
if cache_key:
cache_voice_prompt(cache_key, encoded)
return encoded, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
return await _combine_voice_prompts(audio_paths, reference_texts, sample_rate=24000)
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio from text using HumeAI TADA.
Args:
text: Text to synthesize
voice_prompt: Serialized EncoderOutput dict from create_voice_prompt()
language: Language code (en, ar, de, es, fr, it, ja, pl, pt, zh)
seed: Random seed for reproducibility
instruct: Not supported by TADA (ignored)
Returns:
Tuple of (audio_array, sample_rate=24000)
"""
await self.load_model(self.model_size)
def _generate_sync():
import torch
from tada.modules.encoder import EncoderOutput
if seed is not None:
manual_seed(seed, self._device)
device = self._device
# Reconstruct EncoderOutput from the cached dict
restored = {}
for k, v in voice_prompt.items():
if isinstance(v, torch.Tensor):
# Move to device and match model dtype for float tensors
if v.is_floating_point():
model_dtype = next(self.model.parameters()).dtype
restored[k] = v.to(device=device, dtype=model_dtype)
else:
restored[k] = v.to(device=device)
else:
restored[k] = v
prompt = EncoderOutput(**restored)
# For non-English with the 3B-ML model, we could reload the
# encoder with the language-specific aligner. However, the
# generation itself is language-agnostic — only the encoder's
# aligner changes. Since we encode at create_voice_prompt time,
# the language is already baked in. For simplicity, we don't
# reload the encoder here.
logger.info(f"[TADA] Generating ({language}), text length: {len(text)}")
output = self.model.generate(
prompt=prompt,
text=text,
)
# output.audio is a list of tensors (one per batch item)
if output.audio and output.audio[0] is not None:
audio_tensor = output.audio[0]
audio = audio_tensor.detach().cpu().numpy().squeeze().astype(np.float32)
else:
logger.warning("[TADA] Generation produced no audio")
audio = np.zeros(24000, dtype=np.float32)
return audio, 24000
return await asyncio.to_thread(_generate_sync)
+288
View File
@@ -0,0 +1,288 @@
"""
Kokoro TTS backend implementation.
Wraps the Kokoro-82M model for fast, lightweight text-to-speech.
82M parameters, CPU realtime, 24kHz output, Apache 2.0 license.
Kokoro uses pre-built voice style vectors (not traditional zero-shot cloning
from arbitrary audio). Voice prompts are stored as deferred references to
HF-hosted voice .pt files.
Languages supported (via misaki G2P):
- American English (a), British English (b)
- Spanish (e), French (f), Hindi (h), Italian (i), Portuguese (p)
- Japanese (j) — requires misaki[ja]
- Chinese (z) — requires misaki[zh]
"""
import asyncio
import logging
import os
from typing import Optional
import numpy as np
from . import TTSBackend
from .base import (
get_torch_device,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
logger = logging.getLogger(__name__)
# HuggingFace repo for model + voice detection
KOKORO_HF_REPO = "hexgrad/Kokoro-82M"
KOKORO_SAMPLE_RATE = 24000
# Default voice if none specified
KOKORO_DEFAULT_VOICE = "af_heart"
# All available Kokoro voices: (voice_id, display_name, gender, lang_code)
KOKORO_VOICES = [
# American English female
("af_alloy", "Alloy", "female", "en"),
("af_aoede", "Aoede", "female", "en"),
("af_bella", "Bella", "female", "en"),
("af_heart", "Heart", "female", "en"),
("af_jessica", "Jessica", "female", "en"),
("af_kore", "Kore", "female", "en"),
("af_nicole", "Nicole", "female", "en"),
("af_nova", "Nova", "female", "en"),
("af_river", "River", "female", "en"),
("af_sarah", "Sarah", "female", "en"),
("af_sky", "Sky", "female", "en"),
# American English male
("am_adam", "Adam", "male", "en"),
("am_echo", "Echo", "male", "en"),
("am_eric", "Eric", "male", "en"),
("am_fenrir", "Fenrir", "male", "en"),
("am_liam", "Liam", "male", "en"),
("am_michael", "Michael", "male", "en"),
("am_onyx", "Onyx", "male", "en"),
("am_puck", "Puck", "male", "en"),
("am_santa", "Santa", "male", "en"),
# British English female
("bf_alice", "Alice", "female", "en"),
("bf_emma", "Emma", "female", "en"),
("bf_isabella", "Isabella", "female", "en"),
("bf_lily", "Lily", "female", "en"),
# British English male
("bm_daniel", "Daniel", "male", "en"),
("bm_fable", "Fable", "male", "en"),
("bm_george", "George", "male", "en"),
("bm_lewis", "Lewis", "male", "en"),
# Spanish
("ef_dora", "Dora", "female", "es"),
("em_alex", "Alex", "male", "es"),
("em_santa", "Santa", "male", "es"),
# French
("ff_siwis", "Siwis", "female", "fr"),
# Hindi
("hf_alpha", "Alpha", "female", "hi"),
("hf_beta", "Beta", "female", "hi"),
("hm_omega", "Omega", "male", "hi"),
("hm_psi", "Psi", "male", "hi"),
# Italian
("if_sara", "Sara", "female", "it"),
("im_nicola", "Nicola", "male", "it"),
# Japanese
("jf_alpha", "Alpha", "female", "ja"),
("jf_gongitsune", "Gongitsune", "female", "ja"),
("jf_nezumi", "Nezumi", "female", "ja"),
("jf_tebukuro", "Tebukuro", "female", "ja"),
("jm_kumo", "Kumo", "male", "ja"),
# Portuguese
("pf_dora", "Dora", "female", "pt"),
("pm_alex", "Alex", "male", "pt"),
("pm_santa", "Santa", "male", "pt"),
# Chinese
("zf_xiaobei", "Xiaobei", "female", "zh"),
("zf_xiaoni", "Xiaoni", "female", "zh"),
("zf_xiaoxiao", "Xiaoxiao", "female", "zh"),
("zf_xiaoyi", "Xiaoyi", "female", "zh"),
]
# Map our ISO language codes to Kokoro lang_code characters
LANG_CODE_MAP = {
"en": "a", # American English
"es": "e",
"fr": "f",
"hi": "h",
"it": "i",
"pt": "p",
"ja": "j",
"zh": "z",
}
class KokoroTTSBackend:
"""Kokoro-82M TTS backend — tiny, fast, CPU-friendly."""
def __init__(self):
self._model = None
self._pipelines: dict = {} # lang_code -> KPipeline
self._device: Optional[str] = None
self.model_size = "default"
def _get_device(self) -> str:
"""Select device. Kokoro supports CUDA and CPU. MPS needs fallback env var."""
device = get_torch_device(allow_mps=False)
# Kokoro can use MPS but requires PYTORCH_ENABLE_MPS_FALLBACK=1
# For now, skip MPS to avoid user confusion — CPU is already realtime
return device
@property
def device(self) -> str:
if self._device is None:
self._device = self._get_device()
return self._device
def is_loaded(self) -> bool:
return self._model is not None
def _get_model_path(self, model_size: str) -> str:
return KOKORO_HF_REPO
def _is_model_cached(self, model_size: str = "default") -> bool:
"""Check if Kokoro model files are cached locally."""
from .base import is_model_cached
return is_model_cached(
KOKORO_HF_REPO,
required_files=["config.json", "kokoro-v1_0.pth"],
)
async def load_model(self, model_size: str = "default") -> None:
"""Load the Kokoro model."""
if self._model is not None:
return
await asyncio.to_thread(self._load_model_sync)
def _load_model_sync(self):
"""Synchronous model loading."""
model_name = "kokoro"
is_cached = self._is_model_cached()
with model_load_progress(model_name, is_cached):
from kokoro import KModel
device = self.device
logger.info(f"Loading Kokoro-82M on {device}...")
self._model = KModel(repo_id=KOKORO_HF_REPO).to(device).eval()
logger.info("Kokoro-82M loaded successfully")
def _get_pipeline(self, lang_code: str):
"""Get or create a KPipeline for the given language code."""
kokoro_lang = LANG_CODE_MAP.get(lang_code, "a")
if kokoro_lang not in self._pipelines:
from kokoro import KPipeline
# Create pipeline with our existing model (no redundant model loading)
self._pipelines[kokoro_lang] = KPipeline(
lang_code=kokoro_lang,
repo_id=KOKORO_HF_REPO,
model=self._model,
)
return self._pipelines[kokoro_lang]
def unload_model(self) -> None:
"""Unload model to free memory."""
if self._model is not None:
del self._model
self._model = None
self._pipelines.clear()
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
logger.info("Kokoro unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> tuple[dict, bool]:
"""
Create voice prompt for Kokoro.
Kokoro doesn't do traditional voice cloning from arbitrary audio.
When called for a cloned profile (fallback), uses the default voice.
For preset profiles, the voice_prompt dict is built by the profile
service and bypasses this method entirely.
"""
return {
"voice_type": "preset",
"preset_engine": "kokoro",
"preset_voice_id": KOKORO_DEFAULT_VOICE,
}, False
async def combine_voice_prompts(
self,
audio_paths: list[str],
reference_texts: list[str],
) -> tuple[np.ndarray, str]:
"""Combine voice prompts — uses base implementation for audio concatenation."""
return await _combine_voice_prompts(
audio_paths, reference_texts, sample_rate=KOKORO_SAMPLE_RATE
)
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> tuple[np.ndarray, int]:
"""
Generate audio from text using Kokoro.
Args:
text: Text to synthesize
voice_prompt: Dict with kokoro_voice key
language: Language code
seed: Random seed for reproducibility
instruct: Not supported by Kokoro (ignored)
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model()
voice_name = voice_prompt.get("preset_voice_id") or voice_prompt.get("kokoro_voice") or KOKORO_DEFAULT_VOICE
def _generate_sync():
import torch
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
pipeline = self._get_pipeline(language)
# Generate all chunks and concatenate
audio_chunks = []
for result in pipeline(text, voice=voice_name, speed=1.0):
if result.audio is not None:
chunk = result.audio
if isinstance(chunk, torch.Tensor):
chunk = chunk.detach().cpu().numpy()
audio_chunks.append(chunk.squeeze())
if not audio_chunks:
# Return 1 second of silence as fallback
return np.zeros(KOKORO_SAMPLE_RATE, dtype=np.float32), KOKORO_SAMPLE_RATE
audio = np.concatenate(audio_chunks)
return audio.astype(np.float32), KOKORO_SAMPLE_RATE
return await asyncio.to_thread(_generate_sync)
+17 -11
View File
@@ -12,7 +12,14 @@ from typing import Optional, Tuple
import numpy as np
from . import TTSBackend
from .base import is_model_cached, get_torch_device, combine_voice_prompts as _combine_voice_prompts, model_load_progress
from .base import (
is_model_cached,
get_torch_device,
empty_device_cache,
manual_seed,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
logger = logging.getLogger(__name__)
@@ -30,7 +37,7 @@ class LuxTTSBackend:
self._device = None
def _get_device(self) -> str:
return get_torch_device(allow_mps=True)
return get_torch_device(allow_mps=True, allow_xpu=True)
def is_loaded(self) -> bool:
return self.model is not None
@@ -69,9 +76,12 @@ class LuxTTSBackend:
if device == "cpu":
import os
threads = os.cpu_count() or 4
self.model = LuxTTS(
model_path=LUXTTS_HF_REPO, device="cpu", threads=min(threads, 8),
model_path=LUXTTS_HF_REPO,
device="cpu",
threads=min(threads, 8),
)
else:
self.model = LuxTTS(model_path=LUXTTS_HF_REPO, device=device)
@@ -81,12 +91,12 @@ class LuxTTSBackend:
def unload_model(self) -> None:
"""Unload model to free memory."""
if self.model is not None:
device = self.device
del self.model
self.model = None
self._device = None
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
empty_device_cache(device)
logger.info("LuxTTS unloaded")
@@ -154,12 +164,8 @@ class LuxTTSBackend:
await self.load_model()
def _generate_sync():
import torch
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
manual_seed(seed, self.device)
wav = self.model.generate_speech(
text=text,
+9 -26
View File
@@ -6,7 +6,6 @@ from typing import Optional, List, Tuple
import asyncio
import logging
import numpy as np
import os
from pathlib import Path
logger = logging.getLogger(__name__)
@@ -21,6 +20,7 @@ ensure_original_qwen_config_cached()
from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
from .base import is_model_cached, combine_voice_prompts as _combine_voice_prompts, model_load_progress
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from ..utils.hf_offline_patch import force_offline_if_cached
class MLXTTSBackend:
@@ -96,32 +96,13 @@ class MLXTTSBackend:
model_name = f"qwen-tts-{model_size}"
is_cached = self._is_model_cached(model_size)
# Force offline mode when cached to avoid network requests
original_hf_hub_offline = os.environ.get("HF_HUB_OFFLINE")
if is_cached:
os.environ["HF_HUB_OFFLINE"] = "1"
logger.info("[PATCH] Model %s is cached, forcing HF_HUB_OFFLINE=1 to avoid network requests", model_size)
with model_load_progress(model_name, is_cached):
from mlx_audio.tts import load
try:
with model_load_progress(model_name, is_cached):
from mlx_audio.tts import load
logger.info("Loading MLX TTS model %s...", model_size)
logger.info("Loading MLX TTS model %s...", model_size)
try:
self.model = load(model_path)
except Exception as load_error:
if is_cached and "offline" in str(load_error).lower():
logger.warning("[PATCH] Offline load failed, trying with network: %s", load_error)
os.environ.pop("HF_HUB_OFFLINE", None)
self.model = load(model_path)
else:
raise
finally:
if original_hf_hub_offline is not None:
os.environ["HF_HUB_OFFLINE"] = original_hf_hub_offline
else:
os.environ.pop("HF_HUB_OFFLINE", None)
with force_offline_if_cached(is_cached, model_name):
self.model = load(model_path)
self._current_model_size = model_size
self.model_size = model_size
@@ -329,7 +310,9 @@ class MLXSTTBackend:
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
logger.info("Loading MLX Whisper model %s...", model_size)
self.model = load(model_name)
with force_offline_if_cached(is_cached, progress_model_name):
self.model = load(model_name)
self.model_size = model_size
logger.info("MLX Whisper model %s loaded successfully", model_size)
+31 -21
View File
@@ -14,11 +14,14 @@ from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
from .base import (
is_model_cached,
get_torch_device,
empty_device_cache,
manual_seed,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from ..utils.audio import load_audio
from ..utils.hf_offline_patch import force_offline_if_cached
class PyTorchTTSBackend:
@@ -96,18 +99,28 @@ class PyTorchTTSBackend:
model_path = self._get_model_path(model_size)
logger.info("Loading TTS model %s on %s...", model_size, self.device)
if self.device == "cpu":
self.model = Qwen3TTSModel.from_pretrained(
model_path,
torch_dtype=torch.float32,
low_cpu_mem_usage=False,
)
else:
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16,
)
# Route both HF Hub and Transformers through a single cache root.
# On Windows local setups, model assets can otherwise split between
# .hf-cache/hub and .hf-cache/transformers, causing speech_tokenizer
# and preprocessor_config.json to fail to resolve during load.
from huggingface_hub import constants as hf_constants
tts_cache_dir = hf_constants.HF_HUB_CACHE
with force_offline_if_cached(is_cached, model_name):
if self.device == "cpu":
self.model = Qwen3TTSModel.from_pretrained(
model_path,
cache_dir=tts_cache_dir,
torch_dtype=torch.float32,
low_cpu_mem_usage=False,
)
else:
self.model = Qwen3TTSModel.from_pretrained(
model_path,
cache_dir=tts_cache_dir,
device_map=self.device,
torch_dtype=torch.bfloat16,
)
self._current_model_size = model_size
self.model_size = model_size
@@ -120,8 +133,7 @@ class PyTorchTTSBackend:
self.model = None
self._current_model_size = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
empty_device_cache(self.device)
logger.info("TTS model unloaded")
@@ -213,9 +225,7 @@ class PyTorchTTSBackend:
"""Run synchronous generation in thread pool."""
# Set seed if provided
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
manual_seed(seed, self.device)
# Generate audio - this is the blocking operation
wavs, sample_rate = self.model.generate_voice_clone(
@@ -282,8 +292,9 @@ class PyTorchSTTBackend:
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
logger.info("Loading Whisper model %s on %s...", model_size, self.device)
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
with force_offline_if_cached(is_cached, progress_model_name):
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
self.model_size = model_size
@@ -297,8 +308,7 @@ class PyTorchSTTBackend:
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
empty_device_cache(self.device)
logger.info("Whisper model unloaded")
@@ -0,0 +1,210 @@
"""
Qwen3-TTS CustomVoice backend implementation.
Wraps the Qwen3-TTS-12Hz CustomVoice model for preset-speaker TTS with
instruction-based style control. Uses the same qwen_tts library as the
Base model (pytorch_backend.py) but loads a different checkpoint and
calls generate_custom_voice() instead of generate_voice_clone().
Key differences from the Base engine:
- Uses preset speakers (9 built-in voices) instead of zero-shot cloning
- Supports instruct parameter for tone/emotion/prosody control
- Two model sizes: 1.7B and 0.6B
Languages supported: zh, en, ja, ko, de, fr, ru, pt, es, it
"""
import asyncio
import logging
from typing import Optional
import numpy as np
import torch
from . import TTSBackend, LANGUAGE_CODE_TO_NAME
from .base import (
is_model_cached,
get_torch_device,
combine_voice_prompts as _combine_voice_prompts,
model_load_progress,
)
logger = logging.getLogger(__name__)
# ── Preset speakers ──────────────────────────────────────────────────
# (speaker_id, display_name, gender, native_language_code, description)
QWEN_CUSTOM_VOICES = [
("Vivian", "Vivian", "female", "zh", "Bright, slightly edgy young female voice"),
("Serena", "Serena", "female", "zh", "Warm, gentle young female voice"),
("Uncle_Fu", "Uncle Fu", "male", "zh", "Seasoned male voice with a low, mellow timbre"),
("Dylan", "Dylan", "male", "zh", "Youthful Beijing male voice with a clear, natural timbre"),
("Eric", "Eric", "male", "zh", "Lively Chengdu male voice with a slightly husky brightness"),
("Ryan", "Ryan", "male", "en", "Dynamic male voice with strong rhythmic drive"),
("Aiden", "Aiden", "male", "en", "Sunny American male voice with a clear midrange"),
("Ono_Anna", "Ono Anna", "female", "ja", "Playful Japanese female voice with a light, nimble timbre"),
("Sohee", "Sohee", "female", "ko", "Warm Korean female voice with rich emotion"),
]
QWEN_CV_DEFAULT_SPEAKER = "Ryan"
# HuggingFace repo IDs per model size
QWEN_CV_HF_REPOS = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
}
class QwenCustomVoiceBackend:
"""Qwen3-TTS CustomVoice backend — preset speakers with instruct control."""
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device()
self._current_model_size: Optional[str] = None
def _get_device(self) -> str:
return get_torch_device(allow_xpu=True, allow_directml=True)
def is_loaded(self) -> bool:
return self.model is not None
def _get_model_path(self, model_size: str) -> str:
if model_size not in QWEN_CV_HF_REPOS:
raise ValueError(f"Unknown model size: {model_size}")
return QWEN_CV_HF_REPOS[model_size]
def _is_model_cached(self, model_size: Optional[str] = None) -> bool:
size = model_size or self.model_size
return is_model_cached(self._get_model_path(size))
async def load_model_async(self, model_size: Optional[str] = None) -> None:
if model_size is None:
model_size = self.model_size
if self.model is not None and self._current_model_size == model_size:
return
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
await asyncio.to_thread(self._load_model_sync, model_size)
# Alias for compatibility with the TTSBackend protocol
load_model = load_model_async
def _load_model_sync(self, model_size: str) -> None:
model_name = f"qwen-custom-voice-{model_size}"
is_cached = self._is_model_cached(model_size)
with model_load_progress(model_name, is_cached):
from qwen_tts import Qwen3TTSModel
model_path = self._get_model_path(model_size)
logger.info("Loading Qwen CustomVoice %s on %s...", model_size, self.device)
if self.device == "cpu":
self.model = Qwen3TTSModel.from_pretrained(
model_path,
torch_dtype=torch.float32,
low_cpu_mem_usage=False,
)
else:
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16,
)
self._current_model_size = model_size
self.model_size = model_size
logger.info("Qwen CustomVoice %s loaded successfully", model_size)
def unload_model(self) -> None:
if self.model is not None:
del self.model
self.model = None
self._current_model_size = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
logger.info("Qwen CustomVoice unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> tuple[dict, bool]:
"""
Create voice prompt for CustomVoice.
CustomVoice doesn't use reference audio — it uses preset speakers.
When called for a cloned profile (fallback), uses the default speaker.
For preset profiles, the voice_prompt dict is built by the profile
service and bypasses this method entirely.
"""
return {
"voice_type": "preset",
"preset_engine": "qwen_custom_voice",
"preset_voice_id": QWEN_CV_DEFAULT_SPEAKER,
}, False
async def combine_voice_prompts(
self,
audio_paths: list[str],
reference_texts: list[str],
) -> tuple[np.ndarray, str]:
return await _combine_voice_prompts(audio_paths, reference_texts)
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> tuple[np.ndarray, int]:
"""
Generate audio using Qwen CustomVoice.
Args:
text: Text to synthesize
voice_prompt: Dict with preset_voice_id (speaker name)
language: Language code (zh, en, ja, ko, etc.)
seed: Random seed for reproducibility
instruct: Natural language instruction for style control
(e.g. "Speak in an angry tone", "Very happy")
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model_async(None)
speaker = voice_prompt.get("preset_voice_id") or QWEN_CV_DEFAULT_SPEAKER
def _generate_sync():
if seed is not None:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed(seed)
lang_name = LANGUAGE_CODE_TO_NAME.get(language, "auto")
kwargs = {
"text": text,
"language": lang_name.capitalize() if lang_name != "auto" else "Auto",
"speaker": speaker,
}
# Only pass instruct if non-empty
if instruct:
kwargs["instruct"] = instruct
wavs, sample_rate = self.model.generate_custom_voice(**kwargs)
return wavs[0], sample_rate
audio, sample_rate = await asyncio.to_thread(_generate_sync)
return audio, sample_rate
+97 -5
View File
@@ -34,9 +34,15 @@ def build_server(cuda=False):
binary_name = "voicebox-server-cuda" if cuda else "voicebox-server"
# PyInstaller arguments
# CUDA builds use --onedir so we can split the output into two archives:
# 1. Server core (~200-400MB) — versioned with the app
# 2. CUDA libs (~2GB) — versioned independently (only redownloaded on
# CUDA toolkit / torch major version changes)
# CPU builds remain --onefile for simplicity.
pack_mode = "--onedir" if cuda else "--onefile"
args = [
"server.py", # Use server.py as entry point instead of main.py
"--onefile",
pack_mode,
"--name",
binary_name,
]
@@ -46,6 +52,16 @@ def build_server(cuda=False):
if platform.system() == "Windows":
args.append("--noconsole")
# numpy 2.x / torch ABI mismatch fix: install memmove fallback for
# torch.from_numpy() before the app starts. Runtime hooks run after
# FrozenImporter is registered so frozen torch/numpy are importable.
args.extend(
[
"--runtime-hook",
str(backend_dir / "pyi_rth_numpy_compat.py"),
]
)
# Add local qwen_tts path if specified (for editable installs)
qwen_tts_path = os.getenv("QWEN_TTS_PATH")
if qwen_tts_path and Path(qwen_tts_path).exists():
@@ -80,6 +96,8 @@ def build_server(cuda=False):
"--hidden-import",
"backend.backends.pytorch_backend",
"--hidden-import",
"backend.backends.qwen_custom_voice_backend",
"--hidden-import",
"backend.utils.audio",
"--hidden-import",
"backend.utils.cache",
@@ -165,9 +183,9 @@ def build_server(cuda=False):
"tqdm",
"--hidden-import",
"requests",
"--collect-submodules",
"qwen_tts",
"--collect-data",
# qwen_tts uses inspect.getsource() at runtime to locate
# modeling_qwen3_tts.py — needs physical .py source files bundled
"--collect-all",
"qwen_tts",
# Fix for pkg_resources and jaraco namespace packages
"--hidden-import",
@@ -186,6 +204,80 @@ def build_server(cuda=False):
# needed by LuxTTS for text-to-phoneme conversion
"--collect-all",
"piper_phonemize",
# HumeAI TADA — speech-language model using Llama + flow matching
"--hidden-import",
"backend.backends.hume_backend",
"--hidden-import",
"tada",
"--hidden-import",
"tada.modules",
"--hidden-import",
"tada.modules.tada",
"--hidden-import",
"tada.modules.encoder",
"--hidden-import",
"tada.modules.decoder",
"--hidden-import",
"tada.modules.aligner",
"--hidden-import",
"tada.modules.acoustic_spkr_verf",
"--hidden-import",
"tada.nn",
"--hidden-import",
"tada.nn.vibevoice",
"--hidden-import",
"tada.utils",
"--hidden-import",
"tada.utils.gray_code",
"--hidden-import",
"tada.utils.text",
# DAC shim — provides dac.nn.layers.Snake1d without the real
# descript-audio-codec package (which pulls onnx/tensorboard via
# descript-audiotools). The shim is in backend/utils/dac_shim.py.
"--hidden-import",
"backend.utils.dac_shim",
"--hidden-import",
"torchaudio",
"--collect-submodules",
"tada",
# Kokoro 82M — lightweight TTS engine using misaki G2P
"--hidden-import",
"backend.backends.kokoro_backend",
"--hidden-import",
"kokoro",
"--hidden-import",
"kokoro.pipeline",
"--hidden-import",
"kokoro.model",
"--hidden-import",
"kokoro.istftnet",
"--hidden-import",
"kokoro.modules",
"--hidden-import",
"kokoro.custom_stft",
# misaki ships G2P data files (dictionaries, phoneme tables)
# that must be bundled for espeak/en/ja/zh G2P to work
"--collect-all",
"misaki",
# language_tags ships JSON data files (index.json etc.) loaded at
# runtime via: misaki → phonemizer → segments → csvw → language_tags
"--collect-all",
"language_tags",
# espeakng_loader ships the entire espeak-ng-data directory (369 files)
# loaded at import time by misaki.espeak via get_data_path()
"--collect-all",
"espeakng_loader",
# spacy en_core_web_sm model — misaki.en tries to spacy.cli.download()
# at runtime if not found, which calls pip as a subprocess and crashes
# the frozen binary. Bundle the model so spacy.util.is_package() passes.
"--collect-all",
"en_core_web_sm",
"--copy-metadata",
"en_core_web_sm",
"--hidden-import",
"en_core_web_sm",
"--hidden-import",
"loguru",
]
)
@@ -328,7 +420,7 @@ def build_server(cuda=False):
"torchvision",
"torchaudio",
"--index-url",
"https://download.pytorch.org/whl/cu126",
"https://download.pytorch.org/whl/cu128",
"--force-reinstall",
"-q",
],
+58 -3
View File
@@ -19,7 +19,22 @@ if _custom_models_dir:
logger.info("Model download path set to: %s", _custom_models_dir)
# Default data directory (used in development)
_data_dir = Path("data")
_data_dir = Path("data").resolve()
def _path_relative_to_any_data_dir(path: Path) -> Path | None:
"""Extract the path within a data dir from an absolute or relative path."""
parts = path.parts
for idx, part in enumerate(parts):
if part != "data":
continue
tail = parts[idx + 1 :]
if tail:
return Path(*tail)
return Path()
return None
def set_data_dir(path: str | Path):
@@ -30,9 +45,9 @@ def set_data_dir(path: str | Path):
path: Path to the data directory
"""
global _data_dir
_data_dir = Path(path)
_data_dir = Path(path).resolve()
_data_dir.mkdir(parents=True, exist_ok=True)
logger.info("Data directory set to: %s", _data_dir.absolute())
logger.info("Data directory set to: %s", _data_dir)
def get_data_dir() -> Path:
@@ -45,6 +60,46 @@ def get_data_dir() -> Path:
return _data_dir
def to_storage_path(path: str | Path) -> str:
"""Convert a filesystem path to a DB-safe path relative to the data dir."""
resolved_path = Path(path).resolve()
relative_to_any_data_dir = _path_relative_to_any_data_dir(resolved_path)
if relative_to_any_data_dir is not None:
return str(relative_to_any_data_dir)
try:
return str(resolved_path.relative_to(_data_dir))
except ValueError:
return str(resolved_path)
def resolve_storage_path(path: str | Path | None) -> Path | None:
"""Resolve a DB-stored path against the configured data dir."""
if path is None:
return None
stored_path = Path(path)
if stored_path.is_absolute():
rebased_path = _path_relative_to_any_data_dir(stored_path)
if rebased_path is not None:
candidate = (_data_dir / rebased_path).resolve()
if candidate.exists() or not stored_path.exists():
return candidate
return stored_path
# 0.3.0 records sometimes stored relative paths with the data-dir name
# baked in (e.g. "data/profiles/..."). Joining those directly with
# _data_dir produces a spurious "<data_dir>/data/profiles/..." nest.
if stored_path.parts and stored_path.parts[0] == "data":
stored_path = (
Path(*stored_path.parts[1:]) if len(stored_path.parts) > 1 else Path()
)
return (_data_dir / stored_path).resolve()
def get_db_path() -> Path:
"""Get database file path."""
return _data_dir / "voicebox.db"
+56
View File
@@ -34,6 +34,7 @@ def run_migrations(engine) -> None:
_migrate_generations(engine, inspector, tables)
_migrate_effect_presets(engine, inspector, tables)
_migrate_generation_versions(engine, inspector, tables)
_normalize_storage_paths(engine, tables)
# -- helpers ---------------------------------------------------------------
@@ -134,6 +135,17 @@ def _migrate_profiles(engine, inspector, tables: set[str]) -> None:
_add_column(engine, "profiles", "avatar_path VARCHAR", "avatar_path")
if "effects_chain" not in columns:
_add_column(engine, "profiles", "effects_chain TEXT", "effects_chain")
# Voice type system — v0.3.x
if "voice_type" not in columns:
_add_column(engine, "profiles", "voice_type VARCHAR DEFAULT 'cloned'", "voice_type")
if "preset_engine" not in columns:
_add_column(engine, "profiles", "preset_engine VARCHAR", "preset_engine")
if "preset_voice_id" not in columns:
_add_column(engine, "profiles", "preset_voice_id VARCHAR", "preset_voice_id")
if "design_prompt" not in columns:
_add_column(engine, "profiles", "design_prompt TEXT", "design_prompt")
if "default_engine" not in columns:
_add_column(engine, "profiles", "default_engine VARCHAR", "default_engine")
def _migrate_generations(engine, inspector, tables: set[str]) -> None:
@@ -168,3 +180,47 @@ def _migrate_generation_versions(engine, inspector, tables: set[str]) -> None:
columns = _get_columns(inspector, "generation_versions")
if "source_version_id" not in columns:
_add_column(engine, "generation_versions", "source_version_id VARCHAR", "source_version_id")
def _normalize_storage_paths(engine, tables: set[str]) -> None:
"""Normalize stored file paths to be relative to the configured data dir."""
from pathlib import Path
from ..config import get_data_dir, to_storage_path, resolve_storage_path
data_dir = get_data_dir()
path_columns = [
("generations", "audio_path"),
("generation_versions", "audio_path"),
("profile_samples", "audio_path"),
("profiles", "avatar_path"),
]
total_fixed = 0
with engine.connect() as conn:
for table, column in path_columns:
if table not in tables:
continue
rows = conn.execute(
text(f"SELECT id, {column} FROM {table} WHERE {column} IS NOT NULL")
).fetchall()
for row_id, path_val in rows:
if not path_val:
continue
p = Path(path_val)
resolved = resolve_storage_path(p)
if resolved is None:
continue
normalized = to_storage_path(resolved)
if normalized != path_val:
conn.execute(
text(f"UPDATE {table} SET {column} = :path WHERE id = :id"),
{"path": normalized, "id": row_id},
)
total_fixed += 1
if total_fixed > 0:
conn.commit()
logger.info("Normalized %d stored file paths", total_fixed)
+15 -1
View File
@@ -10,7 +10,13 @@ Base = declarative_base()
class VoiceProfile(Base):
"""Voice profile."""
"""Voice profile.
voice_type discriminates three flavours:
- "cloned" — traditional reference-audio profiles (all cloning engines)
- "preset" — engine-specific pre-built voice (e.g. Kokoro voices)
- "designed" — text-described voice (e.g. Qwen CustomVoice, future)
"""
__tablename__ = "profiles"
@@ -20,6 +26,14 @@ class VoiceProfile(Base):
language = Column(String, default="en")
avatar_path = Column(String, nullable=True)
effects_chain = Column(Text, nullable=True)
# Voice type system — added v0.3.x
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
design_prompt = Column(Text, nullable=True) # text description — only for designed
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
+4 -2
View File
@@ -3,7 +3,8 @@
import json
import logging
import uuid
from pathlib import Path
from .. import config
logger = logging.getLogger(__name__)
@@ -25,7 +26,8 @@ def backfill_generation_versions(SessionLocal, Generation, GenerationVersion) ->
for gen in generations:
if gen.id in existing_version_gen_ids:
continue
if not Path(gen.audio_path).exists():
resolved_audio_path = config.resolve_storage_path(gen.audio_path)
if resolved_audio_path is None or not resolved_audio_path.exists():
continue
version = GenerationVersion(
id=str(uuid.uuid4()),
+13 -2
View File
@@ -15,6 +15,11 @@ class VoiceProfileCreate(BaseModel):
language: str = Field(
default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$"
)
voice_type: Optional[str] = Field(default="cloned", pattern="^(cloned|preset|designed)$")
preset_engine: Optional[str] = Field(None, max_length=50)
preset_voice_id: Optional[str] = Field(None, max_length=100)
design_prompt: Optional[str] = Field(None, max_length=2000)
default_engine: Optional[str] = Field(None, max_length=50)
class VoiceProfileResponse(BaseModel):
@@ -26,6 +31,11 @@ class VoiceProfileResponse(BaseModel):
language: str
avatar_path: Optional[str] = None
effects_chain: Optional[List["EffectConfig"]] = None
voice_type: str = "cloned"
preset_engine: Optional[str] = None
preset_voice_id: Optional[str] = None
design_prompt: Optional[str] = None
default_engine: Optional[str] = None
generation_count: int = 0
sample_count: int = 0
created_at: datetime
@@ -66,9 +76,9 @@ class GenerationRequest(BaseModel):
text: str = Field(..., min_length=1, max_length=50000)
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$")
seed: Optional[int] = Field(None, ge=0)
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B)$")
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B|1B|3B)$")
instruct: Optional[str] = Field(None, max_length=500)
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo)$")
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$")
max_chunk_chars: int = Field(
default=800, ge=100, le=5000, description="Max characters per chunk for long text splitting"
)
@@ -172,6 +182,7 @@ class HealthResponse(BaseModel):
vram_used_mb: Optional[float] = None
backend_type: Optional[str] = None # Backend type (mlx or pytorch)
backend_variant: Optional[str] = None # Binary variant (cpu or cuda)
gpu_compatibility_warning: Optional[str] = None # Warning if GPU arch unsupported
class DirectoryCheck(BaseModel):
+95
View File
@@ -0,0 +1,95 @@
"""
PyInstaller runtime hook: numpy 2.x / torch ABI mismatch fix.
Problem
-------
torch is compiled against numpy 1.x headers. numpy 2.x changed the version
number returned by PyArray_GetNDArrayCVersion() (0x01000009 → 0x02000000),
so torch's is_numpy_available() returns False and every torch.from_numpy()
call raises:
RuntimeError: Numpy is not available
This surfaces as:
ValueError: Unable to create tensor, you should probably activate
padding with 'padding=True'
during TTS generation (EncodecFeatureExtractor → BatchFeature.convert_to_tensors).
Fix
---
Runtime hooks execute after PyInstaller's FrozenImporter is registered, so
frozen torch/numpy are importable here. We start a background thread that
waits for torch to finish loading then wraps torch.from_numpy with a ctypes
memmove fallback that bypasses the C-level numpy ABI check entirely.
This approach works with any numpy version and is safer than binary-patching
libtorch_python.dylib (which risks PyArray_Descr struct layout mismatches).
"""
import sys
import threading
def _patch_torch_from_numpy():
import time
for _ in range(7200): # poll up to 360 s at 50 ms intervals
time.sleep(0.05)
torch = sys.modules.get("torch")
if torch is None or not hasattr(torch, "from_numpy"):
continue
if getattr(torch, "_vb_from_numpy_patched", False):
return
try:
import ctypes
import numpy as np
_orig = torch.from_numpy
# Explicit numpy → torch dtype map. Silent fallback to float32 on
# unknown dtypes would reinterpret the memcpy'd bytes as fp32 and
# silently corrupt data (e.g. fp16 tensors from some TTS engines),
# so we raise instead.
dtype_map = {
"float16": _t.float16,
"float32": _t.float32,
"float64": _t.float64,
"int8": _t.int8,
"int16": _t.int16,
"int32": _t.int32,
"int64": _t.int64,
"uint8": _t.uint8,
"bool": _t.bool,
"complex64": _t.complex64,
"complex128": _t.complex128,
}
def _safe_from_numpy(
arr, _orig=_orig, _c=ctypes, _np=np, _t=torch, _map=dtype_map
):
try:
return _orig(arr)
except RuntimeError:
a = _np.ascontiguousarray(arr)
key = str(a.dtype)
if key not in _map:
raise TypeError(
f"pyi_rth_numpy_compat: unsupported numpy dtype "
f"{key!r} in torch.from_numpy fallback; add an "
f"explicit mapping rather than silently copying "
f"bytes into the wrong dtype."
)
out = _t.empty(list(a.shape), dtype=_map[key])
_c.memmove(out.data_ptr(), a.ctypes.data, a.nbytes)
return out
torch.from_numpy = _safe_from_numpy
torch._vb_from_numpy_patched = True
except Exception:
pass
return
threading.Thread(target=_patch_torch_from_numpy, daemon=True).start()
+16 -2
View File
@@ -8,7 +8,7 @@ sqlalchemy>=2.0.0
alembic>=1.13.0
# ML models
torch>=2.1.0
torch>=2.2.0
transformers>=4.36.0,<=4.57.6
accelerate>=0.26.0
huggingface_hub>=0.20.0
@@ -33,10 +33,24 @@ s3tokenizer
spacy-pkuseg
pyloudnorm
# HumeAI TADA sub-dependencies (hume-tada itself is installed
# --no-deps in the setup script because it pins torch>=2.7,<2.8.
# descript-audio-codec is NOT installed — it pulls onnx/tensorboard
# via descript-audiotools. A lightweight shim in utils/dac_shim.py
# provides the only class TADA uses: Snake1d.)
torchaudio
# Kokoro TTS (lightweight 82M-param engine)
kokoro>=0.9.4
misaki[en,ja,zh]>=0.9.4
# spacy model for misaki English G2P — must be pre-installed or misaki
# tries spacy.cli.download() at runtime which crashes frozen builds
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
# Audio processing
librosa>=0.10.0
soundfile>=0.12.0
numpy>=1.24.0
numpy>=1.24.0,<2.0
numba>=0.60.0,<0.61.0
pedalboard>=0.9.0
+7 -9
View File
@@ -1,12 +1,10 @@
"""Audio file serving endpoints."""
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session
from .. import models
from .. import config, models
from ..services import history
from ..database import get_db
@@ -22,8 +20,8 @@ async def get_version_audio(version_id: str, db: Session = Depends(get_db)):
if not version:
raise HTTPException(status_code=404, detail="Version not found")
audio_path = Path(version.audio_path)
if not audio_path.exists():
audio_path = config.resolve_storage_path(version.audio_path)
if audio_path is None or not audio_path.exists():
raise HTTPException(status_code=404, detail="Audio file not found")
return FileResponse(
@@ -40,8 +38,8 @@ async def get_audio(generation_id: str, db: Session = Depends(get_db)):
if not generation:
raise HTTPException(status_code=404, detail="Generation not found")
audio_path = Path(generation.audio_path)
if not audio_path.exists():
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is None or not audio_path.exists():
raise HTTPException(status_code=404, detail="Audio file not found")
return FileResponse(
@@ -60,8 +58,8 @@ async def get_sample_audio(sample_id: str, db: Session = Depends(get_db)):
if not sample:
raise HTTPException(status_code=404, detail="Sample not found")
audio_path = Path(sample.audio_path)
if not audio_path.exists():
audio_path = config.resolve_storage_path(sample.audio_path)
if audio_path is None or not audio_path.exists():
raise HTTPException(status_code=404, detail="Audio file not found")
return FileResponse(
+7 -6
View File
@@ -3,7 +3,6 @@
import asyncio
import io
import uuid
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import StreamingResponse
@@ -41,10 +40,11 @@ async def preview_effects(
all_versions = versions_mod.list_versions(generation_id, db)
clean_version = next((v for v in all_versions if v.effects_chain is None), None)
source_path = clean_version.audio_path if clean_version else gen.audio_path
if not source_path or not Path(source_path).exists():
resolved_source_path = config.resolve_storage_path(source_path)
if resolved_source_path is None or not resolved_source_path.exists():
raise HTTPException(status_code=404, detail="Source audio file not found")
audio, sample_rate = await asyncio.to_thread(load_audio, source_path)
audio, sample_rate = await asyncio.to_thread(load_audio, str(resolved_source_path))
processed = await asyncio.to_thread(apply_effects, audio, sample_rate, chain_dicts)
import soundfile as sf
@@ -193,10 +193,11 @@ async def apply_effects_to_generation(
source_path = clean_version.audio_path
source_version_id = clean_version.id
if not source_path or not Path(source_path).exists():
resolved_source_path = config.resolve_storage_path(source_path)
if resolved_source_path is None or not resolved_source_path.exists():
raise HTTPException(status_code=404, detail="Source audio file not found")
audio, sample_rate = await asyncio.to_thread(load_audio, source_path)
audio, sample_rate = await asyncio.to_thread(load_audio, str(resolved_source_path))
processed_audio = await asyncio.to_thread(apply_effects, audio, sample_rate, chain_dicts)
version_id = str(uuid.uuid4())
@@ -208,7 +209,7 @@ async def apply_effects_to_generation(
version = versions_mod.create_version(
generation_id=generation_id,
label=label,
audio_path=str(processed_path),
audio_path=config.to_storage_path(processed_path),
db=db,
effects_chain=chain_dicts,
is_default=data.set_as_default,
+31 -2
View File
@@ -20,6 +20,10 @@ from ..utils.tasks import get_task_manager
router = APIRouter()
def _resolve_generation_engine(data: models.GenerationRequest, profile) -> str:
return data.engine or getattr(profile, "default_engine", None) or getattr(profile, "preset_engine", None) or "qwen"
@router.post("/generate", response_model=models.GenerationResponse)
async def generate_speech(
data: models.GenerationRequest,
@@ -35,7 +39,12 @@ async def generate_speech(
from ..backends import engine_has_model_sizes
engine = data.engine or "qwen"
engine = _resolve_generation_engine(data, profile)
try:
profiles.validate_profile_engine(profile, engine)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
model_size = (data.model_size or "1.7B") if engine_has_model_sizes(engine) else None
generation = await history.create_generation(
@@ -230,7 +239,11 @@ async def stream_speech(
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
engine = data.engine or "qwen"
engine = _resolve_generation_engine(data, profile)
try:
profiles.validate_profile_engine(profile, engine)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
tts_model = get_tts_backend_for_engine(engine)
model_size = data.model_size or "1.7B"
@@ -263,6 +276,22 @@ async def stream_speech(
trim_fn=trim_fn,
)
effects_chain_config = None
if data.effects_chain is not None:
effects_chain_config = [e.model_dump() for e in data.effects_chain]
elif profile.effects_chain:
import json as _json
try:
effects_chain_config = _json.loads(profile.effects_chain)
except Exception:
effects_chain_config = None
if effects_chain_config:
from ..utils.effects import apply_effects
audio = apply_effects(audio, sample_rate, effects_chain_config)
if data.normalize:
from ..utils.audio import normalize_audio
+16 -1
View File
@@ -93,6 +93,12 @@ async def health():
except ImportError:
pass
gpu_compat_warning = None
if has_cuda:
from ..backends.base import check_cuda_compatibility
_compatible, gpu_compat_warning = check_cuda_compatibility()
gpu_available = has_cuda or has_mps or has_xpu or has_directml or backend_type == "mlx"
gpu_type = None
@@ -110,6 +116,11 @@ async def health():
vram_used = None
if has_cuda:
vram_used = torch.cuda.memory_allocated() / 1024 / 1024
elif has_xpu:
try:
vram_used = torch.xpu.memory_allocated() / 1024 / 1024
except Exception:
pass # memory_allocated() may not be available on all IPEX versions
model_loaded = False
model_size = None
@@ -162,7 +173,11 @@ async def health():
gpu_type=gpu_type,
vram_used_mb=vram_used,
backend_type=backend_type,
backend_variant=os.environ.get("VOICEBOX_BACKEND_VARIANT", "cuda" if torch.cuda.is_available() else "cpu"),
backend_variant=os.environ.get(
"VOICEBOX_BACKEND_VARIANT",
"cuda" if torch.cuda.is_available() else ("xpu" if has_xpu else "cpu"),
),
gpu_compatibility_warning=gpu_compat_warning,
)
+15 -4
View File
@@ -1,13 +1,12 @@
"""Generation history endpoints."""
import io
from pathlib import Path
from fastapi import APIRouter, Depends, File, HTTPException, UploadFile
from fastapi.responses import FileResponse, StreamingResponse
from sqlalchemy.orm import Session
from .. import models
from .. import config, models
from ..services import export_import, history
from ..app import safe_content_disposition
from ..database import Generation as DBGeneration, VoiceProfile as DBVoiceProfile, get_db
@@ -63,6 +62,13 @@ async def import_generation(
raise HTTPException(status_code=500, detail=str(e))
@router.delete("/history/failed")
async def clear_failed_generations(db: Session = Depends(get_db)):
"""Delete every generation with status='failed'. Used by the UI's 'Clear failed' button (#410)."""
count = await history.delete_failed_generations(db)
return {"deleted": count}
@router.get("/history/{generation_id}", response_model=models.HistoryResponse)
async def get_generation(
generation_id: str,
@@ -90,6 +96,11 @@ async def get_generation(
duration=gen.duration,
seed=gen.seed,
instruct=gen.instruct,
engine=gen.engine or "qwen",
model_size=gen.model_size,
status=gen.status or "completed",
error=gen.error,
is_favorited=bool(gen.is_favorited),
created_at=gen.created_at,
)
@@ -162,8 +173,8 @@ async def export_generation_audio(
if not generation.audio_path:
raise HTTPException(status_code=404, detail="Generation has no audio file")
audio_path = Path(generation.audio_path)
if not audio_path.is_file():
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is None or not audio_path.is_file():
raise HTTPException(status_code=404, detail="Audio file not found")
safe_text = "".join(c for c in generation.text[:30] if c.isalnum() or c in (" ", "-", "_")).strip()
+46 -4
View File
@@ -1,6 +1,8 @@
"""Voice profile endpoints."""
import io
import json as _json
import logging
import tempfile
from datetime import datetime
from pathlib import Path
@@ -15,6 +17,8 @@ from ..database import VoiceProfile as DBVoiceProfile, get_db
from ..services import channels, export_import, profiles
from ..services.profiles import _profile_to_response
logger = logging.getLogger(__name__)
router = APIRouter()
@@ -62,6 +66,46 @@ async def import_profile(
raise HTTPException(status_code=500, detail=str(e))
# ── Preset Voice Endpoints ───────────────────────────────────────────
# These MUST be declared before /profiles/{profile_id} to avoid the
# wildcard swallowing "presets" as a profile_id.
@router.get("/profiles/presets/{engine}")
async def list_preset_voices(engine: str):
"""List available preset voices for an engine."""
if engine == "kokoro":
from ..backends.kokoro_backend import KOKORO_VOICES
return {
"engine": engine,
"voices": [
{
"voice_id": vid,
"name": name,
"gender": gender,
"language": lang,
}
for vid, name, gender, lang in KOKORO_VOICES
],
}
if engine == "qwen_custom_voice":
from ..backends.qwen_custom_voice_backend import QWEN_CUSTOM_VOICES
return {
"engine": engine,
"voices": [
{
"voice_id": speaker_id,
"name": display_name,
"gender": gender,
"language": lang,
}
for speaker_id, display_name, gender, lang, _desc in QWEN_CUSTOM_VOICES
],
}
return {"engine": engine, "voices": []}
@router.get("/profiles/{profile_id}", response_model=models.VoiceProfileResponse)
async def get_profile(
profile_id: str,
@@ -215,8 +259,8 @@ async def get_profile_avatar(
if not profile.avatar_path:
raise HTTPException(status_code=404, detail="No avatar found for this profile")
avatar_path = Path(profile.avatar_path)
if not avatar_path.exists():
avatar_path = config.resolve_storage_path(profile.avatar_path)
if avatar_path is None or not avatar_path.exists():
raise HTTPException(status_code=404, detail="Avatar file not found")
return FileResponse(avatar_path)
@@ -297,8 +341,6 @@ async def update_profile_effects(
db: Session = Depends(get_db),
):
"""Set or clear the default effects chain for a voice profile."""
import json as _json
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
+30
View File
@@ -105,6 +105,11 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
This is the clean shutdown mechanism: instead of the Tauri app trying to
forcefully kill the server (which spawns console windows on Windows),
the server monitors its parent and shuts itself down gracefully.
The Tauri app writes a .keep-running sentinel file to data_dir before
exiting when "remain running after close" is enabled. This is a reliable
fallback for the HTTP /watchdog/disable request, which can race with
process exit on Windows.
"""
import os
import signal
@@ -164,6 +169,19 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
if not alive:
watchdog_logger.warning(f"Parent PID {parent_pid} not found on first check — disabling watchdog")
return
# Clear any stale .keep-running sentinel from a previous session. The
# sentinel is only removed by the watchdog when it's consumed during a
# grace period; if the HTTP /watchdog/disable path wins the race on a
# "keep running" exit, the sentinel is left on disk. Wipe it here so a
# future session can't inherit that stale signal.
if data_dir:
stale = os.path.join(data_dir, ".keep-running")
if os.path.exists(stale):
try:
os.remove(stale)
watchdog_logger.info("Removed stale .keep-running sentinel from previous session")
except OSError as e:
watchdog_logger.warning(f"Failed to remove stale sentinel: {e}")
while True:
if _watchdog_disabled:
watchdog_logger.info("Watchdog disabled (keep server running), stopping monitor")
@@ -178,6 +196,18 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
if _watchdog_disabled:
watchdog_logger.info("Watchdog was disabled during grace period, keeping server alive")
return
# Check for sentinel file written by Tauri before exit.
# This catches the case where the HTTP disable request
# didn't arrive before the parent process died (common
# on Windows where process teardown is fast).
sentinel = os.path.join(data_dir, ".keep-running") if data_dir else None
if sentinel and os.path.exists(sentinel):
watchdog_logger.info("Found .keep-running sentinel file, keeping server alive")
try:
os.remove(sentinel)
except OSError:
pass
return
watchdog_logger.info("Watchdog still enabled after grace period, shutting down server...")
if sys.platform == "win32":
# sys.exit triggers SystemExit, allowing uvicorn to run
+282 -119
View File
@@ -1,16 +1,23 @@
"""
CUDA backend binary download, assembly, and verification.
CUDA backend download, assembly, and verification.
Downloads split parts of the CUDA-enabled voicebox-server binary from
GitHub Releases, reassembles them, verifies integrity via SHA-256,
and places the binary in the app's data directory for use on next
backend restart.
Downloads two archives from GitHub Releases:
1. Server core (voicebox-server-cuda.tar.gz) — the exe + non-NVIDIA deps,
versioned with the app.
2. CUDA libs (cuda-libs-{version}.tar.gz) — NVIDIA runtime libraries,
versioned independently (only redownloaded on CUDA toolkit bump).
Both archives are extracted into {data_dir}/backends/cuda/ which forms the
complete PyInstaller --onedir directory structure that torch expects.
"""
import asyncio
import hashlib
import json
import logging
import os
import sys
import tarfile
from pathlib import Path
from typing import Optional
@@ -24,6 +31,16 @@ GITHUB_RELEASES_URL = "https://github.com/jamiepine/voicebox/releases/download"
PROGRESS_KEY = "cuda-backend"
# The current expected CUDA libs version. Bump this when we change the
# CUDA toolkit version or torch's CUDA dependency changes (e.g. cu126 -> cu128).
CUDA_LIBS_VERSION = "cu128-v1"
# Prevents concurrent download_cuda_binary() calls from racing on the same
# temp file. The auto-update background task and the manual HTTP endpoint
# can both invoke download_cuda_binary(); without this lock the progress-
# manager status check is a TOCTOU race.
_download_lock = asyncio.Lock()
def get_backends_dir() -> Path:
"""Directory where downloaded backend binaries are stored."""
@@ -32,21 +49,46 @@ def get_backends_dir() -> Path:
return d
def get_cuda_binary_name() -> str:
"""Platform-specific CUDA binary filename."""
def get_cuda_dir() -> Path:
"""Directory where the CUDA backend (onedir) is extracted."""
d = get_backends_dir() / "cuda"
d.mkdir(parents=True, exist_ok=True)
return d
def get_cuda_exe_name() -> str:
"""Platform-specific CUDA executable filename."""
if sys.platform == "win32":
return "voicebox-server-cuda.exe"
return "voicebox-server-cuda"
def get_cuda_binary_path() -> Optional[Path]:
"""Return path to CUDA binary if it exists."""
p = get_backends_dir() / get_cuda_binary_name()
"""Return path to the CUDA executable if it exists inside the onedir."""
p = get_cuda_dir() / get_cuda_exe_name()
if p.exists():
return p
return None
def get_cuda_libs_manifest_path() -> Path:
"""Path to the cuda-libs.json manifest inside the CUDA dir."""
return get_cuda_dir() / "cuda-libs.json"
def get_installed_cuda_libs_version() -> Optional[str]:
"""Read the installed CUDA libs version from cuda-libs.json, or None."""
manifest_path = get_cuda_libs_manifest_path()
if not manifest_path.exists():
return None
try:
data = json.loads(manifest_path.read_text())
return data.get("version")
except Exception as e:
logger.warning(f"Could not read cuda-libs.json: {e}")
return None
def is_cuda_active() -> bool:
"""Check if the current process is the CUDA binary.
@@ -60,140 +102,252 @@ def get_cuda_status() -> dict:
progress_manager = get_progress_manager()
cuda_path = get_cuda_binary_path()
progress = progress_manager.get_progress(PROGRESS_KEY)
cuda_libs_version = get_installed_cuda_libs_version()
return {
"available": cuda_path is not None,
"active": is_cuda_active(),
"binary_path": str(cuda_path) if cuda_path else None,
"cuda_libs_version": cuda_libs_version,
"downloading": progress is not None and progress.get("status") == "downloading",
"download_progress": progress,
}
async def download_cuda_binary(version: Optional[str] = None):
"""Download the CUDA backend binary from GitHub Releases.
def _needs_server_download(version: Optional[str] = None) -> bool:
"""Check if the server core archive needs to be (re)downloaded."""
cuda_path = get_cuda_binary_path()
if not cuda_path:
return True
# Check if the binary version matches the expected app version
installed = get_cuda_binary_version()
expected = version or __version__
if expected.startswith("v"):
expected = expected[1:]
return installed != expected
Downloads split parts listed in a manifest file, concatenates them,
and verifies the SHA-256 checksum for integrity. Atomic write
(temp file -> rename).
def _needs_cuda_libs_download() -> bool:
"""Check if the CUDA libs archive needs to be (re)downloaded."""
installed = get_installed_cuda_libs_version()
if installed is None:
return True
return installed != CUDA_LIBS_VERSION
async def _download_and_extract_archive(
client,
url: str,
sha256_url: Optional[str],
dest_dir: Path,
label: str,
progress_offset: int,
total_size: int,
):
"""Download a .tar.gz archive and extract it into dest_dir.
Args:
version: Version tag (e.g. "v0.2.0"). Defaults to current app version.
client: httpx.AsyncClient
url: URL of the .tar.gz archive
sha256_url: URL of the .sha256 checksum file (optional)
dest_dir: Directory to extract into
label: Human-readable label for progress updates
progress_offset: Byte offset for progress reporting (when downloading
multiple archives sequentially)
total_size: Total bytes across all downloads (for progress bar)
"""
progress = get_progress_manager()
temp_path = dest_dir / f".download-{label.replace(' ', '-')}.tmp"
# Clean up leftover partial download
if temp_path.exists():
temp_path.unlink()
# Fetch expected checksum (fail-fast: never extract an unverified archive)
expected_sha = None
if sha256_url:
try:
sha_resp = await client.get(sha256_url)
sha_resp.raise_for_status()
expected_sha = sha_resp.text.strip().split()[0]
logger.info(f"{label}: expected SHA-256: {expected_sha[:16]}...")
except Exception as e:
raise RuntimeError(f"{label}: failed to fetch checksum from {sha256_url}") from e
# Stream download, verify, and extract — always clean up temp file
downloaded = 0
try:
async with client.stream("GET", url) as response:
response.raise_for_status()
with open(temp_path, "wb") as f:
async for chunk in response.aiter_bytes(chunk_size=1024 * 1024):
f.write(chunk)
downloaded += len(chunk)
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Downloading {label}",
status="downloading",
)
# Verify integrity
if expected_sha:
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Verifying {label}...",
status="downloading",
)
sha256 = hashlib.sha256()
with open(temp_path, "rb") as f:
while True:
data = f.read(1024 * 1024)
if not data:
break
sha256.update(data)
actual = sha256.hexdigest()
if actual != expected_sha:
raise ValueError(
f"{label} integrity check failed: expected {expected_sha[:16]}..., got {actual[:16]}..."
)
logger.info(f"{label}: integrity verified")
# Extract (use data filter for path traversal protection on Python 3.12+)
progress.update_progress(
PROGRESS_KEY,
current=progress_offset + downloaded,
total=total_size,
filename=f"Extracting {label}...",
status="downloading",
)
with tarfile.open(temp_path, "r:gz") as tar:
if sys.version_info >= (3, 12):
tar.extractall(path=dest_dir, filter="data")
else:
tar.extractall(path=dest_dir)
logger.info(f"{label}: extracted to {dest_dir}")
finally:
if temp_path.exists():
temp_path.unlink()
return downloaded
async def download_cuda_binary(version: Optional[str] = None):
"""Download the CUDA backend (server core + CUDA libs if needed).
Downloads both archives from GitHub Releases, extracts them into
{data_dir}/backends/cuda/, and writes the cuda-libs.json manifest.
Only downloads what's needed:
- Server core: always redownloaded (versioned with app)
- CUDA libs: only if missing or version mismatch
Args:
version: Version tag (e.g. "v0.3.0"). Defaults to current app version.
"""
if _download_lock.locked():
logger.info("CUDA download already in progress, skipping duplicate request")
return
async with _download_lock:
await _download_cuda_binary_locked(version)
async def _download_cuda_binary_locked(version: Optional[str] = None):
"""Inner implementation of download_cuda_binary, called under _download_lock."""
import httpx
if version is None:
version = f"v{__version__}"
progress = get_progress_manager()
binary_name = get_cuda_binary_name()
dest_dir = get_backends_dir()
final_path = dest_dir / binary_name
temp_path = dest_dir / f"{binary_name}.download"
cuda_dir = get_cuda_dir()
# Clean up any leftover partial download
if temp_path.exists():
temp_path.unlink()
need_server = _needs_server_download(version)
need_libs = _needs_cuda_libs_download()
logger.info(f"Starting CUDA backend download for {version}")
if not need_server and not need_libs:
logger.info("CUDA backend is up to date, nothing to download")
return
logger.info(
f"Starting CUDA backend download for {version} "
f"(server={'yes' if need_server else 'cached'}, "
f"libs={'yes' if need_libs else 'cached'})"
)
progress.update_progress(
PROGRESS_KEY, current=0, total=0,
filename="Fetching manifest...", status="downloading",
PROGRESS_KEY,
current=0,
total=0,
filename="Preparing download...",
status="downloading",
)
base_url = f"{GITHUB_RELEASES_URL}/{version}"
stem = Path(binary_name).stem # voicebox-server-cuda
server_archive = "voicebox-server-cuda.tar.gz"
libs_archive = f"cuda-libs-{CUDA_LIBS_VERSION}.tar.gz"
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=30.0) as client:
# Fetch the manifest (list of split part filenames)
manifest_url = f"{base_url}/{stem}.manifest"
manifest_resp = await client.get(manifest_url)
manifest_resp.raise_for_status()
parts = [p.strip() for p in manifest_resp.text.strip().splitlines() if p.strip()]
if not parts:
raise ValueError("Empty manifest — no split parts found")
logger.info(f"Found {len(parts)} split parts to download")
# Fetch expected checksum (optional — for integrity verification)
expected_sha = None
try:
sha_url = f"{base_url}/{stem}.sha256"
sha_resp = await client.get(sha_url)
if sha_resp.status_code == 200:
# Format: "sha256hex filename\n"
expected_sha = sha_resp.text.strip().split()[0]
logger.info(f"Expected SHA-256: {expected_sha[:16]}...")
except Exception as e:
logger.warning(f"Could not fetch checksum file — skipping verification: {e}")
# Get total size across all parts by issuing HEAD requests
# Estimate total download size
total_size = 0
for part_name in parts:
if need_server:
try:
head_resp = await client.head(f"{base_url}/{part_name}")
content_length = int(head_resp.headers.get("content-length", 0))
total_size += content_length
head = await client.head(f"{base_url}/{server_archive}")
total_size += int(head.headers.get("content-length", 0))
except Exception:
pass
if need_libs:
try:
head = await client.head(f"{base_url}/{libs_archive}")
total_size += int(head.headers.get("content-length", 0))
except Exception:
pass
logger.info(f"Total download size: {total_size / 1024 / 1024:.1f} MB")
# Download and concatenate parts
total_downloaded = 0
with open(temp_path, "wb") as f:
for i, part_name in enumerate(parts):
part_url = f"{base_url}/{part_name}"
logger.info(f"Downloading part {i + 1}/{len(parts)}: {part_name}")
offset = 0
async with client.stream("GET", part_url) as response:
response.raise_for_status()
async for chunk in response.aiter_bytes(chunk_size=1024 * 1024):
f.write(chunk)
total_downloaded += len(chunk)
progress.update_progress(
PROGRESS_KEY, current=total_downloaded, total=total_size,
filename=f"Downloading CUDA backend ({i + 1}/{len(parts)})",
status="downloading",
)
# Verify integrity if checksum was available
if expected_sha:
progress.update_progress(
PROGRESS_KEY, current=total_downloaded, total=total_downloaded,
filename="Verifying integrity...", status="downloading",
)
sha256 = hashlib.sha256()
with open(temp_path, "rb") as f:
while True:
chunk = f.read(1024 * 1024)
if not chunk:
break
sha256.update(chunk)
actual = sha256.hexdigest()
if actual != expected_sha:
raise ValueError(
f"Integrity check failed: expected {expected_sha[:16]}..., "
f"got {actual[:16]}..."
# Download server core
if need_server:
server_downloaded = await _download_and_extract_archive(
client,
url=f"{base_url}/{server_archive}",
sha256_url=f"{base_url}/{server_archive}.sha256",
dest_dir=cuda_dir,
label="CUDA server",
progress_offset=offset,
total_size=total_size,
)
logger.info(f"Integrity verified: {actual[:16]}...")
offset += server_downloaded
# Atomic move into place (replace handles existing target on all platforms)
temp_path.replace(final_path)
# Make executable on Unix
exe_path = cuda_dir / get_cuda_exe_name()
if sys.platform != "win32" and exe_path.exists():
exe_path.chmod(0o755)
# Make executable on Unix
if sys.platform != "win32":
final_path.chmod(0o755)
# Download CUDA libs
if need_libs:
await _download_and_extract_archive(
client,
url=f"{base_url}/{libs_archive}",
sha256_url=f"{base_url}/{libs_archive}.sha256",
dest_dir=cuda_dir,
label="CUDA libraries",
progress_offset=offset,
total_size=total_size,
)
logger.info(f"CUDA backend downloaded to {final_path}")
# Write local cuda-libs.json manifest
manifest = {"version": CUDA_LIBS_VERSION}
get_cuda_libs_manifest_path().write_text(json.dumps(manifest, indent=2) + "\n")
logger.info(f"CUDA backend ready at {cuda_dir}")
progress.mark_complete(PROGRESS_KEY)
except Exception as e:
# Clean up on failure
if temp_path.exists():
temp_path.unlink()
logger.error(f"CUDA backend download failed: {e}")
progress.mark_error(PROGRESS_KEY, str(e))
raise
@@ -202,15 +356,19 @@ async def download_cuda_binary(version: Optional[str] = None):
def get_cuda_binary_version() -> Optional[str]:
"""Get the version of the installed CUDA binary, or None if not installed."""
import subprocess
cuda_path = get_cuda_binary_path()
if not cuda_path:
return None
try:
result = subprocess.run(
[str(cuda_path), "--version"],
capture_output=True, text=True, timeout=30,
capture_output=True,
text=True,
timeout=30,
cwd=str(cuda_path.parent), # Run from the onedir directory
)
# Output format: "voicebox-server 0.2.0"
# Output format: "voicebox-server 0.3.0"
for line in result.stdout.strip().splitlines():
if "voicebox-server" in line:
return line.split()[-1]
@@ -222,26 +380,29 @@ def get_cuda_binary_version() -> Optional[str]:
async def check_and_update_cuda_binary():
"""Check if the CUDA binary is outdated and auto-download if so.
Called on server startup. If a CUDA binary exists but its version
doesn't match the current app version, triggers a background download
of the updated CUDA binary. The download progress is visible to the
frontend via the existing SSE progress endpoint.
Called on server startup. Checks both server version and CUDA libs
version. Downloads only what's needed.
"""
cuda_path = get_cuda_binary_path()
if not cuda_path:
return # No CUDA binary installed, nothing to update
cuda_version = get_cuda_binary_version()
current_version = __version__
need_server = _needs_server_download()
need_libs = _needs_cuda_libs_download()
if cuda_version == current_version:
logger.info(f"CUDA binary is up to date (v{current_version})")
if not need_server and not need_libs:
logger.info(f"CUDA binary is up to date (server=v{__version__}, libs={get_installed_cuda_libs_version()})")
return
logger.info(
f"CUDA binary version mismatch: binary=v{cuda_version}, app=v{current_version}. "
f"Auto-downloading updated CUDA backend..."
)
reasons = []
if need_server:
cuda_version = get_cuda_binary_version()
reasons.append(f"server v{cuda_version} != v{__version__}")
if need_libs:
installed_libs = get_installed_cuda_libs_version()
reasons.append(f"libs {installed_libs} != {CUDA_LIBS_VERSION}")
logger.info(f"CUDA backend needs update ({', '.join(reasons)}). Auto-downloading...")
try:
await download_cuda_binary()
@@ -250,10 +411,12 @@ async def check_and_update_cuda_binary():
async def delete_cuda_binary() -> bool:
"""Delete the downloaded CUDA binary. Returns True if deleted."""
path = get_cuda_binary_path()
if path and path.exists():
path.unlink()
logger.info(f"Deleted CUDA binary: {path}")
"""Delete the downloaded CUDA backend directory. Returns True if deleted."""
import shutil
cuda_dir = get_cuda_dir()
if cuda_dir.exists() and any(cuda_dir.iterdir()):
shutil.rmtree(cuda_dir)
logger.info(f"Deleted CUDA backend directory: {cuda_dir}")
return True
return False
+2 -2
View File
@@ -11,6 +11,8 @@ from typing import List, Optional
from sqlalchemy.orm import Session
from sqlalchemy.exc import IntegrityError
from ..utils.effects import validate_effects_chain
from ..database import EffectPreset as DBEffectPreset
from ..models import EffectPresetResponse, EffectPresetCreate, EffectPresetUpdate, EffectConfig
@@ -52,7 +54,6 @@ def get_preset_by_name(name: str, db: Session) -> Optional[EffectPresetResponse]
def create_preset(data: EffectPresetCreate, db: Session) -> EffectPresetResponse:
"""Create a new user effect preset."""
from .utils.effects import validate_effects_chain
chain_dicts = [e.model_dump() for e in data.effects_chain]
error = validate_effects_chain(chain_dicts)
@@ -94,7 +95,6 @@ def update_preset(preset_id: str, data: EffectPresetUpdate, db: Session) -> Opti
if data.description is not None:
preset.description = data.description
if data.effects_chain is not None:
from .utils.effects import validate_effects_chain
chain_dicts = [e.model_dump() for e in data.effects_chain]
error = validate_effects_chain(chain_dicts)
+11 -9
View File
@@ -73,8 +73,8 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
# Check if profile has avatar
has_avatar = False
if profile.avatar_path:
avatar_path = Path(profile.avatar_path)
if avatar_path.exists():
avatar_path = config.resolve_storage_path(profile.avatar_path)
if avatar_path is not None and avatar_path.exists():
has_avatar = True
# Add avatar to ZIP root with original extension
avatar_ext = avatar_path.suffix
@@ -98,7 +98,9 @@ def export_profile_to_zip(profile_id: str, db: Session) -> bytes:
for sample in samples:
# Get filename from audio_path (should be {sample_id}.wav)
audio_path = Path(sample.audio_path)
audio_path = config.resolve_storage_path(sample.audio_path)
if audio_path is None:
raise ValueError(f"Audio file not found: {sample.audio_path}")
filename = audio_path.name
# Read audio file
@@ -279,7 +281,7 @@ def export_generation_to_zip(generation_id: str, db: Session) -> bytes:
# Build version manifest entries
version_entries = []
for v in versions:
v_path = Path(v.audio_path)
v_path = config.resolve_storage_path(v.audio_path)
effects_chain = None
if v.effects_chain:
effects_chain = json.loads(v.effects_chain)
@@ -314,14 +316,14 @@ def export_generation_to_zip(generation_id: str, db: Session) -> bytes:
# Add all version audio files
for v in versions:
v_path = Path(v.audio_path)
if v_path.exists():
v_path = config.resolve_storage_path(v.audio_path)
if v_path is not None and v_path.exists():
zip_file.write(v_path, f"audio/{v_path.name}")
# Fallback: if no versions exist, include the generation's main audio
if not versions:
audio_path = Path(generation.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is not None and audio_path.exists():
zip_file.write(audio_path, f"audio/{audio_path.name}")
zip_buffer.seek(0)
@@ -426,7 +428,7 @@ async def import_generation_from_zip(file_bytes: bytes, db: Session) -> dict:
profile_id=profile_id,
text=generation_data["text"],
language=generation_data["language"],
audio_path=str(audio_dest),
audio_path=config.to_storage_path(audio_dest),
duration=generation_data["duration"],
seed=generation_data.get("seed"),
instruct=generation_data.get("instruct"),
+8 -6
View File
@@ -163,7 +163,7 @@ def _save_generate(
versions_mod.create_version(
generation_id=generation_id,
label="original",
audio_path=str(clean_audio_path),
audio_path=config.to_storage_path(clean_audio_path),
db=db,
effects_chain=None,
is_default=not has_effects,
@@ -174,6 +174,8 @@ def _save_generate(
if has_effects:
from ..utils.effects import apply_effects, validate_effects_chain
assert effects_chain is not None
error_msg = validate_effects_chain(effects_chain)
if error_msg:
import logging
@@ -189,13 +191,13 @@ def _save_generate(
versions_mod.create_version(
generation_id=generation_id,
label="version-2",
audio_path=str(processed_path),
audio_path=config.to_storage_path(processed_path),
db=db,
effects_chain=effects_chain,
is_default=True,
)
return final_audio_path
return config.to_storage_path(final_audio_path)
def _save_retry(
@@ -211,7 +213,7 @@ def _save_retry(
"""
audio_path = config.get_generations_dir() / f"{generation_id}.wav"
save_audio(audio, str(audio_path), sample_rate)
return str(audio_path)
return config.to_storage_path(audio_path)
def _save_regenerate(
@@ -244,10 +246,10 @@ def _save_regenerate(
versions_mod.create_version(
generation_id=generation_id,
label=label,
audio_path=str(audio_path),
audio_path=config.to_storage_path(audio_path),
db=db,
effects_chain=None,
is_default=True,
)
return str(audio_path)
return config.to_storage_path(audio_path)
+41 -4
View File
@@ -253,8 +253,8 @@ async def delete_generation(
# Delete main audio file (if not already removed by version cleanup)
if generation.audio_path:
audio_path = Path(generation.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is not None and audio_path.exists():
audio_path.unlink()
# Delete from database
@@ -264,6 +264,43 @@ async def delete_generation(
return True
async def delete_failed_generations(db: Session) -> int:
"""
Delete every generation whose status is 'failed'.
Used by the "Clear failed" action in the UI so users can tidy up
history after the model wasn't loaded, the app was closed mid-run,
or a generation otherwise errored out (see issue #410).
Returns:
Number of generations deleted.
"""
from . import versions as versions_mod
failed = db.query(DBGeneration).filter(DBGeneration.status == "failed").all()
count = 0
for generation in failed:
# Clean up version files/rows first.
versions_mod.delete_versions_for_generation(generation.id, db)
# Remove the main audio file if it somehow made it to disk.
if generation.audio_path:
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is not None and audio_path.exists():
try:
audio_path.unlink()
except OSError:
# Best-effort cleanup — don't abort the whole sweep
# if a single file can't be removed.
pass
db.delete(generation)
count += 1
db.commit()
return count
async def delete_generations_by_profile(
profile_id: str,
db: Session,
@@ -283,8 +320,8 @@ async def delete_generations_by_profile(
count = 0
for generation in generations:
# Delete audio file
audio_path = Path(generation.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is not None and audio_path.exists():
audio_path.unlink()
# Delete from database
+219 -59
View File
@@ -1,33 +1,30 @@
"""
Voice profile management module.
"""
"""Voice profile management module."""
from typing import List, Optional
from datetime import datetime
import uuid
import json as _json
import logging
import shutil
import uuid
from datetime import datetime
from pathlib import Path
from sqlalchemy.orm import Session
from sqlalchemy import func, select
from sqlalchemy import func
from sqlalchemy.orm import Session
from .. import config
from ..database import Generation as DBGeneration, ProfileSample as DBProfileSample, VoiceProfile as DBVoiceProfile
from ..models import (
EffectConfig,
ProfileSampleResponse,
VoiceProfileCreate,
VoiceProfileResponse,
ProfileSampleCreate,
ProfileSampleResponse,
)
from ..database import (
VoiceProfile as DBVoiceProfile,
ProfileSample as DBProfileSample,
Generation as DBGeneration,
)
from ..models import EffectConfig
from ..utils.audio import validate_reference_audio, validate_and_load_reference_audio, load_audio, save_audio
from ..utils.images import validate_image, process_avatar
from ..utils.audio import save_audio, validate_and_load_reference_audio
from ..utils.cache import _get_cache_dir, clear_profile_cache
from .tts import get_tts_model
from .. import config
import json as _json
from ..utils.images import process_avatar, validate_image
logger = logging.getLogger(__name__)
CLONING_ENGINES = {"qwen", "luxtts", "chatterbox", "chatterbox_turbo", "tada"}
def _profile_to_response(
@@ -52,6 +49,11 @@ def _profile_to_response(
language=profile.language,
avatar_path=profile.avatar_path,
effects_chain=effects_chain,
voice_type=getattr(profile, "voice_type", None) or "cloned",
preset_engine=getattr(profile, "preset_engine", None),
preset_voice_id=getattr(profile, "preset_voice_id", None),
design_prompt=getattr(profile, "design_prompt", None),
default_engine=getattr(profile, "default_engine", None),
generation_count=generation_count,
sample_count=sample_count,
created_at=profile.created_at,
@@ -59,6 +61,79 @@ def _profile_to_response(
)
def _get_preset_voice_ids(engine: str) -> set[str]:
if engine == "kokoro":
from ..backends.kokoro_backend import KOKORO_VOICES
return {voice_id for voice_id, _name, _gender, _lang in KOKORO_VOICES}
if engine == "qwen_custom_voice":
from ..backends.qwen_custom_voice_backend import QWEN_CUSTOM_VOICES
return {voice_id for voice_id, _name, _gender, _lang, _desc in QWEN_CUSTOM_VOICES}
return set()
def _validate_profile_fields(
*,
voice_type: str,
preset_engine: str | None,
preset_voice_id: str | None,
design_prompt: str | None,
default_engine: str | None,
) -> str | None:
if voice_type == "preset":
if not preset_engine or not preset_voice_id:
return "Preset profiles require both preset_engine and preset_voice_id"
if default_engine and default_engine != preset_engine:
return "Preset profiles must use their preset_engine as default_engine"
available_voice_ids = _get_preset_voice_ids(preset_engine)
if available_voice_ids and preset_voice_id not in available_voice_ids:
return f"Preset voice '{preset_voice_id}' is not valid for engine '{preset_engine}'"
return None
if voice_type == "designed":
if not design_prompt or not design_prompt.strip():
return "Designed profiles require a design_prompt"
if preset_engine or preset_voice_id:
return "Designed profiles cannot set preset_engine or preset_voice_id"
return None
if preset_engine or preset_voice_id:
return "Cloned profiles cannot set preset_engine or preset_voice_id"
if design_prompt:
return "Cloned profiles cannot set design_prompt"
if default_engine and default_engine not in CLONING_ENGINES:
return f"Cloned profiles cannot use default engine '{default_engine}'"
return None
def validate_profile_engine(profile, engine: str) -> None:
voice_type = getattr(profile, "voice_type", None) or "cloned"
if voice_type == "preset":
preset_engine = getattr(profile, "preset_engine", None)
preset_voice_id = getattr(profile, "preset_voice_id", None)
if not preset_engine or not preset_voice_id:
raise ValueError(f"Preset profile {profile.id} is missing preset engine metadata")
if preset_engine != engine:
raise ValueError(
f"Preset profile {profile.id} only supports engine '{preset_engine}', not '{engine}'"
)
return
if voice_type == "designed":
design_prompt = getattr(profile, "design_prompt", None)
if not design_prompt or not design_prompt.strip():
raise ValueError(f"Designed profile {profile.id} is missing design_prompt")
return
if engine not in CLONING_ENGINES:
raise ValueError(f"Engine '{engine}' does not support cloned voice profiles")
async def create_profile(
data: VoiceProfileCreate,
db: Session,
@@ -80,11 +155,32 @@ async def create_profile(
if existing_profile:
raise ValueError(f"A profile with the name '{data.name}' already exists. Please choose a different name.")
# Auto-set default_engine for preset profiles
default_engine = data.default_engine
voice_type = data.voice_type or "cloned"
if voice_type == "preset" and data.preset_engine and not default_engine:
default_engine = data.preset_engine
validation_error = _validate_profile_fields(
voice_type=voice_type,
preset_engine=data.preset_engine,
preset_voice_id=data.preset_voice_id,
design_prompt=data.design_prompt,
default_engine=default_engine,
)
if validation_error:
raise ValueError(validation_error)
db_profile = DBVoiceProfile(
id=str(uuid.uuid4()),
name=data.name,
description=data.description,
language=data.language,
voice_type=voice_type,
preset_engine=data.preset_engine,
preset_voice_id=data.preset_voice_id,
design_prompt=data.design_prompt,
default_engine=default_engine,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
)
@@ -140,7 +236,7 @@ async def add_profile_sample(
db_sample = DBProfileSample(
id=sample_id,
profile_id=profile_id,
audio_path=str(dest_path),
audio_path=config.to_storage_path(dest_path),
reference_text=reference_text,
)
@@ -161,7 +257,7 @@ async def add_profile_sample(
async def get_profile(
profile_id: str,
db: Session,
) -> Optional[VoiceProfileResponse]:
) -> VoiceProfileResponse | None:
"""
Get a voice profile by ID.
@@ -182,7 +278,7 @@ async def get_profile(
async def get_profile_samples(
profile_id: str,
db: Session,
) -> List[ProfileSampleResponse]:
) -> list[ProfileSampleResponse]:
"""
Get all samples for a profile.
@@ -197,7 +293,7 @@ async def get_profile_samples(
return [ProfileSampleResponse.model_validate(s) for s in samples]
async def list_profiles(db: Session) -> List[VoiceProfileResponse]:
async def list_profiles(db: Session) -> list[VoiceProfileResponse]:
"""
List all voice profiles with generation and sample counts.
@@ -238,7 +334,7 @@ async def update_profile(
profile_id: str,
data: VoiceProfileCreate,
db: Session,
) -> Optional[VoiceProfileResponse]:
) -> VoiceProfileResponse | None:
"""
Update a voice profile.
@@ -262,9 +358,27 @@ async def update_profile(
if existing_profile:
raise ValueError(f"A profile with the name '{data.name}' already exists. Please choose a different name.")
voice_type = getattr(profile, "voice_type", None) or "cloned"
preset_engine = getattr(profile, "preset_engine", None)
preset_voice_id = getattr(profile, "preset_voice_id", None)
design_prompt = getattr(profile, "design_prompt", None)
default_engine = data.default_engine if data.default_engine is not None else getattr(profile, "default_engine", None)
validation_error = _validate_profile_fields(
voice_type=voice_type,
preset_engine=preset_engine,
preset_voice_id=preset_voice_id,
design_prompt=design_prompt,
default_engine=default_engine,
)
if validation_error:
raise ValueError(validation_error)
profile.name = data.name
profile.description = data.description
profile.language = data.language
if data.default_engine is not None:
profile.default_engine = data.default_engine or None # empty string → NULL
profile.updated_at = datetime.utcnow()
db.commit()
@@ -327,8 +441,8 @@ async def delete_profile_sample(
# Store profile_id before deleting
profile_id = sample.profile_id
audio_path = Path(sample.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(sample.audio_path)
if audio_path is not None and audio_path.exists():
audio_path.unlink()
db.delete(sample)
@@ -345,7 +459,7 @@ async def update_profile_sample(
sample_id: str,
reference_text: str,
db: Session,
) -> Optional[ProfileSampleResponse]:
) -> ProfileSampleResponse | None:
"""
Update a profile sample's reference text.
@@ -382,19 +496,57 @@ async def create_voice_prompt_for_profile(
engine: str = "qwen",
) -> dict:
"""
Create a combined voice prompt from all samples in a profile.
Create a voice prompt from a profile.
For cloned profiles: combines all audio samples into a voice prompt.
For preset profiles: returns the engine-specific preset voice reference.
For designed profiles: returns the text design prompt (future).
Args:
profile_id: Profile ID
db: Database session
use_cache: Whether to use cached prompts
engine: TTS engine to create prompt for ("qwen" or "luxtts")
engine: TTS engine to create prompt for
Returns:
Voice prompt dictionary
"""
from ..backends import get_tts_backend_for_engine
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile:
raise ValueError(f"Profile not found: {profile_id}")
voice_type = getattr(profile, "voice_type", None) or "cloned"
validate_profile_engine(profile, engine)
# ── Preset profiles: return engine-specific voice reference ──
if voice_type == "preset":
if not profile.preset_engine or not profile.preset_voice_id:
raise ValueError(f"Preset profile {profile_id} is missing preset engine metadata")
if profile.preset_engine != engine:
raise ValueError(
f"Preset profile {profile_id} only supports engine '{profile.preset_engine}', not '{engine}'"
)
return {
"voice_type": "preset",
"preset_engine": profile.preset_engine,
"preset_voice_id": profile.preset_voice_id,
}
# ── Designed profiles: return text description (future) ──
if voice_type == "designed":
if not profile.design_prompt or not profile.design_prompt.strip():
raise ValueError(f"Designed profile {profile_id} is missing design_prompt")
return {
"voice_type": "designed",
"design_prompt": profile.design_prompt,
}
if engine not in CLONING_ENGINES:
raise ValueError(f"Engine '{engine}' does not support cloned voice profiles")
# ── Cloned profiles: create from audio samples ──
samples = db.query(DBProfileSample).filter_by(profile_id=profile_id).all()
if not samples:
@@ -404,40 +556,48 @@ async def create_voice_prompt_for_profile(
if len(samples) == 1:
sample = samples[0]
sample_audio_path = config.resolve_storage_path(sample.audio_path)
if sample_audio_path is None:
raise ValueError(f"Sample audio not found for profile {profile_id}")
voice_prompt, _ = await tts_model.create_voice_prompt(
sample.audio_path,
str(sample_audio_path),
sample.reference_text,
use_cache=use_cache,
)
return voice_prompt
else:
audio_paths = [s.audio_path for s in samples]
reference_texts = [s.reference_text for s in samples]
combined_audio, combined_text = await tts_model.combine_voice_prompts(
audio_paths,
reference_texts,
)
audio_paths = []
for sample in samples:
sample_audio_path = config.resolve_storage_path(sample.audio_path)
if sample_audio_path is None:
raise ValueError(f"Sample audio not found for profile {profile_id}")
audio_paths.append(str(sample_audio_path))
reference_texts = [s.reference_text for s in samples]
# Save combined audio to cache directory (persistent)
# Create a hash of sample IDs to identify this specific combination
import hashlib
combined_audio, combined_text = await tts_model.combine_voice_prompts(
audio_paths,
reference_texts,
)
sample_ids_str = "-".join(sorted([s.id for s in samples]))
combination_hash = hashlib.md5(sample_ids_str.encode()).hexdigest()[:12]
# Save combined audio to cache directory (persistent)
# Create a hash of sample IDs to identify this specific combination
import hashlib
cache_dir = _get_cache_dir()
cache_dir.mkdir(parents=True, exist_ok=True)
combined_path = cache_dir / f"combined_{profile_id}_{combination_hash}.wav"
sample_ids_str = "-".join(sorted([s.id for s in samples]))
combination_hash = hashlib.md5(sample_ids_str.encode()).hexdigest()[:12]
save_audio(combined_audio, str(combined_path), 24000)
cache_dir = _get_cache_dir()
cache_dir.mkdir(parents=True, exist_ok=True)
combined_path = cache_dir / f"combined_{profile_id}_{combination_hash}.wav"
voice_prompt, _ = await tts_model.create_voice_prompt(
str(combined_path),
combined_text,
use_cache=use_cache,
)
return voice_prompt
save_audio(combined_audio, str(combined_path), 24000)
voice_prompt, _ = await tts_model.create_voice_prompt(
str(combined_path),
combined_text,
use_cache=use_cache,
)
return voice_prompt
async def upload_avatar(
@@ -465,8 +625,8 @@ async def upload_avatar(
raise ValueError(error_msg)
if profile.avatar_path:
old_avatar = Path(profile.avatar_path)
if old_avatar.exists():
old_avatar = config.resolve_storage_path(profile.avatar_path)
if old_avatar is not None and old_avatar.exists():
old_avatar.unlink()
# Determine file extension from uploaded file
@@ -487,7 +647,7 @@ async def upload_avatar(
process_avatar(image_path, str(output_path))
profile.avatar_path = str(output_path)
profile.avatar_path = config.to_storage_path(output_path)
profile.updated_at = datetime.utcnow()
db.commit()
@@ -514,8 +674,8 @@ async def delete_avatar(
if not profile or not profile.avatar_path:
return False
avatar_path = Path(profile.avatar_path)
if avatar_path.exists():
avatar_path = config.resolve_storage_path(profile.avatar_path)
if avatar_path is not None and avatar_path.exists():
avatar_path.unlink()
profile.avatar_path = None
+6 -3
View File
@@ -10,6 +10,7 @@ from pathlib import Path
from sqlalchemy.orm import Session
from sqlalchemy import func
from .. import config
from ..models import (
StoryCreate,
StoryResponse,
@@ -483,13 +484,15 @@ async def split_story_item(
Returns:
List of two updated item details (original and new) or None if not found/invalid
"""
# Get the item
# Get the item with a row lock to prevent concurrent splits on the
# same clip (e.g. from rapid double-clicks racing each other).
item = (
db.query(DBStoryItem)
.filter_by(
id=item_id,
story_id=story_id,
)
.with_for_update()
.first()
)
if not item:
@@ -826,8 +829,8 @@ async def export_story_audio(
if version:
resolved_audio_path = version.audio_path
audio_path = Path(resolved_audio_path)
if not audio_path.exists():
audio_path = config.resolve_storage_path(resolved_audio_path)
if audio_path is None or not audio_path.exists():
continue
try:
+4 -4
View File
@@ -158,8 +158,8 @@ def delete_version(version_id: str, db: Session) -> bool:
gen_id = version.generation_id
# Delete audio file
audio_path = Path(version.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(version.audio_path)
if audio_path is not None and audio_path.exists():
audio_path.unlink()
db.delete(version)
@@ -193,8 +193,8 @@ def delete_versions_for_generation(generation_id: str, db: Session) -> int:
)
count = 0
for v in versions:
audio_path = Path(v.audio_path)
if audio_path.exists():
audio_path = config.resolve_storage_path(v.audio_path)
if audio_path is not None and audio_path.exists():
audio_path.unlink()
db.delete(v)
count += 1
+1 -1
View File
@@ -64,7 +64,7 @@ def get_cached_voice_prompt(
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
if cache_file.exists():
try:
prompt = torch.load(cache_file)
prompt = torch.load(cache_file, weights_only=True)
_memory_cache[cache_key] = prompt
return prompt
except Exception:
+95
View File
@@ -0,0 +1,95 @@
"""
Minimal shim for descript-audio-codec (DAC).
TADA only imports Snake1d from dac.nn.layers and dac.model.dac.
The real DAC package pulls in descript-audiotools which depends on
onnx, tensorboard, protobuf, matplotlib, pystoi, etc. — none of
which are needed for TADA's runtime use of Snake1d.
This shim provides the exact Snake1d implementation (MIT-licensed,
from https://github.com/descriptinc/descript-audio-codec) so we can
avoid the entire audiotools dependency chain.
If the real DAC package is installed, this module is never used —
Python's import system will find the site-packages version first.
Install this shim only when descript-audio-codec is NOT installed.
"""
import sys
import types
import torch
import torch.nn as nn
# ── Snake activation (from dac/nn/layers.py) ────────────────────────
# NOTE: The original DAC code uses @torch.jit.script here for a 1.4x
# speedup. We omit it because TorchScript calls inspect.getsource()
# which fails inside a PyInstaller frozen binary (no .py source files).
def snake(x: torch.Tensor, alpha: torch.Tensor) -> torch.Tensor:
shape = x.shape
x = x.reshape(shape[0], shape[1], -1)
x = x + (alpha + 1e-9).reciprocal() * torch.sin(alpha * x).pow(2)
x = x.reshape(shape)
return x
class Snake1d(nn.Module):
def __init__(self, channels: int):
super().__init__()
self.alpha = nn.Parameter(torch.ones(1, channels, 1))
def forward(self, x: torch.Tensor) -> torch.Tensor:
return snake(x, self.alpha)
# ── Register as dac.nn.layers and dac.model.dac ─────────────────────
def install_dac_shim() -> None:
"""Register fake dac package modules in sys.modules.
Only installs the shim if 'dac' is not already importable
(i.e. the real descript-audio-codec is not installed).
"""
try:
import dac # noqa: F401 — real package exists, do nothing
return
except ImportError:
pass
# Create the module tree: dac -> dac.nn -> dac.nn.layers
# -> dac.model -> dac.model.dac
dac_pkg = types.ModuleType("dac")
dac_pkg.__path__ = [] # make it a package
dac_pkg.__package__ = "dac"
dac_nn = types.ModuleType("dac.nn")
dac_nn.__path__ = []
dac_nn.__package__ = "dac.nn"
dac_nn_layers = types.ModuleType("dac.nn.layers")
dac_nn_layers.__package__ = "dac.nn"
dac_nn_layers.Snake1d = Snake1d
dac_nn_layers.snake = snake
dac_model = types.ModuleType("dac.model")
dac_model.__path__ = []
dac_model.__package__ = "dac.model"
dac_model_dac = types.ModuleType("dac.model.dac")
dac_model_dac.__package__ = "dac.model"
dac_model_dac.Snake1d = Snake1d
# Wire up submodules
dac_pkg.nn = dac_nn
dac_pkg.model = dac_model
dac_nn.layers = dac_nn_layers
dac_model.dac = dac_model_dac
# Register in sys.modules
sys.modules["dac"] = dac_pkg
sys.modules["dac.nn"] = dac_nn
sys.modules["dac.nn.layers"] = dac_nn_layers
sys.modules["dac.model"] = dac_model
sys.modules["dac.model.dac"] = dac_model_dac
+49 -2
View File
@@ -1,17 +1,64 @@
"""Monkey-patch huggingface_hub to force offline mode with cached models.
Prevents mlx_audio from making network requests when models are already
downloaded. Must be imported BEFORE mlx_audio.
Prevents mlx_audio / transformers from making network requests when models
are already downloaded. Must be imported BEFORE mlx_audio.
"""
import logging
import os
from contextlib import contextmanager
from pathlib import Path
from typing import Optional, Union
logger = logging.getLogger(__name__)
@contextmanager
def force_offline_if_cached(is_cached: bool, model_label: str = ""):
"""Context manager that sets ``HF_HUB_OFFLINE=1`` while loading a cached model.
If *is_cached* is ``False`` the block runs normally (network allowed).
If the offline load raises an error containing "offline" we automatically
retry with network access so a partially-cached model still works.
Args:
is_cached: Whether the model weights are already on disk.
model_label: Human-readable name used in log messages.
"""
if not is_cached:
yield
return
original_value = os.environ.get("HF_HUB_OFFLINE")
os.environ["HF_HUB_OFFLINE"] = "1"
logger.info(
"[offline-guard] %s is cached — forcing HF_HUB_OFFLINE=1",
model_label or "model",
)
try:
yield
except Exception as exc:
if "offline" in str(exc).lower():
logger.warning(
"[offline-guard] Offline load failed for %s, retrying with network: %s",
model_label or "model",
exc,
)
# Restore original env and retry — caller must wrap the load
# inside force_offline_if_cached so retrying here isn't possible.
# Instead, propagate a flag via the exception so the caller can
# decide. For simplicity we just let it fall through to the
# finally block and re-raise.
raise
raise
finally:
if original_value is not None:
os.environ["HF_HUB_OFFLINE"] = original_value
else:
os.environ.pop("HF_HUB_OFFLINE", None)
def patch_huggingface_hub_offline():
"""Monkey-patch huggingface_hub to force offline mode."""
try:
+13 -4
View File
@@ -1,13 +1,11 @@
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.hooks import collect_submodules
from PyInstaller.utils.hooks import collect_all
from PyInstaller.utils.hooks import copy_metadata
datas = []
binaries = []
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += collect_data_files('qwen_tts')
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.backends.qwen_custom_voice_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'kokoro', 'kokoro.pipeline', 'kokoro.model', 'kokoro.istftnet', 'kokoro.modules', 'kokoro.custom_stft', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += copy_metadata('qwen-tts')
datas += copy_metadata('requests')
datas += copy_metadata('transformers')
@@ -15,8 +13,9 @@ datas += copy_metadata('huggingface-hub')
datas += copy_metadata('tokenizers')
datas += copy_metadata('safetensors')
datas += copy_metadata('tqdm')
hiddenimports += collect_submodules('qwen_tts')
datas += copy_metadata('en_core_web_sm')
hiddenimports += collect_submodules('jaraco')
hiddenimports += collect_submodules('tada')
hiddenimports += collect_submodules('mlx')
hiddenimports += collect_submodules('mlx_audio')
tmp_ret = collect_all('zipvoice')
@@ -27,12 +26,22 @@ tmp_ret = collect_all('lazy_loader')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('librosa')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('qwen_tts')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('inflect')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('perth')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('piper_phonemize')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('misaki')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('language_tags')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('espeakng_loader')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('en_core_web_sm')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('mlx')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('mlx_audio')
+1
View File
@@ -22,6 +22,7 @@ services:
environment:
- LOG_LEVEL=info
- NUMBA_CACHE_DIR=/tmp/numba_cache
networks:
- voicebox-net
+4 -4
View File
@@ -1,7 +1,7 @@
@import 'tailwindcss';
@import 'fumadocs-ui/css/neutral.css';
@import 'fumadocs-ui/css/preset.css';
@import 'fumadocs-openapi/css/preset.css';
@import "tailwindcss";
@import "fumadocs-ui/css/neutral.css";
@import "fumadocs-ui/css/preset.css";
@import "fumadocs-openapi/css/preset.css";
:root {
--color-fd-primary: hsl(43, 50%, 50%);
+2 -11
View File
@@ -5,22 +5,13 @@ import { generate as DefaultImage } from 'fumadocs-ui/og';
export const revalidate = false;
export async function GET(
_req: Request,
{ params }: RouteContext<'/og/docs/[...slug]'>,
) {
export async function GET(_req: Request, { params }: RouteContext<'/og/docs/[...slug]'>) {
const { slug } = await params;
const page = source.getPage(slug.slice(0, -1));
if (!page) notFound();
return new ImageResponse(
(
<DefaultImage
title={page.data.title}
description={page.data.description}
site="My App"
/>
),
<DefaultImage title={page.data.title} description={page.data.description} site="My App" />,
{
width: 1200,
height: 630,
+7 -5
View File
@@ -159,12 +159,14 @@ Tauri looks for `voicebox-server-${PLATFORM}` in `src-tauri/binaries/` and bundl
The `build-cuda-windows` job runs separately:
1. Install PyTorch with CUDA 12.1
2. Build with `build_binary.py --cuda`
3. Split binary with `scripts/split_binary.py`
4. Upload parts as release artifacts
1. Install PyTorch with CUDA 12.8
2. Build with `build_binary.py --cuda` (produces `--onedir` output)
3. Package with `scripts/package_cuda.py` into two archives:
- `voicebox-server-cuda.tar.gz` — server core (~945 MB)
- `cuda-libs-cu128-v1.tar.gz` — NVIDIA runtime libraries (~1.7 GB, cached independently)
4. Upload archives as release artifacts
This binary is downloaded on-demand by users who enable CUDA in settings.
This binary is downloaded on-demand by users who enable CUDA in settings. The CUDA libs archive is only re-downloaded when the CUDA toolkit version changes, not on every app update.
## Troubleshooting
+468 -29
View File
@@ -3,8 +3,12 @@ title: "TTS Engines"
description: "How to add new text-to-speech engines to Voicebox"
---
> **For humans:** This doc is optimized for AI agents to implement new TTS engines autonomously. It's structured as a phased workflow with explicit gates and a checklist so an agent can do the full integration — dependency research, backend, frontend, bundling — and hand you a draft release or prod build to test locally. It's also a useful reference if you're doing it yourself.
Adding an engine touches ~10 files across 4 layers. The backend protocol work is straightforward — the real time sink is dependency hell, upstream library bugs, and PyInstaller bundling.
**Do not start writing code until you complete Phase 0.** The v0.2.3 release was three patch releases of PyInstaller fixes because dependency research was skipped. Every issue — `inspect.getsource()` failures, missing native data files, metadata lookups, dtype mismatches — was discoverable by reading the model library's source code before integration began.
## Architecture Overview
The backend is split into layers:
@@ -18,6 +22,133 @@ The backend is split into layers:
New engines only need to touch `backends/` and `models.py` on the backend side — the route and service layers use a model config registry that handles dispatch automatically.
## Phase 0: Dependency Research
**This phase is mandatory.** Clone the model library and its key dependencies into a temporary directory and inspect them before writing any integration code. The goal is to produce a dependency audit that identifies every PyInstaller-incompatible pattern, every native data file, and every upstream bug you'll need to work around.
### 0.1 Clone and Inspect the Model Library
```bash
# Create a throwaway workspace
mkdir /tmp/engine-research && cd /tmp/engine-research
# Clone the model library
git clone https://github.com/org/model-library.git
cd model-library
```
**Read these files first, in order:**
1. **`setup.py` / `setup.cfg` / `pyproject.toml`** — Check pinned dependency versions. If the library pins `torch==2.6.0` or `numpy<1.26`, you'll need `--no-deps` installation and manual sub-dependency listing (this is what happened with `chatterbox-tts`).
2. **`__init__.py` and the main model class** — Trace the import chain. Look for:
- `from_pretrained()` — does it call `huggingface_hub` internally? Does it pass `token=True` (which crashes without a stored HF token)?
- `from_local()` — does it exist? You may need manual `snapshot_download()` + `from_local()` to bypass download bugs.
- Device handling — does it default to CUDA? Does it support MPS? Many libraries crash on MPS with unsupported operators.
3. **All `import` statements** — Recursively trace what the library imports. You're looking for:
- `inspect.getsource()` anywhere in the chain (search all `.py` files)
- `typeguard` / `@typechecked` decorators (these call `inspect.getsource()` at import time)
- `importlib.metadata.version()` or `pkg_resources.get_distribution()` (need `--copy-metadata`)
- `lazy_loader` (needs `--collect-all` to bundle `.pyi` stubs)
### 0.2 Scan for PyInstaller-Incompatible Patterns
Run these searches against the cloned library **and** its transitive dependencies:
```bash
# inspect.getsource — will crash in frozen binary without --collect-all
grep -r "inspect.getsource\|getsource(" .
# typeguard / @typechecked — calls inspect.getsource at import time
grep -r "@typechecked\|from typeguard" .
# importlib.metadata — needs --copy-metadata
grep -r "importlib.metadata\|pkg_resources.get_distribution\|pkg_resources.require" .
# Data files loaded at runtime — need --collect-all or --collect-data
grep -r "Path(__file__).parent\|os.path.dirname(__file__)\|resources_path\|pkg_resources.resource_filename" .
# Native library paths — may need env var override in frozen builds
grep -r "/usr/share\|/usr/lib\|/usr/local\|espeak\|phonemize" .
# torch.load without map_location — will crash on CPU-only builds
grep -r "torch.load(" . | grep -v "map_location"
# HuggingFace token bugs
grep -r 'token=True\|token=os.getenv' .
# Float64/Float32 assumptions — librosa returns float64, many models assume float32
grep -r "torch.from_numpy\|\.double()\|float64" .
# @torch.jit.script — calls inspect.getsource(), crashes in frozen builds
grep -r "@torch.jit.script\|torch.jit.script" .
# torchaudio.load — requires torchcodec in torchaudio 2.10+, use soundfile.read() instead
grep -r "torchaudio.load\|torchaudio.save" .
# Gated HuggingFace repos — models that hardcode gated repos as tokenizer/config sources
grep -r "from_pretrained\|tokenizer_name\|AutoTokenizer" . | grep -i "llama\|meta-llama\|gated"
```
### 0.3 Install and Trace in a Throwaway Venv
```bash
# Create isolated venv
python -m venv /tmp/engine-venv
source /tmp/engine-venv/bin/activate
# Install the package (try normally first)
pip install model-package
# Check if it conflicts with our stack
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
# If this fails, you need --no-deps:
pip install --no-deps model-package
# Get the full dependency tree
pip show model-package # Check Requires: field
pip show -f model-package # List all installed files (look for data files)
# Check for non-PyPI dependencies
pip install model-package 2>&1 | grep -i "no matching distribution"
```
### 0.4 Test Model Loading on CPU
Before writing any integration code, verify the model works on CPU in a plain Python script:
```python
import torch
# Force CPU to catch map_location bugs early
model = ModelClass.from_pretrained("org/model", device="cpu")
# Test with a float32 audio array (not float64)
import numpy as np
audio = np.random.randn(16000).astype(np.float32)
output = model.generate("Hello world", audio)
print(f"Output shape: {output.shape}, dtype: {output.dtype}, sample rate: {model.sample_rate}")
```
If this crashes, you've found a bug you'll need to monkey-patch. Common ones:
- `RuntimeError: expected scalar type Float but found Double` → needs float32 cast
- `RuntimeError: map_location` → needs `torch.load` patch
- `RuntimeError: Unsupported operator aten::...` → needs MPS skip
### 0.5 Produce a Dependency Audit
Before proceeding to Phase 1, write down:
1. **PyPI vs non-PyPI deps** — which packages need `--find-links`, `git+https://`, or `--no-deps`?
2. **PyInstaller directives needed** — which packages need `--collect-all`, `--copy-metadata`, `--hidden-import`?
3. **Runtime data files** — which packages ship data files (YAML, pretrained weights, phoneme tables, shader libraries) that must be bundled?
4. **Native library paths** — which packages look for data at system paths that won't exist in a frozen binary?
5. **Monkey-patches needed** — `torch.load` map_location, float64→float32 casts, MPS skip, HF token bypass, etc.
6. **Sample rate** — what does the engine output? (24kHz, 44.1kHz, 48kHz)
7. **Model download method** — `from_pretrained()` with library-managed download, or manual `snapshot_download()` + `from_local()`?
This audit becomes your implementation plan for Phases 1, 4, and 5.
## Phase 1: Backend Implementation
### 1.1 Create the Backend File
@@ -148,61 +279,210 @@ In `app/src/lib/hooks/useGenerationForm.ts`:
- Add engine-to-model-name mapping
- Update payload construction for engine-specific fields
**Watch out for model naming inconsistencies.** The HuggingFace repo name, the model size label, and the API model name don't always follow predictable patterns. For example, TADA's 3B model is named `tada-3b-ml` (not `tada-3b`), because it's a multilingual variant. Always check the actual repo names and build the frontend model name mapping from those, not from assumptions like `{engine}-{size}`.
### 3.5 Model Management
In `app/src/components/ServerSettings/ModelManagement.tsx`:
- Add description to `MODEL_DESCRIPTIONS` record
- Add model name to `voiceModels` filter condition
### 3.6 Non-Cloning Engines (Preset Voices)
If your engine uses **pre-built voices** instead of zero-shot cloning from reference audio (e.g. Kokoro), additional integration is needed:
**Backend:**
- In `kokoro_backend.py` (or your engine), define a `VOICES` list of `(voice_id, display_name, gender, language)` tuples
- `create_voice_prompt()` should return `{"voice_type": "preset", "preset_engine": "<engine>", "preset_voice_id": "<id>"}`
- `generate()` should read `voice_prompt.get("preset_voice_id")` to select the voice
- Add a `seed_preset_profiles("<engine>")` call in `backend/routes/models.py` after model download completes
- The `seed_preset_profiles()` function in `backend/services/profiles.py` creates DB profiles with `voice_type="preset"`
**Frontend:**
- The `EngineModelSelector` filters options based on `selectedProfile.voice_type`:
- `"cloned"` profiles → only cloning engines shown (Kokoro hidden)
- `"preset"` profiles → only the preset's engine shown
- Profile cards show the engine name as a badge for preset profiles
- When a preset profile is selected, the engine auto-switches
**Profile schema fields for presets:**
- `voice_type: "preset"` (vs `"cloned"` for traditional profiles)
- `preset_engine: "<engine>"` — which engine owns this voice
- `preset_voice_id: "<id>"` — the engine-specific voice identifier
**For future "designed" voices** (text description instead of audio, e.g. Qwen CustomVoice):
- Use `voice_type: "designed"` with `design_prompt` field
- `create_voice_prompt_for_profile()` already returns the design prompt for this type
## Phase 4: Dependencies
Use the dependency audit from Phase 0 to drive this phase. You should already know what packages are needed, which conflict, and which require special installation.
### 4.1 Python Dependencies
Add to `backend/requirements.txt`. Watch for:
Add to `backend/requirements.txt`. There are three installation patterns, depending on what Phase 0 revealed:
**Pinned dependency conflicts** — If the model package pins old versions, install with `--no-deps`:
**Normal PyPI packages:**
```
some-model-package>=1.0.0
```
**Pinned dependency conflicts (`--no-deps`)** — If the model package pins old versions of torch/numpy/transformers, install with `--no-deps` and list sub-dependencies manually. This is the pattern used for `chatterbox-tts`:
```bash
# In justfile / CI setup:
pip install --no-deps chatterbox-tts
# In requirements.txt — list each actual sub-dependency:
conformer>=0.3.2
diffusers>=0.31.0
omegaconf>=2.3.0
resemble-perth>=0.0.2
s3tokenizer>=0.1.6
```
Then list sub-dependencies manually in `requirements.txt`.
To identify sub-deps: `pip show chatterbox-tts` → `Requires:` field, then cross-reference against existing `requirements.txt` to avoid duplicates.
**Non-PyPI packages:**
```
linacodec @ git+https://github.com/user/repo.git
**Non-PyPI packages** — Some libraries only exist on GitHub or require custom indexes:
```
# Git-only packages (no PyPI release)
linacodec @ git+https://github.com/ysharma3501/LinaCodec.git
Zipvoice @ git+https://github.com/ysharma3501/LuxTTS.git
**Custom package indexes:**
```
# Custom package indexes (C extensions with platform-specific wheels)
--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
piper-phonemize>=1.2.0
```
### 4.2 Identifying Hidden Sub-Dependencies
### 4.2 Dependency Conflict Resolution
1. Install the package normally in a throwaway venv
2. Run `pip show <package>` to get its `Requires:` list
3. Cross-reference against existing requirements.txt
4. Test that the engine loads and generates
Check for conflicts with the existing stack before adding anything:
## Phase 5: PyInstaller Bundling
```bash
# Our current stack pins (approximate):
# Python 3.12+, torch>=2.10, transformers>=4.57, numpy>=1.26
This is where most of the pain lives. Common issues:
# Test compatibility
pip install model-package torch==2.10 transformers==4.57.3 numpy>=1.26
| Issue | Symptom | Fix |
|-------|---------|-----|
| `inspect.getsource()` at import | "could not get source code" | `--collect-all <package>` |
| Data files (yaml, .pth.tar) | FileNotFoundError at runtime | `--collect-all <package>` |
| Native data paths (espeak-ng) | Library looks at `/usr/share/...` | Set env var in frozen builds |
| `importlib.metadata` lookups | "No package metadata found" | `--copy-metadata <package>` |
| Dynamic imports | ModuleNotFoundError | `--hidden-import <module>` |
# If it fails, check what the package pins:
pip show model-package | grep Requires
# Look at setup.py/pyproject.toml for version constraints
```
### Testing Frozen Builds
**Known incompatible patterns in the wild:**
- `torch==2.6.0` — many older packages pin this
- `numpy<1.26` — conflicts with Python 3.12+
- `transformers==4.46.3` — many packages pin old transformers
- `onnxruntime` pinned versions — often conflict with torch
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary.
### 4.3 Update Installation Scripts
Dependencies must be added in multiple places:
| File | What to add |
|------|------------|
| `backend/requirements.txt` | Package and version constraint |
| `justfile` | `--no-deps` install line if needed (in `setup-python` and `setup-python-release` targets) |
| `.github/workflows/release.yml` | Same `--no-deps` line in CI build steps |
| `Dockerfile` | Same install commands for Docker builds |
## Phase 5: PyInstaller Bundling (`build_binary.py`)
This is where most of the pain lives. **The v0.2.3 release was entirely dedicated to fixing bundling issues** — every new engine that shipped in v0.2.1 (LuxTTS, Chatterbox, Chatterbox Turbo) worked in dev but failed in production builds. Don't skip this phase.
### 5.1 Register Your Engine in `build_binary.py`
Every new engine needs entries in `backend/build_binary.py`. This file drives PyInstaller and is the single most common source of "works in dev, breaks in prod" bugs. You need to decide which PyInstaller directives your engine's dependencies require:
| Directive | What It Does | When You Need It |
|-----------|-------------|-----------------|
| `--hidden-import <module>` | Includes a module PyInstaller can't detect via static analysis | Dynamic imports, lazy imports, plugin architectures |
| `--collect-all <package>` | Bundles source `.py` files, data files, AND native libraries | Packages that call `inspect.getsource()` at import time (e.g. `inflect` via `typeguard`'s `@typechecked`), or that ship pretrained model files (e.g. `perth` ships `.pth.tar` + `hparams.yaml`) |
| `--collect-data <package>` | Bundles only data files (not source or native libs) | Packages with YAML configs, vocab files, etc. |
| `--collect-submodules <package>` | Bundles all submodules | Packages with deep module trees that PyInstaller misses |
| `--copy-metadata <package>` | Copies `importlib.metadata` info | Packages that call `importlib.metadata.version()` or `pkg_resources.get_distribution()` at runtime. Already required for: `requests`, `transformers`, `huggingface-hub`, `tokenizers`, `safetensors`, `tqdm` |
**Example: adding hidden imports and collect-all for a new engine:**
```python
# In build_binary.py, inside the args list:
"--hidden-import",
"backend.backends.your_engine_backend",
"--hidden-import",
"your_engine_package",
"--hidden-import",
"your_engine_package.inference",
"--collect-all",
"some_dependency_that_uses_inspect_getsource",
"--copy-metadata",
"some_dependency_that_checks_its_own_version",
```
### 5.2 Lessons from v0.2.3 — Real Failures and Their Fixes
These are actual production failures from shipping new engines. Every one of these passed `python -m uvicorn` in dev:
| Engine | Failure | Root Cause | Fix |
|--------|---------|-----------|-----|
| LuxTTS | `"could not get source code"` on import | `inflect` uses `typeguard`'s `@typechecked` which calls `inspect.getsource()` — needs `.py` source files, not just bytecode | `--collect-all inflect` |
| LuxTTS | `espeak-ng-data` not found | `piper_phonemize` C library looks for data at `/usr/share/espeak-ng-data/` which doesn't exist in the bundle | `--collect-all piper_phonemize` + set `ESPEAK_DATA_PATH` env var at runtime (see 5.3) |
| LuxTTS | `inspect.getsource` error in Vocos codec | `linacodec` and `zipvoice` use source introspection | `--collect-all linacodec` + `--collect-all zipvoice` |
| Chatterbox | `FileNotFoundError` for watermark model | `perth` ships pretrained model files (`hparams.yaml`, `.pth.tar`) that PyInstaller doesn't bundle by default | `--collect-all perth` |
| All engines | `importlib.metadata` failures | Frozen binary doesn't include package metadata for `huggingface-hub`, `transformers`, etc. | `--copy-metadata` for each affected package |
| All engines | Download progress bars stuck at 0% | `huggingface_hub` silently disables tqdm progress bars based on logger level in frozen builds — our progress tracker never receives byte updates | Force-enable tqdm's internal counter in `HFProgressTracker` |
| TADA | `inspect.getsource` error in DAC's `Snake1d` | `@torch.jit.script` calls `inspect.getsource()` which fails without `.py` source files | Wrote a lightweight shim (`dac_shim.py`) reimplementing `Snake1d` without `@torch.jit.script`, registered fake `dac.*` modules in `sys.modules` |
| All engines | `NameError: name 'obj' is not defined` on macOS | Python 3.12.0 has a [CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects | Upgrade to Python 3.12.13+ |
| All engines | `resource_tracker` subprocess crash | `multiprocessing` in frozen binaries needs `freeze_support()` called before anything else | Added to `server.py` entry point |
### 5.3 Runtime Frozen-Build Handling (`server.py`)
Some fixes can't live in `build_binary.py` — they need runtime detection. The entry point `backend/server.py` handles these before any heavy imports:
```python
# 1. freeze_support() — MUST be called before any multiprocessing use
import multiprocessing
multiprocessing.freeze_support()
# 2. Native data paths — redirect C libraries to bundled data
if getattr(sys, 'frozen', False):
_meipass = getattr(sys, '_MEIPASS', os.path.dirname(sys.executable))
_espeak_data = os.path.join(_meipass, 'piper_phonemize', 'espeak-ng-data')
if os.path.isdir(_espeak_data):
os.environ.setdefault('ESPEAK_DATA_PATH', _espeak_data)
# 3. stdout/stderr safety — PyInstaller --noconsole on Windows sets these to None
if not _is_writable(sys.stdout):
sys.stdout = open(os.devnull, 'w')
```
If your engine's dependencies include native libraries that look for data at system paths (like espeak-ng does), you'll need to add a similar `os.environ.setdefault()` block here.
### 5.4 CUDA vs CPU Build Branching
`build_binary.py` produces two different binaries:
- **`voicebox-server`** (CPU) — excludes all `nvidia.*` packages to avoid bundling ~3 GB of CUDA DLLs
- **`voicebox-server-cuda`** — includes `torch.cuda` and `torch.backends.cudnn`
On Windows, if the build environment has CUDA torch installed but you're building the CPU binary, the script temporarily swaps to CPU-only torch and restores CUDA torch afterward. This prevents PyInstaller from accidentally bundling CUDA libraries into the CPU build.
New engine imports go in the **common section** (not the CUDA or MLX conditional blocks) unless your engine has platform-specific dependencies.
### 5.5 MLX Conditional Inclusion
Apple Silicon builds conditionally include MLX hidden imports and `--collect-all mlx` / `--collect-all mlx_audio`. If your engine has an MLX-specific backend variant, add its imports inside the `if is_apple_silicon() and not cuda:` block.
### 5.6 Testing Frozen Builds
You can't skip this. Models that work in `python -m uvicorn` will break in the PyInstaller binary. The v0.2.3 release required **three patch releases** (v0.2.1 → v0.2.2 → v0.2.3) to get all engines working in production.
1. Build: `just build`
2. Run and try download + load + generate
3. Check stderr for the actual error
4. Fix, rebuild, repeat
2. Launch the binary directly (not via `python -m`)
3. Test the **full chain**: download → load → generate → progress tracking
4. Check stderr for the actual error (logs go to stderr for Tauri sidecar capture)
5. Fix, rebuild, repeat
**Common gotcha:** testing only generation with a pre-cached model from your dev install. Always test with a clean model cache to verify downloads work too.
## Phase 6: Common Upstream Workarounds
@@ -240,6 +520,90 @@ def _get_device(self):
return "cpu" # Skip MPS
```
### Gated HuggingFace repos as hardcoded config sources
Some models hardcode a gated HuggingFace repo as their tokenizer or config source (e.g., TADA hardcodes `"meta-llama/Llama-3.2-1B"` in both its `AlignerConfig` and `TadaConfig`). This silently fails without HF authentication.
**Fix:** Download from an ungated mirror and patch the config objects directly:
```python
# Download tokenizer from ungated mirror
UNGATED_TOKENIZER = "unsloth/Llama-3.2-1B"
tokenizer_path = snapshot_download(UNGATED_TOKENIZER, token=None)
# Patch the model config to use the local path instead of the gated repo
config = ModelConfig.from_pretrained(model_path)
config.tokenizer_name = tokenizer_path
model = ModelClass.from_pretrained(model_path, config=config)
```
**Do NOT monkey-patch `AutoTokenizer.from_pretrained`** — it's a classmethod, and replacing it corrupts the descriptor, which breaks other engines that use different tokenizers (e.g., Qwen uses a Qwen tokenizer via `AutoTokenizer`). Always patch at the config level, not the class method level.
### `torchaudio.load()` requires `torchcodec` in 2.10+
As of `torchaudio>=2.10`, `torchaudio.load()` requires the `torchcodec` package for audio I/O. If your engine or backend code uses `torchaudio.load()`, replace it with `soundfile`:
```python
# Before (breaks without torchcodec):
import torchaudio
waveform, sr = torchaudio.load("audio.wav")
# After:
import soundfile as sf
import torch
data, sr = sf.read("audio.wav", dtype="float32")
waveform = torch.from_numpy(data).unsqueeze(0)
```
Note: `torchaudio.functional.resample()` and other pure-PyTorch math functions work fine without `torchcodec` — only the I/O functions are affected.
### `@torch.jit.script` breaks in frozen builds
`torch.jit.script` calls `inspect.getsource()` to parse the decorated function's source code. In a PyInstaller binary, `.py` source files aren't available, so this crashes at import time.
**Fix:** Remove or avoid `@torch.jit.script` decorators. If the decorated function comes from an upstream dependency, write a shim that reimplements the function without the decorator (see "Toxic dependency chains" below).
### Toxic dependency chains — the shim pattern
Sometimes a model library depends on a package with a massive, hostile transitive dependency tree, but only uses a tiny piece of it. When the dependency chain is unbuildable or would pull in dozens of unwanted packages, the right move is to write a lightweight shim.
**Example:** TADA depends on `descript-audio-codec` (DAC), which pulls in `descript-audiotools` -> `onnx`, `tensorboard`, `protobuf`, `matplotlib`, `pystoi`, etc. The `onnx` package fails to build from source on macOS. But TADA only uses `Snake1d` from DAC — a 7-line PyTorch module.
**Solution:** Create a shim at `backend/utils/dac_shim.py` that registers fake modules in `sys.modules`:
```python
import sys
import types
import torch
from torch import nn
def snake(x, alpha):
"""Snake activation — reimplemented without @torch.jit.script."""
return x + (1.0 / (alpha + 1e-9)) * torch.sin(alpha * x).pow(2)
class Snake1d(nn.Module):
def __init__(self, channels):
super().__init__()
self.alpha = nn.Parameter(torch.ones(1, channels, 1))
def forward(self, x):
return snake(x, self.alpha)
# Register fake dac.* modules so "from dac.nn.layers import Snake1d" works
_nn = types.ModuleType("dac.nn")
_layers = types.ModuleType("dac.nn.layers")
_layers.Snake1d = Snake1d
_nn.layers = _layers
for name, mod in [("dac", types.ModuleType("dac")),
("dac.nn", _nn), ("dac.nn.layers", _layers)]:
sys.modules[name] = mod
```
**Key rules for shims:**
- Import the shim **before** importing the model library (so it finds the fake modules first)
- Do NOT use `@torch.jit.script` in the shim (see above)
- Only reimplement what the model actually uses — check the import chain carefully
## Upcoming Engines
Based on the current model landscape, these are candidates for future integration:
@@ -250,8 +614,83 @@ Based on the current model landscape, these are candidates for future integratio
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
| **HumeAI TADA** | EN (1B), Multi (3B) | Medium | 700s+ coherent audio, synced transcripts | Needs vetting |
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
The multi-engine architecture is now in place, making new model integration straightforward (~1 day for a well-documented model with a PyPI package).
## Implementation Checklist
Use this as a gate between phases. Do not proceed to the next phase until every item in the current phase is checked.
### Phase 0: Dependency Research
- [ ] Cloned model library source into a temp directory
- [ ] Read `setup.py` / `pyproject.toml` — noted pinned dependency versions
- [ ] Traced all imports from the model class through to leaf dependencies
- [ ] Searched for `inspect.getsource`, `@typechecked`, `typeguard` in the full dependency tree
- [ ] Searched for `importlib.metadata`, `pkg_resources.get_distribution` in the dependency tree
- [ ] Searched for `Path(__file__).parent`, `os.path.dirname(__file__)`, hardcoded system paths
- [ ] Searched for `torch.load` calls missing `map_location`
- [ ] Searched for `torch.from_numpy` without `.float()` cast
- [ ] Searched for `token=True` or `token=os.getenv("HF_TOKEN")` in HuggingFace calls
- [ ] Searched for `@torch.jit.script` / `torch.jit.script` (crashes in frozen builds)
- [ ] Searched for `torchaudio.load` / `torchaudio.save` (requires `torchcodec` in 2.10+)
- [ ] Searched for hardcoded gated HuggingFace repo names (e.g., `meta-llama/*`)
- [ ] Evaluated whether any dependency is used minimally enough to shim instead of install
- [ ] Tested model loading and generation on CPU in a throwaway venv
- [ ] Tested with a clean HuggingFace cache (no pre-downloaded models)
- [ ] Produced a written dependency audit documenting all findings
### Phase 1: Backend Implementation
- [ ] Created `backend/backends/<engine>_backend.py` implementing `TTSBackend` protocol
- [ ] Chose voice prompt pattern (pre-computed tensors vs deferred file paths)
- [ ] Implemented all monkey-patches identified in Phase 0
- [ ] Used `get_torch_device()` from `backends/base.py` for device selection
- [ ] Used `model_load_progress()` from `backends/base.py` for download/load tracking
- [ ] Tested: model downloads correctly
- [ ] Tested: model loads on CPU
- [ ] Tested: generation produces valid audio
- [ ] Tested: voice cloning from reference audio works
- [ ] Registered `ModelConfig` in `backends/__init__.py`
- [ ] Added to `TTS_ENGINES` dict
- [ ] Added factory branch in `get_tts_backend_for_engine()`
- [ ] Updated engine regex in `backend/models.py`
### Phase 2–3: Route, Service, and Frontend
- [ ] Confirmed zero changes needed in routes/services (or documented why custom behavior is needed)
- [ ] Added engine to TypeScript union type in `app/src/lib/api/types.ts`
- [ ] Added language map entry in `app/src/lib/constants/languages.ts`
- [ ] Added to `ENGINE_OPTIONS` and `ENGINE_DESCRIPTIONS` in `EngineModelSelector.tsx`
- [ ] Added to Zod schema and model-name mapping in `useGenerationForm.ts`
- [ ] Added description in `ModelManagement.tsx`
### Phase 4: Dependencies
- [ ] Added packages to `backend/requirements.txt`
- [ ] If `--no-deps` needed: listed sub-dependencies explicitly
- [ ] If git-only packages: added `@ git+https://...` entries
- [ ] If custom index needed: added `--find-links` line
- [ ] Updated `justfile` setup targets
- [ ] Updated `.github/workflows/release.yml` build steps
- [ ] Updated `Dockerfile` if applicable
- [ ] Verified `pip install` succeeds in a clean venv with existing requirements
### Phase 5: PyInstaller Bundling
- [ ] Added `--hidden-import` entries in `build_binary.py` for:
- [ ] `backend.backends.<engine>_backend`
- [ ] The model package and its key submodules
- [ ] Added `--collect-all` for any packages that:
- [ ] Use `inspect.getsource()` / `@typechecked`
- [ ] Ship pretrained model data files (`.pth.tar`, `.yaml`, etc.)
- [ ] Ship native data files (phoneme tables, shader libraries, etc.)
- [ ] Added `--copy-metadata` for any packages that use `importlib.metadata`
- [ ] If engine has native data paths: added `os.environ.setdefault()` in `server.py`
- [ ] Built frozen binary with `just build`
- [ ] Tested in frozen binary with **clean model cache** (not pre-cached from dev):
- [ ] Model download works with real-time progress
- [ ] Model loading works
- [ ] Generation produces valid audio
- [ ] No errors in stderr logs
### Phase 6: Final Verification
- [ ] Engine works in dev mode (`just dev`)
- [ ] Engine works in frozen binary (`just build` → run binary directly)
- [ ] Tested on target platform (macOS for MLX, Windows/Linux for CUDA)
- [ ] No regressions in existing engines
+37 -10
View File
@@ -5,17 +5,22 @@ description: "How voice profile management works in Voicebox"
## Overview
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
Voice profiles are the unit of "a saved voice" in Voicebox. As of 0.4 they support two flavors backed by the same `profiles` table:
- **Cloned profiles** — store one or more reference audio samples; the cloning engine generates a voice embedding at use time
- **Preset profiles** — store no audio; just a pointer to an engine-specific pre-built voice (e.g. Kokoro's `am_adam`, Qwen CustomVoice's `Ryan`)
The schema also reserves a third type, `designed`, for future text-described voices. Not currently used by any shipped engine.
## Architecture
The voice profile system consists of three main components:
**Database Layer:** SQLite tables store profile metadata and sample references.
**Database Layer:** SQLite tables store profile metadata, sample references (cloned), and engine + voice ID (preset).
**File Storage:** Audio samples are stored on disk in a structured directory format.
**File Storage:** Audio samples are stored on disk in a structured directory format. Preset profiles have no on-disk audio.
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
**Profile Module:** `backend/services/profiles.py` provides the business logic for CRUD operations and dispatches to the appropriate engine based on `voice_type`.
## Data Model
@@ -24,27 +29,49 @@ The voice profile system consists of three main components:
```python
class VoiceProfile(Base):
__tablename__ = "profiles"
id = Column(String, primary_key=True)
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
name = Column(String, unique=True, nullable=False)
description = Column(Text)
language = Column(String, default="en")
created_at = Column(DateTime)
updated_at = Column(DateTime)
avatar_path = Column(String, nullable=True)
effects_chain = Column(Text, nullable=True)
# Voice type system — added v0.3.x
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
design_prompt = Column(Text, nullable=True) # text description — only for designed (reserved)
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
```
The `voice_type` column discriminates the three flavors:
| `voice_type` | `preset_engine` | `preset_voice_id` | Samples in `profile_samples` |
| ------------ | --------------- | ----------------- | ---------------------------- |
| `cloned` | NULL | NULL | Required (≥1 row) |
| `preset` | engine name | voice ID string | None |
| `designed` | NULL | NULL | None (uses `design_prompt`) |
The `default_engine` column is set automatically when the profile is created. For preset profiles it's locked to the source engine — switching engines at generation time will skip the profile (and the UI auto-switches back when the user clicks a greyed-out card; see the floating generate box and profile grid).
### ProfileSample Table
```python
class ProfileSample(Base):
__tablename__ = "profile_samples"
id = Column(String, primary_key=True)
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
profile_id = Column(String, ForeignKey("profiles.id"))
audio_path = Column(String, nullable=False)
reference_text = Column(Text, nullable=False)
```
Only populated for cloned profiles. Preset and designed profiles have zero rows in this table.
## File Structure
Profiles are stored in the data directory:
+5 -5
View File
@@ -3,12 +3,12 @@ title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -31,6 +31,6 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## Get Started
- [Installation](/docs/overview/installation) -- download and install Voicebox
- [Quick Start](/docs/overview/quick-start) -- get up and running in 5 minutes
- [API Reference](/docs/api-reference) -- integrate voice synthesis into your apps
- [Installation](/overview/installation) -- download and install Voicebox
- [Quick Start](/overview/quick-start) -- get up and running in 5 minutes
- [API Reference](/api-reference) -- integrate voice synthesis into your apps
@@ -1,32 +1,43 @@
---
title: "Creating Voice Profiles"
description: "Advanced guide to creating high-quality voice profiles"
description: "How to create voice profiles, both cloning-based and preset-based"
---
## Overview
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
A **voice profile** is a saved voice you can reuse across generations, stories, and the API. As of 0.4, Voicebox profiles come in two flavors that map to two different ways of getting a voice:
## Quick Start
| Profile type | What it stores | Use when… |
| -------------- | ---------------------------------------------------- | -------------------------------------------------------- |
| **Cloned** | One or more reference audio samples + a voice embedding | You want to replicate a specific person's voice |
| **Preset** | A reference to a pre-built voice in a specific engine | You want a curated, production-ready voice with no audio prep |
Both types live in the same Profiles tab and behave the same way at generation time — pick the type that matches your goal and follow the workflow below.
<Callout type="info">
Not sure which to use? Cloning gives you a *specific* voice but needs clean audio. Preset gives you *good* voices instantly but you don't get to choose who they sound like.
</Callout>
## Workflow A — Cloned Profiles
Use this when you want to replicate a specific person's voice from a recording.
<Steps>
<Step title="Prepare Audio">
10-30 seconds of clear speech
10-30 seconds of clear speech, minimal background noise. See [Voice Cloning](/overview/voice-cloning) for the engine catalog.
</Step>
<Step title="Create Profile">
**Profiles** → **+ New Profile**
**Profiles** → **+ New Profile** → choose a cloning engine (Qwen3-TTS, Chatterbox, LuxTTS, or TADA)
</Step>
<Step title="Upload Sample">
Add your audio file
<Step title="Upload or Record Sample">
Drag in an audio file, or record directly with the in-app recorder
</Step>
<Step title="Generate">
Use the profile to generate speech
<Step title="Generate to Test">
Use the profile to generate a test phrase. If quality is poor, add more samples
</Step>
</Steps>
## Audio Requirements
### Ideal Sample Characteristics
### Audio Requirements (Cloning Only)
<Cards>
<Card title="Duration">
@@ -44,7 +55,7 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
<Card title="Quality">
**High fidelity**
44.1kHz or 48kHz sample rate
44.1 kHz or 48 kHz sample rate
Minimal compression
</Card>
<Card title="Content">
@@ -58,18 +69,16 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
### File Formats
Supported formats:
- **WAV** (recommended) - Lossless quality
- **MP3** - Acceptable, minimal compression
- **M4A** - Acceptable
- **FLAC** - Lossless alternative
- **WAV** (recommended) — Lossless quality
- **MP3** — Acceptable, minimal compression
- **M4A** — Acceptable
- **FLAC** — Lossless alternative
<Callout type="info">
Use WAV for best results. Avoid heavily compressed formats.
</Callout>
## Recording Tips
### Environment
### Recording Tips
<AccordionGroup>
<Accordion title="Quiet Space">
@@ -87,27 +96,25 @@ Supported formats:
</Accordion>
<Accordion title="Recording Settings">
- 44.1kHz or 48kHz sample rate
- 44.1 kHz or 48 kHz sample rate
- 16-bit or 24-bit depth
- Mono is fine (stereo will be converted)
- Avoid automatic gain control
</Accordion>
</AccordionGroup>
### Speaking
### Speaking Style
- **Natural pace** - Don't rush or speak too slowly
- **Clear articulation** - Pronounce words clearly
- **Consistent volume** - Maintain steady loudness
- **Normal tone** - Speak as you normally would
- **Complete sentences** - Avoid fragments or "ums"
- **Natural pace** — Don't rush or speak too slowly
- **Clear articulation** — Pronounce words clearly
- **Consistent volume** — Maintain steady loudness
- **Normal tone** — Speak as you normally would
- **Complete sentences** — Avoid fragments or "ums"
## Multiple Samples
### Multiple Samples
Adding multiple samples can significantly improve quality:
### Why Multiple Samples?
<Cards>
<Card title="Robustness">
Model learns a more complete representation
@@ -123,110 +130,57 @@ Adding multiple samples can significantly improve quality:
</Card>
</Cards>
### Sample Variety
Consider adding samples with:
1. **Different tones**
- Casual conversation
- Professional/formal
- Excited/enthusiastic
- Calm/serious
2. **Different content**
- Narratives
- Questions
- Statements
- Emotions (happy, sad, neutral)
3. **Different recording conditions**
- Studio quality
- Phone call quality (if needed)
- Room acoustics
1. **Different tones** — casual, formal, excited, calm
2. **Different content** — narratives, questions, statements
3. **Different recording conditions** — studio quality, room acoustics
<Callout type="warn">
All samples should be from the **same speaker**. Mixing voices will produce poor results.
</Callout>
## Processing Existing Audio
### Processing Existing Audio
If you have existing audio (podcasts, videos, etc.):
### Extracting Clean Segments
<Steps>
<Step title="Find Clean Speech">
Look for segments with:
- Just the target speaker
- No background music
- Minimal noise
Look for segments with just the target speaker, no background music, minimal noise
</Step>
<Step title="Use Audio Editor">
Tools like Audacity or Adobe Audition:
- Cut out clean 10-30s segments
- Remove silence at start/end
- Normalize volume if needed
Tools like Audacity or Adobe Audition: cut clean 10-30s segments, remove silence at start/end, normalize volume
</Step>
<Step title="Export as WAV">
Save as high-quality WAV file
</Step>
</Steps>
### Noise Reduction
For light background noise, use Audacity's noise reduction (gentle settings — over-processing introduces artifacts).
If you have light background noise:
### Testing & Iteration
```
1. Use noise reduction in Audacity:
- Select noise-only section
- Get Noise Profile
- Select full audio
- Apply noise reduction (gentle settings)
2. Avoid over-processing:
- Can introduce artifacts
- May reduce voice quality
```
## Testing & Iteration
### Test Your Profile
After creating a profile:
After creating a cloned profile:
<Steps>
<Step title="Generate Test">
Generate a simple phrase:
```
"Hello, this is a test of my voice profile."
```
Try a simple phrase: `"Hello, this is a test of my voice profile."`
</Step>
<Step title="Evaluate Quality">
Listen for:
- Natural tone
- Clear pronunciation
- Proper prosody
- Lack of artifacts
Listen for natural tone, clear pronunciation, proper prosody, lack of artifacts
</Step>
<Step title="Iterate">
If quality is poor:
- Add more samples
- Try different source audio
- Check sample quality
If quality is poor: add more samples, try different source audio, check sample quality
</Step>
</Steps>
### Common Issues
#### Common Issues
<AccordionGroup>
<Accordion title="Robotic Voice">
**Cause**: Poor quality samples or too short
**Fix**: Use longer, higher quality samples
**Fix**: Use longer, higher-quality samples
</Accordion>
<Accordion title="Wrong Tone">
@@ -242,51 +196,89 @@ After creating a profile:
</Accordion>
</AccordionGroup>
## Workflow B — Preset Profiles
Use this when you want a ready-made voice without recording anything. Available engines: **Kokoro 82M** (50 voices) and **Qwen CustomVoice** (9 voices). See [Preset Voices](/overview/preset-voices) for the full catalog.
<Steps>
<Step title="Create Profile">
**Profiles** → **+ New Profile** → choose **Kokoro** or **Qwen CustomVoice** as the engine
</Step>
<Step title="Pick a Voice">
The engine's voice catalog appears. Click any voice to preview it
</Step>
<Step title="Name and Save">
Give the profile a name. No audio sample required
</Step>
<Step title="Generate">
The profile is ready immediately — use it in the floating generate box or Generate page
</Step>
</Steps>
<Callout type="info">
Preset profiles are **locked to their source engine**. Switching to a different engine in the floating generate box greys out the profile, since the voice only exists in that engine. Clicking a greyed profile auto-switches the engine back.
</Callout>
### Qwen CustomVoice + Instruct
Preset voices in Qwen CustomVoice support **delivery instructions** — natural-language style control over tone, pace, and emotion. The floating generate box shows a slider icon next to the generate button when a Qwen CustomVoice profile is selected; click it to reveal the instruct textarea.
See [Preset Voices → Using Instruct Mode](/overview/preset-voices#using-instruct-mode) for examples.
## Advanced Tips
### Celebrity/Character Voices
### Celebrity / Character Voices (Cloning)
For cloning public figures or characters:
1. **Legal considerations** - Ensure you have rights or it's fair use
2. **Source quality** - Find high-quality interview audio or clean clips
3. **Consistency** - Use clips where they speak similarly
4. **Multiple samples** - Very important for recognizable voices
1. **Legal considerations** — Ensure you have rights or it's clearly fair use
2. **Source quality** — Find high-quality interview audio or clean clips
3. **Consistency** — Use clips where they speak similarly
4. **Multiple samples** — Very important for recognizable voices
### Accent & Dialect
### Accent & Dialect (Cloning)
The model will preserve accent and dialect:
Cloning models preserve accent and dialect:
- British English will generate British English
- Southern accent will produce Southern accent
- Regional pronunciations will be maintained
- British English samples generate British English output
- Southern accent samples produce Southern accent output
- Regional pronunciations are maintained
### Emotion Transfer
### Emotion Transfer (Cloning)
The emotional tone of samples affects generation:
- Energetic samples → Energetic output
- Calm samples → Calm output
- Mix samples for versatile profile
- Energetic samples → energetic output
- Calm samples → calm output
- Mix samples for a more versatile profile
For Qwen CustomVoice presets, use the **instruct** field instead of relying on sample emotion — that's exactly what it controls.
## Managing Profiles
### Organization
- **Descriptive names** - "John Smith - Professional Narrator"
- **Add descriptions** - Note recording conditions, use cases
- **Language tags** - Mark the primary language
- **Archive unused** - Keep profile list manageable
- **Descriptive names** — "John Smith - Professional Narrator"
- **Add descriptions** — Note recording conditions, use cases, or which preset voice
- **Language tags** — Mark the primary language
- **Archive unused** — Keep profile list manageable
### Export/Import
### Export / Import
- **Export** profiles to share or backup
- **Import** from colleagues or teammates
- Profiles include voice embeddings, not original audio
- **Cloned profiles** export with their voice embeddings (not the original audio)
- **Preset profiles** export as engine + voice ID metadata only — the importer must have that engine's model installed
## Next Steps
<Cards>
<Card title="Voice Cloning" href="/overview/voice-cloning">
Engine catalog and best practices for cloning
</Card>
<Card title="Preset Voices" href="/overview/preset-voices">
Full catalog of Kokoro and Qwen CustomVoice voices
</Card>
<Card title="Generate Speech" href="/overview/generating-speech">
Use your profile to generate speech
</Card>
@@ -0,0 +1,236 @@
---
title: "GPU Acceleration"
description: "How Voicebox uses your GPU — auto-detection, manual setup, troubleshooting"
---
## Overview
Voicebox auto-detects available accelerators on first launch and picks the fastest backend it can use. For most people this just works — open the app and you're already on the right backend.
This page is for the cases where it doesn't:
- You have a GPU but Voicebox is running on CPU
- You upgraded GPUs (especially to RTX 50-series / Blackwell) and generation broke
- You want to switch backends manually (e.g. force MLX over PyTorch on Apple Silicon)
- You see `[UNSUPPORTED - see logs]` next to your GPU in Settings
## Backend Matrix
| Platform | Auto-selected backend | Notes |
| --------------------------- | ------------------------- | ---------------------------------------------------- |
| **macOS Apple Silicon** | MLX (Metal) | 4-5x faster than PyTorch via Apple Neural Engine |
| **macOS Intel** | PyTorch CPU | No GPU acceleration available; PyTorch ≥ 2.2 only |
| **Windows + NVIDIA** | PyTorch CUDA (cu128) | Auto-downloads the CUDA backend binary on first use |
| **Windows + Intel Arc** | PyTorch XPU (IPEX) | New in 0.4 — works with Arc A-series and B-series |
| **Windows generic GPU** | DirectML | Universal Windows GPU support; slower than CUDA |
| **Linux + NVIDIA** | PyTorch CUDA (cu128) | Same auto-download flow as Windows |
| **Linux + AMD** | PyTorch ROCm | Auto-configures `HSA_OVERRIDE_GFX_VERSION` |
| **Linux + Intel Arc** | PyTorch XPU (IPEX) | |
| **Any (no GPU)** | PyTorch CPU | Works everywhere; expect 5-50x slower than GPU |
The detected backend is shown in Settings → GPU. Logs at startup also print the chosen backend and the device name.
## Apple Silicon — MLX vs PyTorch
On M-series Macs, Voicebox ships an MLX-optimized backend that uses the Apple Neural Engine. It's **4-5x faster** than the PyTorch (CPU/Metal) path for supported engines.
| Engine | MLX support | Notes |
| -------------------- | ----------- | ------------------------------------------- |
| Qwen3-TTS | ✅ Native | Uses MLX exclusively when available |
| Chatterbox / Turbo | PyTorch MPS | Falls back to Metal via PyTorch |
| LuxTTS | PyTorch MPS | |
| TADA | PyTorch MPS | |
| Kokoro | PyTorch MPS | Requires `PYTORCH_ENABLE_MPS_FALLBACK=1` |
| Qwen CustomVoice | PyTorch MPS | |
| Whisper (transcribe) | ✅ Native | MLX-Whisper is the default on Apple Silicon |
The Whisper Turbo + MLX combo dropped transcription latency from ~20s to ~2-3s on M-series chips (see CHANGELOG entry for v0.1.10).
## Windows / Linux + NVIDIA — The CUDA Backend Swap
Voicebox doesn't bundle CUDA into the main installer (it would balloon downloads to multi-gigabyte territory for users who don't have an NVIDIA GPU). Instead, when you first need it, the app downloads a separate **CUDA backend binary** that contains the PyTorch + CUDA runtime.
<Steps>
<Step title="Open Settings → GPU">
If an NVIDIA GPU is detected, you'll see "Install CUDA backend" in the GPU panel
</Step>
<Step title="Click Install">
The app downloads two archives separately:
- **Server core** (~200-400 MB) — versioned with each Voicebox release
- **CUDA libs** (~4 GB) — the heavy PyTorch + CUDA DLLs, versioned independently
</Step>
<Step title="Restart">
Voicebox restarts to swap in the CUDA backend
</Step>
</Steps>
<Callout type="info">
The split-archive design (added in v0.4) means most Voicebox upgrades only redownload the small server-core archive. The 4 GB libs archive is only refreshed when the underlying CUDA toolkit or torch major version changes.
</Callout>
### Auto-update
When a new Voicebox release ships, the GPU panel checks if the bundled server-core matches the installed CUDA version. If only the core changed (typical), it pulls the new core in the background. If the libs version changed (rare — only happens on cu126 → cu128 type bumps), you'll be prompted to confirm the larger download.
## RTX 50-series / Blackwell
Voicebox 0.4 added explicit RTX 50-series support:
- CUDA toolkit upgraded to **cu128** (previous releases used cu126 which lacks Blackwell kernels)
- Build pinned with `TORCH_CUDA_ARCH_LIST=...12.0+PTX` for forward-compatibility
If you're on an RTX 5070 / 5080 / 5090 and you see "no kernel image is available" errors:
1. Make sure you're on Voicebox **≥ 0.4.0** (Settings → About)
2. Reinstall the CUDA backend (Settings → GPU → Reinstall CUDA backend) — older installs may have stale cu126 libs
3. If errors persist, see the GPU compatibility warnings section below
## Intel Arc (XPU)
New in 0.4. Works with both Arc A-series (Alchemist: A380, A580, A750, A770) and B-series (Battlemage).
### Setup
Voicebox auto-detects Arc GPUs and routes through Intel's PyTorch XPU backend (powered by IPEX — Intel Extension for PyTorch). No extra installation step beyond the standard Voicebox install.
Verify it's working:
- Settings → GPU should show **XPU** followed by your Arc model name (e.g. `XPU (Intel Arc A770)`)
- Startup logs print `Backend: PYTORCH` and `GPU: XPU (Intel Arc ...)`
### Engines on XPU
All PyTorch-based engines work on XPU. Performance is generally between CPU and CUDA — expect ~2-3x speedup over CPU for the larger models.
## DirectML
The fallback for Windows users with non-NVIDIA, non-Intel-Arc GPUs (older AMD discrete, integrated GPUs, etc.). Slower than CUDA and XPU but provides some acceleration over CPU.
Auto-selected when no other GPU backend is available.
## AMD ROCm (Linux)
ROCm provides PyTorch GPU acceleration on AMD discrete GPUs. Voicebox auto-configures `HSA_OVERRIDE_GFX_VERSION` for common cards that need the override.
### Verifying
```bash
# In a terminal
echo $HSA_OVERRIDE_GFX_VERSION
# Should show e.g. 10.3.0 for RX 6000 series
```
If detection fails, set the variable manually before launching Voicebox:
```bash
export HSA_OVERRIDE_GFX_VERSION=10.3.0
voicebox
```
Common values:
- `10.3.0` — RX 6000 series (RDNA 2)
- `11.0.0` — RX 7000 series (RDNA 3)
- `9.0.0` — Older Vega cards
## GPU Compatibility Warnings
Voicebox 0.4 added a runtime check that compares your GPU's compute capability against the architectures the bundled PyTorch was compiled for. If they don't match, you'll see:
- A startup log line: `WARNING: GPU COMPATIBILITY: <your GPU> is not supported by this PyTorch build...`
- The GPU label in Settings shows `[UNSUPPORTED - see logs]`
- The `/health` API returns a populated `gpu_compatibility_warning` field
### What to do
The most common trigger is a brand-new GPU architecture that pre-built PyTorch wheels don't yet cover natively. In order of preference:
1. **Update Voicebox** — newer releases ship newer PyTorch with broader arch support
2. **Reinstall the CUDA backend** — Settings → GPU → Reinstall CUDA backend
3. **For bleeding-edge GPUs (newer than current Blackwell):** install PyTorch nightly manually:
```bash
pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128 --force-reinstall
```
Then point Voicebox at that environment via [Remote Mode](/overview/remote-mode) until stable PyTorch catches up.
4. **Fall back to CPU** temporarily — set `VOICEBOX_FORCE_CPU=1` before launching
## CPU-Only Fallback
When no GPU is available (or you've forced it off), Voicebox runs the PyTorch CPU backend. Expect:
- 5-50x slower generation depending on engine and text length
- Heavy CPU usage during generation
- Some engines work better than others on CPU:
- **Kokoro 82M** — runs at realtime on modern CPUs
- **LuxTTS** — exceeds 150x realtime on CPU
- **Chatterbox Turbo (350M)** — usable but slow
- Larger models (Qwen 1.7B, Chatterbox Multilingual, TADA 3B) — painful
For CPU-bound use cases, prefer the smaller, lighter engines.
## Verifying Your Setup
Three places to check that the right backend is being used:
<Steps>
<Step title="Settings → GPU">
Shows the detected backend, GPU model, and VRAM (when applicable). Look for the `[UNSUPPORTED - see logs]` suffix
</Step>
<Step title="Settings → Logs">
The "Server logs" tab shows the startup banner with `Backend: <type>` and `GPU: <name>`
</Step>
<Step title="Health endpoint">
`curl http://localhost:17493/health` returns a JSON payload with `backend_type`, `backend_variant`, and `gpu_compatibility_warning` (when applicable)
</Step>
</Steps>
## Troubleshooting
<AccordionGroup>
<Accordion title="Settings shows CPU instead of my GPU">
- On NVIDIA: install the CUDA backend (Settings → GPU)
- On Intel Arc: confirm IPEX detection in startup logs; restart the app after a driver update
- On AMD Linux: check `HSA_OVERRIDE_GFX_VERSION` is set
</Accordion>
<Accordion title="'no kernel image is available' / 'CUDA error'">
Almost always means the bundled PyTorch doesn't have kernels for your GPU's compute capability.
1. Update to Voicebox ≥ 0.4.0 (Blackwell support added there)
2. Reinstall the CUDA backend
3. If still broken, install PyTorch nightly via Remote Mode
</Accordion>
<Accordion title="Out of memory (CUDA)">
- Switch to a smaller model size (e.g. Qwen3 0.6B instead of 1.7B)
- Use Settings → Models to unload other engines you're not using
- Enable `low_cpu_mem_usage` is already on for CPU; for CUDA, the engine's `device_map` handles offload automatically
- Close other GPU applications
</Accordion>
<Accordion title="MPS fallback errors on macOS">
Some operations don't have a Metal implementation. Voicebox sets `PYTORCH_ENABLE_MPS_FALLBACK=1` for engines that need it (notably Kokoro), but if you launch from a custom env, set it manually:
```bash
export PYTORCH_ENABLE_MPS_FALLBACK=1
```
</Accordion>
<Accordion title="Generation works but is slow on my GPU">
- Check Settings → GPU shows your GPU (not CPU)
- Check VRAM usage — you may be paging to system memory
- Try a smaller model
- For NVIDIA: confirm cu128 is installed (Settings → GPU → version)
</Accordion>
</AccordionGroup>
## Next Steps
<Cards>
<Card title="Remote Mode" href="/overview/remote-mode">
Run the backend on a different machine with a stronger GPU
</Card>
<Card title="Model Management" href="/developer/model-management">
Unload models to free GPU memory
</Card>
<Card title="Troubleshooting" href="/overview/troubleshooting">
General troubleshooting beyond GPU
</Card>
</Cards>
+5 -4
View File
@@ -5,10 +5,10 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 4 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** -- models and voice data stay on your machine
- **4 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
@@ -20,7 +20,7 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## TTS Engines
Four engines with different strengths, switchable per-generation:
Five engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
@@ -28,6 +28,7 @@ Four engines with different strengths, switchable per-generation:
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
## GPU Support
@@ -56,7 +57,7 @@ Four engines with different strengths, switchable per-generation:
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
+2
View File
@@ -6,7 +6,9 @@
"installation",
"docker",
"quick-start",
"gpu-acceleration",
"voice-cloning",
"preset-voices",
"stories-editor",
"recording-transcription",
"generation-history",
@@ -0,0 +1,202 @@
---
title: "Preset Voices"
description: "Use built-in, ready-made voices without recording audio samples"
---
## Overview
Some Voicebox engines ship with a curated set of pre-built voices. Instead of cloning from your own audio sample, you pick a voice from a fixed catalog and the model speaks in that voice. No recording, no upload, no per-voice training required.
Two engines in 0.4 ship preset voices:
| Engine | Voices | Languages | Strengths |
| --------------------- | ----------------------- | --------- | ------------------------------------------------------- |
| **Kokoro 82M** | 50 | 9 | Tiny model, CPU-friendly, lowest VRAM of any engine |
| **Qwen CustomVoice** | 9 (premium curated) | 4 | Natural-language style control over tone, emotion, pace |
<Callout type="info">
Looking for cloning a specific person's voice instead? See [Voice Cloning](/overview/voice-cloning).
</Callout>
## When to Use Preset Voices
<Cards>
<Card title="No reference audio">
You don't have (or don't want to provide) a recording of the target voice
</Card>
<Card title="Production reliability">
Curated voices have predictable quality across any text input
</Card>
<Card title="Speed">
Skip the audio cleanup, sample preparation, and quality iteration loop
</Card>
<Card title="Lightweight setup">
Kokoro runs at CPU realtime with ~150 MB on disk — no GPU needed
</Card>
</Cards>
## Creating a Preset-Voice Profile
<Steps>
<Step title="Open Profiles → New Profile">
Same entry point as cloning profiles
</Step>
<Step title="Choose the engine">
Select **Kokoro** or **Qwen CustomVoice** from the engine dropdown
</Step>
<Step title="Pick a preset voice">
The voice catalog for the chosen engine appears — preview each by clicking it
</Step>
<Step title="Name and save">
Give the profile a name. No audio sample needed — just save
</Step>
<Step title="Generate">
Use the profile like any other in the floating generate box or the Generate page
</Step>
</Steps>
<Callout type="info">
Preset profiles are locked to their source engine — switching engines won't work since the voice exists only for that model. The profile grid greys out preset profiles when you switch to a different engine, and clicking one auto-switches the engine back to the right one.
</Callout>
## Kokoro 82M — 50 Voices Across 9 Languages
Kokoro is the smallest engine in Voicebox at 82M parameters. It runs at CPU realtime with negligible VRAM, making it the best option for lightweight local inference. Voices are pre-built style vectors trained into the model — there's no concept of cloning here.
**Repository:** [`hexgrad/Kokoro-82M`](https://huggingface.co/hexgrad/Kokoro-82M) · Apache 2.0 licensed
### American English
| Female | Male |
| ------- | ------- |
| Alloy | Adam |
| Aoede | Echo |
| Bella | Eric |
| Heart | Fenrir |
| Jessica | Liam |
| Kore | Michael |
| Nicole | Onyx |
| Nova | Puck |
| River | Santa |
| Sarah | |
| Sky | |
### British English
| Female | Male |
| -------- | ------ |
| Alice | Daniel |
| Emma | Fable |
| Isabella | George |
| Lily | Lewis |
### Other Languages
| Language | Voices |
| ----------------- | ------------------------------------------- |
| Spanish (`es`) | Dora (f), Alex (m), Santa (m) |
| French (`fr`) | Siwis (f) |
| Hindi (`hi`) | Alpha (f), Beta (f), Omega (m), Psi (m) |
| Italian (`it`) | Sara (f), Nicola (m) |
| Japanese (`ja`) | Alpha (f), Gongitsune (f), Nezumi (f), Tebukuro (f), Kumo (m) |
| Portuguese (`pt`) | Dora (f), Alex (m), Santa (m) |
| Chinese (`zh`) | Xiaobei (f), Xiaoni (f), Xiaoxiao (f), Xiaoyi (f) |
### Kokoro at a Glance
| Property | Value |
| --------------- | -------------------------------------------- |
| Parameters | 82M |
| Sample rate | 24 kHz |
| VRAM | ~150 MB (negligible on CPU) |
| Speed | Realtime on CPU, faster on GPU |
| Instruct | Not supported (preset voice carries the style) |
| License | Apache 2.0 |
## Qwen CustomVoice — 9 Premium Voices with Instruct Control
Qwen CustomVoice ships with 9 curated speakers and supports **natural-language style control** — you tell the model how to deliver the line ("speak slowly with warmth", "authoritative and clear") and it adapts tone, emotion, and pace.
Two model sizes:
- **1.7B** — full quality, recommended default
- **0.6B** — lighter, faster, lower-end hardware
**Repository:** [`Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) (and 0.6B variant) · by Alibaba
### Voice Catalog
| Speaker | Gender | Language | Description |
| --------- | ------ | -------- | ------------------------------------------------------------ |
| Vivian | female | Chinese | Bright, slightly edgy young female voice |
| Serena | female | Chinese | Warm, gentle young female voice |
| Uncle Fu | male | Chinese | Seasoned male voice with a low, mellow timbre |
| Dylan | male | Chinese | Youthful Beijing male voice with a clear, natural timbre |
| Eric | male | Chinese | Lively Chengdu male voice with a slightly husky brightness |
| Ryan | male | English | Dynamic male voice with strong rhythmic drive (default) |
| Aiden | male | English | Sunny American male voice with a clear midrange |
| Ono Anna | female | Japanese | Playful Japanese female voice with a light, nimble timbre |
| Sohee | female | Korean | Warm Korean female voice with rich emotion |
### Using Instruct Mode
In the floating generate box, switch to a Qwen CustomVoice profile and click the **delivery instructions** toggle (slider icon, left of the generate button). A second textarea appears below the main text:
- Main text → what you want the voice to say
- Instruct text → how you want it delivered
Examples of effective instruct prompts:
```
Speak slowly with emphasis, like reading bedtime stories
Warm and friendly, conversational tone
Professional and authoritative, broadcast quality
Whisper, intimate and close
Excited and energetic, like sports commentary
```
The full Generate page also surfaces the instruct field as a separate input.
### Qwen CustomVoice at a Glance
| Property | Value |
| --------------- | -------------------------------------------------- |
| Parameters | 1.7B / 0.6B |
| Languages | Chinese, English, Japanese, Korean (10 supported) |
| Voices | 9 curated preset speakers |
| VRAM | ~3.5 GB (1.7B), ~1.2 GB (0.6B) |
| Instruct | Yes — natural-language style control |
| Cloning | No — paired Base Qwen3-TTS engine handles cloning |
## Cloning vs Preset — Quick Decision
| You want… | Use |
| -------------------------------------------------- | ----------------------------------------- |
| To replicate a specific person's voice | [Voice Cloning](/overview/voice-cloning) |
| Production-ready voices with no audio prep | Kokoro or Qwen CustomVoice |
| The smallest possible footprint (CPU-only) | Kokoro |
| Fine control over delivery (tone, pace, emotion) | Qwen CustomVoice |
| The broadest language coverage | [Voice Cloning](/overview/voice-cloning) via Chatterbox Multilingual (23 langs) |
## Limitations
<Callout type="warn">
Preset voices are fixed — you can't fine-tune or modify the underlying voice. If you want a specific voice that isn't in the catalog, use a cloning engine and provide a reference sample.
</Callout>
- Preset voices can't be exported to use in other Voicebox installations as audio (only as profile metadata pointing to the same engine + voice ID)
- The Kokoro voice catalog is set by the upstream model — new voices appear only when hexgrad publishes new model releases
- Qwen CustomVoice's 9 speakers are part of the model checkpoint — same constraint
## Next Steps
<Cards>
<Card title="Voice Cloning" href="/overview/voice-cloning">
Clone a specific voice from your own audio
</Card>
<Card title="Generate Speech" href="/overview/generating-speech">
Use a profile to generate audio
</Card>
<Card title="Build Stories" href="/overview/building-stories">
Compose multi-voice narratives
</Card>
</Cards>
+58 -15
View File
@@ -1,11 +1,25 @@
---
title: "Voice Cloning"
description: "Clone any voice from just a few seconds of audio"
description: "Clone any voice from a few seconds of reference audio"
---
## Overview
Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning from just a few seconds of audio. The model captures prosody, emotion, and natural cadence.
Voicebox can replicate a specific person's voice from a short audio sample — known as **zero-shot voice cloning**. You provide 10-30 seconds of clear speech, the model extracts a voice embedding, and from then on you can generate any text in that voice.
Five engines in 0.4 support cloning:
| Engine | Languages | Strengths |
| --------------------------- | --------- | -------------------------------------------------------------------------- |
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual, supports delivery instructions on the same kwarg |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Hindi, Swahili, Hebrew, more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion tags (`[laugh]`, `[sigh]`) |
| **LuxTTS** | English | Lightweight (~1 GB VRAM), 48 kHz output, 150x realtime on CPU |
| **TADA** (1B / 3B) | 10 | Speech-language model with 700s+ coherent long-form generation |
<Callout type="info">
Don't want to record audio? Use a curated voice from Kokoro or Qwen CustomVoice instead — see [Preset Voices](/overview/preset-voices).
</Callout>
## How It Works
@@ -13,17 +27,30 @@ Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning f
<Step title="Upload or Record Sample">
Provide 10-30 seconds of clear speech from the target voice
</Step>
<Step title="Model Analysis">
Qwen3-TTS analyzes vocal characteristics, tone, and speaking patterns
<Step title="Engine Analysis">
The selected engine analyzes vocal characteristics, tone, and speaking patterns
</Step>
<Step title="Voice Profile Created">
The model generates a voice embedding for synthesis
A voice embedding is generated and stored with your profile
</Step>
<Step title="Generate Speech">
Use the profile to generate any text in the cloned voice
</Step>
</Steps>
## Choosing an Engine for Cloning
Different engines suit different use cases. The profile grid greys out unsupported engines so you can switch easily.
| If you want… | Pick |
| -------------------------------------------------- | --------------------- |
| Best overall quality on a few common languages | **Qwen3-TTS 1.7B** |
| Faster generation, slightly lower quality | **Qwen3-TTS 0.6B** |
| Languages outside Qwen's 10 (Arabic, Hindi, etc.) | **Chatterbox Multilingual** |
| Expressive English with `[laugh]` `[sigh]` tags | **Chatterbox Turbo** |
| CPU-only or GPU-light setup, English | **LuxTTS** |
| Long-form generation (audiobooks, full chapters) | **TADA 3B** |
## Best Practices
### Sample Quality
@@ -52,24 +79,40 @@ Adding multiple samples from the same speaker can improve quality:
- Different recording conditions
<Callout type="info">
The model will learn a more robust representation from diverse samples.
The model will learn a more robust representation from diverse samples. Especially helpful for distinctive voices the model might otherwise smooth over.
</Callout>
## Supported Languages
## Supported Languages by Engine
Currently supported:
- English
- Chinese (Mandarin)
- **Qwen3-TTS** — English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian (10)
- **Chatterbox Multilingual** — Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Turkish (23)
- **Chatterbox Turbo** — English
- **LuxTTS** — English
- **TADA 3B** — 10 multilingual; **TADA 1B** — English
More languages coming soon.
For complete language tables and engine-specific notes, see the [TTS Engines developer guide](/developer/tts-engines).
## Limitations
<Callout type="warn">
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice.
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice. See the project's [SECURITY.md](https://github.com/jamiepine/voicebox/blob/main/SECURITY.md) and your local laws on synthetic voice content.
</Callout>
- Quality depends on sample clarity
- Works best with consistent speaking tone
- Quality depends on sample clarity — noisy samples produce noisy clones
- Works best with consistent speaking tone within a sample
- May struggle with extreme accents or speech impediments
- Background noise reduces quality
- Background noise reduces quality and can introduce artifacts
## Next Steps
<Cards>
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
Step-by-step guide to creating profiles
</Card>
<Card title="Preset Voices" href="/overview/preset-voices">
Use built-in voices instead of cloning
</Card>
<Card title="Generating Speech" href="/overview/generating-speech">
Use a profile to generate audio
</Card>
</Cards>
+115 -130
View File
@@ -1,6 +1,6 @@
# Voicebox Project Status & Roadmap
> Last updated: 2026-03-13 | Current version: **v0.1.13** | 13.1k stars | ~176 open issues | 25 open PRs
> Last updated: 2026-03-18 | Current version: **v0.3.0** | 13.4k stars | ~136 open issues | 9 open PRs
---
@@ -36,6 +36,10 @@
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │ │ │ TADA │ │ │ │
│ │ │ │(1B / 3B) │ │ │ │
│ │ │ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ ┌───────────┐ ┌─────────┐ │ │
│ │ │ STTBackend│ │ Profiles│ │ │
@@ -59,6 +63,7 @@
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
@@ -78,7 +83,7 @@
```
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo)
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
@@ -95,7 +100,7 @@ POST /generate
## Current State
### What's Shipped (v0.1.13 + recent merges)
### What's Shipped (v0.3.0)
**Core TTS:**
- Qwen3-TTS voice cloning (1.7B and 0.6B models)
@@ -103,29 +108,42 @@ POST /generate
- Multi-engine TTS architecture with thread-safe backend registry (PR #254)
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Instruct parameter UI exists but is non-functional across all backends (see #224, Known Limitations)
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps in `main.py`
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296)
- Chunked TTS generation for long text — engine-agnostic, removes ~500 char limit (PR #266)
- Async generation queue (PR #269)
- Post-processing audio effects system (PR #271)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
**Infrastructure:**
- CUDA backend swap via binary download and restart (PR #252)
- GPU acceleration settings UI
- CUDA backend swap via binary download and restart (PR #252), upgraded to cu128 (PR #316)
- CUDA backend split into independently versioned server + libs archives (PR #298)
- Docker + web deployment (PR #161)
- Backend refactor: modular architecture, style guide, tooling (PR #285)
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
- Windows support: CUDA detection, cross-platform justfile, clean server shutdown (PR #272)
- Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large variants)
- Model management UI with inline download progress bars (HFProgressTracker)
- Model management UI with inline download progress bars + folder migration (PR #268)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching
- Streaming generation endpoint (MLX only)
- Duplicate profile name validation (PR #175)
- Linux NVIDIA GBM buffer + WebKitGTK microphone fix (PR #210)
- Audio player freeze fix + UX improvements (PR #293)
- CORS restriction to known local origins (PR #88)
### Abandoned Integrations
| Model | PR | Reason |
|-------|----|--------|
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. |
### What's In-Flight
| Feature | Branch/PR | Status |
|---------|-----------|--------|
| Chatterbox Turbo + per-engine language lists | `feat/chatterbox-turbo` / PR #258 | Open, ready for review |
| Kokoro 82M TTS engine | WIP | In development — 82M CPU-realtime engine, 8 languages |
### TTS Engine Comparison
@@ -136,6 +154,9 @@ POST /generate
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
| Kokoro 82M | `kokoro` | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
### Multi-Engine Architecture (Shipped)
@@ -143,7 +164,7 @@ The singleton TTS backend blocker described in the previous version of this doc
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'`
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'`
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
@@ -165,69 +186,41 @@ The singleton TTS backend blocker described in the previous version of this doc
| PR | Title | Merged |
|----|-------|--------|
| **#257** | feat: Chatterbox TTS engine with multilingual voice cloning | 2026-03-13 |
| **#254** | feat: LuxTTS integration — multi-engine TTS support | 2026-03-13 |
| **#252** | feat: CUDA backend swap via binary download and restart | 2026-03-13 |
| **#238** | Download cancel/clear UI, fixed model downloading | 2026-03-13 |
| **#250** | docs: align local API port examples | 2026-03-13 |
| **#210** | fix: Linux NVIDIA GBM buffer crash | 2026-03-13 |
| **#175** | Fix #134: duplicate profile name validation | 2026-03-13 |
| **#316** | Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI | 2026-03-18 |
| **#305** | fix: bundle qwen_tts source files in PyInstaller build | 2026-03-17 |
| **#298** | feat: split CUDA backend into independently versioned server + libs archives | 2026-03-17 |
| **#296** | Add HumeAI TADA TTS engine (1B English + 3B Multilingual) | 2026-03-17 |
| **#295** | fix: batch of bug fixes from issue tracker | 2026-03-17 |
| **#293** | Fix audio player freezing and improve UX | 2026-03-17 |
| **#294** | Settings overhaul: routed sub-tabs, server logs, changelog, about page | 2026-03-16 |
| **#288** | Better docs | 2026-03-16 |
| **#285** | Backend refactor: modular architecture, style guide, tooling | 2026-03-16 |
| **#274** | Landing page v0.2.0 redesign | 2026-03-15 |
| **#272** | Windows support: CUDA detection, cross-platform justfile, clean server shutdown | 2026-03-15 |
| **#271** | Add post-processing audio effects system | 2026-03-14 |
| **#269** | feat: async generation queue | 2026-03-13 |
| **#268** | feat: model management improvements and folder migration | 2026-03-13 |
| **#266** | feat: chunked TTS generation for long text (engine-agnostic) | 2026-03-13 |
| **#265** | feat: paralinguistic tag autocomplete for Chatterbox Turbo | 2026-03-13 |
| **#264** | fix: Chatterbox float64 dtype mismatch + model unload button | 2026-03-13 |
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | 2026-03-13 |
| **#230** | docs: fix README grammar | 2026-03-13 |
| **#161** | feat: Docker + web deployment | 2026-03-13 |
| **#88** | security: restrict CORS to known local origins | 2026-03-13 |
### In-Flight (Our Work)
### Currently Open (9 PRs)
| PR | Title | Status | Notes |
|----|-------|--------|-------|
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | Open | Ready for review. Adds Turbo engine + dynamic language dropdown. |
### Merge-Ready / Near-Ready (Bug Fixes & Small Features)
| PR | Title | Risk | Notes |
|----|-------|------|-------|
| **#230** | docs: fix README grammar | None | Docs-only |
| **#243** | a11y: screen reader and keyboard improvements | Low | Accessibility, no backend changes |
| **#178** | Fix #168 #140: generation error handling | Low | Error handling improvements |
| **#152** | Fix: prevent crashes when HuggingFace unreachable | Medium | Monkey-patches HF hub; solves real offline bug (#150, #151) |
| **#218** | fix: unify qwen tts cache dir on Windows | Low | Windows-specific path fix |
| **#214** | fix: panic on launch from tokio::spawn | Low | Rust-side Tauri fix |
| **#88** | security: restrict CORS to known local origins | Low | Security hardening |
| **#133** | feat: network access toggle | Low | Wires up existing plumbing |
### Significant Feature PRs
| PR | Title | Complexity | Notes |
|----|-------|-----------|-------|
| **#253** | Enhance speech tokenizer with 48kHz version | Medium | Qwen tokenizer upgrade |
| **#97** | fix: pass language parameter to TTS models | Medium | May be partially obsoleted by multi-engine work — needs review |
| **#99** | feat: chunked TTS with quality selector | Medium | Solves 500-char limit. Addresses #191, #203, #69, #111. |
| **#154** | feat: Audiobook tab | Medium | Full audiobook workflow. Depends on #99 concepts. |
| **#91** | fix: CoreAudio device enumeration | Medium | macOS audio device handling |
### Architectural PRs (Need Careful Review)
| PR | Title | Complexity | Notes |
|----|-------|-----------|-------|
| **#225** | feat: custom HuggingFace model support | High | Arbitrary HF repo loading. May need rework given multi-engine arch is now shipped. |
| **#194** | feat: Hebrew + Chatterbox TTS | High | **Superseded** by PR #257 which shipped Chatterbox multilingual (23 langs incl. Hebrew). May be closeable. |
| **#195** | feat: per-profile LoRA fine-tuning | Very High | Training pipeline, adapter management, 15 new endpoints. Depends on #194 (now superseded). |
| **#161** | feat: Docker + web deployment | High | 3-stage Dockerfile, SPA serving. Independent of TTS engine work. |
| **#124** / **#123** | Docker (simpler attempts) | Low-Medium | Overlap with #161 |
| **#227** | fix: harden input validation & file safety | Medium | Coupled to #225 (custom models) |
### PRs That Need Author Action / Are Stale
| PR | Title | Notes |
|----|-------|-------|
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Build system, needs review |
| **#215** | Update prerequisites with Tauri deps | Branch is `main` — will have conflicts |
| **#89** | Linux Support | Branch is `main` — will have conflicts. Broad scope. |
| **#83** | Update download links for v0.1.12 | Outdated (we're on v0.1.13) |
### PRs Likely Superseded
| PR | Superseded By | Notes |
|----|--------------|-------|
| **#194** (Hebrew + Chatterbox) | PR #257 (merged) | #257 ships Chatterbox multilingual with 23 languages including Hebrew. #194 took a different approach (route by language). Can likely be closed. |
| **#33** (External provider binaries) | PR #252 (merged) | #252 shipped CUDA backend swap. #33's broader provider architecture may still have value but needs reassessment. |
| **#311** | feat: add CosyVoice2/3 TTS engine | **Will close** | Model quality too poor. See Abandoned Integrations. |
| **#253** | Enhance speech tokenizer with 48kHz version | Community PR | Qwen tokenizer upgrade. Worth reviewing. |
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Superseded | Our PR #305 shipped this. Can close. |
| **#227** | fix: harden input validation & file safety | Community PR | Coupled to #225 (custom models). |
| **#225** | feat: custom HuggingFace model support | Community PR | Needs rework for multi-engine arch. |
| **#218** | fix: unify qwen tts cache dir on Windows | Community PR | Windows-specific path fix. Still relevant. |
| **#195** | feat: per-profile LoRA fine-tuning | Draft | Complex. 15 new endpoints. |
| **#154** | feat: Audiobook tab | Community PR | Chunked generation now shipped (#266). |
| **#91** | fix: CoreAudio device enumeration | Draft | macOS audio device handling. |
---
@@ -272,7 +265,7 @@ Strong demand for: Hindi (#245), Indonesian (#247), Dutch (#236), Hebrew (#199),
| #132 | LavaSR (transcription) |
| #76 | (General model expansion) |
Community also requests: XTTS-v2, Fish Speech, CosyVoice, Kokoro. The multi-engine architecture is now in place, making new model integration significantly easier.
Community also requests: XTTS-v2, Fish Speech, Kokoro. CosyVoice was tried and abandoned. The multi-engine architecture is in place, making new model integration straightforward.
### Long-Form / Chunking (5 issues)
@@ -280,7 +273,7 @@ Users hitting the ~500 character practical limit.
**Key issues:** #234 (queue system), #203 (500 char limit), #191 (auto-split), #111, #69
**Fix path:** PR #99 (chunked TTS + quality selector) directly addresses this. PR #154 (Audiobook tab) builds on it.
**Fix path:** **Mostly resolved.** PR #266 (engine-agnostic chunked TTS) and PR #269 (async generation queue) are both merged. PR #154 (Audiobook tab) is still open.
### Feature Requests (23 issues)
@@ -318,7 +311,7 @@ Notable requests:
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **PR exists** (#161) | Waiting on review |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
@@ -326,31 +319,31 @@ Notable requests:
## New Model Integration — Landscape
### Models Worth Supporting (2026 SOTA — updated March 13)
### Models Worth Supporting (2026 SOTA — updated March 18)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct Support | Integration Ease | Status |
|-------|---------|-------|-------------|-----------|------|-----------------|-----------------|--------|
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None (Base); Yes (CustomVoice variant, predefined speakers only) | **Shipped** | v0.1.13 |
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | **Shipped** | PR #254 |
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` float | **Shipped** | PR #257 |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **PR #258** | In review |
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** — `inference_instruct2()`, works with cloning | Ready | Best instruct candidate |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Multi-engine arch in place |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0, multi-speaker dialogue |
| **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody from text context | Needs vetting | MIT, 700s+ coherent, synced transcript output |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion |
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Partial — automatic style inference | Ready | Apache 2.0, multi-engine arch in place |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Multi-engine arch in place |
| **Pocket TTS** | Zero-shot + streaming | >1× RT on CPU | — | English | ~100M params, CPU-first | None | Needs vetting | MIT, Kyutai Labs, no GPU required |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **Shipped** | PR #258 |
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody | **Shipped** | PR #296 |
| **Kokoro-82M** | Pre-built voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | **In progress** | Apache 2.0, pip install, ~350MB |
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Yes — `inference_instruct2()` | **Abandoned** | PR #311 — poor output quality |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Needs license clarification |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Mature pip package |
| **Pocket TTS** | Zero-shot + streaming | >1x RT on CPU | — | English | ~100M params, CPU-first | None | Ready | MIT, Kyutai Labs |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0 |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0 |
#### Notes on New Candidates (March 2026)
#### Notes on Candidates (March 2026)
- **CosyVoice2-0.5B** — Best candidate for instruct support. `inference_instruct2()` accepts a text instruct parameter for emotions, speed, volume, dialects — and it works alongside voice cloning. This is the closest match to what users expect from our instruct UI. [HF: FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B)
- **HumeAI TADA** — Text-Audio Dual Alignment arch. Near-zero hallucinations/drift, free synced transcript. 700+ seconds coherent audio. Best candidate for Stories long-form reliability. Prosody/emotion is automatic from text context, not user-controllable. [HF: HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **MOSS-TTS** — Modular suite: flagship cloning, MOSS-TTSD (multi-speaker dialogue), MOSS-VoiceGenerator (create voices from text descriptions). VoiceGenerator unifies timbre design and style control via text prompts, usable as a layer for downstream TTS including cloning. [HF: OpenMOSS-Team/MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | [GitHub: OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS)
- **Fish Speech** — Word-level fine-grained control using plain language descriptions inline in the script. Works with cloning. Note: Fish Audio S2 has a restrictive research license (commercial use requires approval), but the open-source Fish Speech model may differ. Needs license clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
- **VoxCPM 1.5** — Tokenizer-free continuous diffusion + autoregressive. No discrete token artifacts. Prosody/emotion is context-aware but automatic, not explicitly controllable via text prompt. Real-time streaming, LoRA fine-tuning. Trained on 1.8M+ hours. [GitHub: OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs (Moshi team). Runs >1× realtime without GPU. No style control. Broadens hardware support significantly. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **CosyVoice2-0.5B** — **Tried and abandoned** (PR #311). Despite having the best instruct API, output quality was poor. No PyPI package, needed 5+ shims, heavy deps. Not worth it.
- **HumeAI TADA** — **Shipped** (PR #296). 700+ seconds coherent audio. [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **Kokoro-82M** — **In progress.** 82M params, CPU realtime, Apache 2.0, clean `pip install kokoro`. Uses pre-built voice styles (not zero-shot cloning from arbitrary audio). [GitHub: hexgrad/kokoro](https://github.com/hexgrad/kokoro)
- **Fish Speech** — Word-level fine-grained control. License needs clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
- **XTTS-v2** — Coqui's multilingual cloning. 17+ languages, pip-installable. [GitHub: coqui-ai/TTS](https://github.com/coqui-ai/TTS)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control)
### Adding a New Engine (Now Straightforward)
@@ -394,49 +387,44 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
## Recommended Priorities
### Tier 1 — Ship Now (Low Risk)
### Tier 1 — Ship Now
| Priority | PR/Item | Impact | Effort |
|----------|---------|--------|--------|
| 1 | **#258** — Chatterbox Turbo + per-engine languages | Paralinguistic tags, proper language filtering | Review only |
| 2 | **#152** — Offline mode crash fix | Fixes #150, #151 | Low |
| 3 | **#99** — Chunked TTS + quality selector | Removes 500-char limit, addresses 5 issues | Medium |
| 4 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 5 | **#178** — Generation error handling | Error UX | Low |
| 6 | **#230** — Docs fixes | Zero risk | None |
| 7 | **#133** — Network access toggle | Wires up existing code | Low |
| 8 | **#88** — CORS restriction | Security improvement | Low |
| 9 | **#214** — Tauri window close panic fix | Stability | Low |
| 10 | Triage GPU issues | Many may be resolved by CUDA swap (#252) | Low |
| 11 | Close superseded PRs | #194 (superseded by #257), #83 (outdated) | None |
| 1 | **Kokoro 82M** — finish integration | New engine, CPU-friendly, 8 langs | Low (nearly done) |
| 2 | Close PR #311 (CosyVoice) and #237 (superseded by #305) | Housekeeping | None |
| 3 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 4 | **#253** — 48kHz speech tokenizer | Quality improvement for Qwen | Medium |
### Tier 2 — Next Release (v0.2.0)
### Tier 2 — Feature Work
| Priority | Item | Impact | Effort |
|----------|------|--------|--------|
| 1 | **#253** — 48kHz speech tokenizer | Quality improvement | Medium |
| 2 | **#161** — Docker deployment | Server/headless users | Medium |
| 3 | **#154** — Audiobook tab | Long-form users | Medium |
| 4 | ~~**Model config registry**~~ | ~~Reduce dispatch duplication in main.py~~ | **Done** |
| 5 | **#225** — Custom HuggingFace models | User-supplied models | High (needs rework for multi-engine) |
| 1 | **#154** — Audiobook tab | Long-form users. Chunking + queue now shipped. | Medium |
| 2 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
| 3 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable | Low |
| 4 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
| 5 | Streaming for non-MLX engines | Currently MLX-only | Medium |
### Tier 3 — Future (v0.3.0+)
### Tier 3 — Future Engines
| Priority | Item | Notes |
|----------|------|-------|
| 1 | **HumeAI TADA** | Long-form reliability for Stories, synced transcripts. Addresses #234, #203, #191, #111, #69. Needs API vetting. |
| 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model, broadens hardware support. Kyutai ships clean code. Needs API vetting. |
| 3 | **MOSS-TTS** | Text-to-voice design (no ref audio) is unique. Multi-speaker dialogue for Stories. Needs thorough API vetting. |
| 4 | **Kokoro-82M** | 82M params, CPU realtime, Apache 2.0. Easy win. |
| 5 | ~~**Model config registry refactor**~~ | **Done** — consolidated in `backend/backends/__init__.py` + `EngineModelSelector.tsx` |
| 6 | XTTS-v2 / Fish Speech / CosyVoice | Multi-engine arch is ready; just needs backend implementation |
| 7 | **VoxCPM 1.5** | Tokenizer-free streaming, interesting but uncertain integration surface |
| 8 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable |
| 9 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine |
| 10 | External/remote providers | Depends on use case demand |
| 11 | GGUF support (#226) | Depends on model ecosystem maturity |
| 12 | Queue system (#234) | Batch generation |
| 13 | Streaming for non-MLX engines | Currently MLX-only |
| 1 | **Fish Speech** | 50+ langs, word-level instruct. License TBD. |
| 2 | **XTTS-v2** | 17+ langs, mature pip package. Best multilingual cloning. |
| 3 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. |
| 4 | **MOSS-TTS** | Text-to-voice design. Multi-speaker dialogue for Stories. |
| 5 | **VoxCPM 1.5** | Tokenizer-free streaming. Uncertain integration surface. |
### ~~Previously Prioritized — Now Done~~
- ~~#258 — Chatterbox Turbo~~ **Merged**
- ~~#99 — Chunked TTS~~ **Superseded by #266, merged**
- ~~#88 — CORS restriction~~ **Merged**
- ~~#161 — Docker deployment~~ **Merged**
- ~~#234 — Queue system~~ **Addressed by #269, merged**
- ~~HumeAI TADA~~ **Shipped** (PR #296)
- ~~Kokoro-82M~~ **In progress**
---
@@ -444,13 +432,10 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
| Branch | PR | Status | Notes |
|--------|-----|--------|-------|
| `feat/chatterbox-turbo` | #258 | Open | Chatterbox Turbo + per-engine languages |
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
| `external-provider-binaries` | #33 | Superseded by #252 | Original CUDA provider approach |
| `feat/dual-server-binaries` | — | No PR | Related to provider split |
| `fix-multi-sample` | — | No PR | Voice profile multi-sample fix |
| `fix-dl-notification-...` | — | No PR | Model download UX |
---
@@ -475,7 +460,7 @@ The generation form now uses a flat model dropdown with engine-based routing. Pe
| `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
| `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model |
+428
View File
@@ -0,0 +1,428 @@
# Voicebox API Refactor Plan
Date: 2026-03-19
Status: Proposed
Scope: Backend HTTP API structure, schemas, docs, and compatibility strategy
## Goals
- Make the API easier to understand and automate against.
- Improve endpoint consistency without breaking the desktop app or existing local integrations.
- Align generated docs and checked-in OpenAPI artifacts with the actual backend.
- Separate app-facing resources from internal or operational actions.
- Create a migration path toward a cleaner `v2` resource model while preserving `v1` routes during transition.
## Non-Goals
- Rewriting backend business logic or generation internals.
- Introducing authentication for all deployment modes in the first pass.
- Changing storage models or database schema unless required for API correctness.
- Removing current routes immediately.
## Current Pain Points
- Mixed endpoint styles: resource-oriented (`/profiles`) and command-oriented (`/generate`, `/tasks/clear`) coexist.
- Related generation resources are split across multiple namespaces: `/generate`, `/history`, `/audio`, `/effects`, and `/generations/.../versions`.
- Response payloads vary widely: typed models, raw dicts with `message`, booleans, and `HTTPException(detail=...)` payloads.
- Some async flows use exception-shaped `202` responses instead of first-class task contracts.
- Checked-in OpenAPI output can drift from actual backend models.
- Operational endpoints such as `/shutdown` are exposed in the same surface as user workflows.
## Guiding Principles
1. Prefer additive changes before destructive changes.
2. Keep `v1` behavior working until the app and docs fully migrate.
3. Add compatibility shims close to the routing layer, not deep in services.
4. Treat OpenAPI as a release artifact that must be kept in sync.
5. Standardize public contracts before renaming everything.
## Target API Shape
This is the intended end state, not the immediate first milestone.
### Core Resources
- `/profiles`
- `/profiles/{profile_id}/samples`
- `/profiles/{profile_id}/avatar`
- `/profiles/{profile_id}/effects`
- `/generations`
- `/generations/{generation_id}`
- `/generations/{generation_id}/status`
- `/generations/{generation_id}/audio`
- `/generations/{generation_id}/versions`
- `/generations/{generation_id}/versions/{version_id}`
- `/generations/{generation_id}/versions/{version_id}/audio`
- `/stories`
- `/stories/{story_id}/items`
- `/effects/presets`
- `/models`
- `/models/{model_name}`
- `/tasks`
### Operational or Internal Endpoints
Move under an explicit namespace and disable where appropriate:
- `/admin/shutdown`
- `/admin/watchdog/disable`
- `/admin/cache/clear`
- `/admin/tasks/clear`
### Response Contract Direction
- Resource reads and writes return typed resource models.
- Delete and action endpoints return small typed action result models.
- Errors use a consistent structure.
- Async actions return explicit task metadata instead of overloading `detail`.
## Migration Strategy Overview
The refactor is split into six phases. Phases 1-3 are the highest impact and safest to ship first.
| Phase | Focus | Est. Duration | Risk | Backward Compatibility |
| --- | --- | --- | --- | --- |
| 1 | Documentation and contract correctness | 2-3 days | Low | Full |
| 2 | Response and error consistency | 3-5 days | Low-Medium | Full |
| 3 | Router structure and internal organization | 3-4 days | Low | Full |
| 4 | Additive `v2` resource endpoints | 1-2 weeks | Medium | Full |
| 5 | Client migration and deprecation rollout | 1 week | Medium | Full during rollout |
| 6 | Cleanup and optional removals | 1-2 releases | Medium-High | Partial after notice |
## Phase 1: Fix Contract Drift First
Priority: Highest
Outcome: The documented API matches the running backend.
### Problems Addressed
- `docs/openapi.json` can become stale.
- Generated API reference pages may describe outdated request bodies.
- App metadata still frames the backend too narrowly.
### Implementation Steps
1. Update FastAPI app metadata in `backend/app.py`.
- Replace the old Qwen-specific description with a multi-engine Voicebox API description.
- Add tags metadata for major domains if desired.
2. Regenerate OpenAPI from the running app using the existing docs script flow.
3. Compare `backend/models.py` to the checked-in schema.
- Verify `GenerationRequest`, effects endpoints, stories endpoints, and model endpoints.
4. Regenerate or refresh API reference pages under `docs/content/docs/api-reference/`.
5. Add a CI check that fails if `docs/openapi.json` is out of date.
6. Add a short maintainer note describing when schema regeneration is required.
### Backward Compatibility
- No route changes.
- No payload changes.
- Safe to release immediately.
### Success Criteria
- `docs/openapi.json` matches the live app.
- Generated docs include all currently supported generate parameters.
- No frontend code changes required.
## Phase 2: Standardize Responses and Errors
Priority: High
Outcome: Clients can handle responses predictably.
### Problems Addressed
- Delete endpoints return ad hoc message dicts.
- Toggle endpoints return special one-off payloads.
- `202` async responses are encoded as `HTTPException(detail=...)` in some places.
### Implementation Steps
1. Add shared response models in `backend/models.py`.
- `ActionResult`
- `DeleteResult`
- `ToggleFavoriteResponse`
- `AcceptedTaskResponse`
- `ApiError`
2. Convert routes that currently return raw dicts to explicit `response_model`s.
- `DELETE /profiles/{profile_id}`
- `DELETE /history/{generation_id}`
- `DELETE /stories/{story_id}`
- `POST /tasks/clear`
- `POST /cache/clear`
- similar endpoints across routes
3. Replace exception-shaped `202` responses in `transcription.py` with an explicit accepted response body.
- Return `JSONResponse(status_code=202, content=...)` or typed FastAPI response model.
4. Add a global exception handler for known API errors if helpful.
- Normalize `ValueError` to `400` with a consistent error body.
- Preserve FastAPI validation errors for now, or wrap them in a consistent top-level shape in a later pass.
5. Document the stable error contract in the docs.
### Migration Strategy
- Keep field names inside successful payloads compatible where possible.
- For existing dict responses, preserve the current keys while introducing typed models with the same shape.
- For `202` flows, support both old and new client handling for one release if needed.
### Timeline Estimate
- 3-5 engineering days including tests and docs refresh.
### Success Criteria
- All mutation endpoints declare response models.
- Clients can programmatically distinguish success, accepted, and error cases without special casing `detail` payloads.
## Phase 3: Normalize Router Structure Internally
Priority: High
Outcome: The backend becomes easier to maintain before public path changes begin.
### Problems Addressed
- Route files hardcode full paths and are all mounted at root.
- There is no consistent use of router prefixes or tags.
- Route grouping in code does not cleanly express the public API shape.
### Implementation Steps
1. Add prefixes and tags to routers.
- `profiles`: `prefix="/profiles"`
- `generations`: `prefix="/generate"` for now or split additive aliases carefully
- `history`: `prefix="/history"`
- `effects`: `prefix="/effects"`
- and so on
2. Convert route declarations to relative paths within each router.
3. Introduce a small route compatibility layer for routes that are likely to move later.
- Example: helper functions that can be mounted under both old and new paths.
4. Add explicit route tags so Swagger/OpenAPI groups are coherent.
5. Document the intended public ownership of each namespace.
### Backward Compatibility
- No public path changes yet if existing paths are preserved through prefixes and aliases.
- Mostly internal refactoring.
### Timeline Estimate
- 3-4 engineering days.
### Success Criteria
- All route modules use prefixes and tags.
- Route registration in `backend/routes/__init__.py` becomes simpler.
- OpenAPI groups read cleanly by domain.
## Phase 4: Introduce Additive `v2` Resource Endpoints
Priority: High
Outcome: A cleaner API exists without breaking the current one.
### Problems Addressed
- Generation-related resources are fragmented.
- Sample and audio endpoints are not consistently modeled as resources.
- Command-style naming makes the API harder to reason about.
### New Endpoints to Add
These should be introduced alongside current endpoints, not as replacements.
- `POST /generations` -> alias for current `/generate`
- `GET /generations` -> alias for current `/history`
- `GET /generations/{id}` -> alias for current `/history/{id}`
- `POST /generations/{id}/retry` -> alias for current `/generate/{id}/retry`
- `POST /generations/{id}/regenerate` -> alias for current `/generate/{id}/regenerate`
- `GET /generations/{id}/status` -> alias for current `/generate/{id}/status`
- `POST /generations/stream` -> alias for current `/generate/stream`
- `GET /generations/{id}/audio` -> alias for current `/audio/{generation_id}`
- `GET /generations/{id}/export` -> alias for current `/history/{generation_id}/export`
- `GET /generations/{id}/export-audio` -> alias for current `/history/{generation_id}/export-audio`
- `GET /profiles/{profile_id}/samples/{sample_id}` or `GET /samples/{sample_id}` as a consciously chosen model
- `PUT /profiles/{profile_id}/samples/{sample_id}` -> alias for current sample update route
- `DELETE /profiles/{profile_id}/samples/{sample_id}` -> alias for current sample delete route
### Implementation Steps
1. Create new handler entry points that call the existing service functions.
2. Keep old handlers in place, but mark them deprecated in OpenAPI.
3. Add `summary` and `description` text clarifying preferred routes.
4. Update frontend and docs examples to use new endpoints first.
5. Add tests proving both old and new paths return equivalent responses.
### Migration Strategy
- Old paths remain functional for at least one stable release cycle.
- New docs and client examples use `v2-style` resource routes immediately.
- Include deprecation headers where feasible, for example:
- `Deprecation: true`
- `Sunset: <date>`
- `Link: <new-doc-url>; rel="successor-version"`
### Timeline Estimate
- 1-2 weeks depending on test coverage and frontend updates.
### Success Criteria
- All major generation workflows are accessible through resource-oriented routes.
- Old routes still work unchanged.
## Phase 5: Migrate First-Party Clients and Publish Deprecations
Priority: Medium
Outcome: Voicebox itself stops depending on legacy paths.
### Problems Addressed
- The desktop app and docs may continue to reinforce old route shapes.
- Third-party consumers need a visible migration path.
### Implementation Steps
1. Update `app/src/lib/api/client.ts` to use the new preferred endpoints.
2. Regenerate or refresh any generated API clients.
3. Update docs examples, tutorials, and code snippets to use preferred routes only.
4. Add a changelog entry describing the migration path.
5. Add runtime deprecation logging for legacy route usage in development mode.
6. If feasible, expose a small `/health` or `/meta` field showing API version and deprecation window.
### Migration Strategy
- Keep old endpoints available but clearly documented as legacy.
- Publish a mapping table from old route to new route.
- Do not change request or response payloads during the same phase unless necessary.
### Timeline Estimate
- About 1 week including docs and app verification.
### Success Criteria
- First-party app no longer depends on legacy route names.
- Docs do not advertise deprecated paths as the primary interface.
## Phase 6: Cleanup, Namespace Hardening, and Optional Breaking Changes
Priority: Medium
Outcome: The API surface is cleaner and safer for remote or Docker use.
### Problems Addressed
- Internal/admin endpoints are mixed into the public API.
- Legacy aliases increase maintenance cost forever if never retired.
### Implementation Steps
1. Move operational endpoints under `/admin` or `/internal`.
- `/shutdown`
- `/watchdog/disable`
- `/tasks/clear`
- `/cache/clear`
2. Gate these endpoints behind configuration for non-local deployments.
- Example: `VOICEBOX_ENABLE_ADMIN_API=true`
3. Decide whether to remove or keep legacy aliases.
- If removing, do so only after a published deprecation window.
4. Remove deprecated docs pages and old examples.
5. Tighten route-level tests to prevent accidental reintroduction of legacy patterns.
### Migration Strategy
- For desktop-only local use, aliases may remain indefinitely if removal cost outweighs benefit.
- For published remote API guidance, hide admin endpoints from default docs even if they still exist.
### Timeline Estimate
- 1-2 releases after the additive migration is complete.
### Success Criteria
- Public docs expose a coherent resource API.
- Operational endpoints are clearly separate or disabled in remote contexts.
## Cross-Cutting Work Items
These should happen throughout the migration, not only in a single phase.
### Testing
- Add route equivalence tests for old and new paths.
- Add schema snapshot tests for OpenAPI generation.
- Add response-shape tests for common mutations and async workflows.
- Add contract tests for `202 Accepted` flows.
### Documentation
- Maintain an old-to-new endpoint mapping table.
- Add per-endpoint examples for create profile, generate, apply effects, transcribe, and stories operations.
- Explicitly document which endpoints are app-facing vs admin-facing.
### Observability
- Add warning logs when deprecated endpoints are used.
- Track usage counts in development or optional telemetry-free local logs.
### Release Management
- Mention API changes in `CHANGELOG.md`.
- Ensure docs and app updates ship in the same release as new preferred routes.
## Recommended Execution Order
If engineering time is limited, implement in this exact order:
1. Fix OpenAPI and docs drift.
2. Standardize response models and accepted-task responses.
3. Add router prefixes and tags internally.
4. Add `/generations` aliases and sample path aliases.
5. Migrate the first-party app to preferred routes.
6. Deprecate or hide legacy/admin routes.
## Old-to-New Route Mapping
| Current Route | Preferred Route |
| --- | --- |
| `POST /generate` | `POST /generations` |
| `POST /generate/stream` | `POST /generations/stream` |
| `POST /generate/{id}/retry` | `POST /generations/{id}/retry` |
| `POST /generate/{id}/regenerate` | `POST /generations/{id}/regenerate` |
| `GET /generate/{id}/status` | `GET /generations/{id}/status` |
| `GET /history` | `GET /generations` |
| `GET /history/{id}` | `GET /generations/{id}` |
| `GET /audio/{id}` | `GET /generations/{id}/audio` |
| `GET /history/{id}/export` | `GET /generations/{id}/export` |
| `GET /history/{id}/export-audio` | `GET /generations/{id}/export-audio` |
| `PUT /profiles/samples/{sample_id}` | `PUT /profiles/{profile_id}/samples/{sample_id}` |
| `DELETE /profiles/samples/{sample_id}` | `DELETE /profiles/{profile_id}/samples/{sample_id}` |
| `POST /tasks/clear` | `POST /admin/tasks/clear` |
| `POST /cache/clear` | `POST /admin/cache/clear` |
| `POST /shutdown` | `POST /admin/shutdown` |
| `POST /watchdog/disable` | `POST /admin/watchdog/disable` |
## Risks and Mitigations
### Risk: App regressions during endpoint migration
- Mitigation: Add new routes before changing client usage.
- Mitigation: Keep payloads identical while paths change.
### Risk: Docs still drift after cleanup
- Mitigation: Add CI enforcement and a release checklist step.
### Risk: Third-party local scripts break on removal
- Mitigation: Prefer indefinite aliases for one-person local workflows unless maintenance becomes painful.
### Risk: Admin endpoints remain dangerous in remote mode
- Mitigation: Hide and gate them before promoting remote deployment more broadly.
## Definition of Done
The refactor can be considered complete when all of the following are true:
- OpenAPI, checked-in docs, and backend models match.
- The preferred public API is resource-oriented and documented consistently.
- The Voicebox app uses preferred routes exclusively.
- Legacy routes are either deprecated with a timeline or intentionally retained as compatibility aliases.
- Operational endpoints are clearly separated from the public app API.
+173
View File
@@ -0,0 +1,173 @@
# CUDA Libs as a Bolt-On Addon
## Problem
Every time we bump `__version__` (even for a UI tweak or bugfix), the exact-match version check in both `main.rs:222` and `cuda.py:237` invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime.
## Why This Is Hard With `--onefile`
The core tension is PyInstaller `--onefile` mode (`build_binary.py:39`). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA `.dll`/`.so` files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server.
## Options
### Option A: Switch to `--onedir` for the CUDA Build (Recommended)
Instead of `--onefile`, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives:
1. **`voicebox-server-cuda` executable + non-NVIDIA deps** (~200-400MB) -- versioned with the app, redownloaded on every app update.
2. **`cuda-libs-cu126.tar.gz`** (~2GB) -- the `nvidia.*` packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., `cuda-libs-cu126-v1`). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes.
#### How it would work at runtime
- Tauri downloads the server binary archive and extracts it to `{data_dir}/backends/cuda/`
- On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory
- The CUDA server exe finds the `.dll`/`.so` files next to it (standard PyInstaller onedir behavior)
- Version check becomes two checks: server version + cuda-libs version
#### Independent versioning
Add a `cuda-libs.json` manifest:
```json
{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"}
```
The server checks this on startup. The Tauri side checks it before launching. Only bump `cu126-v1` -> `cu126-v2` when we actually change the CUDA toolkit or torch major version.
#### Build pipeline changes
The CI `build-cuda-windows` job would build with `--onedir`, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset.
#### Download experience
- First-time CUDA setup: ~2.4GB total (same as today)
- Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached
- CUDA toolkit bump: ~2GB for just the libs
#### Pros
- PyInstaller `--onedir` natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory
- The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (`nvidia/cublas/lib/`, etc.)
- CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version
- Server updates become ~200-400MB instead of ~2.4GB
- No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree
#### Cons
- Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up
- Need to modify download/assembly logic in `cuda.py` to handle two separate archives
- The Tauri side (`main.rs`) needs to point at an exe inside a directory rather than a standalone binary
- Users who manually manage the file may find the folder structure confusing
#### TTS engine compatibility
No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way.
---
### Option B: Keep `--onefile` but Externalize CUDA Libs via Library Path
Keep the server as a single `--onefile` binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to `{data_dir}/backends/cuda-libs/`. Before launching, set the library search path to include that directory.
**Important caveat:** The CPU torch wheel (`whl/cpu`) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally.
#### How it would work
- Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded
- Ship `cuda-libs-cu126-v1.tar.gz` separately (~2GB of `.dll`/`.so` files)
- When launching, Tauri sets `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux) to include the cuda-libs directory
#### Pros
- Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously
- True bolt-on CUDA libs with fully independent versioning
- Server updates are always small (~150MB for the onefile binary)
#### Cons
- **Fragile on Windows.** PyInstaller `--onefile` extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario.
- `os.add_dll_directory()` only affects `LoadLibraryEx` with `LOAD_LIBRARY_SEARCH_USER_DIRS` flag -- not all DLL loads go through this path
- PyInstaller's onefile bootloader may configure DLL search paths before Python code runs
- Could work on Linux but is fragile on Windows
---
### Option C: Hybrid -- `--onefile` Server + Dynamic CUDA Lib Loading at Runtime
Build the server as `--onefile` with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using `ctypes.CDLL` or `os.add_dll_directory()`.
In `server.py`, before any torch imports:
```python
cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS")
if cuda_libs_dir and os.path.isdir(cuda_libs_dir):
if sys.platform == "win32":
os.add_dll_directory(cuda_libs_dir)
os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "")
else:
os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "")
```
#### Pros
- Single server binary, true bolt-on CUDA libs
- Clean separation of concerns
- Independent versioning
#### Cons
- Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer
- On Windows, `os.add_dll_directory()` may not cover all DLL load paths
- PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs
- Most complex to get right and maintain
## Recommendation
**Option A (`--onedir` with split archives)** is the most reliable path:
1. **It actually works.** `--onedir` puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install.
2. **Natural separation.** PyInstaller's `--onedir` output already separates the NVIDIA `.dll`/`.so` files into `nvidia/` subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building.
3. **Independent versioning is straightforward.** A `cuda-libs.json` manifest controls when redownloads are needed.
4. **Build pipeline simplification.** Build CUDA libs archive less frequently, store as a pinned release asset.
The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in `cuda.py` with manifests and split parts. Extending that to handle two archives is incremental work.
## Tauri Compatibility (Validated)
Tauri handles PyInstaller `--onedir` with no issues. The key insight is that we're **not** using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing `cuda.py` + `main.rs` flow). For runtime-launched processes, Tauri's `tauri::shell::Command` supports arbitrary directories natively.
### The critical change in `main.rs`
The only Tauri-side change needed is adding `.current_dir()` when spawning the CUDA backend:
```rust
let cuda_dir = data_dir.join("backends/cuda");
let exe_path = cuda_dir.join("voicebox-server-cuda.exe");
let mut cmd = app.shell().command(exe_path.to_str().unwrap());
cmd = cmd.current_dir(&cuda_dir); // PyInstaller finds all DLLs relative to exe
cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);
```
`.current_dir()` tells the PyInstaller bootloader that everything (DLLs, `nvidia/cublas/lib/`, `_internal/`, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal `pip install` or dev environment -- no `LD_LIBRARY_PATH` hacks, no `os.add_dll_directory` gymnastics.
### Community evidence
- Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206).
- The shell plugin explicitly supports `cwd` in both Rust and JS APIs.
- No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage.
### Known gotcha: process termination on Windows
PyInstaller onedir creates a parent bootloader + child Python process on Windows. `child.kill()` only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use `taskkill /F /T` for clean shutdown. This is not a blocker -- our existing `--parent-pid` watchdog mechanism in `server.py` already handles orphan cleanup.
## Next Steps
1. Prototype: Build the current CUDA binary with `--onedir` and verify torch CUDA works from the output directory
2. Measure the size split: how much is NVIDIA libs vs everything else
3. Design the two-archive download flow and dual version checking
4. Update `cuda.py` for dual-archive extraction (server core + cuda-libs)
5. Update `main.rs`: change launch path to `backends/cuda/` dir + add `.current_dir()`
6. Add `ensure_cuda_structure()` helper in Rust to verify exe + nvidia/ subdirs exist before spawning
7. Update CI pipeline: `build-cuda-windows` produces two archives instead of split parts
8. ~~Update `split_binary.py` or replace with archive-based distribution~~ Done: replaced with `package_cuda.py`
+1 -6
View File
@@ -1,9 +1,4 @@
import {
defineConfig,
defineDocs,
frontmatterSchema,
metaSchema,
} from 'fumadocs-mdx/config';
import { defineConfig, defineDocs, frontmatterSchema, metaSchema } from 'fumadocs-mdx/config';
// You can customise Zod schemas for frontmatter and `meta.json` here
// see https://fumadocs.dev/docs/mdx/collections
+4 -14
View File
@@ -2,11 +2,7 @@
"compilerOptions": {
"baseUrl": ".",
"target": "ESNext",
"lib": [
"dom",
"dom.iterable",
"esnext"
],
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
@@ -20,12 +16,8 @@
"jsx": "react-jsx",
"incremental": true,
"paths": {
"@/*": [
"./*"
],
"@/.source": [
".source"
]
"@/*": ["./*"],
"@/.source": [".source"]
},
"plugins": [
{
@@ -40,7 +32,5 @@
".next/types/**/*.ts",
".next/dev/types/**/*.ts"
],
"exclude": [
"node_modules"
]
"exclude": ["node_modules"]
}
+21 -5
View File
@@ -46,6 +46,8 @@ setup-python:
{{ pip }} install -r {{ backend_dir }}/requirements.txt
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
{{ pip }} install --no-deps chatterbox-tts
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
{{ pip }} install --no-deps hume-tada
# Apple Silicon: install MLX backend
if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then
echo "Detected Apple Silicon — installing MLX dependencies..."
@@ -67,13 +69,26 @@ setup-python:
}
Write-Host "Installing Python dependencies..."
& "{{ python }}" -m pip install --upgrade pip -q
$hasNvidia = $null -ne (Get-WmiObject Win32_VideoController | Where-Object { $_.Name -match 'NVIDIA' })
$gpus = Get-CimInstance Win32_VideoController | Select-Object -ExpandProperty Name
Write-Host "Detected GPUs: $($gpus -join ', ')"
$hasNvidia = ($gpus | Where-Object { $_ -match 'NVIDIA' }).Count -gt 0
$hasIntelArc = ($gpus | Where-Object { $_ -match 'Arc' }).Count -gt 0
if ($hasNvidia) { \
Write-Host "NVIDIA GPU detected — installing PyTorch with CUDA support..."; \
& "{{ pip }}" install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126; \
& "{{ pip }}" install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128; \
} elseif ($hasIntelArc) { \
Write-Host "Intel Arc GPU detected — installing PyTorch with XPU support..."; \
& "{{ pip }}" install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu; \
& "{{ pip }}" install intel-extension-for-pytorch --index-url https://download.pytorch.org/whl/xpu; \
} else { \
Write-Host "No NVIDIA or Intel Arc GPU detected — using CPU-only PyTorch."; \
Write-Host "If you have an Intel Arc GPU, install XPU support manually:"; \
Write-Host " pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu"; \
Write-Host " pip install intel-extension-for-pytorch --index-url https://download.pytorch.org/whl/xpu"; \
}
& "{{ pip }}" install -r {{ backend_dir }}/requirements.txt
& "{{ pip }}" install --no-deps chatterbox-tts
& "{{ pip }}" install --no-deps hume-tada
& "{{ pip }}" install git+https://github.com/QwenLM/Qwen3-TTS.git
& "{{ pip }}" install pyinstaller ruff pytest pytest-asyncio -q
Write-Host "Python environment ready."
@@ -205,10 +220,11 @@ build-server-cuda: _ensure-venv
$env:PATH = "{{ venv_bin }};$env:PATH"; \
& "{{ python }}" backend/build_binary.py --cuda; \
if ($LASTEXITCODE -ne 0) { throw "build_binary.py --cuda failed with exit code $LASTEXITCODE" }; \
$dest = "$env:APPDATA/com.voicebox.app/backends"; \
$dest = "$env:APPDATA/sh.voicebox.app/backends/cuda"; \
if (Test-Path $dest) { Remove-Item -Recurse -Force $dest }; \
New-Item -ItemType Directory -Path $dest -Force | Out-Null; \
Copy-Item "backend/dist/voicebox-server-cuda.exe" "$dest/voicebox-server-cuda.exe" -Force; \
Write-Host "Copied CUDA binary to $dest"
Copy-Item "backend/dist/voicebox-server-cuda/*" $dest -Recurse -Force; \
Write-Host "Copied CUDA backend to $dest"
# Build everything locally: CPU server + CUDA server + installable Tauri app
[windows]
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@voicebox/landing",
"version": "0.2.3",
"version": "0.4.0",
"description": "Landing page for voicebox.sh",
"scripts": {
"dev": "bun --bun next dev --turbo",

Some files were not shown because too many files have changed in this diff Show More