Compare commits

...
Author SHA1 Message Date
James PineandClaude Opus 4.7 a6ac8ceba7 fix(landing): address PR #487 review feedback
- Preserve canonical camelCase platform aliases (macArm, macIntel) in the
  /download/[platform] redirect so those URLs don't lose their platform param.
- Add accessible title + role="img" to the inline Windows SVG so it passes
  Biome's a11y rule and announces to screen readers.
- On /api/releases fetch failure, show an explicit error state with a single
  intentional link to GitHub releases — no more silent GitHub fallback or
  disabled-button UX lie. Keeps normies off GitHub unless they opt in.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:31:34 -07:00
James PineandClaude Opus 4.7 a179b826cd docs: consolidate troubleshooting into the MDX docs site + status updates
- Delete docs/TROUBLESHOOTING.md; the canonical troubleshooting guide now
  lives under docs/content/docs/overview/troubleshooting.mdx so it's served
  from docs.voicebox.sh alongside the rest of the docs.
- CONTRIBUTING.md + README.md: repoint "Troubleshooting" references to the
  new MDX path. README gets a top-level callout so users hit the guide
  before filing an issue.
- PROJECT_STATUS.md: refresh issue/PR counts, document the flash-attn
  warning (cosmetic on all platforms; CUDA-only, fallback is PyTorch SDPA
  which is near-FA2 on Ampere+) with per-platform context + community
  Windows wheels + SageAttention/xformers alternatives, add WebAudio
  audio-session bug note (tracked separately in PR #486), and expand the
  Qwen 0.6B→1.7B MLX fallback explanation for triage.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:20:35 -07:00
James PineandClaude Opus 4.7 a641ffc919 fix(landing): route Linux users to /linux-install instead of attempting download
No prebuilt Linux binary exists yet (see /linux-install for build-from-source
instructions). The /download page previously treated Linux like the other
platforms — auto-triggering a non-existent AppImage and offering a dead
manual button.

- /download page: if platform resolves to 'linux' via ?platform or UA detect,
  window.location.replace('/linux-install') — never try to auto-download.
- Manual Linux card: label changed to "Build from source" and links to
  /linux-install (no download attribute, no asset URL).
- /download/linux pretty URL: 307s straight to /linux-install.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:20:20 -07:00
James PineandClaude Opus 4.7 f267cafb80 chore(landing): run dev server on Node instead of Bun runtime
Bun runtime + Next 16 Turbopack dev server intermittently trips a
JavaScriptCore allocator panic ('pas panic: deallocation did fail ...
Alloc bit not set') after a few requests. Dropping --bun keeps Bun as
the package manager but runs next dev on Node, which is stable.

Build + start keep --bun since one-shot invocations don't exhibit the
allocator drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:16:20 -07:00
James PineandClaude Opus 4.7 83cc172f71 fix(landing): route Download CTAs to /download page, not the section anchor
Hero CTA, navbar link, and footer link were all scrolling to #download
(the section at the bottom of the page) instead of going to the new
/download page that triggers the actual download.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:14:52 -07:00
James PineandClaude Opus 4.7 d7285bf23a fix(landing): use official platform brand icons via simple-icons
The hand-rolled Linux SVG path wasn't actually Tux — it was a symmetric
placeholder shape. Apple/Windows were close but not canonical either.

- Apple + Linux: pulled from @icons-pack/react-simple-icons (SiApple, SiLinux).
- Windows: simple-icons drops the Microsoft mark over trademark policy, so
  the Windows 11 flag is inlined from Microsoft's public brand guidance.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:12:56 -07:00
James PineandClaude Opus 4.7 13924741a7 feat(landing): add polished /download page — no more dumping users on GitHub
Users were clicking download, landing on the GitHub releases page, and filing
confused comments along the lines of "I ended up on some blog site called
GitHub." We now route every download CTA through a dedicated /download page
that auto-triggers the platform-specific download and gives users a polished
post-click experience with donate + docs + AI help prompts.

- New /download page:
  - Big app logo + "Your download has started" messaging.
  - Auto-detects platform from ?platform=X or navigator.userAgent.
  - Programmatically clicks a hidden anchor to trigger the file download
    without leaving the page.
  - Platform-specific buttons as a visible fallback for "download not
    working" / manual-pick.
  - Personal donate spiel + Buy Me a Coffee button.
  - Resources grid: docs, DeepWiki ("got questions? ask AI"), GitHub.
- Landing page download section cards now link to /download?platform=X
  instead of the asset URL directly.
- /download/[platform] (used by README/docs links) now redirects to the
  /download page rather than straight to the asset or to GitHub on error.
- Drops unused downloadLinks state from the landing page.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 23:09:03 -07:00
James PineandClaude Opus 4.7 1ca0756dc1 fix(landing): use a realistic UUID for profile_id in API example
Profile IDs are str(uuid.uuid4()), not slugs (see backend/services/profiles.py:175).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 22:57:30 -07:00
James PineandClaude Opus 4.7 cf17d94441 fix(landing): use qwen_custom_voice in API example (instruct is CustomVoice-only)
The curl snippet showed engine: "qwen" alongside an instruct field, but base
Qwen3-TTS has no instruct path — that's a Qwen CustomVoice feature.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 22:56:45 -07:00
James Pine 28aa963b09 readme update 2026-04-18 21:19:51 -07:00
ae91aa9a88 docs: audit mdx docs against multi-engine backend (#484)
* docs: audit mdx docs against multi-engine backend and refresh stale content

Rewrote developer-facing docs that predated the TTSBackend Protocol /
ModelConfig registry refactor (architecture, tts-generation,
model-management, transcription). Updated user-facing docs to reflect all
seven shipped engines (Qwen, Qwen CustomVoice, LuxTTS, Chatterbox,
Chatterbox Turbo, TADA, Kokoro) instead of the outdated "5 engines" claim.

Also fixes:
- Stale app identifier (com.voicebox.app → sh.voicebox.app)
- CUDA backend update flow (now two-archive split, not N-way chunks)
- Whisper model list (removed tiny, added turbo)
- Broken /development/ and /guides/ route links
- Stale just commands and install steps (missing --no-deps chatterbox/tada)
- Removed ASCII art diagrams from README and stories.mdx
- History Generation schema sync with DB model

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: add DeepWiki badge to README

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: address PR review feedback

- architecture.mdx: fix backends/ file list (remove nonexistent qwen_backend.py, rename tada_backend.py → hume_backend.py)
- model-management.mdx: Kokoro language count 9 → 8 (matches ModelConfig)
- model-management.mdx: ProgressManager path services/ → utils/
- tts-generation.mdx: ModelConfig example uses field(default_factory=...) — mutable default would raise at runtime
- tts-generation.mdx: "1080p samples" → "on CUDA" (1080p is video, not audio)
- PROJECT_STATUS.md: replace ASCII architecture diagram with prose (matches no-ASCII-art rule)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(app): guard against undefined engine in FloatingGenerateBox preset check

form.getValues('engine') returns string | undefined; Set<string>.has()
rejects undefined under strict mode. Added a truthy guard before the
preset lookup.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 21:06:06 -07:00
da6070155e landing: three more tutorials, mobile navbar + hero CTA fixes (#483)
- Add three tutorial cards (Danish Sofi, StinkyScrublet, mikbes)
- Navbar: switch parent to flex/justify-between on mobile (grid on sm+),
  unhide Donate button so both CTAs sit on the right, matching desktop
- Hero CTAs: keep Download and GitHub side-by-side on mobile instead of
  stacking vertically

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 19:21:46 -07:00
3c1e8512b9 fix(build): install mlx-audio/mlx-lm with --no-deps to bypass transformers 5.x conflict (#482)
The previous fix (#481) capped transformers at 4.57.6 in requirements-mlx.txt,
but pip's clean resolver in CI can't satisfy that alongside mlx-audio>=0.3.1
(declares `transformers==5.0.0rc3` or `>=5.0.0`) — it backtracks through every
transformers and tokenizers version and exits with `ResolutionImpossible`.

The dev install worked only because mlx-audio 0.4.1 was already present, so
pip never tried to re-resolve.

mlx-audio 0.4.1 + mlx-lm 0.31.1 both declare transformers>=5.x but the API
surface we actually use works fine on 4.57.x in practice (verified across all
engines in dev). Install both --no-deps to bypass the resolver; transitive
runtime deps (huggingface_hub, librosa, numpy, numba, pyloudnorm, etc.) are
already pulled in by requirements.txt.

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 17:43:31 -07:00
bf58750447 fix(build): pin transformers in MLX requirements to prevent 5.x upgrade (#481)
mlx-audio depends on `transformers` with no upper bound. Installing
requirements-mlx.txt after requirements.txt lets pip upgrade transformers
past the 4.57.x cap to 5.x, which breaks three engines in the frozen MLX
bundle:

- qwen-custom-voice: `check_model_inputs` was rewritten to take `func` as
  positional, so `@check_model_inputs()` factory calls fail with
  `TypeError: missing 1 required positional argument: 'func'`
- tada-1b: `PretrainedConfig.__init_subclass__` now applies `@dataclass`,
  which rejects tada's `strides: list = []` mutable default
- luxtts: Whisper init hits `AssertionError` in `torch._refs.normal_`

Restating the same constraint here keeps mlx-audio's transformers
dependency from quietly winning the resolver.

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 17:05:36 -07:00
Jamie PineandGitHub 2d56309bdd Change 'About' link text to 'Models' 2026-04-18 16:46:23 -07:00
James Pine 0445be295c tests and better website 2026-04-18 16:19:12 -07:00
James Pine 8d550a5f7c Bump version: 0.4.0 → 0.4.1 2026-04-18 15:15:58 -07:00
Esteban FraccasciaandGitHub 795bd54381 fix(linux): use pactl to detect PipeWire/PulseAudio monitor for system audio capture (#457)
cpal 0.15 uses ALSA as its Linux backend, which does not expose
PulseAudio/PipeWire monitor sources. The previous approach searched
for 'monitor' in cpal device names, which never matched on most
Linux systems, silently falling back to the microphone input.

This fix:
- Detects the correct monitor source via 'pactl get-default-sink'
  and 'pactl list short sources'
- Sets PULSE_SOURCE env var before cpal initialization so PulseAudio's
  ALSA plugin routes the default input through the monitor
- Preserves the original name-based search as fallback when pactl is
  unavailable
- No new dependencies added

Tested on PipeWire 1.0.5 with Realtek ALC897 (HD-Audio Generic).
2026-04-18 03:15:05 -07:00
a6ab5f3858 Add initial frontend quality gates and TS hardening (#418)
Co-authored-by: Erion De Andrade <[email protected]>
2026-04-18 03:14:43 -07:00
9d7e4a417e fix(api-client): declare moved + errors on migrateModels response type (#470)
ModelManagement.tsx reads migrationResult.moved (added in #433) but
apiClient.migrateModels() was typed as returning only { source, destination }.
The backend actually returns { moved: int, errors: list[str], source, destination }
(backend/routes/models.py:140, 168). Widen the TS return type so the check
typechecks under the new CI gate from #418.

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-18 03:14:08 -07:00
882cabc7d2 fix: warn user when no models to migrate during storage change (#433)
When user attempts to change model storage location with no models
downloaded, the migration API returns moved=0 early. Previously the UI
would still call setCustomModelsDir() and restart the server, causing
unexpected behavior (hang/connection lost).

This change checks migrationResult.moved === 0 and shows a helpful
toast message instead of proceeding with the storage change.

Fixes: #426

Co-authored-by: fuleinist <[email protected]>
2026-04-18 03:13:06 -07:00
9c76b5de2c docs: clarify paralinguistic tag support in quick start (#450)
Co-authored-by: txhno <[email protected]>
2026-04-18 03:12:46 -07:00
Cocoon-BreakandGitHub 4560b7378a fix: delete version rows and files in delete_generations_by_profile (Closes #446) (#447)
Signed-off-by: Cocoon-Break <[email protected]>
2026-04-18 03:12:39 -07:00
高巨龙andGitHub abd9943430 Fix migration dialog hanging when no models are present (#439)
When migrating model path with an empty cache, backend returned early without emitting migration completion SSE, causing frontend overlay to hang. This patch emits complete status for empty migrations.
2026-04-18 03:12:32 -07:00
c8cb12f1bc fix(build): repair frozen-binary imports for kokoro, chatterbox-multilingual, scipy, transformers (#438)
* fix(build): bundle kokoro source files for transformers runtime introspection

transformers opens .py source files at runtime to check attention/MoE
implementation via regex (e.g. _can_set_attn_implementation). PyInstaller's
--hidden-import only bundles .pyc bytecode, so kokoro/modules.py was missing
from the bundle causing a FileNotFoundError on Kokoro model load.

Switch from individual --hidden-import entries to --collect-all kokoro in both
build_binary.py and voicebox-server.spec. The kokoro package is 172K so no
meaningful bundle size impact.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(build): use SPECPATH for runtime hook instead of hardcoded absolute path

The linter expanded runtime_hooks=[] to an absolute /Users/... path which
would break CI and other dev machines. Use os.path.join(SPECPATH, ...) to
mirror the relative approach in build_binary.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(build): runtime hook to work around PyInstaller + Python 3.12 import breakages

Four distinct bundling-specific crashes blocked Kokoro and Qwen CustomVoice
from loading in the frozen binary:

1. torch._dynamo import triggered via class-body decorators
   (@torch._dynamo.allow_in_graph on PreTrainedModel,
   @torch.compiler.disable in flex_attention) pulls in torch._numpy._ufuncs
   which crashes on module load with NameError: name 'name' is not defined.

2. AlbertModel (Kokoro) triggers @auto_docstring -> modeling_auto ->
   GenerationMixin -> candidate_generator -> sklearn -> scipy, which hits
   the same class of bug in scipy.stats._distn_infrastructure (NameError:
   name 'obj' is not defined).

3. AutoModel (Qwen) pulls the same sklearn -> scipy chain directly.

4. librosa (required by most TTS engines) -> scipy.signal -> scipy.stats
   hits the _distn_infrastructure crash regardless of the transformers
   stubs above.

The root cause of (1) and (4) is that PyInstaller's frozen importer runs
module-level `for X in [<list-comp using dir()>]:` loops with an empty
iterable, leaving the loop variable unbound. Trailing `del obj` / unrelated
references then crash.

Fix: a single runtime hook (pyi_rth_torch_compiler_disable.py) installs:

- sys.modules stubs for torch._dynamo and torch._dynamo.config, plus a
  meta-path finder for torch._dynamo.* submodules — voicebox never uses
  torch.compile/dynamo for inference, so a permissive no-op stub (callable
  as decorator, falsey as predicate, context-manager-safe for
  TransformGetItemToIndex) is drop-in safe.
- meta-path finder stubs for transformers.utils.auto_docstring and
  transformers.generation.candidate_generator — both import-chain
  short-circuits; docstrings and speculative decoding aren't used for TTS.
- meta-path finder for scipy.stats._distn_infrastructure that reads the
  real .py source via the wrapped loader's get_source(), replaces the
  bundling-broken `del obj` with `globals().pop('obj', None)`, and
  compile+exec's the patched source. This keeps the real scipy module
  intact so librosa and everything downstream works normally.

Supporting changes:

- backend/pyi_hooks/hook-scipy.stats._distn_infrastructure.py sets
  module_collection_mode = "pyz+py" so the .py source is actually in the
  bundle for the runtime patcher to read.
- build_binary.py and voicebox-server.spec register the runtime hook and
  the new hooks dir.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(build): force transformers torch<2.6 mask path and bundle spacy_pkuseg

- patch transformers.masking_utils to set _is_torch_greater_or_equal_than_2_6
  = False, forcing sdpa_mask_older_torch and avoiding the vmap .item() crash
  that breaks Qwen CustomVoice generation (our torch._dynamo stub can't
  reproduce TransformGetItemToIndex's graph transform).
- add PyInstaller hook to bundle transformers.masking_utils .py source so the
  runtime finder can source-patch it.
- --collect-all spacy_pkuseg so Chatterbox Multilingual can load its Chinese
  segmenter (dicts/default.pkl + native .so extensions).
- add per-finder install diagnostics + _HOOK_VERSION marker to make future
  bundle-only regressions easier to triage.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(build): pass PyInstaller hook paths relative so .spec is portable

Absolute paths ended up in the auto-regenerated voicebox-server.spec
because build_binary.py prefixed every --runtime-hook and
--additional-hooks-dir with str(backend_dir / ...). That broke builds
on any machine whose checkout wasn't at /Users/jamie/... and anyone
invoking pyinstaller voicebox-server.spec directly.

os.chdir(backend_dir) already runs before PyInstaller (same reason
server.py works as a bare filename), so the backend_dir prefix is
unnecessary. Drop it so the generated spec references pyi_hooks/,
pyi_rth_numpy_compat.py, pyi_rth_torch_compiler_disable.py as repo-
relative paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-18 03:12:13 -07:00
Andrew BarnesandGitHub 54a3bf322e fix: add generation cancellation flow (#444) 2026-04-18 02:51:39 -07:00
476abe07fc fix(paths): strip legacy "data/" prefix when resolving stored paths (#440)
0.3.0 sometimes stored relative media paths with the data-dir name baked in
(e.g. "data/profiles/<uuid>/sample.wav"). resolve_storage_path joined those
directly with _data_dir, producing "<data_dir>/data/profiles/..." — a
spurious double nest that breaks file reads after upgrading to 0.4.0.

The 0.4.0 startup migration didn't catch it because resolve_storage_path
produced the buggy double-nested path, to_storage_path saw "data" at the
first (legitimate) index, and the normalized value matched the stored value
so the row was skipped.

Strip any leading "data/" component before joining. This unblocks runtime
reads and lets _normalize_storage_paths rewrite the affected rows on next
startup — no manual migration needed.

Fixes "No such file or directory: '<data_dir>/data/profiles/...'" and
associated 404s on GET /audio/<id> after upgrading from 0.3.0 to 0.4.0.

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-17 17:53:11 -07:00
James PineandClaude Opus 4.6 67bf8e906a docs/landing: update for 0.4.0 — new engines, GPU docs, donate button, voice docs restructure
Docs:
- Add gpu-acceleration.mdx (all 9 platform/GPU combos, CUDA backend swap, Blackwell, XPU, troubleshooting)
- Add preset-voices.mdx (Kokoro 50 voices, Qwen CustomVoice 9 voices, instruct mode docs)
- Restructure voice-cloning.mdx to cover all 5 cloning engines with comparison table
- Restructure creating-voice-profiles.mdx around cloned vs preset workflows
- Update voice-profiles.mdx schema with voice_type discriminator, preset/design columns

Landing:
- Add 3 new engine cards (Qwen CustomVoice, HumeAI TADA, Kokoro) to Multi-Engine section
- Add Donate button (Buy Me a Coffee) to navbar and footer
- Add DONATE_URL constant

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-04-16 19:51:43 -07:00
95 changed files with 5403 additions and 3669 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.4.0
current_version = 0.4.1
commit = True
tag = True
tag_name = v{new_version}
+26
View File
@@ -0,0 +1,26 @@
name: CI
on:
pull_request:
push:
branches:
- main
jobs:
frontend-quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Typecheck app + web
run: bun run typecheck
- name: Build web smoke test
run: bun run build:web
+9
View File
@@ -68,6 +68,15 @@ jobs:
if: matrix.backend == 'mlx'
run: |
pip install -r backend/requirements-mlx.txt
# mlx-audio>=0.3.1 and mlx-lm>=0.31.1 both declare transformers>=5.x,
# which conflicts with our 4.57.x cap. The runtime APIs we use work
# fine on transformers 4.57.x in practice (verified in dev), so install
# them --no-deps. mlx-audio's other runtime deps (huggingface_hub,
# librosa, numpy, numba, pyloudnorm) are already in requirements.txt;
# the rest (sounddevice, miniaudio, protobuf, sentencepiece, pyyaml,
# jinja2) are pulled in by other engines.
pip install --no-deps mlx-lm==0.31.1
pip install --no-deps mlx-audio==0.4.1
- name: Build Python server (Linux/macOS)
if: matrix.platform != 'windows-latest'
+5
View File
@@ -63,3 +63,8 @@ nul
tmp/
temp/
*.tmp
# E2E test artifacts
backend/tests/results/
backend/tests/fixtures/reference_voice.wav
backend/tests/fixtures/reference_voice.txt
+42 -1
View File
@@ -7,6 +7,46 @@
## [Unreleased]
## [0.4.1] - 2026-04-18
A fast follow-up to 0.4.0 focused on making the new engines actually load in the production binary — plus generation cancellation, Linux system-audio capture, and the repo's first PR-time type check. Five first-time contributors shipped in this release.
0.4.0 introduced three new TTS engines, but the frozen PyInstaller binary tripped over several Python-ecosystem quirks that don't show up in the dev venv: `transformers` opening `.py` sources at runtime, `scipy.stats._distn_infrastructure` hitting a frozen-importer `NameError`, and `chatterbox-multilingual` failing to find its Chinese segmenter dictionary. This release patches all of those in one sweep.
### Frozen-Binary Reliability ([#438](https://github.com/jamiepine/voicebox/pull/438))
- **Kokoro** now bundles `.py` sources alongside `.pyc` via `--collect-all kokoro` so `transformers`' `_can_set_attn_implementation` regex scan can read them — previously `FileNotFoundError: kokoro/modules.py` killed Kokoro loading in production builds
- **Chatterbox Multilingual** now bundles `spacy_pkuseg/dicts/default.pkl` and the package's native `.so` extensions via `--collect-all spacy_pkuseg` — previously the Chinese word segmenter crashed with `FileNotFoundError` on first load
- **scipy.stats._distn_infrastructure** — new runtime hook source-patches the trailing `del obj` (which raises `NameError` under PyInstaller's frozen importer because the preceding list comprehension evaluates empty) to `globals().pop('obj', None)`, unblocking `librosa` → `scipy.signal` → `scipy.stats` for every TTS engine that depends on librosa
- **transformers.masking_utils** — same runtime hook forces `_is_torch_greater_or_equal_than_2_6 = False` so the older `sdpa_mask_older_torch` path is selected; the 2.6+ path uses `TransformGetItemToIndex()`, a real `torch._dynamo` graph transform our permissive stub can't reproduce
- **torch._dynamo** — no-op stub replaces the real module before `transformers` imports it, preventing the `torch._numpy._ufuncs` import crash (`NameError: name 'name' is not defined`) that blocked Kokoro and every engine pulling in `flex_attention`
- `.spec` paths are now repo-relative instead of absolute, so the generated spec is portable across machines and CI
### Generation
- **Cancel queued or running generations** ([#444](https://github.com/jamiepine/voicebox/pull/444)) — new `/generate/{id}/cancel` endpoint and a Stop button on the history row while generating. The serial queue now tracks per-ID state (queued / running / cancelled) so queued jobs are skipped before the worker picks them up and running jobs are `.cancel()`-ed mid-flight; `run_generation` catches `CancelledError` and marks the row `failed` with a "cancelled" error.
- **Legacy `data/` path prefix resolution** ([#440](https://github.com/jamiepine/voicebox/pull/440)) — generations stored with the old `data/` prefix under pre-0.4 installs now resolve correctly after the storage root moved, fixing 404s for historical audio.
### Model Migration
- Migration dialog no longer hangs when the cache is empty ([#439](https://github.com/jamiepine/voicebox/pull/439)) — the backend now emits a completion SSE event even when zero models are moved.
- Storage-change flow surfaces a toast when there's nothing to migrate ([#433](https://github.com/jamiepine/voicebox/pull/433)) instead of proceeding with a no-op move and restarting the server.
- Deleting all generations from a voice profile now deletes the associated version files and DB rows too ([#447](https://github.com/jamiepine/voicebox/pull/447)) — previously orphaned versions accumulated in storage.
### Platform
- **Linux system audio capture** ([#457](https://github.com/jamiepine/voicebox/pull/457)) — `cpal`'s ALSA backend doesn't expose PulseAudio/PipeWire monitor sources by name, so the previous device-name search never matched and silently fell back to the microphone. Detection now uses `pactl get-default-sink` + `pactl list short sources` and routes via `PULSE_SOURCE`, with the name-based search retained as a fallback when `pactl` is absent.
### Frontend CI
- First PR-time quality gate ([#418](https://github.com/jamiepine/voicebox/pull/418)) — new `.github/workflows/ci.yml` runs `bun run typecheck` + `bun run build:web` on every PR. Fixed pre-existing type issues that were being suppressed with `@ts-expect-error`, cleaned up a dep-array typo (`[platform.metadata.isTauricheckOnMountcheckForUpdates]`) in `useAutoUpdater`, and removed 100+ lines of dead `ModelItem` code from `ModelManagement.tsx`.
- Follow-up: widened `apiClient.migrateModels()` return type to include `moved` and `errors` so the storage-change handler typechecks against the real backend response ([#470](https://github.com/jamiepine/voicebox/pull/470)).
### Docs
- Clarified in the Quick Start + README that paralinguistic tags (`[laugh]`, `[sigh]`) only work with Chatterbox Turbo; other engines read them as literal text ([#450](https://github.com/jamiepine/voicebox/pull/450)).
### New Contributors
- [@Bortlesboat](https://github.com/Bortlesboat) — generation cancellation (#444)
- [@gaojulong](https://github.com/gaojulong) — migration dialog hang fix (#439)
- [@fuleinist](https://github.com/fuleinist) — migration no-op toast (#433)
- [@erionjuniordeandrade-a11y](https://github.com/erionjuniordeandrade-a11y) — frontend CI + type hardening (#418)
- [@estefrac](https://github.com/estefrac) — Linux pactl system-audio capture (#457)
## [0.4.0] - 2026-04-16
The biggest Voicebox release yet. Three new TTS engines bring the lineup to **seven** — HumeAI TADA, Kokoro 82M, and Qwen CustomVoice join Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo. GPU support broadens to Intel Arc (XPU) and NVIDIA Blackwell (RTX 50-series), with runtime diagnostics that warn when your PyTorch build doesn't match your GPU. The CUDA backend is now split into independently versioned server and library archives, so upgrading no longer redownloads 4 GB of PyTorch/CUDA DLLs.
@@ -555,7 +595,8 @@ The first public release of Voicebox — an open-source voice synthesis studio p
Tauri v2, React, TypeScript, Tailwind CSS, FastAPI, Qwen3-TTS, Whisper, SQLite
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.4.0...HEAD
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.4.1...HEAD
[0.4.1]: https://github.com/jamiepine/voicebox/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/jamiepine/voicebox/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/jamiepine/voicebox/compare/v0.2.3...v0.3.0
[0.2.3]: https://github.com/jamiepine/voicebox/compare/v0.2.2...v0.2.3
+4 -3
View File
@@ -260,7 +260,7 @@ voicebox/
### ✨ New Features
- Check the roadmap in README.md
- Check the roadmap in README.md and the engineering status in [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) before proposing work — it lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
- Discuss major features in an issue first
- Keep features focused and well-scoped
@@ -359,7 +359,7 @@ Releases are managed by maintainers:
## Troubleshooting
See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and solutions.
See [docs/content/docs/overview/troubleshooting.mdx](docs/content/docs/overview/troubleshooting.mdx) for common issues and solutions.
**Quick fixes:**
@@ -372,12 +372,13 @@ See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and sol
- Open an issue for bugs or feature requests
- Check existing issues and discussions
- Review the codebase to understand patterns
- See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues
- See [docs/content/docs/overview/troubleshooting.mdx](docs/content/docs/overview/troubleshooting.mdx) for common issues
## Additional Resources
- [README.md](README.md) - Project overview
- [backend/README.md](backend/README.md) - API documentation
- [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) - Living engineering roadmap: architecture, shipped vs in-flight work, prioritized open issues, candidate TTS engines under evaluation, architectural bottlenecks. Keep this updated when you ship significant features, close or backlog a model integration, or identify new bottlenecks.
- [docs/AUTOUPDATER_QUICKSTART.md](docs/AUTOUPDATER_QUICKSTART.md) - Auto-updater setup
- [SECURITY.md](SECURITY.md) - Security policy
- [CHANGELOG.md](CHANGELOG.md) - Version history
+23 -7
View File
@@ -23,6 +23,9 @@
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
</a>
<a href="https://deepwiki.com/jamiepine/voicebox">
<img src="https://img.shields.io/static/v1?label=Ask&message=DeepWiki&color=5B6EF7" alt="Ask DeepWiki" />
</a>
</p>
<p align="center">
@@ -30,7 +33,8 @@
<a href="https://docs.voicebox.sh">Docs</a> •
<a href="#download">Download</a> •
<a href="#features">Features</a> •
<a href="#api">API</a>
<a href="#api">API</a> •
<a href="docs/content/docs/overview/troubleshooting.mdx">Troubleshooting</a>
</p>
<br/>
@@ -59,13 +63,14 @@
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** — models and voice data stay on your machine
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** — zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** — auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** — multi-track timeline for conversations, podcasts, and narratives
- **API-first** — REST API for integrating voice synthesis into your own projects
@@ -87,25 +92,34 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
> **Linux** — Pre-built binaries are not yet available. See [voicebox.sh/linux-install](https://voicebox.sh/linux-install) for build-from-source instructions.
> **Having trouble?** See the [Troubleshooting Guide](docs/content/docs/overview/troubleshooting.mdx) for common install, generation, model-download, and GPU issues.
---
## Features
### Multi-Engine Voice Cloning
Five TTS engines with different strengths, switchable per-generation:
Seven TTS engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions ("speak slowly", "whisper") |
| **Qwen CustomVoice** | 10 | 9 curated preset voices with natural-language delivery control — no reference audio required |
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
| **Kokoro** | 8 | 50 curated preset voices, tiny 82M model, fast CPU inference |
### Emotions & Paralinguistic Tags
Type `/` in the text input to insert expressive tags that the model synthesizes inline with speech (Chatterbox Turbo):
Only **Chatterbox Turbo** interprets paralinguistic tags like `[laugh]` and
`[sigh]`. Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and HumeAI TADA read them
literally as text.
With **Chatterbox Turbo** selected, type `/` in the text input to open the tag
inserter and add expressive tags inline with speech:
`[laugh]` `[chuckle]` `[gasp]` `[cough]` `[sigh]` `[groan]` `[sniff]` `[shush]` `[clear throat]`
@@ -231,7 +245,7 @@ Full API documentation available at `http://localhost:17493/docs`.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
@@ -250,6 +264,8 @@ Full API documentation available at `http://localhost:17493/docs`.
| **Plugin Architecture** | Extend with custom models and effects |
| **Mobile Companion** | Control Voicebox from your phone |
For the **full engineering status, open-issue triage, and prioritized work queue**, see [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) — a living document that tracks what's shipped, what's in-flight, candidate TTS engines under evaluation, and why we've accepted or backlogged specific integrations.
---
## Development
+3 -3
View File
@@ -6,8 +6,8 @@ We release patches for security vulnerabilities. Which versions are eligible for
| Version | Supported |
| ------- | ------------------ |
| 0.1.x | :white_check_mark: |
| < 0.1 | :x: |
| 0.3.x | :white_check_mark: |
| < 0.3 | :x: |
## Reporting a Vulnerability
@@ -82,7 +82,7 @@ Timeline may vary based on severity and complexity.
## Security Updates
Security updates will be:
- Released as patch versions (e.g., 0.1.1)
- Released as patch versions (e.g., 0.3.2)
- Documented in CHANGELOG.md
- Announced via GitHub releases
- Automatically delivered via auto-updater
+2 -1
View File
@@ -1,11 +1,12 @@
{
"name": "@voicebox/app",
"version": "0.4.0",
"version": "0.4.1",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"typecheck": "tsc -p tsconfig.json --noEmit",
"preview": "vite preview",
"lint": "biome lint src",
"lint:fix": "biome lint --write src",
-3
View File
@@ -121,7 +121,6 @@ function App() {
console.log('Dev mode: Skipping auto-start of server (run it separately)');
setServerReady(true); // Mark as ready so UI doesn't show loading screen
// Mark that server was not started by app (so we don't try to stop it on close)
// @ts-expect-error - adding property to window
window.__voiceboxServerStartedByApp = false;
return;
}
@@ -144,13 +143,11 @@ function App() {
useServerStore.getState().setServerUrl(serverUrl);
setServerReady(true);
// Mark that we started the server (so we know to stop it on close)
// @ts-expect-error - adding property to window
window.__voiceboxServerStartedByApp = true;
})
.catch((error) => {
console.error('Failed to auto-start server:', error);
serverStartingRef.current = false;
// @ts-expect-error - adding property to window
window.__voiceboxServerStartedByApp = false;
// Only fall back to health-check polling when the error indicates the
+1 -1
View File
@@ -124,7 +124,7 @@ export function AudioTab() {
);
}
const handleChannelDelete = async (e, channelId) => {
const handleChannelDelete = async (e: React.MouseEvent, channelId: string) => {
e.stopPropagation();
if (await confirm('Delete this channel?')) {
deleteChannel.mutate(channelId);
@@ -146,7 +146,7 @@ export function FloatingGenerateBox({
// Cloned/designed profile with no default — ensure a compatible (non-preset) engine
const currentEngine = form.getValues('engine');
const presetEngines = new Set(['kokoro', 'qwen_custom_voice']);
if (presetEngines.has(currentEngine)) {
if (currentEngine && presetEngines.has(currentEngine)) {
form.setValue('engine', 'qwen');
}
}
+85 -56
View File
@@ -1,15 +1,14 @@
import { useQueryClient } from '@tanstack/react-query';
import { useMutation, useQueryClient } from '@tanstack/react-query';
import { AnimatePresence, motion } from 'framer-motion';
import {
AlignCenter,
AudioLines,
AudioWaveform,
Download,
FileArchive,
Loader2,
MoreHorizontal,
Play,
RotateCcw,
Square,
Star,
Trash2,
Wand2,
@@ -130,6 +129,23 @@ export function HistoryTable() {
const exportGeneration = useExportGeneration();
const exportGenerationAudio = useExportGenerationAudio();
const importGeneration = useImportGeneration();
const cancelGeneration = useMutation({
mutationFn: (generationId: string) => apiClient.cancelGeneration(generationId),
onSuccess: async (data) => {
await queryClient.invalidateQueries({ queryKey: ['history'] });
toast({
title: 'Cancelling generation',
description: data.message,
});
},
onError: (error) => {
toast({
title: 'Cancel failed',
description: error instanceof Error ? error.message : 'Could not cancel generation',
variant: 'destructive',
});
},
});
const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio);
@@ -483,6 +499,8 @@ export function HistoryTable() {
const isPlayable = !isGenerating && !isFailed;
const hasVersions = gen.versions && gen.versions.length > 1;
const isVersionsExpanded = expandedVersionsId === gen.id;
const isCancelling =
cancelGeneration.isPending && cancelGeneration.variables === gen.id;
return (
<div
key={gen.id}
@@ -631,60 +649,71 @@ export function HistoryTable() {
<Trash2 className="h-2 w-2" />
</Button>
</>
) : isGenerating ? (
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Cancel generation"
disabled={isCancelling}
onClick={() => cancelGeneration.mutate(gen.id)}
>
{isCancelling ? (
<Loader2 className="h-2 w-2 animate-spin" />
) : (
<Square className="h-2 w-2" />
)}
</Button>
) : (
<>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Actions"
disabled={isGenerating}
>
<MoreHorizontal className="h-2 w-2" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem
onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}
>
<Play className="mr-2 h-4 w-4" />
Play
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleDownloadAudio(gen.id, gen.text)}
disabled={exportGenerationAudio.isPending}
>
<Download className="mr-2 h-4 w-4" />
Export Audio
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleExportPackage(gen.id, gen.text)}
disabled={exportGeneration.isPending}
>
<FileArchive className="mr-2 h-4 w-4" />
Export Package
</DropdownMenuItem>
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
<Wand2 className="mr-2 h-4 w-4" />
Apply Effects
</DropdownMenuItem>
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
<RotateCcw className="mr-2 h-4 w-4" />
Regenerate
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
disabled={deleteGeneration.isPending}
// className="text-destructive focus:text-destructive"
>
<Trash2 className="mr-2 h-4 w-4" />
Delete
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="ghost"
size="icon"
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
aria-label="Actions"
disabled={isGenerating}
>
<MoreHorizontal className="h-2 w-2" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}>
<Play className="mr-2 h-4 w-4" />
Play
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleDownloadAudio(gen.id, gen.text)}
disabled={exportGenerationAudio.isPending}
>
<Download className="mr-2 h-4 w-4" />
Export Audio
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleExportPackage(gen.id, gen.text)}
disabled={exportGeneration.isPending}
>
<FileArchive className="mr-2 h-4 w-4" />
Export Package
</DropdownMenuItem>
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
<Wand2 className="mr-2 h-4 w-4" />
Apply Effects
</DropdownMenuItem>
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
<RotateCcw className="mr-2 h-4 w-4" />
Regenerate
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
disabled={deleteGeneration.isPending}
// className="text-destructive focus:text-destructive"
>
<Trash2 className="mr-2 h-4 w-4" />
Delete
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
)}
</div>
</div>
@@ -977,7 +977,19 @@ export function ModelManagement() {
});
try {
// Start the migration (background task)
await apiClient.migrateModels(newDir);
const migrationResult = await apiClient.migrateModels(newDir);
// If no models to migrate, warn user and skip the change
if (migrationResult.moved === 0) {
setMigrating(false);
setMigrationProgress(null);
toast({
title: 'No models to migrate',
description: 'Download at least one model before changing the storage location.',
});
setPendingMigrateDir(null);
return;
}
// Connect to SSE for progress
await new Promise<void>((resolve, reject) => {
@@ -1064,105 +1076,3 @@ export function ModelManagement() {
);
}
interface ModelItemProps {
model: {
model_name: string;
display_name: string;
downloaded: boolean;
downloading?: boolean; // From server - true if download in progress
size_mb?: number;
loaded: boolean;
};
onDownload: () => void;
onDelete: () => void;
isDownloading: boolean; // Local state - true if user just clicked download
formatSize: (sizeMb?: number) => string;
}
function ModelItem({ model, onDownload, onDelete, isDownloading, formatSize }: ModelItemProps) {
// Use server's downloading state OR local state (for immediate feedback before server updates)
const showDownloading = model.downloading || isDownloading;
const statusText = model.loaded
? 'Loaded'
: showDownloading
? 'Downloading'
: model.downloaded
? 'Downloaded'
: 'Not downloaded';
const sizeText =
model.downloaded && model.size_mb && !showDownloading ? `, ${formatSize(model.size_mb)}` : '';
const rowLabel = `${model.display_name}, ${statusText}${sizeText}. Use Tab to reach Download or Delete.`;
return (
<div
className="flex items-center justify-between p-3 border rounded-lg"
role="group"
tabIndex={0}
aria-label={rowLabel}
>
<div className="flex-1">
<div className="flex items-center gap-2">
<span className="font-medium text-sm">{model.display_name}</span>
{model.loaded && (
<Badge variant="default" className="text-xs">
Loaded
</Badge>
)}
{/* Only show Downloaded if actually downloaded AND not downloading */}
{model.downloaded && !model.loaded && !showDownloading && (
<Badge variant="secondary" className="text-xs">
Downloaded
</Badge>
)}
</div>
{model.downloaded && model.size_mb && !showDownloading && (
<div className="text-xs text-muted-foreground mt-1">
Size: {formatSize(model.size_mb)}
</div>
)}
</div>
<div className="flex items-center gap-2">
{model.downloaded && !showDownloading ? (
<div className="flex items-center gap-2">
<div className="flex items-center gap-1 text-sm text-muted-foreground">
<span>Ready</span>
</div>
<Button
size="sm"
onClick={onDelete}
variant="outline"
disabled={model.loaded}
title={model.loaded ? 'Unload model before deleting' : 'Delete model'}
aria-label={
model.loaded ? 'Unload model before deleting' : `Delete ${model.display_name}`
}
>
<Trash2 className="h-4 w-4" />
</Button>
</div>
) : showDownloading ? (
<Button
size="sm"
variant="outline"
disabled
aria-label={`${model.display_name} downloading`}
>
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
Downloading...
</Button>
) : (
<Button
size="sm"
onClick={onDownload}
variant="outline"
aria-label={`Download ${model.display_name}`}
>
<Download className="h-4 w-4 mr-2" />
Download
</Button>
)}
</div>
</div>
);
}
@@ -371,7 +371,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
}
}, [isResizing, handleResizeMove, handleResizeEnd]);
const handleTimelineClick = (e: React.MouseEvent<HTMLDivElement>) => {
const handleTimelineClick = (e: React.MouseEvent<HTMLElement>) => {
if (!tracksRef.current || draggingItem || trimmingItem) return;
const rect = tracksRef.current.getBoundingClientRect();
const x = e.clientX - rect.left + tracksRef.current.scrollLeft;
+13 -4
View File
@@ -5,7 +5,15 @@ import type { UpdateStatus } from '@/platform/types';
// Re-export UpdateStatus for backwards compatibility
export type { UpdateStatus };
export function useAutoUpdater(checkOnMount = false) {
interface UseAutoUpdaterOptions {
checkOnMount?: boolean;
showToast?: boolean;
}
export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false) {
const { checkOnMount } =
typeof options === 'boolean' ? { checkOnMount: options } : { checkOnMount: options.checkOnMount ?? false };
const platform = usePlatform();
const [status, setStatus] = useState<UpdateStatus>(platform.updater.getStatus());
const hasCheckedRef = useRef(false);
@@ -38,10 +46,11 @@ export function useAutoUpdater(checkOnMount = false) {
useEffect(() => {
if (checkOnMount && platform.metadata.isTauri && !hasCheckedRef.current) {
hasCheckedRef.current = true;
checkForUpdates();
checkForUpdates().catch((error) => {
console.error('Auto update check failed:', error);
});
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
return {
status,
+1 -1
View File
@@ -73,7 +73,7 @@ export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false)
}
// Empty dependency array - only run once on mount
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
// Show toast when update is available
useEffect(() => {
+9 -1
View File
@@ -234,6 +234,12 @@ class ApiClient {
});
}
async cancelGeneration(generationId: string): Promise<{ message: string }> {
return this.request<{ message: string }>(`/generate/${generationId}/cancel`, {
method: 'POST',
});
}
async regenerateGeneration(generationId: string): Promise<GenerationResponse> {
return this.request<GenerationResponse>(`/generate/${generationId}/regenerate`, {
method: 'POST',
@@ -384,7 +390,9 @@ class ApiClient {
return this.request<{ path: string }>('/models/cache-dir');
}
async migrateModels(destination: string): Promise<{ source: string; destination: string }> {
async migrateModels(
destination: string,
): Promise<{ source: string; destination: string; moved: number; errors: string[] }> {
return this.request('/models/migrate', {
method: 'POST',
body: JSON.stringify({ destination }),
+1 -1
View File
@@ -1,3 +1,3 @@
# Backend package
__version__ = "0.4.0"
__version__ = "0.4.1"
+23 -12
View File
@@ -55,10 +55,23 @@ def build_server(cuda=False):
# numpy 2.x / torch ABI mismatch fix: install memmove fallback for
# torch.from_numpy() before the app starts. Runtime hooks run after
# FrozenImporter is registered so frozen torch/numpy are importable.
# Paths are passed relative to backend_dir because os.chdir(backend_dir)
# runs before PyInstaller. Absolute paths would get baked into the
# generated .spec, breaking reproducible builds on other machines / CI.
args.extend(
[
"--runtime-hook",
str(backend_dir / "pyi_rth_numpy_compat.py"),
"pyi_rth_numpy_compat.py",
# Stub torch.compiler.disable before transformers imports
# flex_attention, which otherwise triggers torch._dynamo →
# torch._numpy._ufuncs and crashes at module load under
# PyInstaller. See pyi_rth_torch_compiler_disable.py.
"--runtime-hook",
"pyi_rth_torch_compiler_disable.py",
# Per-module collection overrides (e.g. forcing scipy.stats._distn_infrastructure
# to bundle .py source alongside .pyc so the runtime hook can source-patch it).
"--additional-hooks-dir",
"pyi_hooks",
]
)
@@ -125,6 +138,11 @@ def build_server(cuda=False):
"backend.backends.chatterbox_backend",
"--hidden-import",
"backend.backends.chatterbox_turbo_backend",
# chatterbox multilingual uses spacy_pkuseg for Chinese word
# segmentation, which ships pickled dict files (dicts/default.pkl)
# and native .so extensions that --hidden-import alone won't bundle.
"--collect-all",
"spacy_pkuseg",
"--hidden-import",
"backend.backends.luxtts_backend",
"--hidden-import",
@@ -241,20 +259,13 @@ def build_server(cuda=False):
"--collect-submodules",
"tada",
# Kokoro 82M — lightweight TTS engine using misaki G2P
# collect-all is required because transformers introspects .py source
# files at runtime (e.g. _can_set_attn_implementation opens the class
# file); hidden-import alone only bundles bytecode.
"--hidden-import",
"backend.backends.kokoro_backend",
"--hidden-import",
"--collect-all",
"kokoro",
"--hidden-import",
"kokoro.pipeline",
"--hidden-import",
"kokoro.model",
"--hidden-import",
"kokoro.istftnet",
"--hidden-import",
"kokoro.modules",
"--hidden-import",
"kokoro.custom_stft",
# misaki ships G2P data files (dictionaries, phoneme tables)
# that must be bundled for espeak/en/ja/zh G2P to work
"--collect-all",
+8
View File
@@ -89,6 +89,14 @@ def resolve_storage_path(path: str | Path | None) -> Path | None:
return stored_path
# 0.3.0 records sometimes stored relative paths with the data-dir name
# baked in (e.g. "data/profiles/..."). Joining those directly with
# _data_dir produces a spurious "<data_dir>/data/profiles/..." nest.
if stored_path.parts and stored_path.parts[0] == "data":
stored_path = (
Path(*stored_path.parts[1:]) if len(stored_path.parts) > 1 else Path()
)
return (_data_dir / stored_path).resolve()
@@ -0,0 +1,12 @@
"""
Force scipy.stats._distn_infrastructure to be bundled with its .py source file
alongside the .pyc bytecode.
The runtime hook in backend/pyi_rth_torch_compiler_disable.py patches this
module's source at load time (the module has a `del obj` at line 369 that
raises NameError under PyInstaller's frozen importer). That patch reads the
source via loader.get_source(), which only works if the .py file was
actually collected into the bundle.
"""
module_collection_mode = "pyz+py"
@@ -0,0 +1,11 @@
"""
Force transformers.masking_utils to be bundled with its .py source alongside
the .pyc bytecode so the runtime hook in
backend/pyi_rth_torch_compiler_disable.py can source-patch it.
The patch forces the torch<2.6 code path, bypassing `with TransformGetItemToIndex()`
which our torch._dynamo no-op stub can't implement for real — the real context
manager uses dynamo graph transforms to avoid `.item()` calls inside vmap.
"""
module_collection_mode = "pyz+py"
+540
View File
@@ -0,0 +1,540 @@
"""
PyInstaller runtime hook: stub torch._dynamo to a no-op module.
Problem
-------
transformers triggers torch._dynamo import at module-load time (not just
when torch.compile is called) via class-body decorators:
transformers/modeling_utils.py:1984
@torch._dynamo.allow_in_graph
class PreTrainedModel(...)
transformers/integrations/flex_attention.py:61
@torch.compiler.disable(recursive=False)
class WrappedFlexAttention...
The attribute access triggers torch.__getattr__ -> importlib.import_module
-> torch._dynamo -> torch._dynamo.utils imports torch._numpy ->
torch._numpy._ndarray imports torch._numpy._ufuncs, which crashes under
PyInstaller with:
File "torch/_numpy/_ufuncs.py", line 235, in <module>
vars()[name] = deco_binary_ufunc(ufunc)
NameError: name 'name' is not defined
(The module-level `for name in _binary: vars()[name] = ...` pattern works
in a regular venv but fails in the PyInstaller bundle. Root cause is in
PyInstaller's importer / bytecode pipeline and not easily fixed upstream.)
Surfaces as Kokoro failing to load when `from transformers import AlbertModel`
trips the decorator chain.
Fix
---
voicebox never uses torch.compile / torch._dynamo for inference, so we
replace torch._dynamo with a no-op stub module before transformers is
imported. Any attribute access on the stub returns a pass-through callable,
so `@torch._dynamo.allow_in_graph`, `torch._dynamo.is_compiling()`,
`torch._dynamo.mark_static_address(...)`, etc. all work.
This hook is pure sys.modules manipulation — we deliberately do NOT import
torch here. Runtime hooks run before the app starts and before
pyi_rth_numpy_compat has had a chance to patch torch.from_numpy (it runs
in a background thread, waiting for torch to appear in sys.modules).
Eager-importing torch at hook time would trip the numpy ABI issue and
kill the server process at startup.
torch.compiler.disable does not need a separate stub: its implementation
is effectively `import torch._dynamo; return torch._dynamo.disable(...)`,
and since our stub is in sys.modules, that call resolves to our no-op
_NoopDecorator pass-through.
"""
import os
import sys
import tempfile
import types
# Diagnostics — log hook activity to a file alongside the bundle so we can
# see what's happening when the server is run as a sidecar (no stdout for
# runtime hook prints). Safe no-op if the file can't be written.
_DIAG_PATH = os.path.join(tempfile.gettempdir(), "voicebox_rt_hook.log")
def _diag(msg: str) -> None:
try:
with open(_DIAG_PATH, "a", encoding="utf-8") as f:
f.write(msg + "\n")
except Exception:
pass
_HOOK_VERSION = "v6-masking-utils-finder"
_diag(f"=== runtime hook load @ pid={os.getpid()} version={_HOOK_VERSION} ===")
class _NoopDecorator:
"""Multi-role no-op: decorator, falsey predicate, and context manager.
Returned from calls like `torch._dynamo.disable()` (decorator),
`torch._dynamo.is_compiling()` (predicate used in `if not ...`), and
`with torch._dynamo._trace_wrapped_higher_order_op.TransformGetItemToIndex():`
(context manager used to scope an fx graph transformation).
By implementing __call__, __bool__, __enter__, __exit__, and __iter__ we
cover every use pattern we've seen transformers/torch use on a stubbed
object. Anything we haven't covered will raise a clearer error than a
silent wrong-result.
"""
__slots__ = ()
def __call__(self, fn=None, *args, **kwargs):
return fn
def __bool__(self) -> bool:
return False
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
return False # don't suppress exceptions
def __iter__(self):
return iter(())
_noop_decorator_singleton = _NoopDecorator()
def _noop_callable(*args, **kwargs):
# Direct-decorator use: @torch._dynamo.foo (no parens) — fn is positional
if len(args) == 1 and callable(args[0]) and not kwargs:
return args[0]
# Side-effect call with non-callable arg(s), e.g. mark_static_address(tensor)
return _noop_decorator_singleton
class _NoopDynamoModule(types.ModuleType):
"""Permissive stub: every attribute is a pass-through callable.
Covers attributes transformers hits at import time (allow_in_graph) and
runtime (is_compiling, mark_static_address, reset, disable, ...).
Dunder attributes (__file__, __spec__, __loader__, ...) raise
AttributeError so probes like inspect.getmodule() — which does
`hasattr(m, '__file__')` then `os.path.normpath(m.__file__)` — see the
module as having no source file and fall through to its normal
handling, instead of receiving a function and blowing up.
"""
def __getattr__(self, name: str):
if name.startswith("__") and name.endswith("__"):
raise AttributeError(name)
return _noop_callable
class _DynamoLoader:
"""Loader used by _DynamoMetaPathFinder to materialise stub submodules."""
def create_module(self, spec):
return _NoopDynamoModule(spec.name)
def exec_module(self, module):
# Mark every stub submodule as a package so deeper submodule imports
# (`from torch._dynamo.X.Y import Z`) keep working.
module.__path__ = []
class _DynamoMetaPathFinder:
"""Resolve any `torch._dynamo.X[.Y...]` import to a no-op stub module.
Without this, `from torch._dynamo._trace_wrapped_higher_order_op import X`
fails even with torch._dynamo pre-populated in sys.modules — Python's
import machinery checks the parent's __path__ and then looks up the
child, and we need to provide both.
"""
def find_spec(self, fullname, path=None, target=None):
if fullname == "torch._dynamo":
return None # handled by the pre-populated sys.modules entry
if not fullname.startswith("torch._dynamo."):
return None
from importlib.machinery import ModuleSpec
return ModuleSpec(fullname, _DynamoLoader(), is_package=True)
class _TransformersStubFinder:
"""Replace specific transformers submodules with no-op stubs.
Two modules are targeted:
1. transformers.utils.auto_docstring
The real @auto_docstring decorator loads
transformers.models.auto.modeling_auto just to build example docstrings,
which drags in GenerationMixin -> candidate_generator -> sklearn.metrics
-> scipy.stats._distn_infrastructure and trips (2) below. Docstrings
aren't functional for inference, so a pass-through decorator is safe.
2. transformers.generation.candidate_generator
Imported at module scope by transformers.generation.utils. It does
`from sklearn.metrics import roc_curve` at module load, which triggers:
File "scipy/stats/_distn_infrastructure.py", line 369, in <module>
NameError: name 'obj' is not defined
This is a PyInstaller-specific module-load bug (same class as the
torch._numpy._ufuncs crash) where a module-level `for obj in [s for s
in dir() if ...]` loop evaluates to empty in the bundle, leaving `obj`
unbound before `del obj`.
The exports (AssistedCandidateGenerator, EarlyExitCandidateGenerator,
etc.) are speculative-decoding helpers voicebox's TTS engines do not
use; a no-op stub module satisfies the imports.
"""
_STUBBED_MODULES = frozenset(
{
"transformers.utils.auto_docstring",
"transformers.generation.candidate_generator",
}
)
def find_spec(self, fullname, path=None, target=None):
if fullname not in self._STUBBED_MODULES:
return None
from importlib.machinery import ModuleSpec
return ModuleSpec(fullname, _NoopStubLoader(), is_package=False)
class _NoopStubLoader:
def create_module(self, spec):
return _NoopDynamoModule(spec.name)
def exec_module(self, module):
# _NoopDynamoModule.__getattr__ already answers every non-dunder
# attribute with a pass-through callable, which satisfies
# `from stubbed_module import X` for any X.
pass
def _patch_scipy_distn_source(source: str) -> str:
"""Replace the unsafe `del obj` with a no-op that survives when obj is unbound.
Returns the input unchanged if the target line isn't found (e.g. scipy
version has changed).
"""
target = "\ndel obj\n"
replacement = "\nglobals().pop('obj', None)\n"
if target in source:
return source.replace(target, replacement, 1)
return source
def _patch_masking_utils_source(source: str) -> str:
"""Force torch<2.6 code path in transformers.masking_utils.
The torch>=2.6 path uses `with TransformGetItemToIndex():` to allow
`.item()` calls inside vmap. That context manager is implemented via
torch._dynamo graph transforms, which our stub doesn't reproduce — it's
a no-op. The inner `_vmap_for_bhqkv` then crashes with:
RuntimeError: vmap: It looks like you're calling .item() on a Tensor.
Forcing the torch<2.6 flag off selects sdpa_mask_older_torch which uses
a different vmap pattern that does not hit .item() and does not need
TransformGetItemToIndex.
"""
target = 'is_torch_greater_or_equal("2.6", accept_dev=True)'
# Find the specific line that assigns _is_torch_greater_or_equal_than_2_6
if "_is_torch_greater_or_equal_than_2_6 = " + target in source:
return source.replace(
"_is_torch_greater_or_equal_than_2_6 = " + target,
"_is_torch_greater_or_equal_than_2_6 = False",
1,
)
return source
class _SourcePatchingFinder:
"""Generic delegate-and-wrap meta-path finder that patches a module's
source before exec'ing.
Subclasses declare `target` (module fullname) and `patch` (str->str).
Requires the target module's .py source to be bundled (use a PyInstaller
hook setting module_collection_mode = "pyz+py").
"""
target: str
patch_fn: callable = None
def find_spec(self, fullname, path=None, target=None):
if fullname != self.target:
return None
for finder in sys.meta_path:
if finder is self:
continue
find = getattr(finder, "find_spec", None)
if find is None:
continue
try:
real_spec = find(fullname, path, target)
except Exception:
continue
if real_spec is None or real_spec.loader is None:
continue
real_spec.loader = _SourcePatchLoader(real_spec.loader, self.patch_fn)
return real_spec
return None
class _SourcePatchLoader:
"""Delegate loader that reads source via get_source, applies a patch, and
compile/exec's the patched text into module.__dict__.
"""
def __init__(self, inner, patch_fn):
self._inner = inner
self._patch_fn = patch_fn
def __getattr__(self, name):
return getattr(self._inner, name)
def create_module(self, spec):
return self._inner.create_module(spec)
def exec_module(self, module):
source = None
try:
source = self._inner.get_source(module.__name__)
except Exception as e:
_diag(f"[source-patch] get_source({module.__name__}) failed: {e!r}")
if not source:
_diag(
f"[source-patch] no source for {module.__name__}; "
"falling back to inner exec_module (patch NOT applied)"
)
self._inner.exec_module(module)
return
patched = self._patch_fn(source)
_diag(
f"[source-patch] {module.__name__}: "
f"patched={patched is not source}, len={len(patched)}"
)
spec = module.__spec__
if spec is not None and spec.submodule_search_locations is not None:
module.__path__ = spec.submodule_search_locations
filename = getattr(self._inner, "path", module.__name__)
exec(compile(patched, filename, "exec"), module.__dict__)
_diag(f"[source-patch] {module.__name__} OK")
class _MaskingUtilsFinder(_SourcePatchingFinder):
target = "transformers.masking_utils"
patch_fn = staticmethod(_patch_masking_utils_source)
class _ScipyDistnPatchingFinder:
"""Delegate-and-wrap finder for scipy.stats._distn_infrastructure.
That module ends with:
for obj in [s for s in dir() if s.startswith('_doc_')]:
exec('del ' + obj)
del obj
In the PyInstaller bundle the list comprehension evaluates to empty
(module-level dir() under the frozen importer returns a different scope
than CPython's normal module-exec path — same class of bug as the
torch._numpy._ufuncs crash). The for loop body doesn't run, `obj` is
never bound, and the trailing `del obj` raises NameError at module load.
This kills every downstream module: librosa (needed by nearly every TTS
engine for mel filters) -> scipy.signal -> scipy.stats -> here.
Workaround: delegate to the real loader, but pre-bind `obj = None` in the
module namespace before its bytecode runs. If the for loop executes, each
iteration overwrites the sentinel via STORE_NAME (normal behaviour). If it
doesn't, `del obj` removes the sentinel and module load succeeds. The
`_doc_*` cleanup this line was meant to do is purely cosmetic — those vars
stay in the module namespace but nothing references them after this point.
"""
_TARGET = "scipy.stats._distn_infrastructure"
def find_spec(self, fullname, path=None, target=None):
if fullname != self._TARGET:
return None
_diag(f"[scipy-finder] match: {fullname}, path={path!r}")
# Delegate to the other finders to locate the real spec
for finder in sys.meta_path:
if finder is self:
continue
find = getattr(finder, "find_spec", None)
if find is None:
continue
try:
real_spec = find(fullname, path, target)
except Exception as e:
_diag(f"[scipy-finder] inner finder {type(finder).__name__} raised: {e}")
continue
if real_spec is None:
continue
if real_spec.loader is None:
_diag(f"[scipy-finder] {type(finder).__name__} returned spec with loader=None")
continue
_diag(
f"[scipy-finder] wrapped loader from "
f"{type(finder).__name__} -> {type(real_spec.loader).__name__}"
)
real_spec.loader = _ScipyDistnPrebindLoader(real_spec.loader)
return real_spec
_diag("[scipy-finder] NO inner finder returned a spec")
return None
class _ScipyDistnPrebindLoader:
"""Thin wrapper that pre-binds `obj = None` before delegating to the
real PyInstaller loader.
Every other attribute/method delegates to the inner loader — PyiFrozenLoader
is a rich FileLoader/ExecutionLoader with get_code/get_source/get_filename/
is_package/get_resource_reader/etc., any of which Python's import machinery
or 3rd-party code may call on spec.loader. Forwarding via __getattr__
avoids breaking any of those paths (and preserves @_check_name contracts
because the decorated methods run on the inner instance where self.name
matches spec.name).
"""
def __init__(self, inner):
self._inner = inner
def __getattr__(self, name):
# __getattr__ fires only for attrs not already on self, so delegate
# everything that isn't create_module/exec_module (or __getattr__/init).
return getattr(self._inner, name)
def create_module(self, spec):
return self._inner.create_module(spec)
def exec_module(self, module):
# Compile scipy's module source with the problematic line patched.
#
# The real module ends with:
# for obj in [s for s in dir() if s.startswith('_doc_')]:
# exec('del ' + obj)
# del obj
#
# Under PyInstaller's frozen importer, `del obj` raises NameError
# even when we pre-populate module.__dict__['obj'] — the pre-compiled
# .pyc bytecode interacts with the frame setup differently than a
# fresh compile() from source. Easiest robust fix: read the source
# and replace `del obj` with a safe variant before compiling.
#
# Requires the .py source to be bundled alongside the .pyc — see
# backend/pyi_hooks/hook-scipy.stats._distn_infrastructure.py.
source = None
try:
source = self._inner.get_source(module.__name__)
except Exception as e:
_diag(f"[scipy-loader] get_source failed: {e!r}")
if source:
patched = _patch_scipy_distn_source(source)
_diag(
f"[scipy-loader] source-patch path: patched={patched is not source}, "
f"len={len(patched)}"
)
spec = module.__spec__
if spec is not None and spec.submodule_search_locations is not None:
module.__path__ = spec.submodule_search_locations
filename = getattr(self._inner, "path", module.__name__)
bytecode = compile(patched, filename, "exec")
try:
exec(bytecode, module.__dict__)
except Exception as e:
_diag(f"[scipy-loader] patched exec raised {type(e).__name__}: {e!r}")
raise
_diag(f"[scipy-loader] exec_module {module.__name__} OK (source-patched)")
return
# No source available — fall back to the pre-bind approach. This is
# best-effort; if the frozen .pyc really does see a different `obj`
# slot, this will still crash, but we've done all we can without
# source.
_diag("[scipy-loader] no source available; falling back to pre-bind")
module.__dict__["obj"] = None
self._inner.exec_module(module)
def _install_dynamo_stub() -> None:
stub = _NoopDynamoModule("torch._dynamo")
# Mark as a package so `from torch._dynamo.X import Y` imports work
# (Python's import machinery checks parent.__path__ before looking up
# the child).
stub.__path__ = []
# torch._dynamo.config is accessed as a nested attribute namespace
# (e.g. `torch._dynamo.config.capture_scalar_outputs = True`), so use
# a permissive module so any attr read returns a no-op and sets succeed.
stub.config = _NoopDynamoModule("torch._dynamo.config")
stub.config.__path__ = []
sys.modules["torch._dynamo"] = stub
sys.modules["torch._dynamo.config"] = stub.config
# Finders:
# - torch._dynamo.* submodules -> no-op stubs
# - transformers.utils.auto_docstring and
# transformers.generation.candidate_generator -> no-op stubs (both
# paths reach sklearn -> scipy.stats which trips a separate crash)
# - scipy.stats._distn_infrastructure -> real load with `obj` pre-bound,
# so librosa -> scipy.signal -> scipy.stats loads cleanly
for _FinderCls in (
_DynamoMetaPathFinder,
_TransformersStubFinder,
_ScipyDistnPatchingFinder,
_MaskingUtilsFinder,
):
try:
sys.meta_path.insert(0, _FinderCls())
_diag(f"installed finder: {_FinderCls.__name__}")
except Exception as e:
_diag(f"FAILED to install {_FinderCls.__name__}: {e!r}")
_diag(
"final sys.meta_path head: "
+ ", ".join(type(f).__name__ for f in sys.meta_path[:6])
)
# If torch is already imported, also set the attribute on the package so
# `torch._dynamo` resolves to our stub without triggering torch.__getattr__
# (which would lazy-import the real module and crash).
torch_mod = sys.modules.get("torch")
if torch_mod is not None:
torch_mod._dynamo = stub
try:
_install_dynamo_stub()
except Exception as _e:
# Best effort. If this fails the original NameError will surface when
# transformers imports — no worse than not patching at all.
_diag(f"_install_dynamo_stub FAILED: {_e!r}")
# NOTE: we deliberately do NOT import torch or torch.compiler here.
# Runtime hooks run before the app starts and before pyi_rth_numpy_compat
# has had a chance to patch torch.from_numpy (it runs in a background
# thread, waiting for torch to appear in sys.modules). Importing torch
# eagerly at hook time would trip the numpy ABI issue and kill the
# server process at startup.
#
# torch.compiler.disable does not need an explicit stub: its
# implementation is effectively `import torch._dynamo; return
# torch._dynamo.disable(fn, recursive, reason=reason)`, and since our
# stub is installed in sys.modules, that call resolves to our no-op
# _NoopDecorator pass-through.
+11 -1
View File
@@ -2,4 +2,14 @@
# These should only be installed on aarch64-apple-darwin platforms
mlx>=0.30.0
mlx-audio>=0.3.1
# NOTE: mlx-audio is intentionally not listed here. From 0.3.1 onward it
# declares `transformers==5.0.0rc3` / `>=5.0.0`, which conflicts with the
# `transformers<=4.57.6` cap in requirements.txt and breaks CI's clean
# resolver. The mlx-audio API surface we use (mlx_audio.tts.load,
# mlx_audio.stt.load) works fine on transformers 4.57.x in practice.
#
# Install it via `pip install --no-deps mlx-audio==0.4.1` after this file
# (see .github/workflows/release.yml). All other mlx-audio runtime deps
# (huggingface_hub, librosa, miniaudio, mlx-lm, numba, numpy, protobuf,
# pyloudnorm, sounddevice, tqdm) are already in requirements.txt.
+32 -1
View File
@@ -14,7 +14,7 @@ from .. import models
from ..services import history, profiles, tts
from ..database import Generation as DBGeneration, VoiceProfile as DBVoiceProfile, get_db
from ..services.generation import run_generation
from ..services.task_queue import enqueue_generation
from ..services.task_queue import cancel_generation as cancel_generation_job, enqueue_generation
from ..utils.tasks import get_task_manager
router = APIRouter()
@@ -82,6 +82,7 @@ async def generate_speech(
pass
enqueue_generation(
generation_id,
run_generation(
generation_id=generation_id,
profile_id=data.profile_id,
@@ -127,6 +128,7 @@ async def retry_generation(generation_id: str, db: Session = Depends(get_db)):
)
enqueue_generation(
generation_id,
run_generation(
generation_id=generation_id,
profile_id=gen.profile_id,
@@ -170,6 +172,7 @@ async def regenerate_generation(generation_id: str, db: Session = Depends(get_db
version_id = str(uuid.uuid4())
enqueue_generation(
generation_id,
run_generation(
generation_id=generation_id,
profile_id=gen.profile_id,
@@ -187,6 +190,34 @@ async def regenerate_generation(generation_id: str, db: Session = Depends(get_db
return models.GenerationResponse.model_validate(gen)
@router.post("/generate/{generation_id}/cancel")
async def cancel_generation(generation_id: str, db: Session = Depends(get_db)):
"""Cancel a queued or running generation."""
gen = db.query(DBGeneration).filter_by(id=generation_id).first()
if not gen:
raise HTTPException(status_code=404, detail="Generation not found")
if (gen.status or "completed") not in ("loading_model", "generating"):
raise HTTPException(status_code=400, detail="Only active generations can be cancelled")
cancellation_state = cancel_generation_job(generation_id)
if cancellation_state is None:
raise HTTPException(status_code=409, detail="Generation is no longer cancellable")
if cancellation_state == "queued":
task_manager = get_task_manager()
task_manager.complete_generation(generation_id)
await history.update_generation_status(
generation_id=generation_id,
status="failed",
db=db,
error="Generation cancelled",
)
return {"message": "Queued generation cancelled"}
return {"message": "Generation cancellation requested"}
@router.get("/generate/{generation_id}/status")
async def get_generation_status(generation_id: str, db: Session = Depends(get_db)):
"""SSE endpoint that streams generation status updates."""
+3 -2
View File
@@ -135,14 +135,15 @@ async def migrate_models(request: models.ModelMigrateRequest):
if destination.resolve().is_relative_to(source.resolve()):
raise HTTPException(status_code=400, detail="Destination cannot be inside the current cache directory")
progress_manager = get_progress_manager()
model_dirs = [d for d in source.iterdir() if d.name.startswith("models--") and d.is_dir()]
if not model_dirs:
progress_manager.update_progress("migration", 1, 1, status="complete")
progress_manager.mark_complete("migration")
return {"moved": 0, "errors": [], "source": str(source), "destination": str(destination)}
destination.mkdir(parents=True, exist_ok=True)
progress_manager = get_progress_manager()
same_fs = False
try:
same_fs = source.stat().st_dev == destination.stat().st_dev
+8
View File
@@ -16,6 +16,7 @@ Mode differences:
from __future__ import annotations
import asyncio
import traceback
from typing import Literal, Optional
@@ -126,6 +127,13 @@ async def run_generation(
duration=duration,
)
except asyncio.CancelledError:
await history.update_generation_status(
generation_id=generation_id,
status="failed",
db=bg_db,
error="Generation cancelled",
)
except Exception as e:
traceback.print_exc()
await history.update_generation_status(
+4
View File
@@ -319,6 +319,10 @@ async def delete_generations_by_profile(
count = 0
for generation in generations:
# Delete associated version files and rows first
from . import versions as versions_mod
versions_mod.delete_versions_for_generation(generation.id, db)
# Delete audio file
audio_path = config.resolve_storage_path(generation.audio_path)
if audio_path is not None and audio_path.exists():
+67 -7
View File
@@ -5,12 +5,27 @@ to avoid GPU contention.
import asyncio
import traceback
from dataclasses import dataclass
from typing import Coroutine, Literal
# Keep references to fire-and-forget background tasks to prevent GC
_background_tasks: set = set()
@dataclass
class GenerationJob:
"""Queued generation work plus the generation ID it belongs to."""
generation_id: str
coro: Coroutine
# Generation queue — serializes TTS inference to avoid GPU contention
_generation_queue: asyncio.Queue = None # type: ignore # initialized at startup
_generation_worker_task: asyncio.Task | None = None
_queued_generation_ids: set[str] = set()
_running_generation_tasks: dict[str, asyncio.Task] = {}
_cancelled_generation_ids: set[str] = set()
def create_background_task(coro) -> asyncio.Task:
@@ -24,25 +39,70 @@ def create_background_task(coro) -> asyncio.Task:
async def _generation_worker():
"""Worker that processes generation tasks one at a time."""
while True:
coro = await _generation_queue.get()
job = await _generation_queue.get()
try:
await coro
if job.generation_id in _cancelled_generation_ids:
_cancelled_generation_ids.discard(job.generation_id)
job.coro.close()
continue
task = asyncio.create_task(job.coro)
_running_generation_tasks[job.generation_id] = task
_queued_generation_ids.discard(job.generation_id)
try:
await task
except asyncio.CancelledError:
if not task.cancelled():
raise
except Exception:
traceback.print_exc()
finally:
_running_generation_tasks.pop(job.generation_id, None)
_queued_generation_ids.discard(job.generation_id)
_generation_queue.task_done()
def enqueue_generation(coro):
def enqueue_generation(generation_id: str, coro):
"""Add a generation coroutine to the serial queue."""
_generation_queue.put_nowait(coro)
if _generation_queue is None:
raise RuntimeError("Generation queue has not been initialized")
_queued_generation_ids.add(generation_id)
_generation_queue.put_nowait(GenerationJob(generation_id=generation_id, coro=coro))
def init_queue():
def cancel_generation(generation_id: str) -> Literal["queued", "running"] | None:
"""Cancel a queued or running generation if it is still active."""
running_task = _running_generation_tasks.get(generation_id)
if running_task is not None:
running_task.cancel()
return "running"
if generation_id in _queued_generation_ids:
_queued_generation_ids.discard(generation_id)
_cancelled_generation_ids.add(generation_id)
return "queued"
return None
def init_queue(force: bool = False):
"""Initialize the generation queue and start the worker.
Must be called once during application startup (inside a running event loop).
"""
global _generation_queue
global _generation_queue, _generation_worker_task
global _queued_generation_ids, _running_generation_tasks, _cancelled_generation_ids
if _generation_worker_task is not None and not _generation_worker_task.done():
if not force:
return
_generation_worker_task.cancel()
for task in list(_running_generation_tasks.values()):
task.cancel()
_generation_queue = asyncio.Queue()
create_background_task(_generation_worker())
_queued_generation_ids = set()
_running_generation_tasks = {}
_cancelled_generation_ids = set()
_generation_worker_task = create_background_task(_generation_worker())
+220
View File
@@ -0,0 +1,220 @@
# End-to-End Model Generation Test — Design
## Goal
A single script, runnable on macOS and Windows, that exercises every TTS model against the **frozen PyInstaller binary** (not the dev server), captures per-model pass/fail and error messages, and exits non-zero if any model fails. Generation is strictly sequential — one model loaded at a time.
## Test matrix (10 runs)
Derived from `backend/backends/__init__.py:185-316`. Each row maps to one `POST /generate` call.
| # | engine | model_size | profile kind | notes |
|---|-----------------------|------------|--------------|-------|
| 1 | `qwen` | `1.7B` | cloned | reference audio required |
| 2 | `qwen` | `0.6B` | cloned | |
| 3 | `qwen_custom_voice` | `1.7B` | preset | `preset_voice_id="Ryan"` |
| 4 | `qwen_custom_voice` | `0.6B` | preset | `preset_voice_id="Ryan"` |
| 5 | `luxtts` | — | cloned | English only |
| 6 | `chatterbox` | — | cloned | |
| 7 | `chatterbox_turbo` | — | cloned | English only |
| 8 | `tada` | `1B` | cloned | tada-1b, English only |
| 9 | `tada` | `3B` | cloned | tada-3b-ml, multilingual |
| 10| `kokoro` | — | preset | `preset_voice_id="af_heart"` |
Cloned engines (1, 2, 5, 6, 7, 8, 9) share **one** profile created once with the reference WAV. Preset profiles are created separately, one for kokoro and one for qwen_custom_voice.
Language for every run: `en` (covers every engine's supported set).
## End-to-end flow
```
1. Resolve paths → find binary, build if missing
2. Launch binary → spawn with --port --data-dir --parent-pid
3. Wait for /health → poll until status=="healthy" or 120s timeout
4. Create profiles → 1 cloned + 2 preset, via /profiles (+ /samples)
5. For each (engine, model_size) in matrix:
a. Check cache → GET /models/status → cached? short timeout : long
b. POST /generate → get generation_id
c. Stream /status → consume SSE until completed/failed/timeout
d. Record result → {engine, model_size, status, duration, error, elapsed}
6. Write results → JSON + Markdown table to ./results/
7. Shutdown binary → SIGTERM, fall back to kill, verify port freed
8. Exit code → 0 if all passed, 1 otherwise
```
## Binary resolution
Search order — **first hit wins**:
| Platform | Path | Build type |
|----------|------|------------|
| macOS | `backend/dist/voicebox-server-cuda/voicebox-server-cuda` | onedir (CUDA, rarely on Mac) |
| macOS | `backend/dist/voicebox-server` | onefile (CPU) |
| Windows | `backend\dist\voicebox-server-cuda\voicebox-server-cuda.exe` | onedir (CUDA) |
| Windows | `backend\dist\voicebox-server.exe` | onefile (CPU) |
If none exist, run `python backend/build_binary.py` and wait for it to finish (can take 5-20 min). Fail with a clear error if the build itself fails. `--skip-build` flag forces "error out if no binary" instead of building.
## Spawn command
Mirrors Tauri's launch in `tauri/src-tauri/src/main.rs:369-388`:
```
<binary> --host 127.0.0.1 --port <free-port> --data-dir <tempdir> --parent-pid <test-pid>
```
- **Port**: bind to `0` first in Python to grab a free port, then pass that number.
- **Data dir**: `tempfile.mkdtemp(prefix="voicebox-e2e-")`. Deleted after the run unless `--keep-data-dir`. Profiles and generated WAVs land here.
- **Parent PID**: current Python PID — ensures the backend dies if the test crashes (watchdog in `server.py:102-224`).
- **stdout/stderr**: tee to both a log file in `./results/server-<timestamp>.log` and a rolling in-memory buffer. On model failure, last 100 lines of the buffer are attached to that model's error record.
## Profile setup
One cloned profile shared across all cloning engines:
```http
POST /profiles
{
"name": "e2e-cloned",
"voice_type": "cloned",
"language": "en"
}
```
Then:
```http
POST /profiles/{id}/samples (multipart)
file: <reference WAV>
reference_text: <exact transcription>
```
Two preset profiles:
```http
POST /profiles
{ "name": "e2e-kokoro", "voice_type": "preset", "language": "en",
"preset_engine": "kokoro", "preset_voice_id": "af_heart" }
POST /profiles
{ "name": "e2e-qwen-cv", "voice_type": "preset", "language": "en",
"preset_engine": "qwen_custom_voice", "preset_voice_id": "Ryan" }
```
## Generation request (per matrix row)
```http
POST /generate
{
"profile_id": "<appropriate profile>",
"text": "The quick brown fox jumps over the lazy dog.",
"language": "en",
"engine": "<engine>",
"model_size": "<size or omitted>",
"seed": 42,
"normalize": true
}
```
Response `id` feeds into the SSE status loop (`GET /generate/{id}/status`, `routes/generations.py:190-227`). Loop reads lines until a payload with `status in ("completed", "failed")` arrives, then breaks.
## Timeout strategy (split)
Check `GET /models/status` for the target model **before** generation:
| Cached? | Per-model timeout | Rationale |
|---------|-------------------|-----------|
| Yes | **3 minutes** | Inference only; generous for CPU builds |
| No | **20 minutes** | First-run HF download up to 8 GB (tada-3b-ml) |
On timeout: cancel the SSE stream, mark the row `timeout`, and continue to the next row. Don't abort the whole run on one timeout.
## Result format
`./results/e2e-<platform>-<arch>-<timestamp>.json`:
```json
{
"platform": "darwin-arm64",
"binary": "/abs/path/voicebox-server",
"binary_size_mb": 612,
"started_at": "2026-04-16T12:34:56Z",
"finished_at": "...",
"results": [
{
"engine": "qwen",
"model_size": "1.7B",
"status": "passed|failed|timeout",
"generation_id": "...",
"was_cached": true,
"elapsed_seconds": 12.4,
"audio_duration": 3.1,
"audio_path": "/tmp/.../gen.wav",
"error": null,
"server_log_tail": null
}
]
}
```
Companion `./results/e2e-<...>.md`:
```
# Voicebox E2E — darwin-arm64 — 2026-04-16 12:34
| Engine | Size | Status | Elapsed | Error |
|---------------------|------|--------|---------|-------|
| qwen | 1.7B | PASS | 12.4s | |
| qwen | 0.6B | FAIL | 4.1s | CUDA OOM: ... |
...
```
## CLI flags
```
python -m backend.tests.test_all_models_e2e [flags]
--binary PATH Use this binary instead of auto-detecting
--skip-build Error if no binary found (no auto-build)
--reference-wav PATH Reference audio (default: backend/tests/fixtures/reference_voice.wav)
--reference-text STR Transcription (default: read from fixtures/reference_voice.txt)
--only ENGINE[,...] Run only these engines (e.g. kokoro,qwen)
--skip ENGINE[,...] Skip these engines
--keep-data-dir Don't delete tempdir after run
--timeout-cached SEC Override 180
--timeout-download SEC Override 1200
--port N Override auto-picked port
--output-dir PATH Default: backend/tests/results/
```
## File layout
```
backend/tests/
├── E2E_MODEL_TEST_DESIGN.md (this file)
├── test_all_models_e2e.py (main script, ~400-500 LoC)
├── fixtures/
│ ├── reference_voice.wav (user-provided, ~5-15s clean speech)
│ └── reference_voice.txt (exact transcription)
└── results/ (gitignored)
├── e2e-darwin-arm64-<ts>.json
├── e2e-darwin-arm64-<ts>.md
└── server-<ts>.log
```
The script uses only stdlib + `httpx` (or `requests`) + `sseclient-py` — all already in `backend/requirements.txt`. No pytest to keep it invocable as a single command on fresh checkouts.
## Safety & cleanup
- Always kill the spawned binary in a `try/finally`. On Windows, `taskkill /F /T` the whole tree (Tauri does the same).
- Verify the port is free on shutdown (Tauri port-reuse check in `main.rs:114-186` could otherwise pick up a ghost).
- Don't touch the user's HF cache by default — let the server use `HF_HUB_CACHE` / `VOICEBOX_MODELS_DIR`. Passing `--isolated-cache` would point both env vars at the tempdir for a true cold-start run (opt-in only; would re-download every time).
## Non-goals
- Not validating audio quality (no WER, no waveform comparison). Pass = "endpoint returned `completed` and produced a non-empty WAV".
- Not testing STT (Whisper), effects chains, channels, or streaming endpoints.
- Not running on CI today — human-invoked on dev machines. CI integration is a follow-up once the script is stable.
- No model unload between runs — models stay loaded; server manages its own eviction.
- No version-drift check on the binary.
- No `instruct` parameter exercised on qwen_custom_voice runs.
+16
View File
@@ -0,0 +1,16 @@
# E2E Test Fixtures
Place two files here before running `test_all_models_e2e.py`:
- `reference_voice.wav` — a clean speech sample, mono, 16–24 kHz, ~5–15 seconds.
- `reference_voice.txt` — the **exact** transcription of the WAV (single line, no trailing newline required).
These are used to create a cloned voice profile for every cloning-capable engine (qwen, luxtts, chatterbox, chatterbox_turbo, tada). Keep them out of version control if they contain personal audio — this directory is not gitignored by default, so add them to `.gitignore` locally if needed.
You can point the test at different files with:
```
python backend/tests/test_all_models_e2e.py \
--reference-wav /path/to/your.wav \
--reference-text "exact transcription here"
```
+630
View File
@@ -0,0 +1,630 @@
"""
End-to-end model generation test.
Exercises every TTS model against the frozen PyInstaller binary, captures
per-model pass/fail, and writes a JSON + Markdown report.
Usage:
python backend/tests/test_all_models_e2e.py [flags]
See E2E_MODEL_TEST_DESIGN.md for the full design.
"""
from __future__ import annotations
import argparse
import json
import os
import platform
import shutil
import signal
import socket
import subprocess
import sys
import tempfile
import threading
import time
from collections import deque
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
import httpx
REPO_ROOT = Path(__file__).resolve().parents[2]
BACKEND_DIR = REPO_ROOT / "backend"
DIST_DIR = BACKEND_DIR / "dist"
FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
RESULTS_DIR = Path(__file__).resolve().parent / "results"
# ── Test matrix ──────────────────────────────────────────────────────
@dataclass(frozen=True)
class MatrixRow:
label: str # human-readable (appears in report)
engine: str # /generate engine
model_size: Optional[str] # /generate model_size (None = omit)
profile_kind: str # "cloned" | "preset_kokoro" | "preset_qwen_cv"
model_name: str # /models/status key for cache lookup
MATRIX: list[MatrixRow] = [
MatrixRow("qwen 1.7B", "qwen", "1.7B", "cloned", "qwen-tts-1.7B"),
MatrixRow("qwen 0.6B", "qwen", "0.6B", "cloned", "qwen-tts-0.6B"),
MatrixRow("qwen_custom_voice 1.7B", "qwen_custom_voice", "1.7B", "preset_qwen_cv", "qwen-custom-voice-1.7B"),
MatrixRow("qwen_custom_voice 0.6B", "qwen_custom_voice", "0.6B", "preset_qwen_cv", "qwen-custom-voice-0.6B"),
MatrixRow("luxtts", "luxtts", None, "cloned", "luxtts"),
MatrixRow("chatterbox", "chatterbox", None, "cloned", "chatterbox-tts"),
MatrixRow("chatterbox_turbo", "chatterbox_turbo", None, "cloned", "chatterbox-turbo"),
MatrixRow("tada 1B", "tada", "1B", "cloned", "tada-1b"),
MatrixRow("tada 3B", "tada", "3B", "cloned", "tada-3b-ml"),
MatrixRow("kokoro", "kokoro", None, "preset_kokoro", "kokoro"),
]
TEXT = "The quick brown fox jumps over the lazy dog."
DEFAULT_TIMEOUT_CACHED = 180
DEFAULT_TIMEOUT_DOWNLOAD = 1200
HEALTH_TIMEOUT = 120
# ── Result record ────────────────────────────────────────────────────
@dataclass
class ModelResult:
label: str
engine: str
model_size: Optional[str]
status: str # "passed" | "failed" | "timeout"
was_cached: Optional[bool] = None
generation_id: Optional[str] = None
elapsed_seconds: float = 0.0
audio_duration: Optional[float] = None
audio_path: Optional[str] = None
audio_bytes: Optional[int] = None
error: Optional[str] = None
http_status: Optional[int] = None
server_log_tail: Optional[list[str]] = None
# ── Binary resolution ────────────────────────────────────────────────
def find_binary() -> Optional[Path]:
"""Return the first existing binary in priority order, or None."""
is_win = platform.system() == "Windows"
exe = ".exe" if is_win else ""
candidates = [
DIST_DIR / "voicebox-server-cuda" / f"voicebox-server-cuda{exe}",
DIST_DIR / f"voicebox-server{exe}",
]
for c in candidates:
if c.exists() and c.is_file():
return c
return None
def build_binary() -> Path:
"""Invoke build_binary.py and return the resulting binary path."""
print("[build] No frozen binary found — invoking build_binary.py (this may take 5-20 minutes)...", flush=True)
script = BACKEND_DIR / "build_binary.py"
result = subprocess.run(
[sys.executable, str(script)],
cwd=str(BACKEND_DIR),
)
if result.returncode != 0:
raise RuntimeError(f"build_binary.py exited with code {result.returncode}")
found = find_binary()
if found is None:
raise RuntimeError("build_binary.py finished but no binary was found in backend/dist/")
return found
# ── Server spawn + log capture ───────────────────────────────────────
class ServerProcess:
def __init__(self, binary: Path, port: int, data_dir: Path, log_path: Path):
self.binary = binary
self.port = port
self.data_dir = data_dir
self.log_path = log_path
self.proc: Optional[subprocess.Popen] = None
self._log_buffer: deque[str] = deque(maxlen=500)
self._reader_thread: Optional[threading.Thread] = None
def start(self) -> None:
args = [
str(self.binary),
"--host", "127.0.0.1",
"--port", str(self.port),
"--data-dir", str(self.data_dir),
"--parent-pid", str(os.getpid()),
]
print(f"[spawn] {' '.join(args)}", flush=True)
self._log_fh = open(self.log_path, "w", encoding="utf-8", errors="replace")
# Combine stderr into stdout so we get a single ordered stream.
self.proc = subprocess.Popen(
args,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
text=True,
errors="replace",
)
self._reader_thread = threading.Thread(target=self._pump_logs, daemon=True)
self._reader_thread.start()
def _pump_logs(self) -> None:
assert self.proc is not None and self.proc.stdout is not None
for line in self.proc.stdout:
self._log_buffer.append(line.rstrip("\n"))
self._log_fh.write(line)
self._log_fh.flush()
def log_tail(self, n: int = 100) -> list[str]:
tail = list(self._log_buffer)[-n:]
return tail
def is_alive(self) -> bool:
return self.proc is not None and self.proc.poll() is None
def stop(self) -> None:
if self.proc is None:
return
if self.proc.poll() is not None:
return
try:
if platform.system() == "Windows":
subprocess.run(
["taskkill", "/F", "/T", "/PID", str(self.proc.pid)],
capture_output=True,
)
else:
self.proc.send_signal(signal.SIGTERM)
except Exception as e:
print(f"[shutdown] signal failed: {e}", flush=True)
try:
self.proc.wait(timeout=10)
except subprocess.TimeoutExpired:
print("[shutdown] server didn't exit cleanly, killing", flush=True)
self.proc.kill()
try:
self.proc.wait(timeout=5)
except subprocess.TimeoutExpired:
pass
if self._reader_thread is not None:
self._reader_thread.join(timeout=2)
try:
self._log_fh.close()
except Exception:
pass
def pick_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
# ── HTTP helpers ─────────────────────────────────────────────────────
def wait_for_health(base_url: str, server: ServerProcess, timeout: int) -> None:
deadline = time.time() + timeout
with httpx.Client(timeout=5.0) as client:
while time.time() < deadline:
if not server.is_alive():
raise RuntimeError("Server process exited before becoming healthy")
try:
r = client.get(f"{base_url}/health")
if r.status_code == 200 and r.json().get("status") == "healthy":
return
except httpx.HTTPError:
pass
time.sleep(1.0)
raise TimeoutError(f"Server did not become healthy within {timeout}s")
def get_model_cached(client: httpx.Client, base_url: str, model_name: str) -> Optional[bool]:
try:
r = client.get(f"{base_url}/models/status", timeout=30.0)
r.raise_for_status()
for m in r.json().get("models", []):
if m.get("model_name") == model_name:
return bool(m.get("downloaded"))
except httpx.HTTPError:
return None
return None
def create_cloned_profile(client: httpx.Client, base_url: str, wav_path: Path, reference_text: str) -> str:
r = client.post(f"{base_url}/profiles", json={
"name": "e2e-cloned",
"voice_type": "cloned",
"language": "en",
})
r.raise_for_status()
profile_id = r.json()["id"]
with open(wav_path, "rb") as f:
r = client.post(
f"{base_url}/profiles/{profile_id}/samples",
files={"file": (wav_path.name, f, "audio/wav")},
data={"reference_text": reference_text},
timeout=120.0,
)
r.raise_for_status()
return profile_id
def create_preset_profile(client: httpx.Client, base_url: str, name: str, engine: str, voice_id: str) -> str:
r = client.post(f"{base_url}/profiles", json={
"name": name,
"voice_type": "preset",
"language": "en",
"preset_engine": engine,
"preset_voice_id": voice_id,
})
r.raise_for_status()
return r.json()["id"]
def run_one_generation(
client: httpx.Client,
base_url: str,
row: MatrixRow,
profile_id: str,
timeout_s: int,
) -> tuple[str, dict]:
"""Start a generation and stream its status until done/failed/timeout.
Returns (status, payload) where status is "completed" | "failed" | "timeout".
"""
body = {
"profile_id": profile_id,
"text": TEXT,
"language": "en",
"engine": row.engine,
"seed": 42,
"normalize": True,
}
if row.model_size is not None:
body["model_size"] = row.model_size
r = client.post(f"{base_url}/generate", json=body, timeout=30.0)
r.raise_for_status()
gen = r.json()
gen_id = gen["id"]
deadline = time.time() + timeout_s
last_payload: dict = gen
status_url = f"{base_url}/generate/{gen_id}/status"
while time.time() < deadline:
remaining = max(1.0, deadline - time.time())
try:
with client.stream("GET", status_url, timeout=httpx.Timeout(remaining + 5, read=remaining + 5)) as resp:
resp.raise_for_status()
for line in resp.iter_lines():
if not line or not line.startswith("data: "):
continue
try:
payload = json.loads(line[6:])
except json.JSONDecodeError:
continue
last_payload = payload
status = payload.get("status")
if status == "not_found":
return "failed", {"error": "generation not found", **payload}
if status in ("completed", "failed"):
return status, payload
if time.time() >= deadline:
break
except httpx.HTTPError:
time.sleep(1.0)
continue
return "timeout", last_payload
def fetch_audio_info(
client: httpx.Client, base_url: str, generation_id: str, data_dir: Path
) -> tuple[Optional[str], Optional[int]]:
"""Return (audio_path, audio_bytes) for a completed generation.
Server stores audio_path relative to data_dir; resolve it to get a size.
"""
try:
r = client.get(f"{base_url}/history/{generation_id}", timeout=10.0)
if r.status_code != 200:
return None, None
data = r.json()
audio_path = data.get("audio_path")
if not audio_path:
return None, None
p = Path(audio_path)
if not p.is_absolute():
p = data_dir / p
if p.exists():
return str(p), p.stat().st_size
return audio_path, None
except httpx.HTTPError:
return None, None
# ── Report writers ───────────────────────────────────────────────────
def write_reports(
output_dir: Path,
binary: Path,
started_at: datetime,
finished_at: datetime,
results: list[ModelResult],
) -> tuple[Path, Path]:
output_dir.mkdir(parents=True, exist_ok=True)
plat = f"{platform.system().lower()}-{platform.machine().lower()}"
ts = started_at.strftime("%Y%m%d-%H%M%S")
json_path = output_dir / f"e2e-{plat}-{ts}.json"
md_path = output_dir / f"e2e-{plat}-{ts}.md"
doc = {
"platform": plat,
"binary": str(binary),
"binary_size_mb": round(binary.stat().st_size / (1024 * 1024), 1) if binary.exists() else None,
"started_at": started_at.isoformat(),
"finished_at": finished_at.isoformat(),
"elapsed_seconds": (finished_at - started_at).total_seconds(),
"results": [asdict(r) for r in results],
}
json_path.write_text(json.dumps(doc, indent=2))
lines = [
f"# Voicebox E2E — {plat} — {started_at.strftime('%Y-%m-%d %H:%M UTC')}",
"",
f"Binary: `{binary}` ",
f"Elapsed: {doc['elapsed_seconds']:.1f}s",
"",
"| Model | Status | Cached | Elapsed | Audio | Error |",
"|-------|--------|--------|---------|-------|-------|",
]
for r in results:
status_icon = {"passed": "PASS", "failed": "FAIL", "timeout": "TIMEOUT"}.get(r.status, r.status.upper())
cached = "yes" if r.was_cached else ("no" if r.was_cached is False else "?")
audio_col = f"{r.audio_duration:.2f}s" if r.audio_duration else ("—" if r.status != "passed" else "?")
error_col = (r.error or "").replace("\n", " ")[:120]
lines.append(f"| {r.label} | {status_icon} | {cached} | {r.elapsed_seconds:.1f}s | {audio_col} | {error_col} |")
failed_rows = [r for r in results if r.status != "passed"]
if failed_rows:
lines.append("")
lines.append("## Failures")
for r in failed_rows:
lines.append("")
lines.append(f"### {r.label} — {r.status}")
if r.error:
lines.append("")
lines.append("```")
lines.append(r.error)
lines.append("```")
if r.server_log_tail:
lines.append("")
lines.append("<details><summary>server log (last lines)</summary>")
lines.append("")
lines.append("```")
lines.extend(r.server_log_tail)
lines.append("```")
lines.append("</details>")
md_path.write_text("\n".join(lines) + "\n")
return json_path, md_path
# ── Main ─────────────────────────────────────────────────────────────
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="Voicebox E2E model generation test")
p.add_argument("--binary", type=Path, help="Path to voicebox-server binary (overrides auto-detect)")
p.add_argument("--skip-build", action="store_true", help="Error if binary missing instead of building")
p.add_argument(
"--reference-wav",
type=Path,
default=FIXTURES_DIR / "reference_voice.wav",
help="Reference audio for cloning engines",
)
p.add_argument(
"--reference-text",
help="Transcription of reference-wav (default: read from fixtures/reference_voice.txt)",
)
p.add_argument("--only", help="Comma-separated engines to run (e.g. kokoro,qwen)")
p.add_argument("--skip", help="Comma-separated engines to skip")
p.add_argument("--keep-data-dir", action="store_true", help="Don't delete tempdir after run")
p.add_argument("--timeout-cached", type=int, default=DEFAULT_TIMEOUT_CACHED)
p.add_argument("--timeout-download", type=int, default=DEFAULT_TIMEOUT_DOWNLOAD)
p.add_argument("--port", type=int, help="Override auto-picked port")
p.add_argument("--output-dir", type=Path, default=RESULTS_DIR)
return p.parse_args()
def filter_matrix(args: argparse.Namespace) -> list[MatrixRow]:
only = set(x.strip() for x in args.only.split(",")) if args.only else None
skip = set(x.strip() for x in args.skip.split(",")) if args.skip else set()
rows = []
for r in MATRIX:
if only is not None and r.engine not in only:
continue
if r.engine in skip:
continue
rows.append(r)
return rows
def resolve_reference(args: argparse.Namespace) -> tuple[Path, str]:
wav = args.reference_wav
if not wav.exists():
raise FileNotFoundError(
f"Reference WAV not found: {wav}\n"
f"Place a sample at {FIXTURES_DIR / 'reference_voice.wav'} or pass --reference-wav.\n"
f"See backend/tests/fixtures/README.md."
)
if args.reference_text:
text = args.reference_text
else:
txt_path = wav.with_suffix(".txt")
if not txt_path.exists():
raise FileNotFoundError(
f"Reference transcription not found: {txt_path}\n"
f"Create it next to the WAV, or pass --reference-text."
)
text = txt_path.read_text().strip()
if not text:
raise ValueError("Reference transcription is empty")
return wav, text
def main() -> int:
args = parse_args()
rows = filter_matrix(args)
if not rows:
print("No rows selected after --only/--skip filtering", file=sys.stderr)
return 2
# Binary
binary = args.binary or find_binary()
if binary is None:
if args.skip_build:
print("No frozen binary found and --skip-build set. Run: python backend/build_binary.py", file=sys.stderr)
return 2
binary = build_binary()
if not binary.exists():
print(f"Binary path does not exist: {binary}", file=sys.stderr)
return 2
print(f"[binary] {binary}", flush=True)
# Reference audio (only required if any cloning row is in the matrix)
needs_reference = any(r.profile_kind == "cloned" for r in rows)
ref_wav: Optional[Path] = None
ref_text: Optional[str] = None
if needs_reference:
try:
ref_wav, ref_text = resolve_reference(args)
except (FileNotFoundError, ValueError) as e:
print(f"[fixture] {e}", file=sys.stderr)
return 2
print(f"[fixture] reference WAV: {ref_wav}", flush=True)
print(f"[fixture] reference text: {ref_text!r}", flush=True)
# Tempdir + log path
data_dir = Path(tempfile.mkdtemp(prefix="voicebox-e2e-"))
args.output_dir.mkdir(parents=True, exist_ok=True)
ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
log_path = args.output_dir / f"server-{ts}.log"
port = args.port or pick_free_port()
base_url = f"http://127.0.0.1:{port}"
server = ServerProcess(binary=binary, port=port, data_dir=data_dir, log_path=log_path)
started_at = datetime.now(timezone.utc)
results: list[ModelResult] = []
try:
server.start()
print(f"[health] waiting for {base_url}/health ...", flush=True)
wait_for_health(base_url, server, HEALTH_TIMEOUT)
print("[health] ready", flush=True)
with httpx.Client(timeout=30.0) as client:
# Profile setup (only create what's needed)
cloned_profile_id: Optional[str] = None
kokoro_profile_id: Optional[str] = None
qwen_cv_profile_id: Optional[str] = None
needed_kinds = {r.profile_kind for r in rows}
if "cloned" in needed_kinds:
assert ref_wav is not None and ref_text is not None
print("[profile] creating cloned profile...", flush=True)
cloned_profile_id = create_cloned_profile(client, base_url, ref_wav, ref_text)
if "preset_kokoro" in needed_kinds:
print("[profile] creating kokoro preset...", flush=True)
kokoro_profile_id = create_preset_profile(client, base_url, "e2e-kokoro", "kokoro", "af_heart")
if "preset_qwen_cv" in needed_kinds:
print("[profile] creating qwen_custom_voice preset...", flush=True)
qwen_cv_profile_id = create_preset_profile(client, base_url, "e2e-qwen-cv", "qwen_custom_voice", "Ryan")
profile_lookup = {
"cloned": cloned_profile_id,
"preset_kokoro": kokoro_profile_id,
"preset_qwen_cv": qwen_cv_profile_id,
}
# Matrix loop
for row in rows:
print(f"\n[run] {row.label} (engine={row.engine}, size={row.model_size})", flush=True)
profile_id = profile_lookup[row.profile_kind]
assert profile_id is not None
was_cached = get_model_cached(client, base_url, row.model_name)
timeout_s = args.timeout_cached if was_cached else args.timeout_download
print(f"[run] cached={was_cached} timeout={timeout_s}s", flush=True)
t0 = time.time()
result = ModelResult(
label=row.label,
engine=row.engine,
model_size=row.model_size,
status="failed",
was_cached=was_cached,
)
try:
status, payload = run_one_generation(client, base_url, row, profile_id, timeout_s)
result.status = "passed" if status == "completed" else status
result.generation_id = payload.get("id")
result.audio_duration = payload.get("duration")
result.error = payload.get("error")
if status == "completed" and result.generation_id:
audio_path, audio_bytes = fetch_audio_info(
client, base_url, result.generation_id, data_dir
)
result.audio_path = audio_path
result.audio_bytes = audio_bytes
if audio_bytes is not None and audio_bytes == 0:
result.status = "failed"
result.error = (result.error or "") + " (audio file is empty)"
except httpx.HTTPStatusError as e:
result.status = "failed"
result.http_status = e.response.status_code
try:
detail = e.response.json().get("detail")
except Exception:
detail = e.response.text
result.error = f"HTTP {e.response.status_code}: {detail}"
except Exception as e:
result.status = "failed"
result.error = f"{type(e).__name__}: {e}"
result.elapsed_seconds = round(time.time() - t0, 2)
if result.status != "passed":
result.server_log_tail = server.log_tail(100)
print(f"[run] {row.label} → {result.status} in {result.elapsed_seconds}s"
+ (f" ({result.error})" if result.error else ""), flush=True)
results.append(result)
finally:
finished_at = datetime.now(timezone.utc)
server.stop()
if not args.keep_data_dir:
shutil.rmtree(data_dir, ignore_errors=True)
else:
print(f"[cleanup] keeping data dir: {data_dir}", flush=True)
json_path, md_path = write_reports(args.output_dir, binary, started_at, finished_at, results)
print(f"\n[report] {json_path}")
print(f"[report] {md_path}")
print(f"[report] server log: {log_path}")
passed = sum(1 for r in results if r.status == "passed")
failed = len(results) - passed
print(f"\n== {passed} passed, {failed} failed ==")
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,54 @@
import asyncio
import pytest
from backend.services import task_queue
@pytest.mark.asyncio
async def test_cancel_queued_generation_skips_execution():
task_queue.init_queue(force=True)
running_started = asyncio.Event()
release_running = asyncio.Event()
queued_ran = asyncio.Event()
async def running_job():
running_started.set()
await release_running.wait()
async def queued_job():
queued_ran.set()
task_queue.enqueue_generation("gen-running", running_job())
await asyncio.wait_for(running_started.wait(), timeout=1)
task_queue.enqueue_generation("gen-queued", queued_job())
assert task_queue.cancel_generation("gen-queued") == "queued"
release_running.set()
await asyncio.sleep(0.1)
assert not queued_ran.is_set()
@pytest.mark.asyncio
async def test_cancel_running_generation_cancels_task():
task_queue.init_queue(force=True)
running_started = asyncio.Event()
running_cancelled = asyncio.Event()
async def running_job():
running_started.set()
try:
await asyncio.Event().wait()
except asyncio.CancelledError:
running_cancelled.set()
raise
task_queue.enqueue_generation("gen-running", running_job())
await asyncio.wait_for(running_started.wait(), timeout=1)
assert task_queue.cancel_generation("gen-running") == "running"
await asyncio.wait_for(running_cancelled.wait(), timeout=1)
+7 -3
View File
@@ -5,7 +5,7 @@ from PyInstaller.utils.hooks import copy_metadata
datas = []
binaries = []
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.backends.qwen_custom_voice_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'kokoro', 'kokoro.pipeline', 'kokoro.model', 'kokoro.istftnet', 'kokoro.modules', 'kokoro.custom_stft', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.backends.qwen_custom_voice_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
datas += copy_metadata('qwen-tts')
datas += copy_metadata('requests')
datas += copy_metadata('transformers')
@@ -18,6 +18,8 @@ hiddenimports += collect_submodules('jaraco')
hiddenimports += collect_submodules('tada')
hiddenimports += collect_submodules('mlx')
hiddenimports += collect_submodules('mlx_audio')
tmp_ret = collect_all('spacy_pkuseg')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('zipvoice')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('linacodec')
@@ -34,6 +36,8 @@ tmp_ret = collect_all('perth')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('piper_phonemize')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('kokoro')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('misaki')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('language_tags')
@@ -54,9 +58,9 @@ a = Analysis(
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hookspath=['pyi_hooks'],
hooksconfig={},
runtime_hooks=[],
runtime_hooks=['pyi_rth_numpy_compat.py', 'pyi_rth_torch_compiler_disable.py'],
excludes=['nvidia', 'nvidia.cublas', 'nvidia.cuda_cupti', 'nvidia.cuda_nvrtc', 'nvidia.cuda_runtime', 'nvidia.cudnn', 'nvidia.cufft', 'nvidia.curand', 'nvidia.cusolver', 'nvidia.cusparse', 'nvidia.nccl', 'nvidia.nvjitlink', 'nvidia.nvtx'],
noarchive=False,
optimize=0,
+7 -4
View File
@@ -17,7 +17,7 @@
},
"app": {
"name": "@voicebox/app",
"version": "0.2.0",
"version": "0.4.1",
"dependencies": {
"@dnd-kit/core": "^6.3.1",
"@dnd-kit/sortable": "^10.0.0",
@@ -72,9 +72,10 @@
},
"landing": {
"name": "@voicebox/landing",
"version": "0.2.0",
"version": "0.4.1",
"dependencies": {
"@fontsource/space-grotesk": "^5.2.10",
"@icons-pack/react-simple-icons": "^13.13.0",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slot": "^1.2.4",
"autoprefixer": "^10.4.17",
@@ -100,7 +101,7 @@
},
"tauri": {
"name": "@voicebox/tauri",
"version": "0.2.0",
"version": "0.4.1",
"dependencies": {
"@tauri-apps/api": "^2.0.0",
"@tauri-apps/plugin-dialog": "^2.0.0",
@@ -123,7 +124,7 @@
},
"web": {
"name": "@voicebox/web",
"version": "0.2.0",
"version": "0.4.1",
"dependencies": {
"@tanstack/react-query": "^5.0.0",
"react": "^18.3.0",
@@ -287,6 +288,8 @@
"@humanwhocodes/object-schema": ["@humanwhocodes/[email protected]", "", {}, "sha512-93zYdMES/c1D69yZiKDBj0V24vqNzB/koF26KPaagAfd3P/4gUlh3Dys5ogAK+Exi9QyzlD8x/08Zt7wIKcDcA=="],
"@icons-pack/react-simple-icons": ["@icons-pack/[email protected]", "", { "peerDependencies": { "react": "^16.13 || ^17 || ^18 || ^19" } }, "sha512-B5HhQMIpcSH4z8IZ8HFhD59CboHceKYMpPC9kAwGyKntvPdyJJv26DLu4Z1wAjcCLyrJhf11tMhiQGom9Rxb9g=="],
"@img/colour": ["@img/[email protected]", "", {}, "sha512-A5P/LfWGFSl6nsckYtjw9da+19jB8hkJ6ACTGcDfEJ0aE+l2n2El7dsVM7UVHZQ9s2lmYMWlrS21YLy2IR1LUw=="],
"@img/sharp-darwin-arm64": ["@img/[email protected]", "", { "optionalDependencies": { "@img/sharp-libvips-darwin-arm64": "1.2.4" }, "os": "darwin", "cpu": "arm64" }, "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w=="],
+638
View File
@@ -0,0 +1,638 @@
# Voicebox Project Status & Roadmap
> Last updated: 2026-04-18 | Current version: **v0.4.1** | 232 open issues | 12 open PRs
---
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Current State](#current-state)
3. [Open PRs — Triage & Analysis](#open-prs--triage--analysis)
4. [Open Issues — Categorized](#open-issues--categorized)
5. [Existing Plan Documents — Status](#existing-plan-documents--status)
6. [New Model Integration — Landscape](#new-model-integration--landscape)
7. [Architectural Bottlenecks](#architectural-bottlenecks)
8. [Recommended Priorities](#recommended-priorities)
---
## Architecture Overview
**Tauri shell (Rust)** hosts a **React frontend** (`app/`) that talks over HTTP on `localhost:17493` to a **FastAPI backend** (`backend/`).
The backend exposes:
- **`TTSBackend` Protocol** with seven concrete engine implementations:
- Qwen3-TTS (PyTorch or MLX depending on platform)
- Qwen CustomVoice (predefined speakers with instruct)
- LuxTTS (fast, CPU-friendly)
- Chatterbox Multilingual (23 languages)
- Chatterbox Turbo (English, paralinguistic tags)
- TADA (1B English, 3B multilingual via HumeAI)
- Kokoro 82M (pre-built voices, CPU realtime)
- **`STTBackend` Protocol** for Whisper (PyTorch or MLX-Whisper)
- **Profiles / History / Stories** services for persistence and timeline editing
### Key Files
| Layer | File | Purpose |
|-------|------|---------|
| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2850 lines) |
| TTS protocol | `backend/backends/__init__.py:32-101` | `TTSBackend` Protocol definition |
| Model registry | `backend/backends/__init__.py:17-29,153-366` | `ModelConfig` dataclass + registry helpers |
| TTS factory | `backend/backends/__init__.py:382-426` | Thread-safe engine registry (double-checked locking) |
| PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
| MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
| Kokoro | `backend/backends/kokoro_backend.py` | Kokoro 82M — CPU realtime, pre-built voices |
| Qwen CustomVoice | `backend/backends/qwen_custom_voice_backend.py` | Qwen CustomVoice — predefined speakers with instruct |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
| Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
| Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
| Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
| Engine selector | `app/src/components/Generation/EngineModelSelector.tsx` | Shared engine/model dropdown |
| Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
| Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
| GPU acceleration | `app/src/components/ServerSettings/GpuAcceleration.tsx` | CUDA backend swap UI |
| Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission |
| Language constants | `app/src/lib/constants/languages.ts` | Per-engine language maps |
### How TTS Generation Works (Current Flow)
```
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
→ tts_backend.create_voice_prompt(audio_path, reference_text)
7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
8. Post-process: trim_tts_output() for Chatterbox engines
9. Save WAV → data/generations/{id}.wav
10. Insert history record in SQLite
11. Return GenerationResponse
```
---
## Current State
### What's Shipped (v0.4.x)
**New since v0.3.0:**
- Kokoro 82M TTS engine + voice profile type system (PR #325)
- Qwen CustomVoice preset engine — predefined speakers with instruct support (PR #328)
- Intel Arc (XPU) GPU support (PR #320)
- Blackwell GPU (sm_120) CUDA support (PR #401)
- Generation cancellation flow (PR #444)
- Frontend quality gates + TypeScript hardening (PR #418)
- macOS Intel (x86_64) PyTorch compatibility (PR #416)
- Frozen-binary import fixes for Kokoro / Chatterbox Multilingual / scipy / transformers (PR #438)
- Linux PipeWire/PulseAudio monitor detection (PR #457)
- Server survives GUI close on Windows (PR #402)
- GPU arch compatibility warning on startup (catches unsupported PyTorch builds)
- cpal Stream playback reliability (PR #405), clip-splitting stability (PR #403)
- torch.from_numpy crash with numpy 2.x in frozen binary (PR #361)
- Async CUDA download lock (PR #428), NUMBA_CACHE_DIR env var (PR #425)
- "Clear failed" history button (PR #412)
- External server GUI startup + data refresh (PR #319)
- Force offline mode for cached Qwen/Whisper models (PR #318)
- macOS 11 ScreenCaptureKit launch crash fix (PR #424)
**Core TTS (cumulative):**
- Qwen3-TTS voice cloning (1.7B and 0.6B models, MLX + PyTorch)
- Qwen CustomVoice (preset speakers, instruct)
- LuxTTS — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual — 23 languages including Hebrew (PR #257)
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
- HumeAI TADA — 1B English + 3B Multilingual (PR #296)
- Kokoro 82M — CPU-realtime, 8 languages, Apache 2.0 (PR #325)
- Multi-engine architecture with thread-safe backend registry (PR #254)
- Chunked TTS generation — engine-agnostic, removes ~500 char limit (PR #266)
- Async generation queue (PR #269)
- Post-processing audio effects system (PR #271)
- Voice profile type system (preset vs cloned, engine compatibility gating)
- Centralized `ModelConfig` registry — no per-engine dispatch maps
- Shared `EngineModelSelector` component
**Infrastructure (cumulative):**
- CUDA backend swap via binary download (PR #252), cu128 upgrade (PR #316), Blackwell/sm_120 (PR #401)
- CUDA backend split into independently versioned server + libs archives (PR #298)
- Intel Arc XPU support (PR #320)
- Docker + web deployment (PR #161)
- Backend refactor: modular architecture, style guide, tooling (PR #285)
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
- Windows support: CUDA detection, cross-platform justfile, server lifecycle (PR #272, #402)
- Linux audio capture via pactl monitor detection (PR #457)
- macOS Intel x86_64 compatibility (PR #416)
- Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large, turbo variants)
- Model management UI with inline download progress + folder migration (PR #268)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching and cancellation (PR #444)
- Streaming generation endpoint (MLX only)
- Audio player freeze fix + UX improvements (PR #293)
- CORS restriction to known local origins (PR #88)
### Abandoned / Backlogged Integrations
| Model | PR / Branch | Reason |
|-------|-------------|--------|
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. PR should be closed. |
| **VoxCPM 1.5 / VoxCPM2** | `voicebox-new-models` research (2026-04-18) | **Backlogged.** See detailed analysis below. |
#### VoxCPM — Evaluation Notes (2026-04-18)
**Project:** [OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM) — tokenizer-free TTS, 2B params (VoxCPM2), end-to-end diffusion autoregressive architecture, 30 languages, 48 kHz output, Apache 2.0, `pip install voxcpm`.
**Why it looked interesting:**
- Clean PyPI install (`pip install voxcpm`)
- Apache 2.0 — commercially safe
- Voice cloning via `reference_wav_path` with optional `prompt_wav_path` + `prompt_text` for "ultimate" cloning
- Streaming API via `generate_streaming()`
- Zero-shot cloning + style control via parenthetical prefixes in text (`(slightly faster, cheerful tone)...`)
- Relatively high-quality output per demos
**Why we backlogged it:**
- **Effectively CUDA-only.** README states `CUDA ≥ 12.0` as hard requirement. Source code's `from_pretrained(device=None|"auto")` claims "preferring CUDA, then MPS, then CPU," but in practice:
- **MPS (Apple Silicon) broken upstream** — OpenBMB/VoxCPM issues #232 (`NotImplementedError: Output channels > 65536 not supported at the MPS device`) and #248 (`IndexError` on M3 Mac) are both open with no resolution.
- **CPU unsupported in the Python package** — issue #256 shows `voxcpm --device cpu` rejected with `unrecognized arguments`. The only CPU path is the third-party **VoxCPM.cpp** GGML engine, which is a separate ecosystem project, not `pip install voxcpm`.
- **macOS source install fails** — issue #233 open with no resolution.
- Would require CUDA-only gating in UI (new `requires_cuda` flag on `ModelConfig`, lock icon + "Requires NVIDIA GPU" in `ModelManagement.tsx` / `EngineModelSelector.tsx`) plus a hard error at `load_model()` as safety net. Doable but adds first-class platform gating that doesn't exist for any other engine today.
- Voicebox's user base skews Apple Silicon (MLX is a primary backend). Shipping a CUDA-only model sets a precedent worth a separate scoping discussion (see issues #419 engine sprawl, #420 platform tiers, PR #465).
**What would change the decision:**
- Upstream fixes MPS crashes (watch issues #232, #248).
- We define an "experimental / CUDA-only" engine tier as part of issue #419 / PR #465, and decide it's acceptable to ship engines that are hidden on non-NVIDIA platforms.
- VoxCPM.cpp matures into a viable CPU path we can wrap (currently separate project, C++/GGML, unclear ergonomics).
**Integration shape if we revive it:** Zero-shot cloning maps naturally to the Chatterbox-style backend (store `ref_audio` + `ref_text` paths in the voice prompt dict, process at generate time). Est. ~250 lines for `voxcpm_backend.py` + one `ModelConfig` entry + engine registration in `backends/__init__.py`. Frontend UI gating is the bigger lift.
### What's In-Flight
| Feature | Branch/PR | Status |
|---------|-----------|--------|
| Platform support tiers | PR #465, issue #420 | Defining tier-1 (supported) vs tier-2 (community) platforms |
| Engine sprawl cleanup | issue #419 | First-class vs experimental TTS backends distinction |
| Frontend tech-debt burn-down | issue #421 | Biome + a11y debt before gating CI |
| Docker registry auto-publish | PR #463, issue #453 | ghcr.io image on tag push |
| New model research | `voicebox-new-models` branch | Evaluating Fish Speech, XTTS-v2, Pocket TTS, VibeVoice, Fish Audio S2, index-tts2 |
### TTS Engine Comparison
| Engine | Model Name | Profile Type | Languages | Size | Key Features | Instruct Support |
|--------|-----------|--------------|-----------|------|-------------|-----------------|
| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | Cloned | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | Cloned | 10 | ~1.2 GB | Lighter, faster | None |
| Qwen CustomVoice 1.7B | `qwen-custom-voice-1.7B` | Preset | 10 | ~3.5 GB | Predefined speakers, instruct support | **Yes** |
| Qwen CustomVoice 0.6B | `qwen-custom-voice-0.6B` | Preset | 10 | ~1.2 GB | Predefined speakers, instruct support | **Yes** |
| LuxTTS | `luxtts` | Cloned | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | `chatterbox-tts` | Cloned | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) |
| Chatterbox Turbo | `chatterbox-turbo` | Cloned | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only |
| TADA 1B | `tada-1b` | Cloned | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | `tada-3b-ml` | Cloned | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
| Kokoro 82M | `kokoro` | Preset | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
### Multi-Engine Architecture (Shipped)
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro'`
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
- **Profile type system** — preset vs cloned profiles, UI grays out incompatible engines and auto-switches on selection
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
### Known Limitations
- **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
- **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
- **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
- **Instruct parameter partially shipped** (#224, #303): Qwen CustomVoice (PR #328) now provides real instruct support via predefined speakers. Other backends still silently drop the instruct field — the UI exposes the field broadly but most engines ignore it. The floating generate box was patched to restore instruct for CustomVoice (commit `106aec4`).
- **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
- **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.
- **Blackwell (RTX 50-series) CUDA**: cu128 + sm_120 kernel support shipped (PR #401, #316), but users still report `cudaErrorNoKernelImageForDevice` (#417, #400, #396, #395, #390, #362) — likely a stale CUDA binary on upgraded installs. Needs a follow-up diagnostic / forced re-download path.
- **Long text 50k character limit** (#464, #365, #354): Still hit on GPU despite chunking (PR #266). Chunking reliability needs another pass.
- **ROCm on RDNA 3/4** (#469): `HSA_OVERRIDE_GFX_VERSION` is hardcoded and harms newer cards.
- **`flash-attn is not installed` warning on every platform (cosmetic, common user complaint)**: Our transformer-based engines (Chatterbox / Qwen) emit `Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.` on every startup, on every platform — we don't pin `flash-attn` in requirements because installing it is fragile and version-sensitive. Fallback is PyTorch SDPA, which is near-FA2 throughput on Ampere+ and is what actually runs. **Per-platform reality:** (a) **macOS/Apple Silicon** — FlashAttention is CUDA-only, irrelevant here; MLX has its own attention kernels. (b) **Linux** — `pip install flash-attn --no-build-isolation` works but takes 20+ min to compile. (c) **Windows** — no official support (Dao-AILab README still says only "Might work"; source builds routinely fail on recent CUDA/MSVC, issues #1715, #1828, #2395). Windows users can install community prebuilt wheels from `kingbri1/flash-attention` or `bdashore3/flash-attention` (latest v2.8.3, Aug 2025; `win_amd64` wheels for CUDA 12.4/12.8, Torch 2.6–2.9, Python 3.10–3.13) matching their exact CUDA/Torch/Python, or use WSL2. **Native-Windows alternatives worth considering as a build-time swap:** SageAttention (thu-ml, Apache 2.0, claims 2–5× over FA2) and xformers (official Windows wheels). **Action for us:** troubleshooting doc now covers it (see `docs/content/docs/overview/troubleshooting.mdx`), and we should optionally suppress the warning via `logging.getLogger(...).setLevel(ERROR)` at backend import since the fallback is functionally fine.
- **WebAudio playback dies after audio-session interruption** (#41, plus an internal repro where the app is backgrounded long enough): WaveSurfer's `AudioContext` gets suspended by macOS — either because another app grabs the audio output, or because the WKWebView throttles when backgrounded. `play()` resolves and `timeupdate` can still fire, but no audio reaches the output. Only app restart fixes it. **Things already tried that didn't work:** (a) swapping WaveSurfer backend away from WebAudio — introduced more bugs, not an option; (b) remount hook on the player — doesn't help because a freshly-created `AudioContext` is born suspended and only resumes on a user gesture. PR #293 was a prior partial fix that doesn't cover this path. **Next thing to try** (not yet attempted — confirmed via grep of `AudioPlayer.tsx`): call `wavesurfer.getMediaElement().getGainNode().context.resume()` on the play button click (the click itself is a valid user gesture), plus a `visibilitychange` + `statechange` listener as belt-and-suspenders. The `ctx.resume()` pattern already exists in the codebase at `useStoryPlayback.ts:52` — just not wired into the main player.
---
## Open PRs — Triage & Analysis
### Recently Merged (Since Last Update — 2026-03-18 → 2026-04-18)
| PR | Title | Merged |
|----|-------|--------|
| **#481** | fix(build): pin transformers in MLX requirements to prevent 5.x upgrade | 2026-04-19 |
| **#470** | fix(api-client): declare moved + errors on migrateModels response type | 2026-04-18 |
| **#457** | fix(linux): use pactl to detect PipeWire/PulseAudio monitor | 2026-04-18 |
| **#450** | docs: clarify paralinguistic tag support in quick start | 2026-04-18 |
| **#447** | fix: delete version rows and files in delete_generations_by_profile | 2026-04-18 |
| **#444** | Fix generation cancellation flow | 2026-04-18 |
| **#440** | fix(paths): strip legacy "data/" prefix when resolving stored paths | 2026-04-18 |
| **#439** | Fix migration dialog hanging when no models are present | 2026-04-18 |
| **#438** | fix(build): repair frozen-binary imports for kokoro/chatterbox-multilingual/scipy/transformers | 2026-04-18 |
| **#433** | fix: warn user when no models to migrate during storage change | 2026-04-18 |
| **#425** | Add NUMBA_CACHE_DIR environment variable | 2026-04-16 |
| **#424** | fix: avoid ScreenCaptureKit launch crash on macOS 11 | 2026-04-16 |
| **#418** | Frontend quality gates + TypeScript hardening | 2026-04-18 |
| **#416** | fix(deps): relax PyTorch requirement for macOS Intel (x86_64) | 2026-04-16 |
| **#412** | feat(history): add "Clear failed" button | 2026-04-16 |
| **#405** | fix: keep cpal Stream alive until playback completes | 2026-04-16 |
| **#403** | fix: prevent intermittent clip splitting failures | 2026-04-16 |
| **#402** | fix: reliably keep server alive after GUI close on Windows | 2026-04-16 |
| **#401** | feat: add Blackwell GPU (sm_120) CUDA support | 2026-04-16 |
| **#394** | fix(history): populate status/error/engine fields from DB row | 2026-04-16 |
| **#384** | Fix: Resolve ModuleNotFoundError in effects service | 2026-04-16 |
| **#361** | fix: torch.from_numpy crash with numpy 2.x in frozen binary | 2026-04-16 |
| **#345** | Fix: "Failed to Save" preset error by resolving backend import path | 2026-03-22 |
| **#344** | fix: include changelog in docker web build | 2026-03-27 |
| **#332** | Fix links in Get Started section of index.mdx | 2026-03-21 |
| **#328** | feat: add Qwen CustomVoice preset engine | 2026-03-27 |
| **#325** | feat: Kokoro 82M TTS engine + voice profile type system | 2026-03-20 |
| **#321** | fix: allows deletion of failed generations | 2026-03-19 |
| **#320** | feat: Intel Arc (XPU) GPU support | 2026-03-21 |
| **#319** | fix: GUI startup with external server + data refresh on server switch | 2026-03-27 |
| **#318** | fix: force offline mode when loading cached models (Qwen TTS & Whisper) | 2026-03-21 |
| **#316** | Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI | 2026-03-18 |
### Currently Open (12 PRs)
| PR | Title | Status | Notes |
|----|-------|--------|-------|
| **#465** | docs: define tier-1 and tier-2 platform support targets | Community PR | Pairs with issue #420. Important for scoping. |
| **#463** | feat(actions): add docker-registry.yml for automatic ghcr.io publishing | Community PR | Pairs with issue #453. Low risk. |
| **#443** | fix: prevent infinite retry loop in offline mode (#434) | Community PR | Fixes reported bug. |
| **#430** | feat: add MiniMax TTS provider support | Community PR | Cloud TTS provider — new direction (external API). Superset of #331? |
| **#331** | feat: add MiniMax Cloud TTS as a built-in engine | Community PR | Likely superseded by #430. Dedupe. |
| **#311** | feat: add CosyVoice2/3 TTS engine | **Close** | Abandoned — output quality too poor. |
| **#253** | Enhance speech tokenizer with 48kHz version | Community PR | Qwen tokenizer upgrade. Still worth reviewing. |
| **#227** | fix: harden input validation & file safety | Community PR | Coupled to #225 (custom models). |
| **#225** | feat: custom HuggingFace voice model support | Community PR | Needs rework for multi-engine arch. |
| **#195** | feat: per-profile LoRA fine-tuning | Draft | Complex. 15 new endpoints. |
| **#154** | feat: Audiobook tab | Community PR | Chunked generation now shipped (#266). |
| **#91** | fix: CoreAudio device enumeration | Draft | macOS audio device handling. |
---
## Open Issues — Categorized
### GPU / Hardware Detection — still the top category
**RTX 50-series (Blackwell / sm_120) cluster — NEW:** #417, #400, #396, #395, #390, #362 all report `cudaErrorNoKernelImageForDevice` / "no kernel image available." sm_120 support shipped in PR #401 + cu128 in PR #316, but users on upgraded installs still hit it — likely stale CUDA binary. Needs a diagnostic that detects binary/GPU-arch mismatch and prompts re-download.
**AMD / ROCm — NEW:** #469 `HSA_OVERRIDE_GFX_VERSION` is hardcoded and breaks RDNA 3/4 cards. #313 DirectML on AMD Ryzen AI Max+ 395 not working.
**Intel Arc:** PR #320 shipped XPU support — may resolve #119.
**General GPU-not-detected (older):** #368, #310, #330, #324, #326, #355 (multi-GPU / eGPU).
**Fix path:** CUDA backend swap (PR #252) + cu128 (PR #316) + sm_120 (PR #401) + GPU-arch warning (`73170d0`) are all in. Remaining work is diagnostics + re-download prompts for users whose binary predates the kernel updates.
### Model Downloads
Still reported. Users get stuck downloads, can't resume, offline mode edge cases.
**Key issues:** #475 (MAC CustomVoice install error), #449 (infinite loading macOS), #445 (can't download CustomVoice), #462 (Qwen requires internet even when loaded — regression from #150), #434 (infinite retry loop offline — PR #443 open), #432 (storage location change hangs when empty — partly fixed by PR #439/#433), #348 (TADA 3B Multilingual download fails), #336 (TADA model not listed in app), #275 (`No module named 'chatterbox'` on download), #304 (whisper-base feature extractor load error), #287 (macOS ARM `check_model_inputs` ImportError on new version), #181, #180.
**Fix path:** PR #443 addresses infinite offline retry. CustomVoice-specific download failures (#475, #445) need triage — likely related to frozen-binary import fixes in PR #438. TADA cluster (#336, #348) and macOS ARM import regressions (#287, #275, #304) need a dedicated triage pass.
**Qwen 0.6B-downloads-1.7B reports:** **#485** (2026-04-19), **#423** (macOS M1), **#329**. Platform-dependent:
- **On MLX (Apple Silicon) — not a bug.** `mlx-community` only publishes 1.7B-Base-bf16 weights, so the 0.6B Base option intentionally resolves to the same repo (`backend/backends/__init__.py:180` — `# 0.6B not available in MLX, falls back`). UX gap: the selector offers a size that doesn't exist on the active backend. Fix: (a) hide the 0.6B option on MLX, or (b) label it "0.6B (uses 1.7B on Apple Silicon)".
- **On PyTorch (Windows/Linux/CUDA/ROCm/XPU/CPU) — real bug if reported.** Both 0.6B and 1.7B have distinct repos (`Qwen/Qwen3-TTS-12Hz-0.6B-Base` vs `-1.7B-Base`). Triage each report by platform before merging into the MLX cluster.
- **Qwen CustomVoice (either platform)** — no fallback, both sizes always have dedicated repos.
### Language Requests (ongoing)
Strong demand: Hungarian (#479), Indonesian (#458, #247), Thai (#455), Bangla (#454), Arabic (#379), Persian (#162), IndicF5 (#339 — Indian languages), Ukrainian (#109), Chinese UI (#392, #261).
**Fix path:** Chatterbox Multilingual (PR #257) covers Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Still missing: Hungarian, Indonesian, Thai, Bangla, Ukrainian. Issue #411 offers a PR for UI i18n foundation.
### New Model Requests (growing)
| Issue | Model Requested |
|-------|----------------|
| #478 | CosyVoice3 (we tried & abandoned CosyVoice2/3 — see #311) |
| #407, #347 | RVC-style voice-to-voice / seed voice conversion (STS) |
| #385 | Fish Audio S2 |
| #380 | OmniVoice |
| #370 | index-tts2 |
| #364 | Voxtral-TTS |
| #335 | Faster-Qwen-TTS |
| #346 | Multi-model batch request |
| #381 | Microsoft MAI models |
| #339 | IndicF5 |
| #226 | GGUF support |
| #172 | VibeVoice |
| #138 | Export to ONNX/Piper format |
| #132 | LavaSR (transcription) |
| #147 | Facebook Omnilingual ASR |
| #338 | Default voices |
The multi-engine architecture makes integration straightforward — see [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx). Platform-specific gating (e.g. VoxCPM CUDA-only) doesn't exist yet and would need design.
### Platform Scope & Quality Debt — NEW category
Awareness issues filed this cycle — ties into engine sprawl and platform tier work.
- **#419** — Engine sprawl: define first-class vs experimental TTS backends
- **#420** — Formalize tier-1 vs tier-2 platform support targets (PR #465 open)
- **#421** — Track & burn down frontend Biome + a11y debt before gating CI
- **#422** — Code-split web build (main bundle > 1 MB)
### Long-Form / Chunking
Still reported despite chunking + queue being merged.
**Key issues:** #464 (50k char limit on GPU despite 16 GB VRAM — v0.4.0), #365 (FR: >50k chars), #363 (smart chunking to prevent robotic artifacts), #354 (50k limit v0.3.0).
**Fix path:** Chunking (#266) and queue (#269) shipped. Remaining work is raising/removing the 50k guard and tuning chunk boundaries for prosody.
### Feature Requests (ongoing)
Notable:
- **#480** — Noise removal on uploaded recordings
- **#448** — API for non-Qwen models (external integrations)
- **#427** — Task status control
- **#407, #347** — Voice-to-voice / audio-to-audio conversion
- **#387** — Location of downloaded generated voices
- **#383** — Concatenate partial reference audio into generated audio
- **#382** — Lightning.ai support
- **#376** — Remote mode
- **#353** — Audio transcoding
- **#317** — Voice pitch control
- **#189** — "Auto" language option
- **#173** — Vocal intonation/inflection control
- **#165, #270** — Audiobook mode (PR #154 open)
- **#242** — Seed value pinning
- **#228** — Always use 0.6B option
- **#235** — Finetuned Qwen3-TTS tokenizer (PR #253 open)
- **#144** — Copy text to clipboard
### Housekeeping / Triage Needed
| Issue | Reason |
|-------|--------|
| **#431**, **#408** | Spam — Chinese "free Claude API" promos. Close. |
| **#398** ("Excelente") | Non-issue. Close. |
| **#357** | Informational — project featured in Awesome MLX. Close after acknowledgement. |
| **#374**, **#377** | Version-release questions, no bug. Close. |
| **#306** ("voice model"), **#389** ("New model"), **#473** ("New functionality") | Title-only issues, no content. Request details or close. |
| **#309** | Uninstall/cleanup question. Answer and close. |
| **#241** | "How to use in Colab" — support question, not a bug. |
| **#423** / **#485** / **#329** | Platform-dependent. On MLX: not a bug (0.6B weights don't exist upstream, fallback is intentional — fix UX). On PyTorch: real bug if reproducible. Classify each by reporter's platform before deduping. |
| **#336** / **#348** | TADA download/registration cluster — triage together. |
| **#287** / **#275** / **#304** | macOS ARM import regressions on new version — likely one root cause. |
| **#292**, **#349** | Possibly already fixed by merged PRs (#321/#412 and #345). Verify + close. |
**~70 older issues (pre-#170) not individually categorized above.** Most are long-tail support questions or duplicates of problems now addressed by the multi-engine / model-registry work. A dedicated backlog-sweep pass is overdue.
### Bugs (ongoing)
| Category | Issues |
|----------|--------|
| Generation failures | #476, #467, #452, #459 (voice clone fetch error), #468 (tada-1b marked error), #437, #300, #301, #282 |
| Audio quality | #456 (clipping errors v0.4.0), #436 (emotion labels), #333 (pitch/echo), #307 (by-model breakdown), #340 (all generations say "www...") |
| Transcription | #371 (fails every time), #291 (extract transcription from generated audio) |
| Effects / presets | #349 ("Failed to save" when creating effects presets — possibly fixed by merged #345) |
| File ops | #477 (spacy_pkuseg dict missing on frozen Windows build), #472 (storage location change), #283 (allow longer files for voice creation + in-app trim), #350 (failed to add sample) |
| History | #292 (can't delete failed generations — possibly fixed by merged #321/#412) |
| Windows | #466 (install problem), #375 (WinError 5 access denied), #273 (port 8000 conflict), #201 (model doesn't stay loaded) |
| Linux | #471 (thread-safe PULSE_SOURCE), #413 (Arch build), #409 (Kubuntu build), #351, #341 |
| macOS | #441 (older macOS), #369 (malware flag), #334 (microphone permission), #287 (`check_model_inputs` ImportError — regression), #171 (ARM64 binary won't open) |
| Profile/UI | #360 (Kokoro profile hides others — partly addressed by auto-switch), #299 (drag-drop on Win11), #329 (size selector state bug), #393 (stuck loading screen after reinstall to new dir) |
| Integrations | #397 (SAMMI-bot 422 Unprocessable Entity) |
| Audio playback / session | **#41** (macOS: Voicebox goes silent after another app takes audio output; restart restores it) — see deep-dive below |
| Database | #174 (sqlite3 IntegrityError) |
---
## Existing Plan Documents — Status
| Document | Target Version | Status | Relevance |
|----------|---------------|--------|-----------|
| `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially superseded** by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
| `CUDA_BACKEND_SWAP.md` | — | **Shipped** (PR #252) | CUDA binary download + backend restart |
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
---
## New Model Integration — Landscape
### Status Snapshot (2026-04-18)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct | Cross-platform? | Status |
|-------|---------|-------|-------------|-----------|------|----------|-----------------|--------|
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None | MLX + PyTorch | **Shipped** |
| **Qwen CustomVoice** | Preset speakers | Medium | 24 kHz | 10 | Medium | **Yes** | PyTorch | **Shipped** (PR #328) |
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | All | **Shipped** (PR #254) |
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` | CPU/CUDA | **Shipped** (PR #257) |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags | CPU/CUDA | **Shipped** (PR #258) |
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), 10 (3B) | Medium | Partial — prosody | PyTorch | **Shipped** (PR #296) |
| **Kokoro-82M** | Preset voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | All | **Shipped** (PR #325) |
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** | — | **Abandoned** (PR #311) — poor output quality |
| ~~**VoxCPM2**~~ | Zero-shot | ~0.15 RTF streaming | 48 kHz | 30 | Medium | Partial — parenthetical style | **CUDA-only in practice** | **Backlogged** (2026-04-18) — see notes above |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — word-level inline | All | Candidate — license TBD |
| **Fish Audio S2** | — | — | — | — | — | — | — | Candidate (#385) |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref | All | Candidate — CPML license likely blocker |
| **Pocket TTS** (Kyutai) | Zero-shot + streaming | >1x RT on CPU | — | English + several European (FR/DE/PT/IT/ES added by Feb 2026) | ~100M | None | CPU-first | Candidate — MIT |
| **MOSS-TTS-Nano** | Zero-shot | **Realtime on 4 CPU cores** | 48 kHz stereo | 20 | 0.1B | Partial — MOSS-VoiceGenerator companion does text-to-voice design | All (ONNX CPU path dropped 2026-04-17) | **Top candidate** — Apache 2.0, released 2026-04-13, streaming |
| **VibeVoice** (Microsoft) | — | — | — | Multi-speaker long-form (up to 90 min, 4 speakers) | 1.5B | — | — | Candidate (#172) — Stories-editor fit |
| **index-tts2** | — | — | — | — | — | — | — | Candidate (#370) |
| **Voxtral TTS** (Mistral) | Zero-shot (short clips) + 20 preset voices | Single-GPU | — | — | 4B (`Voxtral-4B-TTS-2603`) | Presets + cloning | CUDA (16 GB+ VRAM) | Candidate (#364) — frontier quality claim, open-weight |
| **Dia / Dia2** | — | — | — | — | — | — | — | Watch — emotion-forward, but "rough edges" / artifacts per April reviews |
| **IndicF5** | — | — | — | Indian languages | — | — | — | Candidate (#339) — fills Indic gap |
| **MiniMax Cloud TTS** | — | Cloud | — | — | N/A (API) | — | N/A | Community PR #430, #331 — new direction (external API) |
| **OmniVoice** | — | — | — | — | — | — | — | Candidate (#380) |
| **RVC voice conversion** | N/A (STS) | — | — | — | — | N/A | All | New modality, not TTS (#407, #347) |
**Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control), Faster-Qwen-TTS (#335), Orpheus / Sesame CSM (on-device fine-tuning discussions), Fish Audio S2 Pro / Fish Speech V1.5 (benchmark leader but research/non-commercial license — same blocker as Fish Speech).
**Deep-research pass (2026-04-18):** MOSS-TTS-Nano identified as the freshest high-alignment candidate — verified via [OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) README (0.1B params, Apache 2.0, 48 kHz stereo, 4-core CPU realtime, streaming, released 2026-04-13). Dedicated repo: [OpenMOSS/MOSS-TTS-Nano](https://github.com/OpenMOSS/MOSS-TTS-Nano). Voxtral TTS verified on HF as `mistralai/Voxtral-4B-TTS-2603`.
#### Active Evaluation Criteria (learned from cycle)
1. **Cross-platform first.** MLX is a primary backend for our Apple Silicon user base. CUDA-only models require platform gating that doesn't exist yet — shipping one sets a precedent (see VoxCPM notes, issues #419/#420).
2. **PyPI + Apache/MIT licensing preferred.** Heavy deps, git-only installs, and `--no-deps` workarounds are expensive to maintain (Chatterbox taught us this).
3. **Output quality is non-negotiable.** CosyVoice was abandoned despite the best instruct API.
4. **Instruct support fills a real gap** (#173, #224, #303). Qwen CustomVoice partially addresses it with preset speakers; zero-shot clone-with-instruct is still unmet.
5. **Long-form + streaming are user-requested** (#363, #365, #464). Candidates with native streaming (Pocket TTS, Fish Speech) get extra weight.
### Adding a New Engine (Now Straightforward)
With the model config registry and shared `EngineModelSelector` component, adding a new TTS engine requires:
1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
2. **Register in `backend/backends/__init__.py`** — add `ModelConfig` entry + `TTS_ENGINES` entry + factory elif
3. **Update `backend/models.py`** — add engine name to regex
4. **Update frontend** — add to engine union type, `EngineModelSelector` options, form schema, language map, profile type gating (icons/labels ~9 files per grep of `kokoro`)
`main.py` requires **zero changes** — the registry handles all dispatch automatically.
**Platform gating doesn't exist yet.** If we add a CUDA-only model (e.g. VoxCPM), we need a new `requires_cuda` (or more generally `requires: list[device]`) flag on `ModelConfig`, plumbed through `/models` API and surfaced in `ModelManagement.tsx` and `EngineModelSelector.tsx` as a lock icon + "Requires NVIDIA GPU" state. Backend should hard-error at `load_model()` as a safety net.
Total effort: **~1 day** for a well-documented model with a PyPI package, cross-platform. **~2 days** if platform gating is required. See [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx) for the full guide.
---
## Architectural Bottlenecks
### ~~1. Single Backend Singleton~~ — RESOLVED
The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
### ~~2. `main.py` Dispatch Point Duplication~~ — RESOLVED
Previously, each engine required updates to 6+ hardcoded dispatch maps across `main.py` (~320 lines of if/elif chains). A model config registry in `backend/backends/__init__.py` now centralizes all model metadata (`ModelConfig` dataclass) with helper functions (`load_engine_model()`, `check_model_loaded()`, `engine_needs_trim()`, etc.). Adding a new engine requires zero changes to `main.py`.
### ~~3. Model Config is Scattered~~ — RESOLVED
Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the `ModelConfig` registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in `EngineModelSelector.tsx`.
### 4. Voice Prompt Cache Assumes PyTorch Tensors
`backend/utils/cache.py` uses `torch.save()` / `torch.load()`. LuxTTS, Chatterbox, and Kokoro backends work around this by storing reference audio paths (or preset voice IDs) instead of tensors in their voice prompt dicts. Not ideal but functional.
### 5. ~~Frontend Assumes Qwen Model Sizes~~ — RESOLVED
The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen / Qwen CustomVoice.
### 6. No Platform Gating on Models — NEW
`ModelConfig` has no way to express hardware requirements. Every engine is shown to every user, regardless of whether it'll actually load. Users on non-CUDA platforms discover failure at load time (or not at all — some fall back silently to CPU and never complete). Blocks shipping CUDA-only engines (VoxCPM) and would improve the Intel Arc / ROCm / CPU-only UX today. See `ModelConfig` TODO: add `requires: list[Literal["cuda", "mps", "xpu", "cpu", "rocm"]]` or equivalent, plumb through `/models` API, render in `ModelManagement.tsx` + `EngineModelSelector.tsx`.
### 7. Engine Sprawl — NEW
Seven TTS engines shipped, more candidates queued. Issue #419 asks for a first-class vs experimental distinction. Related: issue #420 asks for formalized platform support tiers. Combined, these would let us ship more engines more confidently with clearer expectations for users.
---
## Recommended Priorities
### Tier 1 — Ship Now
| Priority | PR/Item | Impact | Effort |
|----------|---------|--------|--------|
| 1 | **RTX 50-series / Blackwell diagnostic** — detect stale CUDA binary vs GPU arch, prompt re-download (#417, #400, #396, #395, #390, #362) | Large cluster of user-blocking errors | Medium |
| 2 | **CustomVoice download failures** (#475, #445) | New engine blocked on MAC/Win — regression triage | Medium |
| 3 | **50k char limit on GPU** (#464) | Regression — chunking should handle this | Medium |
| 4 | Close PR #311 (CosyVoice) and dedupe #331/#430 (MiniMax) | Housekeeping | None |
| 5 | **PR #443** — infinite offline retry loop | Bug fix, reviewable | Low |
| 6 | **PR #465** — define tier-1 / tier-2 platforms | Unblocks engine-sprawl decision (#419) | Low |
| 7 | **PR #463** — docker registry auto-publish | Community PR, low risk | Low |
| 8 | **#253** — 48kHz speech tokenizer | Quality improvement for Qwen | Medium |
| 9 | **Kokoro profile UX** (#360) — partially addressed by auto-switch | Polish | Low |
### Tier 2 — Feature Work
| Priority | Item | Impact | Effort |
|----------|------|--------|--------|
| 1 | **Engine tier system** (#419) — first-class vs experimental, platform gating in `ModelConfig` | Unblocks CUDA-only engines (VoxCPM, etc.) and frontend polish | Medium |
| 2 | **Frontend tech-debt burn-down** (#421) + code-split (#422) | Before gating CI on Biome | Medium |
| 3 | **#154** — Audiobook tab | Long-form users. Chunking + queue shipped. | Medium |
| 4 | **UI i18n** (#411 PR offer, #392, #261) | Chinese UI + general localization | Medium |
| 5 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
| 6 | OpenAI-compatible API (plan doc exists) — see also #448 (API for non-Qwen) | Low effort once API is stable | Low |
| 7 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
| 8 | Streaming for non-MLX engines | Currently MLX-only | Medium |
| 9 | Voice-to-voice / RVC (#407, #347) | New modality — different arch shape | High |
### Tier 3 — Future Engines (cross-platform preferred)
| Priority | Item | Notes |
|----------|------|-------|
| 1 | **MOSS-TTS-Nano** | 0.1B, Apache 2.0, 4-core CPU realtime, 48 kHz stereo, streaming, 20 langs, released 2026-04-13. Best alignment with our criteria. Verify install ergonomics before committing. |
| 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. Fills streaming gap without CUDA dependency. Several European langs added by Feb 2026. |
| 3 | **IndicF5** | Fills Indian-language gap (#339). Closes many language-request issues. |
| 4 | **VibeVoice** (Microsoft, #172) | 1.5B, long-form multi-speaker (up to 90 min, 4 speakers). Strong Stories-editor fit. |
| 5 | **Voxtral TTS** (Mistral, #364) | 4B presets+cloning. Frontier quality claim, but 16 GB+ VRAM — would need the platform-tier work first. |
| 6 | **Fish Speech / Fish Audio S2** | 50+ langs, word-level instruct. **License clarification first.** (#385) |
| 7 | **XTTS-v2** | 17+ langs, mature pip. CPML likely kills commercial use — verify. |
| 8 | **index-tts2** (#370) | Unvetted. |
| — | ~~**VoxCPM2**~~ | **Backlogged** — CUDA-only upstream. Revisit when tier system ships or MPS bugs are fixed upstream. |
### ~~Previously Prioritized — Now Done~~
- ~~Kokoro 82M — finish integration~~ **Shipped** (PR #325)
- ~~Qwen CustomVoice~~ **Shipped** (PR #328)
- ~~Intel Arc (XPU) support~~ **Shipped** (PR #320)
- ~~Blackwell CUDA~~ **Shipped** (PR #401, follow-up work open)
- ~~Generation cancellation~~ **Shipped** (PR #444)
- ~~macOS Intel x86_64~~ **Shipped** (PR #416)
---
## Branch Inventory
| Branch | PR | Status | Notes |
|--------|-----|--------|-------|
| `voicebox-new-models` | — | **Active** | New model research (Fish Speech, Pocket TTS, VibeVoice, etc.); VoxCPM evaluated & backlogged |
| `fix/kokoro-pyinstaller-source-files` | — | Active | Kokoro frozen-build source bundling (parent of `voicebox-new-models`) |
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
| `feat/kokoro` | #325 | **Merged** | Kokoro 82M + voice profile type system |
| `feat/qwen-custom-voice` | #328 | **Merged** | Qwen CustomVoice preset engine |
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
---
## Quick Reference: API Endpoints
<details>
<summary>All current endpoints</summary>
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/health` | GET | Health check, model/GPU status |
| `/profiles` | POST, GET | Create/list voice profiles |
| `/profiles/{id}` | GET, PUT, DELETE | Profile CRUD |
| `/profiles/{id}/samples` | POST, GET | Add/list voice samples |
| `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management |
| `/profiles/{id}/export` | GET | Export profile as ZIP |
| `/profiles/import` | POST | Import profile from ZIP |
| `/generate` | POST | Generate speech (engine param selects TTS backend) |
| `/generate/stream` | POST | Stream speech (MLX only) |
| `/history` | GET | List generation history |
| `/history/{id}` | GET, DELETE | Get/delete generation |
| `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
| `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model |
| `/models/load` | POST | Load model into memory |
| `/models/unload` | POST | Unload model |
| `/models/progress/{name}` | GET | SSE download progress |
| `/tasks/active` | GET | Active downloads/generations (with inline progress) |
| `/stories` | POST, GET | Create/list stories |
| `/stories/{id}` | GET, PUT, DELETE | Story CRUD |
| `/stories/{id}/items` | POST, GET | Story items CRUD |
| `/stories/{id}/export` | GET | Export story audio |
| `/channels` | POST, GET | Audio channel CRUD |
| `/channels/{id}` | PUT, DELETE | Channel update/delete |
| `/cache/clear` | POST | Clear voice prompt cache |
| `/server/cuda/status` | GET | CUDA binary availability |
| `/server/cuda/download` | POST | Download CUDA binary |
| `/server/cuda/switch` | POST | Switch to CUDA backend |
</details>
+13 -42
View File
@@ -7,61 +7,32 @@ This directory contains the documentation for Voicebox, built with [Fumadocs](ht
## Development
### Prerequisites
Install Mintlify globally using bun:
```bash
bun add -g mintlify
```
Or use the helper script:
```bash
bun run install:mintlify
```
### Running Locally
From the `docs/` directory:
```bash
bun install
bun run dev
```
This will start the Mintlify dev server.
The docs will be available at `http://localhost:3000`
The docs will be available at `http://localhost:3000`.
### Structure
```
docs/
├── mint.json # Mintlify configuration
├── custom.css # Custom styles
├── overview/ # Getting started & feature docs
├── guides/ # User guides
├── api/ # API reference
├── development/ # Developer documentation
├── logo/ # Logo assets
└── public/ # Static assets
```
- `content/docs/overview/` — user-facing guides (installation, quick start, feature walkthroughs)
- `content/docs/developer/` — architecture, backend internals, and contributor guides
- `content/docs/api-reference/` — auto-generated from the backend's OpenAPI schema
- `content/docs/index.mdx` — landing page
- `public/` — static assets (images, screenshots, videos)
### Writing Docs
- Use `.mdx` files for all documentation pages
- Follow the existing structure in `mint.json` for navigation
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
- Navigation is generated from `content/docs/meta.json` files
- Fumadocs components available: `Callout`, `Cards` / `Card`, `Tabs` / `Tab`, `Steps` / `Step`, `Accordion` / `AccordionGroup`, `Files` / `Folder` / `File`
- API reference pages under `api-reference/` are regenerated from the backend's OpenAPI schema — don't edit them by hand
## Deployment
Docs are automatically deployed when changes are pushed to the main branch.
To manually deploy:
```bash
mintlify deploy
```
## Contributing
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
Docs are automatically deployed when changes land on `main`.
-311
View File
@@ -1,311 +0,0 @@
---
title: "Troubleshooting Guide"
description: "Common issues and solutions for Voicebox"
---
Common issues and solutions for Voicebox.
## Installation Issues
### macOS: "Voicebox cannot be opened because it is from an unidentified developer"
**Solution:**
1. Right-click the `.dmg` file
2. Select "Open"
3. Click "Open" in the security dialog
4. Alternatively, go to System Settings → Privacy & Security → Allow Voicebox
### Windows: "Windows protected your PC"
**Solution:**
1. Click "More info"
2. Click "Run anyway"
3. Windows Defender may flag new software; this is normal for unsigned apps
### Linux: AppImage won't run
**Solution:**
```bash
chmod +x voicebox-*.AppImage
./voicebox-*.AppImage
```
## Runtime Issues
### Server won't start
**Symptoms:** App opens but shows "Server not connected"
**Solutions:**
1. **Check Python installation**
```bash
python --version # Should be 3.11+
```
2. **Check server binary exists**
- Look in `tauri/src-tauri/binaries/` for your platform
- Binary should match your system architecture
3. **Check permissions**
```bash
# macOS/Linux
chmod +x tauri/src-tauri/binaries/voicebox-server-*
```
4. **Check logs**
- macOS: Open Console.app and search for "voicebox"
- Linux: Check `~/.local/share/voicebox/` for logs
- Windows: Check Event Viewer
### "Model download failed"
**Symptoms:** First generation fails with download error
**Solutions:**
1. **Check internet connection**
- Models download from HuggingFace Hub (~2-4GB)
- First download may take several minutes
2. **Check disk space**
- Models are cached in `~/.cache/huggingface/`
- Ensure at least 5GB free space
3. **Manual download** (if automatic fails)
```bash
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
```
### "Out of memory" errors
**Symptoms:** Generation fails with CUDA/VRAM errors
**Solutions:**
1. **Use smaller model**
- Switch to 0.6B model instead of 1.7B
- Settings → Model Management → Load 0.6B
2. **Close other applications**
- Free up GPU memory
- Close browser tabs, other ML apps
3. **Use CPU mode**
- Slower but works without GPU
- Backend automatically falls back to CPU
### MLX "Failed to load the default metallib" error (Apple Silicon)
**Symptoms:** Generation fails with "library not found" or "metallib" errors
**Solutions:**
1. **Rebuild server binary**
```bash
bun run build:server
```
The build script should automatically include MLX Metal shader libraries.
2. **Check MLX installation**
```bash
pip install -r backend/requirements-mlx.txt
```
3. **Verify backend detection**
- Check server logs for "Backend: MLX"
- If showing "Backend: PYTORCH", MLX may not be installed correctly
### Audio playback issues
**Symptoms:** Generated audio won't play
**Solutions:**
1. **Check audio format**
- Audio is saved as WAV files
- Ensure your system supports WAV playback
2. **Try downloading audio**
- Right-click → Download
- Play in external player
3. **Check browser permissions** (web version)
- Allow audio autoplay in browser settings
### Slow generation
**Symptoms:** Generation takes >30 seconds
**Solutions:**
1. **Check backend type** (Apple Silicon)
- Check Settings → Server Status
- Should show "Backend: MLX" on Apple Silicon
- If showing "Backend: PYTORCH", install MLX: `pip install -r backend/requirements-mlx.txt`
- MLX provides 4-5x faster inference on Apple Silicon
2. **Use GPU** (if available)
- Check Settings → Server Status
- Should show "GPU available: true"
- Apple Silicon: Should show "Metal (Apple Silicon via MLX)"
- Windows/Linux: Should show "CUDA" if GPU available
3. **Enable caching**
- Voice prompts are cached automatically
- Second generation with same voice should be faster
4. **Use smaller model**
- 0.6B model is faster than 1.7B
- Quality difference is minimal for most voices
5. **Check system resources**
- Close other CPU/GPU intensive apps
- Ensure adequate RAM (8GB+ recommended)
## API Issues
### "Connection refused" when using API
**Solutions:**
1. **Check server is running**
```bash
curl http://localhost:17493/health
```
2. **Check remote mode**
- If connecting remotely, ensure server is started with `--host 0.0.0.0`
- Check firewall settings
3. **Check port availability**
- The current local app and dev workflow uses port 17493 by default
- Ensure no other service is using it
### CORS errors in browser
**Solutions:**
1. **Use desktop app** (recommended)
- Desktop app doesn't have CORS restrictions
2. **Configure CORS** (for web deployment)
- Update `backend/main.py` CORS settings
- Add your domain to allowed origins
## Update Issues
### "Update check failed"
**Solutions:**
1. **Check internet connection**
- Updates are fetched from GitHub releases
2. **Check GitHub access**
- Ensure `github.com` is accessible
- Check firewall/proxy settings
3. **Manual update**
- Download latest release from GitHub
- Install manually
### "Invalid signature" error
**Solutions:**
1. **Re-download installer**
- Signature may be corrupted
- Download fresh copy from GitHub
2. **Check release integrity**
- Verify `.sig` file matches installer
- Report issue if signature is invalid
## Data Issues
### Profiles disappeared
**Solutions:**
1. **Check data directory**
- macOS: `~/Library/Application Support/voicebox/`
- Windows: `%APPDATA%/voicebox/`
- Linux: `~/.local/share/voicebox/`
2. **Check database**
- Database: `data/voicebox.db`
- Ensure file exists and is readable
3. **Restore from backup**
- Profiles can be exported/imported
- Check for backup files
### "Database locked" error
**Solutions:**
1. **Close other instances**
- Ensure only one Voicebox instance is running
2. **Restart app**
- Close and reopen Voicebox
3. **Check file permissions**
- Ensure database file is writable
- Check directory permissions
## Development Issues
### Build fails
**Solutions:**
1. **Check Rust installation**
```bash
rustc --version
rustup update
```
2. **Check Tauri dependencies**
```bash
cd tauri
bun install
```
3. **Clean build**
```bash
cd tauri/src-tauri
cargo clean
cd ../..
bun run build
```
### API client generation fails
**Solutions:**
1. **Start backend server**
```bash
bun run dev:server
```
2. **Check OpenAPI endpoint**
```bash
curl http://localhost:17493/openapi.json
```
3. **Regenerate client**
```bash
bun run generate:api
```
## Still Having Issues?
1. **Check existing issues**
- Search GitHub issues for similar problems
- Check closed issues for solutions
2. **Create new issue**
- Include:
- OS and version
- Voicebox version
- Steps to reproduce
- Error messages/logs
- Screenshots (if applicable)
3. **Get help**
- Check documentation in `docs/`
- Review `backend/README.md` for API details
- See `CONTRIBUTING.md` for development help
---
For more help, open an issue on [GitHub](https://github.com/jamiepine/voicebox/issues).
+116 -94
View File
@@ -9,9 +9,9 @@ Voicebox uses a client-server architecture with a React frontend and Python back
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
**Backend Layer:** A Python FastAPI server that hosts the REST API, runs a pluggable registry of TTS and STT engines, manages the SQLite database, and handles audio processing.
These two layers communicate via HTTP, with the frontend making API requests to the backend.
These two layers communicate via HTTP on `localhost:17493`, with the frontend making API requests to the backend. In production the backend is compiled with PyInstaller and launched as a Tauri sidecar; in development it's run manually via `uvicorn`.
## Frontend Architecture
@@ -29,43 +29,33 @@ These two layers communicate via HTTP, with the frontend making API requests to
<Files>
<Folder name="app/src" defaultOpen>
<Folder name="components">
<File name="profiles/" />
<File name="generation/" />
<File name="stories/" />
<File name="shared/" />
<File name="Profiles/" />
<File name="Generation/" />
<File name="Stories/" />
<File name="ServerSettings/" />
</Folder>
<Folder name="lib">
<File name="api/" />
<File name="constants/" />
<File name="hooks/" />
<File name="utils/" />
</Folder>
<Folder name="hooks" />
<Folder name="stores" />
</Folder>
</Files>
### State Management
```typescript
// Example: Profile store
const useProfileStore = create((set) => ({
profiles: [],
selectedProfile: null,
setProfiles: (profiles) => set({ profiles }),
selectProfile: (id) => set({ selectedProfile: id })
}))
```
## Backend Architecture
### Tech Stack
- **Framework**: FastAPI (Python 3.11+)
- **TTS Model**: Qwen3-TTS
- **Transcription**: Whisper
- **Database**: SQLite
- **Audio**: librosa, soundfile
- **TTS Engines**: Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro
- **Transcription**: Whisper (PyTorch or MLX-Whisper)
- **Inference Backends**: MLX (Apple Silicon), PyTorch (CUDA / ROCm / XPU / DirectML / CPU)
- **Database**: SQLite via SQLAlchemy
- **Audio**: librosa, soundfile, Pedalboard
### API Structure
### Layout
<Files>
<Folder name="backend" defaultOpen>
@@ -74,21 +64,31 @@ const useProfileStore = create((set) => ({
<File name="config.py" />
<File name="models.py" />
<File name="server.py" />
<File name="build_binary.py" />
<Folder name="routes">
<File name="profiles.py" />
<File name="generate.py" />
<File name="history.py" />
<File name="..." />
<File name="models.py" />
<File name="channels.py" />
</Folder>
<Folder name="services">
<File name="generation.py" />
<File name="task_queue.py" />
<File name="..." />
<File name="profiles.py" />
<File name="channels.py" />
</Folder>
<Folder name="backends">
<File name="__init__.py" />
<File name="base.py" />
<File name="..." />
<File name="pytorch_backend.py" />
<File name="mlx_backend.py" />
<File name="qwen_custom_voice_backend.py" />
<File name="luxtts_backend.py" />
<File name="chatterbox_backend.py" />
<File name="chatterbox_turbo_backend.py" />
<File name="hume_backend.py" />
<File name="kokoro_backend.py" />
</Folder>
<Folder name="database">
<File name="models.py" />
@@ -97,49 +97,75 @@ const useProfileStore = create((set) => ({
<Folder name="utils">
<File name="audio.py" />
<File name="effects.py" />
<File name="..." />
</Folder>
</Folder>
</Files>
### Request Flow
HTTP request → **routes/** (validate input, parse params) → **services/** (business logic, orchestration) → **backends/** (TTS/STT inference) → **utils/** (audio processing)
An HTTP request enters a **route handler**, which validates input and delegates to a **service** function. The service calls into the appropriate **engine backend** via the registry, which runs the actual inference. Audio post-processing runs through **utils** (trim, resample, effects).
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
Route handlers are intentionally thin — they validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
### Multi-Engine Registry
The backend is designed so that adding a new TTS engine only requires touching the `backends/` directory and the central registry. There is no per-engine branching in routes or services.
- **`TTSBackend` Protocol** (`backends/__init__.py`) — defines the contract every engine implements: `load_model`, `create_voice_prompt`, `combine_voice_prompts`, `generate`, `unload_model`, `is_loaded`, `_get_model_path`.
- **`ModelConfig` dataclass** — central metadata record for each model variant: `model_name`, `display_name`, `engine`, `hf_repo_id`, `size_mb`, `needs_trim`, `languages`, `supports_instruct`, etc.
- **`TTS_ENGINES` dict** — maps engine name (`"qwen"`, `"kokoro"`, etc.) to display name.
- **`get_tts_backend_for_engine(engine)`** — thread-safe factory that lazily instantiates and caches the backend for an engine using double-checked locking.
Shipped engines:
| Engine key | Display name | Profile type |
|------------|--------------|--------------|
| `qwen` | Qwen TTS | Cloned |
| `qwen_custom_voice` | Qwen CustomVoice | Preset |
| `luxtts` | LuxTTS | Cloned |
| `chatterbox` | Chatterbox TTS | Cloned |
| `chatterbox_turbo` | Chatterbox Turbo | Cloned |
| `tada` | TADA | Cloned |
| `kokoro` | Kokoro | Preset |
See [TTS Engines](/developer/tts-engines) for the full contract and integration phases, and [PROJECT_STATUS.md](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for candidates under evaluation.
### Key Modules
- **app.py** — FastAPI app factory, CORS, lifecycle events
- **main.py** — Entry point (imports app, runs uvicorn)
- **server.py** — Tauri sidecar launcher, parent-pid watchdog
- **services/generation.py** — Single function handling all generation modes (generate, retry, regenerate)
- **services/task_queue.py** — Serial generation queue for GPU inference
- **backends/__init__.py** — Protocol definitions and backend factory
- **backends/base.py** — Shared utilities across all engine implementations
- **`app.py`** — FastAPI app factory, CORS, lifecycle events
- **`main.py`** — Entry point (imports app, runs uvicorn)
- **`server.py`** — Tauri sidecar launcher, parent-pid watchdog, frozen-build environment setup
- **`services/generation.py`** — Single function handling all generation modes (generate, retry, regenerate)
- **`services/task_queue.py`** — Serial generation queue for GPU inference
- **`backends/__init__.py`** — Protocol definitions, `ModelConfig` registry, and engine factory
- **`backends/base.py`** — Shared utilities across all engine implementations (device selection, progress tracking, output trimming)
### Backend Selection
### Inference Backend Selection
The server detects the best inference backend at startup:
The server detects the best inference backend at startup and uses it for all engines that support it:
| Platform | Backend | Acceleration |
|----------|---------|-------------|
|----------|---------|--------------|
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
| Windows / Linux (NVIDIA) | PyTorch | CUDA (cu128) |
| Linux (AMD) | PyTorch | ROCm |
| Intel Arc | PyTorch | IPEX / XPU |
| Windows (any GPU) | PyTorch | DirectML |
| Windows / Linux (Intel Arc) | PyTorch | XPU (IPEX) |
| Windows (other GPU) | PyTorch | DirectML |
| Any | PyTorch | CPU fallback |
See [GPU Acceleration](/overview/gpu-acceleration) for platform-specific notes and manual overrides.
### Data Model
The database uses three main tables:
Core tables (see `backend/database/models.py`):
**Profile Table:** Stores voice profiles with fields for id, name, and language.
- **`profiles`** — Voice profiles with `voice_type` discriminator (`cloned` | `preset` | `designed`), `preset_engine`, `preset_voice_id`, and `default_engine`.
- **`profile_samples`** — Reference audio clips + transcripts for cloned profiles. Empty for preset profiles.
- **`generations`** — Generated audio with text, engine, model, language, seed, and duration.
- **`generation_versions`** — Processed variants of a generation with different effects chains applied.
- **`audio_channels`** + **`channel_device_mappings`** + **`profile_channel_mappings`** — Multi-output routing.
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
See [Voice Profiles](/developer/voice-profiles) and [Effects Pipeline](/developer/effects-pipeline) for details.
## Desktop App (Tauri)
@@ -148,6 +174,7 @@ The database uses three main tables:
<Files>
<Folder name="tauri/src-tauri" defaultOpen>
<File name="Cargo.toml" />
<File name="tauri.conf.json" />
<File name="src/" />
<Folder name="binaries" />
</Folder>
@@ -158,82 +185,74 @@ The database uses three main tables:
- Launch Python backend as sidecar process
- Native file dialogs
- System tray integration
- Auto-updates
- OS-specific features
- Auto-updates (Tauri updater + custom CUDA backend swap)
- Parent-PID watchdog so the backend exits if the app crashes
## Build Process
### Development
```bash
# Frontend (Vite dev server)
cd app && bun run dev
# Backend (manual start)
cd backend && uvicorn main:app --reload
# Desktop app (connects to manual backend)
bun run dev
just dev # Starts backend + Tauri app
just dev-web # Starts backend + web app (no Tauri)
just dev-backend # Backend only
just dev-frontend # Tauri app only (backend must be running)
```
### Production
```bash
# Build everything (server binary + Tauri app)
bun run build
# Or build separately:
# 1. Build server binary (PyInstaller)
bun run build:server
# 2. Build Tauri app (includes server)
cd tauri && bun run tauri build
just build # CPU server binary + Tauri installer
just build-local # CPU + CUDA binaries + Tauri installer (Windows)
just build-server # Server binary only
just build-tauri # Tauri app only
```
See [Building](/developer/building) for what PyInstaller does and how the CUDA binary is split and packaged separately.
## Data Flow
### Generation Flow
When a user generates speech, the data flows through the following stages:
1. **User Input** - User enters text in a React component
2. **State Update** - Text is stored in Zustand state
3. **API Request** - React Query mutation triggers an API call via fetch
4. **Backend Processing** - FastAPI endpoint receives the request
5. **TTS Generation** - Qwen3-TTS model generates the audio
6. **Storage** - Audio file is saved to disk and a database record is created
7. **Response** - Backend returns the audio URL
8. **Cache Update** - React Query updates its cache with the response
9. **UI Update** - Component re-renders with new data
10. **Playback** - User can play the generated audio
1. **User Input** — text entered in a React component, engine + profile selected
2. **State Update** — Zustand generation form store records the request
3. **API Request** — React Query mutation hits `POST /generate`
4. **Route** — `routes/generate.py` validates input, dispatches to `services/generation.py`
5. **Voice Prompt** — the service creates or retrieves a cached voice prompt via the engine's backend
6. **Queue** — `services/task_queue.py` serializes generation to avoid GPU contention
7. **Inference** — the engine backend runs `generate()` and returns audio + sample rate
8. **Post-process** — optional trim (for engines that need it), effects chain applied per generation version
9. **Storage** — audio written to the generations directory, metadata saved to SQLite
10. **Response** — backend returns the generation record; frontend updates React Query cache and plays audio
## Performance Considerations
### Frontend
- **Code splitting** - Lazy load routes
- **Memoization** - React.memo for heavy components
- **Virtual scrolling** - For large lists
- **Debouncing** - Search and input handling
- **Code splitting** — lazy-load routes
- **Memoization** — `React.memo` for heavy components
- **Virtual scrolling** — for large lists
- **Debouncing** — search and input handling
### Backend
- **Async operations** - All I/O is async
- **Model caching** - Keep TTS model in memory
- **Voice prompt caching** - Reuse embeddings
- **Connection pooling** - Database connections
- **Async I/O** — all I/O is async; inference runs in `asyncio.to_thread`
- **Serial task queue** — avoids multiple engines fighting for the GPU
- **Voice prompt caching** — engine-specific, keyed by audio hash + reference text
- **Model pinning** — only one model per engine loaded at a time; switching unloads the previous one
- **Per-engine backend cache** — engines are only instantiated once per process
## Security
### Current
- Local-only by default
- Local-only by default (bound to `127.0.0.1:17493`)
- No authentication (localhost trust)
- File system sandboxing via Tauri
### Planned
- API key authentication
- API key authentication for remote mode
- User accounts
- Rate limiting
- HTTPS support
@@ -248,17 +267,20 @@ When a user generates speech, the data flows through the following stages:
### Remote Mode
- Backend on separate machine
- Frontend connects via HTTP
- Shared infrastructure possible
- Backend on a separate machine (Docker or bare host)
- Frontend (desktop or web) connects over HTTP
- See [Remote Mode](/overview/remote-mode) and [Docker](/overview/docker)
## Next Steps
<Cards>
<Card title="Development Setup" href="/development/setup">
<Card title="Development Setup" href="/developer/setup">
Set up your dev environment
</Card>
<Card title="Contributing" href="/development/contributing">
<Card title="TTS Engines" href="/developer/tts-engines">
How to add a new engine
</Card>
<Card title="Contributing" href="/developer/contributing">
Contribute to Voicebox
</Card>
</Cards>
+33 -41
View File
@@ -145,64 +145,55 @@ The updater only works in production Tauri builds. It doesn't run during `just d
## CUDA Backend Updates
The CUDA-enabled backend is distributed separately from the main app due to its large size (~2.43 GB). Unlike the Tauri auto-updater, this uses a custom download system built into the Python backend.
The CUDA-enabled backend is distributed separately from the main app because bundling CUDA would bloat the installer by several gigabytes for users who don't have an NVIDIA GPU. Unlike the Tauri auto-updater, the CUDA backend uses a custom download system built into the Python server.
**Size comparison:**
- Standard app bundle: ~410 MB
- CUDA backend binary: ~2.43 GB (6× larger)
**Size comparison (approximate):**
- Standard CPU bundle (in the installer): ~200–400 MB
- CUDA server core: ~945 MB (versioned with each Voicebox release)
- CUDA libs (NVIDIA runtime DLLs): ~1.7 GB (versioned independently, cached across upgrades)
### Why Split?
### Two-archive split
GitHub Releases has file size limits, and the CUDA-enabled `voicebox-server` binary is too large to include in the main Tauri bundle. Instead:
Since v0.4, the CUDA binary is packaged as **two archives** instead of one:
- **Standard release**: Includes CPU-only backend (~50MB)
- **CUDA release**: Split into multiple parts and downloaded on-demand by users who need GPU acceleration
- **Server core** (`voicebox-server-cuda.tar.gz`) — the Python server + PyTorch code, changes every release.
- **CUDA libs** (`cuda-libs-cu128-v1.tar.gz`) — the heavy NVIDIA CUDA/cuDNN DLLs, only re-downloaded when the CUDA toolkit major version changes.
This means most Voicebox upgrades only re-download the ~945 MB server core, not the full ~2.5 GB bundle.
### Download Process
When a user clicks "Enable CUDA" in the settings:
When a user clicks "Install CUDA backend" in Settings → GPU:
1. **Manifest Fetch** - Backend fetches `{version}/voicebox-server-cuda.manifest` from GitHub Releases
2. **Part Download** - Downloads each split part sequentially (e.g., `voicebox-server-cuda.part1`, `.part2`, etc.)
3. **Assembly** - Concatenates parts into a single binary
4. **Verification** - SHA-256 checksum verification (optional, if `.sha256` file exists)
5. **Placement** - Binary moved to `{data_dir}/backends/voicebox-server-cuda.exe`
6. **Restart** - Backend must restart to use the CUDA binary
1. **Server-core archive** — Downloaded from GitHub Releases and extracted.
2. **CUDA libs archive** — Downloaded separately (or reused if the installed version still matches).
3. **Verification** — SHA-256 checksum verification for integrity.
4. **Placement** — Extracted into `{data_dir}/backends/cuda/`.
5. **Restart** — The Voicebox server restarts and swaps in the CUDA backend.
### Auto-Update on Startup
On server startup, `check_and_update_cuda_binary()` compares the installed CUDA binary version with the app version:
```python
# backend/services/cuda.py
cuda_version = get_cuda_binary_version() # runs `voicebox-server-cuda --version`
current_version = __version__
if cuda_version != current_version:
await download_cuda_binary() # Auto-download in background
```
If versions mismatch, the backend automatically downloads the matching CUDA binary version without user intervention.
On startup, the backend compares the installed CUDA server-core version with the current app version. If they differ, the core archive is pulled in the background. If the libs version pinned by the new release also differs (rare — e.g. on a cu126 → cu128 bump), the user is prompted to confirm the larger download.
### Storage Location
Downloaded CUDA binaries are stored in the app's data directory:
Downloaded CUDA binaries live in the app's data directory:
```
{data_dir}/
backends/
voicebox-server-cuda.exe # Windows
voicebox-server-cuda # macOS/Linux
{data_dir}/backends/cuda/
voicebox-server-cuda.exe # Windows
voicebox-server-cuda # macOS/Linux
<NVIDIA CUDA runtime DLLs>
```
### API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/backend/cuda-status` | GET | Check if CUDA binary available/active |
| `/backend/download-cuda` | POST | Start download |
| `/backend/cuda-status` | GET | Check if the CUDA backend is available/active and which versions are installed |
| `/backend/download-cuda` | POST | Trigger server-core + libs download |
| `/backend/cuda-progress` | GET | SSE stream of download progress |
| `/backend/cuda` | DELETE | Remove downloaded binary |
| `/backend/cuda` | DELETE | Remove the downloaded CUDA backend |
### Progress Tracking
@@ -212,15 +203,16 @@ Downloads report progress via Server-Sent Events (SSE):
GET /backend/cuda-progress
event: progress
data: {"current": 52428800, "total": 104857600, "filename": "Downloading CUDA backend (2/4)", "status": "downloading"}
data: {"current": 52428800, "total": 945000000, "filename": "voicebox-server-cuda.tar.gz", "status": "downloading"}
```
The frontend subscribes to this endpoint to show real-time download progress in the UI.
The frontend subscribes to this endpoint to show real-time progress, including which archive (server core vs libs) is currently downloading.
### Release Artifacts
For each release, these CUDA-related files are uploaded to GitHub:
For each CUDA-capable release, these files are uploaded to GitHub:
- `voicebox-server-cuda.manifest` - List of split part filenames
- `voicebox-server-cuda.part1` through `voicebox-server-cuda.partN` - Binary chunks
- `voicebox-server-cuda.sha256` - SHA-256 checksum for integrity verification
- `voicebox-server-cuda.tar.gz` — server-core archive
- `voicebox-server-cuda.tar.gz.sha256` — checksum
- `cuda-libs-cu128-v1.tar.gz` — CUDA runtime libs (only when the libs version bumps)
- `cuda-libs-cu128-v1.tar.gz.sha256` — checksum
+18 -10
View File
@@ -17,9 +17,10 @@ Thank you for your interest in contributing to Voicebox! This guide will help yo
Before you start contributing, make sure you have:
1. **Read the documentation** to understand how Voicebox works
2. **Set up your development environment** - see [Development Setup](/development/setup)
2. **Set up your development environment** — see [Development Setup](/developer/setup)
3. **Explored the codebase** to understand the project structure
4. **Checked existing issues** to see if someone else is working on something similar
4. **Checked [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md)** — the living engineering roadmap that tracks prioritized tasks (Tier 1 → 3), architectural bottlenecks, and candidate TTS engines under evaluation (including why some are backlogged)
5. **Checked existing issues** to see if someone else is working on something similar
## Ways to Contribute
@@ -173,10 +174,15 @@ When creating a pull request:
<File name="stores/" />
</Folder>
<Folder name="backend">
<File name="app.py" />
<File name="main.py" />
<File name="tts.py" />
<File name="database.py" />
<File name="server.py" />
<File name="models.py" />
<Folder name="routes" />
<Folder name="services" />
<Folder name="backends" />
<Folder name="database" />
<Folder name="utils" />
</Folder>
<Folder name="tauri">
<File name="src-tauri/" />
@@ -197,9 +203,10 @@ When creating a pull request:
### New Features
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
- Check [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) and the [roadmap](https://github.com/jamiepine/voicebox#roadmap) before proposing work — the status doc lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
- Discuss major features in an issue first
- Keep features focused and well-scoped
- Adding a new TTS engine? See [TTS Engines](/developer/tts-engines) for the phased workflow
### Documentation
@@ -253,7 +260,7 @@ When adding new API endpoints:
<Step title="Regenerate Client">
```bash
bun run generate:api
just generate-api
```
This updates the TypeScript client with type-safe bindings.
@@ -263,7 +270,7 @@ When adding new API endpoints:
The API documentation is automatically generated from the OpenAPI schema. Ensure your endpoint has proper docstrings and type hints, then regenerate the docs:
```bash
bun run generate:api
just generate-api
```
</Step>
</Steps>
@@ -324,8 +331,9 @@ By contributing, you agree that your contributions will be licensed under the MI
If you have questions:
1. Check the [documentation](/overview/introduction)
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
3. Open a new issue or discussion
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
2. Read [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for current engineering priorities
3. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
4. Open a new issue or discussion
5. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
Thank you for contributing to Voicebox! 🎉
+18 -6
View File
@@ -15,17 +15,24 @@ The history module tracks all generated audio, providing a searchable record of
class Generation(Base):
__tablename__ = "generations"
id = Column(String, primary_key=True)
profile_id = Column(String, ForeignKey("profiles.id"))
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
text = Column(Text, nullable=False)
language = Column(String, default="en")
audio_path = Column(String, nullable=False)
duration = Column(Float, nullable=False)
audio_path = Column(String, nullable=True)
duration = Column(Float, nullable=True)
seed = Column(Integer)
instruct = Column(Text)
created_at = Column(DateTime)
engine = Column(String, default="qwen")
model_size = Column(String, nullable=True)
status = Column(String, default="completed") # pending | completed | failed
error = Column(Text, nullable=True)
is_favorited = Column(Boolean, default=False)
created_at = Column(DateTime, default=datetime.utcnow)
```
Each generation can also have multiple **generation versions** — processed variants with different effects chains applied. The original (`clean`) version plus any number of processed versions live in a separate `generation_versions` table. See [Effects Pipeline](/developer/effects-pipeline).
## File Storage
Generated audio is stored in:
@@ -230,7 +237,12 @@ GET /history?profile_id=uuid&search=hello&limit=50&offset=0
"duration": 1.5,
"seed": 42,
"instruct": null,
"created_at": "2024-01-15T10:30:00Z"
"engine": "qwen",
"model_size": "1.7B",
"status": "completed",
"error": null,
"is_favorited": false,
"created_at": "2026-04-18T10:30:00Z"
}
],
"total": 150
+152 -294
View File
@@ -1,341 +1,199 @@
---
title: "Model Management"
description: "How model downloading, loading, and status tracking works in Voicebox"
description: "How model downloading, loading, and status tracking works across all engines"
---
## Overview
Voicebox manages two types of models:
Voicebox manages two categories of models:
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
**TTS Models** — Seven engines covering zero-shot cloning and preset voices. Each engine may have one or more size variants.
**ASR Models:** Whisper for transcription (tiny through large).
**ASR Models** — Whisper for transcription. Five sizes, plus MLX-Whisper on Apple Silicon for ~8× faster transcription.
Models are downloaded from HuggingFace Hub on first use and cached locally.
Every model is described by a `ModelConfig` entry in `backend/backends/__init__.py`. Models are downloaded from HuggingFace Hub on first use and cached in the platform-standard HF cache.
## Available Models
## Available TTS Models
### TTS Models
| Model | Engine | HuggingFace Repo | Size | VRAM | Languages |
|-------|--------|------------------|------|------|-----------|
| **Qwen TTS 1.7B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 3.5 GB | ~6 GB | 10 |
| **Qwen TTS 0.6B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | 1.2 GB | ~2 GB | 10 |
| **Qwen CustomVoice 1.7B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 3.5 GB | ~6 GB | 10 |
| **Qwen CustomVoice 0.6B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | 1.2 GB | ~2 GB | 10 |
| **LuxTTS** | `luxtts` | `YatharthS/LuxTTS` | 300 MB | ~1 GB | English |
| **Chatterbox Multilingual** | `chatterbox` | `ResembleAI/chatterbox` | 3.2 GB | ~3 GB | 23 |
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 8 |
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
### Whisper Models
## Available Whisper Models
| Model | HuggingFace ID | Size | VRAM |
|-------|----------------|------|------|
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
| base | `openai/whisper-base` | ~300MB | ~1GB |
| small | `openai/whisper-small` | ~500MB | ~2GB |
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
| large | `openai/whisper-large` | ~3GB | ~10GB |
| Model | HuggingFace Repo | Size |
|-------|------------------|------|
| **Whisper Base** | `openai/whisper-base` | ~300 MB |
| **Whisper Small** | `openai/whisper-small` | ~500 MB |
| **Whisper Medium** | `openai/whisper-medium` | ~1.5 GB |
| **Whisper Large** | `openai/whisper-large-v3` | ~3 GB |
| **Whisper Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB |
On Apple Silicon, MLX-Whisper is preferred automatically — see [Transcription](/developer/transcription).
## Model Storage
Models are cached in the HuggingFace cache directory:
Models live in the platform HuggingFace cache:
<Files>
<Folder name="~/.cache/huggingface/hub" defaultOpen>
<File name="models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/" />
<File name="models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/" />
<File name="models--openai--whisper-base/" />
</Folder>
</Files>
| Platform | Path |
|----------|------|
| macOS | `~/.cache/huggingface/hub/` |
| Linux | `~/.cache/huggingface/hub/` |
| Windows | `%USERPROFILE%\.cache\huggingface\hub\` |
| Docker | `/home/voicebox/.cache/huggingface/hub` (volume-mounted) |
Set `VOICEBOX_MODELS_DIR` to override.
## Progress Tracking
### Progress Manager
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
Tracks download progress across all models:
**`ProgressManager`** (`backend/utils/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
**SSE endpoint** — `GET /models/progress/{model_name}` streams updates until `status` is `complete` or `error`.
```python
class ProgressManager:
def __init__(self):
self._progress = {} # model_name -> progress_info
def update_progress(
self,
model_name: str,
current: int,
total: int,
filename: str,
status: str,
):
self._progress[model_name] = {
"current": current,
"total": total,
"filename": filename,
"status": status, # downloading, complete, error
"updated_at": datetime.utcnow(),
}
def get_progress(self, model_name: str) -> Optional[dict]:
return self._progress.get(model_name)
```
### HuggingFace Progress Callback
Hooks into HuggingFace's download system:
```python
class HFProgressTracker:
def __init__(self, callback):
self.callback = callback
@contextmanager
def patch_download(self):
"""Context manager to intercept HF downloads."""
original_download = hf_hub_download
def patched_download(*args, **kwargs):
# Intercept progress
result = original_download(*args, **kwargs)
self.callback(progress_info)
return result
# Apply patch
with patch('huggingface_hub.hf_hub_download', patched_download):
yield
```
### Server-Sent Events (SSE)
Progress is streamed to the frontend:
```python
@app.get("/models/progress/{model_name}")
async def get_model_progress(model_name: str):
async def event_generator():
while True:
progress = progress_manager.get_progress(model_name)
if progress:
yield f"data: {json.dumps(progress)}\n\n"
if progress and progress["status"] in ["complete", "error"]:
break
await asyncio.sleep(0.5)
return StreamingResponse(
event_generator(),
media_type="text/event-stream"
)
```
## Task Manager
Tracks active downloads and generations:
```python
class TaskManager:
def __init__(self):
self._active_downloads = {}
self._active_generations = {}
def start_download(self, model_name: str):
self._active_downloads[model_name] = {
"status": "downloading",
"started_at": datetime.utcnow(),
}
def complete_download(self, model_name: str):
if model_name in self._active_downloads:
del self._active_downloads[model_name]
def get_active_tasks(self) -> dict:
return {
"downloads": list(self._active_downloads.values()),
"generations": list(self._active_generations.values()),
}
# Frontend
const eventSource = new EventSource(`/models/progress/${modelName}`);
eventSource.onmessage = (event) => {
const { current, total, status } = JSON.parse(event.data);
updateProgressBar(current / total);
if (status === "complete") eventSource.close();
};
```
## Model Status
Check which models are downloaded and loaded:
```python
@app.get("/models/status")
async def get_model_status() -> ModelStatusListResponse:
models = []
# Check TTS models
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
downloaded = is_model_downloaded(hf_id)
loaded = tts_model._current_model_size == size
models.append(ModelStatus(
model_name=f"qwen-tts-{size}",
display_name=f"Qwen3-TTS {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=loaded,
))
# Check Whisper models
for size in ["tiny", "base", "small", "medium", "large"]:
hf_id = f"openai/whisper-{size}"
downloaded = is_model_downloaded(hf_id)
models.append(ModelStatus(
model_name=f"whisper-{size}",
display_name=f"Whisper {size}",
downloaded=downloaded,
size_mb=get_model_size_mb(hf_id),
loaded=False, # Whisper is loaded on-demand
))
return ModelStatusListResponse(models=models)
```
## Manual Model Operations
### Load Model
```python
@app.post("/models/load")
async def load_model(model_size: str = "1.7B"):
tts_model = get_tts_model()
await tts_model.load_model_async(model_size)
return {"status": "loaded", "model_size": model_size}
```
### Unload Model
```python
@app.post("/models/unload")
async def unload_model():
tts_model = get_tts_model()
tts_model.unload_model()
return {"status": "unloaded"}
```
### Trigger Download
```python
@app.post("/models/download")
async def trigger_model_download(request: ModelDownloadRequest):
# This triggers the download in background
# Progress is tracked via /models/progress/{model_name}
if request.model_name.startswith("qwen-tts"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_tts_model(size))
elif request.model_name.startswith("whisper"):
size = request.model_name.split("-")[-1]
asyncio.create_task(download_whisper_model(size))
return {"status": "downloading"}
```
### Delete Model
```python
@app.delete("/models/{model_name}")
async def delete_model(model_name: str):
# Find and delete from HuggingFace cache
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
for model_dir in model_dirs:
shutil.rmtree(model_dir)
return {"status": "deleted"}
```
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Get status of all models |
| POST | `/models/load` | Load TTS model |
| POST | `/models/unload` | Unload TTS model |
| POST | `/models/download` | Trigger model download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete downloaded model |
| GET | `/tasks/active` | Get active downloads/generations |
## Response Schemas
### ModelStatus
`GET /models/status` returns every registered model's current state:
```json
{
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen3-TTS 1.7B",
"downloaded": true,
"size_mb": 3400,
"loaded": true
}
```
### ActiveTasksResponse
```json
{
"downloads": [
"models": [
{
"model_name": "whisper-medium",
"status": "downloading",
"started_at": "2024-01-15T10:30:00Z"
}
],
"generations": [
{
"task_id": "uuid",
"profile_id": "uuid",
"text_preview": "Hello world...",
"started_at": "2024-01-15T10:30:00Z"
}
"model_name": "qwen-tts-1.7B",
"display_name": "Qwen TTS 1.7B",
"engine": "qwen",
"downloaded": true,
"size_mb": 3500,
"loaded": true
},
...
]
}
```
## Frontend Integration
The handler iterates `get_all_model_configs()` and calls `check_model_loaded(config)` for each entry, so new engines appear automatically once they're registered in `ModelConfig`.
### Progress Display
## Manual Model Operations
```typescript
// Subscribe to download progress via SSE
const eventSource = new EventSource(`/models/progress/${modelName}`);
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/models/status` | Status of every registered model |
| POST | `/models/load` | Load a TTS model into memory |
| POST | `/models/unload` | Unload a TTS model from memory |
| POST | `/models/download` | Trigger a background download |
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
| DELETE | `/models/{name}` | Delete a downloaded model from cache |
eventSource.onmessage = (event) => {
const progress = JSON.parse(event.data);
updateProgressBar(progress.current / progress.total);
if (progress.status === 'complete') {
eventSource.close();
}
};
### Load
```http
POST /models/load
{
"model_name": "qwen-tts-1.7B"
}
```
### Model Status UI
The route looks up the config, dispatches to `get_model_load_func(config)`, and returns once the model is ready.
```typescript
// Fetch model status
const { data: models } = useQuery({
queryKey: ['models', 'status'],
queryFn: () => api.getModelStatus(),
});
### Unload
// Display download/load buttons based on status
models.map(model => (
<ModelCard
name={model.display_name}
downloaded={model.downloaded}
loaded={model.loaded}
onDownload={() => triggerDownload(model.model_name)}
onLoad={() => loadModel(model.model_name)}
/>
));
```http
POST /models/unload
{
"model_name": "chatterbox-tts"
}
```
Calls `unload_model_by_config(config)`, which routes to the right backend's `unload_model()` and frees GPU memory.
### Download
```http
POST /models/download
{
"model_name": "kokoro"
}
```
Fires off an async download task. Progress is available via the SSE endpoint. Download is triggered automatically on first generation, so this is only needed for pre-warming.
## Preset Voice Seeding
For engines that use preset voices (Kokoro, Qwen CustomVoice), the backend auto-creates a voice profile per preset voice after the model is downloaded. This is driven by `seed_preset_profiles(engine)` in `backend/services/profiles.py`, called from the models route once download completes.
Preset profiles have:
- `voice_type = "preset"`
- `preset_engine` = engine name (`"kokoro"`, `"qwen_custom_voice"`)
- `preset_voice_id` = engine-specific voice ID (`"am_adam"`, `"f000001"`, etc.)
- No `profile_samples` rows — no audio to store
See [Voice Profiles](/developer/voice-profiles) for the schema.
## Adding a New Model
To add a new size variant of an existing engine, just add another `ModelConfig`:
```python
ModelConfig(
model_name="qwen-tts-3B",
display_name="Qwen TTS 3B",
engine="qwen",
hf_repo_id="Qwen/Qwen3-TTS-12Hz-3B-Base",
model_size="3B",
size_mb=7000,
languages=["zh", "en", ...],
),
```
The frontend picks it up via `/models/status`; download/load flow works without further changes.
Adding a whole new engine is a bigger lift — see [TTS Engines](/developer/tts-engines) for the full phased workflow.
## Error Handling
| Error | Cause | Solution |
|-------|-------|----------|
| Download failed | Network issue | Retry download |
| OOM on load | Model too large | Use smaller model |
| Model not found | Cache corrupted | Re-download |
| Slow download | HF rate limit | Wait and retry |
| Error | Cause | Fix |
|-------|-------|-----|
| Download failed | Network / HF rate limit | Retry |
| OOM on load | Not enough VRAM | Use a smaller variant, unload other engines |
| Model not found | Corrupt cache | Re-download via `/models/download` |
| Stuck progress bar in frozen build | `huggingface_hub` tqdm silenced | `HFProgressTracker` force-enables the internal counter |
| GPU architecture unsupported | PyTorch wheel doesn't target your GPU | See [GPU Acceleration](/overview/gpu-acceleration) |
## Next Steps
<Cards>
<Card title="TTS Generation" href="/developer/tts-generation">
How generation flows through the registry
</Card>
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new engine end-to-end
</Card>
<Card title="Transcription" href="/developer/transcription">
Whisper and MLX-Whisper integration
</Card>
</Cards>
+75 -24
View File
@@ -59,24 +59,64 @@ Ensure you have these installed:
## Just Commands
Run `just --list` to see all available commands:
Run `just --list` to see all available commands. Highlights:
### Setup
| Command | Description |
|---------|-------------|
| `just setup` | Full setup (Python venv + JS deps + dev sidecar). Detects Apple Silicon for MLX and NVIDIA/Intel Arc on Windows for accelerated PyTorch. |
| `just setup-python` | Python venv + dependencies only |
| `just setup-js` | `bun install` only |
### Development
| Command | Description |
|---------|-------------|
| `just dev` | Start backend + Tauri desktop app (reuses a running backend if one exists) |
| `just dev-web` | Start backend + web app (no Tauri/Rust build) |
| `just dev-backend` | Backend only |
| `just dev-frontend` | Tauri app only (backend must already be running) |
| `just kill` | Stop all dev processes |
### Build
| Command | Description |
|---------|-------------|
| `just build` | CPU server binary + Tauri installer |
| `just build-local` | **Windows:** CPU + CUDA server binaries + Tauri installer |
| `just build-server` | CPU server binary only |
| `just build-server-cuda` | **Windows:** CUDA server binary only, placed in `%APPDATA%/sh.voicebox.app/backends/cuda` for local testing |
| `just build-tauri` | Tauri app only |
| `just build-web` | Web app only |
### Quality
| Command | Description |
|---------|-------------|
| `just check` | Lint + format + typecheck (Biome + ruff) |
| `just fix` | Auto-fix lint + format issues |
| `just lint` / `just format` | Lint or format only |
| `just test` | Run Python tests (pytest) |
| `just test-models` | End-to-end generation against every TTS engine using the frozen binary |
### Database
| Command | Description |
|---------|-------------|
| `just setup` | Full setup (Python venv + JS deps) |
| `just dev` | Start backend + desktop app |
| `just dev-web` | Start backend + web app (no Tauri) |
| `just dev-backend` | Start backend only |
| `just dev-frontend` | Start desktop app only (backend must be running) |
| `just build` | Build desktop app for production |
| `just build-web` | Build web app for production |
| `just check` | Run all checks (JS + Python lint + format) |
| `just fix` | Fix lint + format issues |
| `just test` | Run Python tests |
| `just db-init` | Initialize SQLite database |
| `just db-reset` | Reset database (delete + reinit) |
| `just clean` | Clean build artifacts |
| `just clean-all` | Nuclear clean (includes node_modules) |
| `just db-reset` | Delete and reinitialize the database |
### Utilities
| Command | Description |
|---------|-------------|
| `just generate-api` | Generate TypeScript API client from the backend's OpenAPI schema |
| `just docs` | Open `http://localhost:17493/docs` in your browser |
| `just logs` | Tail backend logs |
| `just clean` | Remove build artifacts |
| `just clean-python` | Remove the Python venv + `__pycache__` |
| `just clean-all` | Nuclear clean (includes all `node_modules`) |
## Project Structure
@@ -133,10 +173,12 @@ HTTP request → **routes/** (validate input) → **services/** (business logic)
## Model Downloads
Models are automatically downloaded from HuggingFace Hub on first use:
Models are automatically downloaded from HuggingFace Hub on first use, with live progress streamed to the UI:
- **Whisper** (transcription): Auto-downloads on first transcription
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
- **Whisper** (transcription) — auto-downloads on first transcription
- **TTS engines** — auto-download on first generation. Sizes range from 82 M (Kokoro, ~350 MB) to 3 B (TADA, ~8 GB)
See [Model Management](/developer/model-management) for the full list.
<Callout type="warn">
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
@@ -150,7 +192,7 @@ After starting the backend server, generate the TypeScript API client:
just generate-api
```
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`.
## Manual Setup (Advanced)
@@ -186,8 +228,17 @@ pip install -r requirements.txt
# Apple Silicon: install MLX dependencies
pip install -r requirements-mlx.txt
# Install Qwen3-TTS
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
pip install --no-deps chatterbox-tts
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
pip install --no-deps hume-tada
# Install Qwen3-TTS from source
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
# PyInstaller and linting tools
pip install pyinstaller ruff pytest pytest-asyncio
```
### 3. Start Development
@@ -208,17 +259,17 @@ bun run tauri dev
## Next Steps
<Cards>
<Card title="Architecture" href="/development/architecture">
<Card title="Architecture" href="/developer/architecture">
Understand the system architecture
</Card>
<Card title="Contributing" href="/development/contributing">
<Card title="Contributing" href="/developer/contributing">
Read the contribution guidelines
</Card>
<Card title="Building" href="/development/building">
<Card title="Building" href="/developer/building">
Learn how to build production releases
</Card>
<Card title="API Reference" href="/api-reference">
Explore the REST API
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new TTS engine end-to-end
</Card>
</Cards>
+3 -22
View File
@@ -50,34 +50,15 @@ class StoryItem(Base):
### Start Time
`start_time_ms` defines when an item begins on the timeline:
```
Timeline (ms): 0----1000----2000----3000----4000
Item 1: [======]
Item 2: [==========]
Item 3: [====]
```
`start_time_ms` is the absolute position on the timeline where an item begins playing. Items on the same track cannot overlap; items on different tracks can.
### Tracks
Multiple tracks allow overlapping audio:
```
Track 0: [Item 1] [Item 3]
Track 1: [Item 2]
```
A `track` is an integer (0-indexed) that identifies the horizontal row an item sits on. Audio on separate tracks plays concurrently, so tracks are the primary way to layer multiple voices or sound effects.
### Trimming
Trim values cut audio from the start or end without destroying the original:
```
Original: [=========AUDIO=========]
trim_start: ^^
trim_end: ^^
Result: [=====AUDIO=====]
```
`trim_start_ms` and `trim_end_ms` hide the leading/trailing portions of the source generation without modifying the underlying audio file. The effective playback length is `generation.duration * 1000 - trim_start_ms - trim_end_ms`. Trimming is non-destructive — the same generation can be trimmed differently in different stories.
## Core Operations
+94 -233
View File
@@ -5,250 +5,91 @@ description: "How Whisper-based audio transcription works in Voicebox"
## Overview
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
Voicebox uses OpenAI's Whisper for automatic speech recognition (ASR). Transcription powers two flows:
1. **Reference-text auto-fill** — when a user records or uploads a voice sample, the backend transcribes it and populates the `reference_text` field so cloning can use it.
2. **On-demand transcription** — a user-facing `/transcribe` endpoint for arbitrary audio.
On Apple Silicon, the transcription path runs through **MLX-Whisper** (from `mlx-audio`) for ~8× faster inference than PyTorch. Everywhere else it runs through PyTorch's `transformers` Whisper.
## Architecture
The transcription system is built around the `WhisperModel` class:
**Model Loading:** Lazy loading with HuggingFace Hub download.
**Audio Processing:** Resampling and preprocessing for Whisper.
**Inference:** Running transcription with optional language hints.
## WhisperModel Class
Transcription is wired through the same backend abstraction as TTS. The `STTBackend` protocol lives in `backend/backends/__init__.py`:
```python
class WhisperModel:
def __init__(self, model_size: str = "base"):
self.model = None
self.processor = None
self.model_size = model_size
self.device = self._get_device()
@runtime_checkable
class STTBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
model_size: Optional[str] = None,
) -> str: ...
def unload_model(self) -> None: ...
def is_loaded(self) -> bool: ...
```
### Model Sizes
Two implementations ship today:
| Size | Parameters | VRAM | Speed | Quality |
|------|------------|------|-------|---------|
| tiny | 39M | ~1GB | Fastest | Basic |
| base | 74M | ~1GB | Fast | Good |
| small | 244M | ~2GB | Medium | Better |
| medium | 769M | ~5GB | Slow | High |
| large | 1550M | ~10GB | Slowest | Best |
- **`MLXSTTBackend`** (`backends/mlx_backend.py`) — uses `mlx_audio.stt.load()`. Default on Apple Silicon.
- **`PyTorchSTTBackend`** (`backends/pytorch_backend.py`) — uses `transformers.WhisperForConditionalGeneration`. Default everywhere else.
Default is `base` for balance of speed and quality.
`get_stt_backend()` picks the right one based on `get_backend_type()`. `backend/services/transcribe.py` is a thin wrapper that delegates to the backend.
## Model Sizes
Five Whisper variants are registered in `ModelConfig`:
| Model | HuggingFace Repo | Size | Notes |
|-------|------------------|------|-------|
| **Base** | `openai/whisper-base` | ~300 MB | Default; fast, decent quality |
| **Small** | `openai/whisper-small` | ~500 MB | Better quality, still fast |
| **Medium** | `openai/whisper-medium` | ~1.5 GB | High quality |
| **Large** | `openai/whisper-large-v3` | ~3 GB | Best quality, slow on CPU |
| **Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB | Large-tier quality, ~5× faster than Large |
The `tiny` model is **not** exposed — the quality gap to `base` wasn't worth the download.
`Turbo` + MLX-Whisper on Apple Silicon dropped user-facing transcription latency from ~20s to ~2-3s in v0.1.10.
## Language Hints
Whisper can auto-detect language, but providing a hint improves accuracy on short clips:
```python
text = await backend.transcribe(audio_path, language="en")
```
Accepted language codes are the standard Whisper set (99+ languages). The frontend typically passes the profile's language if available, or lets Whisper detect otherwise.
## Model Loading
Models are downloaded from HuggingFace Hub:
Both backends are lazy: the model is loaded on first use and cached in memory. Switching sizes unloads the previous model.
On MLX, the model is loaded via `mlx_audio.stt.load(hf_repo)`. On PyTorch, via:
```python
def load_model(self, model_size: Optional[str] = None):
from transformers import WhisperProcessor, WhisperForConditionalGeneration
model_name = f"openai/whisper-{model_size}"
# Track download progress
progress_manager = get_progress_manager()
task_manager = get_task_manager()
task_manager.start_download(f"whisper-{model_size}")
# Load processor and model
with tracker.patch_download():
self.processor = WhisperProcessor.from_pretrained(model_name)
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
self.model.to(self.device)
# Mark complete
progress_manager.mark_complete(f"whisper-{model_size}")
task_manager.complete_download(f"whisper-{model_size}")
WhisperProcessor.from_pretrained(hf_repo)
WhisperForConditionalGeneration.from_pretrained(hf_repo).to(device)
```
### Async Loading
Like TTS, loading runs in a thread pool:
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self.model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
```
## Transcription
### Basic Transcription
```python
async def transcribe(
self,
audio_path: str,
language: Optional[str] = None,
) -> str:
await self.load_model_async()
def _transcribe_sync():
# Load and resample to 16kHz (Whisper requirement)
audio, sr = load_audio(audio_path, sample_rate=16000)
# Process audio
inputs = self.processor(
audio,
sampling_rate=16000,
return_tensors="pt",
)
inputs = inputs.to(self.device)
# Set language hint if provided
forced_decoder_ids = None
if language:
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language,
task="transcribe",
)
# Generate
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
forced_decoder_ids=forced_decoder_ids,
)
# Decode
transcription = self.processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
return transcription.strip()
return await asyncio.to_thread(_transcribe_sync)
```
### Supported Languages
Whisper supports 99+ languages. Common ones in Voicebox:
| Code | Language |
|------|----------|
| en | English |
| zh | Chinese |
| ja | Japanese |
| ko | Korean |
| de | German |
| fr | French |
| ru | Russian |
| pt | Portuguese |
| es | Spanish |
| it | Italian |
### Language Detection
When no language is specified, Whisper auto-detects:
```python
# Without language hint - auto-detect
transcription = await whisper.transcribe(audio_path)
# With language hint - more accurate for short clips
transcription = await whisper.transcribe(audio_path, language="en")
```
## Transcription with Timestamps
For advanced use cases, word-level timestamps are available:
```python
async def transcribe_with_timestamps(
self,
audio_path: str,
language: Optional[str] = None,
) -> List[Dict[str, any]]:
await self.load_model_async()
def _transcribe_timestamps_sync():
audio, sr = load_audio(audio_path, sample_rate=16000)
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
predicted_ids = self.model.generate(
inputs["input_features"],
return_timestamps=True,
)
# Parse timestamps
return [
{
"text": transcription,
"start": 0.0,
"end": len(audio) / sr,
}
]
return await asyncio.to_thread(_transcribe_timestamps_sync)
```
## Memory Management
### Unloading
Free memory when not needed:
```python
def unload_model(self):
if self.model is not None:
del self.model
del self.processor
self.model = None
self.processor = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Global Instance
A singleton pattern manages the model:
```python
_whisper_model: Optional[WhisperModel] = None
def get_whisper_model() -> WhisperModel:
global _whisper_model
if _whisper_model is None:
_whisper_model = WhisperModel()
return _whisper_model
```
Both load paths use `model_load_progress()` from `backends/base.py` so the frontend sees live download progress on the first use.
## Audio Preprocessing
### Resampling
Whisper expects mono 16 kHz audio. The audio utility in `backend/utils/audio.py` handles resampling and format conversion transparently:
Whisper requires 16kHz audio:
- **Formats:** WAV, MP3, FLAC, OGG, M4A (via soundfile / librosa)
- **Target:** mono, 16 kHz, float32
```python
audio, sr = load_audio(audio_path, sample_rate=16000)
```
### Format Support
The `load_audio` utility handles:
- WAV
- MP3
- FLAC
- OGG
- M4A
All formats are converted to mono 16kHz.
Files longer than Whisper's 30-second window are handled by the underlying library's chunking logic — no explicit splitting in Voicebox code.
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/transcribe` | Transcribe audio file |
| POST | `/transcribe` | Transcribe an uploaded audio file |
### Request
@@ -259,7 +100,8 @@ POST /transcribe
Content-Type: multipart/form-data
file: <audio_file>
language: en (optional)
language: en # optional
model_size: base # optional (default: "base")
```
### Response
@@ -275,25 +117,44 @@ language: en (optional)
### Reference Text for Voice Cloning
1. User records audio sample
2. Audio is sent to `/transcribe`
3. Transcription becomes `reference_text`
4. Both are added to voice profile
Adding a voice sample triggers transcription automatically:
1. User uploads or records audio.
2. The backend writes the audio file and calls `/transcribe` internally (or the frontend calls it separately).
3. The returned text becomes `reference_text` on the new `profile_samples` row.
4. Cloning engines that need reference text (Chatterbox, TADA, etc.) read it from there.
### Quality Tips
- Provide language hint for short audio
- Use clean audio with minimal noise
- Longer audio (>5s) improves accuracy
- Consider `small` or `medium` model for better quality
- Provide a language hint for short clips (under 5 seconds) — auto-detection is unreliable on little audio.
- Use Turbo or Large for noisy audio — Base can hallucinate on hard inputs.
- Prefer clean audio; transcription errors become reference-text errors, which become cloning errors.
## Memory Management
`unload_model()` drops the model reference and clears the CUDA cache if applicable. `/models/unload` wires this up for manual control.
A singleton per backend is returned by `get_stt_backend()` — multiple callers share one Whisper instance.
## Error Handling
Common issues:
| Error | Cause | Solution |
|-------|-------|----------|
| Model not found | First run, download failed | Retry with network |
| OOM | Model too large | Use smaller model |
| Empty result | No speech detected | Check audio has speech |
| Wrong language | Auto-detect failed | Provide language hint |
| Model not found | First run + network failure | Retry; check connectivity |
| OOM on load | Large model on low-VRAM GPU | Switch to Small or Turbo |
| Empty result | No speech in audio | Confirm input has voice; check trim |
| Wrong language | Auto-detect misfired | Pass `language` hint |
## Next Steps
<Cards>
<Card title="Model Management" href="/developer/model-management">
Download / load / unload any model
</Card>
<Card title="Voice Profiles" href="/developer/voice-profiles">
How reference text is stored alongside samples
</Card>
<Card title="GPU Acceleration" href="/overview/gpu-acceleration">
Platform-specific acceleration including MLX-Whisper
</Card>
</Cards>
+17 -10
View File
@@ -604,18 +604,25 @@ for name, mod in [("dac", types.ModuleType("dac")),
- Do NOT use `@torch.jit.script` in the shim (see above)
- Only reimplement what the model actually uses — check the import chain carefully
## Upcoming Engines
## Candidate Engines
Based on the current model landscape, these are candidates for future integration:
The [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) file is the canonical, living list of candidates under evaluation — including why some have been backlogged (e.g. VoxCPM, which is effectively CUDA-only upstream).
| Model | Languages | Size | Key Features | Status |
|-------|-----------|------|--------------|--------|
| **CosyVoice2-0.5B** | Multilingual | ~500MB | Instruct support (`inference_instruct2()`) | Ready |
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
At a glance, current top candidates:
| Model | Tier | Size | Cross-platform? | Key Features |
|-------|------|------|-----------------|--------------|
| **MOSS-TTS-Nano** | 1 | 0.1 B | Yes (CPU realtime) | 48 kHz stereo, Apache 2.0, released 2026-04-13 |
| **Voxtral TTS** | 2 | 4 B | Likely | `mistralai/Voxtral-4B-TTS-2603` — presets + cloning |
| **VibeVoice** | 2 | ~500 M | Yes | Podcast-style multi-speaker dialogue |
| **Dia2** | 3 | TBD | TBD | Successor to the original Dia |
| **Fish Audio S2 Pro** | 3 | Medium | Yes | Word-level control via inline text |
**Backlogged:**
- **VoxCPM** (2B, Apache 2.0) — CUDA ≥12 required upstream; MPS broken in issues #232/#248; CPU path rejected by maintainers (#256). Keep watching for a PR that relaxes the device requirement.
Update `PROJECT_STATUS.md` when you pick one up or mark one as shipped/backlogged.
## Implementation Checklist
+167 -199
View File
@@ -1,283 +1,251 @@
---
title: "TTS Generation"
description: "How text-to-speech generation works in Voicebox"
description: "How text-to-speech generation works across Voicebox's multi-engine backend"
---
## Overview
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
Voicebox ships seven TTS engines — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, TADA, and Kokoro — behind a single `TTSBackend` Protocol. All of them expose the same async interface so the routes and services don't need per-engine branching.
## Architecture
This page covers how generation flows through that abstraction. For the step-by-step guide to adding a new engine, see [TTS Engines](/developer/tts-engines).
The TTS system is built around the `TTSModel` class which manages:
## The `TTSBackend` Protocol
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
**Voice Prompts:** Converting reference audio into embeddings.
**Generation:** Synthesizing speech from text using voice prompts.
## TTSModel Class
Every engine implements the same contract (defined in `backend/backends/__init__.py`):
```python
class TTSModel:
def __init__(self, model_size: str = "1.7B"):
self.model = None
self.model_size = model_size
self.device = self._get_device() # cuda, mps, or cpu
@runtime_checkable
class TTSBackend(Protocol):
async def load_model(self, model_size: str) -> None: ...
async def create_voice_prompt(
self, audio_path: str, reference_text: str, use_cache: bool = True
) -> Tuple[dict, bool]: ...
async def combine_voice_prompts(
self, audio_paths: List[str], reference_texts: List[str]
) -> Tuple[np.ndarray, str]: ...
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]: ...
def unload_model(self) -> None: ...
def is_loaded(self) -> bool: ...
```
### Device Selection
## The `ModelConfig` Registry
The model automatically selects the best available device:
Each downloadable model variant is described by a `ModelConfig` dataclass:
```python
def _get_device(self) -> str:
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return "cpu" # MPS can have issues, use CPU for stability
return "cpu"
@dataclass
class ModelConfig:
model_name: str # "luxtts", "qwen-tts-1.7B", "kokoro"
display_name: str # "LuxTTS (Fast, CPU-friendly)"
engine: str # "luxtts", "qwen", "kokoro"
hf_repo_id: str # "YatharthS/LuxTTS"
model_size: str = "default"
size_mb: int = 0
needs_trim: bool = False
supports_instruct: bool = False
languages: list[str] = field(default_factory=lambda: ["en"])
```
## Model Loading
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
Models are downloaded from HuggingFace Hub on first use:
- `get_all_model_configs()` — every TTS + STT variant
- `get_tts_model_configs()` — only TTS variants
- `get_model_config(model_name)` — lookup by name
- `engine_needs_trim(engine)` — whether output should run through `trim_tts_output()`
- `load_engine_model(engine, model_size)` — downloads + loads, handles engines with multiple sizes
- `get_tts_backend_for_engine(engine)` — thread-safe backend factory with double-checked locking
The `TTS_ENGINES` dict is the canonical list of shipped engine names:
```python
def load_model(self, model_size: Optional[str] = None):
# Model IDs on HuggingFace Hub
hf_model_map = {
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
}
# Load with progress tracking
with tracker.patch_download():
self.model = Qwen3TTSModel.from_pretrained(
model_path,
device_map=self.device,
torch_dtype=torch.bfloat16, # float32 on CPU
)
TTS_ENGINES = {
"qwen": "Qwen TTS",
"qwen_custom_voice": "Qwen CustomVoice",
"luxtts": "LuxTTS",
"chatterbox": "Chatterbox TTS",
"chatterbox_turbo": "Chatterbox Turbo",
"tada": "TADA",
"kokoro": "Kokoro",
}
```
### Async Loading
## Voice Prompt Patterns
Loading runs in a thread pool to avoid blocking the event loop:
Each engine chooses how to represent a voice in the prompt dict returned from `create_voice_prompt()`. Three patterns are in use today:
**Pattern A — Pre-computed tensors** (Qwen3-TTS, LuxTTS)
```python
async def load_model_async(self, model_size: Optional[str] = None):
if self.model is not None and self._current_model_size == model_size:
return
await asyncio.to_thread(self.load_model, model_size)
encoded = model.encode_prompt(audio_path)
return encoded, False # (prompt_dict, was_cached)
```
## Voice Prompt Creation
Voice prompts are created from reference audio and cached for reuse:
**Pattern B — Deferred file paths** (Chatterbox, Chatterbox Turbo, TADA)
```python
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
await self.load_model_async()
# Check cache
if use_cache:
cache_key = get_cache_key(audio_path, reference_text)
cached = get_cached_voice_prompt(cache_key)
if cached:
return cached, True
# Create prompt (blocking, run in thread pool)
voice_prompt = await asyncio.to_thread(
self.model.create_voice_clone_prompt,
ref_audio=audio_path,
ref_text=reference_text,
)
# Cache the result
cache_voice_prompt(cache_key, voice_prompt)
return voice_prompt, False
return {"ref_audio": audio_path, "ref_text": reference_text}, False
```
### Combining Multiple Samples
When a profile has multiple samples, they're combined:
**Pattern C — Preset voice pointer** (Kokoro, Qwen CustomVoice)
```python
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
combined_audio = []
for audio_path in audio_paths:
audio, sr = load_audio(audio_path)
audio = normalize_audio(audio)
combined_audio.append(audio)
# Concatenate and normalize
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
# Combine texts
combined_text = " ".join(reference_texts)
return mixed, combined_text
return {
"voice_type": "preset",
"preset_engine": "kokoro",
"preset_voice_id": "am_adam",
}, False
```
## Speech Generation
Pattern C is the shape used for profiles where `voice_type == "preset"` — there's no cloning step; the engine looks up a baked-in voice by ID.
The core generation function:
Engines that cache voice prompts prefix their cache keys to avoid collisions:
```python
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
await self.load_model_async()
def _generate_sync():
# Set seed for reproducibility
if seed is not None:
torch.manual_seed(seed)
# Generate audio
wavs, sample_rate = self.model.generate_voice_clone(
text=text,
voice_clone_prompt=voice_prompt,
instruct=instruct, # Natural language delivery control
)
return wavs[0], sample_rate
# Run in thread pool
return await asyncio.to_thread(_generate_sync)
cache_key = f"{engine}_{hash(audio_path, reference_text)}"
```
### Instruct Feature
## Device Selection
The `instruct` parameter allows natural language control over speech delivery:
Engines pick their device through `get_torch_device()` in `backends/base.py`, which layers:
1. `VOICEBOX_FORCE_CPU` environment override
2. CUDA (if compiled and available)
3. XPU (Intel Arc via IPEX)
4. MPS (Apple Silicon) — **only for engines that support it**; some (Chatterbox, older Qwen paths) skip MPS and fall back to CPU due to upstream operator gaps
5. CPU
Qwen TTS uses MLX directly on Apple Silicon instead of going through PyTorch — see `mlx_backend.py`.
## Generation Flow
The request path from frontend to audio file:
1. **Request** — `POST /generate` with `GenerationRequest`:
```json
{
"profile_id": "uuid",
"text": "...",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "warm, slightly amused",
"engine": "qwen",
"max_chunk_chars": 800
}
```
The `engine` field is validated against the regex `^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$`.
2. **Route** — `routes/generate.py` validates input and delegates.
3. **Service** — `services/generation.py` fetches the profile, resolves the engine backend via `get_tts_backend_for_engine(engine)`, and ensures the model is loaded (downloading it on first use with live progress).
4. **Voice prompt** — the service calls `create_voice_prompt()` (or the preset equivalent). For cloned profiles with multiple samples, it calls `combine_voice_prompts()` first to merge reference audio.
5. **Queue** — the request is serialized through `services/task_queue.py` to avoid multiple generations fighting for the GPU.
6. **Inference** — the engine's `generate()` returns `(audio_array, sample_rate)`.
7. **Post-process** — if `engine_needs_trim(engine)` is True, `trim_tts_output()` strips trailing silence. Effects chains (if any) are applied per generation version, not the clean version.
8. **Persist** — audio is written to the generations directory, a row is inserted into the `generations` table, and the response includes the generation metadata.
## Chunking for Long Text
Text longer than `max_chunk_chars` (default 800, range 100–5000) is split at sentence boundaries, generated in sequence, and crossfaded together. The chunking behavior is engine-agnostic — it lives in the service layer, not in individual backends.
## Instruct Mode
Two engines support natural-language delivery control via the `instruct` kwarg:
- **Qwen CustomVoice** — `supports_instruct=True`, fully wired to the model's instruct head.
- **Qwen Base** — silently drops the instruct text (`supports_instruct=False`). The frontend hides the instruct input for Base profiles.
```python
# Examples:
instruct = "Speak slowly and clearly"
instruct = "Sound excited and enthusiastic"
instruct = "Whisper softly"
# Good instruct prompts:
"warm and conversational, slight smile"
"whisper, intimate and close"
"authoritative, broadcast quality"
```
## Caching Strategy
Voice prompts are cached to avoid recomputation:
```python
def get_cache_key(audio_path: str, reference_text: str) -> str:
"""Generate cache key from audio hash and text."""
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
return f"{audio_hash}_{text_hash}"
```
Cache is stored in `data/cache/voice_prompts/`.
Other engines ignore `instruct` entirely.
## Memory Management
### Unloading Models
Free VRAM/RAM when not needed:
Models are loaded lazily on first use and kept in memory. Switching between model sizes (e.g. Qwen 1.7B ↔ 0.6B) unloads the previous model before loading the new one to avoid OOM:
```python
def unload_model(self):
if self.model is not None:
del self.model
self.model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
```
### Model Switching
When switching between model sizes (1.7B ↔ 0.6B):
```python
# Unload existing model first
if self.model is not None and self._current_model_size != model_size:
self.unload_model()
```
## Generation Flow
1. **Request** → Validate text and profile ID
2. **Profile** → Load profile samples from database
3. **Voice Prompt** → Create or retrieve cached prompt
4. **Generate** → Run TTS inference
5. **Save** → Write audio to generations directory
6. **Record** → Create history entry in database
7. **Response** → Return audio path and metadata
The model management API (`/models/load`, `/models/unload`) lets users free VRAM manually — see [Model Management](/developer/model-management).
## API Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/generate` | Generate speech from text |
| GET | `/audio/{id}` | Serve generated audio file |
| GET | `/audio/{generation_id}` | Serve generated audio file |
### Request Schema
```json
{
"profile_id": "uuid",
"text": "Text to synthesize",
"language": "en",
"seed": 42,
"model_size": "1.7B",
"instruct": "Speak clearly"
}
```
### Response Schema
### Response schema
```json
{
"id": "generation_uuid",
"profile_id": "profile_uuid",
"text": "Text to synthesize",
"text": "...",
"language": "en",
"audio_path": "/path/to/audio.wav",
"duration": 3.5,
"seed": 42,
"instruct": "Speak clearly",
"created_at": "2024-01-15T10:30:00Z"
"engine": "qwen",
"model_size": "1.7B",
"instruct": "...",
"created_at": "2026-04-18T10:30:00Z"
}
```
## Performance Considerations
### GPU Acceleration
- **CUDA** is the fastest backend for every PyTorch-based engine. Apple Silicon MLX is competitive with CUDA for Qwen TTS specifically.
- **Serial queue** — only one generation runs at a time per process; concurrent requests are queued.
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
- CUDA provides fastest inference
- MPS (Apple Silicon) has stability issues, uses CPU fallback
- CPU inference is slower but always works
### Per-engine VRAM (approximate, on CUDA)
### Batch Size
| Engine | VRAM |
|--------|------|
| Kokoro | ~150 MB |
| LuxTTS | ~1 GB |
| Chatterbox Turbo | ~1.5 GB |
| Qwen 0.6B / Qwen CustomVoice 0.6B | ~2 GB |
| Chatterbox Multilingual | ~3 GB |
| Qwen 1.7B / Qwen CustomVoice 1.7B | ~6 GB |
| TADA 1B | ~4 GB |
| TADA 3B | ~8 GB |
Currently generates one utterance at a time. For long texts, consider:
- Splitting into sentences
- Sequential generation
- Concatenating results
## Next Steps
### Memory Usage
| Model | VRAM/RAM Required |
|-------|-------------------|
| 0.6B | ~2GB |
| 1.7B | ~6GB |
<Cards>
<Card title="TTS Engines" href="/developer/tts-engines">
Add a new engine — full phased workflow
</Card>
<Card title="Model Management" href="/developer/model-management">
Downloading, loading, and unloading models
</Card>
<Card title="Voice Profiles" href="/developer/voice-profiles">
Cloned vs preset profile schema
</Card>
</Cards>
+39 -12
View File
@@ -5,17 +5,22 @@ description: "How voice profile management works in Voicebox"
## Overview
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
Voice profiles are the unit of "a saved voice" in Voicebox. As of 0.4 they support two flavors backed by the same `profiles` table:
- **Cloned profiles** — store one or more reference audio samples; the cloning engine generates a voice embedding at use time
- **Preset profiles** — store no audio; just a pointer to an engine-specific pre-built voice (e.g. Kokoro's `am_adam`, Qwen CustomVoice's `Ryan`)
The schema also reserves a third type, `designed`, for future text-described voices. Not currently used by any shipped engine.
## Architecture
The voice profile system consists of three main components:
**Database Layer:** SQLite tables store profile metadata and sample references.
**Database Layer:** SQLite tables store profile metadata, sample references (cloned), and engine + voice ID (preset).
**File Storage:** Audio samples are stored on disk in a structured directory format.
**File Storage:** Audio samples are stored on disk in a structured directory format. Preset profiles have no on-disk audio.
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
**Profile Module:** `backend/services/profiles.py` provides the business logic for CRUD operations and dispatches to the appropriate engine based on `voice_type`.
## Data Model
@@ -24,27 +29,49 @@ The voice profile system consists of three main components:
```python
class VoiceProfile(Base):
__tablename__ = "profiles"
id = Column(String, primary_key=True)
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
name = Column(String, unique=True, nullable=False)
description = Column(Text)
language = Column(String, default="en")
created_at = Column(DateTime)
updated_at = Column(DateTime)
avatar_path = Column(String, nullable=True)
effects_chain = Column(Text, nullable=True)
# Voice type system — added v0.3.x
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
design_prompt = Column(Text, nullable=True) # text description — only for designed (reserved)
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
```
The `voice_type` column discriminates the three flavors:
| `voice_type` | `preset_engine` | `preset_voice_id` | Samples in `profile_samples` |
| ------------ | --------------- | ----------------- | ---------------------------- |
| `cloned` | NULL | NULL | Required (≥1 row) |
| `preset` | engine name | voice ID string | None |
| `designed` | NULL | NULL | None (uses `design_prompt`) |
The `default_engine` column is set automatically when the profile is created. For preset profiles it's locked to the source engine — switching engines at generation time will skip the profile (and the UI auto-switches back when the user clicks a greyed-out card; see the floating generate box and profile grid).
### ProfileSample Table
```python
class ProfileSample(Base):
__tablename__ = "profile_samples"
id = Column(String, primary_key=True)
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
profile_id = Column(String, ForeignKey("profiles.id"))
audio_path = Column(String, nullable=False)
reference_text = Column(Text, nullable=False)
```
Only populated for cloned profiles. Preset and designed profiles have zero rows in this table.
## File Structure
Profiles are stored in the data directory:
@@ -152,8 +179,8 @@ async def create_voice_prompt_for_profile(
Reference audio is validated before being accepted:
- **Duration:** 3-30 seconds recommended
- **Format:** WAV, MP3, FLAC, OGG supported
- **Sample Rate:** Resampled to 24kHz
- **Format:** WAV, MP3, FLAC, OGG, M4A supported
- **Sample Rate:** Engine-specific — the audio utility resamples to whatever the active engine expects (Whisper uses 16 kHz, most TTS engines use 24 kHz, LuxTTS outputs 48 kHz). Resampling happens on the fly; the stored sample retains its original rate.
- **Channels:** Converted to mono if stereo
## Export/Import
+4 -3
View File
@@ -3,15 +3,16 @@ title: "Voicebox Documentation"
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
---
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
![Voicebox App Screenshot](/images/app-screenshot-1.webp)
- **Complete privacy** -- models and voice data stay on your machine
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
@@ -1,32 +1,43 @@
---
title: "Creating Voice Profiles"
description: "Advanced guide to creating high-quality voice profiles"
description: "How to create voice profiles, both cloning-based and preset-based"
---
## Overview
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
A **voice profile** is a saved voice you can reuse across generations, stories, and the API. As of 0.4, Voicebox profiles come in two flavors that map to two different ways of getting a voice:
## Quick Start
| Profile type | What it stores | Use when… |
| -------------- | ---------------------------------------------------- | -------------------------------------------------------- |
| **Cloned** | One or more reference audio samples + a voice embedding | You want to replicate a specific person's voice |
| **Preset** | A reference to a pre-built voice in a specific engine | You want a curated, production-ready voice with no audio prep |
Both types live in the same Profiles tab and behave the same way at generation time — pick the type that matches your goal and follow the workflow below.
<Callout type="info">
Not sure which to use? Cloning gives you a *specific* voice but needs clean audio. Preset gives you *good* voices instantly but you don't get to choose who they sound like.
</Callout>
## Workflow A — Cloned Profiles
Use this when you want to replicate a specific person's voice from a recording.
<Steps>
<Step title="Prepare Audio">
10-30 seconds of clear speech
10-30 seconds of clear speech, minimal background noise. See [Voice Cloning](/overview/voice-cloning) for the engine catalog.
</Step>
<Step title="Create Profile">
**Profiles** → **+ New Profile**
**Profiles** → **+ New Profile** → choose a cloning engine (Qwen3-TTS, Chatterbox Multilingual, Chatterbox Turbo, LuxTTS, or TADA)
</Step>
<Step title="Upload Sample">
Add your audio file
<Step title="Upload or Record Sample">
Drag in an audio file, or record directly with the in-app recorder
</Step>
<Step title="Generate">
Use the profile to generate speech
<Step title="Generate to Test">
Use the profile to generate a test phrase. If quality is poor, add more samples
</Step>
</Steps>
## Audio Requirements
### Ideal Sample Characteristics
### Audio Requirements (Cloning Only)
<Cards>
<Card title="Duration">
@@ -44,7 +55,7 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
<Card title="Quality">
**High fidelity**
44.1kHz or 48kHz sample rate
44.1 kHz or 48 kHz sample rate
Minimal compression
</Card>
<Card title="Content">
@@ -58,18 +69,16 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
### File Formats
Supported formats:
- **WAV** (recommended) - Lossless quality
- **MP3** - Acceptable, minimal compression
- **M4A** - Acceptable
- **FLAC** - Lossless alternative
- **WAV** (recommended) — Lossless quality
- **MP3** — Acceptable, minimal compression
- **M4A** — Acceptable
- **FLAC** — Lossless alternative
<Callout type="info">
Use WAV for best results. Avoid heavily compressed formats.
</Callout>
## Recording Tips
### Environment
### Recording Tips
<AccordionGroup>
<Accordion title="Quiet Space">
@@ -87,27 +96,25 @@ Supported formats:
</Accordion>
<Accordion title="Recording Settings">
- 44.1kHz or 48kHz sample rate
- 44.1 kHz or 48 kHz sample rate
- 16-bit or 24-bit depth
- Mono is fine (stereo will be converted)
- Avoid automatic gain control
</Accordion>
</AccordionGroup>
### Speaking
### Speaking Style
- **Natural pace** - Don't rush or speak too slowly
- **Clear articulation** - Pronounce words clearly
- **Consistent volume** - Maintain steady loudness
- **Normal tone** - Speak as you normally would
- **Complete sentences** - Avoid fragments or "ums"
- **Natural pace** — Don't rush or speak too slowly
- **Clear articulation** — Pronounce words clearly
- **Consistent volume** — Maintain steady loudness
- **Normal tone** — Speak as you normally would
- **Complete sentences** — Avoid fragments or "ums"
## Multiple Samples
### Multiple Samples
Adding multiple samples can significantly improve quality:
### Why Multiple Samples?
<Cards>
<Card title="Robustness">
Model learns a more complete representation
@@ -123,110 +130,57 @@ Adding multiple samples can significantly improve quality:
</Card>
</Cards>
### Sample Variety
Consider adding samples with:
1. **Different tones**
- Casual conversation
- Professional/formal
- Excited/enthusiastic
- Calm/serious
2. **Different content**
- Narratives
- Questions
- Statements
- Emotions (happy, sad, neutral)
3. **Different recording conditions**
- Studio quality
- Phone call quality (if needed)
- Room acoustics
1. **Different tones** — casual, formal, excited, calm
2. **Different content** — narratives, questions, statements
3. **Different recording conditions** — studio quality, room acoustics
<Callout type="warn">
All samples should be from the **same speaker**. Mixing voices will produce poor results.
</Callout>
## Processing Existing Audio
### Processing Existing Audio
If you have existing audio (podcasts, videos, etc.):
### Extracting Clean Segments
<Steps>
<Step title="Find Clean Speech">
Look for segments with:
- Just the target speaker
- No background music
- Minimal noise
Look for segments with just the target speaker, no background music, minimal noise
</Step>
<Step title="Use Audio Editor">
Tools like Audacity or Adobe Audition:
- Cut out clean 10-30s segments
- Remove silence at start/end
- Normalize volume if needed
Tools like Audacity or Adobe Audition: cut clean 10-30s segments, remove silence at start/end, normalize volume
</Step>
<Step title="Export as WAV">
Save as high-quality WAV file
</Step>
</Steps>
### Noise Reduction
For light background noise, use Audacity's noise reduction (gentle settings — over-processing introduces artifacts).
If you have light background noise:
### Testing & Iteration
```
1. Use noise reduction in Audacity:
- Select noise-only section
- Get Noise Profile
- Select full audio
- Apply noise reduction (gentle settings)
2. Avoid over-processing:
- Can introduce artifacts
- May reduce voice quality
```
## Testing & Iteration
### Test Your Profile
After creating a profile:
After creating a cloned profile:
<Steps>
<Step title="Generate Test">
Generate a simple phrase:
```
"Hello, this is a test of my voice profile."
```
Try a simple phrase: `"Hello, this is a test of my voice profile."`
</Step>
<Step title="Evaluate Quality">
Listen for:
- Natural tone
- Clear pronunciation
- Proper prosody
- Lack of artifacts
Listen for natural tone, clear pronunciation, proper prosody, lack of artifacts
</Step>
<Step title="Iterate">
If quality is poor:
- Add more samples
- Try different source audio
- Check sample quality
If quality is poor: add more samples, try different source audio, check sample quality
</Step>
</Steps>
### Common Issues
#### Common Issues
<AccordionGroup>
<Accordion title="Robotic Voice">
**Cause**: Poor quality samples or too short
**Fix**: Use longer, higher quality samples
**Fix**: Use longer, higher-quality samples
</Accordion>
<Accordion title="Wrong Tone">
@@ -242,51 +196,89 @@ After creating a profile:
</Accordion>
</AccordionGroup>
## Workflow B — Preset Profiles
Use this when you want a ready-made voice without recording anything. Available engines: **Kokoro 82M** (50 voices) and **Qwen CustomVoice** (9 voices). See [Preset Voices](/overview/preset-voices) for the full catalog.
<Steps>
<Step title="Create Profile">
**Profiles** → **+ New Profile** → choose **Kokoro** or **Qwen CustomVoice** as the engine
</Step>
<Step title="Pick a Voice">
The engine's voice catalog appears. Click any voice to preview it
</Step>
<Step title="Name and Save">
Give the profile a name. No audio sample required
</Step>
<Step title="Generate">
The profile is ready immediately — use it in the floating generate box or Generate page
</Step>
</Steps>
<Callout type="info">
Preset profiles are **locked to their source engine**. Switching to a different engine in the floating generate box greys out the profile, since the voice only exists in that engine. Clicking a greyed profile auto-switches the engine back.
</Callout>
### Qwen CustomVoice + Instruct
Preset voices in Qwen CustomVoice support **delivery instructions** — natural-language style control over tone, pace, and emotion. The floating generate box shows a slider icon next to the generate button when a Qwen CustomVoice profile is selected; click it to reveal the instruct textarea.
See [Preset Voices → Using Instruct Mode](/overview/preset-voices#using-instruct-mode) for examples.
## Advanced Tips
### Celebrity/Character Voices
### Celebrity / Character Voices (Cloning)
For cloning public figures or characters:
1. **Legal considerations** - Ensure you have rights or it's fair use
2. **Source quality** - Find high-quality interview audio or clean clips
3. **Consistency** - Use clips where they speak similarly
4. **Multiple samples** - Very important for recognizable voices
1. **Legal considerations** — Ensure you have rights or it's clearly fair use
2. **Source quality** — Find high-quality interview audio or clean clips
3. **Consistency** — Use clips where they speak similarly
4. **Multiple samples** — Very important for recognizable voices
### Accent & Dialect
### Accent & Dialect (Cloning)
The model will preserve accent and dialect:
Cloning models preserve accent and dialect:
- British English will generate British English
- Southern accent will produce Southern accent
- Regional pronunciations will be maintained
- British English samples generate British English output
- Southern accent samples produce Southern accent output
- Regional pronunciations are maintained
### Emotion Transfer
### Emotion Transfer (Cloning)
The emotional tone of samples affects generation:
- Energetic samples → Energetic output
- Calm samples → Calm output
- Mix samples for versatile profile
- Energetic samples → energetic output
- Calm samples → calm output
- Mix samples for a more versatile profile
For Qwen CustomVoice presets, use the **instruct** field instead of relying on sample emotion — that's exactly what it controls.
## Managing Profiles
### Organization
- **Descriptive names** - "John Smith - Professional Narrator"
- **Add descriptions** - Note recording conditions, use cases
- **Language tags** - Mark the primary language
- **Archive unused** - Keep profile list manageable
- **Descriptive names** — "John Smith - Professional Narrator"
- **Add descriptions** — Note recording conditions, use cases, or which preset voice
- **Language tags** — Mark the primary language
- **Archive unused** — Keep profile list manageable
### Export/Import
### Export / Import
- **Export** profiles to share or backup
- **Import** from colleagues or teammates
- Profiles include voice embeddings, not original audio
- **Cloned profiles** export with their voice embeddings (not the original audio)
- **Preset profiles** export as engine + voice ID metadata only — the importer must have that engine's model installed
## Next Steps
<Cards>
<Card title="Voice Cloning" href="/overview/voice-cloning">
Engine catalog and best practices for cloning
</Card>
<Card title="Preset Voices" href="/overview/preset-voices">
Full catalog of Kokoro and Qwen CustomVoice voices
</Card>
<Card title="Generate Speech" href="/overview/generating-speech">
Use your profile to generate speech
</Card>
@@ -79,9 +79,9 @@ Drag generations to the Stories Editor timeline.
History is stored locally:
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
- **Linux**: `~/.config/com.voicebox.app/data/`
- **macOS**: `~/Library/Application Support/sh.voicebox.app/data/`
- **Windows**: `%APPDATA%/sh.voicebox.app/data/`
- **Linux**: `~/.config/sh.voicebox.app/data/`
<Callout type="warn">
Deleting the data directory will remove all history. Export important files first.
@@ -0,0 +1,236 @@
---
title: "GPU Acceleration"
description: "How Voicebox uses your GPU — auto-detection, manual setup, troubleshooting"
---
## Overview
Voicebox auto-detects available accelerators on first launch and picks the fastest backend it can use. For most people this just works — open the app and you're already on the right backend.
This page is for the cases where it doesn't:
- You have a GPU but Voicebox is running on CPU
- You upgraded GPUs (especially to RTX 50-series / Blackwell) and generation broke
- You want to switch backends manually (e.g. force MLX over PyTorch on Apple Silicon)
- You see `[UNSUPPORTED - see logs]` next to your GPU in Settings
## Backend Matrix
| Platform | Auto-selected backend | Notes |
| --------------------------- | ------------------------- | ---------------------------------------------------- |
| **macOS Apple Silicon** | MLX (Metal) | 4-5x faster than PyTorch via Apple Neural Engine |
| **macOS Intel** | PyTorch CPU | No GPU acceleration available; PyTorch ≥ 2.2 only |
| **Windows + NVIDIA** | PyTorch CUDA (cu128) | Auto-downloads the CUDA backend binary on first use |
| **Windows + Intel Arc** | PyTorch XPU (IPEX) | New in 0.4 — works with Arc A-series and B-series |
| **Windows generic GPU** | DirectML | Universal Windows GPU support; slower than CUDA |
| **Linux + NVIDIA** | PyTorch CUDA (cu128) | Same auto-download flow as Windows |
| **Linux + AMD** | PyTorch ROCm | Auto-configures `HSA_OVERRIDE_GFX_VERSION` |
| **Linux + Intel Arc** | PyTorch XPU (IPEX) | |
| **Any (no GPU)** | PyTorch CPU | Works everywhere; expect 5-50x slower than GPU |
The detected backend is shown in Settings → GPU. Logs at startup also print the chosen backend and the device name.
## Apple Silicon — MLX vs PyTorch
On M-series Macs, Voicebox ships an MLX-optimized backend that uses the Apple Neural Engine. It's **4-5x faster** than the PyTorch (CPU/Metal) path for supported engines.
| Engine | MLX support | Notes |
| -------------------- | ----------- | ------------------------------------------- |
| Qwen3-TTS | ✅ Native | Uses MLX exclusively when available |
| Chatterbox / Turbo | PyTorch MPS | Falls back to Metal via PyTorch |
| LuxTTS | PyTorch MPS | |
| TADA | PyTorch MPS | |
| Kokoro | PyTorch MPS | Requires `PYTORCH_ENABLE_MPS_FALLBACK=1` |
| Qwen CustomVoice | PyTorch MPS | |
| Whisper (transcribe) | ✅ Native | MLX-Whisper is the default on Apple Silicon |
The Whisper Turbo + MLX combo dropped transcription latency from ~20s to ~2-3s on M-series chips (see CHANGELOG entry for v0.1.10).
## Windows / Linux + NVIDIA — The CUDA Backend Swap
Voicebox doesn't bundle CUDA into the main installer (it would balloon downloads to multi-gigabyte territory for users who don't have an NVIDIA GPU). Instead, when you first need it, the app downloads a separate **CUDA backend binary** that contains the PyTorch + CUDA runtime.
<Steps>
<Step title="Open Settings → GPU">
If an NVIDIA GPU is detected, you'll see "Install CUDA backend" in the GPU panel
</Step>
<Step title="Click Install">
The app downloads two archives separately:
- **Server core** (~200-400 MB) — versioned with each Voicebox release
- **CUDA libs** (~4 GB) — the heavy PyTorch + CUDA DLLs, versioned independently
</Step>
<Step title="Restart">
Voicebox restarts to swap in the CUDA backend
</Step>
</Steps>
<Callout type="info">
The split-archive design (added in v0.4) means most Voicebox upgrades only redownload the small server-core archive. The 4 GB libs archive is only refreshed when the underlying CUDA toolkit or torch major version changes.
</Callout>
### Auto-update
When a new Voicebox release ships, the GPU panel checks if the bundled server-core matches the installed CUDA version. If only the core changed (typical), it pulls the new core in the background. If the libs version changed (rare — only happens on cu126 → cu128 type bumps), you'll be prompted to confirm the larger download.
## RTX 50-series / Blackwell
Voicebox 0.4 added explicit RTX 50-series support:
- CUDA toolkit upgraded to **cu128** (previous releases used cu126 which lacks Blackwell kernels)
- Build pinned with `TORCH_CUDA_ARCH_LIST=...12.0+PTX` for forward-compatibility
If you're on an RTX 5070 / 5080 / 5090 and you see "no kernel image is available" errors:
1. Make sure you're on Voicebox **≥ 0.4.0** (Settings → About)
2. Reinstall the CUDA backend (Settings → GPU → Reinstall CUDA backend) — older installs may have stale cu126 libs
3. If errors persist, see the GPU compatibility warnings section below
## Intel Arc (XPU)
New in 0.4. Works with both Arc A-series (Alchemist: A380, A580, A750, A770) and B-series (Battlemage).
### Setup
Voicebox auto-detects Arc GPUs and routes through Intel's PyTorch XPU backend (powered by IPEX — Intel Extension for PyTorch). No extra installation step beyond the standard Voicebox install.
Verify it's working:
- Settings → GPU should show **XPU** followed by your Arc model name (e.g. `XPU (Intel Arc A770)`)
- Startup logs print `Backend: PYTORCH` and `GPU: XPU (Intel Arc ...)`
### Engines on XPU
All PyTorch-based engines work on XPU. Performance is generally between CPU and CUDA — expect ~2-3x speedup over CPU for the larger models.
## DirectML
The fallback for Windows users with non-NVIDIA, non-Intel-Arc GPUs (older AMD discrete, integrated GPUs, etc.). Slower than CUDA and XPU but provides some acceleration over CPU.
Auto-selected when no other GPU backend is available.
## AMD ROCm (Linux)
ROCm provides PyTorch GPU acceleration on AMD discrete GPUs. Voicebox auto-configures `HSA_OVERRIDE_GFX_VERSION` for common cards that need the override.
### Verifying
```bash
# In a terminal
echo $HSA_OVERRIDE_GFX_VERSION
# Should show e.g. 10.3.0 for RX 6000 series
```
If detection fails, set the variable manually before launching Voicebox:
```bash
export HSA_OVERRIDE_GFX_VERSION=10.3.0
voicebox
```
Common values:
- `10.3.0` — RX 6000 series (RDNA 2)
- `11.0.0` — RX 7000 series (RDNA 3)
- `9.0.0` — Older Vega cards
## GPU Compatibility Warnings
Voicebox 0.4 added a runtime check that compares your GPU's compute capability against the architectures the bundled PyTorch was compiled for. If they don't match, you'll see:
- A startup log line: `WARNING: GPU COMPATIBILITY: <your GPU> is not supported by this PyTorch build...`
- The GPU label in Settings shows `[UNSUPPORTED - see logs]`
- The `/health` API returns a populated `gpu_compatibility_warning` field
### What to do
The most common trigger is a brand-new GPU architecture that pre-built PyTorch wheels don't yet cover natively. In order of preference:
1. **Update Voicebox** — newer releases ship newer PyTorch with broader arch support
2. **Reinstall the CUDA backend** — Settings → GPU → Reinstall CUDA backend
3. **For bleeding-edge GPUs (newer than current Blackwell):** install PyTorch nightly manually:
```bash
pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128 --force-reinstall
```
Then point Voicebox at that environment via [Remote Mode](/overview/remote-mode) until stable PyTorch catches up.
4. **Fall back to CPU** temporarily — set `VOICEBOX_FORCE_CPU=1` before launching
## CPU-Only Fallback
When no GPU is available (or you've forced it off), Voicebox runs the PyTorch CPU backend. Expect:
- 5-50x slower generation depending on engine and text length
- Heavy CPU usage during generation
- Some engines work better than others on CPU:
- **Kokoro 82M** — runs at realtime on modern CPUs
- **LuxTTS** — exceeds 150x realtime on CPU
- **Chatterbox Turbo (350M)** — usable but slow
- Larger models (Qwen 1.7B, Chatterbox Multilingual, TADA 3B) — painful
For CPU-bound use cases, prefer the smaller, lighter engines.
## Verifying Your Setup
Three places to check that the right backend is being used:
<Steps>
<Step title="Settings → GPU">
Shows the detected backend, GPU model, and VRAM (when applicable). Look for the `[UNSUPPORTED - see logs]` suffix
</Step>
<Step title="Settings → Logs">
The "Server logs" tab shows the startup banner with `Backend: <type>` and `GPU: <name>`
</Step>
<Step title="Health endpoint">
`curl http://localhost:17493/health` returns a JSON payload with `backend_type`, `backend_variant`, and `gpu_compatibility_warning` (when applicable)
</Step>
</Steps>
## Troubleshooting
<AccordionGroup>
<Accordion title="Settings shows CPU instead of my GPU">
- On NVIDIA: install the CUDA backend (Settings → GPU)
- On Intel Arc: confirm IPEX detection in startup logs; restart the app after a driver update
- On AMD Linux: check `HSA_OVERRIDE_GFX_VERSION` is set
</Accordion>
<Accordion title="'no kernel image is available' / 'CUDA error'">
Almost always means the bundled PyTorch doesn't have kernels for your GPU's compute capability.
1. Update to Voicebox ≥ 0.4.0 (Blackwell support added there)
2. Reinstall the CUDA backend
3. If still broken, install PyTorch nightly via Remote Mode
</Accordion>
<Accordion title="Out of memory (CUDA)">
- Switch to a smaller model size (e.g. Qwen3 0.6B instead of 1.7B)
- Use Settings → Models to unload other engines you're not using
- Enable `low_cpu_mem_usage` is already on for CPU; for CUDA, the engine's `device_map` handles offload automatically
- Close other GPU applications
</Accordion>
<Accordion title="MPS fallback errors on macOS">
Some operations don't have a Metal implementation. Voicebox sets `PYTORCH_ENABLE_MPS_FALLBACK=1` for engines that need it (notably Kokoro), but if you launch from a custom env, set it manually:
```bash
export PYTORCH_ENABLE_MPS_FALLBACK=1
```
</Accordion>
<Accordion title="Generation works but is slow on my GPU">
- Check Settings → GPU shows your GPU (not CPU)
- Check VRAM usage — you may be paging to system memory
- Try a smaller model
- For NVIDIA: confirm cu128 is installed (Settings → GPU → version)
</Accordion>
</AccordionGroup>
## Next Steps
<Cards>
<Card title="Remote Mode" href="/overview/remote-mode">
Run the backend on a different machine with a stronger GPU
</Card>
<Card title="Model Management" href="/developer/model-management">
Unload models to free GPU memory
</Card>
<Card title="Troubleshooting" href="/overview/troubleshooting">
General troubleshooting beyond GPU
</Card>
</Cards>
+4 -4
View File
@@ -68,11 +68,11 @@ Voicebox is available for macOS and Windows, with Linux builds coming soon.
When you launch Voicebox for the first time:
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
1. **Model Download** — The TTS engine you generate with first will download its model automatically. Sizes range from ~350 MB (Kokoro) to ~8 GB (TADA 3B). Most users start with Qwen 1.7B (~3.5 GB).
2. **Data Directory** — Voice profiles and generated audio are stored in:
- macOS: `~/Library/Application Support/com.voicebox.app/`
- Windows: `%APPDATA%/com.voicebox.app/`
- Linux: `~/.config/com.voicebox.app/`
- macOS: `~/Library/Application Support/sh.voicebox.app/`
- Windows: `%APPDATA%/sh.voicebox.app/`
- Linux: `~/.config/sh.voicebox.app/`
3. **Backend Server** — The bundled Python server starts automatically
+15 -12
View File
@@ -5,13 +5,14 @@ description: "Voicebox is a local-first voice cloning studio -- a free and open-
## What is Voicebox?
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
- **Complete privacy** -- models and voice data stay on your machine
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
- **API-first** -- REST API for integrating voice synthesis into your own projects
@@ -20,15 +21,17 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
## TTS Engines
Five engines with different strengths, switchable per-generation:
Seven engines with different strengths, switchable per-generation:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions |
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model -- 700s+ coherent audio |
| Engine | Profile Type | Languages | Strengths |
|--------|--------------|-----------|-----------|
| **Qwen3-TTS** (0.6B / 1.7B) | Cloned | 10 | High-quality multilingual cloning |
| **Qwen CustomVoice** (0.6B / 1.7B) | Preset (9 voices) | 10 | Natural-language delivery control (tone, emotion, pace) |
| **LuxTTS** | Cloned | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | Cloned | 23 | Broadest language coverage |
| **Chatterbox Turbo** | Cloned | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | Cloned | 10 | HumeAI speech-language model -- 700s+ coherent audio |
| **Kokoro** | Preset (50 voices) | 9 | 82M parameters, CPU realtime, lowest VRAM of any engine |
## GPU Support
@@ -57,7 +60,7 @@ Five engines with different strengths, switchable per-generation:
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
| Effects | Pedalboard (Spotify) |
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
+2
View File
@@ -6,7 +6,9 @@
"installation",
"docker",
"quick-start",
"gpu-acceleration",
"voice-cloning",
"preset-voices",
"stories-editor",
"recording-transcription",
"generation-history",
@@ -0,0 +1,202 @@
---
title: "Preset Voices"
description: "Use built-in, ready-made voices without recording audio samples"
---
## Overview
Some Voicebox engines ship with a curated set of pre-built voices. Instead of cloning from your own audio sample, you pick a voice from a fixed catalog and the model speaks in that voice. No recording, no upload, no per-voice training required.
Two engines in 0.4 ship preset voices:
| Engine | Voices | Languages | Strengths |
| --------------------- | ----------------------- | --------- | ------------------------------------------------------- |
| **Kokoro 82M** | 50 | 9 | Tiny model, CPU-friendly, lowest VRAM of any engine |
| **Qwen CustomVoice** | 9 (premium curated) | 4 | Natural-language style control over tone, emotion, pace |
<Callout type="info">
Looking for cloning a specific person's voice instead? See [Voice Cloning](/overview/voice-cloning).
</Callout>
## When to Use Preset Voices
<Cards>
<Card title="No reference audio">
You don't have (or don't want to provide) a recording of the target voice
</Card>
<Card title="Production reliability">
Curated voices have predictable quality across any text input
</Card>
<Card title="Speed">
Skip the audio cleanup, sample preparation, and quality iteration loop
</Card>
<Card title="Lightweight setup">
Kokoro runs at CPU realtime with ~150 MB on disk — no GPU needed
</Card>
</Cards>
## Creating a Preset-Voice Profile
<Steps>
<Step title="Open Profiles → New Profile">
Same entry point as cloning profiles
</Step>
<Step title="Choose the engine">
Select **Kokoro** or **Qwen CustomVoice** from the engine dropdown
</Step>
<Step title="Pick a preset voice">
The voice catalog for the chosen engine appears — preview each by clicking it
</Step>
<Step title="Name and save">
Give the profile a name. No audio sample needed — just save
</Step>
<Step title="Generate">
Use the profile like any other in the floating generate box or the Generate page
</Step>
</Steps>
<Callout type="info">
Preset profiles are locked to their source engine — switching engines won't work since the voice exists only for that model. The profile grid greys out preset profiles when you switch to a different engine, and clicking one auto-switches the engine back to the right one.
</Callout>
## Kokoro 82M — 50 Voices Across 9 Languages
Kokoro is the smallest engine in Voicebox at 82M parameters. It runs at CPU realtime with negligible VRAM, making it the best option for lightweight local inference. Voices are pre-built style vectors trained into the model — there's no concept of cloning here.
**Repository:** [`hexgrad/Kokoro-82M`](https://huggingface.co/hexgrad/Kokoro-82M) · Apache 2.0 licensed
### American English
| Female | Male |
| ------- | ------- |
| Alloy | Adam |
| Aoede | Echo |
| Bella | Eric |
| Heart | Fenrir |
| Jessica | Liam |
| Kore | Michael |
| Nicole | Onyx |
| Nova | Puck |
| River | Santa |
| Sarah | |
| Sky | |
### British English
| Female | Male |
| -------- | ------ |
| Alice | Daniel |
| Emma | Fable |
| Isabella | George |
| Lily | Lewis |
### Other Languages
| Language | Voices |
| ----------------- | ------------------------------------------- |
| Spanish (`es`) | Dora (f), Alex (m), Santa (m) |
| French (`fr`) | Siwis (f) |
| Hindi (`hi`) | Alpha (f), Beta (f), Omega (m), Psi (m) |
| Italian (`it`) | Sara (f), Nicola (m) |
| Japanese (`ja`) | Alpha (f), Gongitsune (f), Nezumi (f), Tebukuro (f), Kumo (m) |
| Portuguese (`pt`) | Dora (f), Alex (m), Santa (m) |
| Chinese (`zh`) | Xiaobei (f), Xiaoni (f), Xiaoxiao (f), Xiaoyi (f) |
### Kokoro at a Glance
| Property | Value |
| --------------- | -------------------------------------------- |
| Parameters | 82M |
| Sample rate | 24 kHz |
| VRAM | ~150 MB (negligible on CPU) |
| Speed | Realtime on CPU, faster on GPU |
| Instruct | Not supported (preset voice carries the style) |
| License | Apache 2.0 |
## Qwen CustomVoice — 9 Premium Voices with Instruct Control
Qwen CustomVoice ships with 9 curated speakers and supports **natural-language style control** — you tell the model how to deliver the line ("speak slowly with warmth", "authoritative and clear") and it adapts tone, emotion, and pace.
Two model sizes:
- **1.7B** — full quality, recommended default
- **0.6B** — lighter, faster, lower-end hardware
**Repository:** [`Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) (and 0.6B variant) · by Alibaba
### Voice Catalog
| Speaker | Gender | Language | Description |
| --------- | ------ | -------- | ------------------------------------------------------------ |
| Vivian | female | Chinese | Bright, slightly edgy young female voice |
| Serena | female | Chinese | Warm, gentle young female voice |
| Uncle Fu | male | Chinese | Seasoned male voice with a low, mellow timbre |
| Dylan | male | Chinese | Youthful Beijing male voice with a clear, natural timbre |
| Eric | male | Chinese | Lively Chengdu male voice with a slightly husky brightness |
| Ryan | male | English | Dynamic male voice with strong rhythmic drive (default) |
| Aiden | male | English | Sunny American male voice with a clear midrange |
| Ono Anna | female | Japanese | Playful Japanese female voice with a light, nimble timbre |
| Sohee | female | Korean | Warm Korean female voice with rich emotion |
### Using Instruct Mode
In the floating generate box, switch to a Qwen CustomVoice profile and click the **delivery instructions** toggle (slider icon, left of the generate button). A second textarea appears below the main text:
- Main text → what you want the voice to say
- Instruct text → how you want it delivered
Examples of effective instruct prompts:
```
Speak slowly with emphasis, like reading bedtime stories
Warm and friendly, conversational tone
Professional and authoritative, broadcast quality
Whisper, intimate and close
Excited and energetic, like sports commentary
```
The full Generate page also surfaces the instruct field as a separate input.
### Qwen CustomVoice at a Glance
| Property | Value |
| --------------- | -------------------------------------------------- |
| Parameters | 1.7B / 0.6B |
| Languages | Chinese, English, Japanese, Korean (10 supported) |
| Voices | 9 curated preset speakers |
| VRAM | ~3.5 GB (1.7B), ~1.2 GB (0.6B) |
| Instruct | Yes — natural-language style control |
| Cloning | No — paired Base Qwen3-TTS engine handles cloning |
## Cloning vs Preset — Quick Decision
| You want… | Use |
| -------------------------------------------------- | ----------------------------------------- |
| To replicate a specific person's voice | [Voice Cloning](/overview/voice-cloning) |
| Production-ready voices with no audio prep | Kokoro or Qwen CustomVoice |
| The smallest possible footprint (CPU-only) | Kokoro |
| Fine control over delivery (tone, pace, emotion) | Qwen CustomVoice |
| The broadest language coverage | [Voice Cloning](/overview/voice-cloning) via Chatterbox Multilingual (23 langs) |
## Limitations
<Callout type="warn">
Preset voices are fixed — you can't fine-tune or modify the underlying voice. If you want a specific voice that isn't in the catalog, use a cloning engine and provide a reference sample.
</Callout>
- Preset voices can't be exported to use in other Voicebox installations as audio (only as profile metadata pointing to the same engine + voice ID)
- The Kokoro voice catalog is set by the upstream model — new voices appear only when hexgrad publishes new model releases
- Qwen CustomVoice's 9 speakers are part of the model checkpoint — same constraint
## Next Steps
<Cards>
<Card title="Voice Cloning" href="/overview/voice-cloning">
Clone a specific voice from your own audio
</Card>
<Card title="Generate Speech" href="/overview/generating-speech">
Use a profile to generate audio
</Card>
<Card title="Build Stories" href="/overview/building-stories">
Compose multi-voice narratives
</Card>
</Cards>
@@ -69,6 +69,16 @@ Now let's use your new voice profile to generate speech.
```
Hello! This is my first voice generation with Voicebox.
```
<Callout type="info">
Paralinguistic tags like `[laugh]`, `[sigh]`, and `[gasp]` only work with
**Chatterbox Turbo**. Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and
HumeAI TADA will read those tags literally instead of turning them into
expressive sounds.
</Callout>
To insert supported tags, select **Chatterbox Turbo** and type `/` in the
text input to open the tag inserter.
</Step>
<Step title="Generate">
+9 -1
View File
@@ -31,8 +31,16 @@ In Remote Mode, the Voicebox desktop app (running on your local machine) communi
# Install Python dependencies
pip install -r requirements.txt
# Engines with incompatible transitive pins — install with --no-deps
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
# Qwen3-TTS from source
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
Or just run `just setup` from the repo root, which handles all of this.
</Step>
<Step title="Start the Server">
@@ -135,4 +143,4 @@ Expected performance on various GPUs:
## Troubleshooting
See the [Troubleshooting Guide](/guides/troubleshooting#remote-mode-issues) for common remote mode issues.
See the [Troubleshooting Guide](/overview/troubleshooting) for common issues.
+150 -17
View File
@@ -29,6 +29,14 @@ Windows SmartScreen may warn that the app is unrecognized.
This is expected for unsigned applications. We're working on code signing for future releases.
</Callout>
### Linux: AppImage Won't Run
**Solution:**
```bash
chmod +x voicebox-*.AppImage
./voicebox-*.AppImage
```
## Server Issues
### Backend Server Won't Start
@@ -66,7 +74,7 @@ Windows SmartScreen may warn that the app is unrecognized.
```bash
# macOS/Linux
chmod +x ~/Library/Application\ Support/com.voicebox.app/backend/voicebox-server
chmod +x ~/Library/Application\ Support/sh.voicebox.app/backend/voicebox-server
```
</Accordion>
@@ -75,16 +83,62 @@ Windows SmartScreen may warn that the app is unrecognized.
**macOS:**
```bash
tail -f ~/Library/Application\ Support/com.voicebox.app/logs/server.log
tail -f ~/Library/Application\ Support/sh.voicebox.app/logs/server.log
```
**Windows:**
```bash
type %APPDATA%\com.voicebox.app\logs\server.log
type %APPDATA%\sh.voicebox.app\logs\server.log
```
</Accordion>
</AccordionGroup>
### `flash-attn is not installed` Warning in Server Logs
**Symptoms:**
```
Warning: flash-attn is not installed. Will only run the manual PyTorch version.
Please install flash-attn for faster inference.
```
**This is harmless.** The warning is emitted by our transformer-based engines (Chatterbox / Qwen) on every startup. FlashAttention is an optional acceleration library — when it's not present, PyTorch's built-in scaled-dot-product attention (SDPA) runs instead, which is near-FA2 throughput on modern GPUs. Generation works normally.
**Why it shows up on every platform:**
- **Windows:** `flash-attn` has no official Windows support. The upstream project (Dao-AILab/flash-attention) still only says it *might* work, and source builds typically fail on recent CUDA/MSVC combinations.
- **macOS (Apple Silicon):** FlashAttention is CUDA-only and doesn't apply here at all. MLX has its own optimized attention kernels.
- **Linux:** It's not pinned in our requirements because installing it is fragile and version-sensitive; users who want it install it themselves.
**Solutions (all optional):**
<AccordionGroup>
<Accordion title="Ignore it (recommended)">
PyTorch SDPA is what actually runs the model, and on Ampere/Ada/Hopper GPUs it's within a few percent of FA2 for our workloads. You won't notice a meaningful speed difference.
</Accordion>
<Accordion title="Install flash-attn on Linux">
```bash
pip install flash-attn --no-build-isolation
```
Requires a matching CUDA toolkit. Build can take 20+ minutes.
</Accordion>
<Accordion title="Install flash-attn on Windows (community wheels)">
Official builds don't exist, but community maintainers publish prebuilt wheels:
- [kingbri1/flash-attention releases](https://github.com/kingbri1/flash-attention/releases)
- [bdashore3/flash-attention releases](https://github.com/bdashore3/flash-attention/releases)
Pick the wheel matching your exact CUDA + PyTorch + Python combination. Example:
```bash
pip install https://github.com/kingbri1/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu128torch2.8.0cxx11abiFALSE-cp312-cp312-win_amd64.whl
```
Alternatively, run Voicebox's backend inside WSL2 and use the standard Linux wheels.
</Accordion>
</AccordionGroup>
### Connection Timeout
**Symptoms:**
@@ -105,12 +159,13 @@ Windows SmartScreen may warn that the app is unrecognized.
- Progress indicator stuck at "Loading model..."
**Explanation:**
This is expected behavior. The first generation downloads the Qwen3-TTS model (~2-4GB) and initializes it.
This is expected behavior. The first generation downloads the selected TTS engine's model and initializes it. Sizes range from 350 MB (Kokoro) to 8 GB (TADA 3B).
**Solution:**
- Wait for the initial download to complete
- Subsequent generations will be much faster
- Wait for the initial download to complete (progress is shown in Settings → Models)
- Subsequent generations reuse the cached model and are much faster
- Check your internet connection
- For low-bandwidth setups, start with Kokoro (~350 MB) or LuxTTS (~300 MB)
### Poor Voice Quality
@@ -173,6 +228,34 @@ This is expected behavior. The first generation downloads the Qwen3-TTS model (~
</Accordion>
</AccordionGroup>
### MLX "Failed to load the default metallib" (Apple Silicon)
**Symptoms:**
- Generation fails with "library not found" or "metallib" errors
- Server logs reference missing Metal shader libraries
**Solutions:**
<AccordionGroup>
<Accordion title="Rebuild the Server Binary">
```bash
just build-server
```
The build script bundles MLX Metal shader libraries on Apple Silicon automatically.
</Accordion>
<Accordion title="Reinstall MLX Dependencies">
```bash
pip install -r backend/requirements-mlx.txt
```
</Accordion>
<Accordion title="Verify Backend Detection">
Check Settings → Server Status. Should show **Backend: MLX** on Apple Silicon. If it shows **Backend: PYTORCH**, MLX isn't installed correctly.
</Accordion>
</AccordionGroup>
## Audio Issues
### No Audio Playback
@@ -202,7 +285,7 @@ This is expected behavior. The first generation downloads the Qwen3-TTS model (~
### Backend Won't Start in Dev Mode
**Symptoms:**
- `bun run dev:server` fails
- `just dev-backend` or `just dev` fails
- Import errors or module not found
**Solutions:**
@@ -233,11 +316,19 @@ This is expected behavior. The first generation downloads the Qwen3-TTS model (~
</Accordion>
<Accordion title="Dependencies">
Reinstall dependencies:
Reinstall dependencies — easiest via `just`:
```bash
just setup
```
Or manually:
```bash
cd backend
pip install -r requirements.txt
pip install --no-deps chatterbox-tts
pip install --no-deps hume-tada
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
```
</Accordion>
@@ -310,8 +401,8 @@ bun run tauri build
- Delete the lock file:
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db-wal
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db-shm
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db-wal
```
### Corrupted Database
@@ -328,10 +419,10 @@ bun run tauri build
```bash
# macOS
rm ~/Library/Application\ Support/com.voicebox.app/data/voicebox.db
rm ~/Library/Application\ Support/sh.voicebox.app/data/voicebox.db
# Windows
del %APPDATA%\com.voicebox.app\data\voicebox.db
del %APPDATA%\sh.voicebox.app\data\voicebox.db
```
Restart the app to create a fresh database.
@@ -348,7 +439,26 @@ Restart the app to create a fresh database.
- Check your internet connection
- Check HuggingFace Hub status
- Try using a VPN if HuggingFace is blocked in your region
- Manually download and place in cache directory
- Manually download via the HuggingFace CLI and place in the cache directory:
```bash
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
```
### Qwen 0.6B Downloads the Same Files as 1.7B on Apple Silicon
**Symptoms:**
- You select Qwen 0.6B on an Apple Silicon Mac and the download is the same size as 1.7B
- Generation speed and VRAM usage match 1.7B, not the expected smaller model
**Explanation:**
This is intentional, not a bug. The MLX community only publishes `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` — there is no 0.6B MLX build. Voicebox's model registry falls back to the 1.7B weights when 0.6B is selected on MLX (see `backend/backends/__init__.py`).
**Solution:**
- On Apple Silicon, both size options use the 1.7B model — pick either.
- If you specifically need a smaller model, switch to **Kokoro 82M** (~350 MB) or **LuxTTS** (~300 MB) — both CPU-realtime.
- On Windows/Linux with PyTorch, 0.6B and 1.7B are distinct repos and behave differently.
### Wrong Model Version
@@ -357,10 +467,10 @@ Restart the app to create a fresh database.
- Different voice output
**Solutions:**
Clear the model cache and re-download:
Clear the model cache and re-download. Replace the `Qwen*` glob with the engine org prefix for other engines (`ResembleAI*` for Chatterbox, `HumeAI*` for TADA, `hexgrad*` for Kokoro, etc.) or use `DELETE /models/{name}` via the API.
```bash
# macOS
# macOS / Linux
rm -rf ~/.cache/huggingface/hub/models--Qwen*
# Windows
@@ -395,6 +505,16 @@ rmdir /s %USERPROFILE%\.cache\huggingface\hub\models--Qwen*
<Accordion title="Update GPU Drivers">
Outdated drivers can cause performance issues. Update to the latest NVIDIA drivers.
</Accordion>
<Accordion title="Apple Silicon: Confirm MLX Backend">
Check Settings → Server Status. Should show **Backend: MLX** on Apple Silicon — MLX is 4–5× faster than PyTorch here. If it shows **Backend: PYTORCH**, reinstall MLX:
```bash
pip install -r backend/requirements-mlx.txt
```
GPU availability should read "Metal (Apple Silicon via MLX)".
</Accordion>
</AccordionGroup>
### High Memory Usage
@@ -408,6 +528,21 @@ rmdir /s %USERPROFILE%\.cache\huggingface\hub\models--Qwen*
- Clear generation history
- Restart the app periodically
## Update Issues
### "Update Check Failed"
**Solutions:**
- Confirm your internet connection — updates are fetched from GitHub releases.
- Ensure `github.com` is accessible and not blocked by a firewall or proxy.
- As a fallback, download the latest release from GitHub and install manually.
### "Invalid Signature" Error
**Solutions:**
- Re-download the installer — the signature may have been corrupted in transit.
- Verify the `.sig` file matches the installer; if it doesn't, file an issue.
## Remote Mode Issues
### Can't Connect to Remote Server
@@ -473,5 +608,3 @@ python --version
# GPU info (if generation issues)
nvidia-smi # NVIDIA GPUs
```
For more detailed troubleshooting, see the [TROUBLESHOOTING.md](https://github.com/jamiepine/voicebox/blob/main/docs/TROUBLESHOOTING.md) file in the repository.
+58 -15
View File
@@ -1,11 +1,25 @@
---
title: "Voice Cloning"
description: "Clone any voice from just a few seconds of audio"
description: "Clone any voice from a few seconds of reference audio"
---
## Overview
Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning from just a few seconds of audio. The model captures prosody, emotion, and natural cadence.
Voicebox can replicate a specific person's voice from a short audio sample — known as **zero-shot voice cloning**. You provide 10-30 seconds of clear speech, the model extracts a voice embedding, and from then on you can generate any text in that voice.
Five engines in 0.4 support cloning:
| Engine | Languages | Strengths |
| --------------------------- | --------- | -------------------------------------------------------------------------- |
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual, supports delivery instructions on the same kwarg |
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Hindi, Swahili, Hebrew, more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion tags (`[laugh]`, `[sigh]`) |
| **LuxTTS** | English | Lightweight (~1 GB VRAM), 48 kHz output, 150x realtime on CPU |
| **TADA** (1B / 3B) | 10 | Speech-language model with 700s+ coherent long-form generation |
<Callout type="info">
Don't want to record audio? Use a curated voice from Kokoro or Qwen CustomVoice instead — see [Preset Voices](/overview/preset-voices).
</Callout>
## How It Works
@@ -13,17 +27,30 @@ Voicebox uses **Qwen3-TTS** from Alibaba to achieve near-perfect voice cloning f
<Step title="Upload or Record Sample">
Provide 10-30 seconds of clear speech from the target voice
</Step>
<Step title="Model Analysis">
Qwen3-TTS analyzes vocal characteristics, tone, and speaking patterns
<Step title="Engine Analysis">
The selected engine analyzes vocal characteristics, tone, and speaking patterns
</Step>
<Step title="Voice Profile Created">
The model generates a voice embedding for synthesis
A voice embedding is generated and stored with your profile
</Step>
<Step title="Generate Speech">
Use the profile to generate any text in the cloned voice
</Step>
</Steps>
## Choosing an Engine for Cloning
Different engines suit different use cases. The profile grid greys out unsupported engines so you can switch easily.
| If you want… | Pick |
| -------------------------------------------------- | --------------------- |
| Best overall quality on a few common languages | **Qwen3-TTS 1.7B** |
| Faster generation, slightly lower quality | **Qwen3-TTS 0.6B** |
| Languages outside Qwen's 10 (Arabic, Hindi, etc.) | **Chatterbox Multilingual** |
| Expressive English with `[laugh]` `[sigh]` tags | **Chatterbox Turbo** |
| CPU-only or GPU-light setup, English | **LuxTTS** |
| Long-form generation (audiobooks, full chapters) | **TADA 3B** |
## Best Practices
### Sample Quality
@@ -52,24 +79,40 @@ Adding multiple samples from the same speaker can improve quality:
- Different recording conditions
<Callout type="info">
The model will learn a more robust representation from diverse samples.
The model will learn a more robust representation from diverse samples. Especially helpful for distinctive voices the model might otherwise smooth over.
</Callout>
## Supported Languages
## Supported Languages by Engine
Currently supported:
- English
- Chinese (Mandarin)
- **Qwen3-TTS** — English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian (10)
- **Chatterbox Multilingual** — Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Turkish (23)
- **Chatterbox Turbo** — English
- **LuxTTS** — English
- **TADA 3B** — 10 multilingual; **TADA 1B** — English
More languages coming soon.
For complete language tables and engine-specific notes, see the [TTS Engines developer guide](/developer/tts-engines).
## Limitations
<Callout type="warn">
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice.
Voice cloning should only be used with consent. Ensure you have permission to clone someone's voice. See the project's [SECURITY.md](https://github.com/jamiepine/voicebox/blob/main/SECURITY.md) and your local laws on synthetic voice content.
</Callout>
- Quality depends on sample clarity
- Works best with consistent speaking tone
- Quality depends on sample clarity — noisy samples produce noisy clones
- Works best with consistent speaking tone within a sample
- May struggle with extreme accents or speech impediments
- Background noise reduces quality
- Background noise reduces quality and can introduce artifacts
## Next Steps
<Cards>
<Card title="Creating Voice Profiles" href="/overview/creating-voice-profiles">
Step-by-step guide to creating profiles
</Card>
<Card title="Preset Voices" href="/overview/preset-voices">
Use built-in voices instead of cloning
</Card>
<Card title="Generating Speech" href="/overview/generating-speech">
Use a profile to generate audio
</Card>
</Cards>
-188
View File
@@ -1,188 +0,0 @@
# Code Review: `backend/` Post-Refactor
**Date:** 2026-03-16
**Scope:** Full review of `backend/` after major refactor
## Overall Assessment
The refactor is well-executed. The codebase follows a clean layered architecture (routes -> services -> backends) with good separation of concerns. The code is readable, the module boundaries are sensible, and the migration strategy is pragmatic for a desktop app. Below are findings organized by severity.
---
## Critical Issues
### 1. Double `init_db()` in bundled entry point
**File:** `server.py:261`
`server.py:261` calls `database.init_db()` explicitly, but `app.py:140` also calls `database.init_db()` inside the `startup` event handler. When running via `server.py`, the database gets initialized twice -- once before uvicorn starts and once during the startup event. This is likely benign (idempotent migrations), but the second call recreates the engine and `SessionLocal`, which could cause subtle issues if any sessions were opened between the two calls.
**Recommendation:** Remove the explicit `init_db()` call in `server.py:260-262` and rely solely on the startup event in `app.py`. The same issue exists in `main.py:38`.
### 2. SSE endpoint holds DB session open indefinitely
**File:** `routes/generations.py:179-212`
The `get_generation_status` SSE endpoint receives a `db` session via `Depends(get_db)` but keeps it open for the lifetime of the SSE stream (polling every 1 second). This ties up a SQLite connection for potentially minutes. With SQLite's single-writer model, this is a contention risk.
**Recommendation:** Open and close a short-lived session on each poll iteration instead of holding one via dependency injection:
```python
async def event_stream():
while True:
db = next(get_db())
try:
gen = db.query(DBGeneration).filter_by(id=generation_id).first()
...
finally:
db.close()
await asyncio.sleep(1)
```
---
## High Severity
### 3. `_save_retry` creates no version record
**File:** `services/generation.py:201-214`
`_save_retry` writes the audio file but creates no `GenerationVersion` entry. If the generation previously had versions (from an initial generate that failed mid-effects, for example), the retry result won't appear in the versions list. This creates an inconsistency: some generations have versions, retried ones don't.
**Recommendation:** Create a "clean" version in `_save_retry` the same way `_save_generate` does.
### 4. `datetime.utcnow()` is deprecated
**File:** `services/stories.py` and others
`datetime.utcnow()` is deprecated as of Python 3.12 and returns a naive datetime. Used throughout `services/stories.py` (lines 95, 96, 193, 307, 360, 404, 457, 529, 537, 598, 610, 652, 716, 775) and possibly other service files.
**Recommendation:** Replace with `datetime.now(datetime.UTC)` or `datetime.now(timezone.utc)`.
### 5. `list_stories` N+1 query
**File:** `services/stories.py:122-132`
`list_stories` issues one `COUNT(*)` query per story inside a loop. For N stories, that's N+1 queries.
**Recommendation:** Use a subquery or a single aggregated query:
```python
from sqlalchemy import func
counts = dict(
db.query(DBStoryItem.story_id, func.count(DBStoryItem.id))
.group_by(DBStoryItem.story_id)
.all()
)
```
---
## Medium Severity
### 6. `create_story` queries item count immediately after creation
**File:** `services/stories.py:103`
Line 103 queries the item count for a story that was just created -- it will always be 0. This is wasted I/O.
### 7. Bare `except Exception` with silent `pass`
**File:** `routes/generations.py:69-70`
When parsing a profile's stored `effects_chain` JSON, exceptions are silently swallowed. A corrupt JSON blob would result in no effects being applied with no logging.
**Recommendation:** Log the exception at warning level.
### 8. `update_story_item_times` uses `generation_id` as key
**File:** `services/stories.py:643-649`
`item_map` is keyed by `generation_id`, but the same generation can appear in a story multiple times (via split/duplicate). This would cause key collisions, and only the last item per generation_id would be updatable.
**Recommendation:** Key by `item_id` instead, and change the `StoryItemUpdateTime` model to use `item_id`.
### 9. Thread safety gap in `get_stt_backend`
**File:** `backends/__init__.py:499-520`
`get_stt_backend()` uses no locking (unlike `get_tts_backend_for_engine` which uses `_tts_backends_lock`). A race condition could create duplicate STT backend instances.
**Recommendation:** Add a lock or use the same double-checked locking pattern.
### 10. Unused `_tts_backend` global
**File:** `backends/__init__.py:156`
`_tts_backend` is declared but never read or written outside of `reset_backends()`. All TTS access goes through `_tts_backends` dict. Dead code.
### 11. `trim_story_item` returns `None` for validation errors
**File:** `services/stories.py:448`
Returning `None` for "item not found" and "invalid trim values" is ambiguous. The route handler can't distinguish between a 404 and a 400 response.
**Recommendation:** Raise specific exceptions (e.g., `ValueError` for invalid trim) so the route can return the appropriate HTTP status.
### 12. `load_engine_model` calls different method names
**File:** `backends/__init__.py:340-346`
For Qwen, it calls `load_model_async(model_size)`. For others, it calls `load_model()` with no arguments. But the `TTSBackend` protocol defines `load_model(self, model_size: str)`. This means the protocol signature doesn't match actual usage for either path.
**Recommendation:** Align the protocol definition with actual backend implementations, or add `load_model_async` to the protocol.
---
## Low Severity / Style
### 13. Inconsistent `async` usage in services
Functions like `create_story`, `list_stories`, etc. in `services/stories.py` are `async def` but contain no `await` expressions. They do synchronous SQLAlchemy I/O. While this works (the functions are awaitable), it's misleading -- these will block the event loop during DB access.
This is a known tradeoff with synchronous SQLAlchemy + FastAPI, and acceptable for a single-user desktop app with SQLite, but worth noting for documentation.
### 14. `getattr(item, "version_id", None)` pattern
**File:** `services/stories.py:57, 504, 524, etc.`
Multiple places use `getattr(item, "version_id", None)` on a DB model that has `version_id` as a declared column (from migrations). After the migration runs, this is always a real attribute. The defensive `getattr` is cargo-culted.
**Recommendation:** Access `item.version_id` directly. If the column is missing, the ORM will raise a clear error.
### 15. `reorder_story_items` ignores trim values
**File:** `services/stories.py:707`
When recalculating timecodes, it uses the full `generation.duration` rather than the effective (trimmed) duration. Trimmed items will have larger gaps than intended.
### 16. Module-level `import torch` in `app.py:44`
`import torch` at module level in `app.py` means torch loads on every import of the app module. This is intentional (AMD env vars must be set first), but the comment on line 38 should mention that this is why the import is here and not at the top.
### 17. f-strings in logging in `server.py`
`server.py` uses f-strings in logging calls (e.g., lines 63-66, 252, 256, 264). This evaluates the string even when the log level is filtered out. The rest of the codebase correctly uses `%s` style (e.g., `app.py:131`).
---
## Architecture Observations (Not Issues)
- **Clean layered design**: routes -> services -> backends with Pydantic models as the API contract.
- **Backend abstraction** with `Protocol` classes and a config registry is a solid pattern.
- **Serial generation queue** (`task_queue.py`) is simple and effective for single-GPU serialization.
- **Migration approach** is pragmatic for the use case. The idempotent, check-then-act pattern is reliable.
- **The `generation.py` refactor** (collapsing three closures into `run_generation` with a mode parameter) is a clear improvement.
---
## Summary
| Severity | Count |
|----------|-------|
| Critical | 2 |
| High | 3 |
| Medium | 7 |
| Low/Style | 5 |
The refactor achieved its goals: clear module boundaries, reduced duplication (especially in `generation.py`), and a well-organized backend abstraction. The critical items (double init_db and SSE session leak) should be addressed first, followed by the version consistency issue in retry and the N+1 query.
-87
View File
@@ -1,87 +0,0 @@
# Documentation Migration: Mintlify → Fumadocs
This document summarizes the migration of documentation from `/docs` (Mintlify) to `/docs2` (Fumadocs).
## What Was Done
### 1. Files Copied
- ✅ All 29 MDX files from `/docs` folders (overview, api, developer, plans)
- ✅ All 4 root-level markdown files (AUTOUPDATER.md, AUTOUPDATER_QUICKSTART.md, TROUBLESHOOTING.md, README.md)
- ✅ All images (3 webp files) → `public/images/`
- ✅ All logo files (2 png files) → `public/logo/`
### 2. Component Migration
Created compatibility layer in `components/mintlify-compat.tsx` that maps Mintlify components to Fumadocs equivalents:
- `<Frame>` → Simple div wrapper (images are zoomable by default in Fumadocs)
- `<CardGroup>` → `<Cards>` (Fumadocs component)
- `<Card>` → `<Card>` (with icon string → Lucide icon mapping)
- `<Steps>` / `<Step>` → Direct mapping to Fumadocs components
- `<Tip>`, `<Note>`, `<Info>` → `<Callout type="info">`
- `<Warning>` → `<Callout type="warn">`
- `<Danger>` → `<Callout type="error">`
- `<AccordionGroup>` / `<Accordion>` → HTML `<details>` / `<summary>` elements
### 3. Navigation Structure
Created `meta.json` files for each folder:
- `content/docs/meta.json` - Root documentation
- `content/docs/overview/meta.json` - Overview pages
- `content/docs/api/meta.json` - API reference
- `content/docs/developer/meta.json` - Developer docs
- `content/docs/plans/meta.json` - Plans/roadmap
### 4. Link Fixes
- Fixed incorrect `/guides/...` paths → `/overview/...`
- All internal links now use correct paths
### 5. Branding
- Updated `lib/layout.shared.tsx` to use "Voicebox" as the nav title
## File Structure
```
docs2/
├── components/
│ └── mintlify-compat.tsx # Mintlify → Fumadocs component mappings
├── content/docs/
│ ├── meta.json # Root navigation
│ ├── overview/ # 12 MDX files
│ ├── api/ # 5 MDX files
│ ├── developer/ # 12 MDX files
│ ├── plans/ # 4 MD files
│ └── *.md # 4 root markdown files
├── public/
│ ├── images/ # 3 webp files
│ └── logo/ # 2 png files
└── mdx-components.tsx # MDX component configuration
```
## Icon Mapping
The following icon strings are mapped to Lucide icons:
- `microphone` → Mic
- `film` → Film
- `code` → Code
- `shield` → Shield
- `download` → Download
- `rocket` → Rocket
- `apple` → Apple
- `windows` → Windows
- `server` → Server
- `user` → User
- `waveform` → Waveform
## Next Steps
1. **Test the build**: Run `npm run build` (requires Node.js >= 20.9.0)
2. **Start dev server**: Run `npm run dev` to preview
3. **Customize styling**: Update `app/global.css` if needed
4. **Add more icons**: Extend `iconMap` in `mintlify-compat.tsx` as needed
5. **Review navigation**: Adjust `meta.json` files to customize page order
## Notes
- Image paths (`/images/...`) work as-is since Next.js serves from `public/`
- All Mintlify components are now compatible with Fumadocs
- Navigation structure follows Fumadocs conventions
- No breaking changes to content - all MDX files work with compatibility layer
-482
View File
@@ -1,482 +0,0 @@
# Voicebox Project Status & Roadmap
> Last updated: 2026-03-18 | Current version: **v0.3.0** | 13.4k stars | ~136 open issues | 9 open PRs
---
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Current State](#current-state)
3. [Open PRs — Triage & Analysis](#open-prs--triage--analysis)
4. [Open Issues — Categorized](#open-issues--categorized)
5. [Existing Plan Documents — Status](#existing-plan-documents--status)
6. [New Model Integration — Landscape](#new-model-integration--landscape)
7. [Architectural Bottlenecks](#architectural-bottlenecks)
8. [Recommended Priorities](#recommended-priorities)
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────┐
│ Tauri Shell (Rust) │
│ ┌───────────────────────────────────────────────┐ │
│ │ React Frontend (app/) │ │
│ │ Zustand stores · API client · Generation UI │ │
│ │ Stories Editor · Voice Profiles · Model Mgmt │ │
│ └──────────────────────┬────────────────────────┘ │
│ │ HTTP :17493 │
│ ┌──────────────────────▼────────────────────────┐ │
│ │ FastAPI Backend (backend/) │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ TTSBackend Protocol │ │ │
│ │ │ ┌──────────┐ ┌───────┐ ┌───────────┐ │ │ │
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │ │ │ TADA │ │ │ │
│ │ │ │(1B / 3B) │ │ │ │
│ │ │ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ ┌───────────┐ ┌─────────┐ │ │
│ │ │ STTBackend│ │ Profiles│ │ │
│ │ │ (Whisper) │ │ History │ │ │
│ │ └───────────┘ │ Stories │ │ │
│ │ └─────────┘ │ │
│ └───────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
```
### Key Files
| Layer | File | Purpose |
|-------|------|---------|
| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2850 lines) |
| TTS protocol | `backend/backends/__init__.py:32-101` | `TTSBackend` Protocol definition |
| Model registry | `backend/backends/__init__.py:17-29,153-366` | `ModelConfig` dataclass + registry helpers |
| TTS factory | `backend/backends/__init__.py:382-426` | Thread-safe engine registry (double-checked locking) |
| PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
| MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
| Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
| Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
| Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
| Engine selector | `app/src/components/Generation/EngineModelSelector.tsx` | Shared engine/model dropdown |
| Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
| Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
| GPU acceleration | `app/src/components/ServerSettings/GpuAcceleration.tsx` | CUDA backend swap UI |
| Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission |
| Language constants | `app/src/lib/constants/languages.ts` | Per-engine language maps |
### How TTS Generation Works (Current Flow)
```
POST /generate
1. Look up voice profile from DB
2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo | tada)
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
→ tts_backend.create_voice_prompt(audio_path, reference_text)
7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
8. Post-process: trim_tts_output() for Chatterbox engines
9. Save WAV → data/generations/{id}.wav
10. Insert history record in SQLite
11. Return GenerationResponse
```
---
## Current State
### What's Shipped (v0.3.0)
**Core TTS:**
- Qwen3-TTS voice cloning (1.7B and 0.6B models)
- MLX backend for Apple Silicon, PyTorch for everything else
- Multi-engine TTS architecture with thread-safe backend registry (PR #254)
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
- HumeAI TADA integration — 1B English + 3B Multilingual speech-language model (PR #296)
- Chunked TTS generation for long text — engine-agnostic, removes ~500 char limit (PR #266)
- Async generation queue (PR #269)
- Post-processing audio effects system (PR #271)
- Centralized model config registry (`ModelConfig` dataclass) — no per-engine dispatch maps
- Shared `EngineModelSelector` component — engine/model dropdown defined once, used in both generation forms
**Infrastructure:**
- CUDA backend swap via binary download and restart (PR #252), upgraded to cu128 (PR #316)
- CUDA backend split into independently versioned server + libs archives (PR #298)
- Docker + web deployment (PR #161)
- Backend refactor: modular architecture, style guide, tooling (PR #285)
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
- Windows support: CUDA detection, cross-platform justfile, clean server shutdown (PR #272)
- Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large variants)
- Model management UI with inline download progress bars + folder migration (PR #268)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching
- Streaming generation endpoint (MLX only)
- Audio player freeze fix + UX improvements (PR #293)
- CORS restriction to known local origins (PR #88)
### Abandoned Integrations
| Model | PR | Reason |
|-------|----|--------|
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. |
### What's In-Flight
| Feature | Branch/PR | Status |
|---------|-----------|--------|
| Kokoro 82M TTS engine | WIP | In development — 82M CPU-realtime engine, 8 languages |
### TTS Engine Comparison
| Engine | Model Name | Languages | Size | Key Features | Instruct Support |
|--------|-----------|-----------|------|-------------|-----------------|
| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | 10 | ~1.2 GB | Lighter, faster | None |
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) for expressiveness |
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only, no separate instruct param |
| TADA 1B | `tada-1b` | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
| TADA 3B Multilingual | `tada-3b-ml` | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
| Kokoro 82M | `kokoro` | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
### Multi-Engine Architecture (Shipped)
The singleton TTS backend blocker described in the previous version of this doc has been **resolved**. The architecture now supports:
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada'`
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
### Known Limitations
- **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
- **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
- **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
- **Instruct parameter is non-functional** (#224): The UI exposes an instruct text field, but it's silently dropped by every backend. The Qwen3-TTS Base model we ship only supports voice cloning — instruct requires the separate CustomVoice model variant (`Qwen3-TTS-12Hz-1.7B-CustomVoice`), which uses predefined speakers instead of ref audio. The instruct UI should be hidden until a backend with real support is integrated.
- **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
- **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.
---
## Open PRs — Triage & Analysis
### Recently Merged (Since Last Update)
| PR | Title | Merged |
|----|-------|--------|
| **#316** | Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI | 2026-03-18 |
| **#305** | fix: bundle qwen_tts source files in PyInstaller build | 2026-03-17 |
| **#298** | feat: split CUDA backend into independently versioned server + libs archives | 2026-03-17 |
| **#296** | Add HumeAI TADA TTS engine (1B English + 3B Multilingual) | 2026-03-17 |
| **#295** | fix: batch of bug fixes from issue tracker | 2026-03-17 |
| **#293** | Fix audio player freezing and improve UX | 2026-03-17 |
| **#294** | Settings overhaul: routed sub-tabs, server logs, changelog, about page | 2026-03-16 |
| **#288** | Better docs | 2026-03-16 |
| **#285** | Backend refactor: modular architecture, style guide, tooling | 2026-03-16 |
| **#274** | Landing page v0.2.0 redesign | 2026-03-15 |
| **#272** | Windows support: CUDA detection, cross-platform justfile, clean server shutdown | 2026-03-15 |
| **#271** | Add post-processing audio effects system | 2026-03-14 |
| **#269** | feat: async generation queue | 2026-03-13 |
| **#268** | feat: model management improvements and folder migration | 2026-03-13 |
| **#266** | feat: chunked TTS generation for long text (engine-agnostic) | 2026-03-13 |
| **#265** | feat: paralinguistic tag autocomplete for Chatterbox Turbo | 2026-03-13 |
| **#264** | fix: Chatterbox float64 dtype mismatch + model unload button | 2026-03-13 |
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | 2026-03-13 |
| **#230** | docs: fix README grammar | 2026-03-13 |
| **#161** | feat: Docker + web deployment | 2026-03-13 |
| **#88** | security: restrict CORS to known local origins | 2026-03-13 |
### Currently Open (9 PRs)
| PR | Title | Status | Notes |
|----|-------|--------|-------|
| **#311** | feat: add CosyVoice2/3 TTS engine | **Will close** | Model quality too poor. See Abandoned Integrations. |
| **#253** | Enhance speech tokenizer with 48kHz version | Community PR | Qwen tokenizer upgrade. Worth reviewing. |
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Superseded | Our PR #305 shipped this. Can close. |
| **#227** | fix: harden input validation & file safety | Community PR | Coupled to #225 (custom models). |
| **#225** | feat: custom HuggingFace model support | Community PR | Needs rework for multi-engine arch. |
| **#218** | fix: unify qwen tts cache dir on Windows | Community PR | Windows-specific path fix. Still relevant. |
| **#195** | feat: per-profile LoRA fine-tuning | Draft | Complex. 15 new endpoints. |
| **#154** | feat: Audiobook tab | Community PR | Chunked generation now shipped (#266). |
| **#91** | fix: CoreAudio device enumeration | Draft | macOS audio device handling. |
---
## Open Issues — Categorized
### GPU / Hardware Detection (19 issues)
The single most reported category. Users on Windows with NVIDIA GPUs frequently report "GPU not detected."
**Root causes (likely):**
- PyInstaller binary doesn't bundle CUDA correctly → falls back to CPU
- DirectML/Vulkan path not implemented (AMD on Windows)
- Binary size limit means CUDA can't ship in the main release
**Key issues:** #239, #222, #220, #217, #208, #198, #192, #167, #164, #141, #130, #127
**Fix path:** PR #252 (CUDA backend swap) is now merged. Users can download the CUDA binary separately from the GPU acceleration settings. Many of these issues may now be resolvable — needs triage to confirm.
### Model Downloads (20 issues)
Second most reported. Users get stuck downloads, can't resume, no offline fallback.
**Key issues:** #249, #240, #221, #216, #212, #181, #180, #159, #150, #149, #145, #143, #135, #134
**Fix path:** PR #238 (cancel/clear UI) is now merged. PR #152 (offline crash fix) still open. Inline progress bars now show for all engines. Resume support not yet addressed.
### Language Requests (18 issues)
Strong demand for: Hindi (#245), Indonesian (#247), Dutch (#236), Hebrew (#199), Greek (#188), Portuguese (#183), Persian (#162), and many more.
**Key issues:** #247, #245, #236, #211, #205, #199, #189, #188, #187, #183, #179, #162
**Fix path:** Chatterbox Multilingual (merged via #257) now supports 23 languages including many of the requested ones: Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Per-engine language filtering (PR #258) ensures the UI shows correct options. Several of these issues may be closeable.
### New Model Requests (5 explicit issues)
| Issue | Model Requested |
|-------|----------------|
| #226 | GGUF support |
| #172 | VibeVoice |
| #138 | Export to ONNX/Piper format |
| #132 | LavaSR (transcription) |
| #76 | (General model expansion) |
Community also requests: XTTS-v2, Fish Speech, Kokoro. CosyVoice was tried and abandoned. The multi-engine architecture is in place, making new model integration straightforward.
### Long-Form / Chunking (5 issues)
Users hitting the ~500 character practical limit.
**Key issues:** #234 (queue system), #203 (500 char limit), #191 (auto-split), #111, #69
**Fix path:** **Mostly resolved.** PR #266 (engine-agnostic chunked TTS) and PR #269 (async generation queue) are both merged. PR #154 (Audiobook tab) is still open.
### Feature Requests (23 issues)
Notable requests:
- **#234** — Queue system for batch generation
- **#182** — Concurrent/multi-thread generation
- **#173** — Vocal intonation/inflection control
- **#165** — Audiobook mode
- **#144** — Copy text to clipboard
- **#184** — Cancel button for progress bar
- **#242** — Seed value pinning for consistency
- **#228** — Always use 0.6B option
- **#233** — Transcribe audio API improvements
- **#235** — Finetuned Qwen3-TTS tokenizer
### Bugs (19 issues)
| Category | Issues |
|----------|--------|
| Generation failures | #248 (broken pipe), #219 (unsupported scalarType), #202 (clipping error), #170 (load failed) |
| UI bugs | #231 (history not updating), #190 (mobile landing), #169 (blank interface) |
| File operations | #207 (transcribe file error), #168 (no such file), #142 (download audio fail) |
| Server lifecycle | #166 (server processes remain), #164 (no auto-update) |
| Database | #174 (sqlite3 IntegrityError) |
| Dependency | #131 (numpy ABI mismatch), #209 (import error) |
---
## Existing Plan Documents — Status
| Document | Target Version | Status | Relevance |
|----------|---------------|--------|-----------|
| `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially superseded** by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
| `CUDA_BACKEND_SWAP.md` | — | **Shipped** (PR #252) | CUDA binary download + backend restart |
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
---
## New Model Integration — Landscape
### Models Worth Supporting (2026 SOTA — updated March 18)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct Support | Integration Ease | Status |
|-------|---------|-------|-------------|-----------|------|-----------------|-----------------|--------|
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None (Base); Yes (CustomVoice variant, predefined speakers only) | **Shipped** | v0.1.13 |
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | **Shipped** | PR #254 |
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` float | **Shipped** | PR #257 |
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags only | **Shipped** | PR #258 |
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), Multilingual (3B) | Medium | Partial — automatic prosody | **Shipped** | PR #296 |
| **Kokoro-82M** | Pre-built voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | **In progress** | Apache 2.0, pip install, ~350MB |
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Yes — `inference_instruct2()` | **Abandoned** | PR #311 — poor output quality |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — inline text descriptions, word-level control | Ready | Needs license clarification |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref audio only | Ready | Mature pip package |
| **Pocket TTS** | Zero-shot + streaming | >1x RT on CPU | — | English | ~100M params, CPU-first | None | Ready | MIT, Kyutai Labs |
| **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | **Yes** — text prompts for style + timbre design | Needs vetting | Apache 2.0 |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Partial — automatic context-aware prosody | Needs vetting | Apache 2.0 |
#### Notes on Candidates (March 2026)
- **CosyVoice2-0.5B** — **Tried and abandoned** (PR #311). Despite having the best instruct API, output quality was poor. No PyPI package, needed 5+ shims, heavy deps. Not worth it.
- **HumeAI TADA** — **Shipped** (PR #296). 700+ seconds coherent audio. [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **Kokoro-82M** — **In progress.** 82M params, CPU realtime, Apache 2.0, clean `pip install kokoro`. Uses pre-built voice styles (not zero-shot cloning from arbitrary audio). [GitHub: hexgrad/kokoro](https://github.com/hexgrad/kokoro)
- **Fish Speech** — Word-level fine-grained control. License needs clarification. [fish.audio blog](https://fish.audio/blog/fish-audio-s2-fine-grained-ai-voice-control-at-the-word-level)
- **XTTS-v2** — Coqui's multilingual cloning. 17+ languages, pip-installable. [GitHub: coqui-ai/TTS](https://github.com/coqui-ai/TTS)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control)
### Adding a New Engine (Now Straightforward)
With the model config registry and shared `EngineModelSelector` component, adding a new TTS engine requires:
1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
2. **Register in `backend/backends/__init__.py`** — add `ModelConfig` entry + `TTS_ENGINES` entry + factory elif
3. **Update `backend/models.py`** — add engine name to regex
4. **Update frontend** — add to engine union type, `EngineModelSelector` options, form schema, language map (4 files)
`main.py` requires **zero changes** — the registry handles all dispatch automatically.
Total effort: **~1 day** for a well-documented model with a PyPI package. See `docs/plans/ADDING_TTS_ENGINES.md` for the full guide.
---
## Architectural Bottlenecks
### ~~1. Single Backend Singleton~~ — RESOLVED
The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
### ~~2. `main.py` Dispatch Point Duplication~~ — RESOLVED
Previously, each engine required updates to 6+ hardcoded dispatch maps across `main.py` (~320 lines of if/elif chains). A model config registry in `backend/backends/__init__.py` now centralizes all model metadata (`ModelConfig` dataclass) with helper functions (`load_engine_model()`, `check_model_loaded()`, `engine_needs_trim()`, etc.). Adding a new engine requires zero changes to `main.py`.
### ~~3. Model Config is Scattered~~ — RESOLVED
Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the `ModelConfig` registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in `EngineModelSelector.tsx`.
### 4. Voice Prompt Cache Assumes PyTorch Tensors
`backend/utils/cache.py` uses `torch.save()` / `torch.load()`. LuxTTS and Chatterbox backends work around this by storing reference audio paths instead of tensors in their voice prompt dicts. Not ideal but functional.
### 5. ~~Frontend Assumes Qwen Model Sizes~~ — RESOLVED
The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen.
---
## Recommended Priorities
### Tier 1 — Ship Now
| Priority | PR/Item | Impact | Effort |
|----------|---------|--------|--------|
| 1 | **Kokoro 82M** — finish integration | New engine, CPU-friendly, 8 langs | Low (nearly done) |
| 2 | Close PR #311 (CosyVoice) and #237 (superseded by #305) | Housekeeping | None |
| 3 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 4 | **#253** — 48kHz speech tokenizer | Quality improvement for Qwen | Medium |
### Tier 2 — Feature Work
| Priority | Item | Impact | Effort |
|----------|------|--------|--------|
| 1 | **#154** — Audiobook tab | Long-form users. Chunking + queue now shipped. | Medium |
| 2 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
| 3 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable | Low |
| 4 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
| 5 | Streaming for non-MLX engines | Currently MLX-only | Medium |
### Tier 3 — Future Engines
| Priority | Item | Notes |
|----------|------|-------|
| 1 | **Fish Speech** | 50+ langs, word-level instruct. License TBD. |
| 2 | **XTTS-v2** | 17+ langs, mature pip package. Best multilingual cloning. |
| 3 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. |
| 4 | **MOSS-TTS** | Text-to-voice design. Multi-speaker dialogue for Stories. |
| 5 | **VoxCPM 1.5** | Tokenizer-free streaming. Uncertain integration surface. |
### ~~Previously Prioritized — Now Done~~
- ~~#258 — Chatterbox Turbo~~ **Merged**
- ~~#99 — Chunked TTS~~ **Superseded by #266, merged**
- ~~#88 — CORS restriction~~ **Merged**
- ~~#161 — Docker deployment~~ **Merged**
- ~~#234 — Queue system~~ **Addressed by #269, merged**
- ~~HumeAI TADA~~ **Shipped** (PR #296)
- ~~Kokoro-82M~~ **In progress**
---
## Branch Inventory
| Branch | PR | Status | Notes |
|--------|-----|--------|-------|
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
---
## Quick Reference: API Endpoints
<details>
<summary>All current endpoints</summary>
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/health` | GET | Health check, model/GPU status |
| `/profiles` | POST, GET | Create/list voice profiles |
| `/profiles/{id}` | GET, PUT, DELETE | Profile CRUD |
| `/profiles/{id}/samples` | POST, GET | Add/list voice samples |
| `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management |
| `/profiles/{id}/export` | GET | Export profile as ZIP |
| `/profiles/import` | POST | Import profile from ZIP |
| `/generate` | POST | Generate speech (engine param selects TTS backend) |
| `/generate/stream` | POST | Stream speech (MLX only) |
| `/history` | GET | List generation history |
| `/history/{id}` | GET, DELETE | Get/delete generation |
| `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
| `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model |
| `/models/load` | POST | Load model into memory |
| `/models/unload` | POST | Unload model |
| `/models/progress/{name}` | GET | SSE download progress |
| `/tasks/active` | GET | Active downloads/generations (with inline progress) |
| `/stories` | POST, GET | Create/list stories |
| `/stories/{id}` | GET, PUT, DELETE | Story CRUD |
| `/stories/{id}/items` | POST, GET | Story items CRUD |
| `/stories/{id}/export` | GET | Export story audio |
| `/channels` | POST, GET | Audio channel CRUD |
| `/channels/{id}` | PUT, DELETE | Channel update/delete |
| `/cache/clear` | POST | Clear voice prompt cache |
| `/server/cuda/status` | GET | CUDA binary availability |
| `/server/cuda/download` | POST | Download CUDA binary |
| `/server/cuda/switch` | POST | Switch to CUDA backend |
</details>
-163
View File
@@ -1,163 +0,0 @@
# Voicebox v0.2.0 -- Release Notes
## The story
Voicebox v0.1.x shipped as a single-engine voice cloning app built around Qwen3-TTS. It worked, but it was limited: one model family, 10 languages, English-centric emotion, a synchronous generation pipeline that locked the UI, and a hard ceiling on how much text you could generate at once.
v0.2.0 is a ground-up rethink. Voicebox is now a **multi-engine voice cloning platform**. Four TTS engines. 23 languages. Expressive paralinguistic controls. A full post-processing effects pipeline. Unlimited generation length. Asynchronous everything. And it runs on every major GPU vendor -- NVIDIA, AMD, Intel Arc, Apple Silicon -- plus Docker for headless deployment.
This is the release where Voicebox stops being a proof of concept and starts being a real tool.
---
## Major New Features
### Multi-Engine Architecture
Voicebox now supports **four TTS engines**, each with different strengths. Switch between them per-generation from a single unified interface:
| Engine | Languages | Strengths |
|--------|-----------|-----------|
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions ("speak slowly", "whisper") |
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
| **Chatterbox Multilingual** | 23 | Broadest language coverage -- Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
### Emotions and Paralinguistic Tags (Chatterbox Turbo)
Type `/` in the text input to open an autocomplete for **9 expressive tags** that the model synthesizes inline with speech:
`[laugh]` `[chuckle]` `[gasp]` `[cough]` `[sigh]` `[groan]` `[sniff]` `[shush]` `[clear throat]`
Tags render as inline badges in a rich text editor and serialize cleanly to the API. This makes generated speech sound natural and expressive in a way that plain TTS can't.
### 23 Languages via Chatterbox Multilingual
The Chatterbox Multilingual engine brings zero-shot voice cloning to **23 languages**: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, and Turkish. The language dropdown dynamically filters to show only languages supported by the selected engine.
### Unlimited Generation Length (Auto-Chunking)
Previously, long text would hit model context limits and degrade. Now, text is **automatically split at sentence boundaries** and each chunk is generated independently, then crossfaded back together. This is fully engine-agnostic and works with all four engines.
- **Auto-chunking limit slider** (100-5,000 chars, default 800) -- controls when text gets split
- **Crossfade slider** (0-200ms, default 50ms) -- blends chunk boundaries smoothly, or set to 0 for a hard cut
- **Max text length raised to 50,000 characters** -- generate entire scripts, chapters, or articles in one go
- Smart splitting respects abbreviations (Dr., e.g., a.m.), CJK punctuation, and never breaks inside paralinguistic `[tags]`
### Asynchronous Generation Queue
Generation is now fully **non-blocking**. Submit a generation and immediately start typing the next one -- no more frozen UI waiting for inference to complete.
- Serial execution queue prevents GPU contention across all backends
- Real-time SSE status streaming (`generating` -> `completed` / `failed`)
- Failed generations can be retried without re-entering text
- Stale generations from crashes are auto-recovered on startup
- Generating status pill shown inline in the story editor
### Post-Processing Effects Pipeline
A full audio effects system powered by Spotify's `pedalboard` library. Apply effects after generation, preview them in real time, and build reusable presets -- all without leaving the app.
**8 effects available:**
| Effect | What it does |
|--------|-------------|
| **Pitch Shift** | Shift pitch up or down by up to 12 semitones |
| **Reverb** | Room reverb with configurable size, damping, and wet/dry mix |
| **Delay** | Echo with adjustable delay time, feedback, and mix |
| **Chorus / Flanger** | Modulated delay -- short for metallic flanger, longer for lush chorus |
| **Compressor** | Dynamic range compression with threshold, ratio, attack, and release |
| **Gain** | Volume adjustment from -40 to +40 dB |
| **High-Pass Filter** | Remove low frequencies below a configurable cutoff |
| **Low-Pass Filter** | Remove high frequencies above a configurable cutoff |
**Effects presets** -- Four built-in presets ship out of the box (Robotic, Radio, Echo Chamber, Deep Voice), and you can create unlimited custom presets. Presets are drag-and-drop chains of effects with per-parameter sliders.
**Per-profile default effects** -- Assign an effects chain to a voice profile and it applies automatically to every generation with that voice. Override per-generation from the generate box.
**Live preview** -- Audition any effects chain against an existing generation before committing. The preview streams processed audio without saving anything.
### Generation Versions
Every generation now supports **multiple versions** with full provenance tracking:
- **Original** -- the clean, unprocessed TTS output (always preserved)
- **Effects versions** -- apply different effects chains to create new versions from any source version
- **Takes** -- regenerate with the same text and voice but a new seed for variation
- **Source tracking** -- each version records which version it was derived from
- **Version pinning in stories** -- pin a specific version to a track clip in the story editor, independent of the generation's default
- **Favorites** -- star generations to mark them for quick access
---
## New Platform Support
### Linux (Native)
Full Linux support with `.deb` and `.rpm` packages. Includes PulseAudio/PipeWire audio capture for voice sample recording.
### AMD ROCm GPU Acceleration
AMD GPU users now get hardware-accelerated inference via ROCm, with automatic `HSA_OVERRIDE_GFX_VERSION` configuration for GPUs not officially in the ROCm compatibility list (e.g., RX 6600).
### NVIDIA CUDA Backend Swap
The CPU-only release can download and swap in a CUDA-accelerated backend binary from within the app -- no reinstall required. Handles GitHub's 2GB asset limit by downloading split parts and verifying SHA-256 checksums.
### Intel Arc (XPU) and DirectML
PyTorch backend also supports Intel Arc GPUs via IPEX/XPU and Windows any-GPU via DirectML.
### Docker + Web Deployment
Run Voicebox headless as a Docker container with the full web UI:
```bash
docker compose up
```
3-stage build, non-root runtime, health checks, persistent model cache across rebuilds. Binds to localhost only by default.
---
## Model Management
- **Per-model unload** -- free GPU memory without deleting downloaded models
- **Custom models directory** -- set `VOICEBOX_MODELS_DIR` to store models anywhere
- **Model folder migration** -- move all models to a new location with progress tracking
- **Whisper Turbo** -- added `openai/whisper-large-v3-turbo` as a transcription model option
- **Download cancel/clear UI** -- cancel in-progress downloads, VS Code-style problems panel for errors
---
## Security
- **CORS hardening** -- replaced wildcard `*` with an explicit allowlist of local origins; extensible via `VOICEBOX_CORS_ORIGINS` env var
- **Network access toggle** -- fully disable outbound network requests for air-gapped deployments
## Accessibility
- Comprehensive screen reader support (tested with NVDA/Narrator) across all major UI surfaces
- Keyboard navigation for voice cards, history rows, model management, and story editor
- State-aware `aria-label` attributes on all interactive controls
## Reliability
- **Atomic audio saves** -- two-phase write prevents corrupted files on crash/interrupt
- **Filesystem health endpoint** -- proactive disk space and directory writability checks
- **Errno-specific error messages** -- clear feedback for permission denied, disk full, missing directory
## UX Polish
- Responsive layout with horizontal-scroll voice cards on mobile
- App version shown in sidebar
- Voice card heights normalized
- Audio player title hidden at narrow widths to prevent overflow
---
## Installation
| Platform | Download |
|----------|----------|
| **macOS (Apple Silicon)** | `Voicebox_0.2.0_aarch64.dmg` |
| **macOS (Intel)** | `Voicebox_0.2.0_x64.dmg` |
| **Windows** | `Voicebox_0.2.0_x64_en-US.msi` or `x64-setup.exe` |
| **Linux** | `.deb` / `.rpm` packages |
| **Docker** | `docker compose up` |
The app includes automatic updates -- future patches will be installed automatically.
---
## Video Script Beats
For the marketing video, focus on these six beats:
1. **"Four engines, one app"** -- show the engine dropdown switching between Qwen, LuxTTS, Chatterbox, and Turbo
2. **"23 languages"** -- generate the same voice clone in Arabic, Japanese, Hindi, etc.
3. **"Make it expressive"** -- type `/laugh` and `/sigh` with Chatterbox Turbo, play back the result
4. **"Shape your sound"** -- apply the Robotic or Deep Voice preset, preview it live, then build a custom effects chain with drag-and-drop
5. **"No limits"** -- paste a long script, show it auto-chunk and generate seamlessly
6. **"Queue and go"** -- fire off multiple generations back-to-back without waiting
-67
View File
@@ -1,67 +0,0 @@
# Voicebox Issue Pain Points (Snapshot)
## Scope
- Dataset: **128 total issues** (**107 open**, **21 closed**)
- Source: GitHub issues in `jamiepine/voicebox`
- Classification: keyword/theme clustering
- Note: counts below are **non-exclusive** (one issue can belong to multiple pain points)
## Most Common Pain Points (Open Issues)
| Rank | Pain Point | Open Issues | What users are reporting |
|---|---|---:|---|
| 1 | Model download & offline reliability | **32** | Downloads failing/stalling, cache/offline behavior inconsistent, wrong model size selected, Errno issues |
| 2 | GPU/backend compatibility | **22** | GPU not detected, backend fallback surprises, platform-specific runtime failures (Windows/Mac) |
| 3 | Export/save/file persistence | **15** | Export fails, "failed to fetch/download audio", samples/profiles not saving |
| 4 | Language/accent quality & coverage | **14** | Missing language support, accent mismatch, robotic outputs |
| 5 | Update/restart safety + long-op controls | **4** | Auto-restart without warning, update confusion, lack of cancel/pause controls |
## Representative Issues by Pain Point
### 1) Model download & offline reliability (32)
- [#159](https://github.com/jamiepine/voicebox/issues/159) - Qwen download fails with Errno 22
- [#151](https://github.com/jamiepine/voicebox/issues/151) - Model loading hangs / server crashes
- [#150](https://github.com/jamiepine/voicebox/issues/150) - Internet required despite downloaded models
- [#149](https://github.com/jamiepine/voicebox/issues/149) - Cancel/pause controls for large downloads
- [#96](https://github.com/jamiepine/voicebox/issues/96) - 0.6B selection still uses/downloads 1.7B
### 2) GPU/backend compatibility (22)
- [#164](https://github.com/jamiepine/voicebox/issues/164) - Windows: no GPU usage + multiple breakages
- [#141](https://github.com/jamiepine/voicebox/issues/141) - Using CPU only, GPU not used
- [#131](https://github.com/jamiepine/voicebox/issues/131) - Numpy ABI mismatch in bundled app
- [#130](https://github.com/jamiepine/voicebox/issues/130) - Intel Mac tensor/padding generation error
- [#127](https://github.com/jamiepine/voicebox/issues/127) - GPU not found
### 3) Export/save/file persistence (15)
- [#148](https://github.com/jamiepine/voicebox/issues/148) - Japanese export fails on 0.1.12
- [#143](https://github.com/jamiepine/voicebox/issues/143) - Samples not saving
- [#134](https://github.com/jamiepine/voicebox/issues/134) - Can't save profile
- [#105](https://github.com/jamiepine/voicebox/issues/105) - Export audio fails (failed to fetch)
- [#49](https://github.com/jamiepine/voicebox/issues/49) - Export filename/location ignored on Windows
### 4) Language/accent quality & coverage (14)
- [#162](https://github.com/jamiepine/voicebox/issues/162) - Persian audio request/problem
- [#117](https://github.com/jamiepine/voicebox/issues/117) - Arabic language support
- [#113](https://github.com/jamiepine/voicebox/issues/113) - Polish language support
- [#109](https://github.com/jamiepine/voicebox/issues/109) - Ukrainian support
- [#100](https://github.com/jamiepine/voicebox/issues/100) - Non-US accent quality issues
### 5) Update/restart safety + controls (4)
- [#164](https://github.com/jamiepine/voicebox/issues/164) - Update behavior + usability failures
- [#136](https://github.com/jamiepine/voicebox/issues/136) - Auto-restart without warning
- [#86](https://github.com/jamiepine/voicebox/issues/86) - Unexpected restart with no confirmation
- [#149](https://github.com/jamiepine/voicebox/issues/149) - Need pause/cancel and pre-download confirmation
## Additional Signal
- There is also a large **feature-request/misc** bucket (**36 open**) that is competing with stability triage (audiobook, Linux build, additional ASR/TTS models, integrations).
## Takeaway
Most user pain is concentrated in four stability areas: **download/offline path**, **GPU/backend detection**, **save/export reliability**, and **language/accent correctness**. Addressing those first should reduce the majority of current support friction.
-428
View File
@@ -1,428 +0,0 @@
# Voicebox API Refactor Plan
Date: 2026-03-19
Status: Proposed
Scope: Backend HTTP API structure, schemas, docs, and compatibility strategy
## Goals
- Make the API easier to understand and automate against.
- Improve endpoint consistency without breaking the desktop app or existing local integrations.
- Align generated docs and checked-in OpenAPI artifacts with the actual backend.
- Separate app-facing resources from internal or operational actions.
- Create a migration path toward a cleaner `v2` resource model while preserving `v1` routes during transition.
## Non-Goals
- Rewriting backend business logic or generation internals.
- Introducing authentication for all deployment modes in the first pass.
- Changing storage models or database schema unless required for API correctness.
- Removing current routes immediately.
## Current Pain Points
- Mixed endpoint styles: resource-oriented (`/profiles`) and command-oriented (`/generate`, `/tasks/clear`) coexist.
- Related generation resources are split across multiple namespaces: `/generate`, `/history`, `/audio`, `/effects`, and `/generations/.../versions`.
- Response payloads vary widely: typed models, raw dicts with `message`, booleans, and `HTTPException(detail=...)` payloads.
- Some async flows use exception-shaped `202` responses instead of first-class task contracts.
- Checked-in OpenAPI output can drift from actual backend models.
- Operational endpoints such as `/shutdown` are exposed in the same surface as user workflows.
## Guiding Principles
1. Prefer additive changes before destructive changes.
2. Keep `v1` behavior working until the app and docs fully migrate.
3. Add compatibility shims close to the routing layer, not deep in services.
4. Treat OpenAPI as a release artifact that must be kept in sync.
5. Standardize public contracts before renaming everything.
## Target API Shape
This is the intended end state, not the immediate first milestone.
### Core Resources
- `/profiles`
- `/profiles/{profile_id}/samples`
- `/profiles/{profile_id}/avatar`
- `/profiles/{profile_id}/effects`
- `/generations`
- `/generations/{generation_id}`
- `/generations/{generation_id}/status`
- `/generations/{generation_id}/audio`
- `/generations/{generation_id}/versions`
- `/generations/{generation_id}/versions/{version_id}`
- `/generations/{generation_id}/versions/{version_id}/audio`
- `/stories`
- `/stories/{story_id}/items`
- `/effects/presets`
- `/models`
- `/models/{model_name}`
- `/tasks`
### Operational or Internal Endpoints
Move under an explicit namespace and disable where appropriate:
- `/admin/shutdown`
- `/admin/watchdog/disable`
- `/admin/cache/clear`
- `/admin/tasks/clear`
### Response Contract Direction
- Resource reads and writes return typed resource models.
- Delete and action endpoints return small typed action result models.
- Errors use a consistent structure.
- Async actions return explicit task metadata instead of overloading `detail`.
## Migration Strategy Overview
The refactor is split into six phases. Phases 1-3 are the highest impact and safest to ship first.
| Phase | Focus | Est. Duration | Risk | Backward Compatibility |
| --- | --- | --- | --- | --- |
| 1 | Documentation and contract correctness | 2-3 days | Low | Full |
| 2 | Response and error consistency | 3-5 days | Low-Medium | Full |
| 3 | Router structure and internal organization | 3-4 days | Low | Full |
| 4 | Additive `v2` resource endpoints | 1-2 weeks | Medium | Full |
| 5 | Client migration and deprecation rollout | 1 week | Medium | Full during rollout |
| 6 | Cleanup and optional removals | 1-2 releases | Medium-High | Partial after notice |
## Phase 1: Fix Contract Drift First
Priority: Highest
Outcome: The documented API matches the running backend.
### Problems Addressed
- `docs/openapi.json` can become stale.
- Generated API reference pages may describe outdated request bodies.
- App metadata still frames the backend too narrowly.
### Implementation Steps
1. Update FastAPI app metadata in `backend/app.py`.
- Replace the old Qwen-specific description with a multi-engine Voicebox API description.
- Add tags metadata for major domains if desired.
2. Regenerate OpenAPI from the running app using the existing docs script flow.
3. Compare `backend/models.py` to the checked-in schema.
- Verify `GenerationRequest`, effects endpoints, stories endpoints, and model endpoints.
4. Regenerate or refresh API reference pages under `docs/content/docs/api-reference/`.
5. Add a CI check that fails if `docs/openapi.json` is out of date.
6. Add a short maintainer note describing when schema regeneration is required.
### Backward Compatibility
- No route changes.
- No payload changes.
- Safe to release immediately.
### Success Criteria
- `docs/openapi.json` matches the live app.
- Generated docs include all currently supported generate parameters.
- No frontend code changes required.
## Phase 2: Standardize Responses and Errors
Priority: High
Outcome: Clients can handle responses predictably.
### Problems Addressed
- Delete endpoints return ad hoc message dicts.
- Toggle endpoints return special one-off payloads.
- `202` async responses are encoded as `HTTPException(detail=...)` in some places.
### Implementation Steps
1. Add shared response models in `backend/models.py`.
- `ActionResult`
- `DeleteResult`
- `ToggleFavoriteResponse`
- `AcceptedTaskResponse`
- `ApiError`
2. Convert routes that currently return raw dicts to explicit `response_model`s.
- `DELETE /profiles/{profile_id}`
- `DELETE /history/{generation_id}`
- `DELETE /stories/{story_id}`
- `POST /tasks/clear`
- `POST /cache/clear`
- similar endpoints across routes
3. Replace exception-shaped `202` responses in `transcription.py` with an explicit accepted response body.
- Return `JSONResponse(status_code=202, content=...)` or typed FastAPI response model.
4. Add a global exception handler for known API errors if helpful.
- Normalize `ValueError` to `400` with a consistent error body.
- Preserve FastAPI validation errors for now, or wrap them in a consistent top-level shape in a later pass.
5. Document the stable error contract in the docs.
### Migration Strategy
- Keep field names inside successful payloads compatible where possible.
- For existing dict responses, preserve the current keys while introducing typed models with the same shape.
- For `202` flows, support both old and new client handling for one release if needed.
### Timeline Estimate
- 3-5 engineering days including tests and docs refresh.
### Success Criteria
- All mutation endpoints declare response models.
- Clients can programmatically distinguish success, accepted, and error cases without special casing `detail` payloads.
## Phase 3: Normalize Router Structure Internally
Priority: High
Outcome: The backend becomes easier to maintain before public path changes begin.
### Problems Addressed
- Route files hardcode full paths and are all mounted at root.
- There is no consistent use of router prefixes or tags.
- Route grouping in code does not cleanly express the public API shape.
### Implementation Steps
1. Add prefixes and tags to routers.
- `profiles`: `prefix="/profiles"`
- `generations`: `prefix="/generate"` for now or split additive aliases carefully
- `history`: `prefix="/history"`
- `effects`: `prefix="/effects"`
- and so on
2. Convert route declarations to relative paths within each router.
3. Introduce a small route compatibility layer for routes that are likely to move later.
- Example: helper functions that can be mounted under both old and new paths.
4. Add explicit route tags so Swagger/OpenAPI groups are coherent.
5. Document the intended public ownership of each namespace.
### Backward Compatibility
- No public path changes yet if existing paths are preserved through prefixes and aliases.
- Mostly internal refactoring.
### Timeline Estimate
- 3-4 engineering days.
### Success Criteria
- All route modules use prefixes and tags.
- Route registration in `backend/routes/__init__.py` becomes simpler.
- OpenAPI groups read cleanly by domain.
## Phase 4: Introduce Additive `v2` Resource Endpoints
Priority: High
Outcome: A cleaner API exists without breaking the current one.
### Problems Addressed
- Generation-related resources are fragmented.
- Sample and audio endpoints are not consistently modeled as resources.
- Command-style naming makes the API harder to reason about.
### New Endpoints to Add
These should be introduced alongside current endpoints, not as replacements.
- `POST /generations` -> alias for current `/generate`
- `GET /generations` -> alias for current `/history`
- `GET /generations/{id}` -> alias for current `/history/{id}`
- `POST /generations/{id}/retry` -> alias for current `/generate/{id}/retry`
- `POST /generations/{id}/regenerate` -> alias for current `/generate/{id}/regenerate`
- `GET /generations/{id}/status` -> alias for current `/generate/{id}/status`
- `POST /generations/stream` -> alias for current `/generate/stream`
- `GET /generations/{id}/audio` -> alias for current `/audio/{generation_id}`
- `GET /generations/{id}/export` -> alias for current `/history/{generation_id}/export`
- `GET /generations/{id}/export-audio` -> alias for current `/history/{generation_id}/export-audio`
- `GET /profiles/{profile_id}/samples/{sample_id}` or `GET /samples/{sample_id}` as a consciously chosen model
- `PUT /profiles/{profile_id}/samples/{sample_id}` -> alias for current sample update route
- `DELETE /profiles/{profile_id}/samples/{sample_id}` -> alias for current sample delete route
### Implementation Steps
1. Create new handler entry points that call the existing service functions.
2. Keep old handlers in place, but mark them deprecated in OpenAPI.
3. Add `summary` and `description` text clarifying preferred routes.
4. Update frontend and docs examples to use new endpoints first.
5. Add tests proving both old and new paths return equivalent responses.
### Migration Strategy
- Old paths remain functional for at least one stable release cycle.
- New docs and client examples use `v2-style` resource routes immediately.
- Include deprecation headers where feasible, for example:
- `Deprecation: true`
- `Sunset: <date>`
- `Link: <new-doc-url>; rel="successor-version"`
### Timeline Estimate
- 1-2 weeks depending on test coverage and frontend updates.
### Success Criteria
- All major generation workflows are accessible through resource-oriented routes.
- Old routes still work unchanged.
## Phase 5: Migrate First-Party Clients and Publish Deprecations
Priority: Medium
Outcome: Voicebox itself stops depending on legacy paths.
### Problems Addressed
- The desktop app and docs may continue to reinforce old route shapes.
- Third-party consumers need a visible migration path.
### Implementation Steps
1. Update `app/src/lib/api/client.ts` to use the new preferred endpoints.
2. Regenerate or refresh any generated API clients.
3. Update docs examples, tutorials, and code snippets to use preferred routes only.
4. Add a changelog entry describing the migration path.
5. Add runtime deprecation logging for legacy route usage in development mode.
6. If feasible, expose a small `/health` or `/meta` field showing API version and deprecation window.
### Migration Strategy
- Keep old endpoints available but clearly documented as legacy.
- Publish a mapping table from old route to new route.
- Do not change request or response payloads during the same phase unless necessary.
### Timeline Estimate
- About 1 week including docs and app verification.
### Success Criteria
- First-party app no longer depends on legacy route names.
- Docs do not advertise deprecated paths as the primary interface.
## Phase 6: Cleanup, Namespace Hardening, and Optional Breaking Changes
Priority: Medium
Outcome: The API surface is cleaner and safer for remote or Docker use.
### Problems Addressed
- Internal/admin endpoints are mixed into the public API.
- Legacy aliases increase maintenance cost forever if never retired.
### Implementation Steps
1. Move operational endpoints under `/admin` or `/internal`.
- `/shutdown`
- `/watchdog/disable`
- `/tasks/clear`
- `/cache/clear`
2. Gate these endpoints behind configuration for non-local deployments.
- Example: `VOICEBOX_ENABLE_ADMIN_API=true`
3. Decide whether to remove or keep legacy aliases.
- If removing, do so only after a published deprecation window.
4. Remove deprecated docs pages and old examples.
5. Tighten route-level tests to prevent accidental reintroduction of legacy patterns.
### Migration Strategy
- For desktop-only local use, aliases may remain indefinitely if removal cost outweighs benefit.
- For published remote API guidance, hide admin endpoints from default docs even if they still exist.
### Timeline Estimate
- 1-2 releases after the additive migration is complete.
### Success Criteria
- Public docs expose a coherent resource API.
- Operational endpoints are clearly separate or disabled in remote contexts.
## Cross-Cutting Work Items
These should happen throughout the migration, not only in a single phase.
### Testing
- Add route equivalence tests for old and new paths.
- Add schema snapshot tests for OpenAPI generation.
- Add response-shape tests for common mutations and async workflows.
- Add contract tests for `202 Accepted` flows.
### Documentation
- Maintain an old-to-new endpoint mapping table.
- Add per-endpoint examples for create profile, generate, apply effects, transcribe, and stories operations.
- Explicitly document which endpoints are app-facing vs admin-facing.
### Observability
- Add warning logs when deprecated endpoints are used.
- Track usage counts in development or optional telemetry-free local logs.
### Release Management
- Mention API changes in `CHANGELOG.md`.
- Ensure docs and app updates ship in the same release as new preferred routes.
## Recommended Execution Order
If engineering time is limited, implement in this exact order:
1. Fix OpenAPI and docs drift.
2. Standardize response models and accepted-task responses.
3. Add router prefixes and tags internally.
4. Add `/generations` aliases and sample path aliases.
5. Migrate the first-party app to preferred routes.
6. Deprecate or hide legacy/admin routes.
## Old-to-New Route Mapping
| Current Route | Preferred Route |
| --- | --- |
| `POST /generate` | `POST /generations` |
| `POST /generate/stream` | `POST /generations/stream` |
| `POST /generate/{id}/retry` | `POST /generations/{id}/retry` |
| `POST /generate/{id}/regenerate` | `POST /generations/{id}/regenerate` |
| `GET /generate/{id}/status` | `GET /generations/{id}/status` |
| `GET /history` | `GET /generations` |
| `GET /history/{id}` | `GET /generations/{id}` |
| `GET /audio/{id}` | `GET /generations/{id}/audio` |
| `GET /history/{id}/export` | `GET /generations/{id}/export` |
| `GET /history/{id}/export-audio` | `GET /generations/{id}/export-audio` |
| `PUT /profiles/samples/{sample_id}` | `PUT /profiles/{profile_id}/samples/{sample_id}` |
| `DELETE /profiles/samples/{sample_id}` | `DELETE /profiles/{profile_id}/samples/{sample_id}` |
| `POST /tasks/clear` | `POST /admin/tasks/clear` |
| `POST /cache/clear` | `POST /admin/cache/clear` |
| `POST /shutdown` | `POST /admin/shutdown` |
| `POST /watchdog/disable` | `POST /admin/watchdog/disable` |
## Risks and Mitigations
### Risk: App regressions during endpoint migration
- Mitigation: Add new routes before changing client usage.
- Mitigation: Keep payloads identical while paths change.
### Risk: Docs still drift after cleanup
- Mitigation: Add CI enforcement and a release checklist step.
### Risk: Third-party local scripts break on removal
- Mitigation: Prefer indefinite aliases for one-person local workflows unless maintenance becomes painful.
### Risk: Admin endpoints remain dangerous in remote mode
- Mitigation: Hide and gate them before promoting remote deployment more broadly.
## Definition of Done
The refactor can be considered complete when all of the following are true:
- OpenAPI, checked-in docs, and backend models match.
- The preferred public API is resource-oriented and documented consistently.
- The Voicebox app uses preferred routes exclusively.
- Legacy routes are either deprecated with a timeline or intentionally retained as compatibility aliases.
- Operational endpoints are clearly separated from the public app API.
-173
View File
@@ -1,173 +0,0 @@
# CUDA Libs as a Bolt-On Addon
## Problem
Every time we bump `__version__` (even for a UI tweak or bugfix), the exact-match version check in both `main.rs:222` and `cuda.py:237` invalidates the user's ~2.4GB CUDA binary, forcing a full redownload. The CUDA binary is the entire server rebuilt with NVIDIA libs included -- there's no separation between app logic and the CUDA runtime.
## Why This Is Hard With `--onefile`
The core tension is PyInstaller `--onefile` mode (`build_binary.py:39`). In onefile mode, everything -- Python code, all dependencies, torch, the NVIDIA `.dll`/`.so` files -- gets packed into a single self-extracting archive. There's no concept of "swap out one part." The binary IS the server.
## Options
### Option A: Switch to `--onedir` for the CUDA Build (Recommended)
Instead of `--onefile`, build the CUDA variant as a directory (a folder with the exe + all the shared libs alongside it). Then split the distribution into two archives:
1. **`voicebox-server-cuda` executable + non-NVIDIA deps** (~200-400MB) -- versioned with the app, redownloaded on every app update.
2. **`cuda-libs-cu126.tar.gz`** (~2GB) -- the `nvidia.*` packages (cublas, cudnn, cuda_runtime, etc.), versioned independently (e.g., `cuda-libs-cu126-v1`). Only redownloaded when we bump the CUDA toolkit version or torch's CUDA dependency changes.
#### How it would work at runtime
- Tauri downloads the server binary archive and extracts it to `{data_dir}/backends/cuda/`
- On first CUDA setup (or when cuda-libs version bumps), downloads and extracts the libs archive into the same directory
- The CUDA server exe finds the `.dll`/`.so` files next to it (standard PyInstaller onedir behavior)
- Version check becomes two checks: server version + cuda-libs version
#### Independent versioning
Add a `cuda-libs.json` manifest:
```json
{"version": "cu126-v1", "torch_compat": ">=2.6.0,<2.8.0"}
```
The server checks this on startup. The Tauri side checks it before launching. Only bump `cu126-v1` -> `cu126-v2` when we actually change the CUDA toolkit or torch major version.
#### Build pipeline changes
The CI `build-cuda-windows` job would build with `--onedir`, then separate the output into two archives. The CUDA libs archive could be built less frequently (only when torch/CUDA version changes) and stored as a pinned release asset.
#### Download experience
- First-time CUDA setup: ~2.4GB total (same as today)
- Subsequent app updates: ~200-400MB for the server, CUDA libs stay cached
- CUDA toolkit bump: ~2GB for just the libs
#### Pros
- PyInstaller `--onedir` natively produces this structure -- NVIDIA DLLs end up as discrete files in the output directory
- The separation is natural: PyInstaller puts torch's NVIDIA deps in predictable paths (`nvidia/cublas/lib/`, etc.)
- CUDA libs are highly stable -- only rebundle when changing CUDA toolkit version (e.g., cu126 -> cu128) or major torch version
- Server updates become ~200-400MB instead of ~2.4GB
- No library path hacking needed -- torch finds NVIDIA DLLs because they're in the same directory tree
#### Cons
- Onedir means a folder with hundreds of files instead of a single exe -- more complex to manage, extract, and clean up
- Need to modify download/assembly logic in `cuda.py` to handle two separate archives
- The Tauri side (`main.rs`) needs to point at an exe inside a directory rather than a standalone binary
- Users who manually manage the file may find the folder structure confusing
#### TTS engine compatibility
No issues. The TTS engines are pure Python + torch. They don't care whether NVIDIA libs are inside the binary or sitting next to it -- torch's dynamic loader finds them either way.
---
### Option B: Keep `--onefile` but Externalize CUDA Libs via Library Path
Keep the server as a single `--onefile` binary (with NVIDIA packages excluded, same as the CPU build). Ship the CUDA libs as a separate download that gets extracted to `{data_dir}/backends/cuda-libs/`. Before launching, set the library search path to include that directory.
**Important caveat:** The CPU torch wheel (`whl/cpu`) doesn't have CUDA kernels compiled in -- it's a fundamentally different build. So the binary would need to be built with CUDA-compiled torch but with the NVIDIA runtime libraries excluded. The runtime libs (cublas, cudnn, etc.) would be provided externally.
#### How it would work
- Build ONE "CUDA-ready" server binary with CUDA-compiled torch but NVIDIA runtime packages excluded
- Ship `cuda-libs-cu126-v1.tar.gz` separately (~2GB of `.dll`/`.so` files)
- When launching, Tauri sets `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux) to include the cuda-libs directory
#### Pros
- Single server binary for both CPU and CUDA users -- simplifies build pipeline enormously
- True bolt-on CUDA libs with fully independent versioning
- Server updates are always small (~150MB for the onefile binary)
#### Cons
- **Fragile on Windows.** PyInstaller `--onefile` extracts to a temp directory at runtime and the internal torch may not find externally-placed NVIDIA libs. DLL resolution on Windows is notoriously unreliable in this scenario.
- `os.add_dll_directory()` only affects `LoadLibraryEx` with `LOAD_LIBRARY_SEARCH_USER_DIRS` flag -- not all DLL loads go through this path
- PyInstaller's onefile bootloader may configure DLL search paths before Python code runs
- Could work on Linux but is fragile on Windows
---
### Option C: Hybrid -- `--onefile` Server + Dynamic CUDA Lib Loading at Runtime
Build the server as `--onefile` with CUDA-compiled torch but with NVIDIA packages excluded. At startup, before torch initializes CUDA, explicitly load the NVIDIA shared libraries using `ctypes.CDLL` or `os.add_dll_directory()`.
In `server.py`, before any torch imports:
```python
cuda_libs_dir = os.environ.get("VOICEBOX_CUDA_LIBS")
if cuda_libs_dir and os.path.isdir(cuda_libs_dir):
if sys.platform == "win32":
os.add_dll_directory(cuda_libs_dir)
os.environ["PATH"] = cuda_libs_dir + os.pathsep + os.environ.get("PATH", "")
else:
os.environ["LD_LIBRARY_PATH"] = cuda_libs_dir + ":" + os.environ.get("LD_LIBRARY_PATH", "")
```
#### Pros
- Single server binary, true bolt-on CUDA libs
- Clean separation of concerns
- Independent versioning
#### Cons
- Needs careful testing with each torch version -- CUDA initialization happens deep in C++ extension layer
- On Windows, `os.add_dll_directory()` may not cover all DLL load paths
- PyInstaller's onefile bootloader may have already configured DLL search paths before Python code runs
- Most complex to get right and maintain
## Recommendation
**Option A (`--onedir` with split archives)** is the most reliable path:
1. **It actually works.** `--onedir` puts all files on disk as regular files. Torch finds NVIDIA DLLs because they're in the same directory tree, exactly as they would be in a normal pip install.
2. **Natural separation.** PyInstaller's `--onedir` output already separates the NVIDIA `.dll`/`.so` files into `nvidia/` subdirectories. We can split the output directory into "core" and "nvidia-libs" archives after building.
3. **Independent versioning is straightforward.** A `cuda-libs.json` manifest controls when redownloads are needed.
4. **Build pipeline simplification.** Build CUDA libs archive less frequently, store as a pinned release asset.
The main cost is managing a directory instead of a single file, but we already have sophisticated download/assembly infrastructure in `cuda.py` with manifests and split parts. Extending that to handle two archives is incremental work.
## Tauri Compatibility (Validated)
Tauri handles PyInstaller `--onedir` with no issues. The key insight is that we're **not** using a static sidecar for CUDA -- we're downloading and extracting at runtime (the existing `cuda.py` + `main.rs` flow). For runtime-launched processes, Tauri's `tauri::shell::Command` supports arbitrary directories natively.
### The critical change in `main.rs`
The only Tauri-side change needed is adding `.current_dir()` when spawning the CUDA backend:
```rust
let cuda_dir = data_dir.join("backends/cuda");
let exe_path = cuda_dir.join("voicebox-server-cuda.exe");
let mut cmd = app.shell().command(exe_path.to_str().unwrap());
cmd = cmd.current_dir(&cuda_dir); // PyInstaller finds all DLLs relative to exe
cmd = cmd.args(["--data-dir", &data_dir_str, "--port", &port_str, "--parent-pid", &parent_pid_str]);
```
`.current_dir()` tells the PyInstaller bootloader that everything (DLLs, `nvidia/cublas/lib/`, `_internal/`, torch extensions, etc.) lives relative to the exe. Torch finds the NVIDIA libs exactly as it does in a normal `pip install` or dev environment -- no `LD_LIBRARY_PATH` hacks, no `os.add_dll_directory` gymnastics.
### Community evidence
- Multiple Tauri users run this exact pattern: Nuitka folders (exe + pythonXX.dll + supporting files), multi-file .NET apps, and PyInstaller onedir backends (GitHub issues #5719, discussion #5206).
- The shell plugin explicitly supports `cwd` in both Rust and JS APIs.
- No reports of torch/CUDA-specific breakage -- the onedir layout is identical to what PyInstaller produces in normal usage.
### Known gotcha: process termination on Windows
PyInstaller onedir creates a parent bootloader + child Python process on Windows. `child.kill()` only hits the outer process in some cases (Tauri issue #11686). Mitigation: keep a reference to the parent PID or use `taskkill /F /T` for clean shutdown. This is not a blocker -- our existing `--parent-pid` watchdog mechanism in `server.py` already handles orphan cleanup.
## Next Steps
1. Prototype: Build the current CUDA binary with `--onedir` and verify torch CUDA works from the output directory
2. Measure the size split: how much is NVIDIA libs vs everything else
3. Design the two-archive download flow and dual version checking
4. Update `cuda.py` for dual-archive extraction (server core + cuda-libs)
5. Update `main.rs`: change launch path to `backends/cuda/` dir + add `.current_dir()`
6. Add `ensure_cuda_structure()` helper in Rust to verify exe + nvidia/ subdirs exist before spawning
7. Update CI pipeline: `build-cuda-windows` produces two archives instead of split parts
8. ~~Update `split_binary.py` or replace with archive-based distribution~~ Done: replaced with `package_cuda.py`
+9
View File
@@ -295,6 +295,15 @@ fix-python: _ensure-venv
test: _ensure-venv
{{ venv_bin }}/python -m pytest {{ backend_dir }}/tests -v
# E2E: generate with every TTS model against the frozen binary (pass extra flags like --only kokoro)
[unix]
test-models *ARGS: _ensure-venv
{{ venv_bin }}/python {{ backend_dir }}/tests/test_all_models_e2e.py {{ ARGS }}
[windows]
test-models *ARGS: _ensure-venv
& "{{ python }}" {{ backend_dir }}/tests/test_all_models_e2e.py {{ ARGS }}
# ─── Database ─────────────────────────────────────────────────────────
# Initialize SQLite database
+3 -2
View File
@@ -1,15 +1,16 @@
{
"name": "@voicebox/landing",
"version": "0.4.0",
"version": "0.4.1",
"description": "Landing page for voicebox.sh",
"scripts": {
"dev": "bun --bun next dev --turbo",
"dev": "next dev --turbo",
"build": "bun --bun next build",
"start": "bun --bun next start",
"lint": "next lint"
},
"dependencies": {
"@fontsource/space-grotesk": "^5.2.10",
"@icons-pack/react-simple-icons": "^13.13.0",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slot": "^1.2.4",
"autoprefixer": "^10.4.17",
Binary file not shown.

After

Width:  |  Height:  |  Size: 148 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 149 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 186 KiB

+16 -27
View File
@@ -1,42 +1,31 @@
import { type NextRequest, NextResponse } from 'next/server';
import { getLatestRelease } from '@/lib/releases';
export const dynamic = 'force-dynamic';
const PLATFORM_MAP: Record<
string,
keyof Awaited<ReturnType<typeof getLatestRelease>>['downloadLinks']
> = {
// Pretty URLs from README / docs (e.g. /download/mac-arm) are kept for
// compatibility, but we now always route through the /download page so users
// see context + a donate prompt + resources while the download kicks off.
// The page handles the actual file trigger itself — no more silent redirects
// to GitHub or direct asset URLs.
const PLATFORM_ALIAS: Record<string, string> = {
'mac-arm': 'macArm',
macArm: 'macArm',
'mac-intel': 'macIntel',
macIntel: 'macIntel',
windows: 'windows',
linux: 'linux',
};
export async function GET(
_request: NextRequest,
request: NextRequest,
{ params }: { params: Promise<{ platform: string }> },
) {
const { platform } = await params;
const key = PLATFORM_MAP[platform];
if (!key) {
return NextResponse.json(
{ error: `Unknown platform: ${platform}. Use: ${Object.keys(PLATFORM_MAP).join(', ')}` },
{ status: 404 },
);
}
try {
const release = await getLatestRelease();
const url = release.downloadLinks[key];
if (!url) {
return NextResponse.json({ error: `No download available for ${platform}` }, { status: 404 });
}
return NextResponse.redirect(url);
} catch {
return NextResponse.redirect(`https://github.com/jamiepine/voicebox/releases/latest`);
// No prebuilt Linux binary yet — send straight to the build-from-source page.
if (platform === 'linux') {
return NextResponse.redirect(new URL('/linux-install', request.url), 307);
}
const normalized = PLATFORM_ALIAS[platform];
const target = new URL('/download', request.url);
if (normalized) target.searchParams.set('platform', normalized);
return NextResponse.redirect(target, 307);
}
+313
View File
@@ -0,0 +1,313 @@
'use client';
import {
ArrowLeft,
Bot,
Coffee,
Download as DownloadIcon,
FileText,
Github,
} from 'lucide-react';
import Image from 'next/image';
import Link from 'next/link';
import { useEffect, useMemo, useState } from 'react';
import { AppleIcon, LinuxIcon, WindowsIcon } from '@/components/PlatformIcons';
import { Button } from '@/components/ui/button';
import { DONATE_URL, GITHUB_RELEASES_PAGE, GITHUB_REPO } from '@/lib/constants';
import type { DownloadLinks } from '@/lib/releases';
type Platform = keyof DownloadLinks;
type PlatformMeta = {
key: Platform;
label: string;
description: string;
icon: React.ComponentType<{ className?: string }>;
};
const PLATFORMS: PlatformMeta[] = [
{ key: 'macArm', label: 'macOS', description: 'Apple Silicon', icon: AppleIcon },
{ key: 'macIntel', label: 'macOS', description: 'Intel (x64)', icon: AppleIcon },
{ key: 'windows', label: 'Windows', description: '64-bit (MSI)', icon: WindowsIcon },
{ key: 'linux', label: 'Linux', description: 'Build from source', icon: LinuxIcon },
];
function detectPlatform(): Platform | null {
if (typeof navigator === 'undefined') return null;
const ua = navigator.userAgent;
if (/Windows/i.test(ua)) return 'windows';
if (/Linux/i.test(ua) && !/Android/i.test(ua)) return 'linux';
if (/Mac/i.test(ua)) {
// Apple Silicon Safari reports "Intel" for compat; default to ARM since
// M-series is the majority. Users can click the Intel button if needed.
return 'macArm';
}
return null;
}
function parseQueryPlatform(search: string): Platform | null {
const params = new URLSearchParams(search);
const raw = params.get('platform');
if (!raw) return null;
// Accept both camelCase and hyphenated forms (/download/mac-arm → ?platform=mac-arm).
const normalized = raw
.toLowerCase()
.replace(/[-_\s]/g, '')
.replace('macarm', 'macArm')
.replace('macintel', 'macIntel');
const valid: Platform[] = ['macArm', 'macIntel', 'windows', 'linux'];
return (valid as string[]).includes(normalized) ? (normalized as Platform) : null;
}
export default function DownloadPage() {
const [links, setLinks] = useState<DownloadLinks | null>(null);
const [linksError, setLinksError] = useState(false);
const [platform, setPlatform] = useState<Platform | null>(null);
const [triggered, setTriggered] = useState(false);
useEffect(() => {
const fromQuery = parseQueryPlatform(window.location.search);
const resolved = fromQuery ?? detectPlatform();
// No prebuilt Linux binary yet — send Linux users to the build-from-source
// instructions instead of sitting on /download trying to trigger a
// download that doesn't exist.
if (resolved === 'linux') {
window.location.replace('/linux-install');
return;
}
setPlatform(resolved);
}, []);
useEffect(() => {
let cancelled = false;
fetch('/api/releases')
.then((r) => {
if (!r.ok) throw new Error(`releases ${r.status}`);
return r.json();
})
.then((data) => {
if (cancelled) return;
if (data.downloadLinks) setLinks(data.downloadLinks as DownloadLinks);
})
.catch(() => {
if (!cancelled) setLinksError(true);
});
return () => {
cancelled = true;
};
}, []);
useEffect(() => {
if (triggered || !links || !platform) return;
const url = links[platform];
if (!url) return;
const a = document.createElement('a');
a.href = url;
a.rel = 'noopener';
a.style.display = 'none';
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
setTriggered(true);
}, [triggered, links, platform]);
const activeMeta = useMemo(
() => PLATFORMS.find((p) => p.key === platform) ?? null,
[platform],
);
return (
<div className="min-h-screen bg-background">
{/* Minimal branded header */}
<header className="border-b border-border/50">
<div className="mx-auto flex max-w-5xl items-center justify-between px-6 py-4">
<Link href="/" className="flex items-center gap-2.5">
<Image
src="/voicebox-logo-app.webp"
alt="Voicebox"
width={28}
height={28}
className="h-7 w-7"
/>
<span className="text-[15px] font-semibold text-foreground">Voicebox</span>
</Link>
<Link
href="/"
className="flex items-center gap-1.5 text-sm text-muted-foreground hover:text-foreground transition-colors"
>
<ArrowLeft className="h-3.5 w-3.5" />
Back to voicebox.sh
</Link>
</div>
</header>
<main className="mx-auto max-w-5xl px-6 py-16 md:py-24">
{/* Hero */}
<div className="flex flex-col md:flex-row md:items-center gap-10 md:gap-14">
<Image
src="/voicebox-logo-app.webp"
alt="Voicebox"
width={200}
height={200}
priority
className="h-32 w-32 md:h-44 md:w-44 shrink-0 drop-shadow-2xl"
/>
<div className="flex-1 min-w-0 text-center md:text-left">
{triggered ? (
<>
<h1 className="text-4xl md:text-5xl font-semibold tracking-tight text-foreground mb-4">
Your download has started.
</h1>
<p className="text-lg text-muted-foreground">
{activeMeta
? `Downloading Voicebox for ${activeMeta.label} (${activeMeta.description}). Check your downloads folder.`
: 'Check your downloads folder for Voicebox.'}
</p>
</>
) : (
<>
<h1 className="text-4xl md:text-5xl font-semibold tracking-tight text-foreground mb-4">
{linksError ? "We couldn't load the latest release." : 'Download Voicebox'}
</h1>
<p className="text-lg text-muted-foreground">
{linksError
? 'Our release server is temporarily unreachable. Please try again in a moment.'
: 'Pick your platform to get started.'}
</p>
</>
)}
</div>
</div>
{/* Platform buttons — always visible as a fallback */}
{linksError ? (
<div className="mt-12 rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 text-center">
<p className="text-sm text-muted-foreground mb-4">
If this keeps happening, you can{' '}
<a
href={`${GITHUB_RELEASES_PAGE}/latest`}
target="_blank"
rel="noopener noreferrer"
className="text-accent underline underline-offset-2 hover:text-accent/80"
>
browse releases on GitHub
</a>
{' '}and grab the build for your platform manually.
</p>
</div>
) : (
<div className="mt-12 rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6">
<h2 className="text-sm font-medium text-foreground mb-4">
{triggered ? 'Download not working?' : 'Choose your platform'}
</h2>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3">
{PLATFORMS.map((meta) => {
const isLinux = meta.key === 'linux';
const url = isLinux ? '/linux-install' : links?.[meta.key];
const isActive = meta.key === platform;
const disabled = !isLinux && !url;
return (
<a
key={meta.key}
href={url ?? '#'}
{...(isLinux ? {} : { download: true })}
aria-disabled={disabled}
onClick={(e) => {
if (disabled) e.preventDefault();
}}
className={`flex items-center rounded-xl border px-5 py-4 transition-all group ${
isActive
? 'border-accent/40 bg-accent/5 hover:border-accent/60'
: 'border-border bg-card/40 hover:border-accent/30 hover:bg-card'
} ${disabled ? 'opacity-50 cursor-not-allowed' : ''}`}
>
<meta.icon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4 flex-1">
<div className="text-sm font-medium text-foreground">{meta.label}</div>
<div className="text-xs text-muted-foreground">{meta.description}</div>
</div>
<DownloadIcon className="h-4 w-4 text-muted-foreground/60 group-hover:text-accent transition-colors" />
</a>
);
})}
</div>
</div>
)}
{/* Donate — prominent, heartfelt, post-click context */}
<div className="mt-16 rounded-2xl border border-border bg-gradient-to-br from-card via-card/80 to-background backdrop-blur-sm p-8 md:p-10 overflow-hidden relative">
<div className="absolute top-0 right-0 w-64 h-64 bg-[#FFDD00]/5 rounded-full blur-3xl -translate-y-1/2 translate-x-1/2 pointer-events-none" />
<div className="relative">
<div className="inline-flex items-center gap-2 rounded-full border border-[#FFDD00]/30 bg-[#FFDD00]/10 px-3 py-1 mb-4">
<Coffee className="h-3 w-3 text-[#FFDD00]" />
<span className="text-[11px] font-medium uppercase tracking-wider text-[#FFDD00]">
Hi from the maintainer
</span>
</div>
<h2 className="text-2xl md:text-3xl font-semibold tracking-tight text-foreground mb-4">
Jamie here — Voicebox is a side project.
</h2>
<p className="text-muted-foreground leading-relaxed mb-6 max-w-2xl">
I build and maintain Voicebox in my spare time. It's completely
free, open source, runs entirely on your machine — no accounts, no
cloud, no subscriptions, no upsells. If it saves you an ElevenLabs
bill or just made your day, a coffee genuinely helps me keep
shipping updates, adding new models, and fixing bugs. Every little
bit keeps the lights on.
</p>
<Button asChild size="lg" className="bg-[#FFDD00]/10 border-[#FFDD00]/30 text-[#FFDD00] hover:bg-[#FFDD00]/20 hover:border-[#FFDD00]/50">
<a href={DONATE_URL} target="_blank" rel="noopener noreferrer">
<Coffee className="h-4 w-4 mr-2" />
Buy me a coffee
</a>
</Button>
</div>
</div>
{/* Resources */}
<div className="mt-10">
<h2 className="text-sm font-medium text-foreground mb-4">While you wait</h2>
<div className="grid grid-cols-1 md:grid-cols-3 gap-4">
<a
href="https://docs.voicebox.sh"
target="_blank"
rel="noopener noreferrer"
className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-5 hover:border-accent/30 hover:bg-card transition-all group"
>
<FileText className="h-5 w-5 text-accent mb-3" />
<h3 className="text-sm font-medium text-foreground mb-1">Read the docs</h3>
<p className="text-xs text-muted-foreground leading-relaxed">
Get familiar with Voicebox — setup, voice cloning, the REST API.
</p>
</a>
<a
href="https://deepwiki.com/jamiepine/voicebox"
target="_blank"
rel="noopener noreferrer"
className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-5 hover:border-accent/30 hover:bg-card transition-all group"
>
<Bot className="h-5 w-5 text-accent mb-3" />
<h3 className="text-sm font-medium text-foreground mb-1">Got questions? Ask AI.</h3>
<p className="text-xs text-muted-foreground leading-relaxed">
DeepWiki is an AI that knows Voicebox inside-out. Ask anything.
</p>
</a>
<a
href={GITHUB_REPO}
target="_blank"
rel="noopener noreferrer"
className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-5 hover:border-accent/30 hover:bg-card transition-all group"
>
<Github className="h-5 w-5 text-accent mb-3" />
<h3 className="text-sm font-medium text-foreground mb-1">Source on GitHub</h3>
<p className="text-xs text-muted-foreground leading-relaxed">
Star the repo, file issues, or contribute a PR.
</p>
</a>
</div>
</div>
</main>
</div>
);
}
+447 -300
View File
@@ -1,327 +1,474 @@
'use client';
"use client";
import { Github, Globe, Languages, MessageSquare, Zap } from 'lucide-react';
import { useEffect, useState } from 'react';
import { ControlUI } from '@/components/ControlUI';
import { Features } from '@/components/Features';
import { Footer } from '@/components/Footer';
import { Navbar } from '@/components/Navbar';
import { AppleIcon, LinuxIcon, WindowsIcon } from '@/components/PlatformIcons';
import { VoiceCreator } from '@/components/VoiceCreator';
import { DOWNLOAD_LINKS, GITHUB_REPO } from '@/lib/constants';
import type { DownloadLinks } from '@/lib/releases';
import {
Github,
Globe,
Languages,
MessageSquare,
SlidersHorizontal,
Zap,
} from "lucide-react";
import {useEffect, useState} from "react";
import {ApiSection} from "@/components/ApiSection";
import {ControlUI} from "@/components/ControlUI";
import {Features} from "@/components/Features";
import {Footer} from "@/components/Footer";
import {Navbar} from "@/components/Navbar";
import {AppleIcon, LinuxIcon, WindowsIcon} from "@/components/PlatformIcons";
import {TutorialsSection} from "@/components/TutorialsSection";
import {VoiceCreator} from "@/components/VoiceCreator";
import {GITHUB_REPO} from "@/lib/constants";
export default function Home() {
const [downloadLinks, setDownloadLinks] = useState<DownloadLinks>(DOWNLOAD_LINKS);
const [version, setVersion] = useState<string | null>(null);
const [totalDownloads, setTotalDownloads] = useState<number | null>(null);
const [version, setVersion] = useState<string | null>(null);
const [totalDownloads, setTotalDownloads] = useState<number | null>(null);
useEffect(() => {
fetch('/api/releases')
.then((res) => {
if (!res.ok) throw new Error('Failed to fetch releases');
return res.json();
})
.then((data) => {
if (data.downloadLinks) setDownloadLinks(data.downloadLinks);
if (data.version) setVersion(data.version);
if (data.totalDownloads != null) setTotalDownloads(data.totalDownloads);
})
.catch((error) => {
console.error('Failed to fetch release info:', error);
});
}, []);
useEffect(() => {
fetch("/api/releases")
.then((res) => {
if (!res.ok) throw new Error("Failed to fetch releases");
return res.json();
})
.then((data) => {
if (data.version) setVersion(data.version);
if (data.totalDownloads != null) setTotalDownloads(data.totalDownloads);
})
.catch((error) => {
console.error("Failed to fetch release info:", error);
});
}, []);
return (
<>
<Navbar />
return (
<>
<Navbar />
{/* ── Hero Section ─────────────────────────────────────────────── */}
<section className="relative pt-32 pb-16">
{/* Background glow */}
<div className="hero-glow hero-glow-fade pointer-events-none absolute inset-0 -top-32">
<div className="absolute left-1/2 top-0 -translate-x-1/2 w-[800px] h-[600px] rounded-full bg-accent/15 blur-[150px]" />
<div className="absolute left-1/2 top-12 -translate-x-1/2 w-[500px] h-[400px] rounded-full bg-accent/10 blur-[80px]" />
</div>
{/* ── Hero Section ─────────────────────────────────────────────── */}
<section className="relative pt-32 pb-16">
{/* Background glow */}
<div className="hero-glow hero-glow-fade pointer-events-none absolute inset-0 -top-32">
<div className="absolute left-1/2 top-0 -translate-x-1/2 w-[800px] h-[600px] rounded-full bg-accent/15 blur-[150px]" />
<div className="absolute left-1/2 top-12 -translate-x-1/2 w-[500px] h-[400px] rounded-full bg-accent/10 blur-[80px]" />
</div>
<div className="relative mx-auto max-w-7xl px-6 text-center">
{/* Logo */}
<div
className="fade-in mx-auto mb-8 h-[120px] w-[120px] md:h-[160px] md:w-[160px]"
style={{ animationDelay: '0ms' }}
>
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src="/voicebox-logo-app.webp"
alt="Voicebox"
className="h-full w-full object-contain"
/>
</div>
<div className="relative mx-auto max-w-7xl px-6 text-center">
{/* Logo */}
<div
className="fade-in mx-auto mb-8 h-[120px] w-[120px] md:h-[160px] md:w-[160px]"
style={{animationDelay: "0ms"}}
>
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src="/voicebox-logo-app.webp"
alt="Voicebox"
className="h-full w-full object-contain"
/>
</div>
{/* Headline */}
<div className="fade-in relative" style={{ animationDelay: '100ms' }}>
<h1 className="text-5xl font-bold tracking-tighter leading-[0.9] text-foreground md:text-7xl lg:text-8xl">
Your voice, your machine.
</h1>
</div>
{/* Headline */}
<div className="fade-in relative" style={{animationDelay: "100ms"}}>
<h1 className="text-5xl font-bold tracking-tighter leading-[0.9] text-foreground md:text-7xl lg:text-8xl">
Clone any voice, in seconds.
</h1>
</div>
{/* Subtitle */}
<p
className="fade-in mx-auto mt-6 max-w-2xl text-lg text-muted-foreground md:text-xl"
style={{ animationDelay: '200ms' }}
>
Open source voice cloning studio with support for multiple TTS engines. Clone any voice,
generate natural speech, and compose multi-voice projects — all running locally.
</p>
{/* Subtitle */}
<p
className="fade-in mx-auto mt-6 max-w-2xl text-lg text-muted-foreground md:text-xl"
style={{animationDelay: "200ms"}}
>
Open source voice cloning studio with support for multiple TTS
engines. Clone any voice, generate natural speech, and compose
multi-voice projects. All running{" "}
<b className="text-white">locally on your machine.</b>
</p>
{/* CTAs */}
<div
className="fade-in mt-10 flex flex-col sm:flex-row items-center justify-center gap-4"
style={{ animationDelay: '300ms' }}
>
<a
href="#download"
className="rounded-full bg-accent px-8 py-3.5 text-sm font-semibold uppercase tracking-wider text-white shadow-[0_4px_20px_hsl(43_60%_50%/0.3),inset_0_2px_0_rgba(255,255,255,0.2),inset_0_-2px_0_rgba(0,0,0,0.1)] transition-all hover:bg-accent-faint active:shadow-[0_2px_10px_hsl(43_60%_50%/0.3),inset_0_4px_8px_rgba(0,0,0,0.3)]"
>
Download
</a>
<a
href={GITHUB_REPO}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 rounded-full border border-border/60 bg-card/40 backdrop-blur-sm px-6 py-3 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground hover:border-border"
>
<Github className="h-4 w-4" />
View on GitHub
</a>
</div>
{/* CTAs */}
<div
className="fade-in mt-10 flex flex-row items-center justify-center gap-3 sm:gap-4"
style={{animationDelay: "300ms"}}
>
<a
href="/download"
className="rounded-full bg-accent px-8 py-3.5 text-sm font-semibold uppercase tracking-wider text-white shadow-[0_4px_20px_hsl(43_60%_50%/0.3),inset_0_2px_0_rgba(255,255,255,0.2),inset_0_-2px_0_rgba(0,0,0,0.1)] transition-all hover:bg-accent-faint active:shadow-[0_2px_10px_hsl(43_60%_50%/0.3),inset_0_4px_8px_rgba(0,0,0,0.3)]"
>
Download
</a>
<a
href={GITHUB_REPO}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 rounded-full border border-border/60 bg-card/40 backdrop-blur-sm px-6 py-3 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground hover:border-border"
>
<Github className="h-4 w-4" />
View on GitHub
</a>
</div>
{/* Version + downloads */}
<p
className="fade-in mt-4 text-xs text-muted-foreground/50"
style={{ animationDelay: '400ms' }}
>
{version ?? ''}
{version && totalDownloads != null ? ' \u00b7 ' : ''}
{totalDownloads != null ? `${totalDownloads.toLocaleString()} downloads` : ''}
{version || totalDownloads != null ? ' \u00b7 ' : ''}
macOS, Windows, Linux
</p>
</div>
{/* Version + downloads */}
<p
className="fade-in mt-4 text-xs text-muted-foreground/50"
style={{animationDelay: "400ms"}}
>
{version ?? ""}
{version && totalDownloads != null ? " \u00b7 " : ""}
{totalDownloads != null
? `${totalDownloads.toLocaleString()} downloads`
: ""}
{version || totalDownloads != null ? " \u00b7 " : ""}
macOS, Windows, Linux
</p>
</div>
{/* ── ControlUI mockup ─────────────────────────────────────── */}
<div className="mt-16">
<ControlUI />
</div>
</section>
{/* ── ControlUI mockup ─────────────────────────────────────── */}
<div className="mt-16">
<ControlUI />
</div>
</section>
{/* ── Features ─────────────────────────────────────────────── */}
<Features />
{/* ── Features ─────────────────────────────────────────────── */}
<Features />
{/* ── Voice Creator ────────────────────────────────────────── */}
<VoiceCreator />
{/* ── Voice Creator ────────────────────────────────────────── */}
<VoiceCreator />
{/* ── Models ─────────────────────────────────────────────────── */}
<section id="about" className="border-t border-border py-24">
<div className="mx-auto max-w-5xl px-6">
<div className="text-center mb-14">
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Multi-Engine Architecture
</h2>
<p className="text-muted-foreground max-w-2xl mx-auto">
Choose the right model for every job. All models run locally on your hardware —
download once, use forever.
</p>
</div>
{/* ── Tutorials ────────────────────────────────────────────── */}
<TutorialsSection />
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
{/* Qwen3-TTS */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">Qwen3-TTS</h3>
<span className="text-xs text-muted-foreground/60">by Alibaba</span>
</div>
<div className="flex gap-1.5">
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
1.7B
</span>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
0.6B
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
High-quality multilingual voice cloning with natural prosody. The only engine with
delivery instructions — control tone, pace, and emotion with natural language.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Globe className="h-3 w-3" />
10 languages
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<MessageSquare className="h-3 w-3" />
Delivery instructions
</span>
</div>
</div>
{/* ── API Section ──────────────────────────────────────────── */}
<ApiSection />
{/* Chatterbox */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">Chatterbox</h3>
<span className="text-xs text-muted-foreground/60">by Resemble AI</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Production-grade voice cloning with the broadest language support. 23 languages with
zero-shot cloning and emotion exaggeration control.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Languages className="h-3 w-3" />
23 languages
</span>
</div>
</div>
{/* ── Models ─────────────────────────────────────────────────── */}
<section id="about" className="border-t border-border py-24">
<div className="mx-auto max-w-5xl px-6">
<div className="text-center mb-14">
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Multi-Engine Architecture
</h2>
<p className="text-muted-foreground max-w-2xl mx-auto">
Choose the right model for every job. All models run locally on
your hardware — download once, use forever.
</p>
</div>
{/* Chatterbox Turbo */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">Chatterbox Turbo</h3>
<span className="text-xs text-muted-foreground/60">by Resemble AI</span>
</div>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
350M
</span>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Lightweight and fast. Supports paralinguistic tags — embed [laugh], [sigh], [gasp]
and more directly in your text for expressive, natural speech.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Zap className="h-3 w-3" />
350M params
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<MessageSquare className="h-3 w-3" />
[laugh] [sigh] tags
</span>
</div>
</div>
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
{/* Qwen3-TTS */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
Qwen3-TTS
</h3>
<span className="text-xs text-muted-foreground/60">
by Alibaba
</span>
</div>
<div className="flex gap-1.5">
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
1.7B
</span>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
0.6B
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
High-quality multilingual voice cloning with natural prosody.
The only engine with delivery instructions — control tone, pace,
and emotion with natural language.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Globe className="h-3 w-3" />
10 languages
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<MessageSquare className="h-3 w-3" />
Delivery instructions
</span>
</div>
</div>
{/* LuxTTS */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">LuxTTS</h3>
<span className="text-xs text-muted-foreground/60">by ZipVoice</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Ultra-fast, CPU-friendly voice cloning at 48kHz. Exceeds 150x realtime on CPU with
~1GB VRAM. The fastest engine for quick iterations.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Zap className="h-3 w-3" />
150x realtime
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
48kHz output
</span>
</div>
</div>
</div>
</div>
</section>
{/* Chatterbox */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
Chatterbox
</h3>
<span className="text-xs text-muted-foreground/60">
by Resemble AI
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Production-grade voice cloning with the broadest language
support. 23 languages with zero-shot cloning and emotion
exaggeration control.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Languages className="h-3 w-3" />
23 languages
</span>
</div>
</div>
{/* ── Download Section ─────────────────────────────────────── */}
<section id="download" className="border-t border-border py-24">
<div className="mx-auto max-w-4xl px-6">
<div className="text-center mb-12">
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Download Voicebox
</h2>
<p className="text-muted-foreground">
Available for macOS, Windows, and Linux. No dependencies required.
</p>
</div>
{/* Chatterbox Turbo */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
Chatterbox Turbo
</h3>
<span className="text-xs text-muted-foreground/60">
by Resemble AI
</span>
</div>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
350M
</span>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Lightweight and fast. Supports paralinguistic tags — embed
[laugh], [sigh], [gasp] and more directly in your text for
expressive, natural speech.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Zap className="h-3 w-3" />
350M params
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<MessageSquare className="h-3 w-3" />
[laugh] [sigh] tags
</span>
</div>
</div>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 max-w-2xl mx-auto">
{/* macOS ARM */}
<a
href={downloadLinks.macArm}
download
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<AppleIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">macOS</div>
<div className="text-xs text-muted-foreground">Apple Silicon (ARM)</div>
</div>
</a>
{/* LuxTTS */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
LuxTTS
</h3>
<span className="text-xs text-muted-foreground/60">
by ZipVoice
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Ultra-fast, CPU-friendly voice cloning at 48kHz. Exceeds 150x
realtime on CPU with ~1GB VRAM. The fastest engine for quick
iterations.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Zap className="h-3 w-3" />
150x realtime
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
48kHz output
</span>
</div>
</div>
{/* macOS Intel */}
<a
href={downloadLinks.macIntel}
download
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<AppleIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">macOS</div>
<div className="text-xs text-muted-foreground">Intel (x64)</div>
</div>
</a>
{/* Qwen CustomVoice */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
Qwen CustomVoice
</h3>
<span className="text-xs text-muted-foreground/60">
by Alibaba
</span>
</div>
<div className="flex gap-1.5">
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
1.7B
</span>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
0.6B
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Nine premium preset speakers with natural-language style
control. Tell the model how to deliver — "speak slowly with
warmth", "authoritative and clear" — and it adapts tone,
emotion, and pace.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<SlidersHorizontal className="h-3 w-3" />
Instruct control
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Globe className="h-3 w-3" />
10 languages
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
9 preset voices
</span>
</div>
</div>
{/* Windows */}
<a
href={downloadLinks.windows}
download
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<WindowsIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">Windows</div>
<div className="text-xs text-muted-foreground">64-bit (MSI)</div>
</div>
</a>
{/* HumeAI TADA */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
TADA
</h3>
<span className="text-xs text-muted-foreground/60">
by Hume AI
</span>
</div>
<div className="flex gap-1.5">
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
3B
</span>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
1B
</span>
</div>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Speech-language model with text-acoustic dual alignment. Built
for long-form generation — produces 700s+ of coherent audio
without drift. Multilingual at 3B, English-focused at 1B.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Globe className="h-3 w-3" />
10 languages
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
Long-form coherent
</span>
</div>
</div>
{/* Linux */}
<a
href="/linux-install"
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<LinuxIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">Linux</div>
<div className="text-xs text-muted-foreground">Build from source</div>
</div>
</a>
</div>
{/* Kokoro 82M */}
<div className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-6 transition-colors hover:border-accent/30">
<div className="flex items-start justify-between mb-3">
<div>
<h3 className="text-base font-semibold text-foreground">
Kokoro
</h3>
<span className="text-xs text-muted-foreground/60">
by hexgrad · Apache 2.0
</span>
</div>
<span className="text-[10px] px-2 py-0.5 rounded-full border border-border bg-background text-muted-foreground">
82M
</span>
</div>
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Tiny 82M-parameter TTS that runs at CPU realtime with negligible
VRAM. Pre-built voice styles instead of cloning — pick a voice,
type, generate. Smallest footprint of any engine.
</p>
<div className="flex flex-wrap gap-2">
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
<Zap className="h-3 w-3" />
CPU realtime
</span>
<span className="flex items-center gap-1 text-[11px] text-muted-foreground/70">
Preset voices
</span>
</div>
</div>
</div>
</div>
</section>
{/* GitHub link */}
<div className="mt-6 text-center">
<a
href={`${GITHUB_REPO}/releases`}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-2 text-sm text-muted-foreground hover:text-foreground transition-colors"
>
<Github className="h-4 w-4" />
View all releases on GitHub
</a>
</div>
</div>
</section>
{/* ── Download Section ─────────────────────────────────────── */}
<section id="download" className="border-t border-border py-24">
<div className="mx-auto max-w-4xl px-6">
<div className="text-center mb-12">
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Download Voicebox
</h2>
<p className="text-muted-foreground">
Available for macOS, Windows, and Linux. No dependencies required.
</p>
</div>
{/* ── Footer ───────────────────────────────────────────────── */}
<Footer />
</>
);
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 max-w-2xl mx-auto">
{/* macOS ARM */}
<a
href="/download?platform=macArm"
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<AppleIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">macOS</div>
<div className="text-xs text-muted-foreground">
Apple Silicon (ARM)
</div>
</div>
</a>
{/* macOS Intel */}
<a
href="/download?platform=macIntel"
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<AppleIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">macOS</div>
<div className="text-xs text-muted-foreground">Intel (x64)</div>
</div>
</a>
{/* Windows */}
<a
href="/download?platform=windows"
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<WindowsIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">Windows</div>
<div className="text-xs text-muted-foreground">
64-bit (MSI)
</div>
</div>
</a>
{/* Linux */}
<a
href="/linux-install"
className="flex items-center rounded-xl border border-border bg-card/60 backdrop-blur-sm px-5 py-4 transition-all hover:border-accent/30 hover:bg-card group"
>
<LinuxIcon className="h-6 w-6 shrink-0 text-muted-foreground group-hover:text-foreground transition-colors" />
<div className="ml-4">
<div className="text-sm font-medium">Linux</div>
<div className="text-xs text-muted-foreground">
Build from source
</div>
</div>
</a>
</div>
{/* GitHub link */}
<div className="mt-6 text-center">
<a
href={`${GITHUB_REPO}/releases`}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-2 text-sm text-muted-foreground hover:text-foreground transition-colors"
>
<Github className="h-4 w-4" />
View all releases on GitHub
</a>
</div>
</div>
</section>
{/* ── Footer ───────────────────────────────────────────────── */}
<Footer />
</>
);
}
+200
View File
@@ -0,0 +1,200 @@
"use client";
import {AppWindow, Code2, Gamepad2, Terminal, Wrench} from "lucide-react";
type Endpoint = {
method: "POST" | "GET" | "DELETE" | "PATCH";
path: string;
label: string;
};
const ENDPOINTS: Endpoint[] = [
{method: "POST", path: "/generate", label: "Generate speech"},
{method: "POST", path: "/generate/{id}/cancel", label: "Cancel a generation"},
{method: "GET", path: "/profiles", label: "List voice profiles"},
{method: "POST", path: "/profiles", label: "Create a new profile"},
{method: "GET", path: "/models/status", label: "Model catalog & state"},
{method: "GET", path: "/history", label: "Past generations"},
{method: "GET", path: "/health", label: "Server health"},
];
const METHOD_STYLES: Record<Endpoint["method"], string> = {
POST: "bg-accent/10 text-accent border-accent/20",
GET: "bg-muted text-muted-foreground border-border",
DELETE: "bg-red-500/10 text-red-400 border-red-500/20",
PATCH: "bg-blue-500/10 text-blue-400 border-blue-500/20",
};
const CURL_SNIPPET = `curl -X POST http://127.0.0.1:17493/generate \\
-H "Content-Type: application/json" \\
-d '{
"text": "Welcome to the game, player one.",
"profile_id": "b3f1c2d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d",
"engine": "qwen_custom_voice",
"instruct": "warm, slow, cinematic"
}' \\
--output line.wav`;
const USE_CASES = [
{
icon: Gamepad2,
title: "Games",
description:
"Generate NPC dialogue on the fly, localize characters into new languages, or ship expressive voice lines without a studio.",
},
{
icon: AppWindow,
title: "Apps & agents",
description:
"Give your app or AI agent a voice. Real-time narration, accessibility readouts, voice replies — all running on the user's machine.",
},
{
icon: Wrench,
title: "Scripts & tools",
description:
"Batch-generate audiobook chapters, automate podcast intros, or wire Voicebox into your Stream Deck. It's just a localhost URL.",
},
];
export function ApiSection() {
return (
<section id="api" className="border-t border-border py-24">
<div className="mx-auto max-w-6xl px-6">
{/* Header */}
<div className="text-center mb-14">
<div className="inline-flex items-center gap-2 rounded-full border border-border/60 bg-card/40 backdrop-blur-sm px-3 py-1 mb-4">
<Code2 className="h-3 w-3 text-accent" />
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">
Built-in REST API
</span>
</div>
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Your local voice API
</h2>
<p className="text-muted-foreground max-w-2xl mx-auto">
Every engine you download becomes a REST endpoint on your machine.
Build apps, games, and voice tools with full programmatic control —
no API keys, no rate limits, no per-character fees.
</p>
</div>
{/* Main panel: endpoints + code snippet */}
<div className="grid grid-cols-1 lg:grid-cols-5 gap-5 mb-14">
{/* Endpoint reference */}
<div className="lg:col-span-3 rounded-xl border border-border bg-card/60 backdrop-blur-sm overflow-hidden">
<div className="flex items-center justify-between px-5 py-3 border-b border-border/60 bg-card/40">
<div className="flex items-center gap-2">
<div className="flex gap-1">
<div className="h-2 w-2 rounded-full bg-muted-foreground/30" />
<div className="h-2 w-2 rounded-full bg-muted-foreground/30" />
<div className="h-2 w-2 rounded-full bg-muted-foreground/30" />
</div>
<span className="text-xs font-medium text-foreground ml-2">
API Reference
</span>
</div>
<code className="text-[10px] bg-background border border-border px-1.5 py-0.5 rounded font-mono text-muted-foreground">
http://127.0.0.1:17493
</code>
</div>
<div className="px-5 py-4 space-y-1">
{ENDPOINTS.map((ep) => (
<div
key={`${ep.method}-${ep.path}`}
className="flex items-center gap-3 py-1.5 group"
>
<span
className={`text-[10px] font-mono font-semibold w-12 text-center rounded px-1 py-0.5 border ${METHOD_STYLES[ep.method]}`}
>
{ep.method}
</span>
<code className="text-xs font-mono text-foreground/90">
{ep.path}
</code>
<span className="text-xs text-muted-foreground/60 ml-auto">
{ep.label}
</span>
</div>
))}
</div>
<div className="border-t border-border/60 px-5 py-3 bg-card/40">
<a
href="http://127.0.0.1:17493/docs"
target="_blank"
rel="noopener noreferrer"
className="text-xs text-accent hover:underline"
>
See the full OpenAPI reference at{" "}
<code className="font-mono">/docs</code> when Voicebox is running
→
</a>
</div>
</div>
{/* Code snippet */}
<div className="lg:col-span-2 rounded-xl border border-border bg-card/60 backdrop-blur-sm overflow-hidden flex flex-col">
<div className="flex items-center gap-2 px-4 py-3 border-b border-border/60 bg-card/40">
<Terminal className="h-3.5 w-3.5 text-muted-foreground" />
<span className="text-xs font-medium text-foreground">
Generate a line
</span>
<span className="ml-auto text-[10px] text-muted-foreground/50 font-mono">
curl
</span>
</div>
<pre className="flex-1 p-4 text-[11px] font-mono text-muted-foreground/90 leading-relaxed overflow-x-auto whitespace-pre">
<code>{CURL_SNIPPET}</code>
</pre>
</div>
</div>
{/* Use cases */}
<div className="grid grid-cols-1 md:grid-cols-3 gap-4">
{USE_CASES.map((uc) => {
const Icon = uc.icon;
return (
<div
key={uc.title}
className="rounded-xl border border-border bg-card/60 backdrop-blur-sm p-5 transition-colors hover:border-accent/30"
>
<div className="flex items-center gap-2 mb-2">
<Icon className="h-4 w-4 text-accent" />
<h3 className="text-[15px] font-medium text-foreground">
{uc.title}
</h3>
</div>
<p className="text-sm leading-relaxed text-muted-foreground">
{uc.description}
</p>
</div>
);
})}
</div>
{/* Bottom bar: key selling points */}
<div className="mt-10 flex flex-wrap items-center justify-center gap-x-8 gap-y-2 text-xs text-muted-foreground">
<span className="flex items-center gap-1.5">
<span className="h-1.5 w-1.5 rounded-full bg-accent" />
No API keys
</span>
<span className="flex items-center gap-1.5">
<span className="h-1.5 w-1.5 rounded-full bg-accent" />
No rate limits
</span>
<span className="flex items-center gap-1.5">
<span className="h-1.5 w-1.5 rounded-full bg-accent" />
No per-character fees
</span>
<span className="flex items-center gap-1.5">
<span className="h-1.5 w-1.5 rounded-full bg-accent" />
Works offline
</span>
<span className="flex items-center gap-1.5">
<span className="h-1.5 w-1.5 rounded-full bg-accent" />
Your audio, your machine
</span>
</div>
</div>
</section>
);
}
+14 -3
View File
@@ -1,6 +1,7 @@
import { Coffee } from 'lucide-react';
import Image from 'next/image';
import Link from 'next/link';
import { GITHUB_REPO } from '@/lib/constants';
import { DONATE_URL, GITHUB_REPO } from '@/lib/constants';
export function Footer() {
return (
@@ -19,9 +20,19 @@ export function Footer() {
/>
<span className="text-sm font-semibold">Voicebox</span>
</div>
<p className="text-sm text-muted-foreground leading-relaxed">
<p className="text-sm text-muted-foreground leading-relaxed mb-4">
Open source voice cloning studio. Local-first, free forever.
</p>
<a
href={DONATE_URL}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-2 rounded-lg border border-border/60 bg-card/60 px-3 py-2 text-sm text-muted-foreground transition-colors hover:text-foreground hover:border-[#FFDD00]/40"
aria-label="Donate via Buy Me a Coffee"
>
<Coffee className="h-4 w-4 text-[#FFDD00]" />
<span className="text-[13px] font-medium">Donate</span>
</a>
</div>
{/* Product */}
@@ -34,7 +45,7 @@ export function Footer() {
</a>
</li>
<li>
<a href="#download" className="hover:text-foreground transition-colors">
<a href="/download" className="hover:text-foreground transition-colors">
Download
</a>
</li>
+38 -20
View File
@@ -1,9 +1,9 @@
'use client';
import { Github } from 'lucide-react';
import { Coffee, Github } from 'lucide-react';
import Image from 'next/image';
import { useEffect, useState } from 'react';
import { GITHUB_REPO } from '@/lib/constants';
import { DONATE_URL, GITHUB_REPO } from '@/lib/constants';
function formatStarCount(count: number): string {
if (count >= 1000) {
@@ -32,7 +32,7 @@ export function Navbar() {
return (
<nav className="fixed inset-x-0 top-0 z-50 border-b border-border/50 bg-background/80 backdrop-blur-xl">
<div className="mx-auto grid max-w-7xl grid-cols-3 items-center px-6 py-3">
<div className="mx-auto flex max-w-7xl items-center justify-between px-6 py-3 sm:grid sm:grid-cols-3">
{/* Logo + wordmark */}
<a href="/" className="flex items-center gap-2.5 justify-self-start">
<Image
@@ -57,10 +57,16 @@ export function Navbar() {
href="#about"
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
>
About
Models
</a>
<a
href="#download"
href="#api"
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
>
API
</a>
<a
href="/download"
className="rounded-md px-3 py-1.5 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground"
>
Download
@@ -75,21 +81,33 @@ export function Navbar() {
</a>
</div>
{/* GitHub star button */}
<a
href={GITHUB_REPO}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 justify-self-end rounded-lg border border-border/60 bg-card/60 px-3 py-1.5 text-sm text-muted-foreground transition-colors hover:text-foreground hover:border-border"
>
<Github className="h-4 w-4" />
<span className="text-[13px] font-medium">Star</span>
{starCount !== null && (
<span className="border-l border-border/60 pl-2 text-[13px] font-semibold text-foreground">
{formatStarCount(starCount)}
</span>
)}
</a>
{/* Donate + GitHub star buttons */}
<div className="flex items-center gap-2 justify-self-end">
<a
href={DONATE_URL}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 rounded-lg border border-border/60 bg-card/60 px-3 py-1.5 text-sm text-muted-foreground transition-colors hover:text-foreground hover:border-[#FFDD00]/40"
aria-label="Donate via Buy Me a Coffee"
>
<Coffee className="h-4 w-4 text-[#FFDD00]" />
<span className="text-[13px] font-medium">Donate</span>
</a>
<a
href={GITHUB_REPO}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 rounded-lg border border-border/60 bg-card/60 px-3 py-1.5 text-sm text-muted-foreground transition-colors hover:text-foreground hover:border-border"
>
<Github className="h-4 w-4" />
<span className="text-[13px] font-medium">Star</span>
{starCount !== null && (
<span className="border-l border-border/60 pl-2 text-[13px] font-semibold text-foreground">
{formatStarCount(starCount)}
</span>
)}
</a>
</div>
</div>
</nav>
);
+21 -15
View File
@@ -1,23 +1,29 @@
import { SiApple, SiLinux } from '@icons-pack/react-simple-icons';
// Official brand icons via Simple Icons (apple/linux). Simple Icons drops
// Microsoft's mark due to trademark policy, so the Windows 11 flag is
// inlined from Microsoft's public brand guidance.
export function AppleIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="currentColor">
<path d="M17.05 20.28c-.98.95-2.05.88-3.08.4-1.09-.5-2.08-.48-3.24 0-1.44.62-2.2.44-3.06-.4C2.79 15.25 3.51 7.59 9.05 7.31c1.35.07 2.29.74 3.08.8 1.18-.24 2.31-.93 3.57-.84 1.51.12 2.65.72 3.4 1.8-3.12 1.87-2.38 5.98.48 7.13-.57 1.5-1.31 2.99-2.54 4.09l.01-.01zM12.03 7.25c-.15-2.23 1.66-4.07 3.74-4.25.29 2.58-2.34 4.5-3.74 4.25z" />
</svg>
);
return <SiApple className={className} color="currentColor" />;
}
export function LinuxIcon({ className }: { className?: string }) {
return <SiLinux className={className} color="currentColor" />;
}
export function WindowsIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="currentColor">
<path d="M3 12V6.75l6-1.32v6.48L3 12zm17-9v8.75l-10 .15V5.21L20 3zM3 13l6 .09v7.81l-6-1.15V13zm17 .25V22l-10-1.8v-7.15l10 .15z" />
</svg>
);
}
export function LinuxIcon({ className }: { className?: string }) {
return (
<svg className={className} viewBox="0 0 24 24" fill="currentColor">
<path d="M12.504 0c-.155 0-.315.008-.48.021-4.226.333-3.105 4.807-3.17 6.298-.076 1.092-.3 1.953-1.05 3.02-.885 1.051-2.127 2.75-2.716 4.521-.278.832-.41 1.684-.287 2.489a.424.424 0 00-.11.135c-.26.26-.195.69-.133 1.001.054.27.112.553.077.784-.12.794-.3 1.593-.3 2.406 0 .599.18 1.193.3 1.791.12.599.3 1.193.3 1.792 0 .812.18 1.611.3 2.405.035.23-.023.514-.077.783-.062.312-.127.742.133 1.002a.424.424 0 00.11.135c-.123.805.01 1.657.287 2.489.589 1.771 1.831 3.47 2.716 4.521.75 1.067 0.974 1.928 1.05 3.02.065 1.491-1.056 5.965 3.17 6.298.165.013.325.021.48.021.155 0 .315-.008.48-.021 4.226-.333 3.105-4.807 3.17-6.298.076-1.092.3-1.953 1.05-3.02.885-1.051 2.127-2.75 2.716-4.521.278-.832.41-1.684.287-2.489a.424.424 0 00.11-.135c.26-.26.195-.69.133-1.001-.054-.27-.112-.553-.077-.784.12-.794.3-1.593.3-2.406 0-.599-.18-1.193-.3-1.791-.12-.599-.3-1.193-.3-1.792 0-.812-.18-1.611-.3-2.405-.035-.23.023-.514.077-.783.062-.312.127-.742-.133-1.002a.424.424 0 00-.11-.135c.123-.805-.01-1.657-.287-2.489-.589-1.771-1.831-3.47-2.716-4.521-.75-1.067-.974-1.928-1.05-3.02-.065-1.491 1.056-5.965-3.17-6.298C12.819.008 12.659 0 12.504 0z" />
<svg
className={className}
viewBox="0 0 24 24"
fill="currentColor"
xmlns="http://www.w3.org/2000/svg"
role="img"
aria-label="Windows"
>
<title>Windows</title>
<path d="M0 3.449L9.75 2.1v9.451H0m10.949-9.602L24 0v11.4l-13.051.149M0 12.6h9.75v9.451L0 20.699M10.949 12.6H24V24l-12.9-1.801" />
</svg>
);
}
+145
View File
@@ -0,0 +1,145 @@
"use client";
import {Play, Youtube} from "lucide-react";
type Tutorial = {
id: string;
title: string;
author: string;
thumbnail: string;
};
const TUTORIALS: (Tutorial | null)[] = [
{
id: "sisnzgc73zc",
title: "Free AI Voice Generator on Your PC (Clones Any Voice)",
author: "Kevin Stratvert",
thumbnail: "/tutorials/sisnzgc73zc.jpg",
},
{
id: "woQe90k7g3c",
title: "NEW Voicebox DESTROYS ElevenLabs?",
author: "Julian Goldie SEO",
thumbnail: "/tutorials/woQe90k7g3c.jpg",
},
{
id: "kqxqjRsdD5E",
title: "This Open-Source TTS App Sounds Scary Good (And It's Free)",
author: "Dave Swift",
thumbnail: "/tutorials/kqxqjRsdD5E.jpg",
},
{
id: "yu9QHqOEqqA",
title: "This FREE AI Tool Just Destroyed ElevenLabs Voice Cloning (VoiceBox)",
author: "Danish Sofi",
thumbnail: "/tutorials/yu9QHqOEqqA.jpg",
},
{
id: "RRRBxNXgeKQ",
title: "Get Started with Voicebox: Open-Source Alternative to ElevenLabs Tutorial",
author: "StinkyScrublet",
thumbnail: "/tutorials/RRRBxNXgeKQ.jpg",
},
{
id: "PyMx4L9mky4",
title: "Free AI Voice Generator (Clones Any Voice)",
author: "mikbes",
thumbnail: "/tutorials/PyMx4L9mky4.jpg",
},
];
function TutorialCard({tutorial}: {tutorial: Tutorial}) {
return (
<a
href={`https://www.youtube.com/watch?v=${tutorial.id}`}
target="_blank"
rel="noopener noreferrer"
className="group rounded-xl border border-border bg-card/60 backdrop-blur-sm overflow-hidden transition-all hover:border-accent/30 hover:bg-card"
>
<div className="relative aspect-video overflow-hidden bg-muted">
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src={tutorial.thumbnail}
alt={tutorial.title}
className="h-full w-full object-cover transition-transform duration-500 group-hover:scale-105"
/>
{/* Gradient overlay */}
<div className="absolute inset-0 bg-gradient-to-t from-black/60 via-black/0 to-black/0" />
{/* Play button overlay */}
<div className="absolute inset-0 flex items-center justify-center">
<div className="flex h-14 w-14 items-center justify-center rounded-full bg-black/50 backdrop-blur-md border border-white/20 transition-all group-hover:scale-110 group-hover:bg-accent/90 group-hover:border-accent">
<Play className="h-5 w-5 text-white fill-white ml-0.5" />
</div>
</div>
{/* YouTube badge */}
<div className="absolute top-3 right-3 flex items-center gap-1 rounded bg-black/60 backdrop-blur-sm px-2 py-1">
<Youtube className="h-3 w-3 text-white" />
<span className="text-[10px] font-medium text-white uppercase tracking-wider">
YouTube
</span>
</div>
</div>
<div className="p-4">
<h3 className="text-sm font-medium text-foreground line-clamp-2 leading-snug mb-1.5 group-hover:text-accent transition-colors">
{tutorial.title}
</h3>
<p className="text-xs text-muted-foreground">{tutorial.author}</p>
</div>
</a>
);
}
function TutorialPlaceholder() {
return (
<div className="rounded-xl border border-dashed border-border/60 bg-card/30 backdrop-blur-sm overflow-hidden">
<div className="relative aspect-video overflow-hidden bg-gradient-to-br from-card via-muted/20 to-card">
<div className="absolute inset-0 flex items-center justify-center">
<div className="flex h-14 w-14 items-center justify-center rounded-full border border-border/40 bg-card/40">
<Play className="h-5 w-5 text-muted-foreground/40 fill-muted-foreground/40 ml-0.5" />
</div>
</div>
</div>
<div className="p-4">
<div className="h-3 w-3/4 rounded bg-muted-foreground/10 mb-2" />
<div className="h-2.5 w-1/3 rounded bg-muted-foreground/10" />
<p className="text-[11px] text-muted-foreground/50 mt-3 uppercase tracking-wider">
Coming soon
</p>
</div>
</div>
);
}
export function TutorialsSection() {
return (
<section id="tutorials" className="border-t border-border py-24">
<div className="mx-auto max-w-6xl px-6">
<div className="text-center mb-14">
<div className="inline-flex items-center gap-2 rounded-full border border-border/60 bg-card/40 backdrop-blur-sm px-3 py-1 mb-4">
<Youtube className="h-3 w-3 text-accent" />
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">
Video tutorials
</span>
</div>
<h2 className="text-3xl font-semibold tracking-tight text-foreground md:text-4xl mb-4">
Learn by watching
</h2>
<p className="text-muted-foreground max-w-2xl mx-auto">
Walkthroughs from the community covering setup, voice cloning, and
production workflows.
</p>
</div>
<div className="grid grid-cols-1 md:grid-cols-3 gap-5">
{TUTORIALS.map((tutorial, i) =>
tutorial ? (
<TutorialCard key={tutorial.id} tutorial={tutorial} />
) : (
<TutorialPlaceholder key={`placeholder-${i}`} />
),
)}
</div>
</div>
</section>
);
}
+1
View File
@@ -4,6 +4,7 @@ export const LATEST_VERSION = 'v0.1.0';
export const GITHUB_REPO = 'https://github.com/jamiepine/voicebox';
export const GITHUB_RELEASES_PAGE = `${GITHUB_REPO}/releases`;
export const DONATE_URL = 'https://buymeacoffee.com/jamiepine';
export const DOWNLOAD_LINKS = {
macArm: GITHUB_RELEASES_PAGE,
+3 -2
View File
@@ -1,6 +1,6 @@
{
"name": "voicebox",
"version": "0.4.0",
"version": "0.4.1",
"private": true,
"workspaces": [
"app",
@@ -24,12 +24,13 @@
"update:icons": "./scripts/update-icons.sh",
"convert:assets": "./scripts/convert-assets.sh",
"lint": "biome lint .",
"typecheck": "bunx tsc -p app/tsconfig.json --noEmit && cd web && bunx tsc --noEmit",
"lint:fix": "biome lint --write .",
"format": "biome format --write .",
"format:check": "biome format .",
"check": "biome check .",
"check:fix": "biome check --write .",
"ci": "biome ci ."
"ci": "bun run typecheck && bun run build:web"
},
"devDependencies": {
"@biomejs/biome": "2.3.12",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@voicebox/tauri",
"private": true,
"version": "0.4.0",
"version": "0.4.1",
"type": "module",
"scripts": {
"dev": "vite",
+1 -1
View File
@@ -5041,7 +5041,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "voicebox"
version = "0.3.1"
version = "0.4.1"
dependencies = [
"base64 0.22.1",
"core-foundation-sys",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "voicebox"
version = "0.4.0"
version = "0.4.1"
description = "A production-quality desktop app for Qwen3-TTS voice cloning and generation"
authors = ["you"]
license = ""
+107 -13
View File
@@ -8,12 +8,71 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
/// Try to find a PulseAudio/PipeWire monitor source using `pactl`.
/// Returns the source name (e.g. "alsa_output.pci-0000_0d_00.6.analog-stereo.monitor") if found.
fn find_monitor_source_via_pactl() -> Option<String> {
let output = std::process::Command::new("pactl")
.args(["list", "short", "sources"])
.output()
.ok()?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout);
// First, try to find the monitor of the default sink
let default_sink = std::process::Command::new("pactl")
.args(["get-default-sink"])
.output()
.ok()
.and_then(|o| {
if o.status.success() {
Some(String::from_utf8_lossy(&o.stdout).trim().to_string())
} else {
None
}
});
// If we know the default sink, look for its .monitor specifically
if let Some(sink_name) = &default_sink {
let monitor_name = format!("{}.monitor", sink_name);
for line in stdout.lines() {
let parts: Vec<&str> = line.split('\t').collect();
if parts.len() >= 2 && parts[1] == monitor_name {
eprintln!(
"Linux audio capture: Found default sink monitor via pactl: {}",
monitor_name
);
return Some(monitor_name);
}
}
}
// Fallback: find any .monitor source
for line in stdout.lines() {
let parts: Vec<&str> = line.split('\t').collect();
if parts.len() >= 2 && parts[1].ends_with(".monitor") {
let name = parts[1].to_string();
eprintln!(
"Linux audio capture: Found monitor source via pactl: {}",
name
);
return Some(name);
}
}
None
}
/// Start capturing system audio on Linux using PulseAudio monitor sources.
///
/// PulseAudio exposes "monitor" devices that mirror the output of each sink,
/// allowing us to capture whatever audio is currently playing on the system.
/// We use `cpal` with the default host (which will be PulseAudio or PipeWire
/// on modern Linux) and look for monitor input devices.
/// On modern Linux with PulseAudio or PipeWire, we first try to detect the
/// monitor source via `pactl` and set the `PULSE_SOURCE` environment variable.
/// This tells PulseAudio's ALSA plugin to use the monitor as the default input
/// source for this process. If `pactl` is unavailable, we fall back to searching
/// cpal device names for "monitor".
pub async fn start_capture(
state: &AudioCaptureState,
max_duration_secs: u32,
@@ -42,30 +101,62 @@ pub async fn start_capture(
// Spawn capture on a dedicated thread
thread::spawn(move || {
// Try to set PULSE_SOURCE to a monitor before initializing cpal.
// This tells PulseAudio/PipeWire's ALSA plugin to use the monitor
// as the default input source for this process.
let monitor_source = find_monitor_source_via_pactl();
if let Some(ref source_name) = monitor_source {
eprintln!(
"Linux audio capture: Setting PULSE_SOURCE={}",
source_name
);
std::env::set_var("PULSE_SOURCE", source_name);
}
let host = cpal::default_host();
// Try to find a monitor device for system audio capture.
// On PulseAudio/PipeWire, monitor sources have "monitor" in their name.
let device = {
// Select the capture device.
// If PULSE_SOURCE was set, the default input device IS the monitor.
// Otherwise, fall back to searching device names for "monitor".
let device = if monitor_source.is_some() {
// PULSE_SOURCE was set — default input IS the monitor now
match host.default_input_device() {
Some(d) => {
let name = d.name().unwrap_or_default();
eprintln!(
"Linux audio capture: Using PULSE_SOURCE monitor device: {}",
name
);
d
}
None => {
let error_msg = "No audio input device available".to_string();
eprintln!("{}", error_msg);
*error_arc.lock().unwrap() = Some(error_msg);
return;
}
}
} else {
// pactl not available — try to find monitor by name (original approach)
let mut monitor_device = None;
if let Ok(devices) = host.input_devices() {
for d in devices {
if let Ok(name) = d.name() {
let name_lower = name.to_lowercase();
if name_lower.contains("monitor") {
eprintln!("Linux audio capture: Found monitor device: {}", name);
eprintln!(
"Linux audio capture: Found monitor device by name: {}",
name
);
monitor_device = Some(d);
break;
}
}
}
}
match monitor_device {
Some(d) => d,
None => {
// Fallback to default input device (microphone)
eprintln!("Linux audio capture: No monitor device found, falling back to default input");
match host.default_input_device() {
Some(d) => d,
@@ -266,7 +357,11 @@ pub async fn stop_capture(state: &AudioCaptureState) -> Result<String, String> {
}
pub fn is_supported() -> bool {
// Check if we can find a monitor device for system audio capture
// Check via pactl first (most reliable on modern Linux)
if find_monitor_source_via_pactl().is_some() {
return true;
}
// Fallback: check cpal devices
let host = cpal::default_host();
if let Ok(devices) = host.input_devices() {
for d in devices {
@@ -277,7 +372,6 @@ pub fn is_supported() -> bool {
}
}
}
// Even without a monitor, basic input capture is available
host.default_input_device().is_some()
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "Voicebox",
"version": "0.4.0",
"version": "0.4.1",
"identifier": "sh.voicebox.app",
"build": {
"beforeDevCommand": "bun run dev",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@voicebox/web",
"private": true,
"version": "0.4.0",
"version": "0.4.1",
"type": "module",
"scripts": {
"dev": "vite",
-4
View File
@@ -11,10 +11,6 @@ class WebUpdater implements PlatformUpdater {
private subscribers: Set<(status: UpdateStatus) => void> = new Set();
private notifySubscribers() {
this.subscribers.forEach((callback) => callback(this.status));
}
subscribe(callback: (status: UpdateStatus) => void): () => void {
this.subscribers.add(callback);
callback(this.status);
+3 -2
View File
@@ -18,8 +18,9 @@
"baseUrl": ".",
"paths": {
"@/*": ["../app/src/*"]
}
},
"types": ["vite/client"]
},
"include": ["src"],
"include": ["src", "../app/src/global.d.ts"],
"references": [{ "path": "./tsconfig.node.json" }]
}