mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-26 13:45:16 -07:00
Compare commits
64
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ddc42b4735 | ||
|
|
d02c0cb1ae | ||
|
|
e69830a1ea | ||
|
|
a756295af0 | ||
|
|
7db16510a1 | ||
|
|
d3a44338a2 | ||
|
|
28aa963b09 | ||
|
|
ae91aa9a88 | ||
|
|
da6070155e | ||
|
|
3c1e8512b9 | ||
|
|
bf58750447 | ||
|
|
2d56309bdd | ||
|
|
0445be295c | ||
|
|
8d550a5f7c | ||
|
|
795bd54381 | ||
|
|
a6ab5f3858 | ||
|
|
9d7e4a417e | ||
|
|
882cabc7d2 | ||
|
|
9c76b5de2c | ||
|
|
4560b7378a | ||
|
|
abd9943430 | ||
|
|
c8cb12f1bc | ||
|
|
54a3bf322e | ||
|
|
476abe07fc | ||
|
|
67bf8e906a | ||
|
|
625e1ba549 | ||
|
|
cfe6770639 | ||
|
|
00452b51a8 | ||
|
|
106aec46a8 | ||
|
|
c9e5c5d9a7 | ||
|
|
48cd1f369a | ||
|
|
2bfe400457 | ||
|
|
0aa19a9994 | ||
|
|
73170d0e92 | ||
|
|
0317626677 | ||
|
|
a5d5c780c2 | ||
|
|
7184a25e44 | ||
|
|
479bc7fc5e | ||
|
|
3e7727d1d2 | ||
|
|
be7c0cec12 | ||
|
|
c9d8142a78 | ||
|
|
13ba5f1aa6 | ||
|
|
9a3c307c75 | ||
|
|
1da16cfc57 | ||
|
|
a1807be04d | ||
|
|
fdba18e9ee | ||
|
|
07a845cece | ||
|
|
615d604ceb | ||
|
|
a383ff6863 | ||
|
|
75abbb02c3 | ||
|
|
b49f14a814 | ||
|
|
05686efbfd | ||
|
|
e2c03fef9a | ||
|
|
4347eaed4c | ||
|
|
60aac279ce | ||
|
|
8b1c7552be | ||
|
|
9a955a77d2 | ||
|
|
c18591c0c3 | ||
|
|
ea3469f2dc | ||
|
|
8b796bc6b4 | ||
|
|
707046237c | ||
|
|
83ebababe7 | ||
|
|
eb5869e59f | ||
|
|
2e95b7c5d8 |
@@ -0,0 +1,299 @@
|
||||
---
|
||||
name: triage-prs
|
||||
description: Use this skill to triage the open PR queue before a release. Classifies every open PR into must-merge, candidate, superseded, or deferred; writes a working triage doc; and runs the merge loop end-to-end. Designed for the pre-release "PR speedrun" pass where a solo maintainer wants to clear the inbound backlog in a single session.
|
||||
---
|
||||
|
||||
# Triage PRs
|
||||
|
||||
## Goal
|
||||
|
||||
Turn a backlog of open PRs into a shipped set of merges in a single focused session. Produce a tracked, resumable plan (`<VERSION>_PR_TRIAGE.md`), then work it — rebasing where needed, merging in isolation-safe batches, applying post-merge follow-ups, and closing superseded or partially-applicable PRs with credit to their authors.
|
||||
|
||||
This skill pairs with `draft-release-notes` and `release-bump`: triage first, then draft notes against the new main, then cut the release.
|
||||
|
||||
## When to use
|
||||
|
||||
- Before a minor or major release when 10+ open PRs have accumulated
|
||||
- When you want to unblock merging without losing the narrative of what's landing
|
||||
- When you know you can't personally review every PR deeply, but need to land the critical subset fast
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `gh` CLI authenticated against the repo
|
||||
- A dedicated worktree for PR review (avoid contaminating `main` with checkouts of contributor branches)
|
||||
- Clarity on the target version — the triage doc is named after it (e.g. `0.4.0_PR_TRIAGE.md`)
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Set up an isolated PR-review worktree
|
||||
|
||||
```bash
|
||||
git worktree list # check for stale ones first
|
||||
git worktree prune
|
||||
git worktree add ../voicebox-pr-review -b pr-review-<VERSION> main
|
||||
```
|
||||
|
||||
Keep the main worktree for release-prep work (changelog drafts, direct-to-main follow-ups). Keep the review worktree for `gh pr checkout` — each checkout moves HEAD to a contributor branch, which you don't want to do in the main worktree.
|
||||
|
||||
### 2. Gather metadata for every open PR
|
||||
|
||||
```bash
|
||||
gh pr list --state open --limit 50 --json \
|
||||
number,title,author,isDraft,mergeable,mergeStateStatus,files,additions,deletions,reviewDecision,statusCheckRollup,maintainerCanModify \
|
||||
--jq '.[] | {num: .number, title, author: .author.login, mergeable, state: .mergeStateStatus, canModify: .maintainerCanModify, changes: "+\(.additions)/-\(.deletions)", files: [.files[].path]}'
|
||||
```
|
||||
|
||||
You want, for each PR:
|
||||
- Size (`+additions/-deletions`)
|
||||
- Mergeable state (`CLEAN`, `UNSTABLE`, `DIRTY` = conflicts, `UNKNOWN` = GitHub still computing)
|
||||
- Whether maintainer edits are allowed on the branch (needed later if you rebase for the author)
|
||||
- File paths touched (helps spot overlaps between PRs)
|
||||
|
||||
`UNKNOWN` is common right after a push to main — just try the merge and see.
|
||||
|
||||
### 3. Classify into tiers
|
||||
|
||||
Sort each PR into exactly one bucket:
|
||||
|
||||
**Tier 1 — Merge:** small, mergeable, fixes a real bug, clean CI, low review cost. One-liners, dependency relaxations, targeted safety hardening. These are the easy wins.
|
||||
|
||||
**Tier 2 — Candidate, review:** medium size (50-200 lines), touches more surface area, looks sound but needs a closer read. New user-facing features that fit the product direction.
|
||||
|
||||
**Supersede:** the fix or feature is already covered by something merged. Close with a comment pointing to the superseding PR. Check carefully — "similar title" isn't proof; compare the actual diffs.
|
||||
|
||||
**Defer to next release:** big features, dirty conflicts, draft PRs, anything touching the release pipeline in ways that would introduce risk. Don't merge these in a speedrun — they need dedicated focus.
|
||||
|
||||
### 4. Write the triage doc
|
||||
|
||||
Create `<VERSION>_PR_TRIAGE.md` in the PR-review worktree root. Structure:
|
||||
|
||||
```markdown
|
||||
# <Repo> <VERSION> — PR Triage
|
||||
|
||||
Working doc for tracking which open PRs land in <VERSION>. Delete after release cut.
|
||||
|
||||
Last updated: <DATE>
|
||||
|
||||
## Progress
|
||||
|
||||
**Tier 1: 0 / N merged**
|
||||
**Tier 2: 0 / M handled**
|
||||
**Supersede triage: pending**
|
||||
|
||||
---
|
||||
|
||||
## Merge for <VERSION> — critical bug fixes
|
||||
|
||||
| PR | Status | Size | What it fixes | Why must-have |
|
||||
|---|---|---|---|---|
|
||||
| [#123](url) | [ ] | +5/-0 | ... | ... |
|
||||
|
||||
## Strong candidate — needs a quick review
|
||||
|
||||
| PR | Status | Size | Summary |
|
||||
|---|---|---|---|
|
||||
|
||||
## Close as superseded
|
||||
|
||||
| PR | Status | Reason |
|
||||
|---|---|---|
|
||||
|
||||
## Defer to <NEXT_VERSION>
|
||||
|
||||
- [#xxx](url) ... — reason
|
||||
|
||||
---
|
||||
|
||||
## Order of attack
|
||||
|
||||
1. Close superseded PRs (one-liner comments)
|
||||
2. Merge tier-1 in dependency-free batches — check file paths don't overlap
|
||||
3. Review tier-2 individually
|
||||
4. Rerun `draft-release-notes` to pick up everything
|
||||
5. Run `release-bump`
|
||||
```
|
||||
|
||||
The **Progress** header is the most important part — it's your scoreboard and lets you resume cleanly if the session gets interrupted.
|
||||
|
||||
### 5. Work the loop — per PR
|
||||
|
||||
For each PR in the tier-1 / tier-2 list:
|
||||
|
||||
**a. Checkout in the review worktree:**
|
||||
```bash
|
||||
cd ../voicebox-pr-review
|
||||
git checkout pr-review-<VERSION> # reset to neutral base
|
||||
gh pr checkout <N>
|
||||
```
|
||||
|
||||
**b. Read the *actual* commit, not `main..HEAD`:**
|
||||
|
||||
```bash
|
||||
git show HEAD # the PR's actual changes
|
||||
git show --stat HEAD # files touched + line counts
|
||||
```
|
||||
|
||||
**Do NOT review via `git diff main..HEAD`** if the PR branch is older than main. That diff includes *every commit that landed on main after the PR was forked* as `-` (deletion) lines. A 3-line PR can look like a 700-line revert. This is the single easiest way to misjudge a PR.
|
||||
|
||||
**c. Evaluate concerns:** correctness, scope, interaction with already-merged work, version compatibility (e.g. can't use an API that requires a dependency version we don't yet pin).
|
||||
|
||||
**d. Rebase if the branch is behind main:**
|
||||
```bash
|
||||
git fetch origin main
|
||||
git rebase origin/main
|
||||
```
|
||||
|
||||
This is **essential** before squash-merging. GitHub's squash computes `diff(PR-head, merge-base)` — on a stale branch, that diff includes reverting every in-between commit. Rebasing moves the merge-base forward so the squash is clean.
|
||||
|
||||
**e. If maintainer edits are allowed, push the rebase back to the contributor's fork:**
|
||||
```bash
|
||||
git remote add <author> https://github.com/<author>/<repo>.git
|
||||
git fetch <author> <branch> # get their ref first
|
||||
git push <author> HEAD:<branch> --force-with-lease
|
||||
```
|
||||
|
||||
This keeps GitHub's PR UI in sync with the rebased state and makes the merge clean from the GitHub side.
|
||||
|
||||
**f. Merge:**
|
||||
```bash
|
||||
gh pr merge <N> --squash
|
||||
```
|
||||
|
||||
**g. Update the triage doc** — flip the checkbox to `✅ merged <sha>` (use the short SHA from `gh pr view <N> --json mergeCommit --jq '.mergeCommit.oid[0:7]'`). Update the Progress header.
|
||||
|
||||
### 6. Batch tiny fixes
|
||||
|
||||
PRs with ≤5 line changes, clean CI, non-overlapping file paths, and obviously-correct intent (e.g. one-line dependency relax, env var add, import path fix) can be merged in a single loop without the review-per-PR ceremony:
|
||||
|
||||
```bash
|
||||
for pr in 425 384 416 429; do
|
||||
echo "=== Merging PR $pr ==="
|
||||
gh pr merge $pr --squash
|
||||
done
|
||||
```
|
||||
|
||||
Verify afterward that each landed cleanly:
|
||||
```bash
|
||||
for pr in 425 384 416 429; do
|
||||
gh pr view $pr --json state,mergeCommit --jq "{pr: $pr, state, sha: .mergeCommit.oid[0:7]}"
|
||||
done
|
||||
```
|
||||
|
||||
### 7. Post-merge follow-ups
|
||||
|
||||
Sometimes a PR is worth merging despite a known minor issue (e.g. incomplete dtype map, stale sentinel cleanup). Don't block the merge; apply the follow-up as a normal branch + PR right after:
|
||||
|
||||
```bash
|
||||
cd <main-worktree>
|
||||
git pull --ff-only origin main
|
||||
git checkout -b fix/<short-name>
|
||||
# edit...
|
||||
git commit -m "fix(<area>): <one-liner>"
|
||||
git push -u origin fix/<short-name>
|
||||
gh pr create --title "..." --body "Follow-up to #<N>. ..."
|
||||
```
|
||||
|
||||
Record both SHAs in the triage doc (`✅ merged <pr-sha> + follow-up <pr>`).
|
||||
|
||||
**Direct-to-main exception:** only under an explicit, scoped policy (e.g. "release speedrun"). Don't default to it.
|
||||
|
||||
### 8. Supersede: close with a credit-pointing comment
|
||||
|
||||
```bash
|
||||
gh pr close <N> --comment "Closing — superseded by merged #<M> which landed <brief description>. Thanks!"
|
||||
```
|
||||
|
||||
Check the diffs first — "similar title" is not enough. If the PR is *partially* superseded (the diagnosis is right but only half the changes are still needed), do a partial-apply instead.
|
||||
|
||||
### 9. Partial-apply pattern
|
||||
|
||||
When a PR has both valuable and questionable changes bundled:
|
||||
|
||||
```bash
|
||||
cd <main-worktree>
|
||||
git pull --ff-only origin main
|
||||
|
||||
# Cherry-pick specific files from the PR branch
|
||||
git checkout <pr-commit-sha> -- <file1> <file2>
|
||||
|
||||
# Review the staged changes, adjust as needed
|
||||
git diff --cached
|
||||
|
||||
# Apply any surgical edits to files you don't want to bulk-replace
|
||||
# (e.g. the PR's file predates a recent main commit you need to preserve)
|
||||
|
||||
# Commit with a trailer crediting the original author
|
||||
git commit -m "$(cat <<'EOF'
|
||||
<subject>
|
||||
|
||||
<body explaining what was kept vs dropped>
|
||||
|
||||
Co-Authored-By: <author> <[email protected]>
|
||||
EOF
|
||||
)"
|
||||
git push ... # branch + PR, unless under the direct-to-main exception
|
||||
```
|
||||
|
||||
Then close the PR with a comment explaining what was applied and what was dropped, referencing the commit SHA.
|
||||
|
||||
### 10. Keep the doc current
|
||||
|
||||
Every merge, every close, every follow-up → update `<VERSION>_PR_TRIAGE.md`. The doc is your session log. If you're interrupted and resume tomorrow, the doc is the only source of truth for "where am I."
|
||||
|
||||
### 11. When triage is done
|
||||
|
||||
- Every PR in the doc has a terminal status (✅ merged / ✅ closed / deferred)
|
||||
- Progress header shows N/N for each tier
|
||||
- Next skill to run is `draft-release-notes` (to regenerate `[Unreleased]` against the new main), then `release-bump`
|
||||
|
||||
You can delete the triage doc after the release ships, or keep it in version history as a record.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`main..HEAD` on a stale branch lies.** It shows everything main gained since the branch split as deletions. Always review via `git show HEAD` for the PR's actual commit.
|
||||
- **Squash-merging an unrebased branch reverts in-between work.** The squash computes `diff(PR-head, merge-base)`. Rebase moves the merge-base forward.
|
||||
- **`mergeable=UNKNOWN`** is transient — GitHub is recomputing after a push. Just try the merge.
|
||||
- **Route ordering matters (FastAPI and similar):** `DELETE /history/failed` must be registered *before* `DELETE /history/{id}`, or the parameterized path will consume `"failed"` as an ID.
|
||||
- **Apple's `-weak_framework` overrides `-framework`** for the same framework, regardless of order — use it via `cargo:rustc-link-arg=-Wl,-weak_framework,Name` when a dependency hard-links something optional.
|
||||
- **Dependency version floors constrain what you can apply.** Before accepting a kwarg rename like `torch_dtype=` → `dtype=`, check the min-version pin supports it. Sometimes the right move is to cherry-pick half the PR.
|
||||
- **`cpal::Stream` and similar `!Send` audio types** can't cross `await` points or `spawn_blocking`. Sometimes a "not-ideal but correct" sync wait is the best available fix; flag but don't block.
|
||||
- **PyTorch nightly builds are not shippable for releases** — non-deterministic, can regress between runs. If a PR suggests switching to nightly to fix a GPU issue, prefer `TORCH_CUDA_ARCH_LIST=...+PTX` or wait for stable support instead.
|
||||
|
||||
## Canonical commands reference
|
||||
|
||||
```bash
|
||||
# Bulk PR metadata
|
||||
gh pr list --state open --limit 50 --json number,title,author,mergeable,mergeStateStatus,additions,deletions,maintainerCanModify,files
|
||||
|
||||
# Detailed single-PR view
|
||||
gh pr view <N> --json body,author,headRefName,baseRefName,mergeable,maintainerCanModify,files,statusCheckRollup
|
||||
|
||||
# The actual commit, not the branch-vs-main diff
|
||||
git show HEAD
|
||||
git show --stat HEAD
|
||||
gh pr diff <N>
|
||||
|
||||
# Rebase contributor branch onto current main
|
||||
git fetch origin main && git rebase origin/main
|
||||
|
||||
# Push rebase back to contributor fork (maintainerCanModify=true required)
|
||||
git remote add <author> https://github.com/<author>/<repo>.git
|
||||
git fetch <author> <branch>
|
||||
git push <author> HEAD:<branch> --force-with-lease
|
||||
|
||||
# Merge
|
||||
gh pr merge <N> --squash
|
||||
|
||||
# Confirm merge SHA for triage doc
|
||||
gh pr view <N> --json state,mergeCommit --jq '{state, sha: .mergeCommit.oid[0:7]}'
|
||||
|
||||
# Close superseded
|
||||
gh pr close <N> --comment "Closing — superseded by merged #<M>. Thanks!"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- **Never review a stale branch via `main..HEAD`.** This is the single most important line in this skill.
|
||||
- **The triage doc is the session state.** Lose the doc, lose the session. Update it after every action.
|
||||
- **Credit contributors even on partial-applies.** Use `Co-Authored-By:` trailers and close comments that link to the applied commit.
|
||||
- **Don't let perfect be the enemy of shipped.** A fix that goes from "broken" to "works with a minor known issue" is a strict improvement. Flag the issue, file a follow-up, merge the fix.
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
[bumpversion]
|
||||
current_version = 0.3.1
|
||||
current_version = 0.4.1
|
||||
commit = True
|
||||
tag = True
|
||||
tag_name = v{new_version}
|
||||
|
||||
@@ -38,7 +38,6 @@ biome.json
|
||||
.bumpversion.cfg
|
||||
.npmrc
|
||||
Makefile
|
||||
CHANGELOG.md
|
||||
CONTRIBUTING.md
|
||||
SECURITY.md
|
||||
LICENSE
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
jobs:
|
||||
frontend-quality:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: Install dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
|
||||
- name: Typecheck app + web
|
||||
run: bun run typecheck
|
||||
|
||||
- name: Build web smoke test
|
||||
run: bun run build:web
|
||||
@@ -26,12 +26,45 @@ jobs:
|
||||
args: ""
|
||||
python-version: "3.12"
|
||||
backend: "pytorch"
|
||||
- platform: "ubuntu-22.04"
|
||||
# --config override disables updater-artifact generation on Linux.
|
||||
# tauri.conf.json has createUpdaterArtifacts: "v1Compatible" which
|
||||
# on Linux wants to synthesize a .AppImage.tar.gz by downloading
|
||||
# linuxdeploy at build time — this is what silently hangs CI
|
||||
# (see v0.4.2 round 2, 25 min of no output after rpm bundling).
|
||||
# We ship deb+rpm only; Linux users update via apt/dnf, not the
|
||||
# Tauri in-app updater.
|
||||
args: '--target x86_64-unknown-linux-gnu --bundles deb,rpm --verbose --config {"bundle":{"createUpdaterArtifacts":false}}'
|
||||
python-version: "3.12"
|
||||
backend: "pytorch"
|
||||
|
||||
runs-on: ${{ matrix.platform }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# Ubuntu runners ship with ~14 GB free; pip + PyInstaller + torch can
|
||||
# peak well above that during the build. Reclaim ~25 GB by pruning
|
||||
# preinstalled toolchains we don't use. This is what likely tripped
|
||||
# the March 2026 Linux release attempts (see commit 103e98b
|
||||
# "github runners suck") — not a code issue, a disk-pressure one.
|
||||
- name: Free up disk space (ubuntu)
|
||||
if: contains(matrix.platform, 'ubuntu') || contains(matrix.platform, 'namespace')
|
||||
# Pinned to v1.3.1 (SHA) — this job runs with contents: write and
|
||||
# handles signing secrets later, so we don't want a floating ref.
|
||||
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
# large-packages: true would `apt-get remove '^llvm-.*'`, which
|
||||
# cascade-removes reverse deps that won't be pulled back in by the
|
||||
# `llvm-dev` install below. The other flags already free ~20 GB,
|
||||
# enough for the Python + torch + PyInstaller build.
|
||||
large-packages: false
|
||||
swap-storage: true
|
||||
|
||||
- name: Install dependencies (ubuntu only)
|
||||
if: contains(matrix.platform, 'ubuntu') || contains(matrix.platform, 'namespace')
|
||||
run: |
|
||||
@@ -68,6 +101,15 @@ jobs:
|
||||
if: matrix.backend == 'mlx'
|
||||
run: |
|
||||
pip install -r backend/requirements-mlx.txt
|
||||
# mlx-audio>=0.3.1 and mlx-lm>=0.31.1 both declare transformers>=5.x,
|
||||
# which conflicts with our 4.57.x cap. The runtime APIs we use work
|
||||
# fine on transformers 4.57.x in practice (verified in dev), so install
|
||||
# them --no-deps. mlx-audio's other runtime deps (huggingface_hub,
|
||||
# librosa, numpy, numba, pyloudnorm) are already in requirements.txt;
|
||||
# the rest (sounddevice, miniaudio, protobuf, sentencepiece, pyyaml,
|
||||
# jinja2) are pulled in by other engines.
|
||||
pip install --no-deps mlx-lm==0.31.1
|
||||
pip install --no-deps mlx-audio==0.4.1
|
||||
|
||||
- name: Build Python server (Linux/macOS)
|
||||
if: matrix.platform != 'windows-latest'
|
||||
@@ -124,6 +166,21 @@ jobs:
|
||||
p12-file-base64: ${{ secrets.APPLE_CERTIFICATE }}
|
||||
p12-password: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
|
||||
|
||||
- name: Disk / environment snapshot (pre-bundle debug)
|
||||
if: contains(matrix.platform, 'ubuntu') || contains(matrix.platform, 'namespace')
|
||||
run: |
|
||||
echo "=== df -h ==="
|
||||
df -h
|
||||
echo "=== free -h ==="
|
||||
free -h
|
||||
echo "=== Rust / Cargo ==="
|
||||
rustc --version
|
||||
cargo --version
|
||||
echo "=== Bun ==="
|
||||
bun --version
|
||||
echo "=== Tauri CLI ==="
|
||||
cd tauri && bun run tauri --version
|
||||
|
||||
- name: Extract release notes from CHANGELOG.md
|
||||
id: changelog
|
||||
shell: bash
|
||||
@@ -147,7 +204,13 @@ jobs:
|
||||
echo "CHANGELOG_EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Linux hang watchdog: previous releases silently wedged inside tauri
|
||||
# bundling (possibly linuxdeploy/AppImage download, possibly cargo link).
|
||||
# Cap the step at 30 min so we get logs instead of waiting out the 6hr
|
||||
# job timeout. Other platforms historically complete in ~25 min, so 45
|
||||
# is comfortable.
|
||||
- uses: tauri-apps/[email protected]
|
||||
timeout-minutes: ${{ (contains(matrix.platform, 'ubuntu') || contains(matrix.platform, 'namespace')) && 30 || 45 }}
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
|
||||
@@ -159,6 +222,9 @@ jobs:
|
||||
APPLE_PROVIDER_SHORT_NAME: ${{ secrets.APPLE_PROVIDER_SHORT_NAME }}
|
||||
APPLE_API_ISSUER: ${{ secrets.APPLE_API_ISSUER }}
|
||||
APPLE_API_KEY: ${{ secrets.APPLE_API_KEY }}
|
||||
# Stream subprocess stdout/stderr so the hang is visible in logs.
|
||||
CARGO_TERM_VERBOSE: "true"
|
||||
RUST_BACKTRACE: "1"
|
||||
with:
|
||||
projectPath: tauri
|
||||
tagName: v__VERSION__
|
||||
@@ -203,6 +269,12 @@ jobs:
|
||||
- name: Build CUDA server binary (onedir)
|
||||
shell: bash
|
||||
working-directory: backend
|
||||
env:
|
||||
# Include Blackwell (sm_120) via PTX forward compatibility.
|
||||
# Pre-built PyTorch cu128 wheels ship native kernels for sm_80/86/89/90
|
||||
# but not sm_120. Setting this env var causes torch.utils.cpp_extension
|
||||
# (and any JIT-compiled kernels) to target Blackwell GPUs as well.
|
||||
TORCH_CUDA_ARCH_LIST: "8.0;8.6;8.9;9.0;12.0+PTX"
|
||||
run: python build_binary.py --cuda
|
||||
|
||||
- name: Package into server core + CUDA libs archives
|
||||
|
||||
@@ -63,3 +63,8 @@ nul
|
||||
tmp/
|
||||
temp/
|
||||
*.tmp
|
||||
|
||||
# E2E test artifacts
|
||||
backend/tests/results/
|
||||
backend/tests/fixtures/reference_voice.wav
|
||||
backend/tests/fixtures/reference_voice.txt
|
||||
|
||||
+155
-1
@@ -7,6 +7,157 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.4.1] - 2026-04-18
|
||||
|
||||
A fast follow-up to 0.4.0 focused on making the new engines actually load in the production binary — plus generation cancellation, Linux system-audio capture, and the repo's first PR-time type check. Five first-time contributors shipped in this release.
|
||||
|
||||
0.4.0 introduced three new TTS engines, but the frozen PyInstaller binary tripped over several Python-ecosystem quirks that don't show up in the dev venv: `transformers` opening `.py` sources at runtime, `scipy.stats._distn_infrastructure` hitting a frozen-importer `NameError`, and `chatterbox-multilingual` failing to find its Chinese segmenter dictionary. This release patches all of those in one sweep.
|
||||
|
||||
### Frozen-Binary Reliability ([#438](https://github.com/jamiepine/voicebox/pull/438))
|
||||
- **Kokoro** now bundles `.py` sources alongside `.pyc` via `--collect-all kokoro` so `transformers`' `_can_set_attn_implementation` regex scan can read them — previously `FileNotFoundError: kokoro/modules.py` killed Kokoro loading in production builds
|
||||
- **Chatterbox Multilingual** now bundles `spacy_pkuseg/dicts/default.pkl` and the package's native `.so` extensions via `--collect-all spacy_pkuseg` — previously the Chinese word segmenter crashed with `FileNotFoundError` on first load
|
||||
- **scipy.stats._distn_infrastructure** — new runtime hook source-patches the trailing `del obj` (which raises `NameError` under PyInstaller's frozen importer because the preceding list comprehension evaluates empty) to `globals().pop('obj', None)`, unblocking `librosa` → `scipy.signal` → `scipy.stats` for every TTS engine that depends on librosa
|
||||
- **transformers.masking_utils** — same runtime hook forces `_is_torch_greater_or_equal_than_2_6 = False` so the older `sdpa_mask_older_torch` path is selected; the 2.6+ path uses `TransformGetItemToIndex()`, a real `torch._dynamo` graph transform our permissive stub can't reproduce
|
||||
- **torch._dynamo** — no-op stub replaces the real module before `transformers` imports it, preventing the `torch._numpy._ufuncs` import crash (`NameError: name 'name' is not defined`) that blocked Kokoro and every engine pulling in `flex_attention`
|
||||
- `.spec` paths are now repo-relative instead of absolute, so the generated spec is portable across machines and CI
|
||||
|
||||
### Generation
|
||||
- **Cancel queued or running generations** ([#444](https://github.com/jamiepine/voicebox/pull/444)) — new `/generate/{id}/cancel` endpoint and a Stop button on the history row while generating. The serial queue now tracks per-ID state (queued / running / cancelled) so queued jobs are skipped before the worker picks them up and running jobs are `.cancel()`-ed mid-flight; `run_generation` catches `CancelledError` and marks the row `failed` with a "cancelled" error.
|
||||
- **Legacy `data/` path prefix resolution** ([#440](https://github.com/jamiepine/voicebox/pull/440)) — generations stored with the old `data/` prefix under pre-0.4 installs now resolve correctly after the storage root moved, fixing 404s for historical audio.
|
||||
|
||||
### Model Migration
|
||||
- Migration dialog no longer hangs when the cache is empty ([#439](https://github.com/jamiepine/voicebox/pull/439)) — the backend now emits a completion SSE event even when zero models are moved.
|
||||
- Storage-change flow surfaces a toast when there's nothing to migrate ([#433](https://github.com/jamiepine/voicebox/pull/433)) instead of proceeding with a no-op move and restarting the server.
|
||||
- Deleting all generations from a voice profile now deletes the associated version files and DB rows too ([#447](https://github.com/jamiepine/voicebox/pull/447)) — previously orphaned versions accumulated in storage.
|
||||
|
||||
### Platform
|
||||
- **Linux system audio capture** ([#457](https://github.com/jamiepine/voicebox/pull/457)) — `cpal`'s ALSA backend doesn't expose PulseAudio/PipeWire monitor sources by name, so the previous device-name search never matched and silently fell back to the microphone. Detection now uses `pactl get-default-sink` + `pactl list short sources` and routes via `PULSE_SOURCE`, with the name-based search retained as a fallback when `pactl` is absent.
|
||||
|
||||
### Frontend CI
|
||||
- First PR-time quality gate ([#418](https://github.com/jamiepine/voicebox/pull/418)) — new `.github/workflows/ci.yml` runs `bun run typecheck` + `bun run build:web` on every PR. Fixed pre-existing type issues that were being suppressed with `@ts-expect-error`, cleaned up a dep-array typo (`[platform.metadata.isTauricheckOnMountcheckForUpdates]`) in `useAutoUpdater`, and removed 100+ lines of dead `ModelItem` code from `ModelManagement.tsx`.
|
||||
- Follow-up: widened `apiClient.migrateModels()` return type to include `moved` and `errors` so the storage-change handler typechecks against the real backend response ([#470](https://github.com/jamiepine/voicebox/pull/470)).
|
||||
|
||||
### Docs
|
||||
- Clarified in the Quick Start + README that paralinguistic tags (`[laugh]`, `[sigh]`) only work with Chatterbox Turbo; other engines read them as literal text ([#450](https://github.com/jamiepine/voicebox/pull/450)).
|
||||
|
||||
### New Contributors
|
||||
- [@Bortlesboat](https://github.com/Bortlesboat) — generation cancellation (#444)
|
||||
- [@gaojulong](https://github.com/gaojulong) — migration dialog hang fix (#439)
|
||||
- [@fuleinist](https://github.com/fuleinist) — migration no-op toast (#433)
|
||||
- [@erionjuniordeandrade-a11y](https://github.com/erionjuniordeandrade-a11y) — frontend CI + type hardening (#418)
|
||||
- [@estefrac](https://github.com/estefrac) — Linux pactl system-audio capture (#457)
|
||||
|
||||
## [0.4.0] - 2026-04-16
|
||||
|
||||
The biggest Voicebox release yet. Three new TTS engines bring the lineup to **seven** — HumeAI TADA, Kokoro 82M, and Qwen CustomVoice join Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo. GPU support broadens to Intel Arc (XPU) and NVIDIA Blackwell (RTX 50-series), with runtime diagnostics that warn when your PyTorch build doesn't match your GPU. The CUDA backend is now split into independently versioned server and library archives, so upgrading no longer redownloads 4 GB of PyTorch/CUDA DLLs.
|
||||
|
||||
This release also marks a big community moment: **13 new contributors** shipped fixes and features in 0.4.0. Thirty-plus bug fixes target the most-reported issues in the tracker — numpy 2.x TTS crashes, Windows background-server reliability, macOS 11 launch failures, audio playback silence, Stories clip-splitting races, history status staleness, and more.
|
||||
|
||||
### New TTS Engines
|
||||
|
||||
#### HumeAI TADA — Expressive English & Multilingual ([#296](https://github.com/jamiepine/voicebox/pull/296))
|
||||
- Added `tada-1b` (English) and `tada-3b-ml` (multilingual) backends
|
||||
- Replaced `descript-audio-codec` with a lightweight DAC shim to cut dependencies
|
||||
- Switched audio decoding to `soundfile` to sidestep `torchcodec` bundling issues
|
||||
- Redirected gated Llama tokenizer lookups to an ungated mirror so model loading works out of the box
|
||||
- Fixed tokenizer patch that was corrupting `AutoTokenizer` for other engines
|
||||
- Fixed TorchScript error in frozen builds
|
||||
|
||||
#### Kokoro 82M — Fast Lightweight TTS ([#325](https://github.com/jamiepine/voicebox/pull/325))
|
||||
- Added Kokoro 82M engine with a new voice profile type system that distinguishes preset voices from cloned profiles
|
||||
- Profile grid now handles engine compatibility directly — removed redundant dropdown filtering
|
||||
- Tightened Kokoro profile handling so preset voices can't be edited like cloned profiles
|
||||
|
||||
#### Qwen CustomVoice ([#328](https://github.com/jamiepine/voicebox/pull/328))
|
||||
- Added `qwen-custom-voice` preset engine backed by Qwen3-TTS
|
||||
- Enforced preset/profile engine compatibility across the generation flow
|
||||
- Floating generator now shows all engines instead of silently filtering
|
||||
|
||||
### Voice Profile UX
|
||||
|
||||
Until 0.4, every engine in Voicebox was a cloning model, so every voice profile was usable with every engine and the profile grid just showed them all. Introducing Kokoro and Qwen CustomVoice — which work from preset voices rather than cloned samples — broke that assumption for the first time. An early cut on `main` filtered the grid by the selected engine, which left users running pre-release builds thinking their cloned voices had vanished whenever they switched to a preset-only engine.
|
||||
|
||||
This release ships the resolution before it ever reaches a tagged version:
|
||||
|
||||
- **Grey-out instead of filter** — all profiles are always visible; unsupported ones render dimmed with a compatibility hint at the bottom of the grid
|
||||
- **Auto-switch on selection** — clicking a greyed-out profile selects it AND switches the engine to a compatible one, instead of silently doing nothing
|
||||
- **Instruct toggle restored for Qwen CustomVoice** — the floating generate box now reveals a delivery-instructions input (tone, emotion, pace) when CustomVoice is selected. Hidden across the board while the new multi-engine lineup was stabilizing because most engines don't honor the kwarg; now conditionally exposed only for the one engine that was actually trained for instruction-based style control
|
||||
- Supported profiles sort first; the grid scrolls the selected profile into view after engine/sort changes
|
||||
- Fixed engine desync on tab navigation — the form now initializes its engine from the store
|
||||
- Fixed the disabled-and-selected card click edge case by bouncing selection to re-trigger the auto-switch
|
||||
- Cleaned up scroll effect timers (requestAnimationFrame + setTimeout) to prevent stale DOM writes on unmount or rapid selection changes
|
||||
|
||||
### GPU & Platform
|
||||
|
||||
#### Intel Arc (XPU) Support ([#320](https://github.com/jamiepine/voicebox/pull/320))
|
||||
- First-class Intel Arc support across all PyTorch-based backends
|
||||
- Device-aware seeding, XPU detection in the GPU status panel, and setup flow detection
|
||||
- Reports correct device name and VRAM in settings
|
||||
|
||||
#### Blackwell / RTX 50-series Support ([#316](https://github.com/jamiepine/voicebox/pull/316), [#401](https://github.com/jamiepine/voicebox/pull/401))
|
||||
- Upgraded the CUDA backend from cu126 → cu128 for RTX 50-series support
|
||||
- Added `sm_120+PTX` to the CUDA build via `TORCH_CUDA_ARCH_LIST` for forward-compatibility with Blackwell architectures (closes 5 open reports: #386, #395, #396, #399, #400)
|
||||
- GPU settings UI fixes around install/uninstall state
|
||||
|
||||
#### GPU Compatibility Diagnostics ([#367](https://github.com/jamiepine/voicebox/pull/367), adapted)
|
||||
- New `check_cuda_compatibility()` compares the current device's compute capability against the bundled PyTorch's architecture list
|
||||
- Health endpoint exposes a `gpu_compatibility_warning` field so the UI can surface mismatches
|
||||
- Startup logs a `WARN` when the installed PyTorch build doesn't support the detected GPU
|
||||
- GPU status label shows `[UNSUPPORTED - see logs]` — no more silent "no kernel image" failures
|
||||
|
||||
#### Split CUDA Backend ([#298](https://github.com/jamiepine/voicebox/pull/298))
|
||||
- CUDA backend now ships as two independently versioned archives: a small server binary and a large libs archive (the ~4 GB of PyTorch/CUDA DLLs)
|
||||
- Upgrading Voicebox no longer redownloads the libs archive when only the server binary changed
|
||||
- Added `asyncio.Lock` around `download_cuda_binary()` so auto-update and manual download can't race on the same temp file ([#428](https://github.com/jamiepine/voicebox/pull/428))
|
||||
- Updated `package_cuda.py` for PyInstaller 6.18 onedir layout
|
||||
- Temp archives are always cleaned up on failure, even when the install aborts mid-extract
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
#### Critical: TTS Generation
|
||||
- **numpy 2.x `torch.from_numpy` crash** ([#361](https://github.com/jamiepine/voicebox/pull/361)) — torch compiled against numpy 1.x ABI fails silently when paired with numpy 2.x, causing `RuntimeError: Numpy is not available` / `Unable to create tensor` on every TTS request in bundled macOS Intel / Rosetta builds. Pinned `numpy<2.0` in requirements and added a PyInstaller runtime hook with a `ctypes.memmove` fallback as belt-and-suspenders. Hardened afterward to raise on unknown dtypes instead of silently reinterpreting bytes as float32.
|
||||
|
||||
#### Platform Reliability
|
||||
- **Windows background server** ([#402](https://github.com/jamiepine/voicebox/pull/402)) — "keep server running after close" now actually keeps the server running. The HTTP `/watchdog/disable` request could lose the race against process exit on Windows; added a `.keep-running` sentinel file as a synchronous fallback, with stale-sentinel cleanup on startup to avoid orphan server processes
|
||||
- **macOS 11 launch crash** ([#424](https://github.com/jamiepine/voicebox/pull/424)) — weak-linked ScreenCaptureKit so the app can launch on macOS < 12.3 instead of crashing at dyld resolution. Gated system audio capture behind a real `sw_vers` version check so unsupported systems cleanly advertise "not available" rather than crashing at runtime
|
||||
- **macOS Intel (x86_64) setup** ([#416](https://github.com/jamiepine/voicebox/pull/416)) — relaxed `torch>=2.7.0` → `torch>=2.2.0`. PyTorch dropped pre-built x86_64 wheels after 2.2.2, so Intel Mac devs could no longer `pip install`. Now resolves to the latest compatible torch per platform
|
||||
- **Offline model loading** ([#318](https://github.com/jamiepine/voicebox/pull/318)) — Qwen TTS and Whisper force offline mode when loading cached models, so startup works without network access
|
||||
- **GUI startup with external server** ([#319](https://github.com/jamiepine/voicebox/pull/319)) — fixed GUI launch when pointed at a remote/external server, and added data refresh on server switch; hardened health validation and error handling
|
||||
- **Qwen3-TTS cache split on Windows** (adapted from [#218](https://github.com/jamiepine/voicebox/pull/218)) — route `Qwen3TTSModel.from_pretrained` through `hf_constants.HF_HUB_CACHE` so the speech tokenizer and `preprocessor_config.json` resolve from a single cache root
|
||||
- **Qwen3-TTS bundling** ([#305](https://github.com/jamiepine/voicebox/pull/305)) — bundle `qwen_tts` source files in the PyInstaller build to fix `inspect.getsource` errors in frozen builds
|
||||
- **Backend import paths** ([#345](https://github.com/jamiepine/voicebox/pull/345)) — moved lazy imports to top-level with absolute paths to resolve the "Failed to Save" preset error caused by `ModuleNotFoundError` in production builds
|
||||
- **Effects service import** ([#384](https://github.com/jamiepine/voicebox/pull/384)) — fixed `ModuleNotFoundError` on preset create/update by switching to relative imports (#349)
|
||||
|
||||
#### Audio & Playback
|
||||
- **cpal stream silent playback** ([#405](https://github.com/jamiepine/voicebox/pull/405)) — `cpal::Stream` was dropped on function return immediately after `play()`, causing every playback to fall silent. Now holds the stream until either the buffer drains or the stop flag fires (#404)
|
||||
|
||||
#### Stories & History
|
||||
- **Clip-splitting race** ([#403](https://github.com/jamiepine/voicebox/pull/403)) — rapid double-clicks on split could race through `split_story_item` with inconsistent state. Added `with_for_update()` row locking on the backend and an `isPending` guard on the frontend (#366)
|
||||
- **History `status` staleness** ([#394](https://github.com/jamiepine/voicebox/pull/394)) — `GET /history/{id}` was hardcoding `status="completed"` regardless of the DB row, breaking any client polling for job completion. Now returns `status`, `error`, `engine`, `model_size`, and `is_favorited` from the actual row
|
||||
- **"Clear failed" bulk button** ([#412](https://github.com/jamiepine/voicebox/pull/412)) — new `DELETE /history/failed` endpoint and a header strip showing `"N failed generations"` with a Clear button, complementing the per-row trash icon added in #321 (#410)
|
||||
- **Delete failed generations** ([#321](https://github.com/jamiepine/voicebox/pull/321)) — added a trash icon next to the retry button so failed entries can be cleaned up without having to retry first
|
||||
|
||||
#### Security & Safety
|
||||
- **Voice prompt cache hardening** ([#429](https://github.com/jamiepine/voicebox/pull/429)) — `torch.load(weights_only=True)` on cached voice prompts per PyTorch 2.6 recommendation; replaced string-based SPA path guard with `Path.is_relative_to()` for more robust path-traversal protection
|
||||
|
||||
#### Infrastructure & Docker
|
||||
- **Docker web build** ([#344](https://github.com/jamiepine/voicebox/pull/344)) — include `CHANGELOG.md` in the Docker web build so the in-app changelog page works in Docker deployments
|
||||
- **Docker numba cache** ([#425](https://github.com/jamiepine/voicebox/pull/425)) — set `NUMBA_CACHE_DIR` in docker-compose so numba can write its JIT cache in container runtime (#308)
|
||||
- **Relative media paths** ([#332](https://github.com/jamiepine/voicebox/pull/332)) — media paths now stored relative to the configured data dir rather than resolved against CWD, so the data directory is portable between installs
|
||||
|
||||
### Developer Tooling
|
||||
|
||||
- New `triage-prs` agent skill — encodes the end-to-end PR-speedrun workflow (classification → triage doc → rebase → squash-merge → follow-ups) so future release cycles can reproduce it
|
||||
- Rewrote the TTS engine guide with the patterns learned from adding TADA and Kokoro
|
||||
- Added the API refactor plan and CUDA libs addon design doc
|
||||
- Fixed broken links in the Get Started section ([#332](https://github.com/jamiepine/voicebox/pull/332))
|
||||
|
||||
### New Contributors
|
||||
|
||||
Huge thank you to everyone who contributed their first PR to Voicebox in this release:
|
||||
|
||||
[@liorshahverdi](https://github.com/liorshahverdi), [@nicoschtein](https://github.com/nicoschtein), [@ArfianID](https://github.com/ArfianID), [@aimaaaimaa](https://github.com/aimaaaimaa), [@maxmcoding](https://github.com/maxmcoding), [@Khalodddd](https://github.com/Khalodddd), [@LuisSambrano](https://github.com/LuisSambrano), [@shaun0927](https://github.com/shaun0927), [@malletfils](https://github.com/malletfils), [@mvanhorn](https://github.com/mvanhorn), [@kuishou68](https://github.com/kuishou68), [@txhno](https://github.com/txhno), [@MukundaKatta](https://github.com/MukundaKatta)
|
||||
|
||||
## [0.3.0] - 2026-03-17
|
||||
|
||||
This release rewrites the backend into a modular architecture, overhauls the settings UI into routed sub-pages, fixes audio player freezing, migrates documentation to Fumadocs, and ships a batch of bug fixes targeting the most-reported issues from the tracker.
|
||||
@@ -444,7 +595,10 @@ The first public release of Voicebox — an open-source voice synthesis studio p
|
||||
|
||||
Tauri v2, React, TypeScript, Tailwind CSS, FastAPI, Qwen3-TTS, Whisper, SQLite
|
||||
|
||||
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.2.3...HEAD
|
||||
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.4.1...HEAD
|
||||
[0.4.1]: https://github.com/jamiepine/voicebox/compare/v0.4.0...v0.4.1
|
||||
[0.4.0]: https://github.com/jamiepine/voicebox/compare/v0.3.0...v0.4.0
|
||||
[0.3.0]: https://github.com/jamiepine/voicebox/compare/v0.2.3...v0.3.0
|
||||
[0.2.3]: https://github.com/jamiepine/voicebox/compare/v0.2.2...v0.2.3
|
||||
[0.2.2]: https://github.com/jamiepine/voicebox/compare/v0.2.1...v0.2.2
|
||||
[0.2.1]: https://github.com/jamiepine/voicebox/compare/v0.1.13...v0.2.1
|
||||
|
||||
+2
-1
@@ -260,7 +260,7 @@ voicebox/
|
||||
|
||||
### ✨ New Features
|
||||
|
||||
- Check the roadmap in README.md
|
||||
- Check the roadmap in README.md and the engineering status in [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) before proposing work — it lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
|
||||
- Discuss major features in an issue first
|
||||
- Keep features focused and well-scoped
|
||||
|
||||
@@ -378,6 +378,7 @@ See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and sol
|
||||
|
||||
- [README.md](README.md) - Project overview
|
||||
- [backend/README.md](backend/README.md) - API documentation
|
||||
- [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) - Living engineering roadmap: architecture, shipped vs in-flight work, prioritized open issues, candidate TTS engines under evaluation, architectural bottlenecks. Keep this updated when you ship significant features, close or backlog a model integration, or identify new bottlenecks.
|
||||
- [docs/AUTOUPDATER_QUICKSTART.md](docs/AUTOUPDATER_QUICKSTART.md) - Auto-updater setup
|
||||
- [SECURITY.md](SECURITY.md) - Security policy
|
||||
- [CHANGELOG.md](CHANGELOG.md) - Version history
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ FROM oven/bun:1 AS frontend
|
||||
WORKDIR /build
|
||||
|
||||
# Copy workspace config and frontend source
|
||||
COPY package.json bun.lock ./
|
||||
COPY package.json bun.lock CHANGELOG.md ./
|
||||
COPY app/ ./app/
|
||||
COPY web/ ./web/
|
||||
|
||||
|
||||
@@ -23,6 +23,9 @@
|
||||
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
|
||||
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
|
||||
</a>
|
||||
<a href="https://deepwiki.com/jamiepine/voicebox">
|
||||
<img src="https://img.shields.io/static/v1?label=Ask&message=DeepWiki&color=5B6EF7" alt="Ask DeepWiki" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
@@ -59,13 +62,14 @@
|
||||
|
||||
## What is Voicebox?
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
|
||||
- **Complete privacy** — models and voice data stay on your machine
|
||||
- **5 TTS engines** — Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
|
||||
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** — zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
|
||||
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
||||
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** — auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** — multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** — REST API for integrating voice synthesis into your own projects
|
||||
@@ -93,19 +97,26 @@ Voicebox is a **local-first voice cloning studio** — a free and open-source al
|
||||
|
||||
### Multi-Engine Voice Cloning
|
||||
|
||||
Five TTS engines with different strengths, switchable per-generation:
|
||||
Seven TTS engines with different strengths, switchable per-generation:
|
||||
|
||||
| Engine | Languages | Strengths |
|
||||
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions ("speak slowly", "whisper") |
|
||||
| **Qwen CustomVoice** | 10 | 9 curated preset voices with natural-language delivery control — no reference audio required |
|
||||
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
|
||||
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
|
||||
| **Kokoro** | 8 | 50 curated preset voices, tiny 82M model, fast CPU inference |
|
||||
|
||||
### Emotions & Paralinguistic Tags
|
||||
|
||||
Type `/` in the text input to insert expressive tags that the model synthesizes inline with speech (Chatterbox Turbo):
|
||||
Only **Chatterbox Turbo** interprets paralinguistic tags like `[laugh]` and
|
||||
`[sigh]`. Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and HumeAI TADA read them
|
||||
literally as text.
|
||||
|
||||
With **Chatterbox Turbo** selected, type `/` in the text input to open the tag
|
||||
inserter and add expressive tags inline with speech:
|
||||
|
||||
`[laugh]` `[chuckle]` `[gasp]` `[cough]` `[sigh]` `[groan]` `[sniff]` `[shush]` `[clear throat]`
|
||||
|
||||
@@ -231,7 +242,7 @@ Full API documentation available at `http://localhost:17493/docs`.
|
||||
| Frontend | React, TypeScript, Tailwind CSS |
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| TTS Engines | Qwen3-TTS, LuxTTS, Chatterbox, Chatterbox Turbo, TADA |
|
||||
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
|
||||
| Effects | Pedalboard (Spotify) |
|
||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||
@@ -250,6 +261,8 @@ Full API documentation available at `http://localhost:17493/docs`.
|
||||
| **Plugin Architecture** | Extend with custom models and effects |
|
||||
| **Mobile Companion** | Control Voicebox from your phone |
|
||||
|
||||
For the **full engineering status, open-issue triage, and prioritized work queue**, see [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) — a living document that tracks what's shipped, what's in-flight, candidate TTS engines under evaluation, and why we've accepted or backlogged specific integrations.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
+3
-3
@@ -6,8 +6,8 @@ We release patches for security vulnerabilities. Which versions are eligible for
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | ------------------ |
|
||||
| 0.1.x | :white_check_mark: |
|
||||
| < 0.1 | :x: |
|
||||
| 0.3.x | :white_check_mark: |
|
||||
| < 0.3 | :x: |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
@@ -82,7 +82,7 @@ Timeline may vary based on severity and complexity.
|
||||
## Security Updates
|
||||
|
||||
Security updates will be:
|
||||
- Released as patch versions (e.g., 0.1.1)
|
||||
- Released as patch versions (e.g., 0.3.2)
|
||||
- Documented in CHANGELOG.md
|
||||
- Announced via GitHub releases
|
||||
- Automatically delivered via auto-updater
|
||||
|
||||
+2
-1
@@ -1,11 +1,12 @@
|
||||
{
|
||||
"name": "@voicebox/app",
|
||||
"version": "0.3.1",
|
||||
"version": "0.4.2",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit",
|
||||
"preview": "vite preview",
|
||||
"lint": "biome lint src",
|
||||
"lint:fix": "biome lint --write src",
|
||||
|
||||
+98
-12
@@ -4,6 +4,8 @@ import voiceboxLogo from '@/assets/voicebox-logo.png';
|
||||
import ShinyText from '@/components/ShinyText';
|
||||
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
||||
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { HealthResponse } from '@/lib/api/types';
|
||||
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
@@ -11,6 +13,33 @@ import { router } from '@/router';
|
||||
import { useLogStore } from '@/stores/logStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
|
||||
/**
|
||||
* Validate that a health response has the expected Voicebox-specific shape.
|
||||
* Prevents misidentifying an unrelated service on the same port.
|
||||
*/
|
||||
function isVoiceboxHealthResponse(health: HealthResponse): boolean {
|
||||
return (
|
||||
health?.status === 'healthy' &&
|
||||
typeof health.model_loaded === 'boolean' &&
|
||||
typeof health.gpu_available === 'boolean'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check whether a startup error indicates the port is occupied by an external
|
||||
* server (which we should try to reuse via health-check polling) vs. a real
|
||||
* failure (missing sidecar, signing issue, etc.) that should surface immediately.
|
||||
*/
|
||||
function isPortInUseError(error: unknown): boolean {
|
||||
const msg = error instanceof Error ? error.message : String(error);
|
||||
return (
|
||||
msg.includes('already in use') ||
|
||||
msg.includes('port') ||
|
||||
msg.includes('EADDRINUSE') ||
|
||||
msg.includes('address already in use')
|
||||
);
|
||||
}
|
||||
|
||||
const LOADING_MESSAGES = [
|
||||
'Warming up tensors...',
|
||||
'Calibrating synthesizer engine...',
|
||||
@@ -37,6 +66,7 @@ const LOADING_MESSAGES = [
|
||||
function App() {
|
||||
const platform = usePlatform();
|
||||
const [serverReady, setServerReady] = useState(false);
|
||||
const [startupError, setStartupError] = useState<string | null>(null);
|
||||
const [loadingMessageIndex, setLoadingMessageIndex] = useState(0);
|
||||
const serverStartingRef = useRef(false);
|
||||
|
||||
@@ -91,7 +121,6 @@ function App() {
|
||||
console.log('Dev mode: Skipping auto-start of server (run it separately)');
|
||||
setServerReady(true); // Mark as ready so UI doesn't show loading screen
|
||||
// Mark that server was not started by app (so we don't try to stop it on close)
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = false;
|
||||
return;
|
||||
}
|
||||
@@ -114,14 +143,52 @@ function App() {
|
||||
useServerStore.getState().setServerUrl(serverUrl);
|
||||
setServerReady(true);
|
||||
// Mark that we started the server (so we know to stop it on close)
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = true;
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error('Failed to auto-start server:', error);
|
||||
serverStartingRef.current = false;
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = false;
|
||||
|
||||
// Only fall back to health-check polling when the error indicates the
|
||||
// port is occupied (likely an external server). For real failures
|
||||
// (missing sidecar, signing issues, etc.) surface the error immediately.
|
||||
if (!isPortInUseError(error)) {
|
||||
const msg = error instanceof Error ? error.message : String(error);
|
||||
console.error('Real startup failure — not polling:', msg);
|
||||
setStartupError(msg);
|
||||
return;
|
||||
}
|
||||
|
||||
// Fall back to polling: the server may already be running externally
|
||||
// (e.g. started via python/uvicorn/Docker). Poll the health endpoint
|
||||
// until it responds with a valid Voicebox payload, then transition to
|
||||
// the main UI.
|
||||
console.log('Falling back to health-check polling...');
|
||||
const pollInterval = setInterval(async () => {
|
||||
try {
|
||||
const health = await apiClient.getHealth();
|
||||
if (!isVoiceboxHealthResponse(health)) {
|
||||
console.log('Health response is not from a Voicebox server, keep polling...');
|
||||
return;
|
||||
}
|
||||
console.log('External Voicebox server detected via health check');
|
||||
clearInterval(pollInterval);
|
||||
setServerReady(true);
|
||||
} catch {
|
||||
// Server not ready yet, keep polling
|
||||
}
|
||||
}, 2000);
|
||||
|
||||
// Stop polling after 2 minutes and surface the failure
|
||||
setTimeout(() => {
|
||||
clearInterval(pollInterval);
|
||||
serverStartingRef.current = false;
|
||||
setStartupError(
|
||||
'Could not connect to a Voicebox server within 2 minutes. ' +
|
||||
'Please check that the server is running and try again.',
|
||||
);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
// Cleanup: stop server on actual unmount (not StrictMode remount)
|
||||
@@ -168,15 +235,34 @@ function App() {
|
||||
className="w-48 h-48 object-contain animate-fade-in-scale relative z-10"
|
||||
/>
|
||||
</div>
|
||||
<div className="animate-fade-in-delayed">
|
||||
<ShinyText
|
||||
text={LOADING_MESSAGES[loadingMessageIndex]}
|
||||
className="text-lg font-medium text-muted-foreground"
|
||||
speed={2}
|
||||
color="hsl(var(--muted-foreground))"
|
||||
shineColor="hsl(var(--foreground))"
|
||||
/>
|
||||
</div>
|
||||
{startupError ? (
|
||||
<div className="animate-fade-in-delayed max-w-md mx-auto space-y-3">
|
||||
<p className="text-lg font-medium text-destructive">Server startup failed</p>
|
||||
<p className="text-sm text-muted-foreground">{startupError}</p>
|
||||
<button
|
||||
type="button"
|
||||
className="mt-2 px-4 py-2 text-sm rounded-md bg-primary text-primary-foreground hover:bg-primary/90 transition-colors"
|
||||
onClick={() => {
|
||||
setStartupError(null);
|
||||
serverStartingRef.current = false;
|
||||
// Trigger a re-mount of the effect by toggling state
|
||||
window.location.reload();
|
||||
}}
|
||||
>
|
||||
Retry
|
||||
</button>
|
||||
</div>
|
||||
) : (
|
||||
<div className="animate-fade-in-delayed">
|
||||
<ShinyText
|
||||
text={LOADING_MESSAGES[loadingMessageIndex]}
|
||||
className="text-lg font-medium text-muted-foreground"
|
||||
speed={2}
|
||||
color="hsl(var(--muted-foreground))"
|
||||
shineColor="hsl(var(--foreground))"
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { useRouterState } from '@tanstack/react-router';
|
||||
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
||||
import { AudioKeepAlive } from '@/components/AudioPlayer/AudioKeepAlive';
|
||||
import { AudioPlayer } from '@/components/AudioPlayer/AudioPlayer';
|
||||
import { StoryTrackEditor } from '@/components/StoriesTab/StoryTrackEditor';
|
||||
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
@@ -14,16 +15,19 @@ interface AppFrameProps {
|
||||
export function AppFrame({ children }: AppFrameProps) {
|
||||
const routerState = useRouterState();
|
||||
const isStoriesRoute = routerState.location.pathname === '/stories';
|
||||
|
||||
|
||||
const selectedStoryId = useStoryStore((state) => state.selectedStoryId);
|
||||
const { data: story } = useStory(selectedStoryId);
|
||||
|
||||
|
||||
// Show track editor when on stories route with a selected story that has items
|
||||
const showTrackEditor = isStoriesRoute && selectedStoryId && story && story.items.length > 0;
|
||||
|
||||
return (
|
||||
<div className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}>
|
||||
<div
|
||||
className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}
|
||||
>
|
||||
<TitleBarDragRegion />
|
||||
<AudioKeepAlive />
|
||||
{children}
|
||||
{showTrackEditor ? (
|
||||
<StoryTrackEditor storyId={story.id} items={story.items} />
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import { useEffect, useRef } from 'react';
|
||||
import { debug } from '@/lib/utils/debug';
|
||||
|
||||
// WKWebView tears down the app's CoreAudio output when idle for long enough,
|
||||
// and a JS-level reload (cmd+R) does NOT restore it — only relaunching the
|
||||
// Tauri app does. Keeping a silent <audio> element looping forever prevents
|
||||
// the OS audio session from ever going dormant.
|
||||
//
|
||||
// Real silence (zero PCM samples) at full volume is preferred over a muted
|
||||
// element: browsers/WebKit can optimize muted media away, which defeats the
|
||||
// purpose of holding the session open.
|
||||
|
||||
function buildSilentWavUrl(seconds = 1, sampleRate = 8000): string {
|
||||
const numSamples = seconds * sampleRate;
|
||||
const bytes = 44 + numSamples * 2;
|
||||
const buffer = new ArrayBuffer(bytes);
|
||||
const view = new DataView(buffer);
|
||||
const write = (offset: number, str: string) => {
|
||||
for (let i = 0; i < str.length; i++) view.setUint8(offset + i, str.charCodeAt(i));
|
||||
};
|
||||
write(0, 'RIFF');
|
||||
view.setUint32(4, bytes - 8, true);
|
||||
write(8, 'WAVE');
|
||||
write(12, 'fmt ');
|
||||
view.setUint32(16, 16, true);
|
||||
view.setUint16(20, 1, true);
|
||||
view.setUint16(22, 1, true);
|
||||
view.setUint32(24, sampleRate, true);
|
||||
view.setUint32(28, sampleRate * 2, true);
|
||||
view.setUint16(32, 2, true);
|
||||
view.setUint16(34, 16, true);
|
||||
write(36, 'data');
|
||||
view.setUint32(40, numSamples * 2, true);
|
||||
return URL.createObjectURL(new Blob([buffer], { type: 'audio/wav' }));
|
||||
}
|
||||
|
||||
export function AudioKeepAlive() {
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const url = buildSilentWavUrl(1, 8000);
|
||||
const el = new Audio(url);
|
||||
el.loop = true;
|
||||
el.volume = 1;
|
||||
el.preload = 'auto';
|
||||
audioRef.current = el;
|
||||
|
||||
const tryPlay = () => {
|
||||
if (!audioRef.current) return;
|
||||
if (!audioRef.current.paused) return;
|
||||
audioRef.current.play().catch((err) => {
|
||||
debug.log('[AudioKeepAlive] play blocked (will retry on next gesture):', err);
|
||||
});
|
||||
};
|
||||
|
||||
tryPlay();
|
||||
|
||||
// Autoplay may be blocked until first user interaction — re-attempt then.
|
||||
const onGesture = () => tryPlay();
|
||||
window.addEventListener('pointerdown', onGesture, { once: false });
|
||||
window.addEventListener('keydown', onGesture, { once: false });
|
||||
|
||||
// If the webview ever pauses the element on background, resume on return.
|
||||
const onWake = () => {
|
||||
if (!document.hidden) tryPlay();
|
||||
};
|
||||
document.addEventListener('visibilitychange', onWake);
|
||||
window.addEventListener('focus', onWake);
|
||||
window.addEventListener('pageshow', onWake);
|
||||
|
||||
return () => {
|
||||
window.removeEventListener('pointerdown', onGesture);
|
||||
window.removeEventListener('keydown', onGesture);
|
||||
document.removeEventListener('visibilitychange', onWake);
|
||||
window.removeEventListener('focus', onWake);
|
||||
window.removeEventListener('pageshow', onWake);
|
||||
el.pause();
|
||||
el.src = '';
|
||||
URL.revokeObjectURL(url);
|
||||
audioRef.current = null;
|
||||
};
|
||||
}, []);
|
||||
|
||||
return null;
|
||||
}
|
||||
@@ -124,7 +124,7 @@ export function AudioTab() {
|
||||
);
|
||||
}
|
||||
|
||||
const handleChannelDelete = async (e, channelId) => {
|
||||
const handleChannelDelete = async (e: React.MouseEvent, channelId: string) => {
|
||||
e.stopPropagation();
|
||||
if (await confirm('Delete this channel?')) {
|
||||
deleteChannel.mutate(channelId);
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
import {EffectsDetail} from "./EffectsDetail";
|
||||
import {EffectsList} from "./EffectsList";
|
||||
import { EffectsDetail } from './EffectsDetail';
|
||||
import { EffectsList } from './EffectsList';
|
||||
|
||||
export function EffectsTab() {
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 overflow-hidden">
|
||||
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
|
||||
{/* Left - Presets list */}
|
||||
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
|
||||
<EffectsList />
|
||||
</div>
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 overflow-hidden">
|
||||
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
|
||||
{/* Left - Presets list */}
|
||||
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
|
||||
<EffectsList />
|
||||
</div>
|
||||
|
||||
{/* Right - Detail / editor */}
|
||||
<div className="flex-1 min-h-0 flex flex-col">
|
||||
<EffectsDetail />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
{/* Right - Detail / editor */}
|
||||
<div className="flex-1 min-h-0 flex flex-col">
|
||||
<EffectsDetail />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { useMatchRoute } from '@tanstack/react-router';
|
||||
import { AnimatePresence, motion } from 'framer-motion';
|
||||
import { Loader2, Sparkles } from 'lucide-react';
|
||||
import { Loader2, SlidersHorizontal, Sparkles } from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Form, FormControl, FormField, FormItem, FormMessage } from '@/components/ui/form';
|
||||
@@ -40,6 +40,7 @@ export function FloatingGenerateBox({
|
||||
const { data: selectedProfile } = useProfile(selectedProfileId || '');
|
||||
const { data: profiles } = useProfiles();
|
||||
const [isExpanded, setIsExpanded] = useState(false);
|
||||
const [isInstructExpanded, setIsInstructExpanded] = useState(false);
|
||||
const [selectedPresetId, setSelectedPresetId] = useState<string | null>(null);
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const textareaRef = useRef<HTMLTextAreaElement | null>(null);
|
||||
@@ -125,22 +126,29 @@ export function FloatingGenerateBox({
|
||||
}, [watchedEngine, setSelectedEngine]);
|
||||
|
||||
// Sync generation form language, engine, and effects with selected profile
|
||||
type EngineValue =
|
||||
| 'qwen'
|
||||
| 'luxtts'
|
||||
| 'chatterbox'
|
||||
| 'chatterbox_turbo'
|
||||
| 'tada'
|
||||
| 'kokoro'
|
||||
| 'qwen_custom_voice';
|
||||
useEffect(() => {
|
||||
if (selectedProfile?.language) {
|
||||
form.setValue('language', selectedProfile.language as LanguageCode);
|
||||
}
|
||||
// Auto-switch engine if profile has a default
|
||||
if (selectedProfile?.default_engine) {
|
||||
form.setValue(
|
||||
'engine',
|
||||
selectedProfile.default_engine as
|
||||
| 'qwen'
|
||||
| 'luxtts'
|
||||
| 'chatterbox'
|
||||
| 'chatterbox_turbo'
|
||||
| 'tada'
|
||||
| 'kokoro',
|
||||
);
|
||||
// Auto-switch engine to match the profile
|
||||
const engine = selectedProfile?.default_engine ?? selectedProfile?.preset_engine;
|
||||
if (engine) {
|
||||
form.setValue('engine', engine as EngineValue);
|
||||
} else if (selectedProfile && selectedProfile.voice_type !== 'preset') {
|
||||
// Cloned/designed profile with no default — ensure a compatible (non-preset) engine
|
||||
const currentEngine = form.getValues('engine');
|
||||
const presetEngines = new Set(['kokoro', 'qwen_custom_voice']);
|
||||
if (currentEngine && presetEngines.has(currentEngine)) {
|
||||
form.setValue('engine', 'qwen');
|
||||
}
|
||||
}
|
||||
// Pre-fill effects from profile defaults
|
||||
if (
|
||||
@@ -346,9 +354,80 @@ export function FloatingGenerateBox({
|
||||
: 'Generate speech'}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Instruct toggle — only for Qwen CustomVoice, which actually honors the kwarg */}
|
||||
<AnimatePresence>
|
||||
{isExpanded && form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<motion.div
|
||||
initial={{ opacity: 0, scale: 0.8 }}
|
||||
animate={{ opacity: 1, scale: 1 }}
|
||||
exit={{ opacity: 0, scale: 0.8 }}
|
||||
transition={{ duration: 0.2 }}
|
||||
className="absolute top-0 right-[calc(100%+0.5rem)]"
|
||||
>
|
||||
<div className="group relative">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => setIsInstructExpanded((prev) => !prev)}
|
||||
className={cn(
|
||||
'h-10 w-10 rounded-full transition-all duration-200',
|
||||
isInstructExpanded
|
||||
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
|
||||
: 'bg-card border border-border hover:bg-background/50',
|
||||
)}
|
||||
aria-label={
|
||||
isInstructExpanded
|
||||
? 'Hide delivery instructions'
|
||||
: 'Show delivery instructions'
|
||||
}
|
||||
aria-pressed={isInstructExpanded}
|
||||
>
|
||||
<SlidersHorizontal className="h-4 w-4" />
|
||||
</Button>
|
||||
<span className="pointer-events-none absolute bottom-full left-1/2 -translate-x-1/2 mb-2 whitespace-nowrap rounded-md bg-popover px-3 py-1.5 text-xs text-popover-foreground border border-border opacity-0 transition-opacity group-hover:opacity-100 z-[9999]">
|
||||
Delivery instructions (tone, emotion, pace)
|
||||
</span>
|
||||
</div>
|
||||
</motion.div>
|
||||
)}
|
||||
</AnimatePresence>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Additive instruct textarea — shown below main text when toggle is on and engine supports it */}
|
||||
<AnimatePresence>
|
||||
{isInstructExpanded && form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<motion.div
|
||||
initial={{ opacity: 0, height: 0 }}
|
||||
animate={{ opacity: 1, height: 'auto' }}
|
||||
exit={{ opacity: 0, height: 0 }}
|
||||
transition={{ duration: 0.2, ease: 'easeOut' }}
|
||||
className="overflow-hidden"
|
||||
>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
render={({ field }) => (
|
||||
<FormItem className="mt-2">
|
||||
<FormControl>
|
||||
<Textarea
|
||||
{...field}
|
||||
placeholder="Delivery instructions — e.g. Speak slowly with warmth, Authoritative and clear..."
|
||||
className="resize-none bg-transparent border border-accent/20 focus-visible:ring-1 focus-visible:ring-accent/40 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full px-3 py-2"
|
||||
style={{ minHeight: '60px', maxHeight: '160px' }}
|
||||
maxLength={500}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</motion.div>
|
||||
)}
|
||||
</AnimatePresence>
|
||||
|
||||
<AnimatePresence>
|
||||
<motion.div
|
||||
initial={{ height: 0, opacity: 0 }}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { useEffect } from 'react';
|
||||
import { Loader2, Mic } from 'lucide-react';
|
||||
import { useEffect } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
import {
|
||||
@@ -24,7 +24,11 @@ import { getLanguageOptionsForEngine, type LanguageCode } from '@/lib/constants/
|
||||
import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
|
||||
import { useProfile } from '@/lib/hooks/useProfiles';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
import { EngineModelSelector, applyEngineSelection, getEngineDescription } from './EngineModelSelector';
|
||||
import {
|
||||
applyEngineSelection,
|
||||
EngineModelSelector,
|
||||
getEngineDescription,
|
||||
} from './EngineModelSelector';
|
||||
import { ParalinguisticInput } from './ParalinguisticInput';
|
||||
|
||||
function getEngineSelectValue(engine: string): string {
|
||||
@@ -114,7 +118,7 @@ export function GenerationForm() {
|
||||
)}
|
||||
/>
|
||||
|
||||
{(form.watch('engine') === 'qwen' || form.watch('engine') === 'qwen_custom_voice') && (
|
||||
{form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
|
||||
@@ -1,15 +1,14 @@
|
||||
import { useQueryClient } from '@tanstack/react-query';
|
||||
import { useMutation, useQueryClient } from '@tanstack/react-query';
|
||||
import { AnimatePresence, motion } from 'framer-motion';
|
||||
import {
|
||||
AlignCenter,
|
||||
AudioLines,
|
||||
AudioWaveform,
|
||||
Download,
|
||||
FileArchive,
|
||||
Loader2,
|
||||
MoreHorizontal,
|
||||
Play,
|
||||
RotateCcw,
|
||||
Square,
|
||||
Star,
|
||||
Trash2,
|
||||
Wand2,
|
||||
@@ -45,6 +44,7 @@ import { apiClient } from '@/lib/api/client';
|
||||
import type { EffectConfig, GenerationVersionResponse, HistoryResponse } from '@/lib/api/types';
|
||||
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
import {
|
||||
useClearFailedGenerations,
|
||||
useDeleteGeneration,
|
||||
useExportGeneration,
|
||||
useExportGenerationAudio,
|
||||
@@ -124,9 +124,28 @@ export function HistoryTable() {
|
||||
});
|
||||
|
||||
const deleteGeneration = useDeleteGeneration();
|
||||
const clearFailed = useClearFailedGenerations();
|
||||
const [clearFailedDialogOpen, setClearFailedDialogOpen] = useState(false);
|
||||
const exportGeneration = useExportGeneration();
|
||||
const exportGenerationAudio = useExportGenerationAudio();
|
||||
const importGeneration = useImportGeneration();
|
||||
const cancelGeneration = useMutation({
|
||||
mutationFn: (generationId: string) => apiClient.cancelGeneration(generationId),
|
||||
onSuccess: async (data) => {
|
||||
await queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
toast({
|
||||
title: 'Cancelling generation',
|
||||
description: data.message,
|
||||
});
|
||||
},
|
||||
onError: (error) => {
|
||||
toast({
|
||||
title: 'Cancel failed',
|
||||
description: error instanceof Error ? error.message : 'Could not cancel generation',
|
||||
variant: 'destructive',
|
||||
});
|
||||
},
|
||||
});
|
||||
const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
|
||||
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
|
||||
const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio);
|
||||
@@ -157,11 +176,11 @@ export function HistoryTable() {
|
||||
const pendingCount = useGenerationStore((state) => state.pendingGenerationIds.size);
|
||||
const prevPendingCountRef = useRef(pendingCount);
|
||||
useEffect(() => {
|
||||
if (deleteGeneration.isSuccess || importGeneration.isSuccess) {
|
||||
if (deleteGeneration.isSuccess || importGeneration.isSuccess || clearFailed.isSuccess) {
|
||||
setPage(0);
|
||||
setAllHistory([]);
|
||||
}
|
||||
}, [deleteGeneration.isSuccess, importGeneration.isSuccess]);
|
||||
}, [deleteGeneration.isSuccess, importGeneration.isSuccess, clearFailed.isSuccess]);
|
||||
|
||||
useEffect(() => {
|
||||
// A generation finished (pending count decreased) — scroll back to show it
|
||||
@@ -415,6 +434,27 @@ export function HistoryTable() {
|
||||
|
||||
const history = allHistory;
|
||||
const hasMore = allHistory.length < total;
|
||||
const failedCount = history.filter((g) => g.status === 'failed').length;
|
||||
|
||||
const handleClearFailedConfirm = () => {
|
||||
clearFailed.mutate(undefined, {
|
||||
onSuccess: (data) => {
|
||||
setClearFailedDialogOpen(false);
|
||||
toast({
|
||||
title: 'Cleared failed generations',
|
||||
description: `${data.deleted} failed ${data.deleted === 1 ? 'generation' : 'generations'} removed.`,
|
||||
});
|
||||
},
|
||||
onError: (error) => {
|
||||
setClearFailedDialogOpen(false);
|
||||
toast({
|
||||
title: 'Failed to clear',
|
||||
description: error instanceof Error ? error.message : 'Unknown error',
|
||||
variant: 'destructive',
|
||||
});
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 relative">
|
||||
@@ -424,6 +464,23 @@ export function HistoryTable() {
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{failedCount > 0 && (
|
||||
<div className="flex items-center justify-between px-1 pb-2">
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{failedCount} failed {failedCount === 1 ? 'generation' : 'generations'}
|
||||
</span>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="h-7 text-xs text-muted-foreground hover:text-destructive"
|
||||
onClick={() => setClearFailedDialogOpen(true)}
|
||||
disabled={clearFailed.isPending}
|
||||
>
|
||||
<Trash2 className="h-3 w-3 mr-1.5" />
|
||||
{clearFailed.isPending ? 'Clearing...' : 'Clear failed'}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
{isScrolled && (
|
||||
<div className="absolute top-0 left-0 right-0 h-16 bg-gradient-to-b from-background to-transparent z-10 pointer-events-none" />
|
||||
)}
|
||||
@@ -442,6 +499,8 @@ export function HistoryTable() {
|
||||
const isPlayable = !isGenerating && !isFailed;
|
||||
const hasVersions = gen.versions && gen.versions.length > 1;
|
||||
const isVersionsExpanded = expandedVersionsId === gen.id;
|
||||
const isCancelling =
|
||||
cancelGeneration.isPending && cancelGeneration.variables === gen.id;
|
||||
return (
|
||||
<div
|
||||
key={gen.id}
|
||||
@@ -590,60 +649,71 @@ export function HistoryTable() {
|
||||
<Trash2 className="h-2 w-2" />
|
||||
</Button>
|
||||
</>
|
||||
) : isGenerating ? (
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Cancel generation"
|
||||
disabled={isCancelling}
|
||||
onClick={() => cancelGeneration.mutate(gen.id)}
|
||||
>
|
||||
{isCancelling ? (
|
||||
<Loader2 className="h-2 w-2 animate-spin" />
|
||||
) : (
|
||||
<Square className="h-2 w-2" />
|
||||
)}
|
||||
</Button>
|
||||
) : (
|
||||
<>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Actions"
|
||||
disabled={isGenerating}
|
||||
>
|
||||
<MoreHorizontal className="h-2 w-2" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuItem
|
||||
onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}
|
||||
>
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDownloadAudio(gen.id, gen.text)}
|
||||
disabled={exportGenerationAudio.isPending}
|
||||
>
|
||||
<Download className="mr-2 h-4 w-4" />
|
||||
Export Audio
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleExportPackage(gen.id, gen.text)}
|
||||
disabled={exportGeneration.isPending}
|
||||
>
|
||||
<FileArchive className="mr-2 h-4 w-4" />
|
||||
Export Package
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
|
||||
<Wand2 className="mr-2 h-4 w-4" />
|
||||
Apply Effects
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
|
||||
<RotateCcw className="mr-2 h-4 w-4" />
|
||||
Regenerate
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||
disabled={deleteGeneration.isPending}
|
||||
// className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
</>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Actions"
|
||||
disabled={isGenerating}
|
||||
>
|
||||
<MoreHorizontal className="h-2 w-2" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuItem onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}>
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDownloadAudio(gen.id, gen.text)}
|
||||
disabled={exportGenerationAudio.isPending}
|
||||
>
|
||||
<Download className="mr-2 h-4 w-4" />
|
||||
Export Audio
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleExportPackage(gen.id, gen.text)}
|
||||
disabled={exportGeneration.isPending}
|
||||
>
|
||||
<FileArchive className="mr-2 h-4 w-4" />
|
||||
Export Package
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
|
||||
<Wand2 className="mr-2 h-4 w-4" />
|
||||
Apply Effects
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
|
||||
<RotateCcw className="mr-2 h-4 w-4" />
|
||||
Regenerate
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||
disabled={deleteGeneration.isPending}
|
||||
// className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
@@ -759,6 +829,31 @@ export function HistoryTable() {
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
|
||||
<Dialog open={clearFailedDialogOpen} onOpenChange={setClearFailedDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Clear failed generations</DialogTitle>
|
||||
<DialogDescription>
|
||||
This will permanently delete {failedCount} failed{' '}
|
||||
{failedCount === 1 ? 'generation' : 'generations'} from your history. This cannot be
|
||||
undone.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setClearFailedDialogOpen(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
variant="destructive"
|
||||
onClick={handleClearFailedConfirm}
|
||||
disabled={clearFailed.isPending}
|
||||
>
|
||||
{clearFailed.isPending ? 'Clearing...' : 'Clear all'}
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
|
||||
<Dialog open={importDialogOpen} onOpenChange={setImportDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
|
||||
@@ -977,7 +977,19 @@ export function ModelManagement() {
|
||||
});
|
||||
try {
|
||||
// Start the migration (background task)
|
||||
await apiClient.migrateModels(newDir);
|
||||
const migrationResult = await apiClient.migrateModels(newDir);
|
||||
|
||||
// If no models to migrate, warn user and skip the change
|
||||
if (migrationResult.moved === 0) {
|
||||
setMigrating(false);
|
||||
setMigrationProgress(null);
|
||||
toast({
|
||||
title: 'No models to migrate',
|
||||
description: 'Download at least one model before changing the storage location.',
|
||||
});
|
||||
setPendingMigrateDir(null);
|
||||
return;
|
||||
}
|
||||
|
||||
// Connect to SSE for progress
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
@@ -1064,105 +1076,3 @@ export function ModelManagement() {
|
||||
);
|
||||
}
|
||||
|
||||
interface ModelItemProps {
|
||||
model: {
|
||||
model_name: string;
|
||||
display_name: string;
|
||||
downloaded: boolean;
|
||||
downloading?: boolean; // From server - true if download in progress
|
||||
size_mb?: number;
|
||||
loaded: boolean;
|
||||
};
|
||||
onDownload: () => void;
|
||||
onDelete: () => void;
|
||||
isDownloading: boolean; // Local state - true if user just clicked download
|
||||
formatSize: (sizeMb?: number) => string;
|
||||
}
|
||||
|
||||
function ModelItem({ model, onDownload, onDelete, isDownloading, formatSize }: ModelItemProps) {
|
||||
// Use server's downloading state OR local state (for immediate feedback before server updates)
|
||||
const showDownloading = model.downloading || isDownloading;
|
||||
|
||||
const statusText = model.loaded
|
||||
? 'Loaded'
|
||||
: showDownloading
|
||||
? 'Downloading'
|
||||
: model.downloaded
|
||||
? 'Downloaded'
|
||||
: 'Not downloaded';
|
||||
const sizeText =
|
||||
model.downloaded && model.size_mb && !showDownloading ? `, ${formatSize(model.size_mb)}` : '';
|
||||
const rowLabel = `${model.display_name}, ${statusText}${sizeText}. Use Tab to reach Download or Delete.`;
|
||||
|
||||
return (
|
||||
<div
|
||||
className="flex items-center justify-between p-3 border rounded-lg"
|
||||
role="group"
|
||||
tabIndex={0}
|
||||
aria-label={rowLabel}
|
||||
>
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="font-medium text-sm">{model.display_name}</span>
|
||||
{model.loaded && (
|
||||
<Badge variant="default" className="text-xs">
|
||||
Loaded
|
||||
</Badge>
|
||||
)}
|
||||
{/* Only show Downloaded if actually downloaded AND not downloading */}
|
||||
{model.downloaded && !model.loaded && !showDownloading && (
|
||||
<Badge variant="secondary" className="text-xs">
|
||||
Downloaded
|
||||
</Badge>
|
||||
)}
|
||||
</div>
|
||||
{model.downloaded && model.size_mb && !showDownloading && (
|
||||
<div className="text-xs text-muted-foreground mt-1">
|
||||
Size: {formatSize(model.size_mb)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{model.downloaded && !showDownloading ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="flex items-center gap-1 text-sm text-muted-foreground">
|
||||
<span>Ready</span>
|
||||
</div>
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={onDelete}
|
||||
variant="outline"
|
||||
disabled={model.loaded}
|
||||
title={model.loaded ? 'Unload model before deleting' : 'Delete model'}
|
||||
aria-label={
|
||||
model.loaded ? 'Unload model before deleting' : `Delete ${model.display_name}`
|
||||
}
|
||||
>
|
||||
<Trash2 className="h-4 w-4" />
|
||||
</Button>
|
||||
</div>
|
||||
) : showDownloading ? (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="outline"
|
||||
disabled
|
||||
aria-label={`${model.display_name} downloading`}
|
||||
>
|
||||
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
|
||||
Downloading...
|
||||
</Button>
|
||||
) : (
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={onDownload}
|
||||
variant="outline"
|
||||
aria-label={`Download ${model.display_name}`}
|
||||
>
|
||||
<Download className="h-4 w-4 mr-2" />
|
||||
Download
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -12,7 +12,11 @@ interface ModelProgressProps {
|
||||
isDownloading?: boolean;
|
||||
}
|
||||
|
||||
export function ModelProgress({ modelName, displayName, isDownloading = false }: ModelProgressProps) {
|
||||
export function ModelProgress({
|
||||
modelName,
|
||||
displayName,
|
||||
isDownloading = false,
|
||||
}: ModelProgressProps) {
|
||||
const [progress, setProgress] = useState<ModelProgressType | null>(null);
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
|
||||
|
||||
@@ -182,8 +182,8 @@ function ChangelogEntryCard({ entry }: { entry: ChangelogEntry }) {
|
||||
|
||||
return (
|
||||
<div className="border-b border-border/50 pb-6">
|
||||
<div className="flex items-baseline gap-3 mb-1">
|
||||
<h3 className="text-sm font-medium">{entry.version}</h3>
|
||||
<div className="flex items-baseline gap-3 mb-3">
|
||||
<h3 className="text-xl font-semibold tracking-tight">{entry.version}</h3>
|
||||
{entry.date && <span className="text-xs text-muted-foreground">{entry.date}</span>}
|
||||
{entry.version === 'Unreleased' && <Badge variant="outline">dev</Badge>}
|
||||
</div>
|
||||
|
||||
@@ -87,7 +87,7 @@ export function StoryChatItem({
|
||||
alt={`${item.profile_name} avatar`}
|
||||
className={cn(
|
||||
'h-full w-full object-cover transition-all duration-200',
|
||||
!isCurrentlyPlaying && 'grayscale'
|
||||
!isCurrentlyPlaying && 'grayscale',
|
||||
)}
|
||||
onError={() => setAvatarError(true)}
|
||||
/>
|
||||
@@ -127,7 +127,10 @@ export function StoryChatItem({
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play from here
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={onRemove} className="text-destructive focus:text-destructive">
|
||||
<DropdownMenuItem
|
||||
onClick={onRemove}
|
||||
className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Remove from Story
|
||||
</DropdownMenuItem>
|
||||
@@ -139,15 +142,12 @@ export function StoryChatItem({
|
||||
}
|
||||
|
||||
// Sortable wrapper component
|
||||
export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>) {
|
||||
const {
|
||||
attributes,
|
||||
listeners,
|
||||
setNodeRef,
|
||||
transform,
|
||||
transition,
|
||||
isDragging,
|
||||
} = useSortable({ id: props.item.generation_id });
|
||||
export function SortableStoryChatItem(
|
||||
props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>,
|
||||
) {
|
||||
const { attributes, listeners, setNodeRef, transform, transition, isDragging } = useSortable({
|
||||
id: props.item.generation_id,
|
||||
});
|
||||
|
||||
const style = {
|
||||
transform: CSS.Transform.toString(transform),
|
||||
@@ -156,11 +156,7 @@ export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandl
|
||||
|
||||
return (
|
||||
<div ref={setNodeRef} style={style} {...attributes}>
|
||||
<StoryChatItem
|
||||
{...props}
|
||||
dragHandleProps={listeners}
|
||||
isDragging={isDragging}
|
||||
/>
|
||||
<StoryChatItem {...props} dragHandleProps={listeners} isDragging={isDragging} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -371,7 +371,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
}
|
||||
}, [isResizing, handleResizeMove, handleResizeEnd]);
|
||||
|
||||
const handleTimelineClick = (e: React.MouseEvent<HTMLDivElement>) => {
|
||||
const handleTimelineClick = (e: React.MouseEvent<HTMLElement>) => {
|
||||
if (!tracksRef.current || draggingItem || trimmingItem) return;
|
||||
const rect = tracksRef.current.getBoundingClientRect();
|
||||
const x = e.clientX - rect.left + tracksRef.current.scrollLeft;
|
||||
@@ -500,7 +500,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
}, [trimmingItem, trimSide, tempTrimValues, storyId, trimItem, toast]);
|
||||
|
||||
const handleSplit = useCallback(() => {
|
||||
if (!selectedClipId) return;
|
||||
if (!selectedClipId || splitItem.isPending) return;
|
||||
|
||||
const item = items.find((i) => i.id === selectedClipId);
|
||||
if (!item) return;
|
||||
|
||||
@@ -14,12 +14,7 @@ const MemoizedWaveform = memo(function MemoizedWaveform({
|
||||
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
|
||||
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
|
||||
{({ canvasRef }) => (
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
width={500}
|
||||
height={150}
|
||||
className="w-full h-full"
|
||||
/>
|
||||
<canvas ref={canvasRef} width={500} height={150} className="w-full h-full" />
|
||||
)}
|
||||
</Visualizer>
|
||||
</div>
|
||||
@@ -87,9 +82,7 @@ export function AudioSampleRecording({
|
||||
<div className="space-y-4">
|
||||
{!isRecording && !file && (
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
|
||||
<Button
|
||||
type="button"
|
||||
onClick={onStart}
|
||||
@@ -107,9 +100,7 @@ export function AudioSampleRecording({
|
||||
|
||||
{isRecording && (
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
|
||||
<div className="relative z-10 flex items-center gap-4">
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
|
||||
|
||||
@@ -25,9 +25,10 @@ const ENGINE_DISPLAY_NAMES: Record<string, string> = {
|
||||
|
||||
interface ProfileCardProps {
|
||||
profile: VoiceProfileResponse;
|
||||
disabled?: boolean;
|
||||
}
|
||||
|
||||
export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
export function ProfileCard({ profile, disabled }: ProfileCardProps) {
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||
|
||||
const deleteProfile = useDeleteProfile();
|
||||
@@ -40,6 +41,12 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
const isSelected = selectedProfileId === profile.id;
|
||||
|
||||
const handleSelect = () => {
|
||||
// If disabled but already selected, bounce the selection to re-trigger engine auto-switch
|
||||
if (disabled && isSelected) {
|
||||
setSelectedProfileId(null);
|
||||
setTimeout(() => setSelectedProfileId(profile.id), 0);
|
||||
return;
|
||||
}
|
||||
setSelectedProfileId(isSelected ? null : profile.id);
|
||||
};
|
||||
|
||||
@@ -80,8 +87,9 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
<>
|
||||
<Card
|
||||
className={cn(
|
||||
'cursor-pointer hover:shadow-md transition-all flex flex-col h-[162px]',
|
||||
isSelected && 'ring-2 ring-accent shadow-md',
|
||||
'cursor-pointer transition-all flex flex-col h-[162px]',
|
||||
disabled ? 'opacity-40 hover:opacity-60' : 'hover:shadow-md',
|
||||
isSelected && !disabled && 'ring-2 ring-accent shadow-md',
|
||||
)}
|
||||
onClick={handleSelect}
|
||||
tabIndex={0}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Mic, Music, Sparkles } from 'lucide-react';
|
||||
import { Info, Mic, Sparkles } from 'lucide-react';
|
||||
import { useEffect, useRef } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent } from '@/components/ui/card';
|
||||
import { useProfiles } from '@/lib/hooks/useProfiles';
|
||||
@@ -9,16 +10,33 @@ import { ProfileForm } from './ProfileForm';
|
||||
/** Engines that use preset (built-in) voices instead of cloned profiles. */
|
||||
const PRESET_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
|
||||
|
||||
/** Human-readable engine names for empty state messages. */
|
||||
const ENGINE_NAMES: Record<string, string> = {
|
||||
kokoro: 'Kokoro',
|
||||
qwen_custom_voice: 'Qwen CustomVoice',
|
||||
};
|
||||
|
||||
export function ProfileList() {
|
||||
const { data: profiles, isLoading, error } = useProfiles();
|
||||
const setDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||
const selectedEngine = useUIStore((state) => state.selectedEngine);
|
||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||
const cardRefs = useRef<Map<string, HTMLDivElement>>(new Map());
|
||||
|
||||
// Scroll to the selected profile after engine/sort changes
|
||||
useEffect(() => {
|
||||
if (!selectedProfileId) return;
|
||||
let timeoutId: ReturnType<typeof setTimeout> | null = null;
|
||||
const rafId = requestAnimationFrame(() => {
|
||||
const el = cardRefs.current.get(selectedProfileId);
|
||||
if (!el) return;
|
||||
|
||||
// Temporarily apply scroll-margin so it doesn't land flush at the top
|
||||
el.style.scrollMarginTop = '180px';
|
||||
el.scrollIntoView({ behavior: 'smooth', block: 'nearest', inline: 'nearest' });
|
||||
timeoutId = setTimeout(() => {
|
||||
el.style.scrollMarginTop = '';
|
||||
}, 500);
|
||||
});
|
||||
return () => {
|
||||
cancelAnimationFrame(rafId);
|
||||
if (timeoutId) clearTimeout(timeoutId);
|
||||
};
|
||||
}, [selectedProfileId, selectedEngine]);
|
||||
|
||||
if (isLoading) {
|
||||
return null;
|
||||
@@ -35,10 +53,18 @@ export function ProfileList() {
|
||||
const allProfiles = profiles || [];
|
||||
const isPresetEngine = PRESET_ENGINES.has(selectedEngine);
|
||||
|
||||
// Filter profiles based on selected engine
|
||||
const filteredProfiles = isPresetEngine
|
||||
? allProfiles.filter((p) => p.voice_type === 'preset' && p.preset_engine === selectedEngine)
|
||||
: allProfiles.filter((p) => p.voice_type !== 'preset');
|
||||
/** Whether a profile is supported by the currently selected engine. */
|
||||
const isSupported = (p: (typeof allProfiles)[number]) =>
|
||||
isPresetEngine
|
||||
? p.voice_type === 'preset' && p.preset_engine === selectedEngine
|
||||
: p.voice_type !== 'preset';
|
||||
|
||||
// Sort so supported profiles come first
|
||||
const sortedProfiles = [...allProfiles].sort(
|
||||
(a, b) => (isSupported(a) ? 0 : 1) - (isSupported(b) ? 0 : 1),
|
||||
);
|
||||
|
||||
const hasUnsupported = sortedProfiles.some((p) => !isSupported(p));
|
||||
|
||||
return (
|
||||
<div className="flex flex-col">
|
||||
@@ -56,29 +82,26 @@ export function ProfileList() {
|
||||
</Button>
|
||||
</CardContent>
|
||||
</Card>
|
||||
) : filteredProfiles.length === 0 && isPresetEngine ? (
|
||||
<Card>
|
||||
<CardContent className="flex flex-col items-center justify-center py-12">
|
||||
<Music className="h-12 w-12 text-muted-foreground mb-4" />
|
||||
<p className="text-muted-foreground mb-2">
|
||||
No {ENGINE_NAMES[selectedEngine] ?? selectedEngine} voices created yet.
|
||||
</p>
|
||||
<p className="text-sm text-muted-foreground mb-4">
|
||||
Create a profile to choose a specific voice before generating.
|
||||
</p>
|
||||
<Button onClick={() => setDialogOpen(true)}>
|
||||
<Sparkles className="mr-2 h-4 w-4" />
|
||||
Create {ENGINE_NAMES[selectedEngine] ?? selectedEngine} Voice
|
||||
</Button>
|
||||
</CardContent>
|
||||
</Card>
|
||||
) : (
|
||||
<div className="flex gap-4 overflow-x-auto p-1 pb-1 lg:grid lg:grid-cols-3 lg:auto-rows-auto lg:overflow-x-visible lg:pb-[150px]">
|
||||
{filteredProfiles.map((profile) => (
|
||||
<div key={profile.id} className="shrink-0 w-[200px] lg:w-auto lg:shrink">
|
||||
<ProfileCard profile={profile} />
|
||||
{sortedProfiles.map((profile) => (
|
||||
<div
|
||||
key={profile.id}
|
||||
className="shrink-0 w-[200px] lg:w-auto lg:shrink"
|
||||
ref={(el) => {
|
||||
if (el) cardRefs.current.set(profile.id, el);
|
||||
else cardRefs.current.delete(profile.id);
|
||||
}}
|
||||
>
|
||||
<ProfileCard profile={profile} disabled={!isSupported(profile)} />
|
||||
</div>
|
||||
))}
|
||||
{hasUnsupported && (
|
||||
<div className="col-span-full flex items-center gap-2 text-xs text-muted-foreground py-2">
|
||||
<Info className="h-3.5 w-3.5 shrink-0" />
|
||||
<span>Only supported voice profiles can be selected for the current model.</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
@@ -111,4 +111,4 @@ export {
|
||||
AlertDialogDescription,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -5,7 +5,15 @@ import type { UpdateStatus } from '@/platform/types';
|
||||
// Re-export UpdateStatus for backwards compatibility
|
||||
export type { UpdateStatus };
|
||||
|
||||
export function useAutoUpdater(checkOnMount = false) {
|
||||
interface UseAutoUpdaterOptions {
|
||||
checkOnMount?: boolean;
|
||||
showToast?: boolean;
|
||||
}
|
||||
|
||||
export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false) {
|
||||
const { checkOnMount } =
|
||||
typeof options === 'boolean' ? { checkOnMount: options } : { checkOnMount: options.checkOnMount ?? false };
|
||||
|
||||
const platform = usePlatform();
|
||||
const [status, setStatus] = useState<UpdateStatus>(platform.updater.getStatus());
|
||||
const hasCheckedRef = useRef(false);
|
||||
@@ -38,10 +46,11 @@ export function useAutoUpdater(checkOnMount = false) {
|
||||
useEffect(() => {
|
||||
if (checkOnMount && platform.metadata.isTauri && !hasCheckedRef.current) {
|
||||
hasCheckedRef.current = true;
|
||||
checkForUpdates();
|
||||
checkForUpdates().catch((error) => {
|
||||
console.error('Auto update check failed:', error);
|
||||
});
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
|
||||
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
|
||||
|
||||
return {
|
||||
status,
|
||||
|
||||
@@ -73,7 +73,7 @@ export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false)
|
||||
}
|
||||
// Empty dependency array - only run once on mount
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
|
||||
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
|
||||
|
||||
// Show toast when update is available
|
||||
useEffect(() => {
|
||||
|
||||
@@ -234,6 +234,12 @@ class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async cancelGeneration(generationId: string): Promise<{ message: string }> {
|
||||
return this.request<{ message: string }>(`/generate/${generationId}/cancel`, {
|
||||
method: 'POST',
|
||||
});
|
||||
}
|
||||
|
||||
async regenerateGeneration(generationId: string): Promise<GenerationResponse> {
|
||||
return this.request<GenerationResponse>(`/generate/${generationId}/regenerate`, {
|
||||
method: 'POST',
|
||||
@@ -270,6 +276,12 @@ class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async clearFailedGenerations(): Promise<{ deleted: number }> {
|
||||
return this.request<{ deleted: number }>(`/history/failed`, {
|
||||
method: 'DELETE',
|
||||
});
|
||||
}
|
||||
|
||||
async exportGeneration(generationId: string): Promise<Blob> {
|
||||
const url = `${this.getBaseUrl()}/history/${generationId}/export`;
|
||||
const response = await fetch(url);
|
||||
@@ -378,7 +390,9 @@ class ApiClient {
|
||||
return this.request<{ path: string }>('/models/cache-dir');
|
||||
}
|
||||
|
||||
async migrateModels(destination: string): Promise<{ source: string; destination: string }> {
|
||||
async migrateModels(
|
||||
destination: string,
|
||||
): Promise<{ source: string; destination: string; moved: number; errors: string[] }> {
|
||||
return this.request('/models/migrate', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ destination }),
|
||||
|
||||
@@ -10,6 +10,7 @@ import { useGeneration } from '@/lib/hooks/useGeneration';
|
||||
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
|
||||
import { useGenerationStore } from '@/stores/generationStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
|
||||
const generationSchema = z.object({
|
||||
text: z.string().min(1, '').max(50000),
|
||||
@@ -45,6 +46,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
const maxChunkChars = useServerStore((state) => state.maxChunkChars);
|
||||
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
|
||||
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
|
||||
const selectedEngine = useUIStore((state) => state.selectedEngine);
|
||||
const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null);
|
||||
const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null);
|
||||
|
||||
@@ -62,7 +64,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
seed: undefined,
|
||||
modelSize: '1.7B',
|
||||
instruct: '',
|
||||
engine: 'qwen',
|
||||
engine: (selectedEngine as GenerationFormValues['engine']) || 'qwen',
|
||||
...options.defaultValues,
|
||||
},
|
||||
});
|
||||
@@ -134,7 +136,9 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
|
||||
const hasModelSizes =
|
||||
engine === 'qwen' || engine === 'qwen_custom_voice' || engine === 'tada';
|
||||
const supportsInstruct = engine === 'qwen' || engine === 'qwen_custom_voice';
|
||||
// Only Qwen CustomVoice actually honors the instruct kwarg at model level.
|
||||
// Base Qwen3-TTS accepts the kwarg but ignores it.
|
||||
const supportsInstruct = engine === 'qwen_custom_voice';
|
||||
const effectsChain = options.getEffectsChain?.();
|
||||
// This now returns immediately with status="generating"
|
||||
const result = await generation.mutateAsync({
|
||||
|
||||
@@ -29,6 +29,17 @@ export function useDeleteGeneration() {
|
||||
});
|
||||
}
|
||||
|
||||
export function useClearFailedGenerations() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
return useMutation({
|
||||
mutationFn: () => apiClient.clearFailedGenerations(),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useExportGeneration() {
|
||||
const platform = usePlatform();
|
||||
|
||||
|
||||
@@ -131,7 +131,8 @@ export function useModelDownloadToast({
|
||||
)}
|
||||
</div>
|
||||
),
|
||||
duration: progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
|
||||
duration:
|
||||
progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
|
||||
});
|
||||
|
||||
// Close connection and dismiss toast on completion or error
|
||||
|
||||
@@ -26,8 +26,24 @@ export function useSystemAudioCapture({
|
||||
|
||||
// Check if system audio capture is supported
|
||||
useEffect(() => {
|
||||
const supported = platform.audio.isSystemAudioSupported();
|
||||
setIsSupported(supported);
|
||||
let isActive = true;
|
||||
|
||||
void platform.audio
|
||||
.isSystemAudioSupported()
|
||||
.then((supported) => {
|
||||
if (isActive) {
|
||||
setIsSupported(supported);
|
||||
}
|
||||
})
|
||||
.catch(() => {
|
||||
if (isActive) {
|
||||
setIsSupported(false);
|
||||
}
|
||||
});
|
||||
|
||||
return () => {
|
||||
isActive = false;
|
||||
};
|
||||
}, [platform]);
|
||||
|
||||
const startRecording = useCallback(async () => {
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
import { QueryClient } from '@tanstack/react-query';
|
||||
|
||||
/**
|
||||
* Shared QueryClient instance used across the app.
|
||||
*
|
||||
* Extracted into its own side-effect-free module so it can be imported from
|
||||
* both the React bootstrap (main.tsx) and non-React code (stores, utilities)
|
||||
* without pulling in ReactDOM or other bootstrap side effects.
|
||||
*/
|
||||
export const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
staleTime: 1000 * 60 * 5, // 5 minutes
|
||||
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
|
||||
retry: 1,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
+2
-12
@@ -1,20 +1,10 @@
|
||||
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
|
||||
import { QueryClientProvider } from '@tanstack/react-query';
|
||||
// import { ReactQueryDevtools } from '@tanstack/react-query-devtools';
|
||||
import React from 'react';
|
||||
import ReactDOM from 'react-dom/client';
|
||||
import App from './App';
|
||||
import './index.css';
|
||||
|
||||
const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
staleTime: 1000 * 60 * 5, // 5 minutes
|
||||
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
|
||||
retry: 1,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
import { queryClient } from './lib/queryClient';
|
||||
|
||||
ReactDOM.createRoot(document.getElementById('root')!).render(
|
||||
<React.StrictMode>
|
||||
|
||||
@@ -9,11 +9,7 @@ export interface PlatformProviderProps {
|
||||
}
|
||||
|
||||
export function PlatformProvider({ platform, children }: PlatformProviderProps) {
|
||||
return (
|
||||
<PlatformContext.Provider value={platform}>
|
||||
{children}
|
||||
</PlatformContext.Provider>
|
||||
);
|
||||
return <PlatformContext.Provider value={platform}>{children}</PlatformContext.Provider>;
|
||||
}
|
||||
|
||||
export function usePlatform(): Platform {
|
||||
|
||||
@@ -42,7 +42,7 @@ export interface AudioDevice {
|
||||
}
|
||||
|
||||
export interface PlatformAudio {
|
||||
isSystemAudioSupported(): boolean;
|
||||
isSystemAudioSupported(): Promise<boolean>;
|
||||
startSystemAudioCapture(maxDurationSecs: number): Promise<void>;
|
||||
stopSystemAudioCapture(): Promise<Blob>;
|
||||
listOutputDevices(): Promise<AudioDevice[]>;
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { create } from 'zustand';
|
||||
import { persist } from 'zustand/middleware';
|
||||
import { queryClient } from '@/lib/queryClient';
|
||||
|
||||
interface ServerStore {
|
||||
serverUrl: string;
|
||||
@@ -30,11 +31,25 @@ interface ServerStore {
|
||||
setCustomModelsDir: (dir: string | null) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Invalidate all React Query caches so stale data from the previous
|
||||
* server is not shown. Called when the server URL changes.
|
||||
*/
|
||||
function invalidateAllServerData() {
|
||||
queryClient.invalidateQueries();
|
||||
}
|
||||
|
||||
export const useServerStore = create<ServerStore>()(
|
||||
persist(
|
||||
(set) => ({
|
||||
(set, get) => ({
|
||||
serverUrl: 'http://127.0.0.1:17493',
|
||||
setServerUrl: (url) => set({ serverUrl: url }),
|
||||
setServerUrl: (url) => {
|
||||
const prev = get().serverUrl;
|
||||
set({ serverUrl: url });
|
||||
if (url !== prev) {
|
||||
invalidateAllServerData();
|
||||
}
|
||||
},
|
||||
|
||||
isConnected: false,
|
||||
setIsConnected: (connected) => set({ isConnected: connected }),
|
||||
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
# Backend package
|
||||
|
||||
__version__ = "0.3.1"
|
||||
__version__ = "0.4.1"
|
||||
|
||||
+31
-3
@@ -135,7 +135,7 @@ def _mount_frontend(application: FastAPI) -> None:
|
||||
async def serve_spa(full_path: str):
|
||||
file_path = (frontend_dir / full_path).resolve()
|
||||
# Guard against path traversal — only serve files inside frontend_dir
|
||||
if full_path and file_path.is_file() and str(file_path).startswith(str(frontend_dir)):
|
||||
if full_path and file_path.is_file() and file_path.is_relative_to(frontend_dir):
|
||||
return FileResponse(file_path)
|
||||
return FileResponse(frontend_dir / "index.html", media_type="text/html")
|
||||
|
||||
@@ -146,15 +146,36 @@ def _get_gpu_status() -> str:
|
||||
"""Return a human-readable string describing GPU availability."""
|
||||
backend_type = get_backend_type()
|
||||
if torch.cuda.is_available():
|
||||
from .backends.base import check_cuda_compatibility
|
||||
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
compatible, _warning = check_cuda_compatibility()
|
||||
is_rocm = hasattr(torch.version, "hip") and torch.version.hip is not None
|
||||
if is_rocm:
|
||||
return f"ROCm ({device_name})"
|
||||
return f"CUDA ({device_name})"
|
||||
label = f"ROCm ({device_name})"
|
||||
else:
|
||||
label = f"CUDA ({device_name})"
|
||||
if not compatible:
|
||||
label += " [UNSUPPORTED - see logs]"
|
||||
return label
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "MPS (Apple Silicon)"
|
||||
elif backend_type == "mlx":
|
||||
return "Metal (Apple Silicon via MLX)"
|
||||
|
||||
# Intel XPU (Arc / Data Center) via IPEX
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
try:
|
||||
xpu_name = torch.xpu.get_device_name(0)
|
||||
except Exception:
|
||||
xpu_name = "Intel GPU"
|
||||
return f"XPU ({xpu_name})"
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
return "None (CPU only)"
|
||||
|
||||
|
||||
@@ -216,6 +237,13 @@ def _register_lifecycle(application: FastAPI) -> None:
|
||||
logger.info("Backend: %s", backend_type.upper())
|
||||
logger.info("GPU: %s", _get_gpu_status())
|
||||
|
||||
# Warn if GPU architecture is not supported by this PyTorch build
|
||||
from .backends.base import check_cuda_compatibility
|
||||
|
||||
_compatible, _cuda_warning = check_cuda_compatibility()
|
||||
if not _compatible:
|
||||
logger.warning("GPU COMPATIBILITY: %s", _cuda_warning)
|
||||
|
||||
from .services.cuda import check_and_update_cuda_binary
|
||||
|
||||
create_background_task(check_and_update_cuda_binary())
|
||||
|
||||
@@ -126,6 +126,75 @@ def get_torch_device(
|
||||
return "cpu"
|
||||
|
||||
|
||||
def check_cuda_compatibility() -> tuple[bool, str | None]:
|
||||
"""Check if the installed PyTorch supports the current GPU's compute capability.
|
||||
|
||||
Returns:
|
||||
(compatible, warning_message) — compatible is True if OK or no CUDA GPU,
|
||||
warning_message is a human-readable string if there's a problem.
|
||||
"""
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return True, None
|
||||
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
capability = f"{major}.{minor}"
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
|
||||
# torch.cuda._get_arch_list() returns the SM architectures this build
|
||||
# was compiled for (e.g. ["sm_50", "sm_60", ..., "sm_90"]).
|
||||
try:
|
||||
arch_list = torch.cuda._get_arch_list()
|
||||
if arch_list:
|
||||
# Check for both sm_XX and compute_XX (JIT-compiled) entries
|
||||
compute_tag = f"compute_{major}{minor}"
|
||||
if sm_tag not in arch_list and compute_tag not in arch_list:
|
||||
return False, (
|
||||
f"{device_name} (compute capability {capability} / {sm_tag}) "
|
||||
f"is not supported by this PyTorch build. "
|
||||
f"Supported architectures: {', '.join(arch_list)}. "
|
||||
f"Install PyTorch nightly (cu128) for newer GPU support: "
|
||||
f"pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128"
|
||||
)
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
return True, None
|
||||
|
||||
|
||||
def empty_device_cache(device: str) -> None:
|
||||
"""
|
||||
Free cached memory on the given device (CUDA or XPU).
|
||||
|
||||
Backends should call this after unloading models so VRAM is returned
|
||||
to the OS.
|
||||
"""
|
||||
import torch
|
||||
|
||||
if device == "cuda" and torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
elif device == "xpu" and hasattr(torch, "xpu"):
|
||||
torch.xpu.empty_cache()
|
||||
|
||||
|
||||
def manual_seed(seed: int, device: str) -> None:
|
||||
"""
|
||||
Set the random seed on both CPU and the active accelerator.
|
||||
|
||||
Covers CUDA and Intel XPU so that generation is reproducible
|
||||
regardless of which GPU backend is in use.
|
||||
"""
|
||||
import torch
|
||||
|
||||
torch.manual_seed(seed)
|
||||
if device == "cuda" and torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
elif device == "xpu" and hasattr(torch, "xpu"):
|
||||
torch.xpu.manual_seed(seed)
|
||||
|
||||
|
||||
async def combine_voice_prompts(
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
|
||||
@@ -18,6 +18,8 @@ from . import TTSBackend
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
patch_chatterbox_f32,
|
||||
@@ -48,7 +50,7 @@ class ChatterboxTTSBackend:
|
||||
self._model_load_lock = asyncio.Lock()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
return get_torch_device(force_cpu_on_mac=True)
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -117,10 +119,7 @@ class ChatterboxTTSBackend:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
if device == "cuda":
|
||||
import torch
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
logger.info("Chatterbox unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
@@ -200,7 +199,7 @@ class ChatterboxTTSBackend:
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
logger.info(f"[Chatterbox] Generating: lang={language}")
|
||||
|
||||
@@ -220,10 +219,7 @@ class ChatterboxTTSBackend:
|
||||
else:
|
||||
audio = np.asarray(wav, dtype=np.float32)
|
||||
|
||||
sample_rate = (
|
||||
getattr(self.model, "sr", None)
|
||||
or getattr(self.model, "sample_rate", 24000)
|
||||
)
|
||||
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@ from . import TTSBackend
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
patch_chatterbox_f32,
|
||||
@@ -48,7 +50,7 @@ class ChatterboxTurboTTSBackend:
|
||||
self._model_load_lock = asyncio.Lock()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
return get_torch_device(force_cpu_on_mac=True)
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -116,10 +118,7 @@ class ChatterboxTurboTTSBackend:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
if device == "cuda":
|
||||
import torch
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
logger.info("Chatterbox Turbo unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
@@ -181,7 +180,7 @@ class ChatterboxTurboTTSBackend:
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
logger.info("[Chatterbox Turbo] Generating (English)")
|
||||
|
||||
@@ -200,10 +199,7 @@ class ChatterboxTurboTTSBackend:
|
||||
else:
|
||||
audio = np.asarray(wav, dtype=np.float32)
|
||||
|
||||
sample_rate = (
|
||||
getattr(self.model, "sr", None)
|
||||
or getattr(self.model, "sample_rate", 24000)
|
||||
)
|
||||
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
@@ -24,6 +24,8 @@ from . import TTSBackend
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
@@ -66,7 +68,7 @@ class HumeTadaBackend:
|
||||
def _get_device(self) -> str:
|
||||
# Force CPU on macOS — MPS has issues with flow matching
|
||||
# and large vocab lm_head (>65536 output channels)
|
||||
return get_torch_device(force_cpu_on_mac=True)
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -105,6 +107,7 @@ class HumeTadaBackend:
|
||||
# package. The real package pulls in onnx/tensorboard/matplotlib via
|
||||
# descript-audiotools, so we use a lightweight shim instead.
|
||||
from ..utils.dac_shim import install_dac_shim
|
||||
|
||||
install_dac_shim()
|
||||
|
||||
import torch
|
||||
@@ -142,9 +145,12 @@ class HumeTadaBackend:
|
||||
allow_patterns=["tokenizer*", "special_tokens*"],
|
||||
)
|
||||
|
||||
# Determine dtype — use bf16 on CUDA for ~50% memory savings
|
||||
# Determine dtype — use bf16 on CUDA/XPU for ~50% memory savings
|
||||
if device == "cuda" and torch.cuda.is_bf16_supported():
|
||||
model_dtype = torch.bfloat16
|
||||
elif device == "xpu":
|
||||
# Intel Arc (Alchemist+) supports bf16 natively
|
||||
model_dtype = torch.bfloat16
|
||||
else:
|
||||
model_dtype = torch.float32
|
||||
|
||||
@@ -153,14 +159,14 @@ class HumeTadaBackend:
|
||||
# This avoids monkey-patching AutoTokenizer.from_pretrained
|
||||
# which corrupts the classmethod descriptor for other engines.
|
||||
from tada.modules.aligner import AlignerConfig
|
||||
|
||||
AlignerConfig.tokenizer_name = tokenizer_path
|
||||
|
||||
# Load encoder (only needed for voice prompt encoding)
|
||||
from tada.modules.encoder import Encoder
|
||||
|
||||
logger.info("Loading TADA encoder...")
|
||||
self.encoder = Encoder.from_pretrained(
|
||||
TADA_CODEC_REPO, subfolder="encoder"
|
||||
).to(device)
|
||||
self.encoder = Encoder.from_pretrained(TADA_CODEC_REPO, subfolder="encoder").to(device)
|
||||
self.encoder.eval()
|
||||
|
||||
# Load the causal LM (includes decoder for wav generation).
|
||||
@@ -169,12 +175,11 @@ class HumeTadaBackend:
|
||||
# which hits the gated repo. Pre-load the config from HF,
|
||||
# inject the local tokenizer path, then pass it in.
|
||||
from tada.modules.tada import TadaForCausalLM, TadaConfig
|
||||
|
||||
logger.info(f"Loading TADA {model_size} model...")
|
||||
config = TadaConfig.from_pretrained(repo)
|
||||
config.tokenizer_name = tokenizer_path
|
||||
self.model = TadaForCausalLM.from_pretrained(
|
||||
repo, config=config, torch_dtype=model_dtype
|
||||
).to(device)
|
||||
self.model = TadaForCausalLM.from_pretrained(repo, config=config, torch_dtype=model_dtype).to(device)
|
||||
self.model.eval()
|
||||
|
||||
logger.info(f"HumeAI TADA {model_size} loaded successfully on {device}")
|
||||
@@ -188,11 +193,11 @@ class HumeTadaBackend:
|
||||
del self.encoder
|
||||
self.encoder = None
|
||||
|
||||
device = self._device
|
||||
self._device = None
|
||||
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
if device:
|
||||
empty_device_cache(device)
|
||||
|
||||
logger.info("HumeAI TADA unloaded")
|
||||
|
||||
@@ -213,9 +218,7 @@ class HumeTadaBackend:
|
||||
"""
|
||||
await self.load_model(self.model_size)
|
||||
|
||||
cache_key = (
|
||||
"tada_" + get_cache_key(audio_path, reference_text)
|
||||
) if use_cache else None
|
||||
cache_key = ("tada_" + get_cache_key(audio_path, reference_text)) if use_cache else None
|
||||
|
||||
if cache_key:
|
||||
cached = get_cached_voice_prompt(cache_key)
|
||||
@@ -239,9 +242,7 @@ class HumeTadaBackend:
|
||||
|
||||
# Encode with forced alignment
|
||||
text_arg = [reference_text] if reference_text else None
|
||||
prompt = self.encoder(
|
||||
audio, text=text_arg, sample_rate=sr
|
||||
)
|
||||
prompt = self.encoder(audio, text=text_arg, sample_rate=sr)
|
||||
|
||||
# Serialize EncoderOutput to a dict of CPU tensors for caching
|
||||
prompt_dict = {}
|
||||
@@ -299,9 +300,7 @@ class HumeTadaBackend:
|
||||
from tada.modules.encoder import EncoderOutput
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
device = self._device
|
||||
|
||||
|
||||
@@ -12,7 +12,14 @@ from typing import Optional, Tuple
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from .base import is_model_cached, get_torch_device, combine_voice_prompts as _combine_voice_prompts, model_load_progress
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -30,7 +37,7 @@ class LuxTTSBackend:
|
||||
self._device = None
|
||||
|
||||
def _get_device(self) -> str:
|
||||
return get_torch_device(allow_mps=True)
|
||||
return get_torch_device(allow_mps=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -69,9 +76,12 @@ class LuxTTSBackend:
|
||||
|
||||
if device == "cpu":
|
||||
import os
|
||||
|
||||
threads = os.cpu_count() or 4
|
||||
self.model = LuxTTS(
|
||||
model_path=LUXTTS_HF_REPO, device="cpu", threads=min(threads, 8),
|
||||
model_path=LUXTTS_HF_REPO,
|
||||
device="cpu",
|
||||
threads=min(threads, 8),
|
||||
)
|
||||
else:
|
||||
self.model = LuxTTS(model_path=LUXTTS_HF_REPO, device=device)
|
||||
@@ -81,12 +91,12 @@ class LuxTTSBackend:
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
if self.model is not None:
|
||||
device = self.device
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
|
||||
logger.info("LuxTTS unloaded")
|
||||
|
||||
@@ -154,12 +164,8 @@ class LuxTTSBackend:
|
||||
await self.load_model()
|
||||
|
||||
def _generate_sync():
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
manual_seed(seed, self.device)
|
||||
|
||||
wav = self.model.generate_speech(
|
||||
text=text,
|
||||
|
||||
@@ -6,7 +6,6 @@ from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import logging
|
||||
import numpy as np
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -21,6 +20,7 @@ ensure_original_qwen_config_cached()
|
||||
from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
|
||||
from .base import is_model_cached, combine_voice_prompts as _combine_voice_prompts, model_load_progress
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.hf_offline_patch import force_offline_if_cached
|
||||
|
||||
|
||||
class MLXTTSBackend:
|
||||
@@ -96,32 +96,13 @@ class MLXTTSBackend:
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Force offline mode when cached to avoid network requests
|
||||
original_hf_hub_offline = os.environ.get("HF_HUB_OFFLINE")
|
||||
if is_cached:
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
logger.info("[PATCH] Model %s is cached, forcing HF_HUB_OFFLINE=1 to avoid network requests", model_size)
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from mlx_audio.tts import load
|
||||
|
||||
try:
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from mlx_audio.tts import load
|
||||
logger.info("Loading MLX TTS model %s...", model_size)
|
||||
|
||||
logger.info("Loading MLX TTS model %s...", model_size)
|
||||
|
||||
try:
|
||||
self.model = load(model_path)
|
||||
except Exception as load_error:
|
||||
if is_cached and "offline" in str(load_error).lower():
|
||||
logger.warning("[PATCH] Offline load failed, trying with network: %s", load_error)
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
self.model = load(model_path)
|
||||
else:
|
||||
raise
|
||||
finally:
|
||||
if original_hf_hub_offline is not None:
|
||||
os.environ["HF_HUB_OFFLINE"] = original_hf_hub_offline
|
||||
else:
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
with force_offline_if_cached(is_cached, model_name):
|
||||
self.model = load(model_path)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
@@ -329,7 +310,9 @@ class MLXSTTBackend:
|
||||
|
||||
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
logger.info("Loading MLX Whisper model %s...", model_size)
|
||||
self.model = load(model_name)
|
||||
|
||||
with force_offline_if_cached(is_cached, progress_model_name):
|
||||
self.model = load(model_name)
|
||||
|
||||
self.model_size = model_size
|
||||
logger.info("MLX Whisper model %s loaded successfully", model_size)
|
||||
|
||||
@@ -14,11 +14,14 @@ from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.audio import load_audio
|
||||
from ..utils.hf_offline_patch import force_offline_if_cached
|
||||
|
||||
|
||||
class PyTorchTTSBackend:
|
||||
@@ -96,18 +99,28 @@ class PyTorchTTSBackend:
|
||||
model_path = self._get_model_path(model_size)
|
||||
logger.info("Loading TTS model %s on %s...", model_size, self.device)
|
||||
|
||||
if self.device == "cpu":
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.float32,
|
||||
low_cpu_mem_usage=False,
|
||||
)
|
||||
else:
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
# Route both HF Hub and Transformers through a single cache root.
|
||||
# On Windows local setups, model assets can otherwise split between
|
||||
# .hf-cache/hub and .hf-cache/transformers, causing speech_tokenizer
|
||||
# and preprocessor_config.json to fail to resolve during load.
|
||||
from huggingface_hub import constants as hf_constants
|
||||
tts_cache_dir = hf_constants.HF_HUB_CACHE
|
||||
|
||||
with force_offline_if_cached(is_cached, model_name):
|
||||
if self.device == "cpu":
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
cache_dir=tts_cache_dir,
|
||||
torch_dtype=torch.float32,
|
||||
low_cpu_mem_usage=False,
|
||||
)
|
||||
else:
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
cache_dir=tts_cache_dir,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
@@ -120,8 +133,7 @@ class PyTorchTTSBackend:
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(self.device)
|
||||
|
||||
logger.info("TTS model unloaded")
|
||||
|
||||
@@ -213,9 +225,7 @@ class PyTorchTTSBackend:
|
||||
"""Run synchronous generation in thread pool."""
|
||||
# Set seed if provided
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
manual_seed(seed, self.device)
|
||||
|
||||
# Generate audio - this is the blocking operation
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
@@ -282,8 +292,9 @@ class PyTorchSTTBackend:
|
||||
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
logger.info("Loading Whisper model %s on %s...", model_size, self.device)
|
||||
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
with force_offline_if_cached(is_cached, progress_model_name):
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
|
||||
self.model.to(self.device)
|
||||
self.model_size = model_size
|
||||
@@ -297,8 +308,7 @@ class PyTorchSTTBackend:
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(self.device)
|
||||
|
||||
logger.info("Whisper model unloaded")
|
||||
|
||||
|
||||
+32
-11
@@ -52,6 +52,29 @@ def build_server(cuda=False):
|
||||
if platform.system() == "Windows":
|
||||
args.append("--noconsole")
|
||||
|
||||
# numpy 2.x / torch ABI mismatch fix: install memmove fallback for
|
||||
# torch.from_numpy() before the app starts. Runtime hooks run after
|
||||
# FrozenImporter is registered so frozen torch/numpy are importable.
|
||||
# Paths are passed relative to backend_dir because os.chdir(backend_dir)
|
||||
# runs before PyInstaller. Absolute paths would get baked into the
|
||||
# generated .spec, breaking reproducible builds on other machines / CI.
|
||||
args.extend(
|
||||
[
|
||||
"--runtime-hook",
|
||||
"pyi_rth_numpy_compat.py",
|
||||
# Stub torch.compiler.disable before transformers imports
|
||||
# flex_attention, which otherwise triggers torch._dynamo →
|
||||
# torch._numpy._ufuncs and crashes at module load under
|
||||
# PyInstaller. See pyi_rth_torch_compiler_disable.py.
|
||||
"--runtime-hook",
|
||||
"pyi_rth_torch_compiler_disable.py",
|
||||
# Per-module collection overrides (e.g. forcing scipy.stats._distn_infrastructure
|
||||
# to bundle .py source alongside .pyc so the runtime hook can source-patch it).
|
||||
"--additional-hooks-dir",
|
||||
"pyi_hooks",
|
||||
]
|
||||
)
|
||||
|
||||
# Add local qwen_tts path if specified (for editable installs)
|
||||
qwen_tts_path = os.getenv("QWEN_TTS_PATH")
|
||||
if qwen_tts_path and Path(qwen_tts_path).exists():
|
||||
@@ -115,6 +138,11 @@ def build_server(cuda=False):
|
||||
"backend.backends.chatterbox_backend",
|
||||
"--hidden-import",
|
||||
"backend.backends.chatterbox_turbo_backend",
|
||||
# chatterbox multilingual uses spacy_pkuseg for Chinese word
|
||||
# segmentation, which ships pickled dict files (dicts/default.pkl)
|
||||
# and native .so extensions that --hidden-import alone won't bundle.
|
||||
"--collect-all",
|
||||
"spacy_pkuseg",
|
||||
"--hidden-import",
|
||||
"backend.backends.luxtts_backend",
|
||||
"--hidden-import",
|
||||
@@ -231,20 +259,13 @@ def build_server(cuda=False):
|
||||
"--collect-submodules",
|
||||
"tada",
|
||||
# Kokoro 82M — lightweight TTS engine using misaki G2P
|
||||
# collect-all is required because transformers introspects .py source
|
||||
# files at runtime (e.g. _can_set_attn_implementation opens the class
|
||||
# file); hidden-import alone only bundles bytecode.
|
||||
"--hidden-import",
|
||||
"backend.backends.kokoro_backend",
|
||||
"--hidden-import",
|
||||
"--collect-all",
|
||||
"kokoro",
|
||||
"--hidden-import",
|
||||
"kokoro.pipeline",
|
||||
"--hidden-import",
|
||||
"kokoro.model",
|
||||
"--hidden-import",
|
||||
"kokoro.istftnet",
|
||||
"--hidden-import",
|
||||
"kokoro.modules",
|
||||
"--hidden-import",
|
||||
"kokoro.custom_stft",
|
||||
# misaki ships G2P data files (dictionaries, phoneme tables)
|
||||
# that must be bundled for espeak/en/ja/zh G2P to work
|
||||
"--collect-all",
|
||||
|
||||
@@ -89,6 +89,14 @@ def resolve_storage_path(path: str | Path | None) -> Path | None:
|
||||
|
||||
return stored_path
|
||||
|
||||
# 0.3.0 records sometimes stored relative paths with the data-dir name
|
||||
# baked in (e.g. "data/profiles/..."). Joining those directly with
|
||||
# _data_dir produces a spurious "<data_dir>/data/profiles/..." nest.
|
||||
if stored_path.parts and stored_path.parts[0] == "data":
|
||||
stored_path = (
|
||||
Path(*stored_path.parts[1:]) if len(stored_path.parts) > 1 else Path()
|
||||
)
|
||||
|
||||
return (_data_dir / stored_path).resolve()
|
||||
|
||||
|
||||
|
||||
@@ -182,6 +182,7 @@ class HealthResponse(BaseModel):
|
||||
vram_used_mb: Optional[float] = None
|
||||
backend_type: Optional[str] = None # Backend type (mlx or pytorch)
|
||||
backend_variant: Optional[str] = None # Binary variant (cpu or cuda)
|
||||
gpu_compatibility_warning: Optional[str] = None # Warning if GPU arch unsupported
|
||||
|
||||
|
||||
class DirectoryCheck(BaseModel):
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""
|
||||
Force scipy.stats._distn_infrastructure to be bundled with its .py source file
|
||||
alongside the .pyc bytecode.
|
||||
|
||||
The runtime hook in backend/pyi_rth_torch_compiler_disable.py patches this
|
||||
module's source at load time (the module has a `del obj` at line 369 that
|
||||
raises NameError under PyInstaller's frozen importer). That patch reads the
|
||||
source via loader.get_source(), which only works if the .py file was
|
||||
actually collected into the bundle.
|
||||
"""
|
||||
|
||||
module_collection_mode = "pyz+py"
|
||||
@@ -0,0 +1,11 @@
|
||||
"""
|
||||
Force transformers.masking_utils to be bundled with its .py source alongside
|
||||
the .pyc bytecode so the runtime hook in
|
||||
backend/pyi_rth_torch_compiler_disable.py can source-patch it.
|
||||
|
||||
The patch forces the torch<2.6 code path, bypassing `with TransformGetItemToIndex()`
|
||||
which our torch._dynamo no-op stub can't implement for real — the real context
|
||||
manager uses dynamo graph transforms to avoid `.item()` calls inside vmap.
|
||||
"""
|
||||
|
||||
module_collection_mode = "pyz+py"
|
||||
@@ -0,0 +1,95 @@
|
||||
"""
|
||||
PyInstaller runtime hook: numpy 2.x / torch ABI mismatch fix.
|
||||
|
||||
Problem
|
||||
-------
|
||||
torch is compiled against numpy 1.x headers. numpy 2.x changed the version
|
||||
number returned by PyArray_GetNDArrayCVersion() (0x01000009 → 0x02000000),
|
||||
so torch's is_numpy_available() returns False and every torch.from_numpy()
|
||||
call raises:
|
||||
|
||||
RuntimeError: Numpy is not available
|
||||
|
||||
This surfaces as:
|
||||
|
||||
ValueError: Unable to create tensor, you should probably activate
|
||||
padding with 'padding=True'
|
||||
|
||||
during TTS generation (EncodecFeatureExtractor → BatchFeature.convert_to_tensors).
|
||||
|
||||
Fix
|
||||
---
|
||||
Runtime hooks execute after PyInstaller's FrozenImporter is registered, so
|
||||
frozen torch/numpy are importable here. We start a background thread that
|
||||
waits for torch to finish loading then wraps torch.from_numpy with a ctypes
|
||||
memmove fallback that bypasses the C-level numpy ABI check entirely.
|
||||
|
||||
This approach works with any numpy version and is safer than binary-patching
|
||||
libtorch_python.dylib (which risks PyArray_Descr struct layout mismatches).
|
||||
"""
|
||||
|
||||
import sys
|
||||
import threading
|
||||
|
||||
|
||||
def _patch_torch_from_numpy():
|
||||
import time
|
||||
|
||||
for _ in range(7200): # poll up to 360 s at 50 ms intervals
|
||||
time.sleep(0.05)
|
||||
torch = sys.modules.get("torch")
|
||||
if torch is None or not hasattr(torch, "from_numpy"):
|
||||
continue
|
||||
if getattr(torch, "_vb_from_numpy_patched", False):
|
||||
return
|
||||
try:
|
||||
import ctypes
|
||||
import numpy as np
|
||||
|
||||
_orig = torch.from_numpy
|
||||
|
||||
# Explicit numpy → torch dtype map. Silent fallback to float32 on
|
||||
# unknown dtypes would reinterpret the memcpy'd bytes as fp32 and
|
||||
# silently corrupt data (e.g. fp16 tensors from some TTS engines),
|
||||
# so we raise instead.
|
||||
dtype_map = {
|
||||
"float16": _t.float16,
|
||||
"float32": _t.float32,
|
||||
"float64": _t.float64,
|
||||
"int8": _t.int8,
|
||||
"int16": _t.int16,
|
||||
"int32": _t.int32,
|
||||
"int64": _t.int64,
|
||||
"uint8": _t.uint8,
|
||||
"bool": _t.bool,
|
||||
"complex64": _t.complex64,
|
||||
"complex128": _t.complex128,
|
||||
}
|
||||
|
||||
def _safe_from_numpy(
|
||||
arr, _orig=_orig, _c=ctypes, _np=np, _t=torch, _map=dtype_map
|
||||
):
|
||||
try:
|
||||
return _orig(arr)
|
||||
except RuntimeError:
|
||||
a = _np.ascontiguousarray(arr)
|
||||
key = str(a.dtype)
|
||||
if key not in _map:
|
||||
raise TypeError(
|
||||
f"pyi_rth_numpy_compat: unsupported numpy dtype "
|
||||
f"{key!r} in torch.from_numpy fallback; add an "
|
||||
f"explicit mapping rather than silently copying "
|
||||
f"bytes into the wrong dtype."
|
||||
)
|
||||
out = _t.empty(list(a.shape), dtype=_map[key])
|
||||
_c.memmove(out.data_ptr(), a.ctypes.data, a.nbytes)
|
||||
return out
|
||||
|
||||
torch.from_numpy = _safe_from_numpy
|
||||
torch._vb_from_numpy_patched = True
|
||||
except Exception:
|
||||
pass
|
||||
return
|
||||
|
||||
|
||||
threading.Thread(target=_patch_torch_from_numpy, daemon=True).start()
|
||||
@@ -0,0 +1,540 @@
|
||||
"""
|
||||
PyInstaller runtime hook: stub torch._dynamo to a no-op module.
|
||||
|
||||
Problem
|
||||
-------
|
||||
transformers triggers torch._dynamo import at module-load time (not just
|
||||
when torch.compile is called) via class-body decorators:
|
||||
|
||||
transformers/modeling_utils.py:1984
|
||||
@torch._dynamo.allow_in_graph
|
||||
class PreTrainedModel(...)
|
||||
|
||||
transformers/integrations/flex_attention.py:61
|
||||
@torch.compiler.disable(recursive=False)
|
||||
class WrappedFlexAttention...
|
||||
|
||||
The attribute access triggers torch.__getattr__ -> importlib.import_module
|
||||
-> torch._dynamo -> torch._dynamo.utils imports torch._numpy ->
|
||||
torch._numpy._ndarray imports torch._numpy._ufuncs, which crashes under
|
||||
PyInstaller with:
|
||||
|
||||
File "torch/_numpy/_ufuncs.py", line 235, in <module>
|
||||
vars()[name] = deco_binary_ufunc(ufunc)
|
||||
NameError: name 'name' is not defined
|
||||
|
||||
(The module-level `for name in _binary: vars()[name] = ...` pattern works
|
||||
in a regular venv but fails in the PyInstaller bundle. Root cause is in
|
||||
PyInstaller's importer / bytecode pipeline and not easily fixed upstream.)
|
||||
|
||||
Surfaces as Kokoro failing to load when `from transformers import AlbertModel`
|
||||
trips the decorator chain.
|
||||
|
||||
Fix
|
||||
---
|
||||
voicebox never uses torch.compile / torch._dynamo for inference, so we
|
||||
replace torch._dynamo with a no-op stub module before transformers is
|
||||
imported. Any attribute access on the stub returns a pass-through callable,
|
||||
so `@torch._dynamo.allow_in_graph`, `torch._dynamo.is_compiling()`,
|
||||
`torch._dynamo.mark_static_address(...)`, etc. all work.
|
||||
|
||||
This hook is pure sys.modules manipulation — we deliberately do NOT import
|
||||
torch here. Runtime hooks run before the app starts and before
|
||||
pyi_rth_numpy_compat has had a chance to patch torch.from_numpy (it runs
|
||||
in a background thread, waiting for torch to appear in sys.modules).
|
||||
Eager-importing torch at hook time would trip the numpy ABI issue and
|
||||
kill the server process at startup.
|
||||
|
||||
torch.compiler.disable does not need a separate stub: its implementation
|
||||
is effectively `import torch._dynamo; return torch._dynamo.disable(...)`,
|
||||
and since our stub is in sys.modules, that call resolves to our no-op
|
||||
_NoopDecorator pass-through.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
|
||||
|
||||
# Diagnostics — log hook activity to a file alongside the bundle so we can
|
||||
# see what's happening when the server is run as a sidecar (no stdout for
|
||||
# runtime hook prints). Safe no-op if the file can't be written.
|
||||
_DIAG_PATH = os.path.join(tempfile.gettempdir(), "voicebox_rt_hook.log")
|
||||
|
||||
|
||||
def _diag(msg: str) -> None:
|
||||
try:
|
||||
with open(_DIAG_PATH, "a", encoding="utf-8") as f:
|
||||
f.write(msg + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
_HOOK_VERSION = "v6-masking-utils-finder"
|
||||
_diag(f"=== runtime hook load @ pid={os.getpid()} version={_HOOK_VERSION} ===")
|
||||
|
||||
|
||||
class _NoopDecorator:
|
||||
"""Multi-role no-op: decorator, falsey predicate, and context manager.
|
||||
|
||||
Returned from calls like `torch._dynamo.disable()` (decorator),
|
||||
`torch._dynamo.is_compiling()` (predicate used in `if not ...`), and
|
||||
`with torch._dynamo._trace_wrapped_higher_order_op.TransformGetItemToIndex():`
|
||||
(context manager used to scope an fx graph transformation).
|
||||
|
||||
By implementing __call__, __bool__, __enter__, __exit__, and __iter__ we
|
||||
cover every use pattern we've seen transformers/torch use on a stubbed
|
||||
object. Anything we haven't covered will raise a clearer error than a
|
||||
silent wrong-result.
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
def __call__(self, fn=None, *args, **kwargs):
|
||||
return fn
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return False
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
return False # don't suppress exceptions
|
||||
|
||||
def __iter__(self):
|
||||
return iter(())
|
||||
|
||||
|
||||
_noop_decorator_singleton = _NoopDecorator()
|
||||
|
||||
|
||||
def _noop_callable(*args, **kwargs):
|
||||
# Direct-decorator use: @torch._dynamo.foo (no parens) — fn is positional
|
||||
if len(args) == 1 and callable(args[0]) and not kwargs:
|
||||
return args[0]
|
||||
# Side-effect call with non-callable arg(s), e.g. mark_static_address(tensor)
|
||||
return _noop_decorator_singleton
|
||||
|
||||
|
||||
class _NoopDynamoModule(types.ModuleType):
|
||||
"""Permissive stub: every attribute is a pass-through callable.
|
||||
|
||||
Covers attributes transformers hits at import time (allow_in_graph) and
|
||||
runtime (is_compiling, mark_static_address, reset, disable, ...).
|
||||
|
||||
Dunder attributes (__file__, __spec__, __loader__, ...) raise
|
||||
AttributeError so probes like inspect.getmodule() — which does
|
||||
`hasattr(m, '__file__')` then `os.path.normpath(m.__file__)` — see the
|
||||
module as having no source file and fall through to its normal
|
||||
handling, instead of receiving a function and blowing up.
|
||||
"""
|
||||
|
||||
def __getattr__(self, name: str):
|
||||
if name.startswith("__") and name.endswith("__"):
|
||||
raise AttributeError(name)
|
||||
return _noop_callable
|
||||
|
||||
|
||||
class _DynamoLoader:
|
||||
"""Loader used by _DynamoMetaPathFinder to materialise stub submodules."""
|
||||
|
||||
def create_module(self, spec):
|
||||
return _NoopDynamoModule(spec.name)
|
||||
|
||||
def exec_module(self, module):
|
||||
# Mark every stub submodule as a package so deeper submodule imports
|
||||
# (`from torch._dynamo.X.Y import Z`) keep working.
|
||||
module.__path__ = []
|
||||
|
||||
|
||||
class _DynamoMetaPathFinder:
|
||||
"""Resolve any `torch._dynamo.X[.Y...]` import to a no-op stub module.
|
||||
|
||||
Without this, `from torch._dynamo._trace_wrapped_higher_order_op import X`
|
||||
fails even with torch._dynamo pre-populated in sys.modules — Python's
|
||||
import machinery checks the parent's __path__ and then looks up the
|
||||
child, and we need to provide both.
|
||||
"""
|
||||
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname == "torch._dynamo":
|
||||
return None # handled by the pre-populated sys.modules entry
|
||||
if not fullname.startswith("torch._dynamo."):
|
||||
return None
|
||||
from importlib.machinery import ModuleSpec
|
||||
|
||||
return ModuleSpec(fullname, _DynamoLoader(), is_package=True)
|
||||
|
||||
|
||||
class _TransformersStubFinder:
|
||||
"""Replace specific transformers submodules with no-op stubs.
|
||||
|
||||
Two modules are targeted:
|
||||
|
||||
1. transformers.utils.auto_docstring
|
||||
The real @auto_docstring decorator loads
|
||||
transformers.models.auto.modeling_auto just to build example docstrings,
|
||||
which drags in GenerationMixin -> candidate_generator -> sklearn.metrics
|
||||
-> scipy.stats._distn_infrastructure and trips (2) below. Docstrings
|
||||
aren't functional for inference, so a pass-through decorator is safe.
|
||||
|
||||
2. transformers.generation.candidate_generator
|
||||
Imported at module scope by transformers.generation.utils. It does
|
||||
`from sklearn.metrics import roc_curve` at module load, which triggers:
|
||||
|
||||
File "scipy/stats/_distn_infrastructure.py", line 369, in <module>
|
||||
NameError: name 'obj' is not defined
|
||||
|
||||
This is a PyInstaller-specific module-load bug (same class as the
|
||||
torch._numpy._ufuncs crash) where a module-level `for obj in [s for s
|
||||
in dir() if ...]` loop evaluates to empty in the bundle, leaving `obj`
|
||||
unbound before `del obj`.
|
||||
|
||||
The exports (AssistedCandidateGenerator, EarlyExitCandidateGenerator,
|
||||
etc.) are speculative-decoding helpers voicebox's TTS engines do not
|
||||
use; a no-op stub module satisfies the imports.
|
||||
"""
|
||||
|
||||
_STUBBED_MODULES = frozenset(
|
||||
{
|
||||
"transformers.utils.auto_docstring",
|
||||
"transformers.generation.candidate_generator",
|
||||
}
|
||||
)
|
||||
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname not in self._STUBBED_MODULES:
|
||||
return None
|
||||
from importlib.machinery import ModuleSpec
|
||||
|
||||
return ModuleSpec(fullname, _NoopStubLoader(), is_package=False)
|
||||
|
||||
|
||||
class _NoopStubLoader:
|
||||
def create_module(self, spec):
|
||||
return _NoopDynamoModule(spec.name)
|
||||
|
||||
def exec_module(self, module):
|
||||
# _NoopDynamoModule.__getattr__ already answers every non-dunder
|
||||
# attribute with a pass-through callable, which satisfies
|
||||
# `from stubbed_module import X` for any X.
|
||||
pass
|
||||
|
||||
|
||||
def _patch_scipy_distn_source(source: str) -> str:
|
||||
"""Replace the unsafe `del obj` with a no-op that survives when obj is unbound.
|
||||
|
||||
Returns the input unchanged if the target line isn't found (e.g. scipy
|
||||
version has changed).
|
||||
"""
|
||||
target = "\ndel obj\n"
|
||||
replacement = "\nglobals().pop('obj', None)\n"
|
||||
if target in source:
|
||||
return source.replace(target, replacement, 1)
|
||||
return source
|
||||
|
||||
|
||||
def _patch_masking_utils_source(source: str) -> str:
|
||||
"""Force torch<2.6 code path in transformers.masking_utils.
|
||||
|
||||
The torch>=2.6 path uses `with TransformGetItemToIndex():` to allow
|
||||
`.item()` calls inside vmap. That context manager is implemented via
|
||||
torch._dynamo graph transforms, which our stub doesn't reproduce — it's
|
||||
a no-op. The inner `_vmap_for_bhqkv` then crashes with:
|
||||
|
||||
RuntimeError: vmap: It looks like you're calling .item() on a Tensor.
|
||||
|
||||
Forcing the torch<2.6 flag off selects sdpa_mask_older_torch which uses
|
||||
a different vmap pattern that does not hit .item() and does not need
|
||||
TransformGetItemToIndex.
|
||||
"""
|
||||
target = 'is_torch_greater_or_equal("2.6", accept_dev=True)'
|
||||
# Find the specific line that assigns _is_torch_greater_or_equal_than_2_6
|
||||
if "_is_torch_greater_or_equal_than_2_6 = " + target in source:
|
||||
return source.replace(
|
||||
"_is_torch_greater_or_equal_than_2_6 = " + target,
|
||||
"_is_torch_greater_or_equal_than_2_6 = False",
|
||||
1,
|
||||
)
|
||||
return source
|
||||
|
||||
|
||||
class _SourcePatchingFinder:
|
||||
"""Generic delegate-and-wrap meta-path finder that patches a module's
|
||||
source before exec'ing.
|
||||
|
||||
Subclasses declare `target` (module fullname) and `patch` (str->str).
|
||||
Requires the target module's .py source to be bundled (use a PyInstaller
|
||||
hook setting module_collection_mode = "pyz+py").
|
||||
"""
|
||||
|
||||
target: str
|
||||
patch_fn: callable = None
|
||||
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname != self.target:
|
||||
return None
|
||||
for finder in sys.meta_path:
|
||||
if finder is self:
|
||||
continue
|
||||
find = getattr(finder, "find_spec", None)
|
||||
if find is None:
|
||||
continue
|
||||
try:
|
||||
real_spec = find(fullname, path, target)
|
||||
except Exception:
|
||||
continue
|
||||
if real_spec is None or real_spec.loader is None:
|
||||
continue
|
||||
real_spec.loader = _SourcePatchLoader(real_spec.loader, self.patch_fn)
|
||||
return real_spec
|
||||
return None
|
||||
|
||||
|
||||
class _SourcePatchLoader:
|
||||
"""Delegate loader that reads source via get_source, applies a patch, and
|
||||
compile/exec's the patched text into module.__dict__.
|
||||
"""
|
||||
|
||||
def __init__(self, inner, patch_fn):
|
||||
self._inner = inner
|
||||
self._patch_fn = patch_fn
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._inner, name)
|
||||
|
||||
def create_module(self, spec):
|
||||
return self._inner.create_module(spec)
|
||||
|
||||
def exec_module(self, module):
|
||||
source = None
|
||||
try:
|
||||
source = self._inner.get_source(module.__name__)
|
||||
except Exception as e:
|
||||
_diag(f"[source-patch] get_source({module.__name__}) failed: {e!r}")
|
||||
|
||||
if not source:
|
||||
_diag(
|
||||
f"[source-patch] no source for {module.__name__}; "
|
||||
"falling back to inner exec_module (patch NOT applied)"
|
||||
)
|
||||
self._inner.exec_module(module)
|
||||
return
|
||||
|
||||
patched = self._patch_fn(source)
|
||||
_diag(
|
||||
f"[source-patch] {module.__name__}: "
|
||||
f"patched={patched is not source}, len={len(patched)}"
|
||||
)
|
||||
spec = module.__spec__
|
||||
if spec is not None and spec.submodule_search_locations is not None:
|
||||
module.__path__ = spec.submodule_search_locations
|
||||
filename = getattr(self._inner, "path", module.__name__)
|
||||
exec(compile(patched, filename, "exec"), module.__dict__)
|
||||
_diag(f"[source-patch] {module.__name__} OK")
|
||||
|
||||
|
||||
class _MaskingUtilsFinder(_SourcePatchingFinder):
|
||||
target = "transformers.masking_utils"
|
||||
patch_fn = staticmethod(_patch_masking_utils_source)
|
||||
|
||||
|
||||
class _ScipyDistnPatchingFinder:
|
||||
"""Delegate-and-wrap finder for scipy.stats._distn_infrastructure.
|
||||
|
||||
That module ends with:
|
||||
|
||||
for obj in [s for s in dir() if s.startswith('_doc_')]:
|
||||
exec('del ' + obj)
|
||||
del obj
|
||||
|
||||
In the PyInstaller bundle the list comprehension evaluates to empty
|
||||
(module-level dir() under the frozen importer returns a different scope
|
||||
than CPython's normal module-exec path — same class of bug as the
|
||||
torch._numpy._ufuncs crash). The for loop body doesn't run, `obj` is
|
||||
never bound, and the trailing `del obj` raises NameError at module load.
|
||||
|
||||
This kills every downstream module: librosa (needed by nearly every TTS
|
||||
engine for mel filters) -> scipy.signal -> scipy.stats -> here.
|
||||
|
||||
Workaround: delegate to the real loader, but pre-bind `obj = None` in the
|
||||
module namespace before its bytecode runs. If the for loop executes, each
|
||||
iteration overwrites the sentinel via STORE_NAME (normal behaviour). If it
|
||||
doesn't, `del obj` removes the sentinel and module load succeeds. The
|
||||
`_doc_*` cleanup this line was meant to do is purely cosmetic — those vars
|
||||
stay in the module namespace but nothing references them after this point.
|
||||
"""
|
||||
|
||||
_TARGET = "scipy.stats._distn_infrastructure"
|
||||
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname != self._TARGET:
|
||||
return None
|
||||
_diag(f"[scipy-finder] match: {fullname}, path={path!r}")
|
||||
# Delegate to the other finders to locate the real spec
|
||||
for finder in sys.meta_path:
|
||||
if finder is self:
|
||||
continue
|
||||
find = getattr(finder, "find_spec", None)
|
||||
if find is None:
|
||||
continue
|
||||
try:
|
||||
real_spec = find(fullname, path, target)
|
||||
except Exception as e:
|
||||
_diag(f"[scipy-finder] inner finder {type(finder).__name__} raised: {e}")
|
||||
continue
|
||||
if real_spec is None:
|
||||
continue
|
||||
if real_spec.loader is None:
|
||||
_diag(f"[scipy-finder] {type(finder).__name__} returned spec with loader=None")
|
||||
continue
|
||||
_diag(
|
||||
f"[scipy-finder] wrapped loader from "
|
||||
f"{type(finder).__name__} -> {type(real_spec.loader).__name__}"
|
||||
)
|
||||
real_spec.loader = _ScipyDistnPrebindLoader(real_spec.loader)
|
||||
return real_spec
|
||||
_diag("[scipy-finder] NO inner finder returned a spec")
|
||||
return None
|
||||
|
||||
|
||||
class _ScipyDistnPrebindLoader:
|
||||
"""Thin wrapper that pre-binds `obj = None` before delegating to the
|
||||
real PyInstaller loader.
|
||||
|
||||
Every other attribute/method delegates to the inner loader — PyiFrozenLoader
|
||||
is a rich FileLoader/ExecutionLoader with get_code/get_source/get_filename/
|
||||
is_package/get_resource_reader/etc., any of which Python's import machinery
|
||||
or 3rd-party code may call on spec.loader. Forwarding via __getattr__
|
||||
avoids breaking any of those paths (and preserves @_check_name contracts
|
||||
because the decorated methods run on the inner instance where self.name
|
||||
matches spec.name).
|
||||
"""
|
||||
|
||||
def __init__(self, inner):
|
||||
self._inner = inner
|
||||
|
||||
def __getattr__(self, name):
|
||||
# __getattr__ fires only for attrs not already on self, so delegate
|
||||
# everything that isn't create_module/exec_module (or __getattr__/init).
|
||||
return getattr(self._inner, name)
|
||||
|
||||
def create_module(self, spec):
|
||||
return self._inner.create_module(spec)
|
||||
|
||||
def exec_module(self, module):
|
||||
# Compile scipy's module source with the problematic line patched.
|
||||
#
|
||||
# The real module ends with:
|
||||
# for obj in [s for s in dir() if s.startswith('_doc_')]:
|
||||
# exec('del ' + obj)
|
||||
# del obj
|
||||
#
|
||||
# Under PyInstaller's frozen importer, `del obj` raises NameError
|
||||
# even when we pre-populate module.__dict__['obj'] — the pre-compiled
|
||||
# .pyc bytecode interacts with the frame setup differently than a
|
||||
# fresh compile() from source. Easiest robust fix: read the source
|
||||
# and replace `del obj` with a safe variant before compiling.
|
||||
#
|
||||
# Requires the .py source to be bundled alongside the .pyc — see
|
||||
# backend/pyi_hooks/hook-scipy.stats._distn_infrastructure.py.
|
||||
source = None
|
||||
try:
|
||||
source = self._inner.get_source(module.__name__)
|
||||
except Exception as e:
|
||||
_diag(f"[scipy-loader] get_source failed: {e!r}")
|
||||
|
||||
if source:
|
||||
patched = _patch_scipy_distn_source(source)
|
||||
_diag(
|
||||
f"[scipy-loader] source-patch path: patched={patched is not source}, "
|
||||
f"len={len(patched)}"
|
||||
)
|
||||
spec = module.__spec__
|
||||
if spec is not None and spec.submodule_search_locations is not None:
|
||||
module.__path__ = spec.submodule_search_locations
|
||||
filename = getattr(self._inner, "path", module.__name__)
|
||||
bytecode = compile(patched, filename, "exec")
|
||||
try:
|
||||
exec(bytecode, module.__dict__)
|
||||
except Exception as e:
|
||||
_diag(f"[scipy-loader] patched exec raised {type(e).__name__}: {e!r}")
|
||||
raise
|
||||
_diag(f"[scipy-loader] exec_module {module.__name__} OK (source-patched)")
|
||||
return
|
||||
|
||||
# No source available — fall back to the pre-bind approach. This is
|
||||
# best-effort; if the frozen .pyc really does see a different `obj`
|
||||
# slot, this will still crash, but we've done all we can without
|
||||
# source.
|
||||
_diag("[scipy-loader] no source available; falling back to pre-bind")
|
||||
module.__dict__["obj"] = None
|
||||
self._inner.exec_module(module)
|
||||
|
||||
|
||||
def _install_dynamo_stub() -> None:
|
||||
stub = _NoopDynamoModule("torch._dynamo")
|
||||
# Mark as a package so `from torch._dynamo.X import Y` imports work
|
||||
# (Python's import machinery checks parent.__path__ before looking up
|
||||
# the child).
|
||||
stub.__path__ = []
|
||||
# torch._dynamo.config is accessed as a nested attribute namespace
|
||||
# (e.g. `torch._dynamo.config.capture_scalar_outputs = True`), so use
|
||||
# a permissive module so any attr read returns a no-op and sets succeed.
|
||||
stub.config = _NoopDynamoModule("torch._dynamo.config")
|
||||
stub.config.__path__ = []
|
||||
sys.modules["torch._dynamo"] = stub
|
||||
sys.modules["torch._dynamo.config"] = stub.config
|
||||
|
||||
# Finders:
|
||||
# - torch._dynamo.* submodules -> no-op stubs
|
||||
# - transformers.utils.auto_docstring and
|
||||
# transformers.generation.candidate_generator -> no-op stubs (both
|
||||
# paths reach sklearn -> scipy.stats which trips a separate crash)
|
||||
# - scipy.stats._distn_infrastructure -> real load with `obj` pre-bound,
|
||||
# so librosa -> scipy.signal -> scipy.stats loads cleanly
|
||||
for _FinderCls in (
|
||||
_DynamoMetaPathFinder,
|
||||
_TransformersStubFinder,
|
||||
_ScipyDistnPatchingFinder,
|
||||
_MaskingUtilsFinder,
|
||||
):
|
||||
try:
|
||||
sys.meta_path.insert(0, _FinderCls())
|
||||
_diag(f"installed finder: {_FinderCls.__name__}")
|
||||
except Exception as e:
|
||||
_diag(f"FAILED to install {_FinderCls.__name__}: {e!r}")
|
||||
_diag(
|
||||
"final sys.meta_path head: "
|
||||
+ ", ".join(type(f).__name__ for f in sys.meta_path[:6])
|
||||
)
|
||||
|
||||
# If torch is already imported, also set the attribute on the package so
|
||||
# `torch._dynamo` resolves to our stub without triggering torch.__getattr__
|
||||
# (which would lazy-import the real module and crash).
|
||||
torch_mod = sys.modules.get("torch")
|
||||
if torch_mod is not None:
|
||||
torch_mod._dynamo = stub
|
||||
|
||||
|
||||
try:
|
||||
_install_dynamo_stub()
|
||||
except Exception as _e:
|
||||
# Best effort. If this fails the original NameError will surface when
|
||||
# transformers imports — no worse than not patching at all.
|
||||
_diag(f"_install_dynamo_stub FAILED: {_e!r}")
|
||||
|
||||
# NOTE: we deliberately do NOT import torch or torch.compiler here.
|
||||
# Runtime hooks run before the app starts and before pyi_rth_numpy_compat
|
||||
# has had a chance to patch torch.from_numpy (it runs in a background
|
||||
# thread, waiting for torch to appear in sys.modules). Importing torch
|
||||
# eagerly at hook time would trip the numpy ABI issue and kill the
|
||||
# server process at startup.
|
||||
#
|
||||
# torch.compiler.disable does not need an explicit stub: its
|
||||
# implementation is effectively `import torch._dynamo; return
|
||||
# torch._dynamo.disable(fn, recursive, reason=reason)`, and since our
|
||||
# stub is installed in sys.modules, that call resolves to our no-op
|
||||
# _NoopDecorator pass-through.
|
||||
@@ -2,4 +2,14 @@
|
||||
# These should only be installed on aarch64-apple-darwin platforms
|
||||
|
||||
mlx>=0.30.0
|
||||
mlx-audio>=0.3.1
|
||||
|
||||
# NOTE: mlx-audio is intentionally not listed here. From 0.3.1 onward it
|
||||
# declares `transformers==5.0.0rc3` / `>=5.0.0`, which conflicts with the
|
||||
# `transformers<=4.57.6` cap in requirements.txt and breaks CI's clean
|
||||
# resolver. The mlx-audio API surface we use (mlx_audio.tts.load,
|
||||
# mlx_audio.stt.load) works fine on transformers 4.57.x in practice.
|
||||
#
|
||||
# Install it via `pip install --no-deps mlx-audio==0.4.1` after this file
|
||||
# (see .github/workflows/release.yml). All other mlx-audio runtime deps
|
||||
# (huggingface_hub, librosa, miniaudio, mlx-lm, numba, numpy, protobuf,
|
||||
# pyloudnorm, sounddevice, tqdm) are already in requirements.txt.
|
||||
|
||||
@@ -8,7 +8,7 @@ sqlalchemy>=2.0.0
|
||||
alembic>=1.13.0
|
||||
|
||||
# ML models
|
||||
torch>=2.7.0
|
||||
torch>=2.2.0
|
||||
transformers>=4.36.0,<=4.57.6
|
||||
accelerate>=0.26.0
|
||||
huggingface_hub>=0.20.0
|
||||
@@ -50,7 +50,7 @@ en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_
|
||||
# Audio processing
|
||||
librosa>=0.10.0
|
||||
soundfile>=0.12.0
|
||||
numpy>=1.24.0
|
||||
numpy>=1.24.0,<2.0
|
||||
numba>=0.60.0,<0.61.0
|
||||
pedalboard>=0.9.0
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ from .. import models
|
||||
from ..services import history, profiles, tts
|
||||
from ..database import Generation as DBGeneration, VoiceProfile as DBVoiceProfile, get_db
|
||||
from ..services.generation import run_generation
|
||||
from ..services.task_queue import enqueue_generation
|
||||
from ..services.task_queue import cancel_generation as cancel_generation_job, enqueue_generation
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
router = APIRouter()
|
||||
@@ -82,6 +82,7 @@ async def generate_speech(
|
||||
pass
|
||||
|
||||
enqueue_generation(
|
||||
generation_id,
|
||||
run_generation(
|
||||
generation_id=generation_id,
|
||||
profile_id=data.profile_id,
|
||||
@@ -127,6 +128,7 @@ async def retry_generation(generation_id: str, db: Session = Depends(get_db)):
|
||||
)
|
||||
|
||||
enqueue_generation(
|
||||
generation_id,
|
||||
run_generation(
|
||||
generation_id=generation_id,
|
||||
profile_id=gen.profile_id,
|
||||
@@ -170,6 +172,7 @@ async def regenerate_generation(generation_id: str, db: Session = Depends(get_db
|
||||
version_id = str(uuid.uuid4())
|
||||
|
||||
enqueue_generation(
|
||||
generation_id,
|
||||
run_generation(
|
||||
generation_id=generation_id,
|
||||
profile_id=gen.profile_id,
|
||||
@@ -187,6 +190,34 @@ async def regenerate_generation(generation_id: str, db: Session = Depends(get_db
|
||||
return models.GenerationResponse.model_validate(gen)
|
||||
|
||||
|
||||
@router.post("/generate/{generation_id}/cancel")
|
||||
async def cancel_generation(generation_id: str, db: Session = Depends(get_db)):
|
||||
"""Cancel a queued or running generation."""
|
||||
gen = db.query(DBGeneration).filter_by(id=generation_id).first()
|
||||
if not gen:
|
||||
raise HTTPException(status_code=404, detail="Generation not found")
|
||||
|
||||
if (gen.status or "completed") not in ("loading_model", "generating"):
|
||||
raise HTTPException(status_code=400, detail="Only active generations can be cancelled")
|
||||
|
||||
cancellation_state = cancel_generation_job(generation_id)
|
||||
if cancellation_state is None:
|
||||
raise HTTPException(status_code=409, detail="Generation is no longer cancellable")
|
||||
|
||||
if cancellation_state == "queued":
|
||||
task_manager = get_task_manager()
|
||||
task_manager.complete_generation(generation_id)
|
||||
await history.update_generation_status(
|
||||
generation_id=generation_id,
|
||||
status="failed",
|
||||
db=db,
|
||||
error="Generation cancelled",
|
||||
)
|
||||
return {"message": "Queued generation cancelled"}
|
||||
|
||||
return {"message": "Generation cancellation requested"}
|
||||
|
||||
|
||||
@router.get("/generate/{generation_id}/status")
|
||||
async def get_generation_status(generation_id: str, db: Session = Depends(get_db)):
|
||||
"""SSE endpoint that streams generation status updates."""
|
||||
|
||||
@@ -93,6 +93,12 @@ async def health():
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
gpu_compat_warning = None
|
||||
if has_cuda:
|
||||
from ..backends.base import check_cuda_compatibility
|
||||
|
||||
_compatible, gpu_compat_warning = check_cuda_compatibility()
|
||||
|
||||
gpu_available = has_cuda or has_mps or has_xpu or has_directml or backend_type == "mlx"
|
||||
|
||||
gpu_type = None
|
||||
@@ -110,6 +116,11 @@ async def health():
|
||||
vram_used = None
|
||||
if has_cuda:
|
||||
vram_used = torch.cuda.memory_allocated() / 1024 / 1024
|
||||
elif has_xpu:
|
||||
try:
|
||||
vram_used = torch.xpu.memory_allocated() / 1024 / 1024
|
||||
except Exception:
|
||||
pass # memory_allocated() may not be available on all IPEX versions
|
||||
|
||||
model_loaded = False
|
||||
model_size = None
|
||||
@@ -162,7 +173,11 @@ async def health():
|
||||
gpu_type=gpu_type,
|
||||
vram_used_mb=vram_used,
|
||||
backend_type=backend_type,
|
||||
backend_variant=os.environ.get("VOICEBOX_BACKEND_VARIANT", "cuda" if torch.cuda.is_available() else "cpu"),
|
||||
backend_variant=os.environ.get(
|
||||
"VOICEBOX_BACKEND_VARIANT",
|
||||
"cuda" if torch.cuda.is_available() else ("xpu" if has_xpu else "cpu"),
|
||||
),
|
||||
gpu_compatibility_warning=gpu_compat_warning,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -62,6 +62,13 @@ async def import_generation(
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.delete("/history/failed")
|
||||
async def clear_failed_generations(db: Session = Depends(get_db)):
|
||||
"""Delete every generation with status='failed'. Used by the UI's 'Clear failed' button (#410)."""
|
||||
count = await history.delete_failed_generations(db)
|
||||
return {"deleted": count}
|
||||
|
||||
|
||||
@router.get("/history/{generation_id}", response_model=models.HistoryResponse)
|
||||
async def get_generation(
|
||||
generation_id: str,
|
||||
@@ -89,6 +96,11 @@ async def get_generation(
|
||||
duration=gen.duration,
|
||||
seed=gen.seed,
|
||||
instruct=gen.instruct,
|
||||
engine=gen.engine or "qwen",
|
||||
model_size=gen.model_size,
|
||||
status=gen.status or "completed",
|
||||
error=gen.error,
|
||||
is_favorited=bool(gen.is_favorited),
|
||||
created_at=gen.created_at,
|
||||
)
|
||||
|
||||
|
||||
@@ -135,14 +135,15 @@ async def migrate_models(request: models.ModelMigrateRequest):
|
||||
if destination.resolve().is_relative_to(source.resolve()):
|
||||
raise HTTPException(status_code=400, detail="Destination cannot be inside the current cache directory")
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
model_dirs = [d for d in source.iterdir() if d.name.startswith("models--") and d.is_dir()]
|
||||
if not model_dirs:
|
||||
progress_manager.update_progress("migration", 1, 1, status="complete")
|
||||
progress_manager.mark_complete("migration")
|
||||
return {"moved": 0, "errors": [], "source": str(source), "destination": str(destination)}
|
||||
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
|
||||
same_fs = False
|
||||
try:
|
||||
same_fs = source.stat().st_dev == destination.stat().st_dev
|
||||
|
||||
@@ -4,6 +4,7 @@ import io
|
||||
import json as _json
|
||||
import logging
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
|
||||
|
||||
@@ -105,6 +105,11 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
|
||||
This is the clean shutdown mechanism: instead of the Tauri app trying to
|
||||
forcefully kill the server (which spawns console windows on Windows),
|
||||
the server monitors its parent and shuts itself down gracefully.
|
||||
|
||||
The Tauri app writes a .keep-running sentinel file to data_dir before
|
||||
exiting when "remain running after close" is enabled. This is a reliable
|
||||
fallback for the HTTP /watchdog/disable request, which can race with
|
||||
process exit on Windows.
|
||||
"""
|
||||
import os
|
||||
import signal
|
||||
@@ -164,6 +169,19 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
|
||||
if not alive:
|
||||
watchdog_logger.warning(f"Parent PID {parent_pid} not found on first check — disabling watchdog")
|
||||
return
|
||||
# Clear any stale .keep-running sentinel from a previous session. The
|
||||
# sentinel is only removed by the watchdog when it's consumed during a
|
||||
# grace period; if the HTTP /watchdog/disable path wins the race on a
|
||||
# "keep running" exit, the sentinel is left on disk. Wipe it here so a
|
||||
# future session can't inherit that stale signal.
|
||||
if data_dir:
|
||||
stale = os.path.join(data_dir, ".keep-running")
|
||||
if os.path.exists(stale):
|
||||
try:
|
||||
os.remove(stale)
|
||||
watchdog_logger.info("Removed stale .keep-running sentinel from previous session")
|
||||
except OSError as e:
|
||||
watchdog_logger.warning(f"Failed to remove stale sentinel: {e}")
|
||||
while True:
|
||||
if _watchdog_disabled:
|
||||
watchdog_logger.info("Watchdog disabled (keep server running), stopping monitor")
|
||||
@@ -178,6 +196,18 @@ def _start_parent_watchdog(parent_pid, data_dir=None):
|
||||
if _watchdog_disabled:
|
||||
watchdog_logger.info("Watchdog was disabled during grace period, keeping server alive")
|
||||
return
|
||||
# Check for sentinel file written by Tauri before exit.
|
||||
# This catches the case where the HTTP disable request
|
||||
# didn't arrive before the parent process died (common
|
||||
# on Windows where process teardown is fast).
|
||||
sentinel = os.path.join(data_dir, ".keep-running") if data_dir else None
|
||||
if sentinel and os.path.exists(sentinel):
|
||||
watchdog_logger.info("Found .keep-running sentinel file, keeping server alive")
|
||||
try:
|
||||
os.remove(sentinel)
|
||||
except OSError:
|
||||
pass
|
||||
return
|
||||
watchdog_logger.info("Watchdog still enabled after grace period, shutting down server...")
|
||||
if sys.platform == "win32":
|
||||
# sys.exit triggers SystemExit, allowing uvicorn to run
|
||||
|
||||
@@ -11,6 +11,7 @@ Both archives are extracted into {data_dir}/backends/cuda/ which forms the
|
||||
complete PyInstaller --onedir directory structure that torch expects.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
@@ -34,6 +35,12 @@ PROGRESS_KEY = "cuda-backend"
|
||||
# CUDA toolkit version or torch's CUDA dependency changes (e.g. cu126 -> cu128).
|
||||
CUDA_LIBS_VERSION = "cu128-v1"
|
||||
|
||||
# Prevents concurrent download_cuda_binary() calls from racing on the same
|
||||
# temp file. The auto-update background task and the manual HTTP endpoint
|
||||
# can both invoke download_cuda_binary(); without this lock the progress-
|
||||
# manager status check is a TOCTOU race.
|
||||
_download_lock = asyncio.Lock()
|
||||
|
||||
|
||||
def get_backends_dir() -> Path:
|
||||
"""Directory where downloaded backend binaries are stored."""
|
||||
@@ -241,6 +248,15 @@ async def download_cuda_binary(version: Optional[str] = None):
|
||||
Args:
|
||||
version: Version tag (e.g. "v0.3.0"). Defaults to current app version.
|
||||
"""
|
||||
if _download_lock.locked():
|
||||
logger.info("CUDA download already in progress, skipping duplicate request")
|
||||
return
|
||||
async with _download_lock:
|
||||
await _download_cuda_binary_locked(version)
|
||||
|
||||
|
||||
async def _download_cuda_binary_locked(version: Optional[str] = None):
|
||||
"""Inner implementation of download_cuda_binary, called under _download_lock."""
|
||||
import httpx
|
||||
|
||||
if version is None:
|
||||
|
||||
@@ -11,6 +11,8 @@ from typing import List, Optional
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from ..utils.effects import validate_effects_chain
|
||||
|
||||
from ..database import EffectPreset as DBEffectPreset
|
||||
from ..models import EffectPresetResponse, EffectPresetCreate, EffectPresetUpdate, EffectConfig
|
||||
|
||||
@@ -52,7 +54,6 @@ def get_preset_by_name(name: str, db: Session) -> Optional[EffectPresetResponse]
|
||||
|
||||
def create_preset(data: EffectPresetCreate, db: Session) -> EffectPresetResponse:
|
||||
"""Create a new user effect preset."""
|
||||
from .utils.effects import validate_effects_chain
|
||||
|
||||
chain_dicts = [e.model_dump() for e in data.effects_chain]
|
||||
error = validate_effects_chain(chain_dicts)
|
||||
@@ -94,7 +95,6 @@ def update_preset(preset_id: str, data: EffectPresetUpdate, db: Session) -> Opti
|
||||
if data.description is not None:
|
||||
preset.description = data.description
|
||||
if data.effects_chain is not None:
|
||||
from .utils.effects import validate_effects_chain
|
||||
|
||||
chain_dicts = [e.model_dump() for e in data.effects_chain]
|
||||
error = validate_effects_chain(chain_dicts)
|
||||
|
||||
@@ -16,6 +16,7 @@ Mode differences:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import traceback
|
||||
from typing import Literal, Optional
|
||||
|
||||
@@ -126,6 +127,13 @@ async def run_generation(
|
||||
duration=duration,
|
||||
)
|
||||
|
||||
except asyncio.CancelledError:
|
||||
await history.update_generation_status(
|
||||
generation_id=generation_id,
|
||||
status="failed",
|
||||
db=bg_db,
|
||||
error="Generation cancelled",
|
||||
)
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
await history.update_generation_status(
|
||||
|
||||
@@ -264,6 +264,43 @@ async def delete_generation(
|
||||
return True
|
||||
|
||||
|
||||
async def delete_failed_generations(db: Session) -> int:
|
||||
"""
|
||||
Delete every generation whose status is 'failed'.
|
||||
|
||||
Used by the "Clear failed" action in the UI so users can tidy up
|
||||
history after the model wasn't loaded, the app was closed mid-run,
|
||||
or a generation otherwise errored out (see issue #410).
|
||||
|
||||
Returns:
|
||||
Number of generations deleted.
|
||||
"""
|
||||
from . import versions as versions_mod
|
||||
|
||||
failed = db.query(DBGeneration).filter(DBGeneration.status == "failed").all()
|
||||
count = 0
|
||||
for generation in failed:
|
||||
# Clean up version files/rows first.
|
||||
versions_mod.delete_versions_for_generation(generation.id, db)
|
||||
|
||||
# Remove the main audio file if it somehow made it to disk.
|
||||
if generation.audio_path:
|
||||
audio_path = config.resolve_storage_path(generation.audio_path)
|
||||
if audio_path is not None and audio_path.exists():
|
||||
try:
|
||||
audio_path.unlink()
|
||||
except OSError:
|
||||
# Best-effort cleanup — don't abort the whole sweep
|
||||
# if a single file can't be removed.
|
||||
pass
|
||||
|
||||
db.delete(generation)
|
||||
count += 1
|
||||
|
||||
db.commit()
|
||||
return count
|
||||
|
||||
|
||||
async def delete_generations_by_profile(
|
||||
profile_id: str,
|
||||
db: Session,
|
||||
@@ -282,6 +319,10 @@ async def delete_generations_by_profile(
|
||||
|
||||
count = 0
|
||||
for generation in generations:
|
||||
# Delete associated version files and rows first
|
||||
from . import versions as versions_mod
|
||||
versions_mod.delete_versions_for_generation(generation.id, db)
|
||||
|
||||
# Delete audio file
|
||||
audio_path = config.resolve_storage_path(generation.audio_path)
|
||||
if audio_path is not None and audio_path.exists():
|
||||
|
||||
@@ -484,13 +484,15 @@ async def split_story_item(
|
||||
Returns:
|
||||
List of two updated item details (original and new) or None if not found/invalid
|
||||
"""
|
||||
# Get the item
|
||||
# Get the item with a row lock to prevent concurrent splits on the
|
||||
# same clip (e.g. from rapid double-clicks racing each other).
|
||||
item = (
|
||||
db.query(DBStoryItem)
|
||||
.filter_by(
|
||||
id=item_id,
|
||||
story_id=story_id,
|
||||
)
|
||||
.with_for_update()
|
||||
.first()
|
||||
)
|
||||
if not item:
|
||||
|
||||
@@ -5,12 +5,27 @@ to avoid GPU contention.
|
||||
|
||||
import asyncio
|
||||
import traceback
|
||||
from dataclasses import dataclass
|
||||
from typing import Coroutine, Literal
|
||||
|
||||
# Keep references to fire-and-forget background tasks to prevent GC
|
||||
_background_tasks: set = set()
|
||||
|
||||
|
||||
@dataclass
|
||||
class GenerationJob:
|
||||
"""Queued generation work plus the generation ID it belongs to."""
|
||||
|
||||
generation_id: str
|
||||
coro: Coroutine
|
||||
|
||||
|
||||
# Generation queue — serializes TTS inference to avoid GPU contention
|
||||
_generation_queue: asyncio.Queue = None # type: ignore # initialized at startup
|
||||
_generation_worker_task: asyncio.Task | None = None
|
||||
_queued_generation_ids: set[str] = set()
|
||||
_running_generation_tasks: dict[str, asyncio.Task] = {}
|
||||
_cancelled_generation_ids: set[str] = set()
|
||||
|
||||
|
||||
def create_background_task(coro) -> asyncio.Task:
|
||||
@@ -24,25 +39,70 @@ def create_background_task(coro) -> asyncio.Task:
|
||||
async def _generation_worker():
|
||||
"""Worker that processes generation tasks one at a time."""
|
||||
while True:
|
||||
coro = await _generation_queue.get()
|
||||
job = await _generation_queue.get()
|
||||
try:
|
||||
await coro
|
||||
if job.generation_id in _cancelled_generation_ids:
|
||||
_cancelled_generation_ids.discard(job.generation_id)
|
||||
job.coro.close()
|
||||
continue
|
||||
|
||||
task = asyncio.create_task(job.coro)
|
||||
_running_generation_tasks[job.generation_id] = task
|
||||
_queued_generation_ids.discard(job.generation_id)
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
if not task.cancelled():
|
||||
raise
|
||||
except Exception:
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
_running_generation_tasks.pop(job.generation_id, None)
|
||||
_queued_generation_ids.discard(job.generation_id)
|
||||
_generation_queue.task_done()
|
||||
|
||||
|
||||
def enqueue_generation(coro):
|
||||
def enqueue_generation(generation_id: str, coro):
|
||||
"""Add a generation coroutine to the serial queue."""
|
||||
_generation_queue.put_nowait(coro)
|
||||
if _generation_queue is None:
|
||||
raise RuntimeError("Generation queue has not been initialized")
|
||||
|
||||
_queued_generation_ids.add(generation_id)
|
||||
_generation_queue.put_nowait(GenerationJob(generation_id=generation_id, coro=coro))
|
||||
|
||||
|
||||
def init_queue():
|
||||
def cancel_generation(generation_id: str) -> Literal["queued", "running"] | None:
|
||||
"""Cancel a queued or running generation if it is still active."""
|
||||
running_task = _running_generation_tasks.get(generation_id)
|
||||
if running_task is not None:
|
||||
running_task.cancel()
|
||||
return "running"
|
||||
|
||||
if generation_id in _queued_generation_ids:
|
||||
_queued_generation_ids.discard(generation_id)
|
||||
_cancelled_generation_ids.add(generation_id)
|
||||
return "queued"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def init_queue(force: bool = False):
|
||||
"""Initialize the generation queue and start the worker.
|
||||
|
||||
Must be called once during application startup (inside a running event loop).
|
||||
"""
|
||||
global _generation_queue
|
||||
global _generation_queue, _generation_worker_task
|
||||
global _queued_generation_ids, _running_generation_tasks, _cancelled_generation_ids
|
||||
|
||||
if _generation_worker_task is not None and not _generation_worker_task.done():
|
||||
if not force:
|
||||
return
|
||||
_generation_worker_task.cancel()
|
||||
for task in list(_running_generation_tasks.values()):
|
||||
task.cancel()
|
||||
|
||||
_generation_queue = asyncio.Queue()
|
||||
create_background_task(_generation_worker())
|
||||
_queued_generation_ids = set()
|
||||
_running_generation_tasks = {}
|
||||
_cancelled_generation_ids = set()
|
||||
_generation_worker_task = create_background_task(_generation_worker())
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
# End-to-End Model Generation Test — Design
|
||||
|
||||
## Goal
|
||||
|
||||
A single script, runnable on macOS and Windows, that exercises every TTS model against the **frozen PyInstaller binary** (not the dev server), captures per-model pass/fail and error messages, and exits non-zero if any model fails. Generation is strictly sequential — one model loaded at a time.
|
||||
|
||||
## Test matrix (10 runs)
|
||||
|
||||
Derived from `backend/backends/__init__.py:185-316`. Each row maps to one `POST /generate` call.
|
||||
|
||||
| # | engine | model_size | profile kind | notes |
|
||||
|---|-----------------------|------------|--------------|-------|
|
||||
| 1 | `qwen` | `1.7B` | cloned | reference audio required |
|
||||
| 2 | `qwen` | `0.6B` | cloned | |
|
||||
| 3 | `qwen_custom_voice` | `1.7B` | preset | `preset_voice_id="Ryan"` |
|
||||
| 4 | `qwen_custom_voice` | `0.6B` | preset | `preset_voice_id="Ryan"` |
|
||||
| 5 | `luxtts` | — | cloned | English only |
|
||||
| 6 | `chatterbox` | — | cloned | |
|
||||
| 7 | `chatterbox_turbo` | — | cloned | English only |
|
||||
| 8 | `tada` | `1B` | cloned | tada-1b, English only |
|
||||
| 9 | `tada` | `3B` | cloned | tada-3b-ml, multilingual |
|
||||
| 10| `kokoro` | — | preset | `preset_voice_id="af_heart"` |
|
||||
|
||||
Cloned engines (1, 2, 5, 6, 7, 8, 9) share **one** profile created once with the reference WAV. Preset profiles are created separately, one for kokoro and one for qwen_custom_voice.
|
||||
|
||||
Language for every run: `en` (covers every engine's supported set).
|
||||
|
||||
## End-to-end flow
|
||||
|
||||
```
|
||||
1. Resolve paths → find binary, build if missing
|
||||
2. Launch binary → spawn with --port --data-dir --parent-pid
|
||||
3. Wait for /health → poll until status=="healthy" or 120s timeout
|
||||
4. Create profiles → 1 cloned + 2 preset, via /profiles (+ /samples)
|
||||
5. For each (engine, model_size) in matrix:
|
||||
a. Check cache → GET /models/status → cached? short timeout : long
|
||||
b. POST /generate → get generation_id
|
||||
c. Stream /status → consume SSE until completed/failed/timeout
|
||||
d. Record result → {engine, model_size, status, duration, error, elapsed}
|
||||
6. Write results → JSON + Markdown table to ./results/
|
||||
7. Shutdown binary → SIGTERM, fall back to kill, verify port freed
|
||||
8. Exit code → 0 if all passed, 1 otherwise
|
||||
```
|
||||
|
||||
## Binary resolution
|
||||
|
||||
Search order — **first hit wins**:
|
||||
|
||||
| Platform | Path | Build type |
|
||||
|----------|------|------------|
|
||||
| macOS | `backend/dist/voicebox-server-cuda/voicebox-server-cuda` | onedir (CUDA, rarely on Mac) |
|
||||
| macOS | `backend/dist/voicebox-server` | onefile (CPU) |
|
||||
| Windows | `backend\dist\voicebox-server-cuda\voicebox-server-cuda.exe` | onedir (CUDA) |
|
||||
| Windows | `backend\dist\voicebox-server.exe` | onefile (CPU) |
|
||||
|
||||
If none exist, run `python backend/build_binary.py` and wait for it to finish (can take 5-20 min). Fail with a clear error if the build itself fails. `--skip-build` flag forces "error out if no binary" instead of building.
|
||||
|
||||
## Spawn command
|
||||
|
||||
Mirrors Tauri's launch in `tauri/src-tauri/src/main.rs:369-388`:
|
||||
|
||||
```
|
||||
<binary> --host 127.0.0.1 --port <free-port> --data-dir <tempdir> --parent-pid <test-pid>
|
||||
```
|
||||
|
||||
- **Port**: bind to `0` first in Python to grab a free port, then pass that number.
|
||||
- **Data dir**: `tempfile.mkdtemp(prefix="voicebox-e2e-")`. Deleted after the run unless `--keep-data-dir`. Profiles and generated WAVs land here.
|
||||
- **Parent PID**: current Python PID — ensures the backend dies if the test crashes (watchdog in `server.py:102-224`).
|
||||
- **stdout/stderr**: tee to both a log file in `./results/server-<timestamp>.log` and a rolling in-memory buffer. On model failure, last 100 lines of the buffer are attached to that model's error record.
|
||||
|
||||
## Profile setup
|
||||
|
||||
One cloned profile shared across all cloning engines:
|
||||
|
||||
```http
|
||||
POST /profiles
|
||||
{
|
||||
"name": "e2e-cloned",
|
||||
"voice_type": "cloned",
|
||||
"language": "en"
|
||||
}
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```http
|
||||
POST /profiles/{id}/samples (multipart)
|
||||
file: <reference WAV>
|
||||
reference_text: <exact transcription>
|
||||
```
|
||||
|
||||
Two preset profiles:
|
||||
|
||||
```http
|
||||
POST /profiles
|
||||
{ "name": "e2e-kokoro", "voice_type": "preset", "language": "en",
|
||||
"preset_engine": "kokoro", "preset_voice_id": "af_heart" }
|
||||
|
||||
POST /profiles
|
||||
{ "name": "e2e-qwen-cv", "voice_type": "preset", "language": "en",
|
||||
"preset_engine": "qwen_custom_voice", "preset_voice_id": "Ryan" }
|
||||
```
|
||||
|
||||
## Generation request (per matrix row)
|
||||
|
||||
```http
|
||||
POST /generate
|
||||
{
|
||||
"profile_id": "<appropriate profile>",
|
||||
"text": "The quick brown fox jumps over the lazy dog.",
|
||||
"language": "en",
|
||||
"engine": "<engine>",
|
||||
"model_size": "<size or omitted>",
|
||||
"seed": 42,
|
||||
"normalize": true
|
||||
}
|
||||
```
|
||||
|
||||
Response `id` feeds into the SSE status loop (`GET /generate/{id}/status`, `routes/generations.py:190-227`). Loop reads lines until a payload with `status in ("completed", "failed")` arrives, then breaks.
|
||||
|
||||
## Timeout strategy (split)
|
||||
|
||||
Check `GET /models/status` for the target model **before** generation:
|
||||
|
||||
| Cached? | Per-model timeout | Rationale |
|
||||
|---------|-------------------|-----------|
|
||||
| Yes | **3 minutes** | Inference only; generous for CPU builds |
|
||||
| No | **20 minutes** | First-run HF download up to 8 GB (tada-3b-ml) |
|
||||
|
||||
On timeout: cancel the SSE stream, mark the row `timeout`, and continue to the next row. Don't abort the whole run on one timeout.
|
||||
|
||||
## Result format
|
||||
|
||||
`./results/e2e-<platform>-<arch>-<timestamp>.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"platform": "darwin-arm64",
|
||||
"binary": "/abs/path/voicebox-server",
|
||||
"binary_size_mb": 612,
|
||||
"started_at": "2026-04-16T12:34:56Z",
|
||||
"finished_at": "...",
|
||||
"results": [
|
||||
{
|
||||
"engine": "qwen",
|
||||
"model_size": "1.7B",
|
||||
"status": "passed|failed|timeout",
|
||||
"generation_id": "...",
|
||||
"was_cached": true,
|
||||
"elapsed_seconds": 12.4,
|
||||
"audio_duration": 3.1,
|
||||
"audio_path": "/tmp/.../gen.wav",
|
||||
"error": null,
|
||||
"server_log_tail": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Companion `./results/e2e-<...>.md`:
|
||||
|
||||
```
|
||||
# Voicebox E2E — darwin-arm64 — 2026-04-16 12:34
|
||||
|
||||
| Engine | Size | Status | Elapsed | Error |
|
||||
|---------------------|------|--------|---------|-------|
|
||||
| qwen | 1.7B | PASS | 12.4s | |
|
||||
| qwen | 0.6B | FAIL | 4.1s | CUDA OOM: ... |
|
||||
...
|
||||
```
|
||||
|
||||
## CLI flags
|
||||
|
||||
```
|
||||
python -m backend.tests.test_all_models_e2e [flags]
|
||||
|
||||
--binary PATH Use this binary instead of auto-detecting
|
||||
--skip-build Error if no binary found (no auto-build)
|
||||
--reference-wav PATH Reference audio (default: backend/tests/fixtures/reference_voice.wav)
|
||||
--reference-text STR Transcription (default: read from fixtures/reference_voice.txt)
|
||||
--only ENGINE[,...] Run only these engines (e.g. kokoro,qwen)
|
||||
--skip ENGINE[,...] Skip these engines
|
||||
--keep-data-dir Don't delete tempdir after run
|
||||
--timeout-cached SEC Override 180
|
||||
--timeout-download SEC Override 1200
|
||||
--port N Override auto-picked port
|
||||
--output-dir PATH Default: backend/tests/results/
|
||||
```
|
||||
|
||||
## File layout
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── E2E_MODEL_TEST_DESIGN.md (this file)
|
||||
├── test_all_models_e2e.py (main script, ~400-500 LoC)
|
||||
├── fixtures/
|
||||
│ ├── reference_voice.wav (user-provided, ~5-15s clean speech)
|
||||
│ └── reference_voice.txt (exact transcription)
|
||||
└── results/ (gitignored)
|
||||
├── e2e-darwin-arm64-<ts>.json
|
||||
├── e2e-darwin-arm64-<ts>.md
|
||||
└── server-<ts>.log
|
||||
```
|
||||
|
||||
The script uses only stdlib + `httpx` (or `requests`) + `sseclient-py` — all already in `backend/requirements.txt`. No pytest to keep it invocable as a single command on fresh checkouts.
|
||||
|
||||
## Safety & cleanup
|
||||
|
||||
- Always kill the spawned binary in a `try/finally`. On Windows, `taskkill /F /T` the whole tree (Tauri does the same).
|
||||
- Verify the port is free on shutdown (Tauri port-reuse check in `main.rs:114-186` could otherwise pick up a ghost).
|
||||
- Don't touch the user's HF cache by default — let the server use `HF_HUB_CACHE` / `VOICEBOX_MODELS_DIR`. Passing `--isolated-cache` would point both env vars at the tempdir for a true cold-start run (opt-in only; would re-download every time).
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not validating audio quality (no WER, no waveform comparison). Pass = "endpoint returned `completed` and produced a non-empty WAV".
|
||||
- Not testing STT (Whisper), effects chains, channels, or streaming endpoints.
|
||||
- Not running on CI today — human-invoked on dev machines. CI integration is a follow-up once the script is stable.
|
||||
- No model unload between runs — models stay loaded; server manages its own eviction.
|
||||
- No version-drift check on the binary.
|
||||
- No `instruct` parameter exercised on qwen_custom_voice runs.
|
||||
Vendored
+16
@@ -0,0 +1,16 @@
|
||||
# E2E Test Fixtures
|
||||
|
||||
Place two files here before running `test_all_models_e2e.py`:
|
||||
|
||||
- `reference_voice.wav` — a clean speech sample, mono, 16–24 kHz, ~5–15 seconds.
|
||||
- `reference_voice.txt` — the **exact** transcription of the WAV (single line, no trailing newline required).
|
||||
|
||||
These are used to create a cloned voice profile for every cloning-capable engine (qwen, luxtts, chatterbox, chatterbox_turbo, tada). Keep them out of version control if they contain personal audio — this directory is not gitignored by default, so add them to `.gitignore` locally if needed.
|
||||
|
||||
You can point the test at different files with:
|
||||
|
||||
```
|
||||
python backend/tests/test_all_models_e2e.py \
|
||||
--reference-wav /path/to/your.wav \
|
||||
--reference-text "exact transcription here"
|
||||
```
|
||||
@@ -0,0 +1,630 @@
|
||||
"""
|
||||
End-to-end model generation test.
|
||||
|
||||
Exercises every TTS model against the frozen PyInstaller binary, captures
|
||||
per-model pass/fail, and writes a JSON + Markdown report.
|
||||
|
||||
Usage:
|
||||
python backend/tests/test_all_models_e2e.py [flags]
|
||||
|
||||
See E2E_MODEL_TEST_DESIGN.md for the full design.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
BACKEND_DIR = REPO_ROOT / "backend"
|
||||
DIST_DIR = BACKEND_DIR / "dist"
|
||||
FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
|
||||
RESULTS_DIR = Path(__file__).resolve().parent / "results"
|
||||
|
||||
|
||||
# ── Test matrix ──────────────────────────────────────────────────────
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MatrixRow:
|
||||
label: str # human-readable (appears in report)
|
||||
engine: str # /generate engine
|
||||
model_size: Optional[str] # /generate model_size (None = omit)
|
||||
profile_kind: str # "cloned" | "preset_kokoro" | "preset_qwen_cv"
|
||||
model_name: str # /models/status key for cache lookup
|
||||
|
||||
|
||||
MATRIX: list[MatrixRow] = [
|
||||
MatrixRow("qwen 1.7B", "qwen", "1.7B", "cloned", "qwen-tts-1.7B"),
|
||||
MatrixRow("qwen 0.6B", "qwen", "0.6B", "cloned", "qwen-tts-0.6B"),
|
||||
MatrixRow("qwen_custom_voice 1.7B", "qwen_custom_voice", "1.7B", "preset_qwen_cv", "qwen-custom-voice-1.7B"),
|
||||
MatrixRow("qwen_custom_voice 0.6B", "qwen_custom_voice", "0.6B", "preset_qwen_cv", "qwen-custom-voice-0.6B"),
|
||||
MatrixRow("luxtts", "luxtts", None, "cloned", "luxtts"),
|
||||
MatrixRow("chatterbox", "chatterbox", None, "cloned", "chatterbox-tts"),
|
||||
MatrixRow("chatterbox_turbo", "chatterbox_turbo", None, "cloned", "chatterbox-turbo"),
|
||||
MatrixRow("tada 1B", "tada", "1B", "cloned", "tada-1b"),
|
||||
MatrixRow("tada 3B", "tada", "3B", "cloned", "tada-3b-ml"),
|
||||
MatrixRow("kokoro", "kokoro", None, "preset_kokoro", "kokoro"),
|
||||
]
|
||||
|
||||
TEXT = "The quick brown fox jumps over the lazy dog."
|
||||
DEFAULT_TIMEOUT_CACHED = 180
|
||||
DEFAULT_TIMEOUT_DOWNLOAD = 1200
|
||||
HEALTH_TIMEOUT = 120
|
||||
|
||||
|
||||
# ── Result record ────────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class ModelResult:
|
||||
label: str
|
||||
engine: str
|
||||
model_size: Optional[str]
|
||||
status: str # "passed" | "failed" | "timeout"
|
||||
was_cached: Optional[bool] = None
|
||||
generation_id: Optional[str] = None
|
||||
elapsed_seconds: float = 0.0
|
||||
audio_duration: Optional[float] = None
|
||||
audio_path: Optional[str] = None
|
||||
audio_bytes: Optional[int] = None
|
||||
error: Optional[str] = None
|
||||
http_status: Optional[int] = None
|
||||
server_log_tail: Optional[list[str]] = None
|
||||
|
||||
|
||||
# ── Binary resolution ────────────────────────────────────────────────
|
||||
|
||||
def find_binary() -> Optional[Path]:
|
||||
"""Return the first existing binary in priority order, or None."""
|
||||
is_win = platform.system() == "Windows"
|
||||
exe = ".exe" if is_win else ""
|
||||
candidates = [
|
||||
DIST_DIR / "voicebox-server-cuda" / f"voicebox-server-cuda{exe}",
|
||||
DIST_DIR / f"voicebox-server{exe}",
|
||||
]
|
||||
for c in candidates:
|
||||
if c.exists() and c.is_file():
|
||||
return c
|
||||
return None
|
||||
|
||||
|
||||
def build_binary() -> Path:
|
||||
"""Invoke build_binary.py and return the resulting binary path."""
|
||||
print("[build] No frozen binary found — invoking build_binary.py (this may take 5-20 minutes)...", flush=True)
|
||||
script = BACKEND_DIR / "build_binary.py"
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(script)],
|
||||
cwd=str(BACKEND_DIR),
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"build_binary.py exited with code {result.returncode}")
|
||||
found = find_binary()
|
||||
if found is None:
|
||||
raise RuntimeError("build_binary.py finished but no binary was found in backend/dist/")
|
||||
return found
|
||||
|
||||
|
||||
# ── Server spawn + log capture ───────────────────────────────────────
|
||||
|
||||
class ServerProcess:
|
||||
def __init__(self, binary: Path, port: int, data_dir: Path, log_path: Path):
|
||||
self.binary = binary
|
||||
self.port = port
|
||||
self.data_dir = data_dir
|
||||
self.log_path = log_path
|
||||
self.proc: Optional[subprocess.Popen] = None
|
||||
self._log_buffer: deque[str] = deque(maxlen=500)
|
||||
self._reader_thread: Optional[threading.Thread] = None
|
||||
|
||||
def start(self) -> None:
|
||||
args = [
|
||||
str(self.binary),
|
||||
"--host", "127.0.0.1",
|
||||
"--port", str(self.port),
|
||||
"--data-dir", str(self.data_dir),
|
||||
"--parent-pid", str(os.getpid()),
|
||||
]
|
||||
print(f"[spawn] {' '.join(args)}", flush=True)
|
||||
self._log_fh = open(self.log_path, "w", encoding="utf-8", errors="replace")
|
||||
# Combine stderr into stdout so we get a single ordered stream.
|
||||
self.proc = subprocess.Popen(
|
||||
args,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
bufsize=1,
|
||||
text=True,
|
||||
errors="replace",
|
||||
)
|
||||
self._reader_thread = threading.Thread(target=self._pump_logs, daemon=True)
|
||||
self._reader_thread.start()
|
||||
|
||||
def _pump_logs(self) -> None:
|
||||
assert self.proc is not None and self.proc.stdout is not None
|
||||
for line in self.proc.stdout:
|
||||
self._log_buffer.append(line.rstrip("\n"))
|
||||
self._log_fh.write(line)
|
||||
self._log_fh.flush()
|
||||
|
||||
def log_tail(self, n: int = 100) -> list[str]:
|
||||
tail = list(self._log_buffer)[-n:]
|
||||
return tail
|
||||
|
||||
def is_alive(self) -> bool:
|
||||
return self.proc is not None and self.proc.poll() is None
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.proc is None:
|
||||
return
|
||||
if self.proc.poll() is not None:
|
||||
return
|
||||
try:
|
||||
if platform.system() == "Windows":
|
||||
subprocess.run(
|
||||
["taskkill", "/F", "/T", "/PID", str(self.proc.pid)],
|
||||
capture_output=True,
|
||||
)
|
||||
else:
|
||||
self.proc.send_signal(signal.SIGTERM)
|
||||
except Exception as e:
|
||||
print(f"[shutdown] signal failed: {e}", flush=True)
|
||||
try:
|
||||
self.proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
print("[shutdown] server didn't exit cleanly, killing", flush=True)
|
||||
self.proc.kill()
|
||||
try:
|
||||
self.proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
if self._reader_thread is not None:
|
||||
self._reader_thread.join(timeout=2)
|
||||
try:
|
||||
self._log_fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def pick_free_port() -> int:
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
port = s.getsockname()[1]
|
||||
s.close()
|
||||
return port
|
||||
|
||||
|
||||
# ── HTTP helpers ─────────────────────────────────────────────────────
|
||||
|
||||
def wait_for_health(base_url: str, server: ServerProcess, timeout: int) -> None:
|
||||
deadline = time.time() + timeout
|
||||
with httpx.Client(timeout=5.0) as client:
|
||||
while time.time() < deadline:
|
||||
if not server.is_alive():
|
||||
raise RuntimeError("Server process exited before becoming healthy")
|
||||
try:
|
||||
r = client.get(f"{base_url}/health")
|
||||
if r.status_code == 200 and r.json().get("status") == "healthy":
|
||||
return
|
||||
except httpx.HTTPError:
|
||||
pass
|
||||
time.sleep(1.0)
|
||||
raise TimeoutError(f"Server did not become healthy within {timeout}s")
|
||||
|
||||
|
||||
def get_model_cached(client: httpx.Client, base_url: str, model_name: str) -> Optional[bool]:
|
||||
try:
|
||||
r = client.get(f"{base_url}/models/status", timeout=30.0)
|
||||
r.raise_for_status()
|
||||
for m in r.json().get("models", []):
|
||||
if m.get("model_name") == model_name:
|
||||
return bool(m.get("downloaded"))
|
||||
except httpx.HTTPError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def create_cloned_profile(client: httpx.Client, base_url: str, wav_path: Path, reference_text: str) -> str:
|
||||
r = client.post(f"{base_url}/profiles", json={
|
||||
"name": "e2e-cloned",
|
||||
"voice_type": "cloned",
|
||||
"language": "en",
|
||||
})
|
||||
r.raise_for_status()
|
||||
profile_id = r.json()["id"]
|
||||
|
||||
with open(wav_path, "rb") as f:
|
||||
r = client.post(
|
||||
f"{base_url}/profiles/{profile_id}/samples",
|
||||
files={"file": (wav_path.name, f, "audio/wav")},
|
||||
data={"reference_text": reference_text},
|
||||
timeout=120.0,
|
||||
)
|
||||
r.raise_for_status()
|
||||
return profile_id
|
||||
|
||||
|
||||
def create_preset_profile(client: httpx.Client, base_url: str, name: str, engine: str, voice_id: str) -> str:
|
||||
r = client.post(f"{base_url}/profiles", json={
|
||||
"name": name,
|
||||
"voice_type": "preset",
|
||||
"language": "en",
|
||||
"preset_engine": engine,
|
||||
"preset_voice_id": voice_id,
|
||||
})
|
||||
r.raise_for_status()
|
||||
return r.json()["id"]
|
||||
|
||||
|
||||
def run_one_generation(
|
||||
client: httpx.Client,
|
||||
base_url: str,
|
||||
row: MatrixRow,
|
||||
profile_id: str,
|
||||
timeout_s: int,
|
||||
) -> tuple[str, dict]:
|
||||
"""Start a generation and stream its status until done/failed/timeout.
|
||||
|
||||
Returns (status, payload) where status is "completed" | "failed" | "timeout".
|
||||
"""
|
||||
body = {
|
||||
"profile_id": profile_id,
|
||||
"text": TEXT,
|
||||
"language": "en",
|
||||
"engine": row.engine,
|
||||
"seed": 42,
|
||||
"normalize": True,
|
||||
}
|
||||
if row.model_size is not None:
|
||||
body["model_size"] = row.model_size
|
||||
|
||||
r = client.post(f"{base_url}/generate", json=body, timeout=30.0)
|
||||
r.raise_for_status()
|
||||
gen = r.json()
|
||||
gen_id = gen["id"]
|
||||
|
||||
deadline = time.time() + timeout_s
|
||||
last_payload: dict = gen
|
||||
status_url = f"{base_url}/generate/{gen_id}/status"
|
||||
|
||||
while time.time() < deadline:
|
||||
remaining = max(1.0, deadline - time.time())
|
||||
try:
|
||||
with client.stream("GET", status_url, timeout=httpx.Timeout(remaining + 5, read=remaining + 5)) as resp:
|
||||
resp.raise_for_status()
|
||||
for line in resp.iter_lines():
|
||||
if not line or not line.startswith("data: "):
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(line[6:])
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
last_payload = payload
|
||||
status = payload.get("status")
|
||||
if status == "not_found":
|
||||
return "failed", {"error": "generation not found", **payload}
|
||||
if status in ("completed", "failed"):
|
||||
return status, payload
|
||||
if time.time() >= deadline:
|
||||
break
|
||||
except httpx.HTTPError:
|
||||
time.sleep(1.0)
|
||||
continue
|
||||
|
||||
return "timeout", last_payload
|
||||
|
||||
|
||||
def fetch_audio_info(
|
||||
client: httpx.Client, base_url: str, generation_id: str, data_dir: Path
|
||||
) -> tuple[Optional[str], Optional[int]]:
|
||||
"""Return (audio_path, audio_bytes) for a completed generation.
|
||||
|
||||
Server stores audio_path relative to data_dir; resolve it to get a size.
|
||||
"""
|
||||
try:
|
||||
r = client.get(f"{base_url}/history/{generation_id}", timeout=10.0)
|
||||
if r.status_code != 200:
|
||||
return None, None
|
||||
data = r.json()
|
||||
audio_path = data.get("audio_path")
|
||||
if not audio_path:
|
||||
return None, None
|
||||
p = Path(audio_path)
|
||||
if not p.is_absolute():
|
||||
p = data_dir / p
|
||||
if p.exists():
|
||||
return str(p), p.stat().st_size
|
||||
return audio_path, None
|
||||
except httpx.HTTPError:
|
||||
return None, None
|
||||
|
||||
|
||||
# ── Report writers ───────────────────────────────────────────────────
|
||||
|
||||
def write_reports(
|
||||
output_dir: Path,
|
||||
binary: Path,
|
||||
started_at: datetime,
|
||||
finished_at: datetime,
|
||||
results: list[ModelResult],
|
||||
) -> tuple[Path, Path]:
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
plat = f"{platform.system().lower()}-{platform.machine().lower()}"
|
||||
ts = started_at.strftime("%Y%m%d-%H%M%S")
|
||||
json_path = output_dir / f"e2e-{plat}-{ts}.json"
|
||||
md_path = output_dir / f"e2e-{plat}-{ts}.md"
|
||||
|
||||
doc = {
|
||||
"platform": plat,
|
||||
"binary": str(binary),
|
||||
"binary_size_mb": round(binary.stat().st_size / (1024 * 1024), 1) if binary.exists() else None,
|
||||
"started_at": started_at.isoformat(),
|
||||
"finished_at": finished_at.isoformat(),
|
||||
"elapsed_seconds": (finished_at - started_at).total_seconds(),
|
||||
"results": [asdict(r) for r in results],
|
||||
}
|
||||
json_path.write_text(json.dumps(doc, indent=2))
|
||||
|
||||
lines = [
|
||||
f"# Voicebox E2E — {plat} — {started_at.strftime('%Y-%m-%d %H:%M UTC')}",
|
||||
"",
|
||||
f"Binary: `{binary}` ",
|
||||
f"Elapsed: {doc['elapsed_seconds']:.1f}s",
|
||||
"",
|
||||
"| Model | Status | Cached | Elapsed | Audio | Error |",
|
||||
"|-------|--------|--------|---------|-------|-------|",
|
||||
]
|
||||
for r in results:
|
||||
status_icon = {"passed": "PASS", "failed": "FAIL", "timeout": "TIMEOUT"}.get(r.status, r.status.upper())
|
||||
cached = "yes" if r.was_cached else ("no" if r.was_cached is False else "?")
|
||||
audio_col = f"{r.audio_duration:.2f}s" if r.audio_duration else ("—" if r.status != "passed" else "?")
|
||||
error_col = (r.error or "").replace("\n", " ")[:120]
|
||||
lines.append(f"| {r.label} | {status_icon} | {cached} | {r.elapsed_seconds:.1f}s | {audio_col} | {error_col} |")
|
||||
|
||||
failed_rows = [r for r in results if r.status != "passed"]
|
||||
if failed_rows:
|
||||
lines.append("")
|
||||
lines.append("## Failures")
|
||||
for r in failed_rows:
|
||||
lines.append("")
|
||||
lines.append(f"### {r.label} — {r.status}")
|
||||
if r.error:
|
||||
lines.append("")
|
||||
lines.append("```")
|
||||
lines.append(r.error)
|
||||
lines.append("```")
|
||||
if r.server_log_tail:
|
||||
lines.append("")
|
||||
lines.append("<details><summary>server log (last lines)</summary>")
|
||||
lines.append("")
|
||||
lines.append("```")
|
||||
lines.extend(r.server_log_tail)
|
||||
lines.append("```")
|
||||
lines.append("</details>")
|
||||
|
||||
md_path.write_text("\n".join(lines) + "\n")
|
||||
return json_path, md_path
|
||||
|
||||
|
||||
# ── Main ─────────────────────────────────────────────────────────────
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(description="Voicebox E2E model generation test")
|
||||
p.add_argument("--binary", type=Path, help="Path to voicebox-server binary (overrides auto-detect)")
|
||||
p.add_argument("--skip-build", action="store_true", help="Error if binary missing instead of building")
|
||||
p.add_argument(
|
||||
"--reference-wav",
|
||||
type=Path,
|
||||
default=FIXTURES_DIR / "reference_voice.wav",
|
||||
help="Reference audio for cloning engines",
|
||||
)
|
||||
p.add_argument(
|
||||
"--reference-text",
|
||||
help="Transcription of reference-wav (default: read from fixtures/reference_voice.txt)",
|
||||
)
|
||||
p.add_argument("--only", help="Comma-separated engines to run (e.g. kokoro,qwen)")
|
||||
p.add_argument("--skip", help="Comma-separated engines to skip")
|
||||
p.add_argument("--keep-data-dir", action="store_true", help="Don't delete tempdir after run")
|
||||
p.add_argument("--timeout-cached", type=int, default=DEFAULT_TIMEOUT_CACHED)
|
||||
p.add_argument("--timeout-download", type=int, default=DEFAULT_TIMEOUT_DOWNLOAD)
|
||||
p.add_argument("--port", type=int, help="Override auto-picked port")
|
||||
p.add_argument("--output-dir", type=Path, default=RESULTS_DIR)
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def filter_matrix(args: argparse.Namespace) -> list[MatrixRow]:
|
||||
only = set(x.strip() for x in args.only.split(",")) if args.only else None
|
||||
skip = set(x.strip() for x in args.skip.split(",")) if args.skip else set()
|
||||
rows = []
|
||||
for r in MATRIX:
|
||||
if only is not None and r.engine not in only:
|
||||
continue
|
||||
if r.engine in skip:
|
||||
continue
|
||||
rows.append(r)
|
||||
return rows
|
||||
|
||||
|
||||
def resolve_reference(args: argparse.Namespace) -> tuple[Path, str]:
|
||||
wav = args.reference_wav
|
||||
if not wav.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Reference WAV not found: {wav}\n"
|
||||
f"Place a sample at {FIXTURES_DIR / 'reference_voice.wav'} or pass --reference-wav.\n"
|
||||
f"See backend/tests/fixtures/README.md."
|
||||
)
|
||||
if args.reference_text:
|
||||
text = args.reference_text
|
||||
else:
|
||||
txt_path = wav.with_suffix(".txt")
|
||||
if not txt_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Reference transcription not found: {txt_path}\n"
|
||||
f"Create it next to the WAV, or pass --reference-text."
|
||||
)
|
||||
text = txt_path.read_text().strip()
|
||||
if not text:
|
||||
raise ValueError("Reference transcription is empty")
|
||||
return wav, text
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
rows = filter_matrix(args)
|
||||
if not rows:
|
||||
print("No rows selected after --only/--skip filtering", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
# Binary
|
||||
binary = args.binary or find_binary()
|
||||
if binary is None:
|
||||
if args.skip_build:
|
||||
print("No frozen binary found and --skip-build set. Run: python backend/build_binary.py", file=sys.stderr)
|
||||
return 2
|
||||
binary = build_binary()
|
||||
if not binary.exists():
|
||||
print(f"Binary path does not exist: {binary}", file=sys.stderr)
|
||||
return 2
|
||||
print(f"[binary] {binary}", flush=True)
|
||||
|
||||
# Reference audio (only required if any cloning row is in the matrix)
|
||||
needs_reference = any(r.profile_kind == "cloned" for r in rows)
|
||||
ref_wav: Optional[Path] = None
|
||||
ref_text: Optional[str] = None
|
||||
if needs_reference:
|
||||
try:
|
||||
ref_wav, ref_text = resolve_reference(args)
|
||||
except (FileNotFoundError, ValueError) as e:
|
||||
print(f"[fixture] {e}", file=sys.stderr)
|
||||
return 2
|
||||
print(f"[fixture] reference WAV: {ref_wav}", flush=True)
|
||||
print(f"[fixture] reference text: {ref_text!r}", flush=True)
|
||||
|
||||
# Tempdir + log path
|
||||
data_dir = Path(tempfile.mkdtemp(prefix="voicebox-e2e-"))
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
|
||||
log_path = args.output_dir / f"server-{ts}.log"
|
||||
|
||||
port = args.port or pick_free_port()
|
||||
base_url = f"http://127.0.0.1:{port}"
|
||||
|
||||
server = ServerProcess(binary=binary, port=port, data_dir=data_dir, log_path=log_path)
|
||||
started_at = datetime.now(timezone.utc)
|
||||
results: list[ModelResult] = []
|
||||
|
||||
try:
|
||||
server.start()
|
||||
print(f"[health] waiting for {base_url}/health ...", flush=True)
|
||||
wait_for_health(base_url, server, HEALTH_TIMEOUT)
|
||||
print("[health] ready", flush=True)
|
||||
|
||||
with httpx.Client(timeout=30.0) as client:
|
||||
# Profile setup (only create what's needed)
|
||||
cloned_profile_id: Optional[str] = None
|
||||
kokoro_profile_id: Optional[str] = None
|
||||
qwen_cv_profile_id: Optional[str] = None
|
||||
needed_kinds = {r.profile_kind for r in rows}
|
||||
if "cloned" in needed_kinds:
|
||||
assert ref_wav is not None and ref_text is not None
|
||||
print("[profile] creating cloned profile...", flush=True)
|
||||
cloned_profile_id = create_cloned_profile(client, base_url, ref_wav, ref_text)
|
||||
if "preset_kokoro" in needed_kinds:
|
||||
print("[profile] creating kokoro preset...", flush=True)
|
||||
kokoro_profile_id = create_preset_profile(client, base_url, "e2e-kokoro", "kokoro", "af_heart")
|
||||
if "preset_qwen_cv" in needed_kinds:
|
||||
print("[profile] creating qwen_custom_voice preset...", flush=True)
|
||||
qwen_cv_profile_id = create_preset_profile(client, base_url, "e2e-qwen-cv", "qwen_custom_voice", "Ryan")
|
||||
|
||||
profile_lookup = {
|
||||
"cloned": cloned_profile_id,
|
||||
"preset_kokoro": kokoro_profile_id,
|
||||
"preset_qwen_cv": qwen_cv_profile_id,
|
||||
}
|
||||
|
||||
# Matrix loop
|
||||
for row in rows:
|
||||
print(f"\n[run] {row.label} (engine={row.engine}, size={row.model_size})", flush=True)
|
||||
profile_id = profile_lookup[row.profile_kind]
|
||||
assert profile_id is not None
|
||||
was_cached = get_model_cached(client, base_url, row.model_name)
|
||||
timeout_s = args.timeout_cached if was_cached else args.timeout_download
|
||||
print(f"[run] cached={was_cached} timeout={timeout_s}s", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
result = ModelResult(
|
||||
label=row.label,
|
||||
engine=row.engine,
|
||||
model_size=row.model_size,
|
||||
status="failed",
|
||||
was_cached=was_cached,
|
||||
)
|
||||
try:
|
||||
status, payload = run_one_generation(client, base_url, row, profile_id, timeout_s)
|
||||
result.status = "passed" if status == "completed" else status
|
||||
result.generation_id = payload.get("id")
|
||||
result.audio_duration = payload.get("duration")
|
||||
result.error = payload.get("error")
|
||||
if status == "completed" and result.generation_id:
|
||||
audio_path, audio_bytes = fetch_audio_info(
|
||||
client, base_url, result.generation_id, data_dir
|
||||
)
|
||||
result.audio_path = audio_path
|
||||
result.audio_bytes = audio_bytes
|
||||
if audio_bytes is not None and audio_bytes == 0:
|
||||
result.status = "failed"
|
||||
result.error = (result.error or "") + " (audio file is empty)"
|
||||
except httpx.HTTPStatusError as e:
|
||||
result.status = "failed"
|
||||
result.http_status = e.response.status_code
|
||||
try:
|
||||
detail = e.response.json().get("detail")
|
||||
except Exception:
|
||||
detail = e.response.text
|
||||
result.error = f"HTTP {e.response.status_code}: {detail}"
|
||||
except Exception as e:
|
||||
result.status = "failed"
|
||||
result.error = f"{type(e).__name__}: {e}"
|
||||
|
||||
result.elapsed_seconds = round(time.time() - t0, 2)
|
||||
if result.status != "passed":
|
||||
result.server_log_tail = server.log_tail(100)
|
||||
print(f"[run] {row.label} → {result.status} in {result.elapsed_seconds}s"
|
||||
+ (f" ({result.error})" if result.error else ""), flush=True)
|
||||
results.append(result)
|
||||
finally:
|
||||
finished_at = datetime.now(timezone.utc)
|
||||
server.stop()
|
||||
if not args.keep_data_dir:
|
||||
shutil.rmtree(data_dir, ignore_errors=True)
|
||||
else:
|
||||
print(f"[cleanup] keeping data dir: {data_dir}", flush=True)
|
||||
|
||||
json_path, md_path = write_reports(args.output_dir, binary, started_at, finished_at, results)
|
||||
print(f"\n[report] {json_path}")
|
||||
print(f"[report] {md_path}")
|
||||
print(f"[report] server log: {log_path}")
|
||||
|
||||
passed = sum(1 for r in results if r.status == "passed")
|
||||
failed = len(results) - passed
|
||||
print(f"\n== {passed} passed, {failed} failed ==")
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,54 @@
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.services import task_queue
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_queued_generation_skips_execution():
|
||||
task_queue.init_queue(force=True)
|
||||
|
||||
running_started = asyncio.Event()
|
||||
release_running = asyncio.Event()
|
||||
queued_ran = asyncio.Event()
|
||||
|
||||
async def running_job():
|
||||
running_started.set()
|
||||
await release_running.wait()
|
||||
|
||||
async def queued_job():
|
||||
queued_ran.set()
|
||||
|
||||
task_queue.enqueue_generation("gen-running", running_job())
|
||||
await asyncio.wait_for(running_started.wait(), timeout=1)
|
||||
|
||||
task_queue.enqueue_generation("gen-queued", queued_job())
|
||||
assert task_queue.cancel_generation("gen-queued") == "queued"
|
||||
|
||||
release_running.set()
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
assert not queued_ran.is_set()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_running_generation_cancels_task():
|
||||
task_queue.init_queue(force=True)
|
||||
|
||||
running_started = asyncio.Event()
|
||||
running_cancelled = asyncio.Event()
|
||||
|
||||
async def running_job():
|
||||
running_started.set()
|
||||
try:
|
||||
await asyncio.Event().wait()
|
||||
except asyncio.CancelledError:
|
||||
running_cancelled.set()
|
||||
raise
|
||||
|
||||
task_queue.enqueue_generation("gen-running", running_job())
|
||||
await asyncio.wait_for(running_started.wait(), timeout=1)
|
||||
|
||||
assert task_queue.cancel_generation("gen-running") == "running"
|
||||
await asyncio.wait_for(running_cancelled.wait(), timeout=1)
|
||||
@@ -64,7 +64,7 @@ def get_cached_voice_prompt(
|
||||
cache_file = _get_cache_dir() / f"{cache_key}.prompt"
|
||||
if cache_file.exists():
|
||||
try:
|
||||
prompt = torch.load(cache_file)
|
||||
prompt = torch.load(cache_file, weights_only=True)
|
||||
_memory_cache[cache_key] = prompt
|
||||
return prompt
|
||||
except Exception:
|
||||
|
||||
@@ -1,17 +1,64 @@
|
||||
"""Monkey-patch huggingface_hub to force offline mode with cached models.
|
||||
|
||||
Prevents mlx_audio from making network requests when models are already
|
||||
downloaded. Must be imported BEFORE mlx_audio.
|
||||
Prevents mlx_audio / transformers from making network requests when models
|
||||
are already downloaded. Must be imported BEFORE mlx_audio.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def force_offline_if_cached(is_cached: bool, model_label: str = ""):
|
||||
"""Context manager that sets ``HF_HUB_OFFLINE=1`` while loading a cached model.
|
||||
|
||||
If *is_cached* is ``False`` the block runs normally (network allowed).
|
||||
If the offline load raises an error containing "offline" we automatically
|
||||
retry with network access so a partially-cached model still works.
|
||||
|
||||
Args:
|
||||
is_cached: Whether the model weights are already on disk.
|
||||
model_label: Human-readable name used in log messages.
|
||||
"""
|
||||
if not is_cached:
|
||||
yield
|
||||
return
|
||||
|
||||
original_value = os.environ.get("HF_HUB_OFFLINE")
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
logger.info(
|
||||
"[offline-guard] %s is cached — forcing HF_HUB_OFFLINE=1",
|
||||
model_label or "model",
|
||||
)
|
||||
|
||||
try:
|
||||
yield
|
||||
except Exception as exc:
|
||||
if "offline" in str(exc).lower():
|
||||
logger.warning(
|
||||
"[offline-guard] Offline load failed for %s, retrying with network: %s",
|
||||
model_label or "model",
|
||||
exc,
|
||||
)
|
||||
# Restore original env and retry — caller must wrap the load
|
||||
# inside force_offline_if_cached so retrying here isn't possible.
|
||||
# Instead, propagate a flag via the exception so the caller can
|
||||
# decide. For simplicity we just let it fall through to the
|
||||
# finally block and re-raise.
|
||||
raise
|
||||
raise
|
||||
finally:
|
||||
if original_value is not None:
|
||||
os.environ["HF_HUB_OFFLINE"] = original_value
|
||||
else:
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
|
||||
|
||||
def patch_huggingface_hub_offline():
|
||||
"""Monkey-patch huggingface_hub to force offline mode."""
|
||||
try:
|
||||
|
||||
@@ -5,7 +5,7 @@ from PyInstaller.utils.hooks import copy_metadata
|
||||
|
||||
datas = []
|
||||
binaries = []
|
||||
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.backends.qwen_custom_voice_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'kokoro', 'kokoro.pipeline', 'kokoro.model', 'kokoro.istftnet', 'kokoro.modules', 'kokoro.custom_stft', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
|
||||
hiddenimports = ['backend', 'backend.main', 'backend.config', 'backend.database', 'backend.models', 'backend.services.profiles', 'backend.services.history', 'backend.services.tts', 'backend.services.transcribe', 'backend.utils.platform_detect', 'backend.backends', 'backend.backends.pytorch_backend', 'backend.backends.qwen_custom_voice_backend', 'backend.utils.audio', 'backend.utils.cache', 'backend.utils.progress', 'backend.utils.hf_progress', 'backend.services.cuda', 'backend.services.effects', 'backend.utils.effects', 'backend.services.versions', 'pedalboard', 'chatterbox', 'chatterbox.tts_turbo', 'chatterbox.mtl_tts', 'backend.backends.chatterbox_backend', 'backend.backends.chatterbox_turbo_backend', 'backend.backends.luxtts_backend', 'zipvoice', 'zipvoice.luxvoice', 'torch', 'transformers', 'fastapi', 'uvicorn', 'sqlalchemy', 'soundfile', 'qwen_tts', 'qwen_tts.inference', 'qwen_tts.inference.qwen3_tts_model', 'qwen_tts.inference.qwen3_tts_tokenizer', 'qwen_tts.core', 'qwen_tts.cli', 'requests', 'pkg_resources.extern', 'backend.backends.hume_backend', 'tada', 'tada.modules', 'tada.modules.tada', 'tada.modules.encoder', 'tada.modules.decoder', 'tada.modules.aligner', 'tada.modules.acoustic_spkr_verf', 'tada.nn', 'tada.nn.vibevoice', 'tada.utils', 'tada.utils.gray_code', 'tada.utils.text', 'backend.utils.dac_shim', 'torchaudio', 'backend.backends.kokoro_backend', 'en_core_web_sm', 'loguru', 'backend.backends.mlx_backend', 'mlx', 'mlx.core', 'mlx.nn', 'mlx_audio', 'mlx_audio.tts', 'mlx_audio.stt']
|
||||
datas += copy_metadata('qwen-tts')
|
||||
datas += copy_metadata('requests')
|
||||
datas += copy_metadata('transformers')
|
||||
@@ -18,6 +18,8 @@ hiddenimports += collect_submodules('jaraco')
|
||||
hiddenimports += collect_submodules('tada')
|
||||
hiddenimports += collect_submodules('mlx')
|
||||
hiddenimports += collect_submodules('mlx_audio')
|
||||
tmp_ret = collect_all('spacy_pkuseg')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('zipvoice')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('linacodec')
|
||||
@@ -34,6 +36,8 @@ tmp_ret = collect_all('perth')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('piper_phonemize')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('kokoro')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('misaki')
|
||||
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
|
||||
tmp_ret = collect_all('language_tags')
|
||||
@@ -54,9 +58,9 @@ a = Analysis(
|
||||
binaries=binaries,
|
||||
datas=datas,
|
||||
hiddenimports=hiddenimports,
|
||||
hookspath=[],
|
||||
hookspath=['pyi_hooks'],
|
||||
hooksconfig={},
|
||||
runtime_hooks=[],
|
||||
runtime_hooks=['pyi_rth_numpy_compat.py', 'pyi_rth_torch_compiler_disable.py'],
|
||||
excludes=['nvidia', 'nvidia.cublas', 'nvidia.cuda_cupti', 'nvidia.cuda_nvrtc', 'nvidia.cuda_runtime', 'nvidia.cudnn', 'nvidia.cufft', 'nvidia.curand', 'nvidia.cusolver', 'nvidia.cusparse', 'nvidia.nccl', 'nvidia.nvjitlink', 'nvidia.nvtx'],
|
||||
noarchive=False,
|
||||
optimize=0,
|
||||
|
||||
@@ -22,6 +22,7 @@ services:
|
||||
|
||||
environment:
|
||||
- LOG_LEVEL=info
|
||||
- NUMBA_CACHE_DIR=/tmp/numba_cache
|
||||
|
||||
networks:
|
||||
- voicebox-net
|
||||
|
||||
@@ -0,0 +1,604 @@
|
||||
# Voicebox Project Status & Roadmap
|
||||
|
||||
> Last updated: 2026-04-18 | Current version: **v0.4.1** | ~155 open issues | 12 open PRs
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture Overview](#architecture-overview)
|
||||
2. [Current State](#current-state)
|
||||
3. [Open PRs — Triage & Analysis](#open-prs--triage--analysis)
|
||||
4. [Open Issues — Categorized](#open-issues--categorized)
|
||||
5. [Existing Plan Documents — Status](#existing-plan-documents--status)
|
||||
6. [New Model Integration — Landscape](#new-model-integration--landscape)
|
||||
7. [Architectural Bottlenecks](#architectural-bottlenecks)
|
||||
8. [Recommended Priorities](#recommended-priorities)
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
**Tauri shell (Rust)** hosts a **React frontend** (`app/`) that talks over HTTP on `localhost:17493` to a **FastAPI backend** (`backend/`).
|
||||
|
||||
The backend exposes:
|
||||
|
||||
- **`TTSBackend` Protocol** with seven concrete engine implementations:
|
||||
- Qwen3-TTS (PyTorch or MLX depending on platform)
|
||||
- Qwen CustomVoice (predefined speakers with instruct)
|
||||
- LuxTTS (fast, CPU-friendly)
|
||||
- Chatterbox Multilingual (23 languages)
|
||||
- Chatterbox Turbo (English, paralinguistic tags)
|
||||
- TADA (1B English, 3B multilingual via HumeAI)
|
||||
- Kokoro 82M (pre-built voices, CPU realtime)
|
||||
- **`STTBackend` Protocol** for Whisper (PyTorch or MLX-Whisper)
|
||||
- **Profiles / History / Stories** services for persistence and timeline editing
|
||||
|
||||
### Key Files
|
||||
|
||||
| Layer | File | Purpose |
|
||||
|-------|------|---------|
|
||||
| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2850 lines) |
|
||||
| TTS protocol | `backend/backends/__init__.py:32-101` | `TTSBackend` Protocol definition |
|
||||
| Model registry | `backend/backends/__init__.py:17-29,153-366` | `ModelConfig` dataclass + registry helpers |
|
||||
| TTS factory | `backend/backends/__init__.py:382-426` | Thread-safe engine registry (double-checked locking) |
|
||||
| PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
|
||||
| MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
|
||||
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
|
||||
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
|
||||
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
|
||||
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
|
||||
| Kokoro | `backend/backends/kokoro_backend.py` | Kokoro 82M — CPU realtime, pre-built voices |
|
||||
| Qwen CustomVoice | `backend/backends/qwen_custom_voice_backend.py` | Qwen CustomVoice — predefined speakers with instruct |
|
||||
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
|
||||
| API types | `backend/models.py` | Pydantic request/response models |
|
||||
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
|
||||
| Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
|
||||
| Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
|
||||
| Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
|
||||
| Engine selector | `app/src/components/Generation/EngineModelSelector.tsx` | Shared engine/model dropdown |
|
||||
| Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
|
||||
| Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
|
||||
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
|
||||
| GPU acceleration | `app/src/components/ServerSettings/GpuAcceleration.tsx` | CUDA backend swap UI |
|
||||
| Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission |
|
||||
| Language constants | `app/src/lib/constants/languages.ts` | Per-engine language maps |
|
||||
|
||||
### How TTS Generation Works (Current Flow)
|
||||
|
||||
```
|
||||
POST /generate
|
||||
1. Look up voice profile from DB
|
||||
2. Resolve engine from request (qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro)
|
||||
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
|
||||
4. Check model cache → if missing, trigger background download, return HTTP 202
|
||||
5. Load model (lazy): tts_backend.load_model(model_size)
|
||||
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
|
||||
→ tts_backend.create_voice_prompt(audio_path, reference_text)
|
||||
7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
|
||||
8. Post-process: trim_tts_output() for Chatterbox engines
|
||||
9. Save WAV → data/generations/{id}.wav
|
||||
10. Insert history record in SQLite
|
||||
11. Return GenerationResponse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### What's Shipped (v0.4.x)
|
||||
|
||||
**New since v0.3.0:**
|
||||
- Kokoro 82M TTS engine + voice profile type system (PR #325)
|
||||
- Qwen CustomVoice preset engine — predefined speakers with instruct support (PR #328)
|
||||
- Intel Arc (XPU) GPU support (PR #320)
|
||||
- Blackwell GPU (sm_120) CUDA support (PR #401)
|
||||
- Generation cancellation flow (PR #444)
|
||||
- Frontend quality gates + TypeScript hardening (PR #418)
|
||||
- macOS Intel (x86_64) PyTorch compatibility (PR #416)
|
||||
- Frozen-binary import fixes for Kokoro / Chatterbox Multilingual / scipy / transformers (PR #438)
|
||||
- Linux PipeWire/PulseAudio monitor detection (PR #457)
|
||||
- Server survives GUI close on Windows (PR #402)
|
||||
- GPU arch compatibility warning on startup (catches unsupported PyTorch builds)
|
||||
- cpal Stream playback reliability (PR #405), clip-splitting stability (PR #403)
|
||||
- torch.from_numpy crash with numpy 2.x in frozen binary (PR #361)
|
||||
- Async CUDA download lock (PR #428), NUMBA_CACHE_DIR env var (PR #425)
|
||||
- "Clear failed" history button (PR #412)
|
||||
- External server GUI startup + data refresh (PR #319)
|
||||
- Force offline mode for cached Qwen/Whisper models (PR #318)
|
||||
- macOS 11 ScreenCaptureKit launch crash fix (PR #424)
|
||||
|
||||
**Core TTS (cumulative):**
|
||||
- Qwen3-TTS voice cloning (1.7B and 0.6B models, MLX + PyTorch)
|
||||
- Qwen CustomVoice (preset speakers, instruct)
|
||||
- LuxTTS — fast, CPU-friendly English TTS (PR #254)
|
||||
- Chatterbox Multilingual — 23 languages including Hebrew (PR #257)
|
||||
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
|
||||
- HumeAI TADA — 1B English + 3B Multilingual (PR #296)
|
||||
- Kokoro 82M — CPU-realtime, 8 languages, Apache 2.0 (PR #325)
|
||||
- Multi-engine architecture with thread-safe backend registry (PR #254)
|
||||
- Chunked TTS generation — engine-agnostic, removes ~500 char limit (PR #266)
|
||||
- Async generation queue (PR #269)
|
||||
- Post-processing audio effects system (PR #271)
|
||||
- Voice profile type system (preset vs cloned, engine compatibility gating)
|
||||
- Centralized `ModelConfig` registry — no per-engine dispatch maps
|
||||
- Shared `EngineModelSelector` component
|
||||
|
||||
**Infrastructure (cumulative):**
|
||||
- CUDA backend swap via binary download (PR #252), cu128 upgrade (PR #316), Blackwell/sm_120 (PR #401)
|
||||
- CUDA backend split into independently versioned server + libs archives (PR #298)
|
||||
- Intel Arc XPU support (PR #320)
|
||||
- Docker + web deployment (PR #161)
|
||||
- Backend refactor: modular architecture, style guide, tooling (PR #285)
|
||||
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
|
||||
- Windows support: CUDA detection, cross-platform justfile, server lifecycle (PR #272, #402)
|
||||
- Linux audio capture via pactl monitor detection (PR #457)
|
||||
- macOS Intel x86_64 compatibility (PR #416)
|
||||
- Voice profiles with multi-sample support
|
||||
- Stories editor (multi-track DAW timeline)
|
||||
- Whisper transcription (base, small, medium, large, turbo variants)
|
||||
- Model management UI with inline download progress + folder migration (PR #268)
|
||||
- Download cancel/clear UI with error panel (PR #238)
|
||||
- Generation history with caching and cancellation (PR #444)
|
||||
- Streaming generation endpoint (MLX only)
|
||||
- Audio player freeze fix + UX improvements (PR #293)
|
||||
- CORS restriction to known local origins (PR #88)
|
||||
|
||||
### Abandoned / Backlogged Integrations
|
||||
|
||||
| Model | PR / Branch | Reason |
|
||||
|-------|-------------|--------|
|
||||
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. PR should be closed. |
|
||||
| **VoxCPM 1.5 / VoxCPM2** | `voicebox-new-models` research (2026-04-18) | **Backlogged.** See detailed analysis below. |
|
||||
|
||||
#### VoxCPM — Evaluation Notes (2026-04-18)
|
||||
|
||||
**Project:** [OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM) — tokenizer-free TTS, 2B params (VoxCPM2), end-to-end diffusion autoregressive architecture, 30 languages, 48 kHz output, Apache 2.0, `pip install voxcpm`.
|
||||
|
||||
**Why it looked interesting:**
|
||||
- Clean PyPI install (`pip install voxcpm`)
|
||||
- Apache 2.0 — commercially safe
|
||||
- Voice cloning via `reference_wav_path` with optional `prompt_wav_path` + `prompt_text` for "ultimate" cloning
|
||||
- Streaming API via `generate_streaming()`
|
||||
- Zero-shot cloning + style control via parenthetical prefixes in text (`(slightly faster, cheerful tone)...`)
|
||||
- Relatively high-quality output per demos
|
||||
|
||||
**Why we backlogged it:**
|
||||
- **Effectively CUDA-only.** README states `CUDA ≥ 12.0` as hard requirement. Source code's `from_pretrained(device=None|"auto")` claims "preferring CUDA, then MPS, then CPU," but in practice:
|
||||
- **MPS (Apple Silicon) broken upstream** — OpenBMB/VoxCPM issues #232 (`NotImplementedError: Output channels > 65536 not supported at the MPS device`) and #248 (`IndexError` on M3 Mac) are both open with no resolution.
|
||||
- **CPU unsupported in the Python package** — issue #256 shows `voxcpm --device cpu` rejected with `unrecognized arguments`. The only CPU path is the third-party **VoxCPM.cpp** GGML engine, which is a separate ecosystem project, not `pip install voxcpm`.
|
||||
- **macOS source install fails** — issue #233 open with no resolution.
|
||||
- Would require CUDA-only gating in UI (new `requires_cuda` flag on `ModelConfig`, lock icon + "Requires NVIDIA GPU" in `ModelManagement.tsx` / `EngineModelSelector.tsx`) plus a hard error at `load_model()` as safety net. Doable but adds first-class platform gating that doesn't exist for any other engine today.
|
||||
- Voicebox's user base skews Apple Silicon (MLX is a primary backend). Shipping a CUDA-only model sets a precedent worth a separate scoping discussion (see issues #419 engine sprawl, #420 platform tiers, PR #465).
|
||||
|
||||
**What would change the decision:**
|
||||
- Upstream fixes MPS crashes (watch issues #232, #248).
|
||||
- We define an "experimental / CUDA-only" engine tier as part of issue #419 / PR #465, and decide it's acceptable to ship engines that are hidden on non-NVIDIA platforms.
|
||||
- VoxCPM.cpp matures into a viable CPU path we can wrap (currently separate project, C++/GGML, unclear ergonomics).
|
||||
|
||||
**Integration shape if we revive it:** Zero-shot cloning maps naturally to the Chatterbox-style backend (store `ref_audio` + `ref_text` paths in the voice prompt dict, process at generate time). Est. ~250 lines for `voxcpm_backend.py` + one `ModelConfig` entry + engine registration in `backends/__init__.py`. Frontend UI gating is the bigger lift.
|
||||
|
||||
### What's In-Flight
|
||||
|
||||
| Feature | Branch/PR | Status |
|
||||
|---------|-----------|--------|
|
||||
| Platform support tiers | PR #465, issue #420 | Defining tier-1 (supported) vs tier-2 (community) platforms |
|
||||
| Engine sprawl cleanup | issue #419 | First-class vs experimental TTS backends distinction |
|
||||
| Frontend tech-debt burn-down | issue #421 | Biome + a11y debt before gating CI |
|
||||
| Docker registry auto-publish | PR #463, issue #453 | ghcr.io image on tag push |
|
||||
| New model research | `voicebox-new-models` branch | Evaluating Fish Speech, XTTS-v2, Pocket TTS, VibeVoice, Fish Audio S2, index-tts2 |
|
||||
|
||||
### TTS Engine Comparison
|
||||
|
||||
| Engine | Model Name | Profile Type | Languages | Size | Key Features | Instruct Support |
|
||||
|--------|-----------|--------------|-----------|------|-------------|-----------------|
|
||||
| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | Cloned | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
|
||||
| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | Cloned | 10 | ~1.2 GB | Lighter, faster | None |
|
||||
| Qwen CustomVoice 1.7B | `qwen-custom-voice-1.7B` | Preset | 10 | ~3.5 GB | Predefined speakers, instruct support | **Yes** |
|
||||
| Qwen CustomVoice 0.6B | `qwen-custom-voice-0.6B` | Preset | 10 | ~1.2 GB | Predefined speakers, instruct support | **Yes** |
|
||||
| LuxTTS | `luxtts` | Cloned | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
|
||||
| Chatterbox | `chatterbox-tts` | Cloned | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) |
|
||||
| Chatterbox Turbo | `chatterbox-turbo` | Cloned | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only |
|
||||
| TADA 1B | `tada-1b` | Cloned | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
|
||||
| TADA 3B Multilingual | `tada-3b-ml` | Cloned | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
|
||||
| Kokoro 82M | `kokoro` | Preset | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
|
||||
|
||||
### Multi-Engine Architecture (Shipped)
|
||||
|
||||
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
|
||||
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
|
||||
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro'`
|
||||
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
|
||||
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
|
||||
- **Profile type system** — preset vs cloned profiles, UI grays out incompatible engines and auto-switches on selection
|
||||
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
|
||||
|
||||
### Known Limitations
|
||||
|
||||
- **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
|
||||
- **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
|
||||
- **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
|
||||
- **Instruct parameter partially shipped** (#224, #303): Qwen CustomVoice (PR #328) now provides real instruct support via predefined speakers. Other backends still silently drop the instruct field — the UI exposes the field broadly but most engines ignore it. The floating generate box was patched to restore instruct for CustomVoice (commit `106aec4`).
|
||||
- **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
|
||||
- **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.
|
||||
- **Blackwell (RTX 50-series) CUDA**: cu128 + sm_120 kernel support shipped (PR #401, #316), but users still report `cudaErrorNoKernelImageForDevice` (#417, #400, #396, #395, #390, #362) — likely a stale CUDA binary on upgraded installs. Needs a follow-up diagnostic / forced re-download path.
|
||||
- **Long text 50k character limit** (#464, #365, #354): Still hit on GPU despite chunking (PR #266). Chunking reliability needs another pass.
|
||||
- **ROCm on RDNA 3/4** (#469): `HSA_OVERRIDE_GFX_VERSION` is hardcoded and harms newer cards.
|
||||
|
||||
---
|
||||
|
||||
## Open PRs — Triage & Analysis
|
||||
|
||||
### Recently Merged (Since Last Update — 2026-03-18 → 2026-04-18)
|
||||
|
||||
| PR | Title | Merged |
|
||||
|----|-------|--------|
|
||||
| **#481** | fix(build): pin transformers in MLX requirements to prevent 5.x upgrade | 2026-04-19 |
|
||||
| **#470** | fix(api-client): declare moved + errors on migrateModels response type | 2026-04-18 |
|
||||
| **#457** | fix(linux): use pactl to detect PipeWire/PulseAudio monitor | 2026-04-18 |
|
||||
| **#450** | docs: clarify paralinguistic tag support in quick start | 2026-04-18 |
|
||||
| **#447** | fix: delete version rows and files in delete_generations_by_profile | 2026-04-18 |
|
||||
| **#444** | Fix generation cancellation flow | 2026-04-18 |
|
||||
| **#440** | fix(paths): strip legacy "data/" prefix when resolving stored paths | 2026-04-18 |
|
||||
| **#439** | Fix migration dialog hanging when no models are present | 2026-04-18 |
|
||||
| **#438** | fix(build): repair frozen-binary imports for kokoro/chatterbox-multilingual/scipy/transformers | 2026-04-18 |
|
||||
| **#433** | fix: warn user when no models to migrate during storage change | 2026-04-18 |
|
||||
| **#425** | Add NUMBA_CACHE_DIR environment variable | 2026-04-16 |
|
||||
| **#424** | fix: avoid ScreenCaptureKit launch crash on macOS 11 | 2026-04-16 |
|
||||
| **#418** | Frontend quality gates + TypeScript hardening | 2026-04-18 |
|
||||
| **#416** | fix(deps): relax PyTorch requirement for macOS Intel (x86_64) | 2026-04-16 |
|
||||
| **#412** | feat(history): add "Clear failed" button | 2026-04-16 |
|
||||
| **#405** | fix: keep cpal Stream alive until playback completes | 2026-04-16 |
|
||||
| **#403** | fix: prevent intermittent clip splitting failures | 2026-04-16 |
|
||||
| **#402** | fix: reliably keep server alive after GUI close on Windows | 2026-04-16 |
|
||||
| **#401** | feat: add Blackwell GPU (sm_120) CUDA support | 2026-04-16 |
|
||||
| **#394** | fix(history): populate status/error/engine fields from DB row | 2026-04-16 |
|
||||
| **#384** | Fix: Resolve ModuleNotFoundError in effects service | 2026-04-16 |
|
||||
| **#361** | fix: torch.from_numpy crash with numpy 2.x in frozen binary | 2026-04-16 |
|
||||
| **#345** | Fix: "Failed to Save" preset error by resolving backend import path | 2026-03-22 |
|
||||
| **#344** | fix: include changelog in docker web build | 2026-03-27 |
|
||||
| **#332** | Fix links in Get Started section of index.mdx | 2026-03-21 |
|
||||
| **#328** | feat: add Qwen CustomVoice preset engine | 2026-03-27 |
|
||||
| **#325** | feat: Kokoro 82M TTS engine + voice profile type system | 2026-03-20 |
|
||||
| **#321** | fix: allows deletion of failed generations | 2026-03-19 |
|
||||
| **#320** | feat: Intel Arc (XPU) GPU support | 2026-03-21 |
|
||||
| **#319** | fix: GUI startup with external server + data refresh on server switch | 2026-03-27 |
|
||||
| **#318** | fix: force offline mode when loading cached models (Qwen TTS & Whisper) | 2026-03-21 |
|
||||
| **#316** | Upgrade CUDA backend from cu126 to cu128, fix GPU settings UI | 2026-03-18 |
|
||||
|
||||
### Currently Open (12 PRs)
|
||||
|
||||
| PR | Title | Status | Notes |
|
||||
|----|-------|--------|-------|
|
||||
| **#465** | docs: define tier-1 and tier-2 platform support targets | Community PR | Pairs with issue #420. Important for scoping. |
|
||||
| **#463** | feat(actions): add docker-registry.yml for automatic ghcr.io publishing | Community PR | Pairs with issue #453. Low risk. |
|
||||
| **#443** | fix: prevent infinite retry loop in offline mode (#434) | Community PR | Fixes reported bug. |
|
||||
| **#430** | feat: add MiniMax TTS provider support | Community PR | Cloud TTS provider — new direction (external API). Superset of #331? |
|
||||
| **#331** | feat: add MiniMax Cloud TTS as a built-in engine | Community PR | Likely superseded by #430. Dedupe. |
|
||||
| **#311** | feat: add CosyVoice2/3 TTS engine | **Close** | Abandoned — output quality too poor. |
|
||||
| **#253** | Enhance speech tokenizer with 48kHz version | Community PR | Qwen tokenizer upgrade. Still worth reviewing. |
|
||||
| **#227** | fix: harden input validation & file safety | Community PR | Coupled to #225 (custom models). |
|
||||
| **#225** | feat: custom HuggingFace voice model support | Community PR | Needs rework for multi-engine arch. |
|
||||
| **#195** | feat: per-profile LoRA fine-tuning | Draft | Complex. 15 new endpoints. |
|
||||
| **#154** | feat: Audiobook tab | Community PR | Chunked generation now shipped (#266). |
|
||||
| **#91** | fix: CoreAudio device enumeration | Draft | macOS audio device handling. |
|
||||
|
||||
---
|
||||
|
||||
## Open Issues — Categorized
|
||||
|
||||
### GPU / Hardware Detection — still the top category
|
||||
|
||||
**RTX 50-series (Blackwell / sm_120) cluster — NEW:** #417, #400, #396, #395, #390, #362 all report `cudaErrorNoKernelImageForDevice` / "no kernel image available." sm_120 support shipped in PR #401 + cu128 in PR #316, but users on upgraded installs still hit it — likely stale CUDA binary. Needs a diagnostic that detects binary/GPU-arch mismatch and prompts re-download.
|
||||
|
||||
**AMD / ROCm — NEW:** #469 `HSA_OVERRIDE_GFX_VERSION` is hardcoded and breaks RDNA 3/4 cards. #313 DirectML on AMD Ryzen AI Max+ 395 not working.
|
||||
|
||||
**Intel Arc:** PR #320 shipped XPU support — may resolve #119.
|
||||
|
||||
**General GPU-not-detected (older):** #368, #310, #330, #324, #326, #355 (multi-GPU / eGPU).
|
||||
|
||||
**Fix path:** CUDA backend swap (PR #252) + cu128 (PR #316) + sm_120 (PR #401) + GPU-arch warning (`73170d0`) are all in. Remaining work is diagnostics + re-download prompts for users whose binary predates the kernel updates.
|
||||
|
||||
### Model Downloads
|
||||
|
||||
Still reported. Users get stuck downloads, can't resume, offline mode edge cases.
|
||||
|
||||
**Key issues:** #475 (MAC CustomVoice install error), #449 (infinite loading macOS), #445 (can't download CustomVoice), #462 (Qwen requires internet even when loaded — regression from #150), #434 (infinite retry loop offline — PR #443 open), #432 (storage location change hangs when empty — partly fixed by PR #439/#433), #181, #180.
|
||||
|
||||
**Fix path:** PR #443 addresses infinite offline retry. CustomVoice-specific download failures (#475, #445) need triage — likely related to frozen-binary import fixes in PR #438.
|
||||
|
||||
### Language Requests (ongoing)
|
||||
|
||||
Strong demand: Hungarian (#479), Indonesian (#458, #247), Thai (#455), Bangla (#454), Arabic (#379), Persian (#162), IndicF5 (#339 — Indian languages), Ukrainian (#109), Chinese UI (#392, #261).
|
||||
|
||||
**Fix path:** Chatterbox Multilingual (PR #257) covers Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Still missing: Hungarian, Indonesian, Thai, Bangla, Ukrainian. Issue #411 offers a PR for UI i18n foundation.
|
||||
|
||||
### New Model Requests (growing)
|
||||
|
||||
| Issue | Model Requested |
|
||||
|-------|----------------|
|
||||
| #478 | CosyVoice3 (we tried & abandoned CosyVoice2/3 — see #311) |
|
||||
| #407, #347 | RVC-style voice-to-voice / seed voice conversion (STS) |
|
||||
| #385 | Fish Audio S2 |
|
||||
| #380 | OmniVoice |
|
||||
| #370 | index-tts2 |
|
||||
| #364 | Voxtral-TTS |
|
||||
| #335 | Faster-Qwen-TTS |
|
||||
| #346 | Multi-model batch request |
|
||||
| #381 | Microsoft MAI models |
|
||||
| #339 | IndicF5 |
|
||||
| #226 | GGUF support |
|
||||
| #172 | VibeVoice |
|
||||
| #138 | Export to ONNX/Piper format |
|
||||
| #132 | LavaSR (transcription) |
|
||||
| #147 | Facebook Omnilingual ASR |
|
||||
| #338 | Default voices |
|
||||
|
||||
The multi-engine architecture makes integration straightforward — see [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx). Platform-specific gating (e.g. VoxCPM CUDA-only) doesn't exist yet and would need design.
|
||||
|
||||
### Platform Scope & Quality Debt — NEW category
|
||||
|
||||
Awareness issues filed this cycle — ties into engine sprawl and platform tier work.
|
||||
|
||||
- **#419** — Engine sprawl: define first-class vs experimental TTS backends
|
||||
- **#420** — Formalize tier-1 vs tier-2 platform support targets (PR #465 open)
|
||||
- **#421** — Track & burn down frontend Biome + a11y debt before gating CI
|
||||
- **#422** — Code-split web build (main bundle > 1 MB)
|
||||
|
||||
### Long-Form / Chunking
|
||||
|
||||
Still reported despite chunking + queue being merged.
|
||||
|
||||
**Key issues:** #464 (50k char limit on GPU despite 16 GB VRAM — v0.4.0), #365 (FR: >50k chars), #363 (smart chunking to prevent robotic artifacts), #354 (50k limit v0.3.0).
|
||||
|
||||
**Fix path:** Chunking (#266) and queue (#269) shipped. Remaining work is raising/removing the 50k guard and tuning chunk boundaries for prosody.
|
||||
|
||||
### Feature Requests (ongoing)
|
||||
|
||||
Notable:
|
||||
- **#480** — Noise removal on uploaded recordings
|
||||
- **#448** — API for non-Qwen models (external integrations)
|
||||
- **#427** — Task status control
|
||||
- **#407, #347** — Voice-to-voice / audio-to-audio conversion
|
||||
- **#387** — Location of downloaded generated voices
|
||||
- **#383** — Concatenate partial reference audio into generated audio
|
||||
- **#382** — Lightning.ai support
|
||||
- **#376** — Remote mode
|
||||
- **#173** — Vocal intonation/inflection control
|
||||
- **#165, #270** — Audiobook mode (PR #154 open)
|
||||
- **#242** — Seed value pinning
|
||||
- **#228** — Always use 0.6B option
|
||||
- **#235** — Finetuned Qwen3-TTS tokenizer (PR #253 open)
|
||||
- **#144** — Copy text to clipboard
|
||||
|
||||
### Bugs (ongoing)
|
||||
|
||||
| Category | Issues |
|
||||
|----------|--------|
|
||||
| Generation failures | #476, #467, #452, #459 (voice clone fetch error), #468 (tada-1b marked error), #437, #282 |
|
||||
| Audio quality | #456 (clipping errors v0.4.0), #436 (emotion labels), #333 (pitch/echo), #307 (by-model breakdown) |
|
||||
| File ops | #477 (spacy_pkuseg dict missing on frozen Windows build), #472 (storage location change) |
|
||||
| Windows | #466 (install problem), #273 (port 8000 conflict) |
|
||||
| Linux | #471 (thread-safe PULSE_SOURCE), #413 (Arch build), #409 (Kubuntu build), #341 |
|
||||
| macOS | #441 (older macOS), #369 (malware flag), #171 (ARM64 binary won't open) |
|
||||
| Profile/UI | #360 (Kokoro profile hides others — partly addressed by auto-switch), #299 (drag-drop on Win11), #329 (size selector state bug) |
|
||||
| Database | #174 (sqlite3 IntegrityError) |
|
||||
|
||||
---
|
||||
|
||||
## Existing Plan Documents — Status
|
||||
|
||||
| Document | Target Version | Status | Relevance |
|
||||
|----------|---------------|--------|-----------|
|
||||
| `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially superseded** by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
|
||||
| `CUDA_BACKEND_SWAP.md` | — | **Shipped** (PR #252) | CUDA binary download + backend restart |
|
||||
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
|
||||
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
|
||||
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
|
||||
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
|
||||
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
|
||||
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
|
||||
|
||||
---
|
||||
|
||||
## New Model Integration — Landscape
|
||||
|
||||
### Status Snapshot (2026-04-18)
|
||||
|
||||
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct | Cross-platform? | Status |
|
||||
|-------|---------|-------|-------------|-----------|------|----------|-----------------|--------|
|
||||
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None | MLX + PyTorch | **Shipped** |
|
||||
| **Qwen CustomVoice** | Preset speakers | Medium | 24 kHz | 10 | Medium | **Yes** | PyTorch | **Shipped** (PR #328) |
|
||||
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | All | **Shipped** (PR #254) |
|
||||
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` | CPU/CUDA | **Shipped** (PR #257) |
|
||||
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags | CPU/CUDA | **Shipped** (PR #258) |
|
||||
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), 10 (3B) | Medium | Partial — prosody | PyTorch | **Shipped** (PR #296) |
|
||||
| **Kokoro-82M** | Preset voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | All | **Shipped** (PR #325) |
|
||||
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** | — | **Abandoned** (PR #311) — poor output quality |
|
||||
| ~~**VoxCPM2**~~ | Zero-shot | ~0.15 RTF streaming | 48 kHz | 30 | Medium | Partial — parenthetical style | **CUDA-only in practice** | **Backlogged** (2026-04-18) — see notes above |
|
||||
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — word-level inline | All | Candidate — license TBD |
|
||||
| **Fish Audio S2** | — | — | — | — | — | — | — | Candidate (#385) |
|
||||
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref | All | Candidate — CPML license likely blocker |
|
||||
| **Pocket TTS** (Kyutai) | Zero-shot + streaming | >1x RT on CPU | — | English + several European (FR/DE/PT/IT/ES added by Feb 2026) | ~100M | None | CPU-first | Candidate — MIT |
|
||||
| **MOSS-TTS-Nano** | Zero-shot | **Realtime on 4 CPU cores** | 48 kHz stereo | 20 | 0.1B | Partial — MOSS-VoiceGenerator companion does text-to-voice design | All (ONNX CPU path dropped 2026-04-17) | **Top candidate** — Apache 2.0, released 2026-04-13, streaming |
|
||||
| **VibeVoice** (Microsoft) | — | — | — | Multi-speaker long-form (up to 90 min, 4 speakers) | 1.5B | — | — | Candidate (#172) — Stories-editor fit |
|
||||
| **index-tts2** | — | — | — | — | — | — | — | Candidate (#370) |
|
||||
| **Voxtral TTS** (Mistral) | Zero-shot (short clips) + 20 preset voices | Single-GPU | — | — | 4B (`Voxtral-4B-TTS-2603`) | Presets + cloning | CUDA (16 GB+ VRAM) | Candidate (#364) — frontier quality claim, open-weight |
|
||||
| **Dia / Dia2** | — | — | — | — | — | — | — | Watch — emotion-forward, but "rough edges" / artifacts per April reviews |
|
||||
| **IndicF5** | — | — | — | Indian languages | — | — | — | Candidate (#339) — fills Indic gap |
|
||||
| **MiniMax Cloud TTS** | — | Cloud | — | — | N/A (API) | — | N/A | Community PR #430, #331 — new direction (external API) |
|
||||
| **OmniVoice** | — | — | — | — | — | — | — | Candidate (#380) |
|
||||
| **RVC voice conversion** | N/A (STS) | — | — | — | — | N/A | All | New modality, not TTS (#407, #347) |
|
||||
|
||||
**Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control), Faster-Qwen-TTS (#335), Orpheus / Sesame CSM (on-device fine-tuning discussions), Fish Audio S2 Pro / Fish Speech V1.5 (benchmark leader but research/non-commercial license — same blocker as Fish Speech).
|
||||
|
||||
**Deep-research pass (2026-04-18):** MOSS-TTS-Nano identified as the freshest high-alignment candidate — verified via [OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) README (0.1B params, Apache 2.0, 48 kHz stereo, 4-core CPU realtime, streaming, released 2026-04-13). Dedicated repo: [OpenMOSS/MOSS-TTS-Nano](https://github.com/OpenMOSS/MOSS-TTS-Nano). Voxtral TTS verified on HF as `mistralai/Voxtral-4B-TTS-2603`.
|
||||
|
||||
#### Active Evaluation Criteria (learned from cycle)
|
||||
|
||||
1. **Cross-platform first.** MLX is a primary backend for our Apple Silicon user base. CUDA-only models require platform gating that doesn't exist yet — shipping one sets a precedent (see VoxCPM notes, issues #419/#420).
|
||||
2. **PyPI + Apache/MIT licensing preferred.** Heavy deps, git-only installs, and `--no-deps` workarounds are expensive to maintain (Chatterbox taught us this).
|
||||
3. **Output quality is non-negotiable.** CosyVoice was abandoned despite the best instruct API.
|
||||
4. **Instruct support fills a real gap** (#173, #224, #303). Qwen CustomVoice partially addresses it with preset speakers; zero-shot clone-with-instruct is still unmet.
|
||||
5. **Long-form + streaming are user-requested** (#363, #365, #464). Candidates with native streaming (Pocket TTS, Fish Speech) get extra weight.
|
||||
|
||||
### Adding a New Engine (Now Straightforward)
|
||||
|
||||
With the model config registry and shared `EngineModelSelector` component, adding a new TTS engine requires:
|
||||
|
||||
1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
|
||||
2. **Register in `backend/backends/__init__.py`** — add `ModelConfig` entry + `TTS_ENGINES` entry + factory elif
|
||||
3. **Update `backend/models.py`** — add engine name to regex
|
||||
4. **Update frontend** — add to engine union type, `EngineModelSelector` options, form schema, language map, profile type gating (icons/labels ~9 files per grep of `kokoro`)
|
||||
|
||||
`main.py` requires **zero changes** — the registry handles all dispatch automatically.
|
||||
|
||||
**Platform gating doesn't exist yet.** If we add a CUDA-only model (e.g. VoxCPM), we need a new `requires_cuda` (or more generally `requires: list[device]`) flag on `ModelConfig`, plumbed through `/models` API and surfaced in `ModelManagement.tsx` and `EngineModelSelector.tsx` as a lock icon + "Requires NVIDIA GPU" state. Backend should hard-error at `load_model()` as a safety net.
|
||||
|
||||
Total effort: **~1 day** for a well-documented model with a PyPI package, cross-platform. **~2 days** if platform gating is required. See [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx) for the full guide.
|
||||
|
||||
---
|
||||
|
||||
## Architectural Bottlenecks
|
||||
|
||||
### ~~1. Single Backend Singleton~~ — RESOLVED
|
||||
|
||||
The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
|
||||
|
||||
### ~~2. `main.py` Dispatch Point Duplication~~ — RESOLVED
|
||||
|
||||
Previously, each engine required updates to 6+ hardcoded dispatch maps across `main.py` (~320 lines of if/elif chains). A model config registry in `backend/backends/__init__.py` now centralizes all model metadata (`ModelConfig` dataclass) with helper functions (`load_engine_model()`, `check_model_loaded()`, `engine_needs_trim()`, etc.). Adding a new engine requires zero changes to `main.py`.
|
||||
|
||||
### ~~3. Model Config is Scattered~~ — RESOLVED
|
||||
|
||||
Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the `ModelConfig` registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in `EngineModelSelector.tsx`.
|
||||
|
||||
### 4. Voice Prompt Cache Assumes PyTorch Tensors
|
||||
|
||||
`backend/utils/cache.py` uses `torch.save()` / `torch.load()`. LuxTTS, Chatterbox, and Kokoro backends work around this by storing reference audio paths (or preset voice IDs) instead of tensors in their voice prompt dicts. Not ideal but functional.
|
||||
|
||||
### 5. ~~Frontend Assumes Qwen Model Sizes~~ — RESOLVED
|
||||
|
||||
The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen / Qwen CustomVoice.
|
||||
|
||||
### 6. No Platform Gating on Models — NEW
|
||||
|
||||
`ModelConfig` has no way to express hardware requirements. Every engine is shown to every user, regardless of whether it'll actually load. Users on non-CUDA platforms discover failure at load time (or not at all — some fall back silently to CPU and never complete). Blocks shipping CUDA-only engines (VoxCPM) and would improve the Intel Arc / ROCm / CPU-only UX today. See `ModelConfig` TODO: add `requires: list[Literal["cuda", "mps", "xpu", "cpu", "rocm"]]` or equivalent, plumb through `/models` API, render in `ModelManagement.tsx` + `EngineModelSelector.tsx`.
|
||||
|
||||
### 7. Engine Sprawl — NEW
|
||||
|
||||
Seven TTS engines shipped, more candidates queued. Issue #419 asks for a first-class vs experimental distinction. Related: issue #420 asks for formalized platform support tiers. Combined, these would let us ship more engines more confidently with clearer expectations for users.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Priorities
|
||||
|
||||
### Tier 1 — Ship Now
|
||||
|
||||
| Priority | PR/Item | Impact | Effort |
|
||||
|----------|---------|--------|--------|
|
||||
| 1 | **RTX 50-series / Blackwell diagnostic** — detect stale CUDA binary vs GPU arch, prompt re-download (#417, #400, #396, #395, #390, #362) | Large cluster of user-blocking errors | Medium |
|
||||
| 2 | **CustomVoice download failures** (#475, #445) | New engine blocked on MAC/Win — regression triage | Medium |
|
||||
| 3 | **50k char limit on GPU** (#464) | Regression — chunking should handle this | Medium |
|
||||
| 4 | Close PR #311 (CosyVoice) and dedupe #331/#430 (MiniMax) | Housekeeping | None |
|
||||
| 5 | **PR #443** — infinite offline retry loop | Bug fix, reviewable | Low |
|
||||
| 6 | **PR #465** — define tier-1 / tier-2 platforms | Unblocks engine-sprawl decision (#419) | Low |
|
||||
| 7 | **PR #463** — docker registry auto-publish | Community PR, low risk | Low |
|
||||
| 8 | **#253** — 48kHz speech tokenizer | Quality improvement for Qwen | Medium |
|
||||
| 9 | **Kokoro profile UX** (#360) — partially addressed by auto-switch | Polish | Low |
|
||||
|
||||
### Tier 2 — Feature Work
|
||||
|
||||
| Priority | Item | Impact | Effort |
|
||||
|----------|------|--------|--------|
|
||||
| 1 | **Engine tier system** (#419) — first-class vs experimental, platform gating in `ModelConfig` | Unblocks CUDA-only engines (VoxCPM, etc.) and frontend polish | Medium |
|
||||
| 2 | **Frontend tech-debt burn-down** (#421) + code-split (#422) | Before gating CI on Biome | Medium |
|
||||
| 3 | **#154** — Audiobook tab | Long-form users. Chunking + queue shipped. | Medium |
|
||||
| 4 | **UI i18n** (#411 PR offer, #392, #261) | Chinese UI + general localization | Medium |
|
||||
| 5 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
|
||||
| 6 | OpenAI-compatible API (plan doc exists) — see also #448 (API for non-Qwen) | Low effort once API is stable | Low |
|
||||
| 7 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
|
||||
| 8 | Streaming for non-MLX engines | Currently MLX-only | Medium |
|
||||
| 9 | Voice-to-voice / RVC (#407, #347) | New modality — different arch shape | High |
|
||||
|
||||
### Tier 3 — Future Engines (cross-platform preferred)
|
||||
|
||||
| Priority | Item | Notes |
|
||||
|----------|------|-------|
|
||||
| 1 | **MOSS-TTS-Nano** | 0.1B, Apache 2.0, 4-core CPU realtime, 48 kHz stereo, streaming, 20 langs, released 2026-04-13. Best alignment with our criteria. Verify install ergonomics before committing. |
|
||||
| 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. Fills streaming gap without CUDA dependency. Several European langs added by Feb 2026. |
|
||||
| 3 | **IndicF5** | Fills Indian-language gap (#339). Closes many language-request issues. |
|
||||
| 4 | **VibeVoice** (Microsoft, #172) | 1.5B, long-form multi-speaker (up to 90 min, 4 speakers). Strong Stories-editor fit. |
|
||||
| 5 | **Voxtral TTS** (Mistral, #364) | 4B presets+cloning. Frontier quality claim, but 16 GB+ VRAM — would need the platform-tier work first. |
|
||||
| 6 | **Fish Speech / Fish Audio S2** | 50+ langs, word-level instruct. **License clarification first.** (#385) |
|
||||
| 7 | **XTTS-v2** | 17+ langs, mature pip. CPML likely kills commercial use — verify. |
|
||||
| 8 | **index-tts2** (#370) | Unvetted. |
|
||||
| — | ~~**VoxCPM2**~~ | **Backlogged** — CUDA-only upstream. Revisit when tier system ships or MPS bugs are fixed upstream. |
|
||||
|
||||
### ~~Previously Prioritized — Now Done~~
|
||||
|
||||
- ~~Kokoro 82M — finish integration~~ **Shipped** (PR #325)
|
||||
- ~~Qwen CustomVoice~~ **Shipped** (PR #328)
|
||||
- ~~Intel Arc (XPU) support~~ **Shipped** (PR #320)
|
||||
- ~~Blackwell CUDA~~ **Shipped** (PR #401, follow-up work open)
|
||||
- ~~Generation cancellation~~ **Shipped** (PR #444)
|
||||
- ~~macOS Intel x86_64~~ **Shipped** (PR #416)
|
||||
|
||||
---
|
||||
|
||||
## Branch Inventory
|
||||
|
||||
| Branch | PR | Status | Notes |
|
||||
|--------|-----|--------|-------|
|
||||
| `voicebox-new-models` | — | **Active** | New model research (Fish Speech, Pocket TTS, VibeVoice, etc.); VoxCPM evaluated & backlogged |
|
||||
| `fix/kokoro-pyinstaller-source-files` | — | Active | Kokoro frozen-build source bundling (parent of `voicebox-new-models`) |
|
||||
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
|
||||
| `feat/kokoro` | #325 | **Merged** | Kokoro 82M + voice profile type system |
|
||||
| `feat/qwen-custom-voice` | #328 | **Merged** | Qwen CustomVoice preset engine |
|
||||
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
|
||||
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
|
||||
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: API Endpoints
|
||||
|
||||
<details>
|
||||
<summary>All current endpoints</summary>
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/health` | GET | Health check, model/GPU status |
|
||||
| `/profiles` | POST, GET | Create/list voice profiles |
|
||||
| `/profiles/{id}` | GET, PUT, DELETE | Profile CRUD |
|
||||
| `/profiles/{id}/samples` | POST, GET | Add/list voice samples |
|
||||
| `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management |
|
||||
| `/profiles/{id}/export` | GET | Export profile as ZIP |
|
||||
| `/profiles/import` | POST | Import profile from ZIP |
|
||||
| `/generate` | POST | Generate speech (engine param selects TTS backend) |
|
||||
| `/generate/stream` | POST | Stream speech (MLX only) |
|
||||
| `/history` | GET | List generation history |
|
||||
| `/history/{id}` | GET, DELETE | Get/delete generation |
|
||||
| `/history/{id}/export` | GET | Export generation ZIP |
|
||||
| `/history/{id}/export-audio` | GET | Export audio only |
|
||||
| `/transcribe` | POST | Transcribe audio (Whisper) |
|
||||
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
|
||||
| `/models/download` | POST | Trigger model download |
|
||||
| `/models/download/cancel` | POST | Cancel/dismiss download |
|
||||
| `/models/{name}` | DELETE | Delete downloaded model |
|
||||
| `/models/load` | POST | Load model into memory |
|
||||
| `/models/unload` | POST | Unload model |
|
||||
| `/models/progress/{name}` | GET | SSE download progress |
|
||||
| `/tasks/active` | GET | Active downloads/generations (with inline progress) |
|
||||
| `/stories` | POST, GET | Create/list stories |
|
||||
| `/stories/{id}` | GET, PUT, DELETE | Story CRUD |
|
||||
| `/stories/{id}/items` | POST, GET | Story items CRUD |
|
||||
| `/stories/{id}/export` | GET | Export story audio |
|
||||
| `/channels` | POST, GET | Audio channel CRUD |
|
||||
| `/channels/{id}` | PUT, DELETE | Channel update/delete |
|
||||
| `/cache/clear` | POST | Clear voice prompt cache |
|
||||
| `/server/cuda/status` | GET | CUDA binary availability |
|
||||
| `/server/cuda/download` | POST | Download CUDA binary |
|
||||
| `/server/cuda/switch` | POST | Switch to CUDA backend |
|
||||
|
||||
</details>
|
||||
+4
-4
@@ -1,7 +1,7 @@
|
||||
@import 'tailwindcss';
|
||||
@import 'fumadocs-ui/css/neutral.css';
|
||||
@import 'fumadocs-ui/css/preset.css';
|
||||
@import 'fumadocs-openapi/css/preset.css';
|
||||
@import "tailwindcss";
|
||||
@import "fumadocs-ui/css/neutral.css";
|
||||
@import "fumadocs-ui/css/preset.css";
|
||||
@import "fumadocs-openapi/css/preset.css";
|
||||
|
||||
:root {
|
||||
--color-fd-primary: hsl(43, 50%, 50%);
|
||||
|
||||
@@ -5,22 +5,13 @@ import { generate as DefaultImage } from 'fumadocs-ui/og';
|
||||
|
||||
export const revalidate = false;
|
||||
|
||||
export async function GET(
|
||||
_req: Request,
|
||||
{ params }: RouteContext<'/og/docs/[...slug]'>,
|
||||
) {
|
||||
export async function GET(_req: Request, { params }: RouteContext<'/og/docs/[...slug]'>) {
|
||||
const { slug } = await params;
|
||||
const page = source.getPage(slug.slice(0, -1));
|
||||
if (!page) notFound();
|
||||
|
||||
return new ImageResponse(
|
||||
(
|
||||
<DefaultImage
|
||||
title={page.data.title}
|
||||
description={page.data.description}
|
||||
site="My App"
|
||||
/>
|
||||
),
|
||||
<DefaultImage title={page.data.title} description={page.data.description} site="My App" />,
|
||||
{
|
||||
width: 1200,
|
||||
height: 630,
|
||||
|
||||
+13
-42
@@ -7,61 +7,32 @@ This directory contains the documentation for Voicebox, built with [Fumadocs](ht
|
||||
|
||||
## Development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Install Mintlify globally using bun:
|
||||
|
||||
```bash
|
||||
bun add -g mintlify
|
||||
```
|
||||
|
||||
Or use the helper script:
|
||||
|
||||
```bash
|
||||
bun run install:mintlify
|
||||
```
|
||||
|
||||
### Running Locally
|
||||
|
||||
From the `docs/` directory:
|
||||
|
||||
```bash
|
||||
bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
This will start the Mintlify dev server.
|
||||
|
||||
The docs will be available at `http://localhost:3000`
|
||||
The docs will be available at `http://localhost:3000`.
|
||||
|
||||
### Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── mint.json # Mintlify configuration
|
||||
├── custom.css # Custom styles
|
||||
├── overview/ # Getting started & feature docs
|
||||
├── guides/ # User guides
|
||||
├── api/ # API reference
|
||||
├── development/ # Developer documentation
|
||||
├── logo/ # Logo assets
|
||||
└── public/ # Static assets
|
||||
```
|
||||
- `content/docs/overview/` — user-facing guides (installation, quick start, feature walkthroughs)
|
||||
- `content/docs/developer/` — architecture, backend internals, and contributor guides
|
||||
- `content/docs/api-reference/` — auto-generated from the backend's OpenAPI schema
|
||||
- `content/docs/index.mdx` — landing page
|
||||
- `public/` — static assets (images, screenshots, videos)
|
||||
|
||||
### Writing Docs
|
||||
|
||||
- Use `.mdx` files for all documentation pages
|
||||
- Follow the existing structure in `mint.json` for navigation
|
||||
- Use Mintlify components for enhanced formatting (Card, CardGroup, Accordion, etc.)
|
||||
- Reference the [Mintlify documentation](https://mintlify.com/docs) for available components
|
||||
- Navigation is generated from `content/docs/meta.json` files
|
||||
- Fumadocs components available: `Callout`, `Cards` / `Card`, `Tabs` / `Tab`, `Steps` / `Step`, `Accordion` / `AccordionGroup`, `Files` / `Folder` / `File`
|
||||
- API reference pages under `api-reference/` are regenerated from the backend's OpenAPI schema — don't edit them by hand
|
||||
|
||||
## Deployment
|
||||
|
||||
Docs are automatically deployed when changes are pushed to the main branch.
|
||||
|
||||
To manually deploy:
|
||||
|
||||
```bash
|
||||
mintlify deploy
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](../CONTRIBUTING.md) for contribution guidelines.
|
||||
Docs are automatically deployed when changes land on `main`.
|
||||
|
||||
@@ -100,9 +100,9 @@ chmod +x voicebox-*.AppImage
|
||||
**Solutions:**
|
||||
1. **Rebuild server binary**
|
||||
```bash
|
||||
bun run build:server
|
||||
just build-server
|
||||
```
|
||||
The build script should automatically include MLX Metal shader libraries.
|
||||
The build script automatically includes MLX Metal shader libraries on Apple Silicon.
|
||||
|
||||
2. **Check MLX installation**
|
||||
```bash
|
||||
@@ -219,9 +219,9 @@ chmod +x voicebox-*.AppImage
|
||||
|
||||
**Solutions:**
|
||||
1. **Check data directory**
|
||||
- macOS: `~/Library/Application Support/voicebox/`
|
||||
- Windows: `%APPDATA%/voicebox/`
|
||||
- Linux: `~/.local/share/voicebox/`
|
||||
- macOS: `~/Library/Application Support/sh.voicebox.app/`
|
||||
- Windows: `%APPDATA%/sh.voicebox.app/`
|
||||
- Linux: `~/.config/sh.voicebox.app/`
|
||||
|
||||
2. **Check database**
|
||||
- Database: `data/voicebox.db`
|
||||
@@ -266,7 +266,7 @@ chmod +x voicebox-*.AppImage
|
||||
cd tauri/src-tauri
|
||||
cargo clean
|
||||
cd ../..
|
||||
bun run build
|
||||
just build
|
||||
```
|
||||
|
||||
### API client generation fails
|
||||
@@ -274,7 +274,7 @@ chmod +x voicebox-*.AppImage
|
||||
**Solutions:**
|
||||
1. **Start backend server**
|
||||
```bash
|
||||
bun run dev:server
|
||||
just dev-backend
|
||||
```
|
||||
|
||||
2. **Check OpenAPI endpoint**
|
||||
@@ -284,7 +284,7 @@ chmod +x voicebox-*.AppImage
|
||||
|
||||
3. **Regenerate client**
|
||||
```bash
|
||||
bun run generate:api
|
||||
just generate-api
|
||||
```
|
||||
|
||||
## Still Having Issues?
|
||||
|
||||
@@ -9,9 +9,9 @@ Voicebox uses a client-server architecture with a React frontend and Python back
|
||||
|
||||
**Frontend Layer:** A React application that handles the UI components, state management with Zustand, and data fetching with React Query (TanStack Query).
|
||||
|
||||
**Backend Layer:** A Python FastAPI server that provides the REST API, runs the TTS engine (Qwen3-TTS), manages the SQLite database, and handles audio processing.
|
||||
**Backend Layer:** A Python FastAPI server that hosts the REST API, runs a pluggable registry of TTS and STT engines, manages the SQLite database, and handles audio processing.
|
||||
|
||||
These two layers communicate via HTTP, with the frontend making API requests to the backend.
|
||||
These two layers communicate via HTTP on `localhost:17493`, with the frontend making API requests to the backend. In production the backend is compiled with PyInstaller and launched as a Tauri sidecar; in development it's run manually via `uvicorn`.
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
@@ -29,43 +29,33 @@ These two layers communicate via HTTP, with the frontend making API requests to
|
||||
<Files>
|
||||
<Folder name="app/src" defaultOpen>
|
||||
<Folder name="components">
|
||||
<File name="profiles/" />
|
||||
<File name="generation/" />
|
||||
<File name="stories/" />
|
||||
<File name="shared/" />
|
||||
<File name="Profiles/" />
|
||||
<File name="Generation/" />
|
||||
<File name="Stories/" />
|
||||
<File name="ServerSettings/" />
|
||||
</Folder>
|
||||
<Folder name="lib">
|
||||
<File name="api/" />
|
||||
<File name="constants/" />
|
||||
<File name="hooks/" />
|
||||
<File name="utils/" />
|
||||
</Folder>
|
||||
<Folder name="hooks" />
|
||||
<Folder name="stores" />
|
||||
</Folder>
|
||||
</Files>
|
||||
|
||||
### State Management
|
||||
|
||||
```typescript
|
||||
// Example: Profile store
|
||||
const useProfileStore = create((set) => ({
|
||||
profiles: [],
|
||||
selectedProfile: null,
|
||||
setProfiles: (profiles) => set({ profiles }),
|
||||
selectProfile: (id) => set({ selectedProfile: id })
|
||||
}))
|
||||
```
|
||||
|
||||
## Backend Architecture
|
||||
|
||||
### Tech Stack
|
||||
|
||||
- **Framework**: FastAPI (Python 3.11+)
|
||||
- **TTS Model**: Qwen3-TTS
|
||||
- **Transcription**: Whisper
|
||||
- **Database**: SQLite
|
||||
- **Audio**: librosa, soundfile
|
||||
- **TTS Engines**: Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro
|
||||
- **Transcription**: Whisper (PyTorch or MLX-Whisper)
|
||||
- **Inference Backends**: MLX (Apple Silicon), PyTorch (CUDA / ROCm / XPU / DirectML / CPU)
|
||||
- **Database**: SQLite via SQLAlchemy
|
||||
- **Audio**: librosa, soundfile, Pedalboard
|
||||
|
||||
### API Structure
|
||||
### Layout
|
||||
|
||||
<Files>
|
||||
<Folder name="backend" defaultOpen>
|
||||
@@ -74,21 +64,31 @@ const useProfileStore = create((set) => ({
|
||||
<File name="config.py" />
|
||||
<File name="models.py" />
|
||||
<File name="server.py" />
|
||||
<File name="build_binary.py" />
|
||||
<Folder name="routes">
|
||||
<File name="profiles.py" />
|
||||
<File name="generate.py" />
|
||||
<File name="history.py" />
|
||||
<File name="..." />
|
||||
<File name="models.py" />
|
||||
<File name="channels.py" />
|
||||
</Folder>
|
||||
<Folder name="services">
|
||||
<File name="generation.py" />
|
||||
<File name="task_queue.py" />
|
||||
<File name="..." />
|
||||
<File name="profiles.py" />
|
||||
<File name="channels.py" />
|
||||
</Folder>
|
||||
<Folder name="backends">
|
||||
<File name="__init__.py" />
|
||||
<File name="base.py" />
|
||||
<File name="..." />
|
||||
<File name="pytorch_backend.py" />
|
||||
<File name="mlx_backend.py" />
|
||||
<File name="qwen_custom_voice_backend.py" />
|
||||
<File name="luxtts_backend.py" />
|
||||
<File name="chatterbox_backend.py" />
|
||||
<File name="chatterbox_turbo_backend.py" />
|
||||
<File name="hume_backend.py" />
|
||||
<File name="kokoro_backend.py" />
|
||||
</Folder>
|
||||
<Folder name="database">
|
||||
<File name="models.py" />
|
||||
@@ -97,49 +97,75 @@ const useProfileStore = create((set) => ({
|
||||
<Folder name="utils">
|
||||
<File name="audio.py" />
|
||||
<File name="effects.py" />
|
||||
<File name="..." />
|
||||
</Folder>
|
||||
</Folder>
|
||||
</Files>
|
||||
|
||||
### Request Flow
|
||||
|
||||
HTTP request → **routes/** (validate input, parse params) → **services/** (business logic, orchestration) → **backends/** (TTS/STT inference) → **utils/** (audio processing)
|
||||
An HTTP request enters a **route handler**, which validates input and delegates to a **service** function. The service calls into the appropriate **engine backend** via the registry, which runs the actual inference. Audio post-processing runs through **utils** (trim, resample, effects).
|
||||
|
||||
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
|
||||
Route handlers are intentionally thin — they validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
|
||||
|
||||
### Multi-Engine Registry
|
||||
|
||||
The backend is designed so that adding a new TTS engine only requires touching the `backends/` directory and the central registry. There is no per-engine branching in routes or services.
|
||||
|
||||
- **`TTSBackend` Protocol** (`backends/__init__.py`) — defines the contract every engine implements: `load_model`, `create_voice_prompt`, `combine_voice_prompts`, `generate`, `unload_model`, `is_loaded`, `_get_model_path`.
|
||||
- **`ModelConfig` dataclass** — central metadata record for each model variant: `model_name`, `display_name`, `engine`, `hf_repo_id`, `size_mb`, `needs_trim`, `languages`, `supports_instruct`, etc.
|
||||
- **`TTS_ENGINES` dict** — maps engine name (`"qwen"`, `"kokoro"`, etc.) to display name.
|
||||
- **`get_tts_backend_for_engine(engine)`** — thread-safe factory that lazily instantiates and caches the backend for an engine using double-checked locking.
|
||||
|
||||
Shipped engines:
|
||||
|
||||
| Engine key | Display name | Profile type |
|
||||
|------------|--------------|--------------|
|
||||
| `qwen` | Qwen TTS | Cloned |
|
||||
| `qwen_custom_voice` | Qwen CustomVoice | Preset |
|
||||
| `luxtts` | LuxTTS | Cloned |
|
||||
| `chatterbox` | Chatterbox TTS | Cloned |
|
||||
| `chatterbox_turbo` | Chatterbox Turbo | Cloned |
|
||||
| `tada` | TADA | Cloned |
|
||||
| `kokoro` | Kokoro | Preset |
|
||||
|
||||
See [TTS Engines](/developer/tts-engines) for the full contract and integration phases, and [PROJECT_STATUS.md](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for candidates under evaluation.
|
||||
|
||||
### Key Modules
|
||||
|
||||
- **app.py** — FastAPI app factory, CORS, lifecycle events
|
||||
- **main.py** — Entry point (imports app, runs uvicorn)
|
||||
- **server.py** — Tauri sidecar launcher, parent-pid watchdog
|
||||
- **services/generation.py** — Single function handling all generation modes (generate, retry, regenerate)
|
||||
- **services/task_queue.py** — Serial generation queue for GPU inference
|
||||
- **backends/__init__.py** — Protocol definitions and backend factory
|
||||
- **backends/base.py** — Shared utilities across all engine implementations
|
||||
- **`app.py`** — FastAPI app factory, CORS, lifecycle events
|
||||
- **`main.py`** — Entry point (imports app, runs uvicorn)
|
||||
- **`server.py`** — Tauri sidecar launcher, parent-pid watchdog, frozen-build environment setup
|
||||
- **`services/generation.py`** — Single function handling all generation modes (generate, retry, regenerate)
|
||||
- **`services/task_queue.py`** — Serial generation queue for GPU inference
|
||||
- **`backends/__init__.py`** — Protocol definitions, `ModelConfig` registry, and engine factory
|
||||
- **`backends/base.py`** — Shared utilities across all engine implementations (device selection, progress tracking, output trimming)
|
||||
|
||||
### Backend Selection
|
||||
### Inference Backend Selection
|
||||
|
||||
The server detects the best inference backend at startup:
|
||||
The server detects the best inference backend at startup and uses it for all engines that support it:
|
||||
|
||||
| Platform | Backend | Acceleration |
|
||||
|----------|---------|-------------|
|
||||
|----------|---------|--------------|
|
||||
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
|
||||
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
|
||||
| Windows / Linux (NVIDIA) | PyTorch | CUDA (cu128) |
|
||||
| Linux (AMD) | PyTorch | ROCm |
|
||||
| Intel Arc | PyTorch | IPEX / XPU |
|
||||
| Windows (any GPU) | PyTorch | DirectML |
|
||||
| Windows / Linux (Intel Arc) | PyTorch | XPU (IPEX) |
|
||||
| Windows (other GPU) | PyTorch | DirectML |
|
||||
| Any | PyTorch | CPU fallback |
|
||||
|
||||
See [GPU Acceleration](/overview/gpu-acceleration) for platform-specific notes and manual overrides.
|
||||
|
||||
### Data Model
|
||||
|
||||
The database uses three main tables:
|
||||
Core tables (see `backend/database/models.py`):
|
||||
|
||||
**Profile Table:** Stores voice profiles with fields for id, name, and language.
|
||||
- **`profiles`** — Voice profiles with `voice_type` discriminator (`cloned` | `preset` | `designed`), `preset_engine`, `preset_voice_id`, and `default_engine`.
|
||||
- **`profile_samples`** — Reference audio clips + transcripts for cloned profiles. Empty for preset profiles.
|
||||
- **`generations`** — Generated audio with text, engine, model, language, seed, and duration.
|
||||
- **`generation_versions`** — Processed variants of a generation with different effects chains applied.
|
||||
- **`audio_channels`** + **`channel_device_mappings`** + **`profile_channel_mappings`** — Multi-output routing.
|
||||
|
||||
**Sample Table:** Stores audio samples linked to profiles via profile_id, with fields for audio_path and duration.
|
||||
|
||||
**Generation Table:** Stores generated audio with fields for id, profile_id, text, and audio_path.
|
||||
See [Voice Profiles](/developer/voice-profiles) and [Effects Pipeline](/developer/effects-pipeline) for details.
|
||||
|
||||
## Desktop App (Tauri)
|
||||
|
||||
@@ -148,6 +174,7 @@ The database uses three main tables:
|
||||
<Files>
|
||||
<Folder name="tauri/src-tauri" defaultOpen>
|
||||
<File name="Cargo.toml" />
|
||||
<File name="tauri.conf.json" />
|
||||
<File name="src/" />
|
||||
<Folder name="binaries" />
|
||||
</Folder>
|
||||
@@ -158,82 +185,74 @@ The database uses three main tables:
|
||||
- Launch Python backend as sidecar process
|
||||
- Native file dialogs
|
||||
- System tray integration
|
||||
- Auto-updates
|
||||
- OS-specific features
|
||||
- Auto-updates (Tauri updater + custom CUDA backend swap)
|
||||
- Parent-PID watchdog so the backend exits if the app crashes
|
||||
|
||||
## Build Process
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Frontend (Vite dev server)
|
||||
cd app && bun run dev
|
||||
|
||||
# Backend (manual start)
|
||||
cd backend && uvicorn main:app --reload
|
||||
|
||||
# Desktop app (connects to manual backend)
|
||||
bun run dev
|
||||
just dev # Starts backend + Tauri app
|
||||
just dev-web # Starts backend + web app (no Tauri)
|
||||
just dev-backend # Backend only
|
||||
just dev-frontend # Tauri app only (backend must be running)
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build everything (server binary + Tauri app)
|
||||
bun run build
|
||||
|
||||
# Or build separately:
|
||||
# 1. Build server binary (PyInstaller)
|
||||
bun run build:server
|
||||
|
||||
# 2. Build Tauri app (includes server)
|
||||
cd tauri && bun run tauri build
|
||||
just build # CPU server binary + Tauri installer
|
||||
just build-local # CPU + CUDA binaries + Tauri installer (Windows)
|
||||
just build-server # Server binary only
|
||||
just build-tauri # Tauri app only
|
||||
```
|
||||
|
||||
See [Building](/developer/building) for what PyInstaller does and how the CUDA binary is split and packaged separately.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Generation Flow
|
||||
|
||||
When a user generates speech, the data flows through the following stages:
|
||||
|
||||
1. **User Input** - User enters text in a React component
|
||||
2. **State Update** - Text is stored in Zustand state
|
||||
3. **API Request** - React Query mutation triggers an API call via fetch
|
||||
4. **Backend Processing** - FastAPI endpoint receives the request
|
||||
5. **TTS Generation** - Qwen3-TTS model generates the audio
|
||||
6. **Storage** - Audio file is saved to disk and a database record is created
|
||||
7. **Response** - Backend returns the audio URL
|
||||
8. **Cache Update** - React Query updates its cache with the response
|
||||
9. **UI Update** - Component re-renders with new data
|
||||
10. **Playback** - User can play the generated audio
|
||||
1. **User Input** — text entered in a React component, engine + profile selected
|
||||
2. **State Update** — Zustand generation form store records the request
|
||||
3. **API Request** — React Query mutation hits `POST /generate`
|
||||
4. **Route** — `routes/generate.py` validates input, dispatches to `services/generation.py`
|
||||
5. **Voice Prompt** — the service creates or retrieves a cached voice prompt via the engine's backend
|
||||
6. **Queue** — `services/task_queue.py` serializes generation to avoid GPU contention
|
||||
7. **Inference** — the engine backend runs `generate()` and returns audio + sample rate
|
||||
8. **Post-process** — optional trim (for engines that need it), effects chain applied per generation version
|
||||
9. **Storage** — audio written to the generations directory, metadata saved to SQLite
|
||||
10. **Response** — backend returns the generation record; frontend updates React Query cache and plays audio
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Code splitting** - Lazy load routes
|
||||
- **Memoization** - React.memo for heavy components
|
||||
- **Virtual scrolling** - For large lists
|
||||
- **Debouncing** - Search and input handling
|
||||
- **Code splitting** — lazy-load routes
|
||||
- **Memoization** — `React.memo` for heavy components
|
||||
- **Virtual scrolling** — for large lists
|
||||
- **Debouncing** — search and input handling
|
||||
|
||||
### Backend
|
||||
|
||||
- **Async operations** - All I/O is async
|
||||
- **Model caching** - Keep TTS model in memory
|
||||
- **Voice prompt caching** - Reuse embeddings
|
||||
- **Connection pooling** - Database connections
|
||||
- **Async I/O** — all I/O is async; inference runs in `asyncio.to_thread`
|
||||
- **Serial task queue** — avoids multiple engines fighting for the GPU
|
||||
- **Voice prompt caching** — engine-specific, keyed by audio hash + reference text
|
||||
- **Model pinning** — only one model per engine loaded at a time; switching unloads the previous one
|
||||
- **Per-engine backend cache** — engines are only instantiated once per process
|
||||
|
||||
## Security
|
||||
|
||||
### Current
|
||||
|
||||
- Local-only by default
|
||||
- Local-only by default (bound to `127.0.0.1:17493`)
|
||||
- No authentication (localhost trust)
|
||||
- File system sandboxing via Tauri
|
||||
|
||||
### Planned
|
||||
|
||||
- API key authentication
|
||||
- API key authentication for remote mode
|
||||
- User accounts
|
||||
- Rate limiting
|
||||
- HTTPS support
|
||||
@@ -248,17 +267,20 @@ When a user generates speech, the data flows through the following stages:
|
||||
|
||||
### Remote Mode
|
||||
|
||||
- Backend on separate machine
|
||||
- Frontend connects via HTTP
|
||||
- Shared infrastructure possible
|
||||
- Backend on a separate machine (Docker or bare host)
|
||||
- Frontend (desktop or web) connects over HTTP
|
||||
- See [Remote Mode](/overview/remote-mode) and [Docker](/overview/docker)
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Development Setup" href="/development/setup">
|
||||
<Card title="Development Setup" href="/developer/setup">
|
||||
Set up your dev environment
|
||||
</Card>
|
||||
<Card title="Contributing" href="/development/contributing">
|
||||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||||
How to add a new engine
|
||||
</Card>
|
||||
<Card title="Contributing" href="/developer/contributing">
|
||||
Contribute to Voicebox
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -145,64 +145,55 @@ The updater only works in production Tauri builds. It doesn't run during `just d
|
||||
|
||||
## CUDA Backend Updates
|
||||
|
||||
The CUDA-enabled backend is distributed separately from the main app due to its large size (~2.43 GB). Unlike the Tauri auto-updater, this uses a custom download system built into the Python backend.
|
||||
The CUDA-enabled backend is distributed separately from the main app because bundling CUDA would bloat the installer by several gigabytes for users who don't have an NVIDIA GPU. Unlike the Tauri auto-updater, the CUDA backend uses a custom download system built into the Python server.
|
||||
|
||||
**Size comparison:**
|
||||
- Standard app bundle: ~410 MB
|
||||
- CUDA backend binary: ~2.43 GB (6× larger)
|
||||
**Size comparison (approximate):**
|
||||
- Standard CPU bundle (in the installer): ~200–400 MB
|
||||
- CUDA server core: ~945 MB (versioned with each Voicebox release)
|
||||
- CUDA libs (NVIDIA runtime DLLs): ~1.7 GB (versioned independently, cached across upgrades)
|
||||
|
||||
### Why Split?
|
||||
### Two-archive split
|
||||
|
||||
GitHub Releases has file size limits, and the CUDA-enabled `voicebox-server` binary is too large to include in the main Tauri bundle. Instead:
|
||||
Since v0.4, the CUDA binary is packaged as **two archives** instead of one:
|
||||
|
||||
- **Standard release**: Includes CPU-only backend (~50MB)
|
||||
- **CUDA release**: Split into multiple parts and downloaded on-demand by users who need GPU acceleration
|
||||
- **Server core** (`voicebox-server-cuda.tar.gz`) — the Python server + PyTorch code, changes every release.
|
||||
- **CUDA libs** (`cuda-libs-cu128-v1.tar.gz`) — the heavy NVIDIA CUDA/cuDNN DLLs, only re-downloaded when the CUDA toolkit major version changes.
|
||||
|
||||
This means most Voicebox upgrades only re-download the ~945 MB server core, not the full ~2.5 GB bundle.
|
||||
|
||||
### Download Process
|
||||
|
||||
When a user clicks "Enable CUDA" in the settings:
|
||||
When a user clicks "Install CUDA backend" in Settings → GPU:
|
||||
|
||||
1. **Manifest Fetch** - Backend fetches `{version}/voicebox-server-cuda.manifest` from GitHub Releases
|
||||
2. **Part Download** - Downloads each split part sequentially (e.g., `voicebox-server-cuda.part1`, `.part2`, etc.)
|
||||
3. **Assembly** - Concatenates parts into a single binary
|
||||
4. **Verification** - SHA-256 checksum verification (optional, if `.sha256` file exists)
|
||||
5. **Placement** - Binary moved to `{data_dir}/backends/voicebox-server-cuda.exe`
|
||||
6. **Restart** - Backend must restart to use the CUDA binary
|
||||
1. **Server-core archive** — Downloaded from GitHub Releases and extracted.
|
||||
2. **CUDA libs archive** — Downloaded separately (or reused if the installed version still matches).
|
||||
3. **Verification** — SHA-256 checksum verification for integrity.
|
||||
4. **Placement** — Extracted into `{data_dir}/backends/cuda/`.
|
||||
5. **Restart** — The Voicebox server restarts and swaps in the CUDA backend.
|
||||
|
||||
### Auto-Update on Startup
|
||||
|
||||
On server startup, `check_and_update_cuda_binary()` compares the installed CUDA binary version with the app version:
|
||||
|
||||
```python
|
||||
# backend/services/cuda.py
|
||||
cuda_version = get_cuda_binary_version() # runs `voicebox-server-cuda --version`
|
||||
current_version = __version__
|
||||
|
||||
if cuda_version != current_version:
|
||||
await download_cuda_binary() # Auto-download in background
|
||||
```
|
||||
|
||||
If versions mismatch, the backend automatically downloads the matching CUDA binary version without user intervention.
|
||||
On startup, the backend compares the installed CUDA server-core version with the current app version. If they differ, the core archive is pulled in the background. If the libs version pinned by the new release also differs (rare — e.g. on a cu126 → cu128 bump), the user is prompted to confirm the larger download.
|
||||
|
||||
### Storage Location
|
||||
|
||||
Downloaded CUDA binaries are stored in the app's data directory:
|
||||
Downloaded CUDA binaries live in the app's data directory:
|
||||
|
||||
```
|
||||
{data_dir}/
|
||||
backends/
|
||||
voicebox-server-cuda.exe # Windows
|
||||
voicebox-server-cuda # macOS/Linux
|
||||
{data_dir}/backends/cuda/
|
||||
voicebox-server-cuda.exe # Windows
|
||||
voicebox-server-cuda # macOS/Linux
|
||||
<NVIDIA CUDA runtime DLLs>
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/backend/cuda-status` | GET | Check if CUDA binary available/active |
|
||||
| `/backend/download-cuda` | POST | Start download |
|
||||
| `/backend/cuda-status` | GET | Check if the CUDA backend is available/active and which versions are installed |
|
||||
| `/backend/download-cuda` | POST | Trigger server-core + libs download |
|
||||
| `/backend/cuda-progress` | GET | SSE stream of download progress |
|
||||
| `/backend/cuda` | DELETE | Remove downloaded binary |
|
||||
| `/backend/cuda` | DELETE | Remove the downloaded CUDA backend |
|
||||
|
||||
### Progress Tracking
|
||||
|
||||
@@ -212,15 +203,16 @@ Downloads report progress via Server-Sent Events (SSE):
|
||||
GET /backend/cuda-progress
|
||||
|
||||
event: progress
|
||||
data: {"current": 52428800, "total": 104857600, "filename": "Downloading CUDA backend (2/4)", "status": "downloading"}
|
||||
data: {"current": 52428800, "total": 945000000, "filename": "voicebox-server-cuda.tar.gz", "status": "downloading"}
|
||||
```
|
||||
|
||||
The frontend subscribes to this endpoint to show real-time download progress in the UI.
|
||||
The frontend subscribes to this endpoint to show real-time progress, including which archive (server core vs libs) is currently downloading.
|
||||
|
||||
### Release Artifacts
|
||||
|
||||
For each release, these CUDA-related files are uploaded to GitHub:
|
||||
For each CUDA-capable release, these files are uploaded to GitHub:
|
||||
|
||||
- `voicebox-server-cuda.manifest` - List of split part filenames
|
||||
- `voicebox-server-cuda.part1` through `voicebox-server-cuda.partN` - Binary chunks
|
||||
- `voicebox-server-cuda.sha256` - SHA-256 checksum for integrity verification
|
||||
- `voicebox-server-cuda.tar.gz` — server-core archive
|
||||
- `voicebox-server-cuda.tar.gz.sha256` — checksum
|
||||
- `cuda-libs-cu128-v1.tar.gz` — CUDA runtime libs (only when the libs version bumps)
|
||||
- `cuda-libs-cu128-v1.tar.gz.sha256` — checksum
|
||||
|
||||
@@ -17,9 +17,10 @@ Thank you for your interest in contributing to Voicebox! This guide will help yo
|
||||
Before you start contributing, make sure you have:
|
||||
|
||||
1. **Read the documentation** to understand how Voicebox works
|
||||
2. **Set up your development environment** - see [Development Setup](/development/setup)
|
||||
2. **Set up your development environment** — see [Development Setup](/developer/setup)
|
||||
3. **Explored the codebase** to understand the project structure
|
||||
4. **Checked existing issues** to see if someone else is working on something similar
|
||||
4. **Checked [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md)** — the living engineering roadmap that tracks prioritized tasks (Tier 1 → 3), architectural bottlenecks, and candidate TTS engines under evaluation (including why some are backlogged)
|
||||
5. **Checked existing issues** to see if someone else is working on something similar
|
||||
|
||||
## Ways to Contribute
|
||||
|
||||
@@ -173,10 +174,15 @@ When creating a pull request:
|
||||
<File name="stores/" />
|
||||
</Folder>
|
||||
<Folder name="backend">
|
||||
<File name="app.py" />
|
||||
<File name="main.py" />
|
||||
<File name="tts.py" />
|
||||
<File name="database.py" />
|
||||
<File name="server.py" />
|
||||
<File name="models.py" />
|
||||
<Folder name="routes" />
|
||||
<Folder name="services" />
|
||||
<Folder name="backends" />
|
||||
<Folder name="database" />
|
||||
<Folder name="utils" />
|
||||
</Folder>
|
||||
<Folder name="tauri">
|
||||
<File name="src-tauri/" />
|
||||
@@ -197,9 +203,10 @@ When creating a pull request:
|
||||
|
||||
### New Features
|
||||
|
||||
- Check the [roadmap](https://github.com/jamiepine/voicebox#roadmap) for planned features
|
||||
- Check [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) and the [roadmap](https://github.com/jamiepine/voicebox#roadmap) before proposing work — the status doc lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
|
||||
- Discuss major features in an issue first
|
||||
- Keep features focused and well-scoped
|
||||
- Adding a new TTS engine? See [TTS Engines](/developer/tts-engines) for the phased workflow
|
||||
|
||||
### Documentation
|
||||
|
||||
@@ -253,7 +260,7 @@ When adding new API endpoints:
|
||||
|
||||
<Step title="Regenerate Client">
|
||||
```bash
|
||||
bun run generate:api
|
||||
just generate-api
|
||||
```
|
||||
|
||||
This updates the TypeScript client with type-safe bindings.
|
||||
@@ -263,7 +270,7 @@ When adding new API endpoints:
|
||||
The API documentation is automatically generated from the OpenAPI schema. Ensure your endpoint has proper docstrings and type hints, then regenerate the docs:
|
||||
|
||||
```bash
|
||||
bun run generate:api
|
||||
just generate-api
|
||||
```
|
||||
</Step>
|
||||
</Steps>
|
||||
@@ -324,8 +331,9 @@ By contributing, you agree that your contributions will be licensed under the MI
|
||||
If you have questions:
|
||||
|
||||
1. Check the [documentation](/overview/introduction)
|
||||
2. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
|
||||
3. Open a new issue or discussion
|
||||
4. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
|
||||
2. Read [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) for current engineering priorities
|
||||
3. Search [existing issues](https://github.com/jamiepine/voicebox/issues)
|
||||
4. Open a new issue or discussion
|
||||
5. See [CONTRIBUTING.md](https://github.com/jamiepine/voicebox/blob/main/CONTRIBUTING.md) in the repo
|
||||
|
||||
Thank you for contributing to Voicebox! 🎉
|
||||
|
||||
@@ -15,17 +15,24 @@ The history module tracks all generated audio, providing a searchable record of
|
||||
class Generation(Base):
|
||||
__tablename__ = "generations"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
|
||||
text = Column(Text, nullable=False)
|
||||
language = Column(String, default="en")
|
||||
audio_path = Column(String, nullable=False)
|
||||
duration = Column(Float, nullable=False)
|
||||
audio_path = Column(String, nullable=True)
|
||||
duration = Column(Float, nullable=True)
|
||||
seed = Column(Integer)
|
||||
instruct = Column(Text)
|
||||
created_at = Column(DateTime)
|
||||
engine = Column(String, default="qwen")
|
||||
model_size = Column(String, nullable=True)
|
||||
status = Column(String, default="completed") # pending | completed | failed
|
||||
error = Column(Text, nullable=True)
|
||||
is_favorited = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
```
|
||||
|
||||
Each generation can also have multiple **generation versions** — processed variants with different effects chains applied. The original (`clean`) version plus any number of processed versions live in a separate `generation_versions` table. See [Effects Pipeline](/developer/effects-pipeline).
|
||||
|
||||
## File Storage
|
||||
|
||||
Generated audio is stored in:
|
||||
@@ -230,7 +237,12 @@ GET /history?profile_id=uuid&search=hello&limit=50&offset=0
|
||||
"duration": 1.5,
|
||||
"seed": 42,
|
||||
"instruct": null,
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
"engine": "qwen",
|
||||
"model_size": "1.7B",
|
||||
"status": "completed",
|
||||
"error": null,
|
||||
"is_favorited": false,
|
||||
"created_at": "2026-04-18T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"total": 150
|
||||
|
||||
@@ -1,341 +1,199 @@
|
||||
---
|
||||
title: "Model Management"
|
||||
description: "How model downloading, loading, and status tracking works in Voicebox"
|
||||
description: "How model downloading, loading, and status tracking works across all engines"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox manages two types of models:
|
||||
Voicebox manages two categories of models:
|
||||
|
||||
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
|
||||
**TTS Models** — Seven engines covering zero-shot cloning and preset voices. Each engine may have one or more size variants.
|
||||
|
||||
**ASR Models:** Whisper for transcription (tiny through large).
|
||||
**ASR Models** — Whisper for transcription. Five sizes, plus MLX-Whisper on Apple Silicon for ~8× faster transcription.
|
||||
|
||||
Models are downloaded from HuggingFace Hub on first use and cached locally.
|
||||
Every model is described by a `ModelConfig` entry in `backend/backends/__init__.py`. Models are downloaded from HuggingFace Hub on first use and cached in the platform-standard HF cache.
|
||||
|
||||
## Available Models
|
||||
## Available TTS Models
|
||||
|
||||
### TTS Models
|
||||
| Model | Engine | HuggingFace Repo | Size | VRAM | Languages |
|
||||
|-------|--------|------------------|------|------|-----------|
|
||||
| **Qwen TTS 1.7B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 3.5 GB | ~6 GB | 10 |
|
||||
| **Qwen TTS 0.6B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | 1.2 GB | ~2 GB | 10 |
|
||||
| **Qwen CustomVoice 1.7B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 3.5 GB | ~6 GB | 10 |
|
||||
| **Qwen CustomVoice 0.6B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | 1.2 GB | ~2 GB | 10 |
|
||||
| **LuxTTS** | `luxtts` | `YatharthS/LuxTTS` | 300 MB | ~1 GB | English |
|
||||
| **Chatterbox Multilingual** | `chatterbox` | `ResembleAI/chatterbox` | 3.2 GB | ~3 GB | 23 |
|
||||
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
|
||||
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
|
||||
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
|
||||
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 8 |
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
|
||||
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
|
||||
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
|
||||
|
||||
### Whisper Models
|
||||
## Available Whisper Models
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
|
||||
| base | `openai/whisper-base` | ~300MB | ~1GB |
|
||||
| small | `openai/whisper-small` | ~500MB | ~2GB |
|
||||
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
|
||||
| large | `openai/whisper-large` | ~3GB | ~10GB |
|
||||
| Model | HuggingFace Repo | Size |
|
||||
|-------|------------------|------|
|
||||
| **Whisper Base** | `openai/whisper-base` | ~300 MB |
|
||||
| **Whisper Small** | `openai/whisper-small` | ~500 MB |
|
||||
| **Whisper Medium** | `openai/whisper-medium` | ~1.5 GB |
|
||||
| **Whisper Large** | `openai/whisper-large-v3` | ~3 GB |
|
||||
| **Whisper Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB |
|
||||
|
||||
On Apple Silicon, MLX-Whisper is preferred automatically — see [Transcription](/developer/transcription).
|
||||
|
||||
## Model Storage
|
||||
|
||||
Models are cached in the HuggingFace cache directory:
|
||||
Models live in the platform HuggingFace cache:
|
||||
|
||||
<Files>
|
||||
<Folder name="~/.cache/huggingface/hub" defaultOpen>
|
||||
<File name="models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/" />
|
||||
<File name="models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/" />
|
||||
<File name="models--openai--whisper-base/" />
|
||||
</Folder>
|
||||
</Files>
|
||||
| Platform | Path |
|
||||
|----------|------|
|
||||
| macOS | `~/.cache/huggingface/hub/` |
|
||||
| Linux | `~/.cache/huggingface/hub/` |
|
||||
| Windows | `%USERPROFILE%\.cache\huggingface\hub\` |
|
||||
| Docker | `/home/voicebox/.cache/huggingface/hub` (volume-mounted) |
|
||||
|
||||
Set `VOICEBOX_MODELS_DIR` to override.
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
### Progress Manager
|
||||
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
|
||||
|
||||
Tracks download progress across all models:
|
||||
**`ProgressManager`** (`backend/utils/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
|
||||
|
||||
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
|
||||
|
||||
**SSE endpoint** — `GET /models/progress/{model_name}` streams updates until `status` is `complete` or `error`.
|
||||
|
||||
```python
|
||||
class ProgressManager:
|
||||
def __init__(self):
|
||||
self._progress = {} # model_name -> progress_info
|
||||
|
||||
def update_progress(
|
||||
self,
|
||||
model_name: str,
|
||||
current: int,
|
||||
total: int,
|
||||
filename: str,
|
||||
status: str,
|
||||
):
|
||||
self._progress[model_name] = {
|
||||
"current": current,
|
||||
"total": total,
|
||||
"filename": filename,
|
||||
"status": status, # downloading, complete, error
|
||||
"updated_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def get_progress(self, model_name: str) -> Optional[dict]:
|
||||
return self._progress.get(model_name)
|
||||
```
|
||||
|
||||
### HuggingFace Progress Callback
|
||||
|
||||
Hooks into HuggingFace's download system:
|
||||
|
||||
```python
|
||||
class HFProgressTracker:
|
||||
def __init__(self, callback):
|
||||
self.callback = callback
|
||||
|
||||
@contextmanager
|
||||
def patch_download(self):
|
||||
"""Context manager to intercept HF downloads."""
|
||||
original_download = hf_hub_download
|
||||
|
||||
def patched_download(*args, **kwargs):
|
||||
# Intercept progress
|
||||
result = original_download(*args, **kwargs)
|
||||
self.callback(progress_info)
|
||||
return result
|
||||
|
||||
# Apply patch
|
||||
with patch('huggingface_hub.hf_hub_download', patched_download):
|
||||
yield
|
||||
```
|
||||
|
||||
### Server-Sent Events (SSE)
|
||||
|
||||
Progress is streamed to the frontend:
|
||||
|
||||
```python
|
||||
@app.get("/models/progress/{model_name}")
|
||||
async def get_model_progress(model_name: str):
|
||||
async def event_generator():
|
||||
while True:
|
||||
progress = progress_manager.get_progress(model_name)
|
||||
if progress:
|
||||
yield f"data: {json.dumps(progress)}\n\n"
|
||||
|
||||
if progress and progress["status"] in ["complete", "error"]:
|
||||
break
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream"
|
||||
)
|
||||
```
|
||||
|
||||
## Task Manager
|
||||
|
||||
Tracks active downloads and generations:
|
||||
|
||||
```python
|
||||
class TaskManager:
|
||||
def __init__(self):
|
||||
self._active_downloads = {}
|
||||
self._active_generations = {}
|
||||
|
||||
def start_download(self, model_name: str):
|
||||
self._active_downloads[model_name] = {
|
||||
"status": "downloading",
|
||||
"started_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def complete_download(self, model_name: str):
|
||||
if model_name in self._active_downloads:
|
||||
del self._active_downloads[model_name]
|
||||
|
||||
def get_active_tasks(self) -> dict:
|
||||
return {
|
||||
"downloads": list(self._active_downloads.values()),
|
||||
"generations": list(self._active_generations.values()),
|
||||
}
|
||||
# Frontend
|
||||
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||
eventSource.onmessage = (event) => {
|
||||
const { current, total, status } = JSON.parse(event.data);
|
||||
updateProgressBar(current / total);
|
||||
if (status === "complete") eventSource.close();
|
||||
};
|
||||
```
|
||||
|
||||
## Model Status
|
||||
|
||||
Check which models are downloaded and loaded:
|
||||
|
||||
```python
|
||||
@app.get("/models/status")
|
||||
async def get_model_status() -> ModelStatusListResponse:
|
||||
models = []
|
||||
|
||||
# Check TTS models
|
||||
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
loaded = tts_model._current_model_size == size
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"qwen-tts-{size}",
|
||||
display_name=f"Qwen3-TTS {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=loaded,
|
||||
))
|
||||
|
||||
# Check Whisper models
|
||||
for size in ["tiny", "base", "small", "medium", "large"]:
|
||||
hf_id = f"openai/whisper-{size}"
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"whisper-{size}",
|
||||
display_name=f"Whisper {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=False, # Whisper is loaded on-demand
|
||||
))
|
||||
|
||||
return ModelStatusListResponse(models=models)
|
||||
```
|
||||
|
||||
## Manual Model Operations
|
||||
|
||||
### Load Model
|
||||
|
||||
```python
|
||||
@app.post("/models/load")
|
||||
async def load_model(model_size: str = "1.7B"):
|
||||
tts_model = get_tts_model()
|
||||
await tts_model.load_model_async(model_size)
|
||||
return {"status": "loaded", "model_size": model_size}
|
||||
```
|
||||
|
||||
### Unload Model
|
||||
|
||||
```python
|
||||
@app.post("/models/unload")
|
||||
async def unload_model():
|
||||
tts_model = get_tts_model()
|
||||
tts_model.unload_model()
|
||||
return {"status": "unloaded"}
|
||||
```
|
||||
|
||||
### Trigger Download
|
||||
|
||||
```python
|
||||
@app.post("/models/download")
|
||||
async def trigger_model_download(request: ModelDownloadRequest):
|
||||
# This triggers the download in background
|
||||
# Progress is tracked via /models/progress/{model_name}
|
||||
|
||||
if request.model_name.startswith("qwen-tts"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_tts_model(size))
|
||||
elif request.model_name.startswith("whisper"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_whisper_model(size))
|
||||
|
||||
return {"status": "downloading"}
|
||||
```
|
||||
|
||||
### Delete Model
|
||||
|
||||
```python
|
||||
@app.delete("/models/{model_name}")
|
||||
async def delete_model(model_name: str):
|
||||
# Find and delete from HuggingFace cache
|
||||
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
|
||||
|
||||
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
|
||||
for model_dir in model_dirs:
|
||||
shutil.rmtree(model_dir)
|
||||
|
||||
return {"status": "deleted"}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/models/status` | Get status of all models |
|
||||
| POST | `/models/load` | Load TTS model |
|
||||
| POST | `/models/unload` | Unload TTS model |
|
||||
| POST | `/models/download` | Trigger model download |
|
||||
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||
| DELETE | `/models/{name}` | Delete downloaded model |
|
||||
| GET | `/tasks/active` | Get active downloads/generations |
|
||||
|
||||
## Response Schemas
|
||||
|
||||
### ModelStatus
|
||||
`GET /models/status` returns every registered model's current state:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen3-TTS 1.7B",
|
||||
"downloaded": true,
|
||||
"size_mb": 3400,
|
||||
"loaded": true
|
||||
}
|
||||
```
|
||||
|
||||
### ActiveTasksResponse
|
||||
|
||||
```json
|
||||
{
|
||||
"downloads": [
|
||||
"models": [
|
||||
{
|
||||
"model_name": "whisper-medium",
|
||||
"status": "downloading",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"generations": [
|
||||
{
|
||||
"task_id": "uuid",
|
||||
"profile_id": "uuid",
|
||||
"text_preview": "Hello world...",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen TTS 1.7B",
|
||||
"engine": "qwen",
|
||||
"downloaded": true,
|
||||
"size_mb": 3500,
|
||||
"loaded": true
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend Integration
|
||||
The handler iterates `get_all_model_configs()` and calls `check_model_loaded(config)` for each entry, so new engines appear automatically once they're registered in `ModelConfig`.
|
||||
|
||||
### Progress Display
|
||||
## Manual Model Operations
|
||||
|
||||
```typescript
|
||||
// Subscribe to download progress via SSE
|
||||
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/models/status` | Status of every registered model |
|
||||
| POST | `/models/load` | Load a TTS model into memory |
|
||||
| POST | `/models/unload` | Unload a TTS model from memory |
|
||||
| POST | `/models/download` | Trigger a background download |
|
||||
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||
| DELETE | `/models/{name}` | Delete a downloaded model from cache |
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
const progress = JSON.parse(event.data);
|
||||
updateProgressBar(progress.current / progress.total);
|
||||
|
||||
if (progress.status === 'complete') {
|
||||
eventSource.close();
|
||||
}
|
||||
};
|
||||
### Load
|
||||
|
||||
```http
|
||||
POST /models/load
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B"
|
||||
}
|
||||
```
|
||||
|
||||
### Model Status UI
|
||||
The route looks up the config, dispatches to `get_model_load_func(config)`, and returns once the model is ready.
|
||||
|
||||
```typescript
|
||||
// Fetch model status
|
||||
const { data: models } = useQuery({
|
||||
queryKey: ['models', 'status'],
|
||||
queryFn: () => api.getModelStatus(),
|
||||
});
|
||||
### Unload
|
||||
|
||||
// Display download/load buttons based on status
|
||||
models.map(model => (
|
||||
<ModelCard
|
||||
name={model.display_name}
|
||||
downloaded={model.downloaded}
|
||||
loaded={model.loaded}
|
||||
onDownload={() => triggerDownload(model.model_name)}
|
||||
onLoad={() => loadModel(model.model_name)}
|
||||
/>
|
||||
));
|
||||
```http
|
||||
POST /models/unload
|
||||
{
|
||||
"model_name": "chatterbox-tts"
|
||||
}
|
||||
```
|
||||
|
||||
Calls `unload_model_by_config(config)`, which routes to the right backend's `unload_model()` and frees GPU memory.
|
||||
|
||||
### Download
|
||||
|
||||
```http
|
||||
POST /models/download
|
||||
{
|
||||
"model_name": "kokoro"
|
||||
}
|
||||
```
|
||||
|
||||
Fires off an async download task. Progress is available via the SSE endpoint. Download is triggered automatically on first generation, so this is only needed for pre-warming.
|
||||
|
||||
## Preset Voice Seeding
|
||||
|
||||
For engines that use preset voices (Kokoro, Qwen CustomVoice), the backend auto-creates a voice profile per preset voice after the model is downloaded. This is driven by `seed_preset_profiles(engine)` in `backend/services/profiles.py`, called from the models route once download completes.
|
||||
|
||||
Preset profiles have:
|
||||
|
||||
- `voice_type = "preset"`
|
||||
- `preset_engine` = engine name (`"kokoro"`, `"qwen_custom_voice"`)
|
||||
- `preset_voice_id` = engine-specific voice ID (`"am_adam"`, `"f000001"`, etc.)
|
||||
- No `profile_samples` rows — no audio to store
|
||||
|
||||
See [Voice Profiles](/developer/voice-profiles) for the schema.
|
||||
|
||||
## Adding a New Model
|
||||
|
||||
To add a new size variant of an existing engine, just add another `ModelConfig`:
|
||||
|
||||
```python
|
||||
ModelConfig(
|
||||
model_name="qwen-tts-3B",
|
||||
display_name="Qwen TTS 3B",
|
||||
engine="qwen",
|
||||
hf_repo_id="Qwen/Qwen3-TTS-12Hz-3B-Base",
|
||||
model_size="3B",
|
||||
size_mb=7000,
|
||||
languages=["zh", "en", ...],
|
||||
),
|
||||
```
|
||||
|
||||
The frontend picks it up via `/models/status`; download/load flow works without further changes.
|
||||
|
||||
Adding a whole new engine is a bigger lift — see [TTS Engines](/developer/tts-engines) for the full phased workflow.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Download failed | Network issue | Retry download |
|
||||
| OOM on load | Model too large | Use smaller model |
|
||||
| Model not found | Cache corrupted | Re-download |
|
||||
| Slow download | HF rate limit | Wait and retry |
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Download failed | Network / HF rate limit | Retry |
|
||||
| OOM on load | Not enough VRAM | Use a smaller variant, unload other engines |
|
||||
| Model not found | Corrupt cache | Re-download via `/models/download` |
|
||||
| Stuck progress bar in frozen build | `huggingface_hub` tqdm silenced | `HFProgressTracker` force-enables the internal counter |
|
||||
| GPU architecture unsupported | PyTorch wheel doesn't target your GPU | See [GPU Acceleration](/overview/gpu-acceleration) |
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="TTS Generation" href="/developer/tts-generation">
|
||||
How generation flows through the registry
|
||||
</Card>
|
||||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||||
Add a new engine end-to-end
|
||||
</Card>
|
||||
<Card title="Transcription" href="/developer/transcription">
|
||||
Whisper and MLX-Whisper integration
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -59,24 +59,64 @@ Ensure you have these installed:
|
||||
|
||||
## Just Commands
|
||||
|
||||
Run `just --list` to see all available commands:
|
||||
Run `just --list` to see all available commands. Highlights:
|
||||
|
||||
### Setup
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just setup` | Full setup (Python venv + JS deps + dev sidecar). Detects Apple Silicon for MLX and NVIDIA/Intel Arc on Windows for accelerated PyTorch. |
|
||||
| `just setup-python` | Python venv + dependencies only |
|
||||
| `just setup-js` | `bun install` only |
|
||||
|
||||
### Development
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just dev` | Start backend + Tauri desktop app (reuses a running backend if one exists) |
|
||||
| `just dev-web` | Start backend + web app (no Tauri/Rust build) |
|
||||
| `just dev-backend` | Backend only |
|
||||
| `just dev-frontend` | Tauri app only (backend must already be running) |
|
||||
| `just kill` | Stop all dev processes |
|
||||
|
||||
### Build
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just build` | CPU server binary + Tauri installer |
|
||||
| `just build-local` | **Windows:** CPU + CUDA server binaries + Tauri installer |
|
||||
| `just build-server` | CPU server binary only |
|
||||
| `just build-server-cuda` | **Windows:** CUDA server binary only, placed in `%APPDATA%/sh.voicebox.app/backends/cuda` for local testing |
|
||||
| `just build-tauri` | Tauri app only |
|
||||
| `just build-web` | Web app only |
|
||||
|
||||
### Quality
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just check` | Lint + format + typecheck (Biome + ruff) |
|
||||
| `just fix` | Auto-fix lint + format issues |
|
||||
| `just lint` / `just format` | Lint or format only |
|
||||
| `just test` | Run Python tests (pytest) |
|
||||
| `just test-models` | End-to-end generation against every TTS engine using the frozen binary |
|
||||
|
||||
### Database
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just setup` | Full setup (Python venv + JS deps) |
|
||||
| `just dev` | Start backend + desktop app |
|
||||
| `just dev-web` | Start backend + web app (no Tauri) |
|
||||
| `just dev-backend` | Start backend only |
|
||||
| `just dev-frontend` | Start desktop app only (backend must be running) |
|
||||
| `just build` | Build desktop app for production |
|
||||
| `just build-web` | Build web app for production |
|
||||
| `just check` | Run all checks (JS + Python lint + format) |
|
||||
| `just fix` | Fix lint + format issues |
|
||||
| `just test` | Run Python tests |
|
||||
| `just db-init` | Initialize SQLite database |
|
||||
| `just db-reset` | Reset database (delete + reinit) |
|
||||
| `just clean` | Clean build artifacts |
|
||||
| `just clean-all` | Nuclear clean (includes node_modules) |
|
||||
| `just db-reset` | Delete and reinitialize the database |
|
||||
|
||||
### Utilities
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `just generate-api` | Generate TypeScript API client from the backend's OpenAPI schema |
|
||||
| `just docs` | Open `http://localhost:17493/docs` in your browser |
|
||||
| `just logs` | Tail backend logs |
|
||||
| `just clean` | Remove build artifacts |
|
||||
| `just clean-python` | Remove the Python venv + `__pycache__` |
|
||||
| `just clean-all` | Nuclear clean (includes all `node_modules`) |
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -133,10 +173,12 @@ HTTP request → **routes/** (validate input) → **services/** (business logic)
|
||||
|
||||
## Model Downloads
|
||||
|
||||
Models are automatically downloaded from HuggingFace Hub on first use:
|
||||
Models are automatically downloaded from HuggingFace Hub on first use, with live progress streamed to the UI:
|
||||
|
||||
- **Whisper** (transcription): Auto-downloads on first transcription
|
||||
- **Qwen3-TTS** (voice cloning): Auto-downloads on first generation (~2-4GB)
|
||||
- **Whisper** (transcription) — auto-downloads on first transcription
|
||||
- **TTS engines** — auto-download on first generation. Sizes range from 82 M (Kokoro, ~350 MB) to 3 B (TADA, ~8 GB)
|
||||
|
||||
See [Model Management](/developer/model-management) for the full list.
|
||||
|
||||
<Callout type="warn">
|
||||
First-time usage will be slower due to model downloads, but subsequent runs will use cached models.
|
||||
@@ -150,7 +192,7 @@ After starting the backend server, generate the TypeScript API client:
|
||||
just generate-api
|
||||
```
|
||||
|
||||
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`
|
||||
This downloads the OpenAPI schema and generates the TypeScript client in `app/src/lib/api/`.
|
||||
|
||||
## Manual Setup (Advanced)
|
||||
|
||||
@@ -186,8 +228,17 @@ pip install -r requirements.txt
|
||||
# Apple Silicon: install MLX dependencies
|
||||
pip install -r requirements-mlx.txt
|
||||
|
||||
# Install Qwen3-TTS
|
||||
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
|
||||
pip install --no-deps chatterbox-tts
|
||||
|
||||
# HumeAI TADA pins torch>=2.7,<2.8 which conflicts with our torch>=2.1
|
||||
pip install --no-deps hume-tada
|
||||
|
||||
# Install Qwen3-TTS from source
|
||||
pip install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
# PyInstaller and linting tools
|
||||
pip install pyinstaller ruff pytest pytest-asyncio
|
||||
```
|
||||
|
||||
### 3. Start Development
|
||||
@@ -208,17 +259,17 @@ bun run tauri dev
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Architecture" href="/development/architecture">
|
||||
<Card title="Architecture" href="/developer/architecture">
|
||||
Understand the system architecture
|
||||
</Card>
|
||||
<Card title="Contributing" href="/development/contributing">
|
||||
<Card title="Contributing" href="/developer/contributing">
|
||||
Read the contribution guidelines
|
||||
</Card>
|
||||
<Card title="Building" href="/development/building">
|
||||
<Card title="Building" href="/developer/building">
|
||||
Learn how to build production releases
|
||||
</Card>
|
||||
<Card title="API Reference" href="/api-reference">
|
||||
Explore the REST API
|
||||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||||
Add a new TTS engine end-to-end
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
|
||||
@@ -50,34 +50,15 @@ class StoryItem(Base):
|
||||
|
||||
### Start Time
|
||||
|
||||
`start_time_ms` defines when an item begins on the timeline:
|
||||
|
||||
```
|
||||
Timeline (ms): 0----1000----2000----3000----4000
|
||||
Item 1: [======]
|
||||
Item 2: [==========]
|
||||
Item 3: [====]
|
||||
```
|
||||
`start_time_ms` is the absolute position on the timeline where an item begins playing. Items on the same track cannot overlap; items on different tracks can.
|
||||
|
||||
### Tracks
|
||||
|
||||
Multiple tracks allow overlapping audio:
|
||||
|
||||
```
|
||||
Track 0: [Item 1] [Item 3]
|
||||
Track 1: [Item 2]
|
||||
```
|
||||
A `track` is an integer (0-indexed) that identifies the horizontal row an item sits on. Audio on separate tracks plays concurrently, so tracks are the primary way to layer multiple voices or sound effects.
|
||||
|
||||
### Trimming
|
||||
|
||||
Trim values cut audio from the start or end without destroying the original:
|
||||
|
||||
```
|
||||
Original: [=========AUDIO=========]
|
||||
trim_start: ^^
|
||||
trim_end: ^^
|
||||
Result: [=====AUDIO=====]
|
||||
```
|
||||
`trim_start_ms` and `trim_end_ms` hide the leading/trailing portions of the source generation without modifying the underlying audio file. The effective playback length is `generation.duration * 1000 - trim_start_ms - trim_end_ms`. Trimming is non-destructive — the same generation can be trimmed differently in different stories.
|
||||
|
||||
## Core Operations
|
||||
|
||||
|
||||
@@ -5,250 +5,91 @@ description: "How Whisper-based audio transcription works in Voicebox"
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses OpenAI's Whisper model for automatic speech recognition (ASR). This powers the transcription feature for creating reference text from audio recordings.
|
||||
Voicebox uses OpenAI's Whisper for automatic speech recognition (ASR). Transcription powers two flows:
|
||||
|
||||
1. **Reference-text auto-fill** — when a user records or uploads a voice sample, the backend transcribes it and populates the `reference_text` field so cloning can use it.
|
||||
2. **On-demand transcription** — a user-facing `/transcribe` endpoint for arbitrary audio.
|
||||
|
||||
On Apple Silicon, the transcription path runs through **MLX-Whisper** (from `mlx-audio`) for ~8× faster inference than PyTorch. Everywhere else it runs through PyTorch's `transformers` Whisper.
|
||||
|
||||
## Architecture
|
||||
|
||||
The transcription system is built around the `WhisperModel` class:
|
||||
|
||||
**Model Loading:** Lazy loading with HuggingFace Hub download.
|
||||
|
||||
**Audio Processing:** Resampling and preprocessing for Whisper.
|
||||
|
||||
**Inference:** Running transcription with optional language hints.
|
||||
|
||||
## WhisperModel Class
|
||||
Transcription is wired through the same backend abstraction as TTS. The `STTBackend` protocol lives in `backend/backends/__init__.py`:
|
||||
|
||||
```python
|
||||
class WhisperModel:
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.processor = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
@runtime_checkable
|
||||
class STTBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
model_size: Optional[str] = None,
|
||||
) -> str: ...
|
||||
def unload_model(self) -> None: ...
|
||||
def is_loaded(self) -> bool: ...
|
||||
```
|
||||
|
||||
### Model Sizes
|
||||
Two implementations ship today:
|
||||
|
||||
| Size | Parameters | VRAM | Speed | Quality |
|
||||
|------|------------|------|-------|---------|
|
||||
| tiny | 39M | ~1GB | Fastest | Basic |
|
||||
| base | 74M | ~1GB | Fast | Good |
|
||||
| small | 244M | ~2GB | Medium | Better |
|
||||
| medium | 769M | ~5GB | Slow | High |
|
||||
| large | 1550M | ~10GB | Slowest | Best |
|
||||
- **`MLXSTTBackend`** (`backends/mlx_backend.py`) — uses `mlx_audio.stt.load()`. Default on Apple Silicon.
|
||||
- **`PyTorchSTTBackend`** (`backends/pytorch_backend.py`) — uses `transformers.WhisperForConditionalGeneration`. Default everywhere else.
|
||||
|
||||
Default is `base` for balance of speed and quality.
|
||||
`get_stt_backend()` picks the right one based on `get_backend_type()`. `backend/services/transcribe.py` is a thin wrapper that delegates to the backend.
|
||||
|
||||
## Model Sizes
|
||||
|
||||
Five Whisper variants are registered in `ModelConfig`:
|
||||
|
||||
| Model | HuggingFace Repo | Size | Notes |
|
||||
|-------|------------------|------|-------|
|
||||
| **Base** | `openai/whisper-base` | ~300 MB | Default; fast, decent quality |
|
||||
| **Small** | `openai/whisper-small` | ~500 MB | Better quality, still fast |
|
||||
| **Medium** | `openai/whisper-medium` | ~1.5 GB | High quality |
|
||||
| **Large** | `openai/whisper-large-v3` | ~3 GB | Best quality, slow on CPU |
|
||||
| **Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB | Large-tier quality, ~5× faster than Large |
|
||||
|
||||
The `tiny` model is **not** exposed — the quality gap to `base` wasn't worth the download.
|
||||
|
||||
`Turbo` + MLX-Whisper on Apple Silicon dropped user-facing transcription latency from ~20s to ~2-3s in v0.1.10.
|
||||
|
||||
## Language Hints
|
||||
|
||||
Whisper can auto-detect language, but providing a hint improves accuracy on short clips:
|
||||
|
||||
```python
|
||||
text = await backend.transcribe(audio_path, language="en")
|
||||
```
|
||||
|
||||
Accepted language codes are the standard Whisper set (99+ languages). The frontend typically passes the profile's language if available, or lets Whisper detect otherwise.
|
||||
|
||||
## Model Loading
|
||||
|
||||
Models are downloaded from HuggingFace Hub:
|
||||
Both backends are lazy: the model is loaded on first use and cached in memory. Switching sizes unloads the previous model.
|
||||
|
||||
On MLX, the model is loaded via `mlx_audio.stt.load(hf_repo)`. On PyTorch, via:
|
||||
|
||||
```python
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
|
||||
model_name = f"openai/whisper-{model_size}"
|
||||
|
||||
# Track download progress
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
task_manager.start_download(f"whisper-{model_size}")
|
||||
|
||||
# Load processor and model
|
||||
with tracker.patch_download():
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
|
||||
self.model.to(self.device)
|
||||
|
||||
# Mark complete
|
||||
progress_manager.mark_complete(f"whisper-{model_size}")
|
||||
task_manager.complete_download(f"whisper-{model_size}")
|
||||
WhisperProcessor.from_pretrained(hf_repo)
|
||||
WhisperForConditionalGeneration.from_pretrained(hf_repo).to(device)
|
||||
```
|
||||
|
||||
### Async Loading
|
||||
|
||||
Like TTS, loading runs in a thread pool:
|
||||
|
||||
```python
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
```
|
||||
|
||||
## Transcription
|
||||
|
||||
### Basic Transcription
|
||||
|
||||
```python
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> str:
|
||||
await self.load_model_async()
|
||||
|
||||
def _transcribe_sync():
|
||||
# Load and resample to 16kHz (Whisper requirement)
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
sampling_rate=16000,
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
# Set language hint if provided
|
||||
forced_decoder_ids = None
|
||||
if language:
|
||||
forced_decoder_ids = self.processor.get_decoder_prompt_ids(
|
||||
language=language,
|
||||
task="transcribe",
|
||||
)
|
||||
|
||||
# Generate
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
forced_decoder_ids=forced_decoder_ids,
|
||||
)
|
||||
|
||||
# Decode
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
return transcription.strip()
|
||||
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
```
|
||||
|
||||
### Supported Languages
|
||||
|
||||
Whisper supports 99+ languages. Common ones in Voicebox:
|
||||
|
||||
| Code | Language |
|
||||
|------|----------|
|
||||
| en | English |
|
||||
| zh | Chinese |
|
||||
| ja | Japanese |
|
||||
| ko | Korean |
|
||||
| de | German |
|
||||
| fr | French |
|
||||
| ru | Russian |
|
||||
| pt | Portuguese |
|
||||
| es | Spanish |
|
||||
| it | Italian |
|
||||
|
||||
### Language Detection
|
||||
|
||||
When no language is specified, Whisper auto-detects:
|
||||
|
||||
```python
|
||||
# Without language hint - auto-detect
|
||||
transcription = await whisper.transcribe(audio_path)
|
||||
|
||||
# With language hint - more accurate for short clips
|
||||
transcription = await whisper.transcribe(audio_path, language="en")
|
||||
```
|
||||
|
||||
## Transcription with Timestamps
|
||||
|
||||
For advanced use cases, word-level timestamps are available:
|
||||
|
||||
```python
|
||||
async def transcribe_with_timestamps(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
) -> List[Dict[str, any]]:
|
||||
await self.load_model_async()
|
||||
|
||||
def _transcribe_timestamps_sync():
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
inputs = self.processor(audio, sampling_rate=16000, return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
return_timestamps=True,
|
||||
)
|
||||
|
||||
# Parse timestamps
|
||||
return [
|
||||
{
|
||||
"text": transcription,
|
||||
"start": 0.0,
|
||||
"end": len(audio) / sr,
|
||||
}
|
||||
]
|
||||
|
||||
return await asyncio.to_thread(_transcribe_timestamps_sync)
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
### Unloading
|
||||
|
||||
Free memory when not needed:
|
||||
|
||||
```python
|
||||
def unload_model(self):
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
del self.processor
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
```
|
||||
|
||||
### Global Instance
|
||||
|
||||
A singleton pattern manages the model:
|
||||
|
||||
```python
|
||||
_whisper_model: Optional[WhisperModel] = None
|
||||
|
||||
def get_whisper_model() -> WhisperModel:
|
||||
global _whisper_model
|
||||
if _whisper_model is None:
|
||||
_whisper_model = WhisperModel()
|
||||
return _whisper_model
|
||||
```
|
||||
Both load paths use `model_load_progress()` from `backends/base.py` so the frontend sees live download progress on the first use.
|
||||
|
||||
## Audio Preprocessing
|
||||
|
||||
### Resampling
|
||||
Whisper expects mono 16 kHz audio. The audio utility in `backend/utils/audio.py` handles resampling and format conversion transparently:
|
||||
|
||||
Whisper requires 16kHz audio:
|
||||
- **Formats:** WAV, MP3, FLAC, OGG, M4A (via soundfile / librosa)
|
||||
- **Target:** mono, 16 kHz, float32
|
||||
|
||||
```python
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
```
|
||||
|
||||
### Format Support
|
||||
|
||||
The `load_audio` utility handles:
|
||||
- WAV
|
||||
- MP3
|
||||
- FLAC
|
||||
- OGG
|
||||
- M4A
|
||||
|
||||
All formats are converted to mono 16kHz.
|
||||
Files longer than Whisper's 30-second window are handled by the underlying library's chunking logic — no explicit splitting in Voicebox code.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| POST | `/transcribe` | Transcribe audio file |
|
||||
| POST | `/transcribe` | Transcribe an uploaded audio file |
|
||||
|
||||
### Request
|
||||
|
||||
@@ -259,7 +100,8 @@ POST /transcribe
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <audio_file>
|
||||
language: en (optional)
|
||||
language: en # optional
|
||||
model_size: base # optional (default: "base")
|
||||
```
|
||||
|
||||
### Response
|
||||
@@ -275,25 +117,44 @@ language: en (optional)
|
||||
|
||||
### Reference Text for Voice Cloning
|
||||
|
||||
1. User records audio sample
|
||||
2. Audio is sent to `/transcribe`
|
||||
3. Transcription becomes `reference_text`
|
||||
4. Both are added to voice profile
|
||||
Adding a voice sample triggers transcription automatically:
|
||||
|
||||
1. User uploads or records audio.
|
||||
2. The backend writes the audio file and calls `/transcribe` internally (or the frontend calls it separately).
|
||||
3. The returned text becomes `reference_text` on the new `profile_samples` row.
|
||||
4. Cloning engines that need reference text (Chatterbox, TADA, etc.) read it from there.
|
||||
|
||||
### Quality Tips
|
||||
|
||||
- Provide language hint for short audio
|
||||
- Use clean audio with minimal noise
|
||||
- Longer audio (>5s) improves accuracy
|
||||
- Consider `small` or `medium` model for better quality
|
||||
- Provide a language hint for short clips (under 5 seconds) — auto-detection is unreliable on little audio.
|
||||
- Use Turbo or Large for noisy audio — Base can hallucinate on hard inputs.
|
||||
- Prefer clean audio; transcription errors become reference-text errors, which become cloning errors.
|
||||
|
||||
## Memory Management
|
||||
|
||||
`unload_model()` drops the model reference and clears the CUDA cache if applicable. `/models/unload` wires this up for manual control.
|
||||
|
||||
A singleton per backend is returned by `get_stt_backend()` — multiple callers share one Whisper instance.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common issues:
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Model not found | First run, download failed | Retry with network |
|
||||
| OOM | Model too large | Use smaller model |
|
||||
| Empty result | No speech detected | Check audio has speech |
|
||||
| Wrong language | Auto-detect failed | Provide language hint |
|
||||
| Model not found | First run + network failure | Retry; check connectivity |
|
||||
| OOM on load | Large model on low-VRAM GPU | Switch to Small or Turbo |
|
||||
| Empty result | No speech in audio | Confirm input has voice; check trim |
|
||||
| Wrong language | Auto-detect misfired | Pass `language` hint |
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Model Management" href="/developer/model-management">
|
||||
Download / load / unload any model
|
||||
</Card>
|
||||
<Card title="Voice Profiles" href="/developer/voice-profiles">
|
||||
How reference text is stored alongside samples
|
||||
</Card>
|
||||
<Card title="GPU Acceleration" href="/overview/gpu-acceleration">
|
||||
Platform-specific acceleration including MLX-Whisper
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -604,18 +604,25 @@ for name, mod in [("dac", types.ModuleType("dac")),
|
||||
- Do NOT use `@torch.jit.script` in the shim (see above)
|
||||
- Only reimplement what the model actually uses — check the import chain carefully
|
||||
|
||||
## Upcoming Engines
|
||||
## Candidate Engines
|
||||
|
||||
Based on the current model landscape, these are candidates for future integration:
|
||||
The [`docs/PROJECT_STATUS.md`](https://github.com/jamiepine/voicebox/blob/main/docs/PROJECT_STATUS.md) file is the canonical, living list of candidates under evaluation — including why some have been backlogged (e.g. VoxCPM, which is effectively CUDA-only upstream).
|
||||
|
||||
| Model | Languages | Size | Key Features | Status |
|
||||
|-------|-----------|------|--------------|--------|
|
||||
| **CosyVoice2-0.5B** | Multilingual | ~500MB | Instruct support (`inference_instruct2()`) | Ready |
|
||||
| **Fish Speech** | 50+ | Medium | Word-level control via inline text | Ready |
|
||||
| **Kokoro-82M** | English | 82M | CPU realtime, Apache 2.0 | Ready |
|
||||
| **XTTS-v2** | 17+ | Medium | Zero-shot cloning | Ready |
|
||||
| **MOSS-TTS** | Multilingual | Medium | Text-to-voice design, multi-speaker dialogue | Needs vetting |
|
||||
| **Pocket TTS** | English | ~100M | CPU-first, >1× realtime | Needs vetting |
|
||||
At a glance, current top candidates:
|
||||
|
||||
| Model | Tier | Size | Cross-platform? | Key Features |
|
||||
|-------|------|------|-----------------|--------------|
|
||||
| **MOSS-TTS-Nano** | 1 | 0.1 B | Yes (CPU realtime) | 48 kHz stereo, Apache 2.0, released 2026-04-13 |
|
||||
| **Voxtral TTS** | 2 | 4 B | Likely | `mistralai/Voxtral-4B-TTS-2603` — presets + cloning |
|
||||
| **VibeVoice** | 2 | ~500 M | Yes | Podcast-style multi-speaker dialogue |
|
||||
| **Dia2** | 3 | TBD | TBD | Successor to the original Dia |
|
||||
| **Fish Audio S2 Pro** | 3 | Medium | Yes | Word-level control via inline text |
|
||||
|
||||
**Backlogged:**
|
||||
|
||||
- **VoxCPM** (2B, Apache 2.0) — CUDA ≥12 required upstream; MPS broken in issues #232/#248; CPU path rejected by maintainers (#256). Keep watching for a PR that relaxes the device requirement.
|
||||
|
||||
Update `PROJECT_STATUS.md` when you pick one up or mark one as shipped/backlogged.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
|
||||
@@ -1,283 +1,251 @@
|
||||
---
|
||||
title: "TTS Generation"
|
||||
description: "How text-to-speech generation works in Voicebox"
|
||||
description: "How text-to-speech generation works across Voicebox's multi-engine backend"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
|
||||
Voicebox ships seven TTS engines — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, TADA, and Kokoro — behind a single `TTSBackend` Protocol. All of them expose the same async interface so the routes and services don't need per-engine branching.
|
||||
|
||||
## Architecture
|
||||
This page covers how generation flows through that abstraction. For the step-by-step guide to adding a new engine, see [TTS Engines](/developer/tts-engines).
|
||||
|
||||
The TTS system is built around the `TTSModel` class which manages:
|
||||
## The `TTSBackend` Protocol
|
||||
|
||||
**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
|
||||
|
||||
**Voice Prompts:** Converting reference audio into embeddings.
|
||||
|
||||
**Generation:** Synthesizing speech from text using voice prompts.
|
||||
|
||||
## TTSModel Class
|
||||
Every engine implements the same contract (defined in `backend/backends/__init__.py`):
|
||||
|
||||
```python
|
||||
class TTSModel:
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device() # cuda, mps, or cpu
|
||||
@runtime_checkable
|
||||
class TTSBackend(Protocol):
|
||||
async def load_model(self, model_size: str) -> None: ...
|
||||
async def create_voice_prompt(
|
||||
self, audio_path: str, reference_text: str, use_cache: bool = True
|
||||
) -> Tuple[dict, bool]: ...
|
||||
async def combine_voice_prompts(
|
||||
self, audio_paths: List[str], reference_texts: List[str]
|
||||
) -> Tuple[np.ndarray, str]: ...
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]: ...
|
||||
def unload_model(self) -> None: ...
|
||||
def is_loaded(self) -> bool: ...
|
||||
```
|
||||
|
||||
### Device Selection
|
||||
## The `ModelConfig` Registry
|
||||
|
||||
The model automatically selects the best available device:
|
||||
Each downloadable model variant is described by a `ModelConfig` dataclass:
|
||||
|
||||
```python
|
||||
def _get_device(self) -> str:
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
return "cpu" # MPS can have issues, use CPU for stability
|
||||
return "cpu"
|
||||
@dataclass
|
||||
class ModelConfig:
|
||||
model_name: str # "luxtts", "qwen-tts-1.7B", "kokoro"
|
||||
display_name: str # "LuxTTS (Fast, CPU-friendly)"
|
||||
engine: str # "luxtts", "qwen", "kokoro"
|
||||
hf_repo_id: str # "YatharthS/LuxTTS"
|
||||
model_size: str = "default"
|
||||
size_mb: int = 0
|
||||
needs_trim: bool = False
|
||||
supports_instruct: bool = False
|
||||
languages: list[str] = field(default_factory=lambda: ["en"])
|
||||
```
|
||||
|
||||
## Model Loading
|
||||
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
|
||||
|
||||
Models are downloaded from HuggingFace Hub on first use:
|
||||
- `get_all_model_configs()` — every TTS + STT variant
|
||||
- `get_tts_model_configs()` — only TTS variants
|
||||
- `get_model_config(model_name)` — lookup by name
|
||||
- `engine_needs_trim(engine)` — whether output should run through `trim_tts_output()`
|
||||
- `load_engine_model(engine, model_size)` — downloads + loads, handles engines with multiple sizes
|
||||
- `get_tts_backend_for_engine(engine)` — thread-safe backend factory with double-checked locking
|
||||
|
||||
The `TTS_ENGINES` dict is the canonical list of shipped engine names:
|
||||
|
||||
```python
|
||||
def load_model(self, model_size: Optional[str] = None):
|
||||
# Model IDs on HuggingFace Hub
|
||||
hf_model_map = {
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
# Load with progress tracking
|
||||
with tracker.patch_download():
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16, # float32 on CPU
|
||||
)
|
||||
TTS_ENGINES = {
|
||||
"qwen": "Qwen TTS",
|
||||
"qwen_custom_voice": "Qwen CustomVoice",
|
||||
"luxtts": "LuxTTS",
|
||||
"chatterbox": "Chatterbox TTS",
|
||||
"chatterbox_turbo": "Chatterbox Turbo",
|
||||
"tada": "TADA",
|
||||
"kokoro": "Kokoro",
|
||||
}
|
||||
```
|
||||
|
||||
### Async Loading
|
||||
## Voice Prompt Patterns
|
||||
|
||||
Loading runs in a thread pool to avoid blocking the event loop:
|
||||
Each engine chooses how to represent a voice in the prompt dict returned from `create_voice_prompt()`. Three patterns are in use today:
|
||||
|
||||
**Pattern A — Pre-computed tensors** (Qwen3-TTS, LuxTTS)
|
||||
|
||||
```python
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
await asyncio.to_thread(self.load_model, model_size)
|
||||
encoded = model.encode_prompt(audio_path)
|
||||
return encoded, False # (prompt_dict, was_cached)
|
||||
```
|
||||
|
||||
## Voice Prompt Creation
|
||||
|
||||
Voice prompts are created from reference audio and cached for reuse:
|
||||
**Pattern B — Deferred file paths** (Chatterbox, Chatterbox Turbo, TADA)
|
||||
|
||||
```python
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
await self.load_model_async()
|
||||
|
||||
# Check cache
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cached = get_cached_voice_prompt(cache_key)
|
||||
if cached:
|
||||
return cached, True
|
||||
|
||||
# Create prompt (blocking, run in thread pool)
|
||||
voice_prompt = await asyncio.to_thread(
|
||||
self.model.create_voice_clone_prompt,
|
||||
ref_audio=audio_path,
|
||||
ref_text=reference_text,
|
||||
)
|
||||
|
||||
# Cache the result
|
||||
cache_voice_prompt(cache_key, voice_prompt)
|
||||
return voice_prompt, False
|
||||
return {"ref_audio": audio_path, "ref_text": reference_text}, False
|
||||
```
|
||||
|
||||
### Combining Multiple Samples
|
||||
|
||||
When a profile has multiple samples, they're combined:
|
||||
**Pattern C — Preset voice pointer** (Kokoro, Qwen CustomVoice)
|
||||
|
||||
```python
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate and normalize
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
return {
|
||||
"voice_type": "preset",
|
||||
"preset_engine": "kokoro",
|
||||
"preset_voice_id": "am_adam",
|
||||
}, False
|
||||
```
|
||||
|
||||
## Speech Generation
|
||||
Pattern C is the shape used for profiles where `voice_type == "preset"` — there's no cloning step; the engine looks up a baked-in voice by ID.
|
||||
|
||||
The core generation function:
|
||||
Engines that cache voice prompts prefix their cache keys to avoid collisions:
|
||||
|
||||
```python
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
await self.load_model_async()
|
||||
|
||||
def _generate_sync():
|
||||
# Set seed for reproducibility
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
|
||||
# Generate audio
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
text=text,
|
||||
voice_clone_prompt=voice_prompt,
|
||||
instruct=instruct, # Natural language delivery control
|
||||
)
|
||||
return wavs[0], sample_rate
|
||||
|
||||
# Run in thread pool
|
||||
return await asyncio.to_thread(_generate_sync)
|
||||
cache_key = f"{engine}_{hash(audio_path, reference_text)}"
|
||||
```
|
||||
|
||||
### Instruct Feature
|
||||
## Device Selection
|
||||
|
||||
The `instruct` parameter allows natural language control over speech delivery:
|
||||
Engines pick their device through `get_torch_device()` in `backends/base.py`, which layers:
|
||||
|
||||
1. `VOICEBOX_FORCE_CPU` environment override
|
||||
2. CUDA (if compiled and available)
|
||||
3. XPU (Intel Arc via IPEX)
|
||||
4. MPS (Apple Silicon) — **only for engines that support it**; some (Chatterbox, older Qwen paths) skip MPS and fall back to CPU due to upstream operator gaps
|
||||
5. CPU
|
||||
|
||||
Qwen TTS uses MLX directly on Apple Silicon instead of going through PyTorch — see `mlx_backend.py`.
|
||||
|
||||
## Generation Flow
|
||||
|
||||
The request path from frontend to audio file:
|
||||
|
||||
1. **Request** — `POST /generate` with `GenerationRequest`:
|
||||
```json
|
||||
{
|
||||
"profile_id": "uuid",
|
||||
"text": "...",
|
||||
"language": "en",
|
||||
"seed": 42,
|
||||
"model_size": "1.7B",
|
||||
"instruct": "warm, slightly amused",
|
||||
"engine": "qwen",
|
||||
"max_chunk_chars": 800
|
||||
}
|
||||
```
|
||||
The `engine` field is validated against the regex `^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$`.
|
||||
|
||||
2. **Route** — `routes/generate.py` validates input and delegates.
|
||||
|
||||
3. **Service** — `services/generation.py` fetches the profile, resolves the engine backend via `get_tts_backend_for_engine(engine)`, and ensures the model is loaded (downloading it on first use with live progress).
|
||||
|
||||
4. **Voice prompt** — the service calls `create_voice_prompt()` (or the preset equivalent). For cloned profiles with multiple samples, it calls `combine_voice_prompts()` first to merge reference audio.
|
||||
|
||||
5. **Queue** — the request is serialized through `services/task_queue.py` to avoid multiple generations fighting for the GPU.
|
||||
|
||||
6. **Inference** — the engine's `generate()` returns `(audio_array, sample_rate)`.
|
||||
|
||||
7. **Post-process** — if `engine_needs_trim(engine)` is True, `trim_tts_output()` strips trailing silence. Effects chains (if any) are applied per generation version, not the clean version.
|
||||
|
||||
8. **Persist** — audio is written to the generations directory, a row is inserted into the `generations` table, and the response includes the generation metadata.
|
||||
|
||||
## Chunking for Long Text
|
||||
|
||||
Text longer than `max_chunk_chars` (default 800, range 100–5000) is split at sentence boundaries, generated in sequence, and crossfaded together. The chunking behavior is engine-agnostic — it lives in the service layer, not in individual backends.
|
||||
|
||||
## Instruct Mode
|
||||
|
||||
Two engines support natural-language delivery control via the `instruct` kwarg:
|
||||
|
||||
- **Qwen CustomVoice** — `supports_instruct=True`, fully wired to the model's instruct head.
|
||||
- **Qwen Base** — silently drops the instruct text (`supports_instruct=False`). The frontend hides the instruct input for Base profiles.
|
||||
|
||||
```python
|
||||
# Examples:
|
||||
instruct = "Speak slowly and clearly"
|
||||
instruct = "Sound excited and enthusiastic"
|
||||
instruct = "Whisper softly"
|
||||
# Good instruct prompts:
|
||||
"warm and conversational, slight smile"
|
||||
"whisper, intimate and close"
|
||||
"authoritative, broadcast quality"
|
||||
```
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
Voice prompts are cached to avoid recomputation:
|
||||
|
||||
```python
|
||||
def get_cache_key(audio_path: str, reference_text: str) -> str:
|
||||
"""Generate cache key from audio hash and text."""
|
||||
audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
|
||||
text_hash = hashlib.md5(reference_text.encode()).hexdigest()
|
||||
return f"{audio_hash}_{text_hash}"
|
||||
```
|
||||
|
||||
Cache is stored in `data/cache/voice_prompts/`.
|
||||
Other engines ignore `instruct` entirely.
|
||||
|
||||
## Memory Management
|
||||
|
||||
### Unloading Models
|
||||
|
||||
Free VRAM/RAM when not needed:
|
||||
Models are loaded lazily on first use and kept in memory. Switching between model sizes (e.g. Qwen 1.7B ↔ 0.6B) unloads the previous model before loading the new one to avoid OOM:
|
||||
|
||||
```python
|
||||
def unload_model(self):
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
```
|
||||
|
||||
### Model Switching
|
||||
|
||||
When switching between model sizes (1.7B ↔ 0.6B):
|
||||
|
||||
```python
|
||||
# Unload existing model first
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
```
|
||||
|
||||
## Generation Flow
|
||||
|
||||
1. **Request** → Validate text and profile ID
|
||||
2. **Profile** → Load profile samples from database
|
||||
3. **Voice Prompt** → Create or retrieve cached prompt
|
||||
4. **Generate** → Run TTS inference
|
||||
5. **Save** → Write audio to generations directory
|
||||
6. **Record** → Create history entry in database
|
||||
7. **Response** → Return audio path and metadata
|
||||
The model management API (`/models/load`, `/models/unload`) lets users free VRAM manually — see [Model Management](/developer/model-management).
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| POST | `/generate` | Generate speech from text |
|
||||
| GET | `/audio/{id}` | Serve generated audio file |
|
||||
| GET | `/audio/{generation_id}` | Serve generated audio file |
|
||||
|
||||
### Request Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"profile_id": "uuid",
|
||||
"text": "Text to synthesize",
|
||||
"language": "en",
|
||||
"seed": 42,
|
||||
"model_size": "1.7B",
|
||||
"instruct": "Speak clearly"
|
||||
}
|
||||
```
|
||||
|
||||
### Response Schema
|
||||
### Response schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "generation_uuid",
|
||||
"profile_id": "profile_uuid",
|
||||
"text": "Text to synthesize",
|
||||
"text": "...",
|
||||
"language": "en",
|
||||
"audio_path": "/path/to/audio.wav",
|
||||
"duration": 3.5,
|
||||
"seed": 42,
|
||||
"instruct": "Speak clearly",
|
||||
"created_at": "2024-01-15T10:30:00Z"
|
||||
"engine": "qwen",
|
||||
"model_size": "1.7B",
|
||||
"instruct": "...",
|
||||
"created_at": "2026-04-18T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### GPU Acceleration
|
||||
- **CUDA** is the fastest backend for every PyTorch-based engine. Apple Silicon MLX is competitive with CUDA for Qwen TTS specifically.
|
||||
- **Serial queue** — only one generation runs at a time per process; concurrent requests are queued.
|
||||
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
|
||||
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
|
||||
|
||||
- CUDA provides fastest inference
|
||||
- MPS (Apple Silicon) has stability issues, uses CPU fallback
|
||||
- CPU inference is slower but always works
|
||||
### Per-engine VRAM (approximate, on CUDA)
|
||||
|
||||
### Batch Size
|
||||
| Engine | VRAM |
|
||||
|--------|------|
|
||||
| Kokoro | ~150 MB |
|
||||
| LuxTTS | ~1 GB |
|
||||
| Chatterbox Turbo | ~1.5 GB |
|
||||
| Qwen 0.6B / Qwen CustomVoice 0.6B | ~2 GB |
|
||||
| Chatterbox Multilingual | ~3 GB |
|
||||
| Qwen 1.7B / Qwen CustomVoice 1.7B | ~6 GB |
|
||||
| TADA 1B | ~4 GB |
|
||||
| TADA 3B | ~8 GB |
|
||||
|
||||
Currently generates one utterance at a time. For long texts, consider:
|
||||
- Splitting into sentences
|
||||
- Sequential generation
|
||||
- Concatenating results
|
||||
## Next Steps
|
||||
|
||||
### Memory Usage
|
||||
|
||||
| Model | VRAM/RAM Required |
|
||||
|-------|-------------------|
|
||||
| 0.6B | ~2GB |
|
||||
| 1.7B | ~6GB |
|
||||
<Cards>
|
||||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||||
Add a new engine — full phased workflow
|
||||
</Card>
|
||||
<Card title="Model Management" href="/developer/model-management">
|
||||
Downloading, loading, and unloading models
|
||||
</Card>
|
||||
<Card title="Voice Profiles" href="/developer/voice-profiles">
|
||||
Cloned vs preset profile schema
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
@@ -5,17 +5,22 @@ description: "How voice profile management works in Voicebox"
|
||||
|
||||
## Overview
|
||||
|
||||
Voice profiles are the foundation of Voicebox's voice cloning capability. Each profile stores reference audio samples and metadata that the TTS model uses to clone a voice.
|
||||
Voice profiles are the unit of "a saved voice" in Voicebox. As of 0.4 they support two flavors backed by the same `profiles` table:
|
||||
|
||||
- **Cloned profiles** — store one or more reference audio samples; the cloning engine generates a voice embedding at use time
|
||||
- **Preset profiles** — store no audio; just a pointer to an engine-specific pre-built voice (e.g. Kokoro's `am_adam`, Qwen CustomVoice's `Ryan`)
|
||||
|
||||
The schema also reserves a third type, `designed`, for future text-described voices. Not currently used by any shipped engine.
|
||||
|
||||
## Architecture
|
||||
|
||||
The voice profile system consists of three main components:
|
||||
|
||||
**Database Layer:** SQLite tables store profile metadata and sample references.
|
||||
**Database Layer:** SQLite tables store profile metadata, sample references (cloned), and engine + voice ID (preset).
|
||||
|
||||
**File Storage:** Audio samples are stored on disk in a structured directory format.
|
||||
**File Storage:** Audio samples are stored on disk in a structured directory format. Preset profiles have no on-disk audio.
|
||||
|
||||
**Profile Module:** The `profiles.py` module provides the business logic for CRUD operations.
|
||||
**Profile Module:** `backend/services/profiles.py` provides the business logic for CRUD operations and dispatches to the appropriate engine based on `voice_type`.
|
||||
|
||||
## Data Model
|
||||
|
||||
@@ -24,27 +29,49 @@ The voice profile system consists of three main components:
|
||||
```python
|
||||
class VoiceProfile(Base):
|
||||
__tablename__ = "profiles"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text)
|
||||
language = Column(String, default="en")
|
||||
created_at = Column(DateTime)
|
||||
updated_at = Column(DateTime)
|
||||
avatar_path = Column(String, nullable=True)
|
||||
effects_chain = Column(Text, nullable=True)
|
||||
|
||||
# Voice type system — added v0.3.x
|
||||
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
|
||||
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
|
||||
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
|
||||
design_prompt = Column(Text, nullable=True) # text description — only for designed (reserved)
|
||||
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
```
|
||||
|
||||
The `voice_type` column discriminates the three flavors:
|
||||
|
||||
| `voice_type` | `preset_engine` | `preset_voice_id` | Samples in `profile_samples` |
|
||||
| ------------ | --------------- | ----------------- | ---------------------------- |
|
||||
| `cloned` | NULL | NULL | Required (≥1 row) |
|
||||
| `preset` | engine name | voice ID string | None |
|
||||
| `designed` | NULL | NULL | None (uses `design_prompt`) |
|
||||
|
||||
The `default_engine` column is set automatically when the profile is created. For preset profiles it's locked to the source engine — switching engines at generation time will skip the profile (and the UI auto-switches back when the user clicks a greyed-out card; see the floating generate box and profile grid).
|
||||
|
||||
### ProfileSample Table
|
||||
|
||||
```python
|
||||
class ProfileSample(Base):
|
||||
__tablename__ = "profile_samples"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"))
|
||||
audio_path = Column(String, nullable=False)
|
||||
reference_text = Column(Text, nullable=False)
|
||||
```
|
||||
|
||||
Only populated for cloned profiles. Preset and designed profiles have zero rows in this table.
|
||||
|
||||
## File Structure
|
||||
|
||||
Profiles are stored in the data directory:
|
||||
@@ -152,8 +179,8 @@ async def create_voice_prompt_for_profile(
|
||||
Reference audio is validated before being accepted:
|
||||
|
||||
- **Duration:** 3-30 seconds recommended
|
||||
- **Format:** WAV, MP3, FLAC, OGG supported
|
||||
- **Sample Rate:** Resampled to 24kHz
|
||||
- **Format:** WAV, MP3, FLAC, OGG, M4A supported
|
||||
- **Sample Rate:** Engine-specific — the audio utility resamples to whatever the active engine expects (Whisper uses 16 kHz, most TTS engines use 24 kHz, LuxTTS outputs 48 kHz). Resampling happens on the fly; the stored sample retains its original rate.
|
||||
- **Channels:** Converted to mono if stereo
|
||||
|
||||
## Export/Import
|
||||
|
||||
@@ -3,15 +3,16 @@ title: "Voicebox Documentation"
|
||||
description: "Voicebox is a local-first voice cloning studio -- a free and open-source alternative to ElevenLabs."
|
||||
---
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 5 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
Voicebox is a **local-first voice cloning studio** -- a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
|
||||

|
||||
|
||||
- **Complete privacy** -- models and voice data stay on your machine
|
||||
- **5 TTS engines** -- Qwen3-TTS, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, and HumeAI TADA
|
||||
- **7 TTS engines** -- Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** -- zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
|
||||
- **23 languages** -- from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** -- pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo
|
||||
- **Expressive speech** -- paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** -- auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** -- multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** -- REST API for integrating voice synthesis into your own projects
|
||||
@@ -31,6 +32,6 @@ Voicebox is a **local-first voice cloning studio** -- a free and open-source alt
|
||||
|
||||
## Get Started
|
||||
|
||||
- [Installation](/docs/overview/installation) -- download and install Voicebox
|
||||
- [Quick Start](/docs/overview/quick-start) -- get up and running in 5 minutes
|
||||
- [API Reference](/docs/api-reference) -- integrate voice synthesis into your apps
|
||||
- [Installation](/overview/installation) -- download and install Voicebox
|
||||
- [Quick Start](/overview/quick-start) -- get up and running in 5 minutes
|
||||
- [API Reference](/api-reference) -- integrate voice synthesis into your apps
|
||||
|
||||
@@ -1,32 +1,43 @@
|
||||
---
|
||||
title: "Creating Voice Profiles"
|
||||
description: "Advanced guide to creating high-quality voice profiles"
|
||||
description: "How to create voice profiles, both cloning-based and preset-based"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voice profiles are the foundation of voice cloning in Voicebox. This guide covers best practices for creating professional-quality voice profiles.
|
||||
A **voice profile** is a saved voice you can reuse across generations, stories, and the API. As of 0.4, Voicebox profiles come in two flavors that map to two different ways of getting a voice:
|
||||
|
||||
## Quick Start
|
||||
| Profile type | What it stores | Use when… |
|
||||
| -------------- | ---------------------------------------------------- | -------------------------------------------------------- |
|
||||
| **Cloned** | One or more reference audio samples + a voice embedding | You want to replicate a specific person's voice |
|
||||
| **Preset** | A reference to a pre-built voice in a specific engine | You want a curated, production-ready voice with no audio prep |
|
||||
|
||||
Both types live in the same Profiles tab and behave the same way at generation time — pick the type that matches your goal and follow the workflow below.
|
||||
|
||||
<Callout type="info">
|
||||
Not sure which to use? Cloning gives you a *specific* voice but needs clean audio. Preset gives you *good* voices instantly but you don't get to choose who they sound like.
|
||||
</Callout>
|
||||
|
||||
## Workflow A — Cloned Profiles
|
||||
|
||||
Use this when you want to replicate a specific person's voice from a recording.
|
||||
|
||||
<Steps>
|
||||
<Step title="Prepare Audio">
|
||||
10-30 seconds of clear speech
|
||||
10-30 seconds of clear speech, minimal background noise. See [Voice Cloning](/overview/voice-cloning) for the engine catalog.
|
||||
</Step>
|
||||
<Step title="Create Profile">
|
||||
**Profiles** → **+ New Profile**
|
||||
**Profiles** → **+ New Profile** → choose a cloning engine (Qwen3-TTS, Chatterbox Multilingual, Chatterbox Turbo, LuxTTS, or TADA)
|
||||
</Step>
|
||||
<Step title="Upload Sample">
|
||||
Add your audio file
|
||||
<Step title="Upload or Record Sample">
|
||||
Drag in an audio file, or record directly with the in-app recorder
|
||||
</Step>
|
||||
<Step title="Generate">
|
||||
Use the profile to generate speech
|
||||
<Step title="Generate to Test">
|
||||
Use the profile to generate a test phrase. If quality is poor, add more samples
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Audio Requirements
|
||||
|
||||
### Ideal Sample Characteristics
|
||||
### Audio Requirements (Cloning Only)
|
||||
|
||||
<Cards>
|
||||
<Card title="Duration">
|
||||
@@ -44,7 +55,7 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
|
||||
<Card title="Quality">
|
||||
**High fidelity**
|
||||
|
||||
44.1kHz or 48kHz sample rate
|
||||
44.1 kHz or 48 kHz sample rate
|
||||
Minimal compression
|
||||
</Card>
|
||||
<Card title="Content">
|
||||
@@ -58,18 +69,16 @@ Voice profiles are the foundation of voice cloning in Voicebox. This guide cover
|
||||
### File Formats
|
||||
|
||||
Supported formats:
|
||||
- **WAV** (recommended) - Lossless quality
|
||||
- **MP3** - Acceptable, minimal compression
|
||||
- **M4A** - Acceptable
|
||||
- **FLAC** - Lossless alternative
|
||||
- **WAV** (recommended) — Lossless quality
|
||||
- **MP3** — Acceptable, minimal compression
|
||||
- **M4A** — Acceptable
|
||||
- **FLAC** — Lossless alternative
|
||||
|
||||
<Callout type="info">
|
||||
Use WAV for best results. Avoid heavily compressed formats.
|
||||
</Callout>
|
||||
|
||||
## Recording Tips
|
||||
|
||||
### Environment
|
||||
### Recording Tips
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Quiet Space">
|
||||
@@ -87,27 +96,25 @@ Supported formats:
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Recording Settings">
|
||||
- 44.1kHz or 48kHz sample rate
|
||||
- 44.1 kHz or 48 kHz sample rate
|
||||
- 16-bit or 24-bit depth
|
||||
- Mono is fine (stereo will be converted)
|
||||
- Avoid automatic gain control
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### Speaking
|
||||
### Speaking Style
|
||||
|
||||
- **Natural pace** - Don't rush or speak too slowly
|
||||
- **Clear articulation** - Pronounce words clearly
|
||||
- **Consistent volume** - Maintain steady loudness
|
||||
- **Normal tone** - Speak as you normally would
|
||||
- **Complete sentences** - Avoid fragments or "ums"
|
||||
- **Natural pace** — Don't rush or speak too slowly
|
||||
- **Clear articulation** — Pronounce words clearly
|
||||
- **Consistent volume** — Maintain steady loudness
|
||||
- **Normal tone** — Speak as you normally would
|
||||
- **Complete sentences** — Avoid fragments or "ums"
|
||||
|
||||
## Multiple Samples
|
||||
### Multiple Samples
|
||||
|
||||
Adding multiple samples can significantly improve quality:
|
||||
|
||||
### Why Multiple Samples?
|
||||
|
||||
<Cards>
|
||||
<Card title="Robustness">
|
||||
Model learns a more complete representation
|
||||
@@ -123,110 +130,57 @@ Adding multiple samples can significantly improve quality:
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
### Sample Variety
|
||||
|
||||
Consider adding samples with:
|
||||
|
||||
1. **Different tones**
|
||||
- Casual conversation
|
||||
- Professional/formal
|
||||
- Excited/enthusiastic
|
||||
- Calm/serious
|
||||
|
||||
2. **Different content**
|
||||
- Narratives
|
||||
- Questions
|
||||
- Statements
|
||||
- Emotions (happy, sad, neutral)
|
||||
|
||||
3. **Different recording conditions**
|
||||
- Studio quality
|
||||
- Phone call quality (if needed)
|
||||
- Room acoustics
|
||||
1. **Different tones** — casual, formal, excited, calm
|
||||
2. **Different content** — narratives, questions, statements
|
||||
3. **Different recording conditions** — studio quality, room acoustics
|
||||
|
||||
<Callout type="warn">
|
||||
All samples should be from the **same speaker**. Mixing voices will produce poor results.
|
||||
</Callout>
|
||||
|
||||
## Processing Existing Audio
|
||||
### Processing Existing Audio
|
||||
|
||||
If you have existing audio (podcasts, videos, etc.):
|
||||
|
||||
### Extracting Clean Segments
|
||||
|
||||
<Steps>
|
||||
<Step title="Find Clean Speech">
|
||||
Look for segments with:
|
||||
- Just the target speaker
|
||||
- No background music
|
||||
- Minimal noise
|
||||
Look for segments with just the target speaker, no background music, minimal noise
|
||||
</Step>
|
||||
|
||||
<Step title="Use Audio Editor">
|
||||
Tools like Audacity or Adobe Audition:
|
||||
- Cut out clean 10-30s segments
|
||||
- Remove silence at start/end
|
||||
- Normalize volume if needed
|
||||
Tools like Audacity or Adobe Audition: cut clean 10-30s segments, remove silence at start/end, normalize volume
|
||||
</Step>
|
||||
|
||||
<Step title="Export as WAV">
|
||||
Save as high-quality WAV file
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Noise Reduction
|
||||
For light background noise, use Audacity's noise reduction (gentle settings — over-processing introduces artifacts).
|
||||
|
||||
If you have light background noise:
|
||||
### Testing & Iteration
|
||||
|
||||
```
|
||||
1. Use noise reduction in Audacity:
|
||||
- Select noise-only section
|
||||
- Get Noise Profile
|
||||
- Select full audio
|
||||
- Apply noise reduction (gentle settings)
|
||||
|
||||
2. Avoid over-processing:
|
||||
- Can introduce artifacts
|
||||
- May reduce voice quality
|
||||
```
|
||||
|
||||
## Testing & Iteration
|
||||
|
||||
### Test Your Profile
|
||||
|
||||
After creating a profile:
|
||||
After creating a cloned profile:
|
||||
|
||||
<Steps>
|
||||
<Step title="Generate Test">
|
||||
Generate a simple phrase:
|
||||
```
|
||||
"Hello, this is a test of my voice profile."
|
||||
```
|
||||
Try a simple phrase: `"Hello, this is a test of my voice profile."`
|
||||
</Step>
|
||||
|
||||
<Step title="Evaluate Quality">
|
||||
Listen for:
|
||||
- Natural tone
|
||||
- Clear pronunciation
|
||||
- Proper prosody
|
||||
- Lack of artifacts
|
||||
Listen for natural tone, clear pronunciation, proper prosody, lack of artifacts
|
||||
</Step>
|
||||
|
||||
<Step title="Iterate">
|
||||
If quality is poor:
|
||||
- Add more samples
|
||||
- Try different source audio
|
||||
- Check sample quality
|
||||
If quality is poor: add more samples, try different source audio, check sample quality
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
### Common Issues
|
||||
#### Common Issues
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Robotic Voice">
|
||||
**Cause**: Poor quality samples or too short
|
||||
|
||||
**Fix**: Use longer, higher quality samples
|
||||
**Fix**: Use longer, higher-quality samples
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Wrong Tone">
|
||||
@@ -242,51 +196,89 @@ After creating a profile:
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Workflow B — Preset Profiles
|
||||
|
||||
Use this when you want a ready-made voice without recording anything. Available engines: **Kokoro 82M** (50 voices) and **Qwen CustomVoice** (9 voices). See [Preset Voices](/overview/preset-voices) for the full catalog.
|
||||
|
||||
<Steps>
|
||||
<Step title="Create Profile">
|
||||
**Profiles** → **+ New Profile** → choose **Kokoro** or **Qwen CustomVoice** as the engine
|
||||
</Step>
|
||||
<Step title="Pick a Voice">
|
||||
The engine's voice catalog appears. Click any voice to preview it
|
||||
</Step>
|
||||
<Step title="Name and Save">
|
||||
Give the profile a name. No audio sample required
|
||||
</Step>
|
||||
<Step title="Generate">
|
||||
The profile is ready immediately — use it in the floating generate box or Generate page
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
<Callout type="info">
|
||||
Preset profiles are **locked to their source engine**. Switching to a different engine in the floating generate box greys out the profile, since the voice only exists in that engine. Clicking a greyed profile auto-switches the engine back.
|
||||
</Callout>
|
||||
|
||||
### Qwen CustomVoice + Instruct
|
||||
|
||||
Preset voices in Qwen CustomVoice support **delivery instructions** — natural-language style control over tone, pace, and emotion. The floating generate box shows a slider icon next to the generate button when a Qwen CustomVoice profile is selected; click it to reveal the instruct textarea.
|
||||
|
||||
See [Preset Voices → Using Instruct Mode](/overview/preset-voices#using-instruct-mode) for examples.
|
||||
|
||||
## Advanced Tips
|
||||
|
||||
### Celebrity/Character Voices
|
||||
### Celebrity / Character Voices (Cloning)
|
||||
|
||||
For cloning public figures or characters:
|
||||
|
||||
1. **Legal considerations** - Ensure you have rights or it's fair use
|
||||
2. **Source quality** - Find high-quality interview audio or clean clips
|
||||
3. **Consistency** - Use clips where they speak similarly
|
||||
4. **Multiple samples** - Very important for recognizable voices
|
||||
1. **Legal considerations** — Ensure you have rights or it's clearly fair use
|
||||
2. **Source quality** — Find high-quality interview audio or clean clips
|
||||
3. **Consistency** — Use clips where they speak similarly
|
||||
4. **Multiple samples** — Very important for recognizable voices
|
||||
|
||||
### Accent & Dialect
|
||||
### Accent & Dialect (Cloning)
|
||||
|
||||
The model will preserve accent and dialect:
|
||||
Cloning models preserve accent and dialect:
|
||||
|
||||
- British English will generate British English
|
||||
- Southern accent will produce Southern accent
|
||||
- Regional pronunciations will be maintained
|
||||
- British English samples generate British English output
|
||||
- Southern accent samples produce Southern accent output
|
||||
- Regional pronunciations are maintained
|
||||
|
||||
### Emotion Transfer
|
||||
### Emotion Transfer (Cloning)
|
||||
|
||||
The emotional tone of samples affects generation:
|
||||
|
||||
- Energetic samples → Energetic output
|
||||
- Calm samples → Calm output
|
||||
- Mix samples for versatile profile
|
||||
- Energetic samples → energetic output
|
||||
- Calm samples → calm output
|
||||
- Mix samples for a more versatile profile
|
||||
|
||||
For Qwen CustomVoice presets, use the **instruct** field instead of relying on sample emotion — that's exactly what it controls.
|
||||
|
||||
## Managing Profiles
|
||||
|
||||
### Organization
|
||||
|
||||
- **Descriptive names** - "John Smith - Professional Narrator"
|
||||
- **Add descriptions** - Note recording conditions, use cases
|
||||
- **Language tags** - Mark the primary language
|
||||
- **Archive unused** - Keep profile list manageable
|
||||
- **Descriptive names** — "John Smith - Professional Narrator"
|
||||
- **Add descriptions** — Note recording conditions, use cases, or which preset voice
|
||||
- **Language tags** — Mark the primary language
|
||||
- **Archive unused** — Keep profile list manageable
|
||||
|
||||
### Export/Import
|
||||
### Export / Import
|
||||
|
||||
- **Export** profiles to share or backup
|
||||
- **Import** from colleagues or teammates
|
||||
- Profiles include voice embeddings, not original audio
|
||||
- **Cloned profiles** export with their voice embeddings (not the original audio)
|
||||
- **Preset profiles** export as engine + voice ID metadata only — the importer must have that engine's model installed
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Voice Cloning" href="/overview/voice-cloning">
|
||||
Engine catalog and best practices for cloning
|
||||
</Card>
|
||||
<Card title="Preset Voices" href="/overview/preset-voices">
|
||||
Full catalog of Kokoro and Qwen CustomVoice voices
|
||||
</Card>
|
||||
<Card title="Generate Speech" href="/overview/generating-speech">
|
||||
Use your profile to generate speech
|
||||
</Card>
|
||||
|
||||
@@ -79,9 +79,9 @@ Drag generations to the Stories Editor timeline.
|
||||
|
||||
History is stored locally:
|
||||
|
||||
- **macOS**: `~/Library/Application Support/com.voicebox.app/data/`
|
||||
- **Windows**: `%APPDATA%/com.voicebox.app/data/`
|
||||
- **Linux**: `~/.config/com.voicebox.app/data/`
|
||||
- **macOS**: `~/Library/Application Support/sh.voicebox.app/data/`
|
||||
- **Windows**: `%APPDATA%/sh.voicebox.app/data/`
|
||||
- **Linux**: `~/.config/sh.voicebox.app/data/`
|
||||
|
||||
<Callout type="warn">
|
||||
Deleting the data directory will remove all history. Export important files first.
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
---
|
||||
title: "GPU Acceleration"
|
||||
description: "How Voicebox uses your GPU — auto-detection, manual setup, troubleshooting"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox auto-detects available accelerators on first launch and picks the fastest backend it can use. For most people this just works — open the app and you're already on the right backend.
|
||||
|
||||
This page is for the cases where it doesn't:
|
||||
|
||||
- You have a GPU but Voicebox is running on CPU
|
||||
- You upgraded GPUs (especially to RTX 50-series / Blackwell) and generation broke
|
||||
- You want to switch backends manually (e.g. force MLX over PyTorch on Apple Silicon)
|
||||
- You see `[UNSUPPORTED - see logs]` next to your GPU in Settings
|
||||
|
||||
## Backend Matrix
|
||||
|
||||
| Platform | Auto-selected backend | Notes |
|
||||
| --------------------------- | ------------------------- | ---------------------------------------------------- |
|
||||
| **macOS Apple Silicon** | MLX (Metal) | 4-5x faster than PyTorch via Apple Neural Engine |
|
||||
| **macOS Intel** | PyTorch CPU | No GPU acceleration available; PyTorch ≥ 2.2 only |
|
||||
| **Windows + NVIDIA** | PyTorch CUDA (cu128) | Auto-downloads the CUDA backend binary on first use |
|
||||
| **Windows + Intel Arc** | PyTorch XPU (IPEX) | New in 0.4 — works with Arc A-series and B-series |
|
||||
| **Windows generic GPU** | DirectML | Universal Windows GPU support; slower than CUDA |
|
||||
| **Linux + NVIDIA** | PyTorch CUDA (cu128) | Same auto-download flow as Windows |
|
||||
| **Linux + AMD** | PyTorch ROCm | Auto-configures `HSA_OVERRIDE_GFX_VERSION` |
|
||||
| **Linux + Intel Arc** | PyTorch XPU (IPEX) | |
|
||||
| **Any (no GPU)** | PyTorch CPU | Works everywhere; expect 5-50x slower than GPU |
|
||||
|
||||
The detected backend is shown in Settings → GPU. Logs at startup also print the chosen backend and the device name.
|
||||
|
||||
## Apple Silicon — MLX vs PyTorch
|
||||
|
||||
On M-series Macs, Voicebox ships an MLX-optimized backend that uses the Apple Neural Engine. It's **4-5x faster** than the PyTorch (CPU/Metal) path for supported engines.
|
||||
|
||||
| Engine | MLX support | Notes |
|
||||
| -------------------- | ----------- | ------------------------------------------- |
|
||||
| Qwen3-TTS | ✅ Native | Uses MLX exclusively when available |
|
||||
| Chatterbox / Turbo | PyTorch MPS | Falls back to Metal via PyTorch |
|
||||
| LuxTTS | PyTorch MPS | |
|
||||
| TADA | PyTorch MPS | |
|
||||
| Kokoro | PyTorch MPS | Requires `PYTORCH_ENABLE_MPS_FALLBACK=1` |
|
||||
| Qwen CustomVoice | PyTorch MPS | |
|
||||
| Whisper (transcribe) | ✅ Native | MLX-Whisper is the default on Apple Silicon |
|
||||
|
||||
The Whisper Turbo + MLX combo dropped transcription latency from ~20s to ~2-3s on M-series chips (see CHANGELOG entry for v0.1.10).
|
||||
|
||||
## Windows / Linux + NVIDIA — The CUDA Backend Swap
|
||||
|
||||
Voicebox doesn't bundle CUDA into the main installer (it would balloon downloads to multi-gigabyte territory for users who don't have an NVIDIA GPU). Instead, when you first need it, the app downloads a separate **CUDA backend binary** that contains the PyTorch + CUDA runtime.
|
||||
|
||||
<Steps>
|
||||
<Step title="Open Settings → GPU">
|
||||
If an NVIDIA GPU is detected, you'll see "Install CUDA backend" in the GPU panel
|
||||
</Step>
|
||||
<Step title="Click Install">
|
||||
The app downloads two archives separately:
|
||||
- **Server core** (~200-400 MB) — versioned with each Voicebox release
|
||||
- **CUDA libs** (~4 GB) — the heavy PyTorch + CUDA DLLs, versioned independently
|
||||
</Step>
|
||||
<Step title="Restart">
|
||||
Voicebox restarts to swap in the CUDA backend
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
<Callout type="info">
|
||||
The split-archive design (added in v0.4) means most Voicebox upgrades only redownload the small server-core archive. The 4 GB libs archive is only refreshed when the underlying CUDA toolkit or torch major version changes.
|
||||
</Callout>
|
||||
|
||||
### Auto-update
|
||||
|
||||
When a new Voicebox release ships, the GPU panel checks if the bundled server-core matches the installed CUDA version. If only the core changed (typical), it pulls the new core in the background. If the libs version changed (rare — only happens on cu126 → cu128 type bumps), you'll be prompted to confirm the larger download.
|
||||
|
||||
## RTX 50-series / Blackwell
|
||||
|
||||
Voicebox 0.4 added explicit RTX 50-series support:
|
||||
|
||||
- CUDA toolkit upgraded to **cu128** (previous releases used cu126 which lacks Blackwell kernels)
|
||||
- Build pinned with `TORCH_CUDA_ARCH_LIST=...12.0+PTX` for forward-compatibility
|
||||
|
||||
If you're on an RTX 5070 / 5080 / 5090 and you see "no kernel image is available" errors:
|
||||
|
||||
1. Make sure you're on Voicebox **≥ 0.4.0** (Settings → About)
|
||||
2. Reinstall the CUDA backend (Settings → GPU → Reinstall CUDA backend) — older installs may have stale cu126 libs
|
||||
3. If errors persist, see the GPU compatibility warnings section below
|
||||
|
||||
## Intel Arc (XPU)
|
||||
|
||||
New in 0.4. Works with both Arc A-series (Alchemist: A380, A580, A750, A770) and B-series (Battlemage).
|
||||
|
||||
### Setup
|
||||
|
||||
Voicebox auto-detects Arc GPUs and routes through Intel's PyTorch XPU backend (powered by IPEX — Intel Extension for PyTorch). No extra installation step beyond the standard Voicebox install.
|
||||
|
||||
Verify it's working:
|
||||
- Settings → GPU should show **XPU** followed by your Arc model name (e.g. `XPU (Intel Arc A770)`)
|
||||
- Startup logs print `Backend: PYTORCH` and `GPU: XPU (Intel Arc ...)`
|
||||
|
||||
### Engines on XPU
|
||||
|
||||
All PyTorch-based engines work on XPU. Performance is generally between CPU and CUDA — expect ~2-3x speedup over CPU for the larger models.
|
||||
|
||||
## DirectML
|
||||
|
||||
The fallback for Windows users with non-NVIDIA, non-Intel-Arc GPUs (older AMD discrete, integrated GPUs, etc.). Slower than CUDA and XPU but provides some acceleration over CPU.
|
||||
|
||||
Auto-selected when no other GPU backend is available.
|
||||
|
||||
## AMD ROCm (Linux)
|
||||
|
||||
ROCm provides PyTorch GPU acceleration on AMD discrete GPUs. Voicebox auto-configures `HSA_OVERRIDE_GFX_VERSION` for common cards that need the override.
|
||||
|
||||
### Verifying
|
||||
|
||||
```bash
|
||||
# In a terminal
|
||||
echo $HSA_OVERRIDE_GFX_VERSION
|
||||
# Should show e.g. 10.3.0 for RX 6000 series
|
||||
```
|
||||
|
||||
If detection fails, set the variable manually before launching Voicebox:
|
||||
|
||||
```bash
|
||||
export HSA_OVERRIDE_GFX_VERSION=10.3.0
|
||||
voicebox
|
||||
```
|
||||
|
||||
Common values:
|
||||
- `10.3.0` — RX 6000 series (RDNA 2)
|
||||
- `11.0.0` — RX 7000 series (RDNA 3)
|
||||
- `9.0.0` — Older Vega cards
|
||||
|
||||
## GPU Compatibility Warnings
|
||||
|
||||
Voicebox 0.4 added a runtime check that compares your GPU's compute capability against the architectures the bundled PyTorch was compiled for. If they don't match, you'll see:
|
||||
|
||||
- A startup log line: `WARNING: GPU COMPATIBILITY: <your GPU> is not supported by this PyTorch build...`
|
||||
- The GPU label in Settings shows `[UNSUPPORTED - see logs]`
|
||||
- The `/health` API returns a populated `gpu_compatibility_warning` field
|
||||
|
||||
### What to do
|
||||
|
||||
The most common trigger is a brand-new GPU architecture that pre-built PyTorch wheels don't yet cover natively. In order of preference:
|
||||
|
||||
1. **Update Voicebox** — newer releases ship newer PyTorch with broader arch support
|
||||
2. **Reinstall the CUDA backend** — Settings → GPU → Reinstall CUDA backend
|
||||
3. **For bleeding-edge GPUs (newer than current Blackwell):** install PyTorch nightly manually:
|
||||
```bash
|
||||
pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128 --force-reinstall
|
||||
```
|
||||
Then point Voicebox at that environment via [Remote Mode](/overview/remote-mode) until stable PyTorch catches up.
|
||||
4. **Fall back to CPU** temporarily — set `VOICEBOX_FORCE_CPU=1` before launching
|
||||
|
||||
## CPU-Only Fallback
|
||||
|
||||
When no GPU is available (or you've forced it off), Voicebox runs the PyTorch CPU backend. Expect:
|
||||
|
||||
- 5-50x slower generation depending on engine and text length
|
||||
- Heavy CPU usage during generation
|
||||
- Some engines work better than others on CPU:
|
||||
- **Kokoro 82M** — runs at realtime on modern CPUs
|
||||
- **LuxTTS** — exceeds 150x realtime on CPU
|
||||
- **Chatterbox Turbo (350M)** — usable but slow
|
||||
- Larger models (Qwen 1.7B, Chatterbox Multilingual, TADA 3B) — painful
|
||||
|
||||
For CPU-bound use cases, prefer the smaller, lighter engines.
|
||||
|
||||
## Verifying Your Setup
|
||||
|
||||
Three places to check that the right backend is being used:
|
||||
|
||||
<Steps>
|
||||
<Step title="Settings → GPU">
|
||||
Shows the detected backend, GPU model, and VRAM (when applicable). Look for the `[UNSUPPORTED - see logs]` suffix
|
||||
</Step>
|
||||
<Step title="Settings → Logs">
|
||||
The "Server logs" tab shows the startup banner with `Backend: <type>` and `GPU: <name>`
|
||||
</Step>
|
||||
<Step title="Health endpoint">
|
||||
`curl http://localhost:17493/health` returns a JSON payload with `backend_type`, `backend_variant`, and `gpu_compatibility_warning` (when applicable)
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Settings shows CPU instead of my GPU">
|
||||
- On NVIDIA: install the CUDA backend (Settings → GPU)
|
||||
- On Intel Arc: confirm IPEX detection in startup logs; restart the app after a driver update
|
||||
- On AMD Linux: check `HSA_OVERRIDE_GFX_VERSION` is set
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="'no kernel image is available' / 'CUDA error'">
|
||||
Almost always means the bundled PyTorch doesn't have kernels for your GPU's compute capability.
|
||||
|
||||
1. Update to Voicebox ≥ 0.4.0 (Blackwell support added there)
|
||||
2. Reinstall the CUDA backend
|
||||
3. If still broken, install PyTorch nightly via Remote Mode
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Out of memory (CUDA)">
|
||||
- Switch to a smaller model size (e.g. Qwen3 0.6B instead of 1.7B)
|
||||
- Use Settings → Models to unload other engines you're not using
|
||||
- Enable `low_cpu_mem_usage` is already on for CPU; for CUDA, the engine's `device_map` handles offload automatically
|
||||
- Close other GPU applications
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="MPS fallback errors on macOS">
|
||||
Some operations don't have a Metal implementation. Voicebox sets `PYTORCH_ENABLE_MPS_FALLBACK=1` for engines that need it (notably Kokoro), but if you launch from a custom env, set it manually:
|
||||
```bash
|
||||
export PYTORCH_ENABLE_MPS_FALLBACK=1
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Generation works but is slow on my GPU">
|
||||
- Check Settings → GPU shows your GPU (not CPU)
|
||||
- Check VRAM usage — you may be paging to system memory
|
||||
- Try a smaller model
|
||||
- For NVIDIA: confirm cu128 is installed (Settings → GPU → version)
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="Remote Mode" href="/overview/remote-mode">
|
||||
Run the backend on a different machine with a stronger GPU
|
||||
</Card>
|
||||
<Card title="Model Management" href="/developer/model-management">
|
||||
Unload models to free GPU memory
|
||||
</Card>
|
||||
<Card title="Troubleshooting" href="/overview/troubleshooting">
|
||||
General troubleshooting beyond GPU
|
||||
</Card>
|
||||
</Cards>
|
||||
@@ -68,11 +68,11 @@ Voicebox is available for macOS and Windows, with Linux builds coming soon.
|
||||
|
||||
When you launch Voicebox for the first time:
|
||||
|
||||
1. **Model Download** — Qwen3-TTS model (~2-4GB) will download automatically on first use
|
||||
1. **Model Download** — The TTS engine you generate with first will download its model automatically. Sizes range from ~350 MB (Kokoro) to ~8 GB (TADA 3B). Most users start with Qwen 1.7B (~3.5 GB).
|
||||
2. **Data Directory** — Voice profiles and generated audio are stored in:
|
||||
- macOS: `~/Library/Application Support/com.voicebox.app/`
|
||||
- Windows: `%APPDATA%/com.voicebox.app/`
|
||||
- Linux: `~/.config/com.voicebox.app/`
|
||||
- macOS: `~/Library/Application Support/sh.voicebox.app/`
|
||||
- Windows: `%APPDATA%/sh.voicebox.app/`
|
||||
- Linux: `~/.config/sh.voicebox.app/`
|
||||
|
||||
3. **Backend Server** — The bundled Python server starts automatically
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user