mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-09-26 13:45:16 -07:00
Compare commits
167
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4ffbc03d15 | ||
|
|
a49cc6afbb | ||
|
|
e3f7cd9d00 | ||
|
|
27a5a62581 | ||
|
|
d3a44338a2 | ||
|
|
28aa963b09 | ||
|
|
ae91aa9a88 | ||
|
|
da6070155e | ||
|
|
3c1e8512b9 | ||
|
|
bf58750447 | ||
|
|
2d56309bdd | ||
|
|
0445be295c | ||
|
|
8d550a5f7c | ||
|
|
795bd54381 | ||
|
|
a6ab5f3858 | ||
|
|
9d7e4a417e | ||
|
|
882cabc7d2 | ||
|
|
9c76b5de2c | ||
|
|
4560b7378a | ||
|
|
abd9943430 | ||
|
|
c8cb12f1bc | ||
|
|
54a3bf322e | ||
|
|
476abe07fc | ||
|
|
67bf8e906a | ||
|
|
625e1ba549 | ||
|
|
cfe6770639 | ||
|
|
00452b51a8 | ||
|
|
106aec46a8 | ||
|
|
c9e5c5d9a7 | ||
|
|
48cd1f369a | ||
|
|
2bfe400457 | ||
|
|
0aa19a9994 | ||
|
|
73170d0e92 | ||
|
|
0317626677 | ||
|
|
a5d5c780c2 | ||
|
|
7184a25e44 | ||
|
|
479bc7fc5e | ||
|
|
3e7727d1d2 | ||
|
|
be7c0cec12 | ||
|
|
c9d8142a78 | ||
|
|
13ba5f1aa6 | ||
|
|
9a3c307c75 | ||
|
|
1da16cfc57 | ||
|
|
a1807be04d | ||
|
|
fdba18e9ee | ||
|
|
07a845cece | ||
|
|
615d604ceb | ||
|
|
a383ff6863 | ||
|
|
75abbb02c3 | ||
|
|
b49f14a814 | ||
|
|
05686efbfd | ||
|
|
e2c03fef9a | ||
|
|
4347eaed4c | ||
|
|
60aac279ce | ||
|
|
8b1c7552be | ||
|
|
9a955a77d2 | ||
|
|
c18591c0c3 | ||
|
|
ea3469f2dc | ||
|
|
b108bb1cb1 | ||
|
|
8b796bc6b4 | ||
|
|
e6f419cd70 | ||
|
|
72c13fd3fc | ||
|
|
4e0c731db8 | ||
|
|
d70b878b71 | ||
|
|
a71011741d | ||
|
|
d6f48ace3e | ||
|
|
0fc2192204 | ||
|
|
9e726ad048 | ||
|
|
3584283d84 | ||
|
|
e4def9365f | ||
|
|
707046237c | ||
|
|
12ed2d51ce | ||
|
|
83ebababe7 | ||
|
|
eb5869e59f | ||
|
|
2e95b7c5d8 | ||
|
|
ffc1b54812 | ||
|
|
fc5ed1ff40 | ||
|
|
c9f38dd496 | ||
|
|
58b19e4e9f | ||
|
|
0245c31dba | ||
|
|
81864e831a | ||
|
|
7bd72ea9f7 | ||
|
|
f96eae2567 | ||
|
|
7d53699c96 | ||
|
|
28e91ce2c1 | ||
|
|
88be097b62 | ||
|
|
564d787927 | ||
|
|
2c1ee94891 | ||
|
|
e789c937ad | ||
|
|
273483ffcf | ||
|
|
5774a168a9 | ||
|
|
6bf40bd2d0 | ||
|
|
12cda2e090 | ||
|
|
7a90290a76 | ||
|
|
b02ce8e2f3 | ||
|
|
4e7772a21d | ||
|
|
51fb320b8c | ||
|
|
8ac202aa58 | ||
|
|
ac68052945 | ||
|
|
e601fd2ca4 | ||
|
|
7b25e0ba0b | ||
|
|
a6817cd082 | ||
|
|
df50b8a925 | ||
|
|
a672ac5279 | ||
|
|
d35e6f0cc5 | ||
|
|
b1069b4521 | ||
|
|
f9e1aa153d | ||
|
|
01800f196f | ||
|
|
606da1c894 | ||
|
|
664178f0cf | ||
|
|
f1541701fb | ||
|
|
a2adc3b506 | ||
|
|
15ba824472 | ||
|
|
2ad4776a76 | ||
|
|
a8469b39f1 | ||
|
|
7dd70a52e4 | ||
|
|
1526f2de26 | ||
|
|
2c63dfff25 | ||
|
|
e0a798dc0d | ||
|
|
5933cba8e9 | ||
|
|
1b2d492398 | ||
|
|
faa825290f | ||
|
|
4a8a9eac14 | ||
|
|
3e4d9ff641 | ||
|
|
192979a762 | ||
|
|
e16cc42d53 | ||
|
|
f10e965003 | ||
|
|
a8968d4081 | ||
|
|
7c4afbe4df | ||
|
|
a180fcc56f | ||
|
|
1860b8dc92 | ||
|
|
1597937535 | ||
|
|
ac41a89359 | ||
|
|
60c0fe3b92 | ||
|
|
c99828cf76 | ||
|
|
5c4b979480 | ||
|
|
2d1b0ae820 | ||
|
|
69486c2a77 | ||
|
|
e9f63d6c57 | ||
|
|
8906bee23e | ||
|
|
0dabb121c9 | ||
|
|
944ba227ca | ||
|
|
473bb3e9fb | ||
|
|
0d0b62ea93 | ||
|
|
798cd40f05 | ||
|
|
3187344f01 | ||
|
|
8efcc95606 | ||
|
|
87cab9473d | ||
|
|
7b0fbfb567 | ||
|
|
7c1ea0a1e1 | ||
|
|
b3012ed10c | ||
|
|
88536d27f7 | ||
|
|
89d6e364d4 | ||
|
|
b7781951df | ||
|
|
fe19a9ca47 | ||
|
|
439fedcbf2 | ||
|
|
0813a3d9d6 | ||
|
|
9514c6596c | ||
|
|
4e84415da7 | ||
|
|
82cd4bf2ef | ||
|
|
3c30c5bec1 | ||
|
|
c9d7bc4f27 | ||
|
|
2e6efa00a2 | ||
|
|
788a04f265 | ||
|
|
5cb54ee03c | ||
|
|
0922845101 | ||
|
|
64dd29d35a |
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: add-tts-engine
|
||||
description: Use this skill to add a new TTS engine to Voicebox. It walks through dependency research, backend implementation, frontend wiring, PyInstaller bundling, and frozen-build testing. Always start with Phase 0 (dependency audit) before writing any code.
|
||||
---
|
||||
|
||||
# Add TTS Engine
|
||||
|
||||
## Goal
|
||||
|
||||
Integrate a new text-to-speech engine into Voicebox end-to-end: dependency research, backend protocol implementation, frontend UI wiring, PyInstaller bundling, and frozen-build verification. The user should only need to test the final build locally.
|
||||
|
||||
## Reference Doc
|
||||
|
||||
The full phased guide lives at `docs/content/docs/developer/tts-engines.mdx`. **Read this file in its entirety before starting.** It contains:
|
||||
|
||||
- Phase 0: Dependency research (mandatory before writing code)
|
||||
- Phase 1: Backend implementation (`TTSBackend` protocol)
|
||||
- Phase 2: Route and service integration (usually zero changes)
|
||||
- Phase 3: Frontend integration (5 files)
|
||||
- Phase 4: Dependencies (`requirements.txt`, justfile, CI, Docker)
|
||||
- Phase 5: PyInstaller bundling (`build_binary.py` + `server.py`)
|
||||
- Phase 6: Common upstream workarounds
|
||||
- Implementation checklist (gate between phases)
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Read the guide
|
||||
|
||||
```bash
|
||||
# Read the full TTS engines doc
|
||||
cat docs/content/docs/developer/tts-engines.mdx
|
||||
```
|
||||
|
||||
Internalize all phases, especially Phase 0 and Phase 5. The v0.2.3 release was three patch releases because Phase 0 was skipped.
|
||||
|
||||
### 2. Dependency research (Phase 0)
|
||||
|
||||
Clone the model library into a temporary directory and audit it. Do NOT skip this.
|
||||
|
||||
```bash
|
||||
mkdir /tmp/engine-research && cd /tmp/engine-research
|
||||
git clone <model-library-url>
|
||||
```
|
||||
|
||||
Run the grep searches from Phase 0.2 in the guide against the cloned source and its transitive dependencies. Produce a written dependency audit covering:
|
||||
|
||||
1. PyPI vs non-PyPI packages
|
||||
2. PyInstaller directives needed (`--collect-all`, `--copy-metadata`, `--hidden-import`)
|
||||
3. Runtime data files that must be bundled
|
||||
4. Native library paths that need env var overrides in frozen builds
|
||||
5. Monkey-patches needed (`torch.load`, float64, MPS, HF token)
|
||||
6. Sample rate
|
||||
7. Model download method (`from_pretrained` vs `snapshot_download` + `from_local`)
|
||||
|
||||
Test model loading and generation on CPU in the throwaway venv before proceeding.
|
||||
|
||||
### 3. Implement (Phases 1–4)
|
||||
|
||||
Follow the guide's phases in order. Key files to modify:
|
||||
|
||||
**Backend (Phase 1):**
|
||||
- Create `backend/backends/<engine>_backend.py`
|
||||
- Register in `backend/backends/__init__.py` (ModelConfig + TTS_ENGINES + factory)
|
||||
- Update regex in `backend/models.py`
|
||||
|
||||
**Frontend (Phase 3):**
|
||||
- `app/src/lib/api/types.ts` — engine union type
|
||||
- `app/src/lib/constants/languages.ts` — ENGINE_LANGUAGES
|
||||
- `app/src/components/Generation/EngineModelSelector.tsx` — ENGINE_OPTIONS, ENGINE_DESCRIPTIONS
|
||||
- `app/src/lib/hooks/useGenerationForm.ts` — Zod schema, model-name mapping
|
||||
- `app/src/components/ServerSettings/ModelManagement.tsx` — MODEL_DESCRIPTIONS
|
||||
|
||||
**Dependencies (Phase 4):**
|
||||
- `backend/requirements.txt`
|
||||
- `justfile` (setup-python, setup-python-release targets)
|
||||
- `.github/workflows/release.yml`
|
||||
- `Dockerfile` (if applicable)
|
||||
|
||||
### 4. PyInstaller bundling (Phase 5)
|
||||
|
||||
Register the engine in `backend/build_binary.py`:
|
||||
- `--hidden-import` for the backend module and model package
|
||||
- `--collect-all` for packages using `inspect.getsource`, shipping data files, or native libraries
|
||||
- `--copy-metadata` for packages using `importlib.metadata`
|
||||
|
||||
If the engine has native data paths, add `os.environ.setdefault()` in `backend/server.py` inside the `if getattr(sys, 'frozen', False):` block.
|
||||
|
||||
### 5. Verify in dev mode
|
||||
|
||||
```bash
|
||||
just dev
|
||||
```
|
||||
|
||||
Test the full chain: model download → load → generate → voice cloning.
|
||||
|
||||
### 6. Use the checklist
|
||||
|
||||
Walk through the Implementation Checklist at the bottom of `tts-engines.mdx`. Every item must be checked before handing the build to the user.
|
||||
|
||||
## Key Lessons (from v0.2.3)
|
||||
|
||||
These are the most common failure modes. Phase 0 research catches all of them:
|
||||
|
||||
| Pattern | Symptom in Frozen Build | Fix |
|
||||
|---------|------------------------|-----|
|
||||
| `@typechecked` / `inspect.getsource()` | "could not get source code" | `--collect-all <package>` |
|
||||
| Package ships pretrained model files | `FileNotFoundError` for `.pth.tar`, `.yaml` | `--collect-all <package>` |
|
||||
| C library with hardcoded system paths | `FileNotFoundError` for `/usr/share/...` | `--collect-all` + env var in `server.py` |
|
||||
| `importlib.metadata.version()` | "No package metadata found" | `--copy-metadata <package>` |
|
||||
| `torch.load` without `map_location` | CUDA device not available on CPU build | Monkey-patch `torch.load` |
|
||||
| `torch.from_numpy` on float64 data | dtype mismatch RuntimeError | Cast to `.float()` |
|
||||
| `token=True` in HF download calls | Auth failure without stored HF token | Use `snapshot_download(token=None)` + `from_local()` |
|
||||
|
||||
## Notes
|
||||
|
||||
- The route and service layers have zero per-engine dispatch points. `main.py` requires zero changes.
|
||||
- The model config registry in `backends/__init__.py` handles all dispatch automatically.
|
||||
- Use `get_torch_device()` and `model_load_progress()` from `backends/base.py` — don't reimplement device detection or progress tracking.
|
||||
- Always test with a **clean HuggingFace cache** (no pre-downloaded models from dev).
|
||||
- Do NOT push or create a release. Hand the build to the user for local testing.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: draft-release-notes
|
||||
description: Use this skill to draft or update the [Unreleased] section of CHANGELOG.md from the actual changes since the last tag. Run this at any point during development to keep a working copy of the release narrative. Does NOT bump versions or create tags.
|
||||
---
|
||||
|
||||
# Draft Release Notes
|
||||
|
||||
## Goal
|
||||
|
||||
Update the `[Unreleased]` section at the top of `CHANGELOG.md` with a narrative release story based on the real changes since the last tag. This is a **non-destructive working copy** — run it as many times as you want during development.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Identify the last release tag and gather changes.**
|
||||
|
||||
```bash
|
||||
LAST_TAG=$(git tag --list "v*" --sort=-v:refname | head -n 1)
|
||||
echo "Last tag: $LAST_TAG"
|
||||
```
|
||||
|
||||
Then collect raw material from three sources:
|
||||
|
||||
a. **Commit log since last tag:**
|
||||
```bash
|
||||
git log --oneline "$LAST_TAG"..HEAD
|
||||
```
|
||||
|
||||
b. **GitHub-generated release notes preview** (PR titles, new contributors):
|
||||
```bash
|
||||
gh api repos/:owner/:repo/releases/generate-notes \
|
||||
-f tag_name="vNEXT" \
|
||||
-f target_commitish="$(git rev-parse HEAD)" \
|
||||
-f previous_tag_name="$LAST_TAG" \
|
||||
--jq '.body'
|
||||
```
|
||||
|
||||
c. **Diff stat for theme analysis:**
|
||||
```bash
|
||||
git diff --stat "$LAST_TAG"..HEAD
|
||||
```
|
||||
|
||||
2. **Draft the release narrative.**
|
||||
|
||||
Write markdown for the `[Unreleased]` section following the format below. Do not include the `## [Unreleased]` heading itself — just the body content.
|
||||
|
||||
3. **Update CHANGELOG.md.**
|
||||
|
||||
Replace everything between `## [Unreleased]` and the next `## [` heading with the new draft. Preserve the HTML comment header and all existing release sections below.
|
||||
|
||||
The `[Unreleased]` section must always exist and always be the first section after the header comments.
|
||||
|
||||
4. **Do NOT commit, tag, or bump versions.** Just leave the file modified in the working tree.
|
||||
|
||||
## Release Story Format
|
||||
|
||||
Structure the `[Unreleased]` section like this:
|
||||
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
<One strong opening paragraph: what this release is about and why it matters.
|
||||
Tie it to concrete shipped changes. No vague hype.>
|
||||
|
||||
<One paragraph on major technical shifts, if applicable.>
|
||||
|
||||
### <Feature/Theme Group>
|
||||
- Bullet points with specifics
|
||||
- Reference PRs where available: ([#123](https://github.com/jamiepine/voicebox/pull/123))
|
||||
|
||||
### <Another Group>
|
||||
- ...
|
||||
|
||||
### Bug Fixes
|
||||
- ...
|
||||
```
|
||||
|
||||
### Style Guidelines
|
||||
|
||||
- **Factual and specific.** Every claim should trace to a real commit or PR.
|
||||
- **Narrative over list.** Lead with paragraphs that tell the story, then support with bullets.
|
||||
- **Group by theme, not by commit.** Cluster related changes under descriptive headings.
|
||||
- **Reference PRs** where they exist, but don't fabricate them.
|
||||
- **Skip trivial chores** (typo fixes, CI tweaks) unless they're the bulk of the release.
|
||||
- **Match the voice of existing releases** — look at the v0.2.1 and v0.2.3 entries in CHANGELOG.md for tone reference.
|
||||
|
||||
## When There Are No Changes
|
||||
|
||||
If `git log "$LAST_TAG"..HEAD` is empty, leave the `[Unreleased]` section empty (just the heading) and tell the user there's nothing to draft.
|
||||
|
||||
## Notes
|
||||
|
||||
- This skill only touches the `[Unreleased]` section. It never modifies stamped release sections.
|
||||
- The agent can be asked to run this skill at any point — mid-feature, before a PR, or right before cutting a release.
|
||||
- The `release-bump` skill depends on this draft being up to date before it finalizes.
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: release-bump
|
||||
description: Use this skill to finalize a release. It stamps the [Unreleased] changelog section with a version and date, runs bumpversion to update all version files, and creates the release commit and tag. Only run this when you're ready to ship.
|
||||
---
|
||||
|
||||
# Release Bump
|
||||
|
||||
## Goal
|
||||
|
||||
Finalize the changelog draft, bump the version across all tracked files, and create a tagged release commit. After this skill runs, the repo has a clean release commit and tag ready to push.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `gh` CLI installed and authenticated (`gh auth status`).
|
||||
- `bumpversion` installed (`pip install bumpversion` or available in the project venv).
|
||||
- The `[Unreleased]` section of `CHANGELOG.md` should already contain the release narrative. If it's empty or stale, run the `draft-release-notes` skill first.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Verify the working tree is clean** (except `CHANGELOG.md` which may have the draft).
|
||||
|
||||
```bash
|
||||
git status --porcelain
|
||||
```
|
||||
|
||||
Only `CHANGELOG.md` (and optionally `.agents/` files) should be modified. If there are other uncommitted changes, stop and ask the user to commit or stash them first.
|
||||
|
||||
2. **Determine the bump level.**
|
||||
|
||||
Ask the user if not specified: `patch`, `minor`, or `major`. Check the current version:
|
||||
|
||||
```bash
|
||||
grep '^current_version' .bumpversion.cfg
|
||||
```
|
||||
|
||||
3. **Stamp the changelog.**
|
||||
|
||||
Read the current `[Unreleased]` content from `CHANGELOG.md`. Compute the new version (based on bump level and current version). Then:
|
||||
|
||||
a. Replace the `## [Unreleased]` section body with an empty placeholder.
|
||||
b. Insert a new stamped section immediately after `## [Unreleased]`:
|
||||
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
## [X.Y.Z] - YYYY-MM-DD
|
||||
|
||||
<the content that was in [Unreleased]>
|
||||
```
|
||||
|
||||
c. Update the reference links at the bottom of the file:
|
||||
- Change the `[Unreleased]` link to compare against the new tag
|
||||
- Add a new link for the new version
|
||||
|
||||
```markdown
|
||||
[Unreleased]: https://github.com/jamiepine/voicebox/compare/vX.Y.Z...HEAD
|
||||
[X.Y.Z]: https://github.com/jamiepine/voicebox/compare/vPREVIOUS...vX.Y.Z
|
||||
```
|
||||
|
||||
4. **Stage the changelog.**
|
||||
|
||||
```bash
|
||||
git add CHANGELOG.md
|
||||
```
|
||||
|
||||
5. **Run bumpversion.**
|
||||
|
||||
```bash
|
||||
bumpversion --allow-dirty <patch|minor|major>
|
||||
```
|
||||
|
||||
The `--allow-dirty` flag is needed because `CHANGELOG.md` is already staged. bumpversion will:
|
||||
- Update version strings in all tracked files (see `.bumpversion.cfg`)
|
||||
- Create a commit with message `Bump version: X.Y.Z -> A.B.C`
|
||||
- Create a tag `vA.B.C`
|
||||
|
||||
The staged `CHANGELOG.md` will be included in this commit automatically.
|
||||
|
||||
6. **Verify results.**
|
||||
|
||||
```bash
|
||||
git show --name-only --stat HEAD
|
||||
git tag --list "v*" --sort=-v:refname | head -n 5
|
||||
```
|
||||
|
||||
Confirm the commit contains:
|
||||
- `CHANGELOG.md`
|
||||
- `.bumpversion.cfg`
|
||||
- `tauri/src-tauri/tauri.conf.json`
|
||||
- `tauri/src-tauri/Cargo.toml`
|
||||
- `package.json`
|
||||
- `app/package.json`
|
||||
- `tauri/package.json`
|
||||
- `landing/package.json`
|
||||
- `web/package.json`
|
||||
- `backend/__init__.py`
|
||||
|
||||
Confirm the new tag exists.
|
||||
|
||||
7. **Do NOT push** unless the user explicitly asks. Report the tag name and suggest:
|
||||
|
||||
```
|
||||
Ready to push. When you're ready:
|
||||
git push origin main --follow-tags
|
||||
```
|
||||
|
||||
## Version Calculation Reference
|
||||
|
||||
Given current version `X.Y.Z`:
|
||||
- `patch` -> `X.Y.(Z+1)`
|
||||
- `minor` -> `X.(Y+1).0`
|
||||
- `major` -> `(X+1).0.0`
|
||||
|
||||
## Error Recovery
|
||||
|
||||
- If bumpversion fails, the tag won't exist. Fix the issue and re-run — bumpversion is idempotent as long as the tag doesn't already exist.
|
||||
- If you need to undo a release commit (before pushing): `git tag -d vX.Y.Z && git reset --soft HEAD~1`
|
||||
- Never amend a release commit that has been pushed.
|
||||
|
||||
## Notes
|
||||
|
||||
- When the tag is pushed, the release CI (`.github/workflows/release.yml`) automatically extracts the matching version section from `CHANGELOG.md` and uses it as the GitHub Release body. No manual copy-paste needed.
|
||||
- The release commit message is controlled by `.bumpversion.cfg` (`Bump version: X.Y.Z -> A.B.C`). Do not override it.
|
||||
- If you need to manually update the GitHub Release body after the fact: `gh release edit vX.Y.Z --notes-file <(sed -n '/## \[X.Y.Z\]/,/## \[/p' CHANGELOG.md | head -n -1)`
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
name: triage-prs
|
||||
description: Use this skill to triage the open PR queue before a release. Classifies every open PR into must-merge, candidate, superseded, or deferred; writes a working triage doc; and runs the merge loop end-to-end. Designed for the pre-release "PR speedrun" pass where a solo maintainer wants to clear the inbound backlog in a single session.
|
||||
---
|
||||
|
||||
# Triage PRs
|
||||
|
||||
## Goal
|
||||
|
||||
Turn a backlog of open PRs into a shipped set of merges in a single focused session. Produce a tracked, resumable plan (`<VERSION>_PR_TRIAGE.md`), then work it — rebasing where needed, merging in isolation-safe batches, applying post-merge follow-ups, and closing superseded or partially-applicable PRs with credit to their authors.
|
||||
|
||||
This skill pairs with `draft-release-notes` and `release-bump`: triage first, then draft notes against the new main, then cut the release.
|
||||
|
||||
## When to use
|
||||
|
||||
- Before a minor or major release when 10+ open PRs have accumulated
|
||||
- When you want to unblock merging without losing the narrative of what's landing
|
||||
- When you know you can't personally review every PR deeply, but need to land the critical subset fast
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `gh` CLI authenticated against the repo
|
||||
- A dedicated worktree for PR review (avoid contaminating `main` with checkouts of contributor branches)
|
||||
- Clarity on the target version — the triage doc is named after it (e.g. `0.4.0_PR_TRIAGE.md`)
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Set up an isolated PR-review worktree
|
||||
|
||||
```bash
|
||||
git worktree list # check for stale ones first
|
||||
git worktree prune
|
||||
git worktree add ../voicebox-pr-review -b pr-review-<VERSION> main
|
||||
```
|
||||
|
||||
Keep the main worktree for release-prep work (changelog drafts, direct-to-main follow-ups). Keep the review worktree for `gh pr checkout` — each checkout moves HEAD to a contributor branch, which you don't want to do in the main worktree.
|
||||
|
||||
### 2. Gather metadata for every open PR
|
||||
|
||||
```bash
|
||||
gh pr list --state open --limit 50 --json \
|
||||
number,title,author,isDraft,mergeable,mergeStateStatus,files,additions,deletions,reviewDecision,statusCheckRollup,maintainerCanModify \
|
||||
--jq '.[] | {num: .number, title, author: .author.login, mergeable, state: .mergeStateStatus, canModify: .maintainerCanModify, changes: "+\(.additions)/-\(.deletions)", files: [.files[].path]}'
|
||||
```
|
||||
|
||||
You want, for each PR:
|
||||
- Size (`+additions/-deletions`)
|
||||
- Mergeable state (`CLEAN`, `UNSTABLE`, `DIRTY` = conflicts, `UNKNOWN` = GitHub still computing)
|
||||
- Whether maintainer edits are allowed on the branch (needed later if you rebase for the author)
|
||||
- File paths touched (helps spot overlaps between PRs)
|
||||
|
||||
`UNKNOWN` is common right after a push to main — just try the merge and see.
|
||||
|
||||
### 3. Classify into tiers
|
||||
|
||||
Sort each PR into exactly one bucket:
|
||||
|
||||
**Tier 1 — Merge:** small, mergeable, fixes a real bug, clean CI, low review cost. One-liners, dependency relaxations, targeted safety hardening. These are the easy wins.
|
||||
|
||||
**Tier 2 — Candidate, review:** medium size (50-200 lines), touches more surface area, looks sound but needs a closer read. New user-facing features that fit the product direction.
|
||||
|
||||
**Supersede:** the fix or feature is already covered by something merged. Close with a comment pointing to the superseding PR. Check carefully — "similar title" isn't proof; compare the actual diffs.
|
||||
|
||||
**Defer to next release:** big features, dirty conflicts, draft PRs, anything touching the release pipeline in ways that would introduce risk. Don't merge these in a speedrun — they need dedicated focus.
|
||||
|
||||
### 4. Write the triage doc
|
||||
|
||||
Create `<VERSION>_PR_TRIAGE.md` in the PR-review worktree root. Structure:
|
||||
|
||||
```markdown
|
||||
# <Repo> <VERSION> — PR Triage
|
||||
|
||||
Working doc for tracking which open PRs land in <VERSION>. Delete after release cut.
|
||||
|
||||
Last updated: <DATE>
|
||||
|
||||
## Progress
|
||||
|
||||
**Tier 1: 0 / N merged**
|
||||
**Tier 2: 0 / M handled**
|
||||
**Supersede triage: pending**
|
||||
|
||||
---
|
||||
|
||||
## Merge for <VERSION> — critical bug fixes
|
||||
|
||||
| PR | Status | Size | What it fixes | Why must-have |
|
||||
|---|---|---|---|---|
|
||||
| [#123](url) | [ ] | +5/-0 | ... | ... |
|
||||
|
||||
## Strong candidate — needs a quick review
|
||||
|
||||
| PR | Status | Size | Summary |
|
||||
|---|---|---|---|
|
||||
|
||||
## Close as superseded
|
||||
|
||||
| PR | Status | Reason |
|
||||
|---|---|---|
|
||||
|
||||
## Defer to <NEXT_VERSION>
|
||||
|
||||
- [#xxx](url) ... — reason
|
||||
|
||||
---
|
||||
|
||||
## Order of attack
|
||||
|
||||
1. Close superseded PRs (one-liner comments)
|
||||
2. Merge tier-1 in dependency-free batches — check file paths don't overlap
|
||||
3. Review tier-2 individually
|
||||
4. Rerun `draft-release-notes` to pick up everything
|
||||
5. Run `release-bump`
|
||||
```
|
||||
|
||||
The **Progress** header is the most important part — it's your scoreboard and lets you resume cleanly if the session gets interrupted.
|
||||
|
||||
### 5. Work the loop — per PR
|
||||
|
||||
For each PR in the tier-1 / tier-2 list:
|
||||
|
||||
**a. Checkout in the review worktree:**
|
||||
```bash
|
||||
cd ../voicebox-pr-review
|
||||
git checkout pr-review-<VERSION> # reset to neutral base
|
||||
gh pr checkout <N>
|
||||
```
|
||||
|
||||
**b. Read the *actual* commit, not `main..HEAD`:**
|
||||
|
||||
```bash
|
||||
git show HEAD # the PR's actual changes
|
||||
git show --stat HEAD # files touched + line counts
|
||||
```
|
||||
|
||||
**Do NOT review via `git diff main..HEAD`** if the PR branch is older than main. That diff includes *every commit that landed on main after the PR was forked* as `-` (deletion) lines. A 3-line PR can look like a 700-line revert. This is the single easiest way to misjudge a PR.
|
||||
|
||||
**c. Evaluate concerns:** correctness, scope, interaction with already-merged work, version compatibility (e.g. can't use an API that requires a dependency version we don't yet pin).
|
||||
|
||||
**d. Rebase if the branch is behind main:**
|
||||
```bash
|
||||
git fetch origin main
|
||||
git rebase origin/main
|
||||
```
|
||||
|
||||
This is **essential** before squash-merging. GitHub's squash computes `diff(PR-head, merge-base)` — on a stale branch, that diff includes reverting every in-between commit. Rebasing moves the merge-base forward so the squash is clean.
|
||||
|
||||
**e. If maintainer edits are allowed, push the rebase back to the contributor's fork:**
|
||||
```bash
|
||||
git remote add <author> https://github.com/<author>/<repo>.git
|
||||
git fetch <author> <branch> # get their ref first
|
||||
git push <author> HEAD:<branch> --force-with-lease
|
||||
```
|
||||
|
||||
This keeps GitHub's PR UI in sync with the rebased state and makes the merge clean from the GitHub side.
|
||||
|
||||
**f. Merge:**
|
||||
```bash
|
||||
gh pr merge <N> --squash
|
||||
```
|
||||
|
||||
**g. Update the triage doc** — flip the checkbox to `✅ merged <sha>` (use the short SHA from `gh pr view <N> --json mergeCommit --jq '.mergeCommit.oid[0:7]'`). Update the Progress header.
|
||||
|
||||
### 6. Batch tiny fixes
|
||||
|
||||
PRs with ≤5 line changes, clean CI, non-overlapping file paths, and obviously-correct intent (e.g. one-line dependency relax, env var add, import path fix) can be merged in a single loop without the review-per-PR ceremony:
|
||||
|
||||
```bash
|
||||
for pr in 425 384 416 429; do
|
||||
echo "=== Merging PR $pr ==="
|
||||
gh pr merge $pr --squash
|
||||
done
|
||||
```
|
||||
|
||||
Verify afterward that each landed cleanly:
|
||||
```bash
|
||||
for pr in 425 384 416 429; do
|
||||
gh pr view $pr --json state,mergeCommit --jq "{pr: $pr, state, sha: .mergeCommit.oid[0:7]}"
|
||||
done
|
||||
```
|
||||
|
||||
### 7. Post-merge follow-ups
|
||||
|
||||
Sometimes a PR is worth merging despite a known minor issue (e.g. incomplete dtype map, stale sentinel cleanup). Don't block the merge; apply the follow-up as a normal branch + PR right after:
|
||||
|
||||
```bash
|
||||
cd <main-worktree>
|
||||
git pull --ff-only origin main
|
||||
git checkout -b fix/<short-name>
|
||||
# edit...
|
||||
git commit -m "fix(<area>): <one-liner>"
|
||||
git push -u origin fix/<short-name>
|
||||
gh pr create --title "..." --body "Follow-up to #<N>. ..."
|
||||
```
|
||||
|
||||
Record both SHAs in the triage doc (`✅ merged <pr-sha> + follow-up <pr>`).
|
||||
|
||||
**Direct-to-main exception:** only under an explicit, scoped policy (e.g. "release speedrun"). Don't default to it.
|
||||
|
||||
### 8. Supersede: close with a credit-pointing comment
|
||||
|
||||
```bash
|
||||
gh pr close <N> --comment "Closing — superseded by merged #<M> which landed <brief description>. Thanks!"
|
||||
```
|
||||
|
||||
Check the diffs first — "similar title" is not enough. If the PR is *partially* superseded (the diagnosis is right but only half the changes are still needed), do a partial-apply instead.
|
||||
|
||||
### 9. Partial-apply pattern
|
||||
|
||||
When a PR has both valuable and questionable changes bundled:
|
||||
|
||||
```bash
|
||||
cd <main-worktree>
|
||||
git pull --ff-only origin main
|
||||
|
||||
# Cherry-pick specific files from the PR branch
|
||||
git checkout <pr-commit-sha> -- <file1> <file2>
|
||||
|
||||
# Review the staged changes, adjust as needed
|
||||
git diff --cached
|
||||
|
||||
# Apply any surgical edits to files you don't want to bulk-replace
|
||||
# (e.g. the PR's file predates a recent main commit you need to preserve)
|
||||
|
||||
# Commit with a trailer crediting the original author
|
||||
git commit -m "$(cat <<'EOF'
|
||||
<subject>
|
||||
|
||||
<body explaining what was kept vs dropped>
|
||||
|
||||
Co-Authored-By: <author> <[email protected]>
|
||||
EOF
|
||||
)"
|
||||
git push ... # branch + PR, unless under the direct-to-main exception
|
||||
```
|
||||
|
||||
Then close the PR with a comment explaining what was applied and what was dropped, referencing the commit SHA.
|
||||
|
||||
### 10. Keep the doc current
|
||||
|
||||
Every merge, every close, every follow-up → update `<VERSION>_PR_TRIAGE.md`. The doc is your session log. If you're interrupted and resume tomorrow, the doc is the only source of truth for "where am I."
|
||||
|
||||
### 11. When triage is done
|
||||
|
||||
- Every PR in the doc has a terminal status (✅ merged / ✅ closed / deferred)
|
||||
- Progress header shows N/N for each tier
|
||||
- Next skill to run is `draft-release-notes` (to regenerate `[Unreleased]` against the new main), then `release-bump`
|
||||
|
||||
You can delete the triage doc after the release ships, or keep it in version history as a record.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`main..HEAD` on a stale branch lies.** It shows everything main gained since the branch split as deletions. Always review via `git show HEAD` for the PR's actual commit.
|
||||
- **Squash-merging an unrebased branch reverts in-between work.** The squash computes `diff(PR-head, merge-base)`. Rebase moves the merge-base forward.
|
||||
- **`mergeable=UNKNOWN`** is transient — GitHub is recomputing after a push. Just try the merge.
|
||||
- **Route ordering matters (FastAPI and similar):** `DELETE /history/failed` must be registered *before* `DELETE /history/{id}`, or the parameterized path will consume `"failed"` as an ID.
|
||||
- **Apple's `-weak_framework` overrides `-framework`** for the same framework, regardless of order — use it via `cargo:rustc-link-arg=-Wl,-weak_framework,Name` when a dependency hard-links something optional.
|
||||
- **Dependency version floors constrain what you can apply.** Before accepting a kwarg rename like `torch_dtype=` → `dtype=`, check the min-version pin supports it. Sometimes the right move is to cherry-pick half the PR.
|
||||
- **`cpal::Stream` and similar `!Send` audio types** can't cross `await` points or `spawn_blocking`. Sometimes a "not-ideal but correct" sync wait is the best available fix; flag but don't block.
|
||||
- **PyTorch nightly builds are not shippable for releases** — non-deterministic, can regress between runs. If a PR suggests switching to nightly to fix a GPU issue, prefer `TORCH_CUDA_ARCH_LIST=...+PTX` or wait for stable support instead.
|
||||
|
||||
## Canonical commands reference
|
||||
|
||||
```bash
|
||||
# Bulk PR metadata
|
||||
gh pr list --state open --limit 50 --json number,title,author,mergeable,mergeStateStatus,additions,deletions,maintainerCanModify,files
|
||||
|
||||
# Detailed single-PR view
|
||||
gh pr view <N> --json body,author,headRefName,baseRefName,mergeable,maintainerCanModify,files,statusCheckRollup
|
||||
|
||||
# The actual commit, not the branch-vs-main diff
|
||||
git show HEAD
|
||||
git show --stat HEAD
|
||||
gh pr diff <N>
|
||||
|
||||
# Rebase contributor branch onto current main
|
||||
git fetch origin main && git rebase origin/main
|
||||
|
||||
# Push rebase back to contributor fork (maintainerCanModify=true required)
|
||||
git remote add <author> https://github.com/<author>/<repo>.git
|
||||
git fetch <author> <branch>
|
||||
git push <author> HEAD:<branch> --force-with-lease
|
||||
|
||||
# Merge
|
||||
gh pr merge <N> --squash
|
||||
|
||||
# Confirm merge SHA for triage doc
|
||||
gh pr view <N> --json state,mergeCommit --jq '{state, sha: .mergeCommit.oid[0:7]}'
|
||||
|
||||
# Close superseded
|
||||
gh pr close <N> --comment "Closing — superseded by merged #<M>. Thanks!"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- **Never review a stale branch via `main..HEAD`.** This is the single most important line in this skill.
|
||||
- **The triage doc is the session state.** Lose the doc, lose the session. Update it after every action.
|
||||
- **Credit contributors even on partial-applies.** Use `Co-Authored-By:` trailers and close comments that link to the applied commit.
|
||||
- **Don't let perfect be the enemy of shipped.** A fix that goes from "broken" to "works with a minor known issue" is a strict improvement. Flag the issue, file a follow-up, merge the fix.
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
[bumpversion]
|
||||
current_version = 0.2.3
|
||||
current_version = 0.4.1
|
||||
commit = True
|
||||
tag = True
|
||||
tag_name = v{new_version}
|
||||
|
||||
@@ -38,7 +38,6 @@ biome.json
|
||||
.bumpversion.cfg
|
||||
.npmrc
|
||||
Makefile
|
||||
CHANGELOG.md
|
||||
CONTRIBUTING.md
|
||||
SECURITY.md
|
||||
LICENSE
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
jobs:
|
||||
frontend-quality:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: Install dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
|
||||
- name: Typecheck app + web
|
||||
run: bun run typecheck
|
||||
|
||||
- name: Build web smoke test
|
||||
run: bun run build:web
|
||||
@@ -62,11 +62,21 @@ jobs:
|
||||
pip install pyinstaller
|
||||
pip install -r backend/requirements.txt
|
||||
pip install --no-deps chatterbox-tts
|
||||
pip install --no-deps hume-tada
|
||||
|
||||
- name: Install MLX dependencies (Apple Silicon only)
|
||||
if: matrix.backend == 'mlx'
|
||||
run: |
|
||||
pip install -r backend/requirements-mlx.txt
|
||||
# mlx-audio>=0.3.1 and mlx-lm>=0.31.1 both declare transformers>=5.x,
|
||||
# which conflicts with our 4.57.x cap. The runtime APIs we use work
|
||||
# fine on transformers 4.57.x in practice (verified in dev), so install
|
||||
# them --no-deps. mlx-audio's other runtime deps (huggingface_hub,
|
||||
# librosa, numpy, numba, pyloudnorm) are already in requirements.txt;
|
||||
# the rest (sounddevice, miniaudio, protobuf, sentencepiece, pyyaml,
|
||||
# jinja2) are pulled in by other engines.
|
||||
pip install --no-deps mlx-lm==0.31.1
|
||||
pip install --no-deps mlx-audio==0.4.1
|
||||
|
||||
- name: Build Python server (Linux/macOS)
|
||||
if: matrix.platform != 'windows-latest'
|
||||
@@ -123,6 +133,29 @@ jobs:
|
||||
p12-file-base64: ${{ secrets.APPLE_CERTIFICATE }}
|
||||
p12-password: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
|
||||
|
||||
- name: Extract release notes from CHANGELOG.md
|
||||
id: changelog
|
||||
shell: bash
|
||||
run: |
|
||||
# Get the version from the tag (strip leading 'v')
|
||||
VERSION="${GITHUB_REF_NAME#v}"
|
||||
|
||||
# Extract the section for this version from CHANGELOG.md
|
||||
# Matches from "## [X.Y.Z]" until the next "## [" heading
|
||||
NOTES=$(sed -n "/^## \[${VERSION}\]/,/^## \[/{/^## \[${VERSION}\]/d;/^## \[/d;p;}" CHANGELOG.md)
|
||||
|
||||
# Fall back to a placeholder if the version isn't in the changelog
|
||||
if [ -z "$(echo "$NOTES" | tr -d '[:space:]')" ]; then
|
||||
NOTES="See the assets below to download and install this version."
|
||||
fi
|
||||
|
||||
# Use multiline output syntax
|
||||
{
|
||||
echo "notes<<CHANGELOG_EOF"
|
||||
echo "$NOTES"
|
||||
echo "CHANGELOG_EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: tauri-apps/[email protected]
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -139,17 +172,7 @@ jobs:
|
||||
projectPath: tauri
|
||||
tagName: v__VERSION__
|
||||
releaseName: "voicebox v__VERSION__"
|
||||
releaseBody: |
|
||||
## What's Changed
|
||||
See the assets below to download and install this version.
|
||||
|
||||
### Installation
|
||||
- **macOS (Apple Silicon)**: Download the `aarch64.dmg` file - uses MLX for fast native inference
|
||||
- **macOS (Intel)**: Download the `x64.dmg` file - uses PyTorch
|
||||
- **Windows**: Download the `.msi` installer
|
||||
- **Linux**: Compile from source (see README)
|
||||
|
||||
The app includes automatic updates - future updates will be installed automatically.
|
||||
releaseBody: ${{ steps.changelog.outputs.notes }}
|
||||
releaseDraft: true
|
||||
prerelease: false
|
||||
args: ${{ matrix.args }}
|
||||
@@ -175,43 +198,54 @@ jobs:
|
||||
pip install pyinstaller
|
||||
pip install -r backend/requirements.txt
|
||||
pip install --no-deps chatterbox-tts
|
||||
pip install --no-deps hume-tada
|
||||
|
||||
- name: Install PyTorch with CUDA 12.1
|
||||
- name: Install PyTorch with CUDA 12.8
|
||||
run: |
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu121 --force-reinstall --no-deps
|
||||
pip install torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
|
||||
pip install torchaudio --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps
|
||||
|
||||
- name: Verify CUDA support in torch
|
||||
run: |
|
||||
python -c "import torch; print(f'CUDA available in build: {torch.cuda.is_available()}'); print(f'CUDA version: {torch.version.cuda}')"
|
||||
|
||||
- name: Build CUDA server binary
|
||||
- name: Build CUDA server binary (onedir)
|
||||
shell: bash
|
||||
working-directory: backend
|
||||
env:
|
||||
# Include Blackwell (sm_120) via PTX forward compatibility.
|
||||
# Pre-built PyTorch cu128 wheels ship native kernels for sm_80/86/89/90
|
||||
# but not sm_120. Setting this env var causes torch.utils.cpp_extension
|
||||
# (and any JIT-compiled kernels) to target Blackwell GPUs as well.
|
||||
TORCH_CUDA_ARCH_LIST: "8.0;8.6;8.9;9.0;12.0+PTX"
|
||||
run: python build_binary.py --cuda
|
||||
|
||||
- name: Split binary for GitHub Releases
|
||||
- name: Package into server core + CUDA libs archives
|
||||
shell: bash
|
||||
run: |
|
||||
python scripts/split_binary.py \
|
||||
backend/dist/voicebox-server-cuda.exe \
|
||||
--output release-assets/
|
||||
python scripts/package_cuda.py \
|
||||
backend/dist/voicebox-server-cuda/ \
|
||||
--output release-assets/ \
|
||||
--cuda-libs-version cu128-v1 \
|
||||
--torch-compat ">=2.7.0,<2.11.0"
|
||||
|
||||
- name: Upload split parts to GitHub Release
|
||||
- name: Upload archives to GitHub Release
|
||||
if: startsWith(github.ref, 'refs/tags/')
|
||||
uses: softprops/action-gh-release@v1
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
files: |
|
||||
release-assets/voicebox-server-cuda.part*.exe
|
||||
release-assets/voicebox-server-cuda.sha256
|
||||
release-assets/voicebox-server-cuda.manifest
|
||||
release-assets/voicebox-server-cuda.tar.gz
|
||||
release-assets/voicebox-server-cuda.tar.gz.sha256
|
||||
release-assets/cuda-libs-cu128-v1.tar.gz
|
||||
release-assets/cuda-libs-cu128-v1.tar.gz.sha256
|
||||
release-assets/cuda-libs.json
|
||||
draft: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Upload binary as workflow artifact
|
||||
- name: Upload onedir as workflow artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: voicebox-server-cuda-windows
|
||||
path: backend/dist/voicebox-server-cuda.exe
|
||||
path: backend/dist/voicebox-server-cuda/
|
||||
retention-days: 7
|
||||
|
||||
+14
@@ -49,8 +49,22 @@ logs/
|
||||
# Generated files
|
||||
app/openapi.json
|
||||
tauri/src-tauri/binaries/*
|
||||
tauri/src-tauri/gen/Assets.car
|
||||
tauri/src-tauri/gen/voicebox.icns
|
||||
tauri/src-tauri/gen/partial.plist
|
||||
|
||||
# PyInstaller
|
||||
*.spec
|
||||
|
||||
# Windows artifacts
|
||||
nul
|
||||
|
||||
# Temporary
|
||||
tmp/
|
||||
temp/
|
||||
*.tmp
|
||||
|
||||
# E2E test artifacts
|
||||
backend/tests/results/
|
||||
backend/tests/fixtures/reference_voice.wav
|
||||
backend/tests/fixtures/reference_voice.txt
|
||||
|
||||
+592
-68
@@ -1,94 +1,618 @@
|
||||
<!-- This file is compiled automatically during the release workflow. -->
|
||||
<!-- Do not edit manually — your changes will be overwritten. -->
|
||||
<!-- To update the draft: ask the agent to use the draft-release-notes skill. -->
|
||||
<!-- To finalize a release: ask the agent to use the release-bump skill. -->
|
||||
|
||||
# Changelog
|
||||
|
||||
All notable changes to Voicebox will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- **Profile Name Validation** - Added proper validation to prevent duplicate profile names ([#134](https://github.com/jamiepine/voicebox/issues/134))
|
||||
- Users now receive clear error messages when attempting to create or update profiles with duplicate names
|
||||
- Improved error handling in create and update profile API endpoints
|
||||
- Added comprehensive test suite for duplicate name validation
|
||||
## [0.4.1] - 2026-04-18
|
||||
|
||||
## [0.1.0] - 2026-01-25
|
||||
A fast follow-up to 0.4.0 focused on making the new engines actually load in the production binary — plus generation cancellation, Linux system-audio capture, and the repo's first PR-time type check. Five first-time contributors shipped in this release.
|
||||
|
||||
### Added
|
||||
0.4.0 introduced three new TTS engines, but the frozen PyInstaller binary tripped over several Python-ecosystem quirks that don't show up in the dev venv: `transformers` opening `.py` sources at runtime, `scipy.stats._distn_infrastructure` hitting a frozen-importer `NameError`, and `chatterbox-multilingual` failing to find its Chinese segmenter dictionary. This release patches all of those in one sweep.
|
||||
|
||||
#### Core Features
|
||||
- **Voice Cloning** - Clone voices from audio samples using Qwen3-TTS (1.7B and 0.6B models)
|
||||
- **Voice Profile Management** - Create, edit, and organize voice profiles with multiple samples
|
||||
- **Speech Generation** - Generate high-quality speech from text using cloned voices
|
||||
- **Generation History** - Track all generations with search and filtering capabilities
|
||||
- **Audio Transcription** - Automatic transcription powered by Whisper
|
||||
- **In-App Recording** - Record audio samples directly in the app with waveform visualization
|
||||
### Frozen-Binary Reliability ([#438](https://github.com/jamiepine/voicebox/pull/438))
|
||||
- **Kokoro** now bundles `.py` sources alongside `.pyc` via `--collect-all kokoro` so `transformers`' `_can_set_attn_implementation` regex scan can read them — previously `FileNotFoundError: kokoro/modules.py` killed Kokoro loading in production builds
|
||||
- **Chatterbox Multilingual** now bundles `spacy_pkuseg/dicts/default.pkl` and the package's native `.so` extensions via `--collect-all spacy_pkuseg` — previously the Chinese word segmenter crashed with `FileNotFoundError` on first load
|
||||
- **scipy.stats._distn_infrastructure** — new runtime hook source-patches the trailing `del obj` (which raises `NameError` under PyInstaller's frozen importer because the preceding list comprehension evaluates empty) to `globals().pop('obj', None)`, unblocking `librosa` → `scipy.signal` → `scipy.stats` for every TTS engine that depends on librosa
|
||||
- **transformers.masking_utils** — same runtime hook forces `_is_torch_greater_or_equal_than_2_6 = False` so the older `sdpa_mask_older_torch` path is selected; the 2.6+ path uses `TransformGetItemToIndex()`, a real `torch._dynamo` graph transform our permissive stub can't reproduce
|
||||
- **torch._dynamo** — no-op stub replaces the real module before `transformers` imports it, preventing the `torch._numpy._ufuncs` import crash (`NameError: name 'name' is not defined`) that blocked Kokoro and every engine pulling in `flex_attention`
|
||||
- `.spec` paths are now repo-relative instead of absolute, so the generated spec is portable across machines and CI
|
||||
|
||||
#### Desktop App
|
||||
- **Tauri Desktop App** - Native desktop application for macOS, Windows, and Linux
|
||||
- **Local Server Mode** - Embedded Python server runs automatically
|
||||
- **Remote Server Mode** - Connect to a remote Voicebox server on your network
|
||||
- **Auto-Updates** - Automatic update notifications and installation
|
||||
### Generation
|
||||
- **Cancel queued or running generations** ([#444](https://github.com/jamiepine/voicebox/pull/444)) — new `/generate/{id}/cancel` endpoint and a Stop button on the history row while generating. The serial queue now tracks per-ID state (queued / running / cancelled) so queued jobs are skipped before the worker picks them up and running jobs are `.cancel()`-ed mid-flight; `run_generation` catches `CancelledError` and marks the row `failed` with a "cancelled" error.
|
||||
- **Legacy `data/` path prefix resolution** ([#440](https://github.com/jamiepine/voicebox/pull/440)) — generations stored with the old `data/` prefix under pre-0.4 installs now resolve correctly after the storage root moved, fixing 404s for historical audio.
|
||||
|
||||
#### API
|
||||
- **REST API** - Full REST API for voice synthesis and profile management
|
||||
- **OpenAPI Documentation** - Interactive API docs at `/docs` endpoint
|
||||
- **Type-Safe Client** - Auto-generated TypeScript client from OpenAPI schema
|
||||
### Model Migration
|
||||
- Migration dialog no longer hangs when the cache is empty ([#439](https://github.com/jamiepine/voicebox/pull/439)) — the backend now emits a completion SSE event even when zero models are moved.
|
||||
- Storage-change flow surfaces a toast when there's nothing to migrate ([#433](https://github.com/jamiepine/voicebox/pull/433)) instead of proceeding with a no-op move and restarting the server.
|
||||
- Deleting all generations from a voice profile now deletes the associated version files and DB rows too ([#447](https://github.com/jamiepine/voicebox/pull/447)) — previously orphaned versions accumulated in storage.
|
||||
|
||||
#### Technical
|
||||
- **Voice Prompt Caching** - Fast regeneration with cached voice prompts
|
||||
- **Multi-Sample Support** - Combine multiple audio samples for better voice quality
|
||||
- **GPU/CPU/MPS Support** - Automatic device detection and optimization
|
||||
- **Model Management** - Lazy loading and VRAM management
|
||||
- **SQLite Database** - Local data persistence
|
||||
### Platform
|
||||
- **Linux system audio capture** ([#457](https://github.com/jamiepine/voicebox/pull/457)) — `cpal`'s ALSA backend doesn't expose PulseAudio/PipeWire monitor sources by name, so the previous device-name search never matched and silently fell back to the microphone. Detection now uses `pactl get-default-sink` + `pactl list short sources` and routes via `PULSE_SOURCE`, with the name-based search retained as a fallback when `pactl` is absent.
|
||||
|
||||
### Technical Details
|
||||
### Frontend CI
|
||||
- First PR-time quality gate ([#418](https://github.com/jamiepine/voicebox/pull/418)) — new `.github/workflows/ci.yml` runs `bun run typecheck` + `bun run build:web` on every PR. Fixed pre-existing type issues that were being suppressed with `@ts-expect-error`, cleaned up a dep-array typo (`[platform.metadata.isTauricheckOnMountcheckForUpdates]`) in `useAutoUpdater`, and removed 100+ lines of dead `ModelItem` code from `ModelManagement.tsx`.
|
||||
- Follow-up: widened `apiClient.migrateModels()` return type to include `moved` and `errors` so the storage-change handler typechecks against the real backend response ([#470](https://github.com/jamiepine/voicebox/pull/470)).
|
||||
|
||||
- Built with Tauri v2 (Rust + React)
|
||||
- FastAPI backend with async Python
|
||||
- TypeScript frontend with React Query and Zustand
|
||||
- Qwen3-TTS for voice cloning
|
||||
- Whisper for transcription
|
||||
### Docs
|
||||
- Clarified in the Quick Start + README that paralinguistic tags (`[laugh]`, `[sigh]`) only work with Chatterbox Turbo; other engines read them as literal text ([#450](https://github.com/jamiepine/voicebox/pull/450)).
|
||||
|
||||
### New Contributors
|
||||
- [@Bortlesboat](https://github.com/Bortlesboat) — generation cancellation (#444)
|
||||
- [@gaojulong](https://github.com/gaojulong) — migration dialog hang fix (#439)
|
||||
- [@fuleinist](https://github.com/fuleinist) — migration no-op toast (#433)
|
||||
- [@erionjuniordeandrade-a11y](https://github.com/erionjuniordeandrade-a11y) — frontend CI + type hardening (#418)
|
||||
- [@estefrac](https://github.com/estefrac) — Linux pactl system-audio capture (#457)
|
||||
|
||||
## [0.4.0] - 2026-04-16
|
||||
|
||||
The biggest Voicebox release yet. Three new TTS engines bring the lineup to **seven** — HumeAI TADA, Kokoro 82M, and Qwen CustomVoice join Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and Chatterbox Turbo. GPU support broadens to Intel Arc (XPU) and NVIDIA Blackwell (RTX 50-series), with runtime diagnostics that warn when your PyTorch build doesn't match your GPU. The CUDA backend is now split into independently versioned server and library archives, so upgrading no longer redownloads 4 GB of PyTorch/CUDA DLLs.
|
||||
|
||||
This release also marks a big community moment: **13 new contributors** shipped fixes and features in 0.4.0. Thirty-plus bug fixes target the most-reported issues in the tracker — numpy 2.x TTS crashes, Windows background-server reliability, macOS 11 launch failures, audio playback silence, Stories clip-splitting races, history status staleness, and more.
|
||||
|
||||
### New TTS Engines
|
||||
|
||||
#### HumeAI TADA — Expressive English & Multilingual ([#296](https://github.com/jamiepine/voicebox/pull/296))
|
||||
- Added `tada-1b` (English) and `tada-3b-ml` (multilingual) backends
|
||||
- Replaced `descript-audio-codec` with a lightweight DAC shim to cut dependencies
|
||||
- Switched audio decoding to `soundfile` to sidestep `torchcodec` bundling issues
|
||||
- Redirected gated Llama tokenizer lookups to an ungated mirror so model loading works out of the box
|
||||
- Fixed tokenizer patch that was corrupting `AutoTokenizer` for other engines
|
||||
- Fixed TorchScript error in frozen builds
|
||||
|
||||
#### Kokoro 82M — Fast Lightweight TTS ([#325](https://github.com/jamiepine/voicebox/pull/325))
|
||||
- Added Kokoro 82M engine with a new voice profile type system that distinguishes preset voices from cloned profiles
|
||||
- Profile grid now handles engine compatibility directly — removed redundant dropdown filtering
|
||||
- Tightened Kokoro profile handling so preset voices can't be edited like cloned profiles
|
||||
|
||||
#### Qwen CustomVoice ([#328](https://github.com/jamiepine/voicebox/pull/328))
|
||||
- Added `qwen-custom-voice` preset engine backed by Qwen3-TTS
|
||||
- Enforced preset/profile engine compatibility across the generation flow
|
||||
- Floating generator now shows all engines instead of silently filtering
|
||||
|
||||
### Voice Profile UX
|
||||
|
||||
Until 0.4, every engine in Voicebox was a cloning model, so every voice profile was usable with every engine and the profile grid just showed them all. Introducing Kokoro and Qwen CustomVoice — which work from preset voices rather than cloned samples — broke that assumption for the first time. An early cut on `main` filtered the grid by the selected engine, which left users running pre-release builds thinking their cloned voices had vanished whenever they switched to a preset-only engine.
|
||||
|
||||
This release ships the resolution before it ever reaches a tagged version:
|
||||
|
||||
- **Grey-out instead of filter** — all profiles are always visible; unsupported ones render dimmed with a compatibility hint at the bottom of the grid
|
||||
- **Auto-switch on selection** — clicking a greyed-out profile selects it AND switches the engine to a compatible one, instead of silently doing nothing
|
||||
- **Instruct toggle restored for Qwen CustomVoice** — the floating generate box now reveals a delivery-instructions input (tone, emotion, pace) when CustomVoice is selected. Hidden across the board while the new multi-engine lineup was stabilizing because most engines don't honor the kwarg; now conditionally exposed only for the one engine that was actually trained for instruction-based style control
|
||||
- Supported profiles sort first; the grid scrolls the selected profile into view after engine/sort changes
|
||||
- Fixed engine desync on tab navigation — the form now initializes its engine from the store
|
||||
- Fixed the disabled-and-selected card click edge case by bouncing selection to re-trigger the auto-switch
|
||||
- Cleaned up scroll effect timers (requestAnimationFrame + setTimeout) to prevent stale DOM writes on unmount or rapid selection changes
|
||||
|
||||
### GPU & Platform
|
||||
|
||||
#### Intel Arc (XPU) Support ([#320](https://github.com/jamiepine/voicebox/pull/320))
|
||||
- First-class Intel Arc support across all PyTorch-based backends
|
||||
- Device-aware seeding, XPU detection in the GPU status panel, and setup flow detection
|
||||
- Reports correct device name and VRAM in settings
|
||||
|
||||
#### Blackwell / RTX 50-series Support ([#316](https://github.com/jamiepine/voicebox/pull/316), [#401](https://github.com/jamiepine/voicebox/pull/401))
|
||||
- Upgraded the CUDA backend from cu126 → cu128 for RTX 50-series support
|
||||
- Added `sm_120+PTX` to the CUDA build via `TORCH_CUDA_ARCH_LIST` for forward-compatibility with Blackwell architectures (closes 5 open reports: #386, #395, #396, #399, #400)
|
||||
- GPU settings UI fixes around install/uninstall state
|
||||
|
||||
#### GPU Compatibility Diagnostics ([#367](https://github.com/jamiepine/voicebox/pull/367), adapted)
|
||||
- New `check_cuda_compatibility()` compares the current device's compute capability against the bundled PyTorch's architecture list
|
||||
- Health endpoint exposes a `gpu_compatibility_warning` field so the UI can surface mismatches
|
||||
- Startup logs a `WARN` when the installed PyTorch build doesn't support the detected GPU
|
||||
- GPU status label shows `[UNSUPPORTED - see logs]` — no more silent "no kernel image" failures
|
||||
|
||||
#### Split CUDA Backend ([#298](https://github.com/jamiepine/voicebox/pull/298))
|
||||
- CUDA backend now ships as two independently versioned archives: a small server binary and a large libs archive (the ~4 GB of PyTorch/CUDA DLLs)
|
||||
- Upgrading Voicebox no longer redownloads the libs archive when only the server binary changed
|
||||
- Added `asyncio.Lock` around `download_cuda_binary()` so auto-update and manual download can't race on the same temp file ([#428](https://github.com/jamiepine/voicebox/pull/428))
|
||||
- Updated `package_cuda.py` for PyInstaller 6.18 onedir layout
|
||||
- Temp archives are always cleaned up on failure, even when the install aborts mid-extract
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
#### Critical: TTS Generation
|
||||
- **numpy 2.x `torch.from_numpy` crash** ([#361](https://github.com/jamiepine/voicebox/pull/361)) — torch compiled against numpy 1.x ABI fails silently when paired with numpy 2.x, causing `RuntimeError: Numpy is not available` / `Unable to create tensor` on every TTS request in bundled macOS Intel / Rosetta builds. Pinned `numpy<2.0` in requirements and added a PyInstaller runtime hook with a `ctypes.memmove` fallback as belt-and-suspenders. Hardened afterward to raise on unknown dtypes instead of silently reinterpreting bytes as float32.
|
||||
|
||||
#### Platform Reliability
|
||||
- **Windows background server** ([#402](https://github.com/jamiepine/voicebox/pull/402)) — "keep server running after close" now actually keeps the server running. The HTTP `/watchdog/disable` request could lose the race against process exit on Windows; added a `.keep-running` sentinel file as a synchronous fallback, with stale-sentinel cleanup on startup to avoid orphan server processes
|
||||
- **macOS 11 launch crash** ([#424](https://github.com/jamiepine/voicebox/pull/424)) — weak-linked ScreenCaptureKit so the app can launch on macOS < 12.3 instead of crashing at dyld resolution. Gated system audio capture behind a real `sw_vers` version check so unsupported systems cleanly advertise "not available" rather than crashing at runtime
|
||||
- **macOS Intel (x86_64) setup** ([#416](https://github.com/jamiepine/voicebox/pull/416)) — relaxed `torch>=2.7.0` → `torch>=2.2.0`. PyTorch dropped pre-built x86_64 wheels after 2.2.2, so Intel Mac devs could no longer `pip install`. Now resolves to the latest compatible torch per platform
|
||||
- **Offline model loading** ([#318](https://github.com/jamiepine/voicebox/pull/318)) — Qwen TTS and Whisper force offline mode when loading cached models, so startup works without network access
|
||||
- **GUI startup with external server** ([#319](https://github.com/jamiepine/voicebox/pull/319)) — fixed GUI launch when pointed at a remote/external server, and added data refresh on server switch; hardened health validation and error handling
|
||||
- **Qwen3-TTS cache split on Windows** (adapted from [#218](https://github.com/jamiepine/voicebox/pull/218)) — route `Qwen3TTSModel.from_pretrained` through `hf_constants.HF_HUB_CACHE` so the speech tokenizer and `preprocessor_config.json` resolve from a single cache root
|
||||
- **Qwen3-TTS bundling** ([#305](https://github.com/jamiepine/voicebox/pull/305)) — bundle `qwen_tts` source files in the PyInstaller build to fix `inspect.getsource` errors in frozen builds
|
||||
- **Backend import paths** ([#345](https://github.com/jamiepine/voicebox/pull/345)) — moved lazy imports to top-level with absolute paths to resolve the "Failed to Save" preset error caused by `ModuleNotFoundError` in production builds
|
||||
- **Effects service import** ([#384](https://github.com/jamiepine/voicebox/pull/384)) — fixed `ModuleNotFoundError` on preset create/update by switching to relative imports (#349)
|
||||
|
||||
#### Audio & Playback
|
||||
- **cpal stream silent playback** ([#405](https://github.com/jamiepine/voicebox/pull/405)) — `cpal::Stream` was dropped on function return immediately after `play()`, causing every playback to fall silent. Now holds the stream until either the buffer drains or the stop flag fires (#404)
|
||||
|
||||
#### Stories & History
|
||||
- **Clip-splitting race** ([#403](https://github.com/jamiepine/voicebox/pull/403)) — rapid double-clicks on split could race through `split_story_item` with inconsistent state. Added `with_for_update()` row locking on the backend and an `isPending` guard on the frontend (#366)
|
||||
- **History `status` staleness** ([#394](https://github.com/jamiepine/voicebox/pull/394)) — `GET /history/{id}` was hardcoding `status="completed"` regardless of the DB row, breaking any client polling for job completion. Now returns `status`, `error`, `engine`, `model_size`, and `is_favorited` from the actual row
|
||||
- **"Clear failed" bulk button** ([#412](https://github.com/jamiepine/voicebox/pull/412)) — new `DELETE /history/failed` endpoint and a header strip showing `"N failed generations"` with a Clear button, complementing the per-row trash icon added in #321 (#410)
|
||||
- **Delete failed generations** ([#321](https://github.com/jamiepine/voicebox/pull/321)) — added a trash icon next to the retry button so failed entries can be cleaned up without having to retry first
|
||||
|
||||
#### Security & Safety
|
||||
- **Voice prompt cache hardening** ([#429](https://github.com/jamiepine/voicebox/pull/429)) — `torch.load(weights_only=True)` on cached voice prompts per PyTorch 2.6 recommendation; replaced string-based SPA path guard with `Path.is_relative_to()` for more robust path-traversal protection
|
||||
|
||||
#### Infrastructure & Docker
|
||||
- **Docker web build** ([#344](https://github.com/jamiepine/voicebox/pull/344)) — include `CHANGELOG.md` in the Docker web build so the in-app changelog page works in Docker deployments
|
||||
- **Docker numba cache** ([#425](https://github.com/jamiepine/voicebox/pull/425)) — set `NUMBA_CACHE_DIR` in docker-compose so numba can write its JIT cache in container runtime (#308)
|
||||
- **Relative media paths** ([#332](https://github.com/jamiepine/voicebox/pull/332)) — media paths now stored relative to the configured data dir rather than resolved against CWD, so the data directory is portable between installs
|
||||
|
||||
### Developer Tooling
|
||||
|
||||
- New `triage-prs` agent skill — encodes the end-to-end PR-speedrun workflow (classification → triage doc → rebase → squash-merge → follow-ups) so future release cycles can reproduce it
|
||||
- Rewrote the TTS engine guide with the patterns learned from adding TADA and Kokoro
|
||||
- Added the API refactor plan and CUDA libs addon design doc
|
||||
- Fixed broken links in the Get Started section ([#332](https://github.com/jamiepine/voicebox/pull/332))
|
||||
|
||||
### New Contributors
|
||||
|
||||
Huge thank you to everyone who contributed their first PR to Voicebox in this release:
|
||||
|
||||
[@liorshahverdi](https://github.com/liorshahverdi), [@nicoschtein](https://github.com/nicoschtein), [@ArfianID](https://github.com/ArfianID), [@aimaaaimaa](https://github.com/aimaaaimaa), [@maxmcoding](https://github.com/maxmcoding), [@Khalodddd](https://github.com/Khalodddd), [@LuisSambrano](https://github.com/LuisSambrano), [@shaun0927](https://github.com/shaun0927), [@malletfils](https://github.com/malletfils), [@mvanhorn](https://github.com/mvanhorn), [@kuishou68](https://github.com/kuishou68), [@txhno](https://github.com/txhno), [@MukundaKatta](https://github.com/MukundaKatta)
|
||||
|
||||
## [0.3.0] - 2026-03-17
|
||||
|
||||
This release rewrites the backend into a modular architecture, overhauls the settings UI into routed sub-pages, fixes audio player freezing, migrates documentation to Fumadocs, and ships a batch of bug fixes targeting the most-reported issues from the tracker.
|
||||
|
||||
The backend's 3,000-line monolith `main.py` has been decomposed into domain routers, a services layer, and a proper database package. A style guide and ruff configuration now enforce consistency. On the frontend, settings have been split into dedicated routed pages with server logs, a changelog viewer, and an about page. The audio player no longer freezes mid-playback, and model loading status is now visible in the UI. Seven user-reported bugs have been fixed, including server crashes during sample uploads, generation list staleness, cryptic error messages, and CUDA support for RTX 50-series GPUs.
|
||||
|
||||
### Settings Overhaul ([#294](https://github.com/jamiepine/voicebox/pull/294))
|
||||
- Split settings into routed sub-tabs: General, Generation, GPU, Logs, Changelog, About
|
||||
- Added live server log viewer with auto-scroll
|
||||
- Added in-app changelog page that parses `CHANGELOG.md` at build time
|
||||
- Added About page with version info, license, and generation folder quick-open
|
||||
- Extracted reusable `SettingRow` component for consistent setting layouts
|
||||
|
||||
### Audio Player Fix ([#293](https://github.com/jamiepine/voicebox/pull/293))
|
||||
- Fixed audio player freezing during playback
|
||||
- Improved playback UX with better state management and listener cleanup
|
||||
- Fixed restart race condition during regeneration
|
||||
- Added stable keys for audio element re-rendering
|
||||
- Improved accessibility across player controls
|
||||
|
||||
### Backend Refactor ([#285](https://github.com/jamiepine/voicebox/pull/285))
|
||||
- Extracted all routes from `main.py` into 13 domain routers under `backend/routes/` — `main.py` dropped from ~3,100 lines to ~10
|
||||
- Moved CRUD and service modules into `backend/services/`, platform detection into `backend/utils/`
|
||||
- Split monolithic `database.py` into a `database/` package with separate `models`, `session`, `migrations`, and `seed` modules
|
||||
- Added `backend/STYLE_GUIDE.md` and `pyproject.toml` with ruff linting config
|
||||
- Removed dead code: unused `_get_cuda_dll_excludes`, stale `studio.py`, `example_usage.py`, old `Makefile`
|
||||
- Deduplicated shared logic across TTS backends into `backends/base.py`
|
||||
- Improved startup logging with version, platform, data directory, and database stats
|
||||
- Fixed startup database session leak — sessions now rollback and close in `finally` block
|
||||
- Isolated shutdown unload calls so one backend failure doesn't block the others
|
||||
- Handled null duration in `story_items` migration
|
||||
- Reject model migration when target is a subdirectory of source cache
|
||||
|
||||
### Documentation Rewrite ([#288](https://github.com/jamiepine/voicebox/pull/288))
|
||||
- Migrated docs site from Mintlify to Fumadocs (Next.js-based)
|
||||
- Rewrote introduction and root page with content from README
|
||||
- Added "Edit on GitHub" links and last-updated timestamps on all pages
|
||||
- Generated OpenAPI spec and auto-generated API reference pages
|
||||
- Removed stale planning docs (`CUDA_BACKEND_SWAP`, `EXTERNAL_PROVIDERS`, `MLX_AUDIO`, `TTS_PROVIDER_ARCHITECTURE`, etc.)
|
||||
- Sidebar groups now expand by default; root redirects to `/docs`
|
||||
- Added OG image metadata and `/og` preview page
|
||||
|
||||
### UI & Frontend
|
||||
- Added model loading status indicator and effects preset dropdown ([3187344](https://github.com/jamiepine/voicebox/commit/3187344))
|
||||
- Fixed take-label race condition during regeneration
|
||||
- Added accessible focus styling to select component
|
||||
- Softened select focus indicator opacity
|
||||
- Addressed 4 critical and 12 major issues from CodeRabbit review
|
||||
|
||||
### Bug Fixes ([#295](https://github.com/jamiepine/voicebox/pull/295))
|
||||
- Fixed sample uploads crashing the server — audio decoding now runs in a thread pool instead of blocking the async event loop ([#278](https://github.com/jamiepine/voicebox/issues/278))
|
||||
- Fixed generation list not updating when a generation completes — switched to `refetchQueries` for reliable cache busting, added SSE error fallback, and page reset on completion ([#231](https://github.com/jamiepine/voicebox/issues/231))
|
||||
- Fixed error toasts showing `[object Object]` instead of the actual error message ([#290](https://github.com/jamiepine/voicebox/issues/290))
|
||||
- Added Whisper model selection (`base`, `small`, `medium`, `large`, `turbo`) and expanded language support to the `/transcribe` endpoint ([#233](https://github.com/jamiepine/voicebox/issues/233))
|
||||
- Upgraded CUDA backend build from cu121 to cu126 for RTX 50-series (Blackwell) GPU support ([#289](https://github.com/jamiepine/voicebox/issues/289))
|
||||
- Handled client disconnects in SSE and streaming endpoints to suppress `[Errno 32] Broken Pipe` errors ([#248](https://github.com/jamiepine/voicebox/issues/248))
|
||||
- Fixed Docker build failure from pip hash mismatch on Qwen3-TTS dependencies ([#286](https://github.com/jamiepine/voicebox/issues/286))
|
||||
- Added 50 MB upload size limit with chunked reads to prevent unbounded memory allocation on sample uploads
|
||||
- Eliminated redundant double audio decode in sample processing pipeline
|
||||
|
||||
### Platform Fixes
|
||||
- Replaced `netstat` with `TcpStream` + PowerShell for Windows port detection ([#277](https://github.com/jamiepine/voicebox/pull/277))
|
||||
- Fixed Docker frontend build and cleaned up Docker docs
|
||||
- Fixed macOS download links to use `.dmg` instead of `.app.tar.gz`
|
||||
- Added dynamic download redirect routes to landing site
|
||||
|
||||
### Release Tooling
|
||||
- Added `draft-release-notes` and `release-bump` agent skills
|
||||
- Wired CI release workflow to extract notes from `CHANGELOG.md` for GitHub Releases
|
||||
- Backfilled changelog with all historical releases
|
||||
|
||||
## [0.2.3] - 2026-03-15
|
||||
|
||||
The "it works in dev but not in prod" release. This version fixes a series of PyInstaller bundling issues that prevented model downloading, loading, generation, and progress tracking from working in production builds.
|
||||
|
||||
### Model Downloads Now Actually Work
|
||||
|
||||
The v0.2.1/v0.2.2 builds could not download or load models that weren't already cached from a dev install. This release fixes the entire chain:
|
||||
|
||||
- **Chatterbox, Chatterbox Turbo, and LuxTTS** all download, load, and generate correctly in bundled builds
|
||||
- **Real-time download progress** — byte-level progress bars now work in production. The root cause: `huggingface_hub` silently disables tqdm progress bars based on logger level, which prevented our progress tracker from receiving byte updates. We now force-enable the internal counter regardless.
|
||||
- **Fixed Python 3.12.0 `code.replace()` bug** — the macOS build was on Python 3.12.0, which has a [known CPython bug](https://github.com/pyinstaller/pyinstaller/issues/7992) that corrupts bytecode when PyInstaller rewrites code objects. This caused `NameError: name 'obj' is not defined` crashes during scipy/torch imports. Upgraded to Python 3.12.13.
|
||||
|
||||
### PyInstaller Fixes
|
||||
|
||||
- Collect all `inflect` files — `typeguard`'s `@typechecked` decorator calls `inspect.getsource()` at import time, which needs `.py` source files, not just bytecode. Fixes LuxTTS "could not get source code" error.
|
||||
- Collect all `perth` files — bundles the pretrained watermark model (`hparams.yaml`, `.pth.tar`) needed by Chatterbox at runtime
|
||||
- Collect all `piper_phonemize` files — bundles `espeak-ng-data/` (phoneme tables, language dicts) needed by LuxTTS for text-to-phoneme conversion
|
||||
- Set `ESPEAK_DATA_PATH` in frozen builds so the espeak-ng C library finds the bundled data instead of looking at `/usr/share/espeak-ng-data/`
|
||||
- Collect all `linacodec` files — fixes `inspect.getsource` error in Vocos codec
|
||||
- Collect all `zipvoice` files — fixes source code lookup in LuxTTS voice cloning
|
||||
- Copy metadata for `requests`, `transformers`, `huggingface-hub`, `tokenizers`, `safetensors`, `tqdm` — fixes `importlib.metadata` lookups in frozen binary
|
||||
- Add hidden imports for `chatterbox`, `chatterbox_turbo`, `luxtts`, `zipvoice` backends
|
||||
- Add `multiprocessing.freeze_support()` to fix resource_tracker subprocess crash in frozen binary
|
||||
- `--noconsole` now only applied on Windows — macOS/Linux need stdout/stderr for Tauri sidecar log capture
|
||||
- Hardened `sys.stdout`/`sys.stderr` devnull redirect to test writability, not just `None` check
|
||||
|
||||
### Updater
|
||||
|
||||
- Fixed updater artifact generation with `v1Compatible` for `tauri-action` signature files
|
||||
- Updated `tauri-action` to v0.6 to fix updater JSON and `.sig` generation
|
||||
|
||||
### Other Fixes
|
||||
|
||||
- Full traceback logging on all backend model loading errors (was just `str(e)` before)
|
||||
|
||||
## [0.2.2] - 2026-03-15
|
||||
|
||||
- Fix Chatterbox model support in bundled builds
|
||||
- Fix LuxTTS/ZipVoice support in bundled builds
|
||||
- Auto-update CUDA binary when app version changes
|
||||
- CUDA download progress bar
|
||||
- Fix server process staying alive on macOS (SIGHUP handling, watchdog grace period)
|
||||
- Hide console window when running CUDA binary on Windows
|
||||
|
||||
## [0.2.1] - 2026-03-15
|
||||
|
||||
Voicebox v0.1.x was a single-engine voice cloning app built around Qwen3-TTS. v0.2.0 is a ground-up rethink: four TTS engines, 23 languages, paralinguistic emotion controls, a post-processing effects pipeline, unlimited generation length, an async generation queue, and support for every major GPU vendor. Plus Docker.
|
||||
|
||||
### New TTS Engines
|
||||
|
||||
#### Multi-Engine Architecture
|
||||
|
||||
Voicebox now runs **four independent TTS engines** behind a thread-safe per-engine backend registry. Switch engines per-generation from a single dropdown — no restart required.
|
||||
|
||||
| Engine | Languages | Size | Key Strengths |
|
||||
| --------------------------- | --------- | ------- | --------------------------------------------- |
|
||||
| **Qwen3-TTS 1.7B** | 10 | ~3.5 GB | Highest quality, delivery instructions |
|
||||
| **Qwen3-TTS 0.6B** | 10 | ~1.2 GB | Lighter, faster variant |
|
||||
| **LuxTTS** | English | ~300 MB | CPU-friendly, 48 kHz output, 150x realtime |
|
||||
| **Chatterbox Multilingual** | 23 | ~3.2 GB | Broadest language coverage, zero-shot cloning |
|
||||
| **Chatterbox Turbo** | English | ~1.5 GB | 350M params, low latency, paralinguistic tags |
|
||||
|
||||
#### Chatterbox Multilingual — 23 Languages ([#257](https://github.com/jamiepine/voicebox/pull/257))
|
||||
|
||||
Zero-shot voice cloning in Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, and Turkish.
|
||||
|
||||
#### LuxTTS — Lightweight English TTS ([#254](https://github.com/jamiepine/voicebox/pull/254))
|
||||
|
||||
A fast, CPU-friendly English engine. ~300 MB download, 48 kHz output, runs at 150x realtime on CPU.
|
||||
|
||||
#### Chatterbox Turbo — Expressive English ([#258](https://github.com/jamiepine/voicebox/pull/258))
|
||||
|
||||
A fast 350M-parameter English model with inline paralinguistic tags.
|
||||
|
||||
#### Paralinguistic Tags Autocomplete ([#265](https://github.com/jamiepine/voicebox/pull/265))
|
||||
|
||||
Type `/` in the text input with Chatterbox Turbo selected to open an autocomplete for **9 expressive tags**: `[laugh]` `[chuckle]` `[gasp]` `[cough]` `[sigh]` `[groan]` `[sniff]` `[shush]` `[clear throat]`
|
||||
|
||||
### Generation
|
||||
|
||||
#### Unlimited Generation Length — Auto-Chunking ([#266](https://github.com/jamiepine/voicebox/pull/266))
|
||||
|
||||
Long text is now automatically split at sentence boundaries, generated per-chunk, and crossfaded back together. Engine-agnostic.
|
||||
|
||||
- Auto-chunking limit slider — 100–5,000 chars (default 800)
|
||||
- Crossfade slider — 0–200ms (default 50ms)
|
||||
- Max text length raised to 50,000 characters
|
||||
- Smart splitting respects abbreviations, CJK punctuation, and `[tags]`
|
||||
|
||||
#### Asynchronous Generation Queue ([#269](https://github.com/jamiepine/voicebox/pull/269))
|
||||
|
||||
Generation is now fully non-blocking. Serial execution queue prevents GPU contention. Real-time SSE status streaming.
|
||||
|
||||
#### Generation Versions
|
||||
|
||||
Every generation now supports multiple versions with provenance tracking — original, effects versions, takes, source tracking, version pinning in stories, and favorites.
|
||||
|
||||
### Post-Processing Effects ([#271](https://github.com/jamiepine/voicebox/pull/271))
|
||||
|
||||
A full audio effects system powered by Spotify's `pedalboard` library: Pitch Shift, Reverb, Delay, Chorus/Flanger, Compressor, Gain, High-Pass Filter, Low-Pass Filter. 4 built-in presets, custom presets, per-profile default effects, and live preview.
|
||||
|
||||
### Platform Support
|
||||
|
||||
- macOS (Apple Silicon and Intel)
|
||||
- Windows
|
||||
- Linux (AppImage)
|
||||
- **Windows Support** ([#272](https://github.com/jamiepine/voicebox/pull/272)) — Full Windows support with CUDA GPU detection
|
||||
- **Linux** ([#262](https://github.com/jamiepine/voicebox/pull/262)) — AMD ROCm, NVIDIA GBM fix, WebKitGTK mic access (build from source)
|
||||
- **NVIDIA CUDA Backend Swap** ([#252](https://github.com/jamiepine/voicebox/pull/252)) — Download and swap in CUDA backend from within the app
|
||||
- **Intel Arc (XPU) and DirectML** — PyTorch backend supports Intel Arc and DirectML
|
||||
- **Docker + Web Deployment** ([#161](https://github.com/jamiepine/voicebox/pull/161)) — 3-stage build, non-root runtime, health checks
|
||||
- **Whisper Turbo** — Added `openai/whisper-large-v3-turbo` as a transcription model option
|
||||
|
||||
---
|
||||
### Model Management ([#268](https://github.com/jamiepine/voicebox/pull/268))
|
||||
|
||||
## [Unreleased]
|
||||
Per-model unload, custom models directory, model folder migration, download cancel/clear UI ([#238](https://github.com/jamiepine/voicebox/pull/238)), restructured settings UI.
|
||||
|
||||
### Fixed
|
||||
- Audio export failing when Tauri save dialog returns object instead of string path
|
||||
- OpenAPI client generator script now documents the local backend port and avoids an unused loop variable warning
|
||||
### Security & Reliability
|
||||
|
||||
### Added
|
||||
- **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks
|
||||
- Includes Python version detection and compatibility warnings
|
||||
- Self-documenting help system with `make help`
|
||||
- Colored output for better readability
|
||||
- Supports parallel development server execution
|
||||
- CORS hardening ([#88](https://github.com/jamiepine/voicebox/pull/88))
|
||||
- Network access toggle ([#133](https://github.com/jamiepine/voicebox/pull/133))
|
||||
- Offline crash fix ([#152](https://github.com/jamiepine/voicebox/pull/152))
|
||||
- Atomic audio saves ([#263](https://github.com/jamiepine/voicebox/pull/263))
|
||||
- Filesystem health endpoint
|
||||
- Chatterbox float64 dtype fix ([#264](https://github.com/jamiepine/voicebox/pull/264))
|
||||
|
||||
### Changed
|
||||
- **README** - Added Makefile reference and updated Quick Start with Makefile-based setup instructions alongside manual setup
|
||||
### Accessibility ([#243](https://github.com/jamiepine/voicebox/pull/243))
|
||||
|
||||
---
|
||||
Screen reader support, keyboard navigation, state-aware `aria-label` attributes on all interactive controls.
|
||||
|
||||
## [Unreleased - Planned]
|
||||
### UI Polish
|
||||
|
||||
### Planned
|
||||
- Real-time streaming synthesis
|
||||
- Conversation mode with multiple speakers
|
||||
- Voice effects (pitch shift, reverb, M3GAN-style)
|
||||
- Timeline-based audio editor
|
||||
- Additional voice models (XTTS, Bark)
|
||||
- Voice design from text descriptions
|
||||
- Project system for saving sessions
|
||||
- Plugin architecture
|
||||
- Redesigned landing page ([#274](https://github.com/jamiepine/voicebox/pull/274))
|
||||
- Voices tab overhaul with inline inspector
|
||||
- Responsive layout improvements
|
||||
- Duplicate profile name validation ([#175](https://github.com/jamiepine/voicebox/pull/175))
|
||||
|
||||
---
|
||||
### Community Contributors
|
||||
|
||||
[@haosenwang1018](https://github.com/haosenwang1018), [@Balneario-de-Cofrentes](https://github.com/Balneario-de-Cofrentes), [@ageofalgo](https://github.com/ageofalgo), [@mikeswann](https://github.com/mikeswann), [@rayl15](https://github.com/rayl15), [@mpecanha](https://github.com/mpecanha), [@ways2read](https://github.com/ways2read), [@ieguiguren](https://github.com/ieguiguren), [@Vaibhavee89](https://github.com/Vaibhavee89), [@pandego](https://github.com/pandego), [@luminest-llc](https://github.com/luminest-llc)
|
||||
|
||||
## [0.1.13] - 2026-02-23
|
||||
|
||||
### Stability and reliability
|
||||
|
||||
- [#95](https://github.com/jamiepine/voicebox/pull/95) Fix: selecting 0.6B model still downloads and uses 1.7B
|
||||
- [#93](https://github.com/jamiepine/voicebox/pull/93) fix(mlx): bundle native libs and broaden error handling for Apple Silicon
|
||||
- [#79](https://github.com/jamiepine/voicebox/pull/79) fix: handle non-ASCII filenames in Content-Disposition headers
|
||||
- [#78](https://github.com/jamiepine/voicebox/pull/78) fix: guard getUserMedia call against undefined mediaDevices in non-secure contexts
|
||||
- [#77](https://github.com/jamiepine/voicebox/pull/77) fix: await for confirmation before deleting voices and channels
|
||||
- [#128](https://github.com/jamiepine/voicebox/pull/128) fix: resolve multiple issues (#96, #119, #111, #108, #121, #125, #127)
|
||||
- [#40](https://github.com/jamiepine/voicebox/pull/40) Fix: audio export path resolution
|
||||
|
||||
### Build and packaging
|
||||
|
||||
- [#122](https://github.com/jamiepine/voicebox/pull/122) fix(web): add @tailwindcss/vite plugin to web config
|
||||
- [#126](https://github.com/jamiepine/voicebox/pull/126) Create requirements.txt
|
||||
|
||||
### UX and docs
|
||||
|
||||
- [#44](https://github.com/jamiepine/voicebox/pull/44) Enhances floating generate box UX
|
||||
- [#57](https://github.com/jamiepine/voicebox/pull/57) chore: updates repo URL in README
|
||||
- [#146](https://github.com/jamiepine/voicebox/pull/146) Add Spacebot banner to landing page
|
||||
- [#1](https://github.com/jamiepine/voicebox/pull/1) Improvements
|
||||
|
||||
## [0.1.12] - 2026-01-31
|
||||
|
||||
### Model Download UX Overhaul
|
||||
|
||||
- Real-time download progress tracking with accurate percentage and speed info
|
||||
- No more downloading notifications during generation even when its not downloading
|
||||
- Better error handling and status reporting throughout the download process
|
||||
|
||||
### Other Improvements
|
||||
|
||||
- Enhanced health check endpoint with GPU type information
|
||||
- Improved model caching verification
|
||||
- More reliable SSE progress updates
|
||||
- Actual update notifications — no need to manually check in settings anymore
|
||||
|
||||
## [0.1.11] - 2026-01-30
|
||||
|
||||
- Fixed transcriptions on MLX
|
||||
- Fixed model download progress (finally)
|
||||
|
||||
## [0.1.10] - 2026-01-30
|
||||
|
||||
### Faster generation on Apple Silicon
|
||||
|
||||
Massive speed gains, from around 20s per generation to 2-3s. Added native MLX backend support for Apple Silicon, providing significantly faster TTS and STT generation on M-series macOS machines.
|
||||
|
||||
- **MLX Backend** — New backend implementation optimized for Apple Silicon using MLX framework
|
||||
- **Dynamic Backend Selection** — Automatically detects platform and selects between MLX (macOS) and PyTorch (other platforms)
|
||||
- Refactored TTS and STT logic into modular backend implementations
|
||||
- Updated build process to include MLX-specific dependencies for macOS builds
|
||||
|
||||
## [0.1.9] - 2026-01-30
|
||||
|
||||
### Improved voice profile creation flow
|
||||
|
||||
- Voice create drafts: No longer lose work if you close the modal
|
||||
- Fixed whisper only transcribing English or Chinese, now has support for all languages
|
||||
|
||||
### Improved Stories editor
|
||||
|
||||
- Added spacebar for play/pause
|
||||
- Timeline now auto-scrolls to follow playhead during playback
|
||||
- Fixed misalignment of the items with mouse when picking up
|
||||
- Fixed hitbox for selecting an item
|
||||
- Fixed playhead jumping forward when pressing play
|
||||
|
||||
### Generation box improvements
|
||||
|
||||
- Instruct mode no longer wipes prompt text
|
||||
- Improved UI cleanliness
|
||||
|
||||
### Misc
|
||||
|
||||
- Fixed "Model downloading" toast during generation when model is already downloaded
|
||||
|
||||
## [0.1.8] - 2026-01-29
|
||||
|
||||
### Model Download Timeout Issues
|
||||
|
||||
Fixed critical issue where model downloads would fail with "Failed to fetch" errors on Windows. Refactored download endpoints to return immediately and continue downloads in background.
|
||||
|
||||
### Cross-Platform Cache Path Issues
|
||||
|
||||
Fixed hardcoded `~/.cache/huggingface/hub` paths that don't work on Windows. All cache paths now use `hf_constants.HF_HUB_CACHE` for proper cross-platform support.
|
||||
|
||||
### Windows Process Management
|
||||
|
||||
- Added `/shutdown` endpoint for graceful server shutdown on Windows
|
||||
- Added `gpu_type` field to health check response
|
||||
|
||||
## [0.1.7] - 2026-01-29
|
||||
|
||||
- Trim and split audio clips in Story Editor
|
||||
- Auto-activation of stories in Story Editor with visible playhead
|
||||
- Conditional auto-play support in AudioPlayer for better user control
|
||||
- Refactored audio loading across HistoryTable, SampleList, and generation forms
|
||||
- Audio now only auto-plays when explicitly intended, preventing unexpected playback
|
||||
|
||||
## [0.1.6] - 2026-01-29
|
||||
|
||||
### Introducing Stories
|
||||
|
||||
A full voice editor for composing podcasts and generated conversations.
|
||||
|
||||
- **Stories Editor** — Create multi-voice narratives, podcasts, or conversations with a timeline-based editor
|
||||
- Compose tracks with different voices
|
||||
- Edit and arrange audio segments inline
|
||||
- Build generated conversations with multiple participants
|
||||
- **Improved Voice Generation UI** — Auto-resizing input, default voice selection, better layout
|
||||
- **Track Editor Integration** — Inline track editing within story items
|
||||
|
||||
## [0.1.5] - 2026-01-28
|
||||
|
||||
Fixed recording length limit at 0:29 to auto stop instead of passing the limit and getting an error, which would cause users to lose their recording.
|
||||
|
||||
## [0.1.4] - 2026-01-28
|
||||
|
||||
- Audio channel management system
|
||||
- Native audio playback handling in AudioPlayer component
|
||||
- Refactored ConnectionForm and Checkbox components
|
||||
- Improved layout consistency and responsiveness
|
||||
- Added safe area constants for better responsive design
|
||||
|
||||
## [0.1.3] - 2026-01-27
|
||||
|
||||
- Improved the generate textbox
|
||||
- Maybe fixed Windows autoupdate restarting entire computer
|
||||
|
||||
## [0.1.2] - 2026-01-27
|
||||
|
||||
### Audio Capture & Format Conversion
|
||||
|
||||
- Added audio format conversion util
|
||||
- Enhanced system audio capture on macOS and Windows
|
||||
- Improved audio recording hooks
|
||||
- Added audio input entitlement for macOS
|
||||
- Added audio capture tests
|
||||
|
||||
### Update System
|
||||
|
||||
- Enhanced auto-updater functionality and update status display
|
||||
|
||||
## [0.1.1] - 2026-01-27
|
||||
|
||||
### Platform Support
|
||||
|
||||
- **macOS Audio Capture** — Native audio capture support for sample creation
|
||||
- **Windows Audio Capture** — WASAPI implementation with improved thread safety
|
||||
- **Linux Support** — Temporarily removed builds due to runner disk space constraints
|
||||
|
||||
### Audio Features
|
||||
|
||||
- Play/pause for audio samples across all components
|
||||
- Three new sample components: Recording, System capture, Upload with drag-and-drop
|
||||
- Audio validation, error handling, and consistent cleanup
|
||||
|
||||
### Voice Profile Management
|
||||
|
||||
- Profile import with file size validation (100MB limit)
|
||||
- Enhanced profile form with new audio sample components
|
||||
- Drag-and-drop support for audio file uploads
|
||||
|
||||
### Server Management
|
||||
|
||||
- Changed default URL from `localhost:8000` to `127.0.0.1:17493`
|
||||
- Server reuse logic, "keep server running" preference, orphaned process handling
|
||||
|
||||
### Build & Release
|
||||
|
||||
- Added `.bumpversion.cfg` for automated version management
|
||||
- Enhanced icon generation script for multi-size Windows icons
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fixed date formatting for timezone-less date strings
|
||||
- Fixed getLatestRelease file filtering
|
||||
- Improved audio duration metadata on Windows
|
||||
|
||||
## [0.1.0] - 2026-01-27
|
||||
|
||||
The first public release of Voicebox — an open-source voice synthesis studio powered by Qwen3-TTS.
|
||||
|
||||
### Voice Cloning with Qwen3-TTS
|
||||
|
||||
- Automatic model download from HuggingFace
|
||||
- Multiple model sizes (1.7B and 0.6B)
|
||||
- Voice prompt caching for instant regeneration
|
||||
- English and Chinese support
|
||||
|
||||
### Voice Profile Management
|
||||
|
||||
- Create profiles from audio files or record directly in the app
|
||||
- Multiple samples per profile for higher quality cloning
|
||||
- Import/Export profiles
|
||||
- Automatic transcription via Whisper
|
||||
|
||||
### Speech Generation
|
||||
|
||||
- Simple text-to-speech with profile selection
|
||||
- Seed control for reproducible generations
|
||||
- Long-form support up to 5,000 characters
|
||||
|
||||
### Generation History
|
||||
|
||||
- Full history with metadata
|
||||
- Search by text content
|
||||
- Inline playback and download
|
||||
|
||||
### Flexible Deployment
|
||||
|
||||
- Local mode with bundled backend
|
||||
- Remote mode for GPU servers on your network
|
||||
- One-click server setup
|
||||
|
||||
### Desktop Experience
|
||||
|
||||
- Built with Tauri v2 (Rust) — native performance, not Electron
|
||||
- Cross-platform: macOS and Windows
|
||||
- No Python installation required
|
||||
|
||||
### Tech Stack
|
||||
|
||||
Tauri v2, React, TypeScript, Tailwind CSS, FastAPI, Qwen3-TTS, Whisper, SQLite
|
||||
|
||||
[Unreleased]: https://github.com/jamiepine/voicebox/compare/v0.4.1...HEAD
|
||||
[0.4.1]: https://github.com/jamiepine/voicebox/compare/v0.4.0...v0.4.1
|
||||
[0.4.0]: https://github.com/jamiepine/voicebox/compare/v0.3.0...v0.4.0
|
||||
[0.3.0]: https://github.com/jamiepine/voicebox/compare/v0.2.3...v0.3.0
|
||||
[0.2.3]: https://github.com/jamiepine/voicebox/compare/v0.2.2...v0.2.3
|
||||
[0.2.2]: https://github.com/jamiepine/voicebox/compare/v0.2.1...v0.2.2
|
||||
[0.2.1]: https://github.com/jamiepine/voicebox/compare/v0.1.13...v0.2.1
|
||||
[0.1.13]: https://github.com/jamiepine/voicebox/compare/v0.1.12...v0.1.13
|
||||
[0.1.12]: https://github.com/jamiepine/voicebox/compare/v0.1.11...v0.1.12
|
||||
[0.1.11]: https://github.com/jamiepine/voicebox/compare/v0.1.10...v0.1.11
|
||||
[0.1.10]: https://github.com/jamiepine/voicebox/compare/v0.1.9...v0.1.10
|
||||
[0.1.9]: https://github.com/jamiepine/voicebox/compare/v0.1.8...v0.1.9
|
||||
[0.1.8]: https://github.com/jamiepine/voicebox/compare/v0.1.7...v0.1.8
|
||||
[0.1.7]: https://github.com/jamiepine/voicebox/compare/v0.1.6...v0.1.7
|
||||
[0.1.6]: https://github.com/jamiepine/voicebox/compare/v0.1.5...v0.1.6
|
||||
[0.1.5]: https://github.com/jamiepine/voicebox/compare/v0.1.4...v0.1.5
|
||||
[0.1.4]: https://github.com/jamiepine/voicebox/compare/v0.1.3...v0.1.4
|
||||
[0.1.3]: https://github.com/jamiepine/voicebox/compare/v0.1.2...v0.1.3
|
||||
[0.1.2]: https://github.com/jamiepine/voicebox/compare/v0.1.1...v0.1.2
|
||||
[0.1.1]: https://github.com/jamiepine/voicebox/compare/v0.1.0...v0.1.1
|
||||
[0.1.0]: https://github.com/jamiepine/voicebox/releases/tag/v0.1.0
|
||||
|
||||
+4
-3
@@ -260,7 +260,7 @@ voicebox/
|
||||
|
||||
### ✨ New Features
|
||||
|
||||
- Check the roadmap in README.md
|
||||
- Check the roadmap in README.md and the engineering status in [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) before proposing work — it lists prioritized tasks (Tier 1 → 3), known architectural bottlenecks, and candidate TTS engines already under evaluation (including why some have been backlogged)
|
||||
- Discuss major features in an issue first
|
||||
- Keep features focused and well-scoped
|
||||
|
||||
@@ -359,7 +359,7 @@ Releases are managed by maintainers:
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and solutions.
|
||||
See [docs/content/docs/overview/troubleshooting.mdx](docs/content/docs/overview/troubleshooting.mdx) for common issues and solutions.
|
||||
|
||||
**Quick fixes:**
|
||||
|
||||
@@ -372,12 +372,13 @@ See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and sol
|
||||
- Open an issue for bugs or feature requests
|
||||
- Check existing issues and discussions
|
||||
- Review the codebase to understand patterns
|
||||
- See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues
|
||||
- See [docs/content/docs/overview/troubleshooting.mdx](docs/content/docs/overview/troubleshooting.mdx) for common issues
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [README.md](README.md) - Project overview
|
||||
- [backend/README.md](backend/README.md) - API documentation
|
||||
- [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) - Living engineering roadmap: architecture, shipped vs in-flight work, prioritized open issues, candidate TTS engines under evaluation, architectural bottlenecks. Keep this updated when you ship significant features, close or backlog a model integration, or identify new bottlenecks.
|
||||
- [docs/AUTOUPDATER_QUICKSTART.md](docs/AUTOUPDATER_QUICKSTART.md) - Auto-updater setup
|
||||
- [SECURITY.md](SECURITY.md) - Security policy
|
||||
- [CHANGELOG.md](CHANGELOG.md) - Version history
|
||||
|
||||
+5
-1
@@ -9,7 +9,7 @@ FROM oven/bun:1 AS frontend
|
||||
WORKDIR /build
|
||||
|
||||
# Copy workspace config and frontend source
|
||||
COPY package.json bun.lock ./
|
||||
COPY package.json bun.lock CHANGELOG.md ./
|
||||
COPY app/ ./app/
|
||||
COPY web/ ./web/
|
||||
|
||||
@@ -31,8 +31,12 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN pip install --no-cache-dir --upgrade pip
|
||||
|
||||
COPY backend/requirements.txt .
|
||||
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
||||
RUN pip install --no-cache-dir --prefix=/install --no-deps chatterbox-tts
|
||||
RUN pip install --no-cache-dir --prefix=/install --no-deps hume-tada
|
||||
RUN pip install --no-cache-dir --prefix=/install \
|
||||
git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
|
||||
|
||||
@@ -1,250 +0,0 @@
|
||||
# Voicebox Makefile
|
||||
# Unix-only (macOS/Linux). Windows users should use WSL.
|
||||
|
||||
SHELL := /bin/bash
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
# Directories
|
||||
BACKEND_DIR := backend
|
||||
TAURI_DIR := tauri
|
||||
WEB_DIR := web
|
||||
APP_DIR := app
|
||||
|
||||
# Python (prefer 3.12, fallback to 3.13, then python3)
|
||||
PYTHON := $(shell command -v python3.12 2>/dev/null || command -v python3.13 2>/dev/null || echo python3)
|
||||
VENV := $(CURDIR)/$(BACKEND_DIR)/venv
|
||||
VENV_BIN := $(VENV)/bin
|
||||
PIP := $(VENV_BIN)/pip
|
||||
PYTHON_VENV := $(VENV_BIN)/python
|
||||
|
||||
# Colors for output
|
||||
BLUE := \033[0;34m
|
||||
GREEN := \033[0;32m
|
||||
YELLOW := \033[0;33m
|
||||
NC := \033[0m # No Color
|
||||
|
||||
.PHONY: help
|
||||
help: ## Show this help message
|
||||
@echo -e "$(BLUE)Voicebox$(NC) - Development Commands"
|
||||
@echo ""
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
|
||||
awk 'BEGIN {FS = ":.*?## "}; {printf " $(GREEN)%-20s$(NC) %s\n", $$1, $$2}'
|
||||
|
||||
# =============================================================================
|
||||
# SETUP
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: setup setup-js setup-python setup-rust
|
||||
|
||||
setup: setup-js setup-python ## Full project setup (all dependencies)
|
||||
@echo -e "$(GREEN)✓ Setup complete!$(NC)"
|
||||
@echo -e " Run $(YELLOW)make dev$(NC) to start development servers"
|
||||
|
||||
setup-js: ## Install JavaScript dependencies (bun)
|
||||
@echo -e "$(BLUE)Installing JavaScript dependencies...$(NC)"
|
||||
bun install
|
||||
|
||||
setup-python: $(VENV)/bin/activate ## Set up Python virtual environment and dependencies
|
||||
@echo -e "$(BLUE)Installing Python dependencies...$(NC)"
|
||||
$(PIP) install --upgrade pip
|
||||
$(PIP) install -r $(BACKEND_DIR)/requirements.txt
|
||||
$(PIP) install --no-deps chatterbox-tts
|
||||
@if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \
|
||||
echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \
|
||||
$(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \
|
||||
echo -e "$(GREEN)✓ MLX backend enabled (native Metal acceleration)$(NC)"; \
|
||||
fi
|
||||
$(PIP) install git+https://github.com/QwenLM/Qwen3-TTS.git
|
||||
@echo -e "$(GREEN)✓ Python environment ready$(NC)"
|
||||
|
||||
$(VENV)/bin/activate:
|
||||
@echo -e "$(BLUE)Creating Python virtual environment...$(NC)"
|
||||
@PY_MINOR=$$($(PYTHON) -c "import sys; print(sys.version_info[1])"); \
|
||||
if [ "$$PY_MINOR" -gt 13 ]; then \
|
||||
echo -e "$(YELLOW)Warning: Python 3.$$PY_MINOR detected. ML packages may not be compatible.$(NC)"; \
|
||||
echo -e "$(YELLOW)Recommended: Use Python 3.12 or 3.13 (brew install [email protected])$(NC)"; \
|
||||
fi
|
||||
$(PYTHON) -m venv $(VENV)
|
||||
|
||||
setup-rust: ## Install Rust toolchain (if not present)
|
||||
@command -v rustc >/dev/null 2>&1 || curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
|
||||
# =============================================================================
|
||||
# DEVELOPMENT
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: dev dev-backend dev-frontend dev-web kill-dev
|
||||
|
||||
dev: ## Start backend + desktop app (parallel)
|
||||
@echo -e "$(BLUE)Starting development servers...$(NC)"
|
||||
@echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)"
|
||||
@trap 'kill 0' EXIT; \
|
||||
$(MAKE) dev-backend & \
|
||||
sleep 2 && if [ "$$(uname)" = "Linux" ] && lspci 2>/dev/null | grep -qi nvidia; then \
|
||||
WEBKIT_DISABLE_DMABUF_RENDERER=1 $(MAKE) dev-frontend; \
|
||||
else \
|
||||
$(MAKE) dev-frontend; \
|
||||
fi & \
|
||||
wait
|
||||
|
||||
dev-backend: ## Start FastAPI backend server
|
||||
@echo -e "$(BLUE)Starting backend server on http://localhost:17493$(NC)"
|
||||
$(VENV_BIN)/uvicorn backend.main:app --reload --port 17493
|
||||
|
||||
dev-frontend: ## Start Tauri desktop app
|
||||
@echo -e "$(BLUE)Starting Tauri desktop app...$(NC)"
|
||||
bun run dev
|
||||
|
||||
dev-web: ## Start backend + web app (parallel)
|
||||
@echo -e "$(BLUE)Starting web development servers...$(NC)"
|
||||
@trap 'kill 0' EXIT; \
|
||||
$(MAKE) dev-backend & \
|
||||
sleep 2 && cd $(WEB_DIR) && bun run dev & \
|
||||
wait
|
||||
|
||||
kill-dev: ## Kill all development processes
|
||||
@echo -e "$(YELLOW)Killing development processes...$(NC)"
|
||||
-pkill -f "uvicorn main:app" 2>/dev/null || true
|
||||
-pkill -f "vite" 2>/dev/null || true
|
||||
@echo -e "$(GREEN)✓ Processes killed$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# BUILD
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: build build-server build-tauri build-web
|
||||
|
||||
build: build-server build-tauri ## Build everything (server binary + desktop app)
|
||||
@echo -e "$(GREEN)✓ Build complete!$(NC)"
|
||||
|
||||
build-server: ## Build Python server binary
|
||||
@echo -e "$(BLUE)Building server binary...$(NC)"
|
||||
PATH="$(VENV_BIN):$$PATH" ./scripts/build-server.sh
|
||||
|
||||
build-tauri: ## Build Tauri desktop app
|
||||
@echo -e "$(BLUE)Building Tauri desktop app...$(NC)"
|
||||
cd $(TAURI_DIR) && bun run tauri build
|
||||
|
||||
build-web: ## Build web app
|
||||
@echo -e "$(BLUE)Building web app...$(NC)"
|
||||
cd $(WEB_DIR) && bun run build
|
||||
@echo -e "$(GREEN)✓ Web build output in $(WEB_DIR)/dist/$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# DATABASE & API
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: db-init db-reset generate-api
|
||||
|
||||
db-init: $(VENV)/bin/activate ## Initialize SQLite database
|
||||
@echo -e "$(BLUE)Initializing database...$(NC)"
|
||||
cd $(BACKEND_DIR) && $(PYTHON_VENV) -c "from database import init_db; init_db()"
|
||||
@echo -e "$(GREEN)✓ Database created at $(BACKEND_DIR)/data/voicebox.db$(NC)"
|
||||
|
||||
db-reset: ## Reset database (delete and reinitialize)
|
||||
@echo -e "$(YELLOW)Resetting database...$(NC)"
|
||||
rm -f $(BACKEND_DIR)/data/voicebox.db
|
||||
$(MAKE) db-init
|
||||
|
||||
generate-api: ## Generate TypeScript API client from OpenAPI schema
|
||||
@echo -e "$(BLUE)Generating API client...$(NC)"
|
||||
@echo -e "$(YELLOW)Note: Backend must be running (make dev-backend)$(NC)"
|
||||
./scripts/generate-api.sh
|
||||
@echo -e "$(GREEN)✓ API client generated in $(APP_DIR)/src/lib/api/$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# CODE QUALITY
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: lint format typecheck check
|
||||
|
||||
lint: ## Run linter (Biome)
|
||||
@echo -e "$(BLUE)Linting...$(NC)"
|
||||
bun run lint
|
||||
|
||||
format: ## Format code (Biome)
|
||||
@echo -e "$(BLUE)Formatting...$(NC)"
|
||||
bun run format
|
||||
|
||||
typecheck: ## Run TypeScript type checking
|
||||
@echo -e "$(BLUE)Type checking...$(NC)"
|
||||
bun run tsc --noEmit
|
||||
|
||||
check: ## Run all checks (Biome lint + format + type check)
|
||||
@echo -e "$(BLUE)Running all checks...$(NC)"
|
||||
bun run check
|
||||
@echo -e "$(GREEN)✓ All checks passed$(NC)"
|
||||
|
||||
# =============================================================================
|
||||
# TESTING
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: test test-backend test-frontend
|
||||
|
||||
test: test-backend test-frontend ## Run all tests
|
||||
@echo -e "$(GREEN)✓ All tests passed$(NC)"
|
||||
|
||||
test-backend: ## Run Python backend tests (requires pytest)
|
||||
@echo -e "$(BLUE)Running backend tests...$(NC)"
|
||||
@if [ -f "$(VENV_BIN)/pytest" ]; then \
|
||||
cd $(BACKEND_DIR) && $(VENV_BIN)/pytest -v; \
|
||||
else \
|
||||
echo -e "$(YELLOW)pytest not installed. Run: $(PIP) install pytest$(NC)"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
test-frontend: ## Run frontend tests (requires test script in package.json)
|
||||
@echo -e "$(BLUE)Running frontend tests...$(NC)"
|
||||
@if bun run test --help >/dev/null 2>&1; then \
|
||||
bun run test; \
|
||||
else \
|
||||
echo -e "$(YELLOW)No test script configured$(NC)"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# LOGS & DEBUGGING
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: logs docs
|
||||
|
||||
logs: ## Tail backend logs
|
||||
@echo -e "$(BLUE)Tailing logs (Ctrl+C to stop)...$(NC)"
|
||||
tail -f $(BACKEND_DIR)/logs/*.log 2>/dev/null || echo "No log files found"
|
||||
|
||||
docs: ## Open API documentation (backend must be running)
|
||||
@echo -e "$(BLUE)Opening API docs...$(NC)"
|
||||
open http://localhost:17493/docs 2>/dev/null || xdg-open http://localhost:17493/docs
|
||||
|
||||
# =============================================================================
|
||||
# CLEAN
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: clean clean-python clean-build clean-all
|
||||
|
||||
clean: ## Clean build artifacts
|
||||
@echo -e "$(BLUE)Cleaning build artifacts...$(NC)"
|
||||
rm -rf $(TAURI_DIR)/src-tauri/target/release
|
||||
rm -rf $(WEB_DIR)/dist
|
||||
rm -rf $(APP_DIR)/dist
|
||||
@echo -e "$(GREEN)✓ Build artifacts cleaned$(NC)"
|
||||
|
||||
clean-python: ## Clean Python cache and virtual environment
|
||||
@echo -e "$(BLUE)Cleaning Python files...$(NC)"
|
||||
rm -rf $(VENV)
|
||||
find $(BACKEND_DIR) -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find $(BACKEND_DIR) -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
@echo -e "$(GREEN)✓ Python environment cleaned$(NC)"
|
||||
|
||||
clean-build: ## Clean Rust/Tauri build cache
|
||||
@echo -e "$(BLUE)Cleaning Rust build cache...$(NC)"
|
||||
cd $(TAURI_DIR)/src-tauri && cargo clean
|
||||
@echo -e "$(GREEN)✓ Rust cache cleaned$(NC)"
|
||||
|
||||
clean-all: clean clean-python clean-build ## Nuclear clean (everything)
|
||||
@echo -e "$(BLUE)Cleaning node_modules...$(NC)"
|
||||
rm -rf node_modules
|
||||
rm -rf $(APP_DIR)/node_modules
|
||||
rm -rf $(TAURI_DIR)/node_modules
|
||||
rm -rf $(WEB_DIR)/node_modules
|
||||
@echo -e "$(GREEN)✓ Full clean complete$(NC)"
|
||||
@@ -1,58 +0,0 @@
|
||||
# Voicebox Offline Mode Fix
|
||||
|
||||
## Problem
|
||||
Voicebox crashes when generating speech if HuggingFace is unreachable, even when models are fully cached locally.
|
||||
|
||||
**Root Cause:**
|
||||
- Voicebox downloads `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` (MLX optimized version)
|
||||
- But `mlx_audio.tts.load()` tries to fetch `config.json` from original repo `Qwen/Qwen3-TTS-12Hz-1.7B-Base`
|
||||
- This network request fails → server crashes with `RemoteDisconnected`
|
||||
|
||||
**Related Issues:**
|
||||
- Issue #150: "Internet connection required, even though models are downloaded?"
|
||||
- Issue #151: "API Stability Issues: Model Loading Hangs and Server Crashes"
|
||||
|
||||
## Solution
|
||||
Two-part fix:
|
||||
|
||||
### 1. Monkey-patch huggingface_hub (`backend/utils/hf_offline_patch.py`)
|
||||
- Intercepts cache lookup functions
|
||||
- Forces offline mode early (before mlx_audio imports)
|
||||
- Adds debug logging for cache hits/misses
|
||||
|
||||
### 2. Symlink original repo to MLX version (`ensure_original_qwen_config_cached()`)
|
||||
- When original `Qwen/Qwen3-TTS-12Hz-1.7B-Base` cache doesn't exist
|
||||
- But MLX `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` does exist
|
||||
- Creates a symlink so cache lookups succeed
|
||||
|
||||
## Files Changed
|
||||
- `backend/backends/mlx_backend.py` - Added patch imports at top
|
||||
- `backend/utils/hf_offline_patch.py` - New patch module
|
||||
|
||||
## Testing
|
||||
To test this fix:
|
||||
1. Build Voicebox from source: `make build`
|
||||
2. Disconnect from internet
|
||||
3. Try generating speech
|
||||
4. Should work without network requests
|
||||
|
||||
## Build Instructions
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Build the app
|
||||
make build
|
||||
|
||||
# Or build just the server
|
||||
make build-server
|
||||
```
|
||||
|
||||
## Notes
|
||||
- The patch is applied automatically when `mlx_backend.py` is imported
|
||||
- Set `VOICEBOX_OFFLINE_PATCH=0` to disable the patch
|
||||
- The symlink approach works because the config.json is compatible between versions
|
||||
|
||||
---
|
||||
*Patch contributed by community*
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
<p align="center">
|
||||
<strong>The open-source voice synthesis studio.</strong><br/>
|
||||
Clone voices. Generate speech. Build voice-powered apps.<br/>
|
||||
Clone voices. Generate speech. Apply effects. Build voice-powered apps.<br/>
|
||||
All running locally on your machine.
|
||||
</p>
|
||||
|
||||
@@ -23,14 +23,18 @@
|
||||
<a href="https://github.com/jamiepine/voicebox/blob/main/LICENSE">
|
||||
<img src="https://img.shields.io/github/license/jamiepine/voicebox?style=flat" alt="License" />
|
||||
</a>
|
||||
<a href="https://deepwiki.com/jamiepine/voicebox">
|
||||
<img src="https://img.shields.io/static/v1?label=Ask&message=DeepWiki&color=5B6EF7" alt="Ask DeepWiki" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://voicebox.sh">voicebox.sh</a> •
|
||||
<a href="https://docs.voicebox.sh">Docs</a> •
|
||||
<a href="#download">Download</a> •
|
||||
<a href="#features">Features</a> •
|
||||
<a href="#api">API</a> •
|
||||
<a href="#roadmap">Roadmap</a>
|
||||
<a href="docs/content/docs/overview/troubleshooting.mdx">Troubleshooting</a>
|
||||
</p>
|
||||
|
||||
<br/>
|
||||
@@ -59,96 +63,158 @@
|
||||
|
||||
## What is Voicebox?
|
||||
|
||||
Voicebox is a **local-first voice cloning studio** with DAW-like features for professional voice synthesis. Think of it as a **local, free and open-source alternative to ElevenLabs** — download models, clone voices, and generate speech entirely on your machine.
|
||||
|
||||
Unlike cloud services that lock your voice data behind subscriptions, Voicebox gives you:
|
||||
Voicebox is a **local-first voice cloning studio** — a free and open-source alternative to ElevenLabs. Clone voices from a few seconds of audio or pick from 50+ preset voices, generate speech in 23 languages across 7 TTS engines, apply post-processing effects, and compose multi-voice projects with a timeline editor.
|
||||
|
||||
- **Complete privacy** — models and voice data stay on your machine
|
||||
- **Professional tools** — multi-track timeline editor, audio trimming, conversation mixing
|
||||
- **Model flexibility** — currently powered by Qwen3-TTS, with support for XTTS, Bark, and other models coming soon
|
||||
- **API-first** — use the desktop app or integrate voice synthesis into your own projects
|
||||
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
|
||||
- **Cloning and preset voices** — zero-shot cloning from a reference sample, or curated preset voices via Kokoro (50 voices) and Qwen CustomVoice (9 voices)
|
||||
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
|
||||
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
|
||||
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
|
||||
- **Unlimited length** — auto-chunking with crossfade for scripts, articles, and chapters
|
||||
- **Stories editor** — multi-track timeline for conversations, podcasts, and narratives
|
||||
- **API-first** — REST API for integrating voice synthesis into your own projects
|
||||
- **Native performance** — built with Tauri (Rust), not Electron
|
||||
- **Super fast on Mac** — MLX backend with native Metal acceleration for 4-5x faster inference on Apple Silicon
|
||||
|
||||
Download a voice model, clone any voice from a few seconds of audio, and compose multi-voice projects with studio-grade editing tools. No Python install required, no cloud dependency, no limits.
|
||||
- **Runs everywhere** — macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker
|
||||
|
||||
---
|
||||
|
||||
## Download
|
||||
|
||||
Voicebox is available now for macOS and Windows.
|
||||
| Platform | Download |
|
||||
| --------------------- | ------------------------------------------------------ |
|
||||
| macOS (Apple Silicon) | [Download DMG](https://voicebox.sh/download/mac-arm) |
|
||||
| macOS (Intel) | [Download DMG](https://voicebox.sh/download/mac-intel) |
|
||||
| Windows | [Download MSI](https://voicebox.sh/download/windows) |
|
||||
| Docker | `docker compose up` |
|
||||
|
||||
| Platform | Download |
|
||||
|----------|----------|
|
||||
| macOS (Apple Silicon) | [Voicebox_aarch64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/Voicebox_aarch64.app.tar.gz) |
|
||||
| macOS (Intel) | [Voicebox_x64.app.tar.gz](https://github.com/jamiepine/voicebox/releases/latest/download/Voicebox_x64.app.tar.gz) |
|
||||
| Windows (MSI) | [Latest Windows MSI](https://github.com/jamiepine/voicebox/releases/latest) |
|
||||
| Windows (Setup) | [Latest Windows Setup](https://github.com/jamiepine/voicebox/releases/latest) |
|
||||
> **[View all binaries →](https://github.com/jamiepine/voicebox/releases/latest)**
|
||||
|
||||
> **Linux** — Pre-built binaries are not yet available. Linux users can compile from source, see [Development](#development) below.
|
||||
> **Linux** — Pre-built binaries are not yet available. See [voicebox.sh/linux-install](https://voicebox.sh/linux-install) for build-from-source instructions.
|
||||
|
||||
> **Having trouble?** See the [Troubleshooting Guide](docs/content/docs/overview/troubleshooting.mdx) for common install, generation, model-download, and GPU issues.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
### Voice Cloning with Qwen3-TTS
|
||||
### Multi-Engine Voice Cloning
|
||||
|
||||
Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-perfect voice cloning from just a few seconds of audio.
|
||||
Seven TTS engines with different strengths, switchable per-generation:
|
||||
|
||||
- **Instant cloning** — Upload a sample, get a voice profile
|
||||
- **High fidelity** — Natural prosody, emotion, and cadence
|
||||
- **Multi-language** — English, Chinese, and more coming
|
||||
- **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super-fast generation
|
||||
| Engine | Languages | Strengths |
|
||||
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Qwen3-TTS** (0.6B / 1.7B) | 10 | High-quality multilingual cloning, delivery instructions ("speak slowly", "whisper") |
|
||||
| **Qwen CustomVoice** | 10 | 9 curated preset voices with natural-language delivery control — no reference audio required |
|
||||
| **LuxTTS** | English | Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU |
|
||||
| **Chatterbox Multilingual** | 23 | Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more |
|
||||
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
|
||||
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
|
||||
| **Kokoro** | 8 | 50 curated preset voices, tiny 82M model, fast CPU inference |
|
||||
|
||||
### Emotions & Paralinguistic Tags
|
||||
|
||||
Only **Chatterbox Turbo** interprets paralinguistic tags like `[laugh]` and
|
||||
`[sigh]`. Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and HumeAI TADA read them
|
||||
literally as text.
|
||||
|
||||
With **Chatterbox Turbo** selected, type `/` in the text input to open the tag
|
||||
inserter and add expressive tags inline with speech:
|
||||
|
||||
`[laugh]` `[chuckle]` `[gasp]` `[cough]` `[sigh]` `[groan]` `[sniff]` `[shush]` `[clear throat]`
|
||||
|
||||
### Post-Processing Effects
|
||||
|
||||
8 audio effects powered by Spotify's `pedalboard` library. Apply after generation, preview in real time, build reusable presets.
|
||||
|
||||
| Effect | Description |
|
||||
| ---------------- | --------------------------------------------- |
|
||||
| Pitch Shift | Up or down by up to 12 semitones |
|
||||
| Reverb | Configurable room size, damping, wet/dry mix |
|
||||
| Delay | Echo with adjustable time, feedback, and mix |
|
||||
| Chorus / Flanger | Modulated delay for metallic or lush textures |
|
||||
| Compressor | Dynamic range compression |
|
||||
| Gain | Volume adjustment (-40 to +40 dB) |
|
||||
| High-Pass Filter | Remove low frequencies |
|
||||
| Low-Pass Filter | Remove high frequencies |
|
||||
|
||||
Ships with 4 built-in presets (Robotic, Radio, Echo Chamber, Deep Voice) and supports custom presets. Effects can be assigned per-profile as defaults.
|
||||
|
||||
### Unlimited Generation Length
|
||||
|
||||
Text is automatically split at sentence boundaries and each chunk is generated independently, then crossfaded together. Works with all engines.
|
||||
|
||||
- Configurable auto-chunking limit (100–5,000 chars)
|
||||
- Crossfade slider (0–200ms) for smooth transitions
|
||||
- Max text length: 50,000 characters
|
||||
- Smart splitting respects abbreviations, CJK punctuation, and `[tags]`
|
||||
|
||||
### Generation Versions
|
||||
|
||||
Every generation supports multiple versions with provenance tracking:
|
||||
|
||||
- **Original** — clean TTS output, always preserved
|
||||
- **Effects versions** — apply different effects chains from any source version
|
||||
- **Takes** — regenerate with a new seed for variation
|
||||
- **Source tracking** — each version records its lineage
|
||||
- **Favorites** — star generations for quick access
|
||||
|
||||
### Async Generation Queue
|
||||
|
||||
Generation is non-blocking. Submit and immediately start typing the next one.
|
||||
|
||||
- Serial execution queue prevents GPU contention
|
||||
- Real-time SSE status streaming
|
||||
- Failed generations can be retried
|
||||
- Stale generations from crashes auto-recover on startup
|
||||
|
||||
### Voice Profile Management
|
||||
|
||||
- **Create profiles** from audio files or record directly in-app
|
||||
- **Import/Export** profiles to share or back up
|
||||
- **Multi-sample support** — combine multiple samples for higher quality cloning
|
||||
- **Organize** with descriptions and language tags
|
||||
|
||||
### Speech Generation
|
||||
|
||||
- **Text-to-speech** with any cloned voice
|
||||
- **Batch generation** for long-form content
|
||||
- **Smart caching** — regenerate instantly with voice prompt caching
|
||||
- Create profiles from audio files or record directly in-app
|
||||
- Import/export profiles to share or back up
|
||||
- Multi-sample support for higher quality cloning
|
||||
- Per-profile default effects chains
|
||||
- Organize with descriptions and language tags
|
||||
|
||||
### Stories Editor
|
||||
|
||||
Create multi-voice narratives, podcasts, and conversations with a timeline-based editor.
|
||||
Multi-voice timeline editor for conversations, podcasts, and narratives.
|
||||
|
||||
- **Multi-track composition** — arrange multiple voice tracks in a single project
|
||||
- **Inline audio editing** — trim and split clips directly in the timeline
|
||||
- **Auto-playback** — preview stories with synchronized playhead
|
||||
- **Voice mixing** — build conversations with multiple participants
|
||||
- Multi-track composition with drag-and-drop
|
||||
- Inline audio trimming and splitting
|
||||
- Auto-playback with synchronized playhead
|
||||
- Version pinning per track clip
|
||||
|
||||
### Recording & Transcription
|
||||
|
||||
- **In-app recording** with waveform visualization
|
||||
- **System audio capture** — record desktop audio on macOS and Windows
|
||||
- **Automatic transcription** powered by Whisper
|
||||
- **Export recordings** in multiple formats
|
||||
- In-app recording with waveform visualization
|
||||
- System audio capture (macOS and Windows)
|
||||
- Automatic transcription powered by Whisper (including Whisper Turbo)
|
||||
- Export recordings in multiple formats
|
||||
|
||||
### Generation History
|
||||
### Model Management
|
||||
|
||||
- **Full history** of all generated audio
|
||||
- **Search & filter** by voice, text, or date
|
||||
- **Re-generate** any past generation with one click
|
||||
- Per-model unload to free GPU memory without deleting downloads
|
||||
- Custom models directory via `VOICEBOX_MODELS_DIR`
|
||||
- Model folder migration with progress tracking
|
||||
- Download cancel/clear UI
|
||||
|
||||
### Flexible Deployment
|
||||
### GPU Support
|
||||
|
||||
- **Local mode** — Everything runs on your machine
|
||||
- **Remote mode** — Connect to a GPU server on your network
|
||||
- **One-click server** — Turn any machine into a Voicebox server
|
||||
| Platform | Backend | Notes |
|
||||
| ------------------------ | -------------- | ---------------------------------------------- |
|
||||
| macOS (Apple Silicon) | MLX (Metal) | 4-5x faster via Neural Engine |
|
||||
| Windows / Linux (NVIDIA) | PyTorch (CUDA) | Auto-downloads CUDA binary from within the app |
|
||||
| Linux (AMD) | PyTorch (ROCm) | Auto-configures HSA_OVERRIDE_GFX_VERSION |
|
||||
| Windows (any GPU) | DirectML | Universal Windows GPU support |
|
||||
| Intel Arc | IPEX/XPU | Intel discrete GPU acceleration |
|
||||
| Any | CPU | Works everywhere, just slower |
|
||||
|
||||
---
|
||||
|
||||
## API
|
||||
|
||||
Voicebox exposes a full REST API, so you can integrate voice synthesis into your own apps.
|
||||
|
||||
For the current local app and development workflow, the backend is typically available at `http://localhost:17493`.
|
||||
If you launch the backend manually with a different host or port, use that address instead.
|
||||
Voicebox exposes a full REST API for integrating voice synthesis into your own apps.
|
||||
|
||||
```bash
|
||||
# Generate speech
|
||||
@@ -165,62 +231,40 @@ curl -X POST http://localhost:17493/profiles \
|
||||
-d '{"name": "My Voice", "language": "en"}'
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
**Use cases:** game dialogue, podcast production, accessibility tools, voice assistants, content automation.
|
||||
|
||||
- Game dialogue systems
|
||||
- Podcast/video production pipelines
|
||||
- Accessibility tools
|
||||
- Voice assistants
|
||||
- Content creation automation
|
||||
|
||||
Full API documentation is available at `http://localhost:17493/docs` in the default local workflow, or at `/docs` on whatever server address you configured.
|
||||
Full API documentation available at `http://localhost:17493/docs`.
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|------------|
|
||||
| Desktop App | Tauri (Rust) |
|
||||
| Frontend | React, TypeScript, Tailwind CSS |
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| Voice Model | Qwen3-TTS (PyTorch or MLX) |
|
||||
| Transcription | Whisper (PyTorch or MLX) |
|
||||
| Inference Engine | MLX (Apple Silicon) / PyTorch (Windows/Linux/Intel) |
|
||||
| Database | SQLite |
|
||||
| Audio | WaveSurfer.js, librosa |
|
||||
|
||||
**Why this stack?**
|
||||
|
||||
- **Tauri over Electron** — 10x smaller bundle, native performance, lower memory
|
||||
- **FastAPI** — Async Python with automatic OpenAPI schema generation
|
||||
- **Type-safe end-to-end** — Generated TypeScript client from OpenAPI spec
|
||||
| Layer | Technology |
|
||||
| ------------- | ------------------------------------------------- |
|
||||
| Desktop App | Tauri (Rust) |
|
||||
| Frontend | React, TypeScript, Tailwind CSS |
|
||||
| State | Zustand, React Query |
|
||||
| Backend | FastAPI (Python) |
|
||||
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
|
||||
| Effects | Pedalboard (Spotify) |
|
||||
| Transcription | Whisper / Whisper Turbo (PyTorch or MLX) |
|
||||
| Inference | MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) |
|
||||
| Database | SQLite |
|
||||
| Audio | WaveSurfer.js, librosa |
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
Voicebox is the beginning of something bigger. Here's what's coming:
|
||||
| Feature | Description |
|
||||
| ----------------------- | ---------------------------------------------- |
|
||||
| **Real-time Streaming** | Stream audio as it generates, word by word |
|
||||
| **Voice Design** | Create new voices from text descriptions |
|
||||
| **More Models** | XTTS, Bark, and other open-source voice models |
|
||||
| **Plugin Architecture** | Extend with custom models and effects |
|
||||
| **Mobile Companion** | Control Voicebox from your phone |
|
||||
|
||||
### Coming Soon
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| **Real-time Synthesis** | Stream audio as it generates, word by word |
|
||||
| **Conversation Mode** | Multi-speaker dialogues with automatic turn-taking |
|
||||
| **Voice Effects** | Pitch shift, reverb, M3GAN-style effects |
|
||||
| **Timeline Editor** | Audio studio with word-level precision editing |
|
||||
| **More Models** | XTTS, Bark, and other open-source voice models |
|
||||
|
||||
### Future Vision
|
||||
|
||||
- **Voice Design** — Create new voices from text descriptions
|
||||
- **Project System** — Save and load complex multi-voice sessions
|
||||
- **Plugin Architecture** — Extend with custom models and effects
|
||||
- **Mobile Companion** — Control Voicebox from your phone
|
||||
|
||||
Voicebox aims to be the **one-stop shop for everything voice** — cloning, synthesis, editing, effects, and beyond.
|
||||
For the **full engineering status, open-issue triage, and prioritized work queue**, see [`docs/PROJECT_STATUS.md`](docs/PROJECT_STATUS.md) — a living document that tracks what's shipped, what's in-flight, candidate TTS engines under evaluation, and why we've accepted or backlogged specific integrations.
|
||||
|
||||
---
|
||||
|
||||
@@ -242,14 +286,6 @@ Install [just](https://github.com/casey/just): `brew install just` or `cargo ins
|
||||
|
||||
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org), [Tauri Prerequisites](https://v2.tauri.app/start/prerequisites/), and [Xcode](https://developer.apple.com/xcode/) on macOS.
|
||||
|
||||
### Platform Notes
|
||||
|
||||
| Platform | GPU Backend | Notes |
|
||||
|----------|-------------|-------|
|
||||
| macOS (Apple Silicon) | MLX (Metal) | 4-5x faster inference via Neural Engine |
|
||||
| Windows (NVIDIA) | PyTorch (CUDA) | `just setup` auto-installs CUDA PyTorch |
|
||||
| Windows/Linux (no NVIDIA) | PyTorch (CPU) | Works but slower |
|
||||
|
||||
### Building Locally
|
||||
|
||||
```bash
|
||||
@@ -257,7 +293,11 @@ just build # Build CPU server binary + Tauri app
|
||||
just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app
|
||||
```
|
||||
|
||||
`just build-local` produces a production-ready installer with the CUDA binary pre-placed for GPU switching.
|
||||
### Adding New Voice Models
|
||||
|
||||
The multi-engine architecture makes adding new TTS engines straightforward. A [step-by-step guide](docs/content/docs/developer/tts-engines.mdx) covers the full process: dependency research, backend protocol implementation, frontend wiring, and PyInstaller bundling.
|
||||
|
||||
The guide is optimized for AI coding agents. An [agent skill](.agents/skills/add-tts-engine/SKILL.md) can pick up a model name and handle the entire integration autonomously — you just test the build locally.
|
||||
|
||||
### Project Structure
|
||||
|
||||
|
||||
+3
-3
@@ -6,8 +6,8 @@ We release patches for security vulnerabilities. Which versions are eligible for
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | ------------------ |
|
||||
| 0.1.x | :white_check_mark: |
|
||||
| < 0.1 | :x: |
|
||||
| 0.3.x | :white_check_mark: |
|
||||
| < 0.3 | :x: |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
@@ -82,7 +82,7 @@ Timeline may vary based on severity and complexity.
|
||||
## Security Updates
|
||||
|
||||
Security updates will be:
|
||||
- Released as patch versions (e.g., 0.1.1)
|
||||
- Released as patch versions (e.g., 0.3.2)
|
||||
- Documented in CHANGELOG.md
|
||||
- Announced via GitHub releases
|
||||
- Automatically delivered via auto-updater
|
||||
|
||||
+2
-1
@@ -1,11 +1,12 @@
|
||||
{
|
||||
"name": "@voicebox/app",
|
||||
"version": "0.2.3",
|
||||
"version": "0.4.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit",
|
||||
"preview": "vite preview",
|
||||
"lint": "biome lint src",
|
||||
"lint:fix": "biome lint --write src",
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
import { readFileSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import type { Plugin } from 'vite';
|
||||
|
||||
/** Vite plugin that exposes CHANGELOG.md as `virtual:changelog`. */
|
||||
export function changelogPlugin(repoRoot: string): Plugin {
|
||||
const virtualId = 'virtual:changelog';
|
||||
const resolvedId = '\0' + virtualId;
|
||||
const changelogPath = path.resolve(repoRoot, 'CHANGELOG.md');
|
||||
|
||||
return {
|
||||
name: 'changelog',
|
||||
resolveId(id) {
|
||||
if (id === virtualId) return resolvedId;
|
||||
},
|
||||
load(id) {
|
||||
if (id === resolvedId) {
|
||||
const raw = readFileSync(changelogPath, 'utf-8');
|
||||
return `export default ${JSON.stringify(raw)};`;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
+107
-12
@@ -4,12 +4,42 @@ import voiceboxLogo from '@/assets/voicebox-logo.png';
|
||||
import ShinyText from '@/components/ShinyText';
|
||||
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
||||
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { HealthResponse } from '@/lib/api/types';
|
||||
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import { router } from '@/router';
|
||||
import { useLogStore } from '@/stores/logStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
|
||||
/**
|
||||
* Validate that a health response has the expected Voicebox-specific shape.
|
||||
* Prevents misidentifying an unrelated service on the same port.
|
||||
*/
|
||||
function isVoiceboxHealthResponse(health: HealthResponse): boolean {
|
||||
return (
|
||||
health?.status === 'healthy' &&
|
||||
typeof health.model_loaded === 'boolean' &&
|
||||
typeof health.gpu_available === 'boolean'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check whether a startup error indicates the port is occupied by an external
|
||||
* server (which we should try to reuse via health-check polling) vs. a real
|
||||
* failure (missing sidecar, signing issue, etc.) that should surface immediately.
|
||||
*/
|
||||
function isPortInUseError(error: unknown): boolean {
|
||||
const msg = error instanceof Error ? error.message : String(error);
|
||||
return (
|
||||
msg.includes('already in use') ||
|
||||
msg.includes('port') ||
|
||||
msg.includes('EADDRINUSE') ||
|
||||
msg.includes('address already in use')
|
||||
);
|
||||
}
|
||||
|
||||
const LOADING_MESSAGES = [
|
||||
'Warming up tensors...',
|
||||
'Calibrating synthesizer engine...',
|
||||
@@ -36,6 +66,7 @@ const LOADING_MESSAGES = [
|
||||
function App() {
|
||||
const platform = usePlatform();
|
||||
const [serverReady, setServerReady] = useState(false);
|
||||
const [startupError, setStartupError] = useState<string | null>(null);
|
||||
const [loadingMessageIndex, setLoadingMessageIndex] = useState(0);
|
||||
const serverStartingRef = useRef(false);
|
||||
|
||||
@@ -63,6 +94,14 @@ function App() {
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [platform.lifecycle]);
|
||||
|
||||
// Subscribe to server logs
|
||||
useEffect(() => {
|
||||
const unsubscribe = platform.lifecycle.subscribeToServerLogs((entry) => {
|
||||
useLogStore.getState().addEntry(entry);
|
||||
});
|
||||
return unsubscribe;
|
||||
}, [platform.lifecycle]);
|
||||
|
||||
// Setup window close handler and auto-start server when running in Tauri (production only)
|
||||
useEffect(() => {
|
||||
if (!platform.metadata.isTauri) {
|
||||
@@ -82,7 +121,6 @@ function App() {
|
||||
console.log('Dev mode: Skipping auto-start of server (run it separately)');
|
||||
setServerReady(true); // Mark as ready so UI doesn't show loading screen
|
||||
// Mark that server was not started by app (so we don't try to stop it on close)
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = false;
|
||||
return;
|
||||
}
|
||||
@@ -105,14 +143,52 @@ function App() {
|
||||
useServerStore.getState().setServerUrl(serverUrl);
|
||||
setServerReady(true);
|
||||
// Mark that we started the server (so we know to stop it on close)
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = true;
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error('Failed to auto-start server:', error);
|
||||
serverStartingRef.current = false;
|
||||
// @ts-expect-error - adding property to window
|
||||
window.__voiceboxServerStartedByApp = false;
|
||||
|
||||
// Only fall back to health-check polling when the error indicates the
|
||||
// port is occupied (likely an external server). For real failures
|
||||
// (missing sidecar, signing issues, etc.) surface the error immediately.
|
||||
if (!isPortInUseError(error)) {
|
||||
const msg = error instanceof Error ? error.message : String(error);
|
||||
console.error('Real startup failure — not polling:', msg);
|
||||
setStartupError(msg);
|
||||
return;
|
||||
}
|
||||
|
||||
// Fall back to polling: the server may already be running externally
|
||||
// (e.g. started via python/uvicorn/Docker). Poll the health endpoint
|
||||
// until it responds with a valid Voicebox payload, then transition to
|
||||
// the main UI.
|
||||
console.log('Falling back to health-check polling...');
|
||||
const pollInterval = setInterval(async () => {
|
||||
try {
|
||||
const health = await apiClient.getHealth();
|
||||
if (!isVoiceboxHealthResponse(health)) {
|
||||
console.log('Health response is not from a Voicebox server, keep polling...');
|
||||
return;
|
||||
}
|
||||
console.log('External Voicebox server detected via health check');
|
||||
clearInterval(pollInterval);
|
||||
setServerReady(true);
|
||||
} catch {
|
||||
// Server not ready yet, keep polling
|
||||
}
|
||||
}, 2000);
|
||||
|
||||
// Stop polling after 2 minutes and surface the failure
|
||||
setTimeout(() => {
|
||||
clearInterval(pollInterval);
|
||||
serverStartingRef.current = false;
|
||||
setStartupError(
|
||||
'Could not connect to a Voicebox server within 2 minutes. ' +
|
||||
'Please check that the server is running and try again.',
|
||||
);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
// Cleanup: stop server on actual unmount (not StrictMode remount)
|
||||
@@ -159,15 +235,34 @@ function App() {
|
||||
className="w-48 h-48 object-contain animate-fade-in-scale relative z-10"
|
||||
/>
|
||||
</div>
|
||||
<div className="animate-fade-in-delayed">
|
||||
<ShinyText
|
||||
text={LOADING_MESSAGES[loadingMessageIndex]}
|
||||
className="text-lg font-medium text-muted-foreground"
|
||||
speed={2}
|
||||
color="hsl(var(--muted-foreground))"
|
||||
shineColor="hsl(var(--foreground))"
|
||||
/>
|
||||
</div>
|
||||
{startupError ? (
|
||||
<div className="animate-fade-in-delayed max-w-md mx-auto space-y-3">
|
||||
<p className="text-lg font-medium text-destructive">Server startup failed</p>
|
||||
<p className="text-sm text-muted-foreground">{startupError}</p>
|
||||
<button
|
||||
type="button"
|
||||
className="mt-2 px-4 py-2 text-sm rounded-md bg-primary text-primary-foreground hover:bg-primary/90 transition-colors"
|
||||
onClick={() => {
|
||||
setStartupError(null);
|
||||
serverStartingRef.current = false;
|
||||
// Trigger a re-mount of the effect by toggling state
|
||||
window.location.reload();
|
||||
}}
|
||||
>
|
||||
Retry
|
||||
</button>
|
||||
</div>
|
||||
) : (
|
||||
<div className="animate-fade-in-delayed">
|
||||
<ShinyText
|
||||
text={LOADING_MESSAGES[loadingMessageIndex]}
|
||||
className="text-lg font-medium text-muted-foreground"
|
||||
speed={2}
|
||||
color="hsl(var(--muted-foreground))"
|
||||
shineColor="hsl(var(--foreground))"
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { useRouterState } from '@tanstack/react-router';
|
||||
import { TitleBarDragRegion } from '@/components/TitleBarDragRegion';
|
||||
import { AudioKeepAlive } from '@/components/AudioPlayer/AudioKeepAlive';
|
||||
import { AudioPlayer } from '@/components/AudioPlayer/AudioPlayer';
|
||||
import { StoryTrackEditor } from '@/components/StoriesTab/StoryTrackEditor';
|
||||
import { TOP_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
@@ -14,16 +15,19 @@ interface AppFrameProps {
|
||||
export function AppFrame({ children }: AppFrameProps) {
|
||||
const routerState = useRouterState();
|
||||
const isStoriesRoute = routerState.location.pathname === '/stories';
|
||||
|
||||
|
||||
const selectedStoryId = useStoryStore((state) => state.selectedStoryId);
|
||||
const { data: story } = useStory(selectedStoryId);
|
||||
|
||||
|
||||
// Show track editor when on stories route with a selected story that has items
|
||||
const showTrackEditor = isStoriesRoute && selectedStoryId && story && story.items.length > 0;
|
||||
|
||||
return (
|
||||
<div className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}>
|
||||
<div
|
||||
className={cn('h-screen bg-background flex flex-col overflow-hidden', TOP_SAFE_AREA_PADDING)}
|
||||
>
|
||||
<TitleBarDragRegion />
|
||||
<AudioKeepAlive />
|
||||
{children}
|
||||
{showTrackEditor ? (
|
||||
<StoryTrackEditor storyId={story.id} items={story.items} />
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import { useEffect, useRef } from 'react';
|
||||
import { debug } from '@/lib/utils/debug';
|
||||
|
||||
// WKWebView tears down the app's CoreAudio output when idle for long enough,
|
||||
// and a JS-level reload (cmd+R) does NOT restore it — only relaunching the
|
||||
// Tauri app does. Keeping a silent <audio> element looping forever prevents
|
||||
// the OS audio session from ever going dormant.
|
||||
//
|
||||
// Real silence (zero PCM samples) at full volume is preferred over a muted
|
||||
// element: browsers/WebKit can optimize muted media away, which defeats the
|
||||
// purpose of holding the session open.
|
||||
|
||||
function buildSilentWavUrl(seconds = 1, sampleRate = 8000): string {
|
||||
const numSamples = seconds * sampleRate;
|
||||
const bytes = 44 + numSamples * 2;
|
||||
const buffer = new ArrayBuffer(bytes);
|
||||
const view = new DataView(buffer);
|
||||
const write = (offset: number, str: string) => {
|
||||
for (let i = 0; i < str.length; i++) view.setUint8(offset + i, str.charCodeAt(i));
|
||||
};
|
||||
write(0, 'RIFF');
|
||||
view.setUint32(4, bytes - 8, true);
|
||||
write(8, 'WAVE');
|
||||
write(12, 'fmt ');
|
||||
view.setUint32(16, 16, true);
|
||||
view.setUint16(20, 1, true);
|
||||
view.setUint16(22, 1, true);
|
||||
view.setUint32(24, sampleRate, true);
|
||||
view.setUint32(28, sampleRate * 2, true);
|
||||
view.setUint16(32, 2, true);
|
||||
view.setUint16(34, 16, true);
|
||||
write(36, 'data');
|
||||
view.setUint32(40, numSamples * 2, true);
|
||||
return URL.createObjectURL(new Blob([buffer], { type: 'audio/wav' }));
|
||||
}
|
||||
|
||||
export function AudioKeepAlive() {
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const url = buildSilentWavUrl(1, 8000);
|
||||
const el = new Audio(url);
|
||||
el.loop = true;
|
||||
el.volume = 1;
|
||||
el.preload = 'auto';
|
||||
audioRef.current = el;
|
||||
|
||||
const tryPlay = () => {
|
||||
if (!audioRef.current) return;
|
||||
if (!audioRef.current.paused) return;
|
||||
audioRef.current.play().catch((err) => {
|
||||
debug.log('[AudioKeepAlive] play blocked (will retry on next gesture):', err);
|
||||
});
|
||||
};
|
||||
|
||||
tryPlay();
|
||||
|
||||
// Autoplay may be blocked until first user interaction — re-attempt then.
|
||||
const onGesture = () => tryPlay();
|
||||
window.addEventListener('pointerdown', onGesture, { once: false });
|
||||
window.addEventListener('keydown', onGesture, { once: false });
|
||||
|
||||
// If the webview ever pauses the element on background, resume on return.
|
||||
const onWake = () => {
|
||||
if (!document.hidden) tryPlay();
|
||||
};
|
||||
document.addEventListener('visibilitychange', onWake);
|
||||
window.addEventListener('focus', onWake);
|
||||
window.addEventListener('pageshow', onWake);
|
||||
|
||||
return () => {
|
||||
window.removeEventListener('pointerdown', onGesture);
|
||||
window.removeEventListener('keydown', onGesture);
|
||||
document.removeEventListener('visibilitychange', onWake);
|
||||
window.removeEventListener('focus', onWake);
|
||||
window.removeEventListener('pageshow', onWake);
|
||||
el.pause();
|
||||
el.src = '';
|
||||
URL.revokeObjectURL(url);
|
||||
audioRef.current = null;
|
||||
};
|
||||
}, []);
|
||||
|
||||
return null;
|
||||
}
|
||||
@@ -17,7 +17,6 @@ export function AudioPlayer() {
|
||||
audioUrl,
|
||||
audioId,
|
||||
profileId,
|
||||
title,
|
||||
isPlaying,
|
||||
currentTime,
|
||||
duration,
|
||||
@@ -63,7 +62,7 @@ export function AudioPlayer() {
|
||||
);
|
||||
|
||||
return shouldUseNative;
|
||||
}, [profileChannels, channels, profileId]);
|
||||
}, [profileChannels, channels, platform.metadata.isTauri]);
|
||||
|
||||
const waveformRef = useRef<HTMLDivElement>(null);
|
||||
const wavesurferRef = useRef<WaveSurfer | null>(null);
|
||||
@@ -73,31 +72,21 @@ export function AudioPlayer() {
|
||||
const isUsingNativePlaybackRef = useRef(false);
|
||||
const [isLoading, setIsLoading] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [wsReady, setWsReady] = useState(false);
|
||||
|
||||
// Initialize WaveSurfer (only when audioUrl exists and container is ready)
|
||||
// Create WaveSurfer once when the player becomes visible (audioUrl is set).
|
||||
// This instance is reused for all subsequent audio loads - never destroyed until unmount.
|
||||
useEffect(() => {
|
||||
// Don't initialize if no audioUrl or already initialized
|
||||
if (!audioUrl) {
|
||||
return;
|
||||
}
|
||||
if (!audioUrl) return;
|
||||
if (wavesurferRef.current) return; // already created
|
||||
|
||||
if (wavesurferRef.current) {
|
||||
debug.log('WaveSurfer already initialized, skipping');
|
||||
return;
|
||||
}
|
||||
|
||||
debug.log('Creating NEW WaveSurfer instance');
|
||||
|
||||
// Wait for container to be properly rendered
|
||||
const initWaveSurfer = () => {
|
||||
const container = waveformRef.current;
|
||||
if (!container) {
|
||||
// Container not ready yet, retry
|
||||
setTimeout(initWaveSurfer, 50);
|
||||
return;
|
||||
}
|
||||
|
||||
// Check if container has dimensions and is visible
|
||||
const rect = container.getBoundingClientRect();
|
||||
const style = window.getComputedStyle(container);
|
||||
const isVisible =
|
||||
@@ -107,412 +96,221 @@ export function AudioPlayer() {
|
||||
style.visibility !== 'hidden';
|
||||
|
||||
if (!isVisible) {
|
||||
// Retry after a short delay
|
||||
setTimeout(initWaveSurfer, 50);
|
||||
return;
|
||||
}
|
||||
|
||||
debug.log('Initializing WaveSurfer...', {
|
||||
container,
|
||||
debug.log('Creating WaveSurfer instance', {
|
||||
width: rect.width,
|
||||
height: rect.height,
|
||||
});
|
||||
|
||||
try {
|
||||
// Get computed CSS variable values
|
||||
const root = document.documentElement;
|
||||
const getCSSVar = (varName: string) => {
|
||||
const value = getComputedStyle(root).getPropertyValue(varName).trim();
|
||||
return value ? `hsl(${value})` : '';
|
||||
};
|
||||
|
||||
const waveColor = getCSSVar('--muted');
|
||||
const progressColor = getCSSVar('--accent');
|
||||
const cursorColor = getCSSVar('--accent');
|
||||
|
||||
const wavesurfer = WaveSurfer.create({
|
||||
container: container,
|
||||
waveColor: waveColor,
|
||||
progressColor: progressColor,
|
||||
cursorColor: cursorColor,
|
||||
container,
|
||||
waveColor: getCSSVar('--muted'),
|
||||
progressColor: getCSSVar('--accent'),
|
||||
cursorColor: getCSSVar('--accent'),
|
||||
cursorWidth: 3,
|
||||
barWidth: 2,
|
||||
barRadius: 2,
|
||||
height: 80,
|
||||
normalize: true,
|
||||
// Use MediaElement backend (default). Unlike the WebAudio backend,
|
||||
// MediaElement uses a standard <audio> element for playback which
|
||||
// benefits from the browser/webview's built-in audio session recovery.
|
||||
// This prevents audio loss when another app steals audio output or
|
||||
// the system audio session is interrupted.
|
||||
interact: true, // Enable interaction (click to seek)
|
||||
mediaControls: false, // Don't show native controls
|
||||
interact: true,
|
||||
dragToSeek: { debounceTime: 0 },
|
||||
mediaControls: false,
|
||||
backend: 'WebAudio',
|
||||
});
|
||||
|
||||
wavesurferRef.current = wavesurfer;
|
||||
debug.log('WaveSurfer created successfully');
|
||||
} catch (error) {
|
||||
debug.error('Failed to create WaveSurfer:', error);
|
||||
setError(
|
||||
`Failed to initialize waveform: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
// Wire up event handlers (these persist for the lifetime of the instance)
|
||||
wavesurfer.on('timeupdate', (time) => {
|
||||
const dur = usePlayerStore.getState().duration;
|
||||
if (dur > 0 && time >= dur) {
|
||||
setCurrentTime(dur);
|
||||
const loop = usePlayerStore.getState().isLooping;
|
||||
if (loop) {
|
||||
wavesurfer.seekTo(0);
|
||||
wavesurfer.play().catch((err) => debug.error('Loop play failed:', err));
|
||||
} else {
|
||||
wavesurfer.pause();
|
||||
setIsPlaying(false);
|
||||
}
|
||||
return;
|
||||
}
|
||||
setCurrentTime(time);
|
||||
});
|
||||
|
||||
const wavesurfer = wavesurferRef.current;
|
||||
if (!wavesurfer) return;
|
||||
wavesurfer.on('ready', () => {
|
||||
const dur = wavesurfer.getDuration();
|
||||
setDuration(dur);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setError(null);
|
||||
debug.log('Audio ready, duration:', dur);
|
||||
|
||||
// Update store when time changes, stop if past duration
|
||||
wavesurfer.on('timeupdate', (time) => {
|
||||
const dur = usePlayerStore.getState().duration;
|
||||
if (dur > 0 && time >= dur) {
|
||||
setCurrentTime(dur);
|
||||
wavesurfer.setVolume(usePlayerStore.getState().volume);
|
||||
wavesurfer.setMuted(false);
|
||||
|
||||
// Auto-play if the flag is set (story mode advance or explicit play)
|
||||
const shouldAutoPlayNow = usePlayerStore.getState().shouldAutoPlay;
|
||||
if (shouldAutoPlayNow) {
|
||||
usePlayerStore.getState().clearAutoPlayFlag();
|
||||
wavesurfer.play().catch((err) => {
|
||||
debug.error('Failed to autoplay:', err);
|
||||
});
|
||||
} else {
|
||||
debug.log('Skipping auto-play - shouldAutoPlay is false');
|
||||
}
|
||||
});
|
||||
|
||||
wavesurfer.on('play', () => setIsPlaying(true));
|
||||
wavesurfer.on('pause', () => {
|
||||
setIsPlaying(false);
|
||||
setCurrentTime(wavesurfer.getCurrentTime());
|
||||
});
|
||||
|
||||
wavesurfer.on('seeking', (time) => setCurrentTime(time));
|
||||
|
||||
// Mute audio during drag-to-seek to prevent popping from the WebAudio
|
||||
// backend's hard stop/start cycle on each seek. Unmute with a short
|
||||
// fade-in when the drag ends.
|
||||
const seekMedia = wavesurfer.getMediaElement() as any;
|
||||
const seekGain: GainNode | null = seekMedia?.getGainNode?.() ?? null;
|
||||
if (seekGain) {
|
||||
const ctx = seekGain.context as AudioContext;
|
||||
wavesurfer.on('dragstart', () => {
|
||||
seekGain.gain.cancelScheduledValues(ctx.currentTime);
|
||||
seekGain.gain.setTargetAtTime(0, ctx.currentTime, 0.002);
|
||||
});
|
||||
wavesurfer.on('dragend', () => {
|
||||
seekGain.gain.cancelScheduledValues(ctx.currentTime);
|
||||
seekGain.gain.setTargetAtTime(1, ctx.currentTime, 0.01);
|
||||
});
|
||||
}
|
||||
wavesurfer.on('finish', () => {
|
||||
const loop = usePlayerStore.getState().isLooping;
|
||||
if (loop) {
|
||||
wavesurfer.seekTo(0);
|
||||
wavesurfer.play();
|
||||
wavesurfer.play().catch((err) => debug.error('Loop play failed:', err));
|
||||
} else {
|
||||
wavesurfer.pause();
|
||||
setIsPlaying(false);
|
||||
const onFinish = usePlayerStore.getState().onFinish;
|
||||
if (onFinish) onFinish();
|
||||
}
|
||||
return;
|
||||
}
|
||||
setCurrentTime(time);
|
||||
});
|
||||
|
||||
// Update store when duration is loaded
|
||||
wavesurfer.on('ready', async () => {
|
||||
const dur = wavesurfer.getDuration();
|
||||
setDuration(dur);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setError(null);
|
||||
debug.log('Audio ready, duration:', dur);
|
||||
debug.log('Waveform should be visible now');
|
||||
|
||||
// Ensure volume is set
|
||||
const currentVolume = usePlayerStore.getState().volume;
|
||||
wavesurfer.setVolume(currentVolume);
|
||||
|
||||
// Auto-play when ready - check if we should use native playback
|
||||
// Get current values from the store and queries at runtime (not captured closure values)
|
||||
const currentAudioUrl = usePlayerStore.getState().audioUrl;
|
||||
const currentProfileId = usePlayerStore.getState().profileId;
|
||||
|
||||
debug.log('Auto-play check - capturing runtime values...');
|
||||
|
||||
// Fetch profile channels at runtime (not using captured value)
|
||||
let runtimeProfileChannels = null;
|
||||
let runtimeChannels = null;
|
||||
|
||||
if (platform.metadata.isTauri && currentProfileId) {
|
||||
try {
|
||||
runtimeProfileChannels = await apiClient.getProfileChannels(currentProfileId);
|
||||
debug.log('Runtime profileChannels:', runtimeProfileChannels);
|
||||
|
||||
if (runtimeProfileChannels && runtimeProfileChannels.channel_ids.length > 0) {
|
||||
runtimeChannels = await apiClient.listChannels();
|
||||
debug.log('Runtime channels:', runtimeChannels);
|
||||
}
|
||||
} catch (error) {
|
||||
debug.error('Failed to fetch runtime channel data:', error);
|
||||
}
|
||||
}
|
||||
|
||||
debug.log('Auto-play check:', {
|
||||
isTauri: platform.metadata.isTauri,
|
||||
currentAudioUrl,
|
||||
currentProfileId,
|
||||
hasProfileChannels: !!runtimeProfileChannels,
|
||||
hasChannels: !!runtimeChannels,
|
||||
});
|
||||
|
||||
if (
|
||||
platform.metadata.isTauri &&
|
||||
currentAudioUrl &&
|
||||
currentProfileId &&
|
||||
runtimeProfileChannels &&
|
||||
runtimeChannels
|
||||
) {
|
||||
debug.log('Attempting native audio playback...');
|
||||
|
||||
// Stop any existing native playback first
|
||||
if (isUsingNativePlaybackRef.current) {
|
||||
try {
|
||||
platform.audio.stopPlayback();
|
||||
debug.log('Stopped existing native playback before starting new one');
|
||||
} catch (error) {
|
||||
debug.error('Failed to stop existing playback:', error);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
// Collect all device IDs from assigned channels
|
||||
const assignedChannels = runtimeChannels.filter((ch: any) =>
|
||||
runtimeProfileChannels.channel_ids.includes(ch.id),
|
||||
);
|
||||
debug.log('Assigned channels for playback:', assignedChannels);
|
||||
|
||||
// Check if any assigned channel has non-default devices
|
||||
const shouldUseNative = assignedChannels.some(
|
||||
(ch: any) => ch.device_ids.length > 0 && !ch.is_default,
|
||||
);
|
||||
debug.log('Should use native playback:', shouldUseNative);
|
||||
|
||||
if (!shouldUseNative) {
|
||||
debug.log('No custom devices assigned, using standard playback');
|
||||
isUsingNativePlaybackRef.current = false;
|
||||
} else {
|
||||
const deviceIds = assignedChannels.flatMap((ch: any) => ch.device_ids);
|
||||
debug.log('Device IDs to play to:', deviceIds);
|
||||
|
||||
if (deviceIds.length > 0) {
|
||||
debug.log('Fetching audio data from:', currentAudioUrl);
|
||||
// Fetch audio data
|
||||
const response = await fetch(currentAudioUrl);
|
||||
const audioData = new Uint8Array(await response.arrayBuffer());
|
||||
debug.log('Audio data size:', audioData.length);
|
||||
|
||||
// Play via native audio
|
||||
debug.log('Invoking play_audio_to_devices...');
|
||||
try {
|
||||
await platform.audio.playToDevices(audioData, deviceIds);
|
||||
debug.log('play_audio_to_devices completed successfully');
|
||||
|
||||
// Mark that we're using native playback
|
||||
isUsingNativePlaybackRef.current = true;
|
||||
|
||||
// Mute WaveSurfer's audio output — native handles the actual sound
|
||||
// Keep WaveSurfer running for waveform visualization
|
||||
wavesurfer.setVolume(0);
|
||||
wavesurfer.setMuted(true);
|
||||
|
||||
// Start WaveSurfer playback for visualization (muted)
|
||||
wavesurfer.play().catch((error) => {
|
||||
debug.error('Failed to start WaveSurfer visualization:', error);
|
||||
});
|
||||
|
||||
setIsPlaying(true);
|
||||
debug.log('Auto-playing via native audio routing - SUCCESS');
|
||||
return;
|
||||
} catch (invokeError) {
|
||||
debug.error('play_audio_to_devices invoke failed:', invokeError);
|
||||
throw invokeError;
|
||||
}
|
||||
} else {
|
||||
debug.log('No device IDs found, falling back to WaveSurfer');
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
debug.error(
|
||||
'Native playback failed during auto-play, falling back to WaveSurfer:',
|
||||
error,
|
||||
);
|
||||
isUsingNativePlaybackRef.current = false;
|
||||
// Fall through to WaveSurfer playback
|
||||
}
|
||||
}
|
||||
|
||||
// Standard playback path — ensure WaveSurfer is unmuted
|
||||
if (!isUsingNativePlaybackRef.current) {
|
||||
wavesurfer.setMuted(false);
|
||||
wavesurfer.setVolume(usePlayerStore.getState().volume);
|
||||
}
|
||||
|
||||
// Only auto-play if shouldAutoPlay flag is set (user explicitly clicked to play)
|
||||
const shouldAutoPlayNow = usePlayerStore.getState().shouldAutoPlay;
|
||||
if (shouldAutoPlayNow) {
|
||||
// Clear the flag first
|
||||
usePlayerStore.getState().clearAutoPlayFlag();
|
||||
|
||||
// Use a small delay to ensure audio element is fully ready
|
||||
setTimeout(() => {
|
||||
wavesurfer.play().catch((error) => {
|
||||
debug.error('Failed to autoplay:', error);
|
||||
// Don't show error for autoplay failures (browser restrictions)
|
||||
});
|
||||
}, 100);
|
||||
} else {
|
||||
debug.log('Skipping auto-play - shouldAutoPlay is false');
|
||||
}
|
||||
});
|
||||
|
||||
// Handle play/pause
|
||||
wavesurfer.on('play', () => {
|
||||
setIsPlaying(true);
|
||||
});
|
||||
wavesurfer.on('pause', () => setIsPlaying(false));
|
||||
wavesurfer.on('finish', () => {
|
||||
// Check loop state from store
|
||||
const loop = usePlayerStore.getState().isLooping;
|
||||
if (loop) {
|
||||
wavesurfer.seekTo(0);
|
||||
wavesurfer.play();
|
||||
} else {
|
||||
setIsPlaying(false);
|
||||
// Trigger finish callback if set
|
||||
const onFinish = usePlayerStore.getState().onFinish;
|
||||
if (onFinish) {
|
||||
onFinish();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Handle errors
|
||||
wavesurfer.on('error', (error) => {
|
||||
debug.error('WaveSurfer error:', error);
|
||||
setIsLoading(false);
|
||||
setError(`Audio error: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
|
||||
// Handle loading
|
||||
wavesurfer.on('loading', (percent) => {
|
||||
setIsLoading(true);
|
||||
if (percent === 100) {
|
||||
wavesurfer.on('error', (err) => {
|
||||
debug.error('WaveSurfer error:', err);
|
||||
setIsLoading(false);
|
||||
}
|
||||
});
|
||||
setError(`Audio error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
|
||||
// Load audio immediately if audioUrl is already set
|
||||
if (audioUrl) {
|
||||
debug.log('WaveSurfer ready, loading audio:', audioUrl);
|
||||
loadingRef.current = true;
|
||||
setIsLoading(true);
|
||||
// Stop any current playback before loading new audio
|
||||
if (wavesurfer.isPlaying()) {
|
||||
wavesurfer.pause();
|
||||
}
|
||||
wavesurfer
|
||||
.load(audioUrl)
|
||||
.then(() => {
|
||||
debug.log('Audio loaded into WaveSurfer');
|
||||
loadingRef.current = false;
|
||||
})
|
||||
.catch((error) => {
|
||||
debug.error('Failed to load audio into WaveSurfer:', error);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setError(
|
||||
`Failed to load audio: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
});
|
||||
wavesurfer.on('loading', (percent) => {
|
||||
setIsLoading(true);
|
||||
if (percent === 100) setIsLoading(false);
|
||||
});
|
||||
|
||||
wavesurferRef.current = wavesurfer;
|
||||
setWsReady(true);
|
||||
debug.log('WaveSurfer created successfully');
|
||||
} catch (err) {
|
||||
debug.error('Failed to create WaveSurfer:', err);
|
||||
setError(
|
||||
`Failed to initialize waveform: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Use double requestAnimationFrame to ensure DOM is fully rendered
|
||||
let rafId1: number;
|
||||
let rafId2: number;
|
||||
let timeoutId: number | null = null;
|
||||
|
||||
rafId1 = requestAnimationFrame(() => {
|
||||
rafId2 = requestAnimationFrame(() => {
|
||||
// Add a small delay to ensure container is fully laid out
|
||||
timeoutId = setTimeout(() => {
|
||||
initWaveSurfer();
|
||||
}, 10);
|
||||
});
|
||||
let rafId: number;
|
||||
rafId = requestAnimationFrame(() => {
|
||||
initWaveSurfer();
|
||||
});
|
||||
|
||||
return () => {
|
||||
debug.log('Cleaning up WaveSurfer initialization effect');
|
||||
if (rafId1) cancelAnimationFrame(rafId1);
|
||||
if (rafId2) cancelAnimationFrame(rafId2);
|
||||
if (timeoutId) clearTimeout(timeoutId);
|
||||
cancelAnimationFrame(rafId);
|
||||
};
|
||||
// Only run on mount-like conditions. audioUrl is here so we create the instance
|
||||
// when the player first appears, but we guard against re-creation above.
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [audioUrl, setIsPlaying, setDuration, setCurrentTime]);
|
||||
|
||||
// Destroy WaveSurfer only on unmount
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (wavesurferRef.current) {
|
||||
debug.log('Destroying WaveSurfer instance');
|
||||
debug.log('Destroying WaveSurfer instance (unmount)');
|
||||
try {
|
||||
wavesurferRef.current.destroy();
|
||||
} catch (error) {
|
||||
debug.error('Error destroying WaveSurfer:', error);
|
||||
} catch (err) {
|
||||
debug.error('Error destroying WaveSurfer:', err);
|
||||
}
|
||||
wavesurferRef.current = null;
|
||||
setWsReady(false);
|
||||
}
|
||||
};
|
||||
}, [audioUrl, setIsPlaying, setCurrentTime, setDuration]);
|
||||
}, []);
|
||||
|
||||
// Load audio when URL changes (only if WaveSurfer is already initialized)
|
||||
// Load audio when URL changes (reuses the existing WaveSurfer instance)
|
||||
useEffect(() => {
|
||||
const wavesurfer = wavesurferRef.current;
|
||||
if (!wavesurfer || !wsReady) return;
|
||||
|
||||
if (!audioUrl || !wavesurfer) {
|
||||
// Reset state when no audio or WaveSurfer not ready
|
||||
if (!audioUrl && wavesurfer) {
|
||||
wavesurfer.pause();
|
||||
wavesurfer.seekTo(0);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setDuration(0);
|
||||
setCurrentTime(0);
|
||||
setError(null);
|
||||
// Reset native playback flag
|
||||
isUsingNativePlaybackRef.current = false;
|
||||
}
|
||||
if (!audioUrl) {
|
||||
// No audio - pause and reset
|
||||
wavesurfer.pause();
|
||||
wavesurfer.seekTo(0);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setDuration(0);
|
||||
setCurrentTime(0);
|
||||
setError(null);
|
||||
isUsingNativePlaybackRef.current = false;
|
||||
return;
|
||||
}
|
||||
|
||||
// Stop native playback if it was active
|
||||
if (isUsingNativePlaybackRef.current && platform.metadata.isTauri) {
|
||||
try {
|
||||
platform.audio.stopPlayback();
|
||||
debug.log('Stopped native audio playback');
|
||||
} catch (error) {
|
||||
debug.error('Failed to stop native playback:', error);
|
||||
}
|
||||
}
|
||||
|
||||
// Reset native playback flag when loading new audio
|
||||
// Unmute WaveSurfer if it was muted for native playback
|
||||
if (isUsingNativePlaybackRef.current) {
|
||||
wavesurfer.setMuted(false);
|
||||
wavesurfer.setVolume(usePlayerStore.getState().volume);
|
||||
}
|
||||
// Reset native playback state
|
||||
isUsingNativePlaybackRef.current = false;
|
||||
wavesurfer.setMuted(false);
|
||||
wavesurfer.setVolume(usePlayerStore.getState().volume);
|
||||
|
||||
// CRITICAL: Force stop any current playback and cancel any pending loads
|
||||
// This must happen BEFORE any early returns
|
||||
debug.log('Audio URL changed to:', audioUrl);
|
||||
|
||||
// COMPLETELY stop and destroy the current audio
|
||||
// Stop current playback and reset position before loading new audio.
|
||||
// With the WebAudio backend, pause() accumulates playedDuration internally.
|
||||
// seekTo(0) resets it so the new track starts from the beginning.
|
||||
debug.log('Loading new audio URL:', audioUrl);
|
||||
try {
|
||||
// First pause if playing
|
||||
if (wavesurfer.isPlaying()) {
|
||||
debug.log('Pausing current playback');
|
||||
wavesurfer.pause();
|
||||
}
|
||||
|
||||
// Use empty() to completely destroy the waveform and reset media
|
||||
debug.log('Calling wavesurfer.empty() to destroy audio');
|
||||
wavesurfer.empty();
|
||||
} catch (error) {
|
||||
debug.error('Error stopping previous audio:', error);
|
||||
// Continue anyway to load new audio
|
||||
wavesurfer.seekTo(0);
|
||||
} catch (err) {
|
||||
debug.error('Error resetting before load:', err);
|
||||
}
|
||||
|
||||
// Reset loading state to allow new load (cancel any pending loads)
|
||||
loadingRef.current = false;
|
||||
|
||||
// Now start the new load
|
||||
loadingRef.current = true;
|
||||
setIsLoading(true);
|
||||
setError(null);
|
||||
setCurrentTime(0);
|
||||
setDuration(0);
|
||||
|
||||
// Load new audio
|
||||
debug.log('Starting new audio load for:', audioUrl);
|
||||
wavesurfer
|
||||
.load(audioUrl)
|
||||
.then(() => {
|
||||
debug.log('Audio load promise resolved');
|
||||
// Don't set loading to false here - wait for 'ready' event
|
||||
debug.log('Audio loaded into WaveSurfer');
|
||||
loadingRef.current = false;
|
||||
})
|
||||
.catch((error) => {
|
||||
debug.error('Failed to load audio:', error);
|
||||
debug.error('Audio URL:', audioUrl);
|
||||
.catch((err) => {
|
||||
debug.error('Failed to load audio:', err);
|
||||
loadingRef.current = false;
|
||||
setIsLoading(false);
|
||||
setError(`Failed to load audio: ${error instanceof Error ? error.message : String(error)}`);
|
||||
setError(`Failed to load audio: ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
}, [audioUrl, setCurrentTime, setDuration]);
|
||||
}, [audioUrl, wsReady, setCurrentTime, setDuration]);
|
||||
|
||||
// Sync play/pause state (only when user clicks play/pause button, not auto-sync)
|
||||
// This effect is kept for external state changes but should be minimal
|
||||
@@ -520,7 +318,6 @@ export function AudioPlayer() {
|
||||
if (!wavesurferRef.current || duration === 0) return;
|
||||
|
||||
if (isPlaying && wavesurferRef.current.isPlaying() === false) {
|
||||
// Only auto-play if audio is ready
|
||||
wavesurferRef.current.play().catch((error) => {
|
||||
debug.error('Failed to play:', error);
|
||||
setIsPlaying(false);
|
||||
@@ -534,14 +331,7 @@ export function AudioPlayer() {
|
||||
// Sync volume
|
||||
useEffect(() => {
|
||||
if (wavesurferRef.current) {
|
||||
// If using native playback, keep WaveSurfer muted regardless of volume setting
|
||||
if (isUsingNativePlaybackRef.current) {
|
||||
wavesurferRef.current.setVolume(0);
|
||||
debug.log('Volume sync: Using native playback, keeping WaveSurfer muted');
|
||||
} else {
|
||||
wavesurferRef.current.setVolume(volume);
|
||||
debug.log('Volume synced:', volume);
|
||||
}
|
||||
wavesurferRef.current.setVolume(volume);
|
||||
}
|
||||
}, [volume]);
|
||||
|
||||
@@ -566,7 +356,6 @@ export function AudioPlayer() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Reset to beginning and play
|
||||
debug.log('Restarting current audio from beginning');
|
||||
wavesurfer.seekTo(0);
|
||||
wavesurfer.play().catch((error) => {
|
||||
@@ -575,34 +364,35 @@ export function AudioPlayer() {
|
||||
setError(`Playback error: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
|
||||
// Clear the restart flag
|
||||
clearRestartFlag();
|
||||
}, [shouldRestart, duration, setIsPlaying, clearRestartFlag]);
|
||||
|
||||
// Handle shouldAutoPlay flag - for story mode auto-advance
|
||||
const shouldAutoPlay = usePlayerStore((state) => state.shouldAutoPlay);
|
||||
const clearAutoPlayFlag = usePlayerStore((state) => state.clearAutoPlayFlag);
|
||||
// Auto-play is handled exclusively in the WaveSurfer 'ready' event handler.
|
||||
// A separate effect here would race with the ready event since the WebAudio
|
||||
// backend needs to fully decode the audio before play() works correctly.
|
||||
|
||||
// Spacebar to play/pause (capture phase so it fires before focused elements)
|
||||
useEffect(() => {
|
||||
const wavesurfer = wavesurferRef.current;
|
||||
if (!wavesurfer || !shouldAutoPlay || duration === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Auto-play the newly loaded audio
|
||||
debug.log('Auto-playing next track in story mode');
|
||||
wavesurfer.seekTo(0);
|
||||
wavesurfer.play().catch((error) => {
|
||||
debug.error('Failed to auto-play:', error);
|
||||
setIsPlaying(false);
|
||||
setError(`Playback error: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
|
||||
// Clear the auto-play flag
|
||||
clearAutoPlayFlag();
|
||||
}, [shouldAutoPlay, duration, setIsPlaying, clearAutoPlayFlag]);
|
||||
|
||||
// Handle loop - WaveSurfer handles this via the 'finish' event
|
||||
const onKeyDown = (e: KeyboardEvent) => {
|
||||
if (e.code !== 'Space') return;
|
||||
// Ignore if user is typing in an input/textarea
|
||||
const tag = (e.target as HTMLElement)?.tagName;
|
||||
if (tag === 'INPUT' || tag === 'TEXTAREA' || (e.target as HTMLElement)?.isContentEditable) {
|
||||
return;
|
||||
}
|
||||
if (audioUrl && duration > 0 && wavesurferRef.current) {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
if (wavesurferRef.current.isPlaying()) {
|
||||
wavesurferRef.current.pause();
|
||||
} else {
|
||||
wavesurferRef.current.play().catch((err) => debug.error('Spacebar play failed:', err));
|
||||
}
|
||||
}
|
||||
};
|
||||
document.addEventListener('keydown', onKeyDown, true);
|
||||
return () => document.removeEventListener('keydown', onKeyDown, true);
|
||||
}, [audioUrl, duration]);
|
||||
|
||||
const handlePlayPause = async () => {
|
||||
// Standard WaveSurfer playback (works for both normal and native playback modes)
|
||||
@@ -741,32 +531,32 @@ export function AudioPlayer() {
|
||||
size="icon"
|
||||
onClick={handlePlayPause}
|
||||
disabled={isLoading || duration === 0}
|
||||
className="shrink-0"
|
||||
className={`shrink-0 -mt-2 ${isPlaying ? 'bg-accent text-accent-foreground' : ''}`}
|
||||
title={duration === 0 && !isLoading ? 'Audio not loaded' : ''}
|
||||
aria-label={
|
||||
duration === 0 && !isLoading ? 'Audio not loaded' : isPlaying ? 'Pause' : 'Play'
|
||||
}
|
||||
>
|
||||
{isPlaying ? <Pause className="h-5 w-5" /> : <Play className="h-5 w-5" />}
|
||||
{isPlaying ? (
|
||||
<Pause className="h-5 w-5 fill-current" />
|
||||
) : (
|
||||
<Play className="h-5 w-5 fill-current" />
|
||||
)}
|
||||
</Button>
|
||||
|
||||
{/* Waveform */}
|
||||
<div className="flex-1 min-w-0 flex flex-col gap-1">
|
||||
<div ref={waveformRef} className="w-full min-h-[80px]" />
|
||||
{duration > 0 && (
|
||||
<Slider
|
||||
value={duration > 0 ? [(currentTime / duration) * 100] : [0]}
|
||||
onValueChange={handleSeek}
|
||||
max={100}
|
||||
step={0.1}
|
||||
className="w-full"
|
||||
aria-label="Playback position"
|
||||
aria-valuetext={`${formatAudioDuration(currentTime)} of ${formatAudioDuration(duration)}`}
|
||||
/>
|
||||
)}
|
||||
{isLoading && (
|
||||
<div className="text-xs text-muted-foreground text-center py-2">Loading audio...</div>
|
||||
)}
|
||||
<div ref={waveformRef} className="w-full min-h-[80px] select-none" />
|
||||
<Slider
|
||||
value={duration > 0 ? [(currentTime / duration) * 100] : [0]}
|
||||
onValueChange={handleSeek}
|
||||
max={100}
|
||||
step={0.1}
|
||||
className="w-full"
|
||||
aria-label="Playback position"
|
||||
aria-valuetext={`${formatAudioDuration(currentTime)} of ${formatAudioDuration(duration)}`}
|
||||
/>
|
||||
|
||||
{error && <div className="text-xs text-destructive text-center py-2">{error}</div>}
|
||||
</div>
|
||||
|
||||
@@ -777,19 +567,12 @@ export function AudioPlayer() {
|
||||
<span className="font-mono">{formatAudioDuration(duration)}</span>
|
||||
</div>
|
||||
|
||||
{/* Title */}
|
||||
{title && (
|
||||
<div className="text-sm font-medium truncate max-w-[200px] shrink-0 hidden lg:block">
|
||||
{title}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Loop Button */}
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={toggleLoop}
|
||||
className={isLooping ? 'text-primary' : ''}
|
||||
className={isLooping ? 'bg-accent text-accent-foreground' : ''}
|
||||
title="Toggle loop"
|
||||
aria-label={isLooping ? 'Stop looping' : 'Loop'}
|
||||
>
|
||||
|
||||
@@ -124,7 +124,7 @@ export function AudioTab() {
|
||||
);
|
||||
}
|
||||
|
||||
const handleChannelDelete = async (e, channelId) => {
|
||||
const handleChannelDelete = async (e: React.MouseEvent, channelId: string) => {
|
||||
e.stopPropagation();
|
||||
if (await confirm('Delete this channel?')) {
|
||||
deleteChannel.mutate(channelId);
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
import {EffectsDetail} from "./EffectsDetail";
|
||||
import {EffectsList} from "./EffectsList";
|
||||
import { EffectsDetail } from './EffectsDetail';
|
||||
import { EffectsList } from './EffectsList';
|
||||
|
||||
export function EffectsTab() {
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 overflow-hidden">
|
||||
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
|
||||
{/* Left - Presets list */}
|
||||
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
|
||||
<EffectsList />
|
||||
</div>
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 overflow-hidden">
|
||||
<div className="flex-1 min-h-0 flex gap-6 overflow-hidden">
|
||||
{/* Left - Presets list */}
|
||||
<div className="w-full max-w-[360px] shrink-0 flex flex-col min-h-0">
|
||||
<EffectsList />
|
||||
</div>
|
||||
|
||||
{/* Right - Detail / editor */}
|
||||
<div className="flex-1 min-h-0 flex flex-col">
|
||||
<EffectsDetail />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
{/* Right - Detail / editor */}
|
||||
<div className="flex-1 min-h-0 flex flex-col">
|
||||
<EffectsDetail />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
import { useEffect } from 'react';
|
||||
import type { UseFormReturn } from 'react-hook-form';
|
||||
import { FormControl } from '@/components/ui/form';
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from '@/components/ui/select';
|
||||
import type { VoiceProfileResponse } from '@/lib/api/types';
|
||||
import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
|
||||
import type { GenerationFormValues } from '@/lib/hooks/useGenerationForm';
|
||||
|
||||
/**
|
||||
* Engine/model options and their display metadata.
|
||||
* Adding a new engine means adding one entry here.
|
||||
*/
|
||||
const ENGINE_OPTIONS = [
|
||||
{ value: 'qwen:1.7B', label: 'Qwen3-TTS 1.7B', engine: 'qwen' },
|
||||
{ value: 'qwen:0.6B', label: 'Qwen3-TTS 0.6B', engine: 'qwen' },
|
||||
{ value: 'qwen_custom_voice:1.7B', label: 'Qwen CustomVoice 1.7B', engine: 'qwen_custom_voice' },
|
||||
{ value: 'qwen_custom_voice:0.6B', label: 'Qwen CustomVoice 0.6B', engine: 'qwen_custom_voice' },
|
||||
{ value: 'luxtts', label: 'LuxTTS', engine: 'luxtts' },
|
||||
{ value: 'chatterbox', label: 'Chatterbox', engine: 'chatterbox' },
|
||||
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo', engine: 'chatterbox_turbo' },
|
||||
{ value: 'tada:1B', label: 'TADA 1B', engine: 'tada' },
|
||||
{ value: 'tada:3B', label: 'TADA 3B Multilingual', engine: 'tada' },
|
||||
{ value: 'kokoro', label: 'Kokoro 82M', engine: 'kokoro' },
|
||||
] as const;
|
||||
|
||||
const ENGINE_DESCRIPTIONS: Record<string, string> = {
|
||||
qwen: 'Multi-language, two sizes',
|
||||
qwen_custom_voice: '9 preset voices, instruct control',
|
||||
luxtts: 'Fast, English-focused',
|
||||
chatterbox: '23 languages, incl. Hebrew',
|
||||
chatterbox_turbo: 'English, [laugh] [cough] tags',
|
||||
tada: 'HumeAI, 700s+ coherent audio',
|
||||
kokoro: '82M params, CPU realtime, 8 langs',
|
||||
};
|
||||
|
||||
/** Engines that only support English and should force language to 'en' on select. */
|
||||
const ENGLISH_ONLY_ENGINES = new Set(['luxtts', 'chatterbox_turbo']);
|
||||
|
||||
/** Engines that support cloned (reference audio) profiles. */
|
||||
const CLONING_ENGINES = new Set(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada']);
|
||||
|
||||
function getAvailableOptions(selectedProfile?: VoiceProfileResponse | null) {
|
||||
if (!selectedProfile) return ENGINE_OPTIONS;
|
||||
return ENGINE_OPTIONS.filter((opt) => isProfileCompatibleWithEngine(selectedProfile, opt.engine));
|
||||
}
|
||||
|
||||
function getSelectValue(engine: string, modelSize?: string): string {
|
||||
if (engine === 'qwen') return `qwen:${modelSize || '1.7B'}`;
|
||||
if (engine === 'qwen_custom_voice') return `qwen_custom_voice:${modelSize || '1.7B'}`;
|
||||
if (engine === 'tada') return `tada:${modelSize || '1B'}`;
|
||||
return engine;
|
||||
}
|
||||
|
||||
export function applyEngineSelection(form: UseFormReturn<GenerationFormValues>, value: string) {
|
||||
if (value.startsWith('qwen_custom_voice:')) {
|
||||
const [, modelSize] = value.split(':');
|
||||
form.setValue('engine', 'qwen_custom_voice');
|
||||
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
|
||||
const currentLang = form.getValues('language');
|
||||
const available = getLanguageOptionsForEngine('qwen_custom_voice');
|
||||
if (!available.some((l) => l.value === currentLang)) {
|
||||
form.setValue('language', available[0]?.value ?? 'en');
|
||||
}
|
||||
} else if (value.startsWith('qwen:')) {
|
||||
const [, modelSize] = value.split(':');
|
||||
form.setValue('engine', 'qwen');
|
||||
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
|
||||
// Validate language is supported by Qwen
|
||||
const currentLang = form.getValues('language');
|
||||
const available = getLanguageOptionsForEngine('qwen');
|
||||
if (!available.some((l) => l.value === currentLang)) {
|
||||
form.setValue('language', available[0]?.value ?? 'en');
|
||||
}
|
||||
} else if (value.startsWith('tada:')) {
|
||||
const [, modelSize] = value.split(':');
|
||||
form.setValue('engine', 'tada');
|
||||
form.setValue('modelSize', modelSize as '1B' | '3B');
|
||||
// TADA 1B is English-only; 3B is multilingual
|
||||
if (modelSize === '1B') {
|
||||
form.setValue('language', 'en');
|
||||
} else {
|
||||
const currentLang = form.getValues('language');
|
||||
const available = getLanguageOptionsForEngine('tada');
|
||||
if (!available.some((l) => l.value === currentLang)) {
|
||||
form.setValue('language', available[0]?.value ?? 'en');
|
||||
}
|
||||
}
|
||||
} else {
|
||||
form.setValue('engine', value as GenerationFormValues['engine']);
|
||||
form.setValue('modelSize', undefined as unknown as '1.7B' | '0.6B');
|
||||
if (ENGLISH_ONLY_ENGINES.has(value)) {
|
||||
form.setValue('language', 'en');
|
||||
} else {
|
||||
// If current language isn't supported by the new engine, reset to first available
|
||||
const currentLang = form.getValues('language');
|
||||
const available = getLanguageOptionsForEngine(value);
|
||||
if (!available.some((l) => l.value === currentLang)) {
|
||||
form.setValue('language', available[0]?.value ?? 'en');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
interface EngineModelSelectorProps {
|
||||
form: UseFormReturn<GenerationFormValues>;
|
||||
compact?: boolean;
|
||||
selectedProfile?: VoiceProfileResponse | null;
|
||||
}
|
||||
|
||||
export function EngineModelSelector({ form, compact, selectedProfile }: EngineModelSelectorProps) {
|
||||
const engine = form.watch('engine') || 'qwen';
|
||||
const modelSize = form.watch('modelSize');
|
||||
const selectValue = getSelectValue(engine, modelSize);
|
||||
const availableOptions = getAvailableOptions(selectedProfile);
|
||||
|
||||
const currentEngineAvailable = availableOptions.some((opt) => opt.value === selectValue);
|
||||
|
||||
useEffect(() => {
|
||||
if (!currentEngineAvailable && availableOptions.length > 0) {
|
||||
applyEngineSelection(form, availableOptions[0].value);
|
||||
}
|
||||
}, [availableOptions, currentEngineAvailable, form]);
|
||||
|
||||
const itemClass = compact ? 'text-xs text-muted-foreground' : undefined;
|
||||
const triggerClass = compact
|
||||
? 'h-8 text-xs bg-card border-border rounded-full hover:bg-background/50 transition-all'
|
||||
: undefined;
|
||||
|
||||
return (
|
||||
<Select value={selectValue} onValueChange={(v) => applyEngineSelection(form, v)}>
|
||||
<FormControl>
|
||||
<SelectTrigger className={triggerClass}>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
{availableOptions.map((opt) => (
|
||||
<SelectItem key={opt.value} value={opt.value} className={itemClass}>
|
||||
{opt.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
);
|
||||
}
|
||||
|
||||
/** Returns a human-readable description for the currently selected engine. */
|
||||
export function getEngineDescription(engine: string): string {
|
||||
return ENGINE_DESCRIPTIONS[engine] ?? '';
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a profile is compatible with the currently selected engine.
|
||||
* Useful for UI hints.
|
||||
*/
|
||||
export function isProfileCompatibleWithEngine(
|
||||
profile: VoiceProfileResponse,
|
||||
engine: string,
|
||||
): boolean {
|
||||
const voiceType = profile.voice_type || 'cloned';
|
||||
if (voiceType === 'preset') return profile.preset_engine === engine;
|
||||
if (voiceType === 'cloned') return CLONING_ENGINES.has(engine);
|
||||
return true; // designed — future
|
||||
}
|
||||
@@ -1,8 +1,8 @@
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { useMatchRoute } from '@tanstack/react-router';
|
||||
import { AnimatePresence, motion } from 'framer-motion';
|
||||
import { Loader2, SlidersHorizontal, Sparkles } from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { EffectsChainEditor } from '@/components/Effects/EffectsChainEditor';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Form, FormControl, FormField, FormItem, FormMessage } from '@/components/ui/form';
|
||||
import {
|
||||
@@ -13,7 +13,7 @@ import {
|
||||
SelectValue,
|
||||
} from '@/components/ui/select';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import type { EffectConfig } from '@/lib/api/types';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import { getLanguageOptionsForEngine, type LanguageCode } from '@/lib/constants/languages';
|
||||
import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
|
||||
import { useProfile, useProfiles } from '@/lib/hooks/useProfiles';
|
||||
@@ -22,6 +22,7 @@ import { cn } from '@/lib/utils/cn';
|
||||
import { useGenerationStore } from '@/stores/generationStore';
|
||||
import { useStoryStore } from '@/stores/storyStore';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
import { EngineModelSelector } from './EngineModelSelector';
|
||||
import { ParalinguisticInput } from './ParalinguisticInput';
|
||||
|
||||
interface FloatingGenerateBoxProps {
|
||||
@@ -35,11 +36,12 @@ export function FloatingGenerateBox({
|
||||
}: FloatingGenerateBoxProps) {
|
||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||
const setSelectedProfileId = useUIStore((state) => state.setSelectedProfileId);
|
||||
const setSelectedEngine = useUIStore((state) => state.setSelectedEngine);
|
||||
const { data: selectedProfile } = useProfile(selectedProfileId || '');
|
||||
const { data: profiles } = useProfiles();
|
||||
const [isExpanded, setIsExpanded] = useState(false);
|
||||
const [isInstructMode, setIsInstructMode] = useState(false);
|
||||
const [effectsChain, setEffectsChain] = useState<EffectConfig[]>([]);
|
||||
const [isInstructExpanded, setIsInstructExpanded] = useState(false);
|
||||
const [selectedPresetId, setSelectedPresetId] = useState<string | null>(null);
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const textareaRef = useRef<HTMLTextAreaElement | null>(null);
|
||||
const matchRoute = useMatchRoute();
|
||||
@@ -49,18 +51,33 @@ export function FloatingGenerateBox({
|
||||
const { data: currentStory } = useStory(selectedStoryId);
|
||||
const addPendingStoryAdd = useGenerationStore((s) => s.addPendingStoryAdd);
|
||||
|
||||
// Fetch effect presets for the dropdown
|
||||
const { data: effectPresets } = useQuery({
|
||||
queryKey: ['effectPresets'],
|
||||
queryFn: () => apiClient.listEffectPresets(),
|
||||
});
|
||||
|
||||
// Calculate if track editor is visible (on stories route with items)
|
||||
const hasTrackEditor = isStoriesRoute && currentStory && currentStory.items.length > 0;
|
||||
|
||||
const { form, handleSubmit, isPending } = useGenerationForm({
|
||||
onSuccess: async (generationId) => {
|
||||
setIsExpanded(false);
|
||||
// Defer the story add until TTS completes — useGenerationProgress handles it
|
||||
// Defer the story add until TTS completes -- useGenerationProgress handles it
|
||||
if (isStoriesRoute && selectedStoryId && generationId) {
|
||||
addPendingStoryAdd(generationId, selectedStoryId);
|
||||
}
|
||||
},
|
||||
getEffectsChain: () => (effectsChain.length > 0 ? effectsChain : undefined),
|
||||
getEffectsChain: () => {
|
||||
if (!selectedPresetId) return undefined;
|
||||
// Profile's own effects chain (no matching preset)
|
||||
if (selectedPresetId === '_profile') {
|
||||
return selectedProfile?.effects_chain ?? undefined;
|
||||
}
|
||||
if (!effectPresets) return undefined;
|
||||
const preset = effectPresets.find((p) => p.id === selectedPresetId);
|
||||
return preset?.effects_chain;
|
||||
},
|
||||
});
|
||||
|
||||
// Click away handler to collapse the box
|
||||
@@ -100,12 +117,63 @@ export function FloatingGenerateBox({
|
||||
}
|
||||
}, [selectedProfileId, profiles, setSelectedProfileId]);
|
||||
|
||||
// Sync generation form language with selected profile's language
|
||||
// Sync engine selection to global store so ProfileList can filter
|
||||
const watchedEngine = form.watch('engine');
|
||||
useEffect(() => {
|
||||
if (watchedEngine) {
|
||||
setSelectedEngine(watchedEngine);
|
||||
}
|
||||
}, [watchedEngine, setSelectedEngine]);
|
||||
|
||||
// Sync generation form language, engine, and effects with selected profile
|
||||
type EngineValue =
|
||||
| 'qwen'
|
||||
| 'luxtts'
|
||||
| 'chatterbox'
|
||||
| 'chatterbox_turbo'
|
||||
| 'tada'
|
||||
| 'kokoro'
|
||||
| 'qwen_custom_voice';
|
||||
useEffect(() => {
|
||||
if (selectedProfile?.language) {
|
||||
form.setValue('language', selectedProfile.language as LanguageCode);
|
||||
}
|
||||
}, [selectedProfile, form]);
|
||||
// Auto-switch engine to match the profile
|
||||
const engine = selectedProfile?.default_engine ?? selectedProfile?.preset_engine;
|
||||
if (engine) {
|
||||
form.setValue('engine', engine as EngineValue);
|
||||
} else if (selectedProfile && selectedProfile.voice_type !== 'preset') {
|
||||
// Cloned/designed profile with no default — ensure a compatible (non-preset) engine
|
||||
const currentEngine = form.getValues('engine');
|
||||
const presetEngines = new Set(['kokoro', 'qwen_custom_voice']);
|
||||
if (currentEngine && presetEngines.has(currentEngine)) {
|
||||
form.setValue('engine', 'qwen');
|
||||
}
|
||||
}
|
||||
// Pre-fill effects from profile defaults
|
||||
if (
|
||||
selectedProfile?.effects_chain &&
|
||||
selectedProfile.effects_chain.length > 0 &&
|
||||
effectPresets
|
||||
) {
|
||||
// Try to match against a known preset
|
||||
const profileChainJson = JSON.stringify(selectedProfile.effects_chain);
|
||||
const matchingPreset = effectPresets.find(
|
||||
(p) => JSON.stringify(p.effects_chain) === profileChainJson,
|
||||
);
|
||||
if (matchingPreset) {
|
||||
setSelectedPresetId(matchingPreset.id);
|
||||
} else {
|
||||
// No matching preset — use special value to pass profile chain directly
|
||||
setSelectedPresetId('_profile');
|
||||
}
|
||||
} else if (
|
||||
selectedProfile &&
|
||||
(!selectedProfile.effects_chain || selectedProfile.effects_chain.length === 0)
|
||||
) {
|
||||
setSelectedPresetId(null);
|
||||
}
|
||||
}, [selectedProfile, effectPresets, form]);
|
||||
|
||||
// Auto-resize textarea based on content (only when expanded)
|
||||
useEffect(() => {
|
||||
@@ -188,111 +256,57 @@ export function FloatingGenerateBox({
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)}>
|
||||
<div className="flex gap-2">
|
||||
<motion.div
|
||||
className={cn('flex-1', isExpanded && 'mr-12')}
|
||||
transition={{ duration: 0.3, ease: 'easeOut' }}
|
||||
>
|
||||
{/* Text field - hidden when in instruct mode */}
|
||||
<div style={{ display: isInstructMode ? 'none' : 'block' }}>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="text"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
{form.watch('engine') === 'chatterbox_turbo' ? (
|
||||
<ParalinguisticInput
|
||||
value={field.value}
|
||||
onChange={field.onChange}
|
||||
placeholder={
|
||||
isStoriesRoute && currentStory
|
||||
? `Generate speech for "${currentStory.name}"... (type / for effects)`
|
||||
: selectedProfile
|
||||
? `Type / for effects like [laugh], [sigh]...`
|
||||
: 'Select a voice profile above...'
|
||||
}
|
||||
className="px-3 py-2 resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
overflowY: 'auto',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
) : (
|
||||
<Textarea
|
||||
{...field}
|
||||
ref={(node: HTMLTextAreaElement | null) => {
|
||||
// Store ref for auto-resize (only for active field)
|
||||
if (!isInstructMode) {
|
||||
textareaRef.current = node;
|
||||
}
|
||||
// Forward ref to react-hook-form
|
||||
if (typeof field.ref === 'function') {
|
||||
field.ref(node);
|
||||
}
|
||||
}}
|
||||
placeholder={
|
||||
isStoriesRoute && currentStory
|
||||
? `Generate speech for "${currentStory.name}"...`
|
||||
: selectedProfile
|
||||
? `Generate speech using ${selectedProfile.name}...`
|
||||
: 'Select a voice profile above...'
|
||||
}
|
||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
)}
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
{/* Instruct field - hidden when in text mode */}
|
||||
<div style={{ display: isInstructMode ? 'block' : 'none' }}>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
<motion.div className="flex-1" transition={{ duration: 0.3, ease: 'easeOut' }}>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="text"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormControl>
|
||||
<motion.div
|
||||
animate={{
|
||||
height: isExpanded ? 'auto' : '32px',
|
||||
}}
|
||||
transition={{ duration: 0.15, ease: 'easeOut' }}
|
||||
style={{ overflow: 'hidden' }}
|
||||
>
|
||||
{form.watch('engine') === 'chatterbox_turbo' ? (
|
||||
<ParalinguisticInput
|
||||
value={field.value}
|
||||
onChange={field.onChange}
|
||||
placeholder={
|
||||
isStoriesRoute && currentStory
|
||||
? `Generate speech for "${currentStory.name}"... (type / for effects)`
|
||||
: selectedProfile
|
||||
? `Type / for effects like [laugh], [sigh]...`
|
||||
: 'Select a voice profile above...'
|
||||
}
|
||||
className="px-3 py-2 resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
maxHeight: '300px',
|
||||
overflowY: 'auto',
|
||||
}}
|
||||
disabled={!selectedProfileId}
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
) : (
|
||||
<Textarea
|
||||
{...field}
|
||||
ref={(node: HTMLTextAreaElement | null) => {
|
||||
// Store ref for auto-resize (only for active field)
|
||||
if (isInstructMode) {
|
||||
textareaRef.current = node;
|
||||
}
|
||||
// Forward ref to react-hook-form
|
||||
textareaRef.current = node;
|
||||
if (typeof field.ref === 'function') {
|
||||
field.ref(node);
|
||||
}
|
||||
}}
|
||||
placeholder="e.g. very happy and excited"
|
||||
placeholder={
|
||||
isStoriesRoute && currentStory
|
||||
? `Generate speech for "${currentStory.name}"...`
|
||||
: selectedProfile
|
||||
? `Generate speech using ${selectedProfile.name}...`
|
||||
: 'Select a voice profile above...'
|
||||
}
|
||||
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
|
||||
style={{
|
||||
minHeight: isExpanded ? '100px' : '32px',
|
||||
@@ -302,13 +316,13 @@ export function FloatingGenerateBox({
|
||||
onClick={() => setIsExpanded(true)}
|
||||
onFocus={() => setIsExpanded(true)}
|
||||
/>
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</motion.div>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</motion.div>
|
||||
|
||||
<div className="relative shrink-0">
|
||||
@@ -340,8 +354,10 @@ export function FloatingGenerateBox({
|
||||
: 'Generate speech'}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Instruct toggle — only for Qwen CustomVoice, which actually honors the kwarg */}
|
||||
<AnimatePresence>
|
||||
{isExpanded && form.watch('engine') === 'qwen' && (
|
||||
{isExpanded && form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<motion.div
|
||||
initial={{ opacity: 0, scale: 0.8 }}
|
||||
animate={{ opacity: 1, scale: 1 }}
|
||||
@@ -354,23 +370,24 @@ export function FloatingGenerateBox({
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => setIsInstructMode(!isInstructMode)}
|
||||
onClick={() => setIsInstructExpanded((prev) => !prev)}
|
||||
className={cn(
|
||||
'h-10 w-10 rounded-full transition-all duration-200',
|
||||
isInstructMode
|
||||
isInstructExpanded
|
||||
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
|
||||
: effectsChain.length > 0
|
||||
? 'bg-accent/50 text-accent-foreground border border-accent/50 hover:bg-accent/70'
|
||||
: 'bg-card border border-border hover:bg-background/50',
|
||||
: 'bg-card border border-border hover:bg-background/50',
|
||||
)}
|
||||
aria-label={
|
||||
isInstructMode ? 'Fine tune instructions, on' : 'Fine tune instructions'
|
||||
isInstructExpanded
|
||||
? 'Hide delivery instructions'
|
||||
: 'Show delivery instructions'
|
||||
}
|
||||
aria-pressed={isInstructExpanded}
|
||||
>
|
||||
<SlidersHorizontal className="h-4 w-4" />
|
||||
</Button>
|
||||
<span className="pointer-events-none absolute bottom-full left-1/2 -translate-x-1/2 mb-2 whitespace-nowrap rounded-md bg-popover px-3 py-1.5 text-xs text-popover-foreground border border-border opacity-0 transition-opacity group-hover:opacity-100 z-[9999]">
|
||||
Fine tune instructions & effects
|
||||
Delivery instructions (tone, emotion, pace)
|
||||
</span>
|
||||
</div>
|
||||
</motion.div>
|
||||
@@ -379,19 +396,34 @@ export function FloatingGenerateBox({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Effects chain editor panel - shown alongside instruct */}
|
||||
{/* Additive instruct textarea — shown below main text when toggle is on and engine supports it */}
|
||||
<AnimatePresence>
|
||||
{isExpanded && isInstructMode && (
|
||||
{isInstructExpanded && form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<motion.div
|
||||
initial={{ height: 0, opacity: 0 }}
|
||||
animate={{ height: 'auto', opacity: 1 }}
|
||||
exit={{ height: 0, opacity: 0 }}
|
||||
transition={{ duration: 0.2 }}
|
||||
className="overflow-hidden mt-2"
|
||||
initial={{ opacity: 0, height: 0 }}
|
||||
animate={{ opacity: 1, height: 'auto' }}
|
||||
exit={{ opacity: 0, height: 0 }}
|
||||
transition={{ duration: 0.2, ease: 'easeOut' }}
|
||||
className="overflow-hidden"
|
||||
>
|
||||
<div className="border-t border-border/50 pt-2 pb-1">
|
||||
<EffectsChainEditor value={effectsChain} onChange={setEffectsChain} compact />
|
||||
</div>
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
render={({ field }) => (
|
||||
<FormItem className="mt-2">
|
||||
<FormControl>
|
||||
<Textarea
|
||||
{...field}
|
||||
placeholder="Delivery instructions — e.g. Speak slowly with warmth, Authoritative and clear..."
|
||||
className="resize-none bg-transparent border border-accent/20 focus-visible:ring-1 focus-visible:ring-accent/40 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full px-3 py-2"
|
||||
style={{ minHeight: '60px', maxHeight: '160px' }}
|
||||
maxLength={500}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage className="text-xs" />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</motion.div>
|
||||
)}
|
||||
</AnimatePresence>
|
||||
@@ -454,57 +486,35 @@ export function FloatingGenerateBox({
|
||||
}}
|
||||
/>
|
||||
|
||||
<FormItem className="flex-1 space-y-0">
|
||||
<EngineModelSelector form={form} compact />
|
||||
</FormItem>
|
||||
|
||||
<FormItem className="flex-1 space-y-0">
|
||||
<Select
|
||||
value={
|
||||
form.watch('engine') === 'luxtts'
|
||||
? 'luxtts'
|
||||
: form.watch('engine') === 'chatterbox'
|
||||
? 'chatterbox'
|
||||
: form.watch('engine') === 'chatterbox_turbo'
|
||||
? 'chatterbox_turbo'
|
||||
: `qwen:${form.watch('modelSize') || '1.7B'}`
|
||||
value={selectedPresetId || 'none'}
|
||||
onValueChange={(value) =>
|
||||
setSelectedPresetId(value === 'none' ? null : value)
|
||||
}
|
||||
onValueChange={(value) => {
|
||||
if (value === 'luxtts') {
|
||||
form.setValue('engine', 'luxtts');
|
||||
form.setValue('language', 'en');
|
||||
} else if (value === 'chatterbox') {
|
||||
form.setValue('engine', 'chatterbox');
|
||||
} else if (value === 'chatterbox_turbo') {
|
||||
form.setValue('engine', 'chatterbox_turbo');
|
||||
form.setValue('language', 'en');
|
||||
} else {
|
||||
const [, modelSize] = value.split(':');
|
||||
form.setValue('engine', 'qwen');
|
||||
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
|
||||
}
|
||||
}}
|
||||
>
|
||||
<FormControl>
|
||||
<SelectTrigger className="h-8 text-xs bg-card border-border rounded-full hover:bg-background/50 transition-all">
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectTrigger className="h-8 text-xs bg-card border-border rounded-full hover:bg-background/50 transition-all">
|
||||
<SelectValue placeholder="No effects" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="qwen:1.7B" className="text-xs text-muted-foreground">
|
||||
Qwen3-TTS 1.7B
|
||||
</SelectItem>
|
||||
<SelectItem value="qwen:0.6B" className="text-xs text-muted-foreground">
|
||||
Qwen3-TTS 0.6B
|
||||
</SelectItem>
|
||||
<SelectItem value="luxtts" className="text-xs text-muted-foreground">
|
||||
LuxTTS
|
||||
</SelectItem>
|
||||
<SelectItem value="chatterbox" className="text-xs text-muted-foreground">
|
||||
Chatterbox
|
||||
</SelectItem>
|
||||
<SelectItem
|
||||
value="chatterbox_turbo"
|
||||
className="text-xs text-muted-foreground"
|
||||
>
|
||||
Chatterbox Turbo
|
||||
<SelectItem value="none" className="text-xs">
|
||||
No effects
|
||||
</SelectItem>
|
||||
{selectedProfile?.effects_chain &&
|
||||
selectedProfile.effects_chain.length > 0 && (
|
||||
<SelectItem value="_profile" className="text-xs">
|
||||
Profile default
|
||||
</SelectItem>
|
||||
)}
|
||||
{effectPresets?.map((preset) => (
|
||||
<SelectItem key={preset.id} value={preset.id} className="text-xs">
|
||||
{preset.name}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</FormItem>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Loader2, Mic } from 'lucide-react';
|
||||
import { useEffect } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
import {
|
||||
@@ -19,18 +20,45 @@ import {
|
||||
SelectValue,
|
||||
} from '@/components/ui/select';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
|
||||
import { getLanguageOptionsForEngine, type LanguageCode } from '@/lib/constants/languages';
|
||||
import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
|
||||
import { useProfile } from '@/lib/hooks/useProfiles';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
import {
|
||||
applyEngineSelection,
|
||||
EngineModelSelector,
|
||||
getEngineDescription,
|
||||
} from './EngineModelSelector';
|
||||
import { ParalinguisticInput } from './ParalinguisticInput';
|
||||
|
||||
function getEngineSelectValue(engine: string): string {
|
||||
if (engine === 'qwen') return 'qwen:1.7B';
|
||||
if (engine === 'qwen_custom_voice') return 'qwen_custom_voice:1.7B';
|
||||
if (engine === 'tada') return 'tada:1B';
|
||||
return engine;
|
||||
}
|
||||
|
||||
export function GenerationForm() {
|
||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||
const { data: selectedProfile } = useProfile(selectedProfileId || '');
|
||||
|
||||
const { form, handleSubmit, isPending } = useGenerationForm();
|
||||
|
||||
useEffect(() => {
|
||||
if (!selectedProfile) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (selectedProfile.language) {
|
||||
form.setValue('language', selectedProfile.language as LanguageCode);
|
||||
}
|
||||
|
||||
const preferredEngine = selectedProfile.default_engine || selectedProfile.preset_engine;
|
||||
if (preferredEngine) {
|
||||
applyEngineSelection(form, getEngineSelectValue(preferredEngine));
|
||||
}
|
||||
}, [form, selectedProfile]);
|
||||
|
||||
async function onSubmit(data: Parameters<typeof handleSubmit>[0]) {
|
||||
await handleSubmit(data, selectedProfileId);
|
||||
}
|
||||
@@ -90,7 +118,7 @@ export function GenerationForm() {
|
||||
)}
|
||||
/>
|
||||
|
||||
{form.watch('engine') === 'qwen' && (
|
||||
{form.watch('engine') === 'qwen_custom_voice' && (
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="instruct"
|
||||
@@ -117,53 +145,9 @@ export function GenerationForm() {
|
||||
<div className="grid gap-4 md:grid-cols-3">
|
||||
<FormItem>
|
||||
<FormLabel>Model</FormLabel>
|
||||
<Select
|
||||
value={
|
||||
form.watch('engine') === 'luxtts'
|
||||
? 'luxtts'
|
||||
: form.watch('engine') === 'chatterbox'
|
||||
? 'chatterbox'
|
||||
: form.watch('engine') === 'chatterbox_turbo'
|
||||
? 'chatterbox_turbo'
|
||||
: `qwen:${form.watch('modelSize') || '1.7B'}`
|
||||
}
|
||||
onValueChange={(value) => {
|
||||
if (value === 'luxtts') {
|
||||
form.setValue('engine', 'luxtts');
|
||||
form.setValue('language', 'en');
|
||||
} else if (value === 'chatterbox') {
|
||||
form.setValue('engine', 'chatterbox');
|
||||
} else if (value === 'chatterbox_turbo') {
|
||||
form.setValue('engine', 'chatterbox_turbo');
|
||||
form.setValue('language', 'en');
|
||||
} else {
|
||||
const [, modelSize] = value.split(':');
|
||||
form.setValue('engine', 'qwen');
|
||||
form.setValue('modelSize', modelSize as '1.7B' | '0.6B');
|
||||
}
|
||||
}}
|
||||
>
|
||||
<FormControl>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
<SelectItem value="qwen:1.7B">Qwen3-TTS 1.7B</SelectItem>
|
||||
<SelectItem value="qwen:0.6B">Qwen3-TTS 0.6B</SelectItem>
|
||||
<SelectItem value="luxtts">LuxTTS</SelectItem>
|
||||
<SelectItem value="chatterbox">Chatterbox</SelectItem>
|
||||
<SelectItem value="chatterbox_turbo">Chatterbox Turbo</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<EngineModelSelector form={form} selectedProfile={selectedProfile} />
|
||||
<FormDescription>
|
||||
{form.watch('engine') === 'luxtts'
|
||||
? 'Fast, English-focused'
|
||||
: form.watch('engine') === 'chatterbox'
|
||||
? '23 languages, incl. Hebrew'
|
||||
: form.watch('engine') === 'chatterbox_turbo'
|
||||
? 'English, [laugh] [cough] tags'
|
||||
: 'Multi-language, two sizes'}
|
||||
{getEngineDescription(form.watch('engine') || 'qwen')}
|
||||
</FormDescription>
|
||||
</FormItem>
|
||||
|
||||
|
||||
@@ -1,21 +1,20 @@
|
||||
import { useQueryClient } from '@tanstack/react-query';
|
||||
import { useMutation, useQueryClient } from '@tanstack/react-query';
|
||||
import { AnimatePresence, motion } from 'framer-motion';
|
||||
import {
|
||||
AlignCenter,
|
||||
AudioLines,
|
||||
AudioWaveform,
|
||||
Download,
|
||||
FileArchive,
|
||||
Loader2,
|
||||
MoreHorizontal,
|
||||
Play,
|
||||
RotateCcw,
|
||||
Square,
|
||||
Star,
|
||||
Trash2,
|
||||
Wand2,
|
||||
} from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import Loader from 'react-loaders';
|
||||
|
||||
import { EffectsChainEditor } from '@/components/Effects/EffectsChainEditor';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import {
|
||||
@@ -45,6 +44,7 @@ import { apiClient } from '@/lib/api/client';
|
||||
import type { EffectConfig, GenerationVersionResponse, HistoryResponse } from '@/lib/api/types';
|
||||
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
import {
|
||||
useClearFailedGenerations,
|
||||
useDeleteGeneration,
|
||||
useExportGeneration,
|
||||
useExportGenerationAudio,
|
||||
@@ -56,8 +56,35 @@ import { formatDate, formatDuration, formatEngineName } from '@/lib/utils/format
|
||||
import { useGenerationStore } from '@/stores/generationStore';
|
||||
import { usePlayerStore } from '@/stores/playerStore';
|
||||
|
||||
// OLD TABLE-BASED COMPONENT - REMOVED (can be found in git history)
|
||||
// This is the new alternate history view with fixed height rows
|
||||
// ─── Audio Bars ─────────────────────────────────────────────────────────────
|
||||
|
||||
function AudioBars({ mode }: { mode: 'idle' | 'generating' | 'playing' }) {
|
||||
const barColor = mode !== 'idle' ? 'bg-accent' : 'bg-muted-foreground/40';
|
||||
return (
|
||||
<div className="flex items-center gap-[2px] h-5">
|
||||
{[0, 1, 2, 3, 4].map((i) => (
|
||||
<motion.div
|
||||
key={`${mode}-${i}`}
|
||||
className={`w-[3px] rounded-full ${barColor}`}
|
||||
animate={
|
||||
mode === 'generating'
|
||||
? { height: ['6px', '16px', '6px'] }
|
||||
: mode === 'playing'
|
||||
? { height: ['8px', '14px', '4px', '12px', '8px'] }
|
||||
: { height: '8px' }
|
||||
}
|
||||
transition={
|
||||
mode === 'generating'
|
||||
? { duration: 0.6, repeat: Infinity, delay: i * 0.08, ease: 'easeInOut' }
|
||||
: mode === 'playing'
|
||||
? { duration: 1.2, repeat: Infinity, delay: i * 0.15, ease: 'easeInOut' }
|
||||
: { duration: 0.4, ease: 'easeOut' }
|
||||
}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// NEW ALTERNATE HISTORY VIEW - FIXED HEIGHT ROWS WITH INFINITE SCROLL
|
||||
export function HistoryTable() {
|
||||
@@ -97,9 +124,28 @@ export function HistoryTable() {
|
||||
});
|
||||
|
||||
const deleteGeneration = useDeleteGeneration();
|
||||
const clearFailed = useClearFailedGenerations();
|
||||
const [clearFailedDialogOpen, setClearFailedDialogOpen] = useState(false);
|
||||
const exportGeneration = useExportGeneration();
|
||||
const exportGenerationAudio = useExportGenerationAudio();
|
||||
const importGeneration = useImportGeneration();
|
||||
const cancelGeneration = useMutation({
|
||||
mutationFn: (generationId: string) => apiClient.cancelGeneration(generationId),
|
||||
onSuccess: async (data) => {
|
||||
await queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
toast({
|
||||
title: 'Cancelling generation',
|
||||
description: data.message,
|
||||
});
|
||||
},
|
||||
onError: (error) => {
|
||||
toast({
|
||||
title: 'Cancel failed',
|
||||
description: error instanceof Error ? error.message : 'Could not cancel generation',
|
||||
variant: 'destructive',
|
||||
});
|
||||
},
|
||||
});
|
||||
const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
|
||||
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
|
||||
const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio);
|
||||
@@ -126,13 +172,28 @@ export function HistoryTable() {
|
||||
}
|
||||
}, [historyData, page]);
|
||||
|
||||
// Reset to page 0 when deletions or imports occur
|
||||
// Reset to page 0 when deletions, imports, or generation completions occur
|
||||
const pendingCount = useGenerationStore((state) => state.pendingGenerationIds.size);
|
||||
const prevPendingCountRef = useRef(pendingCount);
|
||||
useEffect(() => {
|
||||
if (deleteGeneration.isSuccess || importGeneration.isSuccess) {
|
||||
if (deleteGeneration.isSuccess || importGeneration.isSuccess || clearFailed.isSuccess) {
|
||||
setPage(0);
|
||||
setAllHistory([]);
|
||||
}
|
||||
}, [deleteGeneration.isSuccess, importGeneration.isSuccess]);
|
||||
}, [deleteGeneration.isSuccess, importGeneration.isSuccess, clearFailed.isSuccess]);
|
||||
|
||||
useEffect(() => {
|
||||
// A generation finished (pending count decreased) — scroll back to show it
|
||||
if (
|
||||
prevPendingCountRef.current > 0 &&
|
||||
pendingCount < prevPendingCountRef.current &&
|
||||
page !== 0
|
||||
) {
|
||||
setPage(0);
|
||||
setAllHistory([]);
|
||||
}
|
||||
prevPendingCountRef.current = pendingCount;
|
||||
}, [pendingCount, page]);
|
||||
|
||||
// Intersection Observer for infinite scroll
|
||||
useEffect(() => {
|
||||
@@ -373,6 +434,27 @@ export function HistoryTable() {
|
||||
|
||||
const history = allHistory;
|
||||
const hasMore = allHistory.length < total;
|
||||
const failedCount = history.filter((g) => g.status === 'failed').length;
|
||||
|
||||
const handleClearFailedConfirm = () => {
|
||||
clearFailed.mutate(undefined, {
|
||||
onSuccess: (data) => {
|
||||
setClearFailedDialogOpen(false);
|
||||
toast({
|
||||
title: 'Cleared failed generations',
|
||||
description: `${data.deleted} failed ${data.deleted === 1 ? 'generation' : 'generations'} removed.`,
|
||||
});
|
||||
},
|
||||
onError: (error) => {
|
||||
setClearFailedDialogOpen(false);
|
||||
toast({
|
||||
title: 'Failed to clear',
|
||||
description: error instanceof Error ? error.message : 'Unknown error',
|
||||
variant: 'destructive',
|
||||
});
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0 relative">
|
||||
@@ -382,6 +464,23 @@ export function HistoryTable() {
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{failedCount > 0 && (
|
||||
<div className="flex items-center justify-between px-1 pb-2">
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{failedCount} failed {failedCount === 1 ? 'generation' : 'generations'}
|
||||
</span>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="h-7 text-xs text-muted-foreground hover:text-destructive"
|
||||
onClick={() => setClearFailedDialogOpen(true)}
|
||||
disabled={clearFailed.isPending}
|
||||
>
|
||||
<Trash2 className="h-3 w-3 mr-1.5" />
|
||||
{clearFailed.isPending ? 'Clearing...' : 'Clear failed'}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
{isScrolled && (
|
||||
<div className="absolute top-0 left-0 right-0 h-16 bg-gradient-to-b from-background to-transparent z-10 pointer-events-none" />
|
||||
)}
|
||||
@@ -394,11 +493,14 @@ export function HistoryTable() {
|
||||
>
|
||||
{history.map((gen) => {
|
||||
const isCurrentlyPlaying = currentAudioId === gen.id && isPlaying;
|
||||
const isGenerating = gen.status === 'generating';
|
||||
const isInProgress = gen.status === 'loading_model' || gen.status === 'generating';
|
||||
const isGenerating = isInProgress;
|
||||
const isFailed = gen.status === 'failed';
|
||||
const isPlayable = !isGenerating && !isFailed;
|
||||
const hasVersions = gen.versions && gen.versions.length > 1;
|
||||
const isVersionsExpanded = expandedVersionsId === gen.id;
|
||||
const isCancelling =
|
||||
cancelGeneration.isPending && cancelGeneration.variables === gen.id;
|
||||
return (
|
||||
<div
|
||||
key={gen.id}
|
||||
@@ -412,7 +514,7 @@ export function HistoryTable() {
|
||||
role={isPlayable ? 'button' : undefined}
|
||||
tabIndex={isPlayable ? 0 : undefined}
|
||||
className={cn(
|
||||
'flex items-stretch gap-4 h-26 p-3',
|
||||
'flex items-stretch gap-4 h-26 p-3 outline-none',
|
||||
isPlayable && 'hover:bg-muted/70 cursor-pointer rounded-md',
|
||||
isVersionsExpanded && 'rounded-b-none',
|
||||
)}
|
||||
@@ -445,12 +547,9 @@ export function HistoryTable() {
|
||||
>
|
||||
{/* Status icon */}
|
||||
<div className="flex items-center shrink-0 w-10 justify-center overflow-hidden">
|
||||
<div className="scale-50">
|
||||
<Loader
|
||||
type={isGenerating ? 'line-scale' : 'line-scale-pulse-out-rapid'}
|
||||
active={isGenerating || isCurrentlyPlaying}
|
||||
/>
|
||||
</div>
|
||||
<AudioBars
|
||||
mode={isGenerating ? 'generating' : isCurrentlyPlaying ? 'playing' : 'idle'}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Left side - Meta information */}
|
||||
@@ -472,8 +571,10 @@ export function HistoryTable() {
|
||||
) : null}
|
||||
</div>
|
||||
<div className="text-xs text-muted-foreground">
|
||||
{isGenerating ? (
|
||||
<span className="text-accent">Generating...</span>
|
||||
{isInProgress ? (
|
||||
<span className="text-accent">
|
||||
{gen.status === 'loading_model' ? 'Loading model...' : 'Generating...'}
|
||||
</span>
|
||||
) : (
|
||||
formatDate(gen.created_at)
|
||||
)}
|
||||
@@ -527,69 +628,92 @@ export function HistoryTable() {
|
||||
)}
|
||||
|
||||
{isFailed ? (
|
||||
<>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Retry generation"
|
||||
onClick={() => handleRetry(gen.id)}
|
||||
>
|
||||
<RotateCcw className="h-2 w-2" />
|
||||
</Button>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Delete generation"
|
||||
disabled={deleteGeneration.isPending}
|
||||
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||
>
|
||||
<Trash2 className="h-2 w-2" />
|
||||
</Button>
|
||||
</>
|
||||
) : isGenerating ? (
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Retry generation"
|
||||
onClick={() => handleRetry(gen.id)}
|
||||
aria-label="Cancel generation"
|
||||
disabled={isCancelling}
|
||||
onClick={() => cancelGeneration.mutate(gen.id)}
|
||||
>
|
||||
<RotateCcw className="h-2 w-2" />
|
||||
{isCancelling ? (
|
||||
<Loader2 className="h-2 w-2 animate-spin" />
|
||||
) : (
|
||||
<Square className="h-2 w-2" />
|
||||
)}
|
||||
</Button>
|
||||
) : (
|
||||
<>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Actions"
|
||||
disabled={isGenerating}
|
||||
>
|
||||
<MoreHorizontal className="h-2 w-2" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuItem
|
||||
onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}
|
||||
>
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDownloadAudio(gen.id, gen.text)}
|
||||
disabled={exportGenerationAudio.isPending}
|
||||
>
|
||||
<Download className="mr-2 h-4 w-4" />
|
||||
Export Audio
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleExportPackage(gen.id, gen.text)}
|
||||
disabled={exportGeneration.isPending}
|
||||
>
|
||||
<FileArchive className="mr-2 h-4 w-4" />
|
||||
Export Package
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
|
||||
<Wand2 className="mr-2 h-4 w-4" />
|
||||
Apply Effects
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
|
||||
<RotateCcw className="mr-2 h-4 w-4" />
|
||||
Regenerate
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||
disabled={deleteGeneration.isPending}
|
||||
// className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
</>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6 text-muted-foreground/50 hover:bg-muted-foreground/20 hover:text-muted-foreground"
|
||||
aria-label="Actions"
|
||||
disabled={isGenerating}
|
||||
>
|
||||
<MoreHorizontal className="h-2 w-2" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuItem onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}>
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDownloadAudio(gen.id, gen.text)}
|
||||
disabled={exportGenerationAudio.isPending}
|
||||
>
|
||||
<Download className="mr-2 h-4 w-4" />
|
||||
Export Audio
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleExportPackage(gen.id, gen.text)}
|
||||
disabled={exportGeneration.isPending}
|
||||
>
|
||||
<FileArchive className="mr-2 h-4 w-4" />
|
||||
Export Package
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleApplyEffects(gen.id)}>
|
||||
<Wand2 className="mr-2 h-4 w-4" />
|
||||
Apply Effects
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={() => handleRegenerate(gen.id)}>
|
||||
<RotateCcw className="mr-2 h-4 w-4" />
|
||||
Regenerate
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem
|
||||
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
|
||||
disabled={deleteGeneration.isPending}
|
||||
// className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
@@ -705,6 +829,31 @@ export function HistoryTable() {
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
|
||||
<Dialog open={clearFailedDialogOpen} onOpenChange={setClearFailedDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Clear failed generations</DialogTitle>
|
||||
<DialogDescription>
|
||||
This will permanently delete {failedCount} failed{' '}
|
||||
{failedCount === 1 ? 'generation' : 'generations'} from your history. This cannot be
|
||||
undone.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setClearFailedDialogOpen(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
variant="destructive"
|
||||
onClick={handleClearFailedConfirm}
|
||||
disabled={clearFailed.isPending}
|
||||
>
|
||||
{clearFailed.isPending ? 'Clearing...' : 'Clear all'}
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
|
||||
<Dialog open={importDialogOpen} onOpenChange={setImportDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
|
||||
@@ -243,7 +243,40 @@ export function GpuAcceleration() {
|
||||
|
||||
{/* Native GPU detected - no CUDA download needed */}
|
||||
|
||||
{/* CUDA download section - only show when no GPU is active (native or CUDA) */}
|
||||
{/* Currently running CUDA - show switch back to CPU */}
|
||||
{isCurrentlyCuda && platform.metadata.isTauri && (
|
||||
<>
|
||||
{restartPhase !== 'idle' ? (
|
||||
<div className="flex items-center gap-2 p-3 rounded-lg bg-primary/5 border">
|
||||
<Loader2 className="h-4 w-4 animate-spin" />
|
||||
<span className="text-sm">
|
||||
{restartPhase === 'stopping' && 'Stopping server...'}
|
||||
{restartPhase === 'waiting' && 'Restarting server...'}
|
||||
{restartPhase === 'ready' && 'Server restarted successfully!'}
|
||||
</span>
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
|
||||
re-download later).
|
||||
</p>
|
||||
<Button onClick={handleSwitchToCpu} variant="outline" className="w-full" size="sm">
|
||||
<RotateCw className="h-4 w-4 mr-2" />
|
||||
Switch to CPU Backend
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
{error && (
|
||||
<div className="flex items-center gap-2 text-sm text-destructive">
|
||||
<AlertCircle className="h-4 w-4 shrink-0" />
|
||||
<span>{error}</span>
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* CUDA download/manage section - show when no native GPU and not currently running CUDA */}
|
||||
{!hasNativeGpu && !isCurrentlyCuda && (
|
||||
<>
|
||||
{/* Download progress (manual download or auto-update) */}
|
||||
@@ -315,7 +348,7 @@ export function GpuAcceleration() {
|
||||
)}
|
||||
|
||||
{/* Downloaded but not active - show switch button */}
|
||||
{cudaAvailable && !isCurrentlyCuda && platform.metadata.isTauri && (
|
||||
{cudaAvailable && platform.metadata.isTauri && (
|
||||
<div className="space-y-3">
|
||||
<p className="text-sm text-muted-foreground">
|
||||
CUDA backend is downloaded and ready. Restart the server to enable GPU
|
||||
@@ -328,27 +361,8 @@ export function GpuAcceleration() {
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Currently active - show switch back to CPU */}
|
||||
{isCurrentlyCuda && platform.metadata.isTauri && (
|
||||
<div className="space-y-3">
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Running with CUDA GPU acceleration. Switch back to CPU if needed (you can
|
||||
re-download later).
|
||||
</p>
|
||||
<Button
|
||||
onClick={handleSwitchToCpu}
|
||||
variant="outline"
|
||||
className="w-full"
|
||||
size="sm"
|
||||
>
|
||||
<RotateCw className="h-4 w-4 mr-2" />
|
||||
Switch to CPU Backend
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Delete option when downloaded (and not active) */}
|
||||
{cudaAvailable && !isCurrentlyCuda && (
|
||||
{cudaAvailable && (
|
||||
<Button
|
||||
onClick={handleDelete}
|
||||
variant="ghost"
|
||||
|
||||
@@ -62,6 +62,16 @@ const MODEL_DESCRIPTIONS: Record<string, string> = {
|
||||
'Production-grade open source TTS by Resemble AI. Supports 23 languages with voice cloning and emotion exaggeration control.',
|
||||
'chatterbox-turbo':
|
||||
'Streamlined 350M parameter TTS by Resemble AI. High-quality English speech with less compute and VRAM than larger models.',
|
||||
'tada-1b':
|
||||
'HumeAI TADA 1B — English speech-language model built on Llama 3.2 1B. Generates 700s+ of coherent audio with synchronized text-acoustic alignment.',
|
||||
'tada-3b-ml':
|
||||
'HumeAI TADA 3B Multilingual — built on Llama 3.2 3B. Supports 10 languages with high-fidelity voice cloning via text-acoustic dual alignment.',
|
||||
kokoro:
|
||||
'Kokoro 82M by hexgrad. Tiny 82M-parameter TTS that runs at CPU realtime. Supports 8 languages with pre-built voice styles. Apache 2.0 licensed.',
|
||||
'qwen-custom-voice-1.7B':
|
||||
'Qwen3-TTS CustomVoice 1.7B by Alibaba. 9 premium preset voices with instruct-based style control for tone, emotion, and prosody. Supports 10 languages.',
|
||||
'qwen-custom-voice-0.6B':
|
||||
'Qwen3-TTS CustomVoice 0.6B by Alibaba. Lightweight version with the same 9 preset voices and instruct control. Faster inference for lower-end hardware.',
|
||||
'whisper-base':
|
||||
'Smallest Whisper model (74M parameters). Fast transcription with moderate accuracy.',
|
||||
'whisper-small':
|
||||
@@ -390,8 +400,11 @@ export function ModelManagement() {
|
||||
modelStatus?.models.filter(
|
||||
(m) =>
|
||||
m.model_name.startsWith('qwen-tts') ||
|
||||
m.model_name.startsWith('qwen-custom-voice') ||
|
||||
m.model_name.startsWith('luxtts') ||
|
||||
m.model_name.startsWith('chatterbox'),
|
||||
m.model_name.startsWith('chatterbox') ||
|
||||
m.model_name.startsWith('tada') ||
|
||||
m.model_name.startsWith('kokoro'),
|
||||
) ?? [];
|
||||
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
|
||||
|
||||
@@ -964,7 +977,19 @@ export function ModelManagement() {
|
||||
});
|
||||
try {
|
||||
// Start the migration (background task)
|
||||
await apiClient.migrateModels(newDir);
|
||||
const migrationResult = await apiClient.migrateModels(newDir);
|
||||
|
||||
// If no models to migrate, warn user and skip the change
|
||||
if (migrationResult.moved === 0) {
|
||||
setMigrating(false);
|
||||
setMigrationProgress(null);
|
||||
toast({
|
||||
title: 'No models to migrate',
|
||||
description: 'Download at least one model before changing the storage location.',
|
||||
});
|
||||
setPendingMigrateDir(null);
|
||||
return;
|
||||
}
|
||||
|
||||
// Connect to SSE for progress
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
@@ -1051,105 +1076,3 @@ export function ModelManagement() {
|
||||
);
|
||||
}
|
||||
|
||||
interface ModelItemProps {
|
||||
model: {
|
||||
model_name: string;
|
||||
display_name: string;
|
||||
downloaded: boolean;
|
||||
downloading?: boolean; // From server - true if download in progress
|
||||
size_mb?: number;
|
||||
loaded: boolean;
|
||||
};
|
||||
onDownload: () => void;
|
||||
onDelete: () => void;
|
||||
isDownloading: boolean; // Local state - true if user just clicked download
|
||||
formatSize: (sizeMb?: number) => string;
|
||||
}
|
||||
|
||||
function ModelItem({ model, onDownload, onDelete, isDownloading, formatSize }: ModelItemProps) {
|
||||
// Use server's downloading state OR local state (for immediate feedback before server updates)
|
||||
const showDownloading = model.downloading || isDownloading;
|
||||
|
||||
const statusText = model.loaded
|
||||
? 'Loaded'
|
||||
: showDownloading
|
||||
? 'Downloading'
|
||||
: model.downloaded
|
||||
? 'Downloaded'
|
||||
: 'Not downloaded';
|
||||
const sizeText =
|
||||
model.downloaded && model.size_mb && !showDownloading ? `, ${formatSize(model.size_mb)}` : '';
|
||||
const rowLabel = `${model.display_name}, ${statusText}${sizeText}. Use Tab to reach Download or Delete.`;
|
||||
|
||||
return (
|
||||
<div
|
||||
className="flex items-center justify-between p-3 border rounded-lg"
|
||||
role="group"
|
||||
tabIndex={0}
|
||||
aria-label={rowLabel}
|
||||
>
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="font-medium text-sm">{model.display_name}</span>
|
||||
{model.loaded && (
|
||||
<Badge variant="default" className="text-xs">
|
||||
Loaded
|
||||
</Badge>
|
||||
)}
|
||||
{/* Only show Downloaded if actually downloaded AND not downloading */}
|
||||
{model.downloaded && !model.loaded && !showDownloading && (
|
||||
<Badge variant="secondary" className="text-xs">
|
||||
Downloaded
|
||||
</Badge>
|
||||
)}
|
||||
</div>
|
||||
{model.downloaded && model.size_mb && !showDownloading && (
|
||||
<div className="text-xs text-muted-foreground mt-1">
|
||||
Size: {formatSize(model.size_mb)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{model.downloaded && !showDownloading ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="flex items-center gap-1 text-sm text-muted-foreground">
|
||||
<span>Ready</span>
|
||||
</div>
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={onDelete}
|
||||
variant="outline"
|
||||
disabled={model.loaded}
|
||||
title={model.loaded ? 'Unload model before deleting' : 'Delete model'}
|
||||
aria-label={
|
||||
model.loaded ? 'Unload model before deleting' : `Delete ${model.display_name}`
|
||||
}
|
||||
>
|
||||
<Trash2 className="h-4 w-4" />
|
||||
</Button>
|
||||
</div>
|
||||
) : showDownloading ? (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="outline"
|
||||
disabled
|
||||
aria-label={`${model.display_name} downloading`}
|
||||
>
|
||||
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
|
||||
Downloading...
|
||||
</Button>
|
||||
) : (
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={onDownload}
|
||||
variant="outline"
|
||||
aria-label={`Download ${model.display_name}`}
|
||||
>
|
||||
<Download className="h-4 w-4 mr-2" />
|
||||
Download
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -12,7 +12,11 @@ interface ModelProgressProps {
|
||||
isDownloading?: boolean;
|
||||
}
|
||||
|
||||
export function ModelProgress({ modelName, displayName, isDownloading = false }: ModelProgressProps) {
|
||||
export function ModelProgress({
|
||||
modelName,
|
||||
displayName,
|
||||
isDownloading = false,
|
||||
}: ModelProgressProps) {
|
||||
const [progress, setProgress] = useState<ModelProgressType | null>(null);
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
import { ArrowUpRight } from 'lucide-react';
|
||||
import type { CSSProperties, ReactNode } from 'react';
|
||||
import { useEffect, useState } from 'react';
|
||||
import voiceboxLogo from '@/assets/voicebox-logo.png';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
|
||||
function FadeIn({ delay = 0, children }: { delay?: number; children: ReactNode }) {
|
||||
return (
|
||||
<div
|
||||
className="animate-[fadeInUp_0.5s_ease_both]"
|
||||
style={{ animationDelay: `${delay}ms` } as CSSProperties}
|
||||
>
|
||||
{children}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function AboutPage() {
|
||||
const platform = usePlatform();
|
||||
const [version, setVersion] = useState('');
|
||||
|
||||
useEffect(() => {
|
||||
platform.metadata
|
||||
.getVersion()
|
||||
.then(setVersion)
|
||||
.catch(() => setVersion(''));
|
||||
}, [platform]);
|
||||
|
||||
return (
|
||||
<>
|
||||
<style>{`
|
||||
@keyframes fadeInUp {
|
||||
from {
|
||||
opacity: 0;
|
||||
transform: translateY(8px);
|
||||
}
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
`}</style>
|
||||
<div className="max-w-md mx-auto h-full flex items-center">
|
||||
<div className="flex flex-col items-center text-center space-y-5">
|
||||
<FadeIn delay={0}>
|
||||
<img src={voiceboxLogo} alt="Voicebox" className="w-20 h-20 object-contain" />
|
||||
</FadeIn>
|
||||
|
||||
<FadeIn delay={80}>
|
||||
<div className="space-y-1.5">
|
||||
<h1 className="text-lg font-semibold">Voicebox</h1>
|
||||
<p className="text-xs text-muted-foreground/60 h-4">
|
||||
{version ? `v${version}` : '\u00A0'}
|
||||
</p>
|
||||
</div>
|
||||
</FadeIn>
|
||||
|
||||
<FadeIn delay={160}>
|
||||
<p className="text-sm text-muted-foreground leading-relaxed max-w-sm">
|
||||
The open-source voice synthesis studio. Clone voices, generate speech, apply effects,
|
||||
and build voice-powered apps — all running locally on your machine.
|
||||
</p>
|
||||
</FadeIn>
|
||||
|
||||
<FadeIn delay={240}>
|
||||
<div className="flex items-center gap-1.5 text-sm text-muted-foreground">
|
||||
<span>Created by</span>
|
||||
<a
|
||||
href="https://github.com/jamiepine"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="text-accent hover:underline"
|
||||
>
|
||||
Jamie Pine
|
||||
</a>
|
||||
</div>
|
||||
</FadeIn>
|
||||
|
||||
<FadeIn delay={320}>
|
||||
<div className="flex flex-wrap justify-center gap-3 pt-2">
|
||||
<a
|
||||
href="https://buymeacoffee.com/jamiepine"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="group inline-flex items-center gap-2 rounded-lg border border-border/60 px-4 py-2 text-sm transition-colors hover:bg-muted/50"
|
||||
>
|
||||
<svg
|
||||
className="h-4 w-4 text-[#FFDD00]"
|
||||
viewBox="0 0 24 24"
|
||||
fill="currentColor"
|
||||
aria-hidden="true"
|
||||
>
|
||||
<path d="m20.216 6.415-.132-.666c-.119-.598-.388-1.163-1.001-1.379-.197-.069-.42-.098-.57-.241-.152-.143-.196-.366-.231-.572-.065-.378-.125-.756-.192-1.133-.057-.325-.102-.69-.25-.987-.195-.4-.597-.634-.996-.788a5.723 5.723 0 0 0-.626-.194c-1-.263-2.05-.36-3.077-.416a25.834 25.834 0 0 0-3.7.062c-.915.083-1.88.184-2.75.5-.318.116-.646.256-.888.501-.297.302-.393.77-.177 1.146.154.267.415.456.692.58.36.162.737.284 1.123.366 1.075.238 2.189.331 3.287.37 1.218.05 2.437.01 3.65-.118.299-.033.598-.073.896-.119.352-.054.578-.513.474-.834-.124-.383-.457-.531-.834-.473-.466.074-.96.108-1.382.146-1.177.08-2.358.082-3.536.006a22.228 22.228 0 0 1-1.157-.107c-.086-.01-.18-.025-.258-.036-.243-.036-.484-.08-.724-.13-.111-.027-.111-.185 0-.212h.005c.277-.06.557-.108.838-.147h.002c.131-.009.263-.032.394-.048a25.076 25.076 0 0 1 3.426-.12c.674.019 1.347.067 2.017.144l.228.031c.267.04.533.088.798.145.392.085.895.113 1.07.542.055.137.08.288.111.431l.319 1.484a.237.237 0 0 1-.199.284h-.003c-.037.006-.075.01-.112.015a36.704 36.704 0 0 1-4.743.295 37.059 37.059 0 0 1-4.699-.304c-.14-.017-.293-.042-.417-.06-.326-.048-.649-.108-.973-.161-.393-.065-.768-.032-1.123.161-.29.16-.527.404-.675.701-.154.316-.199.66-.267 1-.069.34-.176.707-.135 1.056.087.753.613 1.365 1.37 1.502a39.69 39.69 0 0 0 11.343.376.483.483 0 0 1 .535.53l-.071.697-1.018 9.907c-.041.41-.047.832-.125 1.237-.122.637-.553 1.028-1.182 1.171-.577.131-1.165.2-1.756.205-.656.004-1.31-.025-1.966-.022-.699.004-1.556-.06-2.095-.58-.475-.458-.54-1.174-.605-1.793l-.731-7.013-.322-3.094c-.037-.351-.286-.695-.678-.678-.336.015-.718.3-.678.679l.228 2.185.949 9.112c.147 1.344 1.174 2.068 2.446 2.272.742.12 1.503.144 2.257.156.966.016 1.942.053 2.892-.122 1.408-.258 2.465-1.198 2.616-2.657.34-3.332.683-6.663 1.024-9.995l.215-2.087a.484.484 0 0 1 .39-.426c.402-.078.787-.212 1.074-.518.455-.488.546-1.124.385-1.766zm-1.478.772c-.145.137-.363.201-.578.233-2.416.359-4.866.54-7.308.46-1.748-.06-3.477-.254-5.207-.498-.17-.024-.353-.055-.47-.18-.22-.236-.111-.71-.054-.995.052-.26.152-.609.463-.646.484-.057 1.046.148 1.526.22.577.088 1.156.159 1.737.212 2.48.226 5.002.19 7.472-.14.45-.06.899-.13 1.345-.21.399-.072.84-.206 1.08.206.166.281.188.657.162.974a.544.544 0 0 1-.169.364zm-6.159 3.9c-.862.37-1.84.788-3.109.788a5.884 5.884 0 0 1-1.569-.217l.877 9.004c.065.78.717 1.38 1.5 1.38 0 0 1.243.065 1.658.065.447 0 1.786-.065 1.786-.065.783 0 1.434-.6 1.499-1.38l.94-9.95a3.996 3.996 0 0 0-1.322-.238c-.826 0-1.491.284-2.26.613z" />
|
||||
</svg>
|
||||
Buy me a coffee
|
||||
<ArrowUpRight className="h-3.5 w-3.5 text-muted-foreground/40 group-hover:text-muted-foreground transition-colors" />
|
||||
</a>
|
||||
<a
|
||||
href="https://github.com/jamiepine/voicebox"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="group inline-flex items-center gap-2 rounded-lg border border-border/60 px-4 py-2 text-sm transition-colors hover:bg-muted/50"
|
||||
>
|
||||
<svg
|
||||
className="h-4 w-4 text-muted-foreground"
|
||||
viewBox="0 0 24 24"
|
||||
fill="currentColor"
|
||||
aria-hidden="true"
|
||||
>
|
||||
<path d="M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z" />
|
||||
</svg>
|
||||
GitHub
|
||||
<ArrowUpRight className="h-3.5 w-3.5 text-muted-foreground/40 group-hover:text-muted-foreground transition-colors" />
|
||||
</a>
|
||||
</div>
|
||||
</FadeIn>
|
||||
|
||||
<FadeIn delay={400}>
|
||||
<p className="text-xs text-muted-foreground/40 pt-4">
|
||||
Licensed under{' '}
|
||||
<a
|
||||
href="https://github.com/jamiepine/voicebox/blob/main/LICENSE"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="hover:text-muted-foreground/60 transition-colors"
|
||||
>
|
||||
MIT
|
||||
</a>
|
||||
</p>
|
||||
</FadeIn>
|
||||
</div>
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,220 @@
|
||||
import changelogRaw from 'virtual:changelog';
|
||||
import { useMemo, useState } from 'react';
|
||||
import { Badge } from '@/components/ui/badge';
|
||||
import { type ChangelogEntry, parseChangelog } from '@/lib/utils/parseChangelog';
|
||||
|
||||
function renderMarkdown(md: string): React.ReactNode[] {
|
||||
const lines = md.split('\n');
|
||||
const elements: React.ReactNode[] = [];
|
||||
let i = 0;
|
||||
|
||||
while (i < lines.length) {
|
||||
const line = lines[i];
|
||||
|
||||
// Skip empty lines
|
||||
if (line.trim() === '') {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Tables — collect all lines starting with |
|
||||
if (line.trim().startsWith('|')) {
|
||||
const tableLines: string[] = [];
|
||||
while (i < lines.length && lines[i].trim().startsWith('|')) {
|
||||
tableLines.push(lines[i]);
|
||||
i++;
|
||||
}
|
||||
elements.push(renderTable(tableLines, elements.length));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Headings
|
||||
if (line.startsWith('#### ')) {
|
||||
elements.push(
|
||||
<h5 key={elements.length} className="text-sm font-medium mt-5 mb-1">
|
||||
{inlineMarkdown(line.slice(5))}
|
||||
</h5>,
|
||||
);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith('### ')) {
|
||||
elements.push(
|
||||
<h4 key={elements.length} className="text-sm font-medium mt-6 mb-2">
|
||||
{inlineMarkdown(line.slice(4))}
|
||||
</h4>,
|
||||
);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// List items — collect consecutive
|
||||
if (line.startsWith('- ')) {
|
||||
const items: string[] = [];
|
||||
while (i < lines.length && lines[i].startsWith('- ')) {
|
||||
items.push(lines[i].slice(2));
|
||||
i++;
|
||||
}
|
||||
elements.push(
|
||||
<ul key={elements.length} className="space-y-1 my-2">
|
||||
{items.map((item, idx) => (
|
||||
<li key={idx} className="text-sm text-muted-foreground flex gap-2">
|
||||
<span className="text-muted-foreground/50 select-none shrink-0">•</span>
|
||||
<span>{inlineMarkdown(item)}</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Paragraph
|
||||
elements.push(
|
||||
<p key={elements.length} className="text-sm text-muted-foreground my-2">
|
||||
{inlineMarkdown(line)}
|
||||
</p>,
|
||||
);
|
||||
i++;
|
||||
}
|
||||
|
||||
return elements;
|
||||
}
|
||||
|
||||
function renderTable(tableLines: string[], keyBase: number): React.ReactNode {
|
||||
const parseRow = (line: string) =>
|
||||
line
|
||||
.split('|')
|
||||
.slice(1, -1)
|
||||
.map((c) => c.trim());
|
||||
|
||||
const headers = parseRow(tableLines[0]);
|
||||
// Skip separator line (index 1)
|
||||
const rows = tableLines.slice(2).map(parseRow);
|
||||
|
||||
return (
|
||||
<div key={keyBase} className="overflow-x-auto my-3">
|
||||
<table className="text-sm w-full">
|
||||
<thead>
|
||||
<tr className="border-b">
|
||||
{headers.map((h, hIdx) => (
|
||||
<th
|
||||
key={hIdx}
|
||||
className="text-left py-1.5 pr-4 text-muted-foreground font-medium text-xs"
|
||||
>
|
||||
{inlineMarkdown(h)}
|
||||
</th>
|
||||
))}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{rows.map((row, rowIdx) => (
|
||||
<tr key={rowIdx} className="border-b border-border/50">
|
||||
{row.map((cell, cellIdx) => (
|
||||
<td key={cellIdx} className="py-1.5 pr-4 text-muted-foreground">
|
||||
{inlineMarkdown(cell)}
|
||||
</td>
|
||||
))}
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function inlineMarkdown(text: string): React.ReactNode {
|
||||
// Process inline markdown: bold, code, links
|
||||
const parts: React.ReactNode[] = [];
|
||||
// Regex matches: **bold**, `code`, [text](url)
|
||||
const inlineRe = /\*\*(.+?)\*\*|`([^`]+)`|\[([^\]]+)\]\(([^)]+)\)/g;
|
||||
let lastIndex = 0;
|
||||
let match: RegExpExecArray | null = inlineRe.exec(text);
|
||||
|
||||
while (match !== null) {
|
||||
if (match.index > lastIndex) {
|
||||
parts.push(text.slice(lastIndex, match.index));
|
||||
}
|
||||
|
||||
if (match[1] !== undefined) {
|
||||
// Bold
|
||||
parts.push(
|
||||
<strong key={parts.length} className="font-medium text-foreground">
|
||||
{match[1]}
|
||||
</strong>,
|
||||
);
|
||||
} else if (match[2] !== undefined) {
|
||||
// Code
|
||||
parts.push(
|
||||
<code key={parts.length} className="px-1 py-0.5 rounded bg-muted text-xs font-mono">
|
||||
{match[2]}
|
||||
</code>,
|
||||
);
|
||||
} else if (match[3] !== undefined && match[4] !== undefined) {
|
||||
// Link
|
||||
parts.push(
|
||||
<a
|
||||
key={parts.length}
|
||||
href={match[4]}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="text-accent hover:underline"
|
||||
>
|
||||
{match[3]}
|
||||
</a>,
|
||||
);
|
||||
}
|
||||
|
||||
lastIndex = match.index + match[0].length;
|
||||
match = inlineRe.exec(text);
|
||||
}
|
||||
|
||||
if (lastIndex < text.length) {
|
||||
parts.push(text.slice(lastIndex));
|
||||
}
|
||||
|
||||
return parts.length === 1 ? parts[0] : parts;
|
||||
}
|
||||
|
||||
function ChangelogEntryCard({ entry }: { entry: ChangelogEntry }) {
|
||||
const [expanded, setExpanded] = useState(false);
|
||||
const content = useMemo(() => renderMarkdown(entry.body), [entry.body]);
|
||||
const isLong = entry.body.split('\n').length > 12;
|
||||
|
||||
return (
|
||||
<div className="border-b border-border/50 pb-6">
|
||||
<div className="flex items-baseline gap-3 mb-3">
|
||||
<h3 className="text-xl font-semibold tracking-tight">{entry.version}</h3>
|
||||
{entry.date && <span className="text-xs text-muted-foreground">{entry.date}</span>}
|
||||
{entry.version === 'Unreleased' && <Badge variant="outline">dev</Badge>}
|
||||
</div>
|
||||
|
||||
<div className={isLong && !expanded ? 'max-h-48 overflow-hidden relative' : ''}>
|
||||
{content}
|
||||
{isLong && !expanded && (
|
||||
<div className="absolute bottom-0 left-0 right-0 h-16 bg-gradient-to-t from-background to-transparent" />
|
||||
)}
|
||||
</div>
|
||||
|
||||
{isLong && (
|
||||
<button
|
||||
onClick={() => setExpanded(!expanded)}
|
||||
className="text-xs text-accent hover:underline mt-2"
|
||||
>
|
||||
{expanded ? 'Show less' : 'Show more'}
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function ChangelogPage() {
|
||||
const entries = useMemo(() => parseChangelog(changelogRaw), []);
|
||||
|
||||
return (
|
||||
<div className="space-y-6 max-w-2xl">
|
||||
{entries.map((entry) => (
|
||||
<ChangelogEntryCard key={entry.version} entry={entry} />
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,379 @@
|
||||
import { zodResolver } from '@hookform/resolvers/zod';
|
||||
import { AlertCircle, ArrowUpRight, Book, Download, Loader2, RefreshCw } from 'lucide-react';
|
||||
import { useEffect, useState } from 'react';
|
||||
import { useForm } from 'react-hook-form';
|
||||
import * as z from 'zod';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Form, FormControl, FormField, FormItem, FormMessage } from '@/components/ui/form';
|
||||
import { Input } from '@/components/ui/input';
|
||||
import { Progress } from '@/components/ui/progress';
|
||||
import { Toggle } from '@/components/ui/toggle';
|
||||
import { useToast } from '@/components/ui/use-toast';
|
||||
import { useAutoUpdater } from '@/hooks/useAutoUpdater';
|
||||
import { useServerHealth } from '@/lib/hooks/useServer';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { SettingRow, SettingSection } from './SettingRow';
|
||||
|
||||
const connectionSchema = z.object({
|
||||
serverUrl: z.string().url('Please enter a valid URL'),
|
||||
});
|
||||
|
||||
type ConnectionFormValues = z.infer<typeof connectionSchema>;
|
||||
|
||||
export function GeneralPage() {
|
||||
const platform = usePlatform();
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
const setServerUrl = useServerStore((state) => state.setServerUrl);
|
||||
const keepServerRunningOnClose = useServerStore((state) => state.keepServerRunningOnClose);
|
||||
const setKeepServerRunningOnClose = useServerStore((state) => state.setKeepServerRunningOnClose);
|
||||
const mode = useServerStore((state) => state.mode);
|
||||
const setMode = useServerStore((state) => state.setMode);
|
||||
const { toast } = useToast();
|
||||
const { data: health, isLoading, error: healthError } = useServerHealth();
|
||||
|
||||
const form = useForm<ConnectionFormValues>({
|
||||
resolver: zodResolver(connectionSchema),
|
||||
defaultValues: { serverUrl },
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
form.reset({ serverUrl });
|
||||
}, [serverUrl, form]);
|
||||
|
||||
const { isDirty } = form.formState;
|
||||
|
||||
function onSubmit(data: ConnectionFormValues) {
|
||||
setServerUrl(data.serverUrl);
|
||||
form.reset(data);
|
||||
toast({
|
||||
title: 'Server URL updated',
|
||||
description: `Connected to ${data.serverUrl}`,
|
||||
});
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-8 max-w-2xl">
|
||||
<div className="grid grid-cols-2 gap-3">
|
||||
<a
|
||||
href="https://docs.voicebox.sh"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="group flex items-center gap-3 rounded-lg border border-border/60 p-4 transition-colors hover:bg-muted/50"
|
||||
>
|
||||
<Book className="h-5 w-5 shrink-0 text-accent" strokeWidth={2.5} />
|
||||
<div className="min-w-0 flex-1">
|
||||
<div className="text-sm font-medium">Read the Docs</div>
|
||||
<div className="text-xs text-muted-foreground">docs.voicebox.sh</div>
|
||||
</div>
|
||||
<ArrowUpRight className="h-4 w-4 text-muted-foreground/40 group-hover:text-muted-foreground transition-colors" />
|
||||
</a>
|
||||
<a
|
||||
href="https://discord.gg/StkzQasqPS"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="group flex items-center gap-3 rounded-lg border border-border/60 p-4 transition-colors hover:bg-muted/50"
|
||||
>
|
||||
<svg
|
||||
className="h-5 w-5 shrink-0 text-accent"
|
||||
viewBox="0 0 24 24"
|
||||
fill="currentColor"
|
||||
aria-hidden="true"
|
||||
>
|
||||
<path d="M20.317 4.37a19.791 19.791 0 0 0-4.885-1.515.074.074 0 0 0-.079.037c-.21.375-.444.864-.608 1.25a18.27 18.27 0 0 0-5.487 0 12.64 12.64 0 0 0-.617-1.25.077.077 0 0 0-.079-.037A19.736 19.736 0 0 0 3.677 4.37a.07.07 0 0 0-.032.027C.533 9.046-.32 13.58.099 18.057a.082.082 0 0 0 .031.057 19.9 19.9 0 0 0 5.993 3.03.078.078 0 0 0 .084-.028c.462-.63.874-1.295 1.226-1.994a.076.076 0 0 0-.041-.106 13.107 13.107 0 0 1-1.872-.892.077.077 0 0 1-.008-.128 10.2 10.2 0 0 0 .372-.292.074.074 0 0 1 .077-.01c3.928 1.793 8.18 1.793 12.062 0a.074.074 0 0 1 .078.01c.12.098.246.198.373.292a.077.077 0 0 1-.006.127 12.299 12.299 0 0 1-1.873.892.077.077 0 0 0-.041.107c.36.698.772 1.362 1.225 1.993a.076.076 0 0 0 .084.028 19.839 19.839 0 0 0 6.002-3.03.077.077 0 0 0 .032-.054c.5-5.177-.838-9.674-3.549-13.66a.061.061 0 0 0-.031-.03zM8.02 15.33c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.956-2.419 2.157-2.419 1.21 0 2.176 1.095 2.157 2.42 0 1.333-.956 2.418-2.157 2.418zm7.975 0c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.955-2.419 2.157-2.419 1.21 0 2.176 1.095 2.157 2.42 0 1.333-.946 2.418-2.157 2.418z" />
|
||||
</svg>
|
||||
<div className="min-w-0 flex-1">
|
||||
<div className="text-sm font-medium">Join the Discord</div>
|
||||
<div className="text-xs text-muted-foreground">Get help & share voices</div>
|
||||
</div>
|
||||
<ArrowUpRight className="h-4 w-4 text-muted-foreground/40 group-hover:text-muted-foreground transition-colors" />
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<SettingSection>
|
||||
<SettingRow
|
||||
title="Server URL"
|
||||
description="The address of your voicebox backend server."
|
||||
action={
|
||||
<ConnectionStatus health={health} isLoading={isLoading} healthError={healthError} />
|
||||
}
|
||||
>
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)} className="flex gap-2">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="serverUrl"
|
||||
render={({ field }) => (
|
||||
<FormItem className="flex-1">
|
||||
<FormControl>
|
||||
<Input placeholder="http://127.0.0.1:17493" {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
{isDirty && (
|
||||
<Button type="submit" size="sm">
|
||||
Save
|
||||
</Button>
|
||||
)}
|
||||
</form>
|
||||
</Form>
|
||||
</SettingRow>
|
||||
|
||||
<SettingRow
|
||||
title="Keep server running when app closes"
|
||||
description="The server will continue running in the background after closing the app."
|
||||
htmlFor="keepServerRunning"
|
||||
action={
|
||||
<Toggle
|
||||
id="keepServerRunning"
|
||||
checked={keepServerRunningOnClose}
|
||||
onCheckedChange={(checked: boolean) => {
|
||||
setKeepServerRunningOnClose(checked);
|
||||
platform.lifecycle.setKeepServerRunning(checked).catch((error) => {
|
||||
console.error('Failed to sync setting to Rust:', error);
|
||||
setKeepServerRunningOnClose(!checked);
|
||||
toast({
|
||||
title: 'Failed to update setting',
|
||||
description: 'Could not sync setting to backend.',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
});
|
||||
toast({
|
||||
title: 'Setting updated',
|
||||
description: checked
|
||||
? 'Server will continue running when app closes'
|
||||
: 'Server will stop when app closes',
|
||||
});
|
||||
}}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
{platform.metadata.isTauri && (
|
||||
<SettingRow
|
||||
title="Allow network access"
|
||||
description="Makes the server accessible from other devices on your network. Restart the app after changing."
|
||||
htmlFor="allowNetworkAccess"
|
||||
action={
|
||||
<Toggle
|
||||
id="allowNetworkAccess"
|
||||
checked={mode === 'remote'}
|
||||
onCheckedChange={(checked: boolean) => {
|
||||
setMode(checked ? 'remote' : 'local');
|
||||
toast({
|
||||
title: 'Setting updated',
|
||||
description: checked
|
||||
? 'Network access enabled. Restart the app to apply.'
|
||||
: 'Network access disabled. Restart the app to apply.',
|
||||
});
|
||||
}}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</SettingSection>
|
||||
|
||||
<ApiReferenceCard serverUrl={serverUrl} />
|
||||
|
||||
{platform.metadata.isTauri && <UpdatesSection />}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ConnectionStatus({
|
||||
health,
|
||||
isLoading,
|
||||
healthError,
|
||||
}: {
|
||||
health: ReturnType<typeof useServerHealth>['data'];
|
||||
isLoading: boolean;
|
||||
healthError: ReturnType<typeof useServerHealth>['error'];
|
||||
}) {
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 rounded-full border border-border/60 px-3 py-1">
|
||||
<Loader2 className="h-3 w-3 animate-spin text-muted-foreground" />
|
||||
<span className="text-xs text-muted-foreground">Connecting</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (healthError) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 rounded-full border border-destructive/30 px-3 py-1">
|
||||
<span className="relative flex h-2 w-2">
|
||||
<span className="absolute inline-flex h-full w-full rounded-full bg-destructive/40" />
|
||||
<span className="relative inline-flex h-2 w-2 rounded-full bg-destructive" />
|
||||
</span>
|
||||
<span className="text-xs text-destructive">Offline</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (health) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 rounded-full border border-accent/30 px-3 py-1">
|
||||
<span className="relative flex h-2 w-2">
|
||||
<span className="absolute inline-flex h-full w-full animate-ping rounded-full bg-accent/60" />
|
||||
<span className="relative inline-flex h-2 w-2 rounded-full bg-accent shadow-[0_0_6px_1px_hsl(var(--accent)/0.5)]" />
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground">Online</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function UpdatesSection() {
|
||||
const platform = usePlatform();
|
||||
const { status, checkForUpdates, downloadAndInstall, restartAndInstall } = useAutoUpdater(false);
|
||||
const [currentVersion, setCurrentVersion] = useState<string>('');
|
||||
const isDev = !import.meta.env?.PROD;
|
||||
|
||||
useEffect(() => {
|
||||
platform.metadata
|
||||
.getVersion()
|
||||
.then(setCurrentVersion)
|
||||
.catch(() => setCurrentVersion('Unknown'));
|
||||
}, [platform]);
|
||||
|
||||
return (
|
||||
<SettingSection title="App Updates" description={`v${currentVersion}${isDev ? ' (dev)' : ''}`}>
|
||||
{isDev ? (
|
||||
<SettingRow
|
||||
title="Development mode"
|
||||
description="Auto-updates are disabled in development mode."
|
||||
/>
|
||||
) : (
|
||||
<>
|
||||
<SettingRow
|
||||
title="Check for updates"
|
||||
description={
|
||||
status.available
|
||||
? `Version ${status.version} available`
|
||||
: status.checking
|
||||
? 'Checking...'
|
||||
: "You're up to date"
|
||||
}
|
||||
action={
|
||||
<Button
|
||||
onClick={checkForUpdates}
|
||||
disabled={status.checking || status.downloading || status.readyToInstall}
|
||||
variant="outline"
|
||||
size="sm"
|
||||
>
|
||||
<RefreshCw
|
||||
className={`h-3.5 w-3.5 mr-1.5 ${status.checking ? 'animate-spin' : ''}`}
|
||||
/>
|
||||
Check
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
|
||||
{status.error && (
|
||||
<SettingRow title="Update error">
|
||||
<div className="flex items-center gap-2 text-sm text-destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
{status.error}
|
||||
</div>
|
||||
</SettingRow>
|
||||
)}
|
||||
|
||||
{status.available && !status.downloading && !status.readyToInstall && (
|
||||
<SettingRow
|
||||
title={`Update to ${status.version}`}
|
||||
description="Download and install the latest version."
|
||||
action={
|
||||
<Button onClick={downloadAndInstall} size="sm">
|
||||
<Download className="h-3.5 w-3.5 mr-1.5" />
|
||||
Download
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{status.downloading && (
|
||||
<SettingRow title="Downloading update...">
|
||||
<div className="space-y-1.5">
|
||||
<Progress value={status.downloadProgress} />
|
||||
<div className="flex items-center justify-between text-xs text-muted-foreground">
|
||||
{status.downloadedBytes !== undefined &&
|
||||
status.totalBytes !== undefined &&
|
||||
status.totalBytes > 0 ? (
|
||||
<span>
|
||||
{(status.downloadedBytes / 1024 / 1024).toFixed(1)} MB /{' '}
|
||||
{(status.totalBytes / 1024 / 1024).toFixed(1)} MB
|
||||
</span>
|
||||
) : (
|
||||
<span />
|
||||
)}
|
||||
{status.downloadProgress !== undefined && <span>{status.downloadProgress}%</span>}
|
||||
</div>
|
||||
</div>
|
||||
</SettingRow>
|
||||
)}
|
||||
|
||||
{status.readyToInstall && (
|
||||
<SettingRow
|
||||
title="Update ready to install"
|
||||
description={`Version ${status.version} has been downloaded. Restart to complete.`}
|
||||
action={
|
||||
<Button onClick={restartAndInstall} size="sm">
|
||||
<RefreshCw className="h-3.5 w-3.5 mr-1.5" />
|
||||
Restart Now
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</SettingSection>
|
||||
);
|
||||
}
|
||||
|
||||
const API_ENDPOINTS = [
|
||||
{ method: 'POST', path: '/generate', label: 'Generate speech' },
|
||||
{ method: 'GET', path: '/health', label: 'Server status' },
|
||||
{ method: 'GET', path: '/profiles', label: 'List voices' },
|
||||
{ method: 'GET', path: '/history', label: 'Past generations' },
|
||||
];
|
||||
|
||||
function ApiReferenceCard({ serverUrl }: { serverUrl: string }) {
|
||||
return (
|
||||
<div className="rounded-lg border border-border/60 p-4 space-y-3">
|
||||
<div>
|
||||
<h3 className="text-sm font-medium">API Access</h3>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Integrate Voicebox into your workflow via the REST API at{' '}
|
||||
<code className="text-xs bg-muted px-1 py-0.5 rounded font-mono">{serverUrl}</code>
|
||||
</p>
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
{API_ENDPOINTS.map((ep) => (
|
||||
<div key={ep.path} className="flex items-center gap-2.5 py-1">
|
||||
<span
|
||||
className={`text-[10px] font-mono font-semibold w-9 text-center rounded px-1 py-px ${
|
||||
ep.method === 'POST' ? 'bg-accent/10 text-accent' : 'bg-muted text-muted-foreground'
|
||||
}`}
|
||||
>
|
||||
{ep.method}
|
||||
</span>
|
||||
<code className="text-xs font-mono text-muted-foreground">{ep.path}</code>
|
||||
<span className="text-xs text-muted-foreground/50 ml-auto">{ep.label}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
<a
|
||||
href={`${serverUrl}/docs`}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="text-accent hover:underline"
|
||||
>
|
||||
View the full API reference
|
||||
</a>
|
||||
</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
import { FolderOpen } from 'lucide-react';
|
||||
import { useCallback, useEffect, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Slider } from '@/components/ui/slider';
|
||||
import { Toggle } from '@/components/ui/toggle';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { SettingRow, SettingSection } from './SettingRow';
|
||||
|
||||
export function GenerationPage() {
|
||||
const platform = usePlatform();
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
const maxChunkChars = useServerStore((state) => state.maxChunkChars);
|
||||
const setMaxChunkChars = useServerStore((state) => state.setMaxChunkChars);
|
||||
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
|
||||
const setCrossfadeMs = useServerStore((state) => state.setCrossfadeMs);
|
||||
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
|
||||
const setNormalizeAudio = useServerStore((state) => state.setNormalizeAudio);
|
||||
const autoplayOnGenerate = useServerStore((state) => state.autoplayOnGenerate);
|
||||
const setAutoplayOnGenerate = useServerStore((state) => state.setAutoplayOnGenerate);
|
||||
const [opening, setOpening] = useState(false);
|
||||
const [generationsPath, setGenerationsPath] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
fetch(`${serverUrl}/health/filesystem`)
|
||||
.then((res) => res.json())
|
||||
.then((data) => {
|
||||
const genDir = data.directories?.find((d: { path: string }) =>
|
||||
d.path.includes('generations'),
|
||||
);
|
||||
if (genDir?.path) setGenerationsPath(genDir.path);
|
||||
})
|
||||
.catch(() => {});
|
||||
}, [serverUrl]);
|
||||
|
||||
const openGenerationsFolder = useCallback(async () => {
|
||||
if (!generationsPath) return;
|
||||
setOpening(true);
|
||||
try {
|
||||
await platform.filesystem.openPath(generationsPath);
|
||||
} catch (e) {
|
||||
console.error('Failed to open generations folder:', e);
|
||||
} finally {
|
||||
setOpening(false);
|
||||
}
|
||||
}, [platform, generationsPath]);
|
||||
|
||||
return (
|
||||
<div className="space-y-8 max-w-2xl">
|
||||
<SettingSection
|
||||
title="Generation"
|
||||
description="Controls for long text generation. These settings apply to all engines."
|
||||
>
|
||||
<SettingRow
|
||||
title="Auto-chunking limit"
|
||||
description="Long text is split into chunks at sentence boundaries. Lower values can improve quality for long outputs."
|
||||
action={
|
||||
<span className="text-sm tabular-nums text-muted-foreground">
|
||||
{maxChunkChars} chars
|
||||
</span>
|
||||
}
|
||||
>
|
||||
<Slider
|
||||
id="maxChunkChars"
|
||||
value={[maxChunkChars]}
|
||||
onValueChange={([value]) => setMaxChunkChars(value)}
|
||||
min={100}
|
||||
max={5000}
|
||||
step={50}
|
||||
aria-label="Auto-chunking character limit"
|
||||
/>
|
||||
</SettingRow>
|
||||
|
||||
<SettingRow
|
||||
title="Chunk crossfade"
|
||||
description="Blends audio between chunks to smooth transitions. Set to 0 for a hard cut."
|
||||
action={
|
||||
<span className="text-sm tabular-nums text-muted-foreground">
|
||||
{crossfadeMs === 0 ? 'Cut' : `${crossfadeMs}ms`}
|
||||
</span>
|
||||
}
|
||||
>
|
||||
<Slider
|
||||
id="crossfadeMs"
|
||||
value={[crossfadeMs]}
|
||||
onValueChange={([value]) => setCrossfadeMs(value)}
|
||||
min={0}
|
||||
max={200}
|
||||
step={10}
|
||||
aria-label="Chunk crossfade duration"
|
||||
/>
|
||||
</SettingRow>
|
||||
|
||||
<SettingRow
|
||||
title="Normalize audio"
|
||||
description="Adjusts output volume to a consistent level across generations."
|
||||
htmlFor="normalizeAudio"
|
||||
action={
|
||||
<Toggle
|
||||
id="normalizeAudio"
|
||||
checked={normalizeAudio}
|
||||
onCheckedChange={setNormalizeAudio}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
<SettingRow
|
||||
title="Autoplay on generate"
|
||||
description="Automatically play audio when a generation completes."
|
||||
htmlFor="autoplayOnGenerate"
|
||||
action={
|
||||
<Toggle
|
||||
id="autoplayOnGenerate"
|
||||
checked={autoplayOnGenerate}
|
||||
onCheckedChange={setAutoplayOnGenerate}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
<SettingRow
|
||||
title="Generations folder"
|
||||
description={generationsPath ?? 'Where generated audio files are stored on disk.'}
|
||||
action={
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
onClick={openGenerationsFolder}
|
||||
disabled={opening || !generationsPath}
|
||||
>
|
||||
<FolderOpen className="h-3.5 w-3.5 mr-1.5" />
|
||||
Open
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
</SettingSection>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,405 @@
|
||||
import { useQuery, useQueryClient } from '@tanstack/react-query';
|
||||
import { AlertCircle, Cpu, Download, Loader2, RotateCw, Trash2 } from 'lucide-react';
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Progress } from '@/components/ui/progress';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { CudaDownloadProgress, HealthResponse } from '@/lib/api/types';
|
||||
import { useServerHealth } from '@/lib/hooks/useServer';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { SettingRow, SettingSection } from './SettingRow';
|
||||
|
||||
type RestartPhase = 'idle' | 'stopping' | 'waiting' | 'ready';
|
||||
|
||||
function AppleLogo({ className }: { className?: string }) {
|
||||
return (
|
||||
<svg className={className} viewBox="0 0 24 24" fill="currentColor" aria-hidden="true">
|
||||
<path d="M18.71 19.5c-.83 1.24-1.71 2.45-3.05 2.47-1.34.03-1.77-.79-3.29-.79-1.53 0-2 .77-3.27.82-1.31.05-2.3-1.32-3.14-2.53C4.25 17 2.94 12.45 4.7 9.39c.87-1.52 2.43-2.48 4.12-2.51 1.28-.02 2.5.87 3.29.87.78 0 2.26-1.07 3.8-.91.65.03 2.47.26 3.64 1.98-.09.06-2.17 1.28-2.15 3.81.03 3.02 2.65 4.03 2.68 4.04-.03.07-.42 1.44-1.38 2.83M13 3.5c.73-.83 1.94-1.46 2.94-1.5.13 1.17-.34 2.35-1.04 3.19-.69.85-1.83 1.51-2.95 1.42-.15-1.15.41-2.35 1.05-3.11z" />
|
||||
</svg>
|
||||
);
|
||||
}
|
||||
|
||||
function GpuIcon({ className }: { className?: string }) {
|
||||
return (
|
||||
<svg
|
||||
className={className}
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
strokeWidth="1.5"
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
aria-hidden="true"
|
||||
>
|
||||
<rect x="4" y="6" width="16" height="12" rx="2" />
|
||||
<path d="M2 10h2M2 14h2M20 10h2M20 14h2" />
|
||||
<path d="M9 10h6M9 14h4" />
|
||||
</svg>
|
||||
);
|
||||
}
|
||||
|
||||
function GpuInfoCard({ health }: { health: HealthResponse }) {
|
||||
const hasGpu = health.gpu_available && health.gpu_type;
|
||||
|
||||
// Parse GPU name from type string like "CUDA (NVIDIA RTX 4090)" or "MPS (Apple M2 Pro)"
|
||||
const gpuName = hasGpu
|
||||
? health.gpu_type!.replace(/^(CUDA|ROCm|MPS|Metal|XPU|DirectML)\s*\((.+)\)$/, '$2') ||
|
||||
health.gpu_type!
|
||||
: null;
|
||||
const gpuBackend = hasGpu ? health.gpu_type!.replace(/\s*\(.+\)$/, '') : null;
|
||||
const isApple = gpuBackend === 'MPS' || gpuBackend === 'Metal';
|
||||
const showBackendVariant = health.backend_variant && health.backend_variant !== 'cpu';
|
||||
|
||||
return (
|
||||
<div className="rounded-lg border border-border/60 p-4">
|
||||
<div className="flex items-center gap-3">
|
||||
{hasGpu ? (
|
||||
isApple ? (
|
||||
<AppleLogo className="h-5 w-5 shrink-0 text-muted-foreground" />
|
||||
) : (
|
||||
<GpuIcon className="h-5 w-5 shrink-0 text-accent" />
|
||||
)
|
||||
) : (
|
||||
<Cpu className="h-5 w-5 shrink-0 text-muted-foreground" />
|
||||
)}
|
||||
<div className="flex-1 min-w-0 space-y-0.5">
|
||||
<div className="text-sm font-medium">{hasGpu ? gpuName : 'CPU Only'}</div>
|
||||
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted-foreground">
|
||||
{hasGpu ? (
|
||||
<>
|
||||
<span>{gpuBackend}</span>
|
||||
{showBackendVariant && (
|
||||
<>
|
||||
<span className="text-border">|</span>
|
||||
<span className="uppercase">{health.backend_variant}</span>
|
||||
</>
|
||||
)}
|
||||
{health.vram_used_mb != null && health.vram_used_mb > 0 && (
|
||||
<>
|
||||
<span className="text-border">|</span>
|
||||
<span>{health.vram_used_mb.toFixed(0)} MB VRAM</span>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
) : (
|
||||
<span>No GPU acceleration detected</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
{hasGpu && (
|
||||
<div className="flex items-center gap-2 rounded-full border border-accent/30 px-2.5 py-0.5">
|
||||
<span className="relative flex h-1.5 w-1.5">
|
||||
<span className="absolute inline-flex h-full w-full animate-ping rounded-full bg-accent/60" />
|
||||
<span className="relative inline-flex h-1.5 w-1.5 rounded-full bg-accent shadow-[0_0_4px_1px_hsl(var(--accent)/0.4)]" />
|
||||
</span>
|
||||
<span className="text-[10px] font-medium text-muted-foreground">Active</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function GpuPage() {
|
||||
const platform = usePlatform();
|
||||
const queryClient = useQueryClient();
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
const { data: health } = useServerHealth();
|
||||
|
||||
const [restartPhase, setRestartPhase] = useState<RestartPhase>('idle');
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [downloadProgress, setDownloadProgress] = useState<CudaDownloadProgress | null>(null);
|
||||
const healthPollRef = useRef<ReturnType<typeof setInterval> | null>(null);
|
||||
|
||||
const {
|
||||
data: cudaStatus,
|
||||
isLoading: _cudaStatusLoading,
|
||||
refetch: refetchCudaStatus,
|
||||
} = useQuery({
|
||||
queryKey: ['cuda-status', serverUrl],
|
||||
queryFn: () => apiClient.getCudaStatus(),
|
||||
refetchInterval: (query) => (query.state.status === 'pending' ? false : 10000),
|
||||
retry: 1,
|
||||
enabled: !!health,
|
||||
});
|
||||
|
||||
const isCurrentlyCuda = health?.backend_variant === 'cuda';
|
||||
const cudaAvailable = cudaStatus?.available ?? false;
|
||||
const cudaDownloading = cudaStatus?.downloading ?? false;
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (healthPollRef.current) {
|
||||
clearInterval(healthPollRef.current);
|
||||
healthPollRef.current = null;
|
||||
}
|
||||
};
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (!cudaDownloading || !serverUrl) return;
|
||||
|
||||
const eventSource = new EventSource(`${serverUrl}/backend/cuda-progress`);
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
try {
|
||||
const data = JSON.parse(event.data) as CudaDownloadProgress;
|
||||
setDownloadProgress(data);
|
||||
|
||||
if (data.status === 'complete') {
|
||||
eventSource.close();
|
||||
setDownloadProgress(null);
|
||||
refetchCudaStatus();
|
||||
} else if (data.status === 'error') {
|
||||
eventSource.close();
|
||||
setError(data.error || 'Download failed');
|
||||
setDownloadProgress(null);
|
||||
refetchCudaStatus();
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Error parsing CUDA progress event:', e);
|
||||
}
|
||||
};
|
||||
|
||||
eventSource.onerror = () => {
|
||||
eventSource.close();
|
||||
};
|
||||
|
||||
return () => {
|
||||
eventSource.close();
|
||||
};
|
||||
}, [cudaDownloading, serverUrl, refetchCudaStatus]);
|
||||
|
||||
const clearHealthPolling = useCallback(() => {
|
||||
if (healthPollRef.current) {
|
||||
clearInterval(healthPollRef.current);
|
||||
healthPollRef.current = null;
|
||||
}
|
||||
}, []);
|
||||
|
||||
const startHealthPolling = useCallback(() => {
|
||||
clearHealthPolling();
|
||||
|
||||
healthPollRef.current = setInterval(async () => {
|
||||
try {
|
||||
const result = await apiClient.getHealth();
|
||||
if (result.status === 'healthy') {
|
||||
clearHealthPolling();
|
||||
setRestartPhase('ready');
|
||||
queryClient.invalidateQueries();
|
||||
setTimeout(() => setRestartPhase('idle'), 2000);
|
||||
}
|
||||
} catch {
|
||||
// Server still down, keep polling
|
||||
}
|
||||
}, 1000);
|
||||
}, [queryClient, clearHealthPolling]);
|
||||
|
||||
const restartServerWithPolling = useCallback(
|
||||
async (errorMessage: string) => {
|
||||
setRestartPhase('stopping');
|
||||
try {
|
||||
await platform.lifecycle.restartServer();
|
||||
setRestartPhase('waiting');
|
||||
startHealthPolling();
|
||||
} catch (e: unknown) {
|
||||
clearHealthPolling();
|
||||
setRestartPhase('idle');
|
||||
throw new Error(e instanceof Error ? e.message : errorMessage);
|
||||
}
|
||||
},
|
||||
[platform, startHealthPolling, clearHealthPolling],
|
||||
);
|
||||
|
||||
const handleDownload = async () => {
|
||||
setError(null);
|
||||
try {
|
||||
await apiClient.downloadCudaBackend();
|
||||
refetchCudaStatus();
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : 'Failed to start download';
|
||||
if (msg.includes('already downloaded')) {
|
||||
refetchCudaStatus();
|
||||
} else {
|
||||
setError(msg);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const handleRestart = async () => {
|
||||
setError(null);
|
||||
try {
|
||||
await restartServerWithPolling('Restart failed');
|
||||
} catch (e: unknown) {
|
||||
setError(e instanceof Error ? e.message : 'Restart failed');
|
||||
}
|
||||
};
|
||||
|
||||
const handleSwitchToCpu = async () => {
|
||||
setError(null);
|
||||
setRestartPhase('stopping');
|
||||
try {
|
||||
await apiClient.deleteCudaBackend();
|
||||
await restartServerWithPolling('Failed to switch to CPU');
|
||||
} catch (e: unknown) {
|
||||
setError(e instanceof Error ? e.message : 'Failed to switch to CPU');
|
||||
refetchCudaStatus();
|
||||
}
|
||||
};
|
||||
|
||||
const handleDelete = async () => {
|
||||
setError(null);
|
||||
try {
|
||||
await apiClient.deleteCudaBackend();
|
||||
refetchCudaStatus();
|
||||
} catch (e: unknown) {
|
||||
setError(e instanceof Error ? e.message : 'Failed to delete CUDA backend');
|
||||
}
|
||||
};
|
||||
|
||||
const formatBytes = (bytes: number): string => {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return `${(bytes / k ** i).toFixed(1)} ${sizes[i]}`;
|
||||
};
|
||||
|
||||
if (!health) return null;
|
||||
|
||||
const hasNativeGpu =
|
||||
health.gpu_available &&
|
||||
!isCurrentlyCuda &&
|
||||
health.gpu_type &&
|
||||
!health.gpu_type.includes('CUDA');
|
||||
|
||||
return (
|
||||
<div className="space-y-8 max-w-2xl">
|
||||
<GpuInfoCard health={health} />
|
||||
|
||||
{/* CUDA section — only when no native GPU and not already on CUDA */}
|
||||
{!hasNativeGpu && !isCurrentlyCuda && (
|
||||
<SettingSection
|
||||
title="CUDA Backend"
|
||||
description="NVIDIA GPU acceleration via a downloadable CUDA backend."
|
||||
>
|
||||
{/* Download progress */}
|
||||
{cudaDownloading && downloadProgress && (
|
||||
<SettingRow title="Downloading CUDA backend...">
|
||||
<div className="space-y-1.5">
|
||||
<Progress value={downloadProgress.progress} className="h-2" />
|
||||
<div className="flex items-center justify-between text-xs text-muted-foreground">
|
||||
<span>
|
||||
{downloadProgress.filename ||
|
||||
(cudaAvailable ? 'Updating...' : 'Downloading...')}
|
||||
</span>
|
||||
<span>
|
||||
{downloadProgress.total > 0
|
||||
? `${formatBytes(downloadProgress.current)} / ${formatBytes(downloadProgress.total)}`
|
||||
: `${downloadProgress.progress.toFixed(1)}%`}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</SettingRow>
|
||||
)}
|
||||
|
||||
{/* Restart in progress */}
|
||||
{restartPhase !== 'idle' && (
|
||||
<SettingRow
|
||||
title={
|
||||
restartPhase === 'ready'
|
||||
? 'Server restarted successfully'
|
||||
: restartPhase === 'waiting'
|
||||
? 'Restarting server...'
|
||||
: 'Stopping server...'
|
||||
}
|
||||
action={<Loader2 className="h-4 w-4 animate-spin text-muted-foreground" />}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Error */}
|
||||
{error && (
|
||||
<SettingRow title="Error">
|
||||
<div className="flex items-center gap-2 text-sm text-destructive">
|
||||
<AlertCircle className="h-4 w-4 shrink-0" />
|
||||
<span>{error}</span>
|
||||
</div>
|
||||
</SettingRow>
|
||||
)}
|
||||
|
||||
{/* Actions */}
|
||||
{restartPhase === 'idle' && !cudaDownloading && (
|
||||
<>
|
||||
{!cudaAvailable && !isCurrentlyCuda && (
|
||||
<SettingRow
|
||||
title="Download CUDA backend"
|
||||
description="~2.4 GB download. Requires an NVIDIA GPU with CUDA support."
|
||||
action={
|
||||
<Button onClick={handleDownload} size="sm">
|
||||
<Download className="h-3.5 w-3.5 mr-1.5" />
|
||||
Download
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{cudaAvailable && !isCurrentlyCuda && platform.metadata.isTauri && (
|
||||
<SettingRow
|
||||
title="Switch to CUDA backend"
|
||||
description="CUDA backend is downloaded and ready. Restart to enable."
|
||||
action={
|
||||
<Button onClick={handleRestart} size="sm">
|
||||
<RotateCw className="h-3.5 w-3.5 mr-1.5" />
|
||||
Restart
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{isCurrentlyCuda && platform.metadata.isTauri && (
|
||||
<SettingRow
|
||||
title="Switch to CPU backend"
|
||||
description="Disable GPU acceleration. You can re-download CUDA later."
|
||||
action={
|
||||
<Button onClick={handleSwitchToCpu} variant="outline" size="sm">
|
||||
<RotateCw className="h-3.5 w-3.5 mr-1.5" />
|
||||
Switch
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{cudaAvailable && !isCurrentlyCuda && (
|
||||
<SettingRow
|
||||
title="Remove CUDA backend"
|
||||
description="Delete the downloaded CUDA binary to free disk space."
|
||||
action={
|
||||
<Button
|
||||
onClick={handleDelete}
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="text-muted-foreground hover:text-destructive"
|
||||
>
|
||||
<Trash2 className="h-3.5 w-3.5 mr-1.5" />
|
||||
Remove
|
||||
</Button>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</SettingSection>
|
||||
)}
|
||||
|
||||
<p className="text-xs text-muted-foreground/60 leading-relaxed">
|
||||
Voicebox automatically detects and uses the best available GPU on your system. On Apple
|
||||
Silicon Macs, the MLX backend runs natively on the Neural Engine and GPU via Metal
|
||||
Performance Shaders (MPS), with no additional setup required. On Windows and Linux with
|
||||
NVIDIA GPUs, you can download an optional CUDA backend for hardware-accelerated inference.
|
||||
AMD ROCm, Intel XPU, and DirectML are also supported where available through PyTorch. When
|
||||
no GPU is detected, Voicebox falls back to CPU — all engines still work, just slower.
|
||||
</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { type LogEntry, useLogStore } from '@/stores/logStore';
|
||||
|
||||
function formatTime(timestamp: number): string {
|
||||
const d = new Date(timestamp);
|
||||
return d.toLocaleTimeString(undefined, {
|
||||
hour: '2-digit',
|
||||
minute: '2-digit',
|
||||
second: '2-digit',
|
||||
hour12: false,
|
||||
});
|
||||
}
|
||||
|
||||
function LogLine({ entry }: { entry: LogEntry }) {
|
||||
return (
|
||||
<div className="flex gap-3 font-mono text-xs leading-5 hover:bg-muted/30">
|
||||
<span className="text-muted-foreground/50 select-none shrink-0">
|
||||
{formatTime(entry.timestamp)}
|
||||
</span>
|
||||
<span
|
||||
className={cn(
|
||||
'whitespace-pre-wrap break-all',
|
||||
entry.stream === 'stderr' ? 'text-orange-400/80' : 'text-muted-foreground',
|
||||
)}
|
||||
>
|
||||
{entry.line}
|
||||
</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function LogsPage() {
|
||||
const entries = useLogStore((s) => s.entries);
|
||||
const clear = useLogStore((s) => s.clear);
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const [autoScroll, setAutoScroll] = useState(true);
|
||||
|
||||
// Auto-scroll to bottom when new entries arrive
|
||||
useEffect(() => {
|
||||
if (autoScroll && containerRef.current) {
|
||||
containerRef.current.scrollTop = containerRef.current.scrollHeight;
|
||||
}
|
||||
}, [entries.length, autoScroll]);
|
||||
|
||||
// Detect manual scroll to disable auto-scroll
|
||||
const handleScroll = () => {
|
||||
const el = containerRef.current;
|
||||
if (!el) return;
|
||||
const atBottom = el.scrollHeight - el.scrollTop - el.clientHeight < 40;
|
||||
setAutoScroll(atBottom);
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="flex flex-col h-full min-h-0">
|
||||
<div className="flex items-center justify-between mb-3">
|
||||
<div>
|
||||
<h3 className="text-sm font-medium">Server Logs</h3>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
{entries.length} {entries.length === 1 ? 'line' : 'lines'}
|
||||
</p>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{!autoScroll && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
onClick={() => {
|
||||
setAutoScroll(true);
|
||||
containerRef.current?.scrollTo({ top: containerRef.current.scrollHeight });
|
||||
}}
|
||||
>
|
||||
Scroll to bottom
|
||||
</Button>
|
||||
)}
|
||||
<Button variant="outline" size="sm" onClick={clear}>
|
||||
Clear
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div
|
||||
ref={containerRef}
|
||||
onScroll={handleScroll}
|
||||
className="flex-1 min-h-0 overflow-y-auto rounded-md border bg-black/20 p-3"
|
||||
>
|
||||
{entries.length === 0 ? (
|
||||
<div className="text-sm text-muted-foreground/50 font-mono space-y-1">
|
||||
<p>No log output yet.</p>
|
||||
{!import.meta.env?.PROD && (
|
||||
<p>
|
||||
Server logs are only captured when the app manages the server process (production
|
||||
builds).
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
) : (
|
||||
entries.map((entry) => <LogLine key={entry.id} entry={entry} />)
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,35 +1,70 @@
|
||||
import { ConnectionForm } from '@/components/ServerSettings/ConnectionForm';
|
||||
import { GenerationSettings } from '@/components/ServerSettings/GenerationSettings';
|
||||
import { GpuAcceleration } from '@/components/ServerSettings/GpuAcceleration';
|
||||
import { UpdateStatus } from '@/components/ServerSettings/UpdateStatus';
|
||||
import { Link, Outlet, useMatchRoute } from '@tanstack/react-router';
|
||||
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import { usePlayerStore } from '@/stores/playerStore';
|
||||
|
||||
export function ServerTab() {
|
||||
interface SettingsTab {
|
||||
label: string;
|
||||
path:
|
||||
| '/settings'
|
||||
| '/settings/generation'
|
||||
| '/settings/gpu'
|
||||
| '/settings/logs'
|
||||
| '/settings/changelog'
|
||||
| '/settings/about';
|
||||
tauriOnly?: boolean;
|
||||
}
|
||||
|
||||
const tabs: SettingsTab[] = [
|
||||
{ label: 'General', path: '/settings' },
|
||||
{ label: 'Generation', path: '/settings/generation' },
|
||||
{ label: 'GPU', path: '/settings/gpu', tauriOnly: true },
|
||||
{ label: 'Logs', path: '/settings/logs', tauriOnly: true },
|
||||
{ label: 'Changelog', path: '/settings/changelog' },
|
||||
{ label: 'About', path: '/settings/about' },
|
||||
];
|
||||
|
||||
export function SettingsLayout() {
|
||||
const platform = usePlatform();
|
||||
const isPlayerVisible = !!usePlayerStore((state) => state.audioUrl);
|
||||
const matchRoute = useMatchRoute();
|
||||
|
||||
return (
|
||||
<div
|
||||
className={cn('overflow-y-auto flex flex-col', isPlayerVisible && BOTTOM_SAFE_AREA_PADDING)}
|
||||
>
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
<ConnectionForm />
|
||||
<GenerationSettings />
|
||||
{platform.metadata.isTauri && <GpuAcceleration />}
|
||||
{platform.metadata.isTauri && <UpdateStatus />}
|
||||
</div>
|
||||
<div className="py-8 text-center text-sm text-muted-foreground">
|
||||
Created by{' '}
|
||||
<a
|
||||
href="https://github.com/jamiepine"
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="text-accent hover:underline"
|
||||
>
|
||||
Jamie Pine
|
||||
</a>
|
||||
<div className="flex flex-col h-full min-h-0">
|
||||
<nav className="flex gap-1 border-b shrink-0">
|
||||
{tabs.map((tab) => {
|
||||
if (tab.tauriOnly && !platform.metadata.isTauri) return null;
|
||||
|
||||
const isActive =
|
||||
tab.path === '/settings'
|
||||
? matchRoute({ to: tab.path, fuzzy: false })
|
||||
: matchRoute({ to: tab.path });
|
||||
|
||||
return (
|
||||
<Link
|
||||
key={tab.path}
|
||||
to={tab.path}
|
||||
className={cn(
|
||||
'px-4 py-2 text-sm font-medium transition-colors border-b-2 -mb-px',
|
||||
isActive
|
||||
? 'border-accent text-foreground'
|
||||
: 'border-transparent text-muted-foreground hover:text-foreground hover:border-muted-foreground/30',
|
||||
)}
|
||||
>
|
||||
{tab.label}
|
||||
</Link>
|
||||
);
|
||||
})}
|
||||
</nav>
|
||||
|
||||
<div
|
||||
className={cn(
|
||||
'flex-1 overflow-y-auto pt-6 pb-6 px-2 -mx-2',
|
||||
isPlayerVisible && BOTTOM_SAFE_AREA_PADDING,
|
||||
)}
|
||||
>
|
||||
<Outlet />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
import type { ReactNode } from 'react';
|
||||
|
||||
/**
|
||||
* A section header with title and optional description, separated by a border.
|
||||
*/
|
||||
export function SettingSection({
|
||||
title,
|
||||
description,
|
||||
children,
|
||||
}: {
|
||||
title?: string;
|
||||
description?: string;
|
||||
children: ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<div className="space-y-1">
|
||||
{title && <h3 className="text-sm font-medium">{title}</h3>}
|
||||
{description && <p className="text-sm text-muted-foreground">{description}</p>}
|
||||
<div className={`${title || description ? 'pt-3' : ''} space-y-0 divide-y divide-border/60`}>
|
||||
{children}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A single settings row: label+description on the left, action on the right.
|
||||
* Use for toggles, inputs, buttons, badges — any control type.
|
||||
*/
|
||||
export function SettingRow({
|
||||
title,
|
||||
description,
|
||||
htmlFor,
|
||||
action,
|
||||
children,
|
||||
}: {
|
||||
title: string;
|
||||
description?: string;
|
||||
htmlFor?: string;
|
||||
/** Right-aligned control (checkbox, button, badge, etc.) */
|
||||
action?: ReactNode;
|
||||
/** Full-width content rendered below the label row (for sliders, inputs, etc.) */
|
||||
children?: ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<div className="py-3">
|
||||
<div className="flex items-center justify-between gap-8">
|
||||
<div className="min-w-0">
|
||||
<label
|
||||
htmlFor={htmlFor}
|
||||
className={`text-sm font-medium leading-none select-none ${htmlFor ? 'cursor-pointer' : ''}`}
|
||||
>
|
||||
{title}
|
||||
</label>
|
||||
{description && <p className="text-sm text-muted-foreground mt-0.5">{description}</p>}
|
||||
</div>
|
||||
{action && <div className="shrink-0">{action}</div>}
|
||||
</div>
|
||||
{children && <div className="mt-3">{children}</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,7 +1,10 @@
|
||||
import { Link, useMatchRoute } from '@tanstack/react-router';
|
||||
import { AudioLines, Box, Mic, Server, Speaker, Volume2, Wand2 } from 'lucide-react';
|
||||
import { AudioLines, Box, Mic, Settings, Speaker, Volume2, Wand2 } from 'lucide-react';
|
||||
import { useEffect, useState } from 'react';
|
||||
import voiceboxLogo from '@/assets/voicebox-logo.png';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
import type { UpdateStatus } from '@/platform/types';
|
||||
import { usePlayerStore } from '@/stores/playerStore';
|
||||
import { version } from '../../package.json';
|
||||
|
||||
@@ -16,12 +19,16 @@ const tabs = [
|
||||
{ id: 'effects', path: '/effects', icon: Wand2, label: 'Effects' },
|
||||
{ id: 'audio', path: '/audio', icon: Speaker, label: 'Audio' },
|
||||
{ id: 'models', path: '/models', icon: Box, label: 'Models' },
|
||||
{ id: 'server', path: '/server', icon: Server, label: 'Server' },
|
||||
{ id: 'settings', path: '/settings', icon: Settings, label: 'Settings' },
|
||||
];
|
||||
|
||||
export function Sidebar({ isMacOS }: SidebarProps) {
|
||||
const matchRoute = useMatchRoute();
|
||||
const isPlayerOpen = !!usePlayerStore((s) => s.audioUrl);
|
||||
const platform = usePlatform();
|
||||
|
||||
const [updateStatus, setUpdateStatus] = useState<UpdateStatus>(platform.updater.getStatus());
|
||||
useEffect(() => platform.updater.subscribe(setUpdateStatus), [platform.updater]);
|
||||
|
||||
return (
|
||||
<div
|
||||
@@ -47,9 +54,10 @@ export function Sidebar({ isMacOS }: SidebarProps) {
|
||||
<div className="flex flex-col gap-3">
|
||||
{tabs.map((tab, index) => {
|
||||
const Icon = tab.icon;
|
||||
// For index route, use exact match; for others, use default matching
|
||||
const isActive =
|
||||
tab.path === '/' ? matchRoute({ to: '/', exact: true }) : matchRoute({ to: tab.path });
|
||||
tab.path === '/'
|
||||
? matchRoute({ to: '/', fuzzy: false })
|
||||
: matchRoute({ to: tab.path, fuzzy: true });
|
||||
|
||||
// Accent fades as buttons get further from the logo
|
||||
const accentOpacity = Math.max(0.08, 0.5 - index * 0.07);
|
||||
@@ -85,10 +93,18 @@ export function Sidebar({ isMacOS }: SidebarProps) {
|
||||
|
||||
{/* Version */}
|
||||
<div
|
||||
className="mt-auto text-[10px] text-muted-foreground/50 transition-all duration-300"
|
||||
className="mt-auto flex flex-col items-center gap-1.5 transition-all duration-300"
|
||||
style={{ paddingBottom: isPlayerOpen ? '7rem' : undefined }}
|
||||
>
|
||||
v{version}
|
||||
<span className="text-[10px] text-muted-foreground/50">v{version}</span>
|
||||
{updateStatus.available && (
|
||||
<Link
|
||||
to="/settings"
|
||||
className="text-[9px] font-semibold tracking-wide uppercase px-2 py-0.5 rounded-full bg-accent/15 text-accent hover:bg-accent/25 transition-colors"
|
||||
>
|
||||
Update
|
||||
</Link>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -87,7 +87,7 @@ export function StoryChatItem({
|
||||
alt={`${item.profile_name} avatar`}
|
||||
className={cn(
|
||||
'h-full w-full object-cover transition-all duration-200',
|
||||
!isCurrentlyPlaying && 'grayscale'
|
||||
!isCurrentlyPlaying && 'grayscale',
|
||||
)}
|
||||
onError={() => setAvatarError(true)}
|
||||
/>
|
||||
@@ -127,7 +127,10 @@ export function StoryChatItem({
|
||||
<Play className="mr-2 h-4 w-4" />
|
||||
Play from here
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={onRemove} className="text-destructive focus:text-destructive">
|
||||
<DropdownMenuItem
|
||||
onClick={onRemove}
|
||||
className="text-destructive focus:text-destructive"
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Remove from Story
|
||||
</DropdownMenuItem>
|
||||
@@ -139,15 +142,12 @@ export function StoryChatItem({
|
||||
}
|
||||
|
||||
// Sortable wrapper component
|
||||
export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>) {
|
||||
const {
|
||||
attributes,
|
||||
listeners,
|
||||
setNodeRef,
|
||||
transform,
|
||||
transition,
|
||||
isDragging,
|
||||
} = useSortable({ id: props.item.generation_id });
|
||||
export function SortableStoryChatItem(
|
||||
props: Omit<StoryChatItemProps, 'dragHandleProps' | 'isDragging'>,
|
||||
) {
|
||||
const { attributes, listeners, setNodeRef, transform, transition, isDragging } = useSortable({
|
||||
id: props.item.generation_id,
|
||||
});
|
||||
|
||||
const style = {
|
||||
transform: CSS.Transform.toString(transform),
|
||||
@@ -156,11 +156,7 @@ export function SortableStoryChatItem(props: Omit<StoryChatItemProps, 'dragHandl
|
||||
|
||||
return (
|
||||
<div ref={setNodeRef} style={style} {...attributes}>
|
||||
<StoryChatItem
|
||||
{...props}
|
||||
dragHandleProps={listeners}
|
||||
isDragging={isDragging}
|
||||
/>
|
||||
<StoryChatItem {...props} dragHandleProps={listeners} isDragging={isDragging} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -371,7 +371,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
}
|
||||
}, [isResizing, handleResizeMove, handleResizeEnd]);
|
||||
|
||||
const handleTimelineClick = (e: React.MouseEvent<HTMLDivElement>) => {
|
||||
const handleTimelineClick = (e: React.MouseEvent<HTMLElement>) => {
|
||||
if (!tracksRef.current || draggingItem || trimmingItem) return;
|
||||
const rect = tracksRef.current.getBoundingClientRect();
|
||||
const x = e.clientX - rect.left + tracksRef.current.scrollLeft;
|
||||
@@ -500,7 +500,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
|
||||
}, [trimmingItem, trimSide, tempTrimValues, storyId, trimItem, toast]);
|
||||
|
||||
const handleSplit = useCallback(() => {
|
||||
if (!selectedClipId) return;
|
||||
if (!selectedClipId || splitItem.isPending) return;
|
||||
|
||||
const item = items.find((i) => i.id === selectedClipId);
|
||||
if (!item) return;
|
||||
|
||||
@@ -14,12 +14,7 @@ const MemoizedWaveform = memo(function MemoizedWaveform({
|
||||
<div className="absolute inset-0 pointer-events-none flex items-center justify-center opacity-30">
|
||||
<Visualizer audio={audioStream} autoStart strokeColor="#b39a3d">
|
||||
{({ canvasRef }) => (
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
width={500}
|
||||
height={150}
|
||||
className="w-full h-full"
|
||||
/>
|
||||
<canvas ref={canvasRef} width={500} height={150} className="w-full h-full" />
|
||||
)}
|
||||
</Visualizer>
|
||||
</div>
|
||||
@@ -87,9 +82,7 @@ export function AudioSampleRecording({
|
||||
<div className="space-y-4">
|
||||
{!isRecording && !file && (
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-dashed rounded-lg min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
|
||||
<Button
|
||||
type="button"
|
||||
onClick={onStart}
|
||||
@@ -107,9 +100,7 @@ export function AudioSampleRecording({
|
||||
|
||||
{isRecording && (
|
||||
<div className="relative flex flex-col items-center justify-center gap-4 p-4 border-2 border-accent rounded-lg bg-accent/5 min-h-[180px] overflow-hidden">
|
||||
{showWaveform && audioStream && (
|
||||
<MemoizedWaveform audioStream={audioStream} />
|
||||
)}
|
||||
{showWaveform && audioStream && <MemoizedWaveform audioStream={audioStream} />}
|
||||
<div className="relative z-10 flex items-center gap-4">
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="h-3 w-3 rounded-full bg-accent animate-pulse" />
|
||||
|
||||
@@ -17,11 +17,18 @@ import { useDeleteProfile, useExportProfile } from '@/lib/hooks/useProfiles';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
|
||||
/** Human-readable display names for preset engine badges. */
|
||||
const ENGINE_DISPLAY_NAMES: Record<string, string> = {
|
||||
kokoro: 'Kokoro',
|
||||
qwen_custom_voice: 'CustomVoice',
|
||||
};
|
||||
|
||||
interface ProfileCardProps {
|
||||
profile: VoiceProfileResponse;
|
||||
disabled?: boolean;
|
||||
}
|
||||
|
||||
export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
export function ProfileCard({ profile, disabled }: ProfileCardProps) {
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
|
||||
|
||||
const deleteProfile = useDeleteProfile();
|
||||
@@ -34,6 +41,12 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
const isSelected = selectedProfileId === profile.id;
|
||||
|
||||
const handleSelect = () => {
|
||||
// If disabled but already selected, bounce the selection to re-trigger engine auto-switch
|
||||
if (disabled && isSelected) {
|
||||
setSelectedProfileId(null);
|
||||
setTimeout(() => setSelectedProfileId(profile.id), 0);
|
||||
return;
|
||||
}
|
||||
setSelectedProfileId(isSelected ? null : profile.id);
|
||||
};
|
||||
|
||||
@@ -74,8 +87,9 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
<>
|
||||
<Card
|
||||
className={cn(
|
||||
'cursor-pointer hover:shadow-md transition-all flex flex-col h-[162px]',
|
||||
isSelected && 'ring-2 ring-accent shadow-md',
|
||||
'cursor-pointer transition-all flex flex-col h-[162px]',
|
||||
disabled ? 'opacity-40 hover:opacity-60' : 'hover:shadow-md',
|
||||
isSelected && !disabled && 'ring-2 ring-accent shadow-md',
|
||||
)}
|
||||
onClick={handleSelect}
|
||||
tabIndex={0}
|
||||
@@ -97,6 +111,16 @@ export function ProfileCard({ profile }: ProfileCardProps) {
|
||||
<Badge variant="outline" className="text-xs h-5 px-1.5 text-muted-foreground">
|
||||
{profile.language}
|
||||
</Badge>
|
||||
{profile.voice_type === 'preset' && (
|
||||
<Badge variant="secondary" className="text-xs h-5 px-1.5">
|
||||
{ENGINE_DISPLAY_NAMES[profile.preset_engine ?? ''] ?? profile.preset_engine}
|
||||
</Badge>
|
||||
)}
|
||||
{profile.voice_type === 'designed' && (
|
||||
<Badge variant="secondary" className="text-xs h-5 px-1.5">
|
||||
designed
|
||||
</Badge>
|
||||
)}
|
||||
{profile.effects_chain && profile.effects_chain.length > 0 && (
|
||||
<Sparkles className="h-3.5 w-3.5 text-accent fill-accent" />
|
||||
)}
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import { zodResolver } from '@hookform/resolvers/zod';
|
||||
import { Edit2, Mic, Monitor, Upload, X } from 'lucide-react';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { Edit2, Mic, Monitor, Music, Upload, X } from 'lucide-react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { useForm } from 'react-hook-form';
|
||||
import * as z from 'zod';
|
||||
import { EffectsChainEditor } from '@/components/Effects/EffectsChainEditor';
|
||||
import { Badge } from '@/components/ui/badge';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import {
|
||||
Dialog,
|
||||
@@ -15,6 +17,7 @@ import {
|
||||
import {
|
||||
Form,
|
||||
FormControl,
|
||||
FormDescription,
|
||||
FormField,
|
||||
FormItem,
|
||||
FormLabel,
|
||||
@@ -32,7 +35,7 @@ import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { useToast } from '@/components/ui/use-toast';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { EffectConfig } from '@/lib/api/types';
|
||||
import type { EffectConfig, PresetVoice, VoiceType } from '@/lib/api/types';
|
||||
import { LANGUAGE_CODES, LANGUAGE_OPTIONS, type LanguageCode } from '@/lib/constants/languages';
|
||||
import { useAudioPlayer } from '@/lib/hooks/useAudioPlayer';
|
||||
import { useAudioRecording } from '@/lib/hooks/useAudioRecording';
|
||||
@@ -40,6 +43,7 @@ import {
|
||||
useAddSample,
|
||||
useCreateProfile,
|
||||
useDeleteAvatar,
|
||||
useDeleteProfile,
|
||||
useProfile,
|
||||
useUpdateProfile,
|
||||
useUploadAvatar,
|
||||
@@ -56,6 +60,16 @@ import { AudioSampleUpload } from './AudioSampleUpload';
|
||||
import { SampleList } from './SampleList';
|
||||
|
||||
const MAX_AUDIO_DURATION_SECONDS = 30;
|
||||
const PRESET_ONLY_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
|
||||
const DEFAULT_ENGINE_OPTIONS = [
|
||||
{ value: 'qwen', label: 'Qwen3-TTS' },
|
||||
{ value: 'qwen_custom_voice', label: 'Qwen CustomVoice' },
|
||||
{ value: 'luxtts', label: 'LuxTTS' },
|
||||
{ value: 'chatterbox', label: 'Chatterbox' },
|
||||
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
|
||||
{ value: 'tada', label: 'TADA' },
|
||||
{ value: 'kokoro', label: 'Kokoro 82M' },
|
||||
] as const;
|
||||
|
||||
const baseProfileSchema = z.object({
|
||||
name: z.string().min(1, 'Name is required').max(100),
|
||||
@@ -116,20 +130,25 @@ export function ProfileForm() {
|
||||
const createProfile = useCreateProfile();
|
||||
const updateProfile = useUpdateProfile();
|
||||
const addSample = useAddSample();
|
||||
const deleteProfile = useDeleteProfile();
|
||||
const uploadAvatar = useUploadAvatar();
|
||||
const deleteAvatar = useDeleteAvatar();
|
||||
const transcribe = useTranscription();
|
||||
const { toast } = useToast();
|
||||
const [voiceSource, setVoiceSource] = useState<'clone' | 'builtin'>('clone');
|
||||
const [sampleMode, setSampleMode] = useState<'upload' | 'record' | 'system'>('record');
|
||||
const [audioDuration, setAudioDuration] = useState<number | null>(null);
|
||||
const [isValidatingAudio, setIsValidatingAudio] = useState(false);
|
||||
const [avatarPreview, setAvatarPreview] = useState<string | null>(null);
|
||||
const [selectedPresetEngine, setSelectedPresetEngine] = useState<string>('kokoro');
|
||||
const [selectedPresetVoiceId, setSelectedPresetVoiceId] = useState<string>('');
|
||||
const avatarInputRef = useRef<HTMLInputElement>(null);
|
||||
const { isPlaying, playPause, cleanup: cleanupAudio } = useAudioPlayer();
|
||||
const isCreating = !editingProfileId;
|
||||
const serverUrl = useServerStore((state) => state.serverUrl);
|
||||
const [profileEffectsChain, setProfileEffectsChain] = useState<EffectConfig[]>([]);
|
||||
const [effectsDirty, setEffectsDirty] = useState(false);
|
||||
const [defaultEngine, setDefaultEngine] = useState<string>('');
|
||||
|
||||
const form = useForm<ProfileFormValues>({
|
||||
resolver: zodResolver(profileSchema),
|
||||
@@ -239,6 +258,26 @@ export function ProfileForm() {
|
||||
},
|
||||
});
|
||||
|
||||
// Fetch available preset voices for the selected engine
|
||||
const presetEngineToQuery = isCreating
|
||||
? selectedPresetEngine
|
||||
: (editingProfile?.preset_engine ?? '');
|
||||
const { data: presetVoicesData } = useQuery({
|
||||
queryKey: ['presetVoices', presetEngineToQuery],
|
||||
queryFn: () => apiClient.listPresetVoices(presetEngineToQuery),
|
||||
enabled:
|
||||
!!presetEngineToQuery &&
|
||||
((voiceSource === 'builtin' && isCreating) ||
|
||||
(!isCreating && editingProfile?.voice_type === 'preset')),
|
||||
});
|
||||
const presetVoices = presetVoicesData?.voices ?? [];
|
||||
const isSampleBasedProfile = isCreating
|
||||
? voiceSource === 'clone'
|
||||
: editingProfile?.voice_type !== 'preset';
|
||||
const availableDefaultEngines = DEFAULT_ENGINE_OPTIONS.filter(
|
||||
(option) => !isSampleBasedProfile || !PRESET_ONLY_ENGINES.has(option.value),
|
||||
);
|
||||
|
||||
// Show recording errors
|
||||
useEffect(() => {
|
||||
if (recordingError) {
|
||||
@@ -287,6 +326,7 @@ export function ProfileForm() {
|
||||
});
|
||||
setProfileEffectsChain(editingProfile.effects_chain ?? []);
|
||||
setEffectsDirty(false);
|
||||
setDefaultEngine(editingProfile.default_engine ?? '');
|
||||
} else if (profileFormDraft && open) {
|
||||
// Restore from draft when opening in create mode
|
||||
form.reset({
|
||||
@@ -326,6 +366,24 @@ export function ProfileForm() {
|
||||
}
|
||||
}, [editingProfile, profileFormDraft, open, form]);
|
||||
|
||||
useEffect(() => {
|
||||
if (
|
||||
defaultEngine &&
|
||||
!availableDefaultEngines.some((option) => option.value === defaultEngine)
|
||||
) {
|
||||
setDefaultEngine('');
|
||||
}
|
||||
}, [availableDefaultEngines, defaultEngine]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!selectedPresetVoiceId) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (!presetVoices.some((voice: PresetVoice) => voice.voice_id === selectedPresetVoiceId)) {
|
||||
setSelectedPresetVoiceId('');
|
||||
}
|
||||
}, [presetVoices, selectedPresetVoiceId]);
|
||||
async function handleTranscribe() {
|
||||
const file = form.getValues('sampleFile');
|
||||
if (!file) {
|
||||
@@ -415,13 +473,14 @@ export function ProfileForm() {
|
||||
async function onSubmit(data: ProfileFormValues) {
|
||||
try {
|
||||
if (editingProfileId) {
|
||||
// Editing: just update profile
|
||||
// Editing: update profile
|
||||
await updateProfile.mutateAsync({
|
||||
profileId: editingProfileId,
|
||||
data: {
|
||||
name: data.name,
|
||||
description: data.description,
|
||||
language: data.language,
|
||||
default_engine: defaultEngine || undefined,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -464,8 +523,50 @@ export function ProfileForm() {
|
||||
title: 'Voice updated',
|
||||
description: `"${data.name}" has been updated successfully.`,
|
||||
});
|
||||
} else if (voiceSource === 'builtin') {
|
||||
// Creating preset profile from built-in voice
|
||||
if (!selectedPresetVoiceId) {
|
||||
toast({
|
||||
title: 'No voice selected',
|
||||
description: 'Please select a built-in voice.',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const profile = await createProfile.mutateAsync({
|
||||
name: data.name,
|
||||
description: data.description,
|
||||
language: data.language,
|
||||
voice_type: 'preset' as VoiceType,
|
||||
preset_engine: selectedPresetEngine,
|
||||
preset_voice_id: selectedPresetVoiceId,
|
||||
default_engine: selectedPresetEngine,
|
||||
});
|
||||
|
||||
// Handle avatar upload if provided
|
||||
if (data.avatarFile) {
|
||||
try {
|
||||
await uploadAvatar.mutateAsync({
|
||||
profileId: profile.id,
|
||||
file: data.avatarFile,
|
||||
});
|
||||
} catch (avatarError) {
|
||||
toast({
|
||||
title: 'Avatar upload failed',
|
||||
description:
|
||||
avatarError instanceof Error ? avatarError.message : 'Failed to upload avatar',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
toast({
|
||||
title: 'Profile created',
|
||||
description: `"${data.name}" has been created with a built-in voice.`,
|
||||
});
|
||||
} else {
|
||||
// Creating: require sample file and reference text
|
||||
// Creating cloned profile: require sample file and reference text
|
||||
const sampleFile = form.getValues('sampleFile');
|
||||
const referenceText = form.getValues('referenceText');
|
||||
|
||||
@@ -528,6 +629,7 @@ export function ProfileForm() {
|
||||
name: data.name,
|
||||
description: data.description,
|
||||
language: data.language,
|
||||
default_engine: defaultEngine || undefined,
|
||||
});
|
||||
|
||||
// Convert non-WAV uploads to WAV so the backend can always use soundfile.
|
||||
@@ -572,12 +674,32 @@ export function ProfileForm() {
|
||||
description: `"${data.name}" has been created with a sample.`,
|
||||
});
|
||||
} catch (sampleError) {
|
||||
// Profile was created but sample failed - still show error
|
||||
let rollbackSucceeded = false;
|
||||
try {
|
||||
await deleteProfile.mutateAsync(profile.id);
|
||||
rollbackSucceeded = true;
|
||||
} catch (rollbackError) {
|
||||
toast({
|
||||
title: 'Rollback failed',
|
||||
description:
|
||||
rollbackError instanceof Error
|
||||
? rollbackError.message
|
||||
: 'Created profile could not be removed after sample upload failure.',
|
||||
variant: 'destructive',
|
||||
});
|
||||
}
|
||||
|
||||
toast({
|
||||
title: 'Failed to add sample',
|
||||
description: `Profile "${data.name}" was created, but failed to add sample: ${sampleError instanceof Error ? sampleError.message : 'Unknown error'}`,
|
||||
description:
|
||||
sampleError instanceof Error
|
||||
? `${sampleError.message}${rollbackSucceeded ? ' The profile was rolled back.' : ''}`
|
||||
: rollbackSucceeded
|
||||
? 'Failed to add sample. The profile was rolled back.'
|
||||
: 'Failed to add sample.',
|
||||
variant: 'destructive',
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -642,16 +764,16 @@ export function ProfileForm() {
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={handleOpenChange}>
|
||||
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-y-auto">
|
||||
<div className="max-w-5xl max-h-[85vh] mx-auto my-auto w-full flex flex-col">
|
||||
<DialogContent className="max-w-none w-screen h-screen left-0 top-0 translate-x-0 translate-y-0 rounded-none p-6 overflow-hidden">
|
||||
<div className="max-w-5xl h-[85vh] mx-auto my-auto w-full flex flex-col overflow-hidden">
|
||||
<DialogHeader>
|
||||
<DialogTitle className="text-2xl">
|
||||
{editingProfileId ? 'Edit Voice' : 'Clone voice'}
|
||||
{editingProfileId ? 'Edit Voice' : 'Create Voice'}
|
||||
</DialogTitle>
|
||||
<DialogDescription>
|
||||
{editingProfileId
|
||||
? 'Update your voice profile details and manage samples.'
|
||||
: 'Create a new voice profile with an audio sample to clone the voice.'}
|
||||
: 'Create a new voice profile from an audio sample or a built-in voice.'}
|
||||
</DialogDescription>
|
||||
{isCreating && profileFormDraft && (
|
||||
<div className="flex items-center gap-2 pt-2">
|
||||
@@ -682,143 +804,276 @@ export function ProfileForm() {
|
||||
|
||||
<Form {...form}>
|
||||
<form onSubmit={form.handleSubmit(onSubmit)} className="flex-1 min-h-0 flex flex-col">
|
||||
<div className="grid gap-6 grid-cols-2 flex-1 overflow-y-auto min-h-0">
|
||||
<div className="grid gap-6 grid-cols-2 flex-1 min-h-0 overflow-hidden">
|
||||
{/* Left column: Sample management */}
|
||||
<div className="space-y-4 border-r pr-6">
|
||||
<div className="space-y-4 border-r pr-6 overflow-y-auto min-h-0">
|
||||
{isCreating ? (
|
||||
<>
|
||||
<Tabs
|
||||
className="pt-4"
|
||||
value={sampleMode}
|
||||
onValueChange={(v) => {
|
||||
const newMode = v as 'upload' | 'record' | 'system';
|
||||
// Cancel any active recordings when switching modes
|
||||
if (isRecording && newMode !== 'record') {
|
||||
cancelRecording();
|
||||
}
|
||||
if (isSystemRecording && newMode !== 'system') {
|
||||
cancelSystemRecording();
|
||||
}
|
||||
setSampleMode(newMode);
|
||||
}}
|
||||
>
|
||||
<TabsList
|
||||
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||
>
|
||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||
<Upload className="h-4 w-4 shrink-0" />
|
||||
Upload
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="record" className="flex items-center gap-2">
|
||||
<Mic className="h-4 w-4 shrink-0" />
|
||||
Record
|
||||
</TabsTrigger>
|
||||
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||
<Monitor className="h-4 w-4 shrink-0" />
|
||||
System Audio
|
||||
</TabsTrigger>
|
||||
)}
|
||||
</TabsList>
|
||||
{/* Voice source selector */}
|
||||
<div className="flex pt-4 pb-2">
|
||||
<div className="inline-flex rounded-lg border border-border p-0.5 bg-muted/50">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setVoiceSource('clone')}
|
||||
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
|
||||
voiceSource === 'clone'
|
||||
? 'bg-accent text-accent-foreground shadow-sm'
|
||||
: 'text-muted-foreground hover:text-foreground'
|
||||
}`}
|
||||
>
|
||||
<Mic className="h-3.5 w-3.5" />
|
||||
Clone from audio
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setVoiceSource('builtin')}
|
||||
className={`inline-flex items-center gap-2 px-3 py-1.5 text-sm rounded-md transition-colors ${
|
||||
voiceSource === 'builtin'
|
||||
? 'bg-accent text-accent-foreground shadow-sm'
|
||||
: 'text-muted-foreground hover:text-foreground'
|
||||
}`}
|
||||
>
|
||||
<Music className="h-3.5 w-3.5" />
|
||||
Built-in voice
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<TabsContent value="upload" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={({ field: { onChange, name } }) => (
|
||||
<AudioSampleUpload
|
||||
file={selectedFile}
|
||||
onFileChange={onChange}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isValidating={isValidatingAudio}
|
||||
isTranscribing={transcribe.isPending}
|
||||
isDisabled={
|
||||
audioDuration !== null &&
|
||||
audioDuration > MAX_AUDIO_DURATION_SECONDS
|
||||
}
|
||||
fieldName={name}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
{voiceSource === 'builtin' ? (
|
||||
<div className="space-y-4">
|
||||
<FormDescription>
|
||||
Choose a pre-built voice. These don't require an audio sample.
|
||||
</FormDescription>
|
||||
|
||||
<TabsContent value="record" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleRecording
|
||||
file={selectedFile}
|
||||
isRecording={isRecording}
|
||||
duration={duration}
|
||||
onStart={startRecording}
|
||||
onStop={stopRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
|
||||
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||
<TabsContent value="system" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleSystem
|
||||
file={selectedFile}
|
||||
isRecording={isSystemRecording}
|
||||
duration={systemDuration}
|
||||
onStart={startSystemRecording}
|
||||
onStop={stopSystemRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
)}
|
||||
</Tabs>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="referenceText"
|
||||
render={({ field }) => (
|
||||
{/* Engine selector */}
|
||||
<FormItem>
|
||||
<FormLabel>Reference Text</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea
|
||||
placeholder="Enter the exact text spoken in the audio..."
|
||||
className="min-h-[100px]"
|
||||
{...field}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
<FormLabel>Engine</FormLabel>
|
||||
<Select
|
||||
value={selectedPresetEngine}
|
||||
onValueChange={setSelectedPresetEngine}
|
||||
>
|
||||
<FormControl>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
<SelectItem value="kokoro">Kokoro 82M</SelectItem>
|
||||
<SelectItem value="qwen_custom_voice">Qwen CustomVoice</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
{/* Voice picker */}
|
||||
<FormItem>
|
||||
<FormLabel>Voice</FormLabel>
|
||||
<div className="grid grid-cols-2 gap-1.5 max-h-[340px] overflow-y-auto pr-1">
|
||||
{presetVoices.map((voice: PresetVoice) => (
|
||||
<button
|
||||
key={voice.voice_id}
|
||||
type="button"
|
||||
onClick={() => {
|
||||
setSelectedPresetVoiceId(voice.voice_id);
|
||||
// Auto-set language from voice
|
||||
if (voice.language) {
|
||||
form.setValue('language', voice.language as LanguageCode);
|
||||
}
|
||||
}}
|
||||
className={`text-left px-3 py-2 rounded-md border text-sm transition-colors ${
|
||||
selectedPresetVoiceId === voice.voice_id
|
||||
? 'border-accent bg-accent/10 text-accent-foreground'
|
||||
: 'border-border hover:bg-muted'
|
||||
}`}
|
||||
>
|
||||
<div className="font-medium">{voice.name}</div>
|
||||
<div className="flex gap-1.5 mt-0.5">
|
||||
<Badge variant="outline" className="text-[10px] h-4 px-1">
|
||||
{voice.gender}
|
||||
</Badge>
|
||||
<Badge variant="outline" className="text-[10px] h-4 px-1">
|
||||
{voice.language}
|
||||
</Badge>
|
||||
</div>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</FormItem>
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
<Tabs
|
||||
className="pt-0"
|
||||
value={sampleMode}
|
||||
onValueChange={(v) => {
|
||||
const newMode = v as 'upload' | 'record' | 'system';
|
||||
// Cancel any active recordings when switching modes
|
||||
if (isRecording && newMode !== 'record') {
|
||||
cancelRecording();
|
||||
}
|
||||
if (isSystemRecording && newMode !== 'system') {
|
||||
cancelSystemRecording();
|
||||
}
|
||||
setSampleMode(newMode);
|
||||
}}
|
||||
>
|
||||
<TabsList
|
||||
className={`grid w-full ${platform.metadata.isTauri && isSystemAudioSupported ? 'grid-cols-3' : 'grid-cols-2'}`}
|
||||
>
|
||||
<TabsTrigger value="upload" className="flex items-center gap-2">
|
||||
<Upload className="h-4 w-4 shrink-0" />
|
||||
Upload
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="record" className="flex items-center gap-2">
|
||||
<Mic className="h-4 w-4 shrink-0" />
|
||||
Record
|
||||
</TabsTrigger>
|
||||
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||
<TabsTrigger value="system" className="flex items-center gap-2">
|
||||
<Monitor className="h-4 w-4 shrink-0" />
|
||||
System Audio
|
||||
</TabsTrigger>
|
||||
)}
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="upload" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={({ field: { onChange, name } }) => (
|
||||
<AudioSampleUpload
|
||||
file={selectedFile}
|
||||
onFileChange={onChange}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isValidating={isValidatingAudio}
|
||||
isTranscribing={transcribe.isPending}
|
||||
isDisabled={
|
||||
audioDuration !== null &&
|
||||
audioDuration > MAX_AUDIO_DURATION_SECONDS
|
||||
}
|
||||
fieldName={name}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="record" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleRecording
|
||||
file={selectedFile}
|
||||
isRecording={isRecording}
|
||||
duration={duration}
|
||||
onStart={startRecording}
|
||||
onStop={stopRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
|
||||
{platform.metadata.isTauri && isSystemAudioSupported && (
|
||||
<TabsContent value="system" className="space-y-4">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="sampleFile"
|
||||
render={() => (
|
||||
<AudioSampleSystem
|
||||
file={selectedFile}
|
||||
isRecording={isSystemRecording}
|
||||
duration={systemDuration}
|
||||
onStart={startSystemRecording}
|
||||
onStop={stopSystemRecording}
|
||||
onCancel={handleCancelRecording}
|
||||
onTranscribe={handleTranscribe}
|
||||
onPlayPause={handlePlayPause}
|
||||
isPlaying={isPlaying}
|
||||
isTranscribing={transcribe.isPending}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
</TabsContent>
|
||||
)}
|
||||
</Tabs>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="referenceText"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>Reference Text</FormLabel>
|
||||
<FormControl>
|
||||
<Textarea
|
||||
placeholder="Enter the exact text spoken in the audio..."
|
||||
className="min-h-[100px]"
|
||||
{...field}
|
||||
/>
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
) : (
|
||||
// Show sample list when editing
|
||||
editingProfileId && (
|
||||
// Editing mode
|
||||
editingProfileId &&
|
||||
editingProfile &&
|
||||
(editingProfile.voice_type === 'preset' ? (
|
||||
<div className="space-y-4 pt-4">
|
||||
<div className="rounded-lg border border-border p-4 space-y-3">
|
||||
<div className="text-sm font-medium text-muted-foreground">
|
||||
Built-in Voice
|
||||
</div>
|
||||
<div className="flex items-center gap-3">
|
||||
<div className="text-lg font-semibold">
|
||||
{presetVoices.find(
|
||||
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
|
||||
)?.name ?? editingProfile.preset_voice_id}
|
||||
</div>
|
||||
<Badge variant="secondary" className="text-xs">
|
||||
{editingProfile.preset_engine}
|
||||
</Badge>
|
||||
</div>
|
||||
{(() => {
|
||||
const voice = presetVoices.find(
|
||||
(v: PresetVoice) => v.voice_id === editingProfile.preset_voice_id,
|
||||
);
|
||||
return voice ? (
|
||||
<div className="flex gap-1.5">
|
||||
<Badge variant="outline" className="text-xs">
|
||||
{voice.gender}
|
||||
</Badge>
|
||||
<Badge variant="outline" className="text-xs">
|
||||
{voice.language}
|
||||
</Badge>
|
||||
</div>
|
||||
) : null;
|
||||
})()}
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
This profile uses a built-in voice. The voice cannot be changed after
|
||||
creation.
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<div>
|
||||
<SampleList profileId={editingProfileId} />
|
||||
</div>
|
||||
)
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Right column: Profile info */}
|
||||
<div className="space-y-4">
|
||||
<div className="space-y-4 overflow-y-auto min-h-0">
|
||||
{/* Avatar Upload */}
|
||||
<FormField
|
||||
control={form.control}
|
||||
@@ -924,6 +1179,36 @@ export function ProfileForm() {
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormItem>
|
||||
<FormLabel>Default Engine</FormLabel>
|
||||
<Select
|
||||
value={defaultEngine || '_none'}
|
||||
onValueChange={(v) => {
|
||||
setDefaultEngine(v === '_none' ? '' : v);
|
||||
}}
|
||||
disabled={
|
||||
voiceSource === 'builtin' || editingProfile?.voice_type === 'preset'
|
||||
}
|
||||
>
|
||||
<FormControl>
|
||||
<SelectTrigger>
|
||||
<SelectValue placeholder="No preference" />
|
||||
</SelectTrigger>
|
||||
</FormControl>
|
||||
<SelectContent>
|
||||
<SelectItem value="_none">No preference</SelectItem>
|
||||
{availableDefaultEngines.map((option) => (
|
||||
<SelectItem key={option.value} value={option.value}>
|
||||
{option.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Auto-selects this engine when the profile is chosen.
|
||||
</p>
|
||||
</FormItem>
|
||||
|
||||
{editingProfileId && (
|
||||
<div className="space-y-2">
|
||||
<FormLabel>Default Effects</FormLabel>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Mic, Sparkles } from 'lucide-react';
|
||||
import { Info, Mic, Sparkles } from 'lucide-react';
|
||||
import { useEffect, useRef } from 'react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent } from '@/components/ui/card';
|
||||
import { useProfiles } from '@/lib/hooks/useProfiles';
|
||||
@@ -6,9 +7,36 @@ import { useUIStore } from '@/stores/uiStore';
|
||||
import { ProfileCard } from './ProfileCard';
|
||||
import { ProfileForm } from './ProfileForm';
|
||||
|
||||
/** Engines that use preset (built-in) voices instead of cloned profiles. */
|
||||
const PRESET_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
|
||||
|
||||
export function ProfileList() {
|
||||
const { data: profiles, isLoading, error } = useProfiles();
|
||||
const setDialogOpen = useUIStore((state) => state.setProfileDialogOpen);
|
||||
const selectedEngine = useUIStore((state) => state.selectedEngine);
|
||||
const selectedProfileId = useUIStore((state) => state.selectedProfileId);
|
||||
const cardRefs = useRef<Map<string, HTMLDivElement>>(new Map());
|
||||
|
||||
// Scroll to the selected profile after engine/sort changes
|
||||
useEffect(() => {
|
||||
if (!selectedProfileId) return;
|
||||
let timeoutId: ReturnType<typeof setTimeout> | null = null;
|
||||
const rafId = requestAnimationFrame(() => {
|
||||
const el = cardRefs.current.get(selectedProfileId);
|
||||
if (!el) return;
|
||||
|
||||
// Temporarily apply scroll-margin so it doesn't land flush at the top
|
||||
el.style.scrollMarginTop = '180px';
|
||||
el.scrollIntoView({ behavior: 'smooth', block: 'nearest', inline: 'nearest' });
|
||||
timeoutId = setTimeout(() => {
|
||||
el.style.scrollMarginTop = '';
|
||||
}, 500);
|
||||
});
|
||||
return () => {
|
||||
cancelAnimationFrame(rafId);
|
||||
if (timeoutId) clearTimeout(timeoutId);
|
||||
};
|
||||
}, [selectedProfileId, selectedEngine]);
|
||||
|
||||
if (isLoading) {
|
||||
return null;
|
||||
@@ -23,6 +51,20 @@ export function ProfileList() {
|
||||
}
|
||||
|
||||
const allProfiles = profiles || [];
|
||||
const isPresetEngine = PRESET_ENGINES.has(selectedEngine);
|
||||
|
||||
/** Whether a profile is supported by the currently selected engine. */
|
||||
const isSupported = (p: (typeof allProfiles)[number]) =>
|
||||
isPresetEngine
|
||||
? p.voice_type === 'preset' && p.preset_engine === selectedEngine
|
||||
: p.voice_type !== 'preset';
|
||||
|
||||
// Sort so supported profiles come first
|
||||
const sortedProfiles = [...allProfiles].sort(
|
||||
(a, b) => (isSupported(a) ? 0 : 1) - (isSupported(b) ? 0 : 1),
|
||||
);
|
||||
|
||||
const hasUnsupported = sortedProfiles.some((p) => !isSupported(p));
|
||||
|
||||
return (
|
||||
<div className="flex flex-col">
|
||||
@@ -42,11 +84,24 @@ export function ProfileList() {
|
||||
</Card>
|
||||
) : (
|
||||
<div className="flex gap-4 overflow-x-auto p-1 pb-1 lg:grid lg:grid-cols-3 lg:auto-rows-auto lg:overflow-x-visible lg:pb-[150px]">
|
||||
{allProfiles.map((profile) => (
|
||||
<div key={profile.id} className="shrink-0 w-[200px] lg:w-auto lg:shrink">
|
||||
<ProfileCard profile={profile} />
|
||||
{sortedProfiles.map((profile) => (
|
||||
<div
|
||||
key={profile.id}
|
||||
className="shrink-0 w-[200px] lg:w-auto lg:shrink"
|
||||
ref={(el) => {
|
||||
if (el) cardRefs.current.set(profile.id, el);
|
||||
else cardRefs.current.delete(profile.id);
|
||||
}}
|
||||
>
|
||||
<ProfileCard profile={profile} disabled={!isSupported(profile)} />
|
||||
</div>
|
||||
))}
|
||||
{hasUnsupported && (
|
||||
<div className="col-span-full flex items-center gap-2 text-xs text-muted-foreground py-2">
|
||||
<Info className="h-3.5 w-3.5 shrink-0" />
|
||||
<span>Only supported voice profiles can be selected for the current model.</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
@@ -111,4 +111,4 @@ export {
|
||||
AlertDialogDescription,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -16,7 +16,7 @@ const SelectTrigger = React.forwardRef<
|
||||
<SelectPrimitive.Trigger
|
||||
ref={ref}
|
||||
className={cn(
|
||||
'flex h-10 w-full items-center justify-between rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background placeholder:text-muted-foreground focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50 [&>span]:line-clamp-1',
|
||||
'flex h-10 w-full items-center justify-between rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background placeholder:text-muted-foreground focus:outline-none focus:bg-muted/50 disabled:cursor-not-allowed disabled:opacity-50 [&>span]:line-clamp-1',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import * as React from 'react';
|
||||
import * as SliderPrimitive from '@radix-ui/react-slider';
|
||||
import * as React from 'react';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
|
||||
const Slider = React.forwardRef<
|
||||
@@ -14,7 +14,7 @@ const Slider = React.forwardRef<
|
||||
<SliderPrimitive.Track className="relative h-2 w-full grow overflow-hidden rounded-full bg-secondary">
|
||||
<SliderPrimitive.Range className="absolute h-full bg-primary" />
|
||||
</SliderPrimitive.Track>
|
||||
<SliderPrimitive.Thumb className="block h-5 w-5 rounded-full border-2 border-primary bg-background ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 translate-x-0.5" />
|
||||
<SliderPrimitive.Thumb className="block h-0 w-0 outline-none disabled:pointer-events-none disabled:opacity-50 after:block after:h-5 after:w-5 after:rounded-full after:border-2 after:border-primary after:bg-background after:ring-offset-background after:transition-colors after:absolute after:top-1/2 after:left-1/2 after:-translate-x-1/2 after:-translate-y-1/2 focus-visible:after:ring-2 focus-visible:after:ring-ring focus-visible:after:ring-offset-2" />
|
||||
</SliderPrimitive.Root>
|
||||
));
|
||||
Slider.displayName = SliderPrimitive.Root.displayName;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { usePlayerStore } from '@/stores/playerStore';
|
||||
import {
|
||||
Toast,
|
||||
ToastClose,
|
||||
@@ -10,6 +11,7 @@ import { useToast } from './use-toast';
|
||||
|
||||
export function Toaster() {
|
||||
const { toasts } = useToast();
|
||||
const isPlayerOpen = !!usePlayerStore((s) => s.audioUrl);
|
||||
|
||||
return (
|
||||
<ToastProvider>
|
||||
@@ -23,7 +25,7 @@ export function Toaster() {
|
||||
<ToastClose />
|
||||
</Toast>
|
||||
))}
|
||||
<ToastViewport />
|
||||
<ToastViewport className={isPlayerOpen ? 'sm:bottom-44' : ''} />
|
||||
</ToastProvider>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import * as React from 'react';
|
||||
import { cn } from '@/lib/utils/cn';
|
||||
|
||||
export interface ToggleProps {
|
||||
checked?: boolean;
|
||||
onCheckedChange?: (checked: boolean) => void;
|
||||
disabled?: boolean;
|
||||
className?: string;
|
||||
id?: string;
|
||||
}
|
||||
|
||||
const Toggle = React.forwardRef<HTMLButtonElement, ToggleProps>(
|
||||
({ checked = false, onCheckedChange, disabled = false, className, id, ...props }, ref) => {
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
ref={ref}
|
||||
id={id}
|
||||
role="switch"
|
||||
aria-checked={checked}
|
||||
disabled={disabled}
|
||||
onClick={() => {
|
||||
if (!disabled && onCheckedChange) {
|
||||
onCheckedChange(!checked);
|
||||
}
|
||||
}}
|
||||
className={cn(
|
||||
'relative inline-flex h-5 w-9 shrink-0 items-center rounded-full transition-colors',
|
||||
'focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2',
|
||||
checked ? 'bg-accent' : 'bg-muted-foreground/25',
|
||||
disabled ? 'opacity-50 cursor-not-allowed' : 'cursor-pointer',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
>
|
||||
<span
|
||||
className={cn(
|
||||
'pointer-events-none block h-4 w-4 rounded-full bg-white shadow-sm transition-transform',
|
||||
checked ? 'translate-x-[18px]' : 'translate-x-[2px]',
|
||||
)}
|
||||
/>
|
||||
</button>
|
||||
);
|
||||
},
|
||||
);
|
||||
Toggle.displayName = 'Toggle';
|
||||
|
||||
export { Toggle };
|
||||
Vendored
+5
@@ -1,3 +1,8 @@
|
||||
interface Window {
|
||||
__voiceboxServerStartedByApp?: boolean;
|
||||
}
|
||||
|
||||
declare module 'virtual:changelog' {
|
||||
const raw: string;
|
||||
export default raw;
|
||||
}
|
||||
|
||||
@@ -5,7 +5,15 @@ import type { UpdateStatus } from '@/platform/types';
|
||||
// Re-export UpdateStatus for backwards compatibility
|
||||
export type { UpdateStatus };
|
||||
|
||||
export function useAutoUpdater(checkOnMount = false) {
|
||||
interface UseAutoUpdaterOptions {
|
||||
checkOnMount?: boolean;
|
||||
showToast?: boolean;
|
||||
}
|
||||
|
||||
export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false) {
|
||||
const { checkOnMount } =
|
||||
typeof options === 'boolean' ? { checkOnMount: options } : { checkOnMount: options.checkOnMount ?? false };
|
||||
|
||||
const platform = usePlatform();
|
||||
const [status, setStatus] = useState<UpdateStatus>(platform.updater.getStatus());
|
||||
const hasCheckedRef = useRef(false);
|
||||
@@ -38,10 +46,11 @@ export function useAutoUpdater(checkOnMount = false) {
|
||||
useEffect(() => {
|
||||
if (checkOnMount && platform.metadata.isTauri && !hasCheckedRef.current) {
|
||||
hasCheckedRef.current = true;
|
||||
checkForUpdates();
|
||||
checkForUpdates().catch((error) => {
|
||||
console.error('Auto update check failed:', error);
|
||||
});
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
|
||||
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
|
||||
|
||||
return {
|
||||
status,
|
||||
|
||||
@@ -73,7 +73,7 @@ export function useAutoUpdater(options: boolean | UseAutoUpdaterOptions = false)
|
||||
}
|
||||
// Empty dependency array - only run once on mount
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [platform.metadata.isTauricheckOnMountcheckForUpdates]);
|
||||
}, [checkOnMount, checkForUpdates, platform.metadata.isTauri]);
|
||||
|
||||
// Show toast when update is available
|
||||
useEffect(() => {
|
||||
|
||||
+55
-13
@@ -17,6 +17,7 @@ import type {
|
||||
HistoryResponse,
|
||||
ModelDownloadRequest,
|
||||
ModelStatusListResponse,
|
||||
PresetVoice,
|
||||
ProfileSampleResponse,
|
||||
StoryCreate,
|
||||
StoryDetailResponse,
|
||||
@@ -32,8 +33,24 @@ import type {
|
||||
TranscriptionResponse,
|
||||
VoiceProfileCreate,
|
||||
VoiceProfileResponse,
|
||||
WhisperModelSize,
|
||||
} from './types';
|
||||
|
||||
function formatErrorDetail(detail: unknown, fallback: string): string {
|
||||
if (typeof detail === 'string') return detail;
|
||||
if (Array.isArray(detail)) {
|
||||
return detail
|
||||
.map((e: Record<string, unknown>) => e.msg || e.message || JSON.stringify(e))
|
||||
.join('; ');
|
||||
}
|
||||
if (detail && typeof detail === 'object') {
|
||||
const obj = detail as Record<string, unknown>;
|
||||
if (typeof obj.message === 'string') return obj.message;
|
||||
return JSON.stringify(detail);
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
class ApiClient {
|
||||
private getBaseUrl(): string {
|
||||
const serverUrl = useServerStore.getState().serverUrl;
|
||||
@@ -54,7 +71,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -81,6 +98,10 @@ class ApiClient {
|
||||
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`);
|
||||
}
|
||||
|
||||
async listPresetVoices(engine: string): Promise<{ engine: string; voices: PresetVoice[] }> {
|
||||
return this.request<{ engine: string; voices: PresetVoice[] }>(`/profiles/presets/${engine}`);
|
||||
}
|
||||
|
||||
async updateProfile(profileId: string, data: VoiceProfileCreate): Promise<VoiceProfileResponse> {
|
||||
return this.request<VoiceProfileResponse>(`/profiles/${profileId}`, {
|
||||
method: 'PUT',
|
||||
@@ -113,7 +134,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -147,7 +168,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.blob();
|
||||
@@ -167,7 +188,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -187,7 +208,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -213,6 +234,12 @@ class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async cancelGeneration(generationId: string): Promise<{ message: string }> {
|
||||
return this.request<{ message: string }>(`/generate/${generationId}/cancel`, {
|
||||
method: 'POST',
|
||||
});
|
||||
}
|
||||
|
||||
async regenerateGeneration(generationId: string): Promise<GenerationResponse> {
|
||||
return this.request<GenerationResponse>(`/generate/${generationId}/regenerate`, {
|
||||
method: 'POST',
|
||||
@@ -249,6 +276,12 @@ class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async clearFailedGenerations(): Promise<{ deleted: number }> {
|
||||
return this.request<{ deleted: number }>(`/history/failed`, {
|
||||
method: 'DELETE',
|
||||
});
|
||||
}
|
||||
|
||||
async exportGeneration(generationId: string): Promise<Blob> {
|
||||
const url = `${this.getBaseUrl()}/history/${generationId}/export`;
|
||||
const response = await fetch(url);
|
||||
@@ -257,7 +290,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.blob();
|
||||
@@ -271,7 +304,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.blob();
|
||||
@@ -297,7 +330,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -318,12 +351,19 @@ class ApiClient {
|
||||
}
|
||||
|
||||
// Transcription
|
||||
async transcribeAudio(file: File, language?: LanguageCode): Promise<TranscriptionResponse> {
|
||||
async transcribeAudio(
|
||||
file: File,
|
||||
language?: LanguageCode,
|
||||
model?: WhisperModelSize,
|
||||
): Promise<TranscriptionResponse> {
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
if (language) {
|
||||
formData.append('language', language);
|
||||
}
|
||||
if (model) {
|
||||
formData.append('model', model);
|
||||
}
|
||||
|
||||
const url = `${this.getBaseUrl()}/transcribe`;
|
||||
const response = await fetch(url, {
|
||||
@@ -335,7 +375,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.json();
|
||||
@@ -350,7 +390,9 @@ class ApiClient {
|
||||
return this.request<{ path: string }>('/models/cache-dir');
|
||||
}
|
||||
|
||||
async migrateModels(destination: string): Promise<{ source: string; destination: string }> {
|
||||
async migrateModels(
|
||||
destination: string,
|
||||
): Promise<{ source: string; destination: string; moved: number; errors: string[] }> {
|
||||
return this.request('/models/migrate', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ destination }),
|
||||
@@ -608,7 +650,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.blob();
|
||||
@@ -705,7 +747,7 @@ class ApiClient {
|
||||
const error = await response.json().catch(() => ({
|
||||
detail: response.statusText,
|
||||
}));
|
||||
throw new Error(error.detail || `HTTP error! status: ${response.status}`);
|
||||
throw new Error(formatErrorDetail(error.detail, `HTTP error! status: ${response.status}`));
|
||||
}
|
||||
|
||||
return response.blob();
|
||||
|
||||
@@ -1,10 +1,17 @@
|
||||
// API Types matching backend Pydantic models
|
||||
import type { LanguageCode } from '@/lib/constants/languages';
|
||||
|
||||
export type VoiceType = 'cloned' | 'preset' | 'designed';
|
||||
|
||||
export interface VoiceProfileCreate {
|
||||
name: string;
|
||||
description?: string;
|
||||
language: LanguageCode;
|
||||
voice_type?: VoiceType;
|
||||
preset_engine?: string;
|
||||
preset_voice_id?: string;
|
||||
design_prompt?: string;
|
||||
default_engine?: string;
|
||||
}
|
||||
|
||||
export interface VoiceProfileResponse {
|
||||
@@ -14,12 +21,24 @@ export interface VoiceProfileResponse {
|
||||
language: string;
|
||||
avatar_path?: string;
|
||||
effects_chain?: EffectConfig[];
|
||||
voice_type: VoiceType;
|
||||
preset_engine?: string;
|
||||
preset_voice_id?: string;
|
||||
design_prompt?: string;
|
||||
default_engine?: string;
|
||||
generation_count: number;
|
||||
sample_count: number;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface PresetVoice {
|
||||
voice_id: string;
|
||||
name: string;
|
||||
gender: 'male' | 'female';
|
||||
language: string;
|
||||
}
|
||||
|
||||
export interface ProfileSampleCreate {
|
||||
reference_text: string;
|
||||
}
|
||||
@@ -42,8 +61,15 @@ export interface GenerationRequest {
|
||||
text: string;
|
||||
language: LanguageCode;
|
||||
seed?: number;
|
||||
model_size?: '1.7B' | '0.6B';
|
||||
engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo';
|
||||
model_size?: '1.7B' | '0.6B' | '1B' | '3B';
|
||||
engine?:
|
||||
| 'qwen'
|
||||
| 'qwen_custom_voice'
|
||||
| 'luxtts'
|
||||
| 'chatterbox'
|
||||
| 'chatterbox_turbo'
|
||||
| 'tada'
|
||||
| 'kokoro';
|
||||
instruct?: string;
|
||||
max_chunk_chars?: number;
|
||||
crossfade_ms?: number;
|
||||
@@ -73,7 +99,7 @@ export interface GenerationResponse {
|
||||
instruct?: string;
|
||||
engine?: string;
|
||||
model_size?: string;
|
||||
status: 'generating' | 'completed' | 'failed';
|
||||
status: 'loading_model' | 'generating' | 'completed' | 'failed';
|
||||
error?: string;
|
||||
is_favorited?: boolean;
|
||||
created_at: string;
|
||||
@@ -99,8 +125,11 @@ export interface HistoryListResponse {
|
||||
total: number;
|
||||
}
|
||||
|
||||
export type WhisperModelSize = 'base' | 'small' | 'medium' | 'large' | 'turbo';
|
||||
|
||||
export interface TranscriptionRequest {
|
||||
language?: LanguageCode;
|
||||
model?: WhisperModelSize;
|
||||
}
|
||||
|
||||
export interface TranscriptionResponse {
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
* LuxTTS is English-only.
|
||||
* Chatterbox Multilingual supports 23 languages.
|
||||
* Chatterbox Turbo is English-only.
|
||||
* Kokoro supports 8 languages.
|
||||
*/
|
||||
|
||||
/** All languages that any engine supports. */
|
||||
@@ -66,6 +67,9 @@ export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
|
||||
'zh',
|
||||
],
|
||||
chatterbox_turbo: ['en'],
|
||||
tada: ['en', 'ar', 'zh', 'de', 'es', 'fr', 'it', 'ja', 'pl', 'pt'],
|
||||
kokoro: ['en', 'es', 'fr', 'hi', 'it', 'pt', 'ja', 'zh'],
|
||||
qwen_custom_voice: ['zh', 'en', 'ja', 'ko', 'de', 'fr', 'ru', 'pt', 'es', 'it'],
|
||||
} as const;
|
||||
|
||||
/** Helper: get language options for a given engine. */
|
||||
|
||||
@@ -10,14 +10,25 @@ import { useGeneration } from '@/lib/hooks/useGeneration';
|
||||
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
|
||||
import { useGenerationStore } from '@/stores/generationStore';
|
||||
import { useServerStore } from '@/stores/serverStore';
|
||||
import { useUIStore } from '@/stores/uiStore';
|
||||
|
||||
const generationSchema = z.object({
|
||||
text: z.string().min(1, '').max(50000),
|
||||
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
|
||||
seed: z.number().int().optional(),
|
||||
modelSize: z.enum(['1.7B', '0.6B']).optional(),
|
||||
modelSize: z.enum(['1.7B', '0.6B', '1B', '3B']).optional(),
|
||||
instruct: z.string().max(500).optional(),
|
||||
engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo']).optional(),
|
||||
engine: z
|
||||
.enum([
|
||||
'qwen',
|
||||
'qwen_custom_voice',
|
||||
'luxtts',
|
||||
'chatterbox',
|
||||
'chatterbox_turbo',
|
||||
'tada',
|
||||
'kokoro',
|
||||
])
|
||||
.optional(),
|
||||
});
|
||||
|
||||
export type GenerationFormValues = z.infer<typeof generationSchema>;
|
||||
@@ -35,6 +46,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
const maxChunkChars = useServerStore((state) => state.maxChunkChars);
|
||||
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
|
||||
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
|
||||
const selectedEngine = useUIStore((state) => state.selectedEngine);
|
||||
const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null);
|
||||
const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null);
|
||||
|
||||
@@ -52,7 +64,7 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
seed: undefined,
|
||||
modelSize: '1.7B',
|
||||
instruct: '',
|
||||
engine: 'qwen',
|
||||
engine: (selectedEngine as GenerationFormValues['engine']) || 'qwen',
|
||||
...options.defaultValues,
|
||||
},
|
||||
});
|
||||
@@ -79,7 +91,15 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
? 'chatterbox-tts'
|
||||
: engine === 'chatterbox_turbo'
|
||||
? 'chatterbox-turbo'
|
||||
: `qwen-tts-${data.modelSize}`;
|
||||
: engine === 'tada'
|
||||
? data.modelSize === '3B'
|
||||
? 'tada-3b-ml'
|
||||
: 'tada-1b'
|
||||
: engine === 'kokoro'
|
||||
? 'kokoro'
|
||||
: engine === 'qwen_custom_voice'
|
||||
? `qwen-custom-voice-${data.modelSize}`
|
||||
: `qwen-tts-${data.modelSize}`;
|
||||
const displayName =
|
||||
engine === 'luxtts'
|
||||
? 'LuxTTS'
|
||||
@@ -87,9 +107,19 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
? 'Chatterbox TTS'
|
||||
: engine === 'chatterbox_turbo'
|
||||
? 'Chatterbox Turbo'
|
||||
: data.modelSize === '1.7B'
|
||||
? 'Qwen TTS 1.7B'
|
||||
: 'Qwen TTS 0.6B';
|
||||
: engine === 'tada'
|
||||
? data.modelSize === '3B'
|
||||
? 'TADA 3B Multilingual'
|
||||
: 'TADA 1B'
|
||||
: engine === 'kokoro'
|
||||
? 'Kokoro 82M'
|
||||
: engine === 'qwen_custom_voice'
|
||||
? data.modelSize === '1.7B'
|
||||
? 'Qwen CustomVoice 1.7B'
|
||||
: 'Qwen CustomVoice 0.6B'
|
||||
: data.modelSize === '1.7B'
|
||||
? 'Qwen TTS 1.7B'
|
||||
: 'Qwen TTS 0.6B';
|
||||
|
||||
// Check if model needs downloading
|
||||
try {
|
||||
@@ -104,7 +134,11 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
console.error('Failed to check model status:', error);
|
||||
}
|
||||
|
||||
const isQwen = engine === 'qwen';
|
||||
const hasModelSizes =
|
||||
engine === 'qwen' || engine === 'qwen_custom_voice' || engine === 'tada';
|
||||
// Only Qwen CustomVoice actually honors the instruct kwarg at model level.
|
||||
// Base Qwen3-TTS accepts the kwarg but ignores it.
|
||||
const supportsInstruct = engine === 'qwen_custom_voice';
|
||||
const effectsChain = options.getEffectsChain?.();
|
||||
// This now returns immediately with status="generating"
|
||||
const result = await generation.mutateAsync({
|
||||
@@ -112,9 +146,9 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
|
||||
text: data.text,
|
||||
language: data.language,
|
||||
seed: data.seed,
|
||||
model_size: isQwen ? data.modelSize : undefined,
|
||||
model_size: hasModelSizes ? data.modelSize : undefined,
|
||||
engine,
|
||||
instruct: isQwen ? data.instruct || undefined : undefined,
|
||||
instruct: supportsInstruct ? data.instruct || undefined : undefined,
|
||||
max_chunk_chars: maxChunkChars,
|
||||
crossfade_ms: crossfadeMs,
|
||||
normalize: normalizeAudio,
|
||||
|
||||
@@ -8,7 +8,7 @@ import { useServerStore } from '@/stores/serverStore';
|
||||
|
||||
interface GenerationStatusEvent {
|
||||
id: string;
|
||||
status: 'generating' | 'completed' | 'failed' | 'not_found';
|
||||
status: 'loading_model' | 'generating' | 'completed' | 'failed' | 'not_found';
|
||||
duration?: number;
|
||||
error?: string;
|
||||
}
|
||||
@@ -75,8 +75,8 @@ export function useGenerationProgress() {
|
||||
currentSources.delete(id);
|
||||
removePendingGeneration(id);
|
||||
|
||||
// Refresh history to pick up the completed generation
|
||||
queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
// Refetch history to pick up the completed generation
|
||||
queryClient.refetchQueries({ queryKey: ['history'] });
|
||||
|
||||
// If this generation was queued for a story, add it now
|
||||
const storyId = removePendingStoryAdd(id);
|
||||
@@ -120,7 +120,7 @@ export function useGenerationProgress() {
|
||||
removePendingGeneration(id);
|
||||
removePendingStoryAdd(id);
|
||||
|
||||
queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
queryClient.refetchQueries({ queryKey: ['history'] });
|
||||
|
||||
toast({
|
||||
title: data.status === 'not_found' ? 'Generation not found' : 'Generation failed',
|
||||
@@ -134,11 +134,12 @@ export function useGenerationProgress() {
|
||||
};
|
||||
|
||||
source.onerror = () => {
|
||||
// EventSource auto-reconnects, but if we get repeated errors
|
||||
// just clean up
|
||||
// SSE connection dropped — clean up and refresh history so any
|
||||
// completed/failed generation still appears in the list
|
||||
source.close();
|
||||
currentSources.delete(id);
|
||||
removePendingGeneration(id);
|
||||
queryClient.refetchQueries({ queryKey: ['history'] });
|
||||
};
|
||||
|
||||
currentSources.set(id, source);
|
||||
|
||||
@@ -29,6 +29,17 @@ export function useDeleteGeneration() {
|
||||
});
|
||||
}
|
||||
|
||||
export function useClearFailedGenerations() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
return useMutation({
|
||||
mutationFn: () => apiClient.clearFailedGenerations(),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ['history'] });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export function useExportGeneration() {
|
||||
const platform = usePlatform();
|
||||
|
||||
|
||||
@@ -131,7 +131,8 @@ export function useModelDownloadToast({
|
||||
)}
|
||||
</div>
|
||||
),
|
||||
duration: progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
|
||||
duration:
|
||||
progress.status === 'complete' || progress.status === 'error' ? 5000 : Infinity,
|
||||
});
|
||||
|
||||
// Close connection and dismiss toast on completion or error
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useState, useRef, useCallback, useEffect } from 'react';
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { usePlatform } from '@/platform/PlatformContext';
|
||||
|
||||
interface UseSystemAudioCaptureOptions {
|
||||
@@ -26,8 +26,24 @@ export function useSystemAudioCapture({
|
||||
|
||||
// Check if system audio capture is supported
|
||||
useEffect(() => {
|
||||
const supported = platform.audio.isSystemAudioSupported();
|
||||
setIsSupported(supported);
|
||||
let isActive = true;
|
||||
|
||||
void platform.audio
|
||||
.isSystemAudioSupported()
|
||||
.then((supported) => {
|
||||
if (isActive) {
|
||||
setIsSupported(supported);
|
||||
}
|
||||
})
|
||||
.catch(() => {
|
||||
if (isActive) {
|
||||
setIsSupported(false);
|
||||
}
|
||||
});
|
||||
|
||||
return () => {
|
||||
isActive = false;
|
||||
};
|
||||
}, [platform]);
|
||||
|
||||
const startRecording = useCallback(async () => {
|
||||
@@ -94,15 +110,13 @@ export function useSystemAudioCapture({
|
||||
const blob = await platform.audio.stopSystemAudioCapture();
|
||||
|
||||
// Pass the actual recorded duration
|
||||
const recordedDuration = startTimeRef.current
|
||||
? (Date.now() - startTimeRef.current) / 1000
|
||||
const recordedDuration = startTimeRef.current
|
||||
? (Date.now() - startTimeRef.current) / 1000
|
||||
: undefined;
|
||||
onRecordingComplete?.(blob, recordedDuration);
|
||||
} catch (err) {
|
||||
const errorMessage =
|
||||
err instanceof Error
|
||||
? err.message
|
||||
: 'Failed to stop system audio capture.';
|
||||
err instanceof Error ? err.message : 'Failed to stop system audio capture.';
|
||||
setError(errorMessage);
|
||||
}
|
||||
}, [isRecording, onRecordingComplete, platform]);
|
||||
|
||||
@@ -1,10 +1,18 @@
|
||||
import { useMutation } from '@tanstack/react-query';
|
||||
import { apiClient } from '@/lib/api/client';
|
||||
import type { WhisperModelSize } from '@/lib/api/types';
|
||||
import type { LanguageCode } from '@/lib/constants/languages';
|
||||
|
||||
export function useTranscription() {
|
||||
return useMutation({
|
||||
mutationFn: ({ file, language }: { file: File; language?: LanguageCode }) =>
|
||||
apiClient.transcribeAudio(file, language),
|
||||
mutationFn: ({
|
||||
file,
|
||||
language,
|
||||
model,
|
||||
}: {
|
||||
file: File;
|
||||
language?: LanguageCode;
|
||||
model?: WhisperModelSize;
|
||||
}) => apiClient.transcribeAudio(file, language, model),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
import { QueryClient } from '@tanstack/react-query';
|
||||
|
||||
/**
|
||||
* Shared QueryClient instance used across the app.
|
||||
*
|
||||
* Extracted into its own side-effect-free module so it can be imported from
|
||||
* both the React bootstrap (main.tsx) and non-React code (stores, utilities)
|
||||
* without pulling in ReactDOM or other bootstrap side effects.
|
||||
*/
|
||||
export const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
staleTime: 1000 * 60 * 5, // 5 minutes
|
||||
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
|
||||
retry: 1,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,37 @@
|
||||
export interface ChangelogEntry {
|
||||
version: string;
|
||||
date: string | null;
|
||||
body: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a Keep-a-Changelog style markdown string into structured entries.
|
||||
*
|
||||
* Splits on `## [version]` headings and extracts the version + date from each.
|
||||
* The body is the raw markdown between headings (trimmed), with the leading
|
||||
* `# Changelog` title and trailing link references stripped.
|
||||
*/
|
||||
export function parseChangelog(raw: string): ChangelogEntry[] {
|
||||
const entries: ChangelogEntry[] = [];
|
||||
|
||||
// Strip trailing link reference definitions (e.g. [0.1.0]: https://...)
|
||||
const cleaned = raw.replace(/^\[[\w.]+\]:.*$/gm, '').trimEnd();
|
||||
|
||||
// Match `## [version]` or `## [version] - date`
|
||||
const headingRe = /^## \[(.+?)\](?:\s*-\s*(.+))?$/gm;
|
||||
const matches = [...cleaned.matchAll(headingRe)];
|
||||
|
||||
for (let i = 0; i < matches.length; i++) {
|
||||
const match = matches[i];
|
||||
const version = match[1];
|
||||
const date = match[2]?.trim() || null;
|
||||
|
||||
const start = match.index! + match[0].length;
|
||||
const end = i + 1 < matches.length ? matches[i + 1].index! : cleaned.length;
|
||||
const body = cleaned.slice(start, end).trim();
|
||||
|
||||
entries.push({ version, date, body });
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
+2
-12
@@ -1,20 +1,10 @@
|
||||
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
|
||||
import { QueryClientProvider } from '@tanstack/react-query';
|
||||
// import { ReactQueryDevtools } from '@tanstack/react-query-devtools';
|
||||
import React from 'react';
|
||||
import ReactDOM from 'react-dom/client';
|
||||
import App from './App';
|
||||
import './index.css';
|
||||
|
||||
const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
staleTime: 1000 * 60 * 5, // 5 minutes
|
||||
gcTime: 1000 * 60 * 10, // 10 minutes (formerly cacheTime)
|
||||
retry: 1,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
import { queryClient } from './lib/queryClient';
|
||||
|
||||
ReactDOM.createRoot(document.getElementById('root')!).render(
|
||||
<React.StrictMode>
|
||||
|
||||
@@ -9,11 +9,7 @@ export interface PlatformProviderProps {
|
||||
}
|
||||
|
||||
export function PlatformProvider({ platform, children }: PlatformProviderProps) {
|
||||
return (
|
||||
<PlatformContext.Provider value={platform}>
|
||||
{children}
|
||||
</PlatformContext.Provider>
|
||||
);
|
||||
return <PlatformContext.Provider value={platform}>{children}</PlatformContext.Provider>;
|
||||
}
|
||||
|
||||
export function usePlatform(): Platform {
|
||||
|
||||
@@ -42,7 +42,7 @@ export interface AudioDevice {
|
||||
}
|
||||
|
||||
export interface PlatformAudio {
|
||||
isSystemAudioSupported(): boolean;
|
||||
isSystemAudioSupported(): Promise<boolean>;
|
||||
startSystemAudioCapture(maxDurationSecs: number): Promise<void>;
|
||||
stopSystemAudioCapture(): Promise<Blob>;
|
||||
listOutputDevices(): Promise<AudioDevice[]>;
|
||||
@@ -50,12 +50,18 @@ export interface PlatformAudio {
|
||||
stopPlayback(): void;
|
||||
}
|
||||
|
||||
export interface ServerLogEntry {
|
||||
stream: 'stdout' | 'stderr';
|
||||
line: string;
|
||||
}
|
||||
|
||||
export interface PlatformLifecycle {
|
||||
startServer(remote?: boolean, modelsDir?: string | null): Promise<string>;
|
||||
stopServer(): Promise<void>;
|
||||
restartServer(modelsDir?: string | null): Promise<string>;
|
||||
setKeepServerRunning(keep: boolean): Promise<void>;
|
||||
setupWindowCloseHandler(): Promise<void>;
|
||||
subscribeToServerLogs(callback: (entry: ServerLogEntry) => void): () => void;
|
||||
onServerReady?: () => void;
|
||||
}
|
||||
|
||||
|
||||
+72
-6
@@ -1,10 +1,22 @@
|
||||
import { createRootRoute, createRoute, createRouter, Outlet } from '@tanstack/react-router';
|
||||
import {
|
||||
createRootRoute,
|
||||
createRoute,
|
||||
createRouter,
|
||||
Outlet,
|
||||
redirect,
|
||||
} from '@tanstack/react-router';
|
||||
import { AppFrame } from '@/components/AppFrame/AppFrame';
|
||||
import { AudioTab } from '@/components/AudioTab/AudioTab';
|
||||
import { EffectsTab } from '@/components/EffectsTab/EffectsTab';
|
||||
import { MainEditor } from '@/components/MainEditor/MainEditor';
|
||||
import { ModelsTab } from '@/components/ModelsTab/ModelsTab';
|
||||
import { ServerTab } from '@/components/ServerTab/ServerTab';
|
||||
import { AboutPage } from '@/components/ServerTab/AboutPage';
|
||||
import { ChangelogPage } from '@/components/ServerTab/ChangelogPage';
|
||||
import { GeneralPage } from '@/components/ServerTab/GeneralPage';
|
||||
import { GenerationPage } from '@/components/ServerTab/GenerationPage';
|
||||
import { GpuPage } from '@/components/ServerTab/GpuPage';
|
||||
import { LogsPage } from '@/components/ServerTab/LogsPage';
|
||||
import { SettingsLayout } from '@/components/ServerTab/ServerTab';
|
||||
import { Sidebar } from '@/components/Sidebar';
|
||||
import { StoriesTab } from '@/components/StoriesTab/StoriesTab';
|
||||
import { Toaster } from '@/components/ui/toaster';
|
||||
@@ -120,11 +132,57 @@ const modelsRoute = createRoute({
|
||||
component: ModelsTab,
|
||||
});
|
||||
|
||||
// Server route
|
||||
const serverRoute = createRoute({
|
||||
// Settings layout route (parent for sub-tabs)
|
||||
const settingsRoute = createRoute({
|
||||
getParentRoute: () => rootRoute,
|
||||
path: '/settings',
|
||||
component: SettingsLayout,
|
||||
});
|
||||
|
||||
// Settings sub-routes
|
||||
const settingsGeneralRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/',
|
||||
component: GeneralPage,
|
||||
});
|
||||
|
||||
const settingsGenerationRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/generation',
|
||||
component: GenerationPage,
|
||||
});
|
||||
|
||||
const settingsGpuRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/gpu',
|
||||
component: GpuPage,
|
||||
});
|
||||
|
||||
const settingsChangelogRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/changelog',
|
||||
component: ChangelogPage,
|
||||
});
|
||||
|
||||
const settingsLogsRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/logs',
|
||||
component: LogsPage,
|
||||
});
|
||||
|
||||
const settingsAboutRoute = createRoute({
|
||||
getParentRoute: () => settingsRoute,
|
||||
path: '/about',
|
||||
component: AboutPage,
|
||||
});
|
||||
|
||||
// Redirect old /server path to /settings
|
||||
const serverRedirectRoute = createRoute({
|
||||
getParentRoute: () => rootRoute,
|
||||
path: '/server',
|
||||
component: ServerTab,
|
||||
beforeLoad: () => {
|
||||
throw redirect({ to: '/settings' });
|
||||
},
|
||||
});
|
||||
|
||||
// Route tree
|
||||
@@ -135,7 +193,15 @@ const routeTree = rootRoute.addChildren([
|
||||
audioRoute,
|
||||
effectsRoute,
|
||||
modelsRoute,
|
||||
serverRoute,
|
||||
settingsRoute.addChildren([
|
||||
settingsGeneralRoute,
|
||||
settingsGenerationRoute,
|
||||
settingsGpuRoute,
|
||||
settingsLogsRoute,
|
||||
settingsChangelogRoute,
|
||||
settingsAboutRoute,
|
||||
]),
|
||||
serverRedirectRoute,
|
||||
]);
|
||||
|
||||
// Create router
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import { create } from 'zustand';
|
||||
import type { ServerLogEntry } from '@/platform/types';
|
||||
|
||||
const MAX_LOG_ENTRIES = 2000;
|
||||
|
||||
let nextLogEntryId = 0;
|
||||
|
||||
export interface LogEntry extends ServerLogEntry {
|
||||
id: number;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
interface LogStore {
|
||||
entries: LogEntry[];
|
||||
addEntry: (entry: ServerLogEntry) => void;
|
||||
clear: () => void;
|
||||
}
|
||||
|
||||
export const useLogStore = create<LogStore>((set) => ({
|
||||
entries: [],
|
||||
addEntry: (entry) =>
|
||||
set((state) => {
|
||||
const newEntry: LogEntry = { ...entry, id: nextLogEntryId++, timestamp: Date.now() };
|
||||
const entries = [...state.entries, newEntry];
|
||||
if (entries.length > MAX_LOG_ENTRIES) {
|
||||
return { entries: entries.slice(entries.length - MAX_LOG_ENTRIES) };
|
||||
}
|
||||
return { entries };
|
||||
}),
|
||||
clear: () => set({ entries: [] }),
|
||||
}));
|
||||
@@ -1,5 +1,6 @@
|
||||
import { create } from 'zustand';
|
||||
import { persist } from 'zustand/middleware';
|
||||
import { queryClient } from '@/lib/queryClient';
|
||||
|
||||
interface ServerStore {
|
||||
serverUrl: string;
|
||||
@@ -30,11 +31,25 @@ interface ServerStore {
|
||||
setCustomModelsDir: (dir: string | null) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Invalidate all React Query caches so stale data from the previous
|
||||
* server is not shown. Called when the server URL changes.
|
||||
*/
|
||||
function invalidateAllServerData() {
|
||||
queryClient.invalidateQueries();
|
||||
}
|
||||
|
||||
export const useServerStore = create<ServerStore>()(
|
||||
persist(
|
||||
(set) => ({
|
||||
(set, get) => ({
|
||||
serverUrl: 'http://127.0.0.1:17493',
|
||||
setServerUrl: (url) => set({ serverUrl: url }),
|
||||
setServerUrl: (url) => {
|
||||
const prev = get().serverUrl;
|
||||
set({ serverUrl: url });
|
||||
if (url !== prev) {
|
||||
invalidateAllServerData();
|
||||
}
|
||||
},
|
||||
|
||||
isConnected: false,
|
||||
setIsConnected: (connected) => set({ isConnected: connected }),
|
||||
|
||||
@@ -31,6 +31,10 @@ interface UIStore {
|
||||
selectedProfileId: string | null;
|
||||
setSelectedProfileId: (id: string | null) => void;
|
||||
|
||||
// Currently selected engine (synced from generation form)
|
||||
selectedEngine: string;
|
||||
setSelectedEngine: (engine: string) => void;
|
||||
|
||||
// Selected voice in Voices tab inspector
|
||||
selectedVoiceId: string | null;
|
||||
setSelectedVoiceId: (id: string | null) => void;
|
||||
@@ -59,6 +63,9 @@ export const useUIStore = create<UIStore>((set) => ({
|
||||
selectedProfileId: null,
|
||||
setSelectedProfileId: (id) => set({ selectedProfileId: id }),
|
||||
|
||||
selectedEngine: 'qwen',
|
||||
setSelectedEngine: (engine) => set({ selectedEngine: engine }),
|
||||
|
||||
selectedVoiceId: null,
|
||||
setSelectedVoiceId: (id) => set({ selectedVoiceId: id }),
|
||||
|
||||
|
||||
@@ -6,5 +6,5 @@
|
||||
"moduleResolution": "bundler",
|
||||
"allowSyntheticDefaultImports": true
|
||||
},
|
||||
"include": ["vite.config.ts"]
|
||||
"include": ["vite.config.ts", "plugins/**/*.ts"]
|
||||
}
|
||||
|
||||
+2
-1
@@ -2,9 +2,10 @@ import path from 'node:path';
|
||||
import tailwindcss from '@tailwindcss/vite';
|
||||
import react from '@vitejs/plugin-react';
|
||||
import { defineConfig } from 'vite';
|
||||
import { changelogPlugin } from './plugins/changelog';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [tailwindcss(), react()],
|
||||
plugins: [tailwindcss(), react(), changelogPlugin(path.resolve(__dirname, '..'))],
|
||||
resolve: {
|
||||
alias: {
|
||||
'@': path.resolve(__dirname, './src'),
|
||||
|
||||
+107
-434
@@ -1,462 +1,135 @@
|
||||
# voicebox Backend
|
||||
# Voicebox Backend
|
||||
|
||||
Production-quality FastAPI backend for Qwen3-TTS voice cloning.
|
||||
FastAPI server powering voice cloning, speech generation, and audio processing. Runs locally as a Tauri sidecar or standalone via `python -m backend.main`.
|
||||
|
||||
## Features
|
||||
## Running
|
||||
|
||||
- ✅ **Voice Profile Management** - Create, update, delete voice profiles with multi-sample support
|
||||
- ✅ **Voice Cloning** - Generate speech using voice profiles with caching
|
||||
- ✅ **Generation History** - Full history tracking with search and filtering
|
||||
- ✅ **Transcription** - Whisper-based audio transcription
|
||||
- ✅ **Multi-Sample Profiles** - Combine multiple reference samples for better quality
|
||||
- ✅ **Voice Prompt Caching** - Dual memory + disk caching for fast generation
|
||||
- ✅ **Audio Validation** - Automatic validation of reference audio quality
|
||||
- ✅ **Model Management** - Lazy loading and VRAM management
|
||||
```bash
|
||||
# Via justfile (recommended)
|
||||
just dev:server
|
||||
|
||||
# Standalone
|
||||
python -m backend.main --host 127.0.0.1 --port 17493
|
||||
|
||||
# With custom data directory
|
||||
python -m backend.main --data-dir /path/to/data
|
||||
```
|
||||
|
||||
The server auto-initializes the SQLite database on first startup. Models are downloaded from HuggingFace on first use.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
backend/
|
||||
├── main.py # FastAPI app with all routes
|
||||
├── models.py # Pydantic request/response models
|
||||
├── platform_detect.py # Platform detection for backend selection
|
||||
├── tts.py # TTS backend abstraction (delegates to MLX or PyTorch)
|
||||
├── transcribe.py # STT backend abstraction (delegates to MLX or PyTorch)
|
||||
├── backends/ # Backend implementations
|
||||
│ ├── __init__.py # Backend factory and protocols
|
||||
│ ├── mlx_backend.py # MLX backend (Apple Silicon)
|
||||
│ └── pytorch_backend.py # PyTorch backend (Windows/Linux/Intel)
|
||||
├── profiles.py # Voice profile CRUD
|
||||
├── history.py # Generation history
|
||||
├── studio.py # Audio editing (TODO)
|
||||
├── database.py # SQLite ORM
|
||||
└── utils/
|
||||
├── audio.py # Audio processing utilities
|
||||
├── cache.py # Voice prompt caching
|
||||
└── validation.py # Input validation
|
||||
app.py # FastAPI app factory, CORS, lifecycle events
|
||||
main.py # Entry point (imports app, runs uvicorn)
|
||||
config.py # Data directory paths and configuration
|
||||
models.py # Pydantic request/response schemas
|
||||
server.py # Tauri sidecar launcher, parent-pid watchdog
|
||||
|
||||
routes/ # Thin HTTP handlers — validation, delegation, response formatting
|
||||
services/ # Business logic, CRUD, orchestration
|
||||
backends/ # TTS/STT engine implementations (MLX, PyTorch, etc.)
|
||||
database/ # ORM models, session management, migrations, seed data
|
||||
utils/ # Shared utilities (audio, effects, caching, progress tracking)
|
||||
```
|
||||
|
||||
### Backend Selection
|
||||
|
||||
Voicebox automatically selects the best backend based on platform:
|
||||
|
||||
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration (4-5x faster)
|
||||
- **Windows/Linux/Intel Mac**: Uses PyTorch backend (CUDA GPU if available, CPU fallback)
|
||||
|
||||
The backend is detected at runtime via `platform_detect.py`. Both backends implement the same interface, so the API remains consistent across platforms.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Health & Info
|
||||
|
||||
#### `GET /`
|
||||
Root endpoint with version info.
|
||||
|
||||
#### `GET /health`
|
||||
Health check with model status.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"model_loaded": true,
|
||||
"gpu_available": true,
|
||||
"gpu_type": "Metal (Apple Silicon via MLX)",
|
||||
"backend_type": "mlx",
|
||||
"vram_used_mb": null
|
||||
}
|
||||
```
|
||||
|
||||
**Backend Types:**
|
||||
- `"mlx"` - MLX backend (Apple Silicon with Metal acceleration)
|
||||
- `"pytorch"` - PyTorch backend (Windows/Linux/Intel Mac)
|
||||
|
||||
### Voice Profiles
|
||||
|
||||
**Note:** The database is automatically initialized when the server starts. No manual setup required.
|
||||
|
||||
#### `POST /profiles`
|
||||
Create a new voice profile.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "My Voice",
|
||||
"description": "Optional description",
|
||||
"language": "en"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"name": "My Voice",
|
||||
"description": "Optional description",
|
||||
"language": "en",
|
||||
"created_at": "2024-01-01T00:00:00Z",
|
||||
"updated_at": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /profiles`
|
||||
List all voice profiles.
|
||||
|
||||
#### `GET /profiles/{profile_id}`
|
||||
Get a specific profile.
|
||||
|
||||
#### `PUT /profiles/{profile_id}`
|
||||
Update a profile.
|
||||
|
||||
#### `DELETE /profiles/{profile_id}`
|
||||
Delete a profile and all associated samples.
|
||||
|
||||
#### `POST /profiles/{profile_id}/samples`
|
||||
Add a sample to a profile.
|
||||
|
||||
**Form Data:**
|
||||
- `file`: Audio file (WAV, MP3, etc.)
|
||||
- `reference_text`: Transcript of the audio
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "sample-uuid",
|
||||
"profile_id": "profile-uuid",
|
||||
"audio_path": "/path/to/sample.wav",
|
||||
"reference_text": "This is my voice"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /profiles/{profile_id}/samples`
|
||||
List all samples for a profile.
|
||||
|
||||
#### `DELETE /profiles/samples/{sample_id}`
|
||||
Delete a specific sample.
|
||||
|
||||
### Generation
|
||||
|
||||
#### `POST /generate`
|
||||
Generate speech from text using a voice profile.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"profile_id": "uuid",
|
||||
"text": "Hello, this is a test.",
|
||||
"language": "en",
|
||||
"seed": 42
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "generation-uuid",
|
||||
"profile_id": "profile-uuid",
|
||||
"text": "Hello, this is a test.",
|
||||
"language": "en",
|
||||
"audio_path": "/path/to/audio.wav",
|
||||
"duration": 2.5,
|
||||
"seed": 42,
|
||||
"created_at": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### History
|
||||
|
||||
#### `GET /history`
|
||||
List generation history with optional filters.
|
||||
|
||||
**Query Parameters:**
|
||||
- `profile_id` (optional): Filter by profile
|
||||
- `search` (optional): Search in text content
|
||||
- `limit` (default: 50): Results per page
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
#### `GET /history/{generation_id}`
|
||||
Get a specific generation.
|
||||
|
||||
#### `DELETE /history/{generation_id}`
|
||||
Delete a generation.
|
||||
|
||||
#### `GET /history/stats`
|
||||
Get generation statistics.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"total_generations": 100,
|
||||
"total_duration_seconds": 250.5,
|
||||
"generations_by_profile": {
|
||||
"profile-uuid-1": 50,
|
||||
"profile-uuid-2": 50
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Audio Files
|
||||
|
||||
#### `GET /audio/{generation_id}`
|
||||
Download generated audio file.
|
||||
|
||||
Returns WAV file with appropriate headers.
|
||||
|
||||
### Transcription
|
||||
|
||||
#### `POST /transcribe`
|
||||
Transcribe audio file to text.
|
||||
|
||||
**Form Data:**
|
||||
- `file`: Audio file
|
||||
- `language` (optional): Language hint (en or zh)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"text": "Transcribed text here",
|
||||
"duration": 5.5
|
||||
}
|
||||
```
|
||||
|
||||
### Model Management
|
||||
|
||||
#### `POST /models/load`
|
||||
Manually load TTS model.
|
||||
|
||||
**Query Parameters:**
|
||||
- `model_size`: Model size (1.7B or 0.6B)
|
||||
|
||||
#### `POST /models/unload`
|
||||
Unload TTS model to free memory.
|
||||
|
||||
## Database Schema
|
||||
|
||||
### profiles
|
||||
- `id`: UUID primary key
|
||||
- `name`: Profile name (unique)
|
||||
- `description`: Optional description
|
||||
- `language`: Language code (en/zh)
|
||||
- `created_at`: Creation timestamp
|
||||
- `updated_at`: Last update timestamp
|
||||
|
||||
### profile_samples
|
||||
- `id`: UUID primary key
|
||||
- `profile_id`: Foreign key to profiles
|
||||
- `audio_path`: Path to audio file
|
||||
- `reference_text`: Transcript
|
||||
|
||||
### generations
|
||||
- `id`: UUID primary key
|
||||
- `profile_id`: Foreign key to profiles
|
||||
- `text`: Generated text
|
||||
- `language`: Language code
|
||||
- `audio_path`: Path to audio file
|
||||
- `duration`: Duration in seconds
|
||||
- `seed`: Random seed (optional)
|
||||
- `created_at`: Creation timestamp
|
||||
|
||||
### projects
|
||||
- `id`: UUID primary key
|
||||
- `name`: Project name
|
||||
- `data`: JSON data
|
||||
- `created_at`: Creation timestamp
|
||||
- `updated_at`: Last update timestamp
|
||||
|
||||
## File Structure
|
||||
### Request flow
|
||||
|
||||
```
|
||||
data/
|
||||
├── profiles/
|
||||
│ └── {profile_id}/
|
||||
│ ├── {sample_id}.wav
|
||||
│ └── ...
|
||||
├── generations/
|
||||
│ └── {generation_id}.wav
|
||||
├── cache/
|
||||
│ └── {hash}.prompt
|
||||
├── projects/
|
||||
│ └── {project_id}.json
|
||||
└── voicebox.db
|
||||
HTTP request
|
||||
-> routes/ (validate input, parse params)
|
||||
-> services/ (business logic, database queries, orchestration)
|
||||
-> backends/ (TTS/STT inference)
|
||||
-> utils/ (audio processing, effects, caching)
|
||||
```
|
||||
|
||||
## Setup
|
||||
Route handlers are intentionally thin. They validate input, delegate to a service function, and format the response. All business logic lives in `services/`.
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
**Note:** On Apple Silicon, also install MLX dependencies for faster inference:
|
||||
```bash
|
||||
pip install -r requirements-mlx.txt
|
||||
```
|
||||
|
||||
### 2. Download Models (Automatic)
|
||||
|
||||
The Qwen3-TTS models are automatically downloaded from HuggingFace Hub on first use, similar to how Whisper models work.
|
||||
|
||||
**No manual download required!** The models will be cached locally after the first download.
|
||||
|
||||
Available models:
|
||||
- **1.7B** (recommended): `Qwen/Qwen3-TTS-12Hz-1.7B-Base` (~4GB)
|
||||
- **0.6B** (faster): `Qwen/Qwen3-TTS-12Hz-0.6B-Base` (~2GB)
|
||||
|
||||
**Note:** The first generation will take longer as the model downloads. Subsequent generations will use the cached model.
|
||||
|
||||
#### Manual Download (Optional)
|
||||
|
||||
If you prefer to download models manually or have limited internet during runtime:
|
||||
|
||||
```bash
|
||||
# Install huggingface-cli
|
||||
pip install huggingface_hub
|
||||
|
||||
# Download 1.7B model
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base
|
||||
|
||||
# Or use Python
|
||||
python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen3-TTS-12Hz-1.7B-Base')"
|
||||
```
|
||||
|
||||
Models are cached in `~/.cache/huggingface/hub/` by default.
|
||||
|
||||
### 4. Run Server
|
||||
|
||||
```bash
|
||||
# Development (local only)
|
||||
python -m backend.main
|
||||
|
||||
# Production (allow remote access)
|
||||
python -m backend.main --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
The desktop app, web client, and current development workflow use `http://localhost:17493` by default.
|
||||
If you launch the backend manually with a different host or port, substitute that address in the examples below.
|
||||
|
||||
### Creating a Voice Profile
|
||||
|
||||
```bash
|
||||
# 1. Create profile
|
||||
curl -X POST http://localhost:17493/profiles \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "My Voice", "language": "en"}'
|
||||
|
||||
# Response: {"id": "abc-123", ...}
|
||||
|
||||
# 2. Add sample
|
||||
curl -X POST http://localhost:17493/profiles/abc-123/samples \
|
||||
-F "[email protected]" \
|
||||
-F "reference_text=This is my voice sample"
|
||||
```
|
||||
|
||||
### Generating Speech
|
||||
### Key modules
|
||||
|
||||
**services/generation.py** -- Single `run_generation()` function that handles all three generation modes (generate, retry, regenerate). Manages model loading, voice prompt creation, chunked inference, normalization, effects, and version persistence.
|
||||
|
||||
**services/task_queue.py** -- Serial generation queue. Ensures only one GPU inference runs at a time. Background tasks are tracked to prevent garbage collection.
|
||||
|
||||
**backends/__init__.py** -- Protocol definitions (`TTSBackend`, `STTBackend`), model config registry, and factory functions. Adding a new engine means implementing the protocol and registering a config entry.
|
||||
|
||||
**backends/base.py** -- Shared utilities used across all engine implementations: HuggingFace cache checks, device detection, voice prompt combination, progress tracking.
|
||||
|
||||
**database/** -- SQLAlchemy ORM models with a re-exporting `__init__.py` for backward compatibility. Migrations run automatically on startup.
|
||||
|
||||
### Backend selection
|
||||
|
||||
The server detects the best inference backend at startup:
|
||||
|
||||
| Platform | Backend | Acceleration |
|
||||
|----------|---------|-------------|
|
||||
| macOS (Apple Silicon) | MLX | Metal / Neural Engine |
|
||||
| Windows / Linux (NVIDIA) | PyTorch | CUDA |
|
||||
| Linux (AMD) | PyTorch | ROCm |
|
||||
| Intel Arc | PyTorch | IPEX / XPU |
|
||||
| Windows (any GPU) | PyTorch | DirectML |
|
||||
| Any | PyTorch | CPU fallback |
|
||||
|
||||
Detection is handled by `utils/platform_detect.py`. Both backends implement the same `TTSBackend` protocol, so the API layer is engine-agnostic.
|
||||
|
||||
## API
|
||||
|
||||
90 endpoints organized by domain. Full interactive documentation available at `http://localhost:17493/docs` when the server is running.
|
||||
|
||||
| Domain | Prefix | Description |
|
||||
|--------|--------|-------------|
|
||||
| Health | `/`, `/health` | Server status, GPU info, filesystem checks |
|
||||
| Profiles | `/profiles` | Voice profile CRUD, samples, avatars, import/export |
|
||||
| Channels | `/channels` | Audio channel management and voice assignment |
|
||||
| Generation | `/generate` | TTS generation, retry, regenerate, status SSE |
|
||||
| History | `/history` | Generation history, search, favorites, export |
|
||||
| Transcription | `/transcribe` | Whisper-based audio-to-text |
|
||||
| Stories | `/stories` | Multi-track timeline editor, audio export |
|
||||
| Effects | `/effects` | Effect presets, preview, version management |
|
||||
| Audio | `/audio`, `/samples` | Audio file serving |
|
||||
| Models | `/models` | Load, unload, download, migrate, status |
|
||||
| Tasks | `/tasks`, `/cache` | Active task tracking, cache management |
|
||||
| CUDA | `/backend/cuda-*` | CUDA binary download and management |
|
||||
|
||||
### Quick examples
|
||||
|
||||
```bash
|
||||
# Generate speech
|
||||
curl -X POST http://localhost:17493/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"profile_id": "abc-123",
|
||||
"text": "Hello, this is a test.",
|
||||
"language": "en",
|
||||
"seed": 42
|
||||
}'
|
||||
-d '{"text": "Hello world", "profile_id": "...", "language": "en"}'
|
||||
|
||||
# Response: {"id": "gen-456", "audio_path": "/path/to/audio.wav", ...}
|
||||
# List profiles
|
||||
curl http://localhost:17493/profiles
|
||||
|
||||
# Download audio
|
||||
curl http://localhost:17493/audio/gen-456 -o output.wav
|
||||
# Stream generation status (SSE)
|
||||
curl http://localhost:17493/generate/{id}/status
|
||||
```
|
||||
|
||||
### Transcribing Audio
|
||||
## Data directory
|
||||
|
||||
```
|
||||
{data_dir}/
|
||||
voicebox.db # SQLite database
|
||||
profiles/{id}/ # Voice samples per profile
|
||||
generations/ # Generated audio files
|
||||
cache/ # Voice prompt cache (memory + disk)
|
||||
backends/ # Downloaded CUDA binary (if applicable)
|
||||
```
|
||||
|
||||
Default location is the OS-specific app data directory. Override with `--data-dir` or the `VOICEBOX_DATA_DIR` environment variable.
|
||||
|
||||
## Code quality
|
||||
|
||||
Linting and formatting are enforced by [ruff](https://docs.astral.sh/ruff/), configured in `pyproject.toml`. See `STYLE_GUIDE.md` for conventions.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:17493/transcribe \
|
||||
-F "[email protected]" \
|
||||
-F "language=en"
|
||||
|
||||
# Response: {"text": "Transcribed text", "duration": 5.5}
|
||||
just check-python # lint + format check
|
||||
just fix-python # auto-fix lint issues + reformat
|
||||
just test # run pytest
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
## Dependencies
|
||||
|
||||
### Multi-Sample Profiles
|
||||
|
||||
Add multiple samples to a profile for better quality:
|
||||
|
||||
```bash
|
||||
# Add first sample
|
||||
curl -X POST http://localhost:17493/profiles/abc-123/samples \
|
||||
-F "[email protected]" \
|
||||
-F "reference_text=First sample"
|
||||
|
||||
# Add second sample
|
||||
curl -X POST http://localhost:17493/profiles/abc-123/samples \
|
||||
-F "[email protected]" \
|
||||
-F "reference_text=Second sample"
|
||||
|
||||
# Generation will automatically combine all samples
|
||||
```
|
||||
|
||||
### Voice Prompt Caching
|
||||
|
||||
Voice prompts are automatically cached for faster generation:
|
||||
- First generation: ~5-10 seconds (creates prompt)
|
||||
- Subsequent generations: ~1-2 seconds (uses cached prompt)
|
||||
|
||||
Cache is stored in `data/cache/` and persists across server restarts.
|
||||
|
||||
### VRAM Management
|
||||
|
||||
Models are lazy-loaded and can be manually unloaded:
|
||||
|
||||
```bash
|
||||
# Unload TTS model
|
||||
curl -X POST http://localhost:17493/models/unload
|
||||
|
||||
# Load specific model size
|
||||
curl -X POST "http://localhost:17493/models/load?model_size=0.6B"
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
All endpoints return proper HTTP status codes:
|
||||
|
||||
- `200 OK`: Success
|
||||
- `400 Bad Request`: Invalid input
|
||||
- `404 Not Found`: Resource not found
|
||||
- `500 Internal Server Error`: Server error
|
||||
|
||||
Error responses include details:
|
||||
|
||||
```json
|
||||
{
|
||||
"detail": "Profile not found"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Use multi-sample profiles** - Better quality than single sample
|
||||
2. **Let caching work** - Voice prompts are cached automatically
|
||||
3. **Use 0.6B model on CPU** - Faster than 1.7B with acceptable quality
|
||||
4. **Use 1.7B model on GPU** - Best quality, still fast
|
||||
5. **Unload Whisper after transcription** - Frees VRAM for TTS
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] WebSocket support for generation progress
|
||||
- [ ] Batch generation endpoint
|
||||
- [ ] Audio effects (M3GAN, etc.)
|
||||
- [ ] Voice design (text-to-voice)
|
||||
- [ ] Audio studio timeline features
|
||||
- [ ] Project management
|
||||
- [ ] Authentication & rate limiting
|
||||
- [ ] Export/import profiles
|
||||
|
||||
## License
|
||||
|
||||
See main project LICENSE.
|
||||
Runtime dependencies are in `requirements.txt`. macOS-only MLX dependencies are in `requirements-mlx.txt`. Dev tools (ruff, pytest) are installed automatically by `just setup-python`.
|
||||
|
||||
@@ -0,0 +1,404 @@
|
||||
# Python Style Guide
|
||||
|
||||
Target: **Python 3.12+** | Formatter/Linter: **Ruff** | Config: `backend/pyproject.toml`
|
||||
|
||||
This guide codifies the conventions used across the backend, and prescribes the target style for code written during the refactor (Phases 3-6). Existing code should be migrated incrementally -- don't reformat entire files in unrelated PRs.
|
||||
|
||||
---
|
||||
|
||||
## Formatting
|
||||
|
||||
Enforced by `ruff format` (Black-compatible).
|
||||
|
||||
- **Line length**: 120 characters.
|
||||
- **Indent**: 4 spaces. No tabs.
|
||||
- **Trailing commas**: Required on multi-line function signatures, arguments, collections.
|
||||
- **Quotes**: Double quotes (`"`) for strings. Single quotes are acceptable in f-string expressions and dict keys inside f-strings where avoiding escapes improves readability.
|
||||
|
||||
Run: `ruff format backend/`
|
||||
|
||||
---
|
||||
|
||||
## Imports
|
||||
|
||||
Enforced by ruff's `isort` rules (rule set `I`).
|
||||
|
||||
**Grouping** -- three blocks separated by a blank line:
|
||||
|
||||
```python
|
||||
import asyncio # 1. stdlib
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np # 2. third-party
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.config import get_data_dir # 3. local (absolute)
|
||||
from .database import get_db # or relative
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
- Within the `backend` package, use **relative imports** for sibling/child modules: `from .database import get_db`, `from ..utils.audio import load_audio`.
|
||||
- Absolute imports are fine for top-level references from entry points (`main.py`, `server.py`).
|
||||
- Never use wildcard imports (`from module import *`).
|
||||
- One import per line for `from X import Y` when there are 4+ names; below that, comma-separated is fine.
|
||||
- **Lazy imports** are acceptable for heavy dependencies (torch, transformers, mlx) inside functions to reduce startup time. Add a comment: `# lazy: heavy import`.
|
||||
|
||||
---
|
||||
|
||||
## Type Annotations
|
||||
|
||||
Python 3.12 means we use **built-in generics and union syntax natively**. No `from __future__ import annotations`, no `typing.List`/`typing.Dict`.
|
||||
|
||||
```python
|
||||
# Yes
|
||||
def process(items: list[str], config: dict[str, int] | None = None) -> tuple[int, str]: ...
|
||||
|
||||
# No
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
def process(items: List[str], config: Optional[Dict[str, int]] = None) -> Tuple[int, str]: ...
|
||||
```
|
||||
|
||||
**What to annotate:**
|
||||
- All public function signatures (parameters + return type).
|
||||
- Private functions: parameters at minimum; return type encouraged.
|
||||
- Module-level variables: only when the type isn't obvious from the assignment.
|
||||
- Route handlers: parameters are annotated via FastAPI's dependency injection. Add explicit `-> SomeResponse` return types when the route doesn't use `response_model`.
|
||||
|
||||
**Imports from `typing` that are still needed** (no built-in equivalent):
|
||||
`Literal`, `TypeAlias`, `Protocol`, `runtime_checkable`, `Callable`, `Any`, `ClassVar`, `TypeVar`, `overload`, `TYPE_CHECKING`.
|
||||
|
||||
Use `collections.abc` for abstract types: `Sequence`, `Mapping`, `Iterable`, `Iterator`, `Generator`.
|
||||
|
||||
---
|
||||
|
||||
## Naming
|
||||
|
||||
| Thing | Convention | Example |
|
||||
|-------|-----------|---------|
|
||||
| Module | `snake_case` | `task_queue.py` |
|
||||
| Class | `PascalCase` | `ProgressManager` |
|
||||
| Function / method | `snake_case` | `create_profile` |
|
||||
| Variable | `snake_case` | `sample_rate` |
|
||||
| Constant | `UPPER_SNAKE_CASE` | `DEFAULT_SAMPLE_RATE` |
|
||||
| Private | `_leading_underscore` | `_generation_queue` |
|
||||
| Type alias | `PascalCase` | `EffectChain = list[dict[str, Any]]` |
|
||||
|
||||
**Specific conventions:**
|
||||
- Database ORM models imported with `DB` prefix alias: `from .database import VoiceProfile as DBVoiceProfile`.
|
||||
- Pydantic models use descriptive suffixes: `VoiceProfileCreate`, `VoiceProfileResponse`, `GenerationRequest`.
|
||||
- Backend classes use engine-name prefix: `MLXTTSBackend`, `PyTorchSTTBackend`.
|
||||
|
||||
---
|
||||
|
||||
## Docstrings
|
||||
|
||||
**Google style**. Required on all public functions, classes, and modules.
|
||||
|
||||
```python
|
||||
def combine_voice_prompts(
|
||||
profile_dir: Path,
|
||||
*,
|
||||
target_sr: int = 24000,
|
||||
) -> tuple[np.ndarray, int]:
|
||||
"""Load and concatenate all voice prompt files for a profile.
|
||||
|
||||
Reads .wav/.mp3/.flac files from the profile directory, resamples to
|
||||
the target sample rate, normalizes, and concatenates into a single array.
|
||||
|
||||
Args:
|
||||
profile_dir: Path to the voice profile directory containing audio files.
|
||||
target_sr: Target sample rate for the output. Defaults to 24000.
|
||||
|
||||
Returns:
|
||||
Tuple of (concatenated audio array, sample rate).
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If profile_dir does not exist.
|
||||
ValueError: If no valid audio files are found.
|
||||
"""
|
||||
```
|
||||
|
||||
**Short form** is fine for simple functions:
|
||||
|
||||
```python
|
||||
def get_db_path() -> Path:
|
||||
"""Get the path to the SQLite database file."""
|
||||
```
|
||||
|
||||
**When to skip**: Private helpers under ~5 lines where the name and signature make intent obvious.
|
||||
|
||||
**Module docstrings**: A single sentence at the top of every file describing its purpose.
|
||||
|
||||
```python
|
||||
"""Voice profile CRUD operations."""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comments
|
||||
|
||||
Comments explain **why**, not **what**. If the code needs a comment to explain what it does, the code should be rewritten to be clearer. The exceptions are non-obvious performance choices, external constraints, and concurrency/race-condition reasoning -- those always deserve a comment.
|
||||
|
||||
### No section dividers
|
||||
|
||||
Do not use ASCII dividers to create visual sections in files:
|
||||
|
||||
```python
|
||||
# No -- any of these:
|
||||
# ============================================
|
||||
# GENERATION ENDPOINTS
|
||||
# ============================================
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Device detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# --- Load model --------------------------------------------------
|
||||
```
|
||||
|
||||
If a file needs section dividers to be navigable, the file is too long. Split it into modules. Within a function, if you need labeled sections to follow the logic, extract those sections into named functions.
|
||||
|
||||
### Inline comments
|
||||
|
||||
Inline comments (end-of-line) are fine when they add information the code can't express:
|
||||
|
||||
```python
|
||||
# Yes -- explains a non-obvious constraint or gives context:
|
||||
audio, sr = load_audio(path, sr=24000) # Qwen expects 24kHz mono
|
||||
_generation_queue: asyncio.Queue = None # type: ignore # initialized at startup
|
||||
"tauri://localhost", # Tauri webview (macOS)
|
||||
|
||||
# No -- restates the code:
|
||||
# Check if profile name already exists
|
||||
existing = db.query(DBVoiceProfile).filter_by(name=data.name).first()
|
||||
|
||||
# Delete from database
|
||||
db.delete(sample)
|
||||
|
||||
# Update fields
|
||||
profile.name = data.name
|
||||
```
|
||||
|
||||
Delete comments that narrate what the next line of code obviously does. If the function name, variable name, or method call already communicates intent, the comment is noise.
|
||||
|
||||
### Block comments
|
||||
|
||||
Use block comments for **why** explanations -- constraints, workarounds, non-obvious decisions:
|
||||
|
||||
```python
|
||||
# PyInstaller + multiprocessing: child processes re-execute the frozen binary
|
||||
# with internal arguments. freeze_support() handles this and exits early.
|
||||
multiprocessing.freeze_support()
|
||||
|
||||
# Mark any stale "generating" records as failed -- these are leftovers
|
||||
# from a previous process that was killed mid-generation.
|
||||
db.query(Generation).filter_by(status="generating").update({"status": "failed"})
|
||||
```
|
||||
|
||||
Keep block comments tight. Two to three lines is normal. If you need a paragraph, it probably belongs in the docstring or a design doc.
|
||||
|
||||
### Linter/type-checker suppression
|
||||
|
||||
Always add a reason after `noqa` and `type: ignore`:
|
||||
|
||||
```python
|
||||
import intel_extension_for_pytorch # noqa: F401 -- side-effect import enables XPU
|
||||
_queue: asyncio.Queue = None # type: ignore[assignment] # initialized at startup
|
||||
```
|
||||
|
||||
Bare `# noqa` or `# type: ignore` with no explanation are not allowed.
|
||||
|
||||
### TODO / FIXME
|
||||
|
||||
Use sparingly. Every `TODO` must include a brief description of what needs doing. Don't use them as a substitute for tracking work properly:
|
||||
|
||||
```python
|
||||
# TODO: replace with async SQLAlchemy once CRUD modules are migrated (Phase 5)
|
||||
result = await asyncio.to_thread(profiles.get_profile, profile_id, db)
|
||||
```
|
||||
|
||||
Never commit `HACK`, `XXX`, or `FIXME` -- fix the problem or file an issue.
|
||||
|
||||
### Commented-out code
|
||||
|
||||
Delete it. That's what git is for. If you need to document that something was intentionally removed, a short tombstone comment is acceptable:
|
||||
|
||||
```python
|
||||
# Removed config.json-only check -- too lenient, doesn't confirm weights exist.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
The refactor is standardizing on a **two-layer pattern**:
|
||||
|
||||
### 1. Domain layer -- raise plain exceptions
|
||||
|
||||
CRUD modules and services raise `ValueError`, `FileNotFoundError`, or (post-refactor) custom exceptions defined in `backend/errors.py`:
|
||||
|
||||
```python
|
||||
# backend/errors.py (to be created in Phase 4)
|
||||
class NotFoundError(Exception):
|
||||
"""Raised when a requested resource does not exist."""
|
||||
|
||||
class ConflictError(Exception):
|
||||
"""Raised on uniqueness constraint violations."""
|
||||
```
|
||||
|
||||
```python
|
||||
# In a service or CRUD module:
|
||||
raise NotFoundError(f"Profile {profile_id} not found")
|
||||
```
|
||||
|
||||
### 2. Route layer -- translate to HTTPException
|
||||
|
||||
Route handlers catch domain exceptions and convert:
|
||||
|
||||
```python
|
||||
@router.post("/profiles")
|
||||
async def create_profile(data: VoiceProfileCreate, db: Session = Depends(get_db)):
|
||||
try:
|
||||
return await profiles.create_profile(data, db)
|
||||
except ConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=str(e))
|
||||
```
|
||||
|
||||
**Background tasks** catch `Exception` broadly, log with `logger.exception()`, and update the task status to `"failed"`.
|
||||
|
||||
**Never**: silently swallow exceptions, use bare `except:`, or catch `BaseException`.
|
||||
|
||||
---
|
||||
|
||||
## Async
|
||||
|
||||
### Rules for the refactor
|
||||
|
||||
1. **Don't declare `async def` unless the function awaits something.** Several service modules still declare `async def` without awaiting -- these should be migrated to sync functions with `asyncio.to_thread()` at the route layer, or to real async SQLAlchemy.
|
||||
2. **CPU-bound work** (audio processing, numpy operations) goes through `asyncio.to_thread()`:
|
||||
```python
|
||||
audio, sr = await asyncio.to_thread(load_audio, source_path)
|
||||
```
|
||||
3. **GPU-bound TTS inference** is serialized through the generation queue (`services/task_queue.py`). Never call a backend's `generate()` directly from a route handler.
|
||||
4. **Fire-and-forget tasks**: use `asyncio.create_task()` and track the task reference to prevent garbage collection:
|
||||
```python
|
||||
task = asyncio.create_task(some_coro())
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Logging
|
||||
|
||||
Use the `logging` module. Not `print()`.
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
logger.info("Loading model %s on %s", model_name, device)
|
||||
logger.warning("Cache miss for %s, downloading", repo_id)
|
||||
logger.exception("Generation %s failed") # logs traceback automatically
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
- Use `%s`-style placeholders in log calls (not f-strings). This avoids formatting the string if the log level is filtered out.
|
||||
- Use `logger.exception()` inside `except` blocks -- it captures the traceback.
|
||||
- Logger name should be `__name__` (yields `backend.utils.audio`, etc.).
|
||||
- Existing `print()` calls should be migrated to logging as files are touched during the refactor.
|
||||
|
||||
---
|
||||
|
||||
## Constants
|
||||
|
||||
- Define at **module level** in the file where they're primarily used.
|
||||
- Use `UPPER_SNAKE_CASE`.
|
||||
- Shared/cross-cutting constants (sample rates, file size limits, CORS origins) go in `backend/config.py` after Phase 6 consolidation.
|
||||
- Magic numbers in function bodies should be extracted to named constants:
|
||||
```python
|
||||
# No
|
||||
if len(audio) > 24000 * 60 * 10:
|
||||
|
||||
# Yes
|
||||
MAX_AUDIO_DURATION_SAMPLES = SAMPLE_RATE * 60 * 10
|
||||
if len(audio) > MAX_AUDIO_DURATION_SAMPLES:
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Function Signatures
|
||||
|
||||
- **Keyword-only arguments** (after `*`) for functions with 3+ parameters, especially when several share the same type:
|
||||
```python
|
||||
def is_model_cached(
|
||||
hf_repo: str,
|
||||
*,
|
||||
weight_extensions: tuple[str, ...] = (".safetensors", ".bin"),
|
||||
required_files: list[str] | None = None,
|
||||
) -> bool:
|
||||
```
|
||||
- Parameters on **separate lines** when the signature exceeds ~100 characters or has 3+ params.
|
||||
- **Trailing comma** after the last parameter in multi-line signatures.
|
||||
- Default values inline with the parameter.
|
||||
|
||||
---
|
||||
|
||||
## String Formatting
|
||||
|
||||
- **f-strings** for runtime string construction.
|
||||
- **`%s`-style** for `logging` calls (lazy evaluation).
|
||||
- **`.format()`**: avoid; f-strings are preferred.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Framework: **pytest** with `pytest-asyncio`.
|
||||
|
||||
- Test files: `test_<module>.py` in `backend/tests/`.
|
||||
- Use `conftest.py` for shared fixtures (db sessions, test client, mock backends).
|
||||
- Group related tests in classes: `class TestProfileCRUD:`.
|
||||
- Use `@pytest.mark.asyncio` for async tests.
|
||||
- Use `@pytest.mark.parametrize` to reduce repetition.
|
||||
- Manual integration scripts stay in `tests/` but are clearly marked (filename prefix `manual_` or documented in `tests/README.md`).
|
||||
|
||||
---
|
||||
|
||||
## Project Layout
|
||||
|
||||
```
|
||||
backend/
|
||||
app.py # FastAPI app factory, CORS, lifecycle events
|
||||
main.py # Entry point (imports app, runs uvicorn)
|
||||
config.py # Data directory paths
|
||||
models.py # Pydantic request/response schemas
|
||||
server.py # Tauri sidecar launcher, parent-pid watchdog
|
||||
routes/ # Thin HTTP handlers (validation, delegation, response formatting)
|
||||
services/ # Business logic, CRUD, orchestration
|
||||
backends/ # TTS/STT engine implementations
|
||||
database/ # ORM models, session management, migrations, seeds
|
||||
utils/ # Shared utilities (audio, effects, caching, progress)
|
||||
tests/ # pytest suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ruff Adoption
|
||||
|
||||
`pyproject.toml` configures ruff for linting and formatting. Run:
|
||||
|
||||
```bash
|
||||
# Lint (check)
|
||||
ruff check backend/
|
||||
|
||||
# Lint (auto-fix)
|
||||
ruff check backend/ --fix
|
||||
|
||||
# Format
|
||||
ruff format backend/
|
||||
```
|
||||
|
||||
Introduce ruff fixes file-by-file as you touch them. Don't run `--fix` across the entire codebase in one shot -- that creates unreviewable diffs.
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
# Backend package
|
||||
|
||||
__version__ = "0.2.3"
|
||||
__version__ = "0.4.1"
|
||||
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
"""FastAPI application factory, middleware, and lifecycle events."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ColoredFormatter(logging.Formatter):
|
||||
"""Custom formatter to add colors matching uvicorn's style."""
|
||||
|
||||
COLORS = {
|
||||
"DEBUG": "\033[36m", # Cyan
|
||||
"INFO": "\033[32m", # Green
|
||||
"WARNING": "\033[33m", # Yellow
|
||||
"ERROR": "\033[31m", # Red
|
||||
"CRITICAL": "\033[35m", # Magenta
|
||||
}
|
||||
RESET = "\033[0m"
|
||||
|
||||
def format(self, record):
|
||||
log_color = self.COLORS.get(record.levelname, self.RESET)
|
||||
record.levelname = f"{log_color}{record.levelname}{self.RESET}"
|
||||
return super().format(record)
|
||||
|
||||
|
||||
# Configure logging to match uvicorn's format with colors
|
||||
handler = logging.StreamHandler(sys.stderr)
|
||||
handler.setFormatter(ColoredFormatter("%(levelname)s: %(message)s"))
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
handlers=[handler],
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# AMD GPU environment variables must be set before torch import
|
||||
if not os.environ.get("HSA_OVERRIDE_GFX_VERSION"):
|
||||
os.environ["HSA_OVERRIDE_GFX_VERSION"] = "10.3.0"
|
||||
if not os.environ.get("MIOPEN_LOG_LEVEL"):
|
||||
os.environ["MIOPEN_LOG_LEVEL"] = "4"
|
||||
|
||||
import torch
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from urllib.parse import quote
|
||||
|
||||
from . import __version__, config, database
|
||||
from .services import tts, transcribe
|
||||
from .database import get_db
|
||||
from .utils.platform_detect import get_backend_type
|
||||
from .utils.progress import get_progress_manager
|
||||
from .services.task_queue import create_background_task, init_queue
|
||||
from .routes import register_routers
|
||||
|
||||
|
||||
def safe_content_disposition(disposition_type: str, filename: str) -> str:
|
||||
"""Build a Content-Disposition header safe for non-ASCII filenames.
|
||||
|
||||
Uses RFC 5987 ``filename*`` parameter so browsers can decode UTF-8
|
||||
filenames while the ``filename`` fallback stays ASCII-only.
|
||||
"""
|
||||
ascii_name = "".join(c for c in filename if c.isascii() and (c.isalnum() or c in " -_.")).strip() or "download"
|
||||
utf8_name = quote(filename, safe="")
|
||||
return f"{disposition_type}; filename=\"{ascii_name}\"; filename*=UTF-8''{utf8_name}"
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application."""
|
||||
application = FastAPI(
|
||||
title="voicebox API",
|
||||
description="Production-quality Qwen3-TTS voice cloning API",
|
||||
version=__version__,
|
||||
)
|
||||
|
||||
_configure_cors(application)
|
||||
register_routers(application)
|
||||
_register_lifecycle(application)
|
||||
_mount_frontend(application)
|
||||
|
||||
return application
|
||||
|
||||
|
||||
def _configure_cors(application: FastAPI) -> None:
|
||||
"""Set up CORS middleware with local-first defaults."""
|
||||
default_origins = [
|
||||
"http://localhost:5173", # Vite dev server
|
||||
"http://127.0.0.1:5173",
|
||||
"http://localhost:17493",
|
||||
"http://127.0.0.1:17493",
|
||||
"tauri://localhost", # Tauri webview (macOS)
|
||||
"https://tauri.localhost", # Tauri webview (Windows/Linux)
|
||||
"http://tauri.localhost", # Tauri webview (Windows, some builds)
|
||||
]
|
||||
env_origins = os.environ.get("VOICEBOX_CORS_ORIGINS", "")
|
||||
all_origins = default_origins + [o.strip() for o in env_origins.split(",") if o.strip()]
|
||||
|
||||
application.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=all_origins,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
|
||||
def _mount_frontend(application: FastAPI) -> None:
|
||||
"""Serve the built web frontend when present (Docker / web deployment).
|
||||
|
||||
The Dockerfile copies the Vite build output to ``/app/frontend/``. When
|
||||
that directory exists we mount static assets and add a catch-all route so
|
||||
the React SPA handles client-side routing. In dev or API-only mode the
|
||||
directory is absent and this function is a no-op.
|
||||
"""
|
||||
frontend_dir = Path(__file__).resolve().parent.parent / "frontend"
|
||||
if not frontend_dir.is_dir():
|
||||
return
|
||||
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
# Mount hashed assets (JS, CSS, images) that Vite places under /assets
|
||||
assets_dir = frontend_dir / "assets"
|
||||
if assets_dir.is_dir():
|
||||
application.mount(
|
||||
"/assets",
|
||||
StaticFiles(directory=str(assets_dir)),
|
||||
name="frontend-assets",
|
||||
)
|
||||
|
||||
# SPA catch-all: serve files if they exist, otherwise index.html for
|
||||
# client-side routes like /voices, /stories, /models, etc.
|
||||
@application.get("/{full_path:path}")
|
||||
async def serve_spa(full_path: str):
|
||||
file_path = (frontend_dir / full_path).resolve()
|
||||
# Guard against path traversal — only serve files inside frontend_dir
|
||||
if full_path and file_path.is_file() and file_path.is_relative_to(frontend_dir):
|
||||
return FileResponse(file_path)
|
||||
return FileResponse(frontend_dir / "index.html", media_type="text/html")
|
||||
|
||||
logger.info("Frontend: serving SPA from %s", frontend_dir)
|
||||
|
||||
|
||||
def _get_gpu_status() -> str:
|
||||
"""Return a human-readable string describing GPU availability."""
|
||||
backend_type = get_backend_type()
|
||||
if torch.cuda.is_available():
|
||||
from .backends.base import check_cuda_compatibility
|
||||
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
compatible, _warning = check_cuda_compatibility()
|
||||
is_rocm = hasattr(torch.version, "hip") and torch.version.hip is not None
|
||||
if is_rocm:
|
||||
label = f"ROCm ({device_name})"
|
||||
else:
|
||||
label = f"CUDA ({device_name})"
|
||||
if not compatible:
|
||||
label += " [UNSUPPORTED - see logs]"
|
||||
return label
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "MPS (Apple Silicon)"
|
||||
elif backend_type == "mlx":
|
||||
return "Metal (Apple Silicon via MLX)"
|
||||
|
||||
# Intel XPU (Arc / Data Center) via IPEX
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
try:
|
||||
xpu_name = torch.xpu.get_device_name(0)
|
||||
except Exception:
|
||||
xpu_name = "Intel GPU"
|
||||
return f"XPU ({xpu_name})"
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
return "None (CPU only)"
|
||||
|
||||
|
||||
def _register_lifecycle(application: FastAPI) -> None:
|
||||
"""Attach startup and shutdown event handlers."""
|
||||
|
||||
@application.on_event("startup")
|
||||
async def startup_event():
|
||||
import platform
|
||||
import sys
|
||||
|
||||
logger.info("Voicebox v%s starting up", __version__)
|
||||
logger.info(
|
||||
"Python %s on %s %s (%s)",
|
||||
sys.version.split()[0],
|
||||
platform.system(),
|
||||
platform.release(),
|
||||
platform.machine(),
|
||||
)
|
||||
|
||||
database.init_db()
|
||||
|
||||
from .database.session import _db_path
|
||||
|
||||
logger.info("Database: %s", _db_path)
|
||||
logger.info("Data directory: %s", config.get_data_dir())
|
||||
|
||||
init_queue()
|
||||
|
||||
# Mark stale "generating" records as failed -- leftovers from a killed process
|
||||
from sqlalchemy import text as sa_text
|
||||
|
||||
db = next(get_db())
|
||||
try:
|
||||
result = db.execute(
|
||||
sa_text(
|
||||
"UPDATE generations SET status = 'failed', "
|
||||
"error = 'Server was shut down during generation' "
|
||||
"WHERE status IN ('generating', 'loading_model')"
|
||||
)
|
||||
)
|
||||
if result.rowcount > 0:
|
||||
logger.info("Marked %d stale generation(s) as failed", result.rowcount)
|
||||
|
||||
from .database import VoiceProfile as DBVoiceProfile, Generation as DBGeneration
|
||||
|
||||
profile_count = db.query(DBVoiceProfile).count()
|
||||
generation_count = db.query(DBGeneration).count()
|
||||
logger.info("Profiles: %d, Generations: %d", profile_count, generation_count)
|
||||
|
||||
db.commit()
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
logger.warning("Could not clean up stale generations: %s", e)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
backend_type = get_backend_type()
|
||||
logger.info("Backend: %s", backend_type.upper())
|
||||
logger.info("GPU: %s", _get_gpu_status())
|
||||
|
||||
# Warn if GPU architecture is not supported by this PyTorch build
|
||||
from .backends.base import check_cuda_compatibility
|
||||
|
||||
_compatible, _cuda_warning = check_cuda_compatibility()
|
||||
if not _compatible:
|
||||
logger.warning("GPU COMPATIBILITY: %s", _cuda_warning)
|
||||
|
||||
from .services.cuda import check_and_update_cuda_binary
|
||||
|
||||
create_background_task(check_and_update_cuda_binary())
|
||||
|
||||
try:
|
||||
progress_manager = get_progress_manager()
|
||||
progress_manager._set_main_loop(asyncio.get_running_loop())
|
||||
except Exception as e:
|
||||
logger.warning("Could not initialize progress manager event loop: %s", e)
|
||||
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
cache_dir = Path(hf_constants.HF_HUB_CACHE)
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Model cache: %s", cache_dir)
|
||||
except Exception as e:
|
||||
logger.warning("Could not create HuggingFace cache directory: %s", e)
|
||||
|
||||
logger.info("Ready")
|
||||
|
||||
@application.on_event("shutdown")
|
||||
async def shutdown_event():
|
||||
logger.info("Voicebox server shutting down...")
|
||||
try:
|
||||
tts.unload_tts_model()
|
||||
except Exception:
|
||||
logger.exception("Failed to unload TTS model")
|
||||
try:
|
||||
transcribe.unload_whisper_model()
|
||||
except Exception:
|
||||
logger.exception("Failed to unload Whisper model")
|
||||
|
||||
|
||||
app = create_app()
|
||||
+433
-31
@@ -1,25 +1,66 @@
|
||||
"""
|
||||
Backend abstraction layer for TTS and STT.
|
||||
|
||||
Provides a unified interface for MLX and PyTorch backends.
|
||||
Provides a unified interface for MLX and PyTorch backends,
|
||||
and a model config registry that eliminates per-engine dispatch maps.
|
||||
"""
|
||||
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Protocol, Optional, Tuple, List
|
||||
from typing_extensions import runtime_checkable
|
||||
import numpy as np
|
||||
|
||||
from ..platform_detect import get_backend_type
|
||||
from ..utils.platform_detect import get_backend_type
|
||||
|
||||
LANGUAGE_CODE_TO_NAME = {
|
||||
"zh": "chinese",
|
||||
"en": "english",
|
||||
"ja": "japanese",
|
||||
"ko": "korean",
|
||||
"de": "german",
|
||||
"fr": "french",
|
||||
"ru": "russian",
|
||||
"pt": "portuguese",
|
||||
"es": "spanish",
|
||||
"it": "italian",
|
||||
}
|
||||
|
||||
WHISPER_HF_REPOS = {
|
||||
"base": "openai/whisper-base",
|
||||
"small": "openai/whisper-small",
|
||||
"medium": "openai/whisper-medium",
|
||||
"large": "openai/whisper-large-v3",
|
||||
"turbo": "openai/whisper-large-v3-turbo",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelConfig:
|
||||
"""Declarative config for a downloadable model variant."""
|
||||
|
||||
model_name: str # e.g. "luxtts", "chatterbox-tts"
|
||||
display_name: str # e.g. "LuxTTS (Fast, CPU-friendly)"
|
||||
engine: str # e.g. "luxtts", "chatterbox"
|
||||
hf_repo_id: str # e.g. "YatharthS/LuxTTS"
|
||||
model_size: str = "default"
|
||||
size_mb: int = 0
|
||||
needs_trim: bool = False
|
||||
supports_instruct: bool = False
|
||||
languages: list[str] = field(default_factory=lambda: ["en"])
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class TTSBackend(Protocol):
|
||||
"""Protocol for TTS backend implementations."""
|
||||
|
||||
|
||||
# Each backend class should define MODEL_CONFIGS as a class variable:
|
||||
# MODEL_CONFIGS: list[ModelConfig]
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
"""Load TTS model."""
|
||||
...
|
||||
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
@@ -28,12 +69,12 @@ class TTSBackend(Protocol):
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
@@ -41,12 +82,12 @@ class TTSBackend(Protocol):
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple voice prompts.
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio_array, combined_text)
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
@@ -57,24 +98,24 @@ class TTSBackend(Protocol):
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text.
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
...
|
||||
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
...
|
||||
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get model path for a given size.
|
||||
|
||||
|
||||
Returns:
|
||||
Model path or HuggingFace Hub ID
|
||||
"""
|
||||
@@ -84,28 +125,29 @@ class TTSBackend(Protocol):
|
||||
@runtime_checkable
|
||||
class STTBackend(Protocol):
|
||||
"""Protocol for STT (Speech-to-Text) backend implementations."""
|
||||
|
||||
|
||||
async def load_model(self, model_size: str) -> None:
|
||||
"""Load STT model."""
|
||||
...
|
||||
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
model_size: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
...
|
||||
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
...
|
||||
@@ -117,19 +159,360 @@ _tts_backends: dict[str, TTSBackend] = {}
|
||||
_tts_backends_lock = threading.Lock()
|
||||
_stt_backend: Optional[STTBackend] = None
|
||||
|
||||
# Supported TTS engines
|
||||
# Supported TTS engines — keyed by engine name, value is the backend class import path.
|
||||
# The factory function uses this for the if/elif chain; the model configs live on the backend classes.
|
||||
TTS_ENGINES = {
|
||||
"qwen": "Qwen TTS",
|
||||
"qwen_custom_voice": "Qwen CustomVoice",
|
||||
"luxtts": "LuxTTS",
|
||||
"chatterbox": "Chatterbox TTS",
|
||||
"chatterbox_turbo": "Chatterbox Turbo",
|
||||
"tada": "TADA",
|
||||
"kokoro": "Kokoro",
|
||||
}
|
||||
|
||||
|
||||
def _get_qwen_model_configs() -> list[ModelConfig]:
|
||||
"""Return Qwen model configs with backend-aware HF repo IDs."""
|
||||
backend_type = get_backend_type()
|
||||
if backend_type == "mlx":
|
||||
repo_1_7b = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
|
||||
repo_0_6b = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16" # 0.6B not available in MLX, falls back
|
||||
else:
|
||||
repo_1_7b = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
repo_0_6b = "Qwen/Qwen3-TTS-12Hz-0.6B-Base"
|
||||
|
||||
return [
|
||||
ModelConfig(
|
||||
model_name="qwen-tts-1.7B",
|
||||
display_name="Qwen TTS 1.7B",
|
||||
engine="qwen",
|
||||
hf_repo_id=repo_1_7b,
|
||||
model_size="1.7B",
|
||||
size_mb=3500,
|
||||
supports_instruct=False, # Base model drops instruct silently
|
||||
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="qwen-tts-0.6B",
|
||||
display_name="Qwen TTS 0.6B",
|
||||
engine="qwen",
|
||||
hf_repo_id=repo_0_6b,
|
||||
model_size="0.6B",
|
||||
size_mb=1200,
|
||||
supports_instruct=False,
|
||||
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _get_qwen_custom_voice_configs() -> list[ModelConfig]:
|
||||
"""Return Qwen CustomVoice model configs."""
|
||||
return [
|
||||
ModelConfig(
|
||||
model_name="qwen-custom-voice-1.7B",
|
||||
display_name="Qwen CustomVoice 1.7B",
|
||||
engine="qwen_custom_voice",
|
||||
hf_repo_id="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
model_size="1.7B",
|
||||
size_mb=3500,
|
||||
supports_instruct=True,
|
||||
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="qwen-custom-voice-0.6B",
|
||||
display_name="Qwen CustomVoice 0.6B",
|
||||
engine="qwen_custom_voice",
|
||||
hf_repo_id="Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
|
||||
model_size="0.6B",
|
||||
size_mb=1200,
|
||||
supports_instruct=True,
|
||||
languages=["zh", "en", "ja", "ko", "de", "fr", "ru", "pt", "es", "it"],
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _get_non_qwen_tts_configs() -> list[ModelConfig]:
|
||||
"""Return model configs for non-Qwen TTS engines.
|
||||
|
||||
These are static — no backend-type branching needed.
|
||||
"""
|
||||
return [
|
||||
ModelConfig(
|
||||
model_name="luxtts",
|
||||
display_name="LuxTTS (Fast, CPU-friendly)",
|
||||
engine="luxtts",
|
||||
hf_repo_id="YatharthS/LuxTTS",
|
||||
size_mb=300,
|
||||
languages=["en"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="chatterbox-tts",
|
||||
display_name="Chatterbox TTS (Multilingual)",
|
||||
engine="chatterbox",
|
||||
hf_repo_id="ResembleAI/chatterbox",
|
||||
size_mb=3200,
|
||||
needs_trim=True,
|
||||
languages=[
|
||||
"zh",
|
||||
"en",
|
||||
"ja",
|
||||
"ko",
|
||||
"de",
|
||||
"fr",
|
||||
"ru",
|
||||
"pt",
|
||||
"es",
|
||||
"it",
|
||||
"he",
|
||||
"ar",
|
||||
"da",
|
||||
"el",
|
||||
"fi",
|
||||
"hi",
|
||||
"ms",
|
||||
"nl",
|
||||
"no",
|
||||
"pl",
|
||||
"sv",
|
||||
"sw",
|
||||
"tr",
|
||||
],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="chatterbox-turbo",
|
||||
display_name="Chatterbox Turbo (English, Tags)",
|
||||
engine="chatterbox_turbo",
|
||||
hf_repo_id="ResembleAI/chatterbox-turbo",
|
||||
size_mb=1500,
|
||||
needs_trim=True,
|
||||
languages=["en"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="tada-1b",
|
||||
display_name="TADA 1B (English)",
|
||||
engine="tada",
|
||||
hf_repo_id="HumeAI/tada-1b",
|
||||
model_size="1B",
|
||||
size_mb=4000,
|
||||
languages=["en"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="tada-3b-ml",
|
||||
display_name="TADA 3B Multilingual",
|
||||
engine="tada",
|
||||
hf_repo_id="HumeAI/tada-3b-ml",
|
||||
model_size="3B",
|
||||
size_mb=8000,
|
||||
languages=["en", "ar", "zh", "de", "es", "fr", "it", "ja", "pl", "pt"],
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="kokoro",
|
||||
display_name="Kokoro 82M",
|
||||
engine="kokoro",
|
||||
hf_repo_id="hexgrad/Kokoro-82M",
|
||||
size_mb=350,
|
||||
languages=["en", "es", "fr", "hi", "it", "pt", "ja", "zh"],
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _get_whisper_configs() -> list[ModelConfig]:
|
||||
"""Return Whisper STT model configs."""
|
||||
return [
|
||||
ModelConfig(
|
||||
model_name="whisper-base",
|
||||
display_name="Whisper Base",
|
||||
engine="whisper",
|
||||
hf_repo_id="openai/whisper-base",
|
||||
model_size="base",
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="whisper-small",
|
||||
display_name="Whisper Small",
|
||||
engine="whisper",
|
||||
hf_repo_id="openai/whisper-small",
|
||||
model_size="small",
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="whisper-medium",
|
||||
display_name="Whisper Medium",
|
||||
engine="whisper",
|
||||
hf_repo_id="openai/whisper-medium",
|
||||
model_size="medium",
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="whisper-large",
|
||||
display_name="Whisper Large",
|
||||
engine="whisper",
|
||||
hf_repo_id="openai/whisper-large-v3",
|
||||
model_size="large",
|
||||
),
|
||||
ModelConfig(
|
||||
model_name="whisper-turbo",
|
||||
display_name="Whisper Turbo",
|
||||
engine="whisper",
|
||||
hf_repo_id="openai/whisper-large-v3-turbo",
|
||||
model_size="turbo",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def get_all_model_configs() -> list[ModelConfig]:
|
||||
"""Return the full list of model configs (TTS + STT)."""
|
||||
return _get_qwen_model_configs() + _get_qwen_custom_voice_configs() + _get_non_qwen_tts_configs() + _get_whisper_configs()
|
||||
|
||||
|
||||
def get_tts_model_configs() -> list[ModelConfig]:
|
||||
"""Return only TTS model configs."""
|
||||
return _get_qwen_model_configs() + _get_qwen_custom_voice_configs() + _get_non_qwen_tts_configs()
|
||||
|
||||
|
||||
# Lookup helpers — these replace the if/elif chains in main.py
|
||||
|
||||
|
||||
def get_model_config(model_name: str) -> Optional[ModelConfig]:
|
||||
"""Look up a model config by model_name."""
|
||||
for cfg in get_all_model_configs():
|
||||
if cfg.model_name == model_name:
|
||||
return cfg
|
||||
return None
|
||||
|
||||
|
||||
def engine_needs_trim(engine: str) -> bool:
|
||||
"""Whether this engine's output should be run through trim_tts_output."""
|
||||
for cfg in get_tts_model_configs():
|
||||
if cfg.engine == engine:
|
||||
return cfg.needs_trim
|
||||
return False
|
||||
|
||||
|
||||
def engine_has_model_sizes(engine: str) -> bool:
|
||||
"""Whether this engine supports multiple model sizes (only Qwen currently)."""
|
||||
configs = [c for c in get_tts_model_configs() if c.engine == engine]
|
||||
return len(configs) > 1
|
||||
|
||||
|
||||
async def load_engine_model(engine: str, model_size: str = "default") -> None:
|
||||
"""Load a model for the given engine, handling engines with multiple model sizes."""
|
||||
backend = get_tts_backend_for_engine(engine)
|
||||
if engine in ("qwen", "qwen_custom_voice"):
|
||||
await backend.load_model_async(model_size)
|
||||
elif engine == "tada":
|
||||
await backend.load_model(model_size)
|
||||
else:
|
||||
await backend.load_model()
|
||||
|
||||
|
||||
async def ensure_model_cached_or_raise(engine: str, model_size: str = "default") -> None:
|
||||
"""Check if a model is cached, raise HTTPException if not. Used by streaming endpoint."""
|
||||
from fastapi import HTTPException
|
||||
|
||||
backend = get_tts_backend_for_engine(engine)
|
||||
cfg = None
|
||||
for c in get_tts_model_configs():
|
||||
if c.engine == engine and c.model_size == model_size:
|
||||
cfg = c
|
||||
break
|
||||
|
||||
if engine in ("qwen", "qwen_custom_voice", "tada"):
|
||||
if not backend._is_model_cached(model_size):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Model {model_size} is not downloaded yet. Use /generate to trigger a download.",
|
||||
)
|
||||
else:
|
||||
if not backend._is_model_cached():
|
||||
display = cfg.display_name if cfg else engine
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"{display} model is not downloaded yet. Use /generate to trigger a download.",
|
||||
)
|
||||
|
||||
|
||||
def unload_model_by_config(config: ModelConfig) -> bool:
|
||||
"""Unload a model given its config. Returns True if it was loaded, False otherwise."""
|
||||
from . import get_tts_backend_for_engine
|
||||
from ..services import tts, transcribe
|
||||
|
||||
if config.engine == "whisper":
|
||||
whisper_model = transcribe.get_whisper_model()
|
||||
if whisper_model.is_loaded() and whisper_model.model_size == config.model_size:
|
||||
transcribe.unload_whisper_model()
|
||||
return True
|
||||
return False
|
||||
|
||||
if config.engine == "qwen":
|
||||
tts_model = tts.get_tts_model()
|
||||
loaded_size = getattr(tts_model, "_current_model_size", None) or getattr(tts_model, "model_size", None)
|
||||
if tts_model.is_loaded() and loaded_size == config.model_size:
|
||||
tts.unload_tts_model()
|
||||
return True
|
||||
return False
|
||||
|
||||
if config.engine == "qwen_custom_voice":
|
||||
backend = get_tts_backend_for_engine(config.engine)
|
||||
loaded_size = getattr(backend, "_current_model_size", None) or getattr(backend, "model_size", None)
|
||||
if backend.is_loaded() and loaded_size == config.model_size:
|
||||
backend.unload_model()
|
||||
return True
|
||||
return False
|
||||
|
||||
# All other TTS engines
|
||||
backend = get_tts_backend_for_engine(config.engine)
|
||||
if backend.is_loaded():
|
||||
backend.unload_model()
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def check_model_loaded(config: ModelConfig) -> bool:
|
||||
"""Check if a model is currently loaded."""
|
||||
from . import get_tts_backend_for_engine
|
||||
from ..services import tts, transcribe
|
||||
|
||||
try:
|
||||
if config.engine == "whisper":
|
||||
whisper_model = transcribe.get_whisper_model()
|
||||
return whisper_model.is_loaded() and getattr(whisper_model, "model_size", None) == config.model_size
|
||||
|
||||
if config.engine == "qwen":
|
||||
tts_model = tts.get_tts_model()
|
||||
loaded_size = getattr(tts_model, "_current_model_size", None) or getattr(tts_model, "model_size", None)
|
||||
return tts_model.is_loaded() and loaded_size == config.model_size
|
||||
|
||||
if config.engine == "qwen_custom_voice":
|
||||
backend = get_tts_backend_for_engine(config.engine)
|
||||
loaded_size = getattr(backend, "_current_model_size", None) or getattr(backend, "model_size", None)
|
||||
return backend.is_loaded() and loaded_size == config.model_size
|
||||
|
||||
backend = get_tts_backend_for_engine(config.engine)
|
||||
return backend.is_loaded()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def get_model_load_func(config: ModelConfig):
|
||||
"""Return a callable that loads/downloads the model."""
|
||||
from . import get_tts_backend_for_engine
|
||||
from ..services import tts, transcribe
|
||||
|
||||
if config.engine == "whisper":
|
||||
return lambda: transcribe.get_whisper_model().load_model(config.model_size)
|
||||
|
||||
if config.engine == "qwen":
|
||||
return lambda: tts.get_tts_model().load_model(config.model_size)
|
||||
|
||||
if config.engine == "qwen_custom_voice":
|
||||
return lambda: get_tts_backend_for_engine(config.engine).load_model(config.model_size)
|
||||
|
||||
return lambda: get_tts_backend_for_engine(config.engine).load_model()
|
||||
|
||||
|
||||
def get_tts_backend() -> TTSBackend:
|
||||
"""
|
||||
Get or create the default (Qwen) TTS backend instance based on platform.
|
||||
|
||||
|
||||
Returns:
|
||||
TTS backend instance (MLX or PyTorch)
|
||||
"""
|
||||
@@ -139,45 +522,62 @@ def get_tts_backend() -> TTSBackend:
|
||||
def get_tts_backend_for_engine(engine: str) -> TTSBackend:
|
||||
"""
|
||||
Get or create a TTS backend for the given engine.
|
||||
|
||||
|
||||
Args:
|
||||
engine: Engine name ("qwen" or "luxtts")
|
||||
|
||||
engine: Engine name (e.g. "qwen", "luxtts", "chatterbox", "chatterbox_turbo")
|
||||
|
||||
Returns:
|
||||
TTS backend instance
|
||||
"""
|
||||
global _tts_backends
|
||||
|
||||
|
||||
# Fast path: check without lock
|
||||
if engine in _tts_backends:
|
||||
return _tts_backends[engine]
|
||||
|
||||
|
||||
# Slow path: create with lock to avoid duplicate instantiation
|
||||
with _tts_backends_lock:
|
||||
# Double-check after acquiring lock
|
||||
if engine in _tts_backends:
|
||||
return _tts_backends[engine]
|
||||
|
||||
|
||||
if engine == "qwen":
|
||||
backend_type = get_backend_type()
|
||||
if backend_type == "mlx":
|
||||
from .mlx_backend import MLXTTSBackend
|
||||
|
||||
backend = MLXTTSBackend()
|
||||
else:
|
||||
from .pytorch_backend import PyTorchTTSBackend
|
||||
|
||||
backend = PyTorchTTSBackend()
|
||||
elif engine == "luxtts":
|
||||
from .luxtts_backend import LuxTTSBackend
|
||||
|
||||
backend = LuxTTSBackend()
|
||||
elif engine == "chatterbox":
|
||||
from .chatterbox_backend import ChatterboxTTSBackend
|
||||
|
||||
backend = ChatterboxTTSBackend()
|
||||
elif engine == "chatterbox_turbo":
|
||||
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
|
||||
|
||||
backend = ChatterboxTurboTTSBackend()
|
||||
elif engine == "tada":
|
||||
from .hume_backend import HumeTadaBackend
|
||||
|
||||
backend = HumeTadaBackend()
|
||||
elif engine == "kokoro":
|
||||
from .kokoro_backend import KokoroTTSBackend
|
||||
|
||||
backend = KokoroTTSBackend()
|
||||
elif engine == "qwen_custom_voice":
|
||||
from .qwen_custom_voice_backend import QwenCustomVoiceBackend
|
||||
|
||||
backend = QwenCustomVoiceBackend()
|
||||
else:
|
||||
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
|
||||
|
||||
|
||||
_tts_backends[engine] = backend
|
||||
return backend
|
||||
|
||||
@@ -185,22 +585,24 @@ def get_tts_backend_for_engine(engine: str) -> TTSBackend:
|
||||
def get_stt_backend() -> STTBackend:
|
||||
"""
|
||||
Get or create STT backend instance based on platform.
|
||||
|
||||
|
||||
Returns:
|
||||
STT backend instance (MLX or PyTorch)
|
||||
"""
|
||||
global _stt_backend
|
||||
|
||||
|
||||
if _stt_backend is None:
|
||||
backend_type = get_backend_type()
|
||||
|
||||
|
||||
if backend_type == "mlx":
|
||||
from .mlx_backend import MLXSTTBackend
|
||||
|
||||
_stt_backend = MLXSTTBackend()
|
||||
else:
|
||||
from .pytorch_backend import PyTorchSTTBackend
|
||||
|
||||
_stt_backend = PyTorchSTTBackend()
|
||||
|
||||
|
||||
return _stt_backend
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
"""
|
||||
Shared utilities for TTS/STT backend implementations.
|
||||
|
||||
Eliminates duplication of cache checking, device detection,
|
||||
voice prompt combination, and model loading progress tracking.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import platform
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Callable, List, Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_model_cached(
|
||||
hf_repo: str,
|
||||
*,
|
||||
weight_extensions: tuple[str, ...] = (".safetensors", ".bin"),
|
||||
required_files: Optional[list[str]] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Check if a HuggingFace model is fully cached locally.
|
||||
|
||||
Args:
|
||||
hf_repo: HuggingFace repo ID (e.g. "Qwen/Qwen3-TTS-12Hz-1.7B-Base")
|
||||
weight_extensions: File extensions that count as model weights.
|
||||
required_files: If set, check that these specific filenames exist
|
||||
in snapshots instead of checking by extension.
|
||||
|
||||
Returns:
|
||||
True if model is fully cached, False if missing or incomplete.
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + hf_repo.replace("/", "--"))
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
# Incomplete blobs mean a download is still in progress
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
logger.debug(f"Found .incomplete files for {hf_repo}")
|
||||
return False
|
||||
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if not snapshots_dir.exists():
|
||||
return False
|
||||
|
||||
if required_files:
|
||||
# Check that every required filename exists somewhere in snapshots
|
||||
for fname in required_files:
|
||||
if not any(snapshots_dir.rglob(fname)):
|
||||
return False
|
||||
return True
|
||||
|
||||
# Check that at least one weight file exists
|
||||
for ext in weight_extensions:
|
||||
if any(snapshots_dir.rglob(f"*{ext}")):
|
||||
return True
|
||||
|
||||
logger.debug(f"No model weights found for {hf_repo}")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking cache for {hf_repo}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_torch_device(
|
||||
*,
|
||||
allow_xpu: bool = False,
|
||||
allow_directml: bool = False,
|
||||
allow_mps: bool = False,
|
||||
force_cpu_on_mac: bool = False,
|
||||
) -> str:
|
||||
"""
|
||||
Detect the best available torch device.
|
||||
|
||||
Args:
|
||||
allow_xpu: Check for Intel XPU (IPEX) support.
|
||||
allow_directml: Check for DirectML (Windows) support.
|
||||
allow_mps: Allow MPS (Apple Silicon). If False, MPS falls back to CPU.
|
||||
force_cpu_on_mac: Force CPU on macOS regardless of GPU availability.
|
||||
"""
|
||||
if force_cpu_on_mac and platform.system() == "Darwin":
|
||||
return "cpu"
|
||||
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
|
||||
if allow_xpu:
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
return "xpu"
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
if allow_directml:
|
||||
try:
|
||||
import torch_directml
|
||||
|
||||
if torch_directml.device_count() > 0:
|
||||
return torch_directml.device(0)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
if allow_mps:
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "mps"
|
||||
|
||||
return "cpu"
|
||||
|
||||
|
||||
def check_cuda_compatibility() -> tuple[bool, str | None]:
|
||||
"""Check if the installed PyTorch supports the current GPU's compute capability.
|
||||
|
||||
Returns:
|
||||
(compatible, warning_message) — compatible is True if OK or no CUDA GPU,
|
||||
warning_message is a human-readable string if there's a problem.
|
||||
"""
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return True, None
|
||||
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
capability = f"{major}.{minor}"
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
|
||||
# torch.cuda._get_arch_list() returns the SM architectures this build
|
||||
# was compiled for (e.g. ["sm_50", "sm_60", ..., "sm_90"]).
|
||||
try:
|
||||
arch_list = torch.cuda._get_arch_list()
|
||||
if arch_list:
|
||||
# Check for both sm_XX and compute_XX (JIT-compiled) entries
|
||||
compute_tag = f"compute_{major}{minor}"
|
||||
if sm_tag not in arch_list and compute_tag not in arch_list:
|
||||
return False, (
|
||||
f"{device_name} (compute capability {capability} / {sm_tag}) "
|
||||
f"is not supported by this PyTorch build. "
|
||||
f"Supported architectures: {', '.join(arch_list)}. "
|
||||
f"Install PyTorch nightly (cu128) for newer GPU support: "
|
||||
f"pip install torch --index-url https://download.pytorch.org/whl/nightly/cu128"
|
||||
)
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
return True, None
|
||||
|
||||
|
||||
def empty_device_cache(device: str) -> None:
|
||||
"""
|
||||
Free cached memory on the given device (CUDA or XPU).
|
||||
|
||||
Backends should call this after unloading models so VRAM is returned
|
||||
to the OS.
|
||||
"""
|
||||
import torch
|
||||
|
||||
if device == "cuda" and torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
elif device == "xpu" and hasattr(torch, "xpu"):
|
||||
torch.xpu.empty_cache()
|
||||
|
||||
|
||||
def manual_seed(seed: int, device: str) -> None:
|
||||
"""
|
||||
Set the random seed on both CPU and the active accelerator.
|
||||
|
||||
Covers CUDA and Intel XPU so that generation is reproducible
|
||||
regardless of which GPU backend is in use.
|
||||
"""
|
||||
import torch
|
||||
|
||||
torch.manual_seed(seed)
|
||||
if device == "cuda" and torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
elif device == "xpu" and hasattr(torch, "xpu"):
|
||||
torch.xpu.manual_seed(seed)
|
||||
|
||||
|
||||
async def combine_voice_prompts(
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
*,
|
||||
sample_rate: Optional[int] = None,
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference audio samples into one.
|
||||
|
||||
Loads each audio file, normalizes, concatenates, and joins texts.
|
||||
|
||||
Args:
|
||||
audio_paths: Paths to reference audio files.
|
||||
reference_texts: Corresponding transcripts.
|
||||
sample_rate: If set, resample audio to this rate during loading.
|
||||
"""
|
||||
combined_audio = []
|
||||
|
||||
for path in audio_paths:
|
||||
kwargs = {"sample_rate": sample_rate} if sample_rate else {}
|
||||
audio, _sr = load_audio(path, **kwargs)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
|
||||
@contextmanager
|
||||
def model_load_progress(
|
||||
model_name: str,
|
||||
is_cached: bool,
|
||||
filter_non_downloads: Optional[bool] = None,
|
||||
):
|
||||
"""
|
||||
Context manager for model loading with HF download progress tracking.
|
||||
|
||||
Handles the tqdm patching, progress_manager/task_manager lifecycle,
|
||||
and error reporting that every backend duplicates.
|
||||
|
||||
Args:
|
||||
model_name: Progress tracking key (e.g. "qwen-tts-1.7B", "whisper-base").
|
||||
is_cached: Whether the model is already downloaded.
|
||||
filter_non_downloads: Whether to filter non-download tqdm bars.
|
||||
Defaults to `is_cached`.
|
||||
|
||||
Yields:
|
||||
The tracker context (already entered). The caller loads the model
|
||||
inside the `with` block. The tqdm patch is torn down on exit.
|
||||
|
||||
Usage:
|
||||
with model_load_progress("qwen-tts-1.7B", is_cached) as ctx:
|
||||
self.model = SomeModel.from_pretrained(...)
|
||||
"""
|
||||
if filter_non_downloads is None:
|
||||
filter_non_downloads = is_cached
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=filter_non_downloads)
|
||||
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
if not is_cached:
|
||||
task_manager.start_download(model_name)
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0,
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
try:
|
||||
yield tracker_context
|
||||
except Exception as e:
|
||||
# Report error to both managers
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
else:
|
||||
# Only mark complete if we were tracking a download
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
finally:
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
|
||||
def patch_chatterbox_f32(model) -> None:
|
||||
"""
|
||||
Patch float64 -> float32 dtype mismatches in upstream chatterbox.
|
||||
|
||||
librosa.load returns float64 numpy arrays. Multiple upstream code paths
|
||||
convert these to torch tensors via torch.from_numpy() without casting,
|
||||
then matmul against float32 model weights. This patches the two known
|
||||
entry points:
|
||||
|
||||
1. S3Tokenizer.log_mel_spectrogram — audio tensor hits _mel_filters (f32)
|
||||
2. VoiceEncoder.forward — float64 mel spectrograms hit LSTM weights (f32)
|
||||
"""
|
||||
import types
|
||||
|
||||
# Patch S3Tokenizer
|
||||
_tokzr = model.s3gen.tokenizer
|
||||
_orig_log_mel = _tokzr.log_mel_spectrogram.__func__
|
||||
|
||||
def _f32_log_mel(self_tokzr, audio, padding=0):
|
||||
import torch as _torch
|
||||
|
||||
if _torch.is_tensor(audio):
|
||||
audio = audio.float()
|
||||
return _orig_log_mel(self_tokzr, audio, padding)
|
||||
|
||||
_tokzr.log_mel_spectrogram = types.MethodType(_f32_log_mel, _tokzr)
|
||||
|
||||
# Patch VoiceEncoder
|
||||
_ve = model.ve
|
||||
_orig_ve_forward = _ve.forward.__func__
|
||||
|
||||
def _f32_ve_forward(self_ve, mels):
|
||||
return _orig_ve_forward(self_ve, mels.float())
|
||||
|
||||
_ve.forward = types.MethodType(_f32_ve_forward, _ve)
|
||||
@@ -8,7 +8,6 @@ on macOS due to known MPS tensor issues.
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import platform
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, List, Optional, Tuple
|
||||
@@ -16,9 +15,15 @@ from typing import ClassVar, List, Optional, Tuple
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.tasks import get_task_manager
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
patch_chatterbox_f32,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -45,17 +50,7 @@ class ChatterboxTTSBackend:
|
||||
self._model_load_lock = asyncio.Lock()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device. Forces CPU on macOS (MPS issue)."""
|
||||
if platform.system() == "Darwin":
|
||||
return "cpu"
|
||||
try:
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
except ImportError:
|
||||
pass
|
||||
return "cpu"
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -64,33 +59,7 @@ class ChatterboxTTSBackend:
|
||||
return CHATTERBOX_HF_REPO
|
||||
|
||||
def _is_model_cached(self, model_size: str = "default") -> bool:
|
||||
"""Check if the Chatterbox multilingual model is cached locally."""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / (
|
||||
"models--" + CHATTERBOX_HF_REPO.replace("/", "--")
|
||||
)
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
return False
|
||||
|
||||
# Check for multilingual weight files
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
for fname in _MTL_WEIGHT_FILES:
|
||||
if not any(snapshots_dir.rglob(fname)):
|
||||
return False
|
||||
return True
|
||||
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking Chatterbox cache: {e}")
|
||||
return False
|
||||
return is_model_cached(CHATTERBOX_HF_REPO, required_files=_MTL_WEIGHT_FILES)
|
||||
|
||||
async def load_model(self, model_size: str = "default") -> None:
|
||||
"""Load the Chatterbox multilingual model."""
|
||||
@@ -103,133 +72,45 @@ class ChatterboxTTSBackend:
|
||||
|
||||
def _load_model_sync(self):
|
||||
"""Synchronous model loading."""
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = "chatterbox-tts"
|
||||
|
||||
is_cached = self._is_model_cached()
|
||||
|
||||
# Set up HF progress tracking (intercepts tqdm for file-level progress)
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
if not is_cached:
|
||||
task_manager.start_download(model_name)
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0,
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
try:
|
||||
with model_load_progress(model_name, is_cached):
|
||||
device = self._get_device()
|
||||
self._device = device
|
||||
|
||||
logger.info(f"Loading Chatterbox Multilingual TTS on {device}...")
|
||||
|
||||
import torch
|
||||
from chatterbox.mtl_tts import ChatterboxMultilingualTTS
|
||||
|
||||
# Load into a local variable first, apply all patches, then
|
||||
# assign to self.model. This avoids leaving a half-initialised
|
||||
# model on self.model if any patch step raises an exception.
|
||||
#
|
||||
# Monkey-patch torch.load for CPU loading. The model's .pt files
|
||||
# were saved on CUDA; from_pretrained() doesn't pass map_location
|
||||
# so loading on CPU fails without this.
|
||||
try:
|
||||
if device == "cpu":
|
||||
_orig_torch_load = torch.load
|
||||
if device == "cpu":
|
||||
_orig_torch_load = torch.load
|
||||
|
||||
def _patched_load(*args, **kwargs):
|
||||
kwargs.setdefault("map_location", "cpu")
|
||||
return _orig_torch_load(*args, **kwargs)
|
||||
def _patched_load(*args, **kwargs):
|
||||
kwargs.setdefault("map_location", "cpu")
|
||||
return _orig_torch_load(*args, **kwargs)
|
||||
|
||||
with ChatterboxTTSBackend._load_lock:
|
||||
torch.load = _patched_load
|
||||
try:
|
||||
model = ChatterboxMultilingualTTS.from_pretrained(
|
||||
device=device,
|
||||
)
|
||||
finally:
|
||||
torch.load = _orig_torch_load
|
||||
else:
|
||||
model = ChatterboxMultilingualTTS.from_pretrained(
|
||||
device=device,
|
||||
)
|
||||
finally:
|
||||
tracker_context.__exit__(None, None, None)
|
||||
with ChatterboxTTSBackend._load_lock:
|
||||
torch.load = _patched_load
|
||||
try:
|
||||
model = ChatterboxMultilingualTTS.from_pretrained(device=device)
|
||||
finally:
|
||||
torch.load = _orig_torch_load
|
||||
else:
|
||||
model = ChatterboxMultilingualTTS.from_pretrained(device=device)
|
||||
|
||||
# Fix: transformers >= 4.36 defaults LlamaModel to sdpa attention
|
||||
# which doesn't support output_attentions=True (needed by
|
||||
# Chatterbox's AlignmentStreamAnalyzer). Force eager attention.
|
||||
# Fix sdpa attention for output_attentions support
|
||||
t3_tfmr = model.t3.tfmr
|
||||
if hasattr(t3_tfmr, "config") and hasattr(
|
||||
t3_tfmr.config, "_attn_implementation"
|
||||
):
|
||||
if hasattr(t3_tfmr, "config") and hasattr(t3_tfmr.config, "_attn_implementation"):
|
||||
t3_tfmr.config._attn_implementation = "eager"
|
||||
for layer in getattr(t3_tfmr, "layers", []):
|
||||
if hasattr(layer, "self_attn"):
|
||||
layer.self_attn._attn_implementation = "eager"
|
||||
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
# Patch float64 → float32 dtype mismatches in upstream chatterbox.
|
||||
# librosa.load returns float64 numpy; multiple upstream code paths
|
||||
# convert it to a torch tensor via torch.from_numpy() without
|
||||
# casting, then matmul it against float32 model weights.
|
||||
import types
|
||||
|
||||
# Patch S3Tokenizer (used by s3gen.tokenizer)
|
||||
_tokzr = model.s3gen.tokenizer
|
||||
_orig_log_mel = _tokzr.log_mel_spectrogram.__func__
|
||||
|
||||
def _f32_log_mel(self_tokzr, audio, padding=0):
|
||||
import torch as _torch
|
||||
if _torch.is_tensor(audio):
|
||||
audio = audio.float()
|
||||
return _orig_log_mel(self_tokzr, audio, padding)
|
||||
|
||||
_tokzr.log_mel_spectrogram = types.MethodType(_f32_log_mel, _tokzr)
|
||||
|
||||
# Patch VoiceEncoder
|
||||
_ve = model.ve
|
||||
_orig_ve_forward = _ve.forward.__func__
|
||||
|
||||
def _f32_ve_forward(self_ve, mels):
|
||||
return _orig_ve_forward(self_ve, mels.float())
|
||||
|
||||
_ve.forward = types.MethodType(_f32_ve_forward, _ve)
|
||||
|
||||
# All patches applied successfully — publish the model
|
||||
patch_chatterbox_f32(model)
|
||||
self.model = model
|
||||
|
||||
logger.info("Chatterbox Multilingual TTS loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
logger.error(
|
||||
"chatterbox-tts package not found. "
|
||||
"Install with: pip install chatterbox-tts"
|
||||
)
|
||||
if not is_cached:
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
logger.error(f"Failed to load Chatterbox: {e}\n{traceback.format_exc()}")
|
||||
if not is_cached:
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
logger.info("Chatterbox Multilingual TTS loaded successfully")
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
@@ -238,10 +119,7 @@ class ChatterboxTTSBackend:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
if device == "cuda":
|
||||
import torch
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
logger.info("Chatterbox unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
@@ -268,17 +146,7 @@ class ChatterboxTTSBackend:
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""Combine multiple reference samples."""
|
||||
combined_audio = []
|
||||
for path in audio_paths:
|
||||
audio, _sr = load_audio(path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
combined_text = " ".join(reference_texts)
|
||||
return mixed, combined_text
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts)
|
||||
|
||||
# Per-language generation defaults. Lower temp + higher cfg = clearer speech.
|
||||
_LANG_DEFAULTS: ClassVar[dict] = {
|
||||
@@ -331,7 +199,7 @@ class ChatterboxTTSBackend:
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
logger.info(f"[Chatterbox] Generating: lang={language}")
|
||||
|
||||
@@ -351,10 +219,7 @@ class ChatterboxTTSBackend:
|
||||
else:
|
||||
audio = np.asarray(wav, dtype=np.float32)
|
||||
|
||||
sample_rate = (
|
||||
getattr(self.model, "sr", None)
|
||||
or getattr(self.model, "sample_rate", 24000)
|
||||
)
|
||||
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
@@ -8,7 +8,6 @@ Forces CPU on macOS due to known MPS tensor issues.
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import platform
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, List, Optional, Tuple
|
||||
@@ -16,9 +15,15 @@ from typing import ClassVar, List, Optional, Tuple
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.tasks import get_task_manager
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
patch_chatterbox_f32,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -45,17 +50,7 @@ class ChatterboxTurboTTSBackend:
|
||||
self._model_load_lock = asyncio.Lock()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device. Forces CPU on macOS (MPS issue)."""
|
||||
if platform.system() == "Darwin":
|
||||
return "cpu"
|
||||
try:
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
except ImportError:
|
||||
pass
|
||||
return "cpu"
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -64,33 +59,7 @@ class ChatterboxTurboTTSBackend:
|
||||
return CHATTERBOX_TURBO_HF_REPO
|
||||
|
||||
def _is_model_cached(self, model_size: str = "default") -> bool:
|
||||
"""Check if the Chatterbox Turbo model is cached locally."""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / (
|
||||
"models--" + CHATTERBOX_TURBO_HF_REPO.replace("/", "--")
|
||||
)
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
return False
|
||||
|
||||
# Check for turbo weight files
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
for fname in _TURBO_WEIGHT_FILES:
|
||||
if not any(snapshots_dir.rglob(fname)):
|
||||
return False
|
||||
return True
|
||||
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking Chatterbox Turbo cache: {e}")
|
||||
return False
|
||||
return is_model_cached(CHATTERBOX_TURBO_HF_REPO, required_files=_TURBO_WEIGHT_FILES)
|
||||
|
||||
async def load_model(self, model_size: str = "default") -> None:
|
||||
"""Load the Chatterbox Turbo model."""
|
||||
@@ -103,59 +72,24 @@ class ChatterboxTurboTTSBackend:
|
||||
|
||||
def _load_model_sync(self):
|
||||
"""Synchronous model loading."""
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = "chatterbox-turbo"
|
||||
|
||||
is_cached = self._is_model_cached()
|
||||
|
||||
# Set up HF progress tracking (intercepts tqdm for file-level progress)
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
if not is_cached:
|
||||
task_manager.start_download(model_name)
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0,
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
try:
|
||||
with model_load_progress(model_name, is_cached):
|
||||
device = self._get_device()
|
||||
self._device = device
|
||||
|
||||
logger.info(f"Loading Chatterbox Turbo TTS on {device}...")
|
||||
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
from chatterbox.tts_turbo import ChatterboxTurboTTS
|
||||
|
||||
# Download model files ourselves so we can pass token=None
|
||||
# (upstream from_pretrained passes token=True which requires
|
||||
# a stored HF token even though the repo is public).
|
||||
try:
|
||||
local_path = snapshot_download(
|
||||
repo_id=CHATTERBOX_TURBO_HF_REPO,
|
||||
token=None,
|
||||
allow_patterns=[
|
||||
"*.safetensors", "*.json", "*.txt", "*.pt", "*.model",
|
||||
],
|
||||
)
|
||||
finally:
|
||||
tracker_context.__exit__(None, None, None)
|
||||
local_path = snapshot_download(
|
||||
repo_id=CHATTERBOX_TURBO_HF_REPO,
|
||||
token=None,
|
||||
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.pt", "*.model"],
|
||||
)
|
||||
|
||||
# Monkey-patch torch.load for CPU loading. The model's .pt files
|
||||
# were saved on CUDA; from_local() doesn't pass map_location
|
||||
# so loading on CPU fails without this.
|
||||
# Load into a local var, apply patches, then publish to
|
||||
# self.model so a failed patch doesn't leave us half-initialised.
|
||||
if device == "cpu":
|
||||
_orig_torch_load = torch.load
|
||||
|
||||
@@ -166,74 +100,16 @@ class ChatterboxTurboTTSBackend:
|
||||
with ChatterboxTurboTTSBackend._load_lock:
|
||||
torch.load = _patched_load
|
||||
try:
|
||||
model = ChatterboxTurboTTS.from_local(
|
||||
local_path, device,
|
||||
)
|
||||
model = ChatterboxTurboTTS.from_local(local_path, device)
|
||||
finally:
|
||||
torch.load = _orig_torch_load
|
||||
else:
|
||||
model = ChatterboxTurboTTS.from_local(
|
||||
local_path, device,
|
||||
)
|
||||
model = ChatterboxTurboTTS.from_local(local_path, device)
|
||||
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
# Patch float64 → float32 dtype mismatches in upstream chatterbox.
|
||||
# librosa.load returns float64 numpy; multiple upstream code paths
|
||||
# convert it to a torch tensor via torch.from_numpy() without
|
||||
# casting, then matmul it against float32 model weights.
|
||||
# We patch the two known entry points:
|
||||
#
|
||||
# 1. S3Tokenizer.log_mel_spectrogram — the audio tensor from
|
||||
# librosa hits _mel_filters (float32) in a matmul.
|
||||
# 2. VoiceEncoder.forward — float64 mel spectrograms hit the
|
||||
# float32 LSTM weights.
|
||||
import types
|
||||
|
||||
# Patch S3Tokenizer (used by s3gen.tokenizer)
|
||||
_tokzr = model.s3gen.tokenizer
|
||||
_orig_log_mel = _tokzr.log_mel_spectrogram.__func__
|
||||
|
||||
def _f32_log_mel(self_tokzr, audio, padding=0):
|
||||
import torch as _torch
|
||||
if _torch.is_tensor(audio):
|
||||
audio = audio.float()
|
||||
return _orig_log_mel(self_tokzr, audio, padding)
|
||||
|
||||
_tokzr.log_mel_spectrogram = types.MethodType(_f32_log_mel, _tokzr)
|
||||
|
||||
# Patch VoiceEncoder
|
||||
_ve = model.ve
|
||||
_orig_ve_forward = _ve.forward.__func__
|
||||
|
||||
def _f32_ve_forward(self_ve, mels):
|
||||
return _orig_ve_forward(self_ve, mels.float())
|
||||
|
||||
_ve.forward = types.MethodType(_f32_ve_forward, _ve)
|
||||
|
||||
# Only publish after all patches succeed
|
||||
patch_chatterbox_f32(model)
|
||||
self.model = model
|
||||
|
||||
logger.info("Chatterbox Turbo TTS loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
logger.error(
|
||||
"chatterbox-tts package not found. "
|
||||
"Install with: pip install chatterbox-tts"
|
||||
)
|
||||
if not is_cached:
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
logger.error(f"Failed to load Chatterbox Turbo: {e}\n{traceback.format_exc()}")
|
||||
if not is_cached:
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
logger.info("Chatterbox Turbo TTS loaded successfully")
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
@@ -242,10 +118,7 @@ class ChatterboxTurboTTSBackend:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
if device == "cuda":
|
||||
import torch
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
logger.info("Chatterbox Turbo unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
@@ -271,17 +144,7 @@ class ChatterboxTurboTTSBackend:
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""Combine multiple reference samples."""
|
||||
combined_audio = []
|
||||
for path in audio_paths:
|
||||
audio, _sr = load_audio(path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
combined_text = " ".join(reference_texts)
|
||||
return mixed, combined_text
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
@@ -317,7 +180,7 @@ class ChatterboxTurboTTSBackend:
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
logger.info("[Chatterbox Turbo] Generating (English)")
|
||||
|
||||
@@ -336,10 +199,7 @@ class ChatterboxTurboTTSBackend:
|
||||
else:
|
||||
audio = np.asarray(wav, dtype=np.float32)
|
||||
|
||||
sample_rate = (
|
||||
getattr(self.model, "sr", None)
|
||||
or getattr(self.model, "sample_rate", 24000)
|
||||
)
|
||||
sample_rate = getattr(self.model, "sr", None) or getattr(self.model, "sample_rate", 24000)
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
@@ -0,0 +1,346 @@
|
||||
"""
|
||||
HumeAI TADA TTS backend implementation.
|
||||
|
||||
Wraps HumeAI's TADA (Text-Acoustic Dual Alignment) model for
|
||||
high-quality voice cloning. Two model variants:
|
||||
- tada-1b: English-only, ~2B params (Llama 3.2 1B base)
|
||||
- tada-3b-ml: Multilingual, ~4B params (Llama 3.2 3B base)
|
||||
|
||||
Both use a shared encoder/codec (HumeAI/tada-codec). The encoder
|
||||
produces 1:1 aligned token embeddings from reference audio, and the
|
||||
causal LM generates speech via flow-matching diffusion.
|
||||
|
||||
24kHz output, bf16 inference on CUDA, fp32 on CPU.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from typing import ClassVar, List, Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# HuggingFace repos
|
||||
TADA_CODEC_REPO = "HumeAI/tada-codec"
|
||||
TADA_1B_REPO = "HumeAI/tada-1b"
|
||||
TADA_3B_ML_REPO = "HumeAI/tada-3b-ml"
|
||||
|
||||
TADA_MODEL_REPOS = {
|
||||
"1B": TADA_1B_REPO,
|
||||
"3B": TADA_3B_ML_REPO,
|
||||
}
|
||||
|
||||
# Key weight files for cache detection
|
||||
_TADA_MODEL_WEIGHT_FILES = [
|
||||
"model.safetensors",
|
||||
]
|
||||
|
||||
_TADA_CODEC_WEIGHT_FILES = [
|
||||
"encoder/model.safetensors",
|
||||
]
|
||||
|
||||
|
||||
class HumeTadaBackend:
|
||||
"""HumeAI TADA TTS backend for high-quality voice cloning."""
|
||||
|
||||
_load_lock: ClassVar[threading.Lock] = threading.Lock()
|
||||
|
||||
def __init__(self):
|
||||
self.model = None
|
||||
self.encoder = None
|
||||
self.model_size = "1B" # default to 1B
|
||||
self._device = None
|
||||
self._model_load_lock = asyncio.Lock()
|
||||
|
||||
def _get_device(self) -> str:
|
||||
# Force CPU on macOS — MPS has issues with flow matching
|
||||
# and large vocab lm_head (>65536 output channels)
|
||||
return get_torch_device(force_cpu_on_mac=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str = "1B") -> str:
|
||||
return TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||
|
||||
def _is_model_cached(self, model_size: str = "1B") -> bool:
|
||||
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||
model_cached = is_model_cached(repo, required_files=_TADA_MODEL_WEIGHT_FILES)
|
||||
codec_cached = is_model_cached(TADA_CODEC_REPO, required_files=_TADA_CODEC_WEIGHT_FILES)
|
||||
return model_cached and codec_cached
|
||||
|
||||
async def load_model(self, model_size: str = "1B") -> None:
|
||||
"""Load the TADA model and encoder."""
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
async with self._model_load_lock:
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
# Unload existing model if switching sizes
|
||||
if self.model is not None:
|
||||
self.unload_model()
|
||||
self.model_size = model_size
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
def _load_model_sync(self, model_size: str = "1B"):
|
||||
"""Synchronous model loading with progress tracking."""
|
||||
model_name = f"tada-{model_size.lower()}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
repo = TADA_MODEL_REPOS.get(model_size, TADA_1B_REPO)
|
||||
|
||||
with model_load_progress(model_name, is_cached):
|
||||
# Install DAC shim before importing tada — tada's encoder/decoder
|
||||
# import dac.nn.layers.Snake1d which requires the descript-audio-codec
|
||||
# package. The real package pulls in onnx/tensorboard/matplotlib via
|
||||
# descript-audiotools, so we use a lightweight shim instead.
|
||||
from ..utils.dac_shim import install_dac_shim
|
||||
|
||||
install_dac_shim()
|
||||
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
device = self._get_device()
|
||||
self._device = device
|
||||
logger.info(f"Loading HumeAI TADA {model_size} on {device}...")
|
||||
|
||||
# Download codec (encoder + decoder) if not cached
|
||||
logger.info("Downloading TADA codec...")
|
||||
snapshot_download(
|
||||
repo_id=TADA_CODEC_REPO,
|
||||
token=None,
|
||||
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin"],
|
||||
)
|
||||
|
||||
# Download model weights if not cached
|
||||
logger.info(f"Downloading TADA {model_size} model...")
|
||||
snapshot_download(
|
||||
repo_id=repo,
|
||||
token=None,
|
||||
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.bin", "*.model"],
|
||||
)
|
||||
|
||||
# TADA hardcodes "meta-llama/Llama-3.2-1B" as the tokenizer
|
||||
# source in its Aligner and TadaForCausalLM.from_pretrained().
|
||||
# That repo is gated (requires Meta license acceptance).
|
||||
# Download the tokenizer from an ungated mirror and get its
|
||||
# local cache path so we can point TADA at it directly.
|
||||
logger.info("Downloading Llama tokenizer (ungated mirror)...")
|
||||
tokenizer_path = snapshot_download(
|
||||
repo_id="unsloth/Llama-3.2-1B",
|
||||
token=None,
|
||||
allow_patterns=["tokenizer*", "special_tokens*"],
|
||||
)
|
||||
|
||||
# Determine dtype — use bf16 on CUDA/XPU for ~50% memory savings
|
||||
if device == "cuda" and torch.cuda.is_bf16_supported():
|
||||
model_dtype = torch.bfloat16
|
||||
elif device == "xpu":
|
||||
# Intel Arc (Alchemist+) supports bf16 natively
|
||||
model_dtype = torch.bfloat16
|
||||
else:
|
||||
model_dtype = torch.float32
|
||||
|
||||
# Patch the Aligner config class to use the local tokenizer
|
||||
# path instead of the gated "meta-llama/Llama-3.2-1B" default.
|
||||
# This avoids monkey-patching AutoTokenizer.from_pretrained
|
||||
# which corrupts the classmethod descriptor for other engines.
|
||||
from tada.modules.aligner import AlignerConfig
|
||||
|
||||
AlignerConfig.tokenizer_name = tokenizer_path
|
||||
|
||||
# Load encoder (only needed for voice prompt encoding)
|
||||
from tada.modules.encoder import Encoder
|
||||
|
||||
logger.info("Loading TADA encoder...")
|
||||
self.encoder = Encoder.from_pretrained(TADA_CODEC_REPO, subfolder="encoder").to(device)
|
||||
self.encoder.eval()
|
||||
|
||||
# Load the causal LM (includes decoder for wav generation).
|
||||
# TadaForCausalLM.from_pretrained() calls
|
||||
# getattr(config, "tokenizer_name", "meta-llama/Llama-3.2-1B")
|
||||
# which hits the gated repo. Pre-load the config from HF,
|
||||
# inject the local tokenizer path, then pass it in.
|
||||
from tada.modules.tada import TadaForCausalLM, TadaConfig
|
||||
|
||||
logger.info(f"Loading TADA {model_size} model...")
|
||||
config = TadaConfig.from_pretrained(repo)
|
||||
config.tokenizer_name = tokenizer_path
|
||||
self.model = TadaForCausalLM.from_pretrained(repo, config=config, torch_dtype=model_dtype).to(device)
|
||||
self.model.eval()
|
||||
|
||||
logger.info(f"HumeAI TADA {model_size} loaded successfully on {device}")
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model and encoder to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
if self.encoder is not None:
|
||||
del self.encoder
|
||||
self.encoder = None
|
||||
|
||||
device = self._device
|
||||
self._device = None
|
||||
|
||||
if device:
|
||||
empty_device_cache(device)
|
||||
|
||||
logger.info("HumeAI TADA unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio using TADA's encoder.
|
||||
|
||||
TADA's encoder performs forced alignment between audio and text tokens,
|
||||
producing an EncoderOutput with 1:1 token-audio alignment. If no
|
||||
reference_text is provided, the encoder uses built-in ASR (English only).
|
||||
|
||||
We serialize the EncoderOutput to a dict for caching.
|
||||
"""
|
||||
await self.load_model(self.model_size)
|
||||
|
||||
cache_key = ("tada_" + get_cache_key(audio_path, reference_text)) if use_cache else None
|
||||
|
||||
if cache_key:
|
||||
cached = get_cached_voice_prompt(cache_key)
|
||||
if cached is not None and isinstance(cached, dict):
|
||||
return cached, True
|
||||
|
||||
def _encode_sync():
|
||||
import torch
|
||||
import soundfile as sf
|
||||
|
||||
device = self._device
|
||||
|
||||
# Load audio with soundfile (torchaudio 2.10+ requires torchcodec)
|
||||
audio_np, sr = sf.read(str(audio_path), dtype="float32")
|
||||
audio = torch.from_numpy(audio_np).float()
|
||||
if audio.ndim == 1:
|
||||
audio = audio.unsqueeze(0) # (samples,) -> (1, samples)
|
||||
else:
|
||||
audio = audio.T # (samples, channels) -> (channels, samples)
|
||||
audio = audio.to(device)
|
||||
|
||||
# Encode with forced alignment
|
||||
text_arg = [reference_text] if reference_text else None
|
||||
prompt = self.encoder(audio, text=text_arg, sample_rate=sr)
|
||||
|
||||
# Serialize EncoderOutput to a dict of CPU tensors for caching
|
||||
prompt_dict = {}
|
||||
for field_name in prompt.__dataclass_fields__:
|
||||
val = getattr(prompt, field_name)
|
||||
if isinstance(val, torch.Tensor):
|
||||
prompt_dict[field_name] = val.detach().cpu()
|
||||
elif isinstance(val, list):
|
||||
prompt_dict[field_name] = val
|
||||
elif isinstance(val, (int, float)):
|
||||
prompt_dict[field_name] = val
|
||||
else:
|
||||
prompt_dict[field_name] = val
|
||||
return prompt_dict
|
||||
|
||||
encoded = await asyncio.to_thread(_encode_sync)
|
||||
|
||||
if cache_key:
|
||||
cache_voice_prompt(cache_key, encoded)
|
||||
|
||||
return encoded, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts, sample_rate=24000)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text using HumeAI TADA.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Serialized EncoderOutput dict from create_voice_prompt()
|
||||
language: Language code (en, ar, de, es, fr, it, ja, pl, pt, zh)
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Not supported by TADA (ignored)
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate=24000)
|
||||
"""
|
||||
await self.load_model(self.model_size)
|
||||
|
||||
def _generate_sync():
|
||||
import torch
|
||||
from tada.modules.encoder import EncoderOutput
|
||||
|
||||
if seed is not None:
|
||||
manual_seed(seed, self._device)
|
||||
|
||||
device = self._device
|
||||
|
||||
# Reconstruct EncoderOutput from the cached dict
|
||||
restored = {}
|
||||
for k, v in voice_prompt.items():
|
||||
if isinstance(v, torch.Tensor):
|
||||
# Move to device and match model dtype for float tensors
|
||||
if v.is_floating_point():
|
||||
model_dtype = next(self.model.parameters()).dtype
|
||||
restored[k] = v.to(device=device, dtype=model_dtype)
|
||||
else:
|
||||
restored[k] = v.to(device=device)
|
||||
else:
|
||||
restored[k] = v
|
||||
|
||||
prompt = EncoderOutput(**restored)
|
||||
|
||||
# For non-English with the 3B-ML model, we could reload the
|
||||
# encoder with the language-specific aligner. However, the
|
||||
# generation itself is language-agnostic — only the encoder's
|
||||
# aligner changes. Since we encode at create_voice_prompt time,
|
||||
# the language is already baked in. For simplicity, we don't
|
||||
# reload the encoder here.
|
||||
|
||||
logger.info(f"[TADA] Generating ({language}), text length: {len(text)}")
|
||||
|
||||
output = self.model.generate(
|
||||
prompt=prompt,
|
||||
text=text,
|
||||
)
|
||||
|
||||
# output.audio is a list of tensors (one per batch item)
|
||||
if output.audio and output.audio[0] is not None:
|
||||
audio_tensor = output.audio[0]
|
||||
audio = audio_tensor.detach().cpu().numpy().squeeze().astype(np.float32)
|
||||
else:
|
||||
logger.warning("[TADA] Generation produced no audio")
|
||||
audio = np.zeros(24000, dtype=np.float32)
|
||||
|
||||
return audio, 24000
|
||||
|
||||
return await asyncio.to_thread(_generate_sync)
|
||||
@@ -0,0 +1,288 @@
|
||||
"""
|
||||
Kokoro TTS backend implementation.
|
||||
|
||||
Wraps the Kokoro-82M model for fast, lightweight text-to-speech.
|
||||
82M parameters, CPU realtime, 24kHz output, Apache 2.0 license.
|
||||
|
||||
Kokoro uses pre-built voice style vectors (not traditional zero-shot cloning
|
||||
from arbitrary audio). Voice prompts are stored as deferred references to
|
||||
HF-hosted voice .pt files.
|
||||
|
||||
Languages supported (via misaki G2P):
|
||||
- American English (a), British English (b)
|
||||
- Spanish (e), French (f), Hindi (h), Italian (i), Portuguese (p)
|
||||
- Japanese (j) — requires misaki[ja]
|
||||
- Chinese (z) — requires misaki[zh]
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from .base import (
|
||||
get_torch_device,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# HuggingFace repo for model + voice detection
|
||||
KOKORO_HF_REPO = "hexgrad/Kokoro-82M"
|
||||
KOKORO_SAMPLE_RATE = 24000
|
||||
|
||||
# Default voice if none specified
|
||||
KOKORO_DEFAULT_VOICE = "af_heart"
|
||||
|
||||
# All available Kokoro voices: (voice_id, display_name, gender, lang_code)
|
||||
KOKORO_VOICES = [
|
||||
# American English female
|
||||
("af_alloy", "Alloy", "female", "en"),
|
||||
("af_aoede", "Aoede", "female", "en"),
|
||||
("af_bella", "Bella", "female", "en"),
|
||||
("af_heart", "Heart", "female", "en"),
|
||||
("af_jessica", "Jessica", "female", "en"),
|
||||
("af_kore", "Kore", "female", "en"),
|
||||
("af_nicole", "Nicole", "female", "en"),
|
||||
("af_nova", "Nova", "female", "en"),
|
||||
("af_river", "River", "female", "en"),
|
||||
("af_sarah", "Sarah", "female", "en"),
|
||||
("af_sky", "Sky", "female", "en"),
|
||||
# American English male
|
||||
("am_adam", "Adam", "male", "en"),
|
||||
("am_echo", "Echo", "male", "en"),
|
||||
("am_eric", "Eric", "male", "en"),
|
||||
("am_fenrir", "Fenrir", "male", "en"),
|
||||
("am_liam", "Liam", "male", "en"),
|
||||
("am_michael", "Michael", "male", "en"),
|
||||
("am_onyx", "Onyx", "male", "en"),
|
||||
("am_puck", "Puck", "male", "en"),
|
||||
("am_santa", "Santa", "male", "en"),
|
||||
# British English female
|
||||
("bf_alice", "Alice", "female", "en"),
|
||||
("bf_emma", "Emma", "female", "en"),
|
||||
("bf_isabella", "Isabella", "female", "en"),
|
||||
("bf_lily", "Lily", "female", "en"),
|
||||
# British English male
|
||||
("bm_daniel", "Daniel", "male", "en"),
|
||||
("bm_fable", "Fable", "male", "en"),
|
||||
("bm_george", "George", "male", "en"),
|
||||
("bm_lewis", "Lewis", "male", "en"),
|
||||
# Spanish
|
||||
("ef_dora", "Dora", "female", "es"),
|
||||
("em_alex", "Alex", "male", "es"),
|
||||
("em_santa", "Santa", "male", "es"),
|
||||
# French
|
||||
("ff_siwis", "Siwis", "female", "fr"),
|
||||
# Hindi
|
||||
("hf_alpha", "Alpha", "female", "hi"),
|
||||
("hf_beta", "Beta", "female", "hi"),
|
||||
("hm_omega", "Omega", "male", "hi"),
|
||||
("hm_psi", "Psi", "male", "hi"),
|
||||
# Italian
|
||||
("if_sara", "Sara", "female", "it"),
|
||||
("im_nicola", "Nicola", "male", "it"),
|
||||
# Japanese
|
||||
("jf_alpha", "Alpha", "female", "ja"),
|
||||
("jf_gongitsune", "Gongitsune", "female", "ja"),
|
||||
("jf_nezumi", "Nezumi", "female", "ja"),
|
||||
("jf_tebukuro", "Tebukuro", "female", "ja"),
|
||||
("jm_kumo", "Kumo", "male", "ja"),
|
||||
# Portuguese
|
||||
("pf_dora", "Dora", "female", "pt"),
|
||||
("pm_alex", "Alex", "male", "pt"),
|
||||
("pm_santa", "Santa", "male", "pt"),
|
||||
# Chinese
|
||||
("zf_xiaobei", "Xiaobei", "female", "zh"),
|
||||
("zf_xiaoni", "Xiaoni", "female", "zh"),
|
||||
("zf_xiaoxiao", "Xiaoxiao", "female", "zh"),
|
||||
("zf_xiaoyi", "Xiaoyi", "female", "zh"),
|
||||
]
|
||||
|
||||
# Map our ISO language codes to Kokoro lang_code characters
|
||||
LANG_CODE_MAP = {
|
||||
"en": "a", # American English
|
||||
"es": "e",
|
||||
"fr": "f",
|
||||
"hi": "h",
|
||||
"it": "i",
|
||||
"pt": "p",
|
||||
"ja": "j",
|
||||
"zh": "z",
|
||||
}
|
||||
|
||||
|
||||
class KokoroTTSBackend:
|
||||
"""Kokoro-82M TTS backend — tiny, fast, CPU-friendly."""
|
||||
|
||||
def __init__(self):
|
||||
self._model = None
|
||||
self._pipelines: dict = {} # lang_code -> KPipeline
|
||||
self._device: Optional[str] = None
|
||||
self.model_size = "default"
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Select device. Kokoro supports CUDA and CPU. MPS needs fallback env var."""
|
||||
device = get_torch_device(allow_mps=False)
|
||||
# Kokoro can use MPS but requires PYTORCH_ENABLE_MPS_FALLBACK=1
|
||||
# For now, skip MPS to avoid user confusion — CPU is already realtime
|
||||
return device
|
||||
|
||||
@property
|
||||
def device(self) -> str:
|
||||
if self._device is None:
|
||||
self._device = self._get_device()
|
||||
return self._device
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self._model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
return KOKORO_HF_REPO
|
||||
|
||||
def _is_model_cached(self, model_size: str = "default") -> bool:
|
||||
"""Check if Kokoro model files are cached locally."""
|
||||
from .base import is_model_cached
|
||||
|
||||
return is_model_cached(
|
||||
KOKORO_HF_REPO,
|
||||
required_files=["config.json", "kokoro-v1_0.pth"],
|
||||
)
|
||||
|
||||
async def load_model(self, model_size: str = "default") -> None:
|
||||
"""Load the Kokoro model."""
|
||||
if self._model is not None:
|
||||
return
|
||||
await asyncio.to_thread(self._load_model_sync)
|
||||
|
||||
def _load_model_sync(self):
|
||||
"""Synchronous model loading."""
|
||||
model_name = "kokoro"
|
||||
is_cached = self._is_model_cached()
|
||||
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from kokoro import KModel
|
||||
|
||||
device = self.device
|
||||
logger.info(f"Loading Kokoro-82M on {device}...")
|
||||
|
||||
self._model = KModel(repo_id=KOKORO_HF_REPO).to(device).eval()
|
||||
|
||||
logger.info("Kokoro-82M loaded successfully")
|
||||
|
||||
def _get_pipeline(self, lang_code: str):
|
||||
"""Get or create a KPipeline for the given language code."""
|
||||
kokoro_lang = LANG_CODE_MAP.get(lang_code, "a")
|
||||
|
||||
if kokoro_lang not in self._pipelines:
|
||||
from kokoro import KPipeline
|
||||
|
||||
# Create pipeline with our existing model (no redundant model loading)
|
||||
self._pipelines[kokoro_lang] = KPipeline(
|
||||
lang_code=kokoro_lang,
|
||||
repo_id=KOKORO_HF_REPO,
|
||||
model=self._model,
|
||||
)
|
||||
|
||||
return self._pipelines[kokoro_lang]
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
if self._model is not None:
|
||||
del self._model
|
||||
self._model = None
|
||||
self._pipelines.clear()
|
||||
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
logger.info("Kokoro unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt for Kokoro.
|
||||
|
||||
Kokoro doesn't do traditional voice cloning from arbitrary audio.
|
||||
When called for a cloned profile (fallback), uses the default voice.
|
||||
For preset profiles, the voice_prompt dict is built by the profile
|
||||
service and bypasses this method entirely.
|
||||
"""
|
||||
return {
|
||||
"voice_type": "preset",
|
||||
"preset_engine": "kokoro",
|
||||
"preset_voice_id": KOKORO_DEFAULT_VOICE,
|
||||
}, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: list[str],
|
||||
reference_texts: list[str],
|
||||
) -> tuple[np.ndarray, str]:
|
||||
"""Combine voice prompts — uses base implementation for audio concatenation."""
|
||||
return await _combine_voice_prompts(
|
||||
audio_paths, reference_texts, sample_rate=KOKORO_SAMPLE_RATE
|
||||
)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio from text using Kokoro.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Dict with kokoro_voice key
|
||||
language: Language code
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Not supported by Kokoro (ignored)
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
await self.load_model()
|
||||
|
||||
voice_name = voice_prompt.get("preset_voice_id") or voice_prompt.get("kokoro_voice") or KOKORO_DEFAULT_VOICE
|
||||
|
||||
def _generate_sync():
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
|
||||
pipeline = self._get_pipeline(language)
|
||||
|
||||
# Generate all chunks and concatenate
|
||||
audio_chunks = []
|
||||
for result in pipeline(text, voice=voice_name, speed=1.0):
|
||||
if result.audio is not None:
|
||||
chunk = result.audio
|
||||
if isinstance(chunk, torch.Tensor):
|
||||
chunk = chunk.detach().cpu().numpy()
|
||||
audio_chunks.append(chunk.squeeze())
|
||||
|
||||
if not audio_chunks:
|
||||
# Return 1 second of silence as fallback
|
||||
return np.zeros(KOKORO_SAMPLE_RATE, dtype=np.float32), KOKORO_SAMPLE_RATE
|
||||
|
||||
audio = np.concatenate(audio_chunks)
|
||||
return audio.astype(np.float32), KOKORO_SAMPLE_RATE
|
||||
|
||||
return await asyncio.to_thread(_generate_sync)
|
||||
@@ -7,16 +7,20 @@ Wraps the LuxTTS (ZipVoice) model for zero-shot voice cloning.
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Tuple
|
||||
from typing import Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
|
||||
from . import TTSBackend
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -33,14 +37,7 @@ class LuxTTSBackend:
|
||||
self._device = None
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "mps"
|
||||
return "cpu"
|
||||
return get_torch_device(allow_mps=True, allow_xpu=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
@@ -55,35 +52,10 @@ class LuxTTSBackend:
|
||||
return LUXTTS_HF_REPO
|
||||
|
||||
def _is_model_cached(self, model_size: str = "default") -> bool:
|
||||
"""Check if LuxTTS model weights are cached locally."""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
repo_cache = (
|
||||
Path(hf_constants.HF_HUB_CACHE)
|
||||
/ ("models--" + LUXTTS_HF_REPO.replace("/", "--"))
|
||||
)
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
return False
|
||||
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
has_weights = any(snapshots_dir.rglob("*.pt")) or any(
|
||||
snapshots_dir.rglob("*.safetensors")
|
||||
) or any(snapshots_dir.rglob("*.onnx")) or any(
|
||||
snapshots_dir.rglob("*.bin")
|
||||
)
|
||||
return has_weights
|
||||
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking LuxTTS cache: {e}")
|
||||
return False
|
||||
return is_model_cached(
|
||||
LUXTTS_HF_REPO,
|
||||
weight_extensions=(".pt", ".safetensors", ".onnx", ".bin"),
|
||||
)
|
||||
|
||||
async def load_model(self, model_size: str = "default") -> None:
|
||||
"""Load the LuxTTS model."""
|
||||
@@ -93,78 +65,38 @@ class LuxTTSBackend:
|
||||
await asyncio.to_thread(self._load_model_sync)
|
||||
|
||||
def _load_model_sync(self):
|
||||
"""Synchronous model loading."""
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = "luxtts"
|
||||
|
||||
is_cached = self._is_model_cached()
|
||||
|
||||
# Set up HF progress tracking (intercepts tqdm for file-level progress)
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
if not is_cached:
|
||||
task_manager.start_download(model_name)
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0,
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
try:
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from zipvoice.luxvoice import LuxTTS
|
||||
|
||||
device = self.device
|
||||
logger.info(f"Loading LuxTTS on {device}...")
|
||||
|
||||
# LuxTTS constructor downloads model and loads everything
|
||||
try:
|
||||
if device == "cpu":
|
||||
import os
|
||||
threads = os.cpu_count() or 4
|
||||
self.model = LuxTTS(
|
||||
model_path=LUXTTS_HF_REPO,
|
||||
device="cpu",
|
||||
threads=min(threads, 8),
|
||||
)
|
||||
else:
|
||||
self.model = LuxTTS(
|
||||
model_path=LUXTTS_HF_REPO,
|
||||
device=device,
|
||||
)
|
||||
finally:
|
||||
tracker_context.__exit__(None, None, None)
|
||||
if device == "cpu":
|
||||
import os
|
||||
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
threads = os.cpu_count() or 4
|
||||
self.model = LuxTTS(
|
||||
model_path=LUXTTS_HF_REPO,
|
||||
device="cpu",
|
||||
threads=min(threads, 8),
|
||||
)
|
||||
else:
|
||||
self.model = LuxTTS(model_path=LUXTTS_HF_REPO, device=device)
|
||||
|
||||
logger.info("LuxTTS loaded successfully")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
logger.error(f"Failed to load LuxTTS: {e}\n{traceback.format_exc()}")
|
||||
if not is_cached:
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
logger.info("LuxTTS loaded successfully")
|
||||
|
||||
def unload_model(self) -> None:
|
||||
"""Unload model to free memory."""
|
||||
if self.model is not None:
|
||||
device = self.device
|
||||
del self.model
|
||||
self.model = None
|
||||
self._device = None
|
||||
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
empty_device_cache(device)
|
||||
|
||||
logger.info("LuxTTS unloaded")
|
||||
|
||||
@@ -205,28 +137,8 @@ class LuxTTSBackend:
|
||||
|
||||
return encoded, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples.
|
||||
|
||||
LuxTTS doesn't have native multi-prompt support, so we concatenate
|
||||
the audio and let encode_prompt handle the combined clip.
|
||||
"""
|
||||
combined_audio = []
|
||||
for path in audio_paths:
|
||||
audio, _sr = load_audio(path, sample_rate=24000)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
async def combine_voice_prompts(self, audio_paths, reference_texts):
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts, sample_rate=24000)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
@@ -252,12 +164,8 @@ class LuxTTSBackend:
|
||||
await self.load_model()
|
||||
|
||||
def _generate_sync():
|
||||
import torch
|
||||
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
manual_seed(seed, self.device)
|
||||
|
||||
wav = self.model.generate_speech(
|
||||
text=text,
|
||||
|
||||
+93
-341
@@ -4,49 +4,44 @@ MLX backend implementation for TTS and STT using mlx-audio.
|
||||
|
||||
from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import logging
|
||||
import numpy as np
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# PATCH: Import and apply offline patch BEFORE any huggingface_hub usage
|
||||
# This prevents mlx_audio from making network requests when models are cached
|
||||
from ..utils.hf_offline_patch import patch_huggingface_hub_offline, ensure_original_qwen_config_cached
|
||||
|
||||
patch_huggingface_hub_offline()
|
||||
ensure_original_qwen_config_cached()
|
||||
|
||||
from . import TTSBackend, STTBackend
|
||||
from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
|
||||
from .base import is_model_cached, combine_voice_prompts as _combine_voice_prompts, model_load_progress
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
LANGUAGE_CODE_TO_NAME = {
|
||||
"zh": "chinese", "en": "english", "ja": "japanese", "ko": "korean",
|
||||
"de": "german", "fr": "french", "ru": "russian", "pt": "portuguese",
|
||||
"es": "spanish", "it": "italian",
|
||||
}
|
||||
from ..utils.hf_offline_patch import force_offline_if_cached
|
||||
|
||||
|
||||
class MLXTTSBackend:
|
||||
"""MLX-based TTS backend using mlx-audio."""
|
||||
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self._current_model_size = None
|
||||
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get the MLX model path.
|
||||
|
||||
|
||||
Args:
|
||||
model_size: Model size (1.7B or 0.6B)
|
||||
|
||||
|
||||
Returns:
|
||||
HuggingFace Hub model ID for MLX
|
||||
"""
|
||||
@@ -56,187 +51,71 @@ class MLXTTSBackend:
|
||||
# 0.6B not yet converted to MLX format
|
||||
"0.6B": "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16", # Fallback to 1.7B
|
||||
}
|
||||
|
||||
|
||||
if model_size not in mlx_model_map:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
|
||||
|
||||
hf_model_id = mlx_model_map[model_size]
|
||||
print(f"Will download MLX model from HuggingFace Hub: {hf_model_id}")
|
||||
|
||||
logger.info("Will download MLX model from HuggingFace Hub: %s", hf_model_id)
|
||||
|
||||
return hf_model_id
|
||||
|
||||
|
||||
def _is_model_cached(self, model_size: str) -> bool:
|
||||
"""
|
||||
Check if the model is already cached locally AND fully downloaded.
|
||||
|
||||
Args:
|
||||
model_size: Model size to check
|
||||
|
||||
Returns:
|
||||
True if model is fully cached, False if missing or incomplete
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
model_path = self._get_model_path(model_size)
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_path.replace("/", "--"))
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
# Check for .incomplete files - if any exist, download is still in progress
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
print(f"[_is_model_cached] Found .incomplete files for {model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
# Check that actual model weight files exist in snapshots
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
has_weights = (
|
||||
any(snapshots_dir.rglob("*.safetensors")) or
|
||||
any(snapshots_dir.rglob("*.bin")) or
|
||||
any(snapshots_dir.rglob("*.npz"))
|
||||
)
|
||||
if not has_weights:
|
||||
print(f"[_is_model_cached] No model weights found for {model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[_is_model_cached] Error checking cache for {model_size}: {e}")
|
||||
return False
|
||||
|
||||
return is_model_cached(
|
||||
self._get_model_path(model_size),
|
||||
weight_extensions=(".safetensors", ".bin", ".npz"),
|
||||
)
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the MLX TTS model.
|
||||
|
||||
|
||||
Args:
|
||||
model_size: Model size to load (1.7B or 0.6B)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
|
||||
# If already loaded with correct size, return
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
|
||||
# Unload existing model if different size requested
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
# Get model path BEFORE importing mlx_audio
|
||||
model_path = self._get_model_path(model_size)
|
||||
|
||||
# Set up progress tracking
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
|
||||
# Check if model is already cached
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Set up progress callback
|
||||
# If cached: filter out non-download progress
|
||||
# If not cached: report all progress (we're actually downloading)
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
|
||||
print(f"Loading MLX TTS model {model_size}...")
|
||||
|
||||
# Only track download progress if model is NOT cached
|
||||
if not is_cached:
|
||||
# Start tracking download task
|
||||
task_manager.start_download(model_name)
|
||||
|
||||
# Initialize progress state so SSE endpoint has initial data to send
|
||||
# This provides immediate feedback while HuggingFace fetches metadata
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0, # Will be updated once actual total is known
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# IMPORTANT: Patch tqdm BEFORE importing mlx_audio
|
||||
# Otherwise mlx_audio caches reference to original tqdm
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
# PATCH: Force offline mode when model is already cached
|
||||
# This prevents crashes when HuggingFace is unreachable
|
||||
original_hf_hub_offline = os.environ.get("HF_HUB_OFFLINE")
|
||||
if is_cached:
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
print(f"[PATCH] Model {model_size} is cached, forcing HF_HUB_OFFLINE=1 to avoid network requests")
|
||||
|
||||
# Import mlx_audio AFTER patching tqdm
|
||||
model_path = self._get_model_path(model_size)
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from mlx_audio.tts import load
|
||||
|
||||
# Load MLX model (downloads automatically)
|
||||
try:
|
||||
|
||||
logger.info("Loading MLX TTS model %s...", model_size)
|
||||
|
||||
with force_offline_if_cached(is_cached, model_name):
|
||||
self.model = load(model_path)
|
||||
except Exception as load_error:
|
||||
# If offline mode failed, try with network enabled as fallback
|
||||
if is_cached and "offline" in str(load_error).lower():
|
||||
print(f"[PATCH] Offline load failed, trying with network: {load_error}")
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
self.model = load(model_path)
|
||||
else:
|
||||
raise
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
# Restore original HF_HUB_OFFLINE setting
|
||||
if original_hf_hub_offline is not None:
|
||||
os.environ["HF_HUB_OFFLINE"] = original_hf_hub_offline
|
||||
else:
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
|
||||
# Only mark download as complete if we were tracking it
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"MLX TTS model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading MLX TTS model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
logger.info("MLX TTS model %s loaded successfully", model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
print("MLX TTS model unloaded")
|
||||
|
||||
logger.info("MLX TTS model unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
@@ -245,20 +124,20 @@ class MLXTTSBackend:
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
|
||||
MLX backend stores voice prompt as a dict with audio path and text.
|
||||
The actual voice prompt processing happens during generation.
|
||||
|
||||
|
||||
Args:
|
||||
audio_path: Path to reference audio file
|
||||
reference_text: Transcript of reference audio
|
||||
use_cache: Whether to use cached prompt if available
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
|
||||
# Check cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
@@ -272,53 +151,25 @@ class MLXTTSBackend:
|
||||
return cached_prompt, True
|
||||
else:
|
||||
# Cached file no longer exists, invalidate cache
|
||||
print(f"Cached audio file not found: {cached_audio_path}, regenerating prompt")
|
||||
|
||||
logger.warning("Cached audio file not found: %s, regenerating prompt", cached_audio_path)
|
||||
|
||||
# MLX voice prompt format - store audio path and text
|
||||
# The model will process this during generation
|
||||
voice_prompt_items = {
|
||||
"ref_audio": str(audio_path),
|
||||
"ref_text": reference_text,
|
||||
}
|
||||
|
||||
|
||||
# Cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||
|
||||
|
||||
return voice_prompt_items, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples for better quality.
|
||||
|
||||
Args:
|
||||
audio_paths: List of audio file paths
|
||||
reference_texts: List of reference texts
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio, combined_text)
|
||||
"""
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate audio
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
|
||||
async def combine_voice_prompts(self, audio_paths, reference_texts):
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
@@ -342,7 +193,7 @@ class MLXTTSBackend:
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
print(f"Generating audio for text: {text}")
|
||||
logger.info("Generating audio for text: %s", text)
|
||||
|
||||
def _generate_sync():
|
||||
"""Run synchronous generation in thread pool."""
|
||||
@@ -354,20 +205,21 @@ class MLXTTSBackend:
|
||||
# Set seed if provided (MLX uses numpy random)
|
||||
if seed is not None:
|
||||
import mlx.core as mx
|
||||
|
||||
np.random.seed(seed)
|
||||
mx.random.seed(seed)
|
||||
|
||||
|
||||
# Extract voice prompt info
|
||||
ref_audio = voice_prompt.get("ref_audio") or voice_prompt.get("ref_audio_path")
|
||||
ref_text = voice_prompt.get("ref_text", "")
|
||||
|
||||
|
||||
# Validate that the audio file exists
|
||||
if ref_audio and not Path(ref_audio).exists():
|
||||
print(f"Warning: Audio file not found: {ref_audio}")
|
||||
print("This may be due to a cached voice prompt referencing a deleted temp file.")
|
||||
print("Regenerating without voice prompt.")
|
||||
logger.warning("Audio file not found: %s", ref_audio)
|
||||
logger.warning("This may be due to a cached voice prompt referencing a deleted temp file.")
|
||||
logger.warning("Regenerating without voice prompt.")
|
||||
ref_audio = None
|
||||
|
||||
|
||||
# Check if model supports voice cloning via generate method
|
||||
# MLX API may support ref_audio parameter directly
|
||||
try:
|
||||
@@ -375,6 +227,7 @@ class MLXTTSBackend:
|
||||
if ref_audio:
|
||||
# Check if generate accepts ref_audio parameter
|
||||
import inspect
|
||||
|
||||
sig = inspect.signature(self.model.generate)
|
||||
if "ref_audio" in sig.parameters:
|
||||
# Generate with voice cloning
|
||||
@@ -393,18 +246,18 @@ class MLXTTSBackend:
|
||||
sample_rate = result.sample_rate
|
||||
except Exception as e:
|
||||
# If voice cloning fails, try without it
|
||||
print(f"Warning: Voice cloning failed, generating without voice prompt: {e}")
|
||||
logger.warning("Voice cloning failed, generating without voice prompt: %s", e)
|
||||
for result in self.model.generate(text, lang_code=lang):
|
||||
audio_chunks.append(np.array(result.audio))
|
||||
sample_rate = result.sample_rate
|
||||
|
||||
|
||||
# Concatenate all chunks
|
||||
if audio_chunks:
|
||||
audio = np.concatenate([np.asarray(chunk, dtype=np.float32) for chunk in audio_chunks])
|
||||
else:
|
||||
# Fallback: empty audio
|
||||
audio = np.array([], dtype=np.float32)
|
||||
|
||||
|
||||
return audio, sample_rate
|
||||
|
||||
# Run blocking inference in thread pool
|
||||
@@ -413,183 +266,82 @@ class MLXTTSBackend:
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
WHISPER_HF_REPOS = {
|
||||
"base": "openai/whisper-base",
|
||||
"small": "openai/whisper-small",
|
||||
"medium": "openai/whisper-medium",
|
||||
"large": "openai/whisper-large-v3",
|
||||
}
|
||||
|
||||
|
||||
class MLXSTTBackend:
|
||||
"""MLX-based STT backend using mlx-audio Whisper."""
|
||||
|
||||
def __init__(self, model_size: str = "base"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
|
||||
def _is_model_cached(self, model_size: str) -> bool:
|
||||
"""
|
||||
Check if the Whisper model is already cached locally AND fully downloaded.
|
||||
|
||||
Args:
|
||||
model_size: Model size to check
|
||||
|
||||
Returns:
|
||||
True if model is fully cached, False if missing or incomplete
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
hf_repo = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + hf_repo.replace("/", "--"))
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
# Check for .incomplete files - if any exist, download is still in progress
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
print(f"[_is_model_cached] Found .incomplete files for whisper-{model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
# Check that actual model weight files exist in snapshots
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
has_weights = (
|
||||
any(snapshots_dir.rglob("*.safetensors")) or
|
||||
any(snapshots_dir.rglob("*.bin")) or
|
||||
any(snapshots_dir.rglob("*.npz"))
|
||||
)
|
||||
if not has_weights:
|
||||
print(f"[_is_model_cached] No model weights found for whisper-{model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[_is_model_cached] Error checking cache for whisper-{model_size}: {e}")
|
||||
return False
|
||||
|
||||
hf_repo = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
return is_model_cached(hf_repo, weight_extensions=(".safetensors", ".bin", ".npz"))
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the MLX Whisper model.
|
||||
|
||||
|
||||
Args:
|
||||
model_size: Model size (tiny, base, small, medium, large)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
return
|
||||
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Check if model is already cached
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Set up progress callback and tracker
|
||||
# If cached: filter out non-download progress
|
||||
# If not cached: report all progress (we're actually downloading)
|
||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
|
||||
# Patch tqdm BEFORE importing mlx_audio
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
# Import mlx_audio
|
||||
with model_load_progress(progress_model_name, is_cached):
|
||||
from mlx_audio.stt import load
|
||||
|
||||
# MLX Whisper uses the standard OpenAI models
|
||||
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
logger.info("Loading MLX Whisper model %s...", model_size)
|
||||
|
||||
print(f"Loading MLX Whisper model {model_size}...")
|
||||
|
||||
# Only track download progress if model is NOT cached
|
||||
if not is_cached:
|
||||
# Start tracking download task
|
||||
task_manager.start_download(progress_model_name)
|
||||
|
||||
# Initialize progress state so SSE endpoint has initial data to send
|
||||
progress_manager.update_progress(
|
||||
model_name=progress_model_name,
|
||||
current=0,
|
||||
total=0,
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Load the model (tqdm is patched, but filters out non-download progress)
|
||||
try:
|
||||
with force_offline_if_cached(is_cached, progress_model_name):
|
||||
self.model = load(model_name)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
# Only mark download as complete if we were tracking it
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(progress_model_name)
|
||||
task_manager.complete_download(progress_model_name)
|
||||
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"MLX Whisper model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: mlx_audio package not found. Install with: pip install mlx-audio")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading MLX Whisper model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
|
||||
|
||||
self.model_size = model_size
|
||||
logger.info("MLX Whisper model %s loaded successfully", model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
print("MLX Whisper model unloaded")
|
||||
|
||||
logger.info("MLX Whisper model unloaded")
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
model_size: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint (en or zh)
|
||||
language: Optional language hint
|
||||
model_size: Optional model size override
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
await self.load_model_async(model_size)
|
||||
|
||||
def _transcribe_sync():
|
||||
"""Run synchronous transcription in thread pool."""
|
||||
|
||||
+102
-357
@@ -4,67 +4,50 @@ PyTorch backend implementation for TTS and STT.
|
||||
|
||||
from typing import Optional, List, Tuple
|
||||
import asyncio
|
||||
import logging
|
||||
import torch
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
from . import TTSBackend, STTBackend
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
from . import TTSBackend, STTBackend, LANGUAGE_CODE_TO_NAME, WHISPER_HF_REPOS
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
empty_device_cache,
|
||||
manual_seed,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
|
||||
from ..utils.audio import normalize_audio, load_audio
|
||||
from ..utils.progress import get_progress_manager
|
||||
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
|
||||
from ..utils.tasks import get_task_manager
|
||||
|
||||
LANGUAGE_CODE_TO_NAME = {
|
||||
"zh": "chinese", "en": "english", "ja": "japanese", "ko": "korean",
|
||||
"de": "german", "fr": "french", "ru": "russian", "pt": "portuguese",
|
||||
"es": "spanish", "it": "italian",
|
||||
}
|
||||
from ..utils.audio import load_audio
|
||||
from ..utils.hf_offline_patch import force_offline_if_cached
|
||||
|
||||
|
||||
class PyTorchTTSBackend:
|
||||
"""PyTorch-based TTS backend using Qwen3-TTS."""
|
||||
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
self._current_model_size = None
|
||||
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
# Intel Arc / Intel Xe GPU via intel-extension-for-pytorch (IPEX)
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, 'xpu') and torch.xpu.is_available():
|
||||
return "xpu"
|
||||
except ImportError:
|
||||
pass
|
||||
# Any GPU on Windows via DirectML (torch-directml)
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
return torch_directml.device(0)
|
||||
except ImportError:
|
||||
pass
|
||||
# MPS (Apple Silicon) — kept for completeness but MLX backend is preferred
|
||||
if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
return "cpu" # MPS disabled for stability; MLX backend handles Apple Silicon
|
||||
return "cpu"
|
||||
|
||||
return get_torch_device(allow_xpu=True, allow_directml=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
"""
|
||||
Get the HuggingFace Hub model ID.
|
||||
|
||||
|
||||
Args:
|
||||
model_size: Model size (1.7B or 0.6B)
|
||||
|
||||
|
||||
Returns:
|
||||
HuggingFace Hub model ID
|
||||
"""
|
||||
@@ -72,179 +55,88 @@ class PyTorchTTSBackend:
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
|
||||
}
|
||||
|
||||
|
||||
if model_size not in hf_model_map:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
|
||||
|
||||
return hf_model_map[model_size]
|
||||
|
||||
|
||||
def _is_model_cached(self, model_size: str) -> bool:
|
||||
"""
|
||||
Check if the model is already cached locally AND fully downloaded.
|
||||
|
||||
Args:
|
||||
model_size: Model size to check
|
||||
|
||||
Returns:
|
||||
True if model is fully cached, False if missing or incomplete
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
model_path = self._get_model_path(model_size)
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + model_path.replace("/", "--"))
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
# Check for .incomplete files - if any exist, download is still in progress
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
print(f"[_is_model_cached] Found .incomplete files for {model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
# Check that actual model weight files exist in snapshots
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
has_weights = (
|
||||
any(snapshots_dir.rglob("*.safetensors")) or
|
||||
any(snapshots_dir.rglob("*.bin"))
|
||||
)
|
||||
if not has_weights:
|
||||
print(f"[_is_model_cached] No model weights found for {model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[_is_model_cached] Error checking cache for {model_size}: {e}")
|
||||
return False
|
||||
|
||||
return is_model_cached(self._get_model_path(model_size))
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
Lazy load the TTS model with automatic downloading from HuggingFace Hub.
|
||||
|
||||
|
||||
Args:
|
||||
model_size: Model size to load (1.7B or 0.6B)
|
||||
"""
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
|
||||
# If already loaded with correct size, return
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
|
||||
# Unload existing model if different size requested
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
try:
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Check if model is already cached
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Set up progress callback and tracker
|
||||
# If cached: filter out non-download progress (like "Segment 1/1" during generation)
|
||||
# If not cached: report all progress (we're actually downloading)
|
||||
progress_callback = create_hf_progress_callback(model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
|
||||
# Patch tqdm BEFORE importing qwen_tts
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
|
||||
# Import qwen_tts
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
|
||||
# Get model path (local or HuggingFace Hub ID)
|
||||
model_path = self._get_model_path(model_size)
|
||||
logger.info("Loading TTS model %s on %s...", model_size, self.device)
|
||||
|
||||
print(f"Loading TTS model {model_size} on {self.device}...")
|
||||
# Route both HF Hub and Transformers through a single cache root.
|
||||
# On Windows local setups, model assets can otherwise split between
|
||||
# .hf-cache/hub and .hf-cache/transformers, causing speech_tokenizer
|
||||
# and preprocessor_config.json to fail to resolve during load.
|
||||
from huggingface_hub import constants as hf_constants
|
||||
tts_cache_dir = hf_constants.HF_HUB_CACHE
|
||||
|
||||
# Only track download progress if model is NOT cached
|
||||
if not is_cached:
|
||||
# Start tracking download task
|
||||
task_manager.start_download(model_name)
|
||||
|
||||
# Initialize progress state so SSE endpoint has initial data to send
|
||||
progress_manager.update_progress(
|
||||
model_name=model_name,
|
||||
current=0,
|
||||
total=0, # Will be updated once actual total is known
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Load the model (tqdm is patched, but filters out non-download progress)
|
||||
try:
|
||||
# Don't pass device_map on CPU: accelerate's meta-tensor mechanism
|
||||
# causes "Cannot copy out of meta tensor" when moving to CPU.
|
||||
# Instead load directly then call .to(device) if needed.
|
||||
with force_offline_if_cached(is_cached, model_name):
|
||||
if self.device == "cpu":
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
cache_dir=tts_cache_dir,
|
||||
torch_dtype=torch.float32,
|
||||
low_cpu_mem_usage=False,
|
||||
)
|
||||
else:
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
cache_dir=tts_cache_dir,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
# Only mark download as complete if we were tracking it
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(model_name)
|
||||
task_manager.complete_download(model_name)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"TTS model {model_size} loaded successfully")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Error: qwen_tts package not found. Install with: pip install git+https://github.com/QwenLM/Qwen3-TTS.git")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error loading TTS model: {e}")
|
||||
print(f"Tip: The model will be automatically downloaded from HuggingFace Hub on first use.")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
model_name = f"qwen-tts-{model_size}"
|
||||
progress_manager.mark_error(model_name, str(e))
|
||||
task_manager.error_download(model_name, str(e))
|
||||
raise
|
||||
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
logger.info("TTS model %s loaded successfully", model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("TTS model unloaded")
|
||||
|
||||
|
||||
empty_device_cache(self.device)
|
||||
|
||||
logger.info("TTS model unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
@@ -253,17 +145,17 @@ class PyTorchTTSBackend:
|
||||
) -> Tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt from reference audio.
|
||||
|
||||
|
||||
Args:
|
||||
audio_path: Path to reference audio file
|
||||
reference_text: Transcript of reference audio
|
||||
use_cache: Whether to use cached prompt if available
|
||||
|
||||
|
||||
Returns:
|
||||
Tuple of (voice_prompt_dict, was_cached)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
|
||||
# Check cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
@@ -279,7 +171,7 @@ class PyTorchTTSBackend:
|
||||
# Legacy cache format - convert to dict
|
||||
# This shouldn't happen in practice, but handle it
|
||||
return {"prompt": cached_prompt}, True
|
||||
|
||||
|
||||
def _create_prompt_sync():
|
||||
"""Run synchronous voice prompt creation in thread pool."""
|
||||
return self.model.create_voice_clone_prompt(
|
||||
@@ -287,48 +179,24 @@ class PyTorchTTSBackend:
|
||||
ref_text=reference_text,
|
||||
x_vector_only_mode=False,
|
||||
)
|
||||
|
||||
|
||||
# Run blocking operation in thread pool
|
||||
voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
|
||||
|
||||
|
||||
# Cache if enabled
|
||||
if use_cache:
|
||||
cache_key = get_cache_key(audio_path, reference_text)
|
||||
cache_voice_prompt(cache_key, voice_prompt_items)
|
||||
|
||||
|
||||
return voice_prompt_items, False
|
||||
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: List[str],
|
||||
reference_texts: List[str],
|
||||
) -> Tuple[np.ndarray, str]:
|
||||
"""
|
||||
Combine multiple reference samples for better quality.
|
||||
|
||||
Args:
|
||||
audio_paths: List of audio file paths
|
||||
reference_texts: List of reference texts
|
||||
|
||||
Returns:
|
||||
Tuple of (combined_audio, combined_text)
|
||||
"""
|
||||
combined_audio = []
|
||||
|
||||
for audio_path in audio_paths:
|
||||
audio, sr = load_audio(audio_path)
|
||||
audio = normalize_audio(audio)
|
||||
combined_audio.append(audio)
|
||||
|
||||
# Concatenate audio
|
||||
mixed = np.concatenate(combined_audio)
|
||||
mixed = normalize_audio(mixed)
|
||||
|
||||
# Combine texts
|
||||
combined_text = " ".join(reference_texts)
|
||||
|
||||
return mixed, combined_text
|
||||
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
@@ -357,9 +225,7 @@ class PyTorchTTSBackend:
|
||||
"""Run synchronous generation in thread pool."""
|
||||
# Set seed if provided
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
manual_seed(seed, self.device)
|
||||
|
||||
# Generate audio - this is the blocking operation
|
||||
wavs, sample_rate = self.model.generate_voice_clone(
|
||||
@@ -376,15 +242,6 @@ class PyTorchTTSBackend:
|
||||
return audio, sample_rate
|
||||
|
||||
|
||||
WHISPER_HF_REPOS = {
|
||||
"base": "openai/whisper-base",
|
||||
"small": "openai/whisper-small",
|
||||
"medium": "openai/whisper-medium",
|
||||
"large": "openai/whisper-large-v3",
|
||||
"turbo": "openai/whisper-large-v3-turbo",
|
||||
}
|
||||
|
||||
|
||||
class PyTorchSTTBackend:
|
||||
"""PyTorch-based STT backend using Whisper."""
|
||||
|
||||
@@ -393,72 +250,18 @@ class PyTorchSTTBackend:
|
||||
self.processor = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
|
||||
|
||||
def _get_device(self) -> str:
|
||||
"""Get the best available device."""
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
# Intel Arc / Intel Xe GPU via intel-extension-for-pytorch (IPEX)
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, 'xpu') and torch.xpu.is_available():
|
||||
return "xpu"
|
||||
except ImportError:
|
||||
pass
|
||||
# Any GPU on Windows via DirectML (torch-directml)
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
return torch_directml.device(0)
|
||||
except ImportError:
|
||||
pass
|
||||
if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
|
||||
return "cpu" # MPS disabled for stability
|
||||
return "cpu"
|
||||
|
||||
return get_torch_device(allow_xpu=True, allow_directml=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
"""Check if model is loaded."""
|
||||
return self.model is not None
|
||||
|
||||
|
||||
def _is_model_cached(self, model_size: str) -> bool:
|
||||
"""
|
||||
Check if the Whisper model is already cached locally AND fully downloaded.
|
||||
|
||||
Args:
|
||||
model_size: Model size to check
|
||||
|
||||
Returns:
|
||||
True if model is fully cached, False if missing or incomplete
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub import constants as hf_constants
|
||||
hf_repo = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
repo_cache = Path(hf_constants.HF_HUB_CACHE) / ("models--" + hf_repo.replace("/", "--"))
|
||||
|
||||
if not repo_cache.exists():
|
||||
return False
|
||||
|
||||
# Check for .incomplete files - if any exist, download is still in progress
|
||||
blobs_dir = repo_cache / "blobs"
|
||||
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
|
||||
print(f"[_is_model_cached] Found .incomplete files for whisper-{model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
# Check that actual model weight files exist in snapshots
|
||||
snapshots_dir = repo_cache / "snapshots"
|
||||
if snapshots_dir.exists():
|
||||
has_weights = (
|
||||
any(snapshots_dir.rglob("*.safetensors")) or
|
||||
any(snapshots_dir.rglob("*.bin"))
|
||||
)
|
||||
if not has_weights:
|
||||
print(f"[_is_model_cached] No model weights found for whisper-{model_size}, treating as not cached")
|
||||
return False
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[_is_model_cached] Error checking cache for whisper-{model_size}: {e}")
|
||||
return False
|
||||
hf_repo = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
return is_model_cached(hf_repo)
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None):
|
||||
"""
|
||||
@@ -467,95 +270,36 @@ class PyTorchSTTBackend:
|
||||
Args:
|
||||
model_size: Model size (tiny, base, small, medium, large)
|
||||
"""
|
||||
print(f"[DEBUG] load_model_async called with size: {model_size}")
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
print(f"[DEBUG] Model already loaded? {self.model is not None}, current size: {self.model_size}, requested: {model_size}")
|
||||
if self.model is not None and self.model_size == model_size:
|
||||
print(f"[DEBUG] Early return - model already loaded")
|
||||
return
|
||||
|
||||
print(f"[DEBUG] Calling asyncio.to_thread for _load_model_sync")
|
||||
# Run blocking load in thread pool
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
print(f"[DEBUG] asyncio.to_thread completed")
|
||||
|
||||
|
||||
# Alias for compatibility
|
||||
load_model = load_model_async
|
||||
|
||||
|
||||
def _load_model_sync(self, model_size: str):
|
||||
"""Synchronous model loading."""
|
||||
print(f"[DEBUG] _load_model_sync called for Whisper {model_size}")
|
||||
try:
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Check if model is already cached
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
# Set up progress callback and tracker
|
||||
# If cached: filter out non-download progress
|
||||
# If not cached: report all progress (we're actually downloading)
|
||||
progress_callback = create_hf_progress_callback(progress_model_name, progress_manager)
|
||||
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
|
||||
|
||||
# Patch tqdm BEFORE importing transformers
|
||||
print("[DEBUG] Starting tqdm patch BEFORE transformers import")
|
||||
tracker_context = tracker.patch_download()
|
||||
tracker_context.__enter__()
|
||||
print("[DEBUG] tqdm patched, now importing transformers")
|
||||
|
||||
# Import transformers
|
||||
with model_load_progress(progress_model_name, is_cached):
|
||||
from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
||||
|
||||
model_name = WHISPER_HF_REPOS.get(model_size, f"openai/whisper-{model_size}")
|
||||
print(f"[DEBUG] Model name: {model_name}")
|
||||
logger.info("Loading Whisper model %s on %s...", model_size, self.device)
|
||||
|
||||
print(f"Loading Whisper model {model_size} on {self.device}...")
|
||||
|
||||
# Only track download progress if model is NOT cached
|
||||
if not is_cached:
|
||||
# Start tracking download task
|
||||
task_manager.start_download(progress_model_name)
|
||||
|
||||
# Initialize progress state so SSE endpoint has initial data to send
|
||||
progress_manager.update_progress(
|
||||
model_name=progress_model_name,
|
||||
current=0,
|
||||
total=0, # Will be updated once actual total is known
|
||||
filename="Connecting to HuggingFace...",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Load models (tqdm is patched, but filters out non-download progress)
|
||||
try:
|
||||
with force_offline_if_cached(is_cached, progress_model_name):
|
||||
self.processor = WhisperProcessor.from_pretrained(model_name)
|
||||
self.model = WhisperForConditionalGeneration.from_pretrained(model_name)
|
||||
finally:
|
||||
# Exit the patch context
|
||||
tracker_context.__exit__(None, None, None)
|
||||
|
||||
# Only mark download as complete if we were tracking it
|
||||
if not is_cached:
|
||||
progress_manager.mark_complete(progress_model_name)
|
||||
task_manager.complete_download(progress_model_name)
|
||||
|
||||
self.model.to(self.device)
|
||||
self.model_size = model_size
|
||||
|
||||
print(f"Whisper model {model_size} loaded successfully")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error loading Whisper model: {e}")
|
||||
progress_manager = get_progress_manager()
|
||||
task_manager = get_task_manager()
|
||||
progress_model_name = f"whisper-{model_size}"
|
||||
progress_manager.mark_error(progress_model_name, str(e))
|
||||
task_manager.error_download(progress_model_name, str(e))
|
||||
raise
|
||||
|
||||
|
||||
self.model.to(self.device)
|
||||
self.model_size = model_size
|
||||
logger.info("Whisper model %s loaded successfully", model_size)
|
||||
|
||||
def unload_model(self):
|
||||
"""Unload the model to free memory."""
|
||||
if self.model is not None:
|
||||
@@ -563,34 +307,35 @@ class PyTorchSTTBackend:
|
||||
del self.processor
|
||||
self.model = None
|
||||
self.processor = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
print("Whisper model unloaded")
|
||||
|
||||
|
||||
empty_device_cache(self.device)
|
||||
|
||||
logger.info("Whisper model unloaded")
|
||||
|
||||
async def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
model_size: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Transcribe audio to text.
|
||||
|
||||
|
||||
Args:
|
||||
audio_path: Path to audio file
|
||||
language: Optional language hint (en or zh)
|
||||
|
||||
language: Optional language hint
|
||||
model_size: Optional model size override
|
||||
|
||||
Returns:
|
||||
Transcribed text
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
await self.load_model_async(model_size)
|
||||
|
||||
def _transcribe_sync():
|
||||
"""Run synchronous transcription in thread pool."""
|
||||
# Load audio
|
||||
audio, sr = load_audio(audio_path, sample_rate=16000)
|
||||
|
||||
|
||||
# Process audio
|
||||
inputs = self.processor(
|
||||
audio,
|
||||
@@ -598,7 +343,7 @@ class PyTorchSTTBackend:
|
||||
return_tensors="pt",
|
||||
)
|
||||
inputs = inputs.to(self.device)
|
||||
|
||||
|
||||
# Generate transcription
|
||||
# If language is provided, force it; otherwise let Whisper auto-detect
|
||||
generate_kwargs = {}
|
||||
@@ -608,20 +353,20 @@ class PyTorchSTTBackend:
|
||||
task="transcribe",
|
||||
)
|
||||
generate_kwargs["forced_decoder_ids"] = forced_decoder_ids
|
||||
|
||||
|
||||
with torch.no_grad():
|
||||
predicted_ids = self.model.generate(
|
||||
inputs["input_features"],
|
||||
**generate_kwargs,
|
||||
)
|
||||
|
||||
|
||||
# Decode
|
||||
transcription = self.processor.batch_decode(
|
||||
predicted_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
|
||||
return transcription.strip()
|
||||
|
||||
|
||||
# Run blocking transcription in thread pool
|
||||
return await asyncio.to_thread(_transcribe_sync)
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""
|
||||
Qwen3-TTS CustomVoice backend implementation.
|
||||
|
||||
Wraps the Qwen3-TTS-12Hz CustomVoice model for preset-speaker TTS with
|
||||
instruction-based style control. Uses the same qwen_tts library as the
|
||||
Base model (pytorch_backend.py) but loads a different checkpoint and
|
||||
calls generate_custom_voice() instead of generate_voice_clone().
|
||||
|
||||
Key differences from the Base engine:
|
||||
- Uses preset speakers (9 built-in voices) instead of zero-shot cloning
|
||||
- Supports instruct parameter for tone/emotion/prosody control
|
||||
- Two model sizes: 1.7B and 0.6B
|
||||
|
||||
Languages supported: zh, en, ja, ko, de, fr, ru, pt, es, it
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from . import TTSBackend, LANGUAGE_CODE_TO_NAME
|
||||
from .base import (
|
||||
is_model_cached,
|
||||
get_torch_device,
|
||||
combine_voice_prompts as _combine_voice_prompts,
|
||||
model_load_progress,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Preset speakers ──────────────────────────────────────────────────
|
||||
|
||||
# (speaker_id, display_name, gender, native_language_code, description)
|
||||
QWEN_CUSTOM_VOICES = [
|
||||
("Vivian", "Vivian", "female", "zh", "Bright, slightly edgy young female voice"),
|
||||
("Serena", "Serena", "female", "zh", "Warm, gentle young female voice"),
|
||||
("Uncle_Fu", "Uncle Fu", "male", "zh", "Seasoned male voice with a low, mellow timbre"),
|
||||
("Dylan", "Dylan", "male", "zh", "Youthful Beijing male voice with a clear, natural timbre"),
|
||||
("Eric", "Eric", "male", "zh", "Lively Chengdu male voice with a slightly husky brightness"),
|
||||
("Ryan", "Ryan", "male", "en", "Dynamic male voice with strong rhythmic drive"),
|
||||
("Aiden", "Aiden", "male", "en", "Sunny American male voice with a clear midrange"),
|
||||
("Ono_Anna", "Ono Anna", "female", "ja", "Playful Japanese female voice with a light, nimble timbre"),
|
||||
("Sohee", "Sohee", "female", "ko", "Warm Korean female voice with rich emotion"),
|
||||
]
|
||||
|
||||
QWEN_CV_DEFAULT_SPEAKER = "Ryan"
|
||||
|
||||
# HuggingFace repo IDs per model size
|
||||
QWEN_CV_HF_REPOS = {
|
||||
"1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
"0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
|
||||
}
|
||||
|
||||
|
||||
class QwenCustomVoiceBackend:
|
||||
"""Qwen3-TTS CustomVoice backend — preset speakers with instruct control."""
|
||||
|
||||
def __init__(self, model_size: str = "1.7B"):
|
||||
self.model = None
|
||||
self.model_size = model_size
|
||||
self.device = self._get_device()
|
||||
self._current_model_size: Optional[str] = None
|
||||
|
||||
def _get_device(self) -> str:
|
||||
return get_torch_device(allow_xpu=True, allow_directml=True)
|
||||
|
||||
def is_loaded(self) -> bool:
|
||||
return self.model is not None
|
||||
|
||||
def _get_model_path(self, model_size: str) -> str:
|
||||
if model_size not in QWEN_CV_HF_REPOS:
|
||||
raise ValueError(f"Unknown model size: {model_size}")
|
||||
return QWEN_CV_HF_REPOS[model_size]
|
||||
|
||||
def _is_model_cached(self, model_size: Optional[str] = None) -> bool:
|
||||
size = model_size or self.model_size
|
||||
return is_model_cached(self._get_model_path(size))
|
||||
|
||||
async def load_model_async(self, model_size: Optional[str] = None) -> None:
|
||||
if model_size is None:
|
||||
model_size = self.model_size
|
||||
|
||||
if self.model is not None and self._current_model_size == model_size:
|
||||
return
|
||||
|
||||
if self.model is not None and self._current_model_size != model_size:
|
||||
self.unload_model()
|
||||
|
||||
await asyncio.to_thread(self._load_model_sync, model_size)
|
||||
|
||||
# Alias for compatibility with the TTSBackend protocol
|
||||
load_model = load_model_async
|
||||
|
||||
def _load_model_sync(self, model_size: str) -> None:
|
||||
model_name = f"qwen-custom-voice-{model_size}"
|
||||
is_cached = self._is_model_cached(model_size)
|
||||
|
||||
with model_load_progress(model_name, is_cached):
|
||||
from qwen_tts import Qwen3TTSModel
|
||||
|
||||
model_path = self._get_model_path(model_size)
|
||||
logger.info("Loading Qwen CustomVoice %s on %s...", model_size, self.device)
|
||||
|
||||
if self.device == "cpu":
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.float32,
|
||||
low_cpu_mem_usage=False,
|
||||
)
|
||||
else:
|
||||
self.model = Qwen3TTSModel.from_pretrained(
|
||||
model_path,
|
||||
device_map=self.device,
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
|
||||
self._current_model_size = model_size
|
||||
self.model_size = model_size
|
||||
logger.info("Qwen CustomVoice %s loaded successfully", model_size)
|
||||
|
||||
def unload_model(self) -> None:
|
||||
if self.model is not None:
|
||||
del self.model
|
||||
self.model = None
|
||||
self._current_model_size = None
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
logger.info("Qwen CustomVoice unloaded")
|
||||
|
||||
async def create_voice_prompt(
|
||||
self,
|
||||
audio_path: str,
|
||||
reference_text: str,
|
||||
use_cache: bool = True,
|
||||
) -> tuple[dict, bool]:
|
||||
"""
|
||||
Create voice prompt for CustomVoice.
|
||||
|
||||
CustomVoice doesn't use reference audio — it uses preset speakers.
|
||||
When called for a cloned profile (fallback), uses the default speaker.
|
||||
For preset profiles, the voice_prompt dict is built by the profile
|
||||
service and bypasses this method entirely.
|
||||
"""
|
||||
return {
|
||||
"voice_type": "preset",
|
||||
"preset_engine": "qwen_custom_voice",
|
||||
"preset_voice_id": QWEN_CV_DEFAULT_SPEAKER,
|
||||
}, False
|
||||
|
||||
async def combine_voice_prompts(
|
||||
self,
|
||||
audio_paths: list[str],
|
||||
reference_texts: list[str],
|
||||
) -> tuple[np.ndarray, str]:
|
||||
return await _combine_voice_prompts(audio_paths, reference_texts)
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
text: str,
|
||||
voice_prompt: dict,
|
||||
language: str = "en",
|
||||
seed: Optional[int] = None,
|
||||
instruct: Optional[str] = None,
|
||||
) -> tuple[np.ndarray, int]:
|
||||
"""
|
||||
Generate audio using Qwen CustomVoice.
|
||||
|
||||
Args:
|
||||
text: Text to synthesize
|
||||
voice_prompt: Dict with preset_voice_id (speaker name)
|
||||
language: Language code (zh, en, ja, ko, etc.)
|
||||
seed: Random seed for reproducibility
|
||||
instruct: Natural language instruction for style control
|
||||
(e.g. "Speak in an angry tone", "Very happy")
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_array, sample_rate)
|
||||
"""
|
||||
await self.load_model_async(None)
|
||||
|
||||
speaker = voice_prompt.get("preset_voice_id") or QWEN_CV_DEFAULT_SPEAKER
|
||||
|
||||
def _generate_sync():
|
||||
if seed is not None:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
|
||||
lang_name = LANGUAGE_CODE_TO_NAME.get(language, "auto")
|
||||
|
||||
kwargs = {
|
||||
"text": text,
|
||||
"language": lang_name.capitalize() if lang_name != "auto" else "Auto",
|
||||
"speaker": speaker,
|
||||
}
|
||||
|
||||
# Only pass instruct if non-empty
|
||||
if instruct:
|
||||
kwargs["instruct"] = instruct
|
||||
|
||||
wavs, sample_rate = self.model.generate_custom_voice(**kwargs)
|
||||
return wavs[0], sample_rate
|
||||
|
||||
audio, sample_rate = await asyncio.to_thread(_generate_sync)
|
||||
return audio, sample_rate
|
||||
+357
-166
@@ -8,11 +8,14 @@ Usage:
|
||||
|
||||
import PyInstaller.__main__
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_apple_silicon():
|
||||
"""Check if running on Apple Silicon."""
|
||||
@@ -28,155 +31,355 @@ def build_server(cuda=False):
|
||||
"""
|
||||
backend_dir = Path(__file__).parent
|
||||
|
||||
binary_name = 'voicebox-server-cuda' if cuda else 'voicebox-server'
|
||||
binary_name = "voicebox-server-cuda" if cuda else "voicebox-server"
|
||||
|
||||
# PyInstaller arguments
|
||||
# CUDA builds use --onedir so we can split the output into two archives:
|
||||
# 1. Server core (~200-400MB) — versioned with the app
|
||||
# 2. CUDA libs (~2GB) — versioned independently (only redownloaded on
|
||||
# CUDA toolkit / torch major version changes)
|
||||
# CPU builds remain --onefile for simplicity.
|
||||
pack_mode = "--onedir" if cuda else "--onefile"
|
||||
args = [
|
||||
'server.py', # Use server.py as entry point instead of main.py
|
||||
'--onefile',
|
||||
'--name', binary_name,
|
||||
"server.py", # Use server.py as entry point instead of main.py
|
||||
pack_mode,
|
||||
"--name",
|
||||
binary_name,
|
||||
]
|
||||
|
||||
# Hide console window on Windows only. On macOS/Linux the sidecar needs
|
||||
# stdout/stderr for Tauri to capture logs.
|
||||
if platform.system() == "Windows":
|
||||
args.append('--noconsole')
|
||||
args.append("--noconsole")
|
||||
|
||||
# numpy 2.x / torch ABI mismatch fix: install memmove fallback for
|
||||
# torch.from_numpy() before the app starts. Runtime hooks run after
|
||||
# FrozenImporter is registered so frozen torch/numpy are importable.
|
||||
# Paths are passed relative to backend_dir because os.chdir(backend_dir)
|
||||
# runs before PyInstaller. Absolute paths would get baked into the
|
||||
# generated .spec, breaking reproducible builds on other machines / CI.
|
||||
args.extend(
|
||||
[
|
||||
"--runtime-hook",
|
||||
"pyi_rth_numpy_compat.py",
|
||||
# Stub torch.compiler.disable before transformers imports
|
||||
# flex_attention, which otherwise triggers torch._dynamo →
|
||||
# torch._numpy._ufuncs and crashes at module load under
|
||||
# PyInstaller. See pyi_rth_torch_compiler_disable.py.
|
||||
"--runtime-hook",
|
||||
"pyi_rth_torch_compiler_disable.py",
|
||||
# Per-module collection overrides (e.g. forcing scipy.stats._distn_infrastructure
|
||||
# to bundle .py source alongside .pyc so the runtime hook can source-patch it).
|
||||
"--additional-hooks-dir",
|
||||
"pyi_hooks",
|
||||
]
|
||||
)
|
||||
|
||||
# Add local qwen_tts path if specified (for editable installs)
|
||||
qwen_tts_path = os.getenv('QWEN_TTS_PATH')
|
||||
qwen_tts_path = os.getenv("QWEN_TTS_PATH")
|
||||
if qwen_tts_path and Path(qwen_tts_path).exists():
|
||||
args.extend(['--paths', str(qwen_tts_path)])
|
||||
print(f"Using local qwen_tts source from: {qwen_tts_path}")
|
||||
args.extend(["--paths", str(qwen_tts_path)])
|
||||
logger.info("Using local qwen_tts source from: %s", qwen_tts_path)
|
||||
|
||||
# Add common hidden imports
|
||||
args.extend([
|
||||
'--hidden-import', 'backend',
|
||||
'--hidden-import', 'backend.main',
|
||||
'--hidden-import', 'backend.config',
|
||||
'--hidden-import', 'backend.database',
|
||||
'--hidden-import', 'backend.models',
|
||||
'--hidden-import', 'backend.profiles',
|
||||
'--hidden-import', 'backend.history',
|
||||
'--hidden-import', 'backend.tts',
|
||||
'--hidden-import', 'backend.transcribe',
|
||||
'--hidden-import', 'backend.platform_detect',
|
||||
'--hidden-import', 'backend.backends',
|
||||
'--hidden-import', 'backend.backends.pytorch_backend',
|
||||
'--hidden-import', 'backend.utils.audio',
|
||||
'--hidden-import', 'backend.utils.cache',
|
||||
'--hidden-import', 'backend.utils.progress',
|
||||
'--hidden-import', 'backend.utils.hf_progress',
|
||||
'--hidden-import', 'backend.utils.validation',
|
||||
'--hidden-import', 'backend.cuda_download',
|
||||
'--hidden-import', 'backend.effects',
|
||||
'--hidden-import', 'backend.utils.effects',
|
||||
'--hidden-import', 'backend.versions',
|
||||
'--hidden-import', 'pedalboard',
|
||||
'--hidden-import', 'chatterbox',
|
||||
'--hidden-import', 'chatterbox.tts_turbo',
|
||||
'--hidden-import', 'chatterbox.mtl_tts',
|
||||
'--hidden-import', 'backend.backends.chatterbox_backend',
|
||||
'--hidden-import', 'backend.backends.chatterbox_turbo_backend',
|
||||
'--hidden-import', 'backend.backends.luxtts_backend',
|
||||
'--hidden-import', 'zipvoice',
|
||||
'--hidden-import', 'zipvoice.luxvoice',
|
||||
'--collect-all', 'zipvoice',
|
||||
'--collect-all', 'linacodec',
|
||||
'--hidden-import', 'torch',
|
||||
'--hidden-import', 'transformers',
|
||||
'--hidden-import', 'fastapi',
|
||||
'--hidden-import', 'uvicorn',
|
||||
'--hidden-import', 'sqlalchemy',
|
||||
'--hidden-import', 'librosa',
|
||||
'--hidden-import', 'soundfile',
|
||||
'--hidden-import', 'qwen_tts',
|
||||
'--hidden-import', 'qwen_tts.inference',
|
||||
'--hidden-import', 'qwen_tts.inference.qwen3_tts_model',
|
||||
'--hidden-import', 'qwen_tts.inference.qwen3_tts_tokenizer',
|
||||
'--hidden-import', 'qwen_tts.core',
|
||||
'--hidden-import', 'qwen_tts.cli',
|
||||
'--copy-metadata', 'qwen-tts',
|
||||
'--copy-metadata', 'requests',
|
||||
'--copy-metadata', 'transformers',
|
||||
'--copy-metadata', 'huggingface-hub',
|
||||
'--copy-metadata', 'tokenizers',
|
||||
'--copy-metadata', 'safetensors',
|
||||
'--copy-metadata', 'tqdm',
|
||||
'--hidden-import', 'requests',
|
||||
'--collect-submodules', 'qwen_tts',
|
||||
'--collect-data', 'qwen_tts',
|
||||
# Fix for pkg_resources and jaraco namespace packages
|
||||
'--hidden-import', 'pkg_resources.extern',
|
||||
'--collect-submodules', 'jaraco',
|
||||
# inflect uses typeguard @typechecked which calls inspect.getsource()
|
||||
# at import time — needs .py source files, not just .pyc bytecode
|
||||
'--collect-all', 'inflect',
|
||||
# perth ships pretrained watermark model files (hparams.yaml, .pth.tar)
|
||||
# in perth/perth_net/pretrained/ — needed by chatterbox at runtime
|
||||
'--collect-all', 'perth',
|
||||
# piper_phonemize ships espeak-ng-data/ (phoneme tables, language dicts)
|
||||
# needed by LuxTTS for text-to-phoneme conversion
|
||||
'--collect-all', 'piper_phonemize',
|
||||
])
|
||||
args.extend(
|
||||
[
|
||||
"--hidden-import",
|
||||
"backend",
|
||||
"--hidden-import",
|
||||
"backend.main",
|
||||
"--hidden-import",
|
||||
"backend.config",
|
||||
"--hidden-import",
|
||||
"backend.database",
|
||||
"--hidden-import",
|
||||
"backend.models",
|
||||
"--hidden-import",
|
||||
"backend.services.profiles",
|
||||
"--hidden-import",
|
||||
"backend.services.history",
|
||||
"--hidden-import",
|
||||
"backend.services.tts",
|
||||
"--hidden-import",
|
||||
"backend.services.transcribe",
|
||||
"--hidden-import",
|
||||
"backend.utils.platform_detect",
|
||||
"--hidden-import",
|
||||
"backend.backends",
|
||||
"--hidden-import",
|
||||
"backend.backends.pytorch_backend",
|
||||
"--hidden-import",
|
||||
"backend.backends.qwen_custom_voice_backend",
|
||||
"--hidden-import",
|
||||
"backend.utils.audio",
|
||||
"--hidden-import",
|
||||
"backend.utils.cache",
|
||||
"--hidden-import",
|
||||
"backend.utils.progress",
|
||||
"--hidden-import",
|
||||
"backend.utils.hf_progress",
|
||||
"--hidden-import",
|
||||
"backend.services.cuda",
|
||||
"--hidden-import",
|
||||
"backend.services.effects",
|
||||
"--hidden-import",
|
||||
"backend.utils.effects",
|
||||
"--hidden-import",
|
||||
"backend.services.versions",
|
||||
"--hidden-import",
|
||||
"pedalboard",
|
||||
"--hidden-import",
|
||||
"chatterbox",
|
||||
"--hidden-import",
|
||||
"chatterbox.tts_turbo",
|
||||
"--hidden-import",
|
||||
"chatterbox.mtl_tts",
|
||||
"--hidden-import",
|
||||
"backend.backends.chatterbox_backend",
|
||||
"--hidden-import",
|
||||
"backend.backends.chatterbox_turbo_backend",
|
||||
# chatterbox multilingual uses spacy_pkuseg for Chinese word
|
||||
# segmentation, which ships pickled dict files (dicts/default.pkl)
|
||||
# and native .so extensions that --hidden-import alone won't bundle.
|
||||
"--collect-all",
|
||||
"spacy_pkuseg",
|
||||
"--hidden-import",
|
||||
"backend.backends.luxtts_backend",
|
||||
"--hidden-import",
|
||||
"zipvoice",
|
||||
"--hidden-import",
|
||||
"zipvoice.luxvoice",
|
||||
"--collect-all",
|
||||
"zipvoice",
|
||||
"--collect-all",
|
||||
"linacodec",
|
||||
"--hidden-import",
|
||||
"torch",
|
||||
"--hidden-import",
|
||||
"transformers",
|
||||
"--hidden-import",
|
||||
"fastapi",
|
||||
"--hidden-import",
|
||||
"uvicorn",
|
||||
"--hidden-import",
|
||||
"sqlalchemy",
|
||||
# librosa uses lazy_loader which generates .pyi stub files at
|
||||
# install time and reads them at runtime to discover submodules.
|
||||
# --hidden-import alone doesn't bundle the stubs, causing
|
||||
# "Cannot load imports from non-existent stub" at runtime.
|
||||
"--collect-all",
|
||||
"lazy_loader",
|
||||
"--collect-all",
|
||||
"librosa",
|
||||
"--hidden-import",
|
||||
"soundfile",
|
||||
"--hidden-import",
|
||||
"qwen_tts",
|
||||
"--hidden-import",
|
||||
"qwen_tts.inference",
|
||||
"--hidden-import",
|
||||
"qwen_tts.inference.qwen3_tts_model",
|
||||
"--hidden-import",
|
||||
"qwen_tts.inference.qwen3_tts_tokenizer",
|
||||
"--hidden-import",
|
||||
"qwen_tts.core",
|
||||
"--hidden-import",
|
||||
"qwen_tts.cli",
|
||||
"--copy-metadata",
|
||||
"qwen-tts",
|
||||
"--copy-metadata",
|
||||
"requests",
|
||||
"--copy-metadata",
|
||||
"transformers",
|
||||
"--copy-metadata",
|
||||
"huggingface-hub",
|
||||
"--copy-metadata",
|
||||
"tokenizers",
|
||||
"--copy-metadata",
|
||||
"safetensors",
|
||||
"--copy-metadata",
|
||||
"tqdm",
|
||||
"--hidden-import",
|
||||
"requests",
|
||||
# qwen_tts uses inspect.getsource() at runtime to locate
|
||||
# modeling_qwen3_tts.py — needs physical .py source files bundled
|
||||
"--collect-all",
|
||||
"qwen_tts",
|
||||
# Fix for pkg_resources and jaraco namespace packages
|
||||
"--hidden-import",
|
||||
"pkg_resources.extern",
|
||||
"--collect-submodules",
|
||||
"jaraco",
|
||||
# inflect uses typeguard @typechecked which calls inspect.getsource()
|
||||
# at import time — needs .py source files, not just .pyc bytecode
|
||||
"--collect-all",
|
||||
"inflect",
|
||||
# perth ships pretrained watermark model files (hparams.yaml, .pth.tar)
|
||||
# in perth/perth_net/pretrained/ — needed by chatterbox at runtime
|
||||
"--collect-all",
|
||||
"perth",
|
||||
# piper_phonemize ships espeak-ng-data/ (phoneme tables, language dicts)
|
||||
# needed by LuxTTS for text-to-phoneme conversion
|
||||
"--collect-all",
|
||||
"piper_phonemize",
|
||||
# HumeAI TADA — speech-language model using Llama + flow matching
|
||||
"--hidden-import",
|
||||
"backend.backends.hume_backend",
|
||||
"--hidden-import",
|
||||
"tada",
|
||||
"--hidden-import",
|
||||
"tada.modules",
|
||||
"--hidden-import",
|
||||
"tada.modules.tada",
|
||||
"--hidden-import",
|
||||
"tada.modules.encoder",
|
||||
"--hidden-import",
|
||||
"tada.modules.decoder",
|
||||
"--hidden-import",
|
||||
"tada.modules.aligner",
|
||||
"--hidden-import",
|
||||
"tada.modules.acoustic_spkr_verf",
|
||||
"--hidden-import",
|
||||
"tada.nn",
|
||||
"--hidden-import",
|
||||
"tada.nn.vibevoice",
|
||||
"--hidden-import",
|
||||
"tada.utils",
|
||||
"--hidden-import",
|
||||
"tada.utils.gray_code",
|
||||
"--hidden-import",
|
||||
"tada.utils.text",
|
||||
# DAC shim — provides dac.nn.layers.Snake1d without the real
|
||||
# descript-audio-codec package (which pulls onnx/tensorboard via
|
||||
# descript-audiotools). The shim is in backend/utils/dac_shim.py.
|
||||
"--hidden-import",
|
||||
"backend.utils.dac_shim",
|
||||
"--hidden-import",
|
||||
"torchaudio",
|
||||
"--collect-submodules",
|
||||
"tada",
|
||||
# Kokoro 82M — lightweight TTS engine using misaki G2P
|
||||
# collect-all is required because transformers introspects .py source
|
||||
# files at runtime (e.g. _can_set_attn_implementation opens the class
|
||||
# file); hidden-import alone only bundles bytecode.
|
||||
"--hidden-import",
|
||||
"backend.backends.kokoro_backend",
|
||||
"--collect-all",
|
||||
"kokoro",
|
||||
# misaki ships G2P data files (dictionaries, phoneme tables)
|
||||
# that must be bundled for espeak/en/ja/zh G2P to work
|
||||
"--collect-all",
|
||||
"misaki",
|
||||
# language_tags ships JSON data files (index.json etc.) loaded at
|
||||
# runtime via: misaki → phonemizer → segments → csvw → language_tags
|
||||
"--collect-all",
|
||||
"language_tags",
|
||||
# espeakng_loader ships the entire espeak-ng-data directory (369 files)
|
||||
# loaded at import time by misaki.espeak via get_data_path()
|
||||
"--collect-all",
|
||||
"espeakng_loader",
|
||||
# spacy en_core_web_sm model — misaki.en tries to spacy.cli.download()
|
||||
# at runtime if not found, which calls pip as a subprocess and crashes
|
||||
# the frozen binary. Bundle the model so spacy.util.is_package() passes.
|
||||
"--collect-all",
|
||||
"en_core_web_sm",
|
||||
"--copy-metadata",
|
||||
"en_core_web_sm",
|
||||
"--hidden-import",
|
||||
"en_core_web_sm",
|
||||
"--hidden-import",
|
||||
"loguru",
|
||||
]
|
||||
)
|
||||
|
||||
# Add CUDA-specific hidden imports
|
||||
if cuda:
|
||||
print("Building with CUDA support")
|
||||
args.extend([
|
||||
'--hidden-import', 'torch.cuda',
|
||||
'--hidden-import', 'torch.backends.cudnn',
|
||||
])
|
||||
logger.info("Building with CUDA support")
|
||||
args.extend(
|
||||
[
|
||||
"--hidden-import",
|
||||
"torch.cuda",
|
||||
"--hidden-import",
|
||||
"torch.backends.cudnn",
|
||||
]
|
||||
)
|
||||
else:
|
||||
# Exclude NVIDIA CUDA packages from CPU-only builds to keep binary small.
|
||||
# When building from a venv with CUDA torch installed, PyInstaller would
|
||||
# bundle ~3GB of NVIDIA shared libraries. We exclude both the Python
|
||||
# modules and the binary DLLs.
|
||||
nvidia_packages = [
|
||||
'nvidia', 'nvidia.cublas', 'nvidia.cuda_cupti', 'nvidia.cuda_nvrtc',
|
||||
'nvidia.cuda_runtime', 'nvidia.cudnn', 'nvidia.cufft', 'nvidia.curand',
|
||||
'nvidia.cusolver', 'nvidia.cusparse', 'nvidia.nccl', 'nvidia.nvjitlink',
|
||||
'nvidia.nvtx',
|
||||
"nvidia",
|
||||
"nvidia.cublas",
|
||||
"nvidia.cuda_cupti",
|
||||
"nvidia.cuda_nvrtc",
|
||||
"nvidia.cuda_runtime",
|
||||
"nvidia.cudnn",
|
||||
"nvidia.cufft",
|
||||
"nvidia.curand",
|
||||
"nvidia.cusolver",
|
||||
"nvidia.cusparse",
|
||||
"nvidia.nccl",
|
||||
"nvidia.nvjitlink",
|
||||
"nvidia.nvtx",
|
||||
]
|
||||
for pkg in nvidia_packages:
|
||||
args.extend(['--exclude-module', pkg])
|
||||
args.extend(["--exclude-module", pkg])
|
||||
|
||||
# Add MLX-specific imports if building on Apple Silicon (never for CUDA builds)
|
||||
if is_apple_silicon() and not cuda:
|
||||
print("Building for Apple Silicon - including MLX dependencies")
|
||||
args.extend([
|
||||
'--hidden-import', 'backend.backends.mlx_backend',
|
||||
'--hidden-import', 'mlx',
|
||||
'--hidden-import', 'mlx.core',
|
||||
'--hidden-import', 'mlx.nn',
|
||||
'--hidden-import', 'mlx_audio',
|
||||
'--hidden-import', 'mlx_audio.tts',
|
||||
'--hidden-import', 'mlx_audio.stt',
|
||||
'--collect-submodules', 'mlx',
|
||||
'--collect-submodules', 'mlx_audio',
|
||||
# Use --collect-all so PyInstaller bundles both data files AND
|
||||
# native shared libraries (.dylib, .metallib) for MLX.
|
||||
# Previously only --collect-data was used, which caused MLX to
|
||||
# raise OSError at runtime inside the bundled binary because
|
||||
# the Metal shader libraries were missing.
|
||||
'--collect-all', 'mlx',
|
||||
'--collect-all', 'mlx_audio',
|
||||
])
|
||||
logger.info("Building for Apple Silicon - including MLX dependencies")
|
||||
args.extend(
|
||||
[
|
||||
"--hidden-import",
|
||||
"backend.backends.mlx_backend",
|
||||
"--hidden-import",
|
||||
"mlx",
|
||||
"--hidden-import",
|
||||
"mlx.core",
|
||||
"--hidden-import",
|
||||
"mlx.nn",
|
||||
"--hidden-import",
|
||||
"mlx_audio",
|
||||
"--hidden-import",
|
||||
"mlx_audio.tts",
|
||||
"--hidden-import",
|
||||
"mlx_audio.stt",
|
||||
"--collect-submodules",
|
||||
"mlx",
|
||||
"--collect-submodules",
|
||||
"mlx_audio",
|
||||
# Use --collect-all so PyInstaller bundles both data files AND
|
||||
# native shared libraries (.dylib, .metallib) for MLX.
|
||||
# Previously only --collect-data was used, which caused MLX to
|
||||
# raise OSError at runtime inside the bundled binary because
|
||||
# the Metal shader libraries were missing.
|
||||
"--collect-all",
|
||||
"mlx",
|
||||
"--collect-all",
|
||||
"mlx_audio",
|
||||
]
|
||||
)
|
||||
elif not cuda:
|
||||
print("Building for non-Apple Silicon platform - PyTorch only")
|
||||
logger.info("Building for non-Apple Silicon platform - PyTorch only")
|
||||
|
||||
dist_dir = str(backend_dir / 'dist')
|
||||
build_dir = str(backend_dir / 'build')
|
||||
dist_dir = str(backend_dir / "dist")
|
||||
build_dir = str(backend_dir / "build")
|
||||
|
||||
args.extend([
|
||||
'--distpath', dist_dir,
|
||||
'--workpath', build_dir,
|
||||
'--noconfirm',
|
||||
'--clean',
|
||||
])
|
||||
args.extend(
|
||||
[
|
||||
"--distpath",
|
||||
dist_dir,
|
||||
"--workpath",
|
||||
build_dir,
|
||||
"--noconfirm",
|
||||
"--clean",
|
||||
]
|
||||
)
|
||||
|
||||
# Change to backend directory
|
||||
os.chdir(backend_dir)
|
||||
|
||||
|
||||
# For CPU builds on Windows, ensure we're using CPU-only torch.
|
||||
# If CUDA torch is installed (local dev), swap to CPU torch before building,
|
||||
# then restore CUDA torch after. This prevents PyInstaller from bundling
|
||||
@@ -184,17 +387,28 @@ def build_server(cuda=False):
|
||||
restore_cuda = False
|
||||
if not cuda and platform.system() == "Windows":
|
||||
import subprocess
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", "import torch; print(torch.version.cuda or '')"],
|
||||
capture_output=True, text=True
|
||||
[sys.executable, "-c", "import torch; print(torch.version.cuda or '')"], capture_output=True, text=True
|
||||
)
|
||||
has_cuda_torch = bool(result.stdout.strip())
|
||||
if has_cuda_torch:
|
||||
print("CUDA torch detected — installing CPU torch for CPU build...")
|
||||
logger.info("CUDA torch detected — installing CPU torch for CPU build...")
|
||||
subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "torch", "torchvision", "torchaudio",
|
||||
"--index-url", "https://download.pytorch.org/whl/cpu", "--force-reinstall", "-q"],
|
||||
check=True
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"pip",
|
||||
"install",
|
||||
"torch",
|
||||
"torchvision",
|
||||
"torchaudio",
|
||||
"--index-url",
|
||||
"https://download.pytorch.org/whl/cpu",
|
||||
"--force-reinstall",
|
||||
"-q",
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
restore_cuda = True
|
||||
|
||||
@@ -204,57 +418,34 @@ def build_server(cuda=False):
|
||||
finally:
|
||||
# Restore CUDA torch if we swapped it out (even on build failure)
|
||||
if restore_cuda:
|
||||
print("Restoring CUDA torch...")
|
||||
logger.info("Restoring CUDA torch...")
|
||||
import subprocess
|
||||
|
||||
subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "torch", "torchvision", "torchaudio",
|
||||
"--index-url", "https://download.pytorch.org/whl/cu126", "--force-reinstall", "-q"],
|
||||
check=True
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"pip",
|
||||
"install",
|
||||
"torch",
|
||||
"torchvision",
|
||||
"torchaudio",
|
||||
"--index-url",
|
||||
"https://download.pytorch.org/whl/cu128",
|
||||
"--force-reinstall",
|
||||
"-q",
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
|
||||
print(f"Binary built in {backend_dir / 'dist' / binary_name}")
|
||||
|
||||
logger.info("Binary built in %s", backend_dir / "dist" / binary_name)
|
||||
|
||||
|
||||
def _get_cuda_dll_excludes():
|
||||
"""Get list of CUDA DLL filenames to exclude from CPU builds.
|
||||
|
||||
When building locally with CUDA torch installed, PyInstaller bundles ~3GB of
|
||||
CUDA DLLs from torch/lib/. Returns a list of DLL filenames to exclude.
|
||||
"""
|
||||
try:
|
||||
import torch
|
||||
torch_lib = Path(torch.__file__).parent / 'lib'
|
||||
except ImportError:
|
||||
return []
|
||||
|
||||
cuda_prefixes = (
|
||||
'torch_cuda', 'cublas', 'cublasLt', 'cudnn', 'cusparse', 'cufft',
|
||||
'cusolver', 'cusolverMg', 'curand', 'nvrtc', 'nvJitLink', 'nccl',
|
||||
'nvperf', 'nvrtc-builtins',
|
||||
)
|
||||
|
||||
exclude_dlls = []
|
||||
if torch_lib.exists():
|
||||
for f in torch_lib.iterdir():
|
||||
if f.suffix == '.dll' and any(f.name.startswith(p) for p in cuda_prefixes):
|
||||
exclude_dlls.append(f.name)
|
||||
|
||||
if exclude_dlls:
|
||||
total_mb = sum(
|
||||
(torch_lib / dll).stat().st_size
|
||||
for dll in exclude_dlls
|
||||
if (torch_lib / dll).exists()
|
||||
) / 1024 / 1024
|
||||
print(f"CPU build: will exclude {len(exclude_dlls)} CUDA DLLs ({total_mb:.0f} MB)")
|
||||
|
||||
return exclude_dlls
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="Build voicebox-server binary")
|
||||
parser.add_argument(
|
||||
'--cuda',
|
||||
action='store_true',
|
||||
"--cuda",
|
||||
action="store_true",
|
||||
help="Build CUDA-enabled binary (voicebox-server-cuda)",
|
||||
)
|
||||
cli_args = parser.parse_args()
|
||||
|
||||
+69
-4
@@ -4,19 +4,38 @@ Configuration module for voicebox backend.
|
||||
Handles data directory configuration for production bundling.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Allow users to override the HuggingFace model download directory.
|
||||
# Set VOICEBOX_MODELS_DIR to an absolute path before starting the server.
|
||||
# This sets HF_HUB_CACHE so all huggingface_hub downloads go to that path.
|
||||
_custom_models_dir = os.environ.get("VOICEBOX_MODELS_DIR")
|
||||
if _custom_models_dir:
|
||||
os.environ["HF_HUB_CACHE"] = _custom_models_dir
|
||||
print(f"[config] Model download path set to: {_custom_models_dir}")
|
||||
logger.info("Model download path set to: %s", _custom_models_dir)
|
||||
|
||||
# Default data directory (used in development)
|
||||
_data_dir = Path("data")
|
||||
_data_dir = Path("data").resolve()
|
||||
|
||||
|
||||
def _path_relative_to_any_data_dir(path: Path) -> Path | None:
|
||||
"""Extract the path within a data dir from an absolute or relative path."""
|
||||
parts = path.parts
|
||||
for idx, part in enumerate(parts):
|
||||
if part != "data":
|
||||
continue
|
||||
|
||||
tail = parts[idx + 1 :]
|
||||
if tail:
|
||||
return Path(*tail)
|
||||
return Path()
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def set_data_dir(path: str | Path):
|
||||
"""
|
||||
@@ -26,9 +45,10 @@ def set_data_dir(path: str | Path):
|
||||
path: Path to the data directory
|
||||
"""
|
||||
global _data_dir
|
||||
_data_dir = Path(path)
|
||||
_data_dir = Path(path).resolve()
|
||||
_data_dir.mkdir(parents=True, exist_ok=True)
|
||||
print(f"Data directory set to: {_data_dir.absolute()}")
|
||||
logger.info("Data directory set to: %s", _data_dir)
|
||||
|
||||
|
||||
def get_data_dir() -> Path:
|
||||
"""
|
||||
@@ -39,28 +59,73 @@ def get_data_dir() -> Path:
|
||||
"""
|
||||
return _data_dir
|
||||
|
||||
|
||||
def to_storage_path(path: str | Path) -> str:
|
||||
"""Convert a filesystem path to a DB-safe path relative to the data dir."""
|
||||
resolved_path = Path(path).resolve()
|
||||
|
||||
relative_to_any_data_dir = _path_relative_to_any_data_dir(resolved_path)
|
||||
if relative_to_any_data_dir is not None:
|
||||
return str(relative_to_any_data_dir)
|
||||
|
||||
try:
|
||||
return str(resolved_path.relative_to(_data_dir))
|
||||
except ValueError:
|
||||
return str(resolved_path)
|
||||
|
||||
|
||||
def resolve_storage_path(path: str | Path | None) -> Path | None:
|
||||
"""Resolve a DB-stored path against the configured data dir."""
|
||||
if path is None:
|
||||
return None
|
||||
|
||||
stored_path = Path(path)
|
||||
if stored_path.is_absolute():
|
||||
rebased_path = _path_relative_to_any_data_dir(stored_path)
|
||||
if rebased_path is not None:
|
||||
candidate = (_data_dir / rebased_path).resolve()
|
||||
if candidate.exists() or not stored_path.exists():
|
||||
return candidate
|
||||
|
||||
return stored_path
|
||||
|
||||
# 0.3.0 records sometimes stored relative paths with the data-dir name
|
||||
# baked in (e.g. "data/profiles/..."). Joining those directly with
|
||||
# _data_dir produces a spurious "<data_dir>/data/profiles/..." nest.
|
||||
if stored_path.parts and stored_path.parts[0] == "data":
|
||||
stored_path = (
|
||||
Path(*stored_path.parts[1:]) if len(stored_path.parts) > 1 else Path()
|
||||
)
|
||||
|
||||
return (_data_dir / stored_path).resolve()
|
||||
|
||||
|
||||
def get_db_path() -> Path:
|
||||
"""Get database file path."""
|
||||
return _data_dir / "voicebox.db"
|
||||
|
||||
|
||||
def get_profiles_dir() -> Path:
|
||||
"""Get profiles directory path."""
|
||||
path = _data_dir / "profiles"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def get_generations_dir() -> Path:
|
||||
"""Get generations directory path."""
|
||||
path = _data_dir / "generations"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def get_cache_dir() -> Path:
|
||||
"""Get cache directory path."""
|
||||
path = _data_dir / "cache"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def get_models_dir() -> Path:
|
||||
"""Get models directory path."""
|
||||
path = _data_dir / "models"
|
||||
|
||||
@@ -1,259 +0,0 @@
|
||||
"""
|
||||
CUDA backend binary download, assembly, and verification.
|
||||
|
||||
Downloads split parts of the CUDA-enabled voicebox-server binary from
|
||||
GitHub Releases, reassembles them, verifies integrity via SHA-256,
|
||||
and places the binary in the app's data directory for use on next
|
||||
backend restart.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from .config import get_data_dir
|
||||
from .utils.progress import get_progress_manager
|
||||
from . import __version__
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
GITHUB_RELEASES_URL = "https://github.com/jamiepine/voicebox/releases/download"
|
||||
|
||||
PROGRESS_KEY = "cuda-backend"
|
||||
|
||||
|
||||
def get_backends_dir() -> Path:
|
||||
"""Directory where downloaded backend binaries are stored."""
|
||||
d = get_data_dir() / "backends"
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def get_cuda_binary_name() -> str:
|
||||
"""Platform-specific CUDA binary filename."""
|
||||
if sys.platform == "win32":
|
||||
return "voicebox-server-cuda.exe"
|
||||
return "voicebox-server-cuda"
|
||||
|
||||
|
||||
def get_cuda_binary_path() -> Optional[Path]:
|
||||
"""Return path to CUDA binary if it exists."""
|
||||
p = get_backends_dir() / get_cuda_binary_name()
|
||||
if p.exists():
|
||||
return p
|
||||
return None
|
||||
|
||||
|
||||
def is_cuda_active() -> bool:
|
||||
"""Check if the current process is the CUDA binary.
|
||||
|
||||
The CUDA binary sets this env var on startup (see server.py).
|
||||
"""
|
||||
return os.environ.get("VOICEBOX_BACKEND_VARIANT") == "cuda"
|
||||
|
||||
|
||||
def get_cuda_status() -> dict:
|
||||
"""Get current CUDA backend status for the API."""
|
||||
progress_manager = get_progress_manager()
|
||||
cuda_path = get_cuda_binary_path()
|
||||
progress = progress_manager.get_progress(PROGRESS_KEY)
|
||||
|
||||
return {
|
||||
"available": cuda_path is not None,
|
||||
"active": is_cuda_active(),
|
||||
"binary_path": str(cuda_path) if cuda_path else None,
|
||||
"downloading": progress is not None and progress.get("status") == "downloading",
|
||||
"download_progress": progress,
|
||||
}
|
||||
|
||||
|
||||
async def download_cuda_binary(version: Optional[str] = None):
|
||||
"""Download the CUDA backend binary from GitHub Releases.
|
||||
|
||||
Downloads split parts listed in a manifest file, concatenates them,
|
||||
and verifies the SHA-256 checksum for integrity. Atomic write
|
||||
(temp file -> rename).
|
||||
|
||||
Args:
|
||||
version: Version tag (e.g. "v0.2.0"). Defaults to current app version.
|
||||
"""
|
||||
import httpx
|
||||
|
||||
if version is None:
|
||||
version = f"v{__version__}"
|
||||
|
||||
progress = get_progress_manager()
|
||||
binary_name = get_cuda_binary_name()
|
||||
dest_dir = get_backends_dir()
|
||||
final_path = dest_dir / binary_name
|
||||
temp_path = dest_dir / f"{binary_name}.download"
|
||||
|
||||
# Clean up any leftover partial download
|
||||
if temp_path.exists():
|
||||
temp_path.unlink()
|
||||
|
||||
logger.info(f"Starting CUDA backend download for {version}")
|
||||
progress.update_progress(
|
||||
PROGRESS_KEY, current=0, total=0,
|
||||
filename="Fetching manifest...", status="downloading",
|
||||
)
|
||||
|
||||
base_url = f"{GITHUB_RELEASES_URL}/{version}"
|
||||
stem = Path(binary_name).stem # voicebox-server-cuda
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=30.0) as client:
|
||||
# Fetch the manifest (list of split part filenames)
|
||||
manifest_url = f"{base_url}/{stem}.manifest"
|
||||
manifest_resp = await client.get(manifest_url)
|
||||
manifest_resp.raise_for_status()
|
||||
parts = [p.strip() for p in manifest_resp.text.strip().splitlines() if p.strip()]
|
||||
|
||||
if not parts:
|
||||
raise ValueError("Empty manifest — no split parts found")
|
||||
|
||||
logger.info(f"Found {len(parts)} split parts to download")
|
||||
|
||||
# Fetch expected checksum (optional — for integrity verification)
|
||||
expected_sha = None
|
||||
try:
|
||||
sha_url = f"{base_url}/{stem}.sha256"
|
||||
sha_resp = await client.get(sha_url)
|
||||
if sha_resp.status_code == 200:
|
||||
# Format: "sha256hex filename\n"
|
||||
expected_sha = sha_resp.text.strip().split()[0]
|
||||
logger.info(f"Expected SHA-256: {expected_sha[:16]}...")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not fetch checksum file — skipping verification: {e}")
|
||||
|
||||
# Get total size across all parts by issuing HEAD requests
|
||||
total_size = 0
|
||||
for part_name in parts:
|
||||
try:
|
||||
head_resp = await client.head(f"{base_url}/{part_name}")
|
||||
content_length = int(head_resp.headers.get("content-length", 0))
|
||||
total_size += content_length
|
||||
except Exception:
|
||||
pass
|
||||
logger.info(f"Total download size: {total_size / 1024 / 1024:.1f} MB")
|
||||
|
||||
# Download and concatenate parts
|
||||
total_downloaded = 0
|
||||
with open(temp_path, "wb") as f:
|
||||
for i, part_name in enumerate(parts):
|
||||
part_url = f"{base_url}/{part_name}"
|
||||
logger.info(f"Downloading part {i + 1}/{len(parts)}: {part_name}")
|
||||
|
||||
async with client.stream("GET", part_url) as response:
|
||||
response.raise_for_status()
|
||||
async for chunk in response.aiter_bytes(chunk_size=1024 * 1024):
|
||||
f.write(chunk)
|
||||
total_downloaded += len(chunk)
|
||||
progress.update_progress(
|
||||
PROGRESS_KEY, current=total_downloaded, total=total_size,
|
||||
filename=f"Downloading CUDA backend ({i + 1}/{len(parts)})",
|
||||
status="downloading",
|
||||
)
|
||||
|
||||
# Verify integrity if checksum was available
|
||||
if expected_sha:
|
||||
progress.update_progress(
|
||||
PROGRESS_KEY, current=total_downloaded, total=total_downloaded,
|
||||
filename="Verifying integrity...", status="downloading",
|
||||
)
|
||||
sha256 = hashlib.sha256()
|
||||
with open(temp_path, "rb") as f:
|
||||
while True:
|
||||
chunk = f.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
sha256.update(chunk)
|
||||
|
||||
actual = sha256.hexdigest()
|
||||
if actual != expected_sha:
|
||||
raise ValueError(
|
||||
f"Integrity check failed: expected {expected_sha[:16]}..., "
|
||||
f"got {actual[:16]}..."
|
||||
)
|
||||
logger.info(f"Integrity verified: {actual[:16]}...")
|
||||
|
||||
# Atomic move into place (replace handles existing target on all platforms)
|
||||
temp_path.replace(final_path)
|
||||
|
||||
# Make executable on Unix
|
||||
if sys.platform != "win32":
|
||||
final_path.chmod(0o755)
|
||||
|
||||
logger.info(f"CUDA backend downloaded to {final_path}")
|
||||
progress.mark_complete(PROGRESS_KEY)
|
||||
|
||||
except Exception as e:
|
||||
# Clean up on failure
|
||||
if temp_path.exists():
|
||||
temp_path.unlink()
|
||||
logger.error(f"CUDA backend download failed: {e}")
|
||||
progress.mark_error(PROGRESS_KEY, str(e))
|
||||
raise
|
||||
|
||||
|
||||
def get_cuda_binary_version() -> Optional[str]:
|
||||
"""Get the version of the installed CUDA binary, or None if not installed."""
|
||||
import subprocess
|
||||
cuda_path = get_cuda_binary_path()
|
||||
if not cuda_path:
|
||||
return None
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[str(cuda_path), "--version"],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
# Output format: "voicebox-server 0.2.0"
|
||||
for line in result.stdout.strip().splitlines():
|
||||
if "voicebox-server" in line:
|
||||
return line.split()[-1]
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not get CUDA binary version: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def check_and_update_cuda_binary():
|
||||
"""Check if the CUDA binary is outdated and auto-download if so.
|
||||
|
||||
Called on server startup. If a CUDA binary exists but its version
|
||||
doesn't match the current app version, triggers a background download
|
||||
of the updated CUDA binary. The download progress is visible to the
|
||||
frontend via the existing SSE progress endpoint.
|
||||
"""
|
||||
cuda_path = get_cuda_binary_path()
|
||||
if not cuda_path:
|
||||
return # No CUDA binary installed, nothing to update
|
||||
|
||||
cuda_version = get_cuda_binary_version()
|
||||
current_version = __version__
|
||||
|
||||
if cuda_version == current_version:
|
||||
logger.info(f"CUDA binary is up to date (v{current_version})")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
f"CUDA binary version mismatch: binary=v{cuda_version}, app=v{current_version}. "
|
||||
f"Auto-downloading updated CUDA backend..."
|
||||
)
|
||||
|
||||
try:
|
||||
await download_cuda_binary()
|
||||
except Exception as e:
|
||||
logger.error(f"Auto-update of CUDA binary failed: {e}")
|
||||
|
||||
|
||||
async def delete_cuda_binary() -> bool:
|
||||
"""Delete the downloaded CUDA binary. Returns True if deleted."""
|
||||
path = get_cuda_binary_path()
|
||||
if path and path.exists():
|
||||
path.unlink()
|
||||
logger.info(f"Deleted CUDA binary: {path}")
|
||||
return True
|
||||
return False
|
||||
@@ -1,487 +0,0 @@
|
||||
"""
|
||||
SQLite database ORM using SQLAlchemy.
|
||||
"""
|
||||
|
||||
from sqlalchemy import create_engine, Column, String, Integer, Float, DateTime, Text, ForeignKey, Boolean
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import sessionmaker, Session
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
from . import config
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
class VoiceProfile(Base):
|
||||
"""Voice profile database model."""
|
||||
__tablename__ = "profiles"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text)
|
||||
language = Column(String, default="en")
|
||||
avatar_path = Column(String, nullable=True)
|
||||
effects_chain = Column(Text, nullable=True) # JSON-serialized default effects chain
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class ProfileSample(Base):
|
||||
"""Voice profile sample database model."""
|
||||
__tablename__ = "profile_samples"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
|
||||
audio_path = Column(String, nullable=False)
|
||||
reference_text = Column(Text, nullable=False)
|
||||
|
||||
|
||||
class Generation(Base):
|
||||
"""Generation history database model."""
|
||||
__tablename__ = "generations"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
|
||||
text = Column(Text, nullable=False)
|
||||
language = Column(String, default="en")
|
||||
audio_path = Column(String, nullable=True)
|
||||
duration = Column(Float, nullable=True)
|
||||
seed = Column(Integer)
|
||||
instruct = Column(Text)
|
||||
engine = Column(String, default="qwen")
|
||||
model_size = Column(String, nullable=True)
|
||||
status = Column(String, default="completed") # generating, completed, failed
|
||||
error = Column(Text, nullable=True)
|
||||
is_favorited = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class Story(Base):
|
||||
"""Story database model."""
|
||||
__tablename__ = "stories"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
description = Column(Text)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class StoryItem(Base):
|
||||
"""Story item database model (links generations to stories)."""
|
||||
__tablename__ = "story_items"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
story_id = Column(String, ForeignKey("stories.id"), nullable=False)
|
||||
generation_id = Column(String, ForeignKey("generations.id"), nullable=False)
|
||||
version_id = Column(String, ForeignKey("generation_versions.id"), nullable=True) # Pin to specific version, null = use generation default
|
||||
start_time_ms = Column(Integer, nullable=False, default=0) # Milliseconds from story start
|
||||
track = Column(Integer, nullable=False, default=0) # Track number (0 = main track)
|
||||
trim_start_ms = Column(Integer, nullable=False, default=0) # Milliseconds trimmed from start
|
||||
trim_end_ms = Column(Integer, nullable=False, default=0) # Milliseconds trimmed from end
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class Project(Base):
|
||||
"""Audio studio project database model."""
|
||||
__tablename__ = "projects"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
data = Column(Text) # JSON string
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class GenerationVersion(Base):
|
||||
"""A version of a generation's audio (clean, processed, alternate takes)."""
|
||||
__tablename__ = "generation_versions"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
generation_id = Column(String, ForeignKey("generations.id"), nullable=False)
|
||||
label = Column(String, nullable=False) # "clean", "processed", or user-defined
|
||||
audio_path = Column(String, nullable=False)
|
||||
effects_chain = Column(Text, nullable=True) # JSON-serialized effects config, null for clean
|
||||
source_version_id = Column(String, ForeignKey("generation_versions.id"), nullable=True) # Which version was used as input
|
||||
is_default = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class EffectPreset(Base):
|
||||
"""Saved effect chain preset."""
|
||||
__tablename__ = "effect_presets"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text, nullable=True)
|
||||
effects_chain = Column(Text, nullable=False) # JSON-serialized effects config
|
||||
is_builtin = Column(Boolean, default=False)
|
||||
sort_order = Column(Integer, default=100)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class AudioChannel(Base):
|
||||
"""Audio channel (bus) database model."""
|
||||
__tablename__ = "audio_channels"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
is_default = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class ChannelDeviceMapping(Base):
|
||||
"""Mapping between channels and OS audio devices."""
|
||||
__tablename__ = "channel_device_mappings"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"), nullable=False)
|
||||
device_id = Column(String, nullable=False) # OS device identifier
|
||||
|
||||
|
||||
class ProfileChannelMapping(Base):
|
||||
"""Mapping between voice profiles and audio channels (many-to-many)."""
|
||||
__tablename__ = "profile_channel_mappings"
|
||||
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
|
||||
|
||||
|
||||
# Database setup will be initialized in init_db()
|
||||
engine = None
|
||||
SessionLocal = None
|
||||
_db_path = None
|
||||
|
||||
|
||||
def init_db():
|
||||
"""Initialize database tables."""
|
||||
global engine, SessionLocal, _db_path
|
||||
|
||||
_db_path = config.get_db_path()
|
||||
_db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
engine = create_engine(
|
||||
f"sqlite:///{_db_path}",
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
|
||||
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
||||
|
||||
# Run migrations before creating tables
|
||||
_run_migrations(engine)
|
||||
|
||||
Base.metadata.create_all(bind=engine)
|
||||
|
||||
# Create default channel if it doesn't exist
|
||||
db = SessionLocal()
|
||||
try:
|
||||
default_channel = db.query(AudioChannel).filter(AudioChannel.is_default == True).first()
|
||||
if not default_channel:
|
||||
default_channel = AudioChannel(
|
||||
id=str(uuid.uuid4()),
|
||||
name="Default",
|
||||
is_default=True
|
||||
)
|
||||
db.add(default_channel)
|
||||
|
||||
# Assign all existing profiles to default channel
|
||||
profiles = db.query(VoiceProfile).all()
|
||||
for profile in profiles:
|
||||
mapping = ProfileChannelMapping(
|
||||
profile_id=profile.id,
|
||||
channel_id=default_channel.id
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
# Backfill: create "clean" GenerationVersion entries for existing generations
|
||||
_backfill_generation_versions()
|
||||
|
||||
# Seed built-in effect presets
|
||||
_seed_builtin_presets()
|
||||
|
||||
|
||||
def _run_migrations(engine):
|
||||
"""Run database migrations."""
|
||||
from sqlalchemy import inspect, text
|
||||
|
||||
inspector = inspect(engine)
|
||||
|
||||
# Check if story_items table exists
|
||||
if 'story_items' not in inspector.get_table_names():
|
||||
return # Table doesn't exist yet, will be created fresh
|
||||
|
||||
# Get columns in story_items table
|
||||
columns = {col['name'] for col in inspector.get_columns('story_items')}
|
||||
|
||||
# Migration: Remove position column and ensure start_time_ms exists
|
||||
# SQLite doesn't support DROP COLUMN easily, so we recreate the table
|
||||
if 'position' in columns:
|
||||
print("Migrating story_items: removing position column, using start_time_ms")
|
||||
|
||||
with engine.connect() as conn:
|
||||
# Check if start_time_ms already exists
|
||||
has_start_time = 'start_time_ms' in columns
|
||||
|
||||
if not has_start_time:
|
||||
# First, add the new column temporarily
|
||||
conn.execute(text("ALTER TABLE story_items ADD COLUMN start_time_ms INTEGER DEFAULT 0"))
|
||||
|
||||
# Calculate timecodes from position ordering
|
||||
result = conn.execute(text("""
|
||||
SELECT si.id, si.story_id, si.position, g.duration
|
||||
FROM story_items si
|
||||
JOIN generations g ON si.generation_id = g.id
|
||||
ORDER BY si.story_id, si.position
|
||||
"""))
|
||||
|
||||
rows = result.fetchall()
|
||||
|
||||
current_story_id = None
|
||||
current_time_ms = 0
|
||||
|
||||
for row in rows:
|
||||
item_id, story_id, position, duration = row
|
||||
|
||||
if story_id != current_story_id:
|
||||
current_story_id = story_id
|
||||
current_time_ms = 0
|
||||
|
||||
conn.execute(
|
||||
text("UPDATE story_items SET start_time_ms = :time WHERE id = :id"),
|
||||
{"time": current_time_ms, "id": item_id}
|
||||
)
|
||||
|
||||
current_time_ms += int(duration * 1000) + 200
|
||||
|
||||
conn.commit()
|
||||
|
||||
# Now recreate the table without the position column
|
||||
# 1. Create new table
|
||||
conn.execute(text("""
|
||||
CREATE TABLE story_items_new (
|
||||
id VARCHAR PRIMARY KEY,
|
||||
story_id VARCHAR NOT NULL,
|
||||
generation_id VARCHAR NOT NULL,
|
||||
start_time_ms INTEGER NOT NULL DEFAULT 0,
|
||||
created_at DATETIME,
|
||||
FOREIGN KEY (story_id) REFERENCES stories(id),
|
||||
FOREIGN KEY (generation_id) REFERENCES generations(id)
|
||||
)
|
||||
"""))
|
||||
|
||||
# 2. Copy data
|
||||
conn.execute(text("""
|
||||
INSERT INTO story_items_new (id, story_id, generation_id, start_time_ms, created_at)
|
||||
SELECT id, story_id, generation_id, start_time_ms, created_at FROM story_items
|
||||
"""))
|
||||
|
||||
# 3. Drop old table
|
||||
conn.execute(text("DROP TABLE story_items"))
|
||||
|
||||
# 4. Rename new table
|
||||
conn.execute(text("ALTER TABLE story_items_new RENAME TO story_items"))
|
||||
|
||||
conn.commit()
|
||||
print("Migrated story_items table to use start_time_ms (removed position column)")
|
||||
|
||||
# Migration: Add track column if it doesn't exist
|
||||
# Re-check columns after potential position migration
|
||||
columns = {col['name'] for col in inspector.get_columns('story_items')}
|
||||
if 'track' not in columns:
|
||||
print("Migrating story_items: adding track column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE story_items ADD COLUMN track INTEGER NOT NULL DEFAULT 0"))
|
||||
conn.commit()
|
||||
print("Added track column to story_items")
|
||||
|
||||
# Migration: Add trim columns if they don't exist
|
||||
# Re-check columns after potential track migration
|
||||
columns = {col['name'] for col in inspector.get_columns('story_items')}
|
||||
if 'trim_start_ms' not in columns:
|
||||
print("Migrating story_items: adding trim_start_ms column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE story_items ADD COLUMN trim_start_ms INTEGER NOT NULL DEFAULT 0"))
|
||||
conn.commit()
|
||||
print("Added trim_start_ms column to story_items")
|
||||
|
||||
columns = {col['name'] for col in inspector.get_columns('story_items')}
|
||||
if 'trim_end_ms' not in columns:
|
||||
print("Migrating story_items: adding trim_end_ms column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE story_items ADD COLUMN trim_end_ms INTEGER NOT NULL DEFAULT 0"))
|
||||
conn.commit()
|
||||
print("Added trim_end_ms column to story_items")
|
||||
|
||||
# Migration: Add avatar_path to profiles table
|
||||
if 'profiles' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('profiles')}
|
||||
if 'avatar_path' not in columns:
|
||||
print("Migrating profiles: adding avatar_path column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE profiles ADD COLUMN avatar_path VARCHAR"))
|
||||
conn.commit()
|
||||
print("Added avatar_path column to profiles")
|
||||
|
||||
# Migration: Add status and error columns to generations table
|
||||
if 'generations' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('generations')}
|
||||
if 'status' not in columns:
|
||||
print("Migrating generations: adding status column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generations ADD COLUMN status VARCHAR DEFAULT 'completed'"))
|
||||
conn.commit()
|
||||
print("Added status column to generations")
|
||||
if 'error' not in columns:
|
||||
print("Migrating generations: adding error column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generations ADD COLUMN error TEXT"))
|
||||
conn.commit()
|
||||
print("Added error column to generations")
|
||||
if 'engine' not in columns:
|
||||
print("Migrating generations: adding engine column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generations ADD COLUMN engine VARCHAR DEFAULT 'qwen'"))
|
||||
conn.commit()
|
||||
print("Added engine column to generations")
|
||||
# Re-read columns after engine migration (variable name shadows outer `engine`)
|
||||
columns = {col['name'] for col in inspector.get_columns('generations')}
|
||||
if 'model_size' not in columns:
|
||||
print("Migrating generations: adding model_size column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generations ADD COLUMN model_size VARCHAR"))
|
||||
conn.commit()
|
||||
print("Added model_size column to generations")
|
||||
|
||||
# Migration: Add effects_chain to profiles table
|
||||
if 'profiles' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('profiles')}
|
||||
if 'effects_chain' not in columns:
|
||||
print("Migrating profiles: adding effects_chain column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE profiles ADD COLUMN effects_chain TEXT"))
|
||||
conn.commit()
|
||||
print("Added effects_chain column to profiles")
|
||||
|
||||
# Migration: Add sort_order to effect_presets table
|
||||
if 'effect_presets' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('effect_presets')}
|
||||
if 'sort_order' not in columns:
|
||||
print("Migrating effect_presets: adding sort_order column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE effect_presets ADD COLUMN sort_order INTEGER DEFAULT 100"))
|
||||
conn.commit()
|
||||
print("Added sort_order column to effect_presets")
|
||||
|
||||
# Migration: Add version_id column to story_items table
|
||||
if 'story_items' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('story_items')}
|
||||
if 'version_id' not in columns:
|
||||
print("Migrating story_items: adding version_id column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE story_items ADD COLUMN version_id VARCHAR"))
|
||||
conn.commit()
|
||||
print("Added version_id column to story_items")
|
||||
|
||||
# Migration: Add source_version_id to generation_versions table
|
||||
if 'generation_versions' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('generation_versions')}
|
||||
if 'source_version_id' not in columns:
|
||||
print("Migrating generation_versions: adding source_version_id column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generation_versions ADD COLUMN source_version_id VARCHAR"))
|
||||
conn.commit()
|
||||
print("Added source_version_id column to generation_versions")
|
||||
|
||||
if 'generations' in inspector.get_table_names():
|
||||
columns = {col['name'] for col in inspector.get_columns('generations')}
|
||||
if 'is_favorited' not in columns:
|
||||
print("Migrating generations: adding is_favorited column")
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("ALTER TABLE generations ADD COLUMN is_favorited BOOLEAN DEFAULT 0"))
|
||||
conn.commit()
|
||||
print("Added is_favorited column to generations")
|
||||
|
||||
# Migration: Create generation_versions for existing generations
|
||||
# (populate after tables are created, handled in init_db)
|
||||
|
||||
|
||||
def _backfill_generation_versions():
|
||||
"""Create 'clean' version entries for existing generations that don't have any."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
from pathlib import Path as _Path
|
||||
|
||||
# Find generations that have no version entries
|
||||
existing_version_gen_ids = {
|
||||
row[0] for row in db.query(GenerationVersion.generation_id).all()
|
||||
}
|
||||
generations = db.query(Generation).filter(
|
||||
Generation.status == "completed",
|
||||
Generation.audio_path.isnot(None),
|
||||
Generation.audio_path != "",
|
||||
).all()
|
||||
|
||||
count = 0
|
||||
for gen in generations:
|
||||
if gen.id in existing_version_gen_ids:
|
||||
continue
|
||||
if not _Path(gen.audio_path).exists():
|
||||
continue
|
||||
version = GenerationVersion(
|
||||
id=str(uuid.uuid4()),
|
||||
generation_id=gen.id,
|
||||
label="clean",
|
||||
audio_path=gen.audio_path,
|
||||
effects_chain=None,
|
||||
is_default=True,
|
||||
)
|
||||
db.add(version)
|
||||
count += 1
|
||||
|
||||
if count > 0:
|
||||
db.commit()
|
||||
print(f"Backfilled {count} generation version entries")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def _seed_builtin_presets():
|
||||
"""Ensure built-in effect presets exist in the database."""
|
||||
import json
|
||||
from .utils.effects import BUILTIN_PRESETS
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
for idx, (key, preset_data) in enumerate(BUILTIN_PRESETS.items()):
|
||||
sort_order = preset_data.get("sort_order", idx)
|
||||
existing = db.query(EffectPreset).filter_by(name=preset_data["name"]).first()
|
||||
if not existing:
|
||||
preset = EffectPreset(
|
||||
id=str(uuid.uuid4()),
|
||||
name=preset_data["name"],
|
||||
description=preset_data.get("description"),
|
||||
effects_chain=json.dumps(preset_data["effects_chain"]),
|
||||
is_builtin=True,
|
||||
sort_order=sort_order,
|
||||
)
|
||||
db.add(preset)
|
||||
elif existing.sort_order != sort_order:
|
||||
existing.sort_order = sort_order
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def get_db():
|
||||
"""Get database session (generator for dependency injection)."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Database package — ORM models, session management, and migrations.
|
||||
|
||||
Re-exports all public symbols so that ``from .database import get_db``
|
||||
and ``from .database import Generation as DBGeneration`` continue to work
|
||||
without changing any importers.
|
||||
"""
|
||||
|
||||
from .models import (
|
||||
Base,
|
||||
AudioChannel,
|
||||
ChannelDeviceMapping,
|
||||
EffectPreset,
|
||||
Generation,
|
||||
GenerationVersion,
|
||||
ProfileChannelMapping,
|
||||
ProfileSample,
|
||||
Project,
|
||||
Story,
|
||||
StoryItem,
|
||||
VoiceProfile,
|
||||
)
|
||||
from .session import engine, SessionLocal, _db_path, init_db, get_db
|
||||
|
||||
__all__ = [
|
||||
# Models
|
||||
"Base",
|
||||
"AudioChannel",
|
||||
"ChannelDeviceMapping",
|
||||
"EffectPreset",
|
||||
"Generation",
|
||||
"GenerationVersion",
|
||||
"ProfileChannelMapping",
|
||||
"ProfileSample",
|
||||
"Project",
|
||||
"Story",
|
||||
"StoryItem",
|
||||
"VoiceProfile",
|
||||
# Session
|
||||
"engine",
|
||||
"SessionLocal",
|
||||
"_db_path",
|
||||
"init_db",
|
||||
"get_db",
|
||||
]
|
||||
@@ -0,0 +1,226 @@
|
||||
"""Column-level migrations for the voicebox SQLite database.
|
||||
|
||||
Why not Alembic? voicebox is a single-user desktop app shipping as a
|
||||
PyInstaller binary. Every user has exactly one SQLite file. Alembic's
|
||||
strengths -- migration tracking across environments, rollback, team
|
||||
coordination -- don't apply here and would add bundling complexity
|
||||
(alembic.ini, env.py, versions/ directory all need to survive
|
||||
PyInstaller). The column-existence checks below are idempotent, run in
|
||||
<50 ms on startup, and have worked reliably across 12 schema changes.
|
||||
If the project ever moves to a server-based deployment or Postgres, this
|
||||
decision should be revisited.
|
||||
|
||||
Adding a new migration:
|
||||
1. Append a new ``_migrate_*`` helper at the bottom of this file.
|
||||
2. Call it from ``run_migrations()`` in the appropriate spot.
|
||||
3. The helper should check column/table existence before acting
|
||||
(idempotent) and print a short message when it does real work.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import inspect, text
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def run_migrations(engine) -> None:
|
||||
"""Run all schema migrations. Safe to call on every startup."""
|
||||
inspector = inspect(engine)
|
||||
tables = set(inspector.get_table_names())
|
||||
|
||||
_migrate_story_items(engine, inspector, tables)
|
||||
_migrate_profiles(engine, inspector, tables)
|
||||
_migrate_generations(engine, inspector, tables)
|
||||
_migrate_effect_presets(engine, inspector, tables)
|
||||
_migrate_generation_versions(engine, inspector, tables)
|
||||
_normalize_storage_paths(engine, tables)
|
||||
|
||||
|
||||
# -- helpers ---------------------------------------------------------------
|
||||
|
||||
def _get_columns(inspector, table: str) -> set[str]:
|
||||
return {col["name"] for col in inspector.get_columns(table)}
|
||||
|
||||
|
||||
def _add_column(engine, table: str, column_sql: str, label: str) -> None:
|
||||
"""Add a column if it doesn't already exist."""
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text(f"ALTER TABLE {table} ADD COLUMN {column_sql}"))
|
||||
conn.commit()
|
||||
logger.info("Added %s column to %s", label, table)
|
||||
|
||||
|
||||
# -- per-table migrations --------------------------------------------------
|
||||
|
||||
def _migrate_story_items(engine, inspector, tables: set[str]) -> None:
|
||||
if "story_items" not in tables:
|
||||
return
|
||||
|
||||
columns = _get_columns(inspector, "story_items")
|
||||
|
||||
# Replace position-based ordering with absolute timecodes
|
||||
if "position" in columns:
|
||||
logger.info("Migrating story_items: removing position column, using start_time_ms")
|
||||
with engine.connect() as conn:
|
||||
if "start_time_ms" not in columns:
|
||||
conn.execute(text(
|
||||
"ALTER TABLE story_items ADD COLUMN start_time_ms INTEGER DEFAULT 0"
|
||||
))
|
||||
result = conn.execute(text("""
|
||||
SELECT si.id, si.story_id, si.position, g.duration
|
||||
FROM story_items si
|
||||
JOIN generations g ON si.generation_id = g.id
|
||||
ORDER BY si.story_id, si.position
|
||||
"""))
|
||||
current_story_id = None
|
||||
current_time_ms = 0
|
||||
for item_id, story_id, _position, duration in result.fetchall():
|
||||
if story_id != current_story_id:
|
||||
current_story_id = story_id
|
||||
current_time_ms = 0
|
||||
conn.execute(
|
||||
text("UPDATE story_items SET start_time_ms = :time WHERE id = :id"),
|
||||
{"time": current_time_ms, "id": item_id},
|
||||
)
|
||||
current_time_ms += int((duration or 0) * 1000) + 200
|
||||
conn.commit()
|
||||
|
||||
# Recreate table without the position column (SQLite lacks DROP COLUMN)
|
||||
conn.execute(text("""
|
||||
CREATE TABLE story_items_new (
|
||||
id VARCHAR PRIMARY KEY,
|
||||
story_id VARCHAR NOT NULL,
|
||||
generation_id VARCHAR NOT NULL,
|
||||
start_time_ms INTEGER NOT NULL DEFAULT 0,
|
||||
track INTEGER NOT NULL DEFAULT 0,
|
||||
trim_start_ms INTEGER NOT NULL DEFAULT 0,
|
||||
trim_end_ms INTEGER NOT NULL DEFAULT 0,
|
||||
version_id VARCHAR,
|
||||
created_at DATETIME,
|
||||
FOREIGN KEY (story_id) REFERENCES stories(id),
|
||||
FOREIGN KEY (generation_id) REFERENCES generations(id)
|
||||
)
|
||||
"""))
|
||||
conn.execute(text("""
|
||||
INSERT INTO story_items_new (id, story_id, generation_id, start_time_ms, track, trim_start_ms, trim_end_ms, version_id, created_at)
|
||||
SELECT id, story_id, generation_id, start_time_ms,
|
||||
COALESCE(track, 0), COALESCE(trim_start_ms, 0), COALESCE(trim_end_ms, 0), version_id, created_at
|
||||
FROM story_items
|
||||
"""))
|
||||
conn.execute(text("DROP TABLE story_items"))
|
||||
conn.execute(text("ALTER TABLE story_items_new RENAME TO story_items"))
|
||||
conn.commit()
|
||||
|
||||
# Re-read after table recreation
|
||||
columns = _get_columns(inspector, "story_items")
|
||||
|
||||
if "track" not in columns:
|
||||
_add_column(engine, "story_items", "track INTEGER NOT NULL DEFAULT 0", "track")
|
||||
# Re-read so subsequent checks see new columns
|
||||
columns = _get_columns(inspector, "story_items")
|
||||
if "trim_start_ms" not in columns:
|
||||
_add_column(engine, "story_items", "trim_start_ms INTEGER NOT NULL DEFAULT 0", "trim_start_ms")
|
||||
if "trim_end_ms" not in columns:
|
||||
_add_column(engine, "story_items", "trim_end_ms INTEGER NOT NULL DEFAULT 0", "trim_end_ms")
|
||||
if "version_id" not in columns:
|
||||
_add_column(engine, "story_items", "version_id VARCHAR", "version_id")
|
||||
|
||||
|
||||
def _migrate_profiles(engine, inspector, tables: set[str]) -> None:
|
||||
if "profiles" not in tables:
|
||||
return
|
||||
columns = _get_columns(inspector, "profiles")
|
||||
if "avatar_path" not in columns:
|
||||
_add_column(engine, "profiles", "avatar_path VARCHAR", "avatar_path")
|
||||
if "effects_chain" not in columns:
|
||||
_add_column(engine, "profiles", "effects_chain TEXT", "effects_chain")
|
||||
# Voice type system — v0.3.x
|
||||
if "voice_type" not in columns:
|
||||
_add_column(engine, "profiles", "voice_type VARCHAR DEFAULT 'cloned'", "voice_type")
|
||||
if "preset_engine" not in columns:
|
||||
_add_column(engine, "profiles", "preset_engine VARCHAR", "preset_engine")
|
||||
if "preset_voice_id" not in columns:
|
||||
_add_column(engine, "profiles", "preset_voice_id VARCHAR", "preset_voice_id")
|
||||
if "design_prompt" not in columns:
|
||||
_add_column(engine, "profiles", "design_prompt TEXT", "design_prompt")
|
||||
if "default_engine" not in columns:
|
||||
_add_column(engine, "profiles", "default_engine VARCHAR", "default_engine")
|
||||
|
||||
|
||||
def _migrate_generations(engine, inspector, tables: set[str]) -> None:
|
||||
if "generations" not in tables:
|
||||
return
|
||||
columns = _get_columns(inspector, "generations")
|
||||
if "status" not in columns:
|
||||
_add_column(engine, "generations", "status VARCHAR DEFAULT 'completed'", "status")
|
||||
if "error" not in columns:
|
||||
_add_column(engine, "generations", "error TEXT", "error")
|
||||
if "engine" not in columns:
|
||||
_add_column(engine, "generations", "engine VARCHAR DEFAULT 'qwen'", "engine")
|
||||
# Re-read after engine column (variable name shadows outer scope in old code)
|
||||
columns = _get_columns(inspector, "generations")
|
||||
if "model_size" not in columns:
|
||||
_add_column(engine, "generations", "model_size VARCHAR", "model_size")
|
||||
if "is_favorited" not in columns:
|
||||
_add_column(engine, "generations", "is_favorited BOOLEAN DEFAULT 0", "is_favorited")
|
||||
|
||||
|
||||
def _migrate_effect_presets(engine, inspector, tables: set[str]) -> None:
|
||||
if "effect_presets" not in tables:
|
||||
return
|
||||
columns = _get_columns(inspector, "effect_presets")
|
||||
if "sort_order" not in columns:
|
||||
_add_column(engine, "effect_presets", "sort_order INTEGER DEFAULT 100", "sort_order")
|
||||
|
||||
|
||||
def _migrate_generation_versions(engine, inspector, tables: set[str]) -> None:
|
||||
if "generation_versions" not in tables:
|
||||
return
|
||||
columns = _get_columns(inspector, "generation_versions")
|
||||
if "source_version_id" not in columns:
|
||||
_add_column(engine, "generation_versions", "source_version_id VARCHAR", "source_version_id")
|
||||
|
||||
|
||||
def _normalize_storage_paths(engine, tables: set[str]) -> None:
|
||||
"""Normalize stored file paths to be relative to the configured data dir."""
|
||||
from pathlib import Path
|
||||
|
||||
from ..config import get_data_dir, to_storage_path, resolve_storage_path
|
||||
|
||||
data_dir = get_data_dir()
|
||||
|
||||
path_columns = [
|
||||
("generations", "audio_path"),
|
||||
("generation_versions", "audio_path"),
|
||||
("profile_samples", "audio_path"),
|
||||
("profiles", "avatar_path"),
|
||||
]
|
||||
|
||||
total_fixed = 0
|
||||
with engine.connect() as conn:
|
||||
for table, column in path_columns:
|
||||
if table not in tables:
|
||||
continue
|
||||
rows = conn.execute(
|
||||
text(f"SELECT id, {column} FROM {table} WHERE {column} IS NOT NULL")
|
||||
).fetchall()
|
||||
for row_id, path_val in rows:
|
||||
if not path_val:
|
||||
continue
|
||||
p = Path(path_val)
|
||||
resolved = resolve_storage_path(p)
|
||||
if resolved is None:
|
||||
continue
|
||||
|
||||
normalized = to_storage_path(resolved)
|
||||
|
||||
if normalized != path_val:
|
||||
conn.execute(
|
||||
text(f"UPDATE {table} SET {column} = :path WHERE id = :id"),
|
||||
{"path": normalized, "id": row_id},
|
||||
)
|
||||
total_fixed += 1
|
||||
if total_fixed > 0:
|
||||
conn.commit()
|
||||
logger.info("Normalized %d stored file paths", total_fixed)
|
||||
@@ -0,0 +1,169 @@
|
||||
"""ORM model definitions for the voicebox SQLite database."""
|
||||
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import Column, String, Integer, Float, DateTime, Text, ForeignKey, Boolean
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
class VoiceProfile(Base):
|
||||
"""Voice profile.
|
||||
|
||||
voice_type discriminates three flavours:
|
||||
- "cloned" — traditional reference-audio profiles (all cloning engines)
|
||||
- "preset" — engine-specific pre-built voice (e.g. Kokoro voices)
|
||||
- "designed" — text-described voice (e.g. Qwen CustomVoice, future)
|
||||
"""
|
||||
|
||||
__tablename__ = "profiles"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text)
|
||||
language = Column(String, default="en")
|
||||
avatar_path = Column(String, nullable=True)
|
||||
effects_chain = Column(Text, nullable=True)
|
||||
|
||||
# Voice type system — added v0.3.x
|
||||
voice_type = Column(String, default="cloned") # "cloned" | "preset" | "designed"
|
||||
preset_engine = Column(String, nullable=True) # e.g. "kokoro" — only for preset
|
||||
preset_voice_id = Column(String, nullable=True) # e.g. "am_adam" — only for preset
|
||||
design_prompt = Column(Text, nullable=True) # text description — only for designed
|
||||
default_engine = Column(String, nullable=True) # auto-selected engine, locked for preset
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class ProfileSample(Base):
|
||||
"""Audio sample attached to a voice profile."""
|
||||
|
||||
__tablename__ = "profile_samples"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
|
||||
audio_path = Column(String, nullable=False)
|
||||
reference_text = Column(Text, nullable=False)
|
||||
|
||||
|
||||
class Generation(Base):
|
||||
"""A single TTS generation."""
|
||||
|
||||
__tablename__ = "generations"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
|
||||
text = Column(Text, nullable=False)
|
||||
language = Column(String, default="en")
|
||||
audio_path = Column(String, nullable=True)
|
||||
duration = Column(Float, nullable=True)
|
||||
seed = Column(Integer)
|
||||
instruct = Column(Text)
|
||||
engine = Column(String, default="qwen")
|
||||
model_size = Column(String, nullable=True)
|
||||
status = Column(String, default="completed")
|
||||
error = Column(Text, nullable=True)
|
||||
is_favorited = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class Story(Base):
|
||||
"""A story that sequences multiple generations."""
|
||||
|
||||
__tablename__ = "stories"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
description = Column(Text)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class StoryItem(Base):
|
||||
"""Links a generation to a story at a specific timecode."""
|
||||
|
||||
__tablename__ = "story_items"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
story_id = Column(String, ForeignKey("stories.id"), nullable=False)
|
||||
generation_id = Column(String, ForeignKey("generations.id"), nullable=False)
|
||||
version_id = Column(String, ForeignKey("generation_versions.id"), nullable=True)
|
||||
start_time_ms = Column(Integer, nullable=False, default=0)
|
||||
track = Column(Integer, nullable=False, default=0)
|
||||
trim_start_ms = Column(Integer, nullable=False, default=0)
|
||||
trim_end_ms = Column(Integer, nullable=False, default=0)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class Project(Base):
|
||||
"""Audio studio project (JSON blob)."""
|
||||
|
||||
__tablename__ = "projects"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
data = Column(Text)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class GenerationVersion(Base):
|
||||
"""A version of a generation's audio (original, processed, alternate takes)."""
|
||||
|
||||
__tablename__ = "generation_versions"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
generation_id = Column(String, ForeignKey("generations.id"), nullable=False)
|
||||
label = Column(String, nullable=False)
|
||||
audio_path = Column(String, nullable=False)
|
||||
effects_chain = Column(Text, nullable=True)
|
||||
source_version_id = Column(String, ForeignKey("generation_versions.id"), nullable=True)
|
||||
is_default = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class EffectPreset(Base):
|
||||
"""Saved effect chain preset."""
|
||||
|
||||
__tablename__ = "effect_presets"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, unique=True, nullable=False)
|
||||
description = Column(Text, nullable=True)
|
||||
effects_chain = Column(Text, nullable=False)
|
||||
is_builtin = Column(Boolean, default=False)
|
||||
sort_order = Column(Integer, default=100)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class AudioChannel(Base):
|
||||
"""Audio output channel (bus)."""
|
||||
|
||||
__tablename__ = "audio_channels"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
name = Column(String, nullable=False)
|
||||
is_default = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class ChannelDeviceMapping(Base):
|
||||
"""Mapping between a channel and an OS audio device."""
|
||||
|
||||
__tablename__ = "channel_device_mappings"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"), nullable=False)
|
||||
device_id = Column(String, nullable=False)
|
||||
|
||||
|
||||
class ProfileChannelMapping(Base):
|
||||
"""Many-to-many mapping between voice profiles and audio channels."""
|
||||
|
||||
__tablename__ = "profile_channel_mappings"
|
||||
|
||||
profile_id = Column(String, ForeignKey("profiles.id"), primary_key=True)
|
||||
channel_id = Column(String, ForeignKey("audio_channels.id"), primary_key=True)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Post-migration data seeding and backfills."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from .. import config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def backfill_generation_versions(SessionLocal, Generation, GenerationVersion) -> None:
|
||||
"""Create 'clean' version entries for generations that predate the versions feature."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
existing_version_gen_ids = {
|
||||
row[0] for row in db.query(GenerationVersion.generation_id).all()
|
||||
}
|
||||
generations = db.query(Generation).filter(
|
||||
Generation.status == "completed",
|
||||
Generation.audio_path.isnot(None),
|
||||
Generation.audio_path != "",
|
||||
).all()
|
||||
|
||||
count = 0
|
||||
for gen in generations:
|
||||
if gen.id in existing_version_gen_ids:
|
||||
continue
|
||||
resolved_audio_path = config.resolve_storage_path(gen.audio_path)
|
||||
if resolved_audio_path is None or not resolved_audio_path.exists():
|
||||
continue
|
||||
version = GenerationVersion(
|
||||
id=str(uuid.uuid4()),
|
||||
generation_id=gen.id,
|
||||
label="clean",
|
||||
audio_path=gen.audio_path,
|
||||
effects_chain=None,
|
||||
is_default=True,
|
||||
)
|
||||
db.add(version)
|
||||
count += 1
|
||||
|
||||
if count > 0:
|
||||
db.commit()
|
||||
logger.info("Backfilled %d generation version entries", count)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def seed_builtin_presets(SessionLocal, EffectPreset) -> None:
|
||||
"""Ensure built-in effect presets exist in the database."""
|
||||
from ..utils.effects import BUILTIN_PRESETS
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
for idx, (_key, preset_data) in enumerate(BUILTIN_PRESETS.items()):
|
||||
sort_order = preset_data.get("sort_order", idx)
|
||||
existing = db.query(EffectPreset).filter_by(name=preset_data["name"]).first()
|
||||
if not existing:
|
||||
preset = EffectPreset(
|
||||
id=str(uuid.uuid4()),
|
||||
name=preset_data["name"],
|
||||
description=preset_data.get("description"),
|
||||
effects_chain=json.dumps(preset_data["effects_chain"]),
|
||||
is_builtin=True,
|
||||
sort_order=sort_order,
|
||||
)
|
||||
db.add(preset)
|
||||
elif existing.sort_order != sort_order:
|
||||
existing.sort_order = sort_order
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Engine creation, initialization, and session management."""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from .. import config
|
||||
from .models import (
|
||||
Base,
|
||||
AudioChannel,
|
||||
EffectPreset,
|
||||
Generation,
|
||||
GenerationVersion,
|
||||
ProfileChannelMapping,
|
||||
VoiceProfile,
|
||||
)
|
||||
from .migrations import run_migrations
|
||||
from .seed import backfill_generation_versions, seed_builtin_presets
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Initialized by init_db()
|
||||
engine = None
|
||||
SessionLocal = None
|
||||
_db_path = None
|
||||
|
||||
|
||||
def init_db() -> None:
|
||||
"""Initialize the database engine, run migrations, create tables, and seed data."""
|
||||
global engine, SessionLocal, _db_path
|
||||
|
||||
_db_path = config.get_db_path()
|
||||
_db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
engine = create_engine(
|
||||
f"sqlite:///{_db_path}",
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
|
||||
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
||||
|
||||
run_migrations(engine)
|
||||
Base.metadata.create_all(bind=engine)
|
||||
|
||||
# Create default audio channel if it doesn't exist
|
||||
db = SessionLocal()
|
||||
try:
|
||||
default_channel = db.query(AudioChannel).filter(AudioChannel.is_default == True).first()
|
||||
if not default_channel:
|
||||
default_channel = AudioChannel(
|
||||
id=str(uuid.uuid4()),
|
||||
name="Default",
|
||||
is_default=True,
|
||||
)
|
||||
db.add(default_channel)
|
||||
|
||||
for profile in db.query(VoiceProfile).all():
|
||||
db.add(ProfileChannelMapping(
|
||||
profile_id=profile.id,
|
||||
channel_id=default_channel.id,
|
||||
))
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
backfill_generation_versions(SessionLocal, Generation, GenerationVersion)
|
||||
seed_builtin_presets(SessionLocal, EffectPreset)
|
||||
|
||||
|
||||
def get_db():
|
||||
"""Yield a database session (FastAPI dependency)."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
@@ -1,221 +0,0 @@
|
||||
"""
|
||||
Example usage of the voicebox backend API.
|
||||
|
||||
This script demonstrates how to:
|
||||
1. Create a voice profile
|
||||
2. Add samples to the profile
|
||||
3. Generate speech
|
||||
4. List history
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# API base URL
|
||||
BASE_URL = "http://localhost:8000"
|
||||
|
||||
|
||||
def check_health():
|
||||
"""Check if the server is running."""
|
||||
response = requests.get(f"{BASE_URL}/health")
|
||||
data = response.json()
|
||||
print(f"Server status: {data['status']}")
|
||||
print(f"Model loaded: {data['model_loaded']}")
|
||||
print(f"GPU available: {data['gpu_available']}")
|
||||
print()
|
||||
return data
|
||||
|
||||
|
||||
def create_profile(name: str, description: str = None, language: str = "en"):
|
||||
"""Create a new voice profile."""
|
||||
response = requests.post(
|
||||
f"{BASE_URL}/profiles",
|
||||
json={
|
||||
"name": name,
|
||||
"description": description,
|
||||
"language": language,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
profile = response.json()
|
||||
print(f"Created profile: {profile['name']} (ID: {profile['id']})")
|
||||
return profile
|
||||
|
||||
|
||||
def add_sample(profile_id: str, audio_file: str, reference_text: str):
|
||||
"""Add a sample to a voice profile."""
|
||||
with open(audio_file, "rb") as f:
|
||||
files = {"file": f}
|
||||
data = {"reference_text": reference_text}
|
||||
response = requests.post(
|
||||
f"{BASE_URL}/profiles/{profile_id}/samples",
|
||||
files=files,
|
||||
data=data,
|
||||
)
|
||||
response.raise_for_status()
|
||||
sample = response.json()
|
||||
print(f"Added sample: {sample['id']}")
|
||||
return sample
|
||||
|
||||
|
||||
def generate_speech(profile_id: str, text: str, language: str = "en", seed: int = None):
|
||||
"""Generate speech using a voice profile."""
|
||||
print(f"Generating speech: '{text[:50]}...'")
|
||||
start_time = time.time()
|
||||
|
||||
response = requests.post(
|
||||
f"{BASE_URL}/generate",
|
||||
json={
|
||||
"profile_id": profile_id,
|
||||
"text": text,
|
||||
"language": language,
|
||||
"seed": seed,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
generation = response.json()
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
print(f"Generated in {elapsed:.2f}s (duration: {generation['duration']:.2f}s)")
|
||||
print(f"Generation ID: {generation['id']}")
|
||||
return generation
|
||||
|
||||
|
||||
def download_audio(generation_id: str, output_file: str):
|
||||
"""Download generated audio."""
|
||||
response = requests.get(f"{BASE_URL}/audio/{generation_id}")
|
||||
response.raise_for_status()
|
||||
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(response.content)
|
||||
|
||||
print(f"Saved audio to: {output_file}")
|
||||
|
||||
|
||||
def list_profiles():
|
||||
"""List all voice profiles."""
|
||||
response = requests.get(f"{BASE_URL}/profiles")
|
||||
response.raise_for_status()
|
||||
profiles = response.json()
|
||||
|
||||
print(f"Found {len(profiles)} profiles:")
|
||||
for profile in profiles:
|
||||
print(f" - {profile['name']} (ID: {profile['id']})")
|
||||
|
||||
return profiles
|
||||
|
||||
|
||||
def list_history(profile_id: str = None, limit: int = 10):
|
||||
"""List generation history."""
|
||||
params = {"limit": limit}
|
||||
if profile_id:
|
||||
params["profile_id"] = profile_id
|
||||
|
||||
response = requests.get(f"{BASE_URL}/history", params=params)
|
||||
response.raise_for_status()
|
||||
history = response.json()
|
||||
|
||||
print(f"Found {len(history)} generations:")
|
||||
for gen in history:
|
||||
print(f" - {gen['text'][:50]}... ({gen['duration']:.2f}s)")
|
||||
|
||||
return history
|
||||
|
||||
|
||||
def transcribe_audio(audio_file: str, language: str = None):
|
||||
"""Transcribe audio file."""
|
||||
print(f"Transcribing: {audio_file}")
|
||||
|
||||
with open(audio_file, "rb") as f:
|
||||
files = {"file": f}
|
||||
data = {}
|
||||
if language:
|
||||
data["language"] = language
|
||||
|
||||
response = requests.post(
|
||||
f"{BASE_URL}/transcribe",
|
||||
files=files,
|
||||
data=data,
|
||||
)
|
||||
|
||||
response.raise_for_status()
|
||||
result = response.json()
|
||||
|
||||
print(f"Transcription: {result['text']}")
|
||||
print(f"Duration: {result['duration']:.2f}s")
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
"""Run example workflow."""
|
||||
print("=" * 60)
|
||||
print("voicebox Backend API Example")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# 1. Check health
|
||||
print("1. Checking server health...")
|
||||
check_health()
|
||||
|
||||
# 2. Create a profile
|
||||
print("2. Creating voice profile...")
|
||||
profile = create_profile(
|
||||
name="Example Voice",
|
||||
description="A test voice profile",
|
||||
language="en",
|
||||
)
|
||||
profile_id = profile["id"]
|
||||
print()
|
||||
|
||||
# 3. Add samples (you'll need actual audio files)
|
||||
print("3. Adding samples...")
|
||||
print(" (Skipping - add your own audio files here)")
|
||||
# Uncomment and add your audio file:
|
||||
# sample = add_sample(
|
||||
# profile_id,
|
||||
# "path/to/your/sample.wav",
|
||||
# "This is the transcript of the audio",
|
||||
# )
|
||||
print()
|
||||
|
||||
# 4. Generate speech (requires samples to be added first)
|
||||
print("4. Generating speech...")
|
||||
print(" (Skipping - add samples first)")
|
||||
# Uncomment after adding samples:
|
||||
# generation = generate_speech(
|
||||
# profile_id,
|
||||
# "Hello, this is a test of the voice cloning system.",
|
||||
# language="en",
|
||||
# seed=42,
|
||||
# )
|
||||
#
|
||||
# # 5. Download audio
|
||||
# print("\n5. Downloading audio...")
|
||||
# download_audio(generation["id"], "output.wav")
|
||||
print()
|
||||
|
||||
# 6. List profiles
|
||||
print("6. Listing all profiles...")
|
||||
list_profiles()
|
||||
print()
|
||||
|
||||
# 7. List history
|
||||
print("7. Listing generation history...")
|
||||
list_history(limit=5)
|
||||
print()
|
||||
|
||||
# 8. Transcribe audio (you'll need an audio file)
|
||||
print("8. Transcribing audio...")
|
||||
print(" (Skipping - add your own audio file here)")
|
||||
# Uncomment and add your audio file:
|
||||
# transcribe_audio("path/to/audio.wav", language="en")
|
||||
print()
|
||||
|
||||
print("=" * 60)
|
||||
print("Example complete!")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+7
-3139
File diff suppressed because it is too large
Load Diff
@@ -1,48 +0,0 @@
|
||||
"""
|
||||
Database migration script to add instruct column to generations table.
|
||||
|
||||
Run this once to update existing databases:
|
||||
python -m backend.migrate_add_instruct
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def migrate():
|
||||
"""Add instruct column to generations table if it doesn't exist."""
|
||||
# Get data directory
|
||||
data_dir = os.environ.get("VOICEBOX_DATA_DIR")
|
||||
if data_dir:
|
||||
db_path = Path(data_dir) / "voicebox.db"
|
||||
else:
|
||||
db_path = Path.cwd() / "data" / "voicebox.db"
|
||||
|
||||
if not db_path.exists():
|
||||
print(f"Database not found at {db_path}, skipping migration")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if instruct column already exists
|
||||
cursor.execute("PRAGMA table_info(generations)")
|
||||
columns = [row[1] for row in cursor.fetchall()]
|
||||
|
||||
if 'instruct' in columns:
|
||||
print("instruct column already exists, skipping migration")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
# Add instruct column
|
||||
print("Adding instruct column to generations table...")
|
||||
cursor.execute("ALTER TABLE generations ADD COLUMN instruct TEXT")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print("Migration complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user