Compare commits

..
Author SHA1 Message Date
Jamie Pine 49ebf6222e fix SSE cleanup, filter add popover to completed only, derive isGenerating from pending set, handle not_found status 2026-03-13 10:57:28 -07:00
Jamie Pine 509b0e71cc responsive layout fixes, version in sidebar, fixed voice card height, hide player title at small widths 2026-03-13 10:44:53 -07:00
Jamie Pine 81f8be1a94 defer story add until TTS completes, add generating pill to story editor, fix item placement per-track 2026-03-13 10:28:20 -07:00
Jamie Pine 655a60ca81 feat: async generation queue with serial execution
Generations now return immediately with a 'generating' status and appear
in history right away. TTS runs in a serial background queue to avoid
GPU contention. Users can kick off multiple generations without blocking.

- Async POST /generate creates DB record immediately, queues TTS work
- Serial generation queue prevents concurrent GPU access (Metal/CUDA/CPU)
- SSE endpoint GET /generate/{id}/status for real-time completion tracking
- Retry endpoint POST /generate/{id}/retry for failed generations
- Store engine and model_size on generation records for retry support
- History cards show animated loader (react-loaders) for generating/playing
- Failed generations show retry button instead of actions menu
- Model downloads happen inline in the queue instead of rejecting with 202
- Stale 'generating' records marked as failed on server startup
- Autoplay on generate setting (default: on)
- Show engine name on generation cards
- Remove sidebar generation spinner
- Checkbox alignment fix in settings
2026-03-13 10:02:41 -07:00
Jamie PineandGitHub 52285362ce Merge pull request #268 from jamiepine/feat/model-management-improvements
feat: model management improvements and folder migration
2026-03-13 09:16:43 -07:00
Jamie Pine 3ea587797f feat: model management improvements and folder migration
- Add model folder migration with byte-level progress tracking (backend + UI)
- Custom models directory support via VOICEBOX_MODELS_DIR env var passed to sidecar
- Hardcoded model descriptions displayed in model detail cards
- Open model folder button in storage location row
- Remove 'not downloaded' badge from model cards
- Fix server settings scroll offset for audio player
- Fix shell open permission to allow file paths
- Add normalize toggle to generation settings
2026-03-13 08:38:20 -07:00
Jamie PineandGitHub 325714bb83 Merge pull request #266 from jamiepine/feat/chunked-tts
feat: chunked TTS generation for long text (engine-agnostic)
2026-03-13 08:23:53 -07:00
James Pine 9aa7080c51 refactor: restructure server settings and models UI
- Split chunking/crossfade sliders into dedicated GenerationSettings card
- Merge connection status badges into ConnectionForm (remove ServerStatus card)
- 2-column grid layout for the entire settings page
- GPU Acceleration: remove icon, badge, and MLX info card
- Models: merge 'Other Voice Models' into single 'Voice Generation' list
- Model detail: remove 'Downloaded' badge, border above actions, swap
  badges above stats row, match disk size font to stats
2026-03-13 07:26:46 -07:00
James Pine 97292ecef7 feat: add chunk crossfade slider (0ms = hard cut)
Persisted setting (default 50ms) controls how audio chunks are blended
together.  Set to 0 for a clean hard cut with no overlap.
2026-03-13 06:48:06 -07:00
James Pine 837f8525d8 feat: add auto-chunking limit slider to settings
Persisted setting (default 800 chars) controls how long text is split
before generation.  Lower values improve quality for long outputs by
keeping each chunk well within the model's context window.

- Slider in Server Connection settings (100–2000 chars, step 50)
- Stored in localStorage via Zustand persist
- Passed as max_chunk_chars on every generation request
- Frontend text limit raised to 50,000 to match backend
2026-03-13 06:35:39 -07:00
James Pine 70ca7f66cb feat: chunked TTS generation for long text (engine-agnostic)
Text exceeding max_chunk_chars (default 800) is automatically split at
sentence boundaries, generated per-chunk, and concatenated with a 50ms
crossfade.  Works with all engines (Qwen, LuxTTS, Chatterbox, Turbo).

- Abbreviation-aware sentence splitter (Dr., Mr., e.g., decimals)
- CJK sentence-ending punctuation support
- Paralinguistic tag preservation ([laugh], [cough], etc.)
- Per-chunk seed variation to avoid correlated RNG artefacts
- Per-chunk Chatterbox trim (catches hallucination at each boundary)
- max_chunk_chars exposed as per-request param on GenerationRequest
- Text max_length raised to 50,000 characters

Closes #99
2026-03-13 06:21:34 -07:00
Jamie PineandGitHub c12b5d6f0a Merge pull request #265 from jamiepine/feat/paralinguistic-tags
feat: paralinguistic tag autocomplete for Chatterbox Turbo
2026-03-13 05:55:06 -07:00
James Pine 139fa38e3f fix: address review feedback for ParalinguisticInput
- Initialize lastSerializedRef to empty string so first-mount hydration
  always runs (fixes initial value not rendering)
- Guard arrow-key menu nav against empty filteredTags (avoids NaN index)
- Disable ARIA role/multiline and detach event handlers when disabled
- Add onBlur to close autocomplete dropdown when editor loses focus
- Chain exception with 'from e' in unload endpoint for better tracebacks
2026-03-13 05:52:06 -07:00
Jamie PineandGitHub 0e9f5db40f Merge pull request #264 from jamiepine/fix/chatterbox-float64-dtype
fix: Chatterbox float64 dtype mismatch + model unload button
2026-03-13 05:40:46 -07:00
James Pine 2f535a772f fix: load model into local var before patching to avoid half-initialised state
Apply local-var-then-assign pattern to chatterbox_backend.py (multilingual)
to match the turbo backend. Also use _current_model_size fallback in
unload, delete, and status endpoints for consistent Qwen model size checks.
2026-03-13 05:40:18 -07:00
James Pine b420637957 feat: paralinguistic tag autocomplete for Chatterbox Turbo
Type / in the text input when using Chatterbox Turbo to open an
autocomplete dropdown with 9 supported paralinguistic tags ([laugh],
[chuckle], [gasp], [cough], [sigh], [groan], [sniff], [shush],
[clear throat]).

- contentEditable div replaces textarea for Turbo engine only
- Tags render as inline styled badges
- Pasting text with [tag] patterns auto-converts to badges
- Badges serialize back to plain [tag] text for the API
- Dropdown portalled to body, opens above caret to avoid overflow
2026-03-13 05:19:23 -07:00
James Pine bfd7b815a5 fix: patch S3Tokenizer.log_mel_spectrogram for float64→float32 cast
The actual dtype mismatch was in S3Tokenizer.log_mel_spectrogram, not
VoiceEncoder.forward. librosa.load returns float64 numpy, which
torch.from_numpy preserves as double. The STFT output (double) then
hits _mel_filters (float32) in a matmul at s3tokenizer.py:163.

Now patching both entry points after model load:
1. S3Tokenizer.log_mel_spectrogram — cast audio to float32 before STFT
2. VoiceEncoder.forward — cast mels to float32 before LSTM

Remove debug traceback logging (no longer needed).
2026-03-13 05:04:29 -07:00
James Pine cac80f6af0 feat: add per-model unload endpoint and UI button
- POST /models/{model_name}/unload — unloads a specific model from
  memory without deleting from disk, supports all engine types
- Frontend: Unload button in model detail dialog when model is loaded
- Delete button remains disabled while loaded (unload first)
2026-03-13 04:50:56 -07:00
James Pine 47ce4cafdf fix: patch VoiceEncoder.forward to cast float64 mels to float32
The previous approach of patching librosa.load didn't work because
melspectrogram itself performs float64 math (numpy dot, signal.lfilter)
regardless of input dtype. The actual mismatch happens when pack()
creates a float64 tensor from the mel arrays and passes it into the
float32 LSTM weights in VoiceEncoder.forward().

Fix by monkey-patching VoiceEncoder.forward() to call mels.float()
before the LSTM, ensuring the input always matches the model dtype.
2026-03-13 04:41:43 -07:00
James Pine bfe912e41a fix: specify WAV format for atomic save temp file
soundfile cannot infer format from .tmp extension, causing all
generations to fail with 'No format specified and unable to get
format from file extension'
2026-03-13 04:34:26 -07:00
James Pine 5ccf79a8f7 Revert "fix: cast librosa float64 audio to float32 for Chatterbox voice encoder"
This reverts commit 1d32170c2e.
2026-03-13 04:28:00 -07:00
James Pine 1d32170c2e fix: cast librosa float64 audio to float32 for Chatterbox voice encoder
The upstream VoiceEncoder's melspectrogram only casts to float32 when
hp.normalized_mels is True (it defaults to False), so librosa's float64
output flows through as double tensors into float32 model weights,
causing 'expected m1 and m2 to have the same dtype, but got: float !=
double'. Fix by monkey-patching prepare_conditionals in both Chatterbox
and Chatterbox Turbo backends to ensure librosa.load returns float32.
2026-03-13 04:15:20 -07:00
James Pine ca74c155e2 fix: pass language parameter to Qwen TTS models and sync form with profile language
Both PyTorch and MLX backends silently dropped the language parameter —
it was accepted by generate() but never forwarded to the underlying
Qwen3-TTS model, causing it to default to auto-detection which
frequently confuses similar languages (e.g. Portuguese for Spanish).

- Add LANGUAGE_CODE_TO_NAME mapping (ISO 639-1 to full name) to both backends
- PyTorch: pass language= to generate_voice_clone()
- MLX: pass lang_code= to all 4 model.generate() call sites
- Frontend: auto-sync generation form language with selected voice profile

Closes #97
2026-03-13 04:04:04 -07:00
James Pine 1f770a157d fix: mismatched JSX closing tag in ModelManagement 2026-03-13 03:59:30 -07:00
Jamie PineandGitHub d64e24d422 Merge pull request #230 from haosenwang1018/docs/readme-grammar-profile-management
docs: fix minor README grammar in feature bullets
2026-03-13 03:56:55 -07:00
Jamie PineandGitHub 77d86ba835 Merge pull request #88 from Balneario-de-Cofrentes/fix/restrict-cors-origins
security: restrict CORS to known local origins
2026-03-13 03:56:15 -07:00
Jamie PineandGitHub 986a748420 Merge pull request #161 from ageofalgo/feat/docker-web-deployment
feat: add Docker + web deployment support
2026-03-13 03:55:04 -07:00
James Pine 50e01d17f8 fix: remove unused TTS_MODE env var from docker-compose
TTS_MODE is not read by any code in the backend — it only exists in
unimplemented planning docs. Remove it to avoid confusing users.
2026-03-13 03:53:15 -07:00
Jamie PineandGitHub 084c51b983 Merge pull request #215 from mikeswann/main
Update prerequisites in markdown with Tauri deps
2026-03-13 03:52:34 -07:00
Jamie PineandGitHub efbbbc7ec1 Merge branch 'main' into main 2026-03-13 03:52:22 -07:00
Jamie PineandGitHub 8e7f0cb9ad Merge pull request #133 from rayl15/feat/network-access-toggle
feat: add network access toggle to server settings
2026-03-13 03:47:35 -07:00
Jamie PineandGitHub 3357a06cba Merge pull request #263 from jamiepine/fix/atomic-save-error-handling
fix: atomic audio save with error handling and filesystem health endpoint
2026-03-13 03:45:26 -07:00
Jamie PineandGitHub f58c7c1cf3 Merge pull request #262 from jamiepine/feat/linux-rocm-whisper-turbo
feat: Linux support, AMD ROCm, Whisper Turbo, and spawn fix
2026-03-13 03:44:36 -07:00
James Pine ea41213123 fix: atomic audio save with errno-specific error handling and filesystem health endpoint
- save_audio() now writes to .tmp then os.replace() for atomic writes
- /generate endpoint catches OSError with specific messages for ENOENT, EACCES, ENOSPC, and BrokenPipeError
- New /health/filesystem endpoint checks directory existence, write permissions, and disk space
- New DirectoryCheck and FilesystemHealthResponse models

Cherry-picked and expanded from #178 (@Vaibhavee89)
2026-03-13 03:43:42 -07:00
James Pine b5801891b8 feat: Linux support, AMD ROCm, Whisper Turbo, and spawn fix
Cherry-picked and adapted from PR #89 and #214:

- Linux audio capture via PulseAudio/PipeWire monitor sources (cpal)
- AMD ROCm GPU support: HSA_OVERRIDE_GFX_VERSION env var, ROCm detection
- Whisper Turbo model (openai/whisper-large-v3-turbo) in all endpoints
- Cleaner Whisper language handling via generate_kwargs
- tauri::async_runtime::spawn fix to prevent panic on app shutdown
- Enable Linux (ubuntu-22.04) in release CI matrix
2026-03-13 03:35:18 -07:00
Jamie PineandGitHub 8f77c041f5 Merge pull request #152 from mpecanha/fix-offline-mode-crash
Fix: Prevent crashes when HuggingFace is unreachable
2026-03-13 03:31:23 -07:00
James Pine 5a3f3ba030 Merge remote-tracking branch 'origin/main' into feat/docker-web-deployment 2026-03-13 03:21:39 -07:00
Jamie PineandGitHub 3c25ee6e2c Merge pull request #243 from ways2read/a11y/screen-reader-and-keyboard-improvements
a11y: screen reader and keyboard improvements
2026-03-13 03:18:42 -07:00
James Pine b92b0dd508 merge: resolve conflicts with latest main 2026-03-13 03:16:56 -07:00
Jamie PineandGitHub 670900bf5a Merge pull request #258 from jamiepine/feat/chatterbox-turbo
feat: Chatterbox Turbo engine + per-engine language lists
2026-03-13 03:14:44 -07:00
James Pine 219cfb1605 docs: update PROJECT_STATUS.md to reflect multi-engine architecture
- Reflects merged PRs: #254 (LuxTTS/multi-engine), #257 (Chatterbox), #252 (CUDA swap), #238 (download UI)
- Updated architecture diagram to show all 4 TTS engines
- Added TTS engine comparison table and multi-engine architecture section
- Marked resolved bottlenecks (singleton backend, frontend Qwen assumptions)
- Updated PR triage: marked #194 and #33 as superseded
- Added 'Adding a New Engine' guide (now ~1 day effort)
- Updated recommended priorities to reflect current state
- Added new API endpoints (CUDA, cancel, active tasks)
2026-03-13 02:39:10 -07:00
James Pine bf728a780c feat: add Chatterbox Turbo engine and per-engine language lists
- New ChatterboxTurboTTSBackend wrapping ChatterboxTurboTTS (ResembleAI/chatterbox-turbo)
- English-only 350M model with paralinguistic tag support ([laugh], [cough], [chuckle])
- Bypasses upstream token=True bug by calling snapshot_download(token=None) + from_local()
- Same CPU-on-macOS forcing and torch.load monkey-patching as multilingual backend
- Full engine integration: generate, stream, model status/download/delete endpoints
- Language dropdown now shows only languages supported by the selected engine
- Per-engine language maps: Qwen (10), LuxTTS (en), Chatterbox (23), Turbo (en)
- Auto-switches to English when selecting English-only engines
- Backend language regex expanded to accept all 23 Chatterbox languages
2026-03-13 02:35:10 -07:00
Jamie PineandGitHub 3e6513c0fb Merge pull request #257 from jamiepine/feat/chatterbox
feat: Chatterbox TTS engine with multilingual voice cloning
2026-03-13 02:12:56 -07:00
James Pine c54ee14173 fix: model loaded icon uses accent-colored CircleCheck, show size for loaded models, fix generate box overlapping player on stories route 2026-03-13 02:09:32 -07:00
James Pine cc07d4d3c9 fix: download progress tracking for all engines and inline progress UI
- Add HFProgressTracker to LuxTTS and Chatterbox backends so tqdm-based
  file-level download progress reaches the frontend (previously only Qwen
  had this, LuxTTS/Chatterbox showed a static spinner)
- Add progress/current/total/filename fields to ActiveDownloadTask so the
  /tasks/active polling endpoint carries progress data
- Show inline progress bar + bytes in the model list and detail modal,
  poll at 1s during active downloads (5s otherwise)
- Fix GpuAcceleration crash: cudaStatusLoading was referenced before
  initialization in its own useQuery declaration
2026-03-13 02:09:32 -07:00
James Pine 9beb9d7fec fix: install chatterbox-tts with --no-deps to avoid numpy pin conflict
chatterbox-tts 0.1.6 pins numpy<1.26 and torch==2.6 which are
incompatible with Python 3.12+. Install with --no-deps and list
its sub-dependencies explicitly in requirements.txt.

Also removes HFProgressTracker from chatterbox backend to avoid
'generator didn't stop after throw()' errors from tqdm patching.
2026-03-13 02:09:32 -07:00
James Pine 76bb207b2b feat: add Chatterbox TTS engine for multilingual voice cloning
- New ChatterboxTTSBackend wrapping ChatterboxMultilingualTTS (ResembleAI/chatterbox)
- Supports 23 languages including Hebrew, forces CPU on macOS (MPS issue)
- Monkey-patches torch.load for CPU loading, forces eager attention for compatibility
- trim_tts_output utility cuts trailing silence/hallucination from Chatterbox output
- Full engine integration: /generate, /generate/stream, model status/download/delete
- Hebrew (he) added to supported languages in frontend and backend validation
- Single flat model dropdown extended with Chatterbox option in both generation UIs
- ModelManagement UI groups LuxTTS and Chatterbox under 'Other Voice Models' section
2026-03-13 02:09:32 -07:00
Jamie PineandGitHub 3576521d62 Merge pull request #254 from jamiepine/feat/luxtts
feat: LuxTTS integration — multi-engine TTS support
2026-03-13 02:04:46 -07:00
Jamie PineandGitHub 2df4ece388 Merge pull request #210 from ieguiguren/fix/linux-nvidia-gbm-buffer
fix: Linux NVIDIA GBM buffer crash + WebKitGTK microphone access
2026-03-13 01:55:58 -07:00
Jamie PineandGitHub cbb4979ed6 Merge pull request #175 from Vaibhavee89/fix/profile-duplicate-name-validation
Fix #134: Add validation for duplicate profile names
2026-03-13 01:55:30 -07:00
Jamie PineandGitHub 573f82a7e6 Merge pull request #250 from pandego/fix/docs-align-local-port-17493
docs: align local API port examples with current dev flow
2026-03-13 01:53:28 -07:00
pandego 3d2506767d docs: address review nits for API generator 2026-03-13 05:08:25 +01:00
pandego cdef2163c1 docs: align local API port examples with current dev flow 2026-03-12 12:17:50 +01:00
Richard Orme 9955e1dcb7 a11y: address PR feedback and polish docs
- HistoryTable: skip row key handler when focus is on Actions button (Enter/Space)
- StoryList: expose selected story (aria-pressed, 'Selected' in label)
- ProfileCard: skip card key handler when focus is on Export/Edit/Delete
- VoicesTab: keep table semantics; edit button in first cell instead of role=button on row
- PR-ACCESSIBILITY.md: 'Fine-tune' wording, 'focus on the text area' phrasing

Made-with: Cursor
2026-03-07 12:36:02 -08:00
Richard Orme 19a28bf6c5 a11y: screen reader and keyboard improvements
- Audio player: aria-labels for Play/Pause, Loop, Mute, Close; labelled playback and volume sliders
- Generation: aria-labels for Generate speech and Fine tune instructions buttons
- Voice cards: focusable, labelled, Enter/Space to select
- History rows: focusable, labelled, Enter/Space to play; transcript textarea labelled
- Voices tab: focusable rows, labelled, Enter/Space to edit; Actions button labelled
- Model management: focusable model rows and labelled Download/Delete buttons
- Server tab: regions with aria-label and tabIndex for Connection, Status, App Updates
- Stories: focusable story rows, labelled, Enter/Space to select; Actions and track editor buttons labelled
- Voice profile samples: Play/Pause/Stop and mini-player slider labelled

Tested with NVDA and Narrator on Windows. See docs/PR-ACCESSIBILITY.md for full description.

Made-with: Cursor
2026-03-07 12:02:33 -08:00
OpenClaw Bot 3f10a70d4c docs: fix minor grammar in feature bullets 2026-03-04 04:39:28 +00:00
mikeswannGitHubcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
d0dfe78701 Update CONTRIBUTING.md
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-02-28 10:36:34 +01:00
mikeswannandGitHub 172addd918 Update README.md 2026-02-28 00:29:23 +01:00
mikeswannandGitHub ada309cfb9 Update CONTRIBUTING.md 2026-02-28 00:28:01 +01:00
IvanandClaude Opus 4.6 30ee07c2e3 fix: scope DMABUF workaround to Linux+NVIDIA, add origin validation
Address CodeRabbit review feedback:
- Makefile: only set WEBKIT_DISABLE_DMABUF_RENDERER=1 when running on
  Linux with an NVIDIA GPU detected via lspci
- main.rs: validate webview origin before auto-granting microphone
  permission — only allow for trusted local origins (tauri://, localhost,
  127.0.0.1)

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-27 07:01:32 +01:00
IvanandClaude Opus 4.6 d21c63b52c fix: enable microphone access on Linux via WebKitGTK
WebKitGTK denies getUserMedia by default. This adds webkit2gtk as a
Linux dependency and configures the webview to enable media streams
and auto-grant UserMediaPermissionRequest for microphone access.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-27 06:49:08 +01:00
IvanandClaude Opus 4.6 5ad67d7ecb fix: disable DMABUF renderer for NVIDIA GPUs on Linux
WebKitGTK fails to create GBM buffers with NVIDIA proprietary drivers,
resulting in an empty/blank Tauri window. Set WEBKIT_DISABLE_DMABUF_RENDERER=1
in the dev target to work around this.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-27 06:32:02 +01:00
Vaibhavee Singh 6cc96c2614 Fix #134: Add validation for duplicate profile names
- Add validation in create_profile() to check for existing names before insert
- Add validation in update_profile() to prevent renaming to duplicate names
- Improve error handling in API endpoints with user-friendly messages
- Add comprehensive test suite for duplicate name validation
- Update CHANGELOG.md with fix details

This fix prevents database constraint violations and provides clear
error messages when users attempt to create or update profiles with
names that already exist in the database.
2026-02-24 10:17:39 +05:30
Claudio Casale edfc6e99fe feat: add Docker + web deployment support 2026-02-23 12:52:03 +01:00
Makinde d00e28ffda Fix: Prevent crashes when HuggingFace is unreachable
Implements offline mode patch for API stability issues:

- Add hf_offline_patch.py to monkey-patch huggingface_hub
- Force cache-only lookups before mlx_audio imports
- Create symlink from original Qwen repo to MLX community version
  when only MLX version is cached

This fixes:
- Issue #150: Internet required even with cached models
- Issue #151: API crashes when HF network fails

The patch ensures that if models are locally cached, no network
requests are made to HuggingFace during speech generation.
2026-02-22 01:57:02 -08:00
Rahul Sharma 28a4fd4824 feat: add network access toggle to server settings
Exposes the existing remote server mode through a checkbox in Server
Connection settings. When enabled, the server binds to 0.0.0.0 instead
of 127.0.0.1, making it accessible from other devices on the network.

The plumbing already existed (Rust sidecar passes --host 0.0.0.0 when
remote=true, serverStore has mode state, Python backend accepts --host),
but the UI hardcoded startServer(false). This wires it up.

Closes #104
2026-02-21 00:39:39 +05:30
David Gil 80c87c8e2c test: add CORS origin restriction tests
20 tests covering:
- All 6 default local origins are allowed
- Arbitrary external origins are blocked
- Preflight (OPTIONS) requests respect the allowlist
- VOICEBOX_CORS_ORIGINS env var extends the allowlist
- Edge cases: empty env, whitespace trimming, trailing commas

Tests use a minimal FastAPI app mirroring the real CORS config,
so they run without ML dependencies (torch, numpy, etc.).
2026-02-17 22:04:25 +01:00
David Gil 427d811954 security: restrict CORS to known local origins instead of wildcard
The wildcard `allow_origins=["*"]` allows any website the user visits to
make requests to the local voicebox backend, potentially triggering TTS
generation or reading voice profiles without consent.

Restrict to the known Tauri webview and Vite dev server origins by
default. Users running in remote server mode can set
VOICEBOX_CORS_ORIGINS to allow additional origins.
2026-02-17 21:58:08 +01:00
86 changed files with 6469 additions and 1037 deletions
+46
View File
@@ -0,0 +1,46 @@
# Version control
.git
.github
.gitignore
# Desktop-only (not needed in web container)
tauri/
landing/
docs/
mlx-test/
scripts/
# Dependencies & build artifacts (rebuilt in Docker)
node_modules/
__pycache__/
*.pyc
*.pyo
*.egg-info/
dist/
build/
*.spec
# Data (will be bind-mounted)
data/
backend/data/
# IDE & OS
.vscode/
.idea/
*.swp
*.swo
.DS_Store
Thumbs.db
# Config files not needed in container
biome.json
.biomeignore
.bumpversion.cfg
.npmrc
Makefile
CHANGELOG.md
CONTRIBUTING.md
SECURITY.md
LICENSE
README.md
backend/README.md
+4 -4
View File
@@ -22,10 +22,10 @@ jobs:
args: "--target x86_64-apple-darwin" args: "--target x86_64-apple-darwin"
python-version: "3.12" python-version: "3.12"
backend: "pytorch" backend: "pytorch"
# - platform: 'ubuntu-22.04' - platform: "ubuntu-22.04"
# args: '' args: ""
# python-version: '3.12' python-version: "3.12"
# backend: 'pytorch' backend: "pytorch"
- platform: "windows-latest" - platform: "windows-latest"
args: "" args: ""
python-version: "3.12" python-version: "3.12"
+9
View File
@@ -5,6 +5,14 @@ All notable changes to Voicebox will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed
- **Profile Name Validation** - Added proper validation to prevent duplicate profile names ([#134](https://github.com/jamiepine/voicebox/issues/134))
- Users now receive clear error messages when attempting to create or update profiles with duplicate names
- Improved error handling in create and update profile API endpoints
- Added comprehensive test suite for duplicate name validation
## [0.1.0] - 2026-01-25 ## [0.1.0] - 2026-01-25
### Added ### Added
@@ -55,6 +63,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ### Fixed
- Audio export failing when Tauri save dialog returns object instead of string path - Audio export failing when Tauri save dialog returns object instead of string path
- OpenAPI client generator script now documents the local backend port and avoids an unused loop variable warning
### Added ### Added
- **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks - **Makefile** - Comprehensive development workflow automation with commands for setup, development, building, testing, and code quality checks
+2 -1
View File
@@ -27,6 +27,7 @@ Thank you for your interest in contributing to Voicebox! This document provides
```bash ```bash
rustc --version # Check if installed rustc --version # Check if installed
``` ```
- **[Tauri Prerequisites](https://v2.tauri.app/start/prerequisites)** - Tauri-specific system dependencies (varies by OS).
- **Git** - Version control - **Git** - Version control
@@ -426,7 +427,7 @@ See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and sol
- **Backend won't start:** Check Python version (3.11+), ensure venv is activated, install dependencies - **Backend won't start:** Check Python version (3.11+), ensure venv is activated, install dependencies
- **Tauri build fails:** Ensure Rust is installed, clean build with `cd tauri/src-tauri && cargo clean` - **Tauri build fails:** Ensure Rust is installed, clean build with `cd tauri/src-tauri && cargo clean`
- **OpenAPI client generation fails:** Ensure backend is running, check `curl http://localhost:8000/openapi.json` - **OpenAPI client generation fails:** Ensure backend is running, check `curl http://localhost:17493/openapi.json`
## Questions? ## Questions?
+79
View File
@@ -0,0 +1,79 @@
# ============================================================
# Voicebox — Local TTS Server with Web UI (CPU)
# 3-stage build: Frontend → Python deps → Runtime
# ============================================================
# === Stage 1: Build frontend ===
FROM oven/bun:1 AS frontend
WORKDIR /build
# Copy workspace config and frontend source
COPY package.json bun.lock ./
COPY app/ ./app/
COPY web/ ./web/
# Strip workspaces not needed for web build, and fix trailing comma
RUN sed -i '/"tauri"/d; /"landing"/d' package.json && \
sed -i -z 's/,\n ]/\n ]/' package.json
RUN bun install --no-save
# Build frontend (skip tsc — upstream has pre-existing type errors)
RUN cd web && bunx --bun vite build
# === Stage 2: Build Python dependencies ===
FROM python:3.11-slim AS backend-builder
WORKDIR /build
RUN apt-get update && apt-get install -y --no-install-recommends \
git \
build-essential \
&& rm -rf /var/lib/apt/lists/*
COPY backend/requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
RUN pip install --no-cache-dir --prefix=/install \
git+https://github.com/QwenLM/Qwen3-TTS.git
# === Stage 3: Runtime ===
FROM python:3.11-slim
# Create non-root user for security
RUN groupadd -r voicebox && \
useradd -r -g voicebox -m -s /bin/bash voicebox
WORKDIR /app
# Install only runtime system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ffmpeg \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy installed Python packages from builder stage
COPY --from=backend-builder /install /usr/local
# Copy backend application code
COPY --chown=voicebox:voicebox backend/ /app/backend/
# Copy built frontend from frontend stage
COPY --from=frontend --chown=voicebox:voicebox /build/web/dist /app/frontend/
# Create data directories owned by non-root user
RUN mkdir -p /app/data/generations /app/data/profiles /app/data/cache \
&& chown -R voicebox:voicebox /app/data
# Switch to non-root user
USER voicebox
# Expose the API port
EXPOSE 17493
# Health check — auto-restart if the server hangs
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=60s \
CMD curl -f http://localhost:17493/health || exit 1
# Start the FastAPI server
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "17493"]
+6 -1
View File
@@ -48,6 +48,7 @@ setup-python: $(VENV)/bin/activate ## Set up Python virtual environment and depe
@echo -e "$(BLUE)Installing Python dependencies...$(NC)" @echo -e "$(BLUE)Installing Python dependencies...$(NC)"
$(PIP) install --upgrade pip $(PIP) install --upgrade pip
$(PIP) install -r $(BACKEND_DIR)/requirements.txt $(PIP) install -r $(BACKEND_DIR)/requirements.txt
$(PIP) install --no-deps chatterbox-tts
@if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \ @if [ "$$(uname -m)" = "arm64" ] && [ "$$(uname)" = "Darwin" ]; then \
echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \ echo -e "$(BLUE)Detected Apple Silicon - installing MLX dependencies...$(NC)"; \
$(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \ $(PIP) install -r $(BACKEND_DIR)/requirements-mlx.txt; \
@@ -79,7 +80,11 @@ dev: ## Start backend + desktop app (parallel)
@echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)" @echo -e "$(YELLOW)Note: If Tauri fails, run 'make build-server' first or use separate terminals$(NC)"
@trap 'kill 0' EXIT; \ @trap 'kill 0' EXIT; \
$(MAKE) dev-backend & \ $(MAKE) dev-backend & \
sleep 2 && $(MAKE) dev-frontend & \ sleep 2 && if [ "$$(uname)" = "Linux" ] && lspci 2>/dev/null | grep -qi nvidia; then \
WEBKIT_DISABLE_DMABUF_RENDERER=1 $(MAKE) dev-frontend; \
else \
$(MAKE) dev-frontend; \
fi & \
wait wait
dev-backend: ## Start FastAPI backend server dev-backend: ## Start FastAPI backend server
+58
View File
@@ -0,0 +1,58 @@
# Voicebox Offline Mode Fix
## Problem
Voicebox crashes when generating speech if HuggingFace is unreachable, even when models are fully cached locally.
**Root Cause:**
- Voicebox downloads `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` (MLX optimized version)
- But `mlx_audio.tts.load()` tries to fetch `config.json` from original repo `Qwen/Qwen3-TTS-12Hz-1.7B-Base`
- This network request fails → server crashes with `RemoteDisconnected`
**Related Issues:**
- Issue #150: "Internet connection required, even though models are downloaded?"
- Issue #151: "API Stability Issues: Model Loading Hangs and Server Crashes"
## Solution
Two-part fix:
### 1. Monkey-patch huggingface_hub (`backend/utils/hf_offline_patch.py`)
- Intercepts cache lookup functions
- Forces offline mode early (before mlx_audio imports)
- Adds debug logging for cache hits/misses
### 2. Symlink original repo to MLX version (`ensure_original_qwen_config_cached()`)
- When original `Qwen/Qwen3-TTS-12Hz-1.7B-Base` cache doesn't exist
- But MLX `mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16` does exist
- Creates a symlink so cache lookups succeed
## Files Changed
- `backend/backends/mlx_backend.py` - Added patch imports at top
- `backend/utils/hf_offline_patch.py` - New patch module
## Testing
To test this fix:
1. Build Voicebox from source: `make build`
2. Disconnect from internet
3. Try generating speech
4. Should work without network requests
## Build Instructions
```bash
# Install dependencies
pip install -r requirements.txt
# Build the app
make build
# Or build just the server
make build-server
```
## Notes
- The patch is applied automatically when `mlx_backend.py` is imported
- Set `VOICEBOX_OFFLINE_PATCH=0` to disable the patch
- The symlink approach works because the config.json is compatible between versions
---
*Patch contributed by community*
+10 -7
View File
@@ -98,12 +98,12 @@ Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-p
- **Instant cloning** — Upload a sample, get a voice profile - **Instant cloning** — Upload a sample, get a voice profile
- **High fidelity** — Natural prosody, emotion, and cadence - **High fidelity** — Natural prosody, emotion, and cadence
- **Multi-language** — English, Chinese, and more coming - **Multi-language** — English, Chinese, and more coming
- **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super fast generation - **Lightning fast on Mac** — MLX backend leverages Apple Silicon's Neural Engine for super-fast generation
### Voice Profile Management ### Voice Profile Management
- **Create profiles** from audio files or record directly in-app - **Create profiles** from audio files or record directly in-app
- **Import/Export** profiles to share or backup - **Import/Export** profiles to share or back up
- **Multi-sample support** — combine multiple samples for higher quality cloning - **Multi-sample support** — combine multiple samples for higher quality cloning
- **Organize** with descriptions and language tags - **Organize** with descriptions and language tags
@@ -147,17 +147,20 @@ Create multi-voice narratives, podcasts, and conversations with a timeline-based
Voicebox exposes a full REST API, so you can integrate voice synthesis into your own apps. Voicebox exposes a full REST API, so you can integrate voice synthesis into your own apps.
For the current local app and development workflow, the backend is typically available at `http://localhost:17493`.
If you launch the backend manually with a different host or port, use that address instead.
```bash ```bash
# Generate speech # Generate speech
curl -X POST http://localhost:8000/generate \ curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}' -d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}'
# List voice profiles # List voice profiles
curl http://localhost:8000/profiles curl http://localhost:17493/profiles
# Create a profile # Create a profile
curl -X POST http://localhost:8000/profiles \ curl -X POST http://localhost:17493/profiles \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"name": "My Voice", "language": "en"}' -d '{"name": "My Voice", "language": "en"}'
``` ```
@@ -170,7 +173,7 @@ curl -X POST http://localhost:8000/profiles \
- Voice assistants - Voice assistants
- Content creation automation - Content creation automation
Full API documentation available at `http://localhost:8000/docs` when running. Full API documentation is available at `http://localhost:17493/docs` in the default local workflow, or at `/docs` on whatever server address you configured.
--- ---
@@ -239,7 +242,7 @@ Install [just](https://github.com/casey/just): `brew install just` or `cargo ins
Also available via Makefile: `make setup && make dev` (run `make help` for all commands). Also available via Makefile: `make setup && make dev` (run `make help` for all commands).
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org), [XCode on macOS](https://developer.apple.com/xcode/). **Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org), [XCode on macOS](https://developer.apple.com/xcode/), [Tauri Prerequisites](https://v2.tauri.app/start/prerequisites/).
**Performance:** **Performance:**
- **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration for 4-5x faster inference - **Apple Silicon (M1/M2/M3)**: Uses MLX backend with native Metal acceleration for 4-5x faster inference
+4 -2
View File
@@ -93,10 +93,12 @@ function App() {
} }
serverStartingRef.current = true; serverStartingRef.current = true;
console.log('Production mode: Starting bundled server...'); const isRemote = useServerStore.getState().mode === 'remote';
const customModelsDir = useServerStore.getState().customModelsDir;
console.log(`Production mode: Starting bundled server... (remote: ${isRemote})`);
platform.lifecycle platform.lifecycle
.startServer(false) .startServer(isRemote, customModelsDir)
.then((serverUrl) => { .then((serverUrl) => {
console.log('Server is ready at:', serverUrl); console.log('Server is ready at:', serverUrl);
// Update the server URL in the store with the dynamically assigned port // Update the server URL in the store with the dynamically assigned port
+26 -6
View File
@@ -1,17 +1,18 @@
import { useQuery } from '@tanstack/react-query'; import { useQuery } from '@tanstack/react-query';
import { Pause, Play, Repeat, Volume2, VolumeX, X } from 'lucide-react'; import { Pause, Play, Repeat, Volume2, VolumeX, X } from 'lucide-react';
import { useEffect, useMemo, useRef, useState } from 'react'; import { useEffect, useId, useMemo, useRef, useState } from 'react';
import WaveSurfer from 'wavesurfer.js'; import WaveSurfer from 'wavesurfer.js';
import { Button } from '@/components/ui/button'; import { Button } from '@/components/ui/button';
import { Slider } from '@/components/ui/slider'; import { Slider } from '@/components/ui/slider';
import { apiClient } from '@/lib/api/client'; import { apiClient } from '@/lib/api/client';
import { formatAudioDuration } from '@/lib/utils/audio'; import { formatAudioDuration } from '@/lib/utils/audio';
import { debug } from '@/lib/utils/debug'; import { debug } from '@/lib/utils/debug';
import { usePlayerStore } from '@/stores/playerStore';
import { usePlatform } from '@/platform/PlatformContext'; import { usePlatform } from '@/platform/PlatformContext';
import { usePlayerStore } from '@/stores/playerStore';
export function AudioPlayer() { export function AudioPlayer() {
const platform = usePlatform(); const platform = usePlatform();
const volumeLabelId = useId();
const { const {
audioUrl, audioUrl,
audioId, audioId,
@@ -359,7 +360,7 @@ export function AudioPlayer() {
if (shouldAutoPlayNow) { if (shouldAutoPlayNow) {
// Clear the flag first // Clear the flag first
usePlayerStore.getState().clearAutoPlayFlag(); usePlayerStore.getState().clearAutoPlayFlag();
// Use a small delay to ensure audio element is fully ready // Use a small delay to ensure audio element is fully ready
setTimeout(() => { setTimeout(() => {
wavesurfer.play().catch((error) => { wavesurfer.play().catch((error) => {
@@ -664,7 +665,7 @@ export function AudioPlayer() {
// Handle shouldAutoPlay flag - for story mode auto-advance // Handle shouldAutoPlay flag - for story mode auto-advance
const shouldAutoPlay = usePlayerStore((state) => state.shouldAutoPlay); const shouldAutoPlay = usePlayerStore((state) => state.shouldAutoPlay);
const clearAutoPlayFlag = usePlayerStore((state) => state.clearAutoPlayFlag); const clearAutoPlayFlag = usePlayerStore((state) => state.clearAutoPlayFlag);
useEffect(() => { useEffect(() => {
const wavesurfer = wavesurferRef.current; const wavesurfer = wavesurferRef.current;
if (!wavesurfer || !shouldAutoPlay || duration === 0) { if (!wavesurfer || !shouldAutoPlay || duration === 0) {
@@ -831,6 +832,9 @@ export function AudioPlayer() {
disabled={isLoading || duration === 0} disabled={isLoading || duration === 0}
className="shrink-0" className="shrink-0"
title={duration === 0 && !isLoading ? 'Audio not loaded' : ''} title={duration === 0 && !isLoading ? 'Audio not loaded' : ''}
aria-label={
duration === 0 && !isLoading ? 'Audio not loaded' : isPlaying ? 'Pause' : 'Play'
}
> >
{isPlaying ? <Pause className="h-5 w-5" /> : <Play className="h-5 w-5" />} {isPlaying ? <Pause className="h-5 w-5" /> : <Play className="h-5 w-5" />}
</Button> </Button>
@@ -845,6 +849,8 @@ export function AudioPlayer() {
max={100} max={100}
step={0.1} step={0.1}
className="w-full" className="w-full"
aria-label="Playback position"
aria-valuetext={`${formatAudioDuration(currentTime)} of ${formatAudioDuration(duration)}`}
/> />
)} )}
{isLoading && ( {isLoading && (
@@ -862,7 +868,9 @@ export function AudioPlayer() {
{/* Title */} {/* Title */}
{title && ( {title && (
<div className="text-sm font-medium truncate max-w-[200px] shrink-0">{title}</div> <div className="text-sm font-medium truncate max-w-[200px] shrink-0 hidden lg:block">
{title}
</div>
)} )}
{/* Loop Button */} {/* Loop Button */}
@@ -872,26 +880,37 @@ export function AudioPlayer() {
onClick={toggleLoop} onClick={toggleLoop}
className={isLooping ? 'text-primary' : ''} className={isLooping ? 'text-primary' : ''}
title="Toggle loop" title="Toggle loop"
aria-label={isLooping ? 'Stop looping' : 'Loop'}
> >
<Repeat className="h-4 w-4" /> <Repeat className="h-4 w-4" />
</Button> </Button>
{/* Volume Control */} {/* Volume Control */}
<div className="flex items-center gap-2 shrink-0 w-[120px]"> <div
className="flex items-center gap-2 shrink-0 w-[120px]"
role="group"
aria-label="Volume"
>
<Button <Button
variant="ghost" variant="ghost"
size="icon" size="icon"
onClick={() => setVolume(volume > 0 ? 0 : 1)} onClick={() => setVolume(volume > 0 ? 0 : 1)}
className="h-8 w-8" className="h-8 w-8"
aria-label={volume > 0 ? 'Mute' : 'Unmute'}
> >
{volume > 0 ? <Volume2 className="h-4 w-4" /> : <VolumeX className="h-4 w-4" />} {volume > 0 ? <Volume2 className="h-4 w-4" /> : <VolumeX className="h-4 w-4" />}
</Button> </Button>
<span id={volumeLabelId} className="sr-only">
Volume level, {Math.round(volume * 100)}%
</span>
<Slider <Slider
value={[volume * 100]} value={[volume * 100]}
onValueChange={handleVolumeChange} onValueChange={handleVolumeChange}
max={100} max={100}
step={1} step={1}
className="flex-1" className="flex-1"
aria-labelledby={volumeLabelId}
aria-valuetext={`${Math.round(volume * 100)}%`}
/> />
</div> </div>
@@ -902,6 +921,7 @@ export function AudioPlayer() {
onClick={handleClose} onClick={handleClose}
className="shrink-0" className="shrink-0"
title="Close player" title="Close player"
aria-label="Close player"
> >
<X className="h-5 w-5" /> <X className="h-5 w-5" />
</Button> </Button>
+6 -4
View File
@@ -23,8 +23,8 @@ import {
import { apiClient } from '@/lib/api/client'; import { apiClient } from '@/lib/api/client';
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui'; import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
import { usePlayerStore } from '@/stores/playerStore';
import { usePlatform } from '@/platform/PlatformContext'; import { usePlatform } from '@/platform/PlatformContext';
import { usePlayerStore } from '@/stores/playerStore';
interface AudioDevice { interface AudioDevice {
id: string; id: string;
@@ -129,7 +129,7 @@ export function AudioTab() {
if (await confirm('Delete this channel?')) { if (await confirm('Delete this channel?')) {
deleteChannel.mutate(channelId); deleteChannel.mutate(channelId);
} }
} };
const allChannels = channels || []; const allChannels = channels || [];
const allDevices = devices || []; const allDevices = devices || [];
@@ -168,7 +168,7 @@ export function AudioTab() {
</Button> </Button>
</div> </div>
) : ( ) : (
<div className="space-y-3 p-2"> <div className="space-y-3">
{allChannels.map((channel) => { {allChannels.map((channel) => {
const isSelected = selectedChannelId === channel.id; const isSelected = selectedChannelId === channel.id;
return ( return (
@@ -343,7 +343,9 @@ export function AudioTab() {
<div className="flex flex-col items-center justify-center py-12 border-2 border-dashed border-muted rounded-md"> <div className="flex flex-col items-center justify-center py-12 border-2 border-dashed border-muted rounded-md">
<CheckCircle2 className="h-12 w-12 text-muted-foreground mb-4" /> <CheckCircle2 className="h-12 w-12 text-muted-foreground mb-4" />
<p className="text-muted-foreground text-center"> <p className="text-muted-foreground text-center">
{platform.metadata.isTauri ? 'No audio devices found' : 'Audio device selection requires Tauri'} {platform.metadata.isTauri
? 'No audio devices found'
: 'Audio device selection requires Tauri'}
</p> </p>
</div> </div>
)} )}
@@ -12,14 +12,15 @@ import {
SelectValue, SelectValue,
} from '@/components/ui/select'; } from '@/components/ui/select';
import { Textarea } from '@/components/ui/textarea'; import { Textarea } from '@/components/ui/textarea';
import { useToast } from '@/components/ui/use-toast'; import { getLanguageOptionsForEngine, type LanguageCode } from '@/lib/constants/languages';
import { LANGUAGE_OPTIONS } from '@/lib/constants/languages';
import { useGenerationForm } from '@/lib/hooks/useGenerationForm'; import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
import { useProfile, useProfiles } from '@/lib/hooks/useProfiles'; import { useProfile, useProfiles } from '@/lib/hooks/useProfiles';
import { useAddStoryItem, useStory } from '@/lib/hooks/useStories'; import { useStory } from '@/lib/hooks/useStories';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
import { useGenerationStore } from '@/stores/generationStore';
import { useStoryStore } from '@/stores/storyStore'; import { useStoryStore } from '@/stores/storyStore';
import { useUIStore } from '@/stores/uiStore'; import { useUIStore } from '@/stores/uiStore';
import { ParalinguisticInput } from './ParalinguisticInput';
interface FloatingGenerateBoxProps { interface FloatingGenerateBoxProps {
isPlayerOpen?: boolean; isPlayerOpen?: boolean;
@@ -43,8 +44,7 @@ export function FloatingGenerateBox({
const selectedStoryId = useStoryStore((state) => state.selectedStoryId); const selectedStoryId = useStoryStore((state) => state.selectedStoryId);
const trackEditorHeight = useStoryStore((state) => state.trackEditorHeight); const trackEditorHeight = useStoryStore((state) => state.trackEditorHeight);
const { data: currentStory } = useStory(selectedStoryId); const { data: currentStory } = useStory(selectedStoryId);
const addStoryItem = useAddStoryItem(); const addPendingStoryAdd = useGenerationStore((s) => s.addPendingStoryAdd);
const { toast } = useToast();
// Calculate if track editor is visible (on stories route with items) // Calculate if track editor is visible (on stories route with items)
const hasTrackEditor = isStoriesRoute && currentStory && currentStory.items.length > 0; const hasTrackEditor = isStoriesRoute && currentStory && currentStory.items.length > 0;
@@ -52,25 +52,9 @@ export function FloatingGenerateBox({
const { form, handleSubmit, isPending } = useGenerationForm({ const { form, handleSubmit, isPending } = useGenerationForm({
onSuccess: async (generationId) => { onSuccess: async (generationId) => {
setIsExpanded(false); setIsExpanded(false);
// If on stories route and a story is selected, add generation to story // Defer the story add until TTS completes — useGenerationProgress handles it
if (isStoriesRoute && selectedStoryId && generationId) { if (isStoriesRoute && selectedStoryId && generationId) {
try { addPendingStoryAdd(generationId, selectedStoryId);
await addStoryItem.mutateAsync({
storyId: selectedStoryId,
data: { generation_id: generationId },
});
toast({
title: 'Added to story',
description: `Generation added to "${currentStory?.name || 'story'}"`,
});
} catch (error) {
toast({
title: 'Failed to add to story',
description:
error instanceof Error ? error.message : 'Could not add generation to story',
variant: 'destructive',
});
}
} }
}, },
}); });
@@ -112,6 +96,13 @@ export function FloatingGenerateBox({
} }
}, [selectedProfileId, profiles, setSelectedProfileId]); }, [selectedProfileId, profiles, setSelectedProfileId]);
// Sync generation form language with selected profile's language
useEffect(() => {
if (selectedProfile?.language) {
form.setValue('language', selectedProfile.language as LanguageCode);
}
}, [selectedProfile, form]);
// Auto-resize textarea based on content (only when expanded) // Auto-resize textarea based on content (only when expanded)
useEffect(() => { useEffect(() => {
if (!isExpanded) { if (!isExpanded) {
@@ -174,7 +165,7 @@ export function FloatingGenerateBox({
isStoriesRoute isStoriesRoute
? // Position aligned with story list: after sidebar + padding, width 360px ? // Position aligned with story list: after sidebar + padding, width 360px
'left-[calc(5rem+2rem)] w-[360px]' 'left-[calc(5rem+2rem)] w-[360px]'
: 'left-[calc(5rem+2rem)] w-[calc((100%-5rem-4rem)/2-1rem)]', : 'left-[calc(5rem+2rem)] right-8 lg:right-auto lg:w-[calc((100%-5rem-4rem)/2-1rem)]',
)} )}
style={{ style={{
// On stories route: offset by track editor height when visible // On stories route: offset by track editor height when visible
@@ -212,34 +203,57 @@ export function FloatingGenerateBox({
transition={{ duration: 0.15, ease: 'easeOut' }} transition={{ duration: 0.15, ease: 'easeOut' }}
style={{ overflow: 'hidden' }} style={{ overflow: 'hidden' }}
> >
<Textarea {form.watch('engine') === 'chatterbox_turbo' ? (
{...field} <ParalinguisticInput
ref={(node: HTMLTextAreaElement | null) => { value={field.value}
// Store ref for auto-resize (only for active field) onChange={field.onChange}
if (!isInstructMode) { placeholder={
textareaRef.current = node; isStoriesRoute && currentStory
? `Generate speech for "${currentStory.name}"... (type / for effects)`
: selectedProfile
? `Type / for effects like [laugh], [sigh]...`
: 'Select a voice profile above...'
} }
// Forward ref to react-hook-form className="px-3 py-2 resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm w-full"
if (typeof field.ref === 'function') { style={{
field.ref(node); minHeight: isExpanded ? '100px' : '32px',
maxHeight: '300px',
overflowY: 'auto',
}}
disabled={!selectedProfileId}
onClick={() => setIsExpanded(true)}
onFocus={() => setIsExpanded(true)}
/>
) : (
<Textarea
{...field}
ref={(node: HTMLTextAreaElement | null) => {
// Store ref for auto-resize (only for active field)
if (!isInstructMode) {
textareaRef.current = node;
}
// Forward ref to react-hook-form
if (typeof field.ref === 'function') {
field.ref(node);
}
}}
placeholder={
isStoriesRoute && currentStory
? `Generate speech for "${currentStory.name}"...`
: selectedProfile
? `Generate speech using ${selectedProfile.name}...`
: 'Select a voice profile above...'
} }
}} className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full"
placeholder={ style={{
isStoriesRoute && currentStory minHeight: isExpanded ? '100px' : '32px',
? `Generate speech for "${currentStory.name}"...` maxHeight: '300px',
: selectedProfile }}
? `Generate speech using ${selectedProfile.name}...` disabled={!selectedProfileId}
: 'Select a voice profile above...' onClick={() => setIsExpanded(true)}
} onFocus={() => setIsExpanded(true)}
className="resize-none bg-transparent border-none focus-visible:ring-0 focus-visible:ring-offset-0 focus:outline-none focus:ring-0 outline-none ring-0 rounded-2xl text-sm placeholder:text-muted-foreground/60 w-full" />
style={{ )}
minHeight: isExpanded ? '100px' : '32px',
maxHeight: '300px',
}}
disabled={!selectedProfileId}
onClick={() => setIsExpanded(true)}
onFocus={() => setIsExpanded(true)}
/>
</motion.div> </motion.div>
</FormControl> </FormControl>
<FormMessage className="text-xs" /> <FormMessage className="text-xs" />
@@ -300,6 +314,13 @@ export function FloatingGenerateBox({
disabled={isPending || !selectedProfileId} disabled={isPending || !selectedProfileId}
className="h-10 w-10 rounded-full bg-accent hover:bg-accent/90 hover:scale-105 text-accent-foreground shadow-lg hover:shadow-accent/50 transition-all duration-200" className="h-10 w-10 rounded-full bg-accent hover:bg-accent/90 hover:scale-105 text-accent-foreground shadow-lg hover:shadow-accent/50 transition-all duration-200"
size="icon" size="icon"
aria-label={
isPending
? 'Generating...'
: !selectedProfileId
? 'Select a voice profile first'
: 'Generate speech'
}
> >
{isPending ? ( {isPending ? (
<Loader2 className="h-4 w-4 animate-spin" /> <Loader2 className="h-4 w-4 animate-spin" />
@@ -316,7 +337,7 @@ export function FloatingGenerateBox({
</span> </span>
</div> </div>
<AnimatePresence> <AnimatePresence>
{isExpanded && form.watch('engine') !== 'luxtts' && ( {isExpanded && form.watch('engine') === 'qwen' && (
<motion.div <motion.div
initial={{ opacity: 0, scale: 0.8 }} initial={{ opacity: 0, scale: 0.8 }}
animate={{ opacity: 1, scale: 1 }} animate={{ opacity: 1, scale: 1 }}
@@ -336,6 +357,9 @@ export function FloatingGenerateBox({
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90' ? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
: 'bg-card border border-border hover:bg-background/50', : 'bg-card border border-border hover:bg-background/50',
)} )}
aria-label={
isInstructMode ? 'Fine tune instructions, on' : 'Fine tune instructions'
}
> >
<SlidersHorizontal className="h-4 w-4" /> <SlidersHorizontal className="h-4 w-4" />
</Button> </Button>
@@ -381,25 +405,30 @@ export function FloatingGenerateBox({
<FormField <FormField
control={form.control} control={form.control}
name="language" name="language"
render={({ field }) => ( render={({ field }) => {
<FormItem className="flex-1 space-y-0"> const engineLangs = getLanguageOptionsForEngine(
<Select onValueChange={field.onChange} defaultValue={field.value}> form.watch('engine') || 'qwen',
<FormControl> );
<SelectTrigger className="h-8 text-xs bg-card border-border rounded-full hover:bg-background/50 transition-all"> return (
<SelectValue /> <FormItem className="flex-1 space-y-0">
</SelectTrigger> <Select onValueChange={field.onChange} value={field.value}>
</FormControl> <FormControl>
<SelectContent> <SelectTrigger className="h-8 text-xs bg-card border-border rounded-full hover:bg-background/50 transition-all">
{LANGUAGE_OPTIONS.map((lang) => ( <SelectValue />
<SelectItem key={lang.value} value={lang.value} className="text-xs"> </SelectTrigger>
{lang.label} </FormControl>
</SelectItem> <SelectContent>
))} {engineLangs.map((lang) => (
</SelectContent> <SelectItem key={lang.value} value={lang.value} className="text-xs">
</Select> {lang.label}
<FormMessage className="text-xs" /> </SelectItem>
</FormItem> ))}
)} </SelectContent>
</Select>
<FormMessage className="text-xs" />
</FormItem>
);
}}
/> />
<FormItem className="flex-1 space-y-0"> <FormItem className="flex-1 space-y-0">
@@ -407,11 +436,21 @@ export function FloatingGenerateBox({
value={ value={
form.watch('engine') === 'luxtts' form.watch('engine') === 'luxtts'
? 'luxtts' ? 'luxtts'
: `qwen:${form.watch('modelSize') || '1.7B'}` : form.watch('engine') === 'chatterbox'
? 'chatterbox'
: form.watch('engine') === 'chatterbox_turbo'
? 'chatterbox_turbo'
: `qwen:${form.watch('modelSize') || '1.7B'}`
} }
onValueChange={(value) => { onValueChange={(value) => {
if (value === 'luxtts') { if (value === 'luxtts') {
form.setValue('engine', 'luxtts'); form.setValue('engine', 'luxtts');
form.setValue('language', 'en');
} else if (value === 'chatterbox') {
form.setValue('engine', 'chatterbox');
} else if (value === 'chatterbox_turbo') {
form.setValue('engine', 'chatterbox_turbo');
form.setValue('language', 'en');
} else { } else {
const [, modelSize] = value.split(':'); const [, modelSize] = value.split(':');
form.setValue('engine', 'qwen'); form.setValue('engine', 'qwen');
@@ -434,6 +473,15 @@ export function FloatingGenerateBox({
<SelectItem value="luxtts" className="text-xs text-muted-foreground"> <SelectItem value="luxtts" className="text-xs text-muted-foreground">
LuxTTS LuxTTS
</SelectItem> </SelectItem>
<SelectItem value="chatterbox" className="text-xs text-muted-foreground">
Chatterbox
</SelectItem>
<SelectItem
value="chatterbox_turbo"
className="text-xs text-muted-foreground"
>
Chatterbox Turbo
</SelectItem>
</SelectContent> </SelectContent>
</Select> </Select>
</FormItem> </FormItem>
@@ -19,10 +19,11 @@ import {
SelectValue, SelectValue,
} from '@/components/ui/select'; } from '@/components/ui/select';
import { Textarea } from '@/components/ui/textarea'; import { Textarea } from '@/components/ui/textarea';
import { LANGUAGE_OPTIONS } from '@/lib/constants/languages'; import { getLanguageOptionsForEngine } from '@/lib/constants/languages';
import { useGenerationForm } from '@/lib/hooks/useGenerationForm'; import { useGenerationForm } from '@/lib/hooks/useGenerationForm';
import { useProfile } from '@/lib/hooks/useProfiles'; import { useProfile } from '@/lib/hooks/useProfiles';
import { useUIStore } from '@/stores/uiStore'; import { useUIStore } from '@/stores/uiStore';
import { ParalinguisticInput } from './ParalinguisticInput';
export function GenerationForm() { export function GenerationForm() {
const selectedProfileId = useUIStore((state) => state.selectedProfileId); const selectedProfileId = useUIStore((state) => state.selectedProfileId);
@@ -64,19 +65,32 @@ export function GenerationForm() {
<FormItem> <FormItem>
<FormLabel>Text to Speak</FormLabel> <FormLabel>Text to Speak</FormLabel>
<FormControl> <FormControl>
<Textarea {form.watch('engine') === 'chatterbox_turbo' ? (
placeholder="Enter the text you want to generate..." <ParalinguisticInput
className="min-h-[150px]" value={field.value}
{...field} onChange={field.onChange}
/> placeholder="Enter text... type / for effects like [laugh], [sigh]"
className="min-h-[150px] rounded-md border border-input bg-background px-3 py-2"
/>
) : (
<Textarea
placeholder="Enter the text you want to generate..."
className="min-h-[150px]"
{...field}
/>
)}
</FormControl> </FormControl>
<FormDescription>Max 5000 characters</FormDescription> <FormDescription>
{form.watch('engine') === 'chatterbox_turbo'
? 'Max 5000 characters. Type / to insert sound effects.'
: 'Max 5000 characters'}
</FormDescription>
<FormMessage /> <FormMessage />
</FormItem> </FormItem>
)} )}
/> />
{form.watch('engine') !== 'luxtts' && ( {form.watch('engine') === 'qwen' && (
<FormField <FormField
control={form.control} control={form.control}
name="instruct" name="instruct"
@@ -107,11 +121,21 @@ export function GenerationForm() {
value={ value={
form.watch('engine') === 'luxtts' form.watch('engine') === 'luxtts'
? 'luxtts' ? 'luxtts'
: `qwen:${form.watch('modelSize') || '1.7B'}` : form.watch('engine') === 'chatterbox'
? 'chatterbox'
: form.watch('engine') === 'chatterbox_turbo'
? 'chatterbox_turbo'
: `qwen:${form.watch('modelSize') || '1.7B'}`
} }
onValueChange={(value) => { onValueChange={(value) => {
if (value === 'luxtts') { if (value === 'luxtts') {
form.setValue('engine', 'luxtts'); form.setValue('engine', 'luxtts');
form.setValue('language', 'en');
} else if (value === 'chatterbox') {
form.setValue('engine', 'chatterbox');
} else if (value === 'chatterbox_turbo') {
form.setValue('engine', 'chatterbox_turbo');
form.setValue('language', 'en');
} else { } else {
const [, modelSize] = value.split(':'); const [, modelSize] = value.split(':');
form.setValue('engine', 'qwen'); form.setValue('engine', 'qwen');
@@ -128,38 +152,47 @@ export function GenerationForm() {
<SelectItem value="qwen:1.7B">Qwen3-TTS 1.7B</SelectItem> <SelectItem value="qwen:1.7B">Qwen3-TTS 1.7B</SelectItem>
<SelectItem value="qwen:0.6B">Qwen3-TTS 0.6B</SelectItem> <SelectItem value="qwen:0.6B">Qwen3-TTS 0.6B</SelectItem>
<SelectItem value="luxtts">LuxTTS</SelectItem> <SelectItem value="luxtts">LuxTTS</SelectItem>
<SelectItem value="chatterbox">Chatterbox</SelectItem>
<SelectItem value="chatterbox_turbo">Chatterbox Turbo</SelectItem>
</SelectContent> </SelectContent>
</Select> </Select>
<FormDescription> <FormDescription>
{form.watch('engine') === 'luxtts' {form.watch('engine') === 'luxtts'
? 'Fast, English-focused' ? 'Fast, English-focused'
: 'Multi-language, two sizes'} : form.watch('engine') === 'chatterbox'
? '23 languages, incl. Hebrew'
: form.watch('engine') === 'chatterbox_turbo'
? 'English, [laugh] [cough] tags'
: 'Multi-language, two sizes'}
</FormDescription> </FormDescription>
</FormItem> </FormItem>
<FormField <FormField
control={form.control} control={form.control}
name="language" name="language"
render={({ field }) => ( render={({ field }) => {
<FormItem> const engineLangs = getLanguageOptionsForEngine(form.watch('engine') || 'qwen');
<FormLabel>Language</FormLabel> return (
<Select onValueChange={field.onChange} defaultValue={field.value}> <FormItem>
<FormControl> <FormLabel>Language</FormLabel>
<SelectTrigger> <Select onValueChange={field.onChange} value={field.value}>
<SelectValue /> <FormControl>
</SelectTrigger> <SelectTrigger>
</FormControl> <SelectValue />
<SelectContent> </SelectTrigger>
{LANGUAGE_OPTIONS.map((lang) => ( </FormControl>
<SelectItem key={lang.value} value={lang.value}> <SelectContent>
{lang.label} {engineLangs.map((lang) => (
</SelectItem> <SelectItem key={lang.value} value={lang.value}>
))} {lang.label}
</SelectContent> </SelectItem>
</Select> ))}
<FormMessage /> </SelectContent>
</FormItem> </Select>
)} <FormMessage />
</FormItem>
);
}}
/> />
<FormField <FormField
@@ -0,0 +1,422 @@
/**
* ParalinguisticInput — a contentEditable rich text input that renders
* Chatterbox Turbo paralinguistic tags (e.g. [laugh]) as inline badges.
*
* Trigger: typing "/" opens an autocomplete dropdown.
* Paste: pasting text with [tag] patterns auto-converts to badges.
* Output: serializes badges back to plain [tag] text for the API.
*/
import { AnimatePresence, motion } from 'framer-motion';
import { forwardRef, useCallback, useEffect, useImperativeHandle, useRef, useState } from 'react';
import { createPortal } from 'react-dom';
import { cn } from '@/lib/utils/cn';
// ── Tag definitions ─────────────────────────────────────────────────
const PARALINGUISTIC_TAGS = [
{ tag: '[laugh]', label: 'laugh', emoji: '\u{1F602}' },
{ tag: '[chuckle]', label: 'chuckle', emoji: '\u{1F60F}' },
{ tag: '[gasp]', label: 'gasp', emoji: '\u{1F62E}' },
{ tag: '[cough]', label: 'cough', emoji: '\u{1F637}' },
{ tag: '[sigh]', label: 'sigh', emoji: '\u{1F614}' },
{ tag: '[groan]', label: 'groan', emoji: '\u{1F629}' },
{ tag: '[sniff]', label: 'sniff', emoji: '\u{1F443}' },
{ tag: '[shush]', label: 'shush', emoji: '\u{1F92B}' },
{ tag: '[clear throat]', label: 'clear throat', emoji: '\u{1F64A}' },
] as const;
const TAG_REGEX = /\[(laugh|chuckle|gasp|cough|sigh|groan|sniff|shush|clear throat)\]/gi;
// Data attribute used to identify badge spans in the DOM
const BADGE_ATTR = 'data-ptag';
// ── Helpers ─────────────────────────────────────────────────────────
/** Build an inline badge <span> for a tag. */
function makeBadgeHTML(tag: string): string {
const entry = PARALINGUISTIC_TAGS.find((t) => t.tag.toLowerCase() === tag.toLowerCase());
const label = entry?.label ?? tag.replace(/[[\]]/g, '');
const emoji = entry?.emoji ?? '';
// Non-editable inline badge. Zero-width spaces around it let the
// caret sit on either side so the user can type before/after.
return `\u200B<span ${BADGE_ATTR}="${tag}" contenteditable="false" class="ptag-badge">${emoji ? `${emoji}\u00A0` : ''}${label}</span>\u200B`;
}
/** Convert plain text with [tag] patterns into HTML with badge spans. */
function textToHTML(text: string): string {
// Escape HTML entities first
const escaped = text.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
// Replace tag patterns with badge HTML
return escaped.replace(TAG_REGEX, (match) => makeBadgeHTML(match));
}
/** Serialize the contentEditable innerHTML back to plain text with [tag] syntax. */
function htmlToText(container: HTMLElement): string {
let result = '';
for (const node of container.childNodes) {
if (node.nodeType === Node.TEXT_NODE) {
// Strip zero-width spaces we added around badges
result += (node.textContent ?? '').replace(/\u200B/g, '');
} else if (node.nodeType === Node.ELEMENT_NODE) {
const el = node as HTMLElement;
if (el.hasAttribute(BADGE_ATTR)) {
result += el.getAttribute(BADGE_ATTR) ?? '';
} else if (el.tagName === 'BR') {
result += '\n';
} else {
// Recurse for nested elements (e.g. spans from paste)
result += htmlToText(el);
}
}
}
return result;
}
/** Get the text content from the current caret position back to the last
* whitespace or start of container, to detect the "/" trigger. */
function getWordBeforeCaret(_container: HTMLElement): { word: string; range: Range | null } {
const sel = window.getSelection();
if (!sel || sel.rangeCount === 0) return { word: '', range: null };
const range = sel.getRangeAt(0).cloneRange();
range.collapse(true);
// Walk backwards from caret through the text node
const textNode = range.startContainer;
if (textNode.nodeType !== Node.TEXT_NODE) return { word: '', range: null };
const text = textNode.textContent ?? '';
const offset = range.startOffset;
let start = offset;
while (
start > 0 &&
text[start - 1] !== ' ' &&
text[start - 1] !== '\n' &&
text[start - 1] !== '\u00A0'
) {
start--;
}
const word = text.slice(start, offset);
const wordRange = document.createRange();
wordRange.setStart(textNode, start);
wordRange.setEnd(textNode, offset);
return { word, range: wordRange };
}
// ── Component ───────────────────────────────────────────────────────
export interface ParalinguisticInputProps {
value?: string;
onChange?: (value: string) => void;
placeholder?: string;
disabled?: boolean;
className?: string;
style?: React.CSSProperties;
onClick?: () => void;
onFocus?: () => void;
}
export interface ParalinguisticInputRef {
focus: () => void;
element: HTMLDivElement | null;
}
export const ParalinguisticInput = forwardRef<ParalinguisticInputRef, ParalinguisticInputProps>(
function ParalinguisticInput(
{ value, onChange, placeholder, disabled, className, style, onClick, onFocus },
ref,
) {
const editorRef = useRef<HTMLDivElement>(null);
const [showMenu, setShowMenu] = useState(false);
const [menuFilter, setMenuFilter] = useState('');
const [menuIndex, setMenuIndex] = useState(0);
const [menuPosition, setMenuPosition] = useState<{ bottom: number; left: number }>({
bottom: 0,
left: 0,
});
const triggerRangeRef = useRef<Range | null>(null);
const lastSerializedRef = useRef<string>('');
const isComposingRef = useRef(false);
useImperativeHandle(ref, () => ({
focus: () => editorRef.current?.focus(),
element: editorRef.current,
}));
// Filtered tag list for the autocomplete menu
const filteredTags = PARALINGUISTIC_TAGS.filter((t) =>
t.label.toLowerCase().includes(menuFilter.toLowerCase()),
);
// ── Sync external value → editor ──────────────────────────────
useEffect(() => {
const el = editorRef.current;
if (!el) return;
// Only update DOM if the external value differs from what we last emitted
if (value !== undefined && value !== lastSerializedRef.current) {
lastSerializedRef.current = value;
el.innerHTML = value ? textToHTML(value) : '';
}
}, [value]);
// ── Emit plain-text value on input ────────────────────────────
const emitChange = useCallback(() => {
const el = editorRef.current;
if (!el || !onChange) return;
const text = htmlToText(el);
lastSerializedRef.current = text;
onChange(text);
}, [onChange]);
// ── Insert a tag badge at the caret ───────────────────────────
const insertTag = useCallback(
(tag: string) => {
const el = editorRef.current;
if (!el) return;
// Delete the /filter text
const wordRange = triggerRangeRef.current;
if (wordRange) {
wordRange.deleteContents();
}
// Insert badge HTML
const temp = document.createElement('span');
temp.innerHTML = makeBadgeHTML(tag);
const frag = document.createDocumentFragment();
let lastNode: Node | null = null;
while (temp.firstChild) {
lastNode = frag.appendChild(temp.firstChild);
}
const sel = window.getSelection();
if (sel && sel.rangeCount > 0) {
const range = sel.getRangeAt(0);
range.deleteContents();
range.insertNode(frag);
// Move caret after the badge
if (lastNode) {
const newRange = document.createRange();
newRange.setStartAfter(lastNode);
newRange.collapse(true);
sel.removeAllRanges();
sel.addRange(newRange);
}
}
setShowMenu(false);
setMenuFilter('');
emitChange();
el.focus();
},
[emitChange],
);
// ── Handle keydown for autocomplete navigation ────────────────
const handleKeyDown = useCallback(
(e: React.KeyboardEvent) => {
if (showMenu) {
if (filteredTags.length === 0) {
if (e.key === 'Escape') {
e.preventDefault();
setShowMenu(false);
}
return;
}
if (e.key === 'ArrowDown') {
e.preventDefault();
setMenuIndex((i) => (i + 1) % filteredTags.length);
} else if (e.key === 'ArrowUp') {
e.preventDefault();
setMenuIndex((i) => (i - 1 + filteredTags.length) % filteredTags.length);
} else if (e.key === 'Enter' || e.key === 'Tab') {
e.preventDefault();
if (filteredTags[menuIndex]) {
insertTag(filteredTags[menuIndex].tag);
}
} else if (e.key === 'Escape') {
e.preventDefault();
setShowMenu(false);
}
} else {
// Prevent Enter from creating <div> blocks in contentEditable
if (e.key === 'Enter' && !e.shiftKey) {
// Let the form handle submit
}
}
},
[showMenu, filteredTags, menuIndex, insertTag],
);
// ── Handle input (check for / trigger) ────────────────────────
const handleInput = useCallback(() => {
if (isComposingRef.current) return;
const el = editorRef.current;
if (!el) return;
const { word, range } = getWordBeforeCaret(el);
if (word.startsWith('/')) {
const filter = word.slice(1); // strip the /
setMenuFilter(filter);
setMenuIndex(0);
triggerRangeRef.current = range;
// Position the menu above the caret using viewport coords (portalled)
const sel = window.getSelection();
if (sel && sel.rangeCount > 0) {
const rect = sel.getRangeAt(0).getBoundingClientRect();
setMenuPosition({
bottom: window.innerHeight - rect.top + 4,
left: rect.left,
});
}
setShowMenu(true);
} else {
setShowMenu(false);
}
emitChange();
}, [emitChange]);
// ── Handle paste — convert [tag] patterns to badges ───────────
const handlePaste = useCallback(
(e: React.ClipboardEvent) => {
e.preventDefault();
const text = e.clipboardData.getData('text/plain');
if (!text) return;
const el = editorRef.current;
if (!el) return;
const html = textToHTML(text);
// Insert at caret
const sel = window.getSelection();
if (sel && sel.rangeCount > 0) {
const range = sel.getRangeAt(0);
range.deleteContents();
const temp = document.createElement('div');
temp.innerHTML = html;
const frag = document.createDocumentFragment();
let lastNode: Node | null = null;
while (temp.firstChild) {
lastNode = frag.appendChild(temp.firstChild);
}
range.insertNode(frag);
if (lastNode) {
const newRange = document.createRange();
newRange.setStartAfter(lastNode);
newRange.collapse(true);
sel.removeAllRanges();
sel.addRange(newRange);
}
}
emitChange();
},
[emitChange],
);
// ── Show placeholder ──────────────────────────────────────────
const isEmpty = !value || value.trim() === '';
return (
<div className="relative">
{/* Placeholder */}
{isEmpty && placeholder && (
<div
className="pointer-events-none absolute inset-0 text-sm text-muted-foreground/60 px-3 py-2 select-none"
aria-hidden
>
{placeholder}
</div>
)}
{/* Editable area */}
<div
ref={editorRef}
contentEditable={!disabled}
suppressContentEditableWarning
role={disabled ? undefined : 'textbox'}
aria-multiline={disabled ? undefined : true}
aria-placeholder={placeholder}
aria-disabled={disabled}
tabIndex={disabled ? -1 : 0}
className={cn(
'min-h-[32px] text-sm whitespace-pre-wrap break-words outline-none',
'[&_.ptag-badge]:inline-flex [&_.ptag-badge]:items-center [&_.ptag-badge]:rounded-full',
'[&_.ptag-badge]:bg-accent/20 [&_.ptag-badge]:text-accent [&_.ptag-badge]:border [&_.ptag-badge]:border-accent/30',
'[&_.ptag-badge]:px-2 [&_.ptag-badge]:py-0 [&_.ptag-badge]:text-xs [&_.ptag-badge]:font-medium',
'[&_.ptag-badge]:mx-0.5 [&_.ptag-badge]:select-none [&_.ptag-badge]:cursor-default',
'[&_.ptag-badge]:align-baseline',
disabled && 'opacity-50 cursor-not-allowed',
className,
)}
style={style}
onInput={!disabled ? handleInput : undefined}
onKeyDown={!disabled ? handleKeyDown : undefined}
onPaste={!disabled ? handlePaste : undefined}
onClick={!disabled ? onClick : undefined}
onFocus={!disabled ? onFocus : undefined}
onBlur={() => {
setShowMenu(false);
triggerRangeRef.current = null;
}}
onCompositionStart={() => {
isComposingRef.current = true;
}}
onCompositionEnd={() => {
isComposingRef.current = false;
handleInput();
}}
/>
{/* Autocomplete dropdown — portalled to body, positioned above the caret */}
{showMenu &&
filteredTags.length > 0 &&
createPortal(
<AnimatePresence>
<motion.div
initial={{ opacity: 0, y: 4 }}
animate={{ opacity: 1, y: 0 }}
exit={{ opacity: 0, y: 4 }}
transition={{ duration: 0.12 }}
className="fixed z-[9999] min-w-[200px] max-h-[280px] overflow-y-auto rounded-lg border border-border bg-popover shadow-lg"
style={{
bottom: menuPosition.bottom,
left: menuPosition.left,
}}
>
{filteredTags.map((t, i) => (
<button
key={t.tag}
type="button"
className={cn(
'flex items-center gap-2 w-full px-3 py-1.5 text-sm text-left transition-colors',
i === menuIndex
? 'bg-accent/20 text-accent-foreground'
: 'text-popover-foreground hover:bg-muted/50',
)}
onMouseDown={(e) => {
e.preventDefault(); // Keep focus in editor
insertTag(t.tag);
}}
onMouseEnter={() => setMenuIndex(i)}
>
<span className="text-base leading-none">{t.emoji}</span>
<span>{t.label}</span>
<span className="ml-auto text-xs text-muted-foreground font-mono">{t.tag}</span>
</button>
))}
</motion.div>
</AnimatePresence>,
document.body,
)}
</div>
);
},
);
+130 -55
View File
@@ -1,13 +1,15 @@
import { useQueryClient } from '@tanstack/react-query';
import { import {
AudioWaveform,
Download, Download,
FileArchive, FileArchive,
Loader2, Loader2,
MoreHorizontal, MoreHorizontal,
Play, Play,
RotateCcw,
Trash2, Trash2,
} from 'lucide-react'; } from 'lucide-react';
import { useEffect, useRef, useState } from 'react'; import { useEffect, useRef, useState } from 'react';
import Loader from 'react-loaders';
import { Button } from '@/components/ui/button'; import { Button } from '@/components/ui/button';
import { import {
Dialog, Dialog,
@@ -36,7 +38,8 @@ import {
useImportGeneration, useImportGeneration,
} from '@/lib/hooks/useHistory'; } from '@/lib/hooks/useHistory';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
import { formatDate, formatDuration } from '@/lib/utils/format'; import { formatDate, formatDuration, formatEngineName } from '@/lib/utils/format';
import { useGenerationStore } from '@/stores/generationStore';
import { usePlayerStore } from '@/stores/playerStore'; import { usePlayerStore } from '@/stores/playerStore';
// OLD TABLE-BASED COMPONENT - REMOVED (can be found in git history) // OLD TABLE-BASED COMPONENT - REMOVED (can be found in git history)
@@ -54,9 +57,12 @@ export function HistoryTable() {
const [importDialogOpen, setImportDialogOpen] = useState(false); const [importDialogOpen, setImportDialogOpen] = useState(false);
const [selectedFile, setSelectedFile] = useState<File | null>(null); const [selectedFile, setSelectedFile] = useState<File | null>(null);
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false); const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
const [generationToDelete, setGenerationToDelete] = useState<{ id: string; name: string } | null>(null); const [generationToDelete, setGenerationToDelete] = useState<{ id: string; name: string } | null>(
null,
);
const limit = 20; const limit = 20;
const { toast } = useToast(); const { toast } = useToast();
const queryClient = useQueryClient();
const { const {
data: historyData, data: historyData,
@@ -71,6 +77,7 @@ export function HistoryTable() {
const exportGeneration = useExportGeneration(); const exportGeneration = useExportGeneration();
const exportGenerationAudio = useExportGenerationAudio(); const exportGenerationAudio = useExportGenerationAudio();
const importGeneration = useImportGeneration(); const importGeneration = useImportGeneration();
const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay); const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay);
const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio); const restartCurrentAudio = usePlayerStore((state) => state.restartCurrentAudio);
const currentAudioId = usePlayerStore((state) => state.audioId); const currentAudioId = usePlayerStore((state) => state.audioId);
@@ -194,6 +201,20 @@ export function HistoryTable() {
} }
}; };
const handleRetry = async (generationId: string) => {
try {
const result = await apiClient.retryGeneration(generationId);
addPendingGeneration(result.id);
queryClient.invalidateQueries({ queryKey: ['history'] });
} catch (error) {
toast({
title: 'Retry failed',
description: error instanceof Error ? error.message : 'Could not retry generation',
variant: 'destructive',
});
}
};
const handleImportConfirm = () => { const handleImportConfirm = () => {
if (selectedFile) { if (selectedFile) {
importGeneration.mutate(selectedFile, { importGeneration.mutate(selectedFile, {
@@ -250,25 +271,54 @@ export function HistoryTable() {
> >
{history.map((gen) => { {history.map((gen) => {
const isCurrentlyPlaying = currentAudioId === gen.id && isPlaying; const isCurrentlyPlaying = currentAudioId === gen.id && isPlaying;
const isGenerating = gen.status === 'generating';
const isFailed = gen.status === 'failed';
const isPlayable = !isGenerating && !isFailed;
return ( return (
<div <div
key={gen.id} key={gen.id}
role={isPlayable ? 'button' : undefined}
tabIndex={isPlayable ? 0 : undefined}
className={cn( className={cn(
'flex items-stretch gap-4 h-26 border rounded-md p-3 bg-card hover:bg-muted/70 transition-colors text-left w-full', 'flex items-stretch gap-4 h-26 border rounded-md p-3 bg-card transition-colors text-left w-full',
isPlayable && 'hover:bg-muted/70 cursor-pointer',
isCurrentlyPlaying && 'bg-muted/70', isCurrentlyPlaying && 'bg-muted/70',
)} )}
aria-label={
isGenerating
? `Generating speech for ${gen.profile_name}...`
: isFailed
? `Generation failed for ${gen.profile_name}`
: isCurrentlyPlaying
? `Sample from ${gen.profile_name}, ${formatDuration(gen.duration ?? 0)}, ${formatDate(gen.created_at)}. Playing. Press Enter to restart.`
: `Sample from ${gen.profile_name}, ${formatDuration(gen.duration ?? 0)}, ${formatDate(gen.created_at)}. Press Enter to play.`
}
onMouseDown={(e) => { onMouseDown={(e) => {
// Don't trigger play if clicking on textarea or if text is selected if (!isPlayable) return;
const target = e.target as HTMLElement; const target = e.target as HTMLElement;
if (target.closest('textarea') || window.getSelection()?.toString()) { if (target.closest('textarea') || window.getSelection()?.toString()) {
return; return;
} }
handlePlay(gen.id, gen.text, gen.profile_id); handlePlay(gen.id, gen.text, gen.profile_id);
}} }}
onKeyDown={(e) => {
if (!isPlayable) return;
const target = e.target as HTMLElement;
if (target.closest('textarea') || target.closest('button')) return;
if (e.key === 'Enter' || e.key === ' ') {
e.preventDefault();
handlePlay(gen.id, gen.text, gen.profile_id);
}
}}
> >
{/* Waveform icon */} {/* Status icon */}
<div className="flex items-center shrink-0"> <div className="flex items-center shrink-0 w-10 justify-center overflow-hidden">
<AudioWaveform className="h-5 w-5 text-muted-foreground" /> <div className="scale-50">
<Loader
type={isGenerating ? 'line-scale' : 'line-scale-pulse-out-rapid'}
active={isGenerating || isCurrentlyPlaying}
/>
</div>
</div> </div>
{/* Left side - Meta information */} {/* Left side - Meta information */}
@@ -279,11 +329,22 @@ export function HistoryTable() {
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<span className="text-xs text-muted-foreground">{gen.language}</span> <span className="text-xs text-muted-foreground">{gen.language}</span>
<span className="text-xs text-muted-foreground"> <span className="text-xs text-muted-foreground">
{formatDuration(gen.duration)} {formatEngineName(gen.engine, gen.model_size)}
</span> </span>
{isFailed ? (
<span className="text-xs text-destructive">Failed</span>
) : !isGenerating ? (
<span className="text-xs text-muted-foreground">
{formatDuration(gen.duration ?? 0)}
</span>
) : null}
</div> </div>
<div className="text-xs text-muted-foreground"> <div className="text-xs text-muted-foreground">
{formatDate(gen.created_at)} {isGenerating ? (
<span className="text-accent">Generating...</span>
) : (
formatDate(gen.created_at)
)}
</div> </div>
</div> </div>
@@ -293,57 +354,70 @@ export function HistoryTable() {
value={gen.text} value={gen.text}
className="flex-1 resize-none text-sm text-muted-foreground select-text" className="flex-1 resize-none text-sm text-muted-foreground select-text"
readOnly readOnly
aria-label={`Transcript for sample from ${gen.profile_name}, ${formatDuration(gen.duration ?? 0)}`}
/> />
</div> </div>
{/* Far right - Ellipsis actions */} {/* Far right - Actions */}
<div <div
className="w-10 shrink-0 flex justify-end" className="w-10 shrink-0 flex justify-end items-center"
onMouseDown={(e) => e.stopPropagation()} onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()} onClick={(e) => e.stopPropagation()}
> >
<DropdownMenu> {isFailed ? (
<DropdownMenuTrigger asChild> <Button
<Button variant="ghost"
variant="ghost" size="icon"
size="icon" className="h-8 w-8"
className="h-8 w-8" aria-label="Retry generation"
aria-label="Actions" onClick={() => handleRetry(gen.id)}
> >
<MoreHorizontal className="h-4 w-4" /> <RotateCcw className="h-4 w-4" />
</Button> </Button>
</DropdownMenuTrigger> ) : isPlayable ? (
<DropdownMenuContent align="end"> <DropdownMenu>
<DropdownMenuItem <DropdownMenuTrigger asChild>
onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)} <Button
> variant="ghost"
<Play className="mr-2 h-4 w-4" /> size="icon"
Play className="h-8 w-8"
</DropdownMenuItem> aria-label="Actions"
<DropdownMenuItem >
onClick={() => handleDownloadAudio(gen.id, gen.text)} <MoreHorizontal className="h-4 w-4" />
disabled={exportGenerationAudio.isPending} </Button>
> </DropdownMenuTrigger>
<Download className="mr-2 h-4 w-4" /> <DropdownMenuContent align="end">
Export Audio <DropdownMenuItem
</DropdownMenuItem> onClick={() => handlePlay(gen.id, gen.text, gen.profile_id)}
<DropdownMenuItem >
onClick={() => handleExportPackage(gen.id, gen.text)} <Play className="mr-2 h-4 w-4" />
disabled={exportGeneration.isPending} Play
> </DropdownMenuItem>
<FileArchive className="mr-2 h-4 w-4" /> <DropdownMenuItem
Export Package onClick={() => handleDownloadAudio(gen.id, gen.text)}
</DropdownMenuItem> disabled={exportGenerationAudio.isPending}
<DropdownMenuItem >
onClick={() => handleDeleteClick(gen.id, gen.profile_name)} <Download className="mr-2 h-4 w-4" />
disabled={deleteGeneration.isPending} Export Audio
className="text-destructive focus:text-destructive" </DropdownMenuItem>
> <DropdownMenuItem
<Trash2 className="mr-2 h-4 w-4" /> onClick={() => handleExportPackage(gen.id, gen.text)}
Delete disabled={exportGeneration.isPending}
</DropdownMenuItem> >
</DropdownMenuContent> <FileArchive className="mr-2 h-4 w-4" />
</DropdownMenu> Export Package
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => handleDeleteClick(gen.id, gen.profile_name)}
disabled={deleteGeneration.isPending}
className="text-destructive focus:text-destructive"
>
<Trash2 className="mr-2 h-4 w-4" />
Delete
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
) : null}
</div> </div>
</div> </div>
); );
@@ -371,7 +445,8 @@ export function HistoryTable() {
<DialogHeader> <DialogHeader>
<DialogTitle>Delete Generation</DialogTitle> <DialogTitle>Delete Generation</DialogTitle>
<DialogDescription> <DialogDescription>
Are you sure you want to delete this generation from "{generationToDelete?.name}"? This action cannot be undone. Are you sure you want to delete this generation from "{generationToDelete?.name}"?
This action cannot be undone.
</DialogDescription> </DialogDescription>
</DialogHeader> </DialogHeader>
<DialogFooter> <DialogFooter>
+7 -7
View File
@@ -13,7 +13,7 @@ import {
} from '@/components/ui/dialog'; } from '@/components/ui/dialog';
import { useToast } from '@/components/ui/use-toast'; import { useToast } from '@/components/ui/use-toast';
import { ProfileList } from '@/components/VoiceProfiles/ProfileList'; import { ProfileList } from '@/components/VoiceProfiles/ProfileList';
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import { useImportProfile } from '@/lib/hooks/useProfiles'; import { useImportProfile } from '@/lib/hooks/useProfiles';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
import { usePlayerStore } from '@/stores/playerStore'; import { usePlayerStore } from '@/stores/playerStore';
@@ -77,9 +77,9 @@ export function MainEditor() {
return ( return (
// Main view: Profiles top left, Generator bottom left, History right // Main view: Profiles top left, Generator bottom left, History right
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6 h-full min-h-0 overflow-hidden relative"> <div className="grid grid-cols-1 lg:grid-cols-2 lg:gap-6 h-full min-h-0 overflow-hidden relative">
{/* Left Column */} {/* Left Column */}
<div className="flex flex-col min-h-0 overflow-hidden relative"> <div className="flex flex-col min-h-0 overflow-hidden relative lg:overflow-hidden">
{/* Scroll Mask - Always visible, behind content */} {/* Scroll Mask - Always visible, behind content */}
<div className="absolute top-0 left-0 right-0 h-16 bg-gradient-to-b from-background to-transparent z-0 pointer-events-none" /> <div className="absolute top-0 left-0 right-0 h-16 bg-gradient-to-b from-background to-transparent z-0 pointer-events-none" />
@@ -110,10 +110,7 @@ export function MainEditor() {
{/* Scrollable Content */} {/* Scrollable Content */}
<div <div
ref={scrollRef} ref={scrollRef}
className={cn( className={cn('flex-1 min-h-0 overflow-y-auto pt-14 pb-4', isPlayerVisible && 'lg:pb-32')}
'flex-1 min-h-0 overflow-y-auto pt-14',
isPlayerVisible ? BOTTOM_SAFE_AREA_PADDING : 'pb-4',
)}
> >
<div className="flex flex-col gap-6"> <div className="flex flex-col gap-6">
<div className="shrink-0 flex flex-col"> <div className="shrink-0 flex flex-col">
@@ -123,6 +120,9 @@ export function MainEditor() {
</div> </div>
</div> </div>
{/* Divider - single column only */}
{/* <div className="border-t border-border -my-3 lg:hidden" /> */}
{/* Right Column - History */} {/* Right Column - History */}
<div className="flex flex-col min-h-0 overflow-hidden"> <div className="flex flex-col min-h-0 overflow-hidden">
<HistoryTable /> <HistoryTable />
+1 -1
View File
@@ -2,7 +2,7 @@ import { ModelManagement } from '@/components/ServerSettings/ModelManagement';
export function ModelsTab() { export function ModelsTab() {
return ( return (
<div className="space-y-4 overflow-y-auto flex flex-col"> <div className="h-full flex flex-col">
<ModelManagement /> <ModelManagement />
</div> </div>
); );
@@ -1,9 +1,12 @@
import { zodResolver } from '@hookform/resolvers/zod'; import { zodResolver } from '@hookform/resolvers/zod';
import { Loader2, XCircle } from 'lucide-react';
import { useEffect } from 'react'; import { useEffect } from 'react';
import { useForm } from 'react-hook-form'; import { useForm } from 'react-hook-form';
import * as z from 'zod'; import * as z from 'zod';
import { Badge } from '@/components/ui/badge';
import { Button } from '@/components/ui/button'; import { Button } from '@/components/ui/button';
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'; import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
import { Checkbox } from '@/components/ui/checkbox';
import { import {
Form, Form,
FormControl, FormControl,
@@ -14,10 +17,10 @@ import {
FormMessage, FormMessage,
} from '@/components/ui/form'; } from '@/components/ui/form';
import { Input } from '@/components/ui/input'; import { Input } from '@/components/ui/input';
import { Checkbox } from '@/components/ui/checkbox';
import { useToast } from '@/components/ui/use-toast'; import { useToast } from '@/components/ui/use-toast';
import { useServerStore } from '@/stores/serverStore'; import { useServerHealth } from '@/lib/hooks/useServer';
import { usePlatform } from '@/platform/PlatformContext'; import { usePlatform } from '@/platform/PlatformContext';
import { useServerStore } from '@/stores/serverStore';
const connectionSchema = z.object({ const connectionSchema = z.object({
serverUrl: z.string().url('Please enter a valid URL'), serverUrl: z.string().url('Please enter a valid URL'),
@@ -31,7 +34,10 @@ export function ConnectionForm() {
const setServerUrl = useServerStore((state) => state.setServerUrl); const setServerUrl = useServerStore((state) => state.setServerUrl);
const keepServerRunningOnClose = useServerStore((state) => state.keepServerRunningOnClose); const keepServerRunningOnClose = useServerStore((state) => state.keepServerRunningOnClose);
const setKeepServerRunningOnClose = useServerStore((state) => state.setKeepServerRunningOnClose); const setKeepServerRunningOnClose = useServerStore((state) => state.setKeepServerRunningOnClose);
const mode = useServerStore((state) => state.mode);
const setMode = useServerStore((state) => state.setMode);
const { toast } = useToast(); const { toast } = useToast();
const { data: health, isLoading, error: healthError } = useServerHealth();
const form = useForm<ConnectionFormValues>({ const form = useForm<ConnectionFormValues>({
resolver: zodResolver(connectionSchema), resolver: zodResolver(connectionSchema),
@@ -49,7 +55,7 @@ export function ConnectionForm() {
function onSubmit(data: ConnectionFormValues) { function onSubmit(data: ConnectionFormValues) {
setServerUrl(data.serverUrl); setServerUrl(data.serverUrl);
form.reset(data); // Reset form state after successful submission form.reset(data);
toast({ toast({
title: 'Server URL updated', title: 'Server URL updated',
description: `Connected to ${data.serverUrl}`, description: `Connected to ${data.serverUrl}`,
@@ -57,7 +63,7 @@ export function ConnectionForm() {
} }
return ( return (
<Card> <Card role="region" aria-label="Server Connection" tabIndex={0}>
<CardHeader> <CardHeader>
<CardTitle>Server Connection</CardTitle> <CardTitle>Server Connection</CardTitle>
</CardHeader> </CardHeader>
@@ -83,10 +89,42 @@ export function ConnectionForm() {
</form> </form>
</Form> </Form>
{/* Connection status */}
<div className="mt-4">
{isLoading ? (
<div className="flex items-center gap-2">
<Loader2 className="h-4 w-4 animate-spin" />
<span className="text-sm text-muted-foreground">Checking connection...</span>
</div>
) : healthError ? (
<div className="flex items-center gap-2">
<XCircle className="h-4 w-4 text-destructive" />
<span className="text-sm text-destructive">
Connection failed: {healthError.message}
</span>
</div>
) : health ? (
<div className="flex flex-wrap gap-2">
<Badge
variant={health.model_loaded || health.model_downloaded ? 'default' : 'secondary'}
>
{health.model_loaded || health.model_downloaded ? 'Model Ready' : 'No Model'}
</Badge>
<Badge variant={health.gpu_available ? 'default' : 'secondary'}>
GPU: {health.gpu_available ? 'Available' : 'Not Available'}
</Badge>
{health.vram_used_mb && (
<Badge variant="outline">VRAM: {health.vram_used_mb.toFixed(0)} MB</Badge>
)}
</div>
) : null}
</div>
<div className="mt-6 pt-6 border-t"> <div className="mt-6 pt-6 border-t">
<div className="flex items-start space-x-3"> <div className="flex items-start space-x-3">
<Checkbox <Checkbox
id="keepServerRunning" id="keepServerRunning"
className="mt-[6px]"
checked={keepServerRunningOnClose} checked={keepServerRunningOnClose}
onCheckedChange={(checked: boolean) => { onCheckedChange={(checked: boolean) => {
setKeepServerRunningOnClose(checked); setKeepServerRunningOnClose(checked);
@@ -115,6 +153,39 @@ export function ConnectionForm() {
</div> </div>
</div> </div>
</div> </div>
{platform.metadata.isTauri && (
<div className="mt-6 pt-6 border-t">
<div className="flex items-start space-x-3">
<Checkbox
id="allowNetworkAccess"
className="mt-[6px]"
checked={mode === 'remote'}
onCheckedChange={(checked: boolean) => {
setMode(checked ? 'remote' : 'local');
toast({
title: 'Setting updated',
description: checked
? 'Network access enabled. Restart the app to apply.'
: 'Network access disabled. Restart the app to apply.',
});
}}
/>
<div className="space-y-1">
<label
htmlFor="allowNetworkAccess"
className="text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70 cursor-pointer"
>
Allow network access
</label>
<p className="text-sm text-muted-foreground">
Makes the server accessible from other devices on your network. Restart the app
after changing this setting.
</p>
</div>
</div>
</div>
)}
</CardContent> </CardContent>
</Card> </Card>
); );
@@ -0,0 +1,116 @@
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
import { Checkbox } from '@/components/ui/checkbox';
import { Slider } from '@/components/ui/slider';
import { useServerStore } from '@/stores/serverStore';
export function GenerationSettings() {
const maxChunkChars = useServerStore((state) => state.maxChunkChars);
const setMaxChunkChars = useServerStore((state) => state.setMaxChunkChars);
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
const setCrossfadeMs = useServerStore((state) => state.setCrossfadeMs);
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
const setNormalizeAudio = useServerStore((state) => state.setNormalizeAudio);
const autoplayOnGenerate = useServerStore((state) => state.autoplayOnGenerate);
const setAutoplayOnGenerate = useServerStore((state) => state.setAutoplayOnGenerate);
return (
<Card role="region" aria-label="Generation Settings" tabIndex={0}>
<CardHeader>
<CardTitle>Generation Settings</CardTitle>
<CardDescription>
Controls for long text generation. These settings apply to all engines.
</CardDescription>
</CardHeader>
<CardContent>
<div className="space-y-6">
<div className="space-y-3">
<div className="flex items-center justify-between">
<label htmlFor="maxChunkChars" className="text-sm font-medium leading-none">
Auto-chunking limit
</label>
<span className="text-sm tabular-nums text-muted-foreground">
{maxChunkChars} chars
</span>
</div>
<Slider
id="maxChunkChars"
value={[maxChunkChars]}
onValueChange={([value]) => setMaxChunkChars(value)}
min={100}
max={5000}
step={50}
aria-label="Auto-chunking character limit"
/>
<p className="text-sm text-muted-foreground">
Long text is split into chunks at sentence boundaries before generating. Lower values
can improve quality for long outputs.
</p>
</div>
<div className="space-y-3">
<div className="flex items-center justify-between">
<label htmlFor="crossfadeMs" className="text-sm font-medium leading-none">
Chunk crossfade
</label>
<span className="text-sm tabular-nums text-muted-foreground">
{crossfadeMs === 0 ? 'Cut' : `${crossfadeMs}ms`}
</span>
</div>
<Slider
id="crossfadeMs"
value={[crossfadeMs]}
onValueChange={([value]) => setCrossfadeMs(value)}
min={0}
max={200}
step={10}
aria-label="Chunk crossfade duration"
/>
<p className="text-sm text-muted-foreground">
Blends audio between chunks to smooth transitions. Set to 0 for a hard cut.
</p>
</div>
<div className="flex items-start gap-3">
<Checkbox
id="normalizeAudio"
checked={normalizeAudio}
onCheckedChange={setNormalizeAudio}
className="mt-[6px]"
/>
<div className="space-y-1">
<label
htmlFor="normalizeAudio"
className="text-sm font-medium leading-none cursor-pointer"
>
Normalize audio
</label>
<p className="text-sm text-muted-foreground">
Adjusts output volume to a consistent level across generations.
</p>
</div>
</div>
<div className="flex items-start gap-3">
<Checkbox
id="autoplayOnGenerate"
checked={autoplayOnGenerate}
onCheckedChange={setAutoplayOnGenerate}
className="mt-[6px]"
/>
<div className="space-y-1">
<label
htmlFor="autoplayOnGenerate"
className="text-sm font-medium leading-none cursor-pointer"
>
Autoplay on generate
</label>
<p className="text-sm text-muted-foreground">
Automatically play audio when a generation completes.
</p>
</div>
</div>
</div>
</CardContent>
</Card>
);
}
@@ -1,7 +1,6 @@
import { useQuery, useQueryClient } from '@tanstack/react-query'; import { useQuery, useQueryClient } from '@tanstack/react-query';
import { AlertCircle, Cpu, Download, Loader2, RotateCw, Trash2, Zap } from 'lucide-react'; import { AlertCircle, Download, Loader2, RotateCw, Trash2 } from 'lucide-react';
import { useCallback, useEffect, useRef, useState } from 'react'; import { useCallback, useEffect, useRef, useState } from 'react';
import { Badge } from '@/components/ui/badge';
import { Button } from '@/components/ui/button'; import { Button } from '@/components/ui/button';
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'; import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
import { Progress } from '@/components/ui/progress'; import { Progress } from '@/components/ui/progress';
@@ -32,7 +31,7 @@ export function GpuAcceleration() {
} = useQuery({ } = useQuery({
queryKey: ['cuda-status', serverUrl], queryKey: ['cuda-status', serverUrl],
queryFn: () => apiClient.getCudaStatus(), queryFn: () => apiClient.getCudaStatus(),
refetchInterval: cudaStatusLoading ? false : 10000, refetchInterval: (query) => (query.state.status === 'pending' ? false : 10000),
retry: 1, retry: 1,
enabled: !!health, // Only fetch when backend is reachable enabled: !!health, // Only fetch when backend is reachable
}); });
@@ -216,31 +215,19 @@ export function GpuAcceleration() {
return ( return (
<Card> <Card>
<CardHeader> <CardHeader>
<CardTitle className="flex items-center gap-2"> <CardTitle>GPU Acceleration</CardTitle>
<Zap className="h-4 w-4" />
GPU Acceleration
</CardTitle>
</CardHeader> </CardHeader>
<CardContent className="space-y-4"> <CardContent className="space-y-4">
{/* Current status */} {/* Current status */}
<div className="flex items-center justify-between"> <div className="space-y-1">
<div className="space-y-1"> <div className="text-sm font-medium">Backend</div>
<div className="text-sm font-medium">Backend</div> <div className="text-sm text-muted-foreground">
<div className="text-sm text-muted-foreground"> {isCurrentlyCuda
{isCurrentlyCuda ? 'CUDA (GPU accelerated)' : 'CPU'} ? 'CUDA (GPU accelerated)'
</div> : hasNativeGpu
? `${health.backend_type === 'mlx' ? 'MLX' : 'PyTorch'} (GPU accelerated)`
: 'CPU'}
</div> </div>
<Badge variant={isCurrentlyCuda ? 'default' : 'secondary'}>
{isCurrentlyCuda ? (
<>
<Zap className="h-3 w-3 mr-1" /> CUDA
</>
) : (
<>
<Cpu className="h-3 w-3 mr-1" /> CPU
</>
)}
</Badge>
</div> </div>
{/* GPU info from health */} {/* GPU info from health */}
@@ -257,14 +244,6 @@ export function GpuAcceleration() {
)} )}
{/* Native GPU detected - no CUDA download needed */} {/* Native GPU detected - no CUDA download needed */}
{hasNativeGpu && (
<div className="p-3 rounded-lg bg-accent/10 border border-accent/20">
<div className="text-sm">
Your system uses <strong>{health.gpu_type}</strong> for acceleration. No additional
downloads needed.
</div>
</div>
)}
{/* CUDA download section - only show when native GPU is NOT detected (i.e., Windows/Linux NVIDIA users) */} {/* CUDA download section - only show when native GPU is NOT detected (i.e., Windows/Linux NVIDIA users) */}
{!hasNativeGpu && ( {!hasNativeGpu && (
File diff suppressed because it is too large Load Diff
@@ -3,14 +3,13 @@ import { Badge } from '@/components/ui/badge';
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'; import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
import { useServerHealth } from '@/lib/hooks/useServer'; import { useServerHealth } from '@/lib/hooks/useServer';
import { useServerStore } from '@/stores/serverStore'; import { useServerStore } from '@/stores/serverStore';
import { ModelProgress } from './ModelProgress';
export function ServerStatus() { export function ServerStatus() {
const { data: health, isLoading, error } = useServerHealth(); const { data: health, isLoading, error } = useServerHealth();
const serverUrl = useServerStore((state) => state.serverUrl); const serverUrl = useServerStore((state) => state.serverUrl);
return ( return (
<Card> <Card role="region" aria-label="Server Status" tabIndex={0}>
<CardHeader> <CardHeader>
<CardTitle>Server Status</CardTitle> <CardTitle>Server Status</CardTitle>
</CardHeader> </CardHeader>
@@ -20,16 +19,6 @@ export function ServerStatus() {
<div className="font-mono text-sm">{serverUrl}</div> <div className="font-mono text-sm">{serverUrl}</div>
</div> </div>
{/* Model download progress */}
<div className="space-y-2">
<ModelProgress modelName="qwen-tts-1.7B" displayName="Qwen TTS 1.7B" />
<ModelProgress modelName="qwen-tts-0.6B" displayName="Qwen TTS 0.6B" />
<ModelProgress modelName="whisper-base" displayName="Whisper Base" />
<ModelProgress modelName="whisper-small" displayName="Whisper Small" />
<ModelProgress modelName="whisper-medium" displayName="Whisper Medium" />
<ModelProgress modelName="whisper-large" displayName="Whisper Large" />
</div>
{isLoading ? ( {isLoading ? (
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<Loader2 className="h-4 w-4 animate-spin" /> <Loader2 className="h-4 w-4 animate-spin" />
@@ -20,7 +20,11 @@ export function UpdateStatus() {
}, [platform]); }, [platform]);
return ( return (
<Card> <Card
role="region"
aria-label="App Updates"
tabIndex={0}
>
<CardHeader> <CardHeader>
<CardTitle>App Updates</CardTitle> <CardTitle>App Updates</CardTitle>
</CardHeader> </CardHeader>
+11 -5
View File
@@ -1,19 +1,25 @@
import { ConnectionForm } from '@/components/ServerSettings/ConnectionForm'; import { ConnectionForm } from '@/components/ServerSettings/ConnectionForm';
import { GenerationSettings } from '@/components/ServerSettings/GenerationSettings';
import { GpuAcceleration } from '@/components/ServerSettings/GpuAcceleration'; import { GpuAcceleration } from '@/components/ServerSettings/GpuAcceleration';
import { ServerStatus } from '@/components/ServerSettings/ServerStatus';
import { UpdateStatus } from '@/components/ServerSettings/UpdateStatus'; import { UpdateStatus } from '@/components/ServerSettings/UpdateStatus';
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import { cn } from '@/lib/utils/cn';
import { usePlatform } from '@/platform/PlatformContext'; import { usePlatform } from '@/platform/PlatformContext';
import { usePlayerStore } from '@/stores/playerStore';
export function ServerTab() { export function ServerTab() {
const platform = usePlatform(); const platform = usePlatform();
const isPlayerVisible = !!usePlayerStore((state) => state.audioUrl);
return ( return (
<div className="space-y-4 overflow-y-auto flex flex-col"> <div
className={cn('overflow-y-auto flex flex-col', isPlayerVisible && BOTTOM_SAFE_AREA_PADDING)}
>
<div className="grid gap-4 md:grid-cols-2"> <div className="grid gap-4 md:grid-cols-2">
<ConnectionForm /> <ConnectionForm />
<ServerStatus /> <GenerationSettings />
{platform.metadata.isTauri && <GpuAcceleration />}
{platform.metadata.isTauri && <UpdateStatus />}
</div> </div>
{platform.metadata.isTauri && <GpuAcceleration />}
{platform.metadata.isTauri && <UpdateStatus />}
<div className="py-8 text-center text-sm text-muted-foreground"> <div className="py-8 text-center text-sm text-muted-foreground">
Created by{' '} Created by{' '}
<a <a
+11 -22
View File
@@ -1,9 +1,9 @@
import { Link, useMatchRoute } from '@tanstack/react-router'; import { Link, useMatchRoute } from '@tanstack/react-router';
import { Box, BookOpen, Loader2, Mic, Server, Speaker, Volume2 } from 'lucide-react'; import { BookOpen, Box, Mic, Server, Speaker, Volume2 } from 'lucide-react';
import voiceboxLogo from '@/assets/voicebox-logo.png'; import voiceboxLogo from '@/assets/voicebox-logo.png';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
import { useGenerationStore } from '@/stores/generationStore';
import { usePlayerStore } from '@/stores/playerStore'; import { usePlayerStore } from '@/stores/playerStore';
import { version } from '../../package.json';
interface SidebarProps { interface SidebarProps {
isMacOS?: boolean; isMacOS?: boolean;
@@ -19,10 +19,8 @@ const tabs = [
]; ];
export function Sidebar({ isMacOS }: SidebarProps) { export function Sidebar({ isMacOS }: SidebarProps) {
const isGenerating = useGenerationStore((state) => state.isGenerating);
const audioUrl = usePlayerStore((state) => state.audioUrl);
const isPlayerVisible = !!audioUrl;
const matchRoute = useMatchRoute(); const matchRoute = useMatchRoute();
const isPlayerOpen = !!usePlayerStore((s) => s.audioUrl);
return ( return (
<div <div
@@ -42,9 +40,7 @@ export function Sidebar({ isMacOS }: SidebarProps) {
const Icon = tab.icon; const Icon = tab.icon;
// For index route, use exact match; for others, use default matching // For index route, use exact match; for others, use default matching
const isActive = const isActive =
tab.path === '/' tab.path === '/' ? matchRoute({ to: '/', exact: true }) : matchRoute({ to: tab.path });
? matchRoute({ to: '/', exact: true })
: matchRoute({ to: tab.path });
return ( return (
<Link <Link
@@ -64,20 +60,13 @@ export function Sidebar({ isMacOS }: SidebarProps) {
})} })}
</div> </div>
{/* Spacer to push loader to bottom */} {/* Version */}
<div className="flex-1" /> <div
className="mt-auto text-[10px] text-muted-foreground/50 transition-all duration-300"
{/* Generation Loader */} style={{ paddingBottom: isPlayerOpen ? '7rem' : undefined }}
{isGenerating && ( >
<div v{version}
className={cn( </div>
'w-full flex items-center justify-center transition-all duration-200',
isPlayerVisible ? 'mb-[120px]' : 'mb-0',
)}
>
<Loader2 className="h-6 w-6 text-accent animate-spin" />
</div>
)}
</div> </div>
); );
} }
+4 -1
View File
@@ -1,8 +1,11 @@
import { FloatingGenerateBox } from '@/components/Generation/FloatingGenerateBox'; import { FloatingGenerateBox } from '@/components/Generation/FloatingGenerateBox';
import { usePlayerStore } from '@/stores/playerStore';
import { StoryContent } from './StoryContent'; import { StoryContent } from './StoryContent';
import { StoryList } from './StoryList'; import { StoryList } from './StoryList';
export function StoriesTab() { export function StoriesTab() {
const audioUrl = usePlayerStore((state) => state.audioUrl);
return ( return (
<div className="flex flex-col h-full min-h-0 overflow-hidden"> <div className="flex flex-col h-full min-h-0 overflow-hidden">
{/* Main content area */} {/* Main content area */}
@@ -18,7 +21,7 @@ export function StoriesTab() {
</div> </div>
{/* Floating Generate Box - position is managed via storyStore.trackEditorHeight */} {/* Floating Generate Box - position is managed via storyStore.trackEditorHeight */}
<FloatingGenerateBox showVoiceSelector /> <FloatingGenerateBox showVoiceSelector isPlayerOpen={!!audioUrl} />
</div> </div>
</div> </div>
); );
+33 -6
View File
@@ -13,8 +13,11 @@ import {
sortableKeyboardCoordinates, sortableKeyboardCoordinates,
verticalListSortingStrategy, verticalListSortingStrategy,
} from '@dnd-kit/sortable'; } from '@dnd-kit/sortable';
import { Link } from '@tanstack/react-router';
import { AnimatePresence, motion } from 'framer-motion';
import { Download, Plus } from 'lucide-react'; import { Download, Plus } from 'lucide-react';
import { useEffect, useMemo, useRef, useState } from 'react'; import { useEffect, useMemo, useRef, useState } from 'react';
import Loader from 'react-loaders';
import { Button } from '@/components/ui/button'; import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input'; import { Input } from '@/components/ui/input';
import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover'; import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover';
@@ -28,6 +31,7 @@ import {
useStory, useStory,
} from '@/lib/hooks/useStories'; } from '@/lib/hooks/useStories';
import { useStoryPlayback } from '@/lib/hooks/useStoryPlayback'; import { useStoryPlayback } from '@/lib/hooks/useStoryPlayback';
import { useGenerationStore } from '@/stores/generationStore';
import { useStoryStore } from '@/stores/storyStore'; import { useStoryStore } from '@/stores/storyStore';
import { SortableStoryChatItem } from './StoryChatItem'; import { SortableStoryChatItem } from './StoryChatItem';
@@ -40,6 +44,7 @@ export function StoryContent() {
const addStoryItem = useAddStoryItem(); const addStoryItem = useAddStoryItem();
const { toast } = useToast(); const { toast } = useToast();
const scrollRef = useRef<HTMLDivElement>(null); const scrollRef = useRef<HTMLDivElement>(null);
const pendingCount = useGenerationStore((s) => s.pendingGenerationIds.size);
// Add generation popover state // Add generation popover state
const [searchQuery, setSearchQuery] = useState(''); const [searchQuery, setSearchQuery] = useState('');
@@ -53,9 +58,9 @@ export function StoryContent() {
const query = searchQuery.toLowerCase(); const query = searchQuery.toLowerCase();
return historyData.items.filter( return historyData.items.filter(
(gen) => (gen) =>
gen.status === 'completed' &&
!storyGenerationIds.has(gen.id) && !storyGenerationIds.has(gen.id) &&
(gen.text.toLowerCase().includes(query) || (gen.text.toLowerCase().includes(query) || gen.profile_name.toLowerCase().includes(query)),
gen.profile_name.toLowerCase().includes(query)),
); );
}, [historyData, story, searchQuery]); }, [historyData, story, searchQuery]);
@@ -267,7 +272,31 @@ export function StoryContent() {
<p className="text-sm text-muted-foreground mt-1">{story.description}</p> <p className="text-sm text-muted-foreground mt-1">{story.description}</p>
)} )}
</div> </div>
<div className="flex gap-2"> <div className="flex gap-2 items-center">
<AnimatePresence>
{pendingCount > 0 && (
<motion.div
initial={{ opacity: 0, scale: 0.9, width: 0 }}
animate={{ opacity: 1, scale: 1, width: 'auto' }}
exit={{ opacity: 0, scale: 0.9, width: 0 }}
transition={{ duration: 0.2 }}
>
<Link
to="/"
className="flex items-center gap-2 h-8 pl-1.5 pr-3 rounded-full bg-card border border-border hover:bg-muted/50 transition-all duration-200 cursor-pointer"
>
<div className="shrink-0 w-10 h-5 overflow-hidden flex items-center justify-center">
<div className="scale-[0.45]">
<Loader type="line-scale" active />
</div>
</div>
<span className="text-xs text-muted-foreground whitespace-nowrap">
Generating {pendingCount} {pendingCount === 1 ? 'audio' : 'audios'}
</span>
</Link>
</motion.div>
)}
</AnimatePresence>
<Popover open={isAddOpen} onOpenChange={setIsAddOpen}> <Popover open={isAddOpen} onOpenChange={setIsAddOpen}>
<PopoverTrigger asChild> <PopoverTrigger asChild>
<Button variant="outline" size="sm"> <Button variant="outline" size="sm">
@@ -287,9 +316,7 @@ export function StoryContent() {
<div className="max-h-60 overflow-y-auto"> <div className="max-h-60 overflow-y-auto">
{availableGenerations.length === 0 ? ( {availableGenerations.length === 0 ? (
<div className="p-4 text-center text-sm text-muted-foreground"> <div className="p-4 text-center text-sm text-muted-foreground">
{searchQuery {searchQuery ? 'No matching generations found' : 'No available generations'}
? 'No matching generations found'
: 'No available generations'}
</div> </div>
) : ( ) : (
availableGenerations.map((gen) => ( availableGenerations.map((gen) => (
+20 -7
View File
@@ -194,17 +194,29 @@ export function StoryList() {
storyList.map((story) => ( storyList.map((story) => (
<div <div
key={story.id} key={story.id}
role="button"
tabIndex={0}
className={cn( className={cn(
'h-24 p-4 border rounded-2xl transition-colors group flex items-center', 'h-24 p-4 border rounded-2xl transition-colors group flex items-center cursor-pointer',
selectedStoryId === story.id && 'bg-muted border-primary', selectedStoryId === story.id && 'bg-muted border-primary',
)} )}
aria-label={
selectedStoryId === story.id
? `Story ${story.name}, ${story.item_count} ${story.item_count === 1 ? 'item' : 'items'}, ${formatDate(story.updated_at)}. Selected. Press Enter to select.`
: `Story ${story.name}, ${story.item_count} ${story.item_count === 1 ? 'item' : 'items'}, ${formatDate(story.updated_at)}. Press Enter to select.`
}
aria-pressed={selectedStoryId === story.id}
onClick={() => setSelectedStoryId(story.id)}
onKeyDown={(e) => {
if (e.target !== e.currentTarget) return;
if (e.key === 'Enter' || e.key === ' ') {
e.preventDefault();
setSelectedStoryId(story.id);
}
}}
> >
<div className="flex items-start justify-between gap-2 w-full min-w-0"> <div className="flex items-start justify-between gap-2 w-full min-w-0">
<button <div className="flex-1 min-w-0 text-left overflow-hidden">
type="button"
className="flex-1 min-w-0 text-left cursor-pointer overflow-hidden"
onClick={() => setSelectedStoryId(story.id)}
>
<h3 className="font-medium truncate">{story.name}</h3> <h3 className="font-medium truncate">{story.name}</h3>
{story.description && ( {story.description && (
<p className="text-sm text-muted-foreground mt-1 truncate"> <p className="text-sm text-muted-foreground mt-1 truncate">
@@ -218,7 +230,7 @@ export function StoryList() {
<span>•</span> <span>•</span>
<span>{formatDate(story.updated_at)}</span> <span>{formatDate(story.updated_at)}</span>
</div> </div>
</button> </div>
<DropdownMenu> <DropdownMenu>
<DropdownMenuTrigger asChild> <DropdownMenuTrigger asChild>
<Button <Button
@@ -226,6 +238,7 @@ export function StoryList() {
size="icon" size="icon"
className="h-8 w-8 opacity-0 group-hover:opacity-100 transition-opacity" className="h-8 w-8 opacity-0 group-hover:opacity-100 transition-opacity"
onClick={(e) => e.stopPropagation()} onClick={(e) => e.stopPropagation()}
aria-label={`Actions for ${story.name}`}
> >
<MoreHorizontal className="h-4 w-4" /> <MoreHorizontal className="h-4 w-4" />
</Button> </Button>
@@ -736,6 +736,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
className="h-7 w-7" className="h-7 w-7"
onClick={handlePlayPause} onClick={handlePlayPause}
title="Play/Pause (Space)" title="Play/Pause (Space)"
aria-label={isCurrentlyPlaying ? 'Pause' : 'Play'}
> >
{isCurrentlyPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />} {isCurrentlyPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
</Button> </Button>
@@ -745,6 +746,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
className="h-7 w-7" className="h-7 w-7"
onClick={handleStop} onClick={handleStop}
disabled={!isCurrentlyPlaying} disabled={!isCurrentlyPlaying}
aria-label="Stop"
> >
<Square className="h-3 w-3" /> <Square className="h-3 w-3" />
</Button> </Button>
@@ -762,6 +764,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
className="h-7 w-7" className="h-7 w-7"
onClick={handleSplit} onClick={handleSplit}
title="Split at playhead (S)" title="Split at playhead (S)"
aria-label="Split at playhead"
> >
<Scissors className="h-4 w-4" /> <Scissors className="h-4 w-4" />
</Button> </Button>
@@ -771,6 +774,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
className="h-7 w-7" className="h-7 w-7"
onClick={handleDuplicate} onClick={handleDuplicate}
title="Duplicate (Cmd/Ctrl+D)" title="Duplicate (Cmd/Ctrl+D)"
aria-label="Duplicate clip"
> >
<Copy className="h-4 w-4" /> <Copy className="h-4 w-4" />
</Button> </Button>
@@ -780,6 +784,7 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
className="h-7 w-7" className="h-7 w-7"
onClick={handleDelete} onClick={handleDelete}
title="Delete (Delete/Backspace)" title="Delete (Delete/Backspace)"
aria-label="Delete clip"
> >
<Trash2 className="h-4 w-4" /> <Trash2 className="h-4 w-4" />
</Button> </Button>
@@ -789,10 +794,22 @@ export function StoryTrackEditor({ storyId, items }: StoryTrackEditorProps) {
{/* Zoom controls - right side */} {/* Zoom controls - right side */}
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<span className="text-xs text-muted-foreground">Zoom:</span> <span className="text-xs text-muted-foreground">Zoom:</span>
<Button variant="ghost" size="icon" className="h-6 w-6" onClick={handleZoomOut}> <Button
variant="ghost"
size="icon"
className="h-6 w-6"
onClick={handleZoomOut}
aria-label="Zoom out"
>
<Minus className="h-3 w-3" /> <Minus className="h-3 w-3" />
</Button> </Button>
<Button variant="ghost" size="icon" className="h-6 w-6" onClick={handleZoomIn}> <Button
variant="ghost"
size="icon"
className="h-6 w-6"
onClick={handleZoomIn}
aria-label="Zoom in"
>
<Plus className="h-3 w-3" /> <Plus className="h-3 w-3" />
</Button> </Button>
</div> </div>
@@ -140,7 +140,13 @@ export function AudioSampleRecording({
</div> </div>
<p className="text-sm text-muted-foreground text-center">File: {file.name}</p> <p className="text-sm text-muted-foreground text-center">File: {file.name}</p>
<div className="flex gap-2"> <div className="flex gap-2">
<Button type="button" size="icon" variant="outline" onClick={onPlayPause}> <Button
type="button"
size="icon"
variant="outline"
onClick={onPlayPause}
aria-label={isPlaying ? 'Pause' : 'Play'}
>
{isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />} {isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
</Button> </Button>
<Button <Button
@@ -77,7 +77,13 @@ export function AudioSampleSystem({
</div> </div>
<p className="text-sm text-muted-foreground text-center">File: {file.name}</p> <p className="text-sm text-muted-foreground text-center">File: {file.name}</p>
<div className="flex gap-2"> <div className="flex gap-2">
<Button type="button" size="icon" variant="outline" onClick={onPlayPause}> <Button
type="button"
size="icon"
variant="outline"
onClick={onPlayPause}
aria-label={isPlaying ? 'Pause' : 'Play'}
>
{isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />} {isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
</Button> </Button>
<Button <Button
@@ -110,6 +110,7 @@ export function AudioSampleUpload({
variant="outline" variant="outline"
onClick={onPlayPause} onClick={onPlayPause}
disabled={isValidating} disabled={isValidating}
aria-label={isPlaying ? 'Pause' : 'Play'}
> >
{isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />} {isPlaying ? <Pause className="h-4 w-4" /> : <Play className="h-4 w-4" />}
</Button> </Button>
@@ -61,14 +61,32 @@ export function ProfileCard({ profile }: ProfileCardProps) {
exportProfile.mutate(profile.id); exportProfile.mutate(profile.id);
}; };
const handleKeyDown = (e: React.KeyboardEvent) => {
const target = e.target as HTMLElement;
if (target.closest('button')) return;
if (e.key === 'Enter' || e.key === ' ') {
e.preventDefault();
handleSelect();
}
};
const selectLabel = isSelected
? `${profile.name}, ${profile.language}. Selected as voice for generation.`
: `${profile.name}, ${profile.language}. Select as voice for generation.`;
return ( return (
<> <>
<Card <Card
className={cn( className={cn(
'cursor-pointer hover:shadow-md transition-all flex flex-col', 'cursor-pointer hover:shadow-md transition-all flex flex-col h-[162px]',
isSelected && 'ring-2 ring-primary shadow-md', isSelected && 'ring-2 ring-primary shadow-md',
)} )}
onClick={handleSelect} onClick={handleSelect}
tabIndex={0}
role="button"
aria-label={selectLabel}
aria-pressed={isSelected}
onKeyDown={handleKeyDown}
> >
<CardHeader className="p-3 pb-2"> <CardHeader className="p-3 pb-2">
<CardTitle className="flex items-center gap-1.5 text-base font-medium"> <CardTitle className="flex items-center gap-1.5 text-base font-medium">
@@ -41,9 +41,11 @@ export function ProfileList() {
</CardContent> </CardContent>
</Card> </Card>
) : ( ) : (
<div className="grid gap-4 grid-cols-3 auto-rows-auto p-1 pb-[150px]"> <div className="flex gap-4 overflow-x-auto p-1 pb-1 lg:grid lg:grid-cols-3 lg:auto-rows-auto lg:overflow-x-visible lg:pb-[150px]">
{allProfiles.map((profile) => ( {allProfiles.map((profile) => (
<ProfileCard key={profile.id} profile={profile} /> <div key={profile.id} className="shrink-0 w-[200px] lg:w-auto lg:shrink">
<ProfileCard profile={profile} />
</div>
))} ))}
</div> </div>
)} )}
@@ -102,6 +102,7 @@ function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
className="h-7 w-7 shrink-0" className="h-7 w-7 shrink-0"
onClick={handlePlayPause} onClick={handlePlayPause}
disabled={isLoading} disabled={isLoading}
aria-label={isPlaying ? 'Pause sample' : 'Play sample'}
> >
{isPlaying ? <Pause className="h-3.5 w-3.5" /> : <Play className="h-3.5 w-3.5 ml-0.5" />} {isPlaying ? <Pause className="h-3.5 w-3.5" /> : <Play className="h-3.5 w-3.5 ml-0.5" />}
</Button> </Button>
@@ -113,6 +114,8 @@ function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
max={100} max={100}
step={0.1} step={0.1}
className="flex-1" className="flex-1"
aria-label="Sample playback position"
aria-valuetext={`${formatAudioDuration(currentTime)} of ${formatAudioDuration(duration)}`}
/> />
<div className="flex items-center gap-1 text-xs text-muted-foreground shrink-0 min-w-[70px]"> <div className="flex items-center gap-1 text-xs text-muted-foreground shrink-0 min-w-[70px]">
<span className="font-mono">{formatAudioDuration(currentTime)}</span> <span className="font-mono">{formatAudioDuration(currentTime)}</span>
@@ -128,6 +131,7 @@ function MiniSamplePlayer({ audioUrl }: MiniSamplePlayerProps) {
className="h-7 w-7 shrink-0" className="h-7 w-7 shrink-0"
onClick={handleStop} onClick={handleStop}
title="Stop" title="Stop"
aria-label="Stop playback"
> >
<X className="h-3.5 w-3.5" /> <X className="h-3.5 w-3.5" />
</Button> </Button>
+18 -7
View File
@@ -179,25 +179,36 @@ function VoiceRow({
onDelete, onDelete,
}: VoiceRowProps) { }: VoiceRowProps) {
const { data: samples } = useProfileSamples(profile.id); const { data: samples } = useProfileSamples(profile.id);
const sampleCount = samples?.length || 0;
const rowLabel = `${profile.name}, ${profile.language}, ${generationCount} generations, ${sampleCount} samples. Press Enter to edit.`;
return ( return (
<TableRow className="cursor-pointer" onClick={onEdit}> <TableRow className="cursor-pointer" onClick={onEdit}>
<TableCell> <TableCell>
<div className="flex items-center gap-2"> <button
type="button"
className="flex w-full min-w-0 items-center gap-2 text-left focus:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 rounded"
aria-label={rowLabel}
onClick={(e) => {
e.stopPropagation();
onEdit();
}}
>
<div className="h-8 w-8 rounded-lg bg-muted flex items-center justify-center shrink-0"> <div className="h-8 w-8 rounded-lg bg-muted flex items-center justify-center shrink-0">
<Mic className="h-4 w-4 text-muted-foreground" /> <Mic className="h-4 w-4 text-muted-foreground" />
</div> </div>
<div> <div className="min-w-0">
<div className="font-medium">{profile.name}</div> <div className="font-medium truncate">{profile.name}</div>
{profile.description && ( {profile.description && (
<div className="text-sm text-muted-foreground">{profile.description}</div> <div className="text-sm text-muted-foreground truncate">{profile.description}</div>
)} )}
</div> </div>
</div> </button>
</TableCell> </TableCell>
<TableCell onClick={(e) => e.stopPropagation()}>{profile.language}</TableCell> <TableCell onClick={(e) => e.stopPropagation()}>{profile.language}</TableCell>
<TableCell onClick={(e) => e.stopPropagation()}>{generationCount}</TableCell> <TableCell onClick={(e) => e.stopPropagation()}>{generationCount}</TableCell>
<TableCell onClick={(e) => e.stopPropagation()}>{samples?.length || 0}</TableCell> <TableCell onClick={(e) => e.stopPropagation()}>{sampleCount}</TableCell>
<TableCell onClick={(e) => e.stopPropagation()}> <TableCell onClick={(e) => e.stopPropagation()}>
<MultiSelect <MultiSelect
options={channels.map((ch) => ({ options={channels.map((ch) => ({
@@ -213,7 +224,7 @@ function VoiceRow({
<TableCell onClick={(e) => e.stopPropagation()}> <TableCell onClick={(e) => e.stopPropagation()}>
<DropdownMenu> <DropdownMenu>
<DropdownMenuTrigger asChild> <DropdownMenuTrigger asChild>
<Button variant="ghost" size="icon"> <Button variant="ghost" size="icon" aria-label={`Actions for ${profile.name}`}>
<MoreHorizontal className="h-4 w-4" /> <MoreHorizontal className="h-4 w-4" />
</Button> </Button>
</DropdownMenuTrigger> </DropdownMenuTrigger>
+1 -1
View File
@@ -1,5 +1,5 @@
import * as React from 'react';
import { Check } from 'lucide-react'; import { Check } from 'lucide-react';
import * as React from 'react';
import { cn } from '@/lib/utils/cn'; import { cn } from '@/lib/utils/cn';
export interface CheckboxProps { export interface CheckboxProps {
+16
View File
@@ -1,4 +1,5 @@
@import "tailwindcss" source("."); @import "tailwindcss" source(".");
@import "loaders.css/loaders.min.css";
@theme { @theme {
--radius-sm: calc(var(--radius) - 4px); --radius-sm: calc(var(--radius) - 4px);
@@ -155,3 +156,18 @@
animation: fadeIn 0.5s ease-out 0.15s forwards; animation: fadeIn 0.5s ease-out 0.15s forwards;
opacity: 0; opacity: 0;
} }
/* react-loaders */
.line-scale-pulse-out-rapid > div,
.line-scale > div {
background-color: hsl(var(--accent)) !important;
}
.loader-hidden {
display: block;
}
.loader-hidden > div > div {
animation-play-state: paused !important;
background-color: hsl(var(--muted-foreground)) !important;
}
+32
View File
@@ -200,6 +200,12 @@ class ApiClient {
}); });
} }
async retryGeneration(generationId: string): Promise<GenerationResponse> {
return this.request<GenerationResponse>(`/generate/${generationId}/retry`, {
method: 'POST',
});
}
// History // History
async listHistory(query?: HistoryQuery): Promise<HistoryListResponse> { async listHistory(query?: HistoryQuery): Promise<HistoryListResponse> {
const params = new URLSearchParams(); const params = new URLSearchParams();
@@ -278,6 +284,11 @@ class ApiClient {
return response.json(); return response.json();
} }
// Generation status SSE
getGenerationStatusUrl(generationId: string): string {
return `${this.getBaseUrl()}/generate/${generationId}/status`;
}
// Audio // Audio
getAudioUrl(audioId: string): string { getAudioUrl(audioId: string): string {
return `${this.getBaseUrl()}/audio/${audioId}`; return `${this.getBaseUrl()}/audio/${audioId}`;
@@ -316,6 +327,21 @@ class ApiClient {
return this.request<ModelStatusListResponse>('/models/status'); return this.request<ModelStatusListResponse>('/models/status');
} }
async getModelsCacheDir(): Promise<{ path: string }> {
return this.request<{ path: string }>('/models/cache-dir');
}
async migrateModels(destination: string): Promise<{ source: string; destination: string }> {
return this.request('/models/migrate', {
method: 'POST',
body: JSON.stringify({ destination }),
});
}
getMigrationProgressUrl(): string {
return `${this.getBaseUrl()}/models/migrate/progress`;
}
async triggerModelDownload(modelName: string): Promise<{ message: string }> { async triggerModelDownload(modelName: string): Promise<{ message: string }> {
console.log( console.log(
'[API] triggerModelDownload called for:', '[API] triggerModelDownload called for:',
@@ -337,6 +363,12 @@ class ApiClient {
}); });
} }
async unloadModel(modelName: string): Promise<{ message: string }> {
return this.request<{ message: string }>(`/models/${modelName}/unload`, {
method: 'POST',
});
}
async cancelDownload(modelName: string): Promise<{ message: string }> { async cancelDownload(modelName: string): Promise<{ message: string }> {
return this.request<{ message: string }>('/models/download/cancel', { return this.request<{ message: string }>('/models/download/cancel', {
method: 'POST', method: 'POST',
+32 -3
View File
@@ -34,8 +34,11 @@ export interface GenerationRequest {
language: LanguageCode; language: LanguageCode;
seed?: number; seed?: number;
model_size?: '1.7B' | '0.6B'; model_size?: '1.7B' | '0.6B';
engine?: 'qwen' | 'luxtts'; engine?: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo';
instruct?: string; instruct?: string;
max_chunk_chars?: number;
crossfade_ms?: number;
normalize?: boolean;
} }
export interface GenerationResponse { export interface GenerationResponse {
@@ -43,9 +46,14 @@ export interface GenerationResponse {
profile_id: string; profile_id: string;
text: string; text: string;
language: string; language: string;
audio_path: string; audio_path?: string;
duration: number; duration?: number;
seed?: number; seed?: number;
instruct?: string;
engine?: string;
model_size?: string;
status: 'generating' | 'completed' | 'failed';
error?: string;
created_at: string; created_at: string;
} }
@@ -119,12 +127,29 @@ export interface ModelProgress {
export interface ModelStatus { export interface ModelStatus {
model_name: string; model_name: string;
display_name: string; display_name: string;
hf_repo_id?: string; // HuggingFace repository ID
downloaded: boolean; downloaded: boolean;
downloading: boolean; // True if download is in progress downloading: boolean; // True if download is in progress
size_mb?: number; size_mb?: number;
loaded: boolean; loaded: boolean;
} }
export interface HuggingFaceModelInfo {
id: string;
author: string;
lastModified: string;
pipeline_tag?: string;
library_name?: string;
downloads: number;
likes: number;
tags: string[];
cardData?: {
license?: string;
language?: string[];
pipeline_tag?: string;
};
}
export interface ModelStatusListResponse { export interface ModelStatusListResponse {
models: ModelStatus[]; models: ModelStatus[];
} }
@@ -138,6 +163,10 @@ export interface ActiveDownloadTask {
status: string; status: string;
started_at: string; started_at: string;
error?: string; error?: string;
progress?: number; // 0-100 percentage
current?: number; // bytes downloaded
total?: number; // total bytes
filename?: string; // current file being downloaded
} }
export interface ActiveGenerationTask { export interface ActiveGenerationTask {
+72 -12
View File
@@ -1,26 +1,86 @@
/** /**
* Supported languages for Qwen3-TTS * Supported languages for voice generation, per engine.
* Based on: https://github.com/QwenLM/Qwen3-TTS *
* Qwen3-TTS supports 10 languages.
* LuxTTS is English-only.
* Chatterbox Multilingual supports 23 languages.
* Chatterbox Turbo is English-only.
*/ */
export const SUPPORTED_LANGUAGES = { /** All languages that any engine supports. */
zh: 'Chinese', export const ALL_LANGUAGES = {
ar: 'Arabic',
da: 'Danish',
de: 'German',
el: 'Greek',
en: 'English', en: 'English',
es: 'Spanish',
fi: 'Finnish',
fr: 'French',
he: 'Hebrew',
hi: 'Hindi',
it: 'Italian',
ja: 'Japanese', ja: 'Japanese',
ko: 'Korean', ko: 'Korean',
de: 'German', ms: 'Malay',
fr: 'French', nl: 'Dutch',
ru: 'Russian', no: 'Norwegian',
pl: 'Polish',
pt: 'Portuguese', pt: 'Portuguese',
es: 'Spanish', ru: 'Russian',
it: 'Italian', sv: 'Swedish',
sw: 'Swahili',
tr: 'Turkish',
zh: 'Chinese',
} as const; } as const;
export type LanguageCode = keyof typeof SUPPORTED_LANGUAGES; export type LanguageCode = keyof typeof ALL_LANGUAGES;
export const LANGUAGE_CODES = Object.keys(SUPPORTED_LANGUAGES) as LanguageCode[]; /** Per-engine supported language codes. */
export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
qwen: ['zh', 'en', 'ja', 'ko', 'de', 'fr', 'ru', 'pt', 'es', 'it'],
luxtts: ['en'],
chatterbox: [
'ar',
'da',
'de',
'el',
'en',
'es',
'fi',
'fr',
'he',
'hi',
'it',
'ja',
'ko',
'ms',
'nl',
'no',
'pl',
'pt',
'ru',
'sv',
'sw',
'tr',
'zh',
],
chatterbox_turbo: ['en'],
} as const;
/** Helper: get language options for a given engine. */
export function getLanguageOptionsForEngine(engine: string) {
const codes = ENGINE_LANGUAGES[engine] ?? ENGINE_LANGUAGES.qwen;
return codes.map((code) => ({
value: code,
label: ALL_LANGUAGES[code],
}));
}
// ── Backwards-compatible exports used elsewhere ──────────────────────
export const SUPPORTED_LANGUAGES = ALL_LANGUAGES;
export const LANGUAGE_CODES = Object.keys(ALL_LANGUAGES) as LanguageCode[];
export const LANGUAGE_OPTIONS = LANGUAGE_CODES.map((code) => ({ export const LANGUAGE_OPTIONS = LANGUAGE_CODES.map((code) => ({
value: code, value: code,
label: SUPPORTED_LANGUAGES[code], label: ALL_LANGUAGES[code],
})); }));
+33 -21
View File
@@ -8,15 +8,15 @@ import { LANGUAGE_CODES, type LanguageCode } from '@/lib/constants/languages';
import { useGeneration } from '@/lib/hooks/useGeneration'; import { useGeneration } from '@/lib/hooks/useGeneration';
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast'; import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
import { useGenerationStore } from '@/stores/generationStore'; import { useGenerationStore } from '@/stores/generationStore';
import { usePlayerStore } from '@/stores/playerStore'; import { useServerStore } from '@/stores/serverStore';
const generationSchema = z.object({ const generationSchema = z.object({
text: z.string().min(1, 'Text is required').max(5000), text: z.string().min(1, 'Text is required').max(50000),
language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]), language: z.enum(LANGUAGE_CODES as [LanguageCode, ...LanguageCode[]]),
seed: z.number().int().optional(), seed: z.number().int().optional(),
modelSize: z.enum(['1.7B', '0.6B']).optional(), modelSize: z.enum(['1.7B', '0.6B']).optional(),
instruct: z.string().max(500).optional(), instruct: z.string().max(500).optional(),
engine: z.enum(['qwen', 'luxtts']).optional(), engine: z.enum(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo']).optional(),
}); });
export type GenerationFormValues = z.infer<typeof generationSchema>; export type GenerationFormValues = z.infer<typeof generationSchema>;
@@ -29,8 +29,10 @@ interface UseGenerationFormOptions {
export function useGenerationForm(options: UseGenerationFormOptions = {}) { export function useGenerationForm(options: UseGenerationFormOptions = {}) {
const { toast } = useToast(); const { toast } = useToast();
const generation = useGeneration(); const generation = useGeneration();
const setAudioWithAutoPlay = usePlayerStore((state) => state.setAudioWithAutoPlay); const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
const setIsGenerating = useGenerationStore((state) => state.setIsGenerating); const maxChunkChars = useServerStore((state) => state.maxChunkChars);
const crossfadeMs = useServerStore((state) => state.crossfadeMs);
const normalizeAudio = useServerStore((state) => state.normalizeAudio);
const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null); const [downloadingModelName, setDownloadingModelName] = useState<string | null>(null);
const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null); const [downloadingDisplayName, setDownloadingDisplayName] = useState<string | null>(null);
@@ -67,17 +69,27 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
} }
try { try {
setIsGenerating(true);
const engine = data.engine || 'qwen'; const engine = data.engine || 'qwen';
const modelName = engine === 'luxtts' ? 'luxtts' : `qwen-tts-${data.modelSize}`; const modelName =
engine === 'luxtts'
? 'luxtts'
: engine === 'chatterbox'
? 'chatterbox-tts'
: engine === 'chatterbox_turbo'
? 'chatterbox-turbo'
: `qwen-tts-${data.modelSize}`;
const displayName = const displayName =
engine === 'luxtts' engine === 'luxtts'
? 'LuxTTS' ? 'LuxTTS'
: data.modelSize === '1.7B' : engine === 'chatterbox'
? 'Qwen TTS 1.7B' ? 'Chatterbox TTS'
: 'Qwen TTS 0.6B'; : engine === 'chatterbox_turbo'
? 'Chatterbox Turbo'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
// Check if model needs downloading
try { try {
const modelStatus = await apiClient.getModelStatus(); const modelStatus = await apiClient.getModelStatus();
const model = modelStatus.models.find((m) => m.model_name === modelName); const model = modelStatus.models.find((m) => m.model_name === modelName);
@@ -90,24 +102,25 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
console.error('Failed to check model status:', error); console.error('Failed to check model status:', error);
} }
const isQwen = engine === 'qwen';
// This now returns immediately with status="generating"
const result = await generation.mutateAsync({ const result = await generation.mutateAsync({
profile_id: selectedProfileId, profile_id: selectedProfileId,
text: data.text, text: data.text,
language: data.language, language: data.language,
seed: data.seed, seed: data.seed,
model_size: engine === 'luxtts' ? undefined : data.modelSize, model_size: isQwen ? data.modelSize : undefined,
engine, engine,
instruct: engine === 'luxtts' ? undefined : data.instruct || undefined, instruct: isQwen ? data.instruct || undefined : undefined,
max_chunk_chars: maxChunkChars,
crossfade_ms: crossfadeMs,
normalize: normalizeAudio,
}); });
toast({ // Track this generation for SSE status updates
title: 'Generation complete!', addPendingGeneration(result.id);
description: `Audio generated (${result.duration.toFixed(2)}s)`,
});
const audioUrl = apiClient.getAudioUrl(result.id);
setAudioWithAutoPlay(audioUrl, result.id, selectedProfileId, data.text.substring(0, 50));
// Reset form immediately — user can start typing again
form.reset({ form.reset({
text: '', text: '',
language: data.language, language: data.language,
@@ -124,7 +137,6 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
variant: 'destructive', variant: 'destructive',
}); });
} finally { } finally {
setIsGenerating(false);
setDownloadingModelName(null); setDownloadingModelName(null);
setDownloadingDisplayName(null); setDownloadingDisplayName(null);
} }
+154
View File
@@ -0,0 +1,154 @@
import { useQueryClient } from '@tanstack/react-query';
import { useEffect, useRef } from 'react';
import { useToast } from '@/components/ui/use-toast';
import { apiClient } from '@/lib/api/client';
import { useGenerationStore } from '@/stores/generationStore';
import { usePlayerStore } from '@/stores/playerStore';
import { useServerStore } from '@/stores/serverStore';
interface GenerationStatusEvent {
id: string;
status: 'generating' | 'completed' | 'failed' | 'not_found';
duration?: number;
error?: string;
}
/**
* Subscribes to SSE for all pending generations. When a generation completes,
* invalidates the history query, removes it from pending, and auto-plays
* if the player is idle.
*/
export function useGenerationProgress() {
const queryClient = useQueryClient();
const { toast } = useToast();
const pendingIds = useGenerationStore((s) => s.pendingGenerationIds);
const removePendingGeneration = useGenerationStore((s) => s.removePendingGeneration);
const removePendingStoryAdd = useGenerationStore((s) => s.removePendingStoryAdd);
const isPlaying = usePlayerStore((s) => s.isPlaying);
const setAudioWithAutoPlay = usePlayerStore((s) => s.setAudioWithAutoPlay);
const autoplayOnGenerate = useServerStore((s) => s.autoplayOnGenerate);
// Keep refs to avoid stale closures in EventSource handlers
const isPlayingRef = useRef(isPlaying);
const autoplayRef = useRef(autoplayOnGenerate);
isPlayingRef.current = isPlaying;
autoplayRef.current = autoplayOnGenerate;
// Track active EventSource instances
const eventSourcesRef = useRef<Map<string, EventSource>>(new Map());
// Unmount-only cleanup — close all SSE connections when the hook is torn down
useEffect(() => {
const sources = eventSourcesRef.current;
return () => {
for (const source of sources.values()) {
source.close();
}
sources.clear();
};
}, []);
useEffect(() => {
const currentSources = eventSourcesRef.current;
// Close SSE connections for IDs no longer pending
for (const [id, source] of currentSources.entries()) {
if (!pendingIds.has(id)) {
source.close();
currentSources.delete(id);
}
}
// Open SSE connections for new pending IDs
for (const id of pendingIds) {
if (currentSources.has(id)) continue;
const url = apiClient.getGenerationStatusUrl(id);
const source = new EventSource(url);
source.onmessage = (event) => {
try {
const data: GenerationStatusEvent = JSON.parse(event.data);
if (data.status === 'completed') {
source.close();
currentSources.delete(id);
removePendingGeneration(id);
// Refresh history to pick up the completed generation
queryClient.invalidateQueries({ queryKey: ['history'] });
// If this generation was queued for a story, add it now
const storyId = removePendingStoryAdd(id);
if (storyId) {
apiClient
.addStoryItem(storyId, { generation_id: id })
.then(() => {
queryClient.invalidateQueries({ queryKey: ['stories'] });
queryClient.invalidateQueries({ queryKey: ['stories', storyId] });
toast({
title: 'Added to story',
description: data.duration
? `Audio generated (${data.duration.toFixed(2)}s) and added to story`
: 'Audio generated and added to story',
});
})
.catch(() => {
toast({
title: 'Generation complete',
description: 'Audio generated but failed to add to story',
variant: 'destructive',
});
});
} else {
// toast({
// title: 'Generation complete!',
// description: data.duration
// ? `Audio generated (${data.duration.toFixed(2)}s)`
// : 'Audio generated',
// });
}
// Auto-play if enabled and nothing is currently playing
if (autoplayRef.current && !isPlayingRef.current) {
const genAudioUrl = apiClient.getAudioUrl(id);
setAudioWithAutoPlay(genAudioUrl, id, '', '');
}
} else if (data.status === 'failed' || data.status === 'not_found') {
source.close();
currentSources.delete(id);
removePendingGeneration(id);
removePendingStoryAdd(id);
queryClient.invalidateQueries({ queryKey: ['history'] });
toast({
title: data.status === 'not_found' ? 'Generation not found' : 'Generation failed',
description: data.error || 'An error occurred during generation',
variant: 'destructive',
});
}
} catch {
// Ignore parse errors from heartbeats etc
}
};
source.onerror = () => {
// EventSource auto-reconnects, but if we get repeated errors
// just clean up
source.close();
currentSources.delete(id);
removePendingGeneration(id);
};
currentSources.set(id, source);
}
}, [
pendingIds,
removePendingGeneration,
removePendingStoryAdd,
queryClient,
toast,
setAudioWithAutoPlay,
]);
}
+12 -12
View File
@@ -1,23 +1,23 @@
import { useCallback, useEffect, useRef, useState } from 'react'; import { useCallback, useEffect, useRef, useState } from 'react';
import { apiClient } from '@/lib/api/client'; import { apiClient } from '@/lib/api/client';
import { useGenerationStore } from '@/stores/generationStore';
import type { ActiveDownloadTask } from '@/lib/api/types'; import type { ActiveDownloadTask } from '@/lib/api/types';
import { useGenerationStore } from '@/stores/generationStore';
// Polling interval in milliseconds // Polling interval in milliseconds
const POLL_INTERVAL = 2000; const POLL_INTERVAL = 30000;
/** /**
* Hook to monitor active tasks (downloads and generations). * Hook to monitor active tasks (downloads and generations).
* Polls the server periodically to catch downloads triggered from anywhere * Polls the server periodically to catch downloads triggered from anywhere
* (transcription, generation, explicit download, etc.). * (transcription, generation, explicit download, etc.).
* *
* Returns the active downloads so components can render download toasts. * Returns the active downloads so components can render download toasts.
*/ */
export function useRestoreActiveTasks() { export function useRestoreActiveTasks() {
const [activeDownloads, setActiveDownloads] = useState<ActiveDownloadTask[]>([]); const [activeDownloads, setActiveDownloads] = useState<ActiveDownloadTask[]>([]);
const setIsGenerating = useGenerationStore((state) => state.setIsGenerating);
const setActiveGenerationId = useGenerationStore((state) => state.setActiveGenerationId); const setActiveGenerationId = useGenerationStore((state) => state.setActiveGenerationId);
const addPendingGeneration = useGenerationStore((state) => state.addPendingGeneration);
// Track which downloads we've seen to detect new ones // Track which downloads we've seen to detect new ones
const seenDownloadsRef = useRef<Set<string>>(new Set()); const seenDownloadsRef = useRef<Set<string>>(new Set());
@@ -25,15 +25,15 @@ export function useRestoreActiveTasks() {
try { try {
const tasks = await apiClient.getActiveTasks(); const tasks = await apiClient.getActiveTasks();
// Update generation state // Restore pending generations (e.g., after page refresh)
if (tasks.generations.length > 0) { if (tasks.generations.length > 0) {
setIsGenerating(true);
setActiveGenerationId(tasks.generations[0].task_id); setActiveGenerationId(tasks.generations[0].task_id);
for (const gen of tasks.generations) {
addPendingGeneration(gen.task_id);
}
} else { } else {
// Only clear if we were tracking a generation
const currentId = useGenerationStore.getState().activeGenerationId; const currentId = useGenerationStore.getState().activeGenerationId;
if (currentId) { if (currentId) {
setIsGenerating(false);
setActiveGenerationId(null); setActiveGenerationId(null);
} }
} }
@@ -41,14 +41,14 @@ export function useRestoreActiveTasks() {
// Update active downloads // Update active downloads
// Keep track of all active downloads (including new ones) // Keep track of all active downloads (including new ones)
const currentDownloadNames = new Set(tasks.downloads.map((d) => d.model_name)); const currentDownloadNames = new Set(tasks.downloads.map((d) => d.model_name));
// Remove completed downloads from our seen set // Remove completed downloads from our seen set
for (const name of seenDownloadsRef.current) { for (const name of seenDownloadsRef.current) {
if (!currentDownloadNames.has(name)) { if (!currentDownloadNames.has(name)) {
seenDownloadsRef.current.delete(name); seenDownloadsRef.current.delete(name);
} }
} }
// Add new downloads to seen set // Add new downloads to seen set
for (const download of tasks.downloads) { for (const download of tasks.downloads) {
seenDownloadsRef.current.add(download.model_name); seenDownloadsRef.current.add(download.model_name);
@@ -59,7 +59,7 @@ export function useRestoreActiveTasks() {
// Silently fail - server might be temporarily unavailable // Silently fail - server might be temporarily unavailable
console.debug('Failed to fetch active tasks:', error); console.debug('Failed to fetch active tasks:', error);
} }
}, [setIsGenerating, setActiveGenerationId]); }, [setActiveGenerationId, addPendingGeneration]);
useEffect(() => { useEffect(() => {
// Fetch immediately on mount // Fetch immediately on mount
+16 -1
View File
@@ -21,10 +21,25 @@ export function formatDate(date: string | Date): string {
} else { } else {
dateObj = date; dateObj = date;
} }
return formatDistance(dateObj, new Date(), { addSuffix: true }).replace(/^about /i, ''); return formatDistance(dateObj, new Date(), { addSuffix: true }).replace(/^about /i, '');
} }
const ENGINE_DISPLAY_NAMES: Record<string, string> = {
qwen: 'Qwen',
luxtts: 'LuxTTS',
chatterbox: 'Chatterbox',
chatterbox_turbo: 'Chatterbox Turbo',
};
export function formatEngineName(engine?: string, modelSize?: string): string {
const name = ENGINE_DISPLAY_NAMES[engine ?? 'qwen'] ?? engine ?? 'Qwen';
if (engine === 'qwen' && modelSize) {
return `${name} ${modelSize}`;
}
return name;
}
export function formatFileSize(bytes: number): string { export function formatFileSize(bytes: number): string {
if (bytes === 0) return '0 Bytes'; if (bytes === 0) return '0 Bytes';
const k = 1024; const k = 1024;
+2 -2
View File
@@ -49,9 +49,9 @@ export interface PlatformAudio {
} }
export interface PlatformLifecycle { export interface PlatformLifecycle {
startServer(remote?: boolean): Promise<string>; startServer(remote?: boolean, modelsDir?: string | null): Promise<string>;
stopServer(): Promise<void>; stopServer(): Promise<void>;
restartServer(): Promise<string>; restartServer(modelsDir?: string | null): Promise<string>;
setKeepServerRunning(keep: boolean): Promise<void>; setKeepServerRunning(keep: boolean): Promise<void>;
setupWindowCloseHandler(): Promise<void>; setupWindowCloseHandler(): Promise<void>;
onServerReady?: () => void; onServerReady?: () => void;
+5
View File
@@ -8,8 +8,10 @@ import { Sidebar } from '@/components/Sidebar';
import { StoriesTab } from '@/components/StoriesTab/StoriesTab'; import { StoriesTab } from '@/components/StoriesTab/StoriesTab';
import { Toaster } from '@/components/ui/toaster'; import { Toaster } from '@/components/ui/toaster';
import { VoicesTab } from '@/components/VoicesTab/VoicesTab'; import { VoicesTab } from '@/components/VoicesTab/VoicesTab';
import { useGenerationProgress } from '@/lib/hooks/useGenerationProgress';
import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast'; import { useModelDownloadToast } from '@/lib/hooks/useModelDownloadToast';
import { MODEL_DISPLAY_NAMES, useRestoreActiveTasks } from '@/lib/hooks/useRestoreActiveTasks'; import { MODEL_DISPLAY_NAMES, useRestoreActiveTasks } from '@/lib/hooks/useRestoreActiveTasks';
// Simple platform check that works in both web and Tauri // Simple platform check that works in both web and Tauri
const isMacOS = () => navigator.platform.toLowerCase().includes('mac'); const isMacOS = () => navigator.platform.toLowerCase().includes('mac');
@@ -18,6 +20,9 @@ function RootLayout() {
// Monitor active downloads/generations and show toasts for them // Monitor active downloads/generations and show toasts for them
const activeDownloads = useRestoreActiveTasks(); const activeDownloads = useRestoreActiveTasks();
// Subscribe to SSE for pending generations — handles completion, auto-play, and history refresh
useGenerationProgress();
return ( return (
<AppFrame> <AppFrame>
<div className="flex flex-1 min-h-0 overflow-hidden"> <div className="flex flex-1 min-h-0 overflow-hidden">
+47 -4
View File
@@ -1,15 +1,58 @@
import { create } from 'zustand'; import { create } from 'zustand';
interface GenerationState { interface GenerationState {
/** IDs of generations currently in progress */
pendingGenerationIds: Set<string>;
/** Whether any generation is in progress (derived from pendingGenerationIds) */
isGenerating: boolean; isGenerating: boolean;
activeGenerationId: string | null; /** Map of generationId → storyId for deferred story additions */
setIsGenerating: (generating: boolean) => void; pendingStoryAdds: Map<string, string>;
addPendingGeneration: (id: string) => void;
removePendingGeneration: (id: string) => void;
addPendingStoryAdd: (generationId: string, storyId: string) => void;
removePendingStoryAdd: (generationId: string) => string | undefined;
setActiveGenerationId: (id: string | null) => void; setActiveGenerationId: (id: string | null) => void;
activeGenerationId: string | null;
} }
export const useGenerationStore = create<GenerationState>((set) => ({ export const useGenerationStore = create<GenerationState>((set, get) => ({
pendingGenerationIds: new Set(),
isGenerating: false, isGenerating: false,
activeGenerationId: null, activeGenerationId: null,
setIsGenerating: (generating) => set({ isGenerating: generating }), pendingStoryAdds: new Map(),
addPendingGeneration: (id) =>
set((state) => {
const next = new Set(state.pendingGenerationIds);
next.add(id);
return { pendingGenerationIds: next, isGenerating: true };
}),
removePendingGeneration: (id) =>
set((state) => {
const next = new Set(state.pendingGenerationIds);
next.delete(id);
return { pendingGenerationIds: next, isGenerating: next.size > 0 };
}),
addPendingStoryAdd: (generationId, storyId) =>
set((state) => {
const next = new Map(state.pendingStoryAdds);
next.set(generationId, storyId);
return { pendingStoryAdds: next };
}),
removePendingStoryAdd: (generationId) => {
const storyId = get().pendingStoryAdds.get(generationId);
if (storyId) {
set((state) => {
const next = new Map(state.pendingStoryAdds);
next.delete(generationId);
return { pendingStoryAdds: next };
});
}
return storyId;
},
setActiveGenerationId: (id) => set({ activeGenerationId: id }), setActiveGenerationId: (id) => set({ activeGenerationId: id }),
})); }));
+30
View File
@@ -13,6 +13,21 @@ interface ServerStore {
keepServerRunningOnClose: boolean; keepServerRunningOnClose: boolean;
setKeepServerRunningOnClose: (keepRunning: boolean) => void; setKeepServerRunningOnClose: (keepRunning: boolean) => void;
maxChunkChars: number;
setMaxChunkChars: (value: number) => void;
crossfadeMs: number;
setCrossfadeMs: (value: number) => void;
normalizeAudio: boolean;
setNormalizeAudio: (value: boolean) => void;
autoplayOnGenerate: boolean;
setAutoplayOnGenerate: (value: boolean) => void;
customModelsDir: string | null;
setCustomModelsDir: (dir: string | null) => void;
} }
export const useServerStore = create<ServerStore>()( export const useServerStore = create<ServerStore>()(
@@ -29,6 +44,21 @@ export const useServerStore = create<ServerStore>()(
keepServerRunningOnClose: false, keepServerRunningOnClose: false,
setKeepServerRunningOnClose: (keepRunning) => set({ keepServerRunningOnClose: keepRunning }), setKeepServerRunningOnClose: (keepRunning) => set({ keepServerRunningOnClose: keepRunning }),
maxChunkChars: 800,
setMaxChunkChars: (value) => set({ maxChunkChars: value }),
crossfadeMs: 50,
setCrossfadeMs: (value) => set({ crossfadeMs: value }),
normalizeAudio: true,
setNormalizeAudio: (value) => set({ normalizeAudio: value }),
autoplayOnGenerate: true,
setAutoplayOnGenerate: (value) => set({ autoplayOnGenerate: value }),
customModelsDir: null,
setCustomModelsDir: (dir) => set({ customModelsDir: dir }),
}), }),
{ {
name: 'voicebox-server', name: 'voicebox-server',
+12 -9
View File
@@ -334,18 +334,21 @@ python -m backend.main --host 0.0.0.0 --port 8000
## Usage Examples ## Usage Examples
The desktop app, web client, and current development workflow use `http://localhost:17493` by default.
If you launch the backend manually with a different host or port, substitute that address in the examples below.
### Creating a Voice Profile ### Creating a Voice Profile
```bash ```bash
# 1. Create profile # 1. Create profile
curl -X POST http://localhost:8000/profiles \ curl -X POST http://localhost:17493/profiles \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"name": "My Voice", "language": "en"}' -d '{"name": "My Voice", "language": "en"}'
# Response: {"id": "abc-123", ...} # Response: {"id": "abc-123", ...}
# 2. Add sample # 2. Add sample
curl -X POST http://localhost:8000/profiles/abc-123/samples \ curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "file=@sample.wav" \ -F "file=@sample.wav" \
-F "reference_text=This is my voice sample" -F "reference_text=This is my voice sample"
``` ```
@@ -353,7 +356,7 @@ curl -X POST http://localhost:8000/profiles/abc-123/samples \
### Generating Speech ### Generating Speech
```bash ```bash
curl -X POST http://localhost:8000/generate \ curl -X POST http://localhost:17493/generate \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"profile_id": "abc-123", "profile_id": "abc-123",
@@ -365,13 +368,13 @@ curl -X POST http://localhost:8000/generate \
# Response: {"id": "gen-456", "audio_path": "/path/to/audio.wav", ...} # Response: {"id": "gen-456", "audio_path": "/path/to/audio.wav", ...}
# Download audio # Download audio
curl http://localhost:8000/audio/gen-456 -o output.wav curl http://localhost:17493/audio/gen-456 -o output.wav
``` ```
### Transcribing Audio ### Transcribing Audio
```bash ```bash
curl -X POST http://localhost:8000/transcribe \ curl -X POST http://localhost:17493/transcribe \
-F "file=@audio.wav" \ -F "file=@audio.wav" \
-F "language=en" -F "language=en"
@@ -386,12 +389,12 @@ Add multiple samples to a profile for better quality:
```bash ```bash
# Add first sample # Add first sample
curl -X POST http://localhost:8000/profiles/abc-123/samples \ curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "file=@sample1.wav" \ -F "file=@sample1.wav" \
-F "reference_text=First sample" -F "reference_text=First sample"
# Add second sample # Add second sample
curl -X POST http://localhost:8000/profiles/abc-123/samples \ curl -X POST http://localhost:17493/profiles/abc-123/samples \
-F "file=@sample2.wav" \ -F "file=@sample2.wav" \
-F "reference_text=Second sample" -F "reference_text=Second sample"
@@ -412,10 +415,10 @@ Models are lazy-loaded and can be manually unloaded:
```bash ```bash
# Unload TTS model # Unload TTS model
curl -X POST http://localhost:8000/models/unload curl -X POST http://localhost:17493/models/unload
# Load specific model size # Load specific model size
curl -X POST "http://localhost:8000/models/load?model_size=0.6B" curl -X POST "http://localhost:17493/models/load?model_size=0.6B"
``` ```
## Error Handling ## Error Handling
+8
View File
@@ -121,6 +121,8 @@ _stt_backend: Optional[STTBackend] = None
TTS_ENGINES = { TTS_ENGINES = {
"qwen": "Qwen TTS", "qwen": "Qwen TTS",
"luxtts": "LuxTTS", "luxtts": "LuxTTS",
"chatterbox": "Chatterbox TTS",
"chatterbox_turbo": "Chatterbox Turbo",
} }
@@ -167,6 +169,12 @@ def get_tts_backend_for_engine(engine: str) -> TTSBackend:
elif engine == "luxtts": elif engine == "luxtts":
from .luxtts_backend import LuxTTSBackend from .luxtts_backend import LuxTTSBackend
backend = LuxTTSBackend() backend = LuxTTSBackend()
elif engine == "chatterbox":
from .chatterbox_backend import ChatterboxTTSBackend
backend = ChatterboxTTSBackend()
elif engine == "chatterbox_turbo":
from .chatterbox_turbo_backend import ChatterboxTurboTTSBackend
backend = ChatterboxTurboTTSBackend()
else: else:
raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}") raise ValueError(f"Unknown TTS engine: {engine}. Supported: {list(TTS_ENGINES.keys())}")
+360
View File
@@ -0,0 +1,360 @@
"""
Chatterbox TTS backend implementation.
Wraps ChatterboxMultilingualTTS from chatterbox-tts for zero-shot
voice cloning. Supports 23 languages including Hebrew. Forces CPU
on macOS due to known MPS tensor issues.
"""
import asyncio
import logging
import platform
import threading
from pathlib import Path
from typing import ClassVar, List, Optional, Tuple
import numpy as np
from . import TTSBackend
from ..utils.audio import normalize_audio, load_audio
from ..utils.progress import get_progress_manager
from ..utils.tasks import get_task_manager
logger = logging.getLogger(__name__)
CHATTERBOX_HF_REPO = "ResembleAI/chatterbox"
# Files that must be present for the multilingual model
_MTL_WEIGHT_FILES = [
"t3_mtl23ls_v2.safetensors",
"s3gen.pt",
"ve.pt",
]
class ChatterboxTTSBackend:
"""Chatterbox Multilingual TTS backend for voice cloning."""
# Class-level lock for torch.load monkey-patching
_load_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self):
self.model = None
self.model_size = "default"
self._device = None
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
"""Get the best available device. Forces CPU on macOS (MPS issue)."""
if platform.system() == "Darwin":
return "cpu"
try:
import torch
if torch.cuda.is_available():
return "cuda"
except ImportError:
pass
return "cpu"
def is_loaded(self) -> bool:
return self.model is not None
def _get_model_path(self, model_size: str = "default") -> str:
return CHATTERBOX_HF_REPO
def _is_model_cached(self, model_size: str = "default") -> bool:
"""Check if the Chatterbox multilingual model is cached locally."""
try:
from huggingface_hub import constants as hf_constants
repo_cache = Path(hf_constants.HF_HUB_CACHE) / (
"models--" + CHATTERBOX_HF_REPO.replace("/", "--")
)
if not repo_cache.exists():
return False
blobs_dir = repo_cache / "blobs"
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
return False
# Check for multilingual weight files
snapshots_dir = repo_cache / "snapshots"
if snapshots_dir.exists():
for fname in _MTL_WEIGHT_FILES:
if not any(snapshots_dir.rglob(fname)):
return False
return True
return False
except Exception as e:
logger.warning(f"Error checking Chatterbox cache: {e}")
return False
async def load_model(self, model_size: str = "default") -> None:
"""Load the Chatterbox multilingual model."""
if self.model is not None:
return
async with self._model_load_lock:
if self.model is not None:
return
await asyncio.to_thread(self._load_model_sync)
def _load_model_sync(self):
"""Synchronous model loading."""
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = "chatterbox-tts"
is_cached = self._is_model_cached()
# Set up HF progress tracking (intercepts tqdm for file-level progress)
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
tracker_context = tracker.patch_download()
tracker_context.__enter__()
if not is_cached:
task_manager.start_download(model_name)
progress_manager.update_progress(
model_name=model_name,
current=0,
total=0,
filename="Connecting to HuggingFace...",
status="downloading",
)
try:
device = self._get_device()
self._device = device
logger.info(f"Loading Chatterbox Multilingual TTS on {device}...")
import torch
from chatterbox.mtl_tts import ChatterboxMultilingualTTS
# Load into a local variable first, apply all patches, then
# assign to self.model. This avoids leaving a half-initialised
# model on self.model if any patch step raises an exception.
#
# Monkey-patch torch.load for CPU loading. The model's .pt files
# were saved on CUDA; from_pretrained() doesn't pass map_location
# so loading on CPU fails without this.
try:
if device == "cpu":
_orig_torch_load = torch.load
def _patched_load(*args, **kwargs):
kwargs.setdefault("map_location", "cpu")
return _orig_torch_load(*args, **kwargs)
with ChatterboxTTSBackend._load_lock:
torch.load = _patched_load
try:
model = ChatterboxMultilingualTTS.from_pretrained(
device=device,
)
finally:
torch.load = _orig_torch_load
else:
model = ChatterboxMultilingualTTS.from_pretrained(
device=device,
)
finally:
tracker_context.__exit__(None, None, None)
# Fix: transformers >= 4.36 defaults LlamaModel to sdpa attention
# which doesn't support output_attentions=True (needed by
# Chatterbox's AlignmentStreamAnalyzer). Force eager attention.
t3_tfmr = model.t3.tfmr
if hasattr(t3_tfmr, "config") and hasattr(
t3_tfmr.config, "_attn_implementation"
):
t3_tfmr.config._attn_implementation = "eager"
for layer in getattr(t3_tfmr, "layers", []):
if hasattr(layer, "self_attn"):
layer.self_attn._attn_implementation = "eager"
if not is_cached:
progress_manager.mark_complete(model_name)
task_manager.complete_download(model_name)
# Patch float64 → float32 dtype mismatches in upstream chatterbox.
# librosa.load returns float64 numpy; multiple upstream code paths
# convert it to a torch tensor via torch.from_numpy() without
# casting, then matmul it against float32 model weights.
import types
# Patch S3Tokenizer (used by s3gen.tokenizer)
_tokzr = model.s3gen.tokenizer
_orig_log_mel = _tokzr.log_mel_spectrogram.__func__
def _f32_log_mel(self_tokzr, audio, padding=0):
import torch as _torch
if _torch.is_tensor(audio):
audio = audio.float()
return _orig_log_mel(self_tokzr, audio, padding)
_tokzr.log_mel_spectrogram = types.MethodType(_f32_log_mel, _tokzr)
# Patch VoiceEncoder
_ve = model.ve
_orig_ve_forward = _ve.forward.__func__
def _f32_ve_forward(self_ve, mels):
return _orig_ve_forward(self_ve, mels.float())
_ve.forward = types.MethodType(_f32_ve_forward, _ve)
# All patches applied successfully — publish the model
self.model = model
logger.info("Chatterbox Multilingual TTS loaded successfully")
except ImportError as e:
logger.error(
"chatterbox-tts package not found. "
"Install with: pip install chatterbox-tts"
)
if not is_cached:
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
except Exception as e:
logger.error(f"Failed to load Chatterbox: {e}")
if not is_cached:
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
def unload_model(self) -> None:
"""Unload model to free memory."""
if self.model is not None:
device = self._device
del self.model
self.model = None
self._device = None
if device == "cuda":
import torch
torch.cuda.empty_cache()
logger.info("Chatterbox unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
Chatterbox processes reference audio at generation time, so the
prompt just stores the file path. The actual audio is loaded by
model.generate() via audio_prompt_path.
"""
voice_prompt = {
"ref_audio": str(audio_path),
"ref_text": reference_text,
}
return voice_prompt, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""Combine multiple reference samples."""
combined_audio = []
for path in audio_paths:
audio, _sr = load_audio(path)
audio = normalize_audio(audio)
combined_audio.append(audio)
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
combined_text = " ".join(reference_texts)
return mixed, combined_text
# Per-language generation defaults. Lower temp + higher cfg = clearer speech.
_LANG_DEFAULTS: ClassVar[dict] = {
"he": {
"exaggeration": 0.4,
"cfg_weight": 0.7,
"temperature": 0.65,
"repetition_penalty": 2.5,
},
}
_GLOBAL_DEFAULTS: ClassVar[dict] = {
"exaggeration": 0.5,
"cfg_weight": 0.5,
"temperature": 0.8,
"repetition_penalty": 2.0,
}
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio using Chatterbox Multilingual TTS.
Args:
text: Text to synthesize
voice_prompt: Dict with ref_audio path
language: BCP-47 language code
seed: Random seed for reproducibility
instruct: Unused (protocol compatibility)
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model()
ref_audio = voice_prompt.get("ref_audio")
if ref_audio and not Path(ref_audio).exists():
logger.warning(f"Reference audio not found: {ref_audio}")
ref_audio = None
# Merge language-specific defaults with global defaults
lang_defaults = self._LANG_DEFAULTS.get(language, self._GLOBAL_DEFAULTS)
def _generate_sync():
import torch
if seed is not None:
torch.manual_seed(seed)
logger.info(f"[Chatterbox] Generating: lang={language}")
wav = self.model.generate(
text,
language_id=language,
audio_prompt_path=ref_audio,
exaggeration=lang_defaults["exaggeration"],
cfg_weight=lang_defaults["cfg_weight"],
temperature=lang_defaults["temperature"],
repetition_penalty=lang_defaults["repetition_penalty"],
)
# Convert tensor -> numpy
if isinstance(wav, torch.Tensor):
audio = wav.squeeze().cpu().numpy().astype(np.float32)
else:
audio = np.asarray(wav, dtype=np.float32)
sample_rate = (
getattr(self.model, "sr", None)
or getattr(self.model, "sample_rate", 24000)
)
return audio, sample_rate
return await asyncio.to_thread(_generate_sync)
@@ -0,0 +1,345 @@
"""
Chatterbox Turbo TTS backend implementation.
Wraps ChatterboxTurboTTS from chatterbox-tts for fast, English-only
voice cloning with paralinguistic tag support ([laugh], [cough], etc.).
Forces CPU on macOS due to known MPS tensor issues.
"""
import asyncio
import logging
import platform
import threading
from pathlib import Path
from typing import ClassVar, List, Optional, Tuple
import numpy as np
from . import TTSBackend
from ..utils.audio import normalize_audio, load_audio
from ..utils.progress import get_progress_manager
from ..utils.tasks import get_task_manager
logger = logging.getLogger(__name__)
CHATTERBOX_TURBO_HF_REPO = "ResembleAI/chatterbox-turbo"
# Files that must be present for the turbo model
_TURBO_WEIGHT_FILES = [
"t3_turbo_v1.safetensors",
"s3gen_meanflow.safetensors",
"ve.safetensors",
]
class ChatterboxTurboTTSBackend:
"""Chatterbox Turbo TTS backend — fast, English-only, with paralinguistic tags."""
# Class-level lock for torch.load monkey-patching
_load_lock: ClassVar[threading.Lock] = threading.Lock()
def __init__(self):
self.model = None
self.model_size = "default"
self._device = None
self._model_load_lock = asyncio.Lock()
def _get_device(self) -> str:
"""Get the best available device. Forces CPU on macOS (MPS issue)."""
if platform.system() == "Darwin":
return "cpu"
try:
import torch
if torch.cuda.is_available():
return "cuda"
except ImportError:
pass
return "cpu"
def is_loaded(self) -> bool:
return self.model is not None
def _get_model_path(self, model_size: str = "default") -> str:
return CHATTERBOX_TURBO_HF_REPO
def _is_model_cached(self, model_size: str = "default") -> bool:
"""Check if the Chatterbox Turbo model is cached locally."""
try:
from huggingface_hub import constants as hf_constants
repo_cache = Path(hf_constants.HF_HUB_CACHE) / (
"models--" + CHATTERBOX_TURBO_HF_REPO.replace("/", "--")
)
if not repo_cache.exists():
return False
blobs_dir = repo_cache / "blobs"
if blobs_dir.exists() and any(blobs_dir.glob("*.incomplete")):
return False
# Check for turbo weight files
snapshots_dir = repo_cache / "snapshots"
if snapshots_dir.exists():
for fname in _TURBO_WEIGHT_FILES:
if not any(snapshots_dir.rglob(fname)):
return False
return True
return False
except Exception as e:
logger.warning(f"Error checking Chatterbox Turbo cache: {e}")
return False
async def load_model(self, model_size: str = "default") -> None:
"""Load the Chatterbox Turbo model."""
if self.model is not None:
return
async with self._model_load_lock:
if self.model is not None:
return
await asyncio.to_thread(self._load_model_sync)
def _load_model_sync(self):
"""Synchronous model loading."""
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
progress_manager = get_progress_manager()
task_manager = get_task_manager()
model_name = "chatterbox-turbo"
is_cached = self._is_model_cached()
# Set up HF progress tracking (intercepts tqdm for file-level progress)
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
tracker_context = tracker.patch_download()
tracker_context.__enter__()
if not is_cached:
task_manager.start_download(model_name)
progress_manager.update_progress(
model_name=model_name,
current=0,
total=0,
filename="Connecting to HuggingFace...",
status="downloading",
)
try:
device = self._get_device()
self._device = device
logger.info(f"Loading Chatterbox Turbo TTS on {device}...")
import torch
from huggingface_hub import snapshot_download
from chatterbox.tts_turbo import ChatterboxTurboTTS
# Download model files ourselves so we can pass token=None
# (upstream from_pretrained passes token=True which requires
# a stored HF token even though the repo is public).
try:
local_path = snapshot_download(
repo_id=CHATTERBOX_TURBO_HF_REPO,
token=None,
allow_patterns=[
"*.safetensors", "*.json", "*.txt", "*.pt", "*.model",
],
)
finally:
tracker_context.__exit__(None, None, None)
# Monkey-patch torch.load for CPU loading. The model's .pt files
# were saved on CUDA; from_local() doesn't pass map_location
# so loading on CPU fails without this.
# Load into a local var, apply patches, then publish to
# self.model so a failed patch doesn't leave us half-initialised.
if device == "cpu":
_orig_torch_load = torch.load
def _patched_load(*args, **kwargs):
kwargs.setdefault("map_location", "cpu")
return _orig_torch_load(*args, **kwargs)
with ChatterboxTurboTTSBackend._load_lock:
torch.load = _patched_load
try:
model = ChatterboxTurboTTS.from_local(
local_path, device,
)
finally:
torch.load = _orig_torch_load
else:
model = ChatterboxTurboTTS.from_local(
local_path, device,
)
if not is_cached:
progress_manager.mark_complete(model_name)
task_manager.complete_download(model_name)
# Patch float64 → float32 dtype mismatches in upstream chatterbox.
# librosa.load returns float64 numpy; multiple upstream code paths
# convert it to a torch tensor via torch.from_numpy() without
# casting, then matmul it against float32 model weights.
# We patch the two known entry points:
#
# 1. S3Tokenizer.log_mel_spectrogram — the audio tensor from
# librosa hits _mel_filters (float32) in a matmul.
# 2. VoiceEncoder.forward — float64 mel spectrograms hit the
# float32 LSTM weights.
import types
# Patch S3Tokenizer (used by s3gen.tokenizer)
_tokzr = model.s3gen.tokenizer
_orig_log_mel = _tokzr.log_mel_spectrogram.__func__
def _f32_log_mel(self_tokzr, audio, padding=0):
import torch as _torch
if _torch.is_tensor(audio):
audio = audio.float()
return _orig_log_mel(self_tokzr, audio, padding)
_tokzr.log_mel_spectrogram = types.MethodType(_f32_log_mel, _tokzr)
# Patch VoiceEncoder
_ve = model.ve
_orig_ve_forward = _ve.forward.__func__
def _f32_ve_forward(self_ve, mels):
return _orig_ve_forward(self_ve, mels.float())
_ve.forward = types.MethodType(_f32_ve_forward, _ve)
# Only publish after all patches succeed
self.model = model
logger.info("Chatterbox Turbo TTS loaded successfully")
except ImportError as e:
logger.error(
"chatterbox-tts package not found. "
"Install with: pip install chatterbox-tts"
)
if not is_cached:
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
except Exception as e:
logger.error(f"Failed to load Chatterbox Turbo: {e}")
if not is_cached:
progress_manager.mark_error(model_name, str(e))
task_manager.error_download(model_name, str(e))
raise
def unload_model(self) -> None:
"""Unload model to free memory."""
if self.model is not None:
device = self._device
del self.model
self.model = None
self._device = None
if device == "cuda":
import torch
torch.cuda.empty_cache()
logger.info("Chatterbox Turbo unloaded")
async def create_voice_prompt(
self,
audio_path: str,
reference_text: str,
use_cache: bool = True,
) -> Tuple[dict, bool]:
"""
Create voice prompt from reference audio.
Chatterbox Turbo processes reference audio at generation time, so the
prompt just stores the file path.
"""
voice_prompt = {
"ref_audio": str(audio_path),
"ref_text": reference_text,
}
return voice_prompt, False
async def combine_voice_prompts(
self,
audio_paths: List[str],
reference_texts: List[str],
) -> Tuple[np.ndarray, str]:
"""Combine multiple reference samples."""
combined_audio = []
for path in audio_paths:
audio, _sr = load_audio(path)
audio = normalize_audio(audio)
combined_audio.append(audio)
mixed = np.concatenate(combined_audio)
mixed = normalize_audio(mixed)
combined_text = " ".join(reference_texts)
return mixed, combined_text
async def generate(
self,
text: str,
voice_prompt: dict,
language: str = "en",
seed: Optional[int] = None,
instruct: Optional[str] = None,
) -> Tuple[np.ndarray, int]:
"""
Generate audio using Chatterbox Turbo TTS.
Supports paralinguistic tags in text: [laugh], [cough], [chuckle], etc.
Args:
text: Text to synthesize (may include paralinguistic tags)
voice_prompt: Dict with ref_audio path
language: Ignored (Turbo is English-only)
seed: Random seed for reproducibility
instruct: Unused (protocol compatibility)
Returns:
Tuple of (audio_array, sample_rate)
"""
await self.load_model()
ref_audio = voice_prompt.get("ref_audio")
if ref_audio and not Path(ref_audio).exists():
logger.warning(f"Reference audio not found: {ref_audio}")
ref_audio = None
def _generate_sync():
import torch
if seed is not None:
torch.manual_seed(seed)
logger.info("[Chatterbox Turbo] Generating (English)")
wav = self.model.generate(
text,
audio_prompt_path=ref_audio,
temperature=0.8,
top_k=1000,
top_p=0.95,
repetition_penalty=1.2,
)
# Convert tensor -> numpy
if isinstance(wav, torch.Tensor):
audio = wav.squeeze().cpu().numpy().astype(np.float32)
else:
audio = np.asarray(wav, dtype=np.float32)
sample_rate = (
getattr(self.model, "sr", None)
or getattr(self.model, "sample_rate", 24000)
)
return audio, sample_rate
return await asyncio.to_thread(_generate_sync)
+25 -14
View File
@@ -94,19 +94,27 @@ class LuxTTSBackend:
def _load_model_sync(self): def _load_model_sync(self):
"""Synchronous model loading.""" """Synchronous model loading."""
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
progress_manager = get_progress_manager() progress_manager = get_progress_manager()
task_manager = get_task_manager() task_manager = get_task_manager()
model_name = "luxtts" model_name = "luxtts"
is_cached = self._is_model_cached() is_cached = self._is_model_cached()
# Set up HF progress tracking (intercepts tqdm for file-level progress)
progress_callback = create_hf_progress_callback(model_name, progress_manager)
tracker = HFProgressTracker(progress_callback, filter_non_downloads=is_cached)
tracker_context = tracker.patch_download()
tracker_context.__enter__()
if not is_cached: if not is_cached:
task_manager.start_download(model_name) task_manager.start_download(model_name)
progress_manager.update_progress( progress_manager.update_progress(
model_name=model_name, model_name=model_name,
current=0, current=0,
total=0, total=0,
filename="Downloading LuxTTS model...", filename="Connecting to HuggingFace...",
status="downloading", status="downloading",
) )
@@ -117,19 +125,22 @@ class LuxTTSBackend:
logger.info(f"Loading LuxTTS on {device}...") logger.info(f"Loading LuxTTS on {device}...")
# LuxTTS constructor downloads model and loads everything # LuxTTS constructor downloads model and loads everything
if device == "cpu": try:
import os if device == "cpu":
threads = os.cpu_count() or 4 import os
self.model = LuxTTS( threads = os.cpu_count() or 4
model_path=LUXTTS_HF_REPO, self.model = LuxTTS(
device="cpu", model_path=LUXTTS_HF_REPO,
threads=min(threads, 8), device="cpu",
) threads=min(threads, 8),
else: )
self.model = LuxTTS( else:
model_path=LUXTTS_HF_REPO, self.model = LuxTTS(
device=device, model_path=LUXTTS_HF_REPO,
) device=device,
)
finally:
tracker_context.__exit__(None, None, None)
if not is_cached: if not is_cached:
progress_manager.mark_complete(model_name) progress_manager.mark_complete(model_name)
+39 -5
View File
@@ -5,8 +5,15 @@ MLX backend implementation for TTS and STT using mlx-audio.
from typing import Optional, List, Tuple from typing import Optional, List, Tuple
import asyncio import asyncio
import numpy as np import numpy as np
import os
from pathlib import Path from pathlib import Path
# PATCH: Import and apply offline patch BEFORE any huggingface_hub usage
# This prevents mlx_audio from making network requests when models are cached
from ..utils.hf_offline_patch import patch_huggingface_hub_offline, ensure_original_qwen_config_cached
patch_huggingface_hub_offline()
ensure_original_qwen_config_cached()
from . import TTSBackend, STTBackend from . import TTSBackend, STTBackend
from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt from ..utils.cache import get_cache_key, get_cached_voice_prompt, cache_voice_prompt
from ..utils.audio import normalize_audio, load_audio from ..utils.audio import normalize_audio, load_audio
@@ -14,6 +21,12 @@ from ..utils.progress import get_progress_manager
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from ..utils.tasks import get_task_manager from ..utils.tasks import get_task_manager
LANGUAGE_CODE_TO_NAME = {
"zh": "chinese", "en": "english", "ja": "japanese", "ko": "korean",
"de": "german", "fr": "french", "ru": "russian", "pt": "portuguese",
"es": "spanish", "it": "italian",
}
class MLXTTSBackend: class MLXTTSBackend:
"""MLX-based TTS backend using mlx-audio.""" """MLX-based TTS backend using mlx-audio."""
@@ -159,15 +172,35 @@ class MLXTTSBackend:
tracker_context = tracker.patch_download() tracker_context = tracker.patch_download()
tracker_context.__enter__() tracker_context.__enter__()
# PATCH: Force offline mode when model is already cached
# This prevents crashes when HuggingFace is unreachable
original_hf_hub_offline = os.environ.get("HF_HUB_OFFLINE")
if is_cached:
os.environ["HF_HUB_OFFLINE"] = "1"
print(f"[PATCH] Model {model_size} is cached, forcing HF_HUB_OFFLINE=1 to avoid network requests")
# Import mlx_audio AFTER patching tqdm # Import mlx_audio AFTER patching tqdm
from mlx_audio.tts import load from mlx_audio.tts import load
# Load MLX model (downloads automatically) # Load MLX model (downloads automatically)
try: try:
self.model = load(model_path) self.model = load(model_path)
except Exception as load_error:
# If offline mode failed, try with network enabled as fallback
if is_cached and "offline" in str(load_error).lower():
print(f"[PATCH] Offline load failed, trying with network: {load_error}")
os.environ.pop("HF_HUB_OFFLINE", None)
self.model = load(model_path)
else:
raise
finally: finally:
# Exit the patch context # Exit the patch context
tracker_context.__exit__(None, None, None) tracker_context.__exit__(None, None, None)
# Restore original HF_HUB_OFFLINE setting
if original_hf_hub_offline is not None:
os.environ["HF_HUB_OFFLINE"] = original_hf_hub_offline
else:
os.environ.pop("HF_HUB_OFFLINE", None)
# Only mark download as complete if we were tracking it # Only mark download as complete if we were tracking it
if not is_cached: if not is_cached:
@@ -316,7 +349,8 @@ class MLXTTSBackend:
# MLX generate() returns a generator yielding GenerationResult objects # MLX generate() returns a generator yielding GenerationResult objects
audio_chunks = [] audio_chunks = []
sample_rate = 24000 sample_rate = 24000
lang = LANGUAGE_CODE_TO_NAME.get(language, "auto")
# Set seed if provided (MLX uses numpy random) # Set seed if provided (MLX uses numpy random)
if seed is not None: if seed is not None:
import mlx.core as mx import mlx.core as mx
@@ -344,23 +378,23 @@ class MLXTTSBackend:
sig = inspect.signature(self.model.generate) sig = inspect.signature(self.model.generate)
if "ref_audio" in sig.parameters: if "ref_audio" in sig.parameters:
# Generate with voice cloning # Generate with voice cloning
for result in self.model.generate(text, ref_audio=ref_audio, ref_text=ref_text): for result in self.model.generate(text, ref_audio=ref_audio, ref_text=ref_text, lang_code=lang):
audio_chunks.append(np.array(result.audio)) audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate sample_rate = result.sample_rate
else: else:
# Fallback: generate without voice cloning # Fallback: generate without voice cloning
for result in self.model.generate(text): for result in self.model.generate(text, lang_code=lang):
audio_chunks.append(np.array(result.audio)) audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate sample_rate = result.sample_rate
else: else:
# No voice prompt, generate normally # No voice prompt, generate normally
for result in self.model.generate(text): for result in self.model.generate(text, lang_code=lang):
audio_chunks.append(np.array(result.audio)) audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate sample_rate = result.sample_rate
except Exception as e: except Exception as e:
# If voice cloning fails, try without it # If voice cloning fails, try without it
print(f"Warning: Voice cloning failed, generating without voice prompt: {e}") print(f"Warning: Voice cloning failed, generating without voice prompt: {e}")
for result in self.model.generate(text): for result in self.model.generate(text, lang_code=lang):
audio_chunks.append(np.array(result.audio)) audio_chunks.append(np.array(result.audio))
sample_rate = result.sample_rate sample_rate = result.sample_rate
+13 -6
View File
@@ -15,6 +15,12 @@ from ..utils.progress import get_progress_manager
from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback from ..utils.hf_progress import HFProgressTracker, create_hf_progress_callback
from ..utils.tasks import get_task_manager from ..utils.tasks import get_task_manager
LANGUAGE_CODE_TO_NAME = {
"zh": "chinese", "en": "english", "ja": "japanese", "ko": "korean",
"de": "german", "fr": "french", "ru": "russian", "pt": "portuguese",
"es": "spanish", "it": "italian",
}
class PyTorchTTSBackend: class PyTorchTTSBackend:
"""PyTorch-based TTS backend using Qwen3-TTS.""" """PyTorch-based TTS backend using Qwen3-TTS."""
@@ -359,6 +365,7 @@ class PyTorchTTSBackend:
wavs, sample_rate = self.model.generate_voice_clone( wavs, sample_rate = self.model.generate_voice_clone(
text=text, text=text,
voice_clone_prompt=voice_prompt, voice_clone_prompt=voice_prompt,
language=LANGUAGE_CODE_TO_NAME.get(language, "auto"),
instruct=instruct, instruct=instruct,
) )
return wavs[0], sample_rate return wavs[0], sample_rate
@@ -374,6 +381,7 @@ WHISPER_HF_REPOS = {
"small": "openai/whisper-small", "small": "openai/whisper-small",
"medium": "openai/whisper-medium", "medium": "openai/whisper-medium",
"large": "openai/whisper-large-v3", "large": "openai/whisper-large-v3",
"turbo": "openai/whisper-large-v3-turbo",
} }
@@ -591,21 +599,20 @@ class PyTorchSTTBackend:
) )
inputs = inputs.to(self.device) inputs = inputs.to(self.device)
# Set language if provided # Generate transcription
forced_decoder_ids = None # If language is provided, force it; otherwise let Whisper auto-detect
generate_kwargs = {}
if language: if language:
# Support all languages from frontend: en, zh, ja, ko, de, fr, ru, pt, es, it
# Whisper supports these and many more
forced_decoder_ids = self.processor.get_decoder_prompt_ids( forced_decoder_ids = self.processor.get_decoder_prompt_ids(
language=language, language=language,
task="transcribe", task="transcribe",
) )
generate_kwargs["forced_decoder_ids"] = forced_decoder_ids
# Generate transcription
with torch.no_grad(): with torch.no_grad():
predicted_ids = self.model.generate( predicted_ids = self.model.generate(
inputs["input_features"], inputs["input_features"],
forced_decoder_ids=forced_decoder_ids, **generate_kwargs,
) )
# Decode # Decode
+36 -2
View File
@@ -45,10 +45,14 @@ class Generation(Base):
profile_id = Column(String, ForeignKey("profiles.id"), nullable=False) profile_id = Column(String, ForeignKey("profiles.id"), nullable=False)
text = Column(Text, nullable=False) text = Column(Text, nullable=False)
language = Column(String, default="en") language = Column(String, default="en")
audio_path = Column(String, nullable=False) audio_path = Column(String, nullable=True)
duration = Column(Float, nullable=False) duration = Column(Float, nullable=True)
seed = Column(Integer) seed = Column(Integer)
instruct = Column(Text) instruct = Column(Text)
engine = Column(String, default="qwen")
model_size = Column(String, nullable=True)
status = Column(String, default="completed") # generating, completed, failed
error = Column(Text, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow) created_at = Column(DateTime, default=datetime.utcnow)
@@ -288,6 +292,36 @@ def _run_migrations(engine):
conn.commit() conn.commit()
print("Added avatar_path column to profiles") print("Added avatar_path column to profiles")
# Migration: Add status and error columns to generations table
if 'generations' in inspector.get_table_names():
columns = {col['name'] for col in inspector.get_columns('generations')}
if 'status' not in columns:
print("Migrating generations: adding status column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE generations ADD COLUMN status VARCHAR DEFAULT 'completed'"))
conn.commit()
print("Added status column to generations")
if 'error' not in columns:
print("Migrating generations: adding error column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE generations ADD COLUMN error TEXT"))
conn.commit()
print("Added error column to generations")
if 'engine' not in columns:
print("Migrating generations: adding engine column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE generations ADD COLUMN engine VARCHAR DEFAULT 'qwen'"))
conn.commit()
print("Added engine column to generations")
# Re-read columns after engine migration (variable name shadows outer `engine`)
columns = {col['name'] for col in inspector.get_columns('generations')}
if 'model_size' not in columns:
print("Migrating generations: adding model_size column")
with engine.connect() as conn:
conn.execute(text("ALTER TABLE generations ADD COLUMN model_size VARCHAR"))
conn.commit()
print("Added model_size column to generations")
def get_db(): def get_db():
"""Get database session (generator for dependency injection).""" """Get database session (generator for dependency injection)."""
+42 -1
View File
@@ -29,6 +29,10 @@ async def create_generation(
seed: Optional[int], seed: Optional[int],
db: Session, db: Session,
instruct: Optional[str] = None, instruct: Optional[str] = None,
generation_id: Optional[str] = None,
status: str = "completed",
engine: Optional[str] = "qwen",
model_size: Optional[str] = None,
) -> GenerationResponse: ) -> GenerationResponse:
""" """
Create a new generation history entry. Create a new generation history entry.
@@ -42,12 +46,16 @@ async def create_generation(
seed: Random seed used (if any) seed: Random seed used (if any)
db: Database session db: Database session
instruct: Natural language instruction used (if any) instruct: Natural language instruction used (if any)
generation_id: Pre-assigned ID (for async generation flow)
status: Generation status (generating, completed, failed)
engine: TTS engine used (qwen, luxtts, chatterbox, chatterbox_turbo)
model_size: Model size variant (1.7B, 0.6B) — only relevant for qwen
Returns: Returns:
Created generation entry Created generation entry
""" """
db_generation = DBGeneration( db_generation = DBGeneration(
id=str(uuid.uuid4()), id=generation_id or str(uuid.uuid4()),
profile_id=profile_id, profile_id=profile_id,
text=text, text=text,
language=language, language=language,
@@ -55,6 +63,9 @@ async def create_generation(
duration=duration, duration=duration,
seed=seed, seed=seed,
instruct=instruct, instruct=instruct,
engine=engine,
model_size=model_size,
status=status,
created_at=datetime.utcnow(), created_at=datetime.utcnow(),
) )
@@ -65,6 +76,32 @@ async def create_generation(
return GenerationResponse.model_validate(db_generation) return GenerationResponse.model_validate(db_generation)
async def update_generation_status(
generation_id: str,
status: str,
db: Session,
audio_path: Optional[str] = None,
duration: Optional[float] = None,
error: Optional[str] = None,
) -> Optional[GenerationResponse]:
"""Update the status of a generation (used by async generation flow)."""
generation = db.query(DBGeneration).filter_by(id=generation_id).first()
if not generation:
return None
generation.status = status
if audio_path is not None:
generation.audio_path = audio_path
if duration is not None:
generation.duration = duration
if error is not None:
generation.error = error
db.commit()
db.refresh(generation)
return GenerationResponse.model_validate(generation)
async def get_generation( async def get_generation(
generation_id: str, generation_id: str,
db: Session, db: Session,
@@ -143,6 +180,10 @@ async def list_generations(
duration=generation.duration, duration=generation.duration,
seed=generation.seed, seed=generation.seed,
instruct=generation.instruct, instruct=generation.instruct,
engine=generation.engine or "qwen",
model_size=generation.model_size,
status=generation.status or "completed",
error=generation.error,
created_at=generation.created_at, created_at=generation.created_at,
)) ))
+732 -132
View File
File diff suppressed because it is too large Load Diff
+49 -12
View File
@@ -11,7 +11,7 @@ class VoiceProfileCreate(BaseModel):
"""Request model for creating a voice profile.""" """Request model for creating a voice profile."""
name: str = Field(..., min_length=1, max_length=100) name: str = Field(..., min_length=1, max_length=100)
description: Optional[str] = Field(None, max_length=500) description: Optional[str] = Field(None, max_length=500)
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it)$") language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$")
class VoiceProfileResponse(BaseModel): class VoiceProfileResponse(BaseModel):
@@ -52,12 +52,15 @@ class ProfileSampleResponse(BaseModel):
class GenerationRequest(BaseModel): class GenerationRequest(BaseModel):
"""Request model for voice generation.""" """Request model for voice generation."""
profile_id: str profile_id: str
text: str = Field(..., min_length=1, max_length=5000) text: str = Field(..., min_length=1, max_length=50000)
language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it)$") language: str = Field(default="en", pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he)$")
seed: Optional[int] = Field(None, ge=0) seed: Optional[int] = Field(None, ge=0)
model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B)$") model_size: Optional[str] = Field(default="1.7B", pattern="^(1\\.7B|0\\.6B)$")
instruct: Optional[str] = Field(None, max_length=500) instruct: Optional[str] = Field(None, max_length=500)
engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts)$") engine: Optional[str] = Field(default="qwen", pattern="^(qwen|luxtts|chatterbox|chatterbox_turbo)$")
max_chunk_chars: int = Field(default=800, ge=100, le=5000, description="Max characters per chunk for long text splitting")
crossfade_ms: int = Field(default=50, ge=0, le=500, description="Crossfade duration in ms between chunks (0 for hard cut)")
normalize: bool = Field(default=True, description="Normalize output audio volume")
class GenerationResponse(BaseModel): class GenerationResponse(BaseModel):
@@ -66,10 +69,14 @@ class GenerationResponse(BaseModel):
profile_id: str profile_id: str
text: str text: str
language: str language: str
audio_path: str audio_path: Optional[str] = None
duration: float duration: Optional[float] = None
seed: Optional[int] seed: Optional[int] = None
instruct: Optional[str] instruct: Optional[str] = None
engine: Optional[str] = "qwen"
model_size: Optional[str] = None
status: str = "completed"
error: Optional[str] = None
created_at: datetime created_at: datetime
class Config: class Config:
@@ -91,10 +98,14 @@ class HistoryResponse(BaseModel):
profile_name: str profile_name: str
text: str text: str
language: str language: str
audio_path: str audio_path: Optional[str] = None
duration: float duration: Optional[float] = None
seed: Optional[int] seed: Optional[int] = None
instruct: Optional[str] instruct: Optional[str] = None
engine: Optional[str] = "qwen"
model_size: Optional[str] = None
status: str = "completed"
error: Optional[str] = None
created_at: datetime created_at: datetime
class Config: class Config:
@@ -131,10 +142,27 @@ class HealthResponse(BaseModel):
backend_variant: Optional[str] = None # Binary variant (cpu or cuda) backend_variant: Optional[str] = None # Binary variant (cpu or cuda)
class DirectoryCheck(BaseModel):
"""Health status for a single directory."""
path: str
exists: bool
writable: bool
error: Optional[str] = None
class FilesystemHealthResponse(BaseModel):
"""Response model for filesystem health check."""
healthy: bool
disk_free_mb: Optional[float] = None
disk_total_mb: Optional[float] = None
directories: List[DirectoryCheck]
class ModelStatus(BaseModel): class ModelStatus(BaseModel):
"""Response model for model status.""" """Response model for model status."""
model_name: str model_name: str
display_name: str display_name: str
hf_repo_id: Optional[str] = None # HuggingFace repository ID
downloaded: bool downloaded: bool
downloading: bool = False # True if download is in progress downloading: bool = False # True if download is in progress
size_mb: Optional[float] = None size_mb: Optional[float] = None
@@ -151,12 +179,21 @@ class ModelDownloadRequest(BaseModel):
model_name: str model_name: str
class ModelMigrateRequest(BaseModel):
"""Request model for migrating models to a new directory."""
destination: str
class ActiveDownloadTask(BaseModel): class ActiveDownloadTask(BaseModel):
"""Response model for active download task.""" """Response model for active download task."""
model_name: str model_name: str
status: str status: str
started_at: datetime started_at: datetime
error: Optional[str] = None error: Optional[str] = None
progress: Optional[float] = None # 0-100 percentage
current: Optional[int] = None # bytes downloaded
total: Optional[int] = None # total bytes
filename: Optional[str] = None # current file being downloaded
class ActiveGenerationTask(BaseModel): class ActiveGenerationTask(BaseModel):
+27 -10
View File
@@ -38,14 +38,22 @@ async def create_profile(
) -> VoiceProfileResponse: ) -> VoiceProfileResponse:
""" """
Create a new voice profile. Create a new voice profile.
Args: Args:
data: Profile creation data data: Profile creation data
db: Database session db: Database session
Returns: Returns:
Created profile Created profile
Raises:
ValueError: If a profile with the same name already exists
""" """
# Check if profile name already exists
existing_profile = db.query(DBVoiceProfile).filter_by(name=data.name).first()
if existing_profile:
raise ValueError(f"A profile with the name '{data.name}' already exists. Please choose a different name.")
# Create profile in database # Create profile in database
db_profile = DBVoiceProfile( db_profile = DBVoiceProfile(
id=str(uuid.uuid4()), id=str(uuid.uuid4()),
@@ -55,15 +63,15 @@ async def create_profile(
created_at=datetime.utcnow(), created_at=datetime.utcnow(),
updated_at=datetime.utcnow(), updated_at=datetime.utcnow(),
) )
db.add(db_profile) db.add(db_profile)
db.commit() db.commit()
db.refresh(db_profile) db.refresh(db_profile)
# Create profile directory # Create profile directory
profile_dir = _get_profiles_dir() / db_profile.id profile_dir = _get_profiles_dir() / db_profile.id
profile_dir.mkdir(parents=True, exist_ok=True) profile_dir.mkdir(parents=True, exist_ok=True)
return VoiceProfileResponse.model_validate(db_profile) return VoiceProfileResponse.model_validate(db_profile)
@@ -191,28 +199,37 @@ async def update_profile(
) -> Optional[VoiceProfileResponse]: ) -> Optional[VoiceProfileResponse]:
""" """
Update a voice profile. Update a voice profile.
Args: Args:
profile_id: Profile ID profile_id: Profile ID
data: Updated profile data data: Updated profile data
db: Database session db: Database session
Returns: Returns:
Updated profile or None if not found Updated profile or None if not found
Raises:
ValueError: If a profile with the same name already exists (different profile)
""" """
profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first() profile = db.query(DBVoiceProfile).filter_by(id=profile_id).first()
if not profile: if not profile:
return None return None
# Check if the new name conflicts with another profile
if profile.name != data.name:
existing_profile = db.query(DBVoiceProfile).filter_by(name=data.name).first()
if existing_profile:
raise ValueError(f"A profile with the name '{data.name}' already exists. Please choose a different name.")
# Update fields # Update fields
profile.name = data.name profile.name = data.name
profile.description = data.description profile.description = data.description
profile.language = data.language profile.language = data.language
profile.updated_at = datetime.utcnow() profile.updated_at = datetime.utcnow()
db.commit() db.commit()
db.refresh(profile) db.refresh(profile)
return VoiceProfileResponse.model_validate(profile) return VoiceProfileResponse.model_validate(profile)
+12
View File
@@ -21,6 +21,18 @@ qwen-tts>=0.0.5
linacodec @ git+https://github.com/ysharma3501/LinaCodec.git linacodec @ git+https://github.com/ysharma3501/LinaCodec.git
Zipvoice @ git+https://github.com/ysharma3501/LuxTTS.git Zipvoice @ git+https://github.com/ysharma3501/LuxTTS.git
# Chatterbox TTS sub-dependencies (chatterbox-tts itself is installed
# --no-deps in the setup script because it pins numpy<1.26 / torch==2.6
# which are incompatible with Python 3.12+)
conformer>=0.3.2
diffusers>=0.29.0
omegaconf
pykakasi
resemble-perth>=1.0.1
s3tokenizer
spacy-pkuseg
pyloudnorm
# Audio processing # Audio processing
librosa>=0.10.0 librosa>=0.10.0
soundfile>=0.12.0 soundfile>=0.12.0
+6 -6
View File
@@ -270,11 +270,14 @@ async def add_item_to_story(
generation_created_at=generation.created_at, generation_created_at=generation.created_at,
) )
# Get track from data or default to 0
track = data.track if data.track is not None else 0
# Calculate start_time_ms if not provided # Calculate start_time_ms if not provided
if data.start_time_ms is not None: if data.start_time_ms is not None:
start_time_ms = data.start_time_ms start_time_ms = data.start_time_ms
else: else:
# Find the maximum end time (start_time_ms + duration_ms) of existing items # Find the maximum end time on the target track only
existing_items = db.query( existing_items = db.query(
DBStoryItem, DBStoryItem,
DBGeneration DBGeneration
@@ -282,11 +285,11 @@ async def add_item_to_story(
DBGeneration, DBGeneration,
DBStoryItem.generation_id == DBGeneration.id DBStoryItem.generation_id == DBGeneration.id
).filter( ).filter(
DBStoryItem.story_id == story_id DBStoryItem.story_id == story_id,
DBStoryItem.track == track,
).all() ).all()
if not existing_items: if not existing_items:
# First item starts at 0
start_time_ms = 0 start_time_ms = 0
else: else:
max_end_time_ms = 0 max_end_time_ms = 0
@@ -297,9 +300,6 @@ async def add_item_to_story(
# Add 200ms gap after the last item # Add 200ms gap after the last item
start_time_ms = max_end_time_ms + 200 start_time_ms = max_end_time_ms + 200
# Get track from data or default to 0
track = data.track if data.track is not None else 0
# Create item # Create item
item = DBStoryItem( item = DBStoryItem(
id=str(uuid.uuid4()), id=str(uuid.uuid4()),
+162
View File
@@ -0,0 +1,162 @@
"""
Tests for CORS origin restrictions.
Validates that the CORS middleware only allows known local origins
and respects the VOICEBOX_CORS_ORIGINS environment variable.
Uses a minimal FastAPI app that mirrors the exact CORS configuration
from backend/main.py, so tests run without heavy ML dependencies.
Usage:
pip install httpx pytest fastapi starlette
python -m pytest backend/tests/test_cors.py -v
"""
import os
import pytest
from unittest.mock import patch
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from starlette.testclient import TestClient
def _build_app(env_origins: str = "") -> FastAPI:
"""
Build a minimal FastAPI app with the same CORS logic as backend/main.py.
This mirrors the exact code in main.py so the test validates the real
configuration without needing torch/numpy/transformers installed.
"""
app = FastAPI()
_default_origins = [
"http://localhost:5173",
"http://127.0.0.1:5173",
"http://localhost:17493",
"http://127.0.0.1:17493",
"tauri://localhost",
"https://tauri.localhost",
]
_cors_origins = _default_origins + [o.strip() for o in env_origins.split(",") if o.strip()]
app.add_middleware(
CORSMiddleware,
allow_origins=_cors_origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/health")
async def health():
return {"status": "ok"}
return app
@pytest.fixture()
def client():
return TestClient(_build_app())
@pytest.fixture()
def client_with_custom_origins():
return TestClient(_build_app("https://custom.example.com,https://other.example.com"))
def _get_with_origin(client: TestClient, origin: str) -> dict:
"""Send a GET with Origin header, return response headers."""
response = client.get("/health", headers={"Origin": origin})
return dict(response.headers)
def _preflight(client: TestClient, origin: str) -> dict:
"""Send CORS preflight OPTIONS request, return response headers."""
response = client.options(
"/health",
headers={
"Origin": origin,
"Access-Control-Request-Method": "GET",
},
)
return dict(response.headers)
class TestCORSDefaultOrigins:
"""CORS should allow known local origins and block everything else."""
@pytest.mark.parametrize("origin", [
"http://localhost:5173",
"http://127.0.0.1:5173",
"http://localhost:17493",
"http://127.0.0.1:17493",
"tauri://localhost",
"https://tauri.localhost",
])
def test_allowed_origins(self, client, origin):
headers = _get_with_origin(client, origin)
assert headers.get("access-control-allow-origin") == origin
@pytest.mark.parametrize("origin", [
"http://evil.com",
"http://localhost:9999",
"https://attacker.example.com",
"null",
])
def test_blocked_origins(self, client, origin):
headers = _get_with_origin(client, origin)
assert "access-control-allow-origin" not in headers
def test_preflight_allowed(self, client):
headers = _preflight(client, "http://localhost:5173")
assert headers.get("access-control-allow-origin") == "http://localhost:5173"
def test_preflight_blocked(self, client):
headers = _preflight(client, "http://evil.com")
assert "access-control-allow-origin" not in headers
def test_credentials_header_present(self, client):
headers = _get_with_origin(client, "http://localhost:5173")
assert headers.get("access-control-allow-credentials") == "true"
class TestCORSCustomOrigins:
"""VOICEBOX_CORS_ORIGINS env var should extend the allowlist."""
def test_custom_origin_allowed(self, client_with_custom_origins):
headers = _get_with_origin(client_with_custom_origins, "https://custom.example.com")
assert headers.get("access-control-allow-origin") == "https://custom.example.com"
def test_other_custom_origin_allowed(self, client_with_custom_origins):
headers = _get_with_origin(client_with_custom_origins, "https://other.example.com")
assert headers.get("access-control-allow-origin") == "https://other.example.com"
def test_default_origins_still_work(self, client_with_custom_origins):
headers = _get_with_origin(client_with_custom_origins, "http://localhost:5173")
assert headers.get("access-control-allow-origin") == "http://localhost:5173"
def test_unlisted_origin_still_blocked(self, client_with_custom_origins):
headers = _get_with_origin(client_with_custom_origins, "http://evil.com")
assert "access-control-allow-origin" not in headers
class TestCORSEnvVarParsing:
"""Edge cases for VOICEBOX_CORS_ORIGINS parsing."""
def test_empty_env_var(self):
app = _build_app("")
client = TestClient(app)
headers = _get_with_origin(client, "http://evil.com")
assert "access-control-allow-origin" not in headers
def test_whitespace_trimmed(self):
app = _build_app(" https://spaced.example.com ")
client = TestClient(app)
headers = _get_with_origin(client, "https://spaced.example.com")
assert headers.get("access-control-allow-origin") == "https://spaced.example.com"
def test_trailing_comma_ignored(self):
app = _build_app("https://one.example.com,")
client = TestClient(app)
headers = _get_with_origin(client, "https://one.example.com")
assert headers.get("access-control-allow-origin") == "https://one.example.com"
@@ -0,0 +1,217 @@
"""
Tests for profile duplicate name validation.
This test suite verifies that the application correctly handles
duplicate profile names and provides user-friendly error messages.
"""
import pytest
import tempfile
import shutil
from pathlib import Path
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Add parent directory to path to import backend modules
import sys
sys.path.insert(0, str(Path(__file__).parent.parent))
from database import Base, VoiceProfile as DBVoiceProfile
from models import VoiceProfileCreate
from profiles import create_profile, update_profile
@pytest.fixture
def test_db():
"""Create a temporary test database."""
# Create temporary directory for test database
temp_dir = tempfile.mkdtemp()
db_path = Path(temp_dir) / "test.db"
# Create engine and session
engine = create_engine(f"sqlite:///{db_path}")
Base.metadata.create_all(bind=engine)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
db = SessionLocal()
yield db
# Cleanup
db.close()
shutil.rmtree(temp_dir)
@pytest.fixture
def mock_profiles_dir(monkeypatch, tmp_path):
"""Mock the profiles directory to use a temporary path."""
import profiles
monkeypatch.setattr(profiles, '_get_profiles_dir', lambda: tmp_path)
return tmp_path
@pytest.mark.asyncio
async def test_create_profile_duplicate_name_raises_error(test_db, mock_profiles_dir):
"""Test that creating a profile with a duplicate name raises a ValueError."""
# Create first profile
profile_data_1 = VoiceProfileCreate(
name="Test Profile",
description="First profile",
language="en"
)
profile_1 = await create_profile(profile_data_1, test_db)
assert profile_1.name == "Test Profile"
# Try to create second profile with same name
profile_data_2 = VoiceProfileCreate(
name="Test Profile",
description="Second profile",
language="en"
)
with pytest.raises(ValueError) as exc_info:
await create_profile(profile_data_2, test_db)
# Verify error message is user-friendly
assert "already exists" in str(exc_info.value)
assert "Test Profile" in str(exc_info.value)
assert "choose a different name" in str(exc_info.value).lower()
@pytest.mark.asyncio
async def test_create_profile_different_names_succeeds(test_db, mock_profiles_dir):
"""Test that creating profiles with different names succeeds."""
# Create first profile
profile_data_1 = VoiceProfileCreate(
name="Profile One",
description="First profile",
language="en"
)
profile_1 = await create_profile(profile_data_1, test_db)
assert profile_1.name == "Profile One"
# Create second profile with different name
profile_data_2 = VoiceProfileCreate(
name="Profile Two",
description="Second profile",
language="en"
)
profile_2 = await create_profile(profile_data_2, test_db)
assert profile_2.name == "Profile Two"
# Verify both profiles exist
assert profile_1.id != profile_2.id
@pytest.mark.asyncio
async def test_update_profile_to_duplicate_name_raises_error(test_db, mock_profiles_dir):
"""Test that updating a profile to a duplicate name raises a ValueError."""
# Create two profiles with different names
profile_data_1 = VoiceProfileCreate(
name="Profile A",
description="First profile",
language="en"
)
profile_1 = await create_profile(profile_data_1, test_db)
profile_data_2 = VoiceProfileCreate(
name="Profile B",
description="Second profile",
language="en"
)
profile_2 = await create_profile(profile_data_2, test_db)
# Try to update profile_2 to use profile_1's name
update_data = VoiceProfileCreate(
name="Profile A", # Duplicate name
description="Updated description",
language="en"
)
with pytest.raises(ValueError) as exc_info:
await update_profile(profile_2.id, update_data, test_db)
# Verify error message is user-friendly
assert "already exists" in str(exc_info.value)
assert "Profile A" in str(exc_info.value)
@pytest.mark.asyncio
async def test_update_profile_keep_same_name_succeeds(test_db, mock_profiles_dir):
"""Test that updating a profile while keeping the same name succeeds."""
# Create profile
profile_data = VoiceProfileCreate(
name="My Profile",
description="Original description",
language="en"
)
profile = await create_profile(profile_data, test_db)
# Update profile with same name but different description
update_data = VoiceProfileCreate(
name="My Profile", # Same name
description="Updated description",
language="en"
)
updated_profile = await update_profile(profile.id, update_data, test_db)
# Verify update succeeded
assert updated_profile is not None
assert updated_profile.id == profile.id
assert updated_profile.name == "My Profile"
assert updated_profile.description == "Updated description"
@pytest.mark.asyncio
async def test_update_profile_to_new_unique_name_succeeds(test_db, mock_profiles_dir):
"""Test that updating a profile to a new unique name succeeds."""
# Create profile
profile_data = VoiceProfileCreate(
name="Original Name",
description="Profile description",
language="en"
)
profile = await create_profile(profile_data, test_db)
# Update profile with new unique name
update_data = VoiceProfileCreate(
name="New Unique Name",
description="Updated description",
language="en"
)
updated_profile = await update_profile(profile.id, update_data, test_db)
# Verify update succeeded
assert updated_profile is not None
assert updated_profile.id == profile.id
assert updated_profile.name == "New Unique Name"
@pytest.mark.asyncio
async def test_case_sensitive_names_allowed(test_db, mock_profiles_dir):
"""Test that profile names are case-sensitive (e.g., 'Test' and 'test' are different)."""
# Create profile with lowercase name
profile_data_1 = VoiceProfileCreate(
name="test profile",
description="Lowercase",
language="en"
)
profile_1 = await create_profile(profile_data_1, test_db)
# Create profile with different case
profile_data_2 = VoiceProfileCreate(
name="Test Profile",
description="Title case",
language="en"
)
profile_2 = await create_profile(profile_data_2, test_db)
# Both should succeed since SQLite unique constraint is case-sensitive by default
assert profile_1.name == "test profile"
assert profile_2.name == "Test Profile"
assert profile_1.id != profile_2.id
+122 -3
View File
@@ -70,14 +70,133 @@ def save_audio(
sample_rate: int = 24000, sample_rate: int = 24000,
) -> None: ) -> None:
""" """
Save audio file. Save audio file with atomic write and error handling.
Writes to a temporary file first, then atomically renames to the
target path. This prevents corrupted/partial WAV files if the
process is interrupted mid-write.
Args: Args:
audio: Audio array audio: Audio array
path: Output path path: Output path
sample_rate: Sample rate sample_rate: Sample rate
Raises:
OSError: If file cannot be written
""" """
sf.write(path, audio, sample_rate) from pathlib import Path
import os
temp_path = f"{path}.tmp"
try:
# Ensure parent directory exists
Path(path).parent.mkdir(parents=True, exist_ok=True)
# Write to temporary file first (explicit format since .tmp
# extension is not recognised by soundfile)
sf.write(temp_path, audio, sample_rate, format='WAV')
# Atomic rename to final path
os.replace(temp_path, path)
except Exception as e:
# Clean up temp file on failure
try:
if Path(temp_path).exists():
Path(temp_path).unlink()
except Exception:
pass # Best effort cleanup
raise OSError(f"Failed to save audio to {path}: {e}") from e
def trim_tts_output(
audio: np.ndarray,
sample_rate: int = 24000,
frame_ms: int = 20,
silence_threshold_db: float = -40.0,
min_silence_ms: int = 200,
max_internal_silence_ms: int = 1000,
fade_ms: int = 30,
) -> np.ndarray:
"""
Trim trailing silence and post-silence hallucination from TTS output.
Chatterbox sometimes produces ``[speech][silence][hallucinated noise]``.
This detects internal silence gaps longer than *max_internal_silence_ms*
and cuts the audio at that boundary, then trims trailing silence and
applies a short cosine fade-out.
Args:
audio: Input audio array (mono float32)
sample_rate: Sample rate in Hz
frame_ms: Frame size for RMS energy calculation
silence_threshold_db: dB threshold below which a frame is silence
min_silence_ms: Minimum trailing silence to keep
max_internal_silence_ms: Cut after any silence gap longer than this
fade_ms: Cosine fade-out duration in ms
Returns:
Trimmed audio array
"""
frame_len = int(sample_rate * frame_ms / 1000)
if frame_len == 0 or len(audio) < frame_len:
return audio
n_frames = len(audio) // frame_len
threshold_linear = 10 ** (silence_threshold_db / 20)
# Compute per-frame RMS
rms = np.array(
[
np.sqrt(np.mean(audio[i * frame_len : (i + 1) * frame_len] ** 2))
for i in range(n_frames)
]
)
is_speech = rms >= threshold_linear
# Find first speech frame
first_speech = 0
for i, s in enumerate(is_speech):
if s:
first_speech = max(0, i - 1) # keep 1 frame padding
break
# Walk forward from first speech; cut at long internal silence gaps
max_silence_frames = int(max_internal_silence_ms / frame_ms)
consecutive_silence = 0
cut_frame = n_frames
for i in range(first_speech, n_frames):
if is_speech[i]:
consecutive_silence = 0
else:
consecutive_silence += 1
if consecutive_silence >= max_silence_frames:
cut_frame = i - consecutive_silence + 1
break
# Trim trailing silence from the cut point
min_silence_frames = int(min_silence_ms / frame_ms)
end_frame = cut_frame
while end_frame > first_speech and not is_speech[end_frame - 1]:
end_frame -= 1
# Keep a short tail
end_frame = min(end_frame + min_silence_frames, cut_frame)
# Convert frames back to samples
start_sample = first_speech * frame_len
end_sample = min(end_frame * frame_len, len(audio))
trimmed = audio[start_sample:end_sample].copy()
# Cosine fade-out
fade_samples = int(sample_rate * fade_ms / 1000)
if fade_samples > 0 and len(trimmed) > fade_samples:
fade = np.cos(np.linspace(0, np.pi / 2, fade_samples)) ** 2
trimmed[-fade_samples:] *= fade
return trimmed
def validate_reference_audio( def validate_reference_audio(
+302
View File
@@ -0,0 +1,302 @@
"""
Chunked TTS generation utilities.
Splits long text into sentence-boundary chunks, generates audio per-chunk
via any TTSBackend, and concatenates with crossfade. All logic is
engine-agnostic — it wraps the standard ``TTSBackend.generate()`` interface.
Short text (≤ max_chunk_chars) uses the single-shot fast path with zero
overhead.
"""
import logging
import re
from typing import List, Tuple
import numpy as np
logger = logging.getLogger("voicebox.chunked-tts")
# Default chunk size in characters. Can be overridden per-request via
# the ``max_chunk_chars`` field on GenerationRequest.
DEFAULT_MAX_CHUNK_CHARS = 800
# Common abbreviations that should NOT be treated as sentence endings.
# Lowercase for case-insensitive matching.
_ABBREVIATIONS = frozenset(
{
"mr",
"mrs",
"ms",
"dr",
"prof",
"sr",
"jr",
"st",
"ave",
"blvd",
"inc",
"ltd",
"corp",
"dept",
"est",
"approx",
"vs",
"etc",
"e.g",
"i.e",
"a.m",
"p.m",
"u.s",
"u.s.a",
"u.k",
}
)
# Paralinguistic tags used by Chatterbox Turbo. The splitter must never
# cut inside one of these.
_PARA_TAG_RE = re.compile(r"\[[^\]]*\]")
# ---------------------------------------------------------------------------
# Text splitting
# ---------------------------------------------------------------------------
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
Priority: sentence-end (``.!?`` not preceded by an abbreviation and not
inside brackets) → clause boundary (``;:,—``) → whitespace → hard cut.
Paralinguistic tags like ``[laugh]`` are treated as atomic and will not
be split across chunks.
"""
text = text.strip()
if not text:
return []
if len(text) <= max_chars:
return [text]
chunks: List[str] = []
remaining = text
while remaining:
remaining = remaining.lstrip()
if not remaining:
break
if len(remaining) <= max_chars:
chunks.append(remaining)
break
segment = remaining[:max_chars]
# Try to split at the last real sentence ending
split_pos = _find_last_sentence_end(segment)
if split_pos == -1:
split_pos = _find_last_clause_boundary(segment)
if split_pos == -1:
split_pos = segment.rfind(" ")
if split_pos == -1:
# Absolute fallback: hard cut but avoid splitting inside a tag
split_pos = _safe_hard_cut(segment, max_chars)
chunk = remaining[: split_pos + 1].strip()
if chunk:
chunks.append(chunk)
remaining = remaining[split_pos + 1 :]
return chunks
def _find_last_sentence_end(text: str) -> int:
"""Return the index of the last sentence-ending punctuation in *text*.
Skips periods that follow common abbreviations (``Dr.``, ``Mr.``, etc.)
and periods inside bracket tags (``[laugh]``). Also handles CJK
sentence-ending punctuation (``。!?``).
"""
best = -1
# ASCII sentence ends
for m in re.finditer(r"[.!?](?:\s|$)", text):
pos = m.start()
char = text[pos]
# Skip periods after abbreviations
if char == ".":
# Walk backwards to find the preceding word
word_start = pos - 1
while word_start >= 0 and text[word_start].isalpha():
word_start -= 1
word = text[word_start + 1 : pos].lower()
if word in _ABBREVIATIONS:
continue
# Skip decimal numbers (digit immediately before the period)
if word_start >= 0 and text[word_start].isdigit():
continue
# Skip if we're inside a bracket tag
if _inside_bracket_tag(text, pos):
continue
best = pos
# CJK sentence-ending punctuation
for m in re.finditer(r"[\u3002\uff01\uff1f]", text):
if m.start() > best:
best = m.start()
return best
def _find_last_clause_boundary(text: str) -> int:
"""Return the index of the last clause-boundary punctuation."""
best = -1
for m in re.finditer(r"[;:,\u2014](?:\s|$)", text):
pos = m.start()
# Skip if inside a bracket tag
if _inside_bracket_tag(text, pos):
continue
best = pos
return best
def _inside_bracket_tag(text: str, pos: int) -> bool:
"""Return True if *pos* falls inside a ``[...]`` tag."""
for m in _PARA_TAG_RE.finditer(text):
if m.start() < pos < m.end():
return True
return False
def _safe_hard_cut(segment: str, max_chars: int) -> int:
"""Find a hard-cut position that doesn't split a ``[tag]``."""
cut = max_chars - 1
# Check if the cut falls inside a bracket tag; if so, move before it
for m in _PARA_TAG_RE.finditer(segment):
if m.start() < cut < m.end():
return m.start() - 1 if m.start() > 0 else cut
return cut
# ---------------------------------------------------------------------------
# Audio concatenation
# ---------------------------------------------------------------------------
def concatenate_audio_chunks(
chunks: List[np.ndarray],
sample_rate: int,
crossfade_ms: int = 50,
) -> np.ndarray:
"""Concatenate audio arrays with a short crossfade to eliminate clicks.
Each chunk is expected to be a 1-D float32 ndarray at *sample_rate* Hz.
"""
if not chunks:
return np.array([], dtype=np.float32)
if len(chunks) == 1:
return chunks[0]
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
result = np.array(chunks[0], dtype=np.float32, copy=True)
for chunk in chunks[1:]:
if len(chunk) == 0:
continue
overlap = min(crossfade_samples, len(result), len(chunk))
if overlap > 0:
fade_out = np.linspace(1.0, 0.0, overlap, dtype=np.float32)
fade_in = np.linspace(0.0, 1.0, overlap, dtype=np.float32)
result[-overlap:] = result[-overlap:] * fade_out + chunk[:overlap] * fade_in
result = np.concatenate([result, chunk[overlap:]])
else:
result = np.concatenate([result, chunk])
return result
# ---------------------------------------------------------------------------
# Engine-agnostic chunked generation
# ---------------------------------------------------------------------------
async def generate_chunked(
backend,
text: str,
voice_prompt: dict,
language: str = "en",
seed: int | None = None,
instruct: str | None = None,
max_chunk_chars: int = DEFAULT_MAX_CHUNK_CHARS,
crossfade_ms: int = 50,
trim_fn=None,
) -> Tuple[np.ndarray, int]:
"""Generate audio with automatic chunking for long text.
For text shorter than *max_chunk_chars* this is a thin wrapper around
``backend.generate()`` with zero overhead.
For longer text the input is split at natural sentence boundaries,
each chunk is generated independently, optionally trimmed (useful for
Chatterbox engines that hallucinate trailing noise), and the results
are concatenated with a crossfade (or hard cut if *crossfade_ms* is 0).
Parameters
----------
backend : TTSBackend
Any backend implementing the ``generate()`` protocol.
text : str
Input text (may be arbitrarily long).
voice_prompt, language, seed, instruct
Forwarded to ``backend.generate()`` verbatim.
max_chunk_chars : int
Maximum characters per chunk (default 800).
crossfade_ms : int
Crossfade duration in milliseconds between chunks. 0 for a hard
cut with no overlap (default 50).
trim_fn : callable | None
Optional ``(audio, sample_rate) -> audio`` post-processing
function applied to each chunk before concatenation (e.g.
``trim_tts_output`` for Chatterbox engines).
Returns
-------
(audio, sample_rate) : Tuple[np.ndarray, int]
"""
chunks = split_text_into_chunks(text, max_chunk_chars)
if len(chunks) <= 1:
# Short text — single-shot fast path
audio, sample_rate = await backend.generate(
text, voice_prompt, language, seed, instruct,
)
if trim_fn is not None:
audio = trim_fn(audio, sample_rate)
return audio, sample_rate
# Long text — chunked generation
logger.info(
"Splitting %d chars into %d chunks (max %d chars each)",
len(text), len(chunks), max_chunk_chars,
)
audio_chunks: List[np.ndarray] = []
sample_rate: int | None = None
for i, chunk_text in enumerate(chunks):
logger.info(
"Generating chunk %d/%d (%d chars)",
i + 1, len(chunks), len(chunk_text),
)
# Vary the seed per chunk to avoid correlated RNG artefacts,
# but keep it deterministic so the same (text, seed) pair
# always produces the same output.
chunk_seed = (seed + i) if seed is not None else None
chunk_audio, chunk_sr = await backend.generate(
chunk_text, voice_prompt, language, chunk_seed, instruct,
)
if trim_fn is not None:
chunk_audio = trim_fn(chunk_audio, chunk_sr)
audio_chunks.append(np.asarray(chunk_audio, dtype=np.float32))
if sample_rate is None:
sample_rate = chunk_sr
audio = concatenate_audio_chunks(audio_chunks, sample_rate, crossfade_ms=crossfade_ms)
return audio, sample_rate
+100
View File
@@ -0,0 +1,100 @@
"""
Monkey patch for huggingface_hub to force offline mode with cached models.
This prevents mlx_audio from making network requests when models are already downloaded.
"""
import os
from pathlib import Path
from typing import Optional, Union
def patch_huggingface_hub_offline():
"""
Monkey-patch huggingface_hub to force offline mode.
This must be called BEFORE importing mlx_audio.
"""
try:
import huggingface_hub
from huggingface_hub import constants as hf_constants
from huggingface_hub.file_download import _try_to_load_from_cache
# Store original function
original_try_load = _try_to_load_from_cache
def _patched_try_to_load_from_cache(
repo_id: str,
filename: str,
cache_dir: Union[str, Path, None] = None,
revision: Optional[str] = None,
repo_type: Optional[str] = None,
):
"""
Patched version that forces offline mode.
Returns None if not cached (instead of making network request).
"""
# Always use the original function, but we're already in HF_HUB_OFFLINE mode
result = original_try_load(
repo_id=repo_id,
filename=filename,
cache_dir=cache_dir,
revision=revision,
repo_type=repo_type,
)
if result is None:
# File not in cache - log this for debugging
cache_path = Path(hf_constants.HF_HUB_CACHE) / f"models--{repo_id.replace('/', '--')}"
print(f"[HF_PATCH] File not cached: {repo_id}/{filename}")
print(f"[HF_PATCH] Expected at: {cache_path}")
else:
print(f"[HF_PATCH] Cache hit: {repo_id}/{filename}")
return result
# Replace the function
import huggingface_hub.file_download as fd
fd._try_to_load_from_cache = _patched_try_to_load_from_cache
print("[HF_PATCH] huggingface_hub patched for offline mode")
except ImportError:
print("[HF_PATCH] huggingface_hub not found, skipping patch")
except Exception as e:
print(f"[HF_PATCH] Error patching huggingface_hub: {e}")
def ensure_original_qwen_config_cached():
"""
The MLX community model is based on the original Qwen model.
mlx_audio may try to fetch config from the original repo.
We need to ensure that config is available in the cache.
"""
from huggingface_hub import constants as hf_constants
# Original Qwen model that mlx_audio might reference
original_repo = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
mlx_repo = "mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16"
cache_dir = Path(hf_constants.HF_HUB_CACHE)
original_path = cache_dir / f"models--{original_repo.replace('/', '--')}"
mlx_path = cache_dir / f"models--{mlx_repo.replace('/', '--')}"
# If original repo cache doesn't exist but MLX does, create a symlink or copy config
if not original_path.exists() and mlx_path.exists():
print(f"[HF_PATCH] Original repo not cached, but MLX version is")
print(f"[HF_PATCH] Creating symlink from {original_repo} -> {mlx_repo}")
try:
# Create a symlink so the cache lookup succeeds
original_path.parent.mkdir(parents=True, exist_ok=True)
original_path.symlink_to(mlx_path, target_is_directory=True)
print(f"[HF_PATCH] Symlink created successfully")
except Exception as e:
print(f"[HF_PATCH] Could not create symlink: {e}")
# Auto-apply patch when module is imported
if os.environ.get("VOICEBOX_OFFLINE_PATCH", "1") != "0":
patch_huggingface_hub_offline()
ensure_original_qwen_config_cached()
+19 -4
View File
@@ -4,6 +4,10 @@
"workspaces": { "workspaces": {
"": { "": {
"name": "voicebox", "name": "voicebox",
"dependencies": {
"loaders.css": "^0.1.2",
"react-loaders": "^3.0.1",
},
"devDependencies": { "devDependencies": {
"@biomejs/biome": "2.3.12", "@biomejs/biome": "2.3.12",
"@types/node": "^20.0.0", "@types/node": "^20.0.0",
@@ -13,7 +17,7 @@
}, },
"app": { "app": {
"name": "@voicebox/app", "name": "@voicebox/app",
"version": "0.1.11", "version": "0.1.13",
"dependencies": { "dependencies": {
"@dnd-kit/core": "^6.3.1", "@dnd-kit/core": "^6.3.1",
"@dnd-kit/sortable": "^10.0.0", "@dnd-kit/sortable": "^10.0.0",
@@ -68,7 +72,7 @@
}, },
"landing": { "landing": {
"name": "@voicebox/landing", "name": "@voicebox/landing",
"version": "0.1.11", "version": "0.1.13",
"dependencies": { "dependencies": {
"@radix-ui/react-separator": "^1.1.8", "@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slot": "^1.2.4", "@radix-ui/react-slot": "^1.2.4",
@@ -93,7 +97,7 @@
}, },
"tauri": { "tauri": {
"name": "@voicebox/tauri", "name": "@voicebox/tauri",
"version": "0.1.11", "version": "0.1.13",
"dependencies": { "dependencies": {
"@tauri-apps/api": "^2.0.0", "@tauri-apps/api": "^2.0.0",
"@tauri-apps/plugin-dialog": "^2.0.0", "@tauri-apps/plugin-dialog": "^2.0.0",
@@ -116,7 +120,7 @@
}, },
"web": { "web": {
"name": "@voicebox/web", "name": "@voicebox/web",
"version": "0.1.11", "version": "0.1.13",
"dependencies": { "dependencies": {
"@tanstack/react-query": "^5.0.0", "@tanstack/react-query": "^5.0.0",
"react": "^18.3.0", "react": "^18.3.0",
@@ -125,6 +129,7 @@
"zustand": "^4.5.0", "zustand": "^4.5.0",
}, },
"devDependencies": { "devDependencies": {
"@tailwindcss/vite": "^4.0.0",
"@types/react": "^18.3.0", "@types/react": "^18.3.0",
"@types/react-dom": "^18.3.0", "@types/react-dom": "^18.3.0",
"@typescript-eslint/eslint-plugin": "^7.0.0", "@typescript-eslint/eslint-plugin": "^7.0.0",
@@ -677,6 +682,8 @@
"class-variance-authority": ["[email protected]", "", { "dependencies": { "clsx": "^2.1.1" } }, "sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg=="], "class-variance-authority": ["[email protected]", "", { "dependencies": { "clsx": "^2.1.1" } }, "sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg=="],
"classnames": ["[email protected]", "", {}, "sha512-saHYOzhIQs6wy2sVxTM6bUDsQO4F50V9RQ22qBpEdCW+I+/Wmke2HOl6lS6dTpdxVhb88/I6+Hs+438c3lfUow=="],
"client-only": ["[email protected]", "", {}, "sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA=="], "client-only": ["[email protected]", "", {}, "sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA=="],
"clsx": ["[email protected]", "", {}, "sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA=="], "clsx": ["[email protected]", "", {}, "sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA=="],
@@ -873,6 +880,8 @@
"lines-and-columns": ["[email protected]", "", {}, "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg=="], "lines-and-columns": ["[email protected]", "", {}, "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg=="],
"loaders.css": ["[email protected]", "", {}, "sha512-Rhowlq24ey1VOeor+3wYOt9+MjaxBOJm1u4KlQgNC3+0xJ0LS4wq4iG57D/BPzvuD/7HHDGQOWJ+81oR2EI9bQ=="],
"locate-path": ["[email protected]", "", { "dependencies": { "p-locate": "^5.0.0" } }, "sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw=="], "locate-path": ["[email protected]", "", { "dependencies": { "p-locate": "^5.0.0" } }, "sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw=="],
"lodash.merge": ["[email protected]", "", {}, "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ=="], "lodash.merge": ["[email protected]", "", {}, "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ=="],
@@ -959,6 +968,8 @@
"prelude-ls": ["[email protected]", "", {}, "sha512-vkcDPrRZo1QZLbn5RLGPpg/WmIQ65qoWWhcGKf/b5eplkkarX0m9z8ppCat4mlOqUsWpyNuYgO3VRyrYHSzX5g=="], "prelude-ls": ["[email protected]", "", {}, "sha512-vkcDPrRZo1QZLbn5RLGPpg/WmIQ65qoWWhcGKf/b5eplkkarX0m9z8ppCat4mlOqUsWpyNuYgO3VRyrYHSzX5g=="],
"prop-types": ["[email protected]", "", { "dependencies": { "loose-envify": "^1.4.0", "object-assign": "^4.1.1", "react-is": "^16.13.1" } }, "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg=="],
"punycode": ["[email protected]", "", {}, "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg=="], "punycode": ["[email protected]", "", {}, "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg=="],
"queue-microtask": ["[email protected]", "", {}, "sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A=="], "queue-microtask": ["[email protected]", "", {}, "sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A=="],
@@ -969,6 +980,10 @@
"react-hook-form": ["[email protected]", "", { "peerDependencies": { "react": "^16.8.0 || ^17 || ^18 || ^19" } }, "sha512-9SUJKCGKo8HUSsCO+y0CtqkqI5nNuaDqTxyqPsZPqIwudpj4rCrAz/jZV+jn57bx5gtZKOh3neQu94DXMc+w5w=="], "react-hook-form": ["[email protected]", "", { "peerDependencies": { "react": "^16.8.0 || ^17 || ^18 || ^19" } }, "sha512-9SUJKCGKo8HUSsCO+y0CtqkqI5nNuaDqTxyqPsZPqIwudpj4rCrAz/jZV+jn57bx5gtZKOh3neQu94DXMc+w5w=="],
"react-is": ["[email protected]", "", {}, "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ=="],
"react-loaders": ["[email protected]", "", { "dependencies": { "classnames": "^2.2.3" }, "peerDependencies": { "prop-types": ">=15.6.0", "react": ">=15" } }, "sha512-4igMNqs9Fb3d4Z+0UHIGQNJsw/37gX0nUO8QxupnEKRn1dtyYC1LGwk5GuaoDciMQCQc/MmPwb4Fn6ZfdoX1FQ=="],
"react-refresh": ["[email protected]", "", {}, "sha512-z6F7K9bV85EfseRCp2bzrpyQ0Gkw1uLoCel9XBVWPg/TjRj94SkJzUTGfOa4bs7iJvBWtQG0Wq7wnI0syw3EBQ=="], "react-refresh": ["[email protected]", "", {}, "sha512-z6F7K9bV85EfseRCp2bzrpyQ0Gkw1uLoCel9XBVWPg/TjRj94SkJzUTGfOa4bs7iJvBWtQG0Wq7wnI0syw3EBQ=="],
"react-remove-scroll": ["[email protected]", "", { "dependencies": { "react-remove-scroll-bar": "^2.3.7", "react-style-singleton": "^2.2.3", "tslib": "^2.1.0", "use-callback-ref": "^1.3.3", "use-sidecar": "^1.1.3" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-Iqb9NjCCTt6Hf+vOdNIZGdTiH1QSqr27H/Ek9sv/a97gfueI/5h1s3yRi1nngzMUaOOToin5dI1dXKdXiF+u0Q=="], "react-remove-scroll": ["[email protected]", "", { "dependencies": { "react-remove-scroll-bar": "^2.3.7", "react-style-singleton": "^2.2.3", "tslib": "^2.1.0", "use-callback-ref": "^1.3.3", "use-sidecar": "^1.1.3" }, "peerDependencies": { "@types/react": "*", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-Iqb9NjCCTt6Hf+vOdNIZGdTiH1QSqr27H/Ek9sv/a97gfueI/5h1s3yRi1nngzMUaOOToin5dI1dXKdXiF+u0Q=="],
+41
View File
@@ -0,0 +1,41 @@
services:
voicebox:
build: .
container_name: voicebox
restart: unless-stopped
ports:
# Bind to localhost only for security
- "127.0.0.1:17493:17493"
volumes:
# Bind-mount for generated audio (customize the host path as needed)
# Host side: ./output/
# Container side: /app/data/generations/
- ./output:/app/data/generations
# Named volume for profiles, DB, cache (persists across container restarts)
- voicebox-data:/app/data
# HuggingFace model cache (so models aren't re-downloaded on rebuild)
- huggingface-cache:/home/voicebox/.cache/huggingface
environment:
- LOG_LEVEL=info
networks:
- voicebox-net
deploy:
resources:
limits:
cpus: '4'
memory: 8G
networks:
voicebox-net:
driver: bridge
volumes:
voicebox-data:
huggingface-cache:
+70
View File
@@ -0,0 +1,70 @@
# Accessibility: screen reader and keyboard improvements
## Summary
Improvements to support screen reader and keyboard users across the main app surfaces: audio player, generation UI, voice selection, history, voices tab, model management, server tab, and stories.
**Tested with NVDA and Narrator on Windows.**
---
## What changed
### Audio player (after generating audio)
- **Play/Pause, Loop, Mute, Close** – `aria-label` added so each control is announced (e.g. "Play", "Pause", "Loop", "Mute", "Close player").
- **Playback position slider** – `aria-label="Playback position"` and `aria-valuetext` with current/total time (e.g. "0:30 of 2:15").
- **Volume** – Wrapped in a labelled group; volume slider has an associated screen-reader-only label and `aria-valuetext` for the level (e.g. "Volume level, 75%").
### Generation UI (text box and voice choice)
- **Generate speech** (submit) and **Fine-tune instructions** (sliders) – Icon buttons now have `aria-label` (and state for fine-tune, e.g. "Fine-tune instructions, on").
### Voice selection (cards on Generate screen)
- Each **voice card** is focusable (`tabIndex={0}`), has `role="button"`, and an `aria-label` (e.g. "Prashant, en. Select as voice for generation.") with `aria-pressed` when selected.
- **Enter/Space** on the card selects that voice; tab order is card → Export/Edit/Delete.
### History list (generated samples)
- Each **sample row** is focusable with `role="button"` and an `aria-label` (e.g. "Sample from [profile], [duration], [date]. Press Enter to play."); **Enter/Space** plays or restarts.
- **Transcript textarea** has `aria-label` (e.g. "Transcript for sample from [profile], [duration]") so when you focus on the text area, the sample is announced in context.
### Voices tab (table)
- Each **voice row** is focusable with `role="button"` and an `aria-label` (e.g. "[Name], [language], [N] generations, [N] samples. Press Enter to edit."); **Enter/Space** opens edit (except when focus is in a control).
- **Actions** dropdown trigger has `aria-label="Actions for [profile name]"`.
### Model management
- Each **model row** is a focusable region (`tabIndex={0}`, `role="group"`) with an `aria-label` (e.g. "[Model name], [status], [size]. Use Tab to reach Download or Delete.").
- **Download** and **Delete** (and Downloading) buttons have `aria-label` (e.g. "Download [name]", "Delete [name]").
### Server tab (panels)
- **Server Connection**, **Server Status**, and **App Updates** cards are landmarks: `role="region"`, `aria-label`, and `tabIndex={0}` so each panel is focusable and announced (e.g. "Server Connection", "Server Status", "App Updates").
### Stories list
- Each **story row** is a focusable control (`role="button"`, `tabIndex={0}`) with `aria-label` (e.g. "Story [name], [N] items, [date]. Press Enter to select."); **Enter/Space** selects the story. Actions button has `aria-label="Actions for [story name]"`.
### Other controls
- **Story list** – Actions (⋮) button: `aria-label="Actions for [story name]"`.
- **Story track editor** – Play/Pause, Stop, Split, Duplicate, Delete, Zoom in/out: `aria-label` on all icon buttons.
- **Voice profile samples** (SampleList, AudioSampleUpload, AudioSampleRecording, AudioSampleSystem) – Play/Pause and Stop: `aria-label` (e.g. "Play sample", "Pause", "Stop playback").
- **SampleList** mini sample player – Seek slider has `aria-label="Sample playback position"` and `aria-valuetext` for time.
---
## Testing
- **Screen readers:** Tested with **NVDA** and **Narrator** on Windows.
- **Keyboard:** Tab order and Enter/Space activation verified for focusable rows and buttons.
---
## Tech note
- React + TypeScript; Radix UI primitives; labels added via `aria-label`, `aria-labelledby`, `aria-valuetext`, and `role`/`tabIndex` where needed.
- No new dependencies.
+3 -3
View File
@@ -162,7 +162,7 @@ chmod +x voicebox-*.AppImage
**Solutions:** **Solutions:**
1. **Check server is running** 1. **Check server is running**
```bash ```bash
curl http://localhost:8000/health curl http://localhost:17493/health
``` ```
2. **Check remote mode** 2. **Check remote mode**
@@ -170,7 +170,7 @@ chmod +x voicebox-*.AppImage
- Check firewall settings - Check firewall settings
3. **Check port availability** 3. **Check port availability**
- Default port is 8000 - The current local app and dev workflow uses port 17493 by default
- Ensure no other service is using it - Ensure no other service is using it
### CORS errors in browser ### CORS errors in browser
@@ -276,7 +276,7 @@ chmod +x voicebox-*.AppImage
2. **Check OpenAPI endpoint** 2. **Check OpenAPI endpoint**
```bash ```bash
curl http://localhost:8000/openapi.json curl http://localhost:17493/openapi.json
``` ```
3. **Regenerate client** 3. **Regenerate client**
+67
View File
@@ -0,0 +1,67 @@
# Voicebox Issue Pain Points (Snapshot)
## Scope
- Dataset: **128 total issues** (**107 open**, **21 closed**)
- Source: GitHub issues in `jamiepine/voicebox`
- Classification: keyword/theme clustering
- Note: counts below are **non-exclusive** (one issue can belong to multiple pain points)
## Most Common Pain Points (Open Issues)
| Rank | Pain Point | Open Issues | What users are reporting |
|---|---|---:|---|
| 1 | Model download & offline reliability | **32** | Downloads failing/stalling, cache/offline behavior inconsistent, wrong model size selected, Errno issues |
| 2 | GPU/backend compatibility | **22** | GPU not detected, backend fallback surprises, platform-specific runtime failures (Windows/Mac) |
| 3 | Export/save/file persistence | **15** | Export fails, "failed to fetch/download audio", samples/profiles not saving |
| 4 | Language/accent quality & coverage | **14** | Missing language support, accent mismatch, robotic outputs |
| 5 | Update/restart safety + long-op controls | **4** | Auto-restart without warning, update confusion, lack of cancel/pause controls |
## Representative Issues by Pain Point
### 1) Model download & offline reliability (32)
- [#159](https://github.com/jamiepine/voicebox/issues/159) - Qwen download fails with Errno 22
- [#151](https://github.com/jamiepine/voicebox/issues/151) - Model loading hangs / server crashes
- [#150](https://github.com/jamiepine/voicebox/issues/150) - Internet required despite downloaded models
- [#149](https://github.com/jamiepine/voicebox/issues/149) - Cancel/pause controls for large downloads
- [#96](https://github.com/jamiepine/voicebox/issues/96) - 0.6B selection still uses/downloads 1.7B
### 2) GPU/backend compatibility (22)
- [#164](https://github.com/jamiepine/voicebox/issues/164) - Windows: no GPU usage + multiple breakages
- [#141](https://github.com/jamiepine/voicebox/issues/141) - Using CPU only, GPU not used
- [#131](https://github.com/jamiepine/voicebox/issues/131) - Numpy ABI mismatch in bundled app
- [#130](https://github.com/jamiepine/voicebox/issues/130) - Intel Mac tensor/padding generation error
- [#127](https://github.com/jamiepine/voicebox/issues/127) - GPU not found
### 3) Export/save/file persistence (15)
- [#148](https://github.com/jamiepine/voicebox/issues/148) - Japanese export fails on 0.1.12
- [#143](https://github.com/jamiepine/voicebox/issues/143) - Samples not saving
- [#134](https://github.com/jamiepine/voicebox/issues/134) - Can't save profile
- [#105](https://github.com/jamiepine/voicebox/issues/105) - Export audio fails (failed to fetch)
- [#49](https://github.com/jamiepine/voicebox/issues/49) - Export filename/location ignored on Windows
### 4) Language/accent quality & coverage (14)
- [#162](https://github.com/jamiepine/voicebox/issues/162) - Persian audio request/problem
- [#117](https://github.com/jamiepine/voicebox/issues/117) - Arabic language support
- [#113](https://github.com/jamiepine/voicebox/issues/113) - Polish language support
- [#109](https://github.com/jamiepine/voicebox/issues/109) - Ukrainian support
- [#100](https://github.com/jamiepine/voicebox/issues/100) - Non-US accent quality issues
### 5) Update/restart safety + controls (4)
- [#164](https://github.com/jamiepine/voicebox/issues/164) - Update behavior + usability failures
- [#136](https://github.com/jamiepine/voicebox/issues/136) - Auto-restart without warning
- [#86](https://github.com/jamiepine/voicebox/issues/86) - Unexpected restart with no confirmation
- [#149](https://github.com/jamiepine/voicebox/issues/149) - Need pause/cancel and pre-download confirmation
## Additional Signal
- There is also a large **feature-request/misc** bucket (**36 open**) that is competing with stability triage (audiobook, Linux build, additional ASR/TTS models, integrations).
## Takeaway
Most user pain is concentrated in four stability areas: **download/offline path**, **GPU/backend detection**, **save/export reliability**, and **language/accent correctness**. Addressing those first should reduce the majority of current support friction.
+222 -194
View File
@@ -1,6 +1,6 @@
# Voicebox Project Status & Roadmap # Voicebox Project Status & Roadmap
> Last updated: 2026-03-12 | Current version: **v0.1.13** | 13.1k stars | 176 open issues | 28 open PRs > Last updated: 2026-03-13 | Current version: **v0.1.13** | 13.1k stars | ~176 open issues | 25 open PRs
--- ---
@@ -30,14 +30,18 @@
│ │ HTTP :17493 │ │ │ HTTP :17493 │
│ ┌──────────────────────▼────────────────────────┐ │ │ ┌──────────────────────▼────────────────────────┐ │
│ │ FastAPI Backend (backend/) │ │ │ │ FastAPI Backend (backend/) │ │
│ │ ┌─────────────┐ ┌───────────┐ ┌─────────┐ │ │ │ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ TTSBackend │ │ STTBackend│ │ Profiles│ │ │ │ │ │ TTSBackend Protocol │ │ │
│ │ │ (Protocol) │ │ (Whisper) │ │ History │ │ │ │ │ │ ┌──────────┐ ┌───────┐ ┌───────────┐ │ │ │
│ │ │ ┌────────┐ │ └───────────┘ │ Stories │ │ │ │ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
│ │ │ │PyTorch │ │ └─────────┘ │ │ │ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
│ │ │ │or MLX │ │ │ │ │ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
│ │ │ └────────┘ │ │ │ │ │ └─────────────────────────────────────────┘ │ │
│ │ └─────────────┘ │ │ │ │ ┌───────────┐ ┌─────────┐ │ │
│ │ │ STTBackend│ │ Profiles│ │ │
│ │ │ (Whisper) │ │ History │ │ │
│ │ └───────────┘ │ Stories │ │ │
│ │ └─────────┘ │ │
│ └───────────────────────────────────────────────┘ │ │ └───────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────┘
``` ```
@@ -46,131 +50,180 @@
| Layer | File | Purpose | | Layer | File | Purpose |
|-------|------|---------| |-------|------|---------|
| Backend entry | `backend/main.py` | FastAPI app, all API routes (~1700 lines) | | Backend entry | `backend/main.py` | FastAPI app, all API routes (~2100 lines) |
| TTS protocol | `backend/backends/__init__.py:14-81` | `TTSBackend` Protocol definition | | TTS protocol | `backend/backends/__init__.py:14-81` | `TTSBackend` Protocol definition |
| TTS factory | `backend/backends/__init__.py:118-137` | Singleton backend selection (MLX vs PyTorch) | | TTS factory | `backend/backends/__init__.py:138-178` | Thread-safe engine registry (double-checked locking) |
| PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package | | PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
| MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` | | MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch | | Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
| API types | `backend/models.py` | Pydantic request/response models | | API types | `backend/models.py` | Pydantic request/response models |
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
| Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
| Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper | | Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
| Frontend types | `app/src/lib/api/types.ts` | TypeScript API types | | Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
| Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI | | Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status UI | | Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
| GPU acceleration | `app/src/components/ServerSettings/GpuAcceleration.tsx` | CUDA backend swap UI |
| Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission | | Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission |
| Language constants | `app/src/lib/constants/languages.ts` | Per-engine language maps |
### How TTS Generation Works (Current Flow) ### How TTS Generation Works (Current Flow)
``` ```
POST /generate POST /generate
1. Look up voice profile from DB 1. Look up voice profile from DB
2. Check model cache → if missing, trigger background download, return HTTP 202 2. Resolve engine from request (qwen | luxtts | chatterbox | chatterbox_turbo)
3. Load model (lazy): tts_backend.load_model(model_size) 3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
4. Create voice prompt: profiles.create_voice_prompt_for_profile() 4. Check model cache → if missing, trigger background download, return HTTP 202
5. Load model (lazy): tts_backend.load_model(model_size)
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
→ tts_backend.create_voice_prompt(audio_path, reference_text) → tts_backend.create_voice_prompt(audio_path, reference_text)
5. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct) 7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
6. Save WAV → data/generations/{id}.wav 8. Post-process: trim_tts_output() for Chatterbox engines
7. Insert history record in SQLite 9. Save WAV → data/generations/{id}.wav
8. Return GenerationResponse 10. Insert history record in SQLite
11. Return GenerationResponse
``` ```
--- ---
## Current State ## Current State
### What's Shipped (v0.1.13) ### What's Shipped (v0.1.13 + recent merges)
**Core TTS:**
- Qwen3-TTS voice cloning (1.7B and 0.6B models) - Qwen3-TTS voice cloning (1.7B and 0.6B models)
- MLX backend for Apple Silicon, PyTorch for everything else - MLX backend for Apple Silicon, PyTorch for everything else
- Multi-engine TTS architecture with thread-safe backend registry (PR #254)
- LuxTTS integration — fast, CPU-friendly English TTS (PR #254)
- Chatterbox Multilingual TTS — 23 languages including Hebrew (PR #257)
- Delivery instructions (instruct parameter, Qwen only)
- Single flat model dropdown (Qwen 1.7B, Qwen 0.6B, LuxTTS, Chatterbox, Chatterbox Turbo)
**Infrastructure:**
- CUDA backend swap via binary download and restart (PR #252)
- GPU acceleration settings UI
- Voice profiles with multi-sample support - Voice profiles with multi-sample support
- Stories editor (multi-track DAW timeline) - Stories editor (multi-track DAW timeline)
- Whisper transcription (base, small, medium, large variants) - Whisper transcription (base, small, medium, large variants)
- Model management UI with download progress (SSE) - Model management UI with inline download progress bars (HFProgressTracker)
- Download cancel/clear UI with error panel (PR #238)
- Generation history with caching - Generation history with caching
- Streaming generation endpoint (MLX only) - Streaming generation endpoint (MLX only)
- Delivery instructions (instruct parameter) - Duplicate profile name validation (PR #175)
- Linux NVIDIA GBM buffer + WebKitGTK microphone fix (PR #210)
### What's NOT Shipped But Has Code ### What's In-Flight
| Feature | Branch | Status | | Feature | Branch/PR | Status |
|---------|--------|--------| |---------|-----------|--------|
| External provider binaries (CUDA split) | `external-provider-binaries` | PR #33, significant work done, stale since Feb | | Chatterbox Turbo + per-engine language lists | `feat/chatterbox-turbo` / PR #258 | Open, ready for review |
| Dual server binaries | `feat/dual-server-binaries` | Branch exists, no PR |
| Multi-sample fix | `fix-multi-sample` | Branch exists, no PR |
| Model download notification fix | `fix-dl-notification-...` | Branch exists, no PR |
### Hardcoded Qwen3-TTS Assumptions ### TTS Engine Comparison
These are the specific coupling points that block multi-model support: | Engine | Model Name | Languages | Size | Key Features |
|--------|-----------|-----------|------|-------------|
| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Instruct mode, highest quality |
| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | 10 | ~1.2 GB | Lighter, faster |
| LuxTTS | `luxtts` | English | ~300 MB | CPU-friendly, 48 kHz, fast |
| Chatterbox | `chatterbox-tts` | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual |
| Chatterbox Turbo | `chatterbox-turbo` | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency |
| Location | What's Hardcoded | ### Multi-Engine Architecture (Shipped)
|----------|-----------------|
| `backend/models.py:58` | `model_size` regex: `^(1\.7B\|0\.6B)$` | The singleton TTS backend blocker described in the previous version of this doc has been **resolved**. The architecture now supports:
| `backend/main.py:611` | Default: `model_size or "1.7B"` |
| `backend/main.py:1322-1365` | Model status list (2 Qwen + 4 Whisper) | - **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
| `backend/main.py:1523-1548` | Download trigger map | - **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
| `backend/main.py:1597-1628` | Delete map | - **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo'`
| `backend/backends/pytorch_backend.py:65-68` | HF repo ID map | - **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
| `backend/backends/mlx_backend.py:41-44` | MLX repo ID map | - **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
| `backend/backends/__init__.py:118-137` | Single global TTS backend | - **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
| `app/src/lib/hooks/useGenerationForm.ts:17` | `modelSize: z.enum(['1.7B', '0.6B'])` |
| `app/src/lib/hooks/useGenerationForm.ts:70-71` | `modelName = "qwen-tts-${data.modelSize}"` | ### Known Limitations
| `app/src/components/Generation/GenerationForm.tsx:140-141` | Hardcoded "Qwen TTS" labels |
| `app/src/components/ServerSettings/ModelManagement.tsx:166-213` | Filters by `qwen-tts` and `whisper` prefix | - **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
| `backend/utils/cache.py` | Voice prompt cache uses `torch.save()` | - **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
- **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
- **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
- **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.
--- ---
## Open PRs — Triage & Analysis ## Open PRs — Triage & Analysis
### Recently Merged (Since Last Update)
| PR | Title | Merged |
|----|-------|--------|
| **#257** | feat: Chatterbox TTS engine with multilingual voice cloning | 2026-03-13 |
| **#254** | feat: LuxTTS integration — multi-engine TTS support | 2026-03-13 |
| **#252** | feat: CUDA backend swap via binary download and restart | 2026-03-13 |
| **#238** | Download cancel/clear UI, fixed model downloading | 2026-03-13 |
| **#250** | docs: align local API port examples | 2026-03-13 |
| **#210** | fix: Linux NVIDIA GBM buffer crash | 2026-03-13 |
| **#175** | Fix #134: duplicate profile name validation | 2026-03-13 |
### In-Flight (Our Work)
| PR | Title | Status | Notes |
|----|-------|--------|-------|
| **#258** | feat: Chatterbox Turbo engine + per-engine language lists | Open | Ready for review. Adds Turbo engine + dynamic language dropdown. |
### Merge-Ready / Near-Ready (Bug Fixes & Small Features) ### Merge-Ready / Near-Ready (Bug Fixes & Small Features)
| PR | Title | Risk | Notes | | PR | Title | Risk | Notes |
|----|-------|------|-------| |----|-------|------|-------|
| **#250** | docs: align local API port examples | None | Docs-only |
| **#230** | docs: fix README grammar | None | Docs-only | | **#230** | docs: fix README grammar | None | Docs-only |
| **#243** | a11y: screen reader and keyboard improvements | Low | Accessibility, no backend changes | | **#243** | a11y: screen reader and keyboard improvements | Low | Accessibility, no backend changes |
| **#175** | Fix #134: duplicate profile name validation | Low | Simple validation |
| **#178** | Fix #168 #140: generation error handling | Low | Error handling improvements | | **#178** | Fix #168 #140: generation error handling | Low | Error handling improvements |
| **#152** | Fix: prevent crashes when HuggingFace unreachable | Medium | Monkey-patches HF hub; solves real offline bug (#150, #151) | | **#152** | Fix: prevent crashes when HuggingFace unreachable | Medium | Monkey-patches HF hub; solves real offline bug (#150, #151) |
| **#218** | fix: unify qwen tts cache dir on Windows | Low | Windows-specific path fix | | **#218** | fix: unify qwen tts cache dir on Windows | Low | Windows-specific path fix |
| **#214** | fix: panic on launch from tokio::spawn | Low | Rust-side Tauri fix | | **#214** | fix: panic on launch from tokio::spawn | Low | Rust-side Tauri fix |
| **#210** | fix: Linux NVIDIA GBM buffer crash | Low | Linux-specific, narrowly scoped |
| **#88** | security: restrict CORS to known local origins | Low | Security hardening | | **#88** | security: restrict CORS to known local origins | Low | Security hardening |
| **#133** | feat: network access toggle | Low | Wires up existing plumbing |
### Significant Feature PRs ### Significant Feature PRs
| PR | Title | Complexity | Dependencies | Notes | | PR | Title | Complexity | Notes |
|----|-------|-----------|--------------|-------| |----|-------|-----------|-------|
| **#97** | fix: pass language parameter to TTS models | Medium | None | **Critical bug** — language param was silently dropped. Adds `LANGUAGE_CODE_TO_NAME` mapping to both backends. Should be high priority. | | **#253** | Enhance speech tokenizer with 48kHz version | Medium | Qwen tokenizer upgrade |
| **#133** | feat: network access toggle | Low | None | Wires up existing plumbing (`--host 0.0.0.0`). Clean, small. | | **#97** | fix: pass language parameter to TTS models | Medium | May be partially obsoleted by multi-engine work — needs review |
| **#238** | download cancel/clear UI + error panel | Medium | None | Adds cancel buttons, VS Code-style Problems panel, fixes whisper-large repo. Quality-of-life win. | | **#99** | feat: chunked TTS with quality selector | Medium | Solves 500-char limit. Addresses #191, #203, #69, #111. |
| **#99** | feat: chunked TTS with quality selector | Medium | None | Solves the 500-char/2048-token limit. Sentence-aware splitting, crossfade concat, 44.1kHz upsampling. Addresses #191, #203, #69, #111. | | **#154** | feat: Audiobook tab | Medium | Full audiobook workflow. Depends on #99 concepts. |
| **#154** | feat: Audiobook tab | Medium | Depends on #99 concepts | Full audiobook workflow — chunked gen, preview, auto-save to Stories. New route + tab. | | **#91** | fix: CoreAudio device enumeration | Medium | macOS audio device handling |
| **#91** | fix: CoreAudio device enumeration | Medium | None | macOS audio device handling. |
### Architectural PRs (Need Careful Review) ### Architectural PRs (Need Careful Review)
| PR | Title | Complexity | Notes | | PR | Title | Complexity | Notes |
|----|-------|-----------|-------| |----|-------|-----------|-------|
| **#33** | CUDA GPU Support — External Provider Binaries | **Very High** | The big one. Splits monolithic backend into main app + downloadable provider executables (PyTorch CPU, CUDA). New provider management system, CI/CD for R2 uploads, provider settings UI. Created Feb 1, significant codebase. **This is the foundation for multi-model support** but is currently Qwen-only. | | **#225** | feat: custom HuggingFace model support | High | Arbitrary HF repo loading. May need rework given multi-engine arch is now shipped. |
| **#225** | feat: custom HuggingFace model support | High | Adds `custom_models.py`, `custom:<slug>` model IDs, frontend model grouping (Built-in vs Custom). **Takes a different approach than #33** — keeps single backend but allows arbitrary HF repos. These two PRs may conflict architecturally. | | **#194** | feat: Hebrew + Chatterbox TTS | High | **Superseded** by PR #257 which shipped Chatterbox multilingual (23 langs incl. Hebrew). May be closeable. |
| **#194** | feat: Hebrew + Chatterbox TTS | High | **First non-Qwen TTS model.** Adds `ChatterboxTTSBackend` alongside existing backends. Routes by language (`he` → Chatterbox, else → Qwen). Adds Hebrew Whisper models. Includes a lot of cleanup. Important precedent for multi-model. | | **#195** | feat: per-profile LoRA fine-tuning | Very High | Training pipeline, adapter management, 15 new endpoints. Depends on #194 (now superseded). |
| **#195** | feat: per-profile LoRA fine-tuning | **Very High** | Depends on #194. Training pipeline, adapter management, SSE progress, 15 new API endpoints. New DB tables. Forces PyTorch even on MLX systems for adapter inference. | | **#161** | feat: Docker + web deployment | High | 3-stage Dockerfile, SPA serving. Independent of TTS engine work. |
| **#161** | feat: Docker + web deployment | High | 3-stage Dockerfile, SPA serving from FastAPI, docker-compose. Implements the Docker deployment plan. | | **#124** / **#123** | Docker (simpler attempts) | Low-Medium | Overlap with #161 |
| **#124** | Add Dockerfiles + docker-compose + docs | Medium | Earlier, simpler Docker attempt. Overlaps with #161. | | **#227** | fix: harden input validation & file safety | Medium | Coupled to #225 (custom models) |
| **#123** | added docker | Low | Minimal Docker PR. Overlaps with #161 and #124. |
| **#227** | fix: harden input validation & file safety | Medium | Follow-up to #225. Atomic writes, threading locks, input validation. Good hardening but coupled to the custom models feature. |
### PRs That Need Author Action / Are Stale ### PRs That Need Author Action / Are Stale
| PR | Title | Notes | | PR | Title | Notes |
|----|-------|-------| |----|-------|-------|
| **#237** | fix: bundle qwen_tts source files in PyInstaller | Solves #212 but needs review for build system impact | | **#237** | fix: bundle qwen_tts source files in PyInstaller | Build system, needs review |
| **#215** | Update prerequisites with Tauri deps | Branch is `main` — will have conflicts | | **#215** | Update prerequisites with Tauri deps | Branch is `main` — will have conflicts |
| **#89** | Linux Support | Branch is `main` — will have conflicts. Broad scope. | | **#89** | Linux Support | Branch is `main` — will have conflicts. Broad scope. |
| **#83** | Update download links for v0.1.12 | Outdated (we're on v0.1.13) | | **#83** | Update download links for v0.1.12 | Outdated (we're on v0.1.13) |
### PRs Likely Superseded
| PR | Superseded By | Notes |
|----|--------------|-------|
| **#194** (Hebrew + Chatterbox) | PR #257 (merged) | #257 ships Chatterbox multilingual with 23 languages including Hebrew. #194 took a different approach (route by language). Can likely be closed. |
| **#33** (External provider binaries) | PR #252 (merged) | #252 shipped CUDA backend swap. #33's broader provider architecture may still have value but needs reassessment. |
--- ---
## Open Issues — Categorized ## Open Issues — Categorized
@@ -186,15 +239,15 @@ The single most reported category. Users on Windows with NVIDIA GPUs frequently
**Key issues:** #239, #222, #220, #217, #208, #198, #192, #167, #164, #141, #130, #127 **Key issues:** #239, #222, #220, #217, #208, #198, #192, #167, #164, #141, #130, #127
**Fix path:** PR #33 (external provider binaries) is designed to solve this. Ship a small main app, let users download the CUDA provider separately. **Fix path:** PR #252 (CUDA backend swap) is now merged. Users can download the CUDA binary separately from the GPU acceleration settings. Many of these issues may now be resolvable — needs triage to confirm.
### Model Downloads (20 issues) ### Model Downloads (20 issues)
Second most reported. Users get stuck downloads, can't resume, no cancel button, no offline fallback. Second most reported. Users get stuck downloads, can't resume, no offline fallback.
**Key issues:** #249, #240, #221, #216, #212, #181, #180, #159, #150, #149, #145, #143, #135, #134 **Key issues:** #249, #240, #221, #216, #212, #181, #180, #159, #150, #149, #145, #143, #135, #134
**Fix path:** PR #238 (cancel/clear UI), PR #152 (offline crash fix). Resume support not yet addressed. **Fix path:** PR #238 (cancel/clear UI) is now merged. PR #152 (offline crash fix) still open. Inline progress bars now show for all engines. Resume support not yet addressed.
### Language Requests (18 issues) ### Language Requests (18 issues)
@@ -202,7 +255,7 @@ Strong demand for: Hindi (#245), Indonesian (#247), Dutch (#236), Hebrew (#199),
**Key issues:** #247, #245, #236, #211, #205, #199, #189, #188, #187, #183, #179, #162 **Key issues:** #247, #245, #236, #211, #205, #199, #189, #188, #187, #183, #179, #162
**Fix path:** PR #97 (pass language param — currently silently dropped!) is the prerequisite. Qwen3-TTS already supports many languages; the bug is that the language code isn't forwarded. Multi-model (#194 Chatterbox for Hebrew) expands coverage further. **Fix path:** Chatterbox Multilingual (merged via #257) now supports 23 languages including many of the requested ones: Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Per-engine language filtering (PR #258) ensures the UI shows correct options. Several of these issues may be closeable.
### New Model Requests (5 explicit issues) ### New Model Requests (5 explicit issues)
@@ -214,7 +267,7 @@ Strong demand for: Hindi (#245), Indonesian (#247), Dutch (#236), Hebrew (#199),
| #132 | LavaSR (transcription) | | #132 | LavaSR (transcription) |
| #76 | (General model expansion) | | #76 | (General model expansion) |
Community is also vocally requesting: LuxTTS, Chatterbox, XTTS-v2, Fish Speech, CosyVoice, Kokoro on social media and in issue comments. Community also requests: XTTS-v2, Fish Speech, CosyVoice, Kokoro. The multi-engine architecture is now in place, making new model integration significantly easier.
### Long-Form / Chunking (5 issues) ### Long-Form / Chunking (5 issues)
@@ -255,153 +308,128 @@ Notable requests:
| Document | Target Version | Status | Relevance | | Document | Target Version | Status | Relevance |
|----------|---------------|--------|-----------| |----------|---------------|--------|-----------|
| `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially implemented** in PR #33 | Core architecture for multi-model + CUDA distribution | | `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially superseded** by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support. API path inconsistency with provider arch doc (`/v1/` vs `/tts/`) | | `CUDA_BACKEND_SWAP.md` | — | **Shipped** (PR #252) | CUDA binary download + backend restart |
| `MLX_AUDIO.md` | — | **Shipped** (the only one) | MLX backend is live. 0.6B MLX model still missing. | | `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **PR exists** (#161) | Waiting on review. No official images published. | | `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer. Linked to issue #10. Low complexity. | | `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **PR exists** (#161) | Waiting on review |
### Cross-Document Conflicts | `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
1. **API path inconsistency:** Provider arch uses `/tts/generate`, External providers uses `/v1/generate`, OpenAI compat uses `/v1/audio/speech`. Need to reconcile.
2. **Docker vs. Provider split:** Docker doc assumes monolithic backend. Provider arch splits into separate binaries. Need to decide: does Docker run the monolith or individual providers?
3. **Version targeting:** Provider arch targets v0.1.13 (current!) but isn't merged. Everything else targets v0.2.0.
--- ---
## New Model Integration — Landscape ## New Model Integration — Landscape
### Models Worth Supporting (2026 SOTA) ### Models Worth Supporting (2026 SOTA — updated March 13)
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Integration Ease | Repo | | Model | Cloning | Speed | Sample Rate | Languages | VRAM | Integration Ease | Status |
|-------|---------|-------|-------------|-----------|------|-----------------|------| |-------|---------|-------|-------------|-----------|------|-----------------|--------|
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English-first | <1 GB | Easy | `ysharma3501/LuxTTS` | | **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | **Shipped** | v0.1.13 |
| **Chatterbox** | 5s zero-shot | Sub-200ms streaming | 24-48 kHz | 23+ | Low | Medium | `resemble-ai/chatterbox` | | **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | **Shipped** | PR #254 |
| **XTTS-v2** | 6s zero-shot | Fast mid-GPU | 24 kHz | 17+ | Medium | Medium | `coqui/XTTS-v2` | | **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | **Shipped** | PR #257 |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | Medium | `fishaudio/fish-speech` | | **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | **PR #258** | In review |
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Easy | Alibaba HF org | | **HumeAI TADA 1B/3B** | Zero-shot | 5× faster than LLM-TTS | — | EN (1B), Multilingual (3B) | Medium | Needs vetting | MIT, 700s+ coherent, synced transcript output |
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny | Medium | Kokoro repo | | **MOSS-TTS Family** | Zero-shot | — | — | Multilingual | Medium | Needs vetting | Apache 2.0, multi-speaker dialogue, text-to-voice design (no ref audio) |
| **VoxCPM 1.5** | Zero-shot (seconds) | ~0.15 RTF streaming | — | Bilingual (EN/ZH) | Medium | Needs vetting | Apache 2.0, tokenizer-free continuous diffusion, LoRA-friendly |
| **Pocket TTS** | Zero-shot + streaming | >1× RT on CPU | — | English | ~100M params, CPU-first | Needs vetting | MIT, Kyutai Labs, no GPU required |
| **Kokoro-82M** | 3s instant | CPU realtime | 24 kHz | English | Tiny (82M) | Ready | Apache 2.0, multi-engine arch in place |
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Ready | Multi-engine arch in place |
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | Ready | Multi-engine arch in place |
| **CosyVoice2-0.5B** | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | Ready | Multi-engine arch in place |
### What's Needed Architecturally for Multi-Model #### Notes on New Candidates (March 2026)
The current codebase assumes one TTS model family (Qwen3-TTS). Adding any new model requires: - **HumeAI TADA** — Text-Audio Dual Alignment arch. Near-zero hallucinations/drift, free synced transcript. 700+ seconds coherent audio. Best candidate for Stories long-form reliability. [HF: HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | [GitHub: HumeAI/tada](https://github.com/HumeAI/tada)
- **MOSS-TTS** — Modular suite: flagship cloning, MOSS-TTSD (multi-speaker dialogue), MOSS-VoiceGenerator (create voices from text descriptions, no ref audio). Unique UX for Stories voice design. [GitHub: OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS)
- **VoxCPM 1.5** — Tokenizer-free continuous diffusion + autoregressive. No discrete token artifacts. Context-aware prosody/emotion, real-time streaming, LoRA fine-tuning. Trained on 1.8M+ hours. [GitHub: OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM)
- **Pocket TTS** — 100M param CPU-first model from Kyutai Labs (Moshi team). Runs >1× realtime without GPU. Broadens hardware support significantly. [GitHub: kyutai-labs/pocket-tts](https://github.com/kyutai-labs/pocket-tts)
- **Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control)
- **Skipped:** Fish Audio S2 — restrictive research license (commercial use requires approval), despite strong features
1. **Model type concept** — A `model_type` field (e.g. `qwen`, `luxtts`, `chatterbox`) alongside `model_size`. The `GenerationRequest` schema, frontend form, and all model config dicts need updating. ### Adding a New Engine (Now Straightforward)
2. **Multiple backend instances** — The singleton `get_tts_backend()` needs to become a registry. Different models have different voice prompt formats, different inference APIs, different sample rates. With the multi-engine architecture shipped, adding a new TTS engine requires:
3. **Voice prompt format abstraction** — Qwen uses `torch.save()`-serialized tensors. LuxTTS uses `encode_prompt()` returning its own format. Chatterbox uses audio-path-based cloning. The cache system (`backend/utils/cache.py`) needs to handle heterogeneous formats. 1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
2. **Register in `backend/backends/__init__.py`** — add to `TTS_ENGINES` dict + factory function
3. **Update `backend/models.py`** — add engine name to regex
4. **Update `backend/main.py`** — add engine cases in generate, stream, model-status, download, delete (5 dispatch points)
5. **Update frontend** — add to engine union type, form schema, model dropdown, language map (5-6 files)
4. **Sample rate normalization** — Qwen outputs 24 kHz. LuxTTS outputs 48 kHz. The Stories editor and audio pipeline need to handle mixed rates. Total effort: **~1 day** for a well-documented model with a PyPI package.
5. **Per-model capabilities** — Not all models support `instruct` (delivery instructions), not all support streaming, not all support the same languages. The UI needs to adapt.
### PR #194 as Precedent
The Hebrew/Chatterbox PR (#194) is the first attempt at multi-model. It takes a pragmatic approach: route by language (`he` → Chatterbox, else → Qwen). This works for one extra model but doesn't scale — what happens when you want Chatterbox for English too?
### PR #225 as Alternative Approach
The custom HuggingFace models PR (#225) takes a different angle: let users register arbitrary HF repos and attempt to load them through the existing Qwen backend. This is flexible but fragile — it assumes all models have the same API as Qwen3-TTS.
### PR #33 as Foundation
The external provider binaries PR (#33) has the most robust architecture for multi-model, since each provider is a separate process with its own dependencies. But it's complex, currently Qwen-only, and has been stale since early February.
--- ---
## Architectural Bottlenecks ## Architectural Bottlenecks
### 1. Single Backend Singleton ### ~~1. Single Backend Singleton~~ — RESOLVED
**File:** `backend/backends/__init__.py:118-137` The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
The entire TTS system runs through one global `_tts_backend` instance. You literally cannot have two models loaded. This is the #1 blocker for multi-model support. ### 2. `main.py` is 2100+ Lines
### 2. `main.py` is 1700+ Lines All API routes, all model configs, all business logic in one file. Five separate dispatch points for each engine. Any new engine touches this file in 5 places. A model config registry pattern would reduce duplication.
All API routes, all model configs, all business logic in one file. Three separate hardcoded model config dicts that must stay in sync. Any multi-model change touches this file heavily. ### 3. Model Config is Scattered (Improved)
### 3. Model Config is Scattered Model identifiers are still duplicated across `main.py` (3 dicts), backend files, frontend components, and the languages constant. However, the pattern is now consistent and well-understood. A centralized model registry would help but isn't blocking.
Model identifiers, HF repo IDs, display names, and download logic are duplicated across:
- `main.py` (3 separate dicts)
- `pytorch_backend.py` (HF repo map)
- `mlx_backend.py` (MLX repo map)
- `GenerationForm.tsx` (UI labels)
- `useGenerationForm.ts` (validation schema)
- `ModelManagement.tsx` (prefix filters)
There is no single source of truth for "what models does Voicebox support."
### 4. Voice Prompt Cache Assumes PyTorch Tensors ### 4. Voice Prompt Cache Assumes PyTorch Tensors
`backend/utils/cache.py` uses `torch.save()` / `torch.load()` for caching voice prompts. Models that don't use PyTorch tensors (LuxTTS, MLX-native models) can't use this cache. `backend/utils/cache.py` uses `torch.save()` / `torch.load()`. LuxTTS and Chatterbox backends work around this by storing reference audio paths instead of tensors in their voice prompt dicts. Not ideal but functional.
### 5. Frontend Assumes Qwen Model Sizes ### 5. ~~Frontend Assumes Qwen Model Sizes~~ — RESOLVED
The generation form schema (`useGenerationForm.ts:17`) validates `model_size` as `'1.7B' | '0.6B'`. The model management UI filters by string prefix `qwen-tts`. Adding any model requires touching 3-4 frontend files. The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen.
--- ---
## Recommended Priorities ## Recommended Priorities
### Tier 1 — Ship Now (Bug Fixes & Critical Improvements) ### Tier 1 — Ship Now (Low Risk)
These PRs fix real user pain with low risk. Can be reviewed and merged quickly. | Priority | PR/Item | Impact | Effort |
|----------|---------|--------|--------|
| 1 | **#258** — Chatterbox Turbo + per-engine languages | Paralinguistic tags, proper language filtering | Review only |
| 2 | **#152** — Offline mode crash fix | Fixes #150, #151 | Low |
| 3 | **#99** — Chunked TTS + quality selector | Removes 500-char limit, addresses 5 issues | Medium |
| 4 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 5 | **#178** — Generation error handling | Error UX | Low |
| 6 | **#230** — Docs fixes | Zero risk | None |
| 7 | **#133** — Network access toggle | Wires up existing code | Low |
| 8 | **#88** — CORS restriction | Security improvement | Low |
| 9 | **#214** — Tauri window close panic fix | Stability | Low |
| 10 | Triage GPU issues | Many may be resolved by CUDA swap (#252) | Low |
| 11 | Close superseded PRs | #194 (superseded by #257), #83 (outdated) | None |
| Priority | PR | Impact | Effort | ### Tier 2 — Next Release (v0.2.0)
|----------|-----|--------|--------|
| 1 | **#97** — Pass language param to TTS | Fixes all non-English generation (18 language issues) | Low |
| 2 | **#238** — Download cancel/clear UI | Addresses 20 download-related issues | Low |
| 3 | **#152** — Offline mode crash fix | Fixes #150, #151 | Low |
| 4 | **#99** — Chunked TTS + quality selector | Removes 500-char limit, addresses 5 issues | Medium |
| 5 | **#218** — Windows HF cache dir fix | Windows-specific pain | Low |
| 6 | **#175, #178** — Profile validation + error handling | Small fixes | Low |
| 7 | **#250, #230** — Docs fixes | Zero risk | None |
| 8 | **#133** — Network access toggle | Wires up existing code | Low |
| 9 | **#88** — CORS restriction | Security improvement | Low |
| 10 | **#214** — Tauri window close panic fix | Stability | Low |
### Tier 2 — Next Release (v0.2.0 Foundations) | Priority | Item | Impact | Effort |
|----------|------|--------|--------|
These require more review but unlock major capabilities. | 1 | **#253** — 48kHz speech tokenizer | Quality improvement | Medium |
| 2 | **#161** — Docker deployment | Server/headless users | Medium |
| Priority | Item | Impact | Effort | Dependencies | | 3 | **#154** — Audiobook tab | Long-form users | Medium |
|----------|------|--------|--------|-------------| | 4 | **Model config registry** | Reduce 5-dispatch-point duplication in main.py | Medium |
| 1 | **PR #33** — External provider binaries | Solves GPU distribution (19 issues), foundation for multi-model | Very High | Needs rebase, thorough review | | 5 | **#225** — Custom HuggingFace models | User-supplied models | High (needs rework for multi-engine) |
| 2 | **Multi-model abstraction layer** | Required before adding LuxTTS/Chatterbox/etc. | High | Informed by #33, #194, #225 |
| 3 | **PR #161** — Docker deployment | Server/headless users | Medium | Independent of #33 |
| 4 | **PR #194** — Hebrew + Chatterbox | First non-Qwen model, language expansion | High | Should align with multi-model abstraction |
| 5 | **PR #154** — Audiobook tab | Significant feature for long-form users | Medium | Benefits from #99 (chunking) |
### Tier 3 — Future (v0.3.0+) ### Tier 3 — Future (v0.3.0+)
| Item | Notes | | Priority | Item | Notes |
|------|-------| |----------|------|-------|
| LuxTTS integration | 48 kHz, low VRAM, but needs multi-model arch first | | 1 | **HumeAI TADA** | Long-form reliability for Stories, synced transcripts. Addresses #234, #203, #191, #111, #69. Needs API vetting. |
| XTTS-v2 / Fish Speech | Multilingual powerhouses | | 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model, broadens hardware support. Kyutai ships clean code. Needs API vetting. |
| OpenAI-compatible API (plan doc exists) | Low effort once API is stable | | 3 | **MOSS-TTS** | Text-to-voice design (no ref audio) is unique. Multi-speaker dialogue for Stories. Needs thorough API vetting. |
| LoRA fine-tuning (PR #195) | Complex, depends on #194 | | 4 | **Kokoro-82M** | 82M params, CPU realtime, Apache 2.0. Easy win. |
| External/remote providers (plan doc exists) | Depends on provider architecture | | 5 | **Model config registry refactor** | Reduce 5-dispatch-point duplication in main.py — do before adding 3+ more engines |
| GGUF support (#226) | Depends on model ecosystem maturity | | 6 | XTTS-v2 / Fish Speech / CosyVoice | Multi-engine arch is ready; just needs backend implementation |
| Queue system (#234) | Batch generation | | 7 | **VoxCPM 1.5** | Tokenizer-free streaming, interesting but uncertain integration surface |
| Real-time streaming synthesis | MLX-only currently, needs PyTorch path | | 8 | OpenAI-compatible API (plan doc exists) | Low effort once API is stable |
| 9 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine |
### Decision Point: Multi-Model Architecture | 10 | External/remote providers | Depends on use case demand |
| 11 | GGUF support (#226) | Depends on model ecosystem maturity |
Before adding any new TTS model, a decision is needed on *how*: | 12 | Queue system (#234) | Batch generation |
| 13 | Streaming for non-MLX engines | Currently MLX-only |
**Option A — Provider Binary Split (PR #33 approach)**
Each model family is a separate executable/process. Most isolated, most flexible, but most complex. Solves the CUDA distribution problem simultaneously.
**Option B — In-Process Model Registry**
Keep everything in one process but replace the singleton with a registry that can instantiate multiple `TTSBackend` implementations. Simpler, but doesn't solve binary size / CUDA distribution.
**Option C — Hybrid (Recommended)**
Use Option B for lightweight models (LuxTTS, Kokoro — small, CPU-friendly) that can coexist in-process. Use Option A for heavy models (CUDA Qwen3-TTS, Fish Speech) that need their own process/dependencies. The provider architecture from PR #33 becomes the escape hatch for heavy models, while light models are built-in.
This matches how PR #194 already works (Chatterbox loaded in-process alongside Qwen) while keeping the door open for PR #33's provider split.
--- ---
@@ -409,24 +437,20 @@ This matches how PR #194 already works (Chatterbox loaded in-process alongside Q
| Branch | PR | Status | Notes | | Branch | PR | Status | Notes |
|--------|-----|--------|-------| |--------|-----|--------|-------|
| `external-provider-binaries` | #33 | Open, stale | Major architecture work | | `feat/chatterbox-turbo` | #258 | Open | Chatterbox Turbo + per-engine languages |
| `feat/dual-server-binaries` | — | No PR | Related to provider split? | | `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
| `external-provider-binaries` | #33 | Superseded by #252 | Original CUDA provider approach |
| `feat/dual-server-binaries` | — | No PR | Related to provider split |
| `fix-multi-sample` | — | No PR | Voice profile multi-sample fix | | `fix-multi-sample` | — | No PR | Voice profile multi-sample fix |
| `fix-dl-notification-...` | — | No PR | Model download UX | | `fix-dl-notification-...` | — | No PR | Model download UX |
| `improvements` | — | No PR | Unknown scope |
| `stories` | — | No PR | Stories editor work? |
| `windows-server-shutdown` | — | No PR | Windows lifecycle |
| `model-dl-fix` | — | No PR | Model download fix |
| `channels` | — | No PR | Audio channels |
| `audio-export-entitlement-fix` | — | No PR | macOS entitlements |
| `better-docs` | — | No PR | Documentation |
--- ---
## Quick Reference: API Endpoints ## Quick Reference: API Endpoints
<details> <details>
<summary>All current endpoints (v0.1.13)</summary> <summary>All current endpoints</summary>
| Endpoint | Method | Purpose | | Endpoint | Method | Purpose |
|----------|--------|---------| |----------|--------|---------|
@@ -437,20 +461,21 @@ This matches how PR #194 already works (Chatterbox loaded in-process alongside Q
| `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management | | `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management |
| `/profiles/{id}/export` | GET | Export profile as ZIP | | `/profiles/{id}/export` | GET | Export profile as ZIP |
| `/profiles/import` | POST | Import profile from ZIP | | `/profiles/import` | POST | Import profile from ZIP |
| `/generate` | POST | Generate speech | | `/generate` | POST | Generate speech (engine param selects TTS backend) |
| `/generate/stream` | POST | Stream speech (SSE) | | `/generate/stream` | POST | Stream speech (MLX only) |
| `/history` | GET | List generation history | | `/history` | GET | List generation history |
| `/history/{id}` | GET, DELETE | Get/delete generation | | `/history/{id}` | GET, DELETE | Get/delete generation |
| `/history/{id}/export` | GET | Export generation ZIP | | `/history/{id}/export` | GET | Export generation ZIP |
| `/history/{id}/export-audio` | GET | Export audio only | | `/history/{id}/export-audio` | GET | Export audio only |
| `/transcribe` | POST | Transcribe audio (Whisper) | | `/transcribe` | POST | Transcribe audio (Whisper) |
| `/models/status` | GET | All model statuses | | `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, Whisper) |
| `/models/download` | POST | Trigger model download | | `/models/download` | POST | Trigger model download |
| `/models/download/cancel` | POST | Cancel/dismiss download |
| `/models/{name}` | DELETE | Delete downloaded model | | `/models/{name}` | DELETE | Delete downloaded model |
| `/models/load` | POST | Load model into memory | | `/models/load` | POST | Load model into memory |
| `/models/unload` | POST | Unload model | | `/models/unload` | POST | Unload model |
| `/models/progress/{name}` | GET | SSE download progress | | `/models/progress/{name}` | GET | SSE download progress |
| `/tasks/active` | GET | Active downloads/generations | | `/tasks/active` | GET | Active downloads/generations (with inline progress) |
| `/stories` | POST, GET | Create/list stories | | `/stories` | POST, GET | Create/list stories |
| `/stories/{id}` | GET, PUT, DELETE | Story CRUD | | `/stories/{id}` | GET, PUT, DELETE | Story CRUD |
| `/stories/{id}/items` | POST, GET | Story items CRUD | | `/stories/{id}/items` | POST, GET | Story items CRUD |
@@ -458,5 +483,8 @@ This matches how PR #194 already works (Chatterbox loaded in-process alongside Q
| `/channels` | POST, GET | Audio channel CRUD | | `/channels` | POST, GET | Audio channel CRUD |
| `/channels/{id}` | PUT, DELETE | Channel update/delete | | `/channels/{id}` | PUT, DELETE | Channel update/delete |
| `/cache/clear` | POST | Clear voice prompt cache | | `/cache/clear` | POST | Clear voice prompt cache |
| `/server/cuda/status` | GET | CUDA binary availability |
| `/server/cuda/download` | POST | Download CUDA binary |
| `/server/cuda/switch` | POST | Switch to CUDA backend |
</details> </details>
+2
View File
@@ -38,6 +38,8 @@ setup-python:
echo "Installing Python dependencies..." echo "Installing Python dependencies..."
{{ pip }} install --upgrade pip -q {{ pip }} install --upgrade pip -q
{{ pip }} install -r {{ backend_dir }}/requirements.txt {{ pip }} install -r {{ backend_dir }}/requirements.txt
# Chatterbox pins numpy<1.26 / torch==2.6 which break on Python 3.12+
{{ pip }} install --no-deps chatterbox-tts
# Apple Silicon: install MLX backend # Apple Silicon: install MLX backend
if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then if [ "$(uname -m)" = "arm64" ] && [ "$(uname)" = "Darwin" ]; then
echo "Detected Apple Silicon — installing MLX dependencies..." echo "Detected Apple Silicon — installing MLX dependencies..."
+5 -1
View File
@@ -40,5 +40,9 @@
"engines": { "engines": {
"bun": ">=1.0.0" "bun": ">=1.0.0"
}, },
"packageManager": "[email protected]" "packageManager": "[email protected]",
"dependencies": {
"loaders.css": "^0.1.2",
"react-loaders": "^3.0.1"
}
} }
+6 -6
View File
@@ -6,7 +6,7 @@ set -e
echo "Generating OpenAPI client..." echo "Generating OpenAPI client..."
# Check if backend is running # Check if backend is running
if ! curl -s http://localhost:8000/openapi.json > /dev/null 2>&1; then if ! curl -s http://localhost:17493/openapi.json > /dev/null 2>&1; then
echo "Backend not running. Starting backend..." echo "Backend not running. Starting backend..."
cd backend cd backend
@@ -26,19 +26,19 @@ if ! curl -s http://localhost:8000/openapi.json > /dev/null 2>&1; then
# Start backend in background # Start backend in background
echo "Starting backend server..." echo "Starting backend server..."
uvicorn main:app --port 8000 & uvicorn main:app --port 17493 & # Keep the generator on the app's documented local backend port.
BACKEND_PID=$! BACKEND_PID=$!
# Wait for server to be ready # Wait for server to be ready
echo "Waiting for server to start..." echo "Waiting for server to start..."
for i in {1..30}; do for _ in {1..30}; do
if curl -s http://localhost:8000/openapi.json > /dev/null 2>&1; then if curl -s http://localhost:17493/openapi.json > /dev/null 2>&1; then
break break
fi fi
sleep 1 sleep 1
done done
if ! curl -s http://localhost:8000/openapi.json > /dev/null 2>&1; then if ! curl -s http://localhost:17493/openapi.json > /dev/null 2>&1; then
echo "Error: Backend failed to start" echo "Error: Backend failed to start"
kill $BACKEND_PID 2>/dev/null || true kill $BACKEND_PID 2>/dev/null || true
exit 1 exit 1
@@ -52,7 +52,7 @@ fi
# Download OpenAPI schema # Download OpenAPI schema
echo "Downloading OpenAPI schema..." echo "Downloading OpenAPI schema..."
curl -s http://localhost:8000/openapi.json > app/openapi.json curl -s http://localhost:17493/openapi.json > app/openapi.json
# Check if openapi-typescript-codegen is installed # Check if openapi-typescript-codegen is installed
if ! bunx --bun openapi-typescript-codegen --version > /dev/null 2>&1; then if ! bunx --bun openapi-typescript-codegen --version > /dev/null 2>&1; then
+382
View File
@@ -0,0 +1,382 @@
#!/usr/bin/env python3
"""
Test script to observe exactly how HuggingFace reports download progress
for each TTS model. Doesn't load models — just downloads and tracks tqdm.
Usage:
backend/venv/bin/python scripts/test_download_progress.py qwen
backend/venv/bin/python scripts/test_download_progress.py luxtts
backend/venv/bin/python scripts/test_download_progress.py chatterbox
Add --delete to clear cache first and force a real download:
backend/venv/bin/python scripts/test_download_progress.py chatterbox --delete
"""
import os
import shutil
import sys
import time
import threading
from pathlib import Path
from contextlib import contextmanager
# ─── Configuration ────────────────────────────────────────────────────────────
MODELS = {
"qwen": {
"repo_id": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
"method": "from_pretrained",
"description": "Qwen TTS 1.7B (uses transformers from_pretrained)",
},
"luxtts": {
"repo_id": "YatharthS/LuxTTS",
"method": "snapshot_download",
"description": "LuxTTS (uses snapshot_download)",
},
"chatterbox": {
"repo_id": "ResembleAI/chatterbox",
"method": "snapshot_download",
"allow_patterns": [
"ve.pt",
"t3_mtl23ls_v2.safetensors",
"s3gen.pt",
"grapheme_mtl_merged_expanded_v1.json",
"conds.pt",
"Cangjie5_TC.json",
],
"description": "Chatterbox Multilingual (uses snapshot_download with allow_patterns)",
},
}
# ─── Progress tracking (mirrors our HFProgressTracker) ────────────────────────
class ProgressSpy:
"""Intercepts tqdm to see exactly what HF reports."""
def __init__(self):
self._lock = threading.Lock()
self.events = [] # List of dicts: {time, type, ...}
self._original_tqdm_class = None
self._original_tqdm_auto = None
self._patched_modules = {}
self._hf_tqdm_original_update = None
self._start_time = None
def _elapsed(self):
return time.time() - self._start_time if self._start_time else 0
def _log(self, event_type, **kwargs):
entry = {"time": f"{self._elapsed():.1f}s", "type": event_type, **kwargs}
self.events.append(entry)
# Live print
parts = [f"[{entry['time']:>7s}] {event_type:>10s}"]
for k, v in kwargs.items():
if k in ("current", "total") and isinstance(v, (int, float)) and v > 1_000_000:
parts.append(f"{k}={v / 1_000_000:.1f}MB")
else:
parts.append(f"{k}={v}")
print(" ".join(parts), flush=True)
def _create_tracked_tqdm_class(self):
spy = self
original_tqdm = self._original_tqdm_class
class SpyTqdm(original_tqdm):
def __init__(self, *args, **kwargs):
desc = kwargs.get("desc", "")
if not desc and args:
first_arg = args[0]
if isinstance(first_arg, str):
desc = first_arg
filename = ""
if desc:
if ":" in desc:
filename = desc.split(":")[0].strip()
else:
filename = desc.strip()
# Filter out non-standard kwargs
tqdm_kwargs = {
'iterable', 'desc', 'total', 'leave', 'file', 'ncols',
'mininterval', 'maxinterval', 'miniters', 'ascii', 'disable',
'unit', 'unit_scale', 'dynamic_ncols', 'smoothing',
'bar_format', 'initial', 'position', 'postfix',
'unit_divisor', 'write_bytes', 'lock_args', 'nrows',
'colour', 'color', 'delay', 'gui', 'disable_default', 'pos',
}
filtered_kwargs = {k: v for k, v in kwargs.items() if k in tqdm_kwargs}
try:
super().__init__(*args, **filtered_kwargs)
except TypeError:
super().__init__(*args, **kwargs)
self._spy_filename = filename or "unknown"
total = getattr(self, "total", None)
spy._log(
"INIT",
filename=self._spy_filename,
total=total or 0,
unit=kwargs.get("unit", "?"),
unit_scale=kwargs.get("unit_scale", False),
disable=kwargs.get("disable", False),
)
def update(self, n=1):
result = super().update(n)
current = getattr(self, "n", 0)
total = getattr(self, "total", 0)
filename = self._spy_filename
spy._log(
"UPDATE",
filename=filename,
n=n,
current=current,
total=total or 0,
pct=f"{100 * current / total:.1f}%" if total else "?",
)
return result
def close(self):
spy._log("CLOSE", filename=self._spy_filename)
return super().close()
return SpyTqdm
@contextmanager
def patch(self):
"""Context manager that patches tqdm globally — same as HFProgressTracker."""
self._start_time = time.time()
try:
import tqdm as tqdm_module
self._original_tqdm_class = tqdm_module.tqdm
except ImportError:
yield
return
tracked_tqdm = self._create_tracked_tqdm_class()
# Patch tqdm.tqdm
tqdm_module.tqdm = tracked_tqdm
# Patch tqdm.auto.tqdm
self._original_tqdm_auto = None
if hasattr(tqdm_module, "auto") and hasattr(tqdm_module.auto, "tqdm"):
self._original_tqdm_auto = tqdm_module.auto.tqdm
tqdm_module.auto.tqdm = tracked_tqdm
# Patch in sys.modules (same as HFProgressTracker)
tqdm_attr_names = ['tqdm', 'base_tqdm', 'old_tqdm']
patched_count = 0
for module_name in list(sys.modules.keys()):
if "huggingface" in module_name or module_name.startswith("tqdm"):
try:
module = sys.modules[module_name]
for attr_name in tqdm_attr_names:
if hasattr(module, attr_name):
attr = getattr(module, attr_name)
is_tqdm_class = (
attr is self._original_tqdm_class
or (self._original_tqdm_auto and attr is self._original_tqdm_auto)
or (
hasattr(attr, "__name__")
and attr.__name__ == "tqdm"
and hasattr(attr, "update")
)
)
if is_tqdm_class:
key = f"{module_name}.{attr_name}"
self._patched_modules[key] = (module, attr_name, attr)
setattr(module, attr_name, tracked_tqdm)
patched_count += 1
except (AttributeError, TypeError):
pass
# Monkey-patch HF's tqdm.update (same as HFProgressTracker)
try:
from huggingface_hub.utils import tqdm as hf_tqdm_module
if hasattr(hf_tqdm_module, 'tqdm'):
hf_tqdm_class = hf_tqdm_module.tqdm
self._hf_tqdm_original_update = hf_tqdm_class.update
spy = self
def patched_update(tqdm_self, n=1):
result = spy._hf_tqdm_original_update(tqdm_self, n)
desc = getattr(tqdm_self, 'desc', '') or ''
current = getattr(tqdm_self, 'n', 0)
total = getattr(tqdm_self, 'total', 0) or 0
spy._log(
"HF_UPDATE",
desc=desc,
current=current,
total=total,
pct=f"{100 * current / total:.1f}%" if total else "?",
)
return result
hf_tqdm_class.update = patched_update
patched_count += 1
except (ImportError, AttributeError):
pass
print(f"\n=== Patched {patched_count} tqdm references ===\n", flush=True)
try:
yield
finally:
# Restore everything
import tqdm as tqdm_module
tqdm_module.tqdm = self._original_tqdm_class
if self._original_tqdm_auto:
tqdm_module.auto.tqdm = self._original_tqdm_auto
for key, (module, attr_name, original) in self._patched_modules.items():
try:
setattr(module, attr_name, original)
except (AttributeError, TypeError):
pass
if self._hf_tqdm_original_update:
try:
from huggingface_hub.utils import tqdm as hf_tqdm_module
if hasattr(hf_tqdm_module, 'tqdm'):
hf_tqdm_module.tqdm.update = self._hf_tqdm_original_update
except (ImportError, AttributeError):
pass
def summary(self):
print("\n" + "=" * 70)
print("SUMMARY")
print("=" * 70)
inits = [e for e in self.events if e["type"] == "INIT"]
updates = [e for e in self.events if e["type"] in ("UPDATE", "HF_UPDATE")]
print(f"\ntqdm bars created: {len(inits)}")
for e in inits:
print(f" - {e.get('filename', '?'):40s} total={e.get('total', '?')}")
print(f"\nTotal update calls: {len(updates)}")
# Group updates by filename
by_file = {}
for e in updates:
fn = e.get("filename") or e.get("desc", "unknown")
if fn not in by_file:
by_file[fn] = []
by_file[fn].append(e)
for fn, evts in by_file.items():
max_current = max(e.get("current", 0) for e in evts)
max_total = max(e.get("total", 0) for e in evts)
print(f"\n {fn}:")
print(f" updates: {len(evts)}")
print(f" max current: {max_current:,}")
print(f" max total: {max_total:,}")
if max_total > 0 and max_current > 0:
print(f" final pct: {100 * max_current / max_total:.1f}%")
else:
print(f" final pct: NO PROGRESS REPORTED")
# ─── Delete cache ─────────────────────────────────────────────────────────────
def delete_cache(repo_id: str):
from huggingface_hub import constants as hf_constants
cache_dir = Path(hf_constants.HF_HUB_CACHE)
repo_cache = cache_dir / ("models--" + repo_id.replace("/", "--"))
if repo_cache.exists():
print(f"Deleting cache: {repo_cache}")
shutil.rmtree(repo_cache)
print("Deleted.")
else:
print(f"No cache found at {repo_cache}")
# ─── Download functions ───────────────────────────────────────────────────────
def download_qwen(spy: ProgressSpy):
"""Mirrors how pytorch_backend.py downloads Qwen."""
from transformers import AutoModel
repo_id = MODELS["qwen"]["repo_id"]
print(f"Downloading {repo_id} via AutoModel.from_pretrained...")
with spy.patch():
# This is what Qwen3TTSModel.from_pretrained does under the hood
from huggingface_hub import snapshot_download
snapshot_download(repo_id)
def download_luxtts(spy: ProgressSpy):
"""Mirrors how luxtts_backend.py downloads LuxTTS."""
from huggingface_hub import snapshot_download
repo_id = MODELS["luxtts"]["repo_id"]
print(f"Downloading {repo_id} via snapshot_download...")
with spy.patch():
snapshot_download(repo_id)
def download_chatterbox(spy: ProgressSpy):
"""Mirrors how chatterbox_backend.py downloads Chatterbox."""
from huggingface_hub import snapshot_download
cfg = MODELS["chatterbox"]
print(f"Downloading {cfg['repo_id']} via snapshot_download with allow_patterns...")
with spy.patch():
snapshot_download(
repo_id=cfg["repo_id"],
repo_type="model",
revision="main",
allow_patterns=cfg["allow_patterns"],
token=os.getenv("HF_TOKEN"),
)
# ─── Main ─────────────────────────────────────────────────────────────────────
def main():
if len(sys.argv) < 2 or sys.argv[1] not in MODELS:
print(f"Usage: {sys.argv[0]} <{'|'.join(MODELS.keys())}> [--delete]")
sys.exit(1)
model_key = sys.argv[1]
should_delete = "--delete" in sys.argv
cfg = MODELS[model_key]
print(f"\n{'=' * 70}")
print(f"Testing download progress for: {cfg['description']}")
print(f"Repo: {cfg['repo_id']}")
print(f"Method: {cfg['method']}")
print(f"{'=' * 70}\n")
if should_delete:
delete_cache(cfg["repo_id"])
print()
spy = ProgressSpy()
dispatch = {
"qwen": download_qwen,
"luxtts": download_luxtts,
"chatterbox": download_chatterbox,
}
try:
dispatch[model_key](spy)
except Exception as e:
print(f"\n!!! Download failed: {e}")
spy.summary()
if __name__ == "__main__":
main()
+2 -1
View File
@@ -5041,7 +5041,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]] [[package]]
name = "voicebox" name = "voicebox"
version = "0.1.12" version = "0.1.13"
dependencies = [ dependencies = [
"base64 0.22.1", "base64 0.22.1",
"core-foundation-sys", "core-foundation-sys",
@@ -5064,6 +5064,7 @@ dependencies = [
"tauri-plugin-updater", "tauri-plugin-updater",
"tokio", "tokio",
"wasapi", "wasapi",
"webkit2gtk",
"windows 0.62.2", "windows 0.62.2",
] ]
+3
View File
@@ -37,6 +37,9 @@ core-foundation-sys = "0.8"
wasapi = "0.22" wasapi = "0.22"
windows = { version = "0.62", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Com"] } windows = { version = "0.62", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Com"] }
[target.'cfg(target_os = "linux")'.dependencies]
webkit2gtk = "2.0"
[target.'cfg(not(any(target_os = "android", target_os = "ios")))'.dependencies] [target.'cfg(not(any(target_os = "android", target_os = "ios")))'.dependencies]
tauri-plugin-updater = "2.0" tauri-plugin-updater = "2.0"
tauri-plugin-process = "2.0" tauri-plugin-process = "2.0"
Binary file not shown.
+299 -3
View File
@@ -1,16 +1,312 @@
use crate::audio_capture::AudioCaptureState; use crate::audio_capture::AudioCaptureState;
use base64::{engine::general_purpose, Engine as _};
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use cpal::{SampleFormat, StreamConfig};
use hound::{WavSpec, WavWriter};
use std::io::Cursor;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
/// Start capturing system audio on Linux using PulseAudio monitor sources.
///
/// PulseAudio exposes "monitor" devices that mirror the output of each sink,
/// allowing us to capture whatever audio is currently playing on the system.
/// We use `cpal` with the default host (which will be PulseAudio or PipeWire
/// on modern Linux) and look for monitor input devices.
pub async fn start_capture( pub async fn start_capture(
state: &AudioCaptureState, state: &AudioCaptureState,
max_duration_secs: u32, max_duration_secs: u32,
) -> Result<(), String> { ) -> Result<(), String> {
todo!("implement Linux audio capture") // Reset previous samples
state.reset();
let samples = state.samples.clone();
let sample_rate_arc = state.sample_rate.clone();
let channels_arc = state.channels.clone();
let stop_tx = state.stop_tx.clone();
let error_arc = state.error.clone();
// Use AtomicBool for stop signal (works across threads)
let stop_flag = Arc::new(AtomicBool::new(false));
let stop_flag_clone = stop_flag.clone();
// Create tokio channel and spawn a task to bridge it to the AtomicBool
let (tx, mut rx) = tokio::sync::mpsc::channel::<()>(1);
*stop_tx.lock().unwrap() = Some(tx);
tokio::spawn(async move {
rx.recv().await;
stop_flag_clone.store(true, Ordering::Relaxed);
});
// Spawn capture on a dedicated thread
thread::spawn(move || {
let host = cpal::default_host();
// Try to find a monitor device for system audio capture.
// On PulseAudio/PipeWire, monitor sources have "monitor" in their name.
let device = {
let mut monitor_device = None;
if let Ok(devices) = host.input_devices() {
for d in devices {
if let Ok(name) = d.name() {
let name_lower = name.to_lowercase();
if name_lower.contains("monitor") {
eprintln!("Linux audio capture: Found monitor device: {}", name);
monitor_device = Some(d);
break;
}
}
}
}
match monitor_device {
Some(d) => d,
None => {
// Fallback to default input device (microphone)
eprintln!("Linux audio capture: No monitor device found, falling back to default input");
match host.default_input_device() {
Some(d) => d,
None => {
let error_msg = "No audio input device available".to_string();
eprintln!("{}", error_msg);
*error_arc.lock().unwrap() = Some(error_msg);
return;
}
}
}
}
};
let device_name = device.name().unwrap_or_else(|_| "unknown".to_string());
eprintln!("Linux audio capture: Using device: {}", device_name);
// Get supported config
let config = match device.default_input_config() {
Ok(c) => c,
Err(e) => {
let error_msg = format!("Failed to get default input config: {}", e);
eprintln!("{}", error_msg);
*error_arc.lock().unwrap() = Some(error_msg);
return;
}
};
let sample_rate = config.sample_rate().0;
let channels = config.channels();
let sample_format = config.sample_format();
eprintln!(
"Linux audio capture: Config - {}Hz, {} channels, format: {:?}",
sample_rate, channels, sample_format
);
*sample_rate_arc.lock().unwrap() = sample_rate;
*channels_arc.lock().unwrap() = channels;
let stream_config = StreamConfig {
channels,
sample_rate: cpal::SampleRate(sample_rate),
buffer_size: cpal::BufferSize::Default,
};
let samples_clone = samples.clone();
let error_arc_clone = error_arc.clone();
let stop_flag_for_stream = stop_flag.clone();
let err_fn = {
let error_arc = error_arc.clone();
move |err: cpal::StreamError| {
let error_msg = format!("Stream error: {}", err);
eprintln!("{}", error_msg);
*error_arc.lock().unwrap() = Some(error_msg);
}
};
let stream = match sample_format {
SampleFormat::F32 => {
let samples = samples_clone.clone();
let stop = stop_flag_for_stream.clone();
device.build_input_stream(
&stream_config,
move |data: &[f32], _: &cpal::InputCallbackInfo| {
if stop.load(Ordering::Relaxed) {
return;
}
let mut guard = samples.lock().unwrap();
guard.extend_from_slice(data);
},
err_fn,
None,
)
}
SampleFormat::I16 => {
let samples = samples_clone.clone();
let stop = stop_flag_for_stream.clone();
device.build_input_stream(
&stream_config,
move |data: &[i16], _: &cpal::InputCallbackInfo| {
if stop.load(Ordering::Relaxed) {
return;
}
let mut guard = samples.lock().unwrap();
for &s in data {
guard.push(s as f32 / 32768.0);
}
},
err_fn,
None,
)
}
SampleFormat::U16 => {
let samples = samples_clone.clone();
let stop = stop_flag_for_stream.clone();
device.build_input_stream(
&stream_config,
move |data: &[u16], _: &cpal::InputCallbackInfo| {
if stop.load(Ordering::Relaxed) {
return;
}
let mut guard = samples.lock().unwrap();
for &s in data {
guard.push((s as f32 / 32768.0) - 1.0);
}
},
err_fn,
None,
)
}
_ => {
let error_msg = format!("Unsupported sample format: {:?}", sample_format);
eprintln!("{}", error_msg);
*error_arc_clone.lock().unwrap() = Some(error_msg);
return;
}
};
let stream = match stream {
Ok(s) => s,
Err(e) => {
let error_msg = format!("Failed to build input stream: {}", e);
eprintln!("{}", error_msg);
*error_arc_clone.lock().unwrap() = Some(error_msg);
return;
}
};
if let Err(e) = stream.play() {
let error_msg = format!("Failed to start stream: {}", e);
eprintln!("{}", error_msg);
*error_arc_clone.lock().unwrap() = Some(error_msg);
return;
}
eprintln!("Linux audio capture: Stream started successfully");
// Keep thread alive until stop signal
loop {
if stop_flag.load(Ordering::Relaxed) {
break;
}
std::thread::sleep(std::time::Duration::from_millis(100));
}
// Stream will be dropped here, stopping capture
eprintln!("Linux audio capture: Stream stopped");
});
// Spawn timeout task
let stop_tx_clone = state.stop_tx.clone();
tokio::spawn(async move {
tokio::time::sleep(tokio::time::Duration::from_secs(max_duration_secs as u64)).await;
let tx = stop_tx_clone.lock().unwrap().take();
if let Some(tx) = tx {
let _ = tx.send(()).await;
}
});
Ok(())
} }
pub async fn stop_capture(state: &AudioCaptureState) -> Result<String, String> { pub async fn stop_capture(state: &AudioCaptureState) -> Result<String, String> {
todo!("implement Linux audio capture stop") // Signal stop
if let Some(tx) = state.stop_tx.lock().unwrap().take() {
let _ = tx.send(());
}
// Wait a bit for capture to stop
tokio::time::sleep(tokio::time::Duration::from_millis(500)).await;
// Check if there was an error during capture
if let Some(error) = state.error.lock().unwrap().as_ref() {
return Err(error.clone());
}
// Get samples
let samples = state.samples.lock().unwrap().clone();
let sample_rate = *state.sample_rate.lock().unwrap();
let channels = *state.channels.lock().unwrap();
if samples.is_empty() {
return Err(
"No audio samples captured. Make sure audio is playing on your system during recording."
.to_string(),
);
}
// Convert to WAV
let wav_data = samples_to_wav(&samples, sample_rate, channels)?;
// Encode to base64
let base64_data = general_purpose::STANDARD.encode(&wav_data);
Ok(base64_data)
} }
pub fn is_supported() -> bool { pub fn is_supported() -> bool {
false // Check if we can find a monitor device for system audio capture
let host = cpal::default_host();
if let Ok(devices) = host.input_devices() {
for d in devices {
if let Ok(name) = d.name() {
if name.to_lowercase().contains("monitor") {
return true;
}
}
}
}
// Even without a monitor, basic input capture is available
host.default_input_device().is_some()
}
fn samples_to_wav(samples: &[f32], sample_rate: u32, channels: u16) -> Result<Vec<u8>, String> {
let mut buffer = Vec::new();
let cursor = Cursor::new(&mut buffer);
let spec = WavSpec {
channels,
sample_rate,
bits_per_sample: 16,
sample_format: hound::SampleFormat::Int,
};
let mut writer =
WavWriter::new(cursor, spec).map_err(|e| format!("Failed to create WAV writer: {}", e))?;
// Convert f32 samples to i16
for sample in samples {
let clamped = sample.clamp(-1.0, 1.0);
let i16_sample = (clamped * 32767.0) as i16;
writer
.write_sample(i16_sample)
.map_err(|e| format!("Failed to write sample: {}", e))?;
}
writer
.finalize()
.map_err(|e| format!("Failed to finalize WAV: {}", e))?;
Ok(buffer)
} }
+75 -3
View File
@@ -16,6 +16,7 @@ struct ServerState {
child: Mutex<Option<tauri_plugin_shell::process::CommandChild>>, child: Mutex<Option<tauri_plugin_shell::process::CommandChild>>,
server_pid: Mutex<Option<u32>>, server_pid: Mutex<Option<u32>>,
keep_running_on_close: Mutex<bool>, keep_running_on_close: Mutex<bool>,
models_dir: Mutex<Option<String>>,
} }
#[command] #[command]
@@ -23,7 +24,16 @@ async fn start_server(
app: tauri::AppHandle, app: tauri::AppHandle,
state: State<'_, ServerState>, state: State<'_, ServerState>,
remote: Option<bool>, remote: Option<bool>,
models_dir: Option<String>,
) -> Result<String, String> { ) -> Result<String, String> {
// Store models_dir for use on restart (empty string means reset to default)
if let Some(ref dir) = models_dir {
if dir.is_empty() {
*state.models_dir.lock().unwrap() = None;
} else {
*state.models_dir.lock().unwrap() = Some(dir.clone());
}
}
// Check if server is already running (managed by this app instance) // Check if server is already running (managed by this app instance)
if state.child.lock().unwrap().is_some() { if state.child.lock().unwrap().is_some() {
return Ok(format!("http://127.0.0.1:{}", SERVER_PORT)); return Ok(format!("http://127.0.0.1:{}", SERVER_PORT));
@@ -274,6 +284,12 @@ async fn start_server(
let port_str = SERVER_PORT.to_string(); let port_str = SERVER_PORT.to_string();
let is_remote = remote.unwrap_or(false); let is_remote = remote.unwrap_or(false);
// Resolve the custom models directory from the parameter or stored state
let effective_models_dir = models_dir.or_else(|| state.models_dir.lock().unwrap().clone());
if let Some(ref dir) = effective_models_dir {
println!("Custom models directory: {}", dir);
}
// If CUDA binary exists, launch it directly instead of the bundled sidecar // If CUDA binary exists, launch it directly instead of the bundled sidecar
let spawn_result = if let Some(ref cuda_path) = cuda_binary { let spawn_result = if let Some(ref cuda_path) = cuda_binary {
println!("Launching CUDA backend: {:?}", cuda_path); println!("Launching CUDA backend: {:?}", cuda_path);
@@ -282,6 +298,9 @@ async fn start_server(
if is_remote { if is_remote {
cmd = cmd.args(["--host", "0.0.0.0"]); cmd = cmd.args(["--host", "0.0.0.0"]);
} }
if let Some(ref dir) = effective_models_dir {
cmd = cmd.env("VOICEBOX_MODELS_DIR", dir);
}
cmd.spawn() cmd.spawn()
} else { } else {
// Use the bundled CPU sidecar // Use the bundled CPU sidecar
@@ -289,6 +308,9 @@ async fn start_server(
if is_remote { if is_remote {
sidecar = sidecar.args(["--host", "0.0.0.0"]); sidecar = sidecar.args(["--host", "0.0.0.0"]);
} }
if let Some(ref dir) = effective_models_dir {
sidecar = sidecar.env("VOICEBOX_MODELS_DIR", dir);
}
println!("Spawning server process..."); println!("Spawning server process...");
sidecar.spawn() sidecar.spawn()
}; };
@@ -613,9 +635,19 @@ async fn stop_server(state: State<'_, ServerState>) -> Result<(), String> {
async fn restart_server( async fn restart_server(
app: tauri::AppHandle, app: tauri::AppHandle,
state: State<'_, ServerState>, state: State<'_, ServerState>,
models_dir: Option<String>,
) -> Result<String, String> { ) -> Result<String, String> {
println!("restart_server: stopping current server..."); println!("restart_server: stopping current server...");
// Update stored models_dir: empty string means reset to default, non-empty means set
if let Some(ref dir) = models_dir {
if dir.is_empty() {
*state.models_dir.lock().unwrap() = None;
} else {
*state.models_dir.lock().unwrap() = Some(dir.clone());
}
}
// Stop the current server // Stop the current server
stop_server(state.clone()).await?; stop_server(state.clone()).await?;
@@ -623,9 +655,9 @@ async fn restart_server(
println!("restart_server: waiting for port release..."); println!("restart_server: waiting for port release...");
tokio::time::sleep(tokio::time::Duration::from_millis(1000)).await; tokio::time::sleep(tokio::time::Duration::from_millis(1000)).await;
// Start server again (will auto-detect CUDA binary) // Start server again (will auto-detect CUDA binary and use stored models_dir)
println!("restart_server: starting server..."); println!("restart_server: starting server...");
start_server(app, state, None).await start_server(app, state, None, None).await
} }
#[command] #[command]
@@ -686,6 +718,7 @@ pub fn run() {
child: Mutex::new(None), child: Mutex::new(None),
server_pid: Mutex::new(None), server_pid: Mutex::new(None),
keep_running_on_close: Mutex::new(false), keep_running_on_close: Mutex::new(false),
models_dir: Mutex::new(None),
}) })
.manage(audio_capture::AudioCaptureState::new()) .manage(audio_capture::AudioCaptureState::new())
.manage(audio_output::AudioOutputState::new()) .manage(audio_output::AudioOutputState::new())
@@ -714,6 +747,43 @@ pub fn run() {
} }
} }
// Enable microphone access on Linux (WebKitGTK denies getUserMedia by default)
#[cfg(target_os = "linux")]
{
use tauri::Manager;
if let Some(window) = app.get_webview_window("main") {
let _ = window.with_webview(|webview| {
use webkit2gtk::{WebViewExt, SettingsExt, PermissionRequestExt};
use webkit2gtk::glib::ObjectExt;
let wk_webview = webview.inner();
// Enable media stream support in WebKitGTK settings
if let Some(settings) = WebViewExt::settings(&wk_webview) {
settings.set_enable_media_stream(true);
}
// Auto-grant UserMediaPermissionRequest (microphone access)
// Only for trusted local origins (Tauri dev server or custom protocol)
wk_webview.connect_permission_request(move |webview, request: &webkit2gtk::PermissionRequest| {
if request.is::<webkit2gtk::UserMediaPermissionRequest>() {
let uri = WebViewExt::uri(webview).unwrap_or_default();
let is_trusted = uri.starts_with("tauri://")
|| uri.starts_with("https://tauri.localhost")
|| uri.starts_with("http://localhost")
|| uri.starts_with("http://127.0.0.1");
if is_trusted {
request.allow();
return true;
}
request.deny();
return true;
}
false
});
});
}
}
Ok(()) Ok(())
}) })
.invoke_handler(tauri::generate_handler![ .invoke_handler(tauri::generate_handler![
@@ -755,7 +825,9 @@ pub fn run() {
}); });
// Wait for frontend response or timeout // Wait for frontend response or timeout
tokio::spawn(async move { // Use tauri::async_runtime::spawn instead of tokio::spawn to avoid
// panics when the Tokio runtime is being dropped during app shutdown
tauri::async_runtime::spawn(async move {
tokio::select! { tokio::select! {
_ = rx.recv() => { _ = rx.recv() => {
// Frontend responded, close window // Frontend responded, close window
+1 -1
View File
@@ -56,7 +56,7 @@
}, },
"plugins": { "plugins": {
"shell": { "shell": {
"open": true "open": ".*"
}, },
"updater": { "updater": {
"pubkey": "dW50cnVzdGVkIGNvbW1lbnQ6IG1pbmlzaWduIHB1YmxpYyBrZXk6IEUxRENBQkRBQjdBNTM1OTIKUldTU05hVzMycXZjNGJGcUxmcVVocll2QjdSaTJNdlFxR2M3VDJsMnVvbDdyZGRPMmRlOW9aWTcK", "pubkey": "dW50cnVzdGVkIGNvbW1lbnQ6IG1pbmlzaWduIHB1YmxpYyBrZXk6IEUxRENBQkRBQjdBNTM1OTIKUldTU05hVzMycXZjNGJGcUxmcVVocll2QjdSaTJNdlFxR2M3VDJsMnVvbDdyZGRPMmRlOW9aWTcK",
+9 -4
View File
@@ -5,9 +5,12 @@ import type { PlatformLifecycle } from '@/platform/types';
class TauriLifecycle implements PlatformLifecycle { class TauriLifecycle implements PlatformLifecycle {
onServerReady?: () => void; onServerReady?: () => void;
async startServer(remote = false): Promise<string> { async startServer(remote = false, modelsDir?: string | null): Promise<string> {
try { try {
const result = await invoke<string>('start_server', { remote }); const result = await invoke<string>('start_server', {
remote,
modelsDir: modelsDir ?? undefined,
});
console.log('Server started:', result); console.log('Server started:', result);
this.onServerReady?.(); this.onServerReady?.();
return result; return result;
@@ -27,9 +30,11 @@ class TauriLifecycle implements PlatformLifecycle {
} }
} }
async restartServer(): Promise<string> { async restartServer(modelsDir?: string | null): Promise<string> {
try { try {
const result = await invoke<string>('restart_server'); const result = await invoke<string>('restart_server', {
modelsDir: modelsDir ?? undefined,
});
console.log('Server restarted:', result); console.log('Server restarted:', result);
this.onServerReady?.(); this.onServerReady?.();
return result; return result;
+2 -2
View File
@@ -3,7 +3,7 @@ import type { PlatformLifecycle } from '@/platform/types';
class WebLifecycle implements PlatformLifecycle { class WebLifecycle implements PlatformLifecycle {
onServerReady?: () => void; onServerReady?: () => void;
async startServer(_remote = false): Promise<string> { async startServer(_remote = false, _modelsDir?: string | null): Promise<string> {
// Web assumes server is running externally // Web assumes server is running externally
// Return a default URL - this should be configured via env vars // Return a default URL - this should be configured via env vars
const serverUrl = import.meta.env.VITE_SERVER_URL || 'http://localhost:17493'; const serverUrl = import.meta.env.VITE_SERVER_URL || 'http://localhost:17493';
@@ -15,7 +15,7 @@ class WebLifecycle implements PlatformLifecycle {
// No-op for web - server is managed externally // No-op for web - server is managed externally
} }
async restartServer(): Promise<string> { async restartServer(_modelsDir?: string | null): Promise<string> {
// No-op for web - server is managed externally // No-op for web - server is managed externally
return import.meta.env.VITE_SERVER_URL || 'http://localhost:17493'; return import.meta.env.VITE_SERVER_URL || 'http://localhost:17493';
} }