POST /speak is a REST wrapper around voicebox.speak for agents that
don't talk MCP (shell scripts, ACP, A2A). It reads X-Voicebox-Client-Id
and uses it for the same per-client profile resolution + default
personality lookup the MCP tool does (speak.py:39-64), so its callers
are first-class clients — but the ClientIdMiddleware only stamped
last_seen_at on /mcp* paths. REST speak callers showed up as "never
seen" in Settings → MCP despite actively acting on their bindings.
Widen the stamp predicate to an explicit ("/mcp", "/speak") prefix
list, and require a path boundary on match so future routes named
/mcpfoo or /speakers don't silently inherit the stamp via the prefix.
New test_client_id_middleware.py pins the scope with 17 parametrised
cases (both the allowed set and the overlap cases that must not match).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The word-level pass catches single-word Whisper loops ("URL URL URL…")
but misses two common hallucination patterns the PR had to claim as
"edge cases":
1. Multi-word English loops — "thanks for watching thanks for watching…"
× 6 sails through because no two consecutive tokens are identical
after text.split().
2. CJK loops — "謝謝觀看" × 7 sails through because text.split() returns
a single unsplit token for the whole loop (no whitespace between
characters).
Add a character-level second pass: a non-greedy regex finds any 2–60
char substring that repeats min_run+ times immediately after itself and
strips the run. The 2-char floor keeps emphasised single-letter runs
("wooooooow") intact. The 60-char ceiling covers every observed
Whisper tail hallucination ("Please like and subscribe to my
channel.", "Subtitles by the Amara.org community") while staying short
enough that coincidental long-phrase repetition in legitimate speech
doesn't hit the threshold. Whitespace normalisation only runs when the
pass actually stripped something, so untouched transcripts keep their
original spacing.
New test_refinement_collapse.py gives the pre-processor its first
deterministic unit-test coverage: 17 tests pinning the word-level
legacy behaviour plus the new multi-word English / CJK / Japanese /
emphasis-preservation cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- Collapse intent tri-state (respond/rewrite/compose) to `personality: bool` on /generate, /speak, and voicebox.speak. Drop respond entirely; keep compose as a standalone button via /profiles/{id}/compose. Remove /rewrite, /respond, and /speak profile endpoints.
- FloatingGenerateBox: Wand2 persona toggle + Dices compose button appear when the selected profile has a personality. ProfileCard badges Wand2 alongside the effects Sparkles.
- MCP bindings: default_intent column → default_personality: bool. Migration drops the legacy column.
- i18n: en / ja / zh-CN / zh-TW translation files filled out and wired through the capture, server, and profile UI.
```ts
voicebox.speak({
text: "Deploy complete.",
profile: "Morgan",
personality: true, // rewrite through the profile's personality LLM
});
```
Ships the Capture release end to end. Global-hotkey dictation with
synthetic paste into the focused app on macOS and Windows, an on-screen
pill across recording / transcribing / refining, customizable push-to-
talk and toggle chords, and an accessibility-permission prompt scoped to
Settings → Captures with inline re-check feedback.
Voice profiles gain optional personalities that power compose / rewrite /
respond actions via a local Qwen3 LLM — shared with refinement, so there
is one local LLM in the app, not two.
Refinement hardened with deterministic Whisper-loop collapse before the
LLM sees the transcript, per-capture flag snapshots for re-runs, and a
ten-transcript evaluation harness across every bundled refinement size.
Version bump 0.4.5 → 0.5.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(offline): patch transformers mistral-regex check to survive HF failures
transformers 4.57.x's `PreTrainedTokenizerBase._patch_mistral_regex` calls
`huggingface_hub.model_info(repo_id)` unconditionally during any non-local
tokenizer load to probe for Mistral-family models. The call raises on
`HF_HUB_OFFLINE=1`, on network outages, and on slow/blocked HF endpoints,
and transformers doesn't catch any of it — the exception bubbles out of
`from_pretrained` and kills the load for unrelated engines (Qwen TTS,
Qwen CustomVoice, TADA, etc.).
0.4.2's load-time `force_offline_if_cached` guard walked straight into
this trap: on cached online users it flipped `HF_HUB_OFFLINE=1` and
converted a healthy load into a hard crash. 0.4.3's inference-path guard
masked it; #524 removed the inference guard in 0.4.4, and users updating
to 0.4.4 started hitting the same error on the load path instead
(#526).
Fix:
- Wrap `_patch_mistral_regex` so any exception from the inner HF
metadata check is swallowed and the tokenizer is returned unchanged.
Voicebox never loads Mistral models, so the regex rewrite this check
gates is a no-op for us; matches the success-path behavior for
non-Mistral repos (tokenization_utils_base.py:2503).
- Drop the `force_offline_if_cached` wraps from every load path
(pytorch_backend Qwen + Whisper, qwen_custom_voice_backend,
mlx_backend Qwen + Whisper). With the mistral patch in place they
provide zero value and only risk re-introducing the same class of
bug. Helper and its unit tests stay — still correct for targeted
future use.
- Add `backend/tests/test_offline_patch.py` covering
OfflineModeIsEnabled / ConnectionError suppression, success
pass-through, idempotence, and the missing-method no-op path.
Fixes#526.
* fix(offline): install mistral-regex patch for non-MLX backends
The previous commit left the patch wired only through ``mlx_backend.py``'s
existing import of ``hf_offline_patch``. On Windows/Linux/CUDA users who
never load the MLX backend (everyone who hit #526), the patch module was
never imported, so ``patch_transformers_mistral_regex`` never ran and the
crash persisted.
Hoist the import into ``backends/__init__.py``. Every backend imports from
this package, so the module-level patch install runs before any
``from_pretrained`` call regardless of which engine the user picks.
Caught by CodeRabbit and Cursor Bugbot on #530.
* fix(offline): guard inference paths with HF_HUB_OFFLINE (#462)
PR #443 wrapped the model *load* path with `force_offline_if_cached` so
cached models don't phone home at startup. The context manager restores
`HF_HUB_OFFLINE` on exit, which left inference paths (generate,
transcribe, voice-prompt creation) unguarded — and `qwen_tts`,
`mlx_audio`, and `transformers` perform lazy tokenizer/processor/config
lookups during inference. With internet on, those lookups are
near-instant and invisible; with internet off, `requests` hangs on DNS
or connect until the network returns. This is exactly what users in
#462 describe: model shows "Loaded", internet drops, generation
"thinks" forever, internet comes back, generation completes.
Chatterbox and LuxTTS don't exhibit this because their engine libs
resolve everything through already-cached paths at load time.
Fix: wrap each inference-sync body with `force_offline_if_cached(True,
...)`. Since inference only runs after a successful load, weights are
known to be on disk, so `is_cached=True` is unconditional.
Also adds the load-time guard that was missing from
`qwen_custom_voice_backend.py` — CustomVoice previously had no offline
protection at all.
Paths patched:
- PyTorchTTSBackend.create_voice_prompt (create_voice_clone_prompt)
- PyTorchTTSBackend.generate (generate_voice_clone)
- PyTorchSTTBackend.transcribe (Whisper generate + decoder-prompt-ids)
- MLXTTSBackend.generate (mlx_audio generate, all branches)
- MLXSTTBackend.transcribe (mlx_audio whisper generate)
- QwenCustomVoiceBackend._load_model_sync + generate
Does not address the secondary `check_model_inputs() missing 'func'`
error reported in the same issue — that's a `transformers` 5.x
version-skew bug on the install path, separate concern.
Fixes#462.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(offline): mutate cached HF constants + threadsafe refcount
Review feedback on the initial fix surfaced two real issues:
1. ``os.environ`` toggles alone don't flip offline mode.
``huggingface_hub.constants.HF_HUB_OFFLINE`` is read once at import
time into a module-level bool; ``transformers.utils.hub._is_offline_mode``
mirrors that bool at its own import time. The hot paths
(``_http._default_backend_factory`` in huggingface_hub,
``is_offline_mode`` in transformers) read the cached bools — not the
env — so mutating only ``os.environ`` was a no-op.
2. Race condition on concurrent inference. Two threads running inside
``force_offline_if_cached`` via ``asyncio.to_thread`` could have
thread A's ``finally`` strip thread B's offline protection mid-run.
Rewrite the helper to:
- mutate ``huggingface_hub.constants.HF_HUB_OFFLINE`` and
``transformers.utils.hub._is_offline_mode`` directly
- refcount concurrent users under a single ``threading.RLock`` so a
shared offline window is restored only when the last caller exits
- still write ``os.environ`` for anything that reads it dynamically
Also addresses the unused-variable ruff flag on the Whisper transcribe
path (``audio, sr`` → ``audio, _sr``).
New unit tests cover the cached-constant mutation, env propagation,
no-op on ``is_cached=False``, nested contexts, and a threaded race
where a slow thread must retain offline mode after a peer exits.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(offline): atomic entry rollback + tidy test assertions
Review follow-up:
- Wrap the `_offline_refcount == 0` setup in a try/except so any failure
during the cached-constant mutation (including unexpected non-ImportError
like RuntimeError or AttributeError from a half-initialized module)
rolls back *all* partial state before re-raising. Without this, a
mid-setup crash could leave `huggingface_hub.constants.HF_HUB_OFFLINE`
mutated but the refcount at 0 — a persistent offline flag outliving
the process.
- Swap ruff-flagged Yoda comparisons in the new test file (SIM300) and
add a module-level note warning that these tests mutate global state
and are not safe under cross-process parallelism.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* test(offline): make concurrency test deterministic and bounded
Replace the `sleep(0.15)` ordering hack with an explicit `threading.Event`
the fast thread sets in `finally`. The slow thread waits on that event
(bounded), then observes the flag — so we deterministically verify the
slow thread still sees offline mode after the fast thread has exited.
Also add timeouts to `barrier.wait()` and assert `not thread.is_alive()`
after the joins so the test can't hang on an unexpected failure path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* fix(audio): preprocess reference samples instead of rejecting them
Uploaded/recorded voice samples were rejected outright whenever the peak
exceeded 0.99 ("Audio is clipping (reduce input gain)"). That wasn't
actionable: a recording in the app has no pre-gain control, and an
already-captured file can't be re-taken by the user. The Settings
"Normalize audio" toggle only affects generated TTS output, so users who
enabled it expecting it to help with sample uploads were still blocked.
Replace the hard reject with a small, always-on preprocess step that
runs right after load:
- DC-offset removal
- Conservative edge-silence trim (top_db=30) with 100 ms padding kept
- Peak cap at 0.95 if the input peak exceeds that
Duration and RMS checks now run on the preprocessed waveform, so
samples that were previously rejected for being "hot" are accepted and
stored with safe headroom. True in-waveform clipping artifacts still
can't be repaired — peak scaling only prevents downstream re-clipping
during multi-sample combination and TTS inference.
Adds a unit-test file (previously none existed for audio.py).
Fixes#456.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(audio): raise trim threshold, cap pad at net-neutral
Review feedback on the preprocessor:
1. ``trim_top_db=30`` was labelled "conservative" in the docstring but is
actually *more* aggressive than librosa's default of 60. Normal
speech dynamic range sits around 30 dB, so 30 dB would eat quiet
trailing syllables and soft consonants. Raise the default to 40 dB —
below normal speech dynamic range but still catching obvious edge
silence — and fix the docstring.
2. Unconditional 100 ms edge padding ran even when ``librosa.effects.trim``
removed nothing. For a well-recorded 29.9 s upload that path would
push the waveform past the 30 s ceiling and trigger a spurious "too
long" rejection. Only pad when trimming actually shortened the
audio, and cap the pad so the output never exceeds the input length.
Adds a regression test for the net-neutral length behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
The 0.6B slot was aliased to the 1.7B repo as a temporary fallback
because `mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16` wasn't published
when MLX support shipped. That conversion is live now, so use it —
Apple Silicon users picking 0.6B get the actual 0.6B model (1.2 GB
instead of 3.5 GB).
Also drops the now-obsolete troubleshooting entry and updates the
triage notes in PROJECT_STATUS.md.
Fixes#485.
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
- Add validation in create_profile() to check for existing names before insert
- Add validation in update_profile() to prevent renaming to duplicate names
- Improve error handling in API endpoints with user-friendly messages
- Add comprehensive test suite for duplicate name validation
- Update CHANGELOG.md with fix details
This fix prevents database constraint violations and provides clear
error messages when users attempt to create or update profiles with
names that already exist in the database.
20 tests covering:
- All 6 default local origins are allowed
- Arbitrary external origins are blocked
- Preflight (OPTIONS) requests respect the allowlist
- VOICEBOX_CORS_ORIGINS env var extends the allowlist
- Edge cases: empty env, whitespace trimming, trailing commas
Tests use a minimal FastAPI app mirroring the real CORS config,
so they run without ML dependencies (torch, numpy, etc.).
- Rearranged imports for consistency in useModelDownloadToast hook.
- Improved logging in useModelDownloadToast for better debugging during download events.
- Updated progress calculation to handle cases where progress exceeds 100%.
- Enhanced toast notifications to reflect download completion and error states.
- Introduced throttling in ProgressManager to optimize SSE updates and prevent overwhelming clients.
- Added new test scripts for monitoring SSE events during model downloads, ensuring accurate progress reporting.
- Introduced a new directory for manual test scripts aimed at debugging and validating backend functionality.
- Added README.md detailing the purpose and usage of various test scripts, including tests for TTS generation, model downloads, and progress tracking.
- Included an __init__.py file to define the test suite structure and provide context for the tests.
- Introduced methods to check if models are cached locally in MLX and PyTorch backends.
- Enhanced progress tracking during model loading to filter out non-download progress when models are cached.
- Updated HFProgressTracker to conditionally report progress based on download status.
- Added test scripts for monitoring SSE events during model downloads and verifying progress tracking functionality.
- Improved overall error handling and logging for better debugging during model download processes.
