Review follow-ups: clear_voice_prompt_memory_cache() now runs before
backend.unload_model() at all three call sites so the device-resident
prompt tensors are already released when empty_device_cache() /
empty_mlx_cache() runs, instead of going back into the caching allocator
afterwards. empty_mlx_cache's docstring states why it may run off the
MLX worker thread.
Unloading a TTS/Whisper/LLM model on the MLX backend only dropped the
Python reference (`del self.model`). MLX keeps freed array buffers in
its own allocator pool for reuse instead of returning them to the OS,
so the process's memory footprint never actually shrank after unload
on Apple Silicon (the default backend there) until the process exited.
Add empty_mlx_cache() (backend/backends/base.py), wrapping
mx.clear_cache(), and call it from the three MLX unload_model()
implementations: MLXTTSBackend, MLXSTTBackend, MLXQwenLLMBackend.
Separately, the voice-clone prompt cache (backend/utils/cache.py) is a
process-lifetime dict populated by create_voice_prompt() across every
TTS engine, but nothing ever cleared it on model unload — only the
unrelated /tasks/clear-cache endpoint touched it. Add
clear_voice_prompt_memory_cache() (memory only, disk cache untouched
so a later generation still reloads the prompt instead of recomputing
it) and wire it into every TTS unload path (services/tts.py and the
qwen_custom_voice / generic branches of unload_model_by_config).
Whisper and the LLM backends never produce voice prompts, so their
unload paths are left alone.
Testing:
- New unit tests: backend/tests/test_mlx_unload_clears_cache.py,
backend/tests/test_voice_prompt_cache_unload.py (8 tests, all pass).
- Verified end-to-end on Apple Silicon against real cached models
(Qwen TTS 1.7B, Whisper Turbo, Qwen3 0.6B): loaded each via the
running app, unloaded via the real /models/{name}/unload endpoint,
and confirmed via mx.get_cache_memory()/get_active_memory() that the
MLX allocator's cache drops to 0 on every cycle. Ran a real
voice-clone generation end to end and confirmed the in-memory prompt
cache goes from 1 entry to 0 on unload while the on-disk .prompt
file is left intact.
Initialise tts_model before the try so the finally can read its device
without a locals() probe, import empty_device_cache alongside the other
backend imports instead of inside a bare try/except, and flatten the MPS
branch in empty_device_cache.
is_model_cached() marked a model as not-cached whenever any .incomplete
blob existed in its cache dir, even when a completed blob with the same
hash already sat next to it. A retried/concurrent download can leave
this orphan behind after the real transfer already finished, which made
the model appear perpetually "downloading" and re-trigger a full
re-download on every load.
Only .incomplete files with no matching completed blob now count as a
genuinely in-progress download.
Without a trace the Settings GPU label and /health keep reporting CUDA
while generation runs on CPU, so the user has no way to confirm the
override engaged.
The override is documented in gpu-acceleration.mdx and listed as step 1 of
the get_torch_device() precedence in tts-generation.mdx, but grepping the
tree for VOICEBOX_FORCE_CPU matched only those two doc files - nothing read
it. Users whose GPU has no compiled kernels in the bundled PyTorch had no
way to fall back to CPU short of renaming the installed CUDA backend
directory.
Resolve it before torch is imported, so it still works when the installed
build is itself the reason CPU is wanted.
model_load_progress() already received is_cached but never forwarded it
to force_offline_if_cached(), which is fully unit-tested but had zero
callers. A fully-cached model still resolved every config file against
huggingface.co, eating the default 5-retry backoff per file offline.
* feat(windows): add native ROCm support for AMD GPUs
Implements native ROCm architecture for Windows.
- Adds backend build pipeline for voicebox-server-rocm.exe
- Detects AMD GPUs dynamically and routes PyTorch allocations
- Adds automatic download and update logic for ROCm dependencies
- Refactors UI in GpuPage.tsx and GpuAcceleration.tsx to add AMD flows
- Fixes 'Switch to CPU' lock on Windows via Tauri backend_override state
- Resolves PyInstaller/rocm_sdk UnboundLocalError silent crashes
- Resolves Numba/NumPy 2.x incompatibilities during Qwen3-TTS load
- Resolves HF_HUB_OFFLINE Catch-22 for CustomVoice processor caching
* fix(rocm): host libs archive under the app release tag, drop offline-load regression
Align the ROCm libs download with the CUDA pattern: both the server core and
the libs archive are published under the app-version release tag, with the libs
content version encoded in the filename only. The previous code fetched libs
from a separate rocm7.2-v1 tag, which disagreed with the download test.
Also revert the unrelated Qwen CustomVoice changes that wrapped model loading in
force_offline_if_cached (not imported — a NameError on load for every platform)
and re-added a Base-model cache gate. The inference-path offline guard was
deliberately removed previously.
* feat(rocm): gate download on AMD detection and persist the backend variant
The ROCm download section now only shows when the backend reports an AMD GPU on
Windows (new supports_rocm health field, backed by the memoized
is_amd_gpu_windows detection that was previously unused), or when ROCm is already
downloaded/active.
Make the backend override honor a pinned variant: set_backend_override persists
the choice to disk so it survives an app restart, start_server reads it back,
and a cuda/rocm pin now actually selects that variant instead of always
preferring ROCm. A stale pin to a deleted backend self-heals to the default
order rather than forcing CPU. Add the web no-op stub for the new method.
* chore(rocm): drop incomplete vitest harness for the unused GpuAcceleration component
GpuAcceleration.tsx is not routed anywhere (GpuPage is the live settings view),
and the added vitest setup referenced testing-library/vitest deps that were not
in the lockfile, breaking the web typecheck. Remove the dead component's test
and its scaffolding to keep this PR scoped to the ROCm feature.
* ci(rocm): add ROCm release-artifact pipeline
Mirror the CUDA packaging path for ROCm so the runtime download has artifacts to
fetch. scripts/package_rocm.py splits the PyInstaller --rocm onedir into
voicebox-server-rocm.tar.gz (core) + rocm-libs-rocm7.2-v1.tar.gz (AMD runtime:
HIP DLLs, rocBLAS Tensile data, MIOpen kernel DBs) + rocm-libs.json, matching
the names services/rocm.py expects, both under the app-version release tag.
The new build-rocm-windows job in release.yml builds on windows-latest/cp312 and
lets build_binary.py --rocm pull the official AMD Radeon wheels.
The file classifier can't be validated against a real AMD build on CI, so it has
unit coverage (test_package_rocm.py) against a synthetic onedir layout. The
prefixes/dir markers may need a tweak after the first real build on AMD
hardware — the packager hard-fails loudly if it classifies zero ROCm files.
---------
Co-authored-by: Jamie Pine <[email protected]>
Applies the compatibility-checker portion of #367. Adds a
check_cuda_compatibility() helper that compares the current device's
compute capability against torch.cuda._get_arch_list() and returns a
human-readable warning if the PyTorch build doesn't support it.
Wired into three places:
• HealthResponse gains a gpu_compatibility_warning field so clients
can surface the issue in the UI
• Startup logs the warning as WARN level
• _get_gpu_status() appends "[UNSUPPORTED - see logs]" to the GPU
label shown in settings
Skipped #367's other half — the switch from stable to nightly cu128
wheels across release.yml, build_binary.py, and justfile. That's
redundant with #401's TORCH_CUDA_ARCH_LIST=...12.0+PTX approach and
would introduce non-deterministic builds from shifting nightly
releases.
Co-Authored-By: nyzxor <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Auto-detect Intel Arc GPUs during Windows setup and install PyTorch
with XPU support + intel-extension-for-pytorch. Enable allow_xpu=True
on all TTS backends (Chatterbox, Chatterbox Turbo, Hume TADA, LuxTTS)
that previously only supported CUDA. Add shared empty_device_cache()
and manual_seed() helpers in base.py to handle XPU memory management
and reproducible seeding alongside CUDA.
