CodeRabbit follow-up on the previous round: the asyncio.Lock closed the
gap for callers going through generate()/transcribe() consistently, but
unload_model() itself never touched that lock and could still land
between the load future resolving and the generate/transcribe future
being submitted - two separate _run_on_mlx_thread calls, so a real gap
existed at the Python level even though the executor is single-worker.
Fix: generate() and transcribe() each now submit exactly ONE callable
to the MLX executor - a self-healing reload-if-needed-then-infer
function - instead of a load submission followed by a separate infer
submission. This removes the gap structurally: nothing can observe an
intermediate state because there is no intermediate state exposed
across an await boundary. unload_model() now also submits
unconditionally (loaded-check moved inside _unload_model_sync, which
runs atomically with the teardown) rather than racing its own
Python-level self.model check against a load in flight.
Verified: same-size generate, size-switch generate, unload route, and
a post-unload generate all complete clean on the live launchd server.
Two real follow-on issues found by CodeRabbit on PR #989:
1. MLXTTSBackend.load_model_async called self.unload_model() directly
on the event-loop thread before dispatching _load_model_sync to the
MLX worker thread - model teardown ran on the wrong OS thread, same
class of bug the PR itself fixes. Combined unload+load into one
_reload_sync callable submitted as a single MLX-thread operation.
2. Public unload_model() (called synchronously from the /models/unload
routes) also ran on the caller thread. Now submits to the MLX
executor and blocks on the result, so teardown always happens on the
worker thread regardless of caller. Same fix applied to
MLXSTTBackend.
3. Both backends cache self.model on the instance, but generate()/
transcribe() only locked their own internal load step - a
concurrent request for a different model_size could swap self.model
between one request's load and its inference (real race: routes/
generations.py calls load_engine_model() and generate_chunked() as
separate awaited steps with a gap between them). Added a per-backend
asyncio.Lock held across the full load+inference sequence in both
generate() and transcribe(). Note: this closes the race for callers
using the backend's own public methods consistently; the wider
route-level orchestration race (load_engine_model + generate_chunked
as two separate calls) is a follow-up outside this file's scope.
Verified: same-size and cross-size-switch generations both complete
clean after the patch; /models/{name}/unload route returns 200 without
deadlocking the (still-responsive) server.
Qwen3-TTS (and MLX STT) generation crashed with:
"There is no Stream(gpu, N) in current thread."
MLXTTSBackend/MLXSTTBackend dispatched model load and generate/
transcribe as separate asyncio.to_thread() calls, which round-robin
across Python's default multi-worker executor pool. MLX's Metal
backend keeps GPU streams registered per-OS-thread, so a model loaded
on one worker thread and then used for generation on a different
worker thread hits a missing stream and crashes.
Reproduced 100% of the time on macOS/Apple Silicon cloning with both
the 1.7B and 0.6B Qwen3-TTS models; Chatterbox/Kokoro were unaffected
since they use the PyTorch backend, not this module.
Fix: route all four MLX call sites in this file (TTS load, TTS
generate, STT load, STT transcribe) through a dedicated single-worker
ThreadPoolExecutor instead of asyncio.to_thread's shared pool, so
every MLX operation for a given process runs on the same OS thread.
Verified: direct /generate API calls against both model sizes
completed cleanly after the fix (previously failed every time).
