fix(offline): guard inference paths with HF_HUB_OFFLINE (#503)

* fix(offline): guard inference paths with HF_HUB_OFFLINE (#462)

PR #443 wrapped the model *load* path with `force_offline_if_cached` so
cached models don't phone home at startup. The context manager restores
`HF_HUB_OFFLINE` on exit, which left inference paths (generate,
transcribe, voice-prompt creation) unguarded — and `qwen_tts`,
`mlx_audio`, and `transformers` perform lazy tokenizer/processor/config
lookups during inference. With internet on, those lookups are
near-instant and invisible; with internet off, `requests` hangs on DNS
or connect until the network returns. This is exactly what users in
#462 describe: model shows "Loaded", internet drops, generation
"thinks" forever, internet comes back, generation completes.

Chatterbox and LuxTTS don't exhibit this because their engine libs
resolve everything through already-cached paths at load time.

Fix: wrap each inference-sync body with `force_offline_if_cached(True,
...)`. Since inference only runs after a successful load, weights are
known to be on disk, so `is_cached=True` is unconditional.

Also adds the load-time guard that was missing from
`qwen_custom_voice_backend.py` — CustomVoice previously had no offline
protection at all.

Paths patched:
  - PyTorchTTSBackend.create_voice_prompt (create_voice_clone_prompt)
  - PyTorchTTSBackend.generate (generate_voice_clone)
  - PyTorchSTTBackend.transcribe (Whisper generate + decoder-prompt-ids)
  - MLXTTSBackend.generate (mlx_audio generate, all branches)
  - MLXSTTBackend.transcribe (mlx_audio whisper generate)
  - QwenCustomVoiceBackend._load_model_sync + generate

Does not address the secondary `check_model_inputs() missing 'func'`
error reported in the same issue — that's a `transformers` 5.x
version-skew bug on the install path, separate concern.

Fixes #462.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(offline): mutate cached HF constants + threadsafe refcount

Review feedback on the initial fix surfaced two real issues:

1. ``os.environ`` toggles alone don't flip offline mode.
   ``huggingface_hub.constants.HF_HUB_OFFLINE`` is read once at import
   time into a module-level bool; ``transformers.utils.hub._is_offline_mode``
   mirrors that bool at its own import time. The hot paths
   (``_http._default_backend_factory`` in huggingface_hub,
   ``is_offline_mode`` in transformers) read the cached bools — not the
   env — so mutating only ``os.environ`` was a no-op.

2. Race condition on concurrent inference. Two threads running inside
   ``force_offline_if_cached`` via ``asyncio.to_thread`` could have
   thread A's ``finally`` strip thread B's offline protection mid-run.

Rewrite the helper to:
  - mutate ``huggingface_hub.constants.HF_HUB_OFFLINE`` and
    ``transformers.utils.hub._is_offline_mode`` directly
  - refcount concurrent users under a single ``threading.RLock`` so a
    shared offline window is restored only when the last caller exits
  - still write ``os.environ`` for anything that reads it dynamically

Also addresses the unused-variable ruff flag on the Whisper transcribe
path (``audio, sr`` → ``audio, _sr``).

New unit tests cover the cached-constant mutation, env propagation,
no-op on ``is_cached=False``, nested contexts, and a threaded race
where a slow thread must retain offline mode after a peer exits.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(offline): atomic entry rollback + tidy test assertions

Review follow-up:

- Wrap the `_offline_refcount == 0` setup in a try/except so any failure
  during the cached-constant mutation (including unexpected non-ImportError
  like RuntimeError or AttributeError from a half-initialized module)
  rolls back *all* partial state before re-raising. Without this, a
  mid-setup crash could leave `huggingface_hub.constants.HF_HUB_OFFLINE`
  mutated but the refcount at 0 — a persistent offline flag outliving
  the process.
- Swap ruff-flagged Yoda comparisons in the new test file (SIM300) and
  add a module-level note warning that these tests mutate global state
  and are not safe under cross-process parallelism.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(offline): make concurrency test deterministic and bounded

Replace the `sleep(0.15)` ordering hack with an explicit `threading.Event`
the fast thread sets in `finally`. The slow thread waits on that event
(bounded), then observes the flag — so we deterministically verify the
slow thread still sees offline mode after the fast thread has exited.

Also add timeouts to `barrier.wait()` and assert `not thread.is_alive()`
after the joins so the test can't hang on an unexpected failure path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

backend/backends/qwen_custom_voice_backend.py

@@ -28,6 +28,7 @@ from .base import (
     combine_voice_prompts as _combine_voice_prompts,
     model_load_progress,
 )
+from ..utils.hf_offline_patch import force_offline_if_cached
 
 logger = logging.getLogger(__name__)
 
@@ -104,18 +105,19 @@ class QwenCustomVoiceBackend:
             model_path = self._get_model_path(model_size)
             logger.info("Loading Qwen CustomVoice %s on %s...", model_size, self.device)
 
-            if self.device == "cpu":
-                self.model = Qwen3TTSModel.from_pretrained(
-                    model_path,
-                    torch_dtype=torch.float32,
-                    low_cpu_mem_usage=False,
-                )
-            else:
-                self.model = Qwen3TTSModel.from_pretrained(
-                    model_path,
-                    device_map=self.device,
-                    torch_dtype=torch.bfloat16,
-                )
+            with force_offline_if_cached(is_cached, model_name):
+                if self.device == "cpu":
+                    self.model = Qwen3TTSModel.from_pretrained(
+                        model_path,
+                        torch_dtype=torch.float32,
+                        low_cpu_mem_usage=False,
+                    )
+                else:
+                    self.model = Qwen3TTSModel.from_pretrained(
+                        model_path,
+                        device_map=self.device,
+                        torch_dtype=torch.bfloat16,
+                    )
 
         self._current_model_size = model_size
         self.model_size = model_size
@@ -184,6 +186,7 @@ class QwenCustomVoiceBackend:
         await self.load_model_async(None)
 
         speaker = voice_prompt.get("preset_voice_id") or QWEN_CV_DEFAULT_SPEAKER
+        model_name = f"qwen-custom-voice-{self._current_model_size}"
 
         def _generate_sync():
             if seed is not None:
@@ -203,7 +206,11 @@ class QwenCustomVoiceBackend:
             if instruct:
                 kwargs["instruct"] = instruct
 
-            wavs, sample_rate = self.model.generate_custom_voice(**kwargs)
+            # Model is loaded → weights are on disk. Force offline so
+            # lazy tokenizer/config lookups inside qwen_tts don't hang
+            # when the user is disconnected (issue #462).
+            with force_offline_if_cached(True, model_name):
+                wavs, sample_rate = self.model.generate_custom_voice(**kwargs)
             return wavs[0], sample_rate
 
         audio, sample_rate = await asyncio.to_thread(_generate_sync)