fix(offline): guard inference paths with HF_HUB_OFFLINE (#503)

* fix(offline): guard inference paths with HF_HUB_OFFLINE (#462)

PR #443 wrapped the model *load* path with `force_offline_if_cached` so
cached models don't phone home at startup. The context manager restores
`HF_HUB_OFFLINE` on exit, which left inference paths (generate,
transcribe, voice-prompt creation) unguarded — and `qwen_tts`,
`mlx_audio`, and `transformers` perform lazy tokenizer/processor/config
lookups during inference. With internet on, those lookups are
near-instant and invisible; with internet off, `requests` hangs on DNS
or connect until the network returns. This is exactly what users in
#462 describe: model shows "Loaded", internet drops, generation
"thinks" forever, internet comes back, generation completes.

Chatterbox and LuxTTS don't exhibit this because their engine libs
resolve everything through already-cached paths at load time.

Fix: wrap each inference-sync body with `force_offline_if_cached(True,
...)`. Since inference only runs after a successful load, weights are
known to be on disk, so `is_cached=True` is unconditional.

Also adds the load-time guard that was missing from
`qwen_custom_voice_backend.py` — CustomVoice previously had no offline
protection at all.

Paths patched:
  - PyTorchTTSBackend.create_voice_prompt (create_voice_clone_prompt)
  - PyTorchTTSBackend.generate (generate_voice_clone)
  - PyTorchSTTBackend.transcribe (Whisper generate + decoder-prompt-ids)
  - MLXTTSBackend.generate (mlx_audio generate, all branches)
  - MLXSTTBackend.transcribe (mlx_audio whisper generate)
  - QwenCustomVoiceBackend._load_model_sync + generate

Does not address the secondary `check_model_inputs() missing 'func'`
error reported in the same issue — that's a `transformers` 5.x
version-skew bug on the install path, separate concern.

Fixes #462.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(offline): mutate cached HF constants + threadsafe refcount

Review feedback on the initial fix surfaced two real issues:

1. ``os.environ`` toggles alone don't flip offline mode.
   ``huggingface_hub.constants.HF_HUB_OFFLINE`` is read once at import
   time into a module-level bool; ``transformers.utils.hub._is_offline_mode``
   mirrors that bool at its own import time. The hot paths
   (``_http._default_backend_factory`` in huggingface_hub,
   ``is_offline_mode`` in transformers) read the cached bools — not the
   env — so mutating only ``os.environ`` was a no-op.

2. Race condition on concurrent inference. Two threads running inside
   ``force_offline_if_cached`` via ``asyncio.to_thread`` could have
   thread A's ``finally`` strip thread B's offline protection mid-run.

Rewrite the helper to:
  - mutate ``huggingface_hub.constants.HF_HUB_OFFLINE`` and
    ``transformers.utils.hub._is_offline_mode`` directly
  - refcount concurrent users under a single ``threading.RLock`` so a
    shared offline window is restored only when the last caller exits
  - still write ``os.environ`` for anything that reads it dynamically

Also addresses the unused-variable ruff flag on the Whisper transcribe
path (``audio, sr`` → ``audio, _sr``).

New unit tests cover the cached-constant mutation, env propagation,
no-op on ``is_cached=False``, nested contexts, and a threaded race
where a slow thread must retain offline mode after a peer exits.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(offline): atomic entry rollback + tidy test assertions

Review follow-up:

- Wrap the `_offline_refcount == 0` setup in a try/except so any failure
  during the cached-constant mutation (including unexpected non-ImportError
  like RuntimeError or AttributeError from a half-initialized module)
  rolls back *all* partial state before re-raising. Without this, a
  mid-setup crash could leave `huggingface_hub.constants.HF_HUB_OFFLINE`
  mutated but the refcount at 0 — a persistent offline flag outliving
  the process.
- Swap ruff-flagged Yoda comparisons in the new test file (SIM300) and
  add a module-level note warning that these tests mutate global state
  and are not safe under cross-process parallelism.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(offline): make concurrency test deterministic and bounded

Replace the `sleep(0.15)` ordering hack with an explicit `threading.Event`
the fast thread sets in `finally`. The slow thread waits on that event
(bounded), then observes the flag — so we deterministically verify the
slow thread still sees offline mode after the fast thread has exited.

Also add timeouts to `barrier.wait()` and assert `not thread.is_alive()`
after the joins so the test can't hang on an unexpected failure path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

backend/backends/pytorch_backend.py

@@ -172,13 +172,19 @@ class PyTorchTTSBackend:
                     # This shouldn't happen in practice, but handle it
                     return {"prompt": cached_prompt}, True
 
+        model_name = f"qwen-tts-{self._current_model_size}"
+
         def _create_prompt_sync():
             """Run synchronous voice prompt creation in thread pool."""
-            return self.model.create_voice_clone_prompt(
-                ref_audio=str(audio_path),
-                ref_text=reference_text,
-                x_vector_only_mode=False,
-            )
+            # Model is loaded → weights are on disk. Force offline so
+            # lazy tokenizer/config lookups inside qwen_tts don't hang
+            # when the user is disconnected (issue #462).
+            with force_offline_if_cached(True, model_name):
+                return self.model.create_voice_clone_prompt(
+                    ref_audio=str(audio_path),
+                    ref_text=reference_text,
+                    x_vector_only_mode=False,
+                )
 
         # Run blocking operation in thread pool
         voice_prompt_items = await asyncio.to_thread(_create_prompt_sync)
@@ -221,19 +227,24 @@ class PyTorchTTSBackend:
         # Load model
         await self.load_model_async(None)
 
+        model_name = f"qwen-tts-{self._current_model_size}"
+
         def _generate_sync():
             """Run synchronous generation in thread pool."""
             # Set seed if provided
             if seed is not None:
                 manual_seed(seed, self.device)
 
-            # Generate audio - this is the blocking operation
-            wavs, sample_rate = self.model.generate_voice_clone(
-                text=text,
-                voice_clone_prompt=voice_prompt,
-                language=LANGUAGE_CODE_TO_NAME.get(language, "auto"),
-                instruct=instruct,
-            )
+            # Model is loaded → weights are on disk. Force offline so
+            # lazy tokenizer/config lookups inside qwen_tts don't hang
+            # when the user is disconnected (issue #462).
+            with force_offline_if_cached(True, model_name):
+                wavs, sample_rate = self.model.generate_voice_clone(
+                    text=text,
+                    voice_clone_prompt=voice_prompt,
+                    language=LANGUAGE_CODE_TO_NAME.get(language, "auto"),
+                    instruct=instruct,
+                )
             return wavs[0], sample_rate
 
         # Run blocking inference in thread pool to avoid blocking event loop
@@ -331,40 +342,46 @@ class PyTorchSTTBackend:
         """
         await self.load_model_async(model_size)
 
+        progress_model_name = f"whisper-{self.model_size}"
+
         def _transcribe_sync():
             """Run synchronous transcription in thread pool."""
             # Load audio
-            audio, sr = load_audio(audio_path, sample_rate=16000)
+            audio, _sr = load_audio(audio_path, sample_rate=16000)
 
-            # Process audio
-            inputs = self.processor(
-                audio,
-                sampling_rate=16000,
-                return_tensors="pt",
-            )
-            inputs = inputs.to(self.device)
-
-            # Generate transcription
-            # If language is provided, force it; otherwise let Whisper auto-detect
-            generate_kwargs = {}
-            if language:
-                forced_decoder_ids = self.processor.get_decoder_prompt_ids(
-                    language=language,
-                    task="transcribe",
+            # Model is loaded → weights are on disk. Force offline so
+            # `get_decoder_prompt_ids` and any lazy tokenizer lookups
+            # don't hang when the user is disconnected (issue #462).
+            with force_offline_if_cached(True, progress_model_name):
+                # Process audio
+                inputs = self.processor(
+                    audio,
+                    sampling_rate=16000,
+                    return_tensors="pt",
                 )
-                generate_kwargs["forced_decoder_ids"] = forced_decoder_ids
+                inputs = inputs.to(self.device)
 
-            with torch.no_grad():
-                predicted_ids = self.model.generate(
-                    inputs["input_features"],
-                    **generate_kwargs,
-                )
+                # Generate transcription
+                # If language is provided, force it; otherwise let Whisper auto-detect
+                generate_kwargs = {}
+                if language:
+                    forced_decoder_ids = self.processor.get_decoder_prompt_ids(
+                        language=language,
+                        task="transcribe",
+                    )
+                    generate_kwargs["forced_decoder_ids"] = forced_decoder_ids
 
-            # Decode
-            transcription = self.processor.batch_decode(
-                predicted_ids,
-                skip_special_tokens=True,
-            )[0]
+                with torch.no_grad():
+                    predicted_ids = self.model.generate(
+                        inputs["input_features"],
+                        **generate_kwargs,
+                    )
+
+                # Decode
+                transcription = self.processor.batch_decode(
+                    predicted_ids,
+                    skip_special_tokens=True,
+                )[0]
 
             return transcription.strip()