Initial commit (forked from jamiepine/voicebox)

backend/tests/E2E_MODEL_TEST_DESIGN.md

@@ -0,0 +1,220 @@
+# End-to-End Model Generation Test — Design
+
+## Goal
+
+A single script, runnable on macOS and Windows, that exercises every TTS model against the **frozen PyInstaller binary** (not the dev server), captures per-model pass/fail and error messages, and exits non-zero if any model fails. Generation is strictly sequential — one model loaded at a time.
+
+## Test matrix (10 runs)
+
+Derived from `backend/backends/__init__.py:185-316`. Each row maps to one `POST /generate` call.
+
+| # | engine                | model_size | profile kind | notes |
+|---|-----------------------|------------|--------------|-------|
+| 1 | `qwen`                | `1.7B`     | cloned       | reference audio required |
+| 2 | `qwen`                | `0.6B`     | cloned       | |
+| 3 | `qwen_custom_voice`   | `1.7B`     | preset       | `preset_voice_id="Ryan"` |
+| 4 | `qwen_custom_voice`   | `0.6B`     | preset       | `preset_voice_id="Ryan"` |
+| 5 | `luxtts`              | —          | cloned       | English only |
+| 6 | `chatterbox`          | —          | cloned       | |
+| 7 | `chatterbox_turbo`    | —          | cloned       | English only |
+| 8 | `tada`                | `1B`       | cloned       | tada-1b, English only |
+| 9 | `tada`                | `3B`       | cloned       | tada-3b-ml, multilingual |
+| 10| `kokoro`              | —          | preset       | `preset_voice_id="af_heart"` |
+
+Cloned engines (1, 2, 5, 6, 7, 8, 9) share **one** profile created once with the reference WAV. Preset profiles are created separately, one for kokoro and one for qwen_custom_voice.
+
+Language for every run: `en` (covers every engine's supported set).
+
+## End-to-end flow
+
+```
+1. Resolve paths          → find binary, build if missing
+2. Launch binary          → spawn with --port --data-dir --parent-pid
+3. Wait for /health       → poll until status=="healthy" or 120s timeout
+4. Create profiles        → 1 cloned + 2 preset, via /profiles (+ /samples)
+5. For each (engine, model_size) in matrix:
+     a. Check cache       → GET /models/status → cached? short timeout : long
+     b. POST /generate    → get generation_id
+     c. Stream /status    → consume SSE until completed/failed/timeout
+     d. Record result     → {engine, model_size, status, duration, error, elapsed}
+6. Write results          → JSON + Markdown table to ./results/
+7. Shutdown binary        → SIGTERM, fall back to kill, verify port freed
+8. Exit code              → 0 if all passed, 1 otherwise
+```
+
+## Binary resolution
+
+Search order — **first hit wins**:
+
+| Platform | Path | Build type |
+|----------|------|------------|
+| macOS    | `backend/dist/voicebox-server-cuda/voicebox-server-cuda` | onedir (CUDA, rarely on Mac) |
+| macOS    | `backend/dist/voicebox-server`                          | onefile (CPU) |
+| Windows  | `backend\dist\voicebox-server-cuda\voicebox-server-cuda.exe` | onedir (CUDA) |
+| Windows  | `backend\dist\voicebox-server.exe`                      | onefile (CPU) |
+
+If none exist, run `python backend/build_binary.py` and wait for it to finish (can take 5-20 min). Fail with a clear error if the build itself fails. `--skip-build` flag forces "error out if no binary" instead of building.
+
+## Spawn command
+
+Mirrors Tauri's launch in `tauri/src-tauri/src/main.rs:369-388`:
+
+```
+<binary> --host 127.0.0.1 --port <free-port> --data-dir <tempdir> --parent-pid <test-pid>
+```
+
+- **Port**: bind to `0` first in Python to grab a free port, then pass that number.
+- **Data dir**: `tempfile.mkdtemp(prefix="voicebox-e2e-")`. Deleted after the run unless `--keep-data-dir`. Profiles and generated WAVs land here.
+- **Parent PID**: current Python PID — ensures the backend dies if the test crashes (watchdog in `server.py:102-224`).
+- **stdout/stderr**: tee to both a log file in `./results/server-<timestamp>.log` and a rolling in-memory buffer. On model failure, last 100 lines of the buffer are attached to that model's error record.
+
+## Profile setup
+
+One cloned profile shared across all cloning engines:
+
+```http
+POST /profiles
+{
+  "name": "e2e-cloned",
+  "voice_type": "cloned",
+  "language": "en"
+}
+```
+
+Then:
+
+```http
+POST /profiles/{id}/samples   (multipart)
+  file: <reference WAV>
+  reference_text: <exact transcription>
+```
+
+Two preset profiles:
+
+```http
+POST /profiles
+{ "name": "e2e-kokoro",  "voice_type": "preset", "language": "en",
+  "preset_engine": "kokoro",            "preset_voice_id": "af_heart" }
+
+POST /profiles
+{ "name": "e2e-qwen-cv", "voice_type": "preset", "language": "en",
+  "preset_engine": "qwen_custom_voice", "preset_voice_id": "Ryan" }
+```
+
+## Generation request (per matrix row)
+
+```http
+POST /generate
+{
+  "profile_id": "<appropriate profile>",
+  "text": "The quick brown fox jumps over the lazy dog.",
+  "language": "en",
+  "engine": "<engine>",
+  "model_size": "<size or omitted>",
+  "seed": 42,
+  "normalize": true
+}
+```
+
+Response `id` feeds into the SSE status loop (`GET /generate/{id}/status`, `routes/generations.py:190-227`). Loop reads lines until a payload with `status in ("completed", "failed")` arrives, then breaks.
+
+## Timeout strategy (split)
+
+Check `GET /models/status` for the target model **before** generation:
+
+| Cached? | Per-model timeout | Rationale |
+|---------|-------------------|-----------|
+| Yes     | **3 minutes**     | Inference only; generous for CPU builds |
+| No      | **20 minutes**    | First-run HF download up to 8 GB (tada-3b-ml) |
+
+On timeout: cancel the SSE stream, mark the row `timeout`, and continue to the next row. Don't abort the whole run on one timeout.
+
+## Result format
+
+`./results/e2e-<platform>-<arch>-<timestamp>.json`:
+
+```json
+{
+  "platform": "darwin-arm64",
+  "binary": "/abs/path/voicebox-server",
+  "binary_size_mb": 612,
+  "started_at": "2026-04-16T12:34:56Z",
+  "finished_at": "...",
+  "results": [
+    {
+      "engine": "qwen",
+      "model_size": "1.7B",
+      "status": "passed|failed|timeout",
+      "generation_id": "...",
+      "was_cached": true,
+      "elapsed_seconds": 12.4,
+      "audio_duration": 3.1,
+      "audio_path": "/tmp/.../gen.wav",
+      "error": null,
+      "server_log_tail": null
+    }
+  ]
+}
+```
+
+Companion `./results/e2e-<...>.md`:
+
+```
+# Voicebox E2E — darwin-arm64 — 2026-04-16 12:34
+
+| Engine              | Size | Status | Elapsed | Error |
+|---------------------|------|--------|---------|-------|
+| qwen                | 1.7B | PASS   |  12.4s  |       |
+| qwen                | 0.6B | FAIL   |   4.1s  | CUDA OOM: ... |
+...
+```
+
+## CLI flags
+
+```
+python -m backend.tests.test_all_models_e2e [flags]
+
+--binary PATH           Use this binary instead of auto-detecting
+--skip-build            Error if no binary found (no auto-build)
+--reference-wav PATH    Reference audio (default: backend/tests/fixtures/reference_voice.wav)
+--reference-text STR    Transcription (default: read from fixtures/reference_voice.txt)
+--only ENGINE[,...]     Run only these engines (e.g. kokoro,qwen)
+--skip ENGINE[,...]     Skip these engines
+--keep-data-dir         Don't delete tempdir after run
+--timeout-cached SEC    Override 180
+--timeout-download SEC  Override 1200
+--port N                Override auto-picked port
+--output-dir PATH       Default: backend/tests/results/
+```
+
+## File layout
+
+```
+backend/tests/
+├── E2E_MODEL_TEST_DESIGN.md       (this file)
+├── test_all_models_e2e.py         (main script, ~400-500 LoC)
+├── fixtures/
+│   ├── reference_voice.wav        (user-provided, ~5-15s clean speech)
+│   └── reference_voice.txt        (exact transcription)
+└── results/                       (gitignored)
+    ├── e2e-darwin-arm64-<ts>.json
+    ├── e2e-darwin-arm64-<ts>.md
+    └── server-<ts>.log
+```
+
+The script uses only stdlib + `httpx` (or `requests`) + `sseclient-py` — all already in `backend/requirements.txt`. No pytest to keep it invocable as a single command on fresh checkouts.
+
+## Safety & cleanup
+
+- Always kill the spawned binary in a `try/finally`. On Windows, `taskkill /F /T` the whole tree (Tauri does the same).
+- Verify the port is free on shutdown (Tauri port-reuse check in `main.rs:114-186` could otherwise pick up a ghost).
+- Don't touch the user's HF cache by default — let the server use `HF_HUB_CACHE` / `VOICEBOX_MODELS_DIR`. Passing `--isolated-cache` would point both env vars at the tempdir for a true cold-start run (opt-in only; would re-download every time).
+
+## Non-goals
+
+- Not validating audio quality (no WER, no waveform comparison). Pass = "endpoint returned `completed` and produced a non-empty WAV".
+- Not testing STT (Whisper), effects chains, channels, or streaming endpoints.
+- Not running on CI today — human-invoked on dev machines. CI integration is a follow-up once the script is stable.
+- No model unload between runs — models stay loaded; server manages its own eviction.
+- No version-drift check on the binary.
+- No `instruct` parameter exercised on qwen_custom_voice runs.

backend/tests/README.md

@@ -0,0 +1,58 @@
+# Backend Tests
+
+Manual test scripts for debugging and validating backend functionality.
+
+## Test Files
+
+### `test_generation_progress.py`
+Tests TTS generation with SSE progress monitoring to identify UX issues where users see download progress even when the model is already cached.
+
+**Usage:**
+```bash
+cd backend
+python tests/test_generation_progress.py
+```
+
+**Prerequisites:**
+- Server must be running (`python main.py`)
+- At least one voice profile must exist
+
+### `test_real_download.py`
+Tests real model download with SSE progress monitoring.
+
+**Usage:**
+```bash
+cd backend
+# Delete cache first to force fresh download
+rm -rf ~/.cache/huggingface/hub/models--openai--whisper-base
+python tests/test_real_download.py
+```
+
+**Prerequisites:**
+- Server must be running (`python main.py`)
+
+### `test_progress.py`
+Unit tests for ProgressManager and HFProgressTracker functionality.
+
+**Usage:**
+```bash
+cd backend
+python tests/test_progress.py
+```
+
+### `test_check_progress_state.py`
+Debugging script to inspect the internal state of ProgressManager and TaskManager.
+
+**Usage:**
+```bash
+cd backend
+python tests/test_check_progress_state.py
+```
+
+## Notes
+
+These are manual test scripts, not automated unit tests. They're designed for:
+- Debugging progress tracking issues
+- Validating SSE event streams
+- Monitoring real-time download behavior
+- Inspecting internal state during development

backend/tests/__init__.py

@@ -0,0 +1,6 @@
+"""
+Test suite for Voicebox backend.
+
+This directory contains manual test scripts for debugging and validating
+progress tracking, model downloads, and generation functionality.
+"""

backend/tests/fixtures/README.md

@@ -0,0 +1,16 @@
+# E2E Test Fixtures
+
+Place two files here before running `test_all_models_e2e.py`:
+
+- `reference_voice.wav` — a clean speech sample, mono, 16–24 kHz, ~5–15 seconds.
+- `reference_voice.txt` — the **exact** transcription of the WAV (single line, no trailing newline required).
+
+These are used to create a cloned voice profile for every cloning-capable engine (qwen, luxtts, chatterbox, chatterbox_turbo, tada). Keep them out of version control if they contain personal audio — this directory is not gitignored by default, so add them to `.gitignore` locally if needed.
+
+You can point the test at different files with:
+
+```
+python backend/tests/test_all_models_e2e.py \
+  --reference-wav /path/to/your.wav \
+  --reference-text "exact transcription here"
+```

backend/tests/test_all_models_e2e.py

@@ -0,0 +1,630 @@
+"""
+End-to-end model generation test.
+
+Exercises every TTS model against the frozen PyInstaller binary, captures
+per-model pass/fail, and writes a JSON + Markdown report.
+
+Usage:
+    python backend/tests/test_all_models_e2e.py [flags]
+
+See E2E_MODEL_TEST_DESIGN.md for the full design.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import platform
+import shutil
+import signal
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+from collections import deque
+from dataclasses import asdict, dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+import httpx
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+BACKEND_DIR = REPO_ROOT / "backend"
+DIST_DIR = BACKEND_DIR / "dist"
+FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
+RESULTS_DIR = Path(__file__).resolve().parent / "results"
+
+
+# ── Test matrix ──────────────────────────────────────────────────────
+
+@dataclass(frozen=True)
+class MatrixRow:
+    label: str            # human-readable (appears in report)
+    engine: str           # /generate engine
+    model_size: Optional[str]  # /generate model_size (None = omit)
+    profile_kind: str     # "cloned" | "preset_kokoro" | "preset_qwen_cv"
+    model_name: str       # /models/status key for cache lookup
+
+
+MATRIX: list[MatrixRow] = [
+    MatrixRow("qwen 1.7B",              "qwen",              "1.7B", "cloned",          "qwen-tts-1.7B"),
+    MatrixRow("qwen 0.6B",              "qwen",              "0.6B", "cloned",          "qwen-tts-0.6B"),
+    MatrixRow("qwen_custom_voice 1.7B", "qwen_custom_voice", "1.7B", "preset_qwen_cv",  "qwen-custom-voice-1.7B"),
+    MatrixRow("qwen_custom_voice 0.6B", "qwen_custom_voice", "0.6B", "preset_qwen_cv",  "qwen-custom-voice-0.6B"),
+    MatrixRow("luxtts",                 "luxtts",            None,   "cloned",          "luxtts"),
+    MatrixRow("chatterbox",             "chatterbox",        None,   "cloned",          "chatterbox-tts"),
+    MatrixRow("chatterbox_turbo",       "chatterbox_turbo",  None,   "cloned",          "chatterbox-turbo"),
+    MatrixRow("tada 1B",                "tada",              "1B",   "cloned",          "tada-1b"),
+    MatrixRow("tada 3B",                "tada",              "3B",   "cloned",          "tada-3b-ml"),
+    MatrixRow("kokoro",                 "kokoro",            None,   "preset_kokoro",   "kokoro"),
+]
+
+TEXT = "The quick brown fox jumps over the lazy dog."
+DEFAULT_TIMEOUT_CACHED = 180
+DEFAULT_TIMEOUT_DOWNLOAD = 1200
+HEALTH_TIMEOUT = 120
+
+
+# ── Result record ────────────────────────────────────────────────────
+
+@dataclass
+class ModelResult:
+    label: str
+    engine: str
+    model_size: Optional[str]
+    status: str                      # "passed" | "failed" | "timeout"
+    was_cached: Optional[bool] = None
+    generation_id: Optional[str] = None
+    elapsed_seconds: float = 0.0
+    audio_duration: Optional[float] = None
+    audio_path: Optional[str] = None
+    audio_bytes: Optional[int] = None
+    error: Optional[str] = None
+    http_status: Optional[int] = None
+    server_log_tail: Optional[list[str]] = None
+
+
+# ── Binary resolution ────────────────────────────────────────────────
+
+def find_binary() -> Optional[Path]:
+    """Return the first existing binary in priority order, or None."""
+    is_win = platform.system() == "Windows"
+    exe = ".exe" if is_win else ""
+    candidates = [
+        DIST_DIR / "voicebox-server-cuda" / f"voicebox-server-cuda{exe}",
+        DIST_DIR / f"voicebox-server{exe}",
+    ]
+    for c in candidates:
+        if c.exists() and c.is_file():
+            return c
+    return None
+
+
+def build_binary() -> Path:
+    """Invoke build_binary.py and return the resulting binary path."""
+    print("[build] No frozen binary found — invoking build_binary.py (this may take 5-20 minutes)...", flush=True)
+    script = BACKEND_DIR / "build_binary.py"
+    result = subprocess.run(
+        [sys.executable, str(script)],
+        cwd=str(BACKEND_DIR),
+    )
+    if result.returncode != 0:
+        raise RuntimeError(f"build_binary.py exited with code {result.returncode}")
+    found = find_binary()
+    if found is None:
+        raise RuntimeError("build_binary.py finished but no binary was found in backend/dist/")
+    return found
+
+
+# ── Server spawn + log capture ───────────────────────────────────────
+
+class ServerProcess:
+    def __init__(self, binary: Path, port: int, data_dir: Path, log_path: Path):
+        self.binary = binary
+        self.port = port
+        self.data_dir = data_dir
+        self.log_path = log_path
+        self.proc: Optional[subprocess.Popen] = None
+        self._log_buffer: deque[str] = deque(maxlen=500)
+        self._reader_thread: Optional[threading.Thread] = None
+
+    def start(self) -> None:
+        args = [
+            str(self.binary),
+            "--host", "127.0.0.1",
+            "--port", str(self.port),
+            "--data-dir", str(self.data_dir),
+            "--parent-pid", str(os.getpid()),
+        ]
+        print(f"[spawn] {' '.join(args)}", flush=True)
+        self._log_fh = open(self.log_path, "w", encoding="utf-8", errors="replace")
+        # Combine stderr into stdout so we get a single ordered stream.
+        self.proc = subprocess.Popen(
+            args,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            bufsize=1,
+            text=True,
+            errors="replace",
+        )
+        self._reader_thread = threading.Thread(target=self._pump_logs, daemon=True)
+        self._reader_thread.start()
+
+    def _pump_logs(self) -> None:
+        assert self.proc is not None and self.proc.stdout is not None
+        for line in self.proc.stdout:
+            self._log_buffer.append(line.rstrip("\n"))
+            self._log_fh.write(line)
+            self._log_fh.flush()
+
+    def log_tail(self, n: int = 100) -> list[str]:
+        tail = list(self._log_buffer)[-n:]
+        return tail
+
+    def is_alive(self) -> bool:
+        return self.proc is not None and self.proc.poll() is None
+
+    def stop(self) -> None:
+        if self.proc is None:
+            return
+        if self.proc.poll() is not None:
+            return
+        try:
+            if platform.system() == "Windows":
+                subprocess.run(
+                    ["taskkill", "/F", "/T", "/PID", str(self.proc.pid)],
+                    capture_output=True,
+                )
+            else:
+                self.proc.send_signal(signal.SIGTERM)
+        except Exception as e:
+            print(f"[shutdown] signal failed: {e}", flush=True)
+        try:
+            self.proc.wait(timeout=10)
+        except subprocess.TimeoutExpired:
+            print("[shutdown] server didn't exit cleanly, killing", flush=True)
+            self.proc.kill()
+            try:
+                self.proc.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                pass
+        if self._reader_thread is not None:
+            self._reader_thread.join(timeout=2)
+        try:
+            self._log_fh.close()
+        except Exception:
+            pass
+
+
+def pick_free_port() -> int:
+    s = socket.socket()
+    s.bind(("127.0.0.1", 0))
+    port = s.getsockname()[1]
+    s.close()
+    return port
+
+
+# ── HTTP helpers ─────────────────────────────────────────────────────
+
+def wait_for_health(base_url: str, server: ServerProcess, timeout: int) -> None:
+    deadline = time.time() + timeout
+    with httpx.Client(timeout=5.0) as client:
+        while time.time() < deadline:
+            if not server.is_alive():
+                raise RuntimeError("Server process exited before becoming healthy")
+            try:
+                r = client.get(f"{base_url}/health")
+                if r.status_code == 200 and r.json().get("status") == "healthy":
+                    return
+            except httpx.HTTPError:
+                pass
+            time.sleep(1.0)
+    raise TimeoutError(f"Server did not become healthy within {timeout}s")
+
+
+def get_model_cached(client: httpx.Client, base_url: str, model_name: str) -> Optional[bool]:
+    try:
+        r = client.get(f"{base_url}/models/status", timeout=30.0)
+        r.raise_for_status()
+        for m in r.json().get("models", []):
+            if m.get("model_name") == model_name:
+                return bool(m.get("downloaded"))
+    except httpx.HTTPError:
+        return None
+    return None
+
+
+def create_cloned_profile(client: httpx.Client, base_url: str, wav_path: Path, reference_text: str) -> str:
+    r = client.post(f"{base_url}/profiles", json={
+        "name": "e2e-cloned",
+        "voice_type": "cloned",
+        "language": "en",
+    })
+    r.raise_for_status()
+    profile_id = r.json()["id"]
+
+    with open(wav_path, "rb") as f:
+        r = client.post(
+            f"{base_url}/profiles/{profile_id}/samples",
+            files={"file": (wav_path.name, f, "audio/wav")},
+            data={"reference_text": reference_text},
+            timeout=120.0,
+        )
+    r.raise_for_status()
+    return profile_id
+
+
+def create_preset_profile(client: httpx.Client, base_url: str, name: str, engine: str, voice_id: str) -> str:
+    r = client.post(f"{base_url}/profiles", json={
+        "name": name,
+        "voice_type": "preset",
+        "language": "en",
+        "preset_engine": engine,
+        "preset_voice_id": voice_id,
+    })
+    r.raise_for_status()
+    return r.json()["id"]
+
+
+def run_one_generation(
+    client: httpx.Client,
+    base_url: str,
+    row: MatrixRow,
+    profile_id: str,
+    timeout_s: int,
+) -> tuple[str, dict]:
+    """Start a generation and stream its status until done/failed/timeout.
+
+    Returns (status, payload) where status is "completed" | "failed" | "timeout".
+    """
+    body = {
+        "profile_id": profile_id,
+        "text": TEXT,
+        "language": "en",
+        "engine": row.engine,
+        "seed": 42,
+        "normalize": True,
+    }
+    if row.model_size is not None:
+        body["model_size"] = row.model_size
+
+    r = client.post(f"{base_url}/generate", json=body, timeout=30.0)
+    r.raise_for_status()
+    gen = r.json()
+    gen_id = gen["id"]
+
+    deadline = time.time() + timeout_s
+    last_payload: dict = gen
+    status_url = f"{base_url}/generate/{gen_id}/status"
+
+    while time.time() < deadline:
+        remaining = max(1.0, deadline - time.time())
+        try:
+            with client.stream("GET", status_url, timeout=httpx.Timeout(remaining + 5, read=remaining + 5)) as resp:
+                resp.raise_for_status()
+                for line in resp.iter_lines():
+                    if not line or not line.startswith("data: "):
+                        continue
+                    try:
+                        payload = json.loads(line[6:])
+                    except json.JSONDecodeError:
+                        continue
+                    last_payload = payload
+                    status = payload.get("status")
+                    if status == "not_found":
+                        return "failed", {"error": "generation not found", **payload}
+                    if status in ("completed", "failed"):
+                        return status, payload
+                    if time.time() >= deadline:
+                        break
+        except httpx.HTTPError:
+            time.sleep(1.0)
+            continue
+
+    return "timeout", last_payload
+
+
+def fetch_audio_info(
+    client: httpx.Client, base_url: str, generation_id: str, data_dir: Path
+) -> tuple[Optional[str], Optional[int]]:
+    """Return (audio_path, audio_bytes) for a completed generation.
+
+    Server stores audio_path relative to data_dir; resolve it to get a size.
+    """
+    try:
+        r = client.get(f"{base_url}/history/{generation_id}", timeout=10.0)
+        if r.status_code != 200:
+            return None, None
+        data = r.json()
+        audio_path = data.get("audio_path")
+        if not audio_path:
+            return None, None
+        p = Path(audio_path)
+        if not p.is_absolute():
+            p = data_dir / p
+        if p.exists():
+            return str(p), p.stat().st_size
+        return audio_path, None
+    except httpx.HTTPError:
+        return None, None
+
+
+# ── Report writers ───────────────────────────────────────────────────
+
+def write_reports(
+    output_dir: Path,
+    binary: Path,
+    started_at: datetime,
+    finished_at: datetime,
+    results: list[ModelResult],
+) -> tuple[Path, Path]:
+    output_dir.mkdir(parents=True, exist_ok=True)
+    plat = f"{platform.system().lower()}-{platform.machine().lower()}"
+    ts = started_at.strftime("%Y%m%d-%H%M%S")
+    json_path = output_dir / f"e2e-{plat}-{ts}.json"
+    md_path = output_dir / f"e2e-{plat}-{ts}.md"
+
+    doc = {
+        "platform": plat,
+        "binary": str(binary),
+        "binary_size_mb": round(binary.stat().st_size / (1024 * 1024), 1) if binary.exists() else None,
+        "started_at": started_at.isoformat(),
+        "finished_at": finished_at.isoformat(),
+        "elapsed_seconds": (finished_at - started_at).total_seconds(),
+        "results": [asdict(r) for r in results],
+    }
+    json_path.write_text(json.dumps(doc, indent=2))
+
+    lines = [
+        f"# Voicebox E2E — {plat} — {started_at.strftime('%Y-%m-%d %H:%M UTC')}",
+        "",
+        f"Binary: `{binary}`  ",
+        f"Elapsed: {doc['elapsed_seconds']:.1f}s",
+        "",
+        "| Model | Status | Cached | Elapsed | Audio | Error |",
+        "|-------|--------|--------|---------|-------|-------|",
+    ]
+    for r in results:
+        status_icon = {"passed": "PASS", "failed": "FAIL", "timeout": "TIMEOUT"}.get(r.status, r.status.upper())
+        cached = "yes" if r.was_cached else ("no" if r.was_cached is False else "?")
+        audio_col = f"{r.audio_duration:.2f}s" if r.audio_duration else ("—" if r.status != "passed" else "?")
+        error_col = (r.error or "").replace("\n", " ")[:120]
+        lines.append(f"| {r.label} | {status_icon} | {cached} | {r.elapsed_seconds:.1f}s | {audio_col} | {error_col} |")
+
+    failed_rows = [r for r in results if r.status != "passed"]
+    if failed_rows:
+        lines.append("")
+        lines.append("## Failures")
+        for r in failed_rows:
+            lines.append("")
+            lines.append(f"### {r.label} — {r.status}")
+            if r.error:
+                lines.append("")
+                lines.append("```")
+                lines.append(r.error)
+                lines.append("```")
+            if r.server_log_tail:
+                lines.append("")
+                lines.append("<details><summary>server log (last lines)</summary>")
+                lines.append("")
+                lines.append("```")
+                lines.extend(r.server_log_tail)
+                lines.append("```")
+                lines.append("</details>")
+
+    md_path.write_text("\n".join(lines) + "\n")
+    return json_path, md_path
+
+
+# ── Main ─────────────────────────────────────────────────────────────
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Voicebox E2E model generation test")
+    p.add_argument("--binary", type=Path, help="Path to voicebox-server binary (overrides auto-detect)")
+    p.add_argument("--skip-build", action="store_true", help="Error if binary missing instead of building")
+    p.add_argument(
+        "--reference-wav",
+        type=Path,
+        default=FIXTURES_DIR / "reference_voice.wav",
+        help="Reference audio for cloning engines",
+    )
+    p.add_argument(
+        "--reference-text",
+        help="Transcription of reference-wav (default: read from fixtures/reference_voice.txt)",
+    )
+    p.add_argument("--only", help="Comma-separated engines to run (e.g. kokoro,qwen)")
+    p.add_argument("--skip", help="Comma-separated engines to skip")
+    p.add_argument("--keep-data-dir", action="store_true", help="Don't delete tempdir after run")
+    p.add_argument("--timeout-cached", type=int, default=DEFAULT_TIMEOUT_CACHED)
+    p.add_argument("--timeout-download", type=int, default=DEFAULT_TIMEOUT_DOWNLOAD)
+    p.add_argument("--port", type=int, help="Override auto-picked port")
+    p.add_argument("--output-dir", type=Path, default=RESULTS_DIR)
+    return p.parse_args()
+
+
+def filter_matrix(args: argparse.Namespace) -> list[MatrixRow]:
+    only = set(x.strip() for x in args.only.split(",")) if args.only else None
+    skip = set(x.strip() for x in args.skip.split(",")) if args.skip else set()
+    rows = []
+    for r in MATRIX:
+        if only is not None and r.engine not in only:
+            continue
+        if r.engine in skip:
+            continue
+        rows.append(r)
+    return rows
+
+
+def resolve_reference(args: argparse.Namespace) -> tuple[Path, str]:
+    wav = args.reference_wav
+    if not wav.exists():
+        raise FileNotFoundError(
+            f"Reference WAV not found: {wav}\n"
+            f"Place a sample at {FIXTURES_DIR / 'reference_voice.wav'} or pass --reference-wav.\n"
+            f"See backend/tests/fixtures/README.md."
+        )
+    if args.reference_text:
+        text = args.reference_text
+    else:
+        txt_path = wav.with_suffix(".txt")
+        if not txt_path.exists():
+            raise FileNotFoundError(
+                f"Reference transcription not found: {txt_path}\n"
+                f"Create it next to the WAV, or pass --reference-text."
+            )
+        text = txt_path.read_text().strip()
+    if not text:
+        raise ValueError("Reference transcription is empty")
+    return wav, text
+
+
+def main() -> int:
+    args = parse_args()
+    rows = filter_matrix(args)
+    if not rows:
+        print("No rows selected after --only/--skip filtering", file=sys.stderr)
+        return 2
+
+    # Binary
+    binary = args.binary or find_binary()
+    if binary is None:
+        if args.skip_build:
+            print("No frozen binary found and --skip-build set. Run: python backend/build_binary.py", file=sys.stderr)
+            return 2
+        binary = build_binary()
+    if not binary.exists():
+        print(f"Binary path does not exist: {binary}", file=sys.stderr)
+        return 2
+    print(f"[binary] {binary}", flush=True)
+
+    # Reference audio (only required if any cloning row is in the matrix)
+    needs_reference = any(r.profile_kind == "cloned" for r in rows)
+    ref_wav: Optional[Path] = None
+    ref_text: Optional[str] = None
+    if needs_reference:
+        try:
+            ref_wav, ref_text = resolve_reference(args)
+        except (FileNotFoundError, ValueError) as e:
+            print(f"[fixture] {e}", file=sys.stderr)
+            return 2
+        print(f"[fixture] reference WAV:  {ref_wav}", flush=True)
+        print(f"[fixture] reference text: {ref_text!r}", flush=True)
+
+    # Tempdir + log path
+    data_dir = Path(tempfile.mkdtemp(prefix="voicebox-e2e-"))
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+    log_path = args.output_dir / f"server-{ts}.log"
+
+    port = args.port or pick_free_port()
+    base_url = f"http://127.0.0.1:{port}"
+
+    server = ServerProcess(binary=binary, port=port, data_dir=data_dir, log_path=log_path)
+    started_at = datetime.now(timezone.utc)
+    results: list[ModelResult] = []
+
+    try:
+        server.start()
+        print(f"[health] waiting for {base_url}/health ...", flush=True)
+        wait_for_health(base_url, server, HEALTH_TIMEOUT)
+        print("[health] ready", flush=True)
+
+        with httpx.Client(timeout=30.0) as client:
+            # Profile setup (only create what's needed)
+            cloned_profile_id: Optional[str] = None
+            kokoro_profile_id: Optional[str] = None
+            qwen_cv_profile_id: Optional[str] = None
+            needed_kinds = {r.profile_kind for r in rows}
+            if "cloned" in needed_kinds:
+                assert ref_wav is not None and ref_text is not None
+                print("[profile] creating cloned profile...", flush=True)
+                cloned_profile_id = create_cloned_profile(client, base_url, ref_wav, ref_text)
+            if "preset_kokoro" in needed_kinds:
+                print("[profile] creating kokoro preset...", flush=True)
+                kokoro_profile_id = create_preset_profile(client, base_url, "e2e-kokoro", "kokoro", "af_heart")
+            if "preset_qwen_cv" in needed_kinds:
+                print("[profile] creating qwen_custom_voice preset...", flush=True)
+                qwen_cv_profile_id = create_preset_profile(client, base_url, "e2e-qwen-cv", "qwen_custom_voice", "Ryan")
+
+            profile_lookup = {
+                "cloned": cloned_profile_id,
+                "preset_kokoro": kokoro_profile_id,
+                "preset_qwen_cv": qwen_cv_profile_id,
+            }
+
+            # Matrix loop
+            for row in rows:
+                print(f"\n[run] {row.label} (engine={row.engine}, size={row.model_size})", flush=True)
+                profile_id = profile_lookup[row.profile_kind]
+                assert profile_id is not None
+                was_cached = get_model_cached(client, base_url, row.model_name)
+                timeout_s = args.timeout_cached if was_cached else args.timeout_download
+                print(f"[run] cached={was_cached} timeout={timeout_s}s", flush=True)
+
+                t0 = time.time()
+                result = ModelResult(
+                    label=row.label,
+                    engine=row.engine,
+                    model_size=row.model_size,
+                    status="failed",
+                    was_cached=was_cached,
+                )
+                try:
+                    status, payload = run_one_generation(client, base_url, row, profile_id, timeout_s)
+                    result.status = "passed" if status == "completed" else status
+                    result.generation_id = payload.get("id")
+                    result.audio_duration = payload.get("duration")
+                    result.error = payload.get("error")
+                    if status == "completed" and result.generation_id:
+                        audio_path, audio_bytes = fetch_audio_info(
+                            client, base_url, result.generation_id, data_dir
+                        )
+                        result.audio_path = audio_path
+                        result.audio_bytes = audio_bytes
+                        if audio_bytes is not None and audio_bytes == 0:
+                            result.status = "failed"
+                            result.error = (result.error or "") + " (audio file is empty)"
+                except httpx.HTTPStatusError as e:
+                    result.status = "failed"
+                    result.http_status = e.response.status_code
+                    try:
+                        detail = e.response.json().get("detail")
+                    except Exception:
+                        detail = e.response.text
+                    result.error = f"HTTP {e.response.status_code}: {detail}"
+                except Exception as e:
+                    result.status = "failed"
+                    result.error = f"{type(e).__name__}: {e}"
+
+                result.elapsed_seconds = round(time.time() - t0, 2)
+                if result.status != "passed":
+                    result.server_log_tail = server.log_tail(100)
+                print(f"[run] {row.label} → {result.status} in {result.elapsed_seconds}s"
+                      + (f" ({result.error})" if result.error else ""), flush=True)
+                results.append(result)
+    finally:
+        finished_at = datetime.now(timezone.utc)
+        server.stop()
+        if not args.keep_data_dir:
+            shutil.rmtree(data_dir, ignore_errors=True)
+        else:
+            print(f"[cleanup] keeping data dir: {data_dir}", flush=True)
+
+    json_path, md_path = write_reports(args.output_dir, binary, started_at, finished_at, results)
+    print(f"\n[report] {json_path}")
+    print(f"[report] {md_path}")
+    print(f"[report] server log: {log_path}")
+
+    passed = sum(1 for r in results if r.status == "passed")
+    failed = len(results) - passed
+    print(f"\n== {passed} passed, {failed} failed ==")
+    return 0 if failed == 0 else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backend/tests/test_amd_gpu_detect.py

@@ -0,0 +1,96 @@
+"""
+Phase 2.1 Test: AMD GPU detection on Windows.
+
+Validates is_amd_gpu_windows() via mocked WMI and torch queries.
+
+Usage:
+    python -m pytest backend/tests/test_amd_gpu_detect.py -v
+"""
+
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from backend.utils.platform_detect import is_amd_gpu_windows
+
+
+class TestAmdGpuWindows:
+    """Unit tests for is_amd_gpu_windows with mocks."""
+
+    @pytest.fixture(autouse=True)
+    def _clear_detection_cache(self):
+        # is_amd_gpu_windows is memoized; reset between cases so each mock takes effect.
+        is_amd_gpu_windows.cache_clear()
+        yield
+        is_amd_gpu_windows.cache_clear()
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Linux")
+    def test_returns_false_on_linux(self, _mock_system):
+        """Non-Windows platforms should always return False."""
+        assert is_amd_gpu_windows() is False
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        return_value=MagicMock(stdout="1\n", returncode=0),
+    )
+    def test_detects_amd_via_wmi(self, _mock_run, _mock_system):
+        """WMI reporting an AMD adapter should return True."""
+        assert is_amd_gpu_windows() is True
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        return_value=MagicMock(stdout="0\n", returncode=0),
+    )
+    def test_no_amd_via_wmi(self, _mock_run, _mock_system):
+        """WMI reporting zero AMD adapters should return False."""
+        assert is_amd_gpu_windows() is False
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        side_effect=Exception("WMI not available"),
+    )
+    @patch("torch.cuda.is_available", return_value=True)
+    @patch(
+        "torch.cuda.get_device_name",
+        return_value="AMD Radeon RX 7800 XT",
+    )
+    def test_fallback_to_torch_radeon(self, _mock_name, _mock_avail, _mock_run, _mock_system):
+        """When WMI fails, torch.cuda.get_device_name('Radeon') should return True."""
+        assert is_amd_gpu_windows() is True
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        side_effect=Exception("WMI not available"),
+    )
+    @patch("torch.cuda.is_available", return_value=True)
+    @patch(
+        "torch.cuda.get_device_name",
+        return_value="NVIDIA GeForce RTX 4090",
+    )
+    def test_fallback_to_torch_nvidia(self, _mock_name, _mock_avail, _mock_run, _mock_system):
+        """When WMI fails, torch.cuda.get_device_name('NVIDIA') should return False."""
+        assert is_amd_gpu_windows() is False
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        side_effect=Exception("WMI not available"),
+    )
+    @patch("torch.cuda.is_available", return_value=False)
+    def test_no_torch_cuda(self, _mock_avail, _mock_run, _mock_system):
+        """When WMI fails and torch.cuda is unavailable, should return False."""
+        assert is_amd_gpu_windows() is False
+
+    @patch("backend.utils.platform_detect.platform.system", return_value="Windows")
+    @patch(
+        "backend.utils.platform_detect.subprocess.run",
+        side_effect=Exception("WMI not available"),
+    )
+    def test_torch_not_installed(self, _mock_run, _mock_system):
+        """When torch is not installed, should return False without crashing."""
+        with patch.dict("sys.modules", {"torch": None}):
+            assert is_amd_gpu_windows() is False

backend/tests/test_audio_failed_generation.py

@@ -0,0 +1,164 @@
+"""
+Regression tests for GET /audio/{generation_id} on failed generations.
+
+A failed generation stores an empty ``audio_path``. Previously,
+``config.resolve_storage_path("")`` resolved to the data directory itself,
+which exists, so the route's 404 guard passed and ``FileResponse`` raised
+``RuntimeError: File at path .../data is not a file`` — a 500 instead of
+a clean 404.
+
+Usage:
+    python -m pytest backend/tests/test_audio_failed_generation.py -v
+"""
+
+import sys
+from pathlib import Path
+
+import pytest
+from fastapi import FastAPI
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+from starlette.testclient import TestClient
+
+# Repo root on sys.path so ``backend`` imports as a package (the audio
+# routes use package-relative imports).
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+from backend import config
+from backend.database import (
+    Base,
+    Generation,
+    GenerationVersion,
+    ProfileSample,
+    VoiceProfile,
+    get_db,
+)
+from backend.routes.audio import router as audio_router
+
+
+def test_resolve_storage_path_empty_returns_none():
+    """An empty stored path must not resolve to the data dir itself."""
+    assert config.resolve_storage_path("") is None
+    assert config.resolve_storage_path(None) is None
+    # Path("") is truthy, so it must be rejected via its (empty) parts.
+    assert config.resolve_storage_path(Path("")) is None
+
+
+@pytest.fixture
+def client(tmp_path, monkeypatch):
+    """Minimal app with only the audio routes and a temp sqlite DB."""
+    monkeypatch.setattr(config, "_data_dir", tmp_path)
+    # An existing directory that a stored audio_path may wrongly point to.
+    (tmp_path / "somedir").mkdir()
+
+    engine = create_engine(
+        f"sqlite:///{tmp_path / 'test.db'}",
+        connect_args={"check_same_thread": False},
+    )
+    Base.metadata.create_all(bind=engine)
+    testing_session_local = sessionmaker(autocommit=False, autoflush=False, bind=engine)
+
+    session = testing_session_local()
+    profile = VoiceProfile(id="profile-1", name="Test Profile")
+    session.add(profile)
+
+    session.add_all(
+        [
+            Generation(
+                id="gen-failed-empty",
+                profile_id="profile-1",
+                text="failed generation",
+                audio_path="",
+                status="failed",
+                error="engine exploded",
+            ),
+            Generation(
+                id="gen-failed-null",
+                profile_id="profile-1",
+                text="failed generation",
+                audio_path=None,
+                status="failed",
+            ),
+            Generation(
+                id="gen-missing-file",
+                profile_id="profile-1",
+                text="completed but file deleted",
+                audio_path="generations/does-not-exist.wav",
+                status="completed",
+            ),
+            Generation(
+                id="gen-with-version",
+                profile_id="profile-1",
+                text="generation with a broken version",
+                audio_path="somedir",
+                status="completed",
+            ),
+            GenerationVersion(
+                id="version-dir",
+                generation_id="gen-with-version",
+                label="original",
+                audio_path="somedir",
+            ),
+            ProfileSample(
+                id="sample-dir",
+                profile_id="profile-1",
+                audio_path="somedir",
+                reference_text="sample pointing at a directory",
+            ),
+        ]
+    )
+    session.commit()
+    session.close()
+
+    app = FastAPI()
+    app.include_router(audio_router)
+
+    def override_get_db():
+        db = testing_session_local()
+        try:
+            yield db
+        finally:
+            db.close()
+
+    app.dependency_overrides[get_db] = override_get_db
+    return TestClient(app)
+
+
+@pytest.mark.parametrize("generation_id", ["gen-failed-empty", "gen-failed-null"])
+def test_failed_generation_returns_404(client, generation_id):
+    """Failed generations (empty/null audio_path) get a clean 404, not a 500."""
+    response = client.get(f"/audio/{generation_id}")
+    assert response.status_code == 404
+    assert response.json()["detail"] == "Generation failed; no audio available"
+
+
+def test_missing_audio_file_returns_404(client):
+    """A completed generation whose file vanished still 404s."""
+    response = client.get("/audio/gen-missing-file")
+    assert response.status_code == 404
+    assert response.json()["detail"] == "Audio file not found"
+
+
+def test_unknown_generation_returns_404(client):
+    response = client.get("/audio/no-such-generation")
+    assert response.status_code == 404
+    assert response.json()["detail"] == "Generation not found"
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        "/audio/gen-with-version",
+        "/audio/version/version-dir",
+        "/samples/sample-dir",
+    ],
+)
+def test_audio_path_pointing_at_directory_returns_404(client, url):
+    """A stored path resolving to an existing directory must 404, not 500.
+
+    Guards the is_file() checks: a directory passes exists() and would
+    crash FileResponse.
+    """
+    response = client.get(url)
+    assert response.status_code == 404
+    assert response.json()["detail"] == "Audio file not found"

backend/tests/test_audio_preprocess.py

@@ -0,0 +1,112 @@
+"""
+Unit tests for reference-audio preprocessing.
+
+Covers :func:`backend.utils.audio.preprocess_reference_audio` and
+:func:`backend.utils.audio.validate_and_load_reference_audio`.
+"""
+
+import sys
+from pathlib import Path
+
+import numpy as np
+import pytest
+import soundfile as sf
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from utils.audio import (  # noqa: E402
+    preprocess_reference_audio,
+    validate_and_load_reference_audio,
+)
+
+
+SR = 24000
+
+
+def _tone(duration_s: float, amp: float = 0.3, freq: float = 220.0) -> np.ndarray:
+    n = int(duration_s * SR)
+    t = np.arange(n, dtype=np.float32) / SR
+    return (amp * np.sin(2 * np.pi * freq * t)).astype(np.float32)
+
+
+def test_peak_cap_scales_hot_input():
+    audio = _tone(3.0, amp=0.99)
+    out = preprocess_reference_audio(audio, SR)
+    assert np.abs(out).max() <= 0.951
+
+
+def test_peak_cap_leaves_moderate_input_untouched():
+    audio = _tone(3.0, amp=0.5)
+    out = preprocess_reference_audio(audio, SR)
+    assert np.isclose(np.abs(out).max(), 0.5, atol=1e-3)
+
+
+def test_dc_offset_removed():
+    audio = _tone(3.0, amp=0.3) + 0.1
+    out = preprocess_reference_audio(audio, SR)
+    assert abs(float(np.mean(out))) < 1e-3
+
+
+def test_silence_is_trimmed_with_padding_kept():
+    silence = np.zeros(int(SR * 1.0), dtype=np.float32)
+    speech = _tone(3.0, amp=0.3)
+    audio = np.concatenate([silence, speech, silence])
+    out = preprocess_reference_audio(audio, SR)
+    # Most of the 2s of leading/trailing silence should be gone, but the
+    # 3s of speech plus ~200ms of padding should remain.
+    assert len(audio) - len(out) >= SR, "expected >=1s of silence trimmed"
+    assert len(out) >= int(3.0 * SR), "speech body should be preserved"
+
+
+def test_clean_audio_is_not_padded_past_original_length():
+    # Well-recorded audio with no edge silence shouldn't get longer after
+    # preprocessing — otherwise a 29.9 s upload could be pushed past the
+    # 30 s max_duration ceiling downstream.
+    audio = _tone(3.0, amp=0.3)
+    out = preprocess_reference_audio(audio, SR)
+    assert len(out) <= len(audio)
+
+
+def test_empty_input_returns_empty():
+    out = preprocess_reference_audio(np.zeros(0, dtype=np.float32), SR)
+    assert out.size == 0
+
+
+def test_validate_accepts_previously_rejected_hot_file(tmp_path):
+    audio = _tone(3.0, amp=0.995)
+    path = tmp_path / "hot.wav"
+    sf.write(str(path), audio, SR)
+
+    ok, err, out_audio, out_sr = validate_and_load_reference_audio(str(path))
+
+    assert ok, f"expected pass, got error: {err}"
+    assert out_audio is not None
+    assert out_sr == SR
+    assert np.abs(out_audio).max() <= 0.951
+
+
+def test_validate_still_rejects_silent_input(tmp_path):
+    audio = np.zeros(int(SR * 3.0), dtype=np.float32)
+    path = tmp_path / "silent.wav"
+    sf.write(str(path), audio, SR)
+
+    ok, err, _, _ = validate_and_load_reference_audio(str(path))
+
+    assert not ok
+    assert err is not None
+    assert "too short" in err.lower() or "quiet" in err.lower()
+
+
+def test_validate_rejects_too_short(tmp_path):
+    audio = _tone(0.5, amp=0.3)
+    path = tmp_path / "short.wav"
+    sf.write(str(path), audio, SR)
+
+    ok, err, _, _ = validate_and_load_reference_audio(str(path))
+
+    assert not ok
+    assert "too short" in (err or "").lower()
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])

backend/tests/test_audioop_python313.py

@@ -0,0 +1,123 @@
+"""
+Regression tests for issue #852: audioop removed from Python 3.13 stdlib.
+
+Voice sample validation imports audioop transitively (librosa → audioread).
+The audioop-lts backport must be declared in requirements and bundled in
+PyInstaller builds on 3.13+.
+"""
+
+import re
+import sys
+from pathlib import Path
+from unittest.mock import patch
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from build_binary import build_server
+
+
+@pytest.fixture
+def backend_dir():
+    return Path(__file__).parent.parent
+
+
+class TestAudioopRequirements:
+    def test_requirements_declare_audioop_lts_for_python_313(self, backend_dir):
+        content = (backend_dir / "requirements.txt").read_text()
+        assert re.search(
+            r"^audioop-lts.*python_version\s*>=\s*['\"]3\.13['\"]",
+            content,
+            re.MULTILINE,
+        ), "requirements.txt must pin audioop-lts for Python 3.13+"
+
+
+@pytest.mark.skipif(sys.version_info < (3, 13), reason="Python 3.13+ only")
+class TestAudioopRuntime:
+    def test_audioop_importable(self):
+        import audioop  # noqa: F401
+
+    def test_validate_reference_wav_does_not_fail_on_missing_audioop(self, tmp_path):
+        import numpy as np
+        import soundfile as sf
+        from utils.audio import validate_and_load_reference_audio
+
+        sr = 24000
+        t = np.arange(int(sr * 3), dtype=np.float32) / sr
+        audio = (0.3 * np.sin(2 * np.pi * 220 * t)).astype(np.float32)
+        path = tmp_path / "reference.wav"
+        sf.write(str(path), audio, sr)
+
+        ok, err, out_audio, out_sr = validate_and_load_reference_audio(str(path))
+
+        assert ok, err
+        assert out_audio is not None
+        assert out_sr == sr
+        assert "audioop" not in (err or "").lower()
+
+
+class TestAudioopBuildArgs:
+    @staticmethod
+    def _hidden_imports(args):
+        imports = []
+        for i, arg in enumerate(args):
+            if arg == "--hidden-import" and i + 1 < len(args):
+                imports.append(args[i + 1])
+        return imports
+
+    def test_pyinstaller_includes_audioop_on_python_313(self):
+        class FakeVersionInfo(tuple):
+            @property
+            def major(self):
+                return self[0]
+
+            @property
+            def minor(self):
+                return self[1]
+
+            @property
+            def micro(self):
+                return self[2]
+
+        fake_313 = FakeVersionInfo((3, 13, 0, "final", 0))
+
+        with (
+            patch("build_binary.PyInstaller.__main__.run") as mock_run,
+            patch("build_binary.platform.system", return_value="Linux"),
+            patch("build_binary.is_apple_silicon", return_value=False),
+            patch("build_binary.os.chdir"),
+            patch("build_binary.sys.version_info", fake_313),
+        ):
+            build_server()
+            args = mock_run.call_args[0][0]
+
+        assert "audioop" in self._hidden_imports(args)
+
+    def test_pyinstaller_omits_audioop_on_python_312(self):
+        class FakeVersionInfo(tuple):
+            @property
+            def major(self):
+                return self[0]
+
+            @property
+            def minor(self):
+                return self[1]
+
+            @property
+            def micro(self):
+                return self[2]
+
+        fake_312 = FakeVersionInfo((3, 12, 0, "final", 0))
+
+        with (
+            patch("build_binary.PyInstaller.__main__.run") as mock_run,
+            patch("build_binary.platform.system", return_value="Linux"),
+            patch("build_binary.is_apple_silicon", return_value=False),
+            patch("build_binary.os.chdir"),
+            patch("build_binary.sys.version_info", fake_312),
+        ):
+            build_server()
+            args = mock_run.call_args[0][0]
+
+        assert "audioop" not in self._hidden_imports(args)

backend/tests/test_client_id_middleware.py

@@ -0,0 +1,49 @@
+"""Unit tests for the ClientIdMiddleware path predicate.
+
+Locks down which endpoints advance ``last_seen_at`` on the
+``MCPClientBinding`` row. Getting this wrong is silent: the Settings UI
+just shows a stale "last heard from" timestamp and bindings never get
+auto-created for new REST callers.
+"""
+
+import pytest
+
+from backend.mcp_server.context import _is_stamped_path
+
+
+@pytest.mark.parametrize(
+    "path",
+    [
+        "/mcp",
+        "/mcp/",
+        "/mcp/tools/call",
+        "/mcp/bindings",  # admin REST; benign — frontend never sets the header
+        "/speak",
+        "/speak/",
+    ],
+)
+def test_mcp_semantic_paths_are_stamped(path: str) -> None:
+    assert _is_stamped_path(path) is True
+
+
+@pytest.mark.parametrize(
+    "path",
+    [
+        "/",
+        "/health",
+        "/generate",
+        "/captures",
+        "/profiles",
+        "/profiles/abc/compose",
+        "/events/speak",
+        "/tasks/active",
+        "/llm/generate",
+        # Prefix overlap should not match — /speakers is a hypothetical
+        # future endpoint that shouldn't leak the stamp.
+        "/speakers",
+        # Same for anything starting with /mcpfoo.
+        "/mcpfoo",
+    ],
+)
+def test_other_paths_are_not_stamped(path: str) -> None:
+    assert _is_stamped_path(path) is False

backend/tests/test_cors.py

@@ -0,0 +1,162 @@
+"""
+Tests for CORS origin restrictions.
+
+Validates that the CORS middleware only allows known local origins
+and respects the VOICEBOX_CORS_ORIGINS environment variable.
+
+Uses a minimal FastAPI app that mirrors the exact CORS configuration
+from backend/main.py, so tests run without heavy ML dependencies.
+
+Usage:
+    pip install httpx pytest fastapi starlette
+    python -m pytest backend/tests/test_cors.py -v
+"""
+
+import os
+import pytest
+from unittest.mock import patch
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from starlette.testclient import TestClient
+
+
+def _build_app(env_origins: str = "") -> FastAPI:
+    """
+    Build a minimal FastAPI app with the same CORS logic as backend/main.py.
+
+    This mirrors the exact code in main.py so the test validates the real
+    configuration without needing torch/numpy/transformers installed.
+    """
+    app = FastAPI()
+
+    _default_origins = [
+        "http://localhost:5173",
+        "http://127.0.0.1:5173",
+        "http://localhost:17493",
+        "http://127.0.0.1:17493",
+        "tauri://localhost",
+        "https://tauri.localhost",
+    ]
+    _cors_origins = _default_origins + [o.strip() for o in env_origins.split(",") if o.strip()]
+
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=_cors_origins,
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    @app.get("/health")
+    async def health():
+        return {"status": "ok"}
+
+    return app
+
+
+@pytest.fixture()
+def client():
+    return TestClient(_build_app())
+
+
+@pytest.fixture()
+def client_with_custom_origins():
+    return TestClient(_build_app("https://custom.example.com,https://other.example.com"))
+
+
+def _get_with_origin(client: TestClient, origin: str) -> dict:
+    """Send a GET with Origin header, return response headers."""
+    response = client.get("/health", headers={"Origin": origin})
+    return dict(response.headers)
+
+
+def _preflight(client: TestClient, origin: str) -> dict:
+    """Send CORS preflight OPTIONS request, return response headers."""
+    response = client.options(
+        "/health",
+        headers={
+            "Origin": origin,
+            "Access-Control-Request-Method": "GET",
+        },
+    )
+    return dict(response.headers)
+
+
+class TestCORSDefaultOrigins:
+    """CORS should allow known local origins and block everything else."""
+
+    @pytest.mark.parametrize("origin", [
+        "http://localhost:5173",
+        "http://127.0.0.1:5173",
+        "http://localhost:17493",
+        "http://127.0.0.1:17493",
+        "tauri://localhost",
+        "https://tauri.localhost",
+    ])
+    def test_allowed_origins(self, client, origin):
+        headers = _get_with_origin(client, origin)
+        assert headers.get("access-control-allow-origin") == origin
+
+    @pytest.mark.parametrize("origin", [
+        "http://evil.com",
+        "http://localhost:9999",
+        "https://attacker.example.com",
+        "null",
+    ])
+    def test_blocked_origins(self, client, origin):
+        headers = _get_with_origin(client, origin)
+        assert "access-control-allow-origin" not in headers
+
+    def test_preflight_allowed(self, client):
+        headers = _preflight(client, "http://localhost:5173")
+        assert headers.get("access-control-allow-origin") == "http://localhost:5173"
+
+    def test_preflight_blocked(self, client):
+        headers = _preflight(client, "http://evil.com")
+        assert "access-control-allow-origin" not in headers
+
+    def test_credentials_header_present(self, client):
+        headers = _get_with_origin(client, "http://localhost:5173")
+        assert headers.get("access-control-allow-credentials") == "true"
+
+
+class TestCORSCustomOrigins:
+    """VOICEBOX_CORS_ORIGINS env var should extend the allowlist."""
+
+    def test_custom_origin_allowed(self, client_with_custom_origins):
+        headers = _get_with_origin(client_with_custom_origins, "https://custom.example.com")
+        assert headers.get("access-control-allow-origin") == "https://custom.example.com"
+
+    def test_other_custom_origin_allowed(self, client_with_custom_origins):
+        headers = _get_with_origin(client_with_custom_origins, "https://other.example.com")
+        assert headers.get("access-control-allow-origin") == "https://other.example.com"
+
+    def test_default_origins_still_work(self, client_with_custom_origins):
+        headers = _get_with_origin(client_with_custom_origins, "http://localhost:5173")
+        assert headers.get("access-control-allow-origin") == "http://localhost:5173"
+
+    def test_unlisted_origin_still_blocked(self, client_with_custom_origins):
+        headers = _get_with_origin(client_with_custom_origins, "http://evil.com")
+        assert "access-control-allow-origin" not in headers
+
+
+class TestCORSEnvVarParsing:
+    """Edge cases for VOICEBOX_CORS_ORIGINS parsing."""
+
+    def test_empty_env_var(self):
+        app = _build_app("")
+        client = TestClient(app)
+        headers = _get_with_origin(client, "http://evil.com")
+        assert "access-control-allow-origin" not in headers
+
+    def test_whitespace_trimmed(self):
+        app = _build_app("  https://spaced.example.com  ")
+        client = TestClient(app)
+        headers = _get_with_origin(client, "https://spaced.example.com")
+        assert headers.get("access-control-allow-origin") == "https://spaced.example.com"
+
+    def test_trailing_comma_ignored(self):
+        app = _build_app("https://one.example.com,")
+        client = TestClient(app)
+        headers = _get_with_origin(client, "https://one.example.com")
+        assert headers.get("access-control-allow-origin") == "https://one.example.com"

backend/tests/test_cuda_download.py

@@ -0,0 +1,32 @@
+import sys as py_sys
+import types
+
+import pytest
+
+from backend.services import cuda
+
+
+def test_cuda_status_reports_unsupported_linux_download(monkeypatch, tmp_path):
+    monkeypatch.setattr(cuda.sys, "platform", "linux")
+    monkeypatch.setattr(cuda, "get_data_dir", lambda: tmp_path)
+
+    status = cuda.get_cuda_status()
+
+    assert status["available"] is False
+    assert status["download_supported"] is False
+    assert status["unsupported_reason"] == cuda.CUDA_DOWNLOAD_UNSUPPORTED_REASON
+
+
+@pytest.mark.asyncio
+async def test_cuda_download_rejects_linux_before_network(monkeypatch, tmp_path):
+    monkeypatch.setattr(cuda.sys, "platform", "linux")
+    monkeypatch.setattr(cuda, "get_data_dir", lambda: tmp_path)
+
+    class UnexpectedClient:
+        def __init__(self, *args, **kwargs):
+            raise AssertionError("unsupported platforms should not start a release download")
+
+    monkeypatch.setitem(py_sys.modules, "httpx", types.SimpleNamespace(AsyncClient=UnexpectedClient))
+
+    with pytest.raises(RuntimeError, match="currently only published for Windows"):
+        await cuda._download_cuda_binary_locked("v0.5.0")

backend/tests/test_generation_download.py

@@ -0,0 +1,305 @@
+"""
+Test TTS generation with SSE progress monitoring.
+This test captures the exact SSE events triggered during generation
+to identify UX issues where users see download progress even when
+the model is already cached.
+"""
+
+import asyncio
+import json
+import httpx
+from typing import List, Dict, Optional
+from datetime import datetime
+
+
+async def monitor_sse_stream(model_name: str, timeout: int = 120):
+    """Monitor SSE stream for a model during generation."""
+    events: List[Dict] = []
+    url = f"http://localhost:8000/models/progress/{model_name}"
+
+    print(f"[{_timestamp()}] Connecting to SSE endpoint: {url}")
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout) as client:
+            async with client.stream("GET", url) as response:
+                print(f"[{_timestamp()}] SSE connected, status: {response.status_code}")
+
+                if response.status_code != 200:
+                    print(f"[{_timestamp()}] Error: SSE endpoint returned {response.status_code}")
+                    return events
+
+                async for line in response.aiter_lines():
+                    if not line:
+                        continue
+
+                    timestamp = _timestamp()
+
+                    if line.startswith("data: "):
+                        try:
+                            data = json.loads(line[6:])
+                            print(
+                                f"[{timestamp}] → SSE Event: {data['status']:12} {data.get('progress', 0):6.1f}% {data.get('filename', '')}"
+                            )
+                            events.append({**data, "_timestamp": timestamp})
+
+                            # Stop if complete or error
+                            if data.get("status") in ("complete", "error"):
+                                print(f"[{timestamp}] → Model {data['status']}!")
+                                break
+
+                        except json.JSONDecodeError as e:
+                            print(f"[{timestamp}] Error parsing JSON: {e}")
+                            print(f"  Line was: {line}")
+
+                    elif line.startswith(": heartbeat"):
+                        print(f"[{timestamp}] ♥ heartbeat")
+
+    except asyncio.TimeoutError:
+        print(f"[{_timestamp()}] SSE monitoring timed out")
+    except Exception as e:
+        print(f"[{_timestamp()}] SSE error: {e}")
+
+    return events
+
+
+async def trigger_generation(profile_id: str, text: str, model_size: str = "1.7B"):
+    """Trigger TTS generation via the API."""
+    url = "http://localhost:8000/generate"
+
+    print(f"\n[{_timestamp()}] Triggering generation...")
+    print(f"  Profile: {profile_id}")
+    print(f"  Text: {text[:50]}...")
+    print(f"  Model: {model_size}")
+
+    try:
+        async with httpx.AsyncClient(timeout=120) as client:
+            response = await client.post(
+                url,
+                json={
+                    "profile_id": profile_id,
+                    "text": text,
+                    "language": "en",
+                    "model_size": model_size,
+                },
+            )
+
+            print(f"[{_timestamp()}] Response: {response.status_code}")
+
+            if response.status_code == 200:
+                result = response.json()
+                print(f"[{_timestamp()}] ✓ Generation successful!")
+                print(f"  Generation ID: {result.get('id')}")
+                print(f"  Duration: {result.get('duration', 0):.2f}s")
+                return True, result
+            elif response.status_code == 202:
+                # Model is being downloaded
+                result = response.json()
+                print(f"[{_timestamp()}] → Model download in progress")
+                print(f"  Detail: {result}")
+                return False, result
+            else:
+                print(f"[{_timestamp()}] ✗ Error: {response.text}")
+                return False, None
+
+    except Exception as e:
+        print(f"[{_timestamp()}] ✗ Exception: {e}")
+        return False, None
+
+
+async def get_first_profile():
+    """Get the first available voice profile."""
+    url = "http://localhost:8000/profiles"
+
+    try:
+        async with httpx.AsyncClient(timeout=10) as client:
+            response = await client.get(url)
+            if response.status_code == 200:
+                profiles = response.json()
+                if profiles:
+                    return profiles[0]["id"]
+    except Exception as e:
+        print(f"Error getting profiles: {e}")
+
+    return None
+
+
+async def check_server():
+    """Check if the server is running."""
+    try:
+        async with httpx.AsyncClient(timeout=5) as client:
+            response = await client.get("http://localhost:8000/health")
+            return response.status_code == 200
+    except Exception as e:
+        print(f"Server not running: {e}")
+        return False
+
+
+def _timestamp():
+    """Get current timestamp for logging."""
+    return datetime.now().strftime("%H:%M:%S.%f")[:-3]
+
+
+async def test_generation_with_cached_model():
+    """
+    Test Case 1: Generation when model is already cached.
+
+    This should NOT show any download progress events.
+    If it does, that's the UX bug we're trying to fix.
+    """
+    print("\n" + "=" * 80)
+    print("TEST CASE 1: Generation with Cached Model")
+    print("=" * 80)
+    print("Expected: No download progress events (or minimal/instant completion)")
+    print("Actual UX Issue: Users see 'started' and 'finished' events even for cached models")
+    print("=" * 80)
+
+    model_size = "1.7B"
+    model_name = f"qwen-tts-{model_size}"
+
+    # Get a profile
+    profile_id = await get_first_profile()
+    if not profile_id:
+        print("✗ No voice profiles found. Please create a profile first.")
+        return False
+
+    print(f"\nUsing profile: {profile_id}")
+
+    # Start SSE monitor BEFORE triggering generation
+    monitor_task = asyncio.create_task(monitor_sse_stream(model_name, timeout=30))
+
+    # Wait for SSE to connect
+    await asyncio.sleep(1)
+
+    # Trigger generation
+    test_text = "Hello, this is a test of the voice generation system."
+    success, result = await trigger_generation(profile_id, test_text, model_size)
+
+    if not success and result and result.get("downloading"):
+        print("\n⚠ Model is being downloaded. Waiting for download to complete...")
+        # Wait for SSE monitor to capture download events
+        events = await monitor_task
+        return events
+
+    # Wait a bit more to catch any progress events
+    await asyncio.sleep(3)
+
+    # Cancel SSE monitor
+    monitor_task.cancel()
+    try:
+        events = await monitor_task
+    except asyncio.CancelledError:
+        events = []
+
+    return events
+
+
+async def test_generation_with_fresh_download():
+    """
+    Test Case 2: Generation when model needs to be downloaded.
+
+    This SHOULD show download progress events.
+    """
+    print("\n" + "=" * 80)
+    print("TEST CASE 2: Generation with Model Download")
+    print("=" * 80)
+    print("Expected: Download progress events from 0% to 100%")
+    print("=" * 80)
+
+    # Use a different model size to force download
+    model_size = "0.6B"  # Smaller model for faster testing
+    model_name = f"qwen-tts-{model_size}"
+
+    # Get a profile
+    profile_id = await get_first_profile()
+    if not profile_id:
+        print("✗ No voice profiles found. Please create a profile first.")
+        return False
+
+    print(f"\nUsing profile: {profile_id}")
+    print("Note: This will download the model if not cached")
+
+    # Start SSE monitor BEFORE triggering generation
+    monitor_task = asyncio.create_task(monitor_sse_stream(model_name, timeout=300))
+
+    # Wait for SSE to connect
+    await asyncio.sleep(1)
+
+    # Trigger generation
+    test_text = "This should trigger a model download if the model is not cached."
+    success, result = await trigger_generation(profile_id, test_text, model_size)
+
+    if not success and result and result.get("downloading"):
+        print("\n→ Model download initiated. Monitoring progress...")
+        # Wait for download to complete
+        events = await monitor_task
+
+        # Try generation again
+        print(f"\n[{_timestamp()}] Retrying generation after download...")
+        await asyncio.sleep(2)
+        success, result = await trigger_generation(profile_id, test_text, model_size)
+
+        if success:
+            print("✓ Generation successful after download")
+
+        return events
+
+    # If model was already cached
+    await asyncio.sleep(3)
+    monitor_task.cancel()
+    try:
+        events = await monitor_task
+    except asyncio.CancelledError:
+        events = []
+
+    return events
+
+
+async def main():
+    print("=" * 80)
+    print("TTS Generation Progress Test")
+    print("=" * 80)
+    print("Purpose: Capture exact SSE events during generation to identify UX issues")
+    print("=" * 80)
+
+    # Check if server is running
+    print(f"\n[{_timestamp()}] Checking if server is running...")
+    if not await check_server():
+        print("✗ Server is not running on http://localhost:8000")
+        print("\nPlease start the server first:")
+        print("  cd backend && python main.py")
+        return False
+
+    print("✓ Server is running")
+
+    # Test Case 1: Cached model
+    print("\n" + "🧪 " * 20)
+    events_cached = await test_generation_with_cached_model()
+
+    # Results for Test Case 1
+    print("\n" + "=" * 80)
+    print("TEST CASE 1 RESULTS: Generation with Cached Model")
+    print("=" * 80)
+
+    if not events_cached:
+        print("✓ GOOD: No SSE progress events received")
+        print("  This is the expected behavior for a cached model.")
+    else:
+        print(f"⚠ ISSUE FOUND: Received {len(events_cached)} SSE events:")
+        print("\nEvent Timeline:")
+        for i, event in enumerate(events_cached, 1):
+            timestamp = event.pop("_timestamp", "??:??:??.???")
+            print(f"  {i}. [{timestamp}] {event}")
+
+        print("\n⚠ This explains the UX issue!")
+        print("  Users see progress events even when the model is already cached,")
+        print("  making them think the model is downloading again.")
+
+    print("\n" + "=" * 80)
+    print("Test Complete!")
+    print("=" * 80)
+
+    return True
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

backend/tests/test_hume_encode_inference_mode.py

@@ -0,0 +1,68 @@
+"""Ensure TADA voice-prompt encoding disables autograd (#890)."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from unittest.mock import AsyncMock
+
+import numpy as np
+import pytest
+import soundfile as sf
+import torch
+
+from backend.backends.hume_backend import HumeTadaBackend
+
+
+@dataclass
+class _FakeEncoderOutput:
+    emb: torch.Tensor
+
+
+class _GradTrackingEncoder:
+    """Raises unless called under torch.inference_mode()."""
+
+    def __init__(self) -> None:
+        self.called_under_inference_mode = False
+
+    def __call__(self, audio, text=None, sample_rate=None):
+        self.called_under_inference_mode = torch.is_inference_mode_enabled()
+        if not self.called_under_inference_mode:
+            raise AssertionError("encoder forward must run under inference_mode")
+        # Touch a requires_grad tensor the way Snake1d alpha would.
+        alpha = torch.nn.Parameter(torch.ones(1, device=audio.device))
+        _ = audio.mean() * alpha
+        return _FakeEncoderOutput(emb=torch.zeros(1, 4, device=audio.device))
+
+
+@pytest.mark.asyncio
+async def test_create_voice_prompt_runs_encoder_under_inference_mode(tmp_path, monkeypatch):
+    wav = tmp_path / "ref.wav"
+    sf.write(str(wav), np.zeros(24000, dtype=np.float32), 24000)
+
+    backend = HumeTadaBackend()
+    backend.model = object()  # mark loaded
+    backend.model_size = "1B"
+    backend._device = "cpu"
+    encoder = _GradTrackingEncoder()
+    backend.encoder = encoder
+
+    monkeypatch.setattr(backend, "load_model", AsyncMock(return_value=None))
+    monkeypatch.setattr(
+        "backend.backends.hume_backend.get_cached_voice_prompt",
+        lambda key: None,
+    )
+    monkeypatch.setattr(
+        "backend.backends.hume_backend.cache_voice_prompt",
+        lambda key, value: None,
+    )
+
+    prompt, from_cache = await backend.create_voice_prompt(
+        str(wav),
+        reference_text="hello world",
+        use_cache=False,
+    )
+
+    assert from_cache is False
+    assert encoder.called_under_inference_mode is True
+    assert isinstance(prompt["emb"], torch.Tensor)
+    assert prompt["emb"].device.type == "cpu"

backend/tests/test_mcp_speak.py

@@ -0,0 +1,91 @@
+"""Tests for the voicebox.speak MCP tool's ``model_size`` plumbing (issue #884).
+
+The MCP speak path used to build its ``GenerationRequest`` without a
+``model_size``, so every agent-triggered generation silently fell back to the
+schema default ("1.7B") — there was no way to reach 0.6B (or TADA's 1B/3B)
+through MCP. These tests pin the fix: ``_speak`` now forwards ``model_size``
+straight into the request, matching the REST ``/generate`` surface.
+"""
+
+import pytest
+from pydantic import ValidationError
+
+import backend.routes.generations as generations
+from backend.mcp_server import tools
+
+
+class _FakeGeneration:
+    """Minimal stand-in for GenerationResponse consumed by ``_speak_response``."""
+
+    def model_dump(self, mode="json"):
+        return {"id": "gen-test", "status": "generating"}
+
+
+@pytest.fixture
+def captured_request(monkeypatch):
+    """Replace the real (torch-backed) generate_speech with a capturing stub.
+
+    ``_speak`` imports ``generate_speech`` lazily from ``routes.generations``,
+    so patching the attribute on that module intercepts the call and lets us
+    inspect the ``GenerationRequest`` it would have run.
+    """
+    captured = {}
+
+    async def fake_generate_speech(req, db):
+        captured["req"] = req
+        return _FakeGeneration()
+
+    monkeypatch.setattr(generations, "generate_speech", fake_generate_speech)
+    # Isolate the unit from the MCP event bus — _speak_response fires a
+    # speak-start event we don't care about here.
+    monkeypatch.setattr(tools.mcp_events, "publish", lambda *a, **k: None)
+    return captured
+
+
+@pytest.mark.asyncio
+async def test_speak_forwards_explicit_model_size(captured_request):
+    await tools._speak(
+        profile_id="p1",
+        profile_name="Morgan",
+        text="hello",
+        engine="qwen",
+        language="en",
+        personality=False,
+        model_size="0.6B",
+        db=None,
+    )
+    assert captured_request["req"].model_size == "0.6B"
+
+
+@pytest.mark.asyncio
+async def test_speak_omitted_model_size_is_none(captured_request):
+    # Omitted → None; generate_speech normalizes None to the engine default,
+    # so this reproduces the pre-fix behaviour for callers that don't ask.
+    await tools._speak(
+        profile_id="p1",
+        profile_name="Morgan",
+        text="hello",
+        engine="qwen",
+        language="en",
+        personality=False,
+        db=None,
+    )
+    assert captured_request["req"].model_size is None
+
+
+@pytest.mark.asyncio
+async def test_speak_rejects_invalid_model_size(captured_request):
+    # The GenerationRequest schema pattern is the single source of truth for
+    # valid sizes; a bad value is rejected before any generation runs.
+    with pytest.raises(ValidationError):
+        await tools._speak(
+            profile_id="p1",
+            profile_name="Morgan",
+            text="hello",
+            engine="qwen",
+            language="en",
+            personality=False,
+            model_size="9B",
+            db=None,
+        )
+    assert "req" not in captured_request

backend/tests/test_mlx_smoke.py

@@ -0,0 +1,55 @@
+"""
+Smoke test for the MLX backend dependencies on Apple Silicon.
+
+Guards the `--no-deps` install of mlx-audio/mlx-lm done by `just setup-python`
+and release.yml: those packages skip their declared dependencies (transformers
+>=5.x conflict), so a missing transitive dep only surfaces at import time.
+This test fails fast if the MLX STT/TTS entry points the backend uses stop
+importing (e.g. the `miniaudio` regression from issue #505).
+
+Usage:
+    python -m pytest backend/tests/test_mlx_smoke.py -v
+"""
+
+import platform
+import sys
+
+import pytest
+
+pytestmark = pytest.mark.skipif(
+    not (sys.platform == "darwin" and platform.machine() == "arm64"),
+    reason="MLX packages are only installed on Apple Silicon macOS",
+)
+
+
+def test_mlx_core_runs():
+    """The MLX runtime itself works (Metal array op)."""
+    import mlx.core as mx
+
+    assert mx.array([1, 2]).sum().item() == 3
+
+
+def test_mlx_audio_tts_entry_point():
+    """`from mlx_audio.tts import load` — used by MLXBackend.load_model_async."""
+    from mlx_audio.tts import load
+
+    assert callable(load)
+
+
+def test_mlx_audio_stt_entry_point():
+    """`from mlx_audio.stt import load` — used by the Whisper MLX STT path.
+
+    Importing mlx_audio.stt also pulls in miniaudio, so this catches the
+    ModuleNotFoundError from issue #505 on fresh installs.
+    """
+    from mlx_audio.stt import load
+
+    assert callable(load)
+
+
+def test_mlx_lm_entry_points():
+    """`mlx_lm.load` / `mlx_lm.generate` — used by qwen_llm_backend."""
+    from mlx_lm import generate, load
+
+    assert callable(load)
+    assert callable(generate)

backend/tests/test_model_status_pending_downloads.py

@@ -0,0 +1,51 @@
+"""Errored downloads must not be reported as still downloading.
+
+A failed download intentionally stays in the TaskManager with
+``status="error"`` so ``/tasks/active`` can surface the error and retry
+UI — but ``/models/status`` derives its ``downloading`` flag from the
+same list. Without a status filter, one failed download shows the model
+as "downloading" forever and masks its real cache state until the app
+restarts (issue #925, symptom reports like #181).
+"""
+
+from backend.utils.tasks import TaskManager
+
+
+def test_errored_download_is_not_pending():
+    tm = TaskManager()
+    tm.start_download("whisper-turbo")
+    assert [t.model_name for t in tm.get_pending_downloads()] == ["whisper-turbo"]
+
+    tm.error_download("whisper-turbo", "boom")
+
+    assert tm.get_pending_downloads() == []
+    # Still visible to /tasks/active for the error/retry UI.
+    active = tm.get_active_downloads()
+    assert [t.model_name for t in active] == ["whisper-turbo"]
+    assert active[0].status == "error"
+    assert active[0].error == "boom"
+
+
+def test_retry_after_error_is_pending_again():
+    tm = TaskManager()
+    tm.start_download("qwen3-4b")
+    tm.error_download("qwen3-4b", "boom")
+    tm.start_download("qwen3-4b")
+    assert [t.model_name for t in tm.get_pending_downloads()] == ["qwen3-4b"]
+
+
+def test_completed_download_is_removed_everywhere():
+    tm = TaskManager()
+    tm.start_download("whisper-turbo")
+    tm.complete_download("whisper-turbo")
+    assert tm.get_pending_downloads() == []
+    assert tm.get_active_downloads() == []
+
+
+def test_cancel_dismisses_errored_download():
+    tm = TaskManager()
+    tm.start_download("whisper-turbo")
+    tm.error_download("whisper-turbo", "boom")
+    assert tm.cancel_download("whisper-turbo") is True
+    assert tm.get_active_downloads() == []
+    assert tm.get_pending_downloads() == []

backend/tests/test_offline_guard.py

@@ -0,0 +1,118 @@
+"""
+Unit tests for the ``force_offline_if_cached`` helper.
+
+Verifies that the helper mutates the cached module constants in
+``huggingface_hub.constants`` and ``transformers.utils.hub`` — not just
+``os.environ`` — and that concurrent users are refcount-coordinated so
+one thread's exit can't strip another thread's offline protection.
+
+NOTE: These tests mutate process-global state in ``huggingface_hub.constants``
+and ``transformers.utils.hub``. They are not safe under cross-process
+parallelism (e.g. ``pytest-xdist`` with ``--dist=loadfile``/``loadscope``);
+run this file serially.
+"""
+
+import os
+import sys
+import threading
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from utils.hf_offline_patch import force_offline_if_cached  # noqa: E402
+
+
+def _hf_const():
+    import huggingface_hub.constants as hf_const
+
+    return hf_const
+
+
+def _tf_hub():
+    import transformers.utils.hub as tf_hub
+
+    return tf_hub
+
+
+def test_mutates_cached_huggingface_hub_constant():
+    original = _hf_const().HF_HUB_OFFLINE
+    with force_offline_if_cached(True, "t"):
+        assert _hf_const().HF_HUB_OFFLINE is True
+    assert original == _hf_const().HF_HUB_OFFLINE
+
+
+def test_mutates_cached_transformers_constant():
+    original = _tf_hub()._is_offline_mode
+    with force_offline_if_cached(True, "t"):
+        assert _tf_hub()._is_offline_mode is True
+    assert original == _tf_hub()._is_offline_mode
+
+
+def test_sets_env_variable():
+    original = os.environ.get("HF_HUB_OFFLINE")
+    with force_offline_if_cached(True, "t"):
+        assert "1" == os.environ.get("HF_HUB_OFFLINE")
+    assert original == os.environ.get("HF_HUB_OFFLINE")
+
+
+def test_noop_when_not_cached():
+    before = _hf_const().HF_HUB_OFFLINE
+    with force_offline_if_cached(False, "t"):
+        assert before == _hf_const().HF_HUB_OFFLINE
+
+
+def test_nested_contexts_respect_refcount():
+    original = _hf_const().HF_HUB_OFFLINE
+    with force_offline_if_cached(True, "outer"):
+        assert _hf_const().HF_HUB_OFFLINE is True
+        with force_offline_if_cached(True, "inner"):
+            assert _hf_const().HF_HUB_OFFLINE is True
+        # inner exit must not restore while outer is still active
+        assert _hf_const().HF_HUB_OFFLINE is True
+    assert original == _hf_const().HF_HUB_OFFLINE
+
+
+def test_concurrent_threads_share_offline_window():
+    """A slow thread must keep seeing offline mode even if a peer exits first."""
+    original = _hf_const().HF_HUB_OFFLINE
+    observations: list[bool] = []
+    errors: list[Exception] = []
+    barrier = threading.Barrier(2)
+    fast_exited = threading.Event()
+
+    def slow():
+        try:
+            with force_offline_if_cached(True, "slow"):
+                barrier.wait(timeout=5)
+                assert fast_exited.wait(timeout=5), "fast thread did not exit"
+                observations.append(_hf_const().HF_HUB_OFFLINE)
+        except Exception as exc:  # noqa: BLE001
+            errors.append(exc)
+
+    def fast():
+        try:
+            with force_offline_if_cached(True, "fast"):
+                barrier.wait(timeout=5)
+        except Exception as exc:  # noqa: BLE001
+            errors.append(exc)
+        finally:
+            fast_exited.set()
+
+    t_slow = threading.Thread(target=slow)
+    t_fast = threading.Thread(target=fast)
+    t_slow.start()
+    t_fast.start()
+    t_slow.join(timeout=5)
+    t_fast.join(timeout=5)
+
+    assert not t_slow.is_alive(), "slow thread did not finish"
+    assert not t_fast.is_alive(), "fast thread did not finish"
+    assert not errors, errors
+    assert observations == [True], "slow thread lost offline protection"
+    assert original == _hf_const().HF_HUB_OFFLINE
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])

backend/tests/test_offline_patch.py

@@ -0,0 +1,113 @@
+"""
+Unit tests for ``patch_transformers_mistral_regex``.
+
+Verifies that our wrapper around
+``transformers.PreTrainedTokenizerBase._patch_mistral_regex`` catches
+exceptions from the unconditional ``huggingface_hub.model_info()`` lookup
+and returns the tokenizer unchanged — matching the success-path behavior
+for non-Mistral repos (transformers 4.57.3, ``tokenization_utils_base.py:2503``).
+
+NOTE: These tests mutate ``transformers.PreTrainedTokenizerBase`` globally;
+run serially, not under ``pytest-xdist`` with per-worker process isolation.
+"""
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from huggingface_hub.errors import OfflineModeIsEnabled  # noqa: E402
+from transformers.tokenization_utils_base import PreTrainedTokenizerBase  # noqa: E402
+
+import utils.hf_offline_patch as hf_offline_patch  # noqa: E402
+
+
+@pytest.fixture(autouse=True)
+def restore_mistral_regex():
+    """Snapshot the current ``_patch_mistral_regex`` and restore after each test."""
+    saved = PreTrainedTokenizerBase.__dict__.get("_patch_mistral_regex")
+    saved_flag = hf_offline_patch._mistral_regex_patched
+    try:
+        yield
+    finally:
+        if saved is not None:
+            PreTrainedTokenizerBase._patch_mistral_regex = saved
+        hf_offline_patch._mistral_regex_patched = saved_flag
+
+
+def _apply_patch():
+    hf_offline_patch._mistral_regex_patched = False
+    hf_offline_patch.patch_transformers_mistral_regex()
+
+
+def test_suppresses_offline_mode_is_enabled(monkeypatch):
+    _apply_patch()
+
+    import huggingface_hub
+
+    def raise_offline(*_args, **_kwargs):
+        raise OfflineModeIsEnabled("offline")
+
+    monkeypatch.setattr(huggingface_hub, "model_info", raise_offline)
+
+    sentinel = object()
+    result = PreTrainedTokenizerBase._patch_mistral_regex(
+        sentinel, "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
+    )
+    assert result is sentinel
+
+
+def test_suppresses_connection_errors(monkeypatch):
+    _apply_patch()
+
+    import huggingface_hub
+
+    def raise_connection(*_args, **_kwargs):
+        raise ConnectionError("network unreachable")
+
+    monkeypatch.setattr(huggingface_hub, "model_info", raise_connection)
+
+    sentinel = object()
+    result = PreTrainedTokenizerBase._patch_mistral_regex(
+        sentinel, "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
+    )
+    assert result is sentinel
+
+
+def test_passthrough_on_success(monkeypatch):
+    """When model_info returns non-Mistral tags the original falls through and returns the tokenizer unchanged."""
+    _apply_patch()
+
+    import huggingface_hub
+
+    class FakeInfo:
+        tags = ["model-type:qwen", "language:en"]
+
+    monkeypatch.setattr(huggingface_hub, "model_info", lambda *_a, **_kw: FakeInfo())
+
+    sentinel = object()
+    result = PreTrainedTokenizerBase._patch_mistral_regex(
+        sentinel, "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
+    )
+    assert result is sentinel
+
+
+def test_idempotent():
+    _apply_patch()
+    first = PreTrainedTokenizerBase._patch_mistral_regex
+    hf_offline_patch.patch_transformers_mistral_regex()
+    second = PreTrainedTokenizerBase._patch_mistral_regex
+    assert first.__func__ is second.__func__
+
+
+def test_missing_method_is_noop(monkeypatch):
+    monkeypatch.delattr(PreTrainedTokenizerBase, "_patch_mistral_regex", raising=False)
+    hf_offline_patch._mistral_regex_patched = False
+    hf_offline_patch.patch_transformers_mistral_regex()
+    assert hf_offline_patch._mistral_regex_patched is False
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])

backend/tests/test_package_rocm.py

@@ -0,0 +1,121 @@
+"""
+Tests for scripts/package_rocm.py — the ROCm onedir → server + libs splitter.
+
+The classifier can't be validated against a real AMD build on CI hardware, so
+these tests pin the file-classification rules against a synthetic onedir layout
+that mirrors the PyInstaller --rocm output (torch/lib HIP DLLs + bundled
+rocm_sdk runtime packages).
+
+Usage:
+    python -m pytest backend/tests/test_package_rocm.py -v
+"""
+
+import importlib.util
+import tarfile
+from pathlib import Path
+
+import pytest
+
+_PACKAGE_ROCM = Path(__file__).resolve().parents[2] / "scripts" / "package_rocm.py"
+_spec = importlib.util.spec_from_file_location("package_rocm", _PACKAGE_ROCM)
+package_rocm = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(package_rocm)
+
+
+class TestIsRocmFile:
+    """Classification of individual files into core vs ROCm libs."""
+
+    @pytest.mark.parametrize(
+        "rel_path",
+        [
+            "_internal/torch/lib/amdhip64.dll",
+            "_internal/torch/lib/rocblas.dll",
+            "_internal/torch/lib/hipblaslt.dll",
+            "_internal/torch/lib/miopen.dll",
+            "_internal/_rocm_sdk_core/amd_comgr.dll",
+            "_internal/_rocm_sdk_libraries_custom/lib/rocblas/library/TensileLibrary.dat",
+            "_internal/_rocm_sdk_libraries_custom/lib/miopen/db/kernels.kdb",
+            # Windows path separators must be handled too.
+            "_internal\\torch\\lib\\rccl.dll",
+        ],
+    )
+    def test_runtime_files_are_rocm(self, rel_path):
+        assert package_rocm.is_rocm_file(rel_path) is True
+
+    @pytest.mark.parametrize(
+        "rel_path",
+        [
+            "voicebox-server-rocm.exe",
+            "_internal/python312.dll",
+            "_internal/torch/lib/torch_cpu.dll",
+            "_internal/torch/lib/c10.dll",
+            # Pure-python rocm_sdk glue stays in the core, even under an SDK dir.
+            "_internal/rocm_sdk/__init__.py",
+            "_internal/_rocm_sdk_core/_dist_info.py",
+            "_internal/torch/_inductor/codegen/something.py",
+        ],
+    )
+    def test_core_files_are_not_rocm(self, rel_path):
+        assert package_rocm.is_rocm_file(rel_path) is False
+
+
+def _write(path: Path, content: bytes = b"x"):
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_bytes(content)
+
+
+class TestPackage:
+    """End-to-end split of a synthetic onedir into the two archives."""
+
+    def test_split_and_manifest(self, tmp_path):
+        onedir = tmp_path / "voicebox-server-rocm"
+        _write(onedir / "voicebox-server-rocm.exe")
+        _write(onedir / "_internal" / "python312.dll")
+        _write(onedir / "_internal" / "rocm_sdk" / "__init__.py")
+        _write(onedir / "_internal" / "torch" / "lib" / "torch_cpu.dll")
+        _write(onedir / "_internal" / "torch" / "lib" / "amdhip64.dll")
+        _write(onedir / "_internal" / "_rocm_sdk_core" / "miopen.dll")
+        _write(
+            onedir
+            / "_internal"
+            / "_rocm_sdk_libraries_custom"
+            / "lib"
+            / "rocblas"
+            / "library"
+            / "TensileLibrary.dat"
+        )
+
+        out = tmp_path / "release-assets"
+        package_rocm.package(onedir, out, "rocm7.2-v1", ">=2.9.0,<2.10.0")
+
+        server = out / "voicebox-server-rocm.tar.gz"
+        libs = out / "rocm-libs-rocm7.2-v1.tar.gz"
+        assert server.exists()
+        assert libs.exists()
+        assert (out / "voicebox-server-rocm.tar.gz.sha256").exists()
+        assert (out / "rocm-libs-rocm7.2-v1.tar.gz.sha256").exists()
+
+        with tarfile.open(libs) as tar:
+            lib_names = set(tar.getnames())
+        with tarfile.open(server) as tar:
+            core_names = set(tar.getnames())
+
+        assert "_internal/torch/lib/amdhip64.dll" in lib_names
+        assert "_internal/_rocm_sdk_core/miopen.dll" in lib_names
+        assert (
+            "_internal/_rocm_sdk_libraries_custom/lib/rocblas/library/TensileLibrary.dat"
+            in lib_names
+        )
+        assert "voicebox-server-rocm.exe" in core_names
+        assert "_internal/torch/lib/torch_cpu.dll" in core_names
+        assert "_internal/rocm_sdk/__init__.py" in core_names
+        # Archives must be disjoint.
+        assert lib_names.isdisjoint(core_names)
+
+    def test_empty_rocm_set_exits(self, tmp_path):
+        onedir = tmp_path / "voicebox-server-rocm"
+        _write(onedir / "voicebox-server-rocm.exe")
+        _write(onedir / "_internal" / "torch" / "lib" / "torch_cpu.dll")
+
+        with pytest.raises(SystemExit):
+            package_rocm.package(onedir, tmp_path / "out", "rocm7.2-v1", ">=2.9.0,<2.10.0")

backend/tests/test_personality_samples.py

@@ -0,0 +1,344 @@
+"""
+Personality-service sanity sweep — spins up a throwaway profile with a
+fake personality, exercises ``/profiles/{id}/compose`` and the rewrite
+path on ``/generate`` (``personality=true``), and scores each output
+against a handful of deterministic heuristics so a person can eyeball
+quality.
+
+Same philosophy as ``test_refinement_samples.py``: LLM output is
+non-deterministic, "correctness" is subjective, so this is interactive
+evaluation — not a CI pass/fail. Gross failures (prompt-echo, refusal,
+empty output) trip heuristic flags. A human still reads the final
+column.
+
+Usage:
+    # Backend server must be running.
+    python backend/tests/test_personality_samples.py
+
+    # Test just one model size:
+    python backend/tests/test_personality_samples.py --model 4B
+
+    # Dump JSON for diffing against a prior run:
+    python backend/tests/test_personality_samples.py --json out.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import socket
+import sys
+import time
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+from typing import Optional
+
+import httpx
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO_ROOT))
+
+
+# ── Sample personalities ──────────────────────────────────────────────
+
+
+@dataclass(frozen=True)
+class Personality:
+    name: str
+    description: str
+    """Free-form character prompt saved to the profile."""
+    sample_text: str
+    """Input used for rewrite. Picked so each personality has something
+    distinctive to say about it — an ill fit between text and personality
+    makes the transformation more obvious."""
+
+
+PERSONALITIES: tuple[Personality, ...] = (
+    Personality(
+        name="grumpy-pirate",
+        description=(
+            "A grumpy old pirate captain who only speaks in nautical "
+            "metaphors. Keeps things short and salty. Swears by his "
+            "beard and the deep blue."
+        ),
+        sample_text="I need you to install the dependencies before the deploy.",
+    ),
+    Personality(
+        name="victorian-professor",
+        description=(
+            "A stuffy Victorian-era professor of natural philosophy. "
+            "Formal register, long sentences, fond of subordinate "
+            "clauses, occasional Latin asides."
+        ),
+        sample_text="The build is broken, we should roll back to yesterday's version.",
+    ),
+    Personality(
+        name="caffeinated-founder",
+        description=(
+            "A tech-bro startup founder who is always three coffees "
+            "deep, obsessed with disruption and synergy, speaks in "
+            "bullet points even out loud."
+        ),
+        sample_text="The meeting ran long and we didn't get to the roadmap.",
+    ),
+)
+
+
+# ── Scoring heuristics ────────────────────────────────────────────────
+
+
+PROMPT_LEAK_PHRASES = tuple(
+    re.compile(pat, re.IGNORECASE)
+    for pat in (
+        r"^here (?:is|'s) the cleaned",
+        r"^here (?:is|'s) a",
+        r"^as (?:an ai|the character)",
+        r"^character description",
+        r"^task:\s*",
+        r"^output:\s*$",
+        r"^sure,?\s+(?:here|i'?ll|let)",
+    )
+)
+
+
+REFUSAL_PHRASES = tuple(
+    re.compile(pat, re.IGNORECASE)
+    for pat in (
+        r"\bi (?:cannot|can't|won'?t|will not|refuse)\b",
+        r"\bi'?m sorry(?:,|\s+but)",
+        r"\bi apologi[sz]e",
+    )
+)
+
+
+STAGE_DIRECTION_RE = re.compile(r"[\*\(_].{0,60}?[\*\)_]")  # *smiles*, (leans in)
+
+
+@dataclass
+class Scorecard:
+    personality: str
+    endpoint: str
+    model: str
+    input_text: str
+    """Empty for compose."""
+    refined: str
+    latency_ms: int
+    length_chars: int = 0
+    prompt_leak: Optional[str] = None
+    refusal: Optional[str] = None
+    stage_directions: list[str] = field(default_factory=list)
+    flags: list[str] = field(default_factory=list)
+
+
+def first_match(patterns, text: str) -> Optional[str]:
+    s = text.lstrip()
+    for pat in patterns:
+        m = pat.search(s)
+        if m:
+            return m.group(0)
+    return None
+
+
+def score(
+    personality: Personality,
+    endpoint: str,
+    model: str,
+    input_text: str,
+    refined: str,
+    latency_ms: int,
+) -> Scorecard:
+    card = Scorecard(
+        personality=personality.name,
+        endpoint=endpoint,
+        model=model,
+        input_text=input_text,
+        refined=refined,
+        latency_ms=latency_ms,
+        length_chars=len(refined),
+        prompt_leak=first_match(PROMPT_LEAK_PHRASES, refined),
+        refusal=first_match(REFUSAL_PHRASES, refined),
+        stage_directions=STAGE_DIRECTION_RE.findall(refined)[:3],
+    )
+
+    if not refined.strip():
+        card.flags.append("empty-output")
+    if card.prompt_leak:
+        card.flags.append(f"prompt-leak({card.prompt_leak!r})")
+    if card.refusal:
+        card.flags.append(f"refusal({card.refusal!r})")
+    if card.stage_directions:
+        card.flags.append(f"stage-directions={card.stage_directions}")
+
+    return card
+
+
+# ── Runner ────────────────────────────────────────────────────────────
+
+
+DEFAULT_PORTS = (8000, 8765, 8899, 17493)
+THROWAWAY_PROFILE_PREFIX = "personality-harness-"
+KOKORO_PROBE_VOICE = "af_heart"
+"""Any valid kokoro voice id works — compose never calls into TTS, it
+just needs a profile row with a personality attached. We pick a
+known-shipping Kokoro voice so the throwaway profile satisfies the
+preset-engine validator on creation."""
+
+
+def detect_backend_port(hint: Optional[int]) -> int:
+    candidates: list[int] = []
+    if hint is not None:
+        candidates.append(hint)
+    candidates.extend(p for p in DEFAULT_PORTS if p != hint)
+    for port in candidates:
+        try:
+            with socket.create_connection(("127.0.0.1", port), timeout=0.4):
+                pass
+        except OSError:
+            continue
+        try:
+            r = httpx.get(f"http://127.0.0.1:{port}/health", timeout=2.0)
+            if r.status_code == 200 and r.json().get("status") == "healthy":
+                return port
+        except Exception:
+            continue
+    raise SystemExit(
+        "No running Voicebox backend found. Start it (`python backend/main.py`) "
+        f"or pass --port. Tried: {candidates}"
+    )
+
+
+def create_throwaway_profile(
+    client: httpx.Client, port: int, personality: Personality, model: str
+) -> str:
+    """Create a preset Kokoro profile with the test personality. Returns
+    the profile id. Tests delete it in a finally block."""
+    name = f"{THROWAWAY_PROFILE_PREFIX}{personality.name}-{model}-{int(time.time())}"
+    resp = client.post(
+        f"http://127.0.0.1:{port}/profiles",
+        json={
+            "name": name,
+            "description": f"Throwaway profile for personality harness ({model}).",
+            "language": "en",
+            "voice_type": "preset",
+            "preset_engine": "kokoro",
+            "preset_voice_id": KOKORO_PROBE_VOICE,
+            "default_engine": "kokoro",
+            "personality": personality.description,
+        },
+        timeout=30.0,
+    )
+    resp.raise_for_status()
+    return resp.json()["id"]
+
+
+def delete_profile(client: httpx.Client, port: int, profile_id: str) -> None:
+    try:
+        client.delete(f"http://127.0.0.1:{port}/profiles/{profile_id}", timeout=10.0)
+    except Exception as e:
+        print(f"  (warning: failed to delete throwaway profile {profile_id}: {e})")
+
+
+def hit_compose(
+    client: httpx.Client,
+    port: int,
+    profile_id: str,
+) -> tuple[str, int]:
+    start = time.monotonic()
+    url = f"http://127.0.0.1:{port}/profiles/{profile_id}/compose"
+    resp = client.post(url, timeout=180.0)
+    latency_ms = int((time.monotonic() - start) * 1000)
+    resp.raise_for_status()
+    return resp.json().get("text", "").strip(), latency_ms
+
+
+def format_report(cards: list[Scorecard]) -> str:
+    lines: list[str] = ["", "═" * 100]
+    by_model: dict[str, list[Scorecard]] = {}
+    for c in cards:
+        by_model.setdefault(c.model, []).append(c)
+    for model, model_cards in by_model.items():
+        clean = sum(1 for c in model_cards if not c.flags)
+        avg = sum(c.latency_ms for c in model_cards) // max(len(model_cards), 1)
+        lines.append("")
+        lines.append(f"▌{model}  —  {clean}/{len(model_cards)} clean, avg {avg} ms")
+        lines.append("─" * 100)
+        for c in model_cards:
+            status = "✓" if not c.flags else "✗"
+            tag = f"{c.personality} · {c.endpoint}"
+            lines.append(f"  {status} {tag}  ({c.latency_ms} ms)")
+            if c.input_text:
+                lines.append(
+                    f"      in:      {c.input_text[:90]}{'…' if len(c.input_text) > 90 else ''}"
+                )
+            lines.append(
+                f"      out:     {c.refined[:120]}{'…' if len(c.refined) > 120 else ''}"
+            )
+            if c.flags:
+                lines.append(f"      ⚠ {'; '.join(c.flags)}")
+            lines.append("")
+    lines.append("═" * 100)
+    return "\n".join(lines)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--port", type=int, default=None)
+    ap.add_argument("--model", choices=("0.6B", "1.7B", "4B"), action="append")
+    ap.add_argument("--json", type=Path, default=None)
+    args = ap.parse_args()
+
+    models = tuple(args.model) if args.model else ("0.6B", "4B")
+    port = detect_backend_port(args.port)
+    print(f"backend → http://127.0.0.1:{port}")
+    print(f"personalities → {len(PERSONALITIES)}, models → {models}")
+
+    # Model size is set on the capture_settings singleton, not passed
+    # per-request to /profiles/{id}/compose. The harness swaps it
+    # between runs so we probe both sizes cleanly.
+    cards: list[Scorecard] = []
+    with httpx.Client() as client:
+        for model in models:
+            print(f"\n── {model} " + "─" * (80 - len(model) - 4))
+            # Flip the server-side default LLM size for this pass.
+            client.put(
+                f"http://127.0.0.1:{port}/settings/captures",
+                json={"llm_model": model},
+                timeout=10.0,
+            )
+            for personality in PERSONALITIES:
+                print(f"  [{personality.name}] ", end="", flush=True)
+                profile_id = create_throwaway_profile(client, port, personality, model)
+                try:
+                    try:
+                        text, latency = hit_compose(client, port, profile_id)
+                    except Exception as e:
+                        print(f"  compose:ERR ({e})", end="")
+                        continue
+                    card = score(
+                        personality=personality,
+                        endpoint="compose",
+                        model=model,
+                        input_text="",
+                        refined=text,
+                        latency_ms=latency,
+                    )
+                    cards.append(card)
+                    status = "ok" if not card.flags else "⚠"
+                    print(f"  compose:{status} ({latency}ms)", end="")
+                    print()
+                finally:
+                    delete_profile(client, port, profile_id)
+
+    print(format_report(cards))
+
+    if args.json:
+        args.json.write_text(json.dumps([asdict(c) for c in cards], indent=2))
+        print(f"wrote {args.json}")
+
+    return 0 if all(not c.flags for c in cards) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backend/tests/test_profile_duplicate_names.py

@@ -0,0 +1,217 @@
+"""
+Tests for profile duplicate name validation.
+
+This test suite verifies that the application correctly handles
+duplicate profile names and provides user-friendly error messages.
+"""
+
+import pytest
+import tempfile
+import shutil
+from pathlib import Path
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+
+# Add parent directory to path to import backend modules
+import sys
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from database import Base, VoiceProfile as DBVoiceProfile
+from models import VoiceProfileCreate
+from profiles import create_profile, update_profile
+
+
+@pytest.fixture
+def test_db():
+    """Create a temporary test database."""
+    # Create temporary directory for test database
+    temp_dir = tempfile.mkdtemp()
+    db_path = Path(temp_dir) / "test.db"
+
+    # Create engine and session
+    engine = create_engine(f"sqlite:///{db_path}")
+    Base.metadata.create_all(bind=engine)
+    SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
+
+    db = SessionLocal()
+
+    yield db
+
+    # Cleanup
+    db.close()
+    shutil.rmtree(temp_dir)
+
+
+@pytest.fixture
+def mock_profiles_dir(monkeypatch, tmp_path):
+    """Mock the profiles directory to use a temporary path."""
+    from backend import config
+    monkeypatch.setattr(config, 'get_profiles_dir', lambda: tmp_path)
+    return tmp_path
+
+
+@pytest.mark.asyncio
+async def test_create_profile_duplicate_name_raises_error(test_db, mock_profiles_dir):
+    """Test that creating a profile with a duplicate name raises a ValueError."""
+    # Create first profile
+    profile_data_1 = VoiceProfileCreate(
+        name="Test Profile",
+        description="First profile",
+        language="en"
+    )
+
+    profile_1 = await create_profile(profile_data_1, test_db)
+    assert profile_1.name == "Test Profile"
+
+    # Try to create second profile with same name
+    profile_data_2 = VoiceProfileCreate(
+        name="Test Profile",
+        description="Second profile",
+        language="en"
+    )
+
+    with pytest.raises(ValueError) as exc_info:
+        await create_profile(profile_data_2, test_db)
+
+    # Verify error message is user-friendly
+    assert "already exists" in str(exc_info.value)
+    assert "Test Profile" in str(exc_info.value)
+    assert "choose a different name" in str(exc_info.value).lower()
+
+
+@pytest.mark.asyncio
+async def test_create_profile_different_names_succeeds(test_db, mock_profiles_dir):
+    """Test that creating profiles with different names succeeds."""
+    # Create first profile
+    profile_data_1 = VoiceProfileCreate(
+        name="Profile One",
+        description="First profile",
+        language="en"
+    )
+
+    profile_1 = await create_profile(profile_data_1, test_db)
+    assert profile_1.name == "Profile One"
+
+    # Create second profile with different name
+    profile_data_2 = VoiceProfileCreate(
+        name="Profile Two",
+        description="Second profile",
+        language="en"
+    )
+
+    profile_2 = await create_profile(profile_data_2, test_db)
+    assert profile_2.name == "Profile Two"
+
+    # Verify both profiles exist
+    assert profile_1.id != profile_2.id
+
+
+@pytest.mark.asyncio
+async def test_update_profile_to_duplicate_name_raises_error(test_db, mock_profiles_dir):
+    """Test that updating a profile to a duplicate name raises a ValueError."""
+    # Create two profiles with different names
+    profile_data_1 = VoiceProfileCreate(
+        name="Profile A",
+        description="First profile",
+        language="en"
+    )
+    profile_1 = await create_profile(profile_data_1, test_db)
+
+    profile_data_2 = VoiceProfileCreate(
+        name="Profile B",
+        description="Second profile",
+        language="en"
+    )
+    profile_2 = await create_profile(profile_data_2, test_db)
+
+    # Try to update profile_2 to use profile_1's name
+    update_data = VoiceProfileCreate(
+        name="Profile A",  # Duplicate name
+        description="Updated description",
+        language="en"
+    )
+
+    with pytest.raises(ValueError) as exc_info:
+        await update_profile(profile_2.id, update_data, test_db)
+
+    # Verify error message is user-friendly
+    assert "already exists" in str(exc_info.value)
+    assert "Profile A" in str(exc_info.value)
+
+
+@pytest.mark.asyncio
+async def test_update_profile_keep_same_name_succeeds(test_db, mock_profiles_dir):
+    """Test that updating a profile while keeping the same name succeeds."""
+    # Create profile
+    profile_data = VoiceProfileCreate(
+        name="My Profile",
+        description="Original description",
+        language="en"
+    )
+    profile = await create_profile(profile_data, test_db)
+
+    # Update profile with same name but different description
+    update_data = VoiceProfileCreate(
+        name="My Profile",  # Same name
+        description="Updated description",
+        language="en"
+    )
+
+    updated_profile = await update_profile(profile.id, update_data, test_db)
+
+    # Verify update succeeded
+    assert updated_profile is not None
+    assert updated_profile.id == profile.id
+    assert updated_profile.name == "My Profile"
+    assert updated_profile.description == "Updated description"
+
+
+@pytest.mark.asyncio
+async def test_update_profile_to_new_unique_name_succeeds(test_db, mock_profiles_dir):
+    """Test that updating a profile to a new unique name succeeds."""
+    # Create profile
+    profile_data = VoiceProfileCreate(
+        name="Original Name",
+        description="Profile description",
+        language="en"
+    )
+    profile = await create_profile(profile_data, test_db)
+
+    # Update profile with new unique name
+    update_data = VoiceProfileCreate(
+        name="New Unique Name",
+        description="Updated description",
+        language="en"
+    )
+
+    updated_profile = await update_profile(profile.id, update_data, test_db)
+
+    # Verify update succeeded
+    assert updated_profile is not None
+    assert updated_profile.id == profile.id
+    assert updated_profile.name == "New Unique Name"
+
+
+@pytest.mark.asyncio
+async def test_case_sensitive_names_allowed(test_db, mock_profiles_dir):
+    """Test that profile names are case-sensitive (e.g., 'Test' and 'test' are different)."""
+    # Create profile with lowercase name
+    profile_data_1 = VoiceProfileCreate(
+        name="test profile",
+        description="Lowercase",
+        language="en"
+    )
+    profile_1 = await create_profile(profile_data_1, test_db)
+
+    # Create profile with different case
+    profile_data_2 = VoiceProfileCreate(
+        name="Test Profile",
+        description="Title case",
+        language="en"
+    )
+    profile_2 = await create_profile(profile_data_2, test_db)
+
+    # Both should succeed since SQLite unique constraint is case-sensitive by default
+    assert profile_1.name == "test profile"
+    assert profile_2.name == "Test Profile"
+    assert profile_1.id != profile_2.id

backend/tests/test_progress.py

@@ -0,0 +1,313 @@
+"""
+Test script to debug model download progress tracking.
+"""
+
+import asyncio
+import json
+import time
+from typing import List, Dict
+import logging
+
+# Set up logging to see what's happening
+logging.basicConfig(
+    level=logging.DEBUG,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+
+from utils.progress import ProgressManager, get_progress_manager
+from utils.hf_progress import HFProgressTracker, create_hf_progress_callback
+
+
+def test_progress_manager_basic():
+    """Test 1: Basic ProgressManager functionality."""
+    print("\n" + "=" * 60)
+    print("Test 1: ProgressManager Basic Operations")
+    print("=" * 60)
+
+    pm = ProgressManager()
+
+    # Test update_progress
+    pm.update_progress(
+        model_name="test-model",
+        current=50,
+        total=100,
+        filename="test.bin",
+        status="downloading"
+    )
+
+    # Test get_progress
+    progress = pm.get_progress("test-model")
+    print(f"✓ Progress stored: {progress}")
+    assert progress is not None
+    assert progress["progress"] == 50.0
+    assert progress["filename"] == "test.bin"
+    assert progress["status"] == "downloading"
+
+    # Test mark_complete
+    pm.mark_complete("test-model")
+    progress = pm.get_progress("test-model")
+    print(f"✓ Marked complete: {progress}")
+    assert progress["status"] == "complete"
+    assert progress["progress"] == 100.0
+
+    print("✓ Test 1 PASSED\n")
+    return True
+
+
+async def test_progress_manager_sse():
+    """Test 2: ProgressManager SSE streaming."""
+    print("\n" + "=" * 60)
+    print("Test 2: ProgressManager SSE Streaming")
+    print("=" * 60)
+
+    pm = ProgressManager()
+    collected_events: List[Dict] = []
+
+    # Simulate SSE client
+    async def sse_client():
+        """Simulates a frontend SSE connection."""
+        print("  SSE client: Subscribing to test-model-sse...")
+        async for event in pm.subscribe("test-model-sse"):
+            # Parse SSE event
+            if event.startswith("data: "):
+                data = json.loads(event[6:])
+                print(f"  SSE client: Received event: {data['status']} - {data.get('progress', 0):.1f}%")
+                collected_events.append(data)
+
+                # Stop when complete
+                if data.get("status") in ("complete", "error"):
+                    break
+            elif event.startswith(": heartbeat"):
+                print("  SSE client: Received heartbeat")
+
+    # Simulate download progress updates (from backend thread)
+    async def simulate_download():
+        """Simulates backend sending progress updates."""
+        print("  Backend: Starting simulated download...")
+        await asyncio.sleep(0.2)  # Let SSE client subscribe first
+
+        # Send progress updates
+        for i in range(0, 101, 20):
+            print(f"  Backend: Updating progress to {i}%")
+            pm.update_progress(
+                model_name="test-model-sse",
+                current=i,
+                total=100,
+                filename=f"file_{i}.bin",
+                status="downloading" if i < 100 else "downloading"
+            )
+            await asyncio.sleep(0.1)
+
+        # Mark complete
+        print("  Backend: Marking download complete")
+        pm.mark_complete("test-model-sse")
+
+    # Run SSE client and download simulation concurrently
+    await asyncio.gather(
+        sse_client(),
+        simulate_download()
+    )
+
+    # Verify we got events
+    print(f"\n  Collected {len(collected_events)} events")
+    assert len(collected_events) > 0, "Should have received at least one event"
+    assert collected_events[-1]["status"] == "complete", "Last event should be 'complete'"
+
+    print("✓ Test 2 PASSED\n")
+    return True
+
+
+def test_hf_progress_tracker():
+    """Test 3: HFProgressTracker tqdm patching."""
+    print("\n" + "=" * 60)
+    print("Test 3: HFProgressTracker tqdm Patching")
+    print("=" * 60)
+
+    captured_progress: List[tuple] = []
+
+    def progress_callback(downloaded: int, total: int, filename: str):
+        """Capture progress updates."""
+        captured_progress.append((downloaded, total, filename))
+        print(f"  Progress callback: {downloaded}/{total} bytes ({filename})")
+
+    tracker = HFProgressTracker(progress_callback)
+
+    # Simulate a download with tqdm
+    with tracker.patch_download():
+        try:
+            from tqdm import tqdm
+
+            # Simulate downloading a file
+            print("  Simulating download with tqdm...")
+            total_size = 1000
+            with tqdm(total=total_size, desc="model.bin", unit="B", unit_scale=True) as pbar:
+                for chunk in range(0, total_size, 100):
+                    pbar.update(100)
+                    time.sleep(0.01)
+
+            print(f"  Captured {len(captured_progress)} progress updates")
+            assert len(captured_progress) > 0, "Should have captured progress updates"
+
+            # Verify progress increases
+            last_downloaded = 0
+            for downloaded, total, filename in captured_progress:
+                assert downloaded >= last_downloaded, "Downloaded bytes should increase"
+                assert total == total_size, "Total should be consistent"
+                last_downloaded = downloaded
+
+            print("✓ Test 3 PASSED\n")
+            return True
+
+        except ImportError:
+            print("✗ tqdm not available, skipping test\n")
+            return None
+
+
+async def test_full_integration():
+    """Test 4: Full integration test."""
+    print("\n" + "=" * 60)
+    print("Test 4: Full Integration (ProgressManager + HFProgressTracker)")
+    print("=" * 60)
+
+    pm = get_progress_manager()
+    collected_events: List[Dict] = []
+
+    # SSE client
+    async def sse_client():
+        print("  SSE client: Subscribing...")
+        async for event in pm.subscribe("integration-test"):
+            if event.startswith("data: "):
+                data = json.loads(event[6:])
+                print(f"  SSE client: {data['status']} - {data.get('progress', 0):.1f}% - {data.get('filename', '')}")
+                collected_events.append(data)
+                if data.get("status") in ("complete", "error"):
+                    break
+
+    # Simulate backend download with HFProgressTracker
+    async def simulate_real_download():
+        await asyncio.sleep(0.2)  # Let SSE subscribe
+
+        print("  Backend: Starting download with HFProgressTracker...")
+
+        # Set up tracking (like the real backend does)
+        progress_callback = create_hf_progress_callback("integration-test", pm)
+        tracker = HFProgressTracker(progress_callback)
+
+        # Initialize progress
+        pm.update_progress(
+            model_name="integration-test",
+            current=0,
+            total=1,
+            filename="",
+            status="downloading"
+        )
+
+        # Simulate download with tqdm patching
+        with tracker.patch_download():
+            try:
+                from tqdm import tqdm
+
+                # Simulate multi-file download (like HuggingFace does)
+                files = [
+                    ("model.safetensors", 5000),
+                    ("config.json", 1000),
+                    ("tokenizer.json", 500),
+                ]
+
+                for filename, size in files:
+                    print(f"  Backend: Downloading {filename}...")
+                    with tqdm(total=size, desc=filename, unit="B") as pbar:
+                        for chunk in range(0, size, 500):
+                            chunk_size = min(500, size - chunk)
+                            pbar.update(chunk_size)
+                            await asyncio.sleep(0.05)
+
+                # Mark complete
+                print("  Backend: Download complete")
+                pm.mark_complete("integration-test")
+
+            except ImportError:
+                print("  ✗ tqdm not available")
+                pm.mark_error("integration-test", "tqdm not available")
+
+    # Run both
+    await asyncio.gather(
+        sse_client(),
+        simulate_real_download()
+    )
+
+    # Verify
+    print(f"\n  Collected {len(collected_events)} events")
+    if len(collected_events) > 0:
+        print(f"  First event: {collected_events[0]}")
+        print(f"  Last event: {collected_events[-1]}")
+        assert collected_events[-1]["status"] == "complete", "Should end with 'complete'"
+        print("✓ Test 4 PASSED\n")
+        return True
+    else:
+        print("✗ Test 4 FAILED - No events received\n")
+        return False
+
+
+async def main():
+    """Run all tests."""
+    print("\n" + "=" * 60)
+    print("Voicebox Progress Tracking Test Suite")
+    print("=" * 60)
+
+    results = []
+
+    # Test 1: Basic operations
+    try:
+        results.append(("Basic Operations", test_progress_manager_basic()))
+    except Exception as e:
+        print(f"✗ Test 1 FAILED: {e}\n")
+        results.append(("Basic Operations", False))
+
+    # Test 2: SSE streaming
+    try:
+        results.append(("SSE Streaming", await test_progress_manager_sse()))
+    except Exception as e:
+        print(f"✗ Test 2 FAILED: {e}\n")
+        results.append(("SSE Streaming", False))
+
+    # Test 3: tqdm patching
+    try:
+        results.append(("tqdm Patching", test_hf_progress_tracker()))
+    except Exception as e:
+        print(f"✗ Test 3 FAILED: {e}\n")
+        results.append(("tqdm Patching", False))
+
+    # Test 4: Full integration
+    try:
+        results.append(("Full Integration", await test_full_integration()))
+    except Exception as e:
+        print(f"✗ Test 4 FAILED: {e}\n")
+        results.append(("Full Integration", False))
+
+    # Summary
+    print("\n" + "=" * 60)
+    print("Test Results Summary")
+    print("=" * 60)
+
+    for name, result in results:
+        status = "✓ PASS" if result else ("⊘ SKIP" if result is None else "✗ FAIL")
+        print(f"  {status:8} {name}")
+
+    passed = sum(1 for _, r in results if r is True)
+    failed = sum(1 for _, r in results if r is False)
+    skipped = sum(1 for _, r in results if r is None)
+
+    print()
+    print(f"  Total: {len(results)} tests")
+    print(f"  Passed: {passed}")
+    print(f"  Failed: {failed}")
+    print(f"  Skipped: {skipped}")
+    print("=" * 60 + "\n")
+
+    return failed == 0
+
+
+if __name__ == "__main__":
+    success = asyncio.run(main())
+    exit(0 if success else 1)

backend/tests/test_qwen_download.py

@@ -0,0 +1,317 @@
+"""
+Test Qwen TTS model download with SSE progress monitoring.
+
+This specifically tests the MLX TTS backend download progress tracking,
+which requires tqdm to be patched BEFORE mlx_audio is imported.
+
+Usage:
+    cd backend && python -m tests.test_qwen_download
+
+Prerequisites:
+    - Server must be running: cd backend && python main.py
+    - Delete model first for fresh download test:
+      curl -X DELETE http://localhost:8000/models/qwen-tts-0.6B
+"""
+
+import asyncio
+import json
+import httpx
+import time
+from typing import List, Dict, Optional
+
+
+async def monitor_sse_stream(model_name: str, timeout: int = 600) -> List[Dict]:
+    """
+    Monitor SSE stream for a model download.
+    
+    Args:
+        model_name: Name of the model to monitor
+        timeout: Maximum time to wait for download (seconds)
+        
+    Returns:
+        List of SSE events received
+    """
+    events: List[Dict] = []
+    url = f"http://localhost:8000/models/progress/{model_name}"
+    last_progress = -1
+    
+    print(f"\n📡 Connecting to SSE endpoint: {url}")
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout) as client:
+            async with client.stream("GET", url) as response:
+                print(f"   SSE connected, status: {response.status_code}")
+
+                if response.status_code != 200:
+                    print(f"   ❌ Error: SSE endpoint returned {response.status_code}")
+                    return events
+
+                async for line in response.aiter_lines():
+                    if not line:
+                        continue
+
+                    if line.startswith("data: "):
+                        try:
+                            data = json.loads(line[6:])
+                            events.append(data)
+                            
+                            # Print progress (only when it changes significantly)
+                            progress = data.get('progress', 0)
+                            status = data.get('status', 'unknown')
+                            filename = data.get('filename', '')
+                            current = data.get('current', 0)
+                            total = data.get('total', 0)
+                            
+                            # Print every 5% change or status change
+                            if abs(progress - last_progress) >= 5 or status in ('complete', 'error'):
+                                current_mb = current / (1024 * 1024)
+                                total_mb = total / (1024 * 1024)
+                                print(f"   📊 {status:12} {progress:6.1f}% ({current_mb:.1f}MB / {total_mb:.1f}MB) {filename[:50]}")
+                                last_progress = progress
+
+                            # Stop if complete or error
+                            if status in ("complete", "error"):
+                                if status == "complete":
+                                    print(f"   ✅ Download complete!")
+                                else:
+                                    print(f"   ❌ Download error: {data.get('error', 'unknown')}")
+                                break
+
+                        except json.JSONDecodeError as e:
+                            print(f"   ⚠️  Error parsing JSON: {e}")
+
+                    elif line.startswith(": heartbeat"):
+                        # Heartbeat every 1 second, don't spam
+                        pass
+
+    except asyncio.CancelledError:
+        print("   ⏹️  SSE monitor cancelled")
+    except Exception as e:
+        print(f"   ❌ SSE error: {e}")
+
+    return events
+
+
+async def trigger_download(model_name: str) -> bool:
+    """Trigger a model download via the API."""
+    url = "http://localhost:8000/models/download"
+
+    print(f"\n🚀 Triggering download for: {model_name}")
+
+    try:
+        async with httpx.AsyncClient(timeout=30) as client:
+            response = await client.post(url, json={"model_name": model_name})
+            result = response.json()
+            print(f"   Response: {response.status_code} - {result}")
+            return response.status_code == 200
+    except Exception as e:
+        print(f"   ❌ Error triggering download: {e}")
+        return False
+
+
+async def delete_model(model_name: str) -> bool:
+    """Delete a model from cache."""
+    url = f"http://localhost:8000/models/{model_name}"
+
+    print(f"\n🗑️  Deleting model: {model_name}")
+
+    try:
+        async with httpx.AsyncClient(timeout=30) as client:
+            response = await client.delete(url)
+            if response.status_code == 200:
+                print(f"   ✅ Model deleted")
+                return True
+            elif response.status_code == 404:
+                print(f"   ℹ️  Model not found (already deleted)")
+                return True
+            else:
+                print(f"   ⚠️  Delete response: {response.status_code} - {response.text}")
+                return False
+    except Exception as e:
+        print(f"   ❌ Error deleting model: {e}")
+        return False
+
+
+async def check_model_status(model_name: str) -> Optional[Dict]:
+    """Check the status of a model."""
+    try:
+        async with httpx.AsyncClient(timeout=10) as client:
+            response = await client.get("http://localhost:8000/models/status")
+            if response.status_code == 200:
+                data = response.json()
+                for model in data.get("models", []):
+                    if model["model_name"] == model_name:
+                        return model
+    except Exception as e:
+        print(f"   ⚠️  Error checking model status: {e}")
+    return None
+
+
+async def check_server() -> bool:
+    """Check if the server is running."""
+    try:
+        async with httpx.AsyncClient(timeout=5) as client:
+            response = await client.get("http://localhost:8000/health")
+            return response.status_code == 200
+    except Exception:
+        return False
+
+
+async def main():
+    print("=" * 70)
+    print("🧪 Qwen TTS Model Download Progress Test")
+    print("=" * 70)
+    print("\nThis test verifies that MLX TTS download progress tracking works.")
+    print("It specifically tests the tqdm patching for mlx_audio.tts imports.")
+
+    # Check if server is running
+    print("\n📡 Checking if server is running...")
+    if not await check_server():
+        print("   ❌ Server is not running on http://localhost:8000")
+        print("\n   Please start the server first:")
+        print("   cd backend && python main.py")
+        return False
+
+    print("   ✅ Server is running")
+
+    # Test model
+    model_name = "qwen-tts-0.6B"
+    
+    # Check current status
+    print(f"\n📊 Checking status of {model_name}...")
+    status = await check_model_status(model_name)
+    if status:
+        print(f"   Downloaded: {status.get('downloaded', False)}")
+        print(f"   Downloading: {status.get('downloading', False)}")
+        print(f"   Loaded: {status.get('loaded', False)}")
+        if status.get('size_mb'):
+            print(f"   Size: {status['size_mb']:.1f} MB")
+    else:
+        print("   ⚠️  Could not get model status")
+
+    # Ask if user wants to delete first
+    print("\n" + "-" * 70)
+    if status and status.get('downloaded'):
+        print("⚠️  Model is already downloaded. Delete it for a fresh download test?")
+        print("   [y] Yes, delete and download fresh")
+        print("   [n] No, just test SSE connection")
+        print("   [q] Quit")
+        
+        choice = input("\nChoice [y/n/q]: ").strip().lower()
+        
+        if choice == 'q':
+            print("Exiting...")
+            return True
+        
+        if choice == 'y':
+            if not await delete_model(model_name):
+                print("Failed to delete model. Continue anyway? [y/n]")
+                if input().strip().lower() != 'y':
+                    return False
+    else:
+        print("Model not downloaded. Will perform fresh download test.")
+        input("Press Enter to continue...")
+
+    # Run the test
+    print("\n" + "=" * 70)
+    print("🏃 Starting Download Test")
+    print("=" * 70)
+
+    async def run_test():
+        # Start SSE monitor in background FIRST
+        monitor_task = asyncio.create_task(monitor_sse_stream(model_name, timeout=600))
+
+        # Wait for SSE to connect
+        await asyncio.sleep(1)
+
+        # Trigger download
+        success = await trigger_download(model_name)
+
+        if not success:
+            print("   ❌ Failed to trigger download")
+            monitor_task.cancel()
+            try:
+                await monitor_task
+            except asyncio.CancelledError:
+                pass
+            return []
+
+        # Wait for SSE monitor to complete
+        print("\n⏳ Waiting for download to complete (this may take several minutes)...")
+        events = await monitor_task
+
+        return events
+
+    start_time = time.time()
+    events = await run_test()
+    elapsed = time.time() - start_time
+
+    # Results
+    print("\n" + "=" * 70)
+    print("📋 Test Results")
+    print("=" * 70)
+
+    print(f"\n⏱️  Elapsed time: {elapsed:.1f} seconds")
+    print(f"📨 Total SSE events received: {len(events)}")
+
+    if not events:
+        print("\n❌ FAILED - No SSE events received!")
+        print("\nPossible causes:")
+        print("  1. SSE endpoint not working")
+        print("  2. tqdm not patched before mlx_audio import")
+        print("  3. Progress callbacks not firing")
+        print("  4. Model already fully downloaded")
+        print("\nDebug steps:")
+        print("  1. Check server logs for [DEBUG] messages")
+        print("  2. Look for 'tqdm patched' before 'mlx_audio.tts import'")
+        print(f"  3. Delete model: curl -X DELETE http://localhost:8000/models/{model_name}")
+        return False
+
+    # Analyze events
+    first_event = events[0]
+    last_event = events[-1]
+    
+    print(f"\n📊 First event:")
+    print(f"   Status: {first_event.get('status')}")
+    print(f"   Progress: {first_event.get('progress', 0):.1f}%")
+    
+    print(f"\n📊 Last event:")
+    print(f"   Status: {last_event.get('status')}")
+    print(f"   Progress: {last_event.get('progress', 0):.1f}%")
+
+    # Check for expected behaviors
+    has_progress_updates = len(events) > 2
+    has_increasing_progress = False
+    has_complete = any(e.get('status') == 'complete' for e in events)
+    has_100_percent = any(e.get('progress', 0) >= 100 for e in events)
+    
+    # Check if progress increased over time
+    if len(events) >= 2:
+        progress_values = [e.get('progress', 0) for e in events]
+        has_increasing_progress = progress_values[-1] > progress_values[0]
+
+    print("\n📋 Checks:")
+    print(f"   {'✅' if has_progress_updates else '❌'} Multiple progress updates received ({len(events)} events)")
+    print(f"   {'✅' if has_increasing_progress else '❌'} Progress increased over time")
+    print(f"   {'✅' if has_100_percent else '❌'} Reached 100% progress")
+    print(f"   {'✅' if has_complete else '❌'} Received 'complete' status")
+
+    # Overall result
+    success = has_progress_updates and has_complete
+    
+    if success:
+        print("\n" + "=" * 70)
+        print("✅ TEST PASSED - Qwen TTS download progress tracking works!")
+        print("=" * 70)
+    else:
+        print("\n" + "=" * 70)
+        print("❌ TEST FAILED - Progress tracking has issues")
+        print("=" * 70)
+        print("\nCheck the server logs for debug output.")
+
+    return success
+
+
+if __name__ == "__main__":
+    result = asyncio.run(main())
+    exit(0 if result else 1)

backend/tests/test_qwen_runaway_retry.py

@@ -0,0 +1,117 @@
+"""Regression coverage for runaway MLX Qwen TTS output."""
+
+from unittest.mock import patch
+
+import numpy as np
+import pytest
+
+from backend.backends import engine_needs_trim, engine_retries_runaway
+from backend.utils.audio import has_tts_runaway
+from backend.utils.chunked_tts import generate_chunked
+
+SAMPLE_RATE = 1000
+
+
+def test_mlx_qwen_enables_runaway_retry_without_aggressive_trim():
+    with patch("backend.backends.get_backend_type", return_value="mlx"):
+        assert engine_needs_trim("qwen") is False
+        assert engine_retries_runaway("qwen") is True
+
+
+def test_pytorch_qwen_keeps_runaway_retry_disabled():
+    with patch("backend.backends.get_backend_type", return_value="pytorch"):
+        assert engine_needs_trim("qwen") is False
+        assert engine_retries_runaway("qwen") is False
+
+
+def test_detector_flags_long_internal_silence():
+    speech = np.full(2 * SAMPLE_RATE, 0.2, dtype=np.float32)
+    runaway_gap = np.zeros(2500, dtype=np.float32)
+    hallucinated_noise = np.full(2 * SAMPLE_RATE, 0.8, dtype=np.float32)
+    audio = np.concatenate([speech, runaway_gap, hallucinated_noise])
+
+    assert has_tts_runaway(audio, SAMPLE_RATE) is True
+
+
+def test_detector_ignores_normal_internal_pause():
+    speech = np.full(SAMPLE_RATE, 0.2, dtype=np.float32)
+    normal_pause = np.zeros(1200, dtype=np.float32)
+    audio = np.concatenate([speech, normal_pause, speech])
+
+    assert has_tts_runaway(audio, SAMPLE_RATE) is False
+
+
+def test_trailing_silence_is_not_a_runaway():
+    speech = np.full(SAMPLE_RATE, 0.2, dtype=np.float32)
+    trailing_silence = np.zeros(2 * SAMPLE_RATE, dtype=np.float32)
+
+    assert (
+        has_tts_runaway(
+            np.concatenate([speech, trailing_silence]),
+            SAMPLE_RATE,
+        )
+        is False
+    )
+
+
+@pytest.mark.asyncio
+async def test_runaway_chunk_is_retried_as_smaller_chunks():
+    class FakeBackend:
+        def __init__(self):
+            self.calls = []
+
+        async def generate(self, text, *_args):
+            self.calls.append(text)
+            if len(text) > 200:
+                speech = np.full(SAMPLE_RATE, 0.2, dtype=np.float32)
+                silence = np.zeros(2500, dtype=np.float32)
+                noise = np.full(SAMPLE_RATE, 0.8, dtype=np.float32)
+                return np.concatenate([speech, silence, noise]), SAMPLE_RATE
+            return np.full(SAMPLE_RATE, 0.2, dtype=np.float32), SAMPLE_RATE
+
+    backend = FakeBackend()
+    text = f"{'A' * 119}. {'B' * 119}."
+
+    audio, sample_rate = await generate_chunked(
+        backend,
+        text,
+        {},
+        max_chunk_chars=800,
+        crossfade_ms=50,
+        runaway_detector=has_tts_runaway,
+    )
+
+    assert sample_rate == SAMPLE_RATE
+    assert backend.calls == [text, f"{'A' * 119}.", f"{'B' * 119}."]
+    assert len(audio) == 1950
+
+
+@pytest.mark.asyncio
+async def test_persistent_runaway_fails_instead_of_returning_corrupt_audio():
+    class AlwaysRunawayBackend:
+        def __init__(self):
+            self.calls = []
+
+        async def generate(self, text, *_args):
+            self.calls.append(text)
+            speech = np.full(SAMPLE_RATE, 0.2, dtype=np.float32)
+            silence = np.zeros(2500, dtype=np.float32)
+            noise = np.full(SAMPLE_RATE, 0.8, dtype=np.float32)
+            return np.concatenate([speech, silence, noise]), SAMPLE_RATE
+
+    backend = AlwaysRunawayBackend()
+    text = f"{'A' * 119}. {'B' * 119}."
+
+    with pytest.raises(
+        RuntimeError,
+        match="remained unstable after retrying smaller text chunks",
+    ):
+        await generate_chunked(
+            backend,
+            text,
+            {},
+            max_chunk_chars=800,
+            runaway_detector=has_tts_runaway,
+        )
+
+    assert [len(call) for call in backend.calls] == [241, 120, 100]

backend/tests/test_refinement_collapse.py

@@ -0,0 +1,145 @@
+"""Unit tests for ``collapse_repetitive_artifacts``.
+
+The eval harness (``test_refinement_samples.py``) is interactive and
+LLM-dependent; these are the fast, deterministic tests for the
+deterministic pre-processor that runs before the LLM ever sees a
+transcript. They pin the behaviour for both the single-word loops the
+original algorithm handled and the multi-word / CJK / emoji loops the
+character-level pass added.
+"""
+
+from backend.services.refinement import collapse_repetitive_artifacts
+
+
+# ── single-word loops (word-level pass) ─────────────────────────────────
+
+
+def test_single_word_loop_stripped():
+    raw = "Hello " + ("URL " * 8).strip() + " goodbye"
+    assert collapse_repetitive_artifacts(raw) == "Hello goodbye"
+
+
+def test_single_word_loop_with_punctuation_normalized():
+    # URL, URL, URL, URL, URL, URL. — six repeats if you normalize
+    # trailing punctuation; word-level pass strips them all.
+    raw = "Hello URL, URL, URL, URL, URL, URL. goodbye"
+    assert collapse_repetitive_artifacts(raw) == "Hello goodbye"
+
+
+def test_single_word_loop_case_insensitive():
+    raw = "hi " + " ".join(["Url", "URL", "url", "Url", "URL", "url"]) + " bye"
+    assert collapse_repetitive_artifacts(raw) == "hi bye"
+
+
+def test_short_single_word_run_preserved():
+    # Five repeats — below threshold.
+    raw = "no no no no no"
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+def test_rhetorical_repetition_preserved():
+    raw = "I said no, no, no, no, no and she left"
+    # Five repeats of "no" — below threshold.
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+# ── multi-word loops (character-level pass) ─────────────────────────────
+
+
+def test_multi_word_english_loop_stripped():
+    # Classic Whisper tail hallucination. Word-level pass sees no
+    # consecutive identical tokens, so it's the character-level pass's
+    # job to catch this.
+    loop = "thanks for watching " * 6
+    raw = f"Okay so the meeting is at three. {loop}"
+    result = collapse_repetitive_artifacts(raw)
+    assert "thanks for watching" not in result
+    assert "Okay so the meeting is at three" in result
+
+
+def test_three_word_loop_stripped():
+    loop = "please like and " * 7
+    raw = f"The point is clear. {loop}right"
+    result = collapse_repetitive_artifacts(raw)
+    assert "please like and" not in result
+    assert "The point is clear" in result
+
+
+def test_long_phrase_loop_within_60_char_cap():
+    unit = "Please like and subscribe to my channel. "  # 41 chars, within cap
+    raw = "End of video. " + unit * 6
+    result = collapse_repetitive_artifacts(raw)
+    assert unit.strip() not in result
+    assert "End of video" in result
+
+
+def test_multi_word_short_run_preserved():
+    # Five repeats of a multi-word unit — below threshold.
+    raw = "thanks for watching thanks for watching thanks for watching thanks for watching thanks for watching"
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+# ── CJK loops (character-level pass, no whitespace) ──────────────────────
+
+
+def test_cjk_loop_stripped():
+    # Common Chinese Whisper hallucination: "thanks for watching".
+    # text.split() yields one token for the whole loop; only the
+    # character-level pass can catch this.
+    prefix = "會議在三點開始"
+    loop = "謝謝觀看" * 7
+    raw = prefix + loop
+    result = collapse_repetitive_artifacts(raw)
+    assert "謝謝觀看" not in result
+    assert prefix in result
+
+
+def test_japanese_loop_stripped():
+    # Same pattern, kana/kanji mix. "ご視聴ありがとうございました" is a
+    # frequent Japanese Whisper tail hallucination.
+    loop = "ご視聴ありがとうございました" * 6
+    raw = f"明日の会議は午後三時です。{loop}"
+    result = collapse_repetitive_artifacts(raw)
+    assert "ご視聴ありがとうございました" not in result
+    assert "明日の会議は午後三時です" in result
+
+
+def test_cjk_short_run_preserved():
+    # Five repeats — below threshold, stays in.
+    raw = "好好好好好"
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+# ── whitespace / edge cases ──────────────────────────────────────────────
+
+
+def test_empty_string_passes_through():
+    assert collapse_repetitive_artifacts("") == ""
+
+
+def test_below_word_threshold_passes_through_unmodified():
+    raw = "just three words"
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+def test_emphasis_vowel_run_preserved():
+    # "wooooooow" is 1 char (plus 8 o's). Character-level min unit is 2,
+    # so "oo…" doesn't get stripped and this legitimate emphasis stays.
+    raw = "that's wooooooow amazing"
+    assert collapse_repetitive_artifacts(raw) == raw
+
+
+def test_custom_threshold_honored():
+    # With min_run=3, even short rhetorical repetition should now strip.
+    raw = "ha ha ha ha context"
+    result = collapse_repetitive_artifacts(raw, min_run=3)
+    assert "ha ha" not in result
+    assert "context" in result
+
+
+def test_leading_and_trailing_whitespace_stripped_after_collapse():
+    # When character pass fires, the normalised result is stripped so
+    # downstream prompts don't carry edge whitespace from the removal.
+    loop = "loop-phrase " * 7
+    raw = loop
+    assert collapse_repetitive_artifacts(raw) == ""

backend/tests/test_refinement_samples.py

@@ -0,0 +1,453 @@
+"""
+Refinement sanity sweep — runs ten realistic raw transcripts through
+``/llm/generate`` (with the full refinement system prompt) and scores
+each output against a handful of deterministic heuristics so a person
+can eyeball quality at a glance.
+
+This is an interactive evaluation harness, not a pass/fail unit test:
+LLM output is non-deterministic and "correctness" for cleanup is
+subjective. The heuristics catch gross failures (prompt leaks,
+Whisper-loop echoes, the model answering a question instead of
+rewriting it) but a human still has to read the final column.
+
+Usage:
+    # Backend server must be running.
+    python backend/tests/test_refinement_samples.py
+
+    # Hit a non-default port (auto-detected via /health probe when omitted):
+    python backend/tests/test_refinement_samples.py --port 17493
+
+    # Only test one model size:
+    python backend/tests/test_refinement_samples.py --model 4B
+
+    # Dump JSON for diffing against a prior run:
+    python backend/tests/test_refinement_samples.py --json results.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import socket
+import sys
+import time
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+from collections.abc import Iterable
+from typing import Optional
+
+import httpx
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+# Point sys.path at the repo root so ``backend.services.refinement`` resolves
+# as a package. Using backend/ as root breaks the service's own
+# ``from ..backends import …`` relative imports.
+sys.path.insert(0, str(REPO_ROOT))
+
+from backend.services.refinement import (  # noqa: E402
+    build_refinement_prompt,
+    collapse_repetitive_artifacts,
+    REFINEMENT_EXAMPLES,
+    RefinementFlags,
+)
+
+
+# ── Sample inputs ─────────────────────────────────────────────────────
+
+
+@dataclass(frozen=True)
+class Sample:
+    name: str
+    """Short label for the results table."""
+    raw: str
+    """The transcript going into refinement."""
+    category: str
+    """Which prompt behaviour this sample probes."""
+    keep_question_mark: bool = False
+    """Raw ends with '?' and the refined output must too. Guards against
+    the model answering instead of rewriting."""
+    must_contain_substrings: tuple[str, ...] = ()
+    """Tokens that must survive refinement — usually technical terms or
+    names we do NOT want the model to rewrite."""
+    must_not_loop: bool = False
+    """Raw contains an STT-hallucination loop; the pre-processor should
+    strip it before the LLM ever sees it."""
+
+
+SAMPLES: tuple[Sample, ...] = (
+    Sample(
+        name="heavy-fillers",
+        category="smart-cleanup",
+        raw=(
+            "so um yeah like i was thinking that uh maybe we could you know "
+            "try that new restaurant tonight if you're like free"
+        ),
+    ),
+    Sample(
+        name="question-stays-question",
+        category="prompt-hard-rule",
+        keep_question_mark=True,
+        raw=(
+            "what is the best way to um learn rust programming do you think"
+        ),
+    ),
+    Sample(
+        name="self-correction",
+        category="self-correction",
+        raw=(
+            "the meeting is at three pm no wait actually four pm on tuesday"
+        ),
+        # Must keep the *final* time (four pm), not the retracted one. The
+        # prompt says "drop the retracted portion AND the correction cue";
+        # the correct rewrite is "The meeting is at four pm on Tuesday."
+        must_contain_substrings=("four pm", "Tuesday"),
+    ),
+    Sample(
+        name="technical-terms",
+        category="preserve-technical",
+        raw=(
+            "run npm install then cd into src slash components and then "
+            "edit index dot tsx"
+        ),
+        must_contain_substrings=("npm install", "src/components", "index.tsx"),
+    ),
+    Sample(
+        name="whisper-loop-tail",
+        category="pre-process-artifact",
+        must_not_loop=True,
+        raw=(
+            "i was watching a video about machine learning training loops "
+            "and then the audio cut out " + ("URL " * 60)
+        ),
+    ),
+    Sample(
+        name="numbers-and-units",
+        category="smart-cleanup",
+        raw=(
+            "the repo has uh four hundred k stars and like two thousand "
+            "contributors across the whole thing"
+        ),
+        # No "400" assertion — the prompt says "keep the speaker's word
+        # choices", so "four hundred k" is the correct passthrough. This
+        # sample is here to check filler removal, not number normalization.
+    ),
+    Sample(
+        name="imperative-stays-command",
+        category="prompt-hard-rule",
+        raw=(
+            "tell me a joke about programming"
+        ),
+    ),
+    Sample(
+        name="long-monologue-mixed",
+        category="everything",
+        raw=(
+            "okay so um i've been thinking a lot about the roadmap and like "
+            "honestly i think we should push the auth rewrite to q3 no wait "
+            "actually q2 because the compliance deadline is uh mid-april "
+            "and we can't really afford to miss that and then you know we "
+            "still have the payments work to do but that's more of a "
+            "basically a maintenance track not a big migration"
+        ),
+    ),
+    Sample(
+        name="code-mid-speech",
+        category="preserve-technical",
+        raw=(
+            "create a function called handleSubmit that takes uh an event "
+            "parameter and calls event dot prevent default"
+        ),
+        must_contain_substrings=("handleSubmit", "event.preventDefault"),
+    ),
+    Sample(
+        name="short-terse",
+        category="smart-cleanup",
+        raw=(
+            "hey can you send me that file"
+        ),
+    ),
+)
+
+
+# ── Scoring heuristics ────────────────────────────────────────────────
+
+
+FILLER_PATTERNS = tuple(
+    re.compile(rf"\b{word}\b", re.IGNORECASE)
+    for word in (
+        "um", "uh", "er", "hmm", "ah",
+        "like", "you know", "i mean", "basically", "literally",
+    )
+)
+
+PROMPT_LEAK_PHRASES = tuple(
+    re.compile(pat, re.IGNORECASE)
+    for pat in (
+        r"^here (?:is|'s) the cleaned",
+        r"^the cleaned (?:version|transcript)",
+        r"^cleaned (?:version|transcript):",
+        r"^output:\s*$",
+        r"^sure,?\s+(?:here|i'll|let)",
+        # Don't match bare "Okay, so…" — speakers often start with that.
+        # Only flag openings that only a chatty LLM would produce.
+        r"^okay,?\s+(?:here(?:'s)?|i'?ll|let me|i understand|no problem)",
+        r"^i (?:cannot|can't|will not|refuse)",
+        r"^as an ai",
+    )
+)
+
+# Rough-and-ready "did the model answer instead of rewrite" sniff test —
+# matches openings the model would use if it mistook the input for a
+# prompt to respond to.
+ANSWER_LEAK_PHRASES = tuple(
+    re.compile(pat, re.IGNORECASE)
+    for pat in (
+        r"^(?:why did|here's a|the answer is|there once was)",
+        r"^(?:a joke|one joke|programming joke)",
+    )
+)
+
+
+@dataclass
+class Scorecard:
+    name: str
+    category: str
+    model: str
+    raw: str
+    refined: str
+    latency_ms: int
+    filler_count_raw: int = 0
+    filler_count_refined: int = 0
+    length_ratio: float = 0.0
+    has_loop_artifact: bool = False
+    prompt_leak: Optional[str] = None
+    answer_leak: Optional[str] = None
+    missing_substrings: list[str] = field(default_factory=list)
+    missing_question_mark: bool = False
+    flags: list[str] = field(default_factory=list)
+    """Short human-readable failure labels — populated by ``score``."""
+
+
+def count_fillers(text: str) -> int:
+    return sum(len(pat.findall(text)) for pat in FILLER_PATTERNS)
+
+
+def has_loop_run(text: str, threshold: int = 6) -> bool:
+    """Detect 6+ consecutive identical tokens — same heuristic as the
+    pre-processor. If the pre-processor did its job, a raw with a loop
+    tail should come back without one."""
+    tokens = text.split()
+    if len(tokens) < threshold:
+        return False
+    run = 1
+    prev: Optional[str] = None
+    for tok in tokens:
+        key = re.sub(r"[^\w]", "", tok).lower()
+        if key and key == prev:
+            run += 1
+            if run >= threshold:
+                return True
+        else:
+            run = 1
+            prev = key
+    return False
+
+
+def first_match(patterns: Iterable[re.Pattern[str]], text: str) -> Optional[str]:
+    stripped = text.lstrip()
+    for pat in patterns:
+        m = pat.search(stripped)
+        if m:
+            return m.group(0)
+    return None
+
+
+def score(sample: Sample, model: str, refined: str, latency_ms: int) -> Scorecard:
+    # Measure length against the *cleaned* raw so the pre-processor's work
+    # (stripping Whisper loops) doesn't get counted against the refinement.
+    cleaned_raw = collapse_repetitive_artifacts(sample.raw)
+    card = Scorecard(
+        name=sample.name,
+        category=sample.category,
+        model=model,
+        raw=sample.raw,
+        refined=refined,
+        latency_ms=latency_ms,
+        filler_count_raw=count_fillers(sample.raw),
+        filler_count_refined=count_fillers(refined),
+        length_ratio=(len(refined) / max(len(cleaned_raw), 1)),
+        has_loop_artifact=has_loop_run(refined),
+        prompt_leak=first_match(PROMPT_LEAK_PHRASES, refined),
+        answer_leak=first_match(ANSWER_LEAK_PHRASES, refined),
+    )
+
+    for needle in sample.must_contain_substrings:
+        if needle.lower() not in refined.lower():
+            card.missing_substrings.append(needle)
+
+    if sample.keep_question_mark and not refined.rstrip().endswith("?"):
+        card.missing_question_mark = True
+
+    # Roll up human-readable failure labels.
+    if card.prompt_leak:
+        card.flags.append(f"prompt-leak({card.prompt_leak!r})")
+    if card.answer_leak:
+        card.flags.append(f"answer-leak({card.answer_leak!r})")
+    if sample.must_not_loop and card.has_loop_artifact:
+        card.flags.append("loop-echo")
+    if card.missing_substrings:
+        card.flags.append(f"lost-terms={card.missing_substrings}")
+    if card.missing_question_mark:
+        card.flags.append("question→statement")
+    if card.filler_count_raw > 0 and card.filler_count_refined >= card.filler_count_raw:
+        card.flags.append(
+            f"fillers-not-removed({card.filler_count_raw}→{card.filler_count_refined})"
+        )
+    if card.length_ratio < 0.25:
+        card.flags.append(f"too-short({card.length_ratio:.2f})")
+    if card.length_ratio > 1.5:
+        card.flags.append(f"too-long({card.length_ratio:.2f})")
+
+    return card
+
+
+# ── Runner ────────────────────────────────────────────────────────────
+
+
+DEFAULT_PORTS = (8000, 8765, 8899, 17493)
+
+
+def detect_backend_port(hint: Optional[int]) -> int:
+    """Return a port that answers /health, preferring the hint."""
+    candidates: list[int] = []
+    if hint is not None:
+        candidates.append(hint)
+    candidates.extend(p for p in DEFAULT_PORTS if p != hint)
+
+    for port in candidates:
+        try:
+            with socket.create_connection(("127.0.0.1", port), timeout=0.4):
+                pass
+        except OSError:
+            continue
+        try:
+            r = httpx.get(f"http://127.0.0.1:{port}/health", timeout=2.0)
+            if r.status_code == 200 and r.json().get("status") == "healthy":
+                return port
+        except Exception:
+            continue
+    raise SystemExit(
+        "No running Voicebox backend found. Start it (`python backend/main.py`) "
+        f"or pass --port. Tried: {candidates}"
+    )
+
+
+def refine_via_api(client: httpx.Client, port: int, system_prompt: str,
+                   raw: str, model_size: str) -> tuple[str, int]:
+    """Mirror the real ``refine_transcript`` path: deterministic pre-process
+    first, then LLM. We hit ``/llm/generate`` rather than the refinement
+    endpoint because that one takes a capture_id — the pre-process call
+    here keeps the test exercising the full production pipeline without
+    standing up a fake Capture row."""
+    cleaned = collapse_repetitive_artifacts(raw)
+    start = time.monotonic()
+    resp = client.post(
+        f"http://127.0.0.1:{port}/llm/generate",
+        json={
+            "prompt": cleaned,
+            "system": system_prompt[:4000],
+            "model_size": model_size,
+            "max_tokens": 2048,
+            "temperature": 0.2,
+            # Same few-shot pairs the refinement service uses — keeps the
+            # test exercising the full production prompt stack.
+            "examples": [[u, a] for u, a in REFINEMENT_EXAMPLES],
+        },
+        timeout=180.0,
+    )
+    latency_ms = int((time.monotonic() - start) * 1000)
+    resp.raise_for_status()
+    return resp.json().get("text", "").strip(), latency_ms
+
+
+def format_report(cards: list[Scorecard]) -> str:
+    lines: list[str] = []
+    lines.append("")
+    lines.append("═" * 100)
+    by_model: dict[str, list[Scorecard]] = {}
+    for card in cards:
+        by_model.setdefault(card.model, []).append(card)
+
+    for model, model_cards in by_model.items():
+        pass_count = sum(1 for c in model_cards if not c.flags)
+        lines.append("")
+        lines.append(
+            f"▌{model}  —  {pass_count}/{len(model_cards)} clean, "
+            f"avg {sum(c.latency_ms for c in model_cards) // len(model_cards)} ms"
+        )
+        lines.append("─" * 100)
+        for card in model_cards:
+            status = "✓" if not card.flags else "✗"
+            lines.append(f"  {status} {card.name}  ({card.category}, {card.latency_ms} ms)")
+            lines.append(f"      raw:     {card.raw[:90]}{'…' if len(card.raw) > 90 else ''}")
+            lines.append(f"      refined: {card.refined[:90]}{'…' if len(card.refined) > 90 else ''}")
+            lines.append(
+                f"      fillers {card.filler_count_raw}→{card.filler_count_refined}, "
+                f"length×{card.length_ratio:.2f}"
+            )
+            if card.flags:
+                lines.append(f"      ⚠ {'; '.join(card.flags)}")
+            lines.append("")
+    lines.append("═" * 100)
+    return "\n".join(lines)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--port", type=int, default=None,
+                    help="Voicebox backend port (auto-detected if omitted)")
+    ap.add_argument("--model", choices=("0.6B", "1.7B", "4B"), action="append",
+                    help="Refinement model size(s) to test (repeat to run several)")
+    ap.add_argument("--json", type=Path, default=None,
+                    help="Also write results as JSON to this path")
+    args = ap.parse_args()
+
+    models = tuple(args.model) if args.model else ("0.6B", "4B")
+    port = detect_backend_port(args.port)
+    print(f"backend → http://127.0.0.1:{port}")
+    print(f"samples → {len(SAMPLES)}, models → {models}")
+
+    system_prompt = build_refinement_prompt(RefinementFlags())
+
+    cards: list[Scorecard] = []
+    with httpx.Client() as client:
+        for model in models:
+            print(f"\n── {model} " + "─" * (80 - len(model) - 4))
+            for i, sample in enumerate(SAMPLES, 1):
+                print(f"  [{i}/{len(SAMPLES)}] {sample.name} … ", end="", flush=True)
+                try:
+                    refined, latency_ms = refine_via_api(
+                        client, port, system_prompt, sample.raw, model
+                    )
+                except Exception as e:
+                    print(f"ERROR — {e}")
+                    continue
+                card = score(sample, model, refined, latency_ms)
+                cards.append(card)
+                print(f"{latency_ms} ms  " + ("ok" if not card.flags else f"⚠ {'; '.join(card.flags)}"))
+
+    print(format_report(cards))
+
+    if args.json:
+        args.json.write_text(json.dumps([asdict(c) for c in cards], indent=2))
+        print(f"wrote {args.json}")
+
+    # Exit non-zero if any card failed — makes the script CI-friendly if
+    # you ever want to trap regressions.
+    return 0 if all(not c.flags for c in cards) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backend/tests/test_rocm_backends.py

@@ -0,0 +1,68 @@
+"""
+Phase 2.2 Test: Backend ROCm compatibility.
+
+Validates that check_cuda_compatibility() and other backend utilities
+behave correctly on ROCm/AMD hardware.
+
+Usage:
+    python -m pytest backend/tests/test_rocm_backends.py -v
+"""
+
+from unittest.mock import patch
+
+import pytest
+
+
+class TestCheckCudaCompatibility:
+    """Unit tests for check_cuda_compatibility with ROCm awareness."""
+
+    def test_no_gpu_returns_compatible(self):
+        from backend.backends.base import check_cuda_compatibility
+
+        with patch("torch.cuda.is_available", return_value=False):
+            compatible, warning = check_cuda_compatibility()
+            assert compatible is True
+            assert warning is None
+
+    def test_rocm_skips_compute_check(self):
+        """On ROCm, the NVIDIA compute-capability check should be skipped."""
+        from backend.backends.base import check_cuda_compatibility
+
+        with patch("torch.cuda.is_available", return_value=True):
+            with patch("torch.version.hip", "6.2.41133"):
+                compatible, warning = check_cuda_compatibility()
+                assert compatible is True
+                assert warning is None
+
+    def test_cuda_compatible_arch(self):
+        from backend.backends.base import check_cuda_compatibility
+
+        with patch("torch.cuda.is_available", return_value=True):
+            with patch("torch.version.hip", None):
+                with patch("torch.cuda.get_device_capability", return_value=(8, 6)):
+                    with patch("torch.cuda.get_device_name", return_value="NVIDIA GeForce RTX 3060"):
+                        with patch.object(
+                            __import__("torch").cuda, "_get_arch_list",
+                            return_value=["sm_80", "sm_86", "sm_89"],
+                            create=True,
+                        ):
+                            compatible, warning = check_cuda_compatibility()
+                            assert compatible is True
+                            assert warning is None
+
+    def test_cuda_incompatible_arch(self):
+        from backend.backends.base import check_cuda_compatibility
+
+        with patch("torch.cuda.is_available", return_value=True):
+            with patch("torch.version.hip", None):
+                with patch("torch.cuda.get_device_capability", return_value=(9, 0)):
+                    with patch("torch.cuda.get_device_name", return_value="NVIDIA GeForce RTX 4090"):
+                        with patch.object(
+                            __import__("torch").cuda, "_get_arch_list",
+                            return_value=["sm_80", "sm_86"],
+                            create=True,
+                        ):
+                            compatible, warning = check_cuda_compatibility()
+                            assert compatible is False
+                            assert warning is not None
+                            assert "not supported" in warning

backend/tests/test_rocm_build.py

@@ -0,0 +1,129 @@
+"""
+Phase 1.2 Test: ROCm build script configuration.
+
+Validates that build_binary.py --rocm generates the correct PyInstaller
+arguments and optionally performs a true E2E build.
+
+Usage:
+    python -m pytest backend/tests/test_rocm_build.py -v
+    python -m pytest backend/tests/test_rocm_build.py -v -m "slow"    # include E2E
+"""
+
+import subprocess
+import sys
+from pathlib import Path
+from unittest.mock import patch
+
+import pytest
+
+from build_binary import build_server
+
+
+class TestRocmBuildArgs:
+    """Validate PyInstaller arguments for ROCm builds."""
+
+    @pytest.fixture
+    def captured_args(self):
+        """Run build_server(rocm=True) with mocked PyInstaller and return args."""
+        with (
+            patch("build_binary.PyInstaller.__main__.run") as mock_run,
+            patch("build_binary.platform.system", return_value="Linux"),
+            patch("build_binary.os.chdir"),
+        ):
+            build_server(rocm=True)
+            return mock_run.call_args[0][0]
+
+    def test_binary_name(self, captured_args):
+        idx = captured_args.index("--name")
+        assert captured_args[idx + 1] == "voicebox-server-rocm"
+
+    def test_pack_mode_is_onedir(self, captured_args):
+        assert "--onedir" in captured_args
+        assert "--onefile" not in captured_args
+
+    def test_hidden_imports_cuda(self, captured_args):
+        """ROCm builds must include torch.cuda hidden imports."""
+        assert "torch.cuda" in captured_args
+
+    def test_no_cudnn_hidden_import_for_rocm(self, captured_args):
+        """ROCm builds must NOT include NVIDIA-specific cudnn hidden imports."""
+        assert "torch.backends.cudnn" not in captured_args
+
+    def test_nvidia_excludes_present(self, captured_args):
+        """ROCm builds must exclude nvidia packages to avoid bundling ~3GB of bloat."""
+        excludes = []
+        for i, arg in enumerate(captured_args):
+            if arg == "--exclude-module":
+                excludes.append(captured_args[i + 1])
+        assert "nvidia" in excludes
+        assert "nvidia.cudnn" in excludes
+
+
+class TestRocmBuildCli:
+    """Validate CLI argument parsing for --rocm."""
+
+    def test_rocm_flag_parses(self):
+        build_script = Path(__file__).parent.parent / "build_binary.py"
+        result = subprocess.run(
+            [sys.executable, str(build_script), "--rocm", "--help"],
+            capture_output=True,
+            text=True,
+        )
+        assert result.returncode == 0
+        assert "--rocm" in result.stdout
+
+    def test_cannot_combine_cuda_and_rocm(self):
+        """Building with both CUDA and ROCm should raise ValueError."""
+        with pytest.raises(ValueError, match="Cannot build with both CUDA and ROCm"):
+            build_server(cuda=True, rocm=True)
+
+
+@pytest.mark.slow()
+@pytest.mark.skipif(sys.platform != "win32", reason="ROCm build E2E only runs on Windows")
+class TestRocmBuildE2E:
+    """
+    True end-to-end build test.
+    Executes build_binary.py --rocm, verifies the binary exists, and runs it
+    with --help to confirm it boots without import errors.
+    """
+
+    def test_rocm_binary_compiles_and_runs(self, tmp_path):
+        backend_dir = Path(__file__).parent.parent
+        build_script = backend_dir / "build_binary.py"
+        dist_dir = backend_dir / "dist"
+        binary_dir = dist_dir / "voicebox-server-rocm"
+        binary_exe = binary_dir / "voicebox-server-rocm.exe"
+
+        # Clean previous dist if it exists to ensure a fresh build
+        if binary_dir.exists():
+            import shutil
+            shutil.rmtree(binary_dir)
+
+        # Run the full build (this can take several minutes)
+        result = subprocess.run(
+            [sys.executable, str(build_script), "--rocm"],
+            capture_output=True,
+            text=True,
+            cwd=str(backend_dir),
+            timeout=900,
+        )
+
+        assert result.returncode == 0, (
+            f"Build failed with stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
+        )
+        assert binary_exe.exists(), (
+            f"Expected binary not found at {binary_exe}"
+        )
+
+        # Run the binary with --help to ensure it boots without import errors
+        run_result = subprocess.run(
+            [str(binary_exe), "--help"],
+            capture_output=True,
+            text=True,
+            timeout=60,
+        )
+
+        # A frozen binary may not have argparse help, but it should not crash
+        # with a ModuleNotFoundError or similar import error.
+        assert "ModuleNotFoundError" not in run_result.stderr
+        assert "ImportError" not in run_result.stderr

backend/tests/test_rocm_download.py

@@ -0,0 +1,203 @@
+"""
+Tests for the ROCm backend download service.
+
+Mocks httpx to verify download, extraction, and progress reporting
+without hitting the network.
+"""
+
+import json
+import tarfile
+import tempfile
+from io import BytesIO
+from pathlib import Path
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from backend.services import rocm
+from backend.utils.progress import get_progress_manager
+
+
+@pytest.fixture(autouse=True)
+def reset_progress_manager():
+    """Reset the global progress manager before each test."""
+    import backend.utils.progress
+    backend.utils.progress._progress_manager = None
+    yield
+    backend.utils.progress._progress_manager = None
+
+
+@pytest.fixture
+def mock_backends_dir(tmp_path: Path, monkeypatch):
+    """Patch get_data_dir so downloads land in a temp directory."""
+    monkeypatch.setattr(rocm, "get_backends_dir", lambda: tmp_path / "backends")
+    return tmp_path / "backends"
+
+
+@pytest.fixture
+def fake_tar_gz():
+    """Create an in-memory .tar.gz archive containing a dummy file."""
+    buf = BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tar:
+        data = b"fake binary content"
+        info = tarfile.TarInfo(name="voicebox-server-rocm.exe")
+        info.size = len(data)
+        tar.addfile(info, BytesIO(data))
+    buf.seek(0)
+    return buf.read()
+
+
+@pytest.fixture
+def fake_sha256():
+    """Return a dummy SHA-256 hex string."""
+    return "a" * 64
+
+
+class FakeResponse:
+    """Minimal fake for httpx.Response."""
+
+    def __init__(self, content: bytes = b"", status_code: int = 200, headers: dict | None = None):
+        self.content = content
+        self.status_code = status_code
+        self.headers = headers or {}
+
+    def raise_for_status(self):
+        if self.status_code >= 400:
+            raise Exception(f"HTTP {self.status_code}")
+
+    def iter_bytes(self, chunk_size: int = 1024):
+        for i in range(0, len(self.content), chunk_size):
+            yield self.content[i : i + chunk_size]
+
+    async def aiter_bytes(self, chunk_size: int = 1024):
+        for i in range(0, len(self.content), chunk_size):
+            yield self.content[i : i + chunk_size]
+
+    @property
+    def text(self):
+        return self.content.decode()
+
+
+class FakeHttpxClient:
+    """Minimal fake for httpx.AsyncClient."""
+
+    def __init__(self, responses: dict[str, FakeResponse]):
+        self._responses = responses
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *args):
+        return False
+
+    async def head(self, url: str):
+        return self._responses.get(url, FakeResponse(status_code=404))
+
+    async def get(self, url: str):
+        return self._responses.get(url, FakeResponse(status_code=404))
+
+    def stream(self, method: str, url: str):
+        resp = self._responses.get(url, FakeResponse(status_code=404))
+        resp.raise_for_status()
+
+        class _Streamer:
+            async def __aenter__(self):
+                return resp
+
+            async def __aexit__(self, *args):
+                return False
+
+            async def aiter_bytes(self, chunk_size: int = 1024):
+                for i in range(0, len(resp.content), chunk_size):
+                    yield resp.content[i : i + chunk_size]
+
+        return _Streamer()
+
+
+@pytest.mark.asyncio
+async def test_get_rocm_status_not_installed(mock_backends_dir):
+    status = rocm.get_rocm_status()
+    assert status["available"] is False
+    assert status["active"] is False
+    assert status["binary_path"] is None
+    assert status["downloading"] is False
+
+
+@pytest.mark.asyncio
+async def test_download_rocm_binary_progress_reporting(mock_backends_dir, fake_tar_gz, fake_sha256):
+    """
+    Verify that download_rocm_binary():
+    1. Downloads the server archive and ROCm libs archive.
+    2. Extracts them into the backends/rocm directory.
+    3. Reports progress via the progress_manager.
+    """
+    import hashlib
+
+    server_sha = hashlib.sha256(fake_tar_gz).hexdigest()
+    libs_sha = hashlib.sha256(fake_tar_gz).hexdigest()
+
+    responses = {
+        "https://github.com/jamiepine/voicebox/releases/download/v0.2.3/voicebox-server-rocm.tar.gz": FakeResponse(
+            content=fake_tar_gz,
+            headers={"content-length": str(len(fake_tar_gz))},
+        ),
+        "https://github.com/jamiepine/voicebox/releases/download/v0.2.3/voicebox-server-rocm.tar.gz.sha256": FakeResponse(
+            content=f"{server_sha}  voicebox-server-rocm.tar.gz\n".encode(),
+        ),
+        f"https://github.com/jamiepine/voicebox/releases/download/v0.2.3/rocm-libs-{rocm.ROCM_LIBS_VERSION}.tar.gz": FakeResponse(
+            content=fake_tar_gz,
+            headers={"content-length": str(len(fake_tar_gz))},
+        ),
+        f"https://github.com/jamiepine/voicebox/releases/download/v0.2.3/rocm-libs-{rocm.ROCM_LIBS_VERSION}.tar.gz.sha256": FakeResponse(
+            content=f"{libs_sha}  rocm-libs.tar.gz\n".encode(),
+        ),
+    }
+
+    fake_client = FakeHttpxClient(responses)
+
+    with patch("httpx.AsyncClient", return_value=fake_client):
+        await rocm.download_rocm_binary(version="v0.2.3")
+
+    # Verify extraction
+    rocm_dir = rocm.get_rocm_dir()
+    assert (rocm_dir / "voicebox-server-rocm.exe").exists()
+
+    # Verify manifest written
+    manifest_path = rocm.get_rocm_libs_manifest_path()
+    assert manifest_path.exists()
+    data = json.loads(manifest_path.read_text())
+    assert data["version"] == rocm.ROCM_LIBS_VERSION
+
+    # Verify progress was reported
+    progress = get_progress_manager().get_progress("rocm-backend")
+    assert progress is not None
+    assert progress["status"] == "complete"
+    assert progress["progress"] == 100.0
+
+
+@pytest.mark.asyncio
+async def test_is_rocm_active(mock_backends_dir, monkeypatch):
+    monkeypatch.setenv("VOICEBOX_BACKEND_VARIANT", "rocm")
+    assert rocm.is_rocm_active() is True
+
+    monkeypatch.setenv("VOICEBOX_BACKEND_VARIANT", "cpu")
+    assert rocm.is_rocm_active() is False
+
+    monkeypatch.delenv("VOICEBOX_BACKEND_VARIANT", raising=False)
+    assert rocm.is_rocm_active() is False
+
+
+@pytest.mark.asyncio
+async def test_delete_rocm_binary(mock_backends_dir, fake_tar_gz):
+    """Test deleting the ROCm backend directory."""
+    rocm_dir = rocm.get_rocm_dir()
+    rocm_dir.mkdir(parents=True, exist_ok=True)
+    (rocm_dir / "dummy.txt").write_text("hello")
+
+    result = await rocm.delete_rocm_binary()
+    assert result is True
+    assert not rocm_dir.exists()
+
+    # Deleting again should return False
+    result = await rocm.delete_rocm_binary()
+    assert result is False

backend/tests/test_rocm_requirements.py

@@ -0,0 +1,130 @@
+"""
+Phase 1.1 Test: ROCm requirements installation.
+
+Validates that requirements-rocm.txt correctly installs ROCm-enabled PyTorch
+and that torch.cuda.is_available() returns True on AMD hardware.
+
+Usage:
+    python -m pytest backend/tests/test_rocm_requirements.py -v
+"""
+
+import os
+import platform
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import pytest
+
+
+def _has_amd_hardware():
+    """Check if AMD GPU hardware is present on Windows."""
+    if platform.system() != "Windows":
+        return False
+    try:
+        result = subprocess.run(
+            [
+                "powershell",
+                "-Command",
+                "Get-WmiObject Win32_VideoController | "
+                "Where-Object {$_.AdapterCompatibility -like '*AMD*'} | "
+                "Measure-Object | Select-Object -ExpandProperty Count",
+            ],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        return int(result.stdout.strip()) > 0
+    except Exception:
+        return False
+
+
+@pytest.fixture()
+def backend_dir():
+    return Path(__file__).parent.parent
+
+
+class TestRocmRequirements:
+    """Validate requirements-rocm.txt content and installation."""
+
+    def test_requirements_file_exists(self, backend_dir):
+        req_file = backend_dir / "requirements-rocm.txt"
+        assert req_file.exists(), "requirements-rocm.txt must exist"
+
+    def test_requirements_file_content(self, backend_dir):
+        import re
+        req_file = backend_dir / "requirements-rocm.txt"
+        content = req_file.read_text()
+        assert "rocm7.2" in content, "Must point to ROCm 7.2 extra index"
+        # Parse exact package names to avoid false positives from URL substrings
+        package_names = re.findall(r"^([A-Za-z][A-Za-z0-9_-]*)", content, re.MULTILINE)
+        assert "torch" in package_names, "Must include torch package"
+        assert "torchaudio" in package_names, "Must include torchaudio package"
+        assert "torchvision" in package_names, "Must include torchvision package"
+
+    @pytest.mark.timeout(900)
+    @pytest.mark.skipif(
+        not os.environ.get("VOICEBOX_TEST_ROCM_INSTALL"),
+        reason="Set VOICEBOX_TEST_ROCM_INSTALL=1 to run the heavy install test",
+    )
+    def test_rocm_torch_installs_and_detects_amd(self, backend_dir):
+        """
+        Create a temporary venv, install requirements-rocm.txt, and verify
+        torch.cuda.is_available() returns True on AMD hardware.
+        """
+        req_file = backend_dir / "requirements-rocm.txt"
+        has_amd = _has_amd_hardware()
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            venv_dir = Path(tmpdir) / "venv"
+            subprocess.run(
+                [sys.executable, "-m", "venv", str(venv_dir)],
+                check=True,
+            )
+
+            if sys.platform == "win32":
+                venv_python = venv_dir / "Scripts" / "python.exe"
+            else:
+                venv_python = venv_dir / "bin" / "python"
+
+            # Upgrade pip to avoid resolver issues
+            subprocess.run(
+                [str(venv_python), "-m", "pip", "install", "--upgrade", "pip"],
+                check=True,
+            )
+
+            # Install ROCm requirements
+            subprocess.run(
+                [str(venv_python), "-m", "pip", "install", "-r", str(req_file)],
+                check=True,
+            )
+
+            # Verify torch imports and cuda availability
+            result = subprocess.run(
+                [
+                    str(venv_python),
+                    "-c",
+                    "import torch; print(torch.__version__); print(torch.cuda.is_available())",
+                ],
+                capture_output=True,
+                text=True,
+                check=True,
+            )
+
+            lines = result.stdout.strip().splitlines()
+            assert len(lines) >= 2, f"Unexpected output: {result.stdout}"
+            torch_version = lines[0]
+            cuda_available = lines[1] == "True"
+
+            # The honest test: on AMD hardware ROCm torch should report cuda available
+            if has_amd:
+                assert cuda_available, (
+                    f"AMD hardware detected but torch.cuda.is_available() returned False. "
+                    f"torch version: {torch_version}, stderr: {result.stderr}"
+                )
+            else:
+                assert not cuda_available, (
+                    f"No AMD hardware detected but torch.cuda.is_available() returned True. "
+                    f"torch version: {torch_version}"
+                )

backend/tests/test_task_queue_cancellation.py

@@ -0,0 +1,54 @@
+import asyncio
+
+import pytest
+
+from backend.services import task_queue
+
+
+@pytest.mark.asyncio
+async def test_cancel_queued_generation_skips_execution():
+    task_queue.init_queue(force=True)
+
+    running_started = asyncio.Event()
+    release_running = asyncio.Event()
+    queued_ran = asyncio.Event()
+
+    async def running_job():
+        running_started.set()
+        await release_running.wait()
+
+    async def queued_job():
+        queued_ran.set()
+
+    task_queue.enqueue_generation("gen-running", running_job())
+    await asyncio.wait_for(running_started.wait(), timeout=1)
+
+    task_queue.enqueue_generation("gen-queued", queued_job())
+    assert task_queue.cancel_generation("gen-queued") == "queued"
+
+    release_running.set()
+    await asyncio.sleep(0.1)
+
+    assert not queued_ran.is_set()
+
+
+@pytest.mark.asyncio
+async def test_cancel_running_generation_cancels_task():
+    task_queue.init_queue(force=True)
+
+    running_started = asyncio.Event()
+    running_cancelled = asyncio.Event()
+
+    async def running_job():
+        running_started.set()
+        try:
+            await asyncio.Event().wait()
+        except asyncio.CancelledError:
+            running_cancelled.set()
+            raise
+
+    task_queue.enqueue_generation("gen-running", running_job())
+    await asyncio.wait_for(running_started.wait(), timeout=1)
+
+    assert task_queue.cancel_generation("gen-running") == "running"
+    await asyncio.wait_for(running_cancelled.wait(), timeout=1)

backend/tests/test_whisper_download.py

@@ -0,0 +1,178 @@
+"""
+Test real model download with SSE progress monitoring.
+"""
+
+import asyncio
+import json
+import httpx
+import time
+from typing import List, Dict
+
+async def monitor_sse_stream(model_name: str, timeout: int = 300):
+    """Monitor SSE stream for a model download."""
+    events: List[Dict] = []
+    url = f"http://localhost:8000/models/progress/{model_name}"
+
+    print(f"Connecting to SSE endpoint: {url}")
+
+    async with httpx.AsyncClient(timeout=timeout) as client:
+        async with client.stream("GET", url) as response:
+            print(f"SSE connected, status: {response.status_code}")
+
+            if response.status_code != 200:
+                print(f"Error: SSE endpoint returned {response.status_code}")
+                return events
+
+            async for line in response.aiter_lines():
+                if not line:
+                    continue
+
+                print(f"  Raw SSE: {line[:100]}...")  # Print first 100 chars
+
+                if line.startswith("data: "):
+                    try:
+                        data = json.loads(line[6:])
+                        print(f"  → {data['status']:12} {data.get('progress', 0):6.1f}% {data.get('filename', '')}")
+                        events.append(data)
+
+                        # Stop if complete or error
+                        if data.get("status") in ("complete", "error"):
+                            print(f"  Download {data['status']}!")
+                            break
+
+                    except json.JSONDecodeError as e:
+                        print(f"  Error parsing JSON: {e}")
+                        print(f"  Line was: {line}")
+
+                elif line.startswith(": heartbeat"):
+                    print("  ♥ heartbeat")
+
+    return events
+
+
+async def trigger_download(model_name: str):
+    """Trigger a model download via the API."""
+    url = "http://localhost:8000/models/download"
+
+    print(f"\nTriggering download for: {model_name}")
+
+    async with httpx.AsyncClient(timeout=300) as client:
+        response = await client.post(url, json={"model_name": model_name})
+        print(f"Response: {response.status_code} - {response.json()}")
+        return response.status_code == 200
+
+
+async def check_server():
+    """Check if the server is running."""
+    try:
+        async with httpx.AsyncClient(timeout=5) as client:
+            response = await client.get("http://localhost:8000/health")
+            return response.status_code == 200
+    except Exception as e:
+        print(f"Server not running: {e}")
+        return False
+
+
+async def main():
+    print("=" * 60)
+    print("Real Model Download Progress Test")
+    print("=" * 60)
+
+    # Check if server is running
+    print("\nChecking if server is running...")
+    if not await check_server():
+        print("✗ Server is not running on http://localhost:8000")
+        print("\nPlease start the server first:")
+        print("  cd backend && python main.py")
+        return False
+
+    print("✓ Server is running")
+
+    # Choose a small model for testing
+    model_name = "whisper-base"  # ~150MB, faster to download
+    print(f"\nUsing model: {model_name}")
+
+    # Option to delete model first if it exists
+    print("\nDo you want to delete the model first to force a fresh download? (y/n)")
+    # For automated testing, skip deletion prompt
+    # delete_first = input().strip().lower() == 'y'
+    delete_first = False
+
+    if delete_first:
+        print(f"Deleting {model_name}...")
+        async with httpx.AsyncClient(timeout=30) as client:
+            response = await client.delete(f"http://localhost:8000/models/{model_name}")
+            print(f"Delete response: {response.status_code}")
+
+    print("\n" + "=" * 60)
+    print("Starting Test")
+    print("=" * 60)
+
+    # Start monitoring SSE stream BEFORE triggering download
+    async def run_test():
+        # Start SSE monitor in background
+        monitor_task = asyncio.create_task(monitor_sse_stream(model_name))
+
+        # Wait a bit to ensure SSE is connected
+        await asyncio.sleep(1)
+
+        # Trigger download
+        success = await trigger_download(model_name)
+
+        if not success:
+            print("✗ Failed to trigger download")
+            monitor_task.cancel()
+            return False
+
+        # Wait for SSE monitor to complete
+        events = await monitor_task
+
+        return events
+
+    events = await run_test()
+
+    # Results
+    print("\n" + "=" * 60)
+    print("Test Results")
+    print("=" * 60)
+
+    if not events:
+        print("✗ FAILED - No SSE events received!")
+        print("\nPossible causes:")
+        print("  1. SSE endpoint not working")
+        print("  2. Progress updates not being sent")
+        print("  3. Model already downloaded (no progress to report)")
+        print("\nTry deleting the model first to force a fresh download:")
+        print(f"  curl -X DELETE http://localhost:8000/models/{model_name}")
+        return False
+
+    print(f"✓ Received {len(events)} SSE events")
+    print(f"\nFirst event: {events[0]}")
+    print(f"Last event: {events[-1]}")
+
+    # Check if we got meaningful progress
+    has_progress = any(e.get('progress', 0) > 0 for e in events)
+    has_complete = any(e.get('status') == 'complete' for e in events)
+
+    if has_progress:
+        print("✓ Progress updates received")
+    else:
+        print("✗ No progress updates (might be already downloaded)")
+
+    if has_complete:
+        print("✓ Download completed successfully")
+    else:
+        print("✗ Download did not complete")
+
+    success = has_progress and has_complete
+
+    if success:
+        print("\n✓ TEST PASSED - Progress tracking works!")
+    else:
+        print("\n⊘ TEST INCONCLUSIVE - Try with a fresh download")
+
+    return success
+
+
+if __name__ == "__main__":
+    asyncio.run(main())