mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-10-03 09:05:17 -07:00
docs: address PR review feedback
- architecture.mdx: fix backends/ file list (remove nonexistent qwen_backend.py, rename tada_backend.py → hume_backend.py) - model-management.mdx: Kokoro language count 9 → 8 (matches ModelConfig) - model-management.mdx: ProgressManager path services/ → utils/ - tts-generation.mdx: ModelConfig example uses field(default_factory=...) — mutable default would raise at runtime - tts-generation.mdx: "1080p samples" → "on CUDA" (1080p is video, not audio) - PROJECT_STATUS.md: replace ASCII architecture diagram with prose (matches no-ASCII-art rule) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
3412dca662
commit
b91d5d74cb
+14
-30
@@ -19,36 +19,20 @@
|
|||||||
|
|
||||||
## Architecture Overview
|
## Architecture Overview
|
||||||
|
|
||||||
```
|
**Tauri shell (Rust)** hosts a **React frontend** (`app/`) that talks over HTTP on `localhost:17493` to a **FastAPI backend** (`backend/`).
|
||||||
┌─────────────────────────────────────────────────────┐
|
|
||||||
│ Tauri Shell (Rust) │
|
The backend exposes:
|
||||||
│ ┌───────────────────────────────────────────────┐ │
|
|
||||||
│ │ React Frontend (app/) │ │
|
- **`TTSBackend` Protocol** with seven concrete engine implementations:
|
||||||
│ │ Zustand stores · API client · Generation UI │ │
|
- Qwen3-TTS (PyTorch or MLX depending on platform)
|
||||||
│ │ Stories Editor · Voice Profiles · Model Mgmt │ │
|
- Qwen CustomVoice (predefined speakers with instruct)
|
||||||
│ └──────────────────────┬────────────────────────┘ │
|
- LuxTTS (fast, CPU-friendly)
|
||||||
│ │ HTTP :17493 │
|
- Chatterbox Multilingual (23 languages)
|
||||||
│ ┌──────────────────────▼────────────────────────┐ │
|
- Chatterbox Turbo (English, paralinguistic tags)
|
||||||
│ │ FastAPI Backend (backend/) │ │
|
- TADA (1B English, 3B multilingual via HumeAI)
|
||||||
│ │ ┌─────────────────────────────────────────┐ │ │
|
- Kokoro 82M (pre-built voices, CPU realtime)
|
||||||
│ │ │ TTSBackend Protocol │ │ │
|
- **`STTBackend` Protocol** for Whisper (PyTorch or MLX-Whisper)
|
||||||
│ │ │ ┌──────────┐ ┌───────┐ ┌───────────┐ │ │ │
|
- **Profiles / History / Stories** services for persistence and timeline editing
|
||||||
│ │ │ │ Qwen3-TTS│ │LuxTTS │ │Chatterbox │ │ │ │
|
|
||||||
│ │ │ │(Py/MLX) │ │ │ │(MTL+Turbo)│ │ │ │
|
|
||||||
│ │ │ └──────────┘ └───────┘ └───────────┘ │ │ │
|
|
||||||
│ │ │ ┌──────────┐ ┌────────┐ ┌──────────┐ │ │ │
|
|
||||||
│ │ │ │ TADA │ │Kokoro │ │Qwen │ │ │ │
|
|
||||||
│ │ │ │(1B / 3B) │ │(82M) │ │CustomVoice│ │ │ │
|
|
||||||
│ │ │ └──────────┘ └────────┘ └──────────┘ │ │ │
|
|
||||||
│ │ └─────────────────────────────────────────┘ │ │
|
|
||||||
│ │ ┌───────────┐ ┌─────────┐ │ │
|
|
||||||
│ │ │ STTBackend│ │ Profiles│ │ │
|
|
||||||
│ │ │ (Whisper) │ │ History │ │ │
|
|
||||||
│ │ └───────────┘ │ Stories │ │ │
|
|
||||||
│ │ └─────────┘ │ │
|
|
||||||
│ └───────────────────────────────────────────────┘ │
|
|
||||||
└─────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Key Files
|
### Key Files
|
||||||
|
|
||||||
|
|||||||
@@ -83,12 +83,11 @@ These two layers communicate via HTTP on `localhost:17493`, with the frontend ma
|
|||||||
<File name="base.py" />
|
<File name="base.py" />
|
||||||
<File name="pytorch_backend.py" />
|
<File name="pytorch_backend.py" />
|
||||||
<File name="mlx_backend.py" />
|
<File name="mlx_backend.py" />
|
||||||
<File name="qwen_backend.py" />
|
|
||||||
<File name="qwen_custom_voice_backend.py" />
|
<File name="qwen_custom_voice_backend.py" />
|
||||||
<File name="luxtts_backend.py" />
|
<File name="luxtts_backend.py" />
|
||||||
<File name="chatterbox_backend.py" />
|
<File name="chatterbox_backend.py" />
|
||||||
<File name="chatterbox_turbo_backend.py" />
|
<File name="chatterbox_turbo_backend.py" />
|
||||||
<File name="tada_backend.py" />
|
<File name="hume_backend.py" />
|
||||||
<File name="kokoro_backend.py" />
|
<File name="kokoro_backend.py" />
|
||||||
</Folder>
|
</Folder>
|
||||||
<Folder name="database">
|
<Folder name="database">
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ Every model is described by a `ModelConfig` entry in `backend/backends/__init__.
|
|||||||
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
|
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
|
||||||
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
|
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
|
||||||
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
|
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
|
||||||
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 9 |
|
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 8 |
|
||||||
|
|
||||||
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
|
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
|
||||||
|
|
||||||
@@ -59,7 +59,7 @@ Set `VOICEBOX_MODELS_DIR` to override.
|
|||||||
|
|
||||||
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
|
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
|
||||||
|
|
||||||
**`ProgressManager`** (`backend/services/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
|
**`ProgressManager`** (`backend/utils/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
|
||||||
|
|
||||||
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
|
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
|
||||||
|
|
||||||
|
|||||||
@@ -50,7 +50,7 @@ class ModelConfig:
|
|||||||
size_mb: int = 0
|
size_mb: int = 0
|
||||||
needs_trim: bool = False
|
needs_trim: bool = False
|
||||||
supports_instruct: bool = False
|
supports_instruct: bool = False
|
||||||
languages: list[str] = ["en"]
|
languages: list[str] = field(default_factory=lambda: ["en"])
|
||||||
```
|
```
|
||||||
|
|
||||||
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
|
Registry helpers in `backends/__init__.py` replace what used to be per-engine `if/elif` chains:
|
||||||
@@ -223,7 +223,7 @@ The model management API (`/models/load`, `/models/unload`) lets users free VRAM
|
|||||||
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
|
- **Voice prompt caching** saves ~1-2s on repeated generations from the same profile.
|
||||||
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
|
- **Model pinning** — the first load is slow (download + load), subsequent generations reuse the cached model in memory.
|
||||||
|
|
||||||
### Per-engine VRAM (approximate, 1080p samples)
|
### Per-engine VRAM (approximate, on CUDA)
|
||||||
|
|
||||||
| Engine | VRAM |
|
| Engine | VRAM |
|
||||||
|--------|------|
|
|--------|------|
|
||||||
|
|||||||
Reference in New Issue
Block a user