mirror of
https://github.com/jamiepine/voicebox.git
synced 2026-10-02 16:45:15 -07:00
docs: audit mdx docs against multi-engine backend (#484)
* docs: audit mdx docs against multi-engine backend and refresh stale content Rewrote developer-facing docs that predated the TTSBackend Protocol / ModelConfig registry refactor (architecture, tts-generation, model-management, transcription). Updated user-facing docs to reflect all seven shipped engines (Qwen, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro) instead of the outdated "5 engines" claim. Also fixes: - Stale app identifier (com.voicebox.app → sh.voicebox.app) - CUDA backend update flow (now two-archive split, not N-way chunks) - Whisper model list (removed tiny, added turbo) - Broken /development/ and /guides/ route links - Stale just commands and install steps (missing --no-deps chatterbox/tada) - Removed ASCII art diagrams from README and stories.mdx - History Generation schema sync with DB model Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * docs: add DeepWiki badge to README Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * docs: address PR review feedback - architecture.mdx: fix backends/ file list (remove nonexistent qwen_backend.py, rename tada_backend.py → hume_backend.py) - model-management.mdx: Kokoro language count 9 → 8 (matches ModelConfig) - model-management.mdx: ProgressManager path services/ → utils/ - tts-generation.mdx: ModelConfig example uses field(default_factory=...) — mutable default would raise at runtime - tts-generation.mdx: "1080p samples" → "on CUDA" (1080p is video, not audio) - PROJECT_STATUS.md: replace ASCII architecture diagram with prose (matches no-ASCII-art rule) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(app): guard against undefined engine in FloatingGenerateBox preset check form.getValues('engine') returns string | undefined; Set<string>.has() rejects undefined under strict mode. Added a truthy guard before the preset lookup. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Jamie Pine
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
da6070155e
commit
ae91aa9a88
@@ -1,341 +1,199 @@
|
||||
---
|
||||
title: "Model Management"
|
||||
description: "How model downloading, loading, and status tracking works in Voicebox"
|
||||
description: "How model downloading, loading, and status tracking works across all engines"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Voicebox manages two types of models:
|
||||
Voicebox manages two categories of models:
|
||||
|
||||
**TTS Models:** Qwen3-TTS for voice cloning (0.6B and 1.7B variants).
|
||||
**TTS Models** — Seven engines covering zero-shot cloning and preset voices. Each engine may have one or more size variants.
|
||||
|
||||
**ASR Models:** Whisper for transcription (tiny through large).
|
||||
**ASR Models** — Whisper for transcription. Five sizes, plus MLX-Whisper on Apple Silicon for ~8× faster transcription.
|
||||
|
||||
Models are downloaded from HuggingFace Hub on first use and cached locally.
|
||||
Every model is described by a `ModelConfig` entry in `backend/backends/__init__.py`. Models are downloaded from HuggingFace Hub on first use and cached in the platform-standard HF cache.
|
||||
|
||||
## Available Models
|
||||
## Available TTS Models
|
||||
|
||||
### TTS Models
|
||||
| Model | Engine | HuggingFace Repo | Size | VRAM | Languages |
|
||||
|-------|--------|------------------|------|------|-----------|
|
||||
| **Qwen TTS 1.7B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 3.5 GB | ~6 GB | 10 |
|
||||
| **Qwen TTS 0.6B** | `qwen` | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | 1.2 GB | ~2 GB | 10 |
|
||||
| **Qwen CustomVoice 1.7B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 3.5 GB | ~6 GB | 10 |
|
||||
| **Qwen CustomVoice 0.6B** | `qwen_custom_voice` | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | 1.2 GB | ~2 GB | 10 |
|
||||
| **LuxTTS** | `luxtts` | `YatharthS/LuxTTS` | 300 MB | ~1 GB | English |
|
||||
| **Chatterbox Multilingual** | `chatterbox` | `ResembleAI/chatterbox` | 3.2 GB | ~3 GB | 23 |
|
||||
| **Chatterbox Turbo** | `chatterbox_turbo` | `ResembleAI/chatterbox-turbo` | 1.5 GB | ~1.5 GB | English |
|
||||
| **TADA 1B** | `tada` | `HumeAI/tada-1b` | 4 GB | ~4 GB | English |
|
||||
| **TADA 3B Multilingual** | `tada` | `HumeAI/tada-3b-ml` | 8 GB | ~8 GB | 10 |
|
||||
| **Kokoro 82M** | `kokoro` | `hexgrad/Kokoro-82M` | 350 MB | ~150 MB | 8 |
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| 0.6B | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | ~1.2GB | ~2GB |
|
||||
| 1.7B | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ~3.4GB | ~6GB |
|
||||
On Apple Silicon, Qwen TTS uses MLX-optimized repos from `mlx-community` instead of the PyTorch repos. The backend picks automatically via `get_backend_type()`.
|
||||
|
||||
### Whisper Models
|
||||
## Available Whisper Models
|
||||
|
||||
| Model | HuggingFace ID | Size | VRAM |
|
||||
|-------|----------------|------|------|
|
||||
| tiny | `openai/whisper-tiny` | ~150MB | ~1GB |
|
||||
| base | `openai/whisper-base` | ~300MB | ~1GB |
|
||||
| small | `openai/whisper-small` | ~500MB | ~2GB |
|
||||
| medium | `openai/whisper-medium` | ~1.5GB | ~5GB |
|
||||
| large | `openai/whisper-large` | ~3GB | ~10GB |
|
||||
| Model | HuggingFace Repo | Size |
|
||||
|-------|------------------|------|
|
||||
| **Whisper Base** | `openai/whisper-base` | ~300 MB |
|
||||
| **Whisper Small** | `openai/whisper-small` | ~500 MB |
|
||||
| **Whisper Medium** | `openai/whisper-medium` | ~1.5 GB |
|
||||
| **Whisper Large** | `openai/whisper-large-v3` | ~3 GB |
|
||||
| **Whisper Turbo** | `openai/whisper-large-v3-turbo` | ~1.5 GB |
|
||||
|
||||
On Apple Silicon, MLX-Whisper is preferred automatically — see [Transcription](/developer/transcription).
|
||||
|
||||
## Model Storage
|
||||
|
||||
Models are cached in the HuggingFace cache directory:
|
||||
Models live in the platform HuggingFace cache:
|
||||
|
||||
<Files>
|
||||
<Folder name="~/.cache/huggingface/hub" defaultOpen>
|
||||
<File name="models--Qwen--Qwen3-TTS-12Hz-1.7B-Base/" />
|
||||
<File name="models--Qwen--Qwen3-TTS-12Hz-0.6B-Base/" />
|
||||
<File name="models--openai--whisper-base/" />
|
||||
</Folder>
|
||||
</Files>
|
||||
| Platform | Path |
|
||||
|----------|------|
|
||||
| macOS | `~/.cache/huggingface/hub/` |
|
||||
| Linux | `~/.cache/huggingface/hub/` |
|
||||
| Windows | `%USERPROFILE%\.cache\huggingface\hub\` |
|
||||
| Docker | `/home/voicebox/.cache/huggingface/hub` (volume-mounted) |
|
||||
|
||||
Set `VOICEBOX_MODELS_DIR` to override.
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
### Progress Manager
|
||||
Downloads stream progress to the frontend via Server-Sent Events. The progress pipeline has three pieces:
|
||||
|
||||
Tracks download progress across all models:
|
||||
**`ProgressManager`** (`backend/utils/progress.py`) — in-memory map of `model_name → {current, total, filename, status}`.
|
||||
|
||||
**`HFProgressTracker`** — context manager that intercepts HuggingFace Hub downloads to emit byte-level progress. Needed because `huggingface_hub` silently disables tqdm in frozen PyInstaller builds.
|
||||
|
||||
**SSE endpoint** — `GET /models/progress/{model_name}` streams updates until `status` is `complete` or `error`.
|
||||
|
||||
```python
|
||||
class ProgressManager:
|
||||
def __init__(self):
|
||||
self._progress = {} # model_name -> progress_info
|
||||
|
||||
def update_progress(
|
||||
self,
|
||||
model_name: str,
|
||||
current: int,
|
||||
total: int,
|
||||
filename: str,
|
||||
status: str,
|
||||
):
|
||||
self._progress[model_name] = {
|
||||
"current": current,
|
||||
"total": total,
|
||||
"filename": filename,
|
||||
"status": status, # downloading, complete, error
|
||||
"updated_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def get_progress(self, model_name: str) -> Optional[dict]:
|
||||
return self._progress.get(model_name)
|
||||
```
|
||||
|
||||
### HuggingFace Progress Callback
|
||||
|
||||
Hooks into HuggingFace's download system:
|
||||
|
||||
```python
|
||||
class HFProgressTracker:
|
||||
def __init__(self, callback):
|
||||
self.callback = callback
|
||||
|
||||
@contextmanager
|
||||
def patch_download(self):
|
||||
"""Context manager to intercept HF downloads."""
|
||||
original_download = hf_hub_download
|
||||
|
||||
def patched_download(*args, **kwargs):
|
||||
# Intercept progress
|
||||
result = original_download(*args, **kwargs)
|
||||
self.callback(progress_info)
|
||||
return result
|
||||
|
||||
# Apply patch
|
||||
with patch('huggingface_hub.hf_hub_download', patched_download):
|
||||
yield
|
||||
```
|
||||
|
||||
### Server-Sent Events (SSE)
|
||||
|
||||
Progress is streamed to the frontend:
|
||||
|
||||
```python
|
||||
@app.get("/models/progress/{model_name}")
|
||||
async def get_model_progress(model_name: str):
|
||||
async def event_generator():
|
||||
while True:
|
||||
progress = progress_manager.get_progress(model_name)
|
||||
if progress:
|
||||
yield f"data: {json.dumps(progress)}\n\n"
|
||||
|
||||
if progress and progress["status"] in ["complete", "error"]:
|
||||
break
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream"
|
||||
)
|
||||
```
|
||||
|
||||
## Task Manager
|
||||
|
||||
Tracks active downloads and generations:
|
||||
|
||||
```python
|
||||
class TaskManager:
|
||||
def __init__(self):
|
||||
self._active_downloads = {}
|
||||
self._active_generations = {}
|
||||
|
||||
def start_download(self, model_name: str):
|
||||
self._active_downloads[model_name] = {
|
||||
"status": "downloading",
|
||||
"started_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
def complete_download(self, model_name: str):
|
||||
if model_name in self._active_downloads:
|
||||
del self._active_downloads[model_name]
|
||||
|
||||
def get_active_tasks(self) -> dict:
|
||||
return {
|
||||
"downloads": list(self._active_downloads.values()),
|
||||
"generations": list(self._active_generations.values()),
|
||||
}
|
||||
# Frontend
|
||||
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||
eventSource.onmessage = (event) => {
|
||||
const { current, total, status } = JSON.parse(event.data);
|
||||
updateProgressBar(current / total);
|
||||
if (status === "complete") eventSource.close();
|
||||
};
|
||||
```
|
||||
|
||||
## Model Status
|
||||
|
||||
Check which models are downloaded and loaded:
|
||||
|
||||
```python
|
||||
@app.get("/models/status")
|
||||
async def get_model_status() -> ModelStatusListResponse:
|
||||
models = []
|
||||
|
||||
# Check TTS models
|
||||
for size, hf_id in [("1.7B", "Qwen/Qwen3-TTS-12Hz-1.7B-Base"), ...]:
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
loaded = tts_model._current_model_size == size
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"qwen-tts-{size}",
|
||||
display_name=f"Qwen3-TTS {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=loaded,
|
||||
))
|
||||
|
||||
# Check Whisper models
|
||||
for size in ["tiny", "base", "small", "medium", "large"]:
|
||||
hf_id = f"openai/whisper-{size}"
|
||||
downloaded = is_model_downloaded(hf_id)
|
||||
|
||||
models.append(ModelStatus(
|
||||
model_name=f"whisper-{size}",
|
||||
display_name=f"Whisper {size}",
|
||||
downloaded=downloaded,
|
||||
size_mb=get_model_size_mb(hf_id),
|
||||
loaded=False, # Whisper is loaded on-demand
|
||||
))
|
||||
|
||||
return ModelStatusListResponse(models=models)
|
||||
```
|
||||
|
||||
## Manual Model Operations
|
||||
|
||||
### Load Model
|
||||
|
||||
```python
|
||||
@app.post("/models/load")
|
||||
async def load_model(model_size: str = "1.7B"):
|
||||
tts_model = get_tts_model()
|
||||
await tts_model.load_model_async(model_size)
|
||||
return {"status": "loaded", "model_size": model_size}
|
||||
```
|
||||
|
||||
### Unload Model
|
||||
|
||||
```python
|
||||
@app.post("/models/unload")
|
||||
async def unload_model():
|
||||
tts_model = get_tts_model()
|
||||
tts_model.unload_model()
|
||||
return {"status": "unloaded"}
|
||||
```
|
||||
|
||||
### Trigger Download
|
||||
|
||||
```python
|
||||
@app.post("/models/download")
|
||||
async def trigger_model_download(request: ModelDownloadRequest):
|
||||
# This triggers the download in background
|
||||
# Progress is tracked via /models/progress/{model_name}
|
||||
|
||||
if request.model_name.startswith("qwen-tts"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_tts_model(size))
|
||||
elif request.model_name.startswith("whisper"):
|
||||
size = request.model_name.split("-")[-1]
|
||||
asyncio.create_task(download_whisper_model(size))
|
||||
|
||||
return {"status": "downloading"}
|
||||
```
|
||||
|
||||
### Delete Model
|
||||
|
||||
```python
|
||||
@app.delete("/models/{model_name}")
|
||||
async def delete_model(model_name: str):
|
||||
# Find and delete from HuggingFace cache
|
||||
cache_dir = Path.home() / ".cache" / "huggingface" / "hub"
|
||||
|
||||
model_dirs = list(cache_dir.glob(f"models--*--{model_name}*"))
|
||||
for model_dir in model_dirs:
|
||||
shutil.rmtree(model_dir)
|
||||
|
||||
return {"status": "deleted"}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/models/status` | Get status of all models |
|
||||
| POST | `/models/load` | Load TTS model |
|
||||
| POST | `/models/unload` | Unload TTS model |
|
||||
| POST | `/models/download` | Trigger model download |
|
||||
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||
| DELETE | `/models/{name}` | Delete downloaded model |
|
||||
| GET | `/tasks/active` | Get active downloads/generations |
|
||||
|
||||
## Response Schemas
|
||||
|
||||
### ModelStatus
|
||||
`GET /models/status` returns every registered model's current state:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen3-TTS 1.7B",
|
||||
"downloaded": true,
|
||||
"size_mb": 3400,
|
||||
"loaded": true
|
||||
}
|
||||
```
|
||||
|
||||
### ActiveTasksResponse
|
||||
|
||||
```json
|
||||
{
|
||||
"downloads": [
|
||||
"models": [
|
||||
{
|
||||
"model_name": "whisper-medium",
|
||||
"status": "downloading",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
],
|
||||
"generations": [
|
||||
{
|
||||
"task_id": "uuid",
|
||||
"profile_id": "uuid",
|
||||
"text_preview": "Hello world...",
|
||||
"started_at": "2024-01-15T10:30:00Z"
|
||||
}
|
||||
"model_name": "qwen-tts-1.7B",
|
||||
"display_name": "Qwen TTS 1.7B",
|
||||
"engine": "qwen",
|
||||
"downloaded": true,
|
||||
"size_mb": 3500,
|
||||
"loaded": true
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend Integration
|
||||
The handler iterates `get_all_model_configs()` and calls `check_model_loaded(config)` for each entry, so new engines appear automatically once they're registered in `ModelConfig`.
|
||||
|
||||
### Progress Display
|
||||
## Manual Model Operations
|
||||
|
||||
```typescript
|
||||
// Subscribe to download progress via SSE
|
||||
const eventSource = new EventSource(`/models/progress/${modelName}`);
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/models/status` | Status of every registered model |
|
||||
| POST | `/models/load` | Load a TTS model into memory |
|
||||
| POST | `/models/unload` | Unload a TTS model from memory |
|
||||
| POST | `/models/download` | Trigger a background download |
|
||||
| GET | `/models/progress/{name}` | Stream download progress (SSE) |
|
||||
| DELETE | `/models/{name}` | Delete a downloaded model from cache |
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
const progress = JSON.parse(event.data);
|
||||
updateProgressBar(progress.current / progress.total);
|
||||
|
||||
if (progress.status === 'complete') {
|
||||
eventSource.close();
|
||||
}
|
||||
};
|
||||
### Load
|
||||
|
||||
```http
|
||||
POST /models/load
|
||||
{
|
||||
"model_name": "qwen-tts-1.7B"
|
||||
}
|
||||
```
|
||||
|
||||
### Model Status UI
|
||||
The route looks up the config, dispatches to `get_model_load_func(config)`, and returns once the model is ready.
|
||||
|
||||
```typescript
|
||||
// Fetch model status
|
||||
const { data: models } = useQuery({
|
||||
queryKey: ['models', 'status'],
|
||||
queryFn: () => api.getModelStatus(),
|
||||
});
|
||||
### Unload
|
||||
|
||||
// Display download/load buttons based on status
|
||||
models.map(model => (
|
||||
<ModelCard
|
||||
name={model.display_name}
|
||||
downloaded={model.downloaded}
|
||||
loaded={model.loaded}
|
||||
onDownload={() => triggerDownload(model.model_name)}
|
||||
onLoad={() => loadModel(model.model_name)}
|
||||
/>
|
||||
));
|
||||
```http
|
||||
POST /models/unload
|
||||
{
|
||||
"model_name": "chatterbox-tts"
|
||||
}
|
||||
```
|
||||
|
||||
Calls `unload_model_by_config(config)`, which routes to the right backend's `unload_model()` and frees GPU memory.
|
||||
|
||||
### Download
|
||||
|
||||
```http
|
||||
POST /models/download
|
||||
{
|
||||
"model_name": "kokoro"
|
||||
}
|
||||
```
|
||||
|
||||
Fires off an async download task. Progress is available via the SSE endpoint. Download is triggered automatically on first generation, so this is only needed for pre-warming.
|
||||
|
||||
## Preset Voice Seeding
|
||||
|
||||
For engines that use preset voices (Kokoro, Qwen CustomVoice), the backend auto-creates a voice profile per preset voice after the model is downloaded. This is driven by `seed_preset_profiles(engine)` in `backend/services/profiles.py`, called from the models route once download completes.
|
||||
|
||||
Preset profiles have:
|
||||
|
||||
- `voice_type = "preset"`
|
||||
- `preset_engine` = engine name (`"kokoro"`, `"qwen_custom_voice"`)
|
||||
- `preset_voice_id` = engine-specific voice ID (`"am_adam"`, `"f000001"`, etc.)
|
||||
- No `profile_samples` rows — no audio to store
|
||||
|
||||
See [Voice Profiles](/developer/voice-profiles) for the schema.
|
||||
|
||||
## Adding a New Model
|
||||
|
||||
To add a new size variant of an existing engine, just add another `ModelConfig`:
|
||||
|
||||
```python
|
||||
ModelConfig(
|
||||
model_name="qwen-tts-3B",
|
||||
display_name="Qwen TTS 3B",
|
||||
engine="qwen",
|
||||
hf_repo_id="Qwen/Qwen3-TTS-12Hz-3B-Base",
|
||||
model_size="3B",
|
||||
size_mb=7000,
|
||||
languages=["zh", "en", ...],
|
||||
),
|
||||
```
|
||||
|
||||
The frontend picks it up via `/models/status`; download/load flow works without further changes.
|
||||
|
||||
Adding a whole new engine is a bigger lift — see [TTS Engines](/developer/tts-engines) for the full phased workflow.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Download failed | Network issue | Retry download |
|
||||
| OOM on load | Model too large | Use smaller model |
|
||||
| Model not found | Cache corrupted | Re-download |
|
||||
| Slow download | HF rate limit | Wait and retry |
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Download failed | Network / HF rate limit | Retry |
|
||||
| OOM on load | Not enough VRAM | Use a smaller variant, unload other engines |
|
||||
| Model not found | Corrupt cache | Re-download via `/models/download` |
|
||||
| Stuck progress bar in frozen build | `huggingface_hub` tqdm silenced | `HFProgressTracker` force-enables the internal counter |
|
||||
| GPU architecture unsupported | PyTorch wheel doesn't target your GPU | See [GPU Acceleration](/overview/gpu-acceleration) |
|
||||
|
||||
## Next Steps
|
||||
|
||||
<Cards>
|
||||
<Card title="TTS Generation" href="/developer/tts-generation">
|
||||
How generation flows through the registry
|
||||
</Card>
|
||||
<Card title="TTS Engines" href="/developer/tts-engines">
|
||||
Add a new engine end-to-end
|
||||
</Card>
|
||||
<Card title="Transcription" href="/developer/transcription">
|
||||
Whisper and MLX-Whisper integration
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
Reference in New Issue
Block a user