Add initial setup for Fumadocs documentation migration

- Created new directory structure for documentation under `/docs2`. - Added `.gitignore` to exclude build artifacts and dependencies. - Introduced `package.json`, `next.config.mjs`, and `postcss.config.mjs` for project configuration. - Implemented MDX components in `mdx-components.tsx` for rendering documentation. - Migrated existing documentation content and created new files for auto-updater and other features. - Established compatibility layer for Mintlify components in `mintlify-compat.tsx`. - Set up OpenAPI documentation in `openapi.json`. - Updated README and migration guide to reflect new structure and usage instructions. - Ensured all components and pages are ready for development and deployment with Fumadocs.
2026-09-18 14:20:42 -07:00 · 2026-01-30 23:32:45 -08:00
parent 9bde534860
commit 64dd29d35a
99 changed files with 9940 additions and 0 deletions
@@ -0,0 +1,283 @@
+---
+title: "TTS Generation"
+description: "How text-to-speech generation works in Voicebox"
+---
+
+## Overview
+
+Voicebox uses Qwen3-TTS for voice cloning and text-to-speech generation. The TTS module handles model loading, voice prompt creation, and audio synthesis.
+
+## Architecture
+
+The TTS system is built around the `TTSModel` class which manages:
+
+**Model Loading:** Lazy loading with automatic HuggingFace Hub download.
+
+**Voice Prompts:** Converting reference audio into embeddings.
+
+**Generation:** Synthesizing speech from text using voice prompts.
+
+## TTSModel Class
+
+```python
+class TTSModel:
+    def __init__(self, model_size: str = "1.7B"):
+        self.model = None
+        self.model_size = model_size
+        self.device = self._get_device()  # cuda, mps, or cpu
+```
+
+### Device Selection
+
+The model automatically selects the best available device:
+
+```python
+def _get_device(self) -> str:
+    if torch.cuda.is_available():
+        return "cuda"
+    elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
+        return "cpu"  # MPS can have issues, use CPU for stability
+    return "cpu"
+```
+
+## Model Loading
+
+Models are downloaded from HuggingFace Hub on first use:
+
+```python
+def load_model(self, model_size: Optional[str] = None):
+    # Model IDs on HuggingFace Hub
+    hf_model_map = {
+        "1.7B": "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
+        "0.6B": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
+    }
+    
+    # Load with progress tracking
+    with tracker.patch_download():
+        self.model = Qwen3TTSModel.from_pretrained(
+            model_path,
+            device_map=self.device,
+            torch_dtype=torch.bfloat16,  # float32 on CPU
+        )
+```
+
+### Async Loading
+
+Loading runs in a thread pool to avoid blocking the event loop:
+
+```python
+async def load_model_async(self, model_size: Optional[str] = None):
+    if self.model is not None and self._current_model_size == model_size:
+        return
+    await asyncio.to_thread(self.load_model, model_size)
+```
+
+## Voice Prompt Creation
+
+Voice prompts are created from reference audio and cached for reuse:
+
+```python
+async def create_voice_prompt(
+    self,
+    audio_path: str,
+    reference_text: str,
+    use_cache: bool = True,
+) -> Tuple[dict, bool]:
+    await self.load_model_async()
+    
+    # Check cache
+    if use_cache:
+        cache_key = get_cache_key(audio_path, reference_text)
+        cached = get_cached_voice_prompt(cache_key)
+        if cached:
+            return cached, True
+    
+    # Create prompt (blocking, run in thread pool)
+    voice_prompt = await asyncio.to_thread(
+        self.model.create_voice_clone_prompt,
+        ref_audio=audio_path,
+        ref_text=reference_text,
+    )
+    
+    # Cache the result
+    cache_voice_prompt(cache_key, voice_prompt)
+    return voice_prompt, False
+```
+
+### Combining Multiple Samples
+
+When a profile has multiple samples, they're combined:
+
+```python
+async def combine_voice_prompts(
+    self,
+    audio_paths: List[str],
+    reference_texts: List[str],
+) -> Tuple[np.ndarray, str]:
+    combined_audio = []
+    
+    for audio_path in audio_paths:
+        audio, sr = load_audio(audio_path)
+        audio = normalize_audio(audio)
+        combined_audio.append(audio)
+    
+    # Concatenate and normalize
+    mixed = np.concatenate(combined_audio)
+    mixed = normalize_audio(mixed)
+    
+    # Combine texts
+    combined_text = " ".join(reference_texts)
+    
+    return mixed, combined_text
+```
+
+## Speech Generation
+
+The core generation function:
+
+```python
+async def generate(
+    self,
+    text: str,
+    voice_prompt: dict,
+    language: str = "en",
+    seed: Optional[int] = None,
+    instruct: Optional[str] = None,
+) -> Tuple[np.ndarray, int]:
+    await self.load_model_async()
+    
+    def _generate_sync():
+        # Set seed for reproducibility
+        if seed is not None:
+            torch.manual_seed(seed)
+        
+        # Generate audio
+        wavs, sample_rate = self.model.generate_voice_clone(
+            text=text,
+            voice_clone_prompt=voice_prompt,
+            instruct=instruct,  # Natural language delivery control
+        )
+        return wavs[0], sample_rate
+    
+    # Run in thread pool
+    return await asyncio.to_thread(_generate_sync)
+```
+
+### Instruct Feature
+
+The `instruct` parameter allows natural language control over speech delivery:
+
+```python
+# Examples:
+instruct = "Speak slowly and clearly"
+instruct = "Sound excited and enthusiastic"
+instruct = "Whisper softly"
+```
+
+## Caching Strategy
+
+Voice prompts are cached to avoid recomputation:
+
+```python
+def get_cache_key(audio_path: str, reference_text: str) -> str:
+    """Generate cache key from audio hash and text."""
+    audio_hash = hashlib.md5(Path(audio_path).read_bytes()).hexdigest()
+    text_hash = hashlib.md5(reference_text.encode()).hexdigest()
+    return f"{audio_hash}_{text_hash}"
+```
+
+Cache is stored in `data/cache/voice_prompts/`.
+
+## Memory Management
+
+### Unloading Models
+
+Free VRAM/RAM when not needed:
+
+```python
+def unload_model(self):
+    if self.model is not None:
+        del self.model
+        self.model = None
+        
+        if torch.cuda.is_available():
+            torch.cuda.empty_cache()
+```
+
+### Model Switching
+
+When switching between model sizes (1.7B ↔ 0.6B):
+
+```python
+# Unload existing model first
+if self.model is not None and self._current_model_size != model_size:
+    self.unload_model()
+```
+
+## Generation Flow
+
+1. **Request** → Validate text and profile ID
+2. **Profile** → Load profile samples from database
+3. **Voice Prompt** → Create or retrieve cached prompt
+4. **Generate** → Run TTS inference
+5. **Save** → Write audio to generations directory
+6. **Record** → Create history entry in database
+7. **Response** → Return audio path and metadata
+
+## API Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| POST | `/generate` | Generate speech from text |
+| GET | `/audio/{id}` | Serve generated audio file |
+
+### Request Schema
+
+```json
+{
+  "profile_id": "uuid",
+  "text": "Text to synthesize",
+  "language": "en",
+  "seed": 42,
+  "model_size": "1.7B",
+  "instruct": "Speak clearly"
+}
+```
+
+### Response Schema
+
+```json
+{
+  "id": "generation_uuid",
+  "profile_id": "profile_uuid",
+  "text": "Text to synthesize",
+  "language": "en",
+  "audio_path": "/path/to/audio.wav",
+  "duration": 3.5,
+  "seed": 42,
+  "instruct": "Speak clearly",
+  "created_at": "2024-01-15T10:30:00Z"
+}
+```
+
+## Performance Considerations
+
+### GPU Acceleration
+
+- CUDA provides fastest inference
+- MPS (Apple Silicon) has stability issues, uses CPU fallback
+- CPU inference is slower but always works
+
+### Batch Size
+
+Currently generates one utterance at a time. For long texts, consider:
+- Splitting into sentences
+- Sequential generation
+- Concatenating results
+
+### Memory Usage
+
+| Model | VRAM/RAM Required |
+|-------|-------------------|
+| 0.6B | ~2GB |
+| 1.7B | ~6GB |