diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..8b3845a0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,68 @@ +# Changelog + +All notable changes to Voicebox will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.1.0] - 2026-01-25 + +### Added + +#### Core Features +- **Voice Cloning** - Clone voices from audio samples using Qwen3-TTS (1.7B and 0.6B models) +- **Voice Profile Management** - Create, edit, and organize voice profiles with multiple samples +- **Speech Generation** - Generate high-quality speech from text using cloned voices +- **Generation History** - Track all generations with search and filtering capabilities +- **Audio Transcription** - Automatic transcription powered by Whisper +- **In-App Recording** - Record audio samples directly in the app with waveform visualization + +#### Desktop App +- **Tauri Desktop App** - Native desktop application for macOS, Windows, and Linux +- **Local Server Mode** - Embedded Python server runs automatically +- **Remote Server Mode** - Connect to a remote Voicebox server on your network +- **Auto-Updates** - Automatic update notifications and installation + +#### API +- **REST API** - Full REST API for voice synthesis and profile management +- **OpenAPI Documentation** - Interactive API docs at `/docs` endpoint +- **Type-Safe Client** - Auto-generated TypeScript client from OpenAPI schema + +#### Technical +- **Voice Prompt Caching** - Fast regeneration with cached voice prompts +- **Multi-Sample Support** - Combine multiple audio samples for better voice quality +- **GPU/CPU/MPS Support** - Automatic device detection and optimization +- **Model Management** - Lazy loading and VRAM management +- **SQLite Database** - Local data persistence + +### Technical Details + +- Built with Tauri v2 (Rust + React) +- FastAPI backend with async Python +- TypeScript frontend with React Query and Zustand +- Qwen3-TTS for voice cloning +- Whisper for transcription + +### Platform Support + +- macOS (Apple Silicon and Intel) +- Windows +- Linux (AppImage) + +--- + +## [Unreleased] + +### Planned +- Real-time streaming synthesis +- Conversation mode with multiple speakers +- Voice effects (pitch shift, reverb, M3GAN-style) +- Timeline-based audio editor +- Additional voice models (XTTS, Bark) +- Voice design from text descriptions +- Project system for saving sessions +- Plugin architecture + +--- + +[0.1.0]: https://github.com/jamiepine/voicebox/releases/tag/v0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..dbadfd95 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,252 @@ +# Contributing to Voicebox + +Thank you for your interest in contributing to Voicebox! This document provides guidelines and instructions for contributing. + +## Code of Conduct + +- Be respectful and inclusive +- Welcome newcomers and help them learn +- Focus on constructive feedback +- Respect different viewpoints and experiences + +## Getting Started + +### Prerequisites + +- [Bun](https://bun.sh) - Package manager +- [Rust](https://rustup.rs) - For Tauri desktop app +- [Python 3.11+](https://python.org) - For backend +- Git + +### Development Setup + +1. **Fork and clone the repository** + ```bash + git clone https://github.com/YOUR_USERNAME/voicebox.git + cd voicebox + ``` + +2. **Install dependencies** + ```bash + bun install + cd backend && pip install -r requirements.txt && cd .. + ``` + +3. **Set up the database** + ```bash + cd backend + python -c "from database import init_db; init_db()" + ``` + +4. **Start development** + ```bash + # Terminal 1: Backend server + bun run dev:server + + # Terminal 2: Desktop app + bun run dev + ``` + +## Development Workflow + +### 1. Create a Branch + +```bash +git checkout -b feature/your-feature-name +# or +git checkout -b fix/your-bug-fix +``` + +### 2. Make Your Changes + +- Write clean, readable code +- Follow existing code style +- Add comments for complex logic +- Update documentation as needed + +### 3. Test Your Changes + +- Test manually in the app +- Ensure backend API endpoints work +- Check for TypeScript/Python errors +- Verify UI components render correctly + +### 4. Commit Your Changes + +Write clear, descriptive commit messages: + +```bash +git commit -m "Add feature: voice profile export" +git commit -m "Fix: audio playback stops after 30 seconds" +``` + +### 5. Push and Create Pull Request + +```bash +git push origin feature/your-feature-name +``` + +Then create a pull request on GitHub with: +- Clear description of changes +- Screenshots (for UI changes) +- Reference to related issues + +## Code Style + +### TypeScript/React + +- Use TypeScript strict mode +- Follow React best practices +- Use functional components with hooks +- Prefer named exports +- Format with Biome (runs automatically) + +```typescript +// Good +export function ProfileCard({ profile }: { profile: Profile }) { + return
{profile.name}
; +} + +// Avoid +export const ProfileCard = (props) => { ... } +``` + +### Python + +- Follow PEP 8 style guide +- Use type hints +- Use async/await for I/O operations +- Format with Black (if configured) + +```python +# Good +async def create_profile(name: str, language: str) -> Profile: + """Create a new voice profile.""" + ... + +# Avoid +def create_profile(name, language): + ... +``` + +### Rust + +- Follow Rust conventions +- Use meaningful variable names +- Handle errors explicitly +- Format with `rustfmt` + +## Project Structure + +``` +voicebox/ +├── app/ # Shared React frontend +│ └── src/ +│ ├── components/ # UI components +│ ├── lib/ # Utilities and API client +│ └── hooks/ # React hooks +├── backend/ # Python FastAPI server +│ ├── main.py # API routes +│ ├── tts.py # Voice synthesis +│ └── ... +├── tauri/ # Desktop app wrapper +│ └── src-tauri/ # Rust backend +└── scripts/ # Build scripts +``` + +## Areas for Contribution + +### 🐛 Bug Fixes + +- Check existing issues for bugs to fix +- Test your fix thoroughly +- Add tests if possible + +### ✨ New Features + +- Check the roadmap in README.md +- Discuss major features in an issue first +- Keep features focused and well-scoped + +### 📚 Documentation + +- Improve README clarity +- Add code comments +- Write API documentation +- Create tutorials or guides + +### 🎨 UI/UX Improvements + +- Improve accessibility +- Enhance visual design +- Optimize performance +- Add animations/transitions + +### 🔧 Infrastructure + +- Improve build process +- Add CI/CD improvements +- Optimize bundle size +- Add testing infrastructure + +## API Development + +When adding new API endpoints: + +1. **Add route in `backend/main.py`** +2. **Create Pydantic models in `backend/models.py`** +3. **Implement business logic in appropriate module** +4. **Update OpenAPI schema** (automatic with FastAPI) +5. **Regenerate TypeScript client:** + ```bash + bun run generate:api + ``` +6. **Update `backend/README.md`** with endpoint documentation + +## Testing + +Currently, testing is primarily manual. When adding tests: + +- **Backend**: Use pytest for Python tests +- **Frontend**: Use Vitest for React component tests +- **E2E**: Use Playwright for end-to-end tests (future) + +## Pull Request Process + +1. **Update documentation** if needed +2. **Ensure code follows style guidelines** +3. **Test your changes thoroughly** +4. **Update CHANGELOG.md** with your changes +5. **Request review** from maintainers + +### PR Checklist + +- [ ] Code follows style guidelines +- [ ] Documentation updated +- [ ] Changes tested +- [ ] No breaking changes (or documented) +- [ ] CHANGELOG.md updated + +## Release Process + +Releases are managed by maintainers: + +1. Version bump in `tauri.conf.json` and `Cargo.toml` +2. Update CHANGELOG.md +3. Create git tag: `git tag v0.2.0` +4. Push tag: `git push --tags` +5. GitHub Actions builds and releases + +## Questions? + +- Open an issue for bugs or feature requests +- Check existing issues and discussions +- Review the codebase to understand patterns + +## License + +By contributing, you agree that your contributions will be licensed under the MIT License. + +--- + +Thank you for contributing to Voicebox! 🎉 diff --git a/CURRENT_STATE.md b/CURRENT_STATE.md deleted file mode 100644 index 975b1447..00000000 --- a/CURRENT_STATE.md +++ /dev/null @@ -1,447 +0,0 @@ -# voicebox - Current State Overview - -**Last Updated:** January 25, 2026 -**Status:** ✅ MVP Core Features Working - Voice generation from Tauri app successful! - ---- - -## 🎯 What We Have - -### ✅ **Fully Implemented & Working** - -#### **Backend (Python FastAPI)** -- **Voice Profile Management** - - Create, read, update, delete profiles - - Add multiple audio samples per profile - - Multi-reference voice combination (combines multiple samples) - - Profile storage in SQLite + file system (`data/profiles/`) - -- **Voice Generation** - - Qwen3-TTS model integration (1.7B and 0.6B support) - - Automatic model downloading from HuggingFace Hub - - Voice prompt caching for instant re-generation - - Support for English and Chinese - - Seed-based reproducibility - - GPU/CPU/MPS device detection - -- **Generation History** - - Full CRUD operations - - Search by text content - - Filter by profile - - Pagination support - - Statistics endpoint - - Audio file storage (`data/generations/`) - -- **Audio Transcription** - - Whisper integration for speech-to-text - - Language detection/selection - - Used for reference text extraction from samples - -- **Database** - - SQLite with SQLAlchemy ORM - - Tables: `profiles`, `profile_samples`, `generations`, `projects` (ready for future) - - Automatic schema initialization - -- **API Endpoints** - - RESTful API with FastAPI - - OpenAPI schema generation - - CORS enabled - - Health check endpoint - - File serving for audio files - -#### **Frontend (React + TypeScript + Tauri)** -- **Voice Profile UI** - - Profile list with cards - - Create/edit profile dialog - - Upload audio samples with transcription - - Sample management (view/delete) - - Profile detail view - -- **Generation UI** - - Form with profile selection - - Text input (up to 5000 chars) - - Language selection (en/zh) - - Optional seed input - - Loading states and error handling - -- **History UI** - - Table view with pagination - - Search functionality - - Play audio inline - - Download audio files - - Delete generations - -- **Server Settings** - - Connection form (local/remote mode) - - Server status display - - Health check integration - -- **State Management** - - React Query for server state - - Zustand for client state (server URL, connection status) - - Type-safe API client - -- **UI Components** - - shadcn/ui component library - - Tailwind CSS styling - - Responsive design - - Toast notifications - - Form validation with Zod - -#### **Tauri Desktop App** -- **Rust Backend** - - Sidecar management for Python server - - Start/stop server commands - - Remote mode support (0.0.0.0 binding) - - Process lifecycle management - -- **Build System** - - Tauri v2 configuration - - Platform-specific builds - - Dev tools in debug mode - ---- - -## 🏗️ Architecture - -### **Project Structure** -``` -voicebox/ -├── app/ # Shared React frontend -│ ├── src/ -│ │ ├── components/ # React components -│ │ │ ├── VoiceProfiles/ ✅ Complete -│ │ │ ├── Generation/ ✅ Complete -│ │ │ ├── History/ ✅ Complete -│ │ │ ├── ServerSettings/ ✅ Complete -│ │ │ └── AudioStudio/ 📦 Placeholder (future) -│ │ ├── lib/ -│ │ │ ├── api/ # Type-safe API client ✅ -│ │ │ ├── hooks/ # React Query hooks ✅ -│ │ │ └── utils/ # Utilities ✅ -│ │ └── stores/ # Zustand stores ✅ -│ -├── backend/ # Python FastAPI server -│ ├── main.py # FastAPI app + routes ✅ -│ ├── models.py # Pydantic models ✅ -│ ├── database.py # SQLAlchemy ORM ✅ -│ ├── profiles.py # Profile management ✅ -│ ├── history.py # History management ✅ -│ ├── tts.py # Qwen3-TTS integration ✅ -│ ├── transcribe.py # Whisper integration ✅ -│ ├── studio.py # Audio studio (future) -│ └── utils/ -│ ├── audio.py # Audio processing ✅ -│ ├── cache.py # Voice prompt caching ✅ -│ └── validation.py # Validation helpers ✅ -│ -├── tauri/ # Tauri desktop wrapper -│ ├── src/ # React entry point ✅ -│ └── src-tauri/ # Rust backend ✅ -│ └── src/main.rs # Sidecar management ✅ -│ -├── data/ # User data directory -│ ├── profiles/ # Profile audio samples -│ ├── generations/ # Generated audio files -│ ├── cache/ # Cached voice prompts -│ └── voicebox.db # SQLite database -│ -└── scripts/ # Build & generation scripts - ├── generate-api.sh # OpenAPI client generation - └── build-server.sh # Python binary build -``` - -### **Data Flow** - -``` -User Action (Tauri App) - ↓ -React Component (Form Submit) - ↓ -React Query Hook (useGeneration) - ↓ -API Client (apiClient.generateSpeech) - ↓ -HTTP Request → FastAPI Backend - ↓ -Backend Route Handler (/generate) - ↓ -Business Logic: - 1. Get profile from DB - 2. Create voice prompt (with caching) - 3. Generate audio with Qwen3-TTS - 4. Save audio file - 5. Create history entry - ↓ -Response (GenerationResponse) - ↓ -React Query Cache Update - ↓ -UI Refresh (History table updates) -``` - -### **Key Technologies** - -| Layer | Technology | Purpose | -|-------|-----------|---------| -| **Desktop Framework** | Tauri v2 | Native desktop app wrapper | -| **Frontend Framework** | React 18 | UI components | -| **Language** | TypeScript | Type safety | -| **Styling** | Tailwind CSS | Utility-first CSS | -| **UI Components** | shadcn/ui | Component library | -| **State Management** | React Query + Zustand | Server & client state | -| **Form Handling** | React Hook Form + Zod | Form validation | -| **Backend Framework** | FastAPI | Async REST API | -| **Database** | SQLite + SQLAlchemy | Data persistence | -| **ML Models** | Qwen3-TTS + Whisper | Voice cloning + transcription | -| **Audio Processing** | librosa + soundfile | Audio I/O and processing | -| **Package Manager** | Bun | Fast JS/TS package management | -| **Build Tool** | Vite | Frontend bundling | - ---- - -## 🔑 Key Features & Capabilities - -### **1. Voice Profile System** -- **Multi-sample support**: Add multiple audio samples per profile -- **Automatic combination**: Multiple samples are combined for better quality -- **Voice prompt caching**: Re-use voice prompts for instant re-generation -- **Audio validation**: Ensures samples meet quality requirements - -### **2. Generation Pipeline** -- **Lazy model loading**: Model loads on first use -- **Device detection**: Automatically uses GPU if available -- **Caching layer**: Voice prompts cached by audio hash + text -- **Error handling**: Graceful degradation and clear error messages - -### **3. History & Search** -- **Full-text search**: Search generations by text content -- **Pagination**: Efficient loading of large histories -- **Audio playback**: Inline audio player -- **File management**: Download and delete operations - -### **4. Server/Client Architecture** -- **Local mode**: Backend runs alongside Tauri app -- **Remote mode**: Connect to remote GPU machine -- **One-click server**: Start server from UI -- **Connection management**: Persistent server URL storage - ---- - -## 📊 Database Schema - -### **Tables** - -```sql --- Voice Profiles -profiles - - id (PK, UUID) - - name (unique) - - description - - language (en/zh) - - created_at - - updated_at - --- Profile Samples -profile_samples - - id (PK, UUID) - - profile_id (FK → profiles.id) - - audio_path - - reference_text - --- Generations -generations - - id (PK, UUID) - - profile_id (FK → profiles.id) - - text - - language - - audio_path - - duration (seconds) - - seed (optional) - - created_at - --- Projects (ready for future) -projects - - id (PK, UUID) - - name - - data (JSON) - - created_at - - updated_at -``` - ---- - -## 🎨 UI Components Status - -| Component | Status | Features | -|-----------|--------|----------| -| **ProfileList** | ✅ Complete | List, create, empty state | -| **ProfileCard** | ✅ Complete | Display profile info | -| **ProfileForm** | ✅ Complete | Create/edit dialog | -| **ProfileDetail** | ✅ Complete | View samples, add samples | -| **SampleUpload** | ✅ Complete | File upload + transcription | -| **GenerationForm** | ✅ Complete | Full generation form | -| **HistoryTable** | ✅ Complete | Table, search, pagination, play/download | -| **ConnectionForm** | ✅ Complete | Server URL input | -| **ServerStatus** | ✅ Complete | Health check display | -| **AudioStudio** | 📦 Placeholder | Timeline editor (future) | - ---- - -## 🔌 API Endpoints - -### **Profiles** -- `POST /profiles` - Create profile -- `GET /profiles` - List all profiles -- `GET /profiles/{id}` - Get profile -- `PUT /profiles/{id}` - Update profile -- `DELETE /profiles/{id}` - Delete profile -- `POST /profiles/{id}/samples` - Add sample -- `GET /profiles/{id}/samples` - List samples -- `DELETE /profiles/samples/{id}` - Delete sample - -### **Generation** -- `POST /generate` - Generate speech - -### **History** -- `GET /history` - List generations (with filters) -- `GET /history/{id}` - Get generation -- `DELETE /history/{id}` - Delete generation -- `GET /history/stats` - Get statistics - -### **Transcription** -- `POST /transcribe` - Transcribe audio - -### **Audio** -- `GET /audio/{id}` - Serve audio file - -### **Health** -- `GET /health` - Health check with model status - -### **Model Management** -- `POST /models/load` - Load TTS model -- `POST /models/unload` - Unload TTS model - ---- - -## 🚀 What's Next (Planned Features) - -### **Phase 2: Advanced Features** -- [ ] Multi-reference voice combination UI -- [ ] Batch generation (multiple variations) -- [ ] Advanced audio normalization -- [ ] Export options (MP3, OGG, etc.) -- [ ] M3GAN voice effect - -### **Phase 3: Audio Studio** -- [ ] Timeline-based audio editor -- [ ] Word-level timestamps -- [ ] Project system (save/load sessions) -- [ ] Audio effects and filters -- [ ] Multi-track editing - -### **Phase 4: Voice Design** -- [ ] Text-to-voice (no reference needed) -- [ ] Preset voices with style control -- [ ] Conversation mode (multi-speaker) -- [ ] Custom audio effects library - ---- - -## 📝 Code Quality Standards - -- ✅ **Type safety**: TypeScript strict mode, Pydantic models -- ✅ **Modular architecture**: No files over 500 lines -- ✅ **Error handling**: Comprehensive error messages -- ✅ **Caching**: Voice prompt caching for performance -- ✅ **Database**: SQLAlchemy ORM with proper relationships -- ✅ **API design**: RESTful with OpenAPI schema -- ✅ **UI/UX**: Responsive, accessible, loading states - ---- - -## 🧪 Testing Status - -- ✅ **Manual testing**: Voice generation working end-to-end -- 📦 **Unit tests**: Not yet implemented -- 📦 **Integration tests**: Not yet implemented -- 📦 **E2E tests**: Not yet implemented - ---- - -## 📦 Dependencies - -### **Backend** -- FastAPI - Web framework -- SQLAlchemy - ORM -- Pydantic - Validation -- Qwen3-TTS - Voice cloning model -- Whisper - Speech recognition -- librosa - Audio processing -- soundfile - Audio I/O -- PyTorch - ML framework - -### **Frontend** -- React 18 - UI framework -- TypeScript - Type safety -- React Query - Server state -- Zustand - Client state -- React Hook Form - Forms -- Zod - Schema validation -- Tailwind CSS - Styling -- shadcn/ui - Components -- Lucide React - Icons - -### **Desktop** -- Tauri v2 - Desktop framework -- Rust - System backend - ---- - -## 🎯 Current Capabilities Summary - -✅ **Working End-to-End:** -1. Create voice profiles with audio samples -2. Generate speech from text using cloned voices -3. View and manage generation history -4. Play and download generated audio -5. Search and filter history -6. Connect to local or remote backend -7. Automatic model downloading -8. Voice prompt caching for speed - -🎉 **You just successfully generated voice from the Tauri app!** - ---- - -## 🔍 Key Files Reference - -### **Backend Core** -- `backend/main.py` - FastAPI app and routes -- `backend/tts.py` - Qwen3-TTS model wrapper -- `backend/profiles.py` - Profile business logic -- `backend/history.py` - History business logic -- `backend/database.py` - Database models - -### **Frontend Core** -- `app/src/App.tsx` - Main app component -- `app/src/lib/api/client.ts` - API client -- `app/src/lib/hooks/` - React Query hooks -- `app/src/stores/` - Zustand stores - -### **Tauri** -- `tauri/src-tauri/src/main.rs` - Rust backend -- `tauri/src/main.tsx` - React entry point - ---- - -## 💡 Development Workflow - -1. **Start backend**: `bun run dev:server` (or via Tauri) -2. **Start frontend**: `bun run dev` (Tauri) or `bun run dev:web` (web) -3. **Generate API client**: `bun run generate:api` (after backend changes) -4. **Build server binary**: `bun run build:server` (for Tauri bundling) - ---- - -**Ready to build more features! 🚀** diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..5eea3a01 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Voicebox Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 86835a64..e482939a 100644 --- a/README.md +++ b/README.md @@ -182,14 +182,9 @@ Voicebox aims to be the **one-stop shop for everything voice** — cloning, synt ## Development -### Prerequisites +See [SETUP.md](SETUP.md) for detailed setup instructions. -- [Bun](https://bun.sh) (package manager) -- [Rust](https://rustup.rs) (for Tauri) -- [Python 3.11+](https://python.org) (for backend) -- CUDA-capable GPU recommended (CPU inference supported but slower) - -### Setup +### Quick Start ```bash # Clone the repo @@ -206,6 +201,8 @@ cd backend && pip install -r requirements.txt && cd .. bun run dev ``` +**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org). CUDA-capable GPU recommended (CPU inference supported but slower). + ### Project Structure ``` @@ -222,18 +219,22 @@ voicebox/ ## Contributing -Contributions welcome! Whether it's bug fixes, new features, or documentation improvements. +Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. 1. Fork the repo 2. Create a feature branch 3. Make your changes 4. Submit a PR +## Security + +Found a security vulnerability? Please report it responsibly. See [SECURITY.md](SECURITY.md) for details. + --- ## License -MIT License — use it however you want. +MIT License — see [LICENSE](LICENSE) for details. --- diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..bdf1e917 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,92 @@ +# Security Policy + +## Supported Versions + +We release patches for security vulnerabilities. Which versions are eligible for receiving such patches depends on the CVSS v3.0 Rating: + +| Version | Supported | +| ------- | ------------------ | +| 0.1.x | :white_check_mark: | +| < 0.1 | :x: | + +## Reporting a Vulnerability + +If you discover a security vulnerability, please report it responsibly: + +1. **Do not** open a public GitHub issue +2. Email security details to: [security@voicebox.sh](mailto:security@voicebox.sh) +3. Include: + - Description of the vulnerability + - Steps to reproduce + - Potential impact + - Suggested fix (if any) + +We will: +- Acknowledge receipt within 48 hours +- Provide a timeline for addressing the issue +- Keep you informed of progress +- Credit you in the security advisory (if desired) + +## Security Best Practices + +### For Users + +- **Keep Voicebox updated** - Updates include security patches +- **Verify downloads** - Only download from official releases +- **Local processing** - Voice data stays on your machine +- **Network security** - Use HTTPS when connecting to remote servers + +### For Developers + +- **Dependencies** - Keep all dependencies up to date +- **Code review** - All PRs require review before merging +- **Secrets** - Never commit API keys or signing keys +- **Signing** - All releases are cryptographically signed + +## Known Security Considerations + +### Local Processing + +Voicebox processes all audio locally by default. Your voice data never leaves your machine unless you explicitly enable remote server mode. + +### Remote Server Mode + +When connecting to a remote server: +- Ensure the server is on a trusted network +- Use HTTPS for remote connections +- Verify server identity before connecting + +### Auto-Updates + +- Updates are cryptographically signed +- Signature verification happens before installation +- Only HTTPS endpoints are allowed + +### Python Server + +The embedded Python server: +- Runs locally by default (localhost only) +- Can be configured for remote access +- Uses standard FastAPI security practices + +## Disclosure Timeline + +- **Day 0**: Vulnerability reported +- **Day 1-2**: Initial assessment and acknowledgment +- **Day 3-7**: Investigation and fix development +- **Day 8-14**: Testing and release preparation +- **Day 15+**: Public disclosure (if applicable) + +Timeline may vary based on severity and complexity. + +## Security Updates + +Security updates will be: +- Released as patch versions (e.g., 0.1.1) +- Documented in CHANGELOG.md +- Announced via GitHub releases +- Automatically delivered via auto-updater + +--- + +Thank you for helping keep Voicebox secure! 🔒 diff --git a/SETUP.md b/SETUP.md index b99b6580..5bc265ab 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,6 +1,6 @@ -# voicebox Setup Guide +# Development Setup Guide -Quick start guide for setting up the voicebox development environment. +This guide will help you set up the Voicebox development environment. ## Prerequisites @@ -174,20 +174,23 @@ voicebox/ ## Troubleshooting -### Backend won't start +See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues and solutions. + +### Quick Fixes + +**Backend won't start:** - Check Python version: `python --version` (needs 3.11+) - Ensure virtual environment is activated - Install dependencies: `pip install -r requirements.txt` -### Tauri build fails +**Tauri build fails:** - Ensure Rust is installed: `rustc --version` - Install Tauri CLI: `bunx @tauri-apps/cli install` -- Check `tauri/src-tauri/Cargo.toml` for correct dependencies +- Clean build: `cd tauri/src-tauri && cargo clean` -### OpenAPI client generation fails +**OpenAPI client generation fails:** - Ensure backend is running on port 8000 - Check `curl http://localhost:8000/openapi.json` returns valid JSON -- Install openapi-typescript-codegen: `bun add -d openapi-typescript-codegen` ## Model Downloads @@ -199,9 +202,13 @@ First-time usage will be slower due to model downloads, but subsequent runs will ## Next Steps -1. ✅ TTS model loading implemented in `backend/tts.py` -2. ✅ API routes implemented in `backend/main.py` -3. Build React components in `app/src/components/` -4. Connect frontend to backend via generated API client +- Read [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines +- Check [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) if you encounter issues +- Review [backend/README.md](backend/README.md) for API documentation +- See [README.md](README.md) for project overview -See [README.md](./README.md) for architecture details and [docs/](./docs/) for detailed documentation. +## Additional Resources + +- [Auto-Updater Setup](docs/AUTOUPDATER_QUICKSTART.md) - Configure automatic updates +- [Security Policy](SECURITY.md) - Security reporting and best practices +- [Changelog](CHANGELOG.md) - Version history and changes diff --git a/docs/ANALYSIS.md b/docs/ANALYSIS.md deleted file mode 100644 index da3cac90..00000000 --- a/docs/ANALYSIS.md +++ /dev/null @@ -1,1215 +0,0 @@ -# Qwen3-TTS Implementation Analysis - -Comprehensive analysis of five existing Qwen3-TTS implementations to inform the architecture of voicebox. - ---- - -## Projects Analyzed - -1. **voice** - Rust CLI with Python backend -2. **Voice-Clone-Studio** - Feature-rich Gradio app -3. **Qwen3-TTS_server** - FastAPI REST API wrapper -4. **mimic** - Web app with audio studio (conceptually the best) -5. **qwen3-tts-enhanced** - Production-quality single-file Gradio app - ---- - -## 1. voice (Rust CLI) - -**Repository:** `/Users/jamespine/Projects/voice` - -### Architecture - -**Dual-mode design:** -- One-shot mode: Single generation via CLI args -- Server mode: Long-running HTTP server (port 3000) - -**Language split:** -- Rust: CLI, HTTP server, IPC orchestration -- Python: TTS inference, model management - -**IPC Pattern:** -``` -Rust → spawn Python subprocess → JSON over stdin/stdout → Rust -``` - -### Python Backend (`tts.py`) - -**Model Management:** -- Lazy loading via `get_tts_model()` -- Qwen3-TTS CustomVoice models (0.6B/1.7B) -- `torch_dtype=torch.bfloat16` for VRAM efficiency -- Flash Attention 2 with SDPA fallback - -**Voice Profile System:** -- Profiles stored in `~/.config/voice/profiles/` -- Each profile: `audio.wav` + `reference.txt` -- Automatic voice prompt creation and caching -- Hash-based cache invalidation (MD5 of audio + text) - -**Generation Pipeline:** -```python -1. Load model (lazy, cached) -2. Get voice prompt (cached if available) -3. Set seed for reproducibility -4. model.generate_voice_clone(text, language, voice_prompt) -5. Save to temp file -6. Return path as JSON -``` - -**Special Features:** -- M3GAN voice effect (pitch shift + formant preservation) -- Uses `librosa` + `soundfile` for audio processing -- Voice profile listing with metadata - -### Rust Frontend - -**HTTP Server (`main.rs`):** -- Axum web framework -- Single endpoint: `POST /generate` -- JSON request/response -- Spawns Python subprocess per request - -**CLI (`cli.rs`):** -- Clap for argument parsing -- Commands: `generate`, `server`, `list-voices` -- Profile management integrated - -**Process Management:** -```rust -let mut child = Command::new("python3") - .arg("tts.py") - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .spawn()?; -``` - -### Strengths - -1. **Clean separation of concerns** - Rust for I/O, Python for ML -2. **Dual-mode flexibility** - CLI and server in one binary -3. **Voice profile abstraction** - Easy to add/manage voices -4. **M3GAN effect** - Unique feature, well-implemented -5. **Good error handling** - Rust's `Result` type enforced - -### Weaknesses - -1. **Synchronous inference** - Blocks during generation -2. **No concurrent requests** - Server spawns subprocess per request -3. **Limited caching** - Only voice prompts, not models -4. **No generation history** - Fire and forget -5. **Basic HTTP API** - No auth, rate limiting, or WebSocket streaming - -### Key Learnings - -- IPC via JSON stdin/stdout is simple and works -- Voice profile pattern is user-friendly -- For Tauri, we can skip the IPC layer and use direct HTTP/IPC channels -- M3GAN effect is a differentiator worth preserving - ---- - -## 2. Voice-Clone-Studio - -**Repository:** `/Users/jamespine/Projects/Voice-Clone-Studio` - -### Architecture - -**Monolithic Gradio App:** -- Single file: `voice_clone_studio.py` (2,815 lines) -- Gradio for web UI -- Global state for model management -- Tab-based organization (6 tabs) - -### Model Support - -**Dual Engine:** -1. **Qwen3-TTS** - Base, CustomVoice, VoiceDesign -2. **VibeVoice TTS** - 1.5B/Large, multi-speaker - -**Model sizes:** -- Qwen Small: 0.6B (~1GB VRAM) -- Qwen Large: 1.7B (~3GB VRAM) -- VibeVoice Small: 1.5B (~3GB VRAM) -- VibeVoice Large: ~6GB VRAM -- VibeVoice ASR: 7B (~14GB VRAM) - -### Voice Prompt Caching - -**Smart caching system:** -```python -def get_or_create_voice_prompt(audio_path, reference_text): - cache_key = hashlib.md5(audio_bytes + text.encode()).hexdigest() - - # Check in-memory cache - if cache_key in _voice_prompt_cache: - return _voice_prompt_cache[cache_key] - - # Check disk cache - prompt_file = f"{audio_path}.{cache_key}.prompt" - if os.path.exists(prompt_file): - prompt = torch.load(prompt_file) - _voice_prompt_cache[cache_key] = prompt - return prompt - - # Create new - prompt = model.create_voice_clone_prompt(audio, text) - torch.save(prompt, prompt_file) - _voice_prompt_cache[cache_key] = prompt - return prompt -``` - -**Cache invalidation:** -- Hash changes if audio or text changes -- Orphaned `.prompt` files cleaned up on demand - -### Features - -**Tab 1: Voice Clone** -- Clone from samples with Qwen or VibeVoice -- Sample selection dropdown -- Language selection (en/zh) -- Seed control for reproducibility -- Model size selection - -**Tab 2: Conversation** -- Multi-speaker dialogue generation -- Script format: `[1]: Text`, `[2]: Text` -- Automatic speaker assignment -- Pause duration control between speakers - -**Tab 3: Voice Presets** -- 9 pre-built Qwen speakers -- Style control (narrative, conversational, etc.) -- No sample needed - -**Tab 4: Voice Design** -- Generate voices from text descriptions -- "Young female, energetic, bright tone" -- Qwen VoiceDesign model (1.7B only) - -**Tab 5: Prep Samples** -- Audio/video file upload -- Auto-transcription (Whisper or VibeVoice ASR) -- Audio editing (trim, normalize, mono conversion) -- Save as voice sample - -**Tab 6: Output History** -- Browse generated files -- Metadata display (timestamp, seed, engine, text) -- Re-generate from metadata - -### VRAM Management - -**Lazy loading + mutual exclusion:** -```python -def get_tts_model(): - global _tts_model, _whisper_model, _vibe_voice_model - - # Unload ASR to free VRAM - if _whisper_model: - del _whisper_model - _whisper_model = None - if _vibe_voice_model: - del _vibe_voice_model - _vibe_voice_model = None - - torch.cuda.empty_cache() - - if not _tts_model: - _tts_model = load_model(...) - - return _tts_model -``` - -### Audio Processing - -**Pipeline:** -1. **Input normalization:** - - Peak normalization to [-1, 1] - - 0.85 scaling (15% headroom) - - Stereo → mono (average channels) - -2. **Video extraction:** - - ffmpeg subprocess for audio extraction - - 24kHz mono output - -3. **Transcription:** - - Whisper medium model - - VibeVoice ASR with speaker diarization - - Auto-cleans `[Speaker X]:` labels - -4. **Generation:** - - Seed control via `torch.manual_seed()` - - bfloat16 inference - - Direct .wav output (soundfile) - -### Strengths - -1. **Voice prompt caching** - Brilliant UX (⚡ cached indicator) -2. **Dual engine support** - Qwen + VibeVoice flexibility -3. **Feature-rich** - Voice design, presets, conversations, long-form -4. **VRAM efficiency** - Smart loading/unloading -5. **Video support** - Extract audio from video files -6. **Metadata tracking** - Every output has reproducibility data -7. **Sample management** - Integrated prep workspace - -### Weaknesses - -1. **2,815-line single file** - Impossible to maintain -2. **Global state everywhere** - Testing nightmare -3. **Duplicated code** - 5 model loaders with identical fallback logic -4. **No separation of concerns** - UI + logic + audio all mixed -5. **No concurrency** - All operations block UI -6. **No error recovery** - Generic error messages -7. **Hardcoded parameters** - CFG scale, inference steps in function bodies -8. **No tests** - Zero test coverage -9. **No logging** - Only print statements - -### Key Learnings - -**Adopt:** -- Voice prompt caching with hash validation -- Lazy model loading with automatic unloading -- Metadata alongside outputs -- Sample management patterns -- Status indicators (cached vs not) - -**Avoid:** -- Monolithic files over 2,000 lines -- Global mutable state -- Duplicated logic without abstraction -- Hardcoded parameters - ---- - -## 3. Qwen3-TTS_server - -**Repository:** `/Users/jamespine/Projects/Qwen3-TTS_server` - -### Architecture - -**FastAPI REST API:** -- Single server process -- Three endpoints: `/generate`, `/clone`, `/health` -- Deployed on RunPod with Docker -- Port 8000 (configurable) - -### Project Structure - -``` -Qwen3-TTS_server/ -├── main.py # FastAPI app -├── models/ -│ └── tts.py # Model management -├── api/ -│ └── routes.py # Endpoint definitions -├── utils/ -│ └── audio.py # Audio processing -├── config.py # Settings -├── Dockerfile # Multi-stage build -└── requirements.txt # Dependencies -``` - -### API Design - -**POST /generate** -```json -{ - "text": "Hello world", - "language": "en", - "speaker": "default", - "seed": 42 -} - -Response: -{ - "audio_url": "https://...", - "duration": 2.5, - "sample_rate": 24000 -} -``` - -**POST /clone** -```json -{ - "text": "Hello world", - "language": "en", - "reference_audio": "base64_encoded_audio", - "reference_text": "This is my voice", - "seed": 42 -} - -Response: -{ - "audio_url": "https://...", - "duration": 2.5, - "sample_rate": 24000 -} -``` - -**GET /health** -```json -{ - "status": "healthy", - "model_loaded": true, - "gpu_available": true, - "vram_used_mb": 1024 -} -``` - -### Model Management - -**Singleton pattern:** -```python -class ModelManager: - _instance = None - _model = None - - def __new__(cls): - if not cls._instance: - cls._instance = super().__new__(cls) - return cls._instance - - def get_model(self): - if not self._model: - self._model = self._load_model() - return self._model -``` - -**Lazy loading:** -- Model loaded on first request -- Kept in memory for subsequent requests -- No unloading (server dedicated to TTS) - -### Deployment - -**Docker multi-stage build:** -```dockerfile -FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base -# Install dependencies - -FROM base AS builder -# Install Python packages - -FROM base AS runtime -COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 -# Runtime only -``` - -**RunPod integration:** -- Configured for GPU instances -- Automatic model download on startup -- Health checks for orchestration - -### Strengths - -1. **Clean API design** - RESTful, well-documented -2. **Proper separation** - Routes, models, utils in separate modules -3. **Singleton model manager** - Better than global variables -4. **Health endpoint** - Essential for deployment -5. **Docker deployment** - Production-ready containerization -6. **Base64 audio input** - No file upload needed for cloning - -### Weaknesses - -1. **No authentication** - Wide open API -2. **No rate limiting** - DoS vulnerable -3. **Sequential inference** - No request queuing -4. **No caching** - Voice prompts recreated every time -5. **No streaming** - Returns only after full generation -6. **No WebSocket** - Can't send progress updates -7. **Synchronous endpoints** - Blocks during generation -8. **No storage** - Audio URLs expire quickly -9. **Limited error handling** - Generic 500 errors - -### Key Learnings - -**Adopt:** -- FastAPI for REST API -- Modular structure (routes, models, utils) -- Health endpoint for monitoring -- Base64 audio input option -- Docker deployment pattern - -**Improve:** -- Add async/await throughout -- Implement request queue -- Add WebSocket for streaming -- Cache voice prompts -- Authentication and rate limiting - ---- - -## 4. mimic - -**Repository:** `/Users/jamespine/Projects/mimic` - -### Architecture - -**Three-tier web app:** -- Frontend: Vanilla JS (no framework) -- Backend: Python FastAPI -- Database: SQLite - -**Best-structured backend** of all projects analyzed. - -### Project Structure - -``` -mimic/ -├── backend/ -│ ├── main.py # FastAPI app -│ ├── models.py # Pydantic models -│ ├── tts.py # TTS inference -│ ├── transcribe.py # Whisper ASR -│ ├── profiles.py # Voice profile management -│ ├── history.py # Generation history -│ ├── studio.py # Audio studio features -│ └── database.py # SQLite ORM -├── frontend/ -│ ├── index.html -│ ├── app.js # Main app (2,794 lines) -│ ├── studio.js # Audio studio (2,363 lines) -│ ├── profiles.js # Profile management -│ └── history.js # History UI -└── data/ - ├── profiles/ # Voice profiles - ├── generations/ # Generated audio - └── mimic.db # SQLite database -``` - -### Backend Design - -**Async/await throughout:** -```python -@router.post("/generate") -async def generate(request: GenerateRequest): - audio = await tts.generate_async( - text=request.text, - profile_id=request.profile_id, - language=request.language - ) - - history_entry = await db.create_generation( - profile_id=request.profile_id, - text=request.text, - audio_path=audio.path - ) - - return history_entry -``` - -**Modular separation:** -- `models.py` - Pydantic request/response models -- `tts.py` - TTS inference logic only -- `transcribe.py` - ASR logic only -- `profiles.py` - CRUD for voice profiles -- `history.py` - CRUD for generation history -- `studio.py` - Audio editing features -- `database.py` - SQLAlchemy ORM - -### Features - -**Voice Profiles:** -- Create from audio file + reference text -- Multiple samples per profile (combined) -- Metadata: name, description, tags, language -- Thumbnail generation from waveform - -**Generation History:** -- SQLite database with full-text search -- Filters: profile, date range, language -- Regeneration from history -- Export to various formats - -**Audio Studio:** -- Timeline-based editing -- Multiple audio tracks -- Word-level timestamps (Whisper alignment) -- Trim, fade, volume control -- Export with normalization - -**Projects:** -- Save/load studio sessions -- Project metadata and versioning -- Export project as single file - -### Database Schema - -```sql -CREATE TABLE profiles ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE, - description TEXT, - language TEXT, - created_at TIMESTAMP, - updated_at TIMESTAMP -); - -CREATE TABLE profile_samples ( - id INTEGER PRIMARY KEY, - profile_id INTEGER, - audio_path TEXT, - reference_text TEXT, - FOREIGN KEY (profile_id) REFERENCES profiles(id) -); - -CREATE TABLE generations ( - id INTEGER PRIMARY KEY, - profile_id INTEGER, - text TEXT, - language TEXT, - audio_path TEXT, - duration REAL, - seed INTEGER, - created_at TIMESTAMP, - FOREIGN KEY (profile_id) REFERENCES profiles(id) -); - -CREATE TABLE projects ( - id INTEGER PRIMARY KEY, - name TEXT, - data JSON, - created_at TIMESTAMP, - updated_at TIMESTAMP -); -``` - -### Frontend Design - -**Major issue: Monolithic classes** - -**app.js (2,794 lines):** -```javascript -class MimicApp { - constructor() { - this.profiles = []; - this.generations = []; - this.currentProfile = null; - this.currentGeneration = null; - // ... 50+ properties - } - - // 80+ methods, no organization - async loadProfiles() { ... } - async createProfile() { ... } - async deleteProfile() { ... } - async generate() { ... } - async loadHistory() { ... } - // ... hundreds more lines -} -``` - -**studio.js (2,363 lines):** -```javascript -class AudioStudio { - constructor() { - this.wavesurfer = null; - this.timeline = null; - this.tracks = []; - this.regions = []; - this.words = []; - // ... 40+ properties - } - - // 60+ methods, all mixed - initWavesurfer() { ... } - addTrack() { ... } - removeTrack() { ... } - playPause() { ... } - exportAudio() { ... } - // ... hundreds more lines -} -``` - -**Global state everywhere:** -```javascript -let app = null; -let studio = null; -let currentProfile = null; -let isGenerating = false; -``` - -**No module system:** -- All files loaded via `