feat(mcp): local MCP server exposes voicebox.* tools to AI agents

Mounts FastMCP at /mcp (Streamable HTTP) so Claude Code, Cursor,
Windsurf, and the VS Code MCP extensions can call voicebox.speak,
voicebox.transcribe, voicebox.list_captures, and voicebox.list_profiles
against the running Voicebox server.

Backend
- new backend/mcp_server package (tools, middleware, profile resolve,
  pub/sub events); named mcp_server to avoid shadowing the installed mcp
  PyPI package FastMCP imports internally
- app.py migrated from @app.on_event to lifespan= so FastMCP's session
  manager cohabits with Voicebox's startup/shutdown
- new MCPClientBinding table + /mcp/bindings CRUD; ClientIdMiddleware
  reads X-Voicebox-Client-Id into a ContextVar and stamps last_seen_at
- profile resolution precedence: explicit -> per-client binding ->
  capture_settings.default_playback_voice_id
- POST /speak REST wrapper for non-MCP callers (shell, ACP, A2A)
- GET /events/speak SSE broadcasts speak-start / speak-end so the pill
  surfaces agent-initiated speech
- backend/mcp_shim proxy (plain httpx) for stdio-only MCP clients
- PyInstaller spec updates + new --shim build target (~18 MB)

Frontend
- Settings -> MCP page with HTTP / stdio / claude-mcp-add copy snippets,
  default voice picker, per-client bindings table, connection status
- useMCPBindings, useSpeakEvents hooks
- CapturePill gains 'speaking' state; DictateWindow subscribes to SSE
  and emits dictate:show so the Rust side surfaces the pill window

Native
- tauri.conf.json externalBin now includes voicebox-mcp
- show_dictate_window helper + dictate:show listener in main.rs
- (also in this commit: InputMonitoringGate UX, hotkey_monitor tweaks,
  landing footer/navbar updates, new overview docs for captures /
  dictation / mcp-server / voice-personalities)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
James Pine
2026-04-22 22:05:30 -07:00
co-authored by Claude Opus 4.7
parent 87c582ad54
commit 0cef2c9fe1
52 changed files with 4094 additions and 336 deletions
+64
View File
@@ -248,6 +248,7 @@ class CaptureSettingsResponse(BaseModel):
preserve_technical: bool = True
allow_auto_paste: bool = True
default_playback_voice_id: Optional[str] = None
hotkey_enabled: bool = False
chord_push_to_talk_keys: List[str] = Field(default_factory=lambda: ["MetaRight", "AltGr"])
chord_toggle_to_talk_keys: List[str] = Field(
default_factory=lambda: ["MetaRight", "AltGr", "Space"]
@@ -269,6 +270,7 @@ class CaptureSettingsUpdate(BaseModel):
preserve_technical: Optional[bool] = None
allow_auto_paste: Optional[bool] = None
default_playback_voice_id: Optional[str] = None
hotkey_enabled: Optional[bool] = None
chord_push_to_talk_keys: Optional[List[str]] = Field(default=None, min_length=1, max_length=6)
chord_toggle_to_talk_keys: Optional[List[str]] = Field(default=None, min_length=1, max_length=6)
@@ -294,6 +296,68 @@ class GenerationSettingsUpdate(BaseModel):
autoplay_on_generate: Optional[bool] = None
class MCPClientBindingResponse(BaseModel):
"""Per-MCP-client voice binding — what voice / engine / intent the server
should use when a given client_id calls voicebox.speak without args."""
client_id: str
label: Optional[str] = None
profile_id: Optional[str] = None
default_engine: Optional[str] = Field(
None,
pattern="^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$",
)
default_intent: Optional[str] = Field(
None, pattern="^(respond|rewrite|compose)$"
)
last_seen_at: Optional[datetime] = None
created_at: datetime
updated_at: datetime
class Config:
from_attributes = True
class MCPClientBindingUpsert(BaseModel):
"""Create or update a binding. Matched by ``client_id``."""
client_id: str = Field(..., min_length=1, max_length=64)
label: Optional[str] = Field(None, max_length=128)
profile_id: Optional[str] = None
default_engine: Optional[str] = Field(
None,
pattern="^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$",
)
default_intent: Optional[str] = Field(
None, pattern="^(respond|rewrite|compose)$"
)
class MCPClientBindingListResponse(BaseModel):
items: List[MCPClientBindingResponse]
class SpeakRequest(BaseModel):
"""Body for POST /speak — non-MCP REST surface that mirrors voicebox.speak."""
text: str = Field(..., min_length=1, max_length=10000)
profile: Optional[str] = Field(
None,
description="Voice profile name or id. Falls back to per-client binding, then default.",
)
engine: Optional[str] = Field(
None,
pattern="^(qwen|qwen_custom_voice|luxtts|chatterbox|chatterbox_turbo|tada|kokoro)$",
)
intent: Optional[str] = Field(
None, pattern="^(respond|rewrite|compose)$"
)
language: Optional[str] = Field(
None,
pattern="^(zh|en|ja|ko|de|fr|ru|pt|es|it|he|ar|da|el|fi|hi|ms|nl|no|pl|sv|sw|tr)$",
)
class LLMGenerateRequest(BaseModel):
"""Request model for LLM text generation."""