feat(mcp): Rust-owned speaking pill with self-contained audio playback

The pill window now surfaces for agent-initiated speech without main-window
involvement. Rust subscribes to /events/speak via a tokio task + reqwest
streaming body (speak_monitor.rs), shows the pill, and forwards events to
the dictate webview over Tauri's event bus. The pill plays audio via a
plain HTMLAudioElement and emits dictate:hide when playback ends. The
pill stays hidden through the ~1 s generation wait and only surfaces when
audio actually starts, with the counter armed at that moment.

Fixes a shared-dict mutation in mcp_server/events.publish() that caused
the second subscriber (Rust speak_monitor) to receive `event: message`
instead of named speak-start/speak-end frames. Also teaches the speak_monitor
parser to handle CRLF framing (sse-starlette default). Main-window
AudioPlayer now skips autoplay for source in {mcp, rest} to avoid
double-play when both windows are alive.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
James Pine
2026-04-23 01:46:17 -07:00
co-authored by Claude Opus 4.7
parent 0cef2c9fe1
commit 6b75e097e1
14 changed files with 387 additions and 192 deletions
+8 -2
View File
@@ -25,9 +25,15 @@ def unsubscribe(queue: asyncio.Queue[dict[str, Any]]) -> None:
def publish(kind: str, payload: dict[str, Any]) -> None:
"""Fan out to all current subscribers. Non-blocking; drops on full queue."""
event = {"kind": kind, **payload}
"""Fan out to all current subscribers. Non-blocking; drops on full queue.
Each subscriber gets its own dict copy — the SSE consumer calls
``event.pop("kind", ...)``, so sharing a single dict between queues
would mean the first consumer to drain its queue strips ``kind`` from
the object the next consumer later reads.
"""
for queue in list(_subscribers):
event = {"kind": kind, **payload}
try:
queue.put_nowait(event)
except asyncio.QueueFull:
+3
View File
@@ -237,6 +237,9 @@ async def get_generation_status(generation_id: str, db: Session = Depends(get_db
"status": gen.status or "completed",
"duration": gen.duration,
"error": gen.error,
# Agent-originated sources ("mcp", "rest") skip main-window
# autoplay — the floating pill plays those directly.
"source": gen.source,
}
yield f"data: {json.dumps(payload)}\n\n"