Labyricorn/voicebox
1
0
Fork 0
mirror of https://github.com/jamiepine/voicebox.git synced 2026-09-17 05:40:42 -07:00
Code Issues Packages Projects Releases Wiki Activity

feat: 0.5.0 Capture release — dictation, MCP, personalities (#544)

Browse Source 
* feat(capture): dictation, personalities, 0.5.0

Ships the Capture release end to end. Global-hotkey dictation with
synthetic paste into the focused app on macOS and Windows, an on-screen
pill across recording / transcribing / refining, customizable push-to-
talk and toggle chords, and an accessibility-permission prompt scoped to
Settings → Captures with inline re-check feedback.

Voice profiles gain optional personalities that power compose / rewrite /
respond actions via a local Qwen3 LLM — shared with refinement, so there
is one local LLM in the app, not two.

Refinement hardened with deterministic Whisper-loop collapse before the
LLM sees the transcript, per-capture flag snapshots for re-runs, and a
ten-transcript evaluation harness across every bundled refinement size.

Version bump 0.4.5 → 0.5.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(mcp): local MCP server exposes voicebox.* tools to AI agents

Mounts FastMCP at /mcp (Streamable HTTP) so Claude Code, Cursor,
Windsurf, and the VS Code MCP extensions can call voicebox.speak,
voicebox.transcribe, voicebox.list_captures, and voicebox.list_profiles
against the running Voicebox server.

Backend
- new backend/mcp_server package (tools, middleware, profile resolve,
  pub/sub events); named mcp_server to avoid shadowing the installed mcp
  PyPI package FastMCP imports internally
- app.py migrated from @app.on_event to lifespan= so FastMCP's session
  manager cohabits with Voicebox's startup/shutdown
- new MCPClientBinding table + /mcp/bindings CRUD; ClientIdMiddleware
  reads X-Voicebox-Client-Id into a ContextVar and stamps last_seen_at
- profile resolution precedence: explicit -> per-client binding ->
  capture_settings.default_playback_voice_id
- POST /speak REST wrapper for non-MCP callers (shell, ACP, A2A)
- GET /events/speak SSE broadcasts speak-start / speak-end so the pill
  surfaces agent-initiated speech
- backend/mcp_shim proxy (plain httpx) for stdio-only MCP clients
- PyInstaller spec updates + new --shim build target (~18 MB)

Frontend
- Settings -> MCP page with HTTP / stdio / claude-mcp-add copy snippets,
  default voice picker, per-client bindings table, connection status
- useMCPBindings, useSpeakEvents hooks
- CapturePill gains 'speaking' state; DictateWindow subscribes to SSE
  and emits dictate:show so the Rust side surfaces the pill window

Native
- tauri.conf.json externalBin now includes voicebox-mcp
- show_dictate_window helper + dictate:show listener in main.rs
- (also in this commit: InputMonitoringGate UX, hotkey_monitor tweaks,
  landing footer/navbar updates, new overview docs for captures /
  dictation / mcp-server / voice-personalities)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(mcp): Rust-owned speaking pill with self-contained audio playback

The pill window now surfaces for agent-initiated speech without main-window
involvement. Rust subscribes to /events/speak via a tokio task + reqwest
streaming body (speak_monitor.rs), shows the pill, and forwards events to
the dictate webview over Tauri's event bus. The pill plays audio via a
plain HTMLAudioElement and emits dictate:hide when playback ends. The
pill stays hidden through the ~1 s generation wait and only surfaces when
audio actually starts, with the counter armed at that moment.

Fixes a shared-dict mutation in mcp_server/events.publish() that caused
the second subscriber (Rust speak_monitor) to receive `event: message`
instead of named speak-start/speak-end frames. Also teaches the speak_monitor
parser to handle CRLF framing (sse-starlette default). Main-window
AudioPlayer now skips autoplay for source in {mcp, rest} to avoid
double-play when both windows are alive.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* readme and dev script

* feat(capture): gate global hotkey on dictation readiness checklist

Stops the "stuck pill" failure where pressing the chord with missing
STT/LLM models triggers a recording that has nowhere to land. The
hotkey now stays disarmed until every gate (models downloaded, Input
Monitoring + Accessibility granted) is green; the empty-state checklist
in CapturesTab surfaces each unmet gate with a one-click action and
auto-arms the chord once everything turns green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* color

* model download status

* progress

* personality: bool API, i18n across the app

- Collapse intent tri-state (respond/rewrite/compose) to `personality: bool` on /generate, /speak, and voicebox.speak. Drop respond entirely; keep compose as a standalone button via /profiles/{id}/compose. Remove /rewrite, /respond, and /speak profile endpoints.
- FloatingGenerateBox: Wand2 persona toggle + Dices compose button appear when the selected profile has a personality. ProfileCard badges Wand2 alongside the effects Sparkles.
- MCP bindings: default_intent column → default_personality: bool. Migration drops the legacy column.
- i18n: en / ja / zh-CN / zh-TW translation files filled out and wired through the capture, server, and profile UI.

```ts
voicebox.speak({
  text: "Deploy complete.",
  profile: "Morgan",
  personality: true, // rewrite through the profile's personality LLM
});
```

* i18n: GenerationPage sidebar copy

* fix: BOOL import for windows crate 0.62

BOOL moved from Win32::Foundation to windows::core in 0.62.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(capture): layout-aware V keycode for synthetic paste on macOS

macOS apps match Cmd+V against the layout-translated character via NSMenu
key equivalents, so posting kVK_ANSI_V (= 9, the QWERTY V position) on
Dvorak produces Cmd+. and never triggers Paste. New keyboard_layout
module resolves the active layout's V keycode via
TISCopyCurrentKeyboardLayoutInputSource + UCKeyTranslate, caches it in an
AtomicU16, and refreshes on kTISNotifySelectedKeyboardInputSourceChanged.
All TIS calls run on the main thread (init from Tauri setup; observer
callback delivered to the main runloop); synthetic_keys::send_paste
reads the cached value once per paste. Falls back to kVK_ANSI_V when
resolution fails or the active input source carries no Unicode key
layout data.

Windows is intentionally left on hardcoded VK_V — SendInput delivers
WM_KEYDOWN with wParam = VK_V to the target regardless of the active
layout, which is why `Send "^v"` works for AutoHotkey on Dvorak Windows.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(capture): cooperative app activation for synthetic paste on macOS 14+

macOS 14 deprecated NSRunningApplication.activateWithOptions: in favour
of a cooperative-activation pattern: the caller first yields activation
rights to the target, then the target activate()s against the tightened
Sonoma foreground rules. Without the yield, activate() on 14+ sometimes
silently fails or only bounces the dock icon — the exact "paste lands in
the wrong app" symptom we were previously one API break away from.

activate_pid now discovers the 14+ selector via respondsToSelector: and
branches: on 14+ it yieldActivationToApplication:'s from
NSRunningApplication.current then calls -activate on the target; on
11–13 it stays on -activateWithOptions: (still the only option). Both
branches propagate the BOOL return — if activation is refused we error
out before clobbering the clipboard instead of silently proceeding.

The respondsToSelector: result is cached in a OnceLock so the probe
isn't repeated on every paste.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(capture): conditional clipboard restore + always-attempt on paste failure

Two bugs in paste_final_text' clipboard handling:

1. Restore was unconditional. If the user ⌘C'd in the target app during
   the 400 ms paste-consume window — or a clipboard history tool (Paste,
   Pastebot, Maccy) or Universal Clipboard sync snapshotted our staged
   text — the blind restore overwrote their newer content with the
   pre-paste snapshot, silently losing user data.

2. send_paste' errors were propagated with ? before the restore, so a
   CGEventPost / SendInput failure left the user's clipboard stuck on
   the transcript.

Fix folds both into one pattern: capture the post-write change count,
re-read it after paste-consume, restore only when they match (plus treat
a change-count read failure as "unknown, don't overwrite"). Isolate
send_paste's error so the restore runs regardless of paste success, then
propagate the paste error after.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(deps): pin rdev to jamiepine/rdev fork

Upstream Narsil/rdev has shipped no release since 2023-06 (crates.io
still serves 0.5.3), so the Sonoma main-thread fix we depend on — PR
#147, applied at hotkey_monitor.rs:184 — is only reachable via a git
pin. A pin to a third-party repo breaks the build whenever the remote
force-pushes, renames, or is taken down, and Cargo does not durably
cache git-dep archives the way it does crates.io tarballs.

Forking to jamiepine/rdev at the same SHA removes that failure mode
without changing crate behavior and gives us a place to cherry-pick
future OS-compatibility fixes on our own timeline. The SHA was verified
to exist on the fork before re-pinning.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(mcp): idle timeout + escalating backoff for speak-SSE monitor

Two reliability gaps in the /events/speak subscriber:

1. resp.chunk().await had no idle timeout. A backend that accepts the
   TCP connection but stops producing frames (deadlocked SSE endpoint,
   zombie process) would block the task forever without reconnecting.
   The pill window would never surface for agent-initiated speech and
   there would be nothing to log. Backend emits a `:ping` heartbeat
   every 15 s, so 45 s without any data is now treated as a dead
   stream — the task errors out and the reconnect loop takes over.

2. Flat 2 s backoff escalates nowhere. Logs fill with reconnect lines
   when the backend is down for minutes, and a backend that accepts +
   immediately closes connections (no data) spins the loop tightly.
   Backoff now escalates 500 ms → 30 s on unproductive rounds and
   resets only when at least one frame arrives (the connection was
   genuinely productive, not just accepted).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(refinement): character-level loop collapse + pytest coverage

The word-level pass catches single-word Whisper loops ("URL URL URL…")
but misses two common hallucination patterns the PR had to claim as
"edge cases":

1. Multi-word English loops — "thanks for watching thanks for watching…"
   × 6 sails through because no two consecutive tokens are identical
   after text.split().
2. CJK loops — "謝謝觀看" × 7 sails through because text.split() returns
   a single unsplit token for the whole loop (no whitespace between
   characters).

Add a character-level second pass: a non-greedy regex finds any 2–60
char substring that repeats min_run+ times immediately after itself and
strips the run. The 2-char floor keeps emphasised single-letter runs
("wooooooow") intact. The 60-char ceiling covers every observed
Whisper tail hallucination ("Please like and subscribe to my
channel.", "Subtitles by the Amara.org community") while staying short
enough that coincidental long-phrase repetition in legitimate speech
doesn't hit the threshold. Whitespace normalisation only runs when the
pass actually stripped something, so untouched transcripts keep their
original spacing.

New test_refinement_collapse.py gives the pre-processor its first
deterministic unit-test coverage: 17 tests pinning the word-level
legacy behaviour plus the new multi-word English / CJK / Japanese /
emphasis-preservation cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(db): graceful fallback when SQLite < 3.35 on MCP bindings migration

SQLite gained ALTER TABLE … DROP COLUMN in 3.35 (Mar 2021). Production
PyInstaller builds bundle Python 3.12 which links to SQLite 3.40+ so
that path is always safe, but a dev running the backend directly on
Ubuntu 20.04 (3.31) or Debian 11 (3.34) would crash on first startup
trying to drop the legacy default_intent column.

Add _supports_drop_column(engine) — returns True on non-SQLite
dialects (Postgres / MySQL have supported DROP COLUMN for decades) and
gates on the runtime sqlite_version for SQLite. When unsupported, log a
warning and leave the unused column in place: SQLAlchemy only maps
declared columns, so a stray default_intent column does no reads or
writes and can't interfere with runtime behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(mcp): correct lifespan shutdown order — drain MCP before unloading models

The inline lifespan ran _run_shutdown inside the MCP context, so the
TTS / Whisper / LLM models were unloaded *before* FastMCP's __aexit__
got a chance to cancel its in-flight session tasks. Any MCP request
mid-generate at shutdown time would crash on "model unloaded" instead
of receiving a clean session-cancelled error.

Rewire via compose_lifespan (which was already defined in
mcp_server.server for exactly this purpose but never used):
AsyncExitStack enters factories in order and exits in LIFO, so
MCP teardown fires first — cancelling sessions — and _run_shutdown
runs after nothing is holding the models. Smoke test shows the
log order flipped as expected:

  Ready
  StreamableHTTP session manager started
  ... running ...
  StreamableHTTP session manager shutting down   ← was last, now first
  Voicebox server shutting down...               ← was first, now last

As a side benefit, _run_shutdown is now paired with _run_startup via
try/finally inside voicebox_lifespan, so a partial startup (models
half-loaded, MCP __aenter__ fails) still unloads whatever was loaded
instead of leaking it to process exit.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(mcp): stamp last_seen_at on /speak too + tighten path predicate

POST /speak is a REST wrapper around voicebox.speak for agents that
don't talk MCP (shell scripts, ACP, A2A). It reads X-Voicebox-Client-Id
and uses it for the same per-client profile resolution + default
personality lookup the MCP tool does (speak.py:39-64), so its callers
are first-class clients — but the ClientIdMiddleware only stamped
last_seen_at on /mcp* paths. REST speak callers showed up as "never
seen" in Settings → MCP despite actively acting on their bindings.

Widen the stamp predicate to an explicit ("/mcp", "/speak") prefix
list, and require a path boundary on match so future routes named
/mcpfoo or /speakers don't silently inherit the stamp via the prefix.
New test_client_id_middleware.py pins the scope with 17 parametrised
cases (both the allowed set and the overlap cases that must not match).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(captures): scrubbable WaveSurfer player for capture detail view

Replace the placeholder fake-waveform + play button in CapturesTab's
audio card with a real CaptureInlinePlayer (wavesurfer.js). The player
renders the actual waveform, lets users scrub through the clip, and
shows a proper current/total timestamp pair in place of the
duration-only label.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(ui): persist selectedProfileId across sessions

Wrap useUIStore in zustand/middleware's persist under the key
voicebox-ui. partialize only selectedProfileId so volatile UI state
(dialog open flags, form drafts, engine/voice pickers, sidebar) stays
in-memory as before — but reopening the app no longer loses whichever
profile the user was last working with.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(captures): mirror readiness checklist into the settings sidebar

The six-gate checklist only rendered in the CapturesTab empty state, so
a user already on the settings page had no single surface showing which
gate was red — the inline InputMonitoringNotice covered one, the model
pickers covered another, and Accessibility was only hinted at by the
auto-paste toggle. Mirror the same component into the right sidebar of
the settings page so every gate (STT model, LLM model, Input Monitoring,
Accessibility, plus the hotkey toggle in the main column) is always
visible while the user configures dictation.

New compact prop on DictationReadinessChecklist drops the centered
header and empty-state max-width so it fits the 280 px sidebar next to
the existing About / Differences blocks. Callers in compact mode own
the heading — CapturesPage reuses the existing captures.readiness.title
key (present in en / ja / zh-CN / zh-TW already) as an h3 matching the
sibling sidebar sections.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(captures): move sidebar checklist below differences + hide when all green

Two small follow-ups to the sidebar checklist placement. Move it below
the What's different section so the sticky top of the sidebar stays the
page's narrative context (About → differences) and the checklist reads
as a status panel rather than preamble. Gate the whole block on
!readiness.allReady so once every gate is green the sidebar drops back
to just About + What's different — no value in real estate full of
checkmarks.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(captures): refetch readiness immediately after STT/LLM model swap

useCaptureSettings updated its own cache optimistically but never
invalidated ['capture-readiness'], so for up to 5 s (the poll interval)
after switching stt_model or llm_model the checklist kept showing the
previous model's ready/missing state. The backend endpoint resolves
the model live on each call — it was just the frontend cache that
lagged. Invalidate in onSettled only when the patch touched a model
field, so unrelated updates (chord keys, toggles) don't pay for a
refetch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(captures): hide macOS-only copy when running on Windows / Linux

Two surfaces leaked macOS-specific copy onto other platforms:

1. The Input Monitoring + Accessibility rows in the readiness
   checklist rendered everywhere. On Windows/Linux the Rust permission
   stubs return true, so the rows showed as permanent green checkmarks
   with copy like "macOS allows Voicebox to detect your global
   shortcut." — nonsense when you're on Windows. Gate both rows on a
   userAgent-based isMacOS check so they only render where the
   underlying TCC permission actually exists.

2. The global-shortcut setting description ended with "macOS will ask
   for Input Monitoring permission the first time you turn this on."
   That sentence rendered on every platform. The readiness checklist
   already surfaces the TCC requirement at the right moment on macOS,
   so the description doesn't need the platform note — drop it from
   en / ja / zh-CN / zh-TW.

Other macOS strings (AccessibilityNotice, InputMonitoringNotice, their
"stillMissing" hints) are already gated behind the Rust permission
booleans returning false, which never happens on Windows/Linux, so they
stay inert without further changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(capture): swap the rdev fork for keytap 0.2, delete local chord state machine

Dep swap:
- Drop the git-pinned jamiepine/rdev fork we were carrying since the
  upstream crate is abandoned.
- Depend on keytap 0.2 from crates.io — our own cross-platform global
  keyboard tap crate. Clean shutdown via Drop, Sonoma-safe by design
  (no TSMGetInputSourceProperty calls off the main thread, so
  `set_is_main_thread(false)` is gone), and properly versioned.

Chord engine rewrite:
- Delete hotkey_monitor.rs's internal Chord state machine (Match enum,
  KeyEvent enum, step()/classify() methods, associated unit tests).
  keytap's ChordMatcher subsumes it: Momentary chord for PTT,
  add_toggle() for Toggle-to-talk, longest-match resolution, sticky-end
  for Toggle. Net: -80 LOC in hotkey_monitor.rs; the remaining module
  is the dispatcher loop + Effect→Tauri translation.
- Preserve the PTT→Toggle "RestartRecording" upgrade signal. keytap
  emits End(PTT)+Start(Toggle) atomically (same Instant) when the held
  set upgrades from a shorter chord to a longer superset. The
  dispatcher peeks at the matcher with a 5 ms recv_timeout after any
  End and coalesces the pair into Effect::RestartRecording so the
  frontend still gets the "discard the transition-moment audio" signal
  instead of an unrelated Stop+Start pair.
- HotkeyMonitor::update_bindings now actually tears down the tap on
  empty bindings instead of leaving an idle CGEventTap around. New
  bindings rebuild the matcher and the dispatcher thread from scratch.

key_codes.rs:
- Rewrite the browser-code → Key table against keytap's cleaner Key
  variant names (`A`..`Z` not `KeyA`..`KeyZ`, `Digit0`..`Digit9` not
  `Num0`..`Num9`, `ArrowUp` not `UpArrow`, `AltLeft`/`AltRight` instead
  of `Alt`/`AltGr`, `Period` not `Dot`, …). On-disk chord string
  format (W3C `KeyboardEvent.code` identifiers) is unchanged, so
  capture_settings rows written before the swap round-trip identically.
  Legacy aliases (`Alt`, `AltGr`, `Num0`, `UpArrow`, `Dot`, …) kept for
  forward-compat on old rows.

main.rs / input_monitoring.rs:
- Update the few doc comments that referenced `rdev::listen` to
  describe keytap's Tap; no behavioural change.
- build_chord_bindings now imports from keytap::Key.
- enable_hotkey / disable_hotkey / update_chord_bindings reach into
  HotkeyMonitor via &mut since apply()/update_bindings() now mutate.

Tests live in keytap now (22 chord-related tests in keytap 0.2,
including the PTT→Toggle upgrade scenario that used to be tested in
hotkey_monitor.rs). Voicebox's hotkey_monitor.rs is thin enough that
local testing would be trivia.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(deps): bump keytap 0.2 → 0.4 for macOS modifier-events fix

0.2 read CGEventFlags via CGEventGetIntegerValueField(event, 0x81),
which is not a valid CGEventField id — macOS silently returned 0, so
FlagsChanged events produced no KeyDown / KeyUp for any modifier key
and the PTT / toggle chords never armed on macOS. 0.4 uses the
documented CGEventGetFlags(event) API.

0.3 (tracing / serde / Fn / IntlBackslash) is picked up as a free
consequence; no API surface we depend on changed.

* perf(captures): stop polling readiness once both models are green

useQuery was firing GET /capture/readiness every 5s forever, and also
on every window focus. Once stt.ready and llm.ready are both true the
answer can only change when the user swaps a model in settings, and
useSettings already invalidates the query on that path — the polling
was pure noise.

Gate both refetchInterval and refetchOnWindowFocus on "not fully ready"
so we fall silent once the checklist is green.

* feat(ui): theme settings, stories polish, track editor restructure

- add dark/light/system theme with persisted choice + OS change listener
- restyle stories sidebar (search, item layout, border) to match captures
- move floating generate box to right column of stories, add top fade mask
- story track editor: sticky track labels aligned via flex rows, custom scrollbar with left/right zoom handles
- capture pill light mode pass, fix inline waveform progress color
- pull mcp_server hidden imports into the pyinstaller spec
- notarization doc draft

* fix mlx llm bundling

* feat(ui): shared ListPane primitive + misc polish

ListPane is a compound component (Header / TitleRow / Title / Actions /
Search / Scroll) that owns the relative wrapper, faded right divider
(50px top fade), top scroll mask, and absolute-positioned header used by
every list-detail tab. Wires up CapturesTab, StoryList, and EffectsList.
EffectsTab gets -mx-8 / pr-8 to match the edge-to-edge layout used
elsewhere.

Other changes:
- MCPPage: native <select> → shadcn <Select> for default voice and
  per-binding voice pickers
- Button outline variant: add hover:border-accent
- Drop hover:text-destructive from trailing delete buttons
  (HistoryTable, GpuAcceleration, GpuPage, EffectsChainEditor,
  EffectsDetail)
- HistoryTable empty state moved behind t('history.empty')
- StoryContent scroll padding pt-14 → pt-16
- backend health reports the captures dir
- landing CapturesMockup: "Send to" → "Export" with Download icon
- CHANGELOG: drop [Unreleased] personality section

* fix(captures): Play As autoplay + default voice + orphan recovery

- Hand /generate ids to the global SSE watcher so playback fires on completion. The mutation onSuccess was checking audio_path on a queued row, which is always empty — autoplay never ran.
- Bind the Play As voice selection to capture_settings.default_playback_voice_id, kept in sync with the Settings → Captures and Settings → MCP pickers. Picking from the split-button dropdown writes back to settings.
- Extract AudioBars from HistoryTable into a shared component; use it for the Play As generating state in place of Loader2.
- Stop the active-state hover from flashing white text when the button is in its lighter accent/10 fill.
- Drop the gradient avatar swatches from the Settings → Captures voice dropdown.
- Backend: when the gen worker exits without writing a terminal status (e.g. SQLite lock racing the failed-status write inside its own exception handler), the cancel endpoint now flips the row to failed instead of 409-ing. Worker also force-fails on its way out as a belt-and-suspenders.

* fix(captures+chord): Stop button stops, ChordPicker accepts shorter chords

Two unrelated correctness bugs caught in PR review:

- The Play As "Stop" button was wired to handlePlayAs() unconditionally, so clicking it during playback kicked a fresh generation instead of halting. Now pauses the player when the click came from the main button while playbackState is 'playing'. Picking a different voice from the dropdown still kicks a new generation as before.
- ChordPicker tracked the peak set of held keys but seeded the peak from initialKeys, so a user who opened the picker with a 3-key chord saved couldn't replace it with a 2-key chord — the candidate length never beat the seed. The peak now resets on the first press of a fresh sequence (when no keys were held immediately prior), then grows monotonically within that hold.

* fix(settings): honor explicit null on nullable fields, ignore on the rest

Routes were calling model_dump(exclude_none=True), which drops every
client-sent null before it reaches the service. The service then layered
on its own `if value is not None` guard. Net effect: setting a nullable
column back to null was a no-op — the MCPPage default-voice picker sends
null when the user picks "no default" and the row was silently keeping
whatever was there before.

Switched the routes to exclude_unset=True so absent fields stay absent
but explicit nulls survive the dump, and centralised the per-field
nullability check in the service. The check inspects the SQLAlchemy
column metadata so non-nullable columns (stt_model, llm_model, the chord
key lists) still drop nulls instead of crashing the request, while
default_playback_voice_id can finally be cleared.

* fix(captures): clean up audio files when create_capture fails

The create flow wrote raw audio (and a transcoded .wav for non-wav
sources) to data/captures before the DB row was committed, so any
failure between the write and the commit — a webm that decoded to a
0-length array, a whisper model that errored mid-transcribe, a SQLite
contention on the commit — left the audio on disk with nothing pointing
at it. Over enough flaky uploads the directory grows without bound.

Now every path written before the commit is tracked in a list, and the
whole stretch from the first write to db.commit() runs inside a
try/except that unlinks each tracked file on raise and re-raises. The
transcode branch removes the raw file from the cleanup list only when
the unlink actually succeeds, so an OSError on the raw-path delete
still hands cleanup the original blob to retry.

* fix(mcp): restrict voicebox.transcribe(audio_path=...) to loopback

audio_path mode took any absolute filesystem path and returned its
decoded contents as transcribed text with no caller verification beyond
the existence/size checks. The X-Voicebox-Client-Id middleware records
the header but never rejects an absent or fake one, so a Voicebox bound
to 0.0.0.0 (the documented "remote access" mode) was effectively an
unauthenticated arbitrary-local-file read primitive.

The middleware now stashes the request's remote address in a ContextVar
alongside the existing client_id, and audio_path mode refuses anything
that doesn't parse as a loopback address (IPv4 127.0.0.0/8, IPv6 ::1).
audio_base64 mode is unchanged — that path was always bounded to bytes
the caller already has.

Loopback callers (the Tauri webview, local CLI scripts, MCP clients on
the same machine) keep working. Remote callers now have to send the
audio over the wire if they want it transcribed.

* fix: PR review nits — response shape, landing copy, form reset

- /llm/generate's "model is downloading" branch was raising HTTPException(202, detail={...}), which wraps the payload in {"detail": ...} and forces clients to parse a success status as if it were an error. Switched to JSONResponse so the payload sits at the top level.
- The landing page's "Language Models" card advertised "Qwen 3.5" with sizes 4B/2B/0.8B; we ship Qwen3 at 0.6B/1.7B/4B. Aligned to what's actually in the binary.
- ProfileForm's discard-draft button reset the form without touching `personality` or `avatarFile`, so stale persona text and an attached avatar would survive the discard. The other three resets in the file already include both fields — this brings the discard path in line.

* perf(mcp): move last_seen_at stamp off the request path

ClientIdMiddleware was running the SQLAlchemy SELECT/INSERT/UPDATE/COMMIT
inline on the event loop after every /mcp/* and /speak request. SQLite
serialises writes, so concurrent MCP traffic queued behind the stamp
write — the response sat waiting on a side-effect that the client never
needs in band, and SSE streams would stall briefly per request.

The middleware now hands the stamp to asyncio.to_thread via a fire-and-
forget create_task so the response returns immediately and the write
runs on the default executor. A module-level set keeps strong refs to
in-flight tasks (per asyncio docs) so the GC can't collect them mid-
write. The fallback path runs the stamp inline if no loop is available
(tests/oddball callers) rather than silently dropping it.

* fix(dictate): force-dismiss the speaking pill when SSE never comes back

The pill subscribed to /generation/{id}/status to know when to start
playback, but EventSource.onerror was a no-op — auto-reconnect was the
intended recovery for transient drops. The gap: if the backend deletes
the gen row mid-flight or the connection silently dies in a way the
browser keeps retrying without ever getting a status event, the pill
sits in 'speaking' forever and the user has no way to clear it.

Added a 60-second hard cap that arms when the SSE opens and clears the
moment any real status event lands. If it fires while the pill is still
on the same id and audio never started, it force-dismisses. Same idea
as the existing post-speak-end 15s grace, but covers the case where the
backend never says anything at all.

* fix: i18n cleanup + readiness checklist effect cadence + ChordPicker shadow

- DictationReadinessChecklist was constructing downloadByModel as a fresh Map every render and listing it in the cleanup effect's deps. With the 1 s polling cadence and arbitrary parent rerenders the effect ran more often than it needed to. Memoised the Map on activeTasks; the effect now keys off the memo's identity.
- zh-CN persona tooltipActive/ariaLabelActive matched their inactive twins byte-for-byte ("以人物设定朗读"). The other locales differentiate the active state with a -ing / -中 suffix; zh-CN now reads "正以人物设定朗读" when active.
- personalityPlaceholder was a ~290-character paragraph that doubled as both the example text and the explanation, repeating most of what personalityHint already said. Trimmed to the example only and folded the explanation + leave-blank consequence into the hint, across all four locales.
- Refinement model size keys were size06 / size17 / size4. Renamed the 4B variant to size40 so the decimal padding is consistent.
- ChordPicker's open-effect bound a window.setTimeout id to a local `t`, shadowing the i18n `t` from useTranslation. Renamed to timeoutId.

* perf(settings): persist generation sliders on release, not per pointer-move

Both sliders on the generation settings page were calling update() —
which is a React Query mutation that PATCHes /settings/generation —
inside onValueChange. Dragging the chunk-limit slider from 800 to 3000
fired a request per pointer-move pixel, and a mid-drag failure plus
optimistic rollback would leave persisted state visibly out of sync
with the thumb position.

Local state now mirrors each slider during a drag and the persist
happens once on Radix's onValueCommit (pointer-up / keyboard-release).
useEffects keep the local state in sync if the persisted value changes
out-of-band — another window editing the same setting still updates the
slider position cleanly.

* chore(backend): Ruff lint pass — deprecated APIs, exception leaks, dead patterns

Mechanical sweep of items called out in the PR review:

- qwen_llm_backend: AutoModelForCausalLM.from_pretrained(torch_dtype=…) is deprecated in transformers ≥4.41 in favor of dtype=. Renamed.
- routes/llm: try/except around backend.generate() raised HTTPException(500, detail=str(e)) which leaks stack traces / paths to clients and trips Ruff B904. Now logs the original exception server-side and hands the client a generic message; chained via `from e` to preserve traceback context.
- mcp_bindings + mcp_server/context: datetime.utcnow() is deprecated since 3.12. Switched the two assignment sites to datetime.now(timezone.utc). The schema-level `default=datetime.utcnow` defaults in database/models.py are left for a later schema-aware pass.
- routes/generations: `logger = …` sat between two import blocks (Ruff E402). Moved below imports.
- mcp_server/server + tests/test_refinement_samples: typing.Callable / typing.Iterable have been preferred-via collections.abc since 3.9 (Ruff UP035).
- routes/events: `except asyncio.TimeoutError` aliases plain `TimeoutError` since 3.11 (UP041).
- services/captures: hoisted WHISPER_NATIVE_FORMATS to module scope (was a function-local UPPER_SNAKE that tripped N806) and replaced the raw_path.unlink try/except OSError-pass with contextlib.suppress (SIM105). Semantic equivalence preserved — written_files.remove(raw_path) still only runs when unlink succeeds because it sits inside the suppressed block after the unlink call.
- database/migrations: hoisted the duplicate `import sqlite3` from inside two helper bodies to a single module-level import.

* feat(stories): regenerate action on clips and the chat list dropdown

The track editor's clip toolbar now has a regenerate icon next to Delete; clicking it kicks a fresh take of the selected clip's underlying generation through the same /generate/{id}/regenerate path the History table uses, and pushes the id into the global pending set so the SSE watcher picks it up. The chat list's per-item dropdown gets the same action between Play-from-here and Remove. Translation keys added under storyContent.itemActions / storyContent.toast across all four locales.

* feat(stories): import external audio into the timeline (drag-drop + picker)

You can now drop a music file onto the story content area or pick one through the new "Import audio" button in the add-clip popover. Both call POST /generate/import which writes the file to data/generations/<id>.<ext>, probes duration via librosa, and inserts a Generation row pointing at a singleton "Imported Audio" profile (created lazily on first import). The existing addStoryItem flow takes over from there — the timeline doesn't care that the row didn't come out of TTS.

Engine field on the row is "import"; it's surfaced on StoryItemDetail so the chat list shows a music icon instead of the (missing) profile avatar and both the dropdown and the track-editor toolbar hide the Regenerate action — there's nothing to regenerate. Accepted formats: wav/mp3/flac/ogg/m4a/aac/webm, capped at 200 MB. Translation keys added across en/ja/zh-CN/zh-TW.

* fix(audio): serve real Content-Type so imports decode in WaveSurfer

/audio/{id} and /audio/version/{id} hardcoded media_type="audio/wav" on
the FileResponse. That was a no-op when every generation came out of
TTS (everything on disk was a .wav anyway), but imported audio keeps
its source format — .mp3 / .m4a / .ogg — and the WaveSurfer MediaElement
backend uses an <audio> tag that checks Content-Type before letting the
clip play, so an MP3 announced as audio/wav silently failed to load.

Both endpoints now derive the type via mimetypes.guess_type and fall
back to audio/wav for unknown suffixes. Download filenames also keep
the real extension instead of always saying ".wav".

* feat(stories): zoom bar bounds tracked to project length, default 60s scope

The track editor's zoom was clamped to a hardcoded [10, 200] pixels-per-second range, which had no relationship to the project — on a 4-minute story a "max zoom out" of 200 px/s still required scrolling, and on a 5-second story you could zoom all the way in to where every clip was a tiny sliver. Reframed the bounds in the unit the user actually thinks in: how many seconds of timeline are visible at once. Min scope is 10 s (most zoomed in), max scope is the entire project, and the default lands on a 60 s scope (or the full project, whichever is shorter) once the editor measures its visible track width on first mount.

The pixels-per-second value still lives in component state (because every downstream calculation already uses it) but minPps/maxPps are computed from `containerWidth − LABEL_COL_WIDTH` and the project's effective duration, so the +/- buttons and the edge-drag handles on the scrollbar all clamp to bounds that move with the project. Re-clamping fires whenever those bounds shift — adding a long clip or resizing the window pulls the current zoom inside the new range instead of leaving the user parked outside it.

* fix(stories): show the source filename on imported clips

Imports were rendering as "Imported Audio" everywhere because every
import points at the singleton voice profile. The filename was already
being stored on the generation row (in the `text` field), so the chat
item title and the timeline clip label now read from `text` when
`engine === 'import'` and fall back to the profile name otherwise. The
chat item also drops the language pill (always "en" on imports — not
informative) and skips the transcript textarea since imports have no
spoken text to show.

* fix(stories): round split_time_ms before posting

handleSplit was sending currentTimeMs - item.start_time_ms straight to the backend, which rejects it because StoryItemSplit.split_time_ms is typed as int and the playhead's currentTimeMs is a float (it's driven from HTMLAudioElement.currentTime, which carries sub-millisecond precision). Pydantic surfaced the mismatch as "Input should be a valid integer, got a number with a fractional part" and the toast read "Failed to split clip". Math.round at the call site, matching what the trim and move handlers already do.

* feat(stories): per-clip volume control on the timeline

Each story item now carries a volume column (linear gain, default 1.0,
clamped 0.0–2.0 server-side). New PUT /stories/{}/items/{}/volume route
+ useUpdateStoryItemVolume hook + a Volume2 icon in the clip-edit
toolbar that opens a popover with a 0–200% slider. Local slider state
drives the visual during a drag; the persist fires once on
onValueCommit, mirroring the generation-page slider pattern.

Web Audio playback inserts a per-clip GainNode between source and
master so volume changes apply live without re-decoding the buffer
(source -> clipGain -> masterGain -> destination). Server-side
mixdown in export multiplies the trimmed clip by its volume before
summing into the timeline. Split + duplicate carry the volume forward
to the new clips so trimming a faded section keeps the level you set.

Migration adds the volume column with default 1.0 so existing rows
read as full volume.

* fix(stories): mute the clip waveform's media element so it can't bleed audio

The clip waveforms drawn inside each timeline track use WaveSurfer with the default MediaElement backend, which creates an internal <audio> element to drive playback timing. Web Audio in useStoryPlayback is what actually produces sound, but WaveSurfer's element was happily preloading and — after the first user gesture unlocked browser autoplay — playing the source URL through the page output too.

For TTS clips it was masked: they're short, both sources start at the same time, and stopping the BufferSourceNode at pause coincides with the natural end of the audio element. For long imports (a four-minute MP3) the BufferSourceNode stops on pause but WaveSurfer's element keeps going on its own track — which is exactly the "music keeps playing when I pause" symptom.

Hand WaveSurfer a muted <audio> element via the `media` option so the visual still loads peaks but the element itself can never produce sound. preload="metadata" keeps the load lightweight.

* fix(stories): hard-cut the audio graph on stop so long imports actually halt

source.stop() was the only thing happening when a clip was halted, and on long imported buffers (multi-minute MP3s scheduled via source.start with a duration argument) it was silently failing to halt the buffer in some browsers — pause left the music playing and seek stacked another source on top of the original. The mute-the-WaveSurfer-element fix was a different bug along the same path; this is the one that actually addresses the duplicated audio.

ActiveSource now carries the per-clip GainNode alongside the source, and stopSource detaches the onended handler before calling stop() (so the natural-end callback can't race with explicit teardown and re-delete a freshly rescheduled entry at the same id), then disconnects both nodes inside their own try/catch blocks. Even when stop() doesn't actually halt the buffer the graph is severed — no path from source to destination, no audio.

* feat(stories): add empty tracks above/below the timeline

Tiny + strips sit at the top of the topmost label cell and the bottom of the bottommost one, sticky-positioned in the label column so they follow horizontal scroll. Clicking either extends the visible track stack in that direction by one — above adds max(existing)+1, below adds min(existing)-1. Both compute against the full set (defaults + item-derived + previously-added) so successive clicks keep extending instead of fighting over the same number.

Empty extras live in component state because a track only earns its keep once a clip lands on it. Once one does, item.track carries the number forward and the row keeps deriving from items naturally; if nothing lands there before reload, the empty row simply isn't there next time, which matches what the user expects of an unused affordance.

* fix(mcp): bundle stdio shim sidecar

* fix(captures): allow dictation without paste permission

* fix(mcp): preserve speak engine defaults

* fix(captures): use platform hotkey defaults

* fix(mcp): preload speak pill window

* fix(captures): hide unwired storage settings

* feat(sponsors): add /sponsors page, homepage promo, and in-app strip

* style(landing): drop pill chrome from /download maintainer kicker

* changelog

* better naming for sponsors

* windows keybind note

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Jamie Pine
2026-04-25 15:46:35 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 627d40b42d
commit 7df366d0c8
171 changed files with 20611 additions and 1293 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tauri/src-tauri/src/accessibility.rs
+42
View File
@@ -0,0 +1,42 @@
//! Platform permission gate for the auto-paste pipeline.
//!
//! On macOS, posting synthetic keyboard events and reading focused-UI state
//! via the AX API both require the host process to be listed under System
//! Settings → Privacy & Security → Accessibility. Without that trust,
//! `CGEventPost` silently drops events and `AXUIElementCopyAttributeValue`
//! returns an error. We surface a boolean check up front so the paste
//! pipeline can short-circuit with a clear "grant permission" message
//! instead of running through the full save → write → post → restore dance
//! with nothing to show for it.
//!
//! Windows has no equivalent user-facing permission — `SendInput` and
//! UIAutomation work for any non-elevated target out of the box. (UAC /
//! UIPI still blocks sending input *into* an elevated target window from a
//! non-elevated process, but that's per-target, not a global switch, and
//! there's no Settings pane to send users to.) So the Windows branch just
//! returns `true`.
#[cfg(target_os = "macos")]
mod ffi {
#[link(name = "ApplicationServices", kind = "framework")]
extern "C" {
/// Returns true when the current process is listed in Accessibility.
/// No prompt side-effect.
pub fn AXIsProcessTrusted() -> bool;
}
}
#[cfg(target_os = "macos")]
pub fn is_trusted() -> bool {
unsafe { ffi::AXIsProcessTrusted() }
}
#[cfg(target_os = "windows")]
pub fn is_trusted() -> bool {
true
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn is_trusted() -> bool {
false
}
tauri/src-tauri/src/clipboard.rs
+718
View File
@@ -0,0 +1,718 @@
//! Snapshot / write / restore helpers around the system clipboard.
//!
//! Used by the auto-paste flow: before synthesising the paste accelerator
//! into a foreign app we need to (1) remember what the user had on the
//! clipboard, (2) stage our transcribed text, (3) paste, (4) put the
//! original contents back. Missing step 4 turns every dictation into a
//! silent clipboard-stomp.
//!
//! On **macOS** the snapshot walks `NSPasteboard.pasteboardItems` and
//! copies every `(UTI, data)` pair into an owned `Vec<u8>`, so restore
//! rebuilds the full multi-type payload — not just the plain-text
//! fallback. Images, styled text, file-reference lists all survive the
//! round-trip.
//!
//! On **Windows** the snapshot walks `EnumClipboardFormats` and copies the
//! HGLOBAL payload for every advertised format. GDI-handle formats (DIB
//! bitmap, metafile, enhanced metafile, palette), owner-display variants,
//! and the private-/GDI-object format ranges are skipped — those can't be
//! round-tripped across processes without synthesising the underlying
//! kernel/GDI objects, which isn't worth the complexity for a dictation
//! clipboard guard. CF_UNICODETEXT, CF_HDROP, CF_DIB (bitmap data in
//! memory, not a handle), CF_DIBV5, and every registered format (HTML
//! Format, Rich Text Format, FileGroupDescriptor, etc.) all survive.
//!
//! On **macOS** every entry point manages its own `NSAutoreleasePool`
//! because the Tauri command runtime threads don't have one by default —
//! without it, every autoreleased `NSString` / `NSData` we touch would
//! leak for the life of the process. On Windows, HGLOBAL ownership
//! transfers to the clipboard on `SetClipboardData` success, so we only
//! free handles we allocated but didn't hand off.
#[cfg(target_os = "macos")]
use objc::runtime::Object;
#[cfg(target_os = "macos")]
use objc::{class, msg_send, sel, sel_impl};
/// One full-fidelity snapshot of the general pasteboard. Hold on to the value
/// until the paste has landed, then pass it to [`restore_clipboard`].
#[derive(Debug, Clone)]
pub struct ClipboardSnapshot {
/// Outer vec: pasteboard items. Inner: `(uti, raw bytes)` per type. We
/// store the raw UTI string and the raw `NSData` payload so we can rebuild
/// the item with `setData:forType:` without interpreting the contents.
items: Vec<Vec<(String, Vec<u8>)>>,
/// `NSPasteboard.changeCount` at the moment of capture. Incremented by AppKit
/// on every mutation from any process, so a caller can decide whether a
/// restore is still safe (change_count == expected) or whether someone
/// else wrote to the clipboard in the interim and we should back off.
change_count: i64,
}
impl ClipboardSnapshot {
pub fn change_count(&self) -> i64 {
self.change_count
}
pub fn item_count(&self) -> usize {
self.items.len()
}
}
#[cfg(target_os = "macos")]
type Id = *mut Object;
/// RAII wrapper so the pool drains even on early return / `?` propagation.
#[cfg(target_os = "macos")]
struct AutoreleasePool {
pool: Id,
}
#[cfg(target_os = "macos")]
impl AutoreleasePool {
unsafe fn new() -> Self {
let pool: Id = msg_send![class!(NSAutoreleasePool), alloc];
let pool: Id = msg_send![pool, init];
Self { pool }
}
}
#[cfg(target_os = "macos")]
impl Drop for AutoreleasePool {
fn drop(&mut self) {
unsafe {
let _: () = msg_send![self.pool, drain];
}
}
}
/// Build an autoreleased `NSString` from a Rust `&str` without scanning for
/// interior nulls (which is what `initWithUTF8String:` would require).
#[cfg(target_os = "macos")]
unsafe fn ns_string(s: &str) -> Id {
// NSUTF8StringEncoding = 4.
let obj: Id = msg_send![class!(NSString), alloc];
let obj: Id = msg_send![
obj,
initWithBytes: s.as_ptr()
length: s.len()
encoding: 4u64
];
let _: () = msg_send![obj, autorelease];
obj
}
#[cfg(target_os = "macos")]
unsafe fn ns_string_to_rust(s: Id) -> Option<String> {
if s.is_null() {
return None;
}
let bytes: *const i8 = msg_send![s, UTF8String];
if bytes.is_null() {
return None;
}
std::ffi::CStr::from_ptr(bytes)
.to_str()
.ok()
.map(|x| x.to_owned())
}
#[cfg(target_os = "macos")]
unsafe fn general_pasteboard() -> Result<Id, String> {
let pb: Id = msg_send![class!(NSPasteboard), generalPasteboard];
if pb.is_null() {
return Err("NSPasteboard generalPasteboard returned nil".into());
}
Ok(pb)
}
/// Read the pasteboard's current change count without snapshotting contents.
///
/// AppKit increments this every time any process writes to the general
/// pasteboard, so it's a cheap way to detect "did someone clobber my staged
/// text before the paste landed?".
#[cfg(target_os = "macos")]
pub fn current_change_count() -> Result<i64, String> {
unsafe {
let _pool = AutoreleasePool::new();
let pb = general_pasteboard()?;
let c: i64 = msg_send![pb, changeCount];
Ok(c)
}
}
/// Capture every item on the general pasteboard into an owned snapshot.
#[cfg(target_os = "macos")]
pub fn save_clipboard() -> Result<ClipboardSnapshot, String> {
unsafe {
let _pool = AutoreleasePool::new();
let pb = general_pasteboard()?;
let change_count: i64 = msg_send![pb, changeCount];
let items: Id = msg_send![pb, pasteboardItems];
if items.is_null() {
return Ok(ClipboardSnapshot {
items: Vec::new(),
change_count,
});
}
let count: usize = msg_send![items, count];
let mut saved: Vec<Vec<(String, Vec<u8>)>> = Vec::with_capacity(count);
for i in 0..count {
let item: Id = msg_send![items, objectAtIndex: i];
if item.is_null() {
continue;
}
let types: Id = msg_send![item, types];
if types.is_null() {
continue;
}
let type_count: usize = msg_send![types, count];
let mut pairs: Vec<(String, Vec<u8>)> = Vec::with_capacity(type_count);
for j in 0..type_count {
let t: Id = msg_send![types, objectAtIndex: j];
let Some(type_str) = ns_string_to_rust(t) else {
continue;
};
let data: Id = msg_send![item, dataForType: t];
if data.is_null() {
// Type advertised but no concrete data (lazy provider).
// Skipping is safer than trying to force it to materialise.
continue;
}
let length: usize = msg_send![data, length];
let bytes_ptr: *const u8 = msg_send![data, bytes];
let bytes = if bytes_ptr.is_null() || length == 0 {
Vec::new()
} else {
std::slice::from_raw_parts(bytes_ptr, length).to_vec()
};
pairs.push((type_str, bytes));
}
saved.push(pairs);
}
Ok(ClipboardSnapshot {
items: saved,
change_count,
})
}
}
/// Replace the pasteboard contents with a single plain-text string. Returns
/// the post-write change count so a later restore can verify nothing else
/// touched the clipboard in between.
#[cfg(target_os = "macos")]
pub fn write_text(text: &str) -> Result<i64, String> {
unsafe {
let _pool = AutoreleasePool::new();
let pb = general_pasteboard()?;
let _new_count: i64 = msg_send![pb, clearContents];
let ns_text = ns_string(text);
// `public.utf8-plain-text` is the raw UTI behind `NSPasteboardTypeString`
// and works for every text-aware paste target we care about.
let ns_type = ns_string("public.utf8-plain-text");
let ok: bool = msg_send![pb, setString: ns_text forType: ns_type];
if !ok {
return Err("NSPasteboard setString:forType: returned NO".into());
}
let after: i64 = msg_send![pb, changeCount];
Ok(after)
}
}
/// Rebuild the pasteboard from a snapshot, replacing whatever is on it now.
///
/// Does not consult the change count — callers that want safe restore should
/// compare [`current_change_count`] against the value returned by
/// [`write_text`] first.
#[cfg(target_os = "macos")]
pub fn restore_clipboard(snapshot: &ClipboardSnapshot) -> Result<(), String> {
unsafe {
let _pool = AutoreleasePool::new();
let pb = general_pasteboard()?;
let _: i64 = msg_send![pb, clearContents];
if snapshot.items.is_empty() {
return Ok(());
}
let array: Id = msg_send![class!(NSMutableArray), array];
for pairs in &snapshot.items {
let item: Id = msg_send![class!(NSPasteboardItem), alloc];
let item: Id = msg_send![item, init];
let _: () = msg_send![item, autorelease];
for (uti, bytes) in pairs {
let ns_type = ns_string(uti);
let data: Id = msg_send![
class!(NSData),
dataWithBytes: bytes.as_ptr()
length: bytes.len()
];
let _ok: bool = msg_send![item, setData: data forType: ns_type];
}
let _: () = msg_send![array, addObject: item];
}
let ok: bool = msg_send![pb, writeObjects: array];
if !ok {
return Err("NSPasteboard writeObjects: returned NO".into());
}
Ok(())
}
}
#[cfg(target_os = "windows")]
mod win {
//! Windows clipboard implementation.
//!
//! The snapshot is structured so it mirrors the macOS `Vec<Vec<_>>`
//! shape: a single outer "item" holding one `(format-name, bytes)`
//! pair per enumerated format. Windows has no notion of multiple
//! pasteboard items, so there's always exactly one or zero outer
//! entries — enough to keep `item_count()` meaningful without
//! fan-out.
//!
//! Format IDs are serialised as strings so the snapshot type can stay
//! platform-neutral. Predefined formats use their canonical
//! identifier (`"CF_UNICODETEXT"`, `"CF_HDROP"`, `"CF_DIB"`, …);
//! registered formats use their string name from
//! `GetClipboardFormatNameW` (`"HTML Format"`, `"Rich Text
//! Format"`, …). Restore reverses the mapping with a lookup table
//! for the predefined IDs and `RegisterClipboardFormatW` for the
//! rest.
//!
//! Skipped format classes:
//! - CF_BITMAP (2), CF_METAFILEPICT (3), CF_PALETTE (9),
//! CF_ENHMETAFILE (14) — HGLOBAL's actually an HBITMAP /
//! HENHMETAFILE, not raw memory. Rebuilding them across processes
//! is possible but not worth it for clipboard stashing.
//! - CF_OWNERDISPLAY (0x80) and the CF_DSPxxx variants (0x81–0x8E) —
//! the owner draws these on demand. No data to snapshot.
//! - CF_PRIVATEFIRST..CF_PRIVATELAST (0x200–0x2FF) — app-private,
//! meaningless to restore from a different process.
//! - CF_GDIOBJFIRST..CF_GDIOBJLAST (0x300–0x3FF) — GDI handles.
//!
//! Text formats that Windows auto-synthesises (CF_TEXT, CF_OEMTEXT,
//! CF_LOCALE) are also skipped during save: `SetClipboardData` on
//! CF_UNICODETEXT regenerates them lazily on restore.
use std::thread;
use std::time::Duration;
use windows::core::PCWSTR;
use windows::Win32::Foundation::{GlobalFree, HANDLE, HGLOBAL, HWND};
use windows::Win32::System::DataExchange::{
CloseClipboard, EmptyClipboard, EnumClipboardFormats, GetClipboardData,
GetClipboardFormatNameW, GetClipboardSequenceNumber, OpenClipboard,
RegisterClipboardFormatW, SetClipboardData,
};
use windows::Win32::System::Memory::{
GlobalAlloc, GlobalLock, GlobalSize, GlobalUnlock, GLOBAL_ALLOC_FLAGS,
};
// `windows` 0.62 doesn't re-export every predefined clipboard format
// under a stable feature flag, so the values are pinned inline.
// These numbers are ABI-stable back to Windows 3.1 — verified against
// winuser.h.
pub const CF_TEXT: u32 = 1;
pub const CF_BITMAP: u32 = 2;
pub const CF_METAFILEPICT: u32 = 3;
pub const CF_SYLK: u32 = 4;
pub const CF_DIF: u32 = 5;
pub const CF_TIFF: u32 = 6;
pub const CF_OEMTEXT: u32 = 7;
pub const CF_DIB: u32 = 8;
pub const CF_PALETTE: u32 = 9;
pub const CF_PENDATA: u32 = 10;
pub const CF_RIFF: u32 = 11;
pub const CF_WAVE: u32 = 12;
pub const CF_UNICODETEXT: u32 = 13;
pub const CF_ENHMETAFILE: u32 = 14;
pub const CF_HDROP: u32 = 15;
pub const CF_LOCALE: u32 = 16;
pub const CF_DIBV5: u32 = 17;
pub const CF_OWNERDISPLAY: u32 = 0x0080;
pub const CF_DSPTEXT: u32 = 0x0081;
pub const CF_DSPBITMAP: u32 = 0x0082;
pub const CF_DSPMETAFILEPICT: u32 = 0x0083;
pub const CF_DSPENHMETAFILE: u32 = 0x008E;
pub const CF_PRIVATEFIRST: u32 = 0x0200;
pub const CF_PRIVATELAST: u32 = 0x02FF;
pub const CF_GDIOBJFIRST: u32 = 0x0300;
pub const CF_GDIOBJLAST: u32 = 0x03FF;
/// `GlobalAlloc` movable-memory flag — `GMEM_MOVEABLE` (0x0002).
/// Required for HGLOBAL handles destined for `SetClipboardData`; fixed
/// allocations are rejected.
const GMEM_MOVEABLE: GLOBAL_ALLOC_FLAGS = GLOBAL_ALLOC_FLAGS(0x0002);
/// Map a predefined clipboard format ID to its canonical identifier
/// string. Registered formats (IDs >= 0xC000) aren't handled here —
/// the caller resolves those via `GetClipboardFormatNameW`.
pub fn predefined_name(id: u32) -> Option<&'static str> {
Some(match id {
CF_TEXT => "CF_TEXT",
CF_BITMAP => "CF_BITMAP",
CF_METAFILEPICT => "CF_METAFILEPICT",
CF_SYLK => "CF_SYLK",
CF_DIF => "CF_DIF",
CF_TIFF => "CF_TIFF",
CF_OEMTEXT => "CF_OEMTEXT",
CF_DIB => "CF_DIB",
CF_PALETTE => "CF_PALETTE",
CF_PENDATA => "CF_PENDATA",
CF_RIFF => "CF_RIFF",
CF_WAVE => "CF_WAVE",
CF_UNICODETEXT => "CF_UNICODETEXT",
CF_ENHMETAFILE => "CF_ENHMETAFILE",
CF_HDROP => "CF_HDROP",
CF_LOCALE => "CF_LOCALE",
CF_DIBV5 => "CF_DIBV5",
CF_OWNERDISPLAY => "CF_OWNERDISPLAY",
CF_DSPTEXT => "CF_DSPTEXT",
CF_DSPBITMAP => "CF_DSPBITMAP",
CF_DSPMETAFILEPICT => "CF_DSPMETAFILEPICT",
CF_DSPENHMETAFILE => "CF_DSPENHMETAFILE",
_ => return None,
})
}
/// Reverse of [`predefined_name`].
pub fn predefined_id(name: &str) -> Option<u32> {
Some(match name {
"CF_TEXT" => CF_TEXT,
"CF_BITMAP" => CF_BITMAP,
"CF_METAFILEPICT" => CF_METAFILEPICT,
"CF_SYLK" => CF_SYLK,
"CF_DIF" => CF_DIF,
"CF_TIFF" => CF_TIFF,
"CF_OEMTEXT" => CF_OEMTEXT,
"CF_DIB" => CF_DIB,
"CF_PALETTE" => CF_PALETTE,
"CF_PENDATA" => CF_PENDATA,
"CF_RIFF" => CF_RIFF,
"CF_WAVE" => CF_WAVE,
"CF_UNICODETEXT" => CF_UNICODETEXT,
"CF_ENHMETAFILE" => CF_ENHMETAFILE,
"CF_HDROP" => CF_HDROP,
"CF_LOCALE" => CF_LOCALE,
"CF_DIBV5" => CF_DIBV5,
"CF_OWNERDISPLAY" => CF_OWNERDISPLAY,
"CF_DSPTEXT" => CF_DSPTEXT,
"CF_DSPBITMAP" => CF_DSPBITMAP,
"CF_DSPMETAFILEPICT" => CF_DSPMETAFILEPICT,
"CF_DSPENHMETAFILE" => CF_DSPENHMETAFILE,
_ => return None,
})
}
/// Returns true for predefined formats whose payload is a GDI handle
/// or owner-display sentinel rather than plain memory — callers must
/// skip these during snapshot because GlobalSize/GlobalLock wouldn't
/// return usable bytes.
pub fn is_skipped_format(id: u32) -> bool {
matches!(
id,
CF_BITMAP
| CF_METAFILEPICT
| CF_PALETTE
| CF_ENHMETAFILE
| CF_OWNERDISPLAY
| CF_DSPTEXT
| CF_DSPBITMAP
| CF_DSPMETAFILEPICT
| CF_DSPENHMETAFILE
) || (CF_PRIVATEFIRST..=CF_PRIVATELAST).contains(&id)
|| (CF_GDIOBJFIRST..=CF_GDIOBJLAST).contains(&id)
}
/// Auto-synthesised formats that Windows regenerates from
/// CF_UNICODETEXT on demand. Safe to skip during save; restore
/// lets `SetClipboardData(CF_UNICODETEXT)` re-derive them.
pub fn is_auto_synthesised(id: u32) -> bool {
matches!(id, CF_TEXT | CF_OEMTEXT | CF_LOCALE)
}
/// RAII wrapper around `OpenClipboard` / `CloseClipboard`.
///
/// The clipboard is a global exclusive resource — only one process at
/// a time holds the handle. `OpenClipboard` fails with
/// ERROR_ACCESS_DENIED when another process is mid-paste; the retry
/// loop here absorbs the common transient case without bubbling a
/// user-visible error.
pub struct ClipboardGuard;
impl ClipboardGuard {
pub fn open() -> Result<Self, String> {
const MAX_ATTEMPTS: usize = 10;
const RETRY_DELAY: Duration = Duration::from_millis(10);
let mut last_err: Option<windows::core::Error> = None;
for _ in 0..MAX_ATTEMPTS {
let result = unsafe { OpenClipboard(Some(HWND(std::ptr::null_mut()))) };
match result {
Ok(()) => return Ok(Self),
Err(e) => {
last_err = Some(e);
thread::sleep(RETRY_DELAY);
}
}
}
Err(format!(
"OpenClipboard failed after {} retries ({:?}). Another process likely holds the clipboard open.",
MAX_ATTEMPTS, last_err
))
}
}
impl Drop for ClipboardGuard {
fn drop(&mut self) {
unsafe {
let _ = CloseClipboard();
}
}
}
/// Read the full payload for `format` from the currently open
/// clipboard into an owned `Vec<u8>`. Returns `Ok(None)` when the
/// clipboard advertises the format but provides no concrete data
/// (delay-rendered format that's never been realised).
pub fn read_format_bytes(format: u32) -> Result<Option<Vec<u8>>, String> {
unsafe {
let handle = GetClipboardData(format)
.map_err(|e| format!("GetClipboardData({format}) failed: {e}"))?;
if handle.is_invalid() {
return Ok(None);
}
let hglobal = HGLOBAL(handle.0);
let size = GlobalSize(hglobal);
if size == 0 {
return Ok(Some(Vec::new()));
}
let ptr = GlobalLock(hglobal);
if ptr.is_null() {
return Err(format!(
"GlobalLock returned null for format {format} (size {size})"
));
}
let bytes = std::slice::from_raw_parts(ptr as *const u8, size).to_vec();
let _ = GlobalUnlock(hglobal);
Ok(Some(bytes))
}
}
/// Look up the name for a registered format ID (>= 0xC000). Returns
/// `None` for unnamed predefined IDs — the caller should have used
/// [`predefined_name`] first.
pub fn registered_name(id: u32) -> Option<String> {
let mut buf = [0u16; 256];
let len = unsafe { GetClipboardFormatNameW(id, &mut buf) };
if len <= 0 {
return None;
}
String::from_utf16(&buf[..len as usize]).ok()
}
/// Allocate a movable HGLOBAL, copy `bytes` in, return the handle
/// ready for `SetClipboardData`. On success ownership transfers to
/// the clipboard; on failure the caller must `GlobalFree`.
pub fn allocate_global(bytes: &[u8]) -> Result<HGLOBAL, String> {
if bytes.is_empty() {
// `GlobalAlloc(_, 0)` returns NULL, which `SetClipboardData`
// would then reject as an invalid handle. Pad to one byte so
// the format still round-trips (the receiving app already
// has to handle zero-content payloads via GlobalSize).
return allocate_global(&[0u8]);
}
unsafe {
let hglobal = GlobalAlloc(GMEM_MOVEABLE, bytes.len())
.map_err(|e| format!("GlobalAlloc({}) failed: {e}", bytes.len()))?;
let ptr = GlobalLock(hglobal);
if ptr.is_null() {
let _ = GlobalFree(Some(hglobal));
return Err("GlobalLock returned null after GlobalAlloc".into());
}
std::ptr::copy_nonoverlapping(bytes.as_ptr(), ptr as *mut u8, bytes.len());
let _ = GlobalUnlock(hglobal);
Ok(hglobal)
}
}
/// Push one format's payload onto the currently open clipboard.
/// On `SetClipboardData` success the HGLOBAL becomes the clipboard's
/// responsibility — do not free. On failure, free it ourselves.
pub fn put_format(format: u32, bytes: &[u8]) -> Result<(), String> {
let hglobal = allocate_global(bytes)?;
let handle = HANDLE(hglobal.0);
unsafe {
match SetClipboardData(format, Some(handle)) {
Ok(_) => Ok(()),
Err(e) => {
let _ = GlobalFree(Some(hglobal));
Err(format!("SetClipboardData({format}) failed: {e}"))
}
}
}
}
/// UTF-16 encode `s` with a trailing null code unit and push it as
/// CF_UNICODETEXT.
pub fn put_unicode_text(s: &str) -> Result<(), String> {
let mut utf16: Vec<u16> = s.encode_utf16().collect();
utf16.push(0);
let bytes: &[u8] = unsafe {
std::slice::from_raw_parts(
utf16.as_ptr() as *const u8,
utf16.len() * std::mem::size_of::<u16>(),
)
};
put_format(CF_UNICODETEXT, bytes)
}
/// Walk every format currently on the clipboard. `EnumClipboardFormats(0)`
/// returns the first; each subsequent call with the previous format
/// returns the next, until it returns 0 (or an error).
pub fn enumerate_formats() -> Vec<u32> {
let mut out = Vec::new();
let mut current = 0u32;
loop {
let next = unsafe { EnumClipboardFormats(current) };
if next == 0 {
break;
}
out.push(next);
current = next;
}
out
}
/// Resolve a snapshot's format name back to the u32 format ID.
/// Registered names (anything not predefined) go through
/// `RegisterClipboardFormatW`, which is idempotent — the same name
/// yields the same ID within a Windows session.
pub fn resolve_format_id(name: &str) -> Result<u32, String> {
if let Some(id) = predefined_id(name) {
return Ok(id);
}
let wide: Vec<u16> = name.encode_utf16().chain(std::iter::once(0)).collect();
let id = unsafe { RegisterClipboardFormatW(PCWSTR(wide.as_ptr())) };
if id == 0 {
return Err(format!("RegisterClipboardFormatW failed for {name:?}"));
}
Ok(id)
}
pub fn sequence_number() -> u32 {
unsafe { GetClipboardSequenceNumber() }
}
pub fn empty() -> Result<(), String> {
unsafe { EmptyClipboard().map_err(|e| format!("EmptyClipboard failed: {e}")) }
}
}
#[cfg(target_os = "windows")]
pub fn current_change_count() -> Result<i64, String> {
Ok(win::sequence_number() as i64)
}
#[cfg(target_os = "windows")]
pub fn save_clipboard() -> Result<ClipboardSnapshot, String> {
let change_count = win::sequence_number() as i64;
let _guard = win::ClipboardGuard::open()?;
let formats = win::enumerate_formats();
let mut pairs: Vec<(String, Vec<u8>)> = Vec::with_capacity(formats.len());
for id in formats {
if win::is_skipped_format(id) || win::is_auto_synthesised(id) {
continue;
}
let name = match win::predefined_name(id) {
Some(n) => n.to_string(),
None => match win::registered_name(id) {
Some(n) => n,
None => continue,
},
};
match win::read_format_bytes(id) {
Ok(Some(bytes)) => pairs.push((name, bytes)),
Ok(None) => {}
Err(_) => {
// Single-format read failure (delay-render that never
// materialises, ACL-restricted format, etc.) shouldn't
// abort the whole snapshot — drop this format and keep
// going so the user's other clipboard contents still
// survive the round-trip.
continue;
}
}
}
let items = if pairs.is_empty() {
Vec::new()
} else {
vec![pairs]
};
Ok(ClipboardSnapshot {
items,
change_count,
})
}
#[cfg(target_os = "windows")]
pub fn write_text(text: &str) -> Result<i64, String> {
let _guard = win::ClipboardGuard::open()?;
win::empty()?;
win::put_unicode_text(text)?;
// `GetClipboardSequenceNumber` reflects the post-write value as soon
// as `SetClipboardData` returns.
Ok(win::sequence_number() as i64)
}
#[cfg(target_os = "windows")]
pub fn restore_clipboard(snapshot: &ClipboardSnapshot) -> Result<(), String> {
let _guard = win::ClipboardGuard::open()?;
win::empty()?;
for pairs in &snapshot.items {
for (name, bytes) in pairs {
let id = match win::resolve_format_id(name) {
Ok(id) => id,
Err(_) => continue,
};
// Per-format failures here also don't abort the whole
// restore — better to get the user's text content back even
// if a weird custom format can't be rehydrated.
let _ = win::put_format(id, bytes);
}
}
Ok(())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn current_change_count() -> Result<i64, String> {
Err("clipboard snapshot is not yet implemented on this platform".into())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn save_clipboard() -> Result<ClipboardSnapshot, String> {
Err("clipboard snapshot is not yet implemented on this platform".into())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn write_text(_text: &str) -> Result<i64, String> {
Err("clipboard snapshot is not yet implemented on this platform".into())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn restore_clipboard(_snapshot: &ClipboardSnapshot) -> Result<(), String> {
Err("clipboard snapshot is not yet implemented on this platform".into())
}
tauri/src-tauri/src/focus_capture.rs
+535
View File
@@ -0,0 +1,535 @@
//! Captures the focused-UI snapshot at chord-start so auto-paste can land
//! in the user's original text field even after focus drifts during
//! transcription / refinement.
//!
//! We don't try to re-focus a specific sub-element on restore — many apps
//! expose complex focus hierarchies that don't respond consistently to
//! programmatic focus pokes. Bringing the owning *window* to the
//! foreground is enough: the window's own focus manager restores its
//! last-focused field, which is what every well-behaved paste-buffer tool
//! does and what users expect.
//!
//! - **macOS** — `AXUIElementCopyAttributeValue(kAXFocusedUIElement)` +
//! `AXUIElementGetPid` + NSRunningApplication activation. Activation
//! uses the cooperative-activation pattern on macOS 14+ (the caller
//! `yieldActivationToApplication:`s, then the target `activate`s) and
//! falls back to the pre-Sonoma `activateWithOptions:` on 11–13. See
//! `activate_pid` for the rationale.
//! - **Windows** — `GetForegroundWindow` + `GetWindowThreadProcessId` for
//! the top-level HWND and PID; UIAutomation's `IUIAutomation::GetFocusedElement`
//! for best-effort control-class (skipped silently if COM isn't usable).
//! Activation walks top-level windows for the saved PID and calls
//! `SetForegroundWindow`, bracketed by the `AttachThreadInput` dance
//! so Windows' foreground-lock rules don't silently swallow the
//! activation into a taskbar flash.
//!
//! PID + bundle id + role are all captured for diagnostics — the bundle
//! id lets step 6 (internal direct injection) detect "focus was inside
//! Voicebox itself" and short-circuit the synthetic-paste path. On
//! Windows, `bundle_id` holds the lowercased exe basename (`"voicebox.exe"`)
//! since there's no equivalent of macOS' reverse-DNS bundle identifier.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct FocusSnapshot {
pub pid: i32,
pub bundle_id: Option<String>,
pub role: Option<String>,
}
#[cfg(target_os = "macos")]
use core_foundation_sys::base::{kCFAllocatorDefault, CFRelease};
#[cfg(target_os = "macos")]
use core_foundation_sys::string::{
kCFStringEncodingUTF8, CFStringCreateWithCString, CFStringGetCString, CFStringGetLength,
CFStringRef,
};
#[cfg(target_os = "macos")]
use objc::runtime::Object;
#[cfg(target_os = "macos")]
use objc::{class, msg_send, sel, sel_impl};
#[cfg(target_os = "macos")]
type Id = *mut Object;
#[cfg(target_os = "macos")]
mod ffi {
use core_foundation_sys::base::CFTypeRef;
use core_foundation_sys::string::CFStringRef;
pub type AXError = i32;
pub const AX_ERROR_SUCCESS: AXError = 0;
pub type AXUIElementRef = *const std::ffi::c_void;
pub type Pid = i32;
#[link(name = "ApplicationServices", kind = "framework")]
extern "C" {
pub fn AXUIElementCreateSystemWide() -> AXUIElementRef;
pub fn AXUIElementCopyAttributeValue(
element: AXUIElementRef,
attribute: CFStringRef,
value: *mut CFTypeRef,
) -> AXError;
pub fn AXUIElementGetPid(element: AXUIElementRef, pid: *mut Pid) -> AXError;
}
// AX attribute keys are exposed as C macros that expand to CFSTR(...)
// literals, not as linkable symbols — build the CFStrings at runtime
// instead (see `cf_string_const` in focus_capture.rs).
}
#[cfg(target_os = "macos")]
struct AutoreleasePool {
pool: Id,
}
#[cfg(target_os = "macos")]
impl AutoreleasePool {
unsafe fn new() -> Self {
let pool: Id = msg_send![class!(NSAutoreleasePool), alloc];
let pool: Id = msg_send![pool, init];
Self { pool }
}
}
#[cfg(target_os = "macos")]
impl Drop for AutoreleasePool {
fn drop(&mut self) {
unsafe {
let _: () = msg_send![self.pool, drain];
}
}
}
#[cfg(target_os = "macos")]
unsafe fn ns_string_to_rust(s: Id) -> Option<String> {
if s.is_null() {
return None;
}
let bytes: *const i8 = msg_send![s, UTF8String];
if bytes.is_null() {
return None;
}
std::ffi::CStr::from_ptr(bytes)
.to_str()
.ok()
.map(|x| x.to_owned())
}
/// Build a `+1` retained CFString from an ASCII constant. Caller owns the
/// returned reference and must `CFRelease` it. Used for AX attribute keys
/// (`"AXFocusedUIElement"`, `"AXRole"`) because those aren't exported as
/// linker symbols — Apple ships them as `CFSTR(...)` macros.
#[cfg(target_os = "macos")]
unsafe fn cf_string_const(s: &str) -> Option<CFStringRef> {
let cstr = std::ffi::CString::new(s).ok()?;
let result = CFStringCreateWithCString(kCFAllocatorDefault, cstr.as_ptr(), kCFStringEncodingUTF8);
if result.is_null() {
None
} else {
Some(result)
}
}
#[cfg(target_os = "macos")]
unsafe fn cfstring_to_rust(s: CFStringRef) -> Option<String> {
if s.is_null() {
return None;
}
let len = CFStringGetLength(s);
if len == 0 {
return Some(String::new());
}
// CFStringGetLength is in UTF-16 code units; UTF-8 can need up to 4
// bytes per unit plus the trailing NUL.
let max_bytes = (len * 4 + 1) as usize;
let mut buf = vec![0u8; max_bytes];
let ok = CFStringGetCString(
s,
buf.as_mut_ptr() as *mut i8,
max_bytes as isize,
kCFStringEncodingUTF8,
);
if ok == 0 {
return None;
}
let cstr = std::ffi::CStr::from_ptr(buf.as_ptr() as *const i8);
cstr.to_str().ok().map(|x| x.to_owned())
}
#[cfg(target_os = "macos")]
unsafe fn bundle_id_for_pid(pid: i32) -> Option<String> {
let _pool = AutoreleasePool::new();
let app: Id = msg_send![
class!(NSRunningApplication),
runningApplicationWithProcessIdentifier: pid
];
if app.is_null() {
return None;
}
let bundle: Id = msg_send![app, bundleIdentifier];
ns_string_to_rust(bundle)
}
/// Read the system-wide focused UI element's PID, bundle id, and AX role.
///
/// Returns an error when no element is focused (e.g. Dock has focus) or
/// when Accessibility permission is missing — `AXUIElementCopyAttributeValue`
/// returns `-25204 kAXErrorAPIDisabled` in that case.
#[cfg(target_os = "macos")]
pub fn capture_focus() -> Result<FocusSnapshot, String> {
use ffi::*;
unsafe {
let system_wide = AXUIElementCreateSystemWide();
if system_wide.is_null() {
return Err("AXUIElementCreateSystemWide returned null".into());
}
let _sys_guard = scopeguard::guard(system_wide, |e| {
CFRelease(e as *const std::ffi::c_void)
});
let focused_attr = cf_string_const("AXFocusedUIElement")
.ok_or("Failed to build AXFocusedUIElement CFString")?;
let _focused_attr_guard =
scopeguard::guard(focused_attr, |s| CFRelease(s as *const std::ffi::c_void));
let mut focused: *const std::ffi::c_void = std::ptr::null();
let err = AXUIElementCopyAttributeValue(
system_wide,
focused_attr,
&mut focused as *mut _,
);
if err != AX_ERROR_SUCCESS || focused.is_null() {
return Err(format!(
"No focused element (AXError {}). Verify Accessibility permission is granted and a focused text field exists.",
err
));
}
let _focus_guard = scopeguard::guard(focused, |e| CFRelease(e));
let focused_elem = focused as AXUIElementRef;
let mut pid: Pid = 0;
let err = AXUIElementGetPid(focused_elem, &mut pid);
if err != AX_ERROR_SUCCESS {
return Err(format!("AXUIElementGetPid failed (AXError {})", err));
}
let role = {
let role_attr = cf_string_const("AXRole");
match role_attr {
Some(role_attr) => {
let _role_attr_guard = scopeguard::guard(role_attr, |s| {
CFRelease(s as *const std::ffi::c_void)
});
let mut role_value: *const std::ffi::c_void = std::ptr::null();
let err = AXUIElementCopyAttributeValue(
focused_elem,
role_attr,
&mut role_value as *mut _,
);
if err == AX_ERROR_SUCCESS && !role_value.is_null() {
let _role_guard = scopeguard::guard(role_value, |e| CFRelease(e));
cfstring_to_rust(role_value as CFStringRef)
} else {
None
}
}
None => None,
}
};
let bundle_id = bundle_id_for_pid(pid);
Ok(FocusSnapshot {
pid,
bundle_id,
role,
})
}
}
/// Bring the app owning `pid` to the foreground, re-activating its
/// last-focused window. Paired with [`capture_focus`] at chord-start so a
/// post-transcription synthetic ⌘V lands where the user started, not
/// wherever focus drifted to during the transcribe / refine window.
///
/// macOS 14 (Sonoma) deprecated `activateWithOptions:` in favour of a
/// cooperative-activation pattern: the caller first invokes
/// `yieldActivationToApplication:` on its own `NSRunningApplication` to
/// grant the target activation rights, then the target's `activate`
/// succeeds against the tightened Sonoma foreground rules. Without the
/// yield, `activate` on 14+ sometimes silently fails or only bounces the
/// dock icon — exactly the "paste lands in the wrong app" symptom we're
/// trying to prevent. The yield is discovered at runtime via
/// `respondsToSelector:` so we don't need an operatingSystemVersion probe
/// and the pre-Sonoma path stays identical.
///
/// The BOOL return of both `activate` and `activateWithOptions:` is now
/// propagated — if the system refuses activation (target quit mid-
/// transcription, trust revoked, cooperative-activation refused) the
/// caller aborts before clobbering the clipboard.
#[cfg(target_os = "macos")]
pub fn activate_pid(pid: i32) -> Result<(), String> {
unsafe {
let _pool = AutoreleasePool::new();
let target: Id = msg_send![
class!(NSRunningApplication),
runningApplicationWithProcessIdentifier: pid
];
if target.is_null() {
return Err(format!("No running application for PID {}", pid));
}
let activated: bool = if can_yield_activation() {
let current: Id =
msg_send![class!(NSRunningApplication), currentApplication];
if !current.is_null() {
let _: () = msg_send![current, yieldActivationToApplication: target];
}
msg_send![target, activate]
} else {
// NSApplicationActivateIgnoringOtherApps = 1 << 1 = 2.
msg_send![target, activateWithOptions: 2u64]
};
if !activated {
return Err(format!(
"NSRunningApplication activate returned false for PID {} — the target may have quit mid-transcription, Accessibility is no longer trusted, or the system refused cooperative activation.",
pid
));
}
Ok(())
}
}
/// `true` when `NSRunningApplication` responds to
/// `yieldActivationToApplication:` — the macOS 14+ discriminator for the
/// cooperative-activation APIs. Cached since the answer doesn't change
/// over a process's lifetime and the objc_msgSend probe is otherwise
/// repeated on every paste.
#[cfg(target_os = "macos")]
fn can_yield_activation() -> bool {
use std::sync::OnceLock;
static CACHED: OnceLock<bool> = OnceLock::new();
*CACHED.get_or_init(|| unsafe {
let current: Id = msg_send![class!(NSRunningApplication), currentApplication];
if current.is_null() {
return false;
}
let responds: bool = msg_send![
current,
respondsToSelector: sel!(yieldActivationToApplication:)
];
responds
})
}
#[cfg(target_os = "windows")]
mod win {
use std::path::Path;
use windows::core::{IUnknown, BOOL, BSTR, PWSTR};
use windows::Win32::Foundation::{CloseHandle, HWND, LPARAM};
use windows::Win32::System::Com::{
CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_MULTITHREADED,
};
use windows::Win32::System::Threading::{
AttachThreadInput, GetCurrentThreadId, OpenProcess, QueryFullProcessImageNameW,
PROCESS_NAME_FORMAT, PROCESS_QUERY_LIMITED_INFORMATION,
};
use windows::Win32::UI::Accessibility::{CUIAutomation, IUIAutomation, IUIAutomationElement};
use windows::Win32::UI::WindowsAndMessaging::{
EnumWindows, GetForegroundWindow, GetWindow, GetWindowThreadProcessId, IsWindowVisible,
SetForegroundWindow, GW_OWNER,
};
/// Read the PID that owns `hwnd`. Returns 0 on failure.
pub unsafe fn hwnd_pid(hwnd: HWND) -> u32 {
let mut pid: u32 = 0;
let _ = GetWindowThreadProcessId(hwnd, Some(&mut pid as *mut _));
pid
}
/// Query a PID's executable path and return its lowercased basename
/// (e.g. `"voicebox.exe"`). This is the Windows analogue of macOS'
/// `bundleIdentifier`, just less globally unique — two apps with the
/// same exe name can collide, but that's rare enough to accept for
/// the self-paste short-circuit.
pub fn exe_basename(pid: u32) -> Option<String> {
unsafe {
let handle = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, false, pid).ok()?;
let mut buf = [0u16; 1024];
let mut size = buf.len() as u32;
let ok = QueryFullProcessImageNameW(
handle,
PROCESS_NAME_FORMAT(0),
PWSTR(buf.as_mut_ptr()),
&mut size,
);
let _ = CloseHandle(handle);
if ok.is_err() || size == 0 {
return None;
}
let full = String::from_utf16(&buf[..size as usize]).ok()?;
let basename = Path::new(&full)
.file_name()
.and_then(|s| s.to_str())
.map(|s| s.to_ascii_lowercase())?;
Some(basename)
}
}
/// Best-effort `UIAutomation::GetFocusedElement().CurrentClassName()`.
/// Returns `None` when COM init, CoCreateInstance, or any UIA call
/// fails — role info is nice-to-have, not load-bearing for paste.
pub fn focused_control_class() -> Option<String> {
unsafe {
// MTA per-thread init. Ignore HRESULT: S_OK / S_FALSE /
// RPC_E_CHANGED_MODE are all benign for our uses here, and
// we deliberately never call CoUninitialize (the Tauri
// runtime thread lives for the life of the process, so
// leaving COM init in place is fine).
let _ = CoInitializeEx(None, COINIT_MULTITHREADED);
let automation: IUIAutomation =
CoCreateInstance(&CUIAutomation, None::<&IUnknown>, CLSCTX_INPROC_SERVER).ok()?;
let element: IUIAutomationElement = automation.GetFocusedElement().ok()?;
// UIAutomationElement's CurrentClassName allocates a BSTR
// the caller has to drop. `BSTR` in `windows` crate is a
// Drop-wrapped owned string, so just returning `.to_string()`
// is safe.
let class: BSTR = element.CurrentClassName().ok()?;
let s = class.to_string();
if s.is_empty() {
None
} else {
Some(s)
}
}
}
/// Find a visible top-level window owned by `pid`. Returns the first
/// match via `EnumWindows`. Top-level ≡ no owner window.
pub fn find_top_level_window(pid: u32) -> Option<HWND> {
struct Ctx {
target_pid: u32,
found: Option<HWND>,
}
let mut ctx = Ctx {
target_pid: pid,
found: None,
};
unsafe extern "system" fn callback(hwnd: HWND, lparam: LPARAM) -> BOOL {
let ctx = &mut *(lparam.0 as *mut Ctx);
if hwnd_pid(hwnd) != ctx.target_pid {
return BOOL(1);
}
// Skip tool windows / invisible shells. `GetWindow(GW_OWNER)`
// is non-null for modal dialogs and other secondary windows;
// we want the real app frame, which has no owner.
if !IsWindowVisible(hwnd).as_bool() {
return BOOL(1);
}
if !GetWindow(hwnd, GW_OWNER).unwrap_or(HWND(std::ptr::null_mut())).is_invalid() {
return BOOL(1);
}
ctx.found = Some(hwnd);
BOOL(0)
}
unsafe {
let _ = EnumWindows(
Some(callback),
LPARAM(&mut ctx as *mut _ as isize),
);
}
ctx.found
}
/// Bring `hwnd` to the foreground reliably.
///
/// Plain `SetForegroundWindow` loses to Windows' foreground-lock
/// rules — when our process isn't already foreground it can't hand
/// focus to another app. The documented workaround is to attach the
/// current thread's input queue to the current foreground window's
/// thread for the duration of the call, which temporarily lets us
/// share that thread's "last user activity" stamp.
pub fn activate_hwnd(hwnd: HWND) -> Result<(), String> {
unsafe {
let fg = GetForegroundWindow();
if fg == hwnd {
return Ok(());
}
let our_thread = GetCurrentThreadId();
let fg_thread = if fg.is_invalid() {
0
} else {
let mut _pid: u32 = 0;
GetWindowThreadProcessId(fg, Some(&mut _pid as *mut _))
};
let attached = fg_thread != 0
&& fg_thread != our_thread
&& AttachThreadInput(our_thread, fg_thread, true).as_bool();
let ok = SetForegroundWindow(hwnd).as_bool();
if attached {
let _ = AttachThreadInput(our_thread, fg_thread, false);
}
if !ok {
return Err(format!(
"SetForegroundWindow failed for HWND {:?} — Windows foreground-lock may have denied the activation.",
hwnd.0
));
}
Ok(())
}
}
}
#[cfg(target_os = "windows")]
pub fn capture_focus() -> Result<FocusSnapshot, String> {
use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
unsafe {
let hwnd = GetForegroundWindow();
if hwnd.is_invalid() {
return Err(
"GetForegroundWindow returned null — the desktop has no focused window (secure attention sequence, lock screen, or no user session)."
.into(),
);
}
let pid = win::hwnd_pid(hwnd);
if pid == 0 {
return Err("GetWindowThreadProcessId returned PID 0 for the foreground window".into());
}
let bundle_id = win::exe_basename(pid);
let role = win::focused_control_class();
Ok(FocusSnapshot {
pid: pid as i32,
bundle_id,
role,
})
}
}
#[cfg(target_os = "windows")]
pub fn activate_pid(pid: i32) -> Result<(), String> {
if pid <= 0 {
return Err(format!("Cannot activate invalid PID {pid}"));
}
let hwnd = win::find_top_level_window(pid as u32)
.ok_or_else(|| format!("No visible top-level window for PID {pid}"))?;
win::activate_hwnd(hwnd)
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn capture_focus() -> Result<FocusSnapshot, String> {
Err("focus capture is not yet implemented on this platform".into())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn activate_pid(_pid: i32) -> Result<(), String> {
Err("app activation is not yet implemented on this platform".into())
}
tauri/src-tauri/src/hotkey_monitor.rs
+287
View File
@@ -0,0 +1,287 @@
//! Global hotkey → dictation effect bridge.
//!
//! Thin adapter from `keytap::chord::ChordMatcher` to Tauri events. keytap
//! owns the OS event tap + the chord state machine (Momentary vs Toggle,
//! longest-match resolution, sticky-toggle semantics); this module's only
//! job is:
//!
//! 1. Build a `ChordMatcher` from the user's saved PTT + Toggle chords.
//! 2. Translate `ChordEvent` → voicebox's [`Effect`] on a dispatcher
//! thread.
//! 3. Fan [`Effect`]s out into Tauri events + dictate-window show/hide.
//!
//! The [`Effect::RestartRecording`] signal is emitted when keytap fires
//! `End(PTT)` and `Start(Toggle)` with the *same* [`Instant`] — which
//! happens when the held set upgrades from a shorter chord to a longer
//! superset in a single event (the classic PTT→hands-free transition).
//! We detect the pair with a 5 ms peek on the matcher's receiver and
//! coalesce into one `Restart` so hosts can discard the transition-
//! moment audio rather than treat it as an unrelated Stop+Start pair.
//!
//! Left- and right-hand modifier variants are kept distinct all the way
//! down to the OS event tap (keytap's core promise). Defaults bind to
//! right-hand Cmd + right-hand Option on macOS / right-hand Ctrl +
//! right-hand Shift on Windows so the usual left-hand shortcuts stay
//! with the OS / app.
use std::collections::{HashMap, HashSet};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread::{self, JoinHandle};
use std::time::Duration;
use keytap::chord::{Chord, ChordEvent, ChordMatcher};
use keytap::{Key, RecvTimeoutError};
use tauri::{AppHandle, Emitter, Manager};
use crate::focus_capture;
use crate::DICTATE_WINDOW_LABEL;
// ========================================================================
// Public types
// ========================================================================
/// Semantic action a chord can be bound to. `PushToTalk` = hold chord to
/// record, release to stop. `ToggleToTalk` = press chord to start recording,
/// press again to stop.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum ChordAction {
PushToTalk,
ToggleToTalk,
}
/// Effect produced after the chord matcher resolves an event. Hosts
/// translate these into UI / recorder calls.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Effect {
StartRecording(ChordAction),
StopRecording(ChordAction),
/// Emitted when a push-to-talk chord is "upgraded" into the toggle
/// chord mid-hold — hosts may want to discard the captured audio and
/// restart so the transition moment isn't in the recording.
RestartRecording(ChordAction),
}
/// Chord key sets from capture settings. Both actions use the same
/// `HashSet<Key>` shape so callers don't need to know about keytap's
/// `Chord` type.
pub type Bindings = HashMap<ChordAction, HashSet<Key>>;
// ========================================================================
// Monitor
// ========================================================================
pub struct HotkeyMonitor {
app: AppHandle,
active: Option<Active>,
}
struct Active {
dispatcher: JoinHandle<()>,
shutdown: Arc<AtomicBool>,
}
impl HotkeyMonitor {
/// Build the monitor with initial bindings. Equivalent to constructing
/// an empty monitor and calling [`Self::update_bindings`] once.
pub fn spawn(app: AppHandle, bindings: Bindings) -> Self {
let mut m = Self { app, active: None };
m.apply(bindings);
m
}
/// Swap in a fresh set of chord bindings. Tears down the existing
/// `ChordMatcher` (which stops keytap's chord worker thread and
/// closes the OS tap) and spawns a new one. No-op for the "all
/// empty" case so "disable hotkey" doesn't keep a tap running for
/// no reason.
pub fn update_bindings(&mut self, bindings: Bindings) {
self.apply(bindings);
}
fn apply(&mut self, bindings: Bindings) {
// Tear down any existing matcher + dispatcher first. The
// dispatcher sees the shutdown flag on its next recv_timeout
// (≤100ms) and returns; joining waits for that. Dropping the
// ChordMatcher stops keytap's chord-worker thread and the
// underlying Tap.
if let Some(active) = self.active.take() {
active.shutdown.store(true, Ordering::Relaxed);
let _ = active.dispatcher.join();
}
if bindings.values().all(|set| set.is_empty()) {
return;
}
let matcher = match build_matcher(&bindings) {
Ok(m) => m,
Err(err) => {
eprintln!(
"HotkeyMonitor: ChordMatcher build failed ({err}). Global chord detection is disabled. On macOS, grant Input Monitoring in System Settings → Privacy & Security → Input Monitoring and relaunch."
);
return;
}
};
let shutdown = Arc::new(AtomicBool::new(false));
let shutdown_for_thread = shutdown.clone();
let app = self.app.clone();
let dispatcher = thread::Builder::new()
.name("voicebox-hotkey-dispatcher".into())
.spawn(move || dispatcher_loop(app, matcher, shutdown_for_thread))
.expect("spawn hotkey dispatcher thread");
self.active = Some(Active { dispatcher, shutdown });
}
}
impl Drop for HotkeyMonitor {
fn drop(&mut self) {
if let Some(active) = self.active.take() {
active.shutdown.store(true, Ordering::Relaxed);
let _ = active.dispatcher.join();
}
}
}
// ========================================================================
// Matcher construction + dispatch
// ========================================================================
fn build_matcher(bindings: &Bindings) -> Result<ChordMatcher<ChordAction>, keytap::Error> {
let mut builder = ChordMatcher::builder();
if let Some(keys) = bindings.get(&ChordAction::PushToTalk) {
if !keys.is_empty() {
builder = builder.add(
ChordAction::PushToTalk,
Chord::of(keys.iter().copied()),
);
}
}
if let Some(keys) = bindings.get(&ChordAction::ToggleToTalk) {
if !keys.is_empty() {
builder = builder.add_toggle(
ChordAction::ToggleToTalk,
Chord::of(keys.iter().copied()),
);
}
}
builder.build()
}
fn dispatcher_loop(
app: AppHandle,
matcher: ChordMatcher<ChordAction>,
shutdown: Arc<AtomicBool>,
) {
while !shutdown.load(Ordering::Relaxed) {
match matcher.recv_timeout(Duration::from_millis(100)) {
Ok(event) => process_event(&app, &matcher, event),
Err(RecvTimeoutError::Timeout) => continue,
Err(RecvTimeoutError::Disconnected) => break,
}
}
}
/// Turn a single [`ChordEvent`] into zero or one [`Effect`]s, peeking at
/// the matcher once for a same-Instant follow-up so upgrade transitions
/// coalesce into [`Effect::RestartRecording`] instead of a Stop+Start
/// pair.
fn process_event(
app: &AppHandle,
matcher: &ChordMatcher<ChordAction>,
event: ChordEvent<ChordAction>,
) {
match event {
ChordEvent::Start { id, .. } => {
apply_effect(app, Effect::StartRecording(id));
}
ChordEvent::End { id: end_id, time: end_time } => {
// Peek for an immediately-following Start. keytap emits
// End+Start atomically (same Instant) when the held set
// transitions between registered chords — our 5 ms window
// is well under perceptible latency but far longer than the
// channel hop between keytap's chord worker and our
// dispatcher.
match matcher.recv_timeout(Duration::from_millis(5)) {
Ok(ChordEvent::Start { id: start_id, time: start_time })
if start_time == end_time =>
{
apply_effect(app, Effect::RestartRecording(start_id));
}
Ok(other) => {
apply_effect(app, Effect::StopRecording(end_id));
// The peeked event wasn't a transition partner;
// process it in its own right. Recursion depth is
// bounded by the number of back-to-back chord
// events, in practice 1–2.
process_event(app, matcher, other);
}
Err(_) => {
apply_effect(app, Effect::StopRecording(end_id));
}
}
}
}
}
// ========================================================================
// Effect → Tauri
// ========================================================================
fn apply_effect(app: &AppHandle, effect: Effect) {
match effect {
Effect::StartRecording(_) => {
// Snapshot focus BEFORE we touch the window — any AppKit
// reshuffle triggered by set_position / show could in principle
// steal key focus and poison the reading. In practice those
// calls leave keyWindow alone, but capturing first is free.
let focus = focus_capture::capture_focus().ok();
if let Some(window) = app.get_webview_window(DICTATE_WINDOW_LABEL) {
// The previous hide-cycle parked the window off-screen and
// made it click-through — undo both before showing, so the
// pill lands at top-center and the user can actually click
// the error pill / stop button.
//
// `current_monitor()` returns None when the window is off
// any display (our hide handler parks it at -10_000, -10_000
// precisely so it never intercepts clicks), so fall back to
// the primary monitor for the reposition.
let monitor = window
.current_monitor()
.ok()
.flatten()
.or_else(|| window.primary_monitor().ok().flatten());
if let Some(monitor) = monitor {
let monitor_pos = monitor.position();
let monitor_size = monitor.size();
if let Ok(win_size) = window.outer_size() {
let x = monitor_pos.x
+ (monitor_size.width as i32 - win_size.width as i32) / 2;
let y = monitor_pos.y + (monitor_size.height as f64 * 0.04) as i32;
let _ = window.set_position(tauri::PhysicalPosition::new(x, y));
}
}
let _ = window.set_ignore_cursor_events(false);
// Deliberately no set_focus() — taking key focus would yank
// it out of whatever app the user was typing in, which is
// the opposite of what a dictation overlay should do.
let _ = window.show();
let payload = serde_json::json!({ "focus": focus });
let _ = window.emit("dictate:start", payload);
}
}
Effect::StopRecording(_) => {
if let Some(window) = app.get_webview_window(DICTATE_WINDOW_LABEL) {
let _ = window.emit("dictate:stop", ());
}
}
Effect::RestartRecording(_) => {
if let Some(window) = app.get_webview_window(DICTATE_WINDOW_LABEL) {
let _ = window.emit("dictate:restart", ());
}
}
}
}
tauri/src-tauri/src/input_monitoring.rs
+82
View File
@@ -0,0 +1,82 @@
//! Platform permission gate for the global keyboard tap.
//!
//! On macOS 10.15+, creating a CGEventTap that observes keyboard events
//! requires the host process to be listed under System Settings → Privacy &
//! Security → Input Monitoring. Without that trust, keytap's `Tap` returns
//! a permission error and no key events ever flow through the chord engine.
//!
//! The relevant TCC pair lives in IOKit, mirroring `AXIsProcessTrusted` /
//! `AXIsProcessTrustedWithOptions` on the Accessibility side:
//!
//! - `IOHIDCheckAccess(kIOHIDRequestTypeListenEvent)` — read the current
//! grant without prompting. We call this from the Captures settings UI
//! so the row can show "granted" / "missing" without surprising the user.
//! - `IOHIDRequestAccess(kIOHIDRequestTypeListenEvent)` — fire the
//! "Voicebox would like to receive keystrokes from any application"
//! dialog and add Voicebox to the Input Monitoring pane (toggle off).
//! Returns true when access is already granted; otherwise returns false
//! and queues the prompt. The user still has to flip the toggle on; this
//! just gets us into the list.
//!
//! `enable_hotkey` calls `request` on first invocation so the prompt fires
//! from a deterministic, user-initiated point (the Captures toggle) instead
//! of as a side-effect of keytap's `Tap` creating its CGEventTap.
//!
//! Windows / Linux don't gate keyboard taps behind a TCC-style permission,
//! so those branches return `true`.
#[cfg(target_os = "macos")]
mod ffi {
use std::os::raw::c_uint;
/// `kIOHIDRequestTypeListenEvent` from `<IOKit/hidsystem/IOHIDLib.h>` —
/// the request-type discriminator for "I want to read keyboard / mouse
/// events created by other processes."
pub const REQUEST_TYPE_LISTEN_EVENT: c_uint = 1;
/// `kIOHIDAccessTypeGranted` from `IOHIDLib.h`. The other values are
/// `Denied = 1` and `Unknown = 2`; we only ever care about the granted
/// case so they don't get their own constants.
pub const ACCESS_TYPE_GRANTED: c_uint = 0;
#[link(name = "IOKit", kind = "framework")]
extern "C" {
/// Returns the current access state as an `IOHIDAccessType` enum
/// (Granted=0, Denied=1, Unknown=2). No prompt side-effect.
///
/// Declared as `c_uint` rather than `bool`: the C signature returns
/// the full enum, and reading a 3-valued enum into Rust's 1-bit
/// `bool` is undefined behaviour that silently inverts our gate.
pub fn IOHIDCheckAccess(request_type: c_uint) -> c_uint;
/// Returns true when access is already granted; otherwise queues
/// the system prompt and returns false synchronously. Safe to call
/// repeatedly — once the entry exists in the Input Monitoring pane
/// macOS won't re-prompt. Real `Boolean` (UInt8) return on the C
/// side, so `bool` here is correct.
pub fn IOHIDRequestAccess(request_type: c_uint) -> bool;
}
}
#[cfg(target_os = "macos")]
pub fn is_trusted() -> bool {
unsafe { ffi::IOHIDCheckAccess(ffi::REQUEST_TYPE_LISTEN_EVENT) == ffi::ACCESS_TYPE_GRANTED }
}
/// Fire the Input Monitoring prompt if not already granted. Returns the
/// current grant state; a `false` here means the prompt was queued and the
/// user needs to flip the toggle in System Settings before key events flow.
#[cfg(target_os = "macos")]
pub fn request() -> bool {
unsafe { ffi::IOHIDRequestAccess(ffi::REQUEST_TYPE_LISTEN_EVENT) }
}
#[cfg(not(target_os = "macos"))]
pub fn is_trusted() -> bool {
true
}
#[cfg(not(target_os = "macos"))]
pub fn request() -> bool {
true
}
tauri/src-tauri/src/key_codes.rs
+94
View File
@@ -0,0 +1,94 @@
//! Stable string ↔ `keytap::Key` mapping for chord persistence.
//!
//! The frontend captures keypresses through the browser keyboard API (which
//! exposes `event.code` like `"MetaRight"`, `"AltRight"`, `"Space"`, `"KeyA"`)
//! and stores chords in capture_settings as JSON arrays of canonical names.
//! On the way back the same names need to round-trip into `keytap::Key`
//! variants the chord engine actually matches against.
//!
//! Input strings follow the W3C `KeyboardEvent.code` identifiers exactly —
//! `"MetaRight"`, `"AltRight"`, `"KeyA"`, `"Digit0"`, `"ArrowUp"`, … —
//! which is also what the browser emits natively, so on-disk chords
//! round-trip without translation on the frontend side. Legacy aliases
//! (`"Alt"` / `"AltGr"` / `"Num0"` / `"UpArrow"` / …) are accepted too so
//! older capture_settings rows written before the keytap swap keep working.
use keytap::Key;
/// Resolve a canonical key name to its `keytap::Key`. Returns `None` for
/// names that don't have a corresponding variant — the command surface
/// rejects those so we never silently drop keys from a chord.
pub fn key_from_str(name: &str) -> Option<Key> {
Some(match name {
// Modifiers — left/right distinction matters for chord defaults.
"AltLeft" | "Alt" => Key::AltLeft,
"AltRight" | "AltGr" => Key::AltRight,
"ControlLeft" => Key::ControlLeft,
"ControlRight" => Key::ControlRight,
"MetaLeft" => Key::MetaLeft,
"MetaRight" => Key::MetaRight,
"ShiftLeft" => Key::ShiftLeft,
"ShiftRight" => Key::ShiftRight,
"CapsLock" => Key::CapsLock,
// Whitespace / navigation
"Space" => Key::Space,
"Tab" => Key::Tab,
"Enter" | "Return" => Key::Enter,
"Backspace" => Key::Backspace,
"Delete" => Key::Delete,
"Escape" => Key::Escape,
"Insert" => Key::Insert,
"Home" => Key::Home,
"End" => Key::End,
"PageUp" => Key::PageUp,
"PageDown" => Key::PageDown,
"ArrowUp" | "UpArrow" => Key::ArrowUp,
"ArrowDown" | "DownArrow" => Key::ArrowDown,
"ArrowLeft" | "LeftArrow" => Key::ArrowLeft,
"ArrowRight" | "RightArrow" => Key::ArrowRight,
// Function row
"F1" => Key::F1, "F2" => Key::F2, "F3" => Key::F3, "F4" => Key::F4,
"F5" => Key::F5, "F6" => Key::F6, "F7" => Key::F7, "F8" => Key::F8,
"F9" => Key::F9, "F10" => Key::F10, "F11" => Key::F11, "F12" => Key::F12,
// Digits
"Digit0" | "Num0" => Key::Digit0,
"Digit1" | "Num1" => Key::Digit1,
"Digit2" | "Num2" => Key::Digit2,
"Digit3" | "Num3" => Key::Digit3,
"Digit4" | "Num4" => Key::Digit4,
"Digit5" | "Num5" => Key::Digit5,
"Digit6" | "Num6" => Key::Digit6,
"Digit7" | "Num7" => Key::Digit7,
"Digit8" | "Num8" => Key::Digit8,
"Digit9" | "Num9" => Key::Digit9,
// Letters — browser emits "KeyA"; keytap uses the bare letter.
"KeyA" => Key::A, "KeyB" => Key::B, "KeyC" => Key::C,
"KeyD" => Key::D, "KeyE" => Key::E, "KeyF" => Key::F,
"KeyG" => Key::G, "KeyH" => Key::H, "KeyI" => Key::I,
"KeyJ" => Key::J, "KeyK" => Key::K, "KeyL" => Key::L,
"KeyM" => Key::M, "KeyN" => Key::N, "KeyO" => Key::O,
"KeyP" => Key::P, "KeyQ" => Key::Q, "KeyR" => Key::R,
"KeyS" => Key::S, "KeyT" => Key::T, "KeyU" => Key::U,
"KeyV" => Key::V, "KeyW" => Key::W, "KeyX" => Key::X,
"KeyY" => Key::Y, "KeyZ" => Key::Z,
// Punctuation / symbols
"Backquote" | "BackQuote" => Key::Backtick,
"Minus" => Key::Minus,
"Equal" => Key::Equal,
"BracketLeft" | "LeftBracket" => Key::BracketLeft,
"BracketRight" | "RightBracket" => Key::BracketRight,
"Semicolon" | "SemiColon" => Key::Semicolon,
"Quote" => Key::Quote,
"Backslash" | "BackSlash" => Key::Backslash,
"Comma" => Key::Comma,
"Period" | "Dot" => Key::Period,
"Slash" => Key::Slash,
_ => return None,
})
}
tauri/src-tauri/src/keyboard_layout.rs
+183
View File
@@ -0,0 +1,183 @@
//! Layout-aware resolution of the keycode whose current-layout translation
//! is `'v'`. Drives [`crate::synthetic_keys::send_paste`] so the synthetic
//! Cmd+V it posts is interpreted as Paste by the focused app regardless of
//! the user's active keyboard layout (Dvorak, Colemak, AZERTY, …).
//!
//! macOS apps process Cmd+V via NSMenu key equivalents, which match against
//! `[NSEvent charactersIgnoringModifiers]` — i.e. the layout-translated
//! character, not the raw keycode. Posting `kVK_ANSI_V` (= 9, the QWERTY V
//! position) on Dvorak therefore produces Cmd+. and never triggers Paste.
//!
//! All TIS calls happen on the main thread: once at startup via [`init`]
//! from Tauri's setup hook, and again from the
//! `kTISNotifySelectedKeyboardInputSourceChanged` distributed notification
//! (delivered to the main runloop). The hot path ([`paste_keycode_v`])
//! only reads an [`AtomicU16`], so paste latency is unchanged.
//!
//! Windows is intentionally not covered here. `SendInput` with
//! `wVk = VK_V` delivers `WM_KEYDOWN` to the target with `wParam = VK_V`
//! regardless of the active layout — most Windows apps treat that as
//! Ctrl+V. AutoHotkey relies on the same behaviour.
use std::sync::atomic::{AtomicU16, Ordering};
/// `kVK_ANSI_V` — the keycode for the physical V key on a US QWERTY
/// layout. Used as the fallback whenever live resolution can't produce a
/// better answer (no Unicode key layout data, lookup failure, non-macOS).
const FALLBACK_V_KEYCODE: u16 = 9;
static V_KEYCODE: AtomicU16 = AtomicU16::new(FALLBACK_V_KEYCODE);
/// Returns the keycode whose current-layout translation is `'v'`. Falls
/// back to `kVK_ANSI_V` when resolution hasn't run, the active input
/// source carries no Unicode key layout data, or no keycode in the layout
/// produces `v`.
pub fn paste_keycode_v() -> u16 {
V_KEYCODE.load(Ordering::Relaxed)
}
#[cfg(target_os = "macos")]
pub fn init() {
macos::init();
}
#[cfg(not(target_os = "macos"))]
pub fn init() {}
#[cfg(target_os = "macos")]
mod macos {
use super::{FALLBACK_V_KEYCODE, V_KEYCODE};
use core_foundation_sys::base::CFRelease;
use core_foundation_sys::data::{CFDataGetBytePtr, CFDataRef};
use core_foundation_sys::dictionary::CFDictionaryRef;
use core_foundation_sys::notification_center::{
CFNotificationCenterAddObserver, CFNotificationCenterGetDistributedCenter,
CFNotificationCenterRef, CFNotificationName,
CFNotificationSuspensionBehaviorDeliverImmediately,
};
use core_foundation_sys::string::CFStringRef;
use std::ffi::c_void;
use std::ptr;
use std::sync::atomic::Ordering;
type TISInputSourceRef = *mut c_void;
/// `kUCKeyActionDown`.
const K_UC_KEY_ACTION_DOWN: u16 = 0;
/// `kUCKeyTranslateNoDeadKeysMask` — collapse dead-key state machine so
/// a single call gives us the bare character. V is never a dead key on
/// any layout we care about, but the flag costs nothing and removes
/// any chance of ambiguous output.
const K_UC_KEY_TRANSLATE_NO_DEAD_KEYS_MASK: u32 = 1;
/// Standard US-style virtual keycodes occupy 0..0x7F. We iterate the
/// full range so non-US-extended layouts (ISO, JIS) can still be
/// resolved if their `v` lives outside the ANSI range.
const MAX_KEYCODE: u16 = 127;
const TARGET_CHAR: u16 = b'v' as u16;
#[link(name = "Carbon", kind = "framework")]
extern "C" {
fn TISCopyCurrentKeyboardLayoutInputSource() -> TISInputSourceRef;
fn TISGetInputSourceProperty(
source: TISInputSourceRef,
key: CFStringRef,
) -> *mut c_void;
fn LMGetKbdType() -> u8;
fn UCKeyTranslate(
keyboard_layout: *const u8,
virtual_key_code: u16,
key_action: u16,
modifier_key_state: u32,
keyboard_type: u32,
key_translate_options: u32,
dead_key_state: *mut u32,
max_string_length: usize,
actual_string_length: *mut usize,
unicode_string: *mut u16,
) -> i32;
static kTISPropertyUnicodeKeyLayoutData: CFStringRef;
static kTISNotifySelectedKeyboardInputSourceChanged: CFStringRef;
}
pub fn init() {
resolve_into_cache();
register_layout_change_observer();
}
fn resolve_into_cache() {
let kc = resolve_v_keycode().unwrap_or(FALLBACK_V_KEYCODE);
V_KEYCODE.store(kc, Ordering::Relaxed);
}
fn resolve_v_keycode() -> Option<u16> {
unsafe {
let source = TISCopyCurrentKeyboardLayoutInputSource();
if source.is_null() {
return None;
}
let _src_guard = scopeguard::guard(source, |s| CFRelease(s as *const c_void));
let layout_data_ptr =
TISGetInputSourceProperty(source, kTISPropertyUnicodeKeyLayoutData);
if layout_data_ptr.is_null() {
return None;
}
let layout_bytes = CFDataGetBytePtr(layout_data_ptr as CFDataRef);
if layout_bytes.is_null() {
return None;
}
let kbd_type = LMGetKbdType() as u32;
for keycode in 0..=MAX_KEYCODE {
let mut dead_key_state: u32 = 0;
let mut chars: [u16; 4] = [0; 4];
let mut actual_len: usize = 0;
let status = UCKeyTranslate(
layout_bytes,
keycode,
K_UC_KEY_ACTION_DOWN,
0, // no modifiers
kbd_type,
K_UC_KEY_TRANSLATE_NO_DEAD_KEYS_MASK,
&mut dead_key_state,
chars.len(),
&mut actual_len,
chars.as_mut_ptr(),
);
if status == 0 && actual_len == 1 && chars[0] == TARGET_CHAR {
return Some(keycode);
}
}
None
}
}
extern "C" fn layout_changed(
_center: CFNotificationCenterRef,
_observer: *mut c_void,
_name: CFNotificationName,
_object: *const c_void,
_user_info: CFDictionaryRef,
) {
resolve_into_cache();
}
fn register_layout_change_observer() {
unsafe {
let center = CFNotificationCenterGetDistributedCenter();
if center.is_null() {
return;
}
CFNotificationCenterAddObserver(
center,
ptr::null(),
layout_changed,
kTISNotifySelectedKeyboardInputSourceChanged,
ptr::null(),
CFNotificationSuspensionBehaviorDeliverImmediately,
);
}
}
}
tauri/src-tauri/src/main.rs
+580 -3
View File
@@ -1,16 +1,123 @@
// Prevents additional console window on Windows in release, DO NOT REMOVE!!
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
mod accessibility;
mod audio_capture;
mod audio_output;
mod clipboard;
mod focus_capture;
#[cfg(desktop)]
mod hotkey_monitor;
mod input_monitoring;
#[cfg(desktop)]
mod key_codes;
mod keyboard_layout;
mod speak_monitor;
mod synthetic_keys;
use std::sync::Mutex;
use tauri::{command, State, Manager, WindowEvent, Emitter, Listener, RunEvent};
use tauri::{command, State, Manager, WindowEvent, Emitter, Listener, RunEvent, WebviewUrl, WebviewWindowBuilder, PhysicalPosition};
use tauri_plugin_shell::ShellExt;
use tokio::sync::mpsc;
pub const DICTATE_WINDOW_LABEL: &str = "dictate";
const DICTATE_WINDOW_WIDTH: f64 = 420.0;
const DICTATE_WINDOW_HEIGHT: f64 = 64.0;
/// Create the floating dictate webview hidden. The HotkeyMonitor shows it on
/// chord-start; the frontend hides it when the capture pipeline finishes.
/// Building it at setup avoids a race where the first chord or agent-speech
/// event fires before the webview subscribes to the `dictate:*` events.
#[cfg(desktop)]
fn build_dictate_window(app: &tauri::AppHandle) -> tauri::Result<tauri::WebviewWindow> {
let window = WebviewWindowBuilder::new(
app,
DICTATE_WINDOW_LABEL,
WebviewUrl::App("?view=dictate".into()),
)
.title("Voicebox Dictate")
.inner_size(DICTATE_WINDOW_WIDTH, DICTATE_WINDOW_HEIGHT)
.decorations(false)
.transparent(true)
.always_on_top(true)
// Follow the user across macOS Spaces / virtual desktops instead of
// being pinned to the Space where the window was first created.
.visible_on_all_workspaces(true)
.skip_taskbar(true)
.resizable(false)
.shadow(false)
.visible(false)
.build()?;
if let Some(monitor) = window.current_monitor()? {
let monitor_size = monitor.size();
let win_size = window.outer_size()?;
let x = (monitor_size.width as i32 - win_size.width as i32) / 2;
let y = (monitor_size.height as f64 * 0.04) as i32;
window.set_position(PhysicalPosition::new(x, y))?;
}
Ok(window)
}
/// Position, undo click-through, and show the dictate pill window.
///
/// The hide path parks the window at (-10_000, -10_000) and toggles
/// `ignore_cursor_events(true)` so invisible click targets don't leak; we
/// undo both here. Mirrors the logic the hotkey_monitor's
/// `Effect::StartRecording` path runs, minus the focus snapshot — this is
/// for agent-initiated speech, not dictation, so there's no focused text
/// field to paste into.
/// Build the pill webview if it doesn't exist yet. Idempotent — used by
/// agent-speech to prime the webview on speak-start so its listeners can
/// register before the actual show arrives from `audio.onplaying`.
#[cfg(desktop)]
pub fn ensure_dictate_window(app: &tauri::AppHandle) {
if app.get_webview_window(DICTATE_WINDOW_LABEL).is_none() {
if let Err(e) = build_dictate_window(app) {
eprintln!("ensure_dictate_window: failed to build pill: {e}");
}
}
}
#[cfg(desktop)]
pub fn show_dictate_window(app: &tauri::AppHandle) {
// Build on demand so agent-initiated speech works before the user has
// enabled the global hotkey (the hotkey path is the other place this
// window gets built, see `enable_hotkey`).
let window = match app.get_webview_window(DICTATE_WINDOW_LABEL) {
Some(w) => w,
None => match build_dictate_window(app) {
Ok(w) => w,
Err(e) => {
eprintln!("show_dictate_window: failed to build pill window: {e}");
return;
}
},
};
// current_monitor() returns None when the window has been parked
// off any display by the hide path; fall back to the primary.
let monitor = window
.current_monitor()
.ok()
.flatten()
.or_else(|| window.primary_monitor().ok().flatten());
if let Some(monitor) = monitor {
let monitor_pos = monitor.position();
let monitor_size = monitor.size();
if let Ok(win_size) = window.outer_size() {
let x = monitor_pos.x
+ (monitor_size.width as i32 - win_size.width as i32) / 2;
let y = monitor_pos.y + (monitor_size.height as f64 * 0.04) as i32;
let _ = window.set_position(PhysicalPosition::new(x, y));
}
}
let _ = window.set_ignore_cursor_events(false);
let _ = window.show();
}
const LEGACY_PORT: u16 = 8000;
const SERVER_PORT: u16 = 17493;
pub(crate) const SERVER_PORT: u16 = 17493;
/// Find a voicebox-server process listening on a given port (Windows only).
///
@@ -709,6 +816,418 @@ fn stop_audio_playback(
state.stop_all_playback()
}
/// Identifier of the Voicebox app itself — used to short-circuit auto-paste
/// when the user fires a chord while focus was inside one of our own
/// windows. Paste into Voicebox-internal targets is step 6 territory and
/// goes through a different (JS-side) injection path.
///
/// Value matches what `focus_capture::capture_focus` writes into
/// `FocusSnapshot::bundle_id` on the current platform — reverse-DNS bundle
/// id on macOS, lowercased exe basename on Windows/Linux.
#[cfg(target_os = "macos")]
const VOICEBOX_BUNDLE_ID: &str = "sh.voicebox.app";
#[cfg(target_os = "windows")]
const VOICEBOX_BUNDLE_ID: &str = "voicebox.exe";
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
const VOICEBOX_BUNDLE_ID: &str = "voicebox";
/// Milliseconds to wait between activating the target app and firing the
/// synthetic ⌘V, giving AppKit time to finish re-ordering windows and
/// restoring its last-focused field.
const POST_ACTIVATE_SETTLE_MS: u64 = 120;
/// Milliseconds the staged text lives on the clipboard after the paste
/// keystroke, before we restore the user's original clipboard contents.
/// Too short and slow apps haven't consumed the paste yet; too long and
/// the user sees our text if they look at their clipboard manager.
const PASTE_CONSUME_MS: u64 = 400;
/// Reports whether the process currently has macOS Accessibility trust.
/// Used by the settings UI and the paste debug harness to decide whether
/// synthetic key events will actually land.
#[command]
fn check_accessibility_permission() -> bool {
accessibility::is_trusted()
}
/// Reports whether the process can observe global keyboard events. Read by
/// the Captures settings UI to surface a "missing — open Settings" hint
/// beside the hotkey toggle. No prompt side-effect.
#[command]
fn check_input_monitoring_permission() -> bool {
input_monitoring::is_trusted()
}
/// Holds the lazily-spawned global hotkey monitor. The monitor is `None`
/// until the user opts in via the Captures settings toggle — that opt-in is
/// what triggers the macOS Input Monitoring TCC prompt, so a fresh-install
/// user who never enables the hotkey never sees the prompt.
///
/// Disabling the hotkey clears the monitor's internal `ChordMatcher` so
/// keytap's event tap is released while Tauri still owns this `HotkeyState`
/// for the rest of the process. A subsequent enable re-arms without
/// re-prompting for the Input Monitoring permission.
#[cfg(desktop)]
#[derive(Default)]
pub struct HotkeyState {
monitor: Mutex<Option<hotkey_monitor::HotkeyMonitor>>,
}
#[cfg(desktop)]
fn build_chord_bindings(
push_to_talk: &[String],
toggle_to_talk: &[String],
) -> Result<hotkey_monitor::Bindings, String> {
use hotkey_monitor::{Bindings, ChordAction};
use keytap::Key;
use std::collections::HashSet;
fn build_chord(name: &str, names: &[String]) -> Result<HashSet<Key>, String> {
if names.is_empty() {
return Err(format!("{name} chord must have at least one key"));
}
let mut chord = HashSet::new();
for raw in names {
let key = key_codes::key_from_str(raw)
.ok_or_else(|| format!("Unsupported key in {name} chord: {raw}"))?;
chord.insert(key);
}
Ok(chord)
}
let push_chord = build_chord("push-to-talk", push_to_talk)?;
let toggle_chord = build_chord("toggle-to-talk", toggle_to_talk)?;
let mut bindings = Bindings::new();
bindings.insert(ChordAction::PushToTalk, push_chord);
bindings.insert(ChordAction::ToggleToTalk, toggle_chord);
Ok(bindings)
}
/// Spawn the global hotkey monitor on first call; subsequent calls just push
/// the new bindings into the existing monitor. Idempotent on purpose — the
/// frontend invokes this both at startup (when `capture_settings.hotkey_enabled`
/// is true) and from the settings toggle.
///
/// On macOS this is the call that triggers the "Voicebox would like to receive
/// keystrokes from any application" TCC prompt, since keytap's `Tap` creates
/// the CGEventTap inside `HotkeyMonitor::spawn`.
#[cfg(desktop)]
#[command]
fn enable_hotkey(
app: tauri::AppHandle,
state: State<'_, HotkeyState>,
push_to_talk: Vec<String>,
toggle_to_talk: Vec<String>,
) -> Result<(), String> {
let bindings = build_chord_bindings(&push_to_talk, &toggle_to_talk)?;
// Fire the Input Monitoring TCC prompt explicitly from the user's
// toggle click, before keytap's Tap would do it implicitly via
// CGEventTap creation. Two reasons: (1) the prompt timing becomes
// deterministic — it appears in response to a click instead of as a
// mysterious side-effect of "the app started"; (2) on subsequent
// launches we can short-circuit the spawn entirely if the user
// revoked the grant, instead of relying on the tap silently failing.
// The call returns the current grant state; we ignore it because
// keytap surfaces its own error via stderr, and the settings UI
// polls `check_input_monitoring_permission` separately.
let _ = input_monitoring::request();
// The dictate pill webview must exist before the first chord fires so it
// can subscribe to `dictate:start`. Build it here (idempotent — Tauri
// returns the existing window when one with this label already exists).
if app.get_webview_window(DICTATE_WINDOW_LABEL).is_none() {
if let Err(e) = build_dictate_window(&app) {
eprintln!("Failed to build dictate window: {}", e);
}
}
let mut slot = state.monitor.lock().map_err(|e| e.to_string())?;
match slot.as_mut() {
Some(monitor) => monitor.update_bindings(bindings),
None => {
*slot = Some(hotkey_monitor::HotkeyMonitor::spawn(app, bindings));
}
}
Ok(())
}
/// Quiet the global hotkey. Tears down the `ChordMatcher` (which stops
/// keytap's chord worker and closes the OS event tap) but keeps the
/// `HotkeyMonitor` handle around so a subsequent `enable_hotkey` re-arms
/// without re-prompting for Input Monitoring permission.
#[cfg(desktop)]
#[command]
fn disable_hotkey(state: State<'_, HotkeyState>) -> Result<(), String> {
let mut slot = state.monitor.lock().map_err(|e| e.to_string())?;
if let Some(monitor) = slot.as_mut() {
monitor.update_bindings(hotkey_monitor::Bindings::new());
}
Ok(())
}
/// Push a new chord configuration into the running `HotkeyMonitor`. Called
/// by the chord-picker UI when the user edits the chord. No-ops when the
/// monitor isn't spawned — the picker is gated behind the enable toggle, so
/// this can only happen if the frontend races; the next `enable_hotkey` will
/// pick up the saved chords.
///
/// Returns an error when a key name doesn't map to a `keytap::Key`, so the
/// picker UI can surface "this key isn't supported" instead of silently
/// dropping it from the chord.
#[cfg(desktop)]
#[command]
fn update_chord_bindings(
state: State<'_, HotkeyState>,
push_to_talk: Vec<String>,
toggle_to_talk: Vec<String>,
) -> Result<(), String> {
let bindings = build_chord_bindings(&push_to_talk, &toggle_to_talk)?;
let mut slot = state.monitor.lock().map_err(|e| e.to_string())?;
if let Some(monitor) = slot.as_mut() {
monitor.update_bindings(bindings);
}
Ok(())
}
/// Open the Privacy & Security → Accessibility pane in System Settings so
/// the user can grant the permission. The URL scheme is stable across
/// macOS 10.14–15; no-op on other platforms.
#[command]
fn open_accessibility_settings(app: tauri::AppHandle) -> Result<(), String> {
#[cfg(target_os = "macos")]
{
let url = "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility";
app.shell()
.open(url, None)
.map_err(|e| format!("Failed to open Accessibility settings: {e}"))?;
Ok(())
}
#[cfg(not(target_os = "macos"))]
{
let _ = app;
Err("Accessibility settings pane is only implemented on macOS".into())
}
}
/// Open the Privacy & Security → Input Monitoring pane in System Settings.
/// Used by the Captures settings UI when the toggle is on but the grant
/// is missing, so the user can flip the system toggle without hunting.
#[command]
fn open_input_monitoring_settings(app: tauri::AppHandle) -> Result<(), String> {
#[cfg(target_os = "macos")]
{
let url = "x-apple.systempreferences:com.apple.preference.security?Privacy_ListenEvent";
app.shell()
.open(url, None)
.map_err(|e| format!("Failed to open Input Monitoring settings: {e}"))?;
Ok(())
}
#[cfg(not(target_os = "macos"))]
{
let _ = app;
Err("Input Monitoring settings pane is only implemented on macOS".into())
}
}
/// Deliver `text` into the UI that had focus when the chord fired.
///
/// Pipeline: activate the captured PID → settle → save the user's
/// clipboard → write `text` → fire ⌘V → wait for the target to consume it
/// → conditionally restore the original clipboard.
///
/// The restore is conditional on `NSPasteboard.changeCount` (or the
/// Windows sequence number) matching the value captured right after
/// `write_text`: if something else wrote to the clipboard during the
/// paste-consume window — the user's own ⌘C in the target app, a
/// clipboard history tool (Paste, Pastebot, Maccy), Universal Clipboard
/// sync, 1Password inserting a secret — their newer content takes
/// priority over our snapshot and is preserved. A
/// [`clipboard::current_change_count`] read failure is treated the same
/// way: unknown state is safer than an unconditional overwrite.
///
/// `send_paste` failure is isolated from the restore decision: we always
/// attempt the conditional restore before propagating the paste error,
/// so a failed `CGEventPost` / `SendInput` never leaves the user's
/// clipboard stuck on the transcript.
///
/// Skips (returns `false`) without touching anything when:
/// - `focus.bundle_id` is Voicebox itself — step 6 will inject directly
/// into our own webview; pasting would just double-insert or miss the
/// real target.
/// - Accessibility is not trusted — `CGEventPost` would silently drop the
/// keystroke, leaving the user's clipboard clobbered with nothing to
/// show for it.
///
/// Returns `true` when the paste sequence completed end-to-end.
#[command]
async fn paste_final_text(
text: String,
focus: focus_capture::FocusSnapshot,
) -> Result<bool, String> {
if focus.bundle_id.as_deref() == Some(VOICEBOX_BUNDLE_ID) {
return Ok(false);
}
if !accessibility::is_trusted() {
return Err(
"Accessibility permission required for auto-paste. Open System Settings → Privacy & Security → Accessibility and enable Voicebox."
.into(),
);
}
focus_capture::activate_pid(focus.pid)?;
tokio::time::sleep(std::time::Duration::from_millis(POST_ACTIVATE_SETTLE_MS)).await;
let snapshot = clipboard::save_clipboard()?;
let after_write = clipboard::write_text(&text)?;
let paste_result = synthetic_keys::send_paste();
tokio::time::sleep(std::time::Duration::from_millis(PASTE_CONSUME_MS)).await;
let safe_to_restore = matches!(
clipboard::current_change_count(),
Ok(current) if current == after_write
);
if safe_to_restore {
clipboard::restore_clipboard(&snapshot)?;
} else {
eprintln!(
"[voicebox] clipboard mutated during paste window — skipping restore to preserve newer content"
);
}
paste_result?;
Ok(true)
}
/// Inspect the currently focused UI element. Returns the owning app's PID,
/// bundle id, and AX role. Useful for sanity-checking the focus pipeline
/// before committing to a paste.
#[command]
fn debug_capture_focus() -> Result<focus_capture::FocusSnapshot, String> {
focus_capture::capture_focus()
}
/// Full auto-paste rehearsal: snapshot the focus target now, sleep
/// `drift_ms` so the user can deliberately switch to a different app
/// (proving we don't paste into whichever window is frontmost when the
/// transcribe finishes), then activate the captured PID, stage `text`,
/// fire ⌘V, and restore the clipboard.
#[command]
async fn debug_focus_roundtrip(
text: String,
drift_ms: u64,
post_paste_delay_ms: u64,
) -> Result<serde_json::Value, String> {
if !accessibility::is_trusted() {
return Err(
"Accessibility permission not granted. Open System Settings → Privacy & Security → Accessibility and enable Voicebox."
.into(),
);
}
let snapshot = focus_capture::capture_focus()?;
tokio::time::sleep(std::time::Duration::from_millis(drift_ms)).await;
focus_capture::activate_pid(snapshot.pid)?;
// Give AppKit a beat to process the activation before the synthetic
// Cmd+V arrives — without this the paste sometimes races ahead of the
// window-ordering animation and lands in the previous frontmost app.
tokio::time::sleep(std::time::Duration::from_millis(120)).await;
let clip = clipboard::save_clipboard()?;
let after_write = clipboard::write_text(&text)?;
synthetic_keys::send_paste()?;
tokio::time::sleep(std::time::Duration::from_millis(post_paste_delay_ms)).await;
let before_restore = clipboard::current_change_count()?;
clipboard::restore_clipboard(&clip)?;
Ok(serde_json::json!({
"focus": snapshot,
"change_count_after_write": after_write,
"change_count_before_restore": before_restore,
"clobbered_during_paste": before_restore != after_write,
}))
}
/// End-to-end smoke test for the auto-paste pipeline: save the user's
/// clipboard, stage `text`, optionally wait `pre_paste_delay_ms` so the
/// caller has time to focus the target app, synthesise ⌘V, wait
/// `post_paste_delay_ms` for the target app to consume the event, and put
/// the original clipboard back.
///
/// Short-circuits when Accessibility permission is missing — without it
/// `CGEventPost` silently drops events, so running the full sequence
/// would just clobber the clipboard with nothing to show for it.
#[command]
async fn debug_paste_text(
text: String,
pre_paste_delay_ms: u64,
post_paste_delay_ms: u64,
) -> Result<serde_json::Value, String> {
if !accessibility::is_trusted() {
return Err(
"Accessibility permission not granted. Open System Settings → Privacy & Security → Accessibility and enable Voicebox, then try again."
.into(),
);
}
let snapshot = clipboard::save_clipboard()?;
let before = snapshot.change_count();
let after_write = clipboard::write_text(&text)?;
tokio::time::sleep(std::time::Duration::from_millis(pre_paste_delay_ms)).await;
synthetic_keys::send_paste()?;
tokio::time::sleep(std::time::Duration::from_millis(post_paste_delay_ms)).await;
let before_restore = clipboard::current_change_count()?;
clipboard::restore_clipboard(&snapshot)?;
let after_restore = clipboard::current_change_count()?;
Ok(serde_json::json!({
"change_count_before": before,
"change_count_after_write": after_write,
"change_count_before_restore": before_restore,
"change_count_after_restore": after_restore,
"clobbered_during_paste": before_restore != after_write,
}))
}
/// Manual smoke test for the clipboard snapshot/restore primitives used by
/// the auto-paste pipeline. Stages `text` on the pasteboard, waits
/// `hold_ms` so the caller can ⌘V into another app, then puts the original
/// clipboard contents back. The return value reports the change-count deltas
/// so the harness can verify no third party mutated the clipboard mid-paste.
#[command]
async fn debug_clipboard_roundtrip(
text: String,
hold_ms: u64,
) -> Result<serde_json::Value, String> {
let snapshot = clipboard::save_clipboard()?;
let before = snapshot.change_count();
let item_count = snapshot.item_count();
let after_write = clipboard::write_text(&text)?;
tokio::time::sleep(std::time::Duration::from_millis(hold_ms)).await;
let before_restore = clipboard::current_change_count()?;
clipboard::restore_clipboard(&snapshot)?;
let after_restore = clipboard::current_change_count()?;
Ok(serde_json::json!({
"saved_items": item_count,
"change_count_before": before,
"change_count_after_write": after_write,
"change_count_before_restore": before_restore,
"change_count_after_restore": after_restore,
"clobbered_during_hold": before_restore != after_write,
}))
}
#[cfg_attr(mobile, tauri::mobile_entry_point)]
pub fn run() {
tauri::Builder::default()
@@ -728,6 +1247,52 @@ pub fn run() {
{
app.handle().plugin(tauri_plugin_updater::Builder::new().build())?;
app.handle().plugin(tauri_plugin_process::init())?;
// Resolve the active keyboard layout's V keycode now, on
// the main thread, and register an observer for layout
// changes. The synthetic-paste hot path then only reads an
// atomic. See keyboard_layout.rs for why this matters
// (Cmd+V is matched by translated character, not keycode,
// so QWERTY keycode 9 produces Cmd+. on Dvorak).
keyboard_layout::init();
// HotkeyMonitor is spawned lazily via the `enable_hotkey`
// command — see HotkeyState. The hidden dictate webview is
// safe to build up front because it does not create the global
// keyboard tap or trigger the macOS Input Monitoring prompt.
app.manage(HotkeyState::default());
// The frontend emits `dictate:hide` whenever the pill cycle
// finishes (rest-fade → hidden). `hide()` alone has been
// unreliable for transparent always-on-top windows on macOS
// — the NSWindow lingers as an invisible click target that
// steals focus to the Voicebox app when the user clicks
// where it used to be. Park the window off-screen and mark
// it click-through as well, so even if `hide()` no-ops the
// user sees and interacts with nothing.
let handle_for_hide = app.handle().clone();
app.handle().listen("dictate:hide", move |_event| {
if let Some(window) = handle_for_hide.get_webview_window(DICTATE_WINDOW_LABEL) {
let _ = window.set_ignore_cursor_events(true);
let _ = window.set_position(PhysicalPosition::new(-10_000, -10_000));
let _ = window.hide();
}
});
// Agent-initiated speech (voicebox.speak over MCP or POST /speak)
// pops the pill up so the user can see what's coming out of their
// machine. The `dictate:show` listener is kept for any frontend
// caller that wants to force-surface the pill directly, but the
// primary source is `speak_monitor` below — Rust subscribes to
// the backend /events/speak SSE stream so the pill surfaces even
// when no JS window is active.
let handle_for_show = app.handle().clone();
app.handle().listen("dictate:show", move |_event| {
show_dictate_window(&handle_for_show);
});
ensure_dictate_window(app.handle());
speak_monitor::spawn_speak_monitor(app.handle().clone());
}
// Hide title bar icon on Windows
@@ -797,7 +1362,19 @@ pub fn run() {
is_system_audio_supported,
list_audio_output_devices,
play_audio_to_devices,
stop_audio_playback
stop_audio_playback,
debug_clipboard_roundtrip,
debug_paste_text,
debug_capture_focus,
debug_focus_roundtrip,
check_accessibility_permission,
check_input_monitoring_permission,
open_accessibility_settings,
open_input_monitoring_settings,
paste_final_text,
enable_hotkey,
disable_hotkey,
update_chord_bindings
])
.on_window_event({
let closing = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
tauri/src-tauri/src/speak_monitor.rs
+180
View File
@@ -0,0 +1,180 @@
//! Rust-side subscriber for the backend `/events/speak` SSE stream.
//!
//! Owns the pill-window lifecycle for agent-initiated speech. The dictate
//! webview used to do this itself via `EventSource`, but hidden WebKit
//! windows on macOS throttle long-lived network connections, so speak events
//! never reached the pill. Tauri's event bus, on the other hand, reliably
//! delivers events to hidden webviews (the chord path proves it), so we
//! subscribe here and fan out via `emit`.
//!
//! Flow:
//! backend speak-start → show dictate window + emit("dictate:speak-start")
//! backend speak-end → emit("dictate:speak-end")
//! The pill webview handles the rest (audio playback, then emits
//! `dictate:hide` back to Rust when the audio element's `ended` fires).
//!
//! Reconnect policy: idle-timeout + escalating backoff. The stream is
//! infinite by design, so a successful round means "we were receiving
//! frames and then the backend closed the connection" (typically a
//! server restart) — reset backoff and reconnect quickly. A failure or
//! a round that produced no frames escalates backoff up to a 30 s cap so
//! long-term outages stop filling stderr with reconnect log lines.
//!
//! The idle timeout guards against the worst silent-failure mode: a
//! backend that accepts the TCP connection but stops producing frames
//! (deadlocked SSE endpoint, zombie process). Without a timeout the
//! `chunk().await` blocks forever and the task never notices. The
//! backend emits a `:ping` comment every 15 s, so 45 s without any data
//! is a reliable signal the stream is dead.
use std::time::Duration;
use tauri::{AppHandle, Emitter};
use crate::{ensure_dictate_window, SERVER_PORT};
const INITIAL_BACKOFF: Duration = Duration::from_millis(500);
const MAX_BACKOFF: Duration = Duration::from_secs(30);
/// Backend emits a `:ping` heartbeat every 15 s. Giving the stream 45 s
/// of idle budget absorbs one missed heartbeat (slow GC pause, brief
/// backend stall) without being so long that a truly dead stream blocks
/// the pill from surfacing for minutes.
const STREAM_IDLE_TIMEOUT: Duration = Duration::from_secs(45);
pub fn spawn_speak_monitor(app: AppHandle) {
tauri::async_runtime::spawn(async move {
run(app).await;
});
}
async fn run(app: AppHandle) {
let url = format!("http://127.0.0.1:{}/events/speak", SERVER_PORT);
let client = match reqwest::Client::builder().build() {
Ok(c) => c,
Err(e) => {
eprintln!("speak_monitor: failed to build HTTP client: {e}");
return;
}
};
let mut backoff = INITIAL_BACKOFF;
let mut attempt: u32 = 0;
loop {
let stream_result = stream_once(&client, &url, &app).await;
let had_success = matches!(stream_result, Ok(true));
if had_success {
backoff = INITIAL_BACKOFF;
attempt = 0;
} else {
attempt += 1;
let reason = match stream_result {
Ok(_) => "stream closed without data".to_string(),
Err(e) => format!("stream err: {e}"),
};
eprintln!(
"speak_monitor: {reason} (attempt {attempt}, retry in {:?})",
backoff
);
}
tokio::time::sleep(backoff).await;
if !had_success {
backoff = (backoff * 2).min(MAX_BACKOFF);
}
}
}
/// Consume the SSE stream until it closes or errors. Returns `Ok(true)`
/// if at least one frame was received (the connection was genuinely
/// productive), `Ok(false)` on a clean but empty close, and `Err` for
/// any connection or parse failure.
async fn stream_once(
client: &reqwest::Client,
url: &str,
app: &AppHandle,
) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
let mut resp = client
.get(url)
.header("Accept", "text/event-stream")
.send()
.await?;
if !resp.status().is_success() {
return Err(format!("speak_monitor: backend returned {}", resp.status()).into());
}
let mut buf = String::new();
let mut saw_data = false;
loop {
let chunk = match tokio::time::timeout(STREAM_IDLE_TIMEOUT, resp.chunk()).await {
Ok(Ok(Some(chunk))) => chunk,
Ok(Ok(None)) => return Ok(saw_data),
Ok(Err(e)) => return Err(Box::new(e)),
Err(_) => {
return Err(format!(
"no data for {:?} (heartbeat should arrive every 15 s)",
STREAM_IDLE_TIMEOUT
)
.into())
}
};
saw_data = true;
buf.push_str(std::str::from_utf8(&chunk)?);
// sse-starlette emits CRLF framing; the spec also permits LF, so
// handle either. Drain whichever separator appears first.
loop {
let crlf = buf.find("\r\n\r\n");
let lf = buf.find("\n\n");
let (end, sep_len) = match (crlf, lf) {
(Some(c), Some(l)) if c <= l => (c, 4),
(Some(c), None) => (c, 4),
(_, Some(l)) => (l, 2),
(None, None) => break,
};
let frame: String = buf.drain(..end + sep_len).collect();
if let Some((event, data)) = parse_frame(&frame) {
dispatch(app, &event, &data);
}
}
}
}
/// Parse a single SSE frame into (event_name, data_json).
///
/// Returns None for comment-only frames (lines starting with `:`) and
/// for frames without a recognizable `event:` or `data:` line.
fn parse_frame(frame: &str) -> Option<(String, String)> {
let mut event: Option<String> = None;
let mut data_lines: Vec<&str> = Vec::new();
for line in frame.lines() {
if line.is_empty() || line.starts_with(':') {
continue;
}
if let Some(rest) = line.strip_prefix("event:") {
event = Some(rest.trim().to_string());
} else if let Some(rest) = line.strip_prefix("data:") {
data_lines.push(rest.trim_start());
}
}
let event = event?;
let data = data_lines.join("\n");
Some((event, data))
}
fn dispatch(app: &AppHandle, event: &str, data: &str) {
match event {
"speak-start" => {
// Defensive for dev/restart paths where the setup-created pill
// is not present — but don't *show* it here. The pill
// surfaces itself from `audio.onplaying` via `dictate:show`, so
// users never see the empty-silent generation window.
ensure_dictate_window(app);
let _ = app.emit("dictate:speak-start", data.to_string());
}
"speak-end" => {
let _ = app.emit("dictate:speak-end", data.to_string());
}
// `ready` and `ping` are heartbeats; ignore.
_ => {}
}
}
tauri/src-tauri/src/synthetic_keys.rs
+205
View File
@@ -0,0 +1,205 @@
//! Synthetic keyboard event posting for the auto-paste pipeline.
//!
//! `send_paste` fires the four-event paste sequence onto the OS input
//! pipeline so the focused app performs its native paste action against
//! whatever the clipboard module has just staged.
//!
//! - **macOS** — Cmd down, V down with Cmd flag, V up with Cmd flag, Cmd
//! up via `CGEventPost` at `kCGHIDEventTap`. Accessibility permission is
//! load-bearing: without it the system swallows the events silently, so
//! callers must gate on [`crate::accessibility::is_trusted`].
//! - **Windows** — Ctrl down, V down, V up, Ctrl up via `SendInput`. No
//! permission gate, but UAC/UIPI blocks delivery into elevated target
//! windows when we run non-elevated — nothing we can do short of also
//! running elevated.
//!
//! On macOS the V keycode is resolved per-layout by
//! [`crate::keyboard_layout`] — Cmd+V is matched against the layout-
//! translated character via NSMenu key equivalents, so hardcoding
//! `kVK_ANSI_V` (the QWERTY V position) would fire Cmd+. on Dvorak. The
//! resolved keycode is read once per paste from an atomic; the cache is
//! primed at startup and refreshed on layout change.
//!
//! Windows hardcodes `VK_V`. `SendInput` with `wVk = VK_V` makes the
//! target receive `WM_KEYDOWN` with `wParam = VK_V` regardless of the
//! active layout, and most Windows apps treat that as Ctrl+V (the same
//! reason `Send "^v"` works in AutoHotkey on Dvorak Windows).
#[cfg(target_os = "macos")]
use std::ffi::c_void;
#[cfg(target_os = "macos")]
mod ffi {
use std::ffi::c_void;
#[repr(C)]
pub struct CGEvent {
_opaque: [u8; 0],
}
pub type CGEventRef = *mut CGEvent;
#[repr(C)]
pub struct CGEventSource {
_opaque: [u8; 0],
}
pub type CGEventSourceRef = *mut CGEventSource;
pub type CGEventTapLocation = u32;
pub type CGKeyCode = u16;
pub type CGEventFlags = u64;
pub type CGEventSourceStateID = i32;
/// `kCGHIDEventTap` — posted events enter at the HID level so every
/// downstream tap (including the target app) sees them exactly as if the
/// hardware had produced them.
pub const K_CG_HID_EVENT_TAP: CGEventTapLocation = 0;
/// `kCGEventSourceStateHIDSystemState` — mimics hardware, which is what
/// we want: modifier bookkeeping inside target apps stays consistent.
pub const K_CG_EVENT_SOURCE_STATE_HID_SYSTEM_STATE: CGEventSourceStateID = 1;
/// `kCGEventFlagMaskCommand` — the Cmd modifier bit inside `CGEventFlags`.
pub const K_CG_EVENT_FLAG_MASK_COMMAND: CGEventFlags = 0x00100000;
/// `kVK_Command` (left Cmd).
pub const KEYCODE_LEFT_CMD: CGKeyCode = 0x37;
#[link(name = "CoreGraphics", kind = "framework")]
extern "C" {
pub fn CGEventSourceCreate(state_id: CGEventSourceStateID) -> CGEventSourceRef;
pub fn CGEventCreateKeyboardEvent(
source: CGEventSourceRef,
virtual_key: CGKeyCode,
key_down: bool,
) -> CGEventRef;
pub fn CGEventSetFlags(event: CGEventRef, flags: CGEventFlags);
pub fn CGEventPost(tap: CGEventTapLocation, event: CGEventRef);
}
#[link(name = "CoreFoundation", kind = "framework")]
extern "C" {
pub fn CFRelease(cf: *const c_void);
}
}
/// Post the four-event Cmd+V sequence to the HID event tap.
///
/// Returns after the events are queued — there's no completion callback,
/// so callers should sleep briefly afterwards to let the target app
/// process the paste before any follow-up (e.g. clipboard restore).
#[cfg(target_os = "macos")]
pub fn send_paste() -> Result<(), String> {
use ffi::*;
let v_keycode = crate::keyboard_layout::paste_keycode_v();
unsafe {
let source = CGEventSourceCreate(K_CG_EVENT_SOURCE_STATE_HID_SYSTEM_STATE);
if source.is_null() {
return Err("CGEventSourceCreate returned null".into());
}
let _source_guard = scopeguard::guard(source, |s| CFRelease(s as *const c_void));
let events = [
(KEYCODE_LEFT_CMD, true, 0),
(v_keycode, true, K_CG_EVENT_FLAG_MASK_COMMAND),
(v_keycode, false, K_CG_EVENT_FLAG_MASK_COMMAND),
(KEYCODE_LEFT_CMD, false, 0),
];
// Build the four events up front so CFRelease happens after all posts.
// Posting in a loop that interleaved create → post → release would
// work, but keeping the events alive for the full sequence matches
// the pattern CGEventPost's docs show and is easier to reason about.
let mut guards = Vec::with_capacity(events.len());
let mut created = Vec::with_capacity(events.len());
for (key, down, flags) in events {
let event = CGEventCreateKeyboardEvent(source, key, down);
if event.is_null() {
return Err(format!(
"CGEventCreateKeyboardEvent(key={}, down={}) returned null",
key, down
));
}
let guard = scopeguard::guard(event, |e| CFRelease(e as *const c_void));
if flags != 0 {
CGEventSetFlags(event, flags);
}
created.push(event);
guards.push(guard);
}
for event in created {
CGEventPost(K_CG_HID_EVENT_TAP, event);
}
drop(guards);
Ok(())
}
}
#[cfg(target_os = "windows")]
mod win {
use windows::Win32::UI::Input::KeyboardAndMouse::{
INPUT, INPUT_0, INPUT_KEYBOARD, KEYBDINPUT, KEYBD_EVENT_FLAGS, KEYEVENTF_KEYUP,
VIRTUAL_KEY,
};
pub fn make_key(vk: VIRTUAL_KEY, up: bool) -> INPUT {
let flags = if up {
KEYEVENTF_KEYUP
} else {
KEYBD_EVENT_FLAGS(0)
};
INPUT {
r#type: INPUT_KEYBOARD,
Anonymous: INPUT_0 {
ki: KEYBDINPUT {
wVk: vk,
wScan: 0,
dwFlags: flags,
time: 0,
dwExtraInfo: 0,
},
},
}
}
}
#[cfg(target_os = "windows")]
pub fn send_paste() -> Result<(), String> {
use windows::Win32::UI::Input::KeyboardAndMouse::{
SendInput, INPUT, VK_CONTROL, VK_V,
};
// Four-event Ctrl+V sequence. Matches the macOS CGEvent pattern: the
// modifier brackets the letter so the target app sees a fully formed
// accelerator rather than a lone V. `dwExtraInfo` is zero — we're not
// tagging these as "ours" because no consumer in the paste path needs
// to distinguish synthetic events from hardware ones.
let events = [
win::make_key(VK_CONTROL, false),
win::make_key(VK_V, false),
win::make_key(VK_V, true),
win::make_key(VK_CONTROL, true),
];
unsafe {
let sent = SendInput(&events, std::mem::size_of::<INPUT>() as i32);
if sent as usize != events.len() {
return Err(format!(
"SendInput delivered {} of {} events — the input desktop may be locked (secure attention sequence) or a higher-integrity window is intercepting.",
sent,
events.len()
));
}
}
Ok(())
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
pub fn send_paste() -> Result<(), String> {
Err("synthetic paste is not yet implemented on this platform".into())
}