From 4247a89fc69b21d95165a543b72c5277d2d36a19 Mon Sep 17 00:00:00 2001 From: Sami Ahmed Date: Tue, 11 Aug 2026 03:42:07 -0700 Subject: [PATCH] Add voice-to-Hermes pipeline architecture with live OS modification design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Voice-hermes-pipeline.md: full architecture for SamiType → Hermes direct command - Live transcript overlay with inline correction - Live OS modification via voice (add widgets, change themes, move elements) - Phased rollout plan (POC → correction → context → full integration) --- docs/voice-hermes-pipeline.md | 256 ++++++++++++++++++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 docs/voice-hermes-pipeline.md diff --git a/docs/voice-hermes-pipeline.md b/docs/voice-hermes-pipeline.md new file mode 100644 index 0000000..2893b1c --- /dev/null +++ b/docs/voice-hermes-pipeline.md @@ -0,0 +1,256 @@ +# SamiType Voice → Hermes Direct Command Pipeline + +## The Vision + +Speak naturally → see your words transcribed live → correct if needed → Hermes executes. + +This is NOT a simple voice command router ("open Firefox"). This is a **voice-driven coding/system administration interface** where natural language spoken commands go directly to Hermes (the AI agent) for execution — writing code, managing servers, configuring the OS, anything Hermes can do. + +## The Flow + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Voice Command Pipeline │ +│ │ +│ 1. SPEAK 2. SEE 3. CORRECT 4. EXECUTE │ +│ │ +│ ┌──────┐ ┌────────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Mic │───▶│ Live │───▶│ Inline │───▶│ Hermes │ │ +│ │ │ │ Transcript │ │ Edit │ │ API │ │ +│ │ │ │ (streaming)│ │ (optional)│ │ │ │ +│ └──────┘ └────────────┘ └──────────┘ └────┬─────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ Result │ │ +│ │ (visual │ │ +│ │ + TTS) │ │ +│ └──────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +### Step-by-step + +1. **Speak** — Press hotkey or say wake word "Computer". Start talking naturally. +2. **Live Transcript** — Words appear on screen as you speak them, streaming, like live captions. A floating overlay bar at the top or bottom of the screen. +3. **Correct** — When you stop speaking, the transcript stays visible for 2 seconds. If SamiType misheard something: + - Click/tap the word to fix it + - Or press Tab to edit the full text + - Or just say "scratch that" to restart + - Or press Enter immediately to send as-is +4. **Execute** — Corrected text goes to Hermes API as a task. Hermes starts working. +5. **Result** — A notification shows "Hermes is working on: [task]". When done, result appears + optional TTS summary. + +## Example Sessions + +### Coding +``` +You: "Computer, create a Python script that monitors disk usage + and sends me a Telegram alert if it goes above 80 percent" + +[Live transcript appears word by word] +[2s pause — you see it's correct, press Enter] + +Hermes: "Writing disk monitor script..." +Hermes: "Done. Script at /usr/local/bin/disk-monitor.py, + systemd service enabled. Want me to test it?" +``` + +### System Admin +``` +You: "Computer, check if the triangl... no, the DashCaddy server + is responding" + +[Transcript: "check if the triangl... no, the DashCaddy server is responding"] +[You edit "triangl" → delete that part] + +Corrected: "check if the DashCaddy server is responding" +[Enter] + +Hermes: "DashCaddy is up. Response time 42ms. + Dashboard loads in 1.2s. All containers healthy." +``` + +### OS Control +``` +You: "Computer, install VLC and set it as default media player" + +[Live transcript — correct, Enter] + +Hermes: "Installing VLC... done. Set as default for video/x-matroska, + video/mp4, and audio/mpeg." +``` + +## Technical Architecture + +### Component 1: SamiType Streaming STT +- **Streaming mode**: Outputs partial transcriptions every ~200ms +- **WebSocket or Unix socket** → UI overlay receives chunks +- **Confidence scores** per word — low-confidence words highlighted in red +- **VAD (Voice Activity Detection)**: Auto-detects when you stop talking → 2s correction window + +### Component 2: Live Transcript Overlay UI +Lightweight floating widget — options: + +**Option A: KDE Plasma Widget (native)** +- Always-on-top floating panel +- QML-based, renders streaming text +- Click-to-edit inline +- Keyboard shortcuts: Enter=submit, Esc=cancel, Tab=edit mode +- Semi-transparent when idle, solid when active + +**Option B: Electron/Tauri app (cross-platform)** +- Floating window, borderless, always-on-top +- Web tech (React/Svelte) for rapid iteration +- Works on any desktop, not just KDE + +**Option C: Terminal overlay (tmux/terminal)** +- Simplest — a tmux pane or terminal window +- Good for development, not daily use + +**Recommendation**: Start with B (Tauri — lightweight, ~10MB, Rust backend), graduate to A (native KDE widget) when SamiType is mature. + +### Component 3: Correction Layer +``` +Transcript: "create a python script that monitors [disk] usage" + highlighted = low confidence ^ + +Click "disk" → text field appears → type "disc" → Enter +Or: ignore, press Enter to send as-is (Hermes will understand context anyway) +``` + +- **Low-confidence words** (< 80% certainty) highlighted in amber/red +- **Click any word** to edit inline +- **Tab** → enter full-edit mode (entire text becomes editable) +- **Auto-submit timer**: 2s after speech ends, auto-sends unless editing +- **"Scratch that"** → clears and restarts listening + +### Component 4: Hermes API Bridge +```python +# Voice command → Hermes HTTP API +import requests + +def send_to_hermes(transcript: str): + response = requests.post("http://localhost:8765/command", json={ + "text": transcript, + "source": "voice", + "context": { + "hostname": "samios", + "user": "sami", + "cwd": os.getcwd(), + "active_app": get_active_window() + } + }) + return response.json() + +# Hermes processes it as a normal task — same as if Sami typed it +``` + +### Component 5: Result Display +- **Notification**: KDE notification "Hermes: [task summary]" with status +- **TTS**: Optional spoken confirmation using SamiType TTS +- **Full output**: Click notification → opens Konsole with full output log + +## Implementation Phases + +### Phase 1: Proof of Concept (1-2 weeks) +- [ ] SamiType running in streaming mode, outputting to stdout/JSON +- [ ] Simple Python/Tauri overlay showing live transcript +- [ ] Enter key → send to Hermes CLI (`hermes ask "transcript text"`) +- [ ] Basic notification on completion +- **Milestone**: Voice "create a file called test.txt" → Hermes creates it → notification + +### Phase 2: Correction + Polish (2-3 weeks) +- [ ] Low-confidence word highlighting +- [ ] Inline word editing +- [ ] 2s auto-submit timer with visual countdown +- [ ] "Scratch that" voice command to restart +- [ ] Result TTS via SamiType +- **Milestone**: Natural correction flow, reliable daily use + +### Phase 3: Context Awareness (3-4 weeks) +- [ ] Active window/app context passed to Hermes +- [ ] "Fix this function" when code editor is focused → Hermes sees the file +- [ ] "What's wrong with this page" when browser is focused → Hermes checks URL +- [ ] Multi-turn: "Now make it async" (Hermes remembers previous command) + +### Phase 4: Full Integration (ongoing) +- [ ] Native KDE Plasma widget +- [ ] Systemd service (always available) +- [ ] Wake word "Computer" (openWakeWord) +- [ ] Custom command shortcuts ("Computer, deploy" = predefined multi-step task) +- [ ] Voice history log + replay + +## Live OS Modification (Killer Feature) + +The most powerful use case: **designing and modifying the OS in real-time via voice.** + +``` +You: "Computer, add a weather widget to the top right corner" +[Enter] +Hermes: "Adding weather widget..." +→ KDE plasma widget installed, positioned, configured +→ Widget appears on your screen 3 seconds later + +You: "Move it down a bit" +[Enter] +→ Widget shifts down 50px + +You: "Make it show 5 days instead of 3" +[Enter] +→ Widget reconfigures, shows 5-day forecast +``` + +### How This Works + +1. **Hermes receives voice command** with desktop context (active screen, resolution, current layout) +2. **Hermes generates the config change** (Plasma widget config, kwin rule, CSS/SVG theme edit) +3. **Hermes pushes the change** via: + - `qdbus` calls to Plasma (move/resize/configure widgets) + - File writes to `~/.config/` (theme changes, panel configs) + - `kstart` / `kquitapp` to reload components +4. **Change appears live** — no reboot, no logout, instant visual update + +### Supported Live Modifications +| Voice Command | Mechanism | +|---------------|-----------| +| "Add [widget] to [position]" | Install + `plasma-shell` script | +| "Move [element] [direction]" | `qdbus org.kde.plasmashell` | +| "Change wallpaper to [name]" | `plasma-apply-wallpaper` | +| "Make panel transparent" | Edit panel config → `qdbus` reload | +| "Change accent color to blue" | `plasma-apply-colorscheme` | +| "Increase font size" | Edit `kdeglobals` → reload KWin | +| "Add a system tray for [app]" | Install + configure tray widget | +| "Switch to dark mode" | `plasma-apply-colorscheme Win11Dark` | +| "Add keyboard shortcut [key] for [action]" | Write `kglobalshortcutsrc` entry | + +### The Iteration Loop +``` +Speak → See change → "No, not like that" → Hermes adjusts → "Yes, perfect" +``` + +This is the same loop you and I already do over Telegram, but: +- **Instant** (no typing, no copy-paste) +- **Visual** (you see the result immediately on your screen) +- **Iterative** ("move it left", "bigger", "more transparent") +- **Contextual** (Hermes sees your current desktop state) + +## Key Design Decisions + +**Why Hermes API and not local command parsing?** +- Hermes understands natural language ("make the thing faster" → identifies what "the thing" is from context) +- Hermes can write code, manage servers, fix bugs — not just launch apps +- Hermes has memory of prior commands and project context +- Simple command parsing can't do "refactor the auth module to use JWT" + +**Why correction instead of just sending raw transcript?** +- SamiType will occasionally mishear technical terms (package names, URLs, code syntax) +- 2-second glance + quick fix prevents cascading errors +- Hermes is smart enough to handle most errors, but "install postgress" vs "install postgresql" matters +- You said you want this — the ability to see and fix in real-time + +**Why Tauri for the overlay?** +- ~10MB binary (vs Electron's ~150MB) +- Rust backend handles audio socket efficiently +- Web frontend (React/Svelte) for rapid UI iteration +- Cross-platform if SamiOS ever runs outside WSL