Add voice-to-Hermes pipeline architecture with live OS modification design
SamiOS CI / lint-and-test (push) Successful in 6s
SamiOS CI / lint-and-test (push) Successful in 6s
- Voice-hermes-pipeline.md: full architecture for SamiType → Hermes direct command - Live transcript overlay with inline correction - Live OS modification via voice (add widgets, change themes, move elements) - Phased rollout plan (POC → correction → context → full integration)
This commit is contained in:
Sami Ahmed
@@ -0,0 +1,256 @@
|
||||
# SamiType Voice → Hermes Direct Command Pipeline
|
||||
|
||||
## The Vision
|
||||
|
||||
Speak naturally → see your words transcribed live → correct if needed → Hermes executes.
|
||||
|
||||
This is NOT a simple voice command router ("open Firefox"). This is a **voice-driven coding/system administration interface** where natural language spoken commands go directly to Hermes (the AI agent) for execution — writing code, managing servers, configuring the OS, anything Hermes can do.
|
||||
|
||||
## The Flow
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Voice Command Pipeline │
|
||||
│ │
|
||||
│ 1. SPEAK 2. SEE 3. CORRECT 4. EXECUTE │
|
||||
│ │
|
||||
│ ┌──────┐ ┌────────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Mic │───▶│ Live │───▶│ Inline │───▶│ Hermes │ │
|
||||
│ │ │ │ Transcript │ │ Edit │ │ API │ │
|
||||
│ │ │ │ (streaming)│ │ (optional)│ │ │ │
|
||||
│ └──────┘ └────────────┘ └──────────┘ └────┬─────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────┐ │
|
||||
│ │ Result │ │
|
||||
│ │ (visual │ │
|
||||
│ │ + TTS) │ │
|
||||
│ └──────────┘ │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Step-by-step
|
||||
|
||||
1. **Speak** — Press hotkey or say wake word "Computer". Start talking naturally.
|
||||
2. **Live Transcript** — Words appear on screen as you speak them, streaming, like live captions. A floating overlay bar at the top or bottom of the screen.
|
||||
3. **Correct** — When you stop speaking, the transcript stays visible for 2 seconds. If SamiType misheard something:
|
||||
- Click/tap the word to fix it
|
||||
- Or press Tab to edit the full text
|
||||
- Or just say "scratch that" to restart
|
||||
- Or press Enter immediately to send as-is
|
||||
4. **Execute** — Corrected text goes to Hermes API as a task. Hermes starts working.
|
||||
5. **Result** — A notification shows "Hermes is working on: [task]". When done, result appears + optional TTS summary.
|
||||
|
||||
## Example Sessions
|
||||
|
||||
### Coding
|
||||
```
|
||||
You: "Computer, create a Python script that monitors disk usage
|
||||
and sends me a Telegram alert if it goes above 80 percent"
|
||||
|
||||
[Live transcript appears word by word]
|
||||
[2s pause — you see it's correct, press Enter]
|
||||
|
||||
Hermes: "Writing disk monitor script..."
|
||||
Hermes: "Done. Script at /usr/local/bin/disk-monitor.py,
|
||||
systemd service enabled. Want me to test it?"
|
||||
```
|
||||
|
||||
### System Admin
|
||||
```
|
||||
You: "Computer, check if the triangl... no, the DashCaddy server
|
||||
is responding"
|
||||
|
||||
[Transcript: "check if the triangl... no, the DashCaddy server is responding"]
|
||||
[You edit "triangl" → delete that part]
|
||||
|
||||
Corrected: "check if the DashCaddy server is responding"
|
||||
[Enter]
|
||||
|
||||
Hermes: "DashCaddy is up. Response time 42ms.
|
||||
Dashboard loads in 1.2s. All containers healthy."
|
||||
```
|
||||
|
||||
### OS Control
|
||||
```
|
||||
You: "Computer, install VLC and set it as default media player"
|
||||
|
||||
[Live transcript — correct, Enter]
|
||||
|
||||
Hermes: "Installing VLC... done. Set as default for video/x-matroska,
|
||||
video/mp4, and audio/mpeg."
|
||||
```
|
||||
|
||||
## Technical Architecture
|
||||
|
||||
### Component 1: SamiType Streaming STT
|
||||
- **Streaming mode**: Outputs partial transcriptions every ~200ms
|
||||
- **WebSocket or Unix socket** → UI overlay receives chunks
|
||||
- **Confidence scores** per word — low-confidence words highlighted in red
|
||||
- **VAD (Voice Activity Detection)**: Auto-detects when you stop talking → 2s correction window
|
||||
|
||||
### Component 2: Live Transcript Overlay UI
|
||||
Lightweight floating widget — options:
|
||||
|
||||
**Option A: KDE Plasma Widget (native)**
|
||||
- Always-on-top floating panel
|
||||
- QML-based, renders streaming text
|
||||
- Click-to-edit inline
|
||||
- Keyboard shortcuts: Enter=submit, Esc=cancel, Tab=edit mode
|
||||
- Semi-transparent when idle, solid when active
|
||||
|
||||
**Option B: Electron/Tauri app (cross-platform)**
|
||||
- Floating window, borderless, always-on-top
|
||||
- Web tech (React/Svelte) for rapid iteration
|
||||
- Works on any desktop, not just KDE
|
||||
|
||||
**Option C: Terminal overlay (tmux/terminal)**
|
||||
- Simplest — a tmux pane or terminal window
|
||||
- Good for development, not daily use
|
||||
|
||||
**Recommendation**: Start with B (Tauri — lightweight, ~10MB, Rust backend), graduate to A (native KDE widget) when SamiType is mature.
|
||||
|
||||
### Component 3: Correction Layer
|
||||
```
|
||||
Transcript: "create a python script that monitors [disk] usage"
|
||||
highlighted = low confidence ^
|
||||
|
||||
Click "disk" → text field appears → type "disc" → Enter
|
||||
Or: ignore, press Enter to send as-is (Hermes will understand context anyway)
|
||||
```
|
||||
|
||||
- **Low-confidence words** (< 80% certainty) highlighted in amber/red
|
||||
- **Click any word** to edit inline
|
||||
- **Tab** → enter full-edit mode (entire text becomes editable)
|
||||
- **Auto-submit timer**: 2s after speech ends, auto-sends unless editing
|
||||
- **"Scratch that"** → clears and restarts listening
|
||||
|
||||
### Component 4: Hermes API Bridge
|
||||
```python
|
||||
# Voice command → Hermes HTTP API
|
||||
import requests
|
||||
|
||||
def send_to_hermes(transcript: str):
|
||||
response = requests.post("http://localhost:8765/command", json={
|
||||
"text": transcript,
|
||||
"source": "voice",
|
||||
"context": {
|
||||
"hostname": "samios",
|
||||
"user": "sami",
|
||||
"cwd": os.getcwd(),
|
||||
"active_app": get_active_window()
|
||||
}
|
||||
})
|
||||
return response.json()
|
||||
|
||||
# Hermes processes it as a normal task — same as if Sami typed it
|
||||
```
|
||||
|
||||
### Component 5: Result Display
|
||||
- **Notification**: KDE notification "Hermes: [task summary]" with status
|
||||
- **TTS**: Optional spoken confirmation using SamiType TTS
|
||||
- **Full output**: Click notification → opens Konsole with full output log
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Proof of Concept (1-2 weeks)
|
||||
- [ ] SamiType running in streaming mode, outputting to stdout/JSON
|
||||
- [ ] Simple Python/Tauri overlay showing live transcript
|
||||
- [ ] Enter key → send to Hermes CLI (`hermes ask "transcript text"`)
|
||||
- [ ] Basic notification on completion
|
||||
- **Milestone**: Voice "create a file called test.txt" → Hermes creates it → notification
|
||||
|
||||
### Phase 2: Correction + Polish (2-3 weeks)
|
||||
- [ ] Low-confidence word highlighting
|
||||
- [ ] Inline word editing
|
||||
- [ ] 2s auto-submit timer with visual countdown
|
||||
- [ ] "Scratch that" voice command to restart
|
||||
- [ ] Result TTS via SamiType
|
||||
- **Milestone**: Natural correction flow, reliable daily use
|
||||
|
||||
### Phase 3: Context Awareness (3-4 weeks)
|
||||
- [ ] Active window/app context passed to Hermes
|
||||
- [ ] "Fix this function" when code editor is focused → Hermes sees the file
|
||||
- [ ] "What's wrong with this page" when browser is focused → Hermes checks URL
|
||||
- [ ] Multi-turn: "Now make it async" (Hermes remembers previous command)
|
||||
|
||||
### Phase 4: Full Integration (ongoing)
|
||||
- [ ] Native KDE Plasma widget
|
||||
- [ ] Systemd service (always available)
|
||||
- [ ] Wake word "Computer" (openWakeWord)
|
||||
- [ ] Custom command shortcuts ("Computer, deploy" = predefined multi-step task)
|
||||
- [ ] Voice history log + replay
|
||||
|
||||
## Live OS Modification (Killer Feature)
|
||||
|
||||
The most powerful use case: **designing and modifying the OS in real-time via voice.**
|
||||
|
||||
```
|
||||
You: "Computer, add a weather widget to the top right corner"
|
||||
[Enter]
|
||||
Hermes: "Adding weather widget..."
|
||||
→ KDE plasma widget installed, positioned, configured
|
||||
→ Widget appears on your screen 3 seconds later
|
||||
|
||||
You: "Move it down a bit"
|
||||
[Enter]
|
||||
→ Widget shifts down 50px
|
||||
|
||||
You: "Make it show 5 days instead of 3"
|
||||
[Enter]
|
||||
→ Widget reconfigures, shows 5-day forecast
|
||||
```
|
||||
|
||||
### How This Works
|
||||
|
||||
1. **Hermes receives voice command** with desktop context (active screen, resolution, current layout)
|
||||
2. **Hermes generates the config change** (Plasma widget config, kwin rule, CSS/SVG theme edit)
|
||||
3. **Hermes pushes the change** via:
|
||||
- `qdbus` calls to Plasma (move/resize/configure widgets)
|
||||
- File writes to `~/.config/` (theme changes, panel configs)
|
||||
- `kstart` / `kquitapp` to reload components
|
||||
4. **Change appears live** — no reboot, no logout, instant visual update
|
||||
|
||||
### Supported Live Modifications
|
||||
| Voice Command | Mechanism |
|
||||
|---------------|-----------|
|
||||
| "Add [widget] to [position]" | Install + `plasma-shell` script |
|
||||
| "Move [element] [direction]" | `qdbus org.kde.plasmashell` |
|
||||
| "Change wallpaper to [name]" | `plasma-apply-wallpaper` |
|
||||
| "Make panel transparent" | Edit panel config → `qdbus` reload |
|
||||
| "Change accent color to blue" | `plasma-apply-colorscheme` |
|
||||
| "Increase font size" | Edit `kdeglobals` → reload KWin |
|
||||
| "Add a system tray for [app]" | Install + configure tray widget |
|
||||
| "Switch to dark mode" | `plasma-apply-colorscheme Win11Dark` |
|
||||
| "Add keyboard shortcut [key] for [action]" | Write `kglobalshortcutsrc` entry |
|
||||
|
||||
### The Iteration Loop
|
||||
```
|
||||
Speak → See change → "No, not like that" → Hermes adjusts → "Yes, perfect"
|
||||
```
|
||||
|
||||
This is the same loop you and I already do over Telegram, but:
|
||||
- **Instant** (no typing, no copy-paste)
|
||||
- **Visual** (you see the result immediately on your screen)
|
||||
- **Iterative** ("move it left", "bigger", "more transparent")
|
||||
- **Contextual** (Hermes sees your current desktop state)
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
**Why Hermes API and not local command parsing?**
|
||||
- Hermes understands natural language ("make the thing faster" → identifies what "the thing" is from context)
|
||||
- Hermes can write code, manage servers, fix bugs — not just launch apps
|
||||
- Hermes has memory of prior commands and project context
|
||||
- Simple command parsing can't do "refactor the auth module to use JWT"
|
||||
|
||||
**Why correction instead of just sending raw transcript?**
|
||||
- SamiType will occasionally mishear technical terms (package names, URLs, code syntax)
|
||||
- 2-second glance + quick fix prevents cascading errors
|
||||
- Hermes is smart enough to handle most errors, but "install postgress" vs "install postgresql" matters
|
||||
- You said you want this — the ability to see and fix in real-time
|
||||
|
||||
**Why Tauri for the overlay?**
|
||||
- ~10MB binary (vs Electron's ~150MB)
|
||||
- Rust backend handles audio socket efficiently
|
||||
- Web frontend (React/Svelte) for rapid UI iteration
|
||||
- Cross-platform if SamiOS ever runs outside WSL
|
||||
Reference in New Issue
Block a user