samios/docs/voice-hermes-pipeline.md

# SamiType Voice → Hermes Direct Command Pipeline

## The Vision

Speak naturally → see your words transcribed live → correct if needed → Hermes executes.

This is NOT a simple voice command router ("open Firefox"). This is a **voice-driven coding/system administration interface** where natural language spoken commands go directly to Hermes (the AI agent) for execution — writing code, managing servers, configuring the OS, anything Hermes can do.

## The Flow

```
┌──────────────────────────────────────────────────────────────┐
│                    Voice Command Pipeline                      │
│                                                               │
│  1. SPEAK          2. SEE          3. CORRECT      4. EXECUTE │
│                                                               │
│  ┌──────┐    ┌────────────┐    ┌──────────┐    ┌──────────┐ │
│  │ Mic  │───▶│ Live       │───▶│ Inline   │───▶│ Hermes   │ │
│  │      │    │ Transcript │    │ Edit     │    │ API      │ │
│  │      │    │ (streaming)│    │ (optional)│   │          │ │
│  └──────┘    └────────────┘    └──────────┘    └────┬─────┘ │
│                                                    │        │
│                                                    ▼        │
│                                              ┌──────────┐  │
│                                              │  Result  │  │
│                                              │ (visual  │  │
│                                              │  + TTS)  │  │
│                                              └──────────┘  │
└──────────────────────────────────────────────────────────────┘
```

### Step-by-step

1. **Speak** — Press hotkey or say wake word "Computer". Start talking naturally.
2. **Live Transcript** — Words appear on screen as you speak them, streaming, like live captions. A floating overlay bar at the top or bottom of the screen.
3. **Correct** — When you stop speaking, the transcript stays visible for 2 seconds. If SamiType misheard something:
   - Click/tap the word to fix it
   - Or press Tab to edit the full text
   - Or just say "scratch that" to restart
   - Or press Enter immediately to send as-is
4. **Execute** — Corrected text goes to Hermes API as a task. Hermes starts working.
5. **Result** — A notification shows "Hermes is working on: [task]". When done, result appears + optional TTS summary.

## Example Sessions

### Coding
```
You: "Computer, create a Python script that monitors disk usage 
      and sends me a Telegram alert if it goes above 80 percent"

[Live transcript appears word by word]
[2s pause — you see it's correct, press Enter]

Hermes: "Writing disk monitor script..."
Hermes: "Done. Script at /usr/local/bin/disk-monitor.py, 
         systemd service enabled. Want me to test it?"
```

### System Admin
```
You: "Computer, check if the triangl... no, the DashCaddy server 
      is responding"

[Transcript: "check if the triangl... no, the DashCaddy server is responding"]
[You edit "triangl" → delete that part]

Corrected: "check if the DashCaddy server is responding"
[Enter]

Hermes: "DashCaddy is up. Response time 42ms. 
         Dashboard loads in 1.2s. All containers healthy."
```

### OS Control
```
You: "Computer, install VLC and set it as default media player"

[Live transcript — correct, Enter]

Hermes: "Installing VLC... done. Set as default for video/x-matroska, 
         video/mp4, and audio/mpeg."
```

## Technical Architecture

### Component 1: SamiType Streaming STT
- **Streaming mode**: Outputs partial transcriptions every ~200ms
- **WebSocket or Unix socket** → UI overlay receives chunks
- **Confidence scores** per word — low-confidence words highlighted in red
- **VAD (Voice Activity Detection)**: Auto-detects when you stop talking → 2s correction window

### Component 2: Live Transcript Overlay UI
Lightweight floating widget — options:

**Option A: KDE Plasma Widget (native)**
- Always-on-top floating panel
- QML-based, renders streaming text
- Click-to-edit inline
- Keyboard shortcuts: Enter=submit, Esc=cancel, Tab=edit mode
- Semi-transparent when idle, solid when active

**Option B: Electron/Tauri app (cross-platform)**
- Floating window, borderless, always-on-top
- Web tech (React/Svelte) for rapid iteration
- Works on any desktop, not just KDE

**Option C: Terminal overlay (tmux/terminal)**
- Simplest — a tmux pane or terminal window
- Good for development, not daily use

**Recommendation**: Start with B (Tauri — lightweight, ~10MB, Rust backend), graduate to A (native KDE widget) when SamiType is mature.

### Component 3: Correction Layer
```
Transcript: "create a python script that monitors [disk] usage"
                                   highlighted = low confidence ^
                                   
Click "disk" → text field appears → type "disc" → Enter
Or: ignore, press Enter to send as-is (Hermes will understand context anyway)
```

- **Low-confidence words** (< 80% certainty) highlighted in amber/red
- **Click any word** to edit inline
- **Tab** → enter full-edit mode (entire text becomes editable)
- **Auto-submit timer**: 2s after speech ends, auto-sends unless editing
- **"Scratch that"** → clears and restarts listening

### Component 4: Hermes API Bridge
```python
# Voice command → Hermes HTTP API
import requests

def send_to_hermes(transcript: str):
    response = requests.post("http://localhost:8765/command", json={
        "text": transcript,
        "source": "voice",
        "context": {
            "hostname": "samios",
            "user": "sami",
            "cwd": os.getcwd(),
            "active_app": get_active_window()
        }
    })
    return response.json()

# Hermes processes it as a normal task — same as if Sami typed it
```

### Component 5: Result Display
- **Notification**: KDE notification "Hermes: [task summary]" with status
- **TTS**: Optional spoken confirmation using SamiType TTS
- **Full output**: Click notification → opens Konsole with full output log

## Implementation Phases

### Phase 1: Proof of Concept (1-2 weeks)
- [ ] SamiType running in streaming mode, outputting to stdout/JSON
- [ ] Simple Python/Tauri overlay showing live transcript
- [ ] Enter key → send to Hermes CLI (`hermes ask "transcript text"`)
- [ ] Basic notification on completion
- **Milestone**: Voice "create a file called test.txt" → Hermes creates it → notification

### Phase 2: Correction + Polish (2-3 weeks)
- [ ] Low-confidence word highlighting
- [ ] Inline word editing
- [ ] 2s auto-submit timer with visual countdown
- [ ] "Scratch that" voice command to restart
- [ ] Result TTS via SamiType
- **Milestone**: Natural correction flow, reliable daily use

### Phase 3: Context Awareness (3-4 weeks)
- [ ] Active window/app context passed to Hermes
- [ ] "Fix this function" when code editor is focused → Hermes sees the file
- [ ] "What's wrong with this page" when browser is focused → Hermes checks URL
- [ ] Multi-turn: "Now make it async" (Hermes remembers previous command)

### Phase 4: Full Integration (ongoing)
- [ ] Native KDE Plasma widget
- [ ] Systemd service (always available)
- [ ] Wake word "Computer" (openWakeWord)
- [ ] Custom command shortcuts ("Computer, deploy" = predefined multi-step task)
- [ ] Voice history log + replay

## Live OS Modification (Killer Feature)

The most powerful use case: **designing and modifying the OS in real-time via voice.**

```
You: "Computer, add a weather widget to the top right corner"
[Enter]
Hermes: "Adding weather widget..."
→ KDE plasma widget installed, positioned, configured
→ Widget appears on your screen 3 seconds later

You: "Move it down a bit"
[Enter]
→ Widget shifts down 50px

You: "Make it show 5 days instead of 3"
[Enter]
→ Widget reconfigures, shows 5-day forecast
```

### How This Works

1. **Hermes receives voice command** with desktop context (active screen, resolution, current layout)
2. **Hermes generates the config change** (Plasma widget config, kwin rule, CSS/SVG theme edit)
3. **Hermes pushes the change** via:
   - `qdbus` calls to Plasma (move/resize/configure widgets)
   - File writes to `~/.config/` (theme changes, panel configs)
   - `kstart` / `kquitapp` to reload components
4. **Change appears live** — no reboot, no logout, instant visual update

### Supported Live Modifications
| Voice Command | Mechanism |
|---------------|-----------|
| "Add [widget] to [position]" | Install + `plasma-shell` script |
| "Move [element] [direction]" | `qdbus org.kde.plasmashell` |
| "Change wallpaper to [name]" | `plasma-apply-wallpaper` |
| "Make panel transparent" | Edit panel config → `qdbus` reload |
| "Change accent color to blue" | `plasma-apply-colorscheme` |
| "Increase font size" | Edit `kdeglobals` → reload KWin |
| "Add a system tray for [app]" | Install + configure tray widget |
| "Switch to dark mode" | `plasma-apply-colorscheme Win11Dark` |
| "Add keyboard shortcut [key] for [action]" | Write `kglobalshortcutsrc` entry |

### The Iteration Loop
```
Speak → See change → "No, not like that" → Hermes adjusts → "Yes, perfect"
```

This is the same loop you and I already do over Telegram, but:
- **Instant** (no typing, no copy-paste)
- **Visual** (you see the result immediately on your screen)
- **Iterative** ("move it left", "bigger", "more transparent")
- **Contextual** (Hermes sees your current desktop state)

## Dictation Mode (LibreOffice Integration)

**The flow:** "Open LibreOffice" → wait for it to load → see a blank page → start dictating → words appear in the document live.

This is different from command mode — it's **free-form dictation** that goes directly into whatever text field has focus, not to Hermes.

### How It Works

```
You: "Computer, open LibreOffice Writer"
[Enter → command mode → Hermes/WM launches LibreOffice]

You: "Computer, start dictation"
[Enter → DICTATION MODE ACTIVATED]

You: "Dear team, I wanted to follow up on the quarterly report.
      The numbers are looking strong with revenue up fifteen percent
      year over year..."

[Words stream DIRECTLY into the LibreOffice document as you speak them]
[Live transcript appears in a small floating bar for correction]
[Low-confidence words highlighted — click to fix]

You: "new paragraph"
[Paragraph break inserted in document]

You: "stop dictation"

You: "Computer, save the document"
[Enter → command mode → file saved]
```

### Two Distinct Modes

| Mode | Trigger | What Happens | Target |
|------|---------|-------------|--------|
| **Command Mode** | "Computer, [instruction]" | Goes to Hermes for execution | Hermes API |
| **Dictation Mode** | "Computer, start dictation" | Words type into focused app | Active text field (LibreOffice, Kate, browser, any app) |
| **End Dictation** | "stop dictation" or hotkey | Returns to idle | — |

### Dictation UI

A minimal floating bar at the bottom of the screen during dictation:

```
┌──────────────────────────────────────────────────────────┐
│ 🔴 DICTATING │ Dear team, I wanted to follow [up] on... │
│              │                            ^low confidence │
│              │ [Enter=send] [Esc=stop] [Tab=edit]        │
└──────────────────────────────────────────────────────────┘
```

- Last ~10 words shown in the bar (scrolling)
- Low-confidence words highlighted in amber
- Click a highlighted word → alternatives appear → pick or type correction
- The corrected text is what actually gets typed into the document

### Dictation Commands (Voice)

| Spoken | Action |
|--------|--------|
| "new paragraph" | Enter key |
| "new line" | Shift+Enter |
| "comma" | , |
| "period" | . |
| "question mark" | ? |
| "exclamation mark" | ! |
| "colon" | : |
| "semicolon" | ; |
| "open quote" / "close quote" | " " |
| "open paren" / "close paren" | ( ) |
| "dash" | — |
| "tab" | Tab key |
| "capitalize [word]" | Capitalizes next word |
| "all caps [word]" | ALL CAPS |
| "scratch that" | Deletes last sentence |
| "undo that" | Ctrl+Z |
| "select [word/phrase]" | Highlights text |
| "delete [word]" | Removes word |
| "stop dictation" | Exits dictation mode |

### Technical Implementation

```python
# Dictation mode handler
class DictationMode:
    def __init__(self):
        self.active = False
        self.sami_type = SamiTypeStreaming()
        self.correction_window = CorrectionOverlay()
        
    def start(self):
        """Enter dictation mode"""
        self.active = True
        self.correction_window.show()
        # SamiType streams partial transcriptions
        self.sami_type.on_partial(self._on_partial)
        self.sami_type.on_final(self._on_final)
        
    def _on_final(self, text: str, confidence: float):
        """Final transcription (after VAD detects end of utterance)"""
        # Check for dictation commands first
        cmd = self._parse_command(text)
        if cmd:
            self._execute_command(cmd)
            return
        
        # Type into active window via xdotool/wl-copy
        self._type_text(text)
        
        # Show in correction bar for 2s
        self.correction_window.show_text(text, confidence)
        
    def _type_text(self, text: str):
        """Type text into the focused application"""
        import subprocess
        # X11: xdotool type
        subprocess.run(["xdotool", "type", "--delay", "0", text + " "])
        # Wayland: wl-copy + wl-paste, or ydotool
        
    def _parse_command(self, text: str):
        """Check if spoken text is a dictation command"""
        commands = {
            "new paragraph": "KEY_Return",
            "new line": "KEY_Shift+Return", 
            "comma": ",",
            "period": ".",
            "scratch that": "UNDO_SENTENCE",
            "stop dictation": "EXIT",
        }
        return commands.get(text.strip().lower())
```

### Supported Applications

Dictation mode works with **any application** that accepts text input:
- **LibreOffice Writer** — documents
- **Kate** — code files
- **Konsole** — terminal commands
- **Firefox** — web forms, URL bar, search
- **Dolphin** — file rename, search
- **Any X11/Wayland app** — xdotool/ydotool types into focused field

## Key Design Decisions

**Why Hermes API and not local command parsing?**
- Hermes understands natural language ("make the thing faster" → identifies what "the thing" is from context)
- Hermes can write code, manage servers, fix bugs — not just launch apps
- Hermes has memory of prior commands and project context
- Simple command parsing can't do "refactor the auth module to use JWT"

**Why correction instead of just sending raw transcript?**
- SamiType will occasionally mishear technical terms (package names, URLs, code syntax)
- 2-second glance + quick fix prevents cascading errors
- Hermes is smart enough to handle most errors, but "install postgress" vs "install postgresql" matters
- You said you want this — the ability to see and fix in real-time

**Why Tauri for the overlay?**
- ~10MB binary (vs Electron's ~150MB)
- Rust backend handles audio socket efficiently
- Web frontend (React/Svelte) for rapid UI iteration
- Cross-platform if SamiOS ever runs outside WSL