samios/docs/voice-hermes-pipeline.md

SamiType Voice → Hermes Direct Command Pipeline

The Vision

Speak naturally → see your words transcribed live → correct if needed → Hermes executes.

This is NOT a simple voice command router ("open Firefox"). This is a voice-driven coding/system administration interface where natural language spoken commands go directly to Hermes (the AI agent) for execution — writing code, managing servers, configuring the OS, anything Hermes can do.

The Flow

┌──────────────────────────────────────────────────────────────┐
│                    Voice Command Pipeline                      │
│                                                               │
│  1. SPEAK          2. SEE          3. CORRECT      4. EXECUTE │
│                                                               │
│  ┌──────┐    ┌────────────┐    ┌──────────┐    ┌──────────┐ │
│  │ Mic  │───▶│ Live       │───▶│ Inline   │───▶│ Hermes   │ │
│  │      │    │ Transcript │    │ Edit     │    │ API      │ │
│  │      │    │ (streaming)│    │ (optional)│   │          │ │
│  └──────┘    └────────────┘    └──────────┘    └────┬─────┘ │
│                                                    │        │
│                                                    ▼        │
│                                              ┌──────────┐  │
│                                              │  Result  │  │
│                                              │ (visual  │  │
│                                              │  + TTS)  │  │
│                                              └──────────┘  │
└──────────────────────────────────────────────────────────────┘

Step-by-step

1. Speak — Press hotkey or say wake word "Computer". Start talking naturally.
2. Live Transcript — Words appear on screen as you speak them, streaming, like live captions. A floating overlay bar at the top or bottom of the screen.
3. Correct — When you stop speaking, the transcript stays visible for 2 seconds. If SamiType misheard something:
   • Click/tap the word to fix it
   • Or press Tab to edit the full text
   • Or just say "scratch that" to restart
   • Or press Enter immediately to send as-is
4. Execute — Corrected text goes to Hermes API as a task. Hermes starts working.
5. Result — A notification shows "Hermes is working on: [task]". When done, result appears + optional TTS summary.

Example Sessions

Coding

You: "Computer, create a Python script that monitors disk usage 
      and sends me a Telegram alert if it goes above 80 percent"

[Live transcript appears word by word]
[2s pause — you see it's correct, press Enter]

Hermes: "Writing disk monitor script..."
Hermes: "Done. Script at /usr/local/bin/disk-monitor.py, 
         systemd service enabled. Want me to test it?"

System Admin

You: "Computer, check if the triangl... no, the DashCaddy server 
      is responding"

[Transcript: "check if the triangl... no, the DashCaddy server is responding"]
[You edit "triangl" → delete that part]

Corrected: "check if the DashCaddy server is responding"
[Enter]

Hermes: "DashCaddy is up. Response time 42ms. 
         Dashboard loads in 1.2s. All containers healthy."

OS Control

You: "Computer, install VLC and set it as default media player"

[Live transcript — correct, Enter]

Hermes: "Installing VLC... done. Set as default for video/x-matroska, 
         video/mp4, and audio/mpeg."

Technical Architecture

Component 1: SamiType Streaming STT

• Streaming mode: Outputs partial transcriptions every ~200ms
• WebSocket or Unix socket → UI overlay receives chunks
• Confidence scores per word — low-confidence words highlighted in red
• VAD (Voice Activity Detection): Auto-detects when you stop talking → 2s correction window

Component 2: Live Transcript Overlay UI

Lightweight floating widget — options:

Option A: KDE Plasma Widget (native)

• Always-on-top floating panel
• QML-based, renders streaming text
• Click-to-edit inline
• Keyboard shortcuts: Enter=submit, Esc=cancel, Tab=edit mode
• Semi-transparent when idle, solid when active

Option B: Electron/Tauri app (cross-platform)

• Floating window, borderless, always-on-top
• Web tech (React/Svelte) for rapid iteration
• Works on any desktop, not just KDE

Option C: Terminal overlay (tmux/terminal)

• Simplest — a tmux pane or terminal window
• Good for development, not daily use

Recommendation: Start with B (Tauri — lightweight, ~10MB, Rust backend), graduate to A (native KDE widget) when SamiType is mature.

Component 3: Correction Layer

Transcript: "create a python script that monitors [disk] usage"
                                   highlighted = low confidence ^
                                   
Click "disk" → text field appears → type "disc" → Enter
Or: ignore, press Enter to send as-is (Hermes will understand context anyway)

• Low-confidence words (< 80% certainty) highlighted in amber/red
• Click any word to edit inline
• Tab → enter full-edit mode (entire text becomes editable)
• Auto-submit timer: 2s after speech ends, auto-sends unless editing
• "Scratch that" → clears and restarts listening

Component 4: Hermes API Bridge

# Voice command → Hermes HTTP API
import requests

def send_to_hermes(transcript: str):
    response = requests.post("http://localhost:8765/command", json={
        "text": transcript,
        "source": "voice",
        "context": {
            "hostname": "samios",
            "user": "sami",
            "cwd": os.getcwd(),
            "active_app": get_active_window()
        }
    })
    return response.json()

# Hermes processes it as a normal task — same as if Sami typed it

Component 5: Result Display

• Notification: KDE notification "Hermes: [task summary]" with status
• TTS: Optional spoken confirmation using SamiType TTS
• Full output: Click notification → opens Konsole with full output log

Implementation Phases

Phase 1: Proof of Concept (1-2 weeks)

• SamiType running in streaming mode, outputting to stdout/JSON
• Simple Python/Tauri overlay showing live transcript
• Enter key → send to Hermes CLI (hermes ask "transcript text")
• Basic notification on completion
• Milestone: Voice "create a file called test.txt" → Hermes creates it → notification

Phase 2: Correction + Polish (2-3 weeks)

• Low-confidence word highlighting
• Inline word editing
• 2s auto-submit timer with visual countdown
• "Scratch that" voice command to restart
• Result TTS via SamiType
• Milestone: Natural correction flow, reliable daily use

Phase 3: Context Awareness (3-4 weeks)

• Active window/app context passed to Hermes
• "Fix this function" when code editor is focused → Hermes sees the file
• "What's wrong with this page" when browser is focused → Hermes checks URL
• Multi-turn: "Now make it async" (Hermes remembers previous command)

Phase 4: Full Integration (ongoing)

• Native KDE Plasma widget
• Systemd service (always available)
• Wake word "Computer" (openWakeWord)
• Custom command shortcuts ("Computer, deploy" = predefined multi-step task)
• Voice history log + replay

Live OS Modification (Killer Feature)

The most powerful use case: designing and modifying the OS in real-time via voice.

You: "Computer, add a weather widget to the top right corner"
[Enter]
Hermes: "Adding weather widget..."
→ KDE plasma widget installed, positioned, configured
→ Widget appears on your screen 3 seconds later

You: "Move it down a bit"
[Enter]
→ Widget shifts down 50px

You: "Make it show 5 days instead of 3"
[Enter]
→ Widget reconfigures, shows 5-day forecast

How This Works

1. Hermes receives voice command with desktop context (active screen, resolution, current layout)
2. Hermes generates the config change (Plasma widget config, kwin rule, CSS/SVG theme edit)
3. Hermes pushes the change via:
   • qdbus calls to Plasma (move/resize/configure widgets)
   • File writes to ~/.config/ (theme changes, panel configs)
   • kstart / kquitapp to reload components
4. Change appears live — no reboot, no logout, instant visual update

Supported Live Modifications

Voice Command	Mechanism
"Add [widget] to [position]"	Install + plasma-shell script
"Move [element] [direction]"	qdbus org.kde.plasmashell
"Change wallpaper to [name]"	plasma-apply-wallpaper
"Make panel transparent"	Edit panel config → qdbus reload
"Change accent color to blue"	plasma-apply-colorscheme
"Increase font size"	Edit kdeglobals → reload KWin
"Add a system tray for [app]"	Install + configure tray widget
"Switch to dark mode"	plasma-apply-colorscheme Win11Dark
"Add keyboard shortcut [key] for [action]"	Write kglobalshortcutsrc entry

The Iteration Loop

Speak → See change → "No, not like that" → Hermes adjusts → "Yes, perfect"

This is the same loop you and I already do over Telegram, but:

• Instant (no typing, no copy-paste)
• Visual (you see the result immediately on your screen)
• Iterative ("move it left", "bigger", "more transparent")
• Contextual (Hermes sees your current desktop state)

Dictation Mode (LibreOffice Integration)

The flow: "Open LibreOffice" → wait for it to load → see a blank page → start dictating → words appear in the document live.

This is different from command mode — it's free-form dictation that goes directly into whatever text field has focus, not to Hermes.

How It Works

You: "Computer, open LibreOffice Writer"
[Enter → command mode → Hermes/WM launches LibreOffice]

You: "Computer, start dictation"
[Enter → DICTATION MODE ACTIVATED]

You: "Dear team, I wanted to follow up on the quarterly report.
      The numbers are looking strong with revenue up fifteen percent
      year over year..."

[Words stream DIRECTLY into the LibreOffice document as you speak them]
[Live transcript appears in a small floating bar for correction]
[Low-confidence words highlighted — click to fix]

You: "new paragraph"
[Paragraph break inserted in document]

You: "stop dictation"

You: "Computer, save the document"
[Enter → command mode → file saved]

Two Distinct Modes

Mode	Trigger	What Happens	Target
Command Mode	"Computer, [instruction]"	Goes to Hermes for execution	Hermes API
Dictation Mode	"Computer, start dictation"	Words type into focused app	Active text field (LibreOffice, Kate, browser, any app)
End Dictation	"stop dictation" or hotkey	Returns to idle	—

Dictation UI

A minimal floating bar at the bottom of the screen during dictation:

┌──────────────────────────────────────────────────────────┐
│ 🔴 DICTATING │ Dear team, I wanted to follow [up] on... │
│              │                            ^low confidence │
│              │ [Enter=send] [Esc=stop] [Tab=edit]        │
└──────────────────────────────────────────────────────────┘

• Last ~10 words shown in the bar (scrolling)
• Low-confidence words highlighted in amber
• Click a highlighted word → alternatives appear → pick or type correction
• The corrected text is what actually gets typed into the document

Dictation Commands (Voice)

Spoken	Action
"new paragraph"	Enter key
"new line"	Shift+Enter
"comma"	,
"period"	.
"question mark"	?
"exclamation mark"	!
"colon"	:
"semicolon"	;
"open quote" / "close quote"	" "
"open paren" / "close paren"	( )
"dash"	—
"tab"	Tab key
"capitalize [word]"	Capitalizes next word
"all caps [word]"	ALL CAPS
"scratch that"	Deletes last sentence
"undo that"	Ctrl+Z
"select [word/phrase]"	Highlights text
"delete [word]"	Removes word
"stop dictation"	Exits dictation mode

Technical Implementation

# Dictation mode handler
class DictationMode:
    def __init__(self):
        self.active = False
        self.sami_type = SamiTypeStreaming()
        self.correction_window = CorrectionOverlay()
        
    def start(self):
        """Enter dictation mode"""
        self.active = True
        self.correction_window.show()
        # SamiType streams partial transcriptions
        self.sami_type.on_partial(self._on_partial)
        self.sami_type.on_final(self._on_final)
        
    def _on_final(self, text: str, confidence: float):
        """Final transcription (after VAD detects end of utterance)"""
        # Check for dictation commands first
        cmd = self._parse_command(text)
        if cmd:
            self._execute_command(cmd)
            return
        
        # Type into active window via xdotool/wl-copy
        self._type_text(text)
        
        # Show in correction bar for 2s
        self.correction_window.show_text(text, confidence)
        
    def _type_text(self, text: str):
        """Type text into the focused application"""
        import subprocess
        # X11: xdotool type
        subprocess.run(["xdotool", "type", "--delay", "0", text + " "])
        # Wayland: wl-copy + wl-paste, or ydotool
        
    def _parse_command(self, text: str):
        """Check if spoken text is a dictation command"""
        commands = {
            "new paragraph": "KEY_Return",
            "new line": "KEY_Shift+Return", 
            "comma": ",",
            "period": ".",
            "scratch that": "UNDO_SENTENCE",
            "stop dictation": "EXIT",
        }
        return commands.get(text.strip().lower())

Supported Applications

Dictation mode works with any application that accepts text input:

• LibreOffice Writer — documents
• Kate — code files
• Konsole — terminal commands
• Firefox — web forms, URL bar, search
• Dolphin — file rename, search
• Any X11/Wayland app — xdotool/ydotool types into focused field

Key Design Decisions

Why Hermes API and not local command parsing?

• Hermes understands natural language ("make the thing faster" → identifies what "the thing" is from context)
• Hermes can write code, manage servers, fix bugs — not just launch apps
• Hermes has memory of prior commands and project context
• Simple command parsing can't do "refactor the auth module to use JWT"

Why correction instead of just sending raw transcript?

• SamiType will occasionally mishear technical terms (package names, URLs, code syntax)
• 2-second glance + quick fix prevents cascading errors
• Hermes is smart enough to handle most errors, but "install postgress" vs "install postgresql" matters
• You said you want this — the ability to see and fix in real-time

Why Tauri for the overlay?

• ~10MB binary (vs Electron's ~150MB)
• Rust backend handles audio socket efficiently
• Web frontend (React/Svelte) for rapid UI iteration
• Cross-platform if SamiOS ever runs outside WSL