samios/docs/voice-integration-architecture.md

SamiType × SamiOS Voice Integration Architecture

Vision

Voice-first OS control — "Computer, open Firefox", "Computer, go to youtube.com", "Computer, volume up" — Star Trek computer style, powered by SamiType (Sami's custom speech-to-text engine).

Architecture Overview

┌─────────────────────────────────────────────────┐
│                  SamiOS Desktop                  │
│                                                  │
│  ┌──────────┐   ┌──────────┐   ┌─────────────┐ │
│  │  Audio   │──▶│ SamiType │──▶│  Command    │ │
│  │  Capture │   │   STT    │   │  Router     │ │
│  └──────────┘   └──────────┘   └──────┬──────┘ │
│                                       │         │
│                    ┌──────────────────┼─────┐   │
│                    ▼                  ▼     ▼   │
│              ┌──────────┐  ┌──────┐ ┌──────┐  │
│              │  KDE/WM  │  │ Web  │ │ Sys  │  │
│              │  Control │  │ Nav  │ │ Ctrl │  │
│              └──────────┘  └──────┘ └──────┘  │
│                                                  │
│  ┌──────────────────────────────────────────┐   │
│  │        SamiType TTS (Voice Response)     │   │
│  └──────────────────────────────────────────┘   │
└─────────────────────────────────────────────────┘

Components

1. Wake Word Engine

• Trigger: "Computer" (configurable)
• Implementation: Porcupine or openWakeWord (offline, low-latency)
• Always listening via systemd service, minimal CPU (< 2%)
• Push-to-talk fallback (keyboard shortcut or dedicated button)

2. SamiType STT (Speech-to-Text)

• Sami's existing speech-to-text engine
• Offline-first — models run locally, no cloud dependency
• Streaming mode — starts transcribing before user finishes speaking
• Outputs text + confidence score

3. Command Router

• Parses SamiType transcription → structured command
• Intent classification:
  • Navigation: "go to [website]", "open [app]", "search for [query]"
  • System: "volume [up/down/mute]", "brightness [N]", "screenshot", "lock screen"
  • Window: "minimize", "maximize", "close window", "switch to [app]"
  • File: "open [file]", "create folder [name]", "move [file] to [location]"
  • Query: "what time is it", "what's the weather", "system status"
• Fallback: If no command match → pass to web search or LLM

4. Execution Layer

• KDE/WM Control: kdialog, qdbus, kstart, wmctrl, xdotool
• Web Navigation: xdg-open, firefox --new-tab [url]
• System Control: pactl (volume), brightnessctl, systemctl, loginctl

5. Voice Response (TTS)

• SamiType TTS engine for spoken confirmations
• "Opening Firefox", "Volume set to 50%", "Screenshot saved"
• Subtle — not chatty, only speaks when action confirmed or error

Command Examples

Voice Input	Action
"Computer, open Firefox"	Launch Firefox
"Computer, go to youtube.com"	Open youtube.com in default browser
"Computer, volume up"	Increase volume by 10%
"Computer, mute"	Mute audio
"Computer, take a screenshot"	spectacle -f (fullscreen capture)
"Computer, lock screen"	loginctl lock-session
"Computer, what time is it"	TTS: "It's 3:42 PM"
"Computer, minimize all windows"	Show desktop (Meta+D)
"Computer, switch to Firefox"	Activate Firefox window
"Computer, check for updates"	Run pacman -Syu, TTS result

Integration Points

SamiType → SamiOS

samiType.onTranscription(text, confidence) {
    if (confidence < 0.6) return;  // ignore low-confidence
    
    command = CommandRouter.parse(text);
    if (command) {
        result = command.execute();
        if (result.shouldSpeak) {
            samiTypeTTS.speak(result.message);
        }
    }
}

systemd Service

[Unit]
Description=SamiOS Voice Assistant (SamiType)
After=pipewire.service

[Service]
ExecStart=/usr/local/bin/samios-voice
Restart=always
User=sami

[Install]
WantedBy=default.target

Phased Rollout

Phase 1: Foundation (MVP)

• SamiType running as systemd service in SamiOS
• Push-to-talk activation (keyboard shortcut)
• Basic command set: open apps, web nav, volume
• Text output in notification (no TTS yet)

Phase 2: Always-On Wake Word

• Wake word "Computer" detection
• Expanded command set: window management, screenshots
• TTS voice feedback

Phase 3: Natural Language

• LLM-powered intent parsing for complex commands
• "Computer, find all PDFs from last week and move them to Documents"
• Context awareness (current app, current selection)

Phase 4: Full Voice OS

• Voice-controlled settings (display, network, bluetooth)
• Dictation mode (SamiType → any text field)
• Custom command creation ("Computer, when I say X, do Y")
• Multi-turn voice interactions

Dependencies

• SamiType (STT + TTS)
• pipewire (audio capture/playback)
• KDE Plasma (window/desktop control via qdbus)
• wmctrl / xdotool (X11 window manipulation)
• Optional: openWakeWord or Porcupine (wake word detection)

Security Considerations

• All processing is local (no cloud, no data leaves the machine)
• Microphone indicator (notification when listening)
• Manual disable switch (system tray toggle)
• No voice data stored unless explicitly saved by user