Add dictation mode design (LibreOffice + any-app voice typing), developer guide
SamiOS CI / lint-and-test (push) Successful in 6s
SamiOS CI / lint-and-test (push) Successful in 6s
- Dictation mode: separate from command mode, types directly into focused app - Punctuation/formatting voice commands (new paragraph, comma, scratch that) - Floating correction bar with low-confidence highlighting - Technical implementation sketch (xdotool/ydotool integration) - Developer guide with project structure, build commands, font policy
This commit is contained in:
Sami Ahmed
@@ -0,0 +1,113 @@
|
||||
# SamiOS Developer Guide
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- Arch Linux (or WSL Arch) with `base-devel`, `git`, `archiso`
|
||||
- Gitea account on `git.sami`
|
||||
|
||||
### Clone
|
||||
```bash
|
||||
git clone https://git.sami/sami7777/samios.git
|
||||
cd samios
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
```
|
||||
samios/
|
||||
├── branding/ # Visual identity assets
|
||||
│ ├── icons/ # Square + launcher icons (PNG + SVG)
|
||||
│ ├── logos/ # Full lockups (face + pyramid)
|
||||
│ ├── wallpapers/ # Desktop wallpapers
|
||||
│ ├── themes/ # SDDM theme
|
||||
│ ├── grub-theme/ # GRUB boot theme
|
||||
│ └── README.md # Brand guidelines
|
||||
├── docs/ # Documentation
|
||||
│ ├── installation.md # User install guide
|
||||
│ ├── roadmap.md # Development roadmap
|
||||
│ ├── voice-integration-architecture.md
|
||||
│ └── voice-hermes-pipeline.md
|
||||
├── packaging/
|
||||
│ ├── archiso/ # ISO build profile
|
||||
│ │ ├── airootfs/ # Root filesystem overlay
|
||||
│ │ ├── packages.x86_64 # Package list
|
||||
│ │ ├── profiledef.sh # ISO profile
|
||||
│ │ └── build.sh # Build script
|
||||
│ ├── packages/ # Custom PKGBUILDs
|
||||
│ │ └── samios-branding/
|
||||
│ └── scripts/ # Setup and installer scripts
|
||||
│ ├── samios-installer.sh
|
||||
│ ├── samios-desktop-setup.sh
|
||||
│ └── kdeglobals-samios
|
||||
├── tests/ # Test suite
|
||||
│ ├── run_tests.sh
|
||||
│ ├── test_profiledef.sh
|
||||
│ ├── test_packages.sh
|
||||
│ └── ...
|
||||
├── Makefile # Build/test/lint targets
|
||||
└── .gitea/workflows/ # CI pipeline
|
||||
```
|
||||
|
||||
## Development Commands
|
||||
|
||||
```bash
|
||||
make test # Run test suite
|
||||
make lint # Shellcheck on all scripts
|
||||
make check # Full CI (lint + test)
|
||||
make build # Build ISO (requires archiso)
|
||||
make clean # Clean build artifacts
|
||||
```
|
||||
|
||||
## Building the ISO
|
||||
|
||||
```bash
|
||||
# Install archiso
|
||||
sudo pacman -S archiso
|
||||
|
||||
# Build
|
||||
cd packaging/archiso
|
||||
sudo ./build.sh
|
||||
|
||||
# Output
|
||||
ls out/samios-*.iso
|
||||
```
|
||||
|
||||
## Font Policy
|
||||
|
||||
- **Default UI font:** Sami Grotesk (Helvetica-like sans-serif)
|
||||
- **Secondary font:** Sami Sans
|
||||
- **Excluded:** Any fonts with "7777" in the filename (personal branding)
|
||||
- Font config: `packaging/archiso/airootfs/etc/fonts/conf.d/10-samios-fonts.conf`
|
||||
|
||||
## Adding Packages
|
||||
|
||||
Edit `packaging/archiso/packages.x86_64` and add the package name. Test with:
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
## Modifying Branding
|
||||
|
||||
1. Place new assets in `branding/` (follow existing structure)
|
||||
2. Update `branding/README.md` with new entries
|
||||
3. If changing icons, generate all sizes from the SVG source
|
||||
4. Test the ISO build
|
||||
|
||||
## Voice Integration
|
||||
|
||||
See `docs/voice-hermes-pipeline.md` for the SamiType voice command architecture.
|
||||
|
||||
## CI/CD
|
||||
|
||||
Gitea Actions workflow at `.gitea/workflows/ci.yml` runs on every push:
|
||||
- Shellcheck on all `.sh` files
|
||||
- Profile structure validation
|
||||
- Test suite execution
|
||||
|
||||
## Release Process
|
||||
|
||||
1. Update version in `profiledef.sh`, `samios` CLI, and `README.md`
|
||||
2. Update `docs/roadmap.md` with completed items
|
||||
3. Commit: `git commit -m "vX.Y.Z: Description"`
|
||||
4. Tag: `git tag vX.Y.Z`
|
||||
5. Push: `git push origin master --tags`
|
||||
@@ -235,6 +235,149 @@ This is the same loop you and I already do over Telegram, but:
|
||||
- **Iterative** ("move it left", "bigger", "more transparent")
|
||||
- **Contextual** (Hermes sees your current desktop state)
|
||||
|
||||
## Dictation Mode (LibreOffice Integration)
|
||||
|
||||
**The flow:** "Open LibreOffice" → wait for it to load → see a blank page → start dictating → words appear in the document live.
|
||||
|
||||
This is different from command mode — it's **free-form dictation** that goes directly into whatever text field has focus, not to Hermes.
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
You: "Computer, open LibreOffice Writer"
|
||||
[Enter → command mode → Hermes/WM launches LibreOffice]
|
||||
|
||||
You: "Computer, start dictation"
|
||||
[Enter → DICTATION MODE ACTIVATED]
|
||||
|
||||
You: "Dear team, I wanted to follow up on the quarterly report.
|
||||
The numbers are looking strong with revenue up fifteen percent
|
||||
year over year..."
|
||||
|
||||
[Words stream DIRECTLY into the LibreOffice document as you speak them]
|
||||
[Live transcript appears in a small floating bar for correction]
|
||||
[Low-confidence words highlighted — click to fix]
|
||||
|
||||
You: "new paragraph"
|
||||
[Paragraph break inserted in document]
|
||||
|
||||
You: "stop dictation"
|
||||
|
||||
You: "Computer, save the document"
|
||||
[Enter → command mode → file saved]
|
||||
```
|
||||
|
||||
### Two Distinct Modes
|
||||
|
||||
| Mode | Trigger | What Happens | Target |
|
||||
|------|---------|-------------|--------|
|
||||
| **Command Mode** | "Computer, [instruction]" | Goes to Hermes for execution | Hermes API |
|
||||
| **Dictation Mode** | "Computer, start dictation" | Words type into focused app | Active text field (LibreOffice, Kate, browser, any app) |
|
||||
| **End Dictation** | "stop dictation" or hotkey | Returns to idle | — |
|
||||
|
||||
### Dictation UI
|
||||
|
||||
A minimal floating bar at the bottom of the screen during dictation:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ 🔴 DICTATING │ Dear team, I wanted to follow [up] on... │
|
||||
│ │ ^low confidence │
|
||||
│ │ [Enter=send] [Esc=stop] [Tab=edit] │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- Last ~10 words shown in the bar (scrolling)
|
||||
- Low-confidence words highlighted in amber
|
||||
- Click a highlighted word → alternatives appear → pick or type correction
|
||||
- The corrected text is what actually gets typed into the document
|
||||
|
||||
### Dictation Commands (Voice)
|
||||
|
||||
| Spoken | Action |
|
||||
|--------|--------|
|
||||
| "new paragraph" | Enter key |
|
||||
| "new line" | Shift+Enter |
|
||||
| "comma" | , |
|
||||
| "period" | . |
|
||||
| "question mark" | ? |
|
||||
| "exclamation mark" | ! |
|
||||
| "colon" | : |
|
||||
| "semicolon" | ; |
|
||||
| "open quote" / "close quote" | " " |
|
||||
| "open paren" / "close paren" | ( ) |
|
||||
| "dash" | — |
|
||||
| "tab" | Tab key |
|
||||
| "capitalize [word]" | Capitalizes next word |
|
||||
| "all caps [word]" | ALL CAPS |
|
||||
| "scratch that" | Deletes last sentence |
|
||||
| "undo that" | Ctrl+Z |
|
||||
| "select [word/phrase]" | Highlights text |
|
||||
| "delete [word]" | Removes word |
|
||||
| "stop dictation" | Exits dictation mode |
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
# Dictation mode handler
|
||||
class DictationMode:
|
||||
def __init__(self):
|
||||
self.active = False
|
||||
self.sami_type = SamiTypeStreaming()
|
||||
self.correction_window = CorrectionOverlay()
|
||||
|
||||
def start(self):
|
||||
"""Enter dictation mode"""
|
||||
self.active = True
|
||||
self.correction_window.show()
|
||||
# SamiType streams partial transcriptions
|
||||
self.sami_type.on_partial(self._on_partial)
|
||||
self.sami_type.on_final(self._on_final)
|
||||
|
||||
def _on_final(self, text: str, confidence: float):
|
||||
"""Final transcription (after VAD detects end of utterance)"""
|
||||
# Check for dictation commands first
|
||||
cmd = self._parse_command(text)
|
||||
if cmd:
|
||||
self._execute_command(cmd)
|
||||
return
|
||||
|
||||
# Type into active window via xdotool/wl-copy
|
||||
self._type_text(text)
|
||||
|
||||
# Show in correction bar for 2s
|
||||
self.correction_window.show_text(text, confidence)
|
||||
|
||||
def _type_text(self, text: str):
|
||||
"""Type text into the focused application"""
|
||||
import subprocess
|
||||
# X11: xdotool type
|
||||
subprocess.run(["xdotool", "type", "--delay", "0", text + " "])
|
||||
# Wayland: wl-copy + wl-paste, or ydotool
|
||||
|
||||
def _parse_command(self, text: str):
|
||||
"""Check if spoken text is a dictation command"""
|
||||
commands = {
|
||||
"new paragraph": "KEY_Return",
|
||||
"new line": "KEY_Shift+Return",
|
||||
"comma": ",",
|
||||
"period": ".",
|
||||
"scratch that": "UNDO_SENTENCE",
|
||||
"stop dictation": "EXIT",
|
||||
}
|
||||
return commands.get(text.strip().lower())
|
||||
```
|
||||
|
||||
### Supported Applications
|
||||
|
||||
Dictation mode works with **any application** that accepts text input:
|
||||
- **LibreOffice Writer** — documents
|
||||
- **Kate** — code files
|
||||
- **Konsole** — terminal commands
|
||||
- **Firefox** — web forms, URL bar, search
|
||||
- **Dolphin** — file rename, search
|
||||
- **Any X11/Wayland app** — xdotool/ydotool types into focused field
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
**Why Hermes API and not local command parsing?**
|
||||
|
||||
Reference in New Issue
Block a user