From 08b63db324b9ca6503fec3f35109848b60fde34e Mon Sep 17 00:00:00 2001 From: Carlo1911 Date: Tue, 4 Aug 2026 16:11:42 -0500 Subject: [PATCH] docs: update README and SKILL documentation with minimum configuration details and usage examples --- README.md | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ SKILL.md | 38 ++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) diff --git a/README.md b/README.md index 1556ef1..a44aec5 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,32 @@ Reads past sessions, analyzes them against loaded skills, generates structured p **No daemons. No external services. No GPU.** Just your host agent, a scheduled job, and Python stdlib. +## Minimum Configuration + +Everything below has a sane default, but two things are **not optional** if you want the +pipeline to actually work end to end rather than silently doing nothing or getting stuck: + +| Variable | Required when | Why | +|---|---|---| +| `SKILL_EVOLUTION_HOST` | You're on anything other than Hermes | Defaults to `hermes`. There is no auto-detection — running on Claude Code without setting this to `claude_code` reads (and would try to write) the wrong host's session DB and skills tree. | +| A provider credential (`ANTHROPIC_API_KEY` by default, or the key matching `SKILL_EVOLUTION_PROVIDER`) | Always, unless you only ever run `deterministic`-only evaluation | The evaluation gate's `llm_judge` (part of the default evaluator set) fails **closed** without a working provider — every proposal will fail the gate forever, even after you approve it, with no error beyond "gate failed" in the result. | + +Everything else is optional and only needs to be set to change a default: + +| Variable | Default | Purpose | +|---|---|---| +| `SKILL_EVOLUTION_REPO` | current working directory | Where a cron job finds `scripts/` | +| `SKILL_EVOLUTION_PROPOSALS_DIR` | `./proposals/` | Where proposal files are read/written | +| `SKILL_EVOLUTION_AUTO_APPLY` / `SKILL_EVOLUTION_MIN_CONFIDENCE` | off / `0.85` | Read by *your* apply step, not by the scripts themselves — see "Auto-Apply" below | +| `SKILL_EVOLUTION_EVALUATORS` | `deterministic,llm_judge,regression` | Which evaluators the gate runs | +| `SKILL_EVOLUTION_GATE_STRICTNESS` | `strict` | `strict` (all must pass) or `majority` | +| `SKILL_EVOLUTION_PROVIDER` | `claude` | `claude` / `ollama` / `opencode` / `openai` / `gemini` | +| `SKILL_EVOLUTION_CLAUDE_CODE_HOME` | `~/.claude` | Only read when `SKILL_EVOLUTION_HOST=claude_code` | +| `SKILL_EVOLUTION_HISTORY_PATH` | `./eval_history.jsonl` | Evaluation history file | + +See `CLAUDE.md` and `SKILL.md` for the full list — this table is the minimum you need to +read before your first run, not an exhaustive reference. + ## Quick Start **Hermes:** @@ -40,6 +66,60 @@ export SKILL_EVOLUTION_AUTO_APPLY=true export SKILL_EVOLUTION_MIN_CONFIDENCE=0.85 ``` +## Usage Example + +A full manual walkthrough on Claude Code, from "do I have anything new to analyze" to +"a proposal is actually applied." This mirrors what a scheduled job automates, one step +at a time. + +```bash +# 0. Configure the host and a provider for the evaluation gate (see "Minimum Configuration") +export SKILL_EVOLUTION_HOST=claude_code +export ANTHROPIC_API_KEY=sk-ant-... + +# 1. Check how many unprocessed sessions exist, without marking anything as processed +python3 scripts/fetch_sessions.py --dry-run --lookback-hours 72 +# → "14 unprocessed sessions found" + +# 2. In a Claude Code session with the skill loaded, ask: +# "Run skill evolution analysis on my recent sessions" +# +# Under the hood, that prompt runs the same pipeline this repo ships as scripts — +# fetch_sessions.py's output formatted by analyze.py, plus skill_index.py's skill +# index — and feeds both to the host agent's own reasoning (the LLM analysis step +# is not a script in this repo; the agent performs it directly): +python3 scripts/fetch_sessions.py --lookback-hours 72 | python3 scripts/analyze.py > /tmp/sessions.txt +python3 scripts/skill_index.py > /tmp/skills.json +# The agent then writes one proposal file per finding into ./proposals/, and marks +# those sessions processed. + +# 3. Inspect what got proposed +python3 scripts/proposal.py --list +python3 scripts/proposal.py --show 20260804-001 + +# 4. Review the proposal file and, if you agree with it, approve it by hand: +# edit proposals/20260804-001.md, change `status: proposed` to `status: approved` + +# 5. Apply it. There is no CLI for this on purpose — apply_proposal() is meant to be +# invoked by a human or by a step you write, never by the analysis session itself. +# This call runs the full evaluation gate first and only mutates the skill if it passes: +python3 -c " +import sys; sys.path.insert(0, 'scripts') +from proposal import load_proposal, apply_proposal +p = load_proposal('proposals/20260804-001.md') +result = apply_proposal(p, min_confidence=0.85) +print('can_apply:', result['can_apply']) +print('reason:', result.get('reason', result.get('evaluation_results'))) +" +``` + +What `can_apply` means depends on the host: on Claude Code, `True` means the skill file +was already rewritten on disk (`applied_by: direct`); on Hermes, `True` means the result +carries `skill_manage` instruction dicts that a separate agent step still has to execute +(`applied_by: agent`) — `apply_proposal()` never calls `skill_manage` itself. `False` means +the gate failed (check `evaluation_results` for which evaluator), the host can't write +skills, or the proposal itself is invalid (e.g. a `create_new` with a placeholder body). + ## Scheduled Runs This repo doesn't ship a ready-made cron/job prompt — that's operator-specific (how you @@ -73,6 +153,7 @@ skill-evolution/ │ ├── skill-evolution-fetch.sh # Cron wrapper (Hermes) │ └── skill-quality-report.sh # Cron wrapper for quality reports └── tests/ # pytest suite (one file per evaluator/feature area) +``` ## How It Works diff --git a/SKILL.md b/SKILL.md index 9e5539a..bd82d24 100644 --- a/SKILL.md +++ b/SKILL.md @@ -24,6 +24,23 @@ session analyzer are written in host-agnostic language and use the env vars No daemons. No external services. No GPU. Just your host agent, a scheduled job, and a few Python scripts. +## Minimum Configuration + +The full knob list lives in "Evaluation Environment Variables" below, but two things are +**not optional** — skip either one and the pipeline runs without erroring while quietly +doing the wrong thing: + +- **`SKILL_EVOLUTION_HOST`** — defaults to `hermes`. There is no auto-detection: on + Claude Code, forgetting to set this to `claude_code` means every read (and any write) + targets Hermes's session DB and skills tree instead of `~/.claude`. +- **A provider credential** for the evaluation gate — `ANTHROPIC_API_KEY` by default + (`SKILL_EVOLUTION_PROVIDER=claude`), or the key matching whichever provider you pick + instead. Without one, `llm_judge` fails **closed** on every call, which means the gate + fails on every proposal forever — even after you set `status: approved` by hand. There + is no error message pointing back at this; the gate just never passes. + +Everything else has a default you can leave alone for a first run. + ## How It Works ``` @@ -126,6 +143,26 @@ mutation instructions if it passes. On Hermes, those instructions are `skill_man calls; on Claude Code, they are direct file writes. Nothing in this repo invokes that step for you. +There is no CLI for this step on purpose — invoke it directly: + +```bash +python3 -c " +import sys; sys.path.insert(0, 'scripts') +from proposal import load_proposal, apply_proposal +p = load_proposal('proposals/20260804-001.md') +result = apply_proposal(p, min_confidence=0.85) +print('can_apply:', result['can_apply']) +print('reason:', result.get('reason', result.get('evaluation_results'))) +" +``` + +`can_apply: True` means different things per host: on Claude Code the skill file was +already rewritten (`applied_by: direct`); on Hermes the result carries `skill_manage` +instruction dicts a separate agent step still has to execute (`applied_by: agent`) — +`apply_proposal()` never calls `skill_manage` itself. `False` means the gate failed +(check `evaluation_results`), the host can't write skills, or the proposal itself is +invalid (e.g. a `create_new` with a placeholder body). + ## Scheduled Runs (Cron) Schedule a job on whichever host you're using. This repo doesn't ship a job prompt — @@ -409,6 +446,7 @@ raising out of whichever component reads it first. | Problem | Likely fix | |---------|-----------| | "No sessions found" | Check that the host adapter sees your session database: `hermes sessions list` (Hermes) or look under `~/.claude/projects/` (Claude Code) | +| A proposal stays `proposed` forever, even after you approve it | Check that a provider credential is set (`ANTHROPIC_API_KEY` by default) — without one, `llm_judge` fails closed on every call and the gate never passes. Run `python3 scripts/evaluate.py --list-evaluators` to see the resolved set, or drop `llm_judge` from `SKILL_EVOLUTION_EVALUATORS` if you only want the deterministic checks | | Cron timeout 3600s / script auto-recurses | `scripts/skill-evolution-fetch.sh` must exec `fetch_sessions.py` found relative to its own location — if your deployed wrapper instead execs a copy of itself (e.g. a self-referential install script), you get an infinite `exec` loop until the job times out. Fix: make sure the deployed wrapper matches this repo's `scripts/skill-evolution-fetch.sh` | | Proposals feel low quality | Try a stronger model in the cron job | | Script won't import | Check the path: Python can't import from directories with hyphens |