sami7777/skill-evolution
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README and SKILL documentation with minimum configuration details and usage examples

Browse Source
This commit is contained in:
Carlo1911
2026-08-04 16:11:42 -05:00
parent 18df2fe7b4
commit 08b63db324
2 changed files with 119 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+81
View File
@@ -6,6 +6,32 @@ Reads past sessions, analyzes them against loaded skills, generates structured p
**No daemons. No external services. No GPU.** Just your host agent, a scheduled job, and Python stdlib.
## Minimum Configuration
Everything below has a sane default, but two things are **not optional** if you want the
pipeline to actually work end to end rather than silently doing nothing or getting stuck:
| Variable | Required when | Why |
|---|---|---|
| `SKILL_EVOLUTION_HOST` | You're on anything other than Hermes | Defaults to `hermes`. There is no auto-detection — running on Claude Code without setting this to `claude_code` reads (and would try to write) the wrong host's session DB and skills tree. |
| A provider credential (`ANTHROPIC_API_KEY` by default, or the key matching `SKILL_EVOLUTION_PROVIDER`) | Always, unless you only ever run `deterministic`-only evaluation | The evaluation gate's `llm_judge` (part of the default evaluator set) fails **closed** without a working provider — every proposal will fail the gate forever, even after you approve it, with no error beyond "gate failed" in the result. |
Everything else is optional and only needs to be set to change a default:
| Variable | Default | Purpose |
|---|---|---|
| `SKILL_EVOLUTION_REPO` | current working directory | Where a cron job finds `scripts/` |
| `SKILL_EVOLUTION_PROPOSALS_DIR` | `./proposals/` | Where proposal files are read/written |
| `SKILL_EVOLUTION_AUTO_APPLY` / `SKILL_EVOLUTION_MIN_CONFIDENCE` | off / `0.85` | Read by *your* apply step, not by the scripts themselves — see "Auto-Apply" below |
| `SKILL_EVOLUTION_EVALUATORS` | `deterministic,llm_judge,regression` | Which evaluators the gate runs |
| `SKILL_EVOLUTION_GATE_STRICTNESS` | `strict` | `strict` (all must pass) or `majority` |
| `SKILL_EVOLUTION_PROVIDER` | `claude` | `claude` / `ollama` / `opencode` / `openai` / `gemini` |
| `SKILL_EVOLUTION_CLAUDE_CODE_HOME` | `~/.claude` | Only read when `SKILL_EVOLUTION_HOST=claude_code` |
| `SKILL_EVOLUTION_HISTORY_PATH` | `./eval_history.jsonl` | Evaluation history file |
See `CLAUDE.md` and `SKILL.md` for the full list — this table is the minimum you need to
read before your first run, not an exhaustive reference.
## Quick Start
**Hermes:**
@@ -40,6 +66,60 @@ export SKILL_EVOLUTION_AUTO_APPLY=true
export SKILL_EVOLUTION_MIN_CONFIDENCE=0.85
```
## Usage Example
A full manual walkthrough on Claude Code, from "do I have anything new to analyze" to
"a proposal is actually applied." This mirrors what a scheduled job automates, one step
at a time.
```bash
# 0. Configure the host and a provider for the evaluation gate (see "Minimum Configuration")
export SKILL_EVOLUTION_HOST=claude_code
export ANTHROPIC_API_KEY=sk-ant-...
# 1. Check how many unprocessed sessions exist, without marking anything as processed
python3 scripts/fetch_sessions.py --dry-run --lookback-hours 72
# → "14 unprocessed sessions found"
# 2. In a Claude Code session with the skill loaded, ask:
# "Run skill evolution analysis on my recent sessions"
#
# Under the hood, that prompt runs the same pipeline this repo ships as scripts —
# fetch_sessions.py's output formatted by analyze.py, plus skill_index.py's skill
# index — and feeds both to the host agent's own reasoning (the LLM analysis step
# is not a script in this repo; the agent performs it directly):
python3 scripts/fetch_sessions.py --lookback-hours 72 | python3 scripts/analyze.py > /tmp/sessions.txt
python3 scripts/skill_index.py > /tmp/skills.json
# The agent then writes one proposal file per finding into ./proposals/, and marks
# those sessions processed.
# 3. Inspect what got proposed
python3 scripts/proposal.py --list
python3 scripts/proposal.py --show 20260804-001
# 4. Review the proposal file and, if you agree with it, approve it by hand:
# edit proposals/20260804-001.md, change `status: proposed` to `status: approved`
# 5. Apply it. There is no CLI for this on purpose — apply_proposal() is meant to be
# invoked by a human or by a step you write, never by the analysis session itself.
# This call runs the full evaluation gate first and only mutates the skill if it passes:
python3 -c "
import sys; sys.path.insert(0, 'scripts')
from proposal import load_proposal, apply_proposal
p = load_proposal('proposals/20260804-001.md')
result = apply_proposal(p, min_confidence=0.85)
print('can_apply:', result['can_apply'])
print('reason:', result.get('reason', result.get('evaluation_results')))
"
```
What `can_apply` means depends on the host: on Claude Code, `True` means the skill file
was already rewritten on disk (`applied_by: direct`); on Hermes, `True` means the result
carries `skill_manage` instruction dicts that a separate agent step still has to execute
(`applied_by: agent`) — `apply_proposal()` never calls `skill_manage` itself. `False` means
the gate failed (check `evaluation_results` for which evaluator), the host can't write
skills, or the proposal itself is invalid (e.g. a `create_new` with a placeholder body).
## Scheduled Runs
This repo doesn't ship a ready-made cron/job prompt — that's operator-specific (how you
@@ -73,6 +153,7 @@ skill-evolution/
│ ├── skill-evolution-fetch.sh # Cron wrapper (Hermes)
│ └── skill-quality-report.sh # Cron wrapper for quality reports
└── tests/ # pytest suite (one file per evaluator/feature area)
```
## How It Works
SKILL.md
+38
View File
@@ -24,6 +24,23 @@ session analyzer are written in host-agnostic language and use the env vars
No daemons. No external services. No GPU. Just your host agent, a scheduled job, and a
few Python scripts.
## Minimum Configuration
The full knob list lives in "Evaluation Environment Variables" below, but two things are
**not optional** — skip either one and the pipeline runs without erroring while quietly
doing the wrong thing:
- **`SKILL_EVOLUTION_HOST`** — defaults to `hermes`. There is no auto-detection: on
Claude Code, forgetting to set this to `claude_code` means every read (and any write)
targets Hermes's session DB and skills tree instead of `~/.claude`.
- **A provider credential** for the evaluation gate — `ANTHROPIC_API_KEY` by default
(`SKILL_EVOLUTION_PROVIDER=claude`), or the key matching whichever provider you pick
instead. Without one, `llm_judge` fails **closed** on every call, which means the gate
fails on every proposal forever — even after you set `status: approved` by hand. There
is no error message pointing back at this; the gate just never passes.
Everything else has a default you can leave alone for a first run.
## How It Works
```
@@ -126,6 +143,26 @@ mutation instructions if it passes. On Hermes, those instructions are `skill_man
calls; on Claude Code, they are direct file writes. Nothing in this repo invokes that
step for you.
There is no CLI for this step on purpose — invoke it directly:
```bash
python3 -c "
import sys; sys.path.insert(0, 'scripts')
from proposal import load_proposal, apply_proposal
p = load_proposal('proposals/20260804-001.md')
result = apply_proposal(p, min_confidence=0.85)
print('can_apply:', result['can_apply'])
print('reason:', result.get('reason', result.get('evaluation_results')))
"
```
`can_apply: True` means different things per host: on Claude Code the skill file was
already rewritten (`applied_by: direct`); on Hermes the result carries `skill_manage`
instruction dicts a separate agent step still has to execute (`applied_by: agent`) —
`apply_proposal()` never calls `skill_manage` itself. `False` means the gate failed
(check `evaluation_results`), the host can't write skills, or the proposal itself is
invalid (e.g. a `create_new` with a placeholder body).
## Scheduled Runs (Cron)
Schedule a job on whichever host you're using. This repo doesn't ship a job prompt —
@@ -409,6 +446,7 @@ raising out of whichever component reads it first.
| Problem | Likely fix |
|---------|-----------|
| "No sessions found" | Check that the host adapter sees your session database: `hermes sessions list` (Hermes) or look under `~/.claude/projects/` (Claude Code) |
| A proposal stays `proposed` forever, even after you approve it | Check that a provider credential is set (`ANTHROPIC_API_KEY` by default) — without one, `llm_judge` fails closed on every call and the gate never passes. Run `python3 scripts/evaluate.py --list-evaluators` to see the resolved set, or drop `llm_judge` from `SKILL_EVOLUTION_EVALUATORS` if you only want the deterministic checks |
| Cron timeout 3600s / script auto-recurses | `scripts/skill-evolution-fetch.sh` must exec `fetch_sessions.py` found relative to its own location — if your deployed wrapper instead execs a copy of itself (e.g. a self-referential install script), you get an infinite `exec` loop until the job times out. Fix: make sure the deployed wrapper matches this repo's `scripts/skill-evolution-fetch.sh` |
| Proposals feel low quality | Try a stronger model in the cron job |
| Script won't import | Check the path: Python can't import from directories with hyphens |