Repository navigation
fix(skill-creator): swap real SKILL.md instead of temp .claude/commands - #1027
Open
john-savepoint wants to merge 2 commits into
Open
john-savepoint wants to merge 2 commits into
john-savepoint wants to merge 2 commits into
Conversation
Phase 3 trigger eval (run_eval.py) returns recall=0% structurally because the detection mechanism writes a temp .claude/commands/<hash>.md file and expects modern Claude Code to surface it as an invokable skill in 'available_skills'. That assumption is stale — CC 2.x discovers skills from ~/.claude/skills/<name>/SKILL.md only; .claude/commands/ files are slash commands now, not router-visible skills. Net: subprocess never invokes the temp skill, never matches the hash in tool_input → recall=0% across all queries / models / hook configs. This PR replaces the broken mechanism with a real-world test path: - swap_skill_description() atomically writes the candidate description into the real ~/.claude/skills/<name>/SKILL.md frontmatter as a YAML block scalar, preserving all other fields and the entire body. - run_eval() does a single swap per iteration, runs all parallel workers against the swapped skill, restores from saved original via try/finally. - atexit + SIGINT/SIGTERM/SIGHUP handlers restore on crash. - Sibling .md.swap-backup file written before swap as belt-and-suspenders. - run_single_query() detects by skill name (not hash): Skill tool with input.skill == skill_name OR Read tool with /skills/<name>/SKILL.md. - Drops cargo-cult CLAUDECODE env strip — verified via CC source: that env var is only WRITTEN by parent CC for the hint protocol; no guard reads it. Subprocess works fine with it set. run_loop.py updated to pass skill_path through to run_eval. The new frontmatter replacer handles all four YAML description forms (single-line, quoted, |, >, |-, >-) — verified by 4 unit tests. Smoke-tested on a real installed skill: 2/2 queries pass, recall went from 0% → 100% on should-trigger queries; precision 100% on should-not. SKILL.md byte-identical after restore; backup file removed. This unblocks the entire Phase 3 optimization loop. Without it, run_loop.py burns ~30s per query producing useless data because the underlying run_eval can never observe a positive trigger.
This was referenced Apr 25, 2026
Without this, concurrent swap_skill_description() calls (e.g. two 'python -m scripts.run_loop' processes from different tmux windows targeting the same skill) race on the underlying SKILL.md file: 1. Run A swaps in description-A → SKILL.md = A 2. Run B swaps in description-B → SKILL.md = B (overwrites A) 3. Run A restores its saved 'original' → SKILL.md back to original 4. Run B restores its 'original' (which was A's swap) → SKILL.md left in description-A state, or worse interleavings. The in-process atexit hashmap can't help across processes. fcntl.flock on a sidecar .md.swap-lock file does — second process blocks at acquire until first releases, naturally serializing the swaps. Implementation: - _acquire_flock opens <skill>.md.swap-lock with LOCK_EX (blocking) - swap_skill_description acquires before mutating; restore releases - _restore_all_active_swaps (atexit / signal handler) releases all held locks on crash - Falls back to in-process-only locking on Windows (no fcntl) Use case: parallel skill upgrades across multiple tmux windows. Different skills run fully concurrent (locks are per-SKILL.md path). Same skill queues cleanly on the lock instead of corrupting.
This was referenced May 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 trigger eval (
scripts/run_eval.py) returnsrecall=0%structurally because the detection mechanism writes a temp.claude/commands/<hash>.mdfile and expects modern Claude Code to surface it as an invokable skill inavailable_skills. That assumption is stale — CC 2.x discovers skills from~/.claude/skills/<name>/SKILL.mdonly;.claude/commands/files are slash commands now, not router-visible skills. Net: the subprocess never invokes the temp skill, never matches the hash intool_input→recall=0%across all queries / models / hook configs.What changes
Replace the temp-command mechanism with a real-world test path:
swap_skill_description()atomically writes the candidate description into the real~/.claude/skills/<name>/SKILL.mdfrontmatter as a YAML block scalar, preserving all other fields and the entire body.run_eval()does a single swap per iteration, runs all parallel workers against the swapped skill, restores from saved original viatry/finally.atexit+SIGINT/SIGTERM/SIGHUPhandlers restore on crash..md.swap-backupfile written before swap as belt-and-suspenders.run_single_query()detects by skill name (not hash):Skilltool withinput.skill == skill_name, ORReadtool with/skills/<name>/SKILL.md.CLAUDECODEenv strip — verified via CC source: that env var is only written by parent CC for the hint protocol; no guard reads it. Subprocess works fine with it set.scripts/run_loop.pyupdated to passskill_paththrough torun_eval.Verification
Frontmatter replacer handles all four YAML description forms (single-line, quoted,
|,>,|-,>-) — verified by 4 unit tests.Smoke-tested on a real installed skill (
medusa-pro): 2/2 queries pass, recall went from 0% → 100% on should-trigger queries; precision 100% on should-not. SKILL.md byte-identical after restore; backup file removed.Why it matters
This unblocks the entire Phase 3 optimization loop. Without it,
run_loop.pyburns ~30s per query producing useless data because the underlyingrun_evalcan never observe a positive trigger. Anyone runningskill-creatoragainst a recent CC version (2.0+) sees this.Test plan
python -m scripts.run_eval --skill-path ~/.claude/skills/<any> --eval-set <eval.json> --runs-per-query 1 --num-workers 1against a known-good description and confirm should-trigger queries trigger..md.swap-backupis cleaned up after a successful run._active_swapsregistry.