Skip to content

fix(skill-creator): swap real SKILL.md instead of temp .claude/commands - #1027

Open
john-savepoint wants to merge 2 commits into
anthropics:mainfrom
john-savepoint:fix/skill-discovery-real-skill-md
Open

john-savepoint wants to merge 2 commits into
anthropics:mainfrom
john-savepoint:fix/skill-discovery-real-skill-md

Conversation

@john-savepoint

Copy link
Copy Markdown

Summary

Phase 3 trigger eval (scripts/run_eval.py) returns recall=0% structurally because the detection mechanism writes a temp .claude/commands/<hash>.md file and expects modern Claude Code to surface it as an invokable skill in available_skills. That assumption is stale — CC 2.x discovers skills from ~/.claude/skills/<name>/SKILL.md only; .claude/commands/ files are slash commands now, not router-visible skills. Net: the subprocess never invokes the temp skill, never matches the hash in tool_input → recall=0% across all queries / models / hook configs.

What changes

Replace the temp-command mechanism with a real-world test path:

  • swap_skill_description() atomically writes the candidate description into the real ~/.claude/skills/<name>/SKILL.md frontmatter as a YAML block scalar, preserving all other fields and the entire body.
  • run_eval() does a single swap per iteration, runs all parallel workers against the swapped skill, restores from saved original via try/finally.
  • atexit + SIGINT/SIGTERM/SIGHUP handlers restore on crash.
  • Sibling .md.swap-backup file written before swap as belt-and-suspenders.
  • run_single_query() detects by skill name (not hash): Skill tool with input.skill == skill_name, OR Read tool with /skills/<name>/SKILL.md.
  • Drops cargo-cult CLAUDECODE env strip — verified via CC source: that env var is only written by parent CC for the hint protocol; no guard reads it. Subprocess works fine with it set.

scripts/run_loop.py updated to pass skill_path through to run_eval.

Verification

Frontmatter replacer handles all four YAML description forms (single-line, quoted, |, >, |-, >-) — verified by 4 unit tests.

Smoke-tested on a real installed skill (medusa-pro): 2/2 queries pass, recall went from 0% → 100% on should-trigger queries; precision 100% on should-not. SKILL.md byte-identical after restore; backup file removed.

Why it matters

This unblocks the entire Phase 3 optimization loop. Without it, run_loop.py burns ~30s per query producing useless data because the underlying run_eval can never observe a positive trigger. Anyone running skill-creator against a recent CC version (2.0+) sees this.

Test plan

  • Run python -m scripts.run_eval --skill-path ~/.claude/skills/<any> --eval-set <eval.json> --runs-per-query 1 --num-workers 1 against a known-good description and confirm should-trigger queries trigger.
  • Verify SKILL.md byte-identical pre/post run.
  • Verify .md.swap-backup is cleaned up after a successful run.
  • Send SIGINT mid-run; verify SKILL.md restored from _active_swaps registry.

Phase 3 trigger eval (run_eval.py) returns recall=0% structurally because
the detection mechanism writes a temp .claude/commands/<hash>.md file and
expects modern Claude Code to surface it as an invokable skill in
'available_skills'. That assumption is stale — CC 2.x discovers skills
from ~/.claude/skills/<name>/SKILL.md only; .claude/commands/ files are
slash commands now, not router-visible skills. Net: subprocess never
invokes the temp skill, never matches the hash in tool_input → recall=0%
across all queries / models / hook configs.

This PR replaces the broken mechanism with a real-world test path:

- swap_skill_description() atomically writes the candidate description
  into the real ~/.claude/skills/<name>/SKILL.md frontmatter as a YAML
  block scalar, preserving all other fields and the entire body.
- run_eval() does a single swap per iteration, runs all parallel workers
  against the swapped skill, restores from saved original via try/finally.
- atexit + SIGINT/SIGTERM/SIGHUP handlers restore on crash.
- Sibling .md.swap-backup file written before swap as belt-and-suspenders.
- run_single_query() detects by skill name (not hash): Skill tool with
  input.skill == skill_name OR Read tool with /skills/<name>/SKILL.md.
- Drops cargo-cult CLAUDECODE env strip — verified via CC source: that
  env var is only WRITTEN by parent CC for the hint protocol; no guard
  reads it. Subprocess works fine with it set.

run_loop.py updated to pass skill_path through to run_eval.

The new frontmatter replacer handles all four YAML description forms
(single-line, quoted, |, >, |-, >-) — verified by 4 unit tests.

Smoke-tested on a real installed skill: 2/2 queries pass, recall went
from 0% → 100% on should-trigger queries; precision 100% on should-not.
SKILL.md byte-identical after restore; backup file removed.

This unblocks the entire Phase 3 optimization loop. Without it,
run_loop.py burns ~30s per query producing useless data because the
underlying run_eval can never observe a positive trigger.
Without this, concurrent swap_skill_description() calls (e.g. two
'python -m scripts.run_loop' processes from different tmux windows
targeting the same skill) race on the underlying SKILL.md file:

1. Run A swaps in description-A → SKILL.md = A
2. Run B swaps in description-B → SKILL.md = B  (overwrites A)
3. Run A restores its saved 'original' → SKILL.md back to original
4. Run B restores its 'original' (which was A's swap) → SKILL.md
   left in description-A state, or worse interleavings.

The in-process atexit hashmap can't help across processes. fcntl.flock
on a sidecar .md.swap-lock file does — second process blocks at
acquire until first releases, naturally serializing the swaps.

Implementation:
- _acquire_flock opens <skill>.md.swap-lock with LOCK_EX (blocking)
- swap_skill_description acquires before mutating; restore releases
- _restore_all_active_swaps (atexit / signal handler) releases all
  held locks on crash
- Falls back to in-process-only locking on Windows (no fcntl)

Use case: parallel skill upgrades across multiple tmux windows.
Different skills run fully concurrent (locks are per-SKILL.md path).
Same skill queues cleanly on the lock instead of corrupting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant