Repository navigation
fix(skill-creator): run_eval trigger detection misses real skill name and bails on first non-Skill tool - #1323
Polluelo978 wants to merge 1 commit into
Conversation
…n-Skill tool in run_eval run_eval.py::run_single_query failed to detect skill triggering, making run_loop.py report recall=0% for every should-trigger query. With all candidates tied at 0, the loop returns the original description and never actually optimizes. Two root causes in the stream detection: 1. Only the registered temp command name (clean_name) was matched, but an installed skill fires under its real name (skill_name) -> never matched. 2. The first non-Skill/Read tool_use caused an immediate `return False`, missing a later Skill/Read call (real runs often call Bash/WebFetch first). Fix: - Match skill_name in addition to clean_name (streaming path and fallback). - Don't return False on the first non-Skill/Read tool; reset pending state and keep scanning until type == "result". Verified on a skill + eval set (3 runs/query, --model claude-sonnet-4-6 --timeout 120): recall 0% -> 100%, precision 100%, held-out test 8/8.
|
Strong +1 — this fixes a bug that fully breaks the description optimizer in a common setup. With a user-global After this PR's scan-past-non-skill-tools logic, the measurements become meaningful again — two skills I tested split to a true 17% vs 100% recall instead of a uniform 0% (the 17% one is a skill whose task the model self-serves; the 100% one needs a specialized methodology, so it consults reliably). Same intent as the earlier #1208. |
|
Update: PR #1298 (MartinCajiao) addresses the same detection issues plus the architectural cause (artifact install path), Windows stream reading, and parallel worker race. It field-scopes the trigger match (vs. substring here) and is more complete overall. Worth reviewing #1298 as the potential merge target if maintainers are looking at both. |
| # Don't bail: other tools (Bash/WebFetch/etc.) | ||
| # may precede a later Skill/Read call. Reset the | ||
| # pending state and keep scanning. | ||
| pending_tool_name = None |
3 participants
Problem
scripts/run_eval.py::run_single_queryfails to detect that a skill triggered, causing the description-optimization loop (run_loop.py) to report recall=0% for every should-trigger query. With all candidates tied at 0, the loop just returns the original description and never actually optimizes.Two root causes in the stream detection:
{skill}-skill-{uuid}and only checks whether that random name (clean_name) appears in the tool input. But an installed skill fires under its real name (e.g.cve-intel), so the random name never matches -> always False.Bash/WebFetch/etc. before consulting the skill. Thecontent_block_starthandler doesreturn Falseon the first such tool, missing a later Skill/Read call.Fix
skill_namein addition to the registeredclean_name, in both the streaming path and the full-assistant-message fallback.type == "result".Early-return on a positive match keeps positives fast; negatives run to
result(short Q&A), bounded by--timeout.Evidence (same skill + eval set, 3 runs/query,
--model claude-sonnet-4-6 --timeout 120)After the fix the loop can measure description quality as intended (held-out test 8/8).
Notes
clean_namematching is preserved;skill_namematching is additive.skills/skill-creator/scripts/run_eval.py.