Repository navigation
fix(skill-creator): match real skill name in eval trigger detection (#1338) - #1340
Closed
KhalifaGad wants to merge 1 commit into
Closed
KhalifaGad wants to merge 1 commit into
KhalifaGad wants to merge 1 commit into
Conversation
run_eval's trigger detector searched for clean_name — the randomized
command filename '<skill_name>-skill-<uuid>' — inside the model's skill
invocation. But the model invokes the skill by its real name (e.g.
{"skill": "<skill_name>"}), which never contains the randomized
suffix, so the match was always False: every query (including obvious
matches) scored 'not triggered', making run_eval/run_loop and the
description optimizer non-functional on current Claude Code.
Add a field-scoped _skill_triggered() helper used by both the streamed
and full-message paths: exact-match the real skill name (or clean_name)
on a Skill call's 'skill' field, and match Read calls on file_path.
Field-scoping avoids the inverse failure — a short, common skill name
(e.g. 'test', 'pdf') substring-matching echoed argument text and
over-counting triggers. The stream path now only ever early-returns
True, leaving the full-message path authoritative, so it cannot
reintroduce the original false-negative.
Refs anthropics#1338
Author
|
Closing as a duplicate — I only found the prior art after opening this. #556 already tracks this bug, #1298 is a comprehensive rework of exactly this path, and #1323 is essentially this same single-file fix. Another competing patch doesn't help. I've left the field-scoped matcher + a short-name false-positive demo as a suggestion on #1298 instead. Apologies for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the trigger-detection bug reported in #1338.
Problem
run_eval.py'srun_single_query()installs the skill under a randomized command name stored inclean_name:Detection then checks whether
clean_nameappears in the skill invocation. But Claude Code invokes the skill by its real name, not the randomized filename. Capturedtool_usefrom a real run:{"skill": "change-delivery", "args": "..."}"change-delivery-skill-38eea5"is never a substring of"change-delivery", so the match isFalse100% of the time. Every query — including textbook matches — scores "not triggered," makingrun_eval,run_loop, and the description optimizer non-functional on current Claude Code: every candidate description scores identically (100% precision / 0% recall), so the optimizer has no signal.Fix
Add a field-scoped
_skill_triggered()helper, used by both detection paths (streamed events and full assistant message):skill_name) — orclean_name— on theskillfield.file_path.Field-scoping (rather than substring-matching the whole serialized input) avoids the inverse failure mode: a short, common skill name like
testorpdfwould otherwise match echoed argument text and over-count triggers. The streamed early-detection is preserved, but the stream path can now only early-returnTrue— the full-message path stays authoritative, so it can't reintroduce the original false-negative.Verification
Claude Code
2.1.140, modelclaude-opus-4-8, the only change being this patch:clean_nameonly)The remaining 2 should-triggers are genuine model behavior (it self-serves those tasks), not detection misses. Additionally a topically-adjacent negative (
"rename our deliveryPlan variable to plan…") stays correctly untriggered, and a unit check confirms apdf-named skill no longer false-positives on aconvert report.pdfargument.Note for maintainers
Verified on Claude Code
2.1.140, where the model invokes via{"skill": "<skill_name>"}. The helper exact-matches that field (plusclean_name); if other CLI versions use a different invocation shape, only_skill_triggered()needs adjusting.Refs #1338