Skip to content

fix(skill-creator): match real skill name in eval trigger detection (#1338) - #1340

Closed
KhalifaGad wants to merge 1 commit into
anthropics:mainfrom
KhalifaGad:fix/eval-trigger-name-match
Closed

KhalifaGad wants to merge 1 commit into
anthropics:mainfrom
KhalifaGad:fix/eval-trigger-name-match

Conversation

@KhalifaGad

Copy link
Copy Markdown

Fixes the trigger-detection bug reported in #1338.

Supersedes #1339, which was auto-closed after an accidental force-push corrupted its branch history (blowing the diff up to the whole tree). History rebuilt; this PR is the intended single-file change.

Problem

run_eval.py's run_single_query() installs the skill under a randomized command name stored in clean_name:

clean_name = f"{skill_name}-skill-{unique_id}"   # e.g. "change-delivery-skill-38eea5"

Detection then checks whether clean_name appears in the skill invocation. But Claude Code invokes the skill by its real name, not the randomized filename. Captured tool_use from a real run:

{"skill": "change-delivery", "args": "..."}

"change-delivery-skill-38eea5" is never a substring of "change-delivery", so the match is False 100% of the time. Every query — including textbook matches — scores "not triggered," making run_eval, run_loop, and the description optimizer non-functional on current Claude Code: every candidate description scores identically (100% precision / 0% recall), so the optimizer has no signal.

Fix

Add a field-scoped _skill_triggered() helper, used by both detection paths (streamed events and full assistant message):

  • Skill calls: exact-match the real skill name (skill_name) — or clean_name — on the skill field.
  • Read calls: match on file_path.

Field-scoping (rather than substring-matching the whole serialized input) avoids the inverse failure mode: a short, common skill name like test or pdf would otherwise match echoed argument text and over-count triggers. The streamed early-detection is preserved, but the stream path can now only early-return True — the full-message path stays authoritative, so it can't reintroduce the original false-negative.

Verification

Claude Code 2.1.140, model claude-opus-4-8, the only change being this patch:

Matcher should-trigger should-not-trigger
before (clean_name only) 0 / 10 10 / 10 (pass only because nothing ever triggers)
after (this PR) 8 / 10 10 / 10

The remaining 2 should-triggers are genuine model behavior (it self-serves those tasks), not detection misses. Additionally a topically-adjacent negative ("rename our deliveryPlan variable to plan…") stays correctly untriggered, and a unit check confirms a pdf-named skill no longer false-positives on a convert report.pdf argument.

Note for maintainers

Verified on Claude Code 2.1.140, where the model invokes via {"skill": "<skill_name>"}. The helper exact-matches that field (plus clean_name); if other CLI versions use a different invocation shape, only _skill_triggered() needs adjusting.

Refs #1338

run_eval's trigger detector searched for clean_name — the randomized
command filename '<skill_name>-skill-<uuid>' — inside the model's skill
invocation. But the model invokes the skill by its real name (e.g.
{"skill": "<skill_name>"}), which never contains the randomized
suffix, so the match was always False: every query (including obvious
matches) scored 'not triggered', making run_eval/run_loop and the
description optimizer non-functional on current Claude Code.

Add a field-scoped _skill_triggered() helper used by both the streamed
and full-message paths: exact-match the real skill name (or clean_name)
on a Skill call's 'skill' field, and match Read calls on file_path.
Field-scoping avoids the inverse failure — a short, common skill name
(e.g. 'test', 'pdf') substring-matching echoed argument text and
over-counting triggers. The stream path now only ever early-returns
True, leaving the full-message path authoritative, so it cannot
reintroduce the original false-negative.

Refs anthropics#1338
@KhalifaGad

Copy link
Copy Markdown
Author

Closing as a duplicate — I only found the prior art after opening this. #556 already tracks this bug, #1298 is a comprehensive rework of exactly this path, and #1323 is essentially this same single-file fix. Another competing patch doesn't help. I've left the field-scoped matcher + a short-name false-positive demo as a suggestion on #1298 instead. Apologies for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant