Skip to content

docs(mcp-builder): make evaluation command examples runnable - #1921

Open
rudycelekli wants to merge 1 commit into
anthropics:mainfrom
rudycelekli:docs/mcp-evaluation-positional-first
Open

rudycelekli wants to merge 1 commit into
anthropics:mainfrom
rudycelekli:docs/mcp-evaluation-positional-first

Conversation

@rudycelekli

Copy link
Copy Markdown

Problem

Four of the six command examples in skills/mcp-builder/reference/evaluation.md fail before evaluation starts. Argparse options -a, -e, and -H accept one or more values, so when an example puts evaluation.xml after the final such option, argparse consumes it and reports the required eval_file is missing. The repeated -e and -H spellings also drop earlier values on current main (tracked separately by #1896).

Change

Put the XML positional argument first in every example and group multiple environment variables or headers under one flag. Add a short rule explaining the order.

Verification

I extracted all six Bash examples and parsed them with the script's current argparse declarations. On main, four failed with the following arguments are required: eval_file; after this edit, all six parse, and the multi-value examples retain both environment variables and both headers. git diff --check passes.

@98zc5g5jyw-arch

Copy link
Copy Markdown

Reviewed at head 271b027.

Scope: skills/mcp-builder/reference/evaluation.md (+15/−15), docs only. Single commit, nothing else vs merge-base 3337550.

The failure is real at the parent — every Bash block containing evaluation.py was extracted from the merge-base doc and its tokens fed to the script's real argparse (-a/--args, -e/--env, -H/--header are nargs='+'; ArgumentParser.parse_args is spied so main() can never reach an MCP server):

  • 4 of 6 examples exit 2 with evaluation.py: error: the following arguments are required: eval_file: the stdio quickstart (-t stdio -c python -a my_mcp_server.py evaluation.xml), the env variant (-e DEBUG=true evaluation.xml), and both SSE/HTTP examples (-H "Authorization: Bearer ***" evaluation.xml). The trailing path is consumed by the last multi-value flag.
  • The two examples that end on a single-value flag (-o … evaluation.xml) parse fine at the parent — the "four of six" claim is exact.
  • The script itself is byte-identical at both states (only the .md changed), so both sweeps were parsed by the same argparse declarations — the delta is purely the command text. No evaluation is ever executed: the parse spy exits right after parse_args.

Fix verified (head)

  • ✅ All 6 examples now parse: eval_file resolves correctly in each (evaluation.xml, my_evaluation.xml), transports/args/output come through as written.
  • ✅ Multi-value examples keep every value — asserted from the parsed namespace, not the raw text: -e API_KEY=abc123 DEBUG=true → env == ['API_KEY=abc123', 'DEBUG=true']; -H "Authorization: Bearer ***" "X-Custom-Header: value" → headers == ['Authorization: Bearer ***', 'X-Custom-Header: value'].
  • ✅ Every flag used in the examples (-t -c -a -e -u -H -o) exists in the script's --help with the documented ARGS [ARGS ...] / ENV [ENV ...] / HEADERS [HEADERS ...] forms, and the doc's usage block matches the real help output. Both added rules are accurate for all four failing cases.
  • ✅ Markdown hygiene: zero trailing-whitespace lines, final newline added (the parent had none), fence tags untouched.

Notes (non-blocking):

  1. The same-class defect still lives in the script itself: the --help epilog examples at scripts/evaluation.py lines 312 and 315 (-a my_server.py eval.xml and -H "Authorization: Bearer ***" eval.xml) fail identically at this head with required: eval_file; only the third epilog example parses. A docs-only PR cannot fix these — flagging for a follow-up in the script.
  2. The "one flag, many values" form also sidesteps the repeated-flag overwrite tracked in fix(mcp-builder): accumulate repeated -e/--env and -H/--header values #1896: with the path first, -e A=1 -e B=2 still yields env == ['B=2'] and repeated -H yields only the last header (verified against this head).
  3. No other reference/*.md in the repo embeds runnable evaluation.py commands, so the sweep is complete.

No remaining findings at 271b027.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants