Skip to content

fix(mcp-builder): accumulate repeated -e/--env and -H/--header values - #1896

Closed
sxh313 wants to merge 1 commit into
anthropics:mainfrom
sxh313:fix/mcp-eval-repeatable-flags
Closed

sxh313 wants to merge 1 commit into
anthropics:mainfrom
sxh313:fix/mcp-eval-repeatable-flags

Conversation

@sxh313

@sxh313 sxh313 commented Sep 27, 2026

Copy link
Copy Markdown

Closes #1895

The change

One file, two lines — skills/mcp-builder/scripts/evaluation.py lines 329 and 333:

-    stdio_group.add_argument("-e", "--env", nargs="+", help="Environment variables in KEY=VALUE format (stdio only)")
+    stdio_group.add_argument("-e", "--env", action="extend", nargs="+", help="Environment variables in KEY=VALUE format (stdio only)")
-    remote_group.add_argument("-H", "--header", nargs="+", dest="headers", help="HTTP headers in 'Key: Value' format (sse/http only)")
+    remote_group.add_argument("-H", "--header", action="extend", nargs="+", dest="headers", help="HTTP headers in 'Key: Value' format (sse/http only)")

nargs="+" with the default store action rebinds args.env / args.headers on
every occurrence, so repeating the flag discards everything before the last
occurrence. reference/evaluation.md tells users to repeat both flags
(:440-446 for -e, :454-459 for -H), and nothing reports the loss —
parse_env_vars() / parse_headers() only warn about malformed values, and
API_KEY=abc123 is well-formed.

Red / green, measured

Both runs use the same script and the same case table; the "red" file is
git show origin/main:skills/mcp-builder/scripts/evaluation.py extracted explicitly
(so it is not affected by the working tree). Only anthropic and connections are
stubbed — this box has no ANTHROPIC_API_KEY and neither dependency installed, so
the observable output is the argument set main() hands to create_connection().
origin/main's blob and the branch differ on exactly lines 329 and 333, verified by
line-by-line comparison.

CASE 1: two -e, as reference/evaluation.md:440-446 tells you to write it
  RED_main      env={'DEBUG': 'true'}                      <- API_KEY lost
  GREEN_branch  env={'API_KEY': 'abc123', 'DEBUG': 'true'}
CASE 2: two -H, as reference/evaluation.md:454-459 tells you to write it
  RED_main      headers={'X-Custom-Header': 'value'}        <- Authorization lost
  GREEN_branch  headers={'Authorization': 'Bearer token123', 'X-Custom-Header': 'value'}
CASE 3: REGRESSION - one -e carrying two values
  RED_main      env={'API_KEY': 'abc123', 'DEBUG': 'true'}
  GREEN_branch  env={'API_KEY': 'abc123', 'DEBUG': 'true'}   identical
CASE 4: MIXED - one -e with two values plus a second -e
  RED_main      env={'C': '3'}                              <- 2 of 3 lost
  GREEN_branch  env={'A': '1', 'B': '2', 'C': '3'}
CASE 5: REGRESSION - neither -e nor -H given
  RED_main      env=None headers=None
  GREEN_branch  env=None headers=None                        identical

Cases 1, 2 and 4 are the defect; 3 and 5 are the two shapes that already worked,
re-run to show extend adds rather than changes them. default=None is safe with
extend (case 5): argparse's _copy_items turns None into [] before extending,
and main() guards with if args.env / if args.headers.

$ python -m py_compile skills/mcp-builder/scripts/evaluation.py     # branch
py_compile OK
$ python -m py_compile evaluation_main.py                           # origin/main bytes
py_compile OK

action="extend" needs Python 3.8+; this module already requires 3.10+ (-> str | None at line 79, list[str]/dict[str, str] at 275/290), so it adds no floor.

Scope

-a/--args (line 328) has the same declaration and the same overwrite behaviour, but
no documented usage repeats -a, so it is left untouched here rather than changed on
speculation. It is named in #1895.

Also recorded in #1895 and deliberately not part of this diff: 4 of the 6
python scripts/evaluation.py ... lines in reference/evaluation.md exit with
error: the following arguments are required: eval_file before reaching this code,
because the last nargs="+" option eats the trailing positional. The minimal repair
there is prose (move the operand, or add --), so it is disclosed rather than
bundled. That is why the cases above put the XML file first — it is the shape that
parses on both main and the branch, so the comparison isolates this change.

Duplicates

24 open PRs touch this file (#657 #747 #1321 #1377 #1386 #1397 #1588 #1602 #1674
#1676 #1683 #1690 #1720 #1724 #1780 #1837–#1845); all 24 were re-confirmed open and
all 24 patches re-downloaded before filing. Grepping added/removed lines for the two
options and the two parse helpers, only #747 rewrites these declarations, and it is a
pure line-wrap reformat that keeps nargs="+" and adds no action= — the overwrite
survives it. No PR touches parse_headers() or parse_env_vars().

Both options were declared with nargs="+" and argparse's default store
action, so a second occurrence replaced the first instead of extending the
list. reference/evaluation.md documents the repeated form for both flags, so
-e API_KEY=... -e DEBUG=true launched the server with only DEBUG, and
-H "Authorization: ..." -H "X-Custom-Header: ..." sent no Authorization
header - silently, with no warning from parse_env_vars()/parse_headers().

action="extend" keeps the single-occurrence form working and makes the
documented repeated form accumulate.
@98zc5g5jyw-arch

Copy link
Copy Markdown

Reviewed at head d3e839cb7f6b.

  • ✅ Two-state end-to-end repro (Python 3.11.15) — identical harness driven against the origin/main blob (sha256 49ed1d17…) and this branch (d42125c2…, matches the worktree file); values below are the env/headers kwargs actually passed to create_connection:
    • eval.xml -e API_KEY=abc123 -e DEBUG=true → main {'DEBUG': 'true'}; branch {'API_KEY': 'abc123', 'DEBUG': 'true'}
    • eval.xml -H "Authorization: Bearer token123" -H "X-Custom-Header: value" → main {'X-Custom-Header': 'value'}; branch keeps both
    • eval.xml -e A=1 B=2 -e C=3 → main {'C': '3'}; branch {'A': '1', 'B': '2', 'C': '3'}
    • Single-occurrence (two values after one -e) and no-flag runs are identical on both ({'API_KEY': 'abc123', 'DEBUG': 'true'} / None), so existing usage is unchanged
  • ✅ Scope — git diff origin/main...HEAD: 1 file, +2/−2, one hunk @@ -326,11 +326,11 @@; only lines 329 and 333 change (action="extend" added); -a/--args at line 328 untouched
  • ✅ Docs support it — reference/evaluation.md L444–445 repeats -e and L457–458 repeats -H; both examples silently lose all but the last value on main
  • ✅ Interpreter floor — action="extend" needs 3.8+; the module already requires 3.10+ (str | None, L79) and 3.9+ (asyncio.to_thread); python3 -m py_compile passes on both blobs
  • ✅ Sweep — grep -rn 'nargs="+"' skills/ finds only evaluation.py:328/329/333, and no doc example repeats -a (single -a … at L434/L443/L525/L566), consistent with leaving it as store
  • ℹ️ The disclosed positional-arg shape (evaluation.xml after a nargs="+" option, as in the doc examples) fails identically on both sides (SystemExit(2), "the following arguments are required: eval_file") — pre-existing, unaffected by this PR
  • ℹ️ Issue mcp-builder: repeated -e/--env and -H/--header flags in evaluation.py silently drop all but the last value #1895 (OPEN) matches exactly the two dropped-value behaviors reproduced above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mcp-builder: repeated -e/--env and -H/--header flags in evaluation.py silently drop all but the last value

2 participants