fix(errors): match an HTTP status code in an error message as a whole…

… word categorize_error matched the raw substrings "401", "402", "429", "502", "503" and "504", so a port or a GUID holding those digits changed the category. "Delegate SDK init failed: connect ECONNREFUSED 127.0.0.1:54013" became a non-retryable AGENT_AUTH_ERROR, and so did the same text on the Delegate send path. The Delegate init tests now also check that each case is retryable or not, and DELEGATE.md says which crashes are retried. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Mihaiii · Oct 6, 2026 · Sep 29, 2026 · Sep 29, 2026 · Sep 29, 2026 · Sep 29, 2026
commit 92dacee7a0c98eb158237f831aff131c96e84698
@@ -821,8 +821,11 @@ tail at WARNING instead.
 `_INIT_CONFIG_ERROR_MARKERS` (auth missing/rejected, missing slugs, unknown env) or on a whole-word
 401/403. A bare `"401"` substring also matched ports and GUIDs, such as `127.0.0.1:54013`, and ended a
 transient failure with no retry. Any other init error and the 60 s init deadline raise a retryable
-`AgentCrashError`. A mid-run respawn keeps the same classification: it once made every
-`AgentConfigError` retryable, so an expired token ended the task at `start()` but was retried later.
+`AgentCrashError`. The shared categorizer matches status codes as whole words too
+(`errors/categorization.py::_mentions_status`): the `AgentCrashError` goes through `categorize_error`,
+whose raw `"401"` substring made the same port and GUID cases a non-retryable auth error. A mid-run
+respawn keeps the same classification: it once made every `AgentConfigError` retryable, so an expired
+token ended the task at `start()` but was retried later.
 
 **Clear the process handle on every path that leaves the host dead or dying** — EOF, `error` frame,
 timeout (pre-check AND mid-`wait_for`), stop, `max_turns`. A shipped bug left it set after a mid-read

@@ -135,7 +135,7 @@ A wall-clock deadline (`timeout`) is enforced both between reads (a top-of-loop
 
 On any crash (host death, an `error` frame during a turn, or an unexpected exception), the agent:
 1. Sets `pending_turn` to a `crashed=True` TurnRecord with captured telemetry.
-2. Raises `AgentCrashError` (retryable) or `AgentConfigError` (non-retryable — missing Node/SDK install, or an init error that a retry cannot fix: missing or rejected auth, missing org/tenant slugs, or an unknown `DELEGATE_ENV`). Any other init error, and an init that does not respond within 60 s, is retryable.
+2. Raises `AgentCrashError` (retryable) or `AgentConfigError` (non-retryable — missing Node/SDK install, or an init error that a retry cannot fix: missing or rejected auth, missing org/tenant slugs, or an unknown `DELEGATE_ENV`). Any other init error, and an init that does not respond within 60 s, is retryable, unless its message matches a category that is not retried (a timeout, or an auth, billing or content-filter error). An HTTP status code matches only as a whole word, so a port or a GUID that contains `401` stays retryable.
 3. The orchestrator reads `pending_turn` and calls `discard_pending_turn()` to roll back state.
 
 The crash reason never includes the host's stderr. The agent logs the last 20 stderr lines at WARNING instead, because the error categorizer matches words in the reason, and stderr contains the sandbox path, which contains the task id.
@@ -170,7 +170,7 @@ Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).
 1. **No multi-generation transcript splitting.** This agent builds one `AssistantMessage` per `communicate()` call, not one per backend round-trip. The host does send the signals a split needs (`isStepStart` on `message` events and per-round-trip `turnUsages` on `result`), but this agent does not use them yet.
 2. **`max_turns` is enforced by coder_eval, not by the host.** The host's `maxSteps` option on `send` does not stop the turn (confirmed live: `maxSteps: 2` ran 7 steps and only reported `maxStepsReached: true`), so this agent does not send it. See [Run-Limit Parity](HARNESS_PARITY.md).
    A turn cut at `max_turns`, or by a cooperative early stop, keeps the token usage and cost of every model call that finished before the cut, as on the other agents. The host sends a `usage` frame for each call before that call's tool results. The call in progress at the cut has no usage. A host that does not send `usage` frames reports usage only on its final `result` frame, which a cut turn never gets: that turn has no usage, and the agent logs a warning.
-3. **Only three backend failures get a specific diagnosis.** A Cloudflare WAF block page is reported as a content-filter failure and is not retried: the same prompt or tool result is blocked again. An SSE connect timeout is reported as a connection failure and is retried. A session conflict ("A reply is already being generated") is retried in a new conversation. There is no first-response stall detection: a stalled turn ends at its turn timeout. Every other crash ends the turn as a retryable `AgentCrashError`.
+3. **Only three backend failures get a specific diagnosis.** A Cloudflare WAF block page is reported as a content-filter failure and is not retried: the same prompt or tool result is blocked again. An SSE connect timeout is reported as a connection failure and is retried. A session conflict ("A reply is already being generated") is retried in a new conversation. There is no first-response stall detection: a stalled turn ends at its turn timeout. Every other crash ends the turn as an `AgentCrashError`, which is retried unless its message matches a category that is not retried (a timeout, or an auth, billing or content-filter error).
 4. **`sdk_options` accepts only `effort`.** Reasoning effort uses the same `sdk_options.effort` key as Claude Code, so `-D agent.sdk_options.effort=high` works for both agents. Any other key, or an `effort` that is not a string, is a validation error. The host checks the tier itself: it logs a tier it does not recognize and uses the model's default. `project_id`, `session_id` and `enable_computer_use` are typed config fields.
 5. **`enable_computer_use: true` requires local permissions and is unavailable on Linux.** macOS needs Accessibility + Screen Recording grants; the SDK throws unconditionally on Linux when this is enabled.
 

@@ -5,6 +5,7 @@
 """
 
 import logging
+import re
 from typing import Any
 
 from .agent import AgentConfigError, AgentCrashError
@@ -68,6 +69,11 @@ def _categorize_by_exception_type(error: Exception, component: str) -> ErrorCate
     return None
 
 
+def _mentions_status(error_str: str, *codes: str) -> bool:
+    """True when an HTTP status code appears as a whole word, so a port or a GUID holding its digits does not match."""
+    return any(re.search(rf"\b{code}\b", error_str) for code in codes)
+
+
 def _categorize_by_message(error_str: str, component: str) -> ErrorCategory | None:
     """String-pattern matching on the already-lowercased error message.
 
@@ -78,20 +84,23 @@ def _categorize_by_message(error_str: str, component: str) -> ErrorCategory | No
     reaches ``PACKAGE_INSTALL_ERROR`` (retryable) instead of ``UNKNOWN``.
     """
     # Authentication errors
-    if any(pat in error_str for pat in ["authentication", "unauthorized", "invalid api key", "401"]):
+    if any(pat in error_str for pat in ["authentication", "unauthorized", "invalid api key"]) or _mentions_status(
+        error_str, "401"
+    ):
         return ErrorCategory.AGENT_AUTH_ERROR
 
     # Billing/credit errors (NOT retryable). Broad patterns are intentional: a false
     # positive skips one retry, a false negative wastes every retry on an error that
     # will never succeed.
     if any(
-        pat in error_str
-        for pat in ["credit", "billing", "payment", "insufficient", "402", "quota exceeded", "spending limit"]
-    ):
+        pat in error_str for pat in ["credit", "billing", "payment", "insufficient", "quota exceeded", "spending limit"]
+    ) or _mentions_status(error_str, "402"):
         return ErrorCategory.AGENT_BILLING_ERROR
 
     # Rate limiting
-    if any(pat in error_str for pat in ["rate limit", "429", "ratelimit", "too many requests"]):
+    if any(pat in error_str for pat in ["rate limit", "ratelimit", "too many requests"]) or _mentions_status(
+        error_str, "429"
+    ):
         return ErrorCategory.AGENT_RATE_LIMIT
 
     # Timeouts
@@ -105,7 +114,9 @@ def _categorize_by_message(error_str: str, component: str) -> ErrorCategory | No
         return ErrorCategory.AGENT_INVALID_OUTPUT
 
     # API/Network errors
-    if any(pat in error_str for pat in ["api error", "connection", "network", "502", "503", "504"]):
+    if any(pat in error_str for pat in ["api error", "connection", "network"]) or _mentions_status(
+        error_str, "502", "503", "504"
+    ):
         if component == "agent":
             return ErrorCategory.AGENT_API_ERROR
         return None  # fall through: let the component group categorize it

@@ -26,6 +26,7 @@
 from coder_eval.errors import AgentConfigError, AgentCrashError, TurnTimeoutError
 from coder_eval.errors.categories import ErrorCategory
 from coder_eval.errors.categorization import categorize_error
+from coder_eval.errors.retry import should_retry
 from coder_eval.models import AgentKind, DelegateAgentConfig
 from coder_eval.reports.markdown import collect_agent_settings_rows
 from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent
@@ -215,9 +216,11 @@ async def test_only_an_init_error_a_retry_cannot_fix_is_non_retryable(
     ):
         patch_exec([_line({"type": "error", "message": host_message, "stack": "Error: ..."})])
         agent = DelegateAgent(_config())
-        with pytest.raises(error_type, match="Delegate SDK init failed"):
+        with pytest.raises(error_type, match="Delegate SDK init failed") as excinfo:
             await agent.start(str(tmp_path))
         assert agent._process is None
+        category = categorize_error(excinfo.value, {"component": "agent"})
+        assert should_retry(category, 0) is (error_type is AgentCrashError), category
 
     async def test_a_config_init_error_names_the_variables_coder_eval_reads(self, patch_exec, tmp_path):
         """The host's message names the host's own variables, which coder_eval keeps out of its env."""

@@ -9,6 +9,7 @@
 
 import pytest
 
+from coder_eval.errors import AgentCrashError
 from coder_eval.errors.categories import RETRY_CONFIG, ErrorCategory
 from coder_eval.errors.categorization import categorize_error
 from coder_eval.errors.executor import execute_with_retry
@@ -163,6 +164,21 @@ def test_categorize_error_timeout_component_specific():
         # String matching - rate limit
         (Exception("Rate limit exceeded"), {}, None, ErrorCategory.AGENT_RATE_LIMIT),
         (Exception("429 Too Many Requests"), {}, None, ErrorCategory.AGENT_RATE_LIMIT),
+        # String matching - a status code matches only as a whole word
+        (Exception("HTTP 401: token expired"), {}, None, ErrorCategory.AGENT_AUTH_ERROR),
+        (
+            AgentCrashError("init failed: connect ECONNREFUSED 127.0.0.1:54013"),
+            {"component": "agent"},
+            None,
+            ErrorCategory.AGENT_CRASH,
+        ),
+        (
+            AgentCrashError("tenant c7a3f401-0000-4000-8000-000000000402 is not reachable"),
+            {"component": "agent"},
+            None,
+            ErrorCategory.AGENT_CRASH,
+        ),
+        (AgentCrashError("upstream port 54290 closed"), {"component": "agent"}, None, ErrorCategory.AGENT_CRASH),
         # String matching - disk
         (Exception("No space left on device"), {}, None, ErrorCategory.DISK_FULL),
         # Component-specific matching