Anthropic system updates reject otherwise recoverable tool history #51764

This issue might be a duplicate of or related to existing issues. Please check:

• fix(vertex): preserve tool continuations across system updates #43478: Similar provider-lowering failure when a chronological system update intersects a local tool continuation.
• Background service resumes sessions still running on another server that shares the database #51213: Reports the same Anthropic error during shared-server recovery, though its focus is execution ownership.
• fix(provider): Anthropic normalize splits tool-call from tool-result causing dangling tool_use error #25774: Anthropic history normalization can split tool calls from their results and cause related validation failures.

Thanks! #43478 concerns provider placement after a tool result; this repro has the update between the call and its available or repaired result. #51213 concerns competing execution owners (fixed by #51215), and #25774 concerns V1 normalizeMessages splitting co-located tool parts. #51765 addresses the separate deterministic V2 lowering failure; the existing repro comment on #51213 links back here.

This should be impossible to hit in opencode unless:

• you are doing it with the ai pkg itself
• you have plugins mutating the message orders

Did you encounter this in opencode???

Yes, we hit it in opencode (2.0.15 with a small unrelated patch), in a real session. Plugins were loaded, but none that reorder messages as far as we can tell.

The trigger was the #51213 situation: the same session was being driven by two opencode server processes at once. One appended an environment system update while the other's tool call was still outstanding, so the misordering came from session writes, not from a plugin. The snippet in the issue uses the ai package directly only to reproduce the rejection without two servers.

If #51215 (one execution owner per session) makes that interleaving unreachable, this is probably moot and I'm happy to close it and #51765.

@rekram1-node allow me a couple of hours to be more confident about the above (written by my orchestrator). but this is my initial diagnosis :)

please don't hesetate to tell me if the orchestrator is being annoying or unhelpful

Follow-up after checking the stored session history rather than inferring.

The session's environment updates alternate between two renderings that differ only in the temp-directory line, which is fixed per process: one server had TMPDIR set, the other (a background service) didn't. They interleave within seconds. At 02:21:00 an assistant step issued a tool call; at 02:21:02 an environment update from the other server was appended and that server started a step, which failed with this error; at 02:21:05 the tool result landed, the first server appended its own environment update and continued normally. The second failure (02:27:35) has the same shape.

So the misordering is in the stored history itself (not the ai package, not a plugin), and it came from two servers driving one session (#51213) on a build without single-owner handling. I found no single-server path. If #51215 makes this unreachable, I'm happy to close this and #51765.

fixed by #52421 and #52426, closing. the two-servers-on-one-session part is tracked in #51215.