Context
From a 90-day retrospective of 271 logged incidents building two production SaaS apps with Claude Code. Pattern report; sibling reports linked at the bottom.
Type of Behavior Issue
Claude made incorrect assumptions (a recursive one: code written by an LLM over-trusts LLMs as components).
What Claude Actually Did
Whenever Claude generated code that calls an LLM as part of our product, it wrote the integration the way it writes calls to deterministic APIs — assuming the callee returns complete, ordered, contract-abiding output:
- Batch outputs aligned by array index instead of business keys. Green under every well-formed response; fails only on reordering/omission in production. After we learned this and wrote it into the project's AGENTS.md, the pattern still re-occurred on the next new batch call (our log: "every new batch call re-makes this mistake"). 3 incidents over 9 weeks.
- No handling for partial coverage — the callee omitting items it deems marginal.
- A "do not repeat these" exclusion list was copied verbatim into the output by the callee (3 of 16 generated items identical to the exclusion list) — generated code had assumed instruction-following was a guarantee.
- Judgment gates designed two-state (pass/fail) for an inherently three-state call (pass / fail / cannot-judge): the day the judge service failed, every item collapsed into "fail" and publishing halted site-wide.
Expected Behavior
When generating code that consumes model output, Claude should default to defensive integration: business-key alignment, partial-coverage handling ("unprocessed" marking + retry), a modeled third state for judge-style calls, and malformed-response test cases (shuffled / missing / extra items). This seems like a training-level prior worth fixing — the current prior treats LLM callees like deterministic services.
Reproducibility
Pattern-level: recurred even after the lesson was written into project instructions Claude reads every session.
Model / Version / Platform
Opus (most sessions; some Sonnet) · Claude Code 2.1.235 · Anthropic API · macOS
Impact
High — one incident published mismatched content to a real third-party platform (instantly removed by moderators); another halted all publishing site-wide.
Context
From a 90-day retrospective of 271 logged incidents building two production SaaS apps with Claude Code. Pattern report; sibling reports linked at the bottom.
Type of Behavior Issue
Claude made incorrect assumptions (a recursive one: code written by an LLM over-trusts LLMs as components).
What Claude Actually Did
Whenever Claude generated code that calls an LLM as part of our product, it wrote the integration the way it writes calls to deterministic APIs — assuming the callee returns complete, ordered, contract-abiding output:
Expected Behavior
When generating code that consumes model output, Claude should default to defensive integration: business-key alignment, partial-coverage handling ("unprocessed" marking + retry), a modeled third state for judge-style calls, and malformed-response test cases (shuffled / missing / extra items). This seems like a training-level prior worth fixing — the current prior treats LLM callees like deterministic services.
Reproducibility
Pattern-level: recurred even after the lesson was written into project instructions Claude reads every session.
Model / Version / Platform
Opus (most sessions; some Sonnet) · Claude Code 2.1.235 · Anthropic API · macOS
Impact
High — one incident published mismatched content to a real third-party platform (instantly removed by moderators); another halted all publishing site-wide.