Bring me an agent PR you don't trust #17

CounterProof is doing public Reality Probes on real AI-assisted pull requests.

If you are reviewing a PR and your reaction is something like:

• “the agent says tests passed, but I do not know if they would fail before the fix”
• “the test is green, but the build/test harness changed too”
• “CI is green, but the production/runtime oracle disagrees”
• “this only reproduces under a specific environment”
• “the suite passed, but I am not sure it covers the reviewer concern”
• “the PR claims live-provider behavior that the unit test does not actually establish”

drop the PR here.

What I will do

For a good candidate, I will try to produce one of these — and publish the evidence:

1. Regression Witness
   same changed test: HEAD passes / BASE fails.

2. NOT WITNESSED
   same test passes on both sides, so the submitted regression does not distinguish the fix.

3. INCONCLUSIVE
   compile/setup/adapter/infrastructure failure prevents a behavioral claim.

4. Proof Integrity finding
   the PR changes CI, test support, fixtures, or other evidence-producing machinery.

5. Proof-boundary note
   the submitted test proves one claim, while the reviewer concern is about another.

I will not use a green receipt to say “merge this PR.” The goal is narrower: make the evidence boundary visible.

Already tested on real PRs

• rundef/async_rithmic#53 — genuine before/after regression witness.
• openai/codex-plugin-cc#731 — genuine witness for the original bug, but not for the later permissions/atomicity review concerns.
• openai/codex-plugin-cc#456 — direct environment reproduction succeeds, but ordinary Regression Witness is withheld because the test harness itself changes.
• anthropics/claude-code#89404 — reviewer evidence shows a deeper problem: the submitted test oracle can disagree with the product's authoritative parser.

Best submission

Just paste:

PR:
What claim do you not trust?
What would convince you?

No need to install CounterProof first.

If the PR is not suitable for deterministic replay, I will say so instead of forcing it into the tool.

One extra question we learned to ask

A real external reviewer asked CounterProof to separate a red→green submitted test from the product's authoritative oracle.

So when you submit a case, include this if you know it:

Authoritative oracle:
What real product/runtime/provider behavior should the submitted test agree with?

Examples: the product's own parser, a live provider API, browser-visible behavior, or “no independent oracle known yet.”