Skip to content

Commit 6067cda

Browse files
hippoleyhippoley
authored
feat: publish Claude oracle-disagreement flagship (#124)
Publish a reviewer-supplied external-oracle contradiction artifact for anthropics/claude-code#89404, bind it into the claim matrix, Reality Lab, README flagship narrative, and regression tests without claiming an independent Claude runtime replay.
1 parent e9a420f commit 6067cda

6 files changed

Lines changed: 177 additions & 3 deletions

File tree

‎README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,38 @@ The same QEMU artifacts now feed three evidence layers: the Bluefin-specific con
5656

5757
---
5858

59+
60+
## Flagship boundary: green judge, contradicted product oracle
61+
62+
A second flagship case shows the opposite failure mode: **green submitted evidence can still be wrong about product truth**.
63+
64+
In [anthropics/claude-code#89404](https://github.com/anthropics/claude-code/pull/89404), the submitted validator suite reported 5/5 passing. Reviewer-supplied measurements showed two stronger facts:
65+
66+
```text
67+
submitted judge 5/5 PASS
68+
multi-line regression suite stays green after extraction revert
69+
product oracle claude plugin validate --json → REJECT
70+
oracle alignment CONTRADICTED
71+
```
72+
73+
CounterProof publishes this as an `ORACLE_CONTRADICTED_BY_PRODUCT` artifact. It is intentionally marked as **reviewer-supplied external-oracle evidence**; CounterProof does not claim it independently executed the Claude Code binary.
74+
75+
This case complements Bluefin:
76+
77+
```text
78+
Bluefin
79+
same oracle + controlled intervention + recovery
80+
→ positive causal witness
81+
82+
Claude #89404
83+
green submitted judge + rejecting product oracle
84+
→ negative proof boundary
85+
```
86+
87+
**[Read the oracle-disagreement artifact →](reality/claude-code-89404/ORACLE_DISAGREEMENT.md)** · **[Read the machine receipt →](examples/claim_matrix/receipts/claude-code-89404-oracle.json)**
88+
89+
---
90+
5991
## Tested on real agent PRs
6092

6193
CounterProof is not developed only against fixtures. New proof semantics are tested against public AI-assisted pull requests where a reviewer has a concrete reason not to trust a green check.

‎docs/REALITY_LAB.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ The rule is simple:
1414
| [openai/codex-plugin-cc#731](https://github.com/openai/codex-plugin-cc/pull/731) | Do the changed tests prove the original fail-open bug, and do they also answer later reviewer concerns? | **WITNESSED + proof boundary** | The tests prove the original bug, but not later 0600/atomic-write concerns. Green evidence must not silently expand its claim. |
1515
| [openai/codex-plugin-cc#456](https://github.com/openai/codex-plugin-cc/pull/456) | What if the thing being fixed is the test harness/environment isolation itself? | **NOT WITNESSED + REVIEW REQUIRED** | Direct BASE/HEAD reproduction supports the bug, but ordinary witness is withheld because changed test support is part of the fix. |
1616
| [MetrolistGroup/Metrolist#4097](https://github.com/MetrolistGroup/Metrolist/pull/4097) | Does a BASE non-zero exit always count as regression evidence? | **INCONCLUSIVE** | No. The BASE failed during Kotlin test compilation, not at a behavioral assertion. This produced the structured `json-v1` result protocol. |
17-
| [anthropics/claude-code#89404](https://github.com/anthropics/claude-code/pull/89404) | What if the submitted tests are internally green but disagree with the product's authoritative parser? | **Open Reality Probe** | This is the emerging oracle-alignment problem: red→green evidence can still be weak if the judge does not represent product truth. |
17+
| [anthropics/claude-code#89404](https://github.com/anthropics/claude-code/pull/89404) | What if the submitted tests are internally green but disagree with the product's authoritative parser? | **[ORACLE CONTRADICTED](../reality/claude-code-89404/ORACLE_DISAGREEMENT.md)** | Reviewer-supplied product-oracle evidence shows `claude plugin validate --json` rejects fixtures the submitted validator accepts, while the submitted suite can remain 5/5 green even when the multi-line extraction behavior regresses. CounterProof publishes this as an external-oracle contradiction, not as an independently executed Claude runtime replay. |
1818
| [continuedev/continue#12576](https://github.com/continuedev/continue/pull/12576) | Can CounterProof replay a package-local Vitest regression inside a large JS/TS monorepo? | **[WITNESSED](../reality/continue-12576/WITNESS.md)** | The real PR exposed two product gaps first: `*.vitest.ts` discovery and package-local `core/node_modules` closure. After both fixes, HEAD was 5/5 PASS and BASE was 4/5 PASS, 1 FAIL; the BASE failure directly hits the legacy `tabAutocompleteModel` migration claim. |
1919
| [topoteretes/cognee#5161](https://github.com/topoteretes/cognee/pull/5161) | What if local safety tests pass but a human reviewer reproduces a broader end-to-end failure mode those tests never exercise? | **Human oracle CONTRADICTED** | The vector adapter refusal test is narrow; reviewer evidence shows graph deletion can happen before the refusal propagates. CounterProof records the human reproduction as an external oracle instead of pretending it ran PostgreSQL itself. |
2020
| [vercel/ai#17096](https://github.com/vercel/ai/pull/17096) | What if two newly added tests have different evidentiary value, while a stronger production claim lives outside the test suite? | **[WITNESSED + NOT WITNESSED + UNVERIFIED](../reality/vercel-ai-17096/CLAIM_MATRIX.md)** | Host→bridge sandbox forwarding is red→green; the new schema test passes on BASE and HEAD; real Claude Agent SDK enforcement remains an external production oracle. |

‎examples/claim_matrix/claude-code-89404.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,11 @@ claims:
77
submitted_test_evidence: UNPROVEN
88
oracle_alignment: CONTRADICTED
99
oracle_probe: "claude plugin validate rejects fixtures that the submitted validator treats as valid"
10-
oracle_source_url: "https://github.com/anthropics/claude-code/pull/89404"
11-
note: "Reviewer evidence establishes product-oracle contradiction; CounterProof has not independently replayed BASE/HEAD for this row."
10+
oracle_source_url: "https://github.com/anthropics/claude-code/pull/89404#pullrequestreview-5190939241"
11+
receipt_file: "receipts/claude-code-89404-oracle.json"
12+
receipt_expected_verdicts:
13+
- ORACLE_CONTRADICTED_BY_PRODUCT
14+
note: "Reviewer evidence establishes product-oracle contradiction; CounterProof has not independently executed the Claude binary for this row."
1215
- id: multiline-description-regression
1316
claim: "The submitted suite protects the multi-line description extraction behavior"
1417
submitted_test_evidence: UNPROVEN

‎examples/claim_matrix/receipts/claude-code-89404-oracle.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"schema_version": 1,
3+
"case": "anthropics/claude-code#89404",
4+
"verdict": "ORACLE_CONTRADICTED_BY_PRODUCT",
5+
"evidence_kind": "REVIEWER_SUPPLIED_EXTERNAL_ORACLE",
6+
"source_pr": "https://github.com/anthropics/claude-code/pull/89404",
7+
"candidate": {
8+
"base_sha": "8b6ef81f636a7697e5ae2338428fa0b272993845",
9+
"head_sha": "0989f29b0bca412edc58018c462cae7344bb1a2f"
10+
},
11+
"submitted_judge": {
12+
"test": "plugins/plugin-dev/skills/agent-development/scripts/validate-agent.test.sh",
13+
"reported_result": "5/5 PASS",
14+
"limitation": "Reviewer reproduced that reverting the multi-line description extraction still leaves the suite green while the false <example> warning returns."
15+
},
16+
"authoritative_oracle": {
17+
"command": "claude plugin validate --json",
18+
"reported_versions": [
19+
"2.1.268",
20+
"2.1.270"
21+
],
22+
"result": "REJECT",
23+
"reason": "The product parser rejects fixtures that the submitted validator test treats as valid.",
24+
"review_source": "https://github.com/anthropics/claude-code/pull/89404#pullrequestreview-5190939241",
25+
"inline_source": "https://github.com/anthropics/claude-code/pull/89404#discussion_r3999827406",
26+
"followup_source": "https://github.com/anthropics/claude-code/pull/89404#issuecomment-5555827628"
27+
},
28+
"conclusion": "The submitted judge is not aligned with the product parser for the claimed-valid fixtures. A green submitted suite must not be promoted to product-level proof.",
29+
"does_not_claim": [
30+
"CounterProof independently executed the Claude Code binary.",
31+
"The entire pull request is invalid.",
32+
"Every submitted regression assertion is false."
33+
]
34+
}

‎reality/claude-code-89404/ORACLE_DISAGREEMENT.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Claude Code #89404 — oracle disagreement proof
2+
3+
CounterProof uses this case as a flagship **negative proof boundary**.
4+
5+
The submitted regression suite reports green, but reviewer-supplied evidence shows
6+
that the product's own parser rejects fixtures the submitted validator treats as
7+
valid.
8+
9+
## Candidate identity
10+
11+
- BASE: `8b6ef81f636a7697e5ae2338428fa0b272993845`
12+
- HEAD: `0989f29b0bca412edc58018c462cae7344bb1a2f`
13+
14+
## Submitted judge
15+
16+
`validate-agent.test.sh` reports 5/5 passing.
17+
18+
A reviewer also reproduced that reverting the multi-line description extraction
19+
back to the original single-line form still leaves that suite green while the false
20+
`<example>` warning returns.
21+
22+
That means the submitted judge does not actually guard the claimed multi-line
23+
description behavior.
24+
25+
## Authoritative oracle
26+
27+
Reviewer measurements used the product command:
28+
29+
```text
30+
claude plugin validate --json
31+
```
32+
33+
on Claude Code 2.1.268 and 2.1.270.
34+
35+
The product parser rejected the agent fixtures that the submitted validator test
36+
treated as valid.
37+
38+
Primary external evidence:
39+
40+
- review: https://github.com/anthropics/claude-code/pull/89404#pullrequestreview-5190939241
41+
- inline parser finding: https://github.com/anthropics/claude-code/pull/89404#discussion_r3999827406
42+
- reviewer agreement on product-parser alignment: https://github.com/anthropics/claude-code/pull/89404#issuecomment-5555827628
43+
44+
## CounterProof conclusion
45+
46+
```text
47+
submitted judge GREEN
48+
authoritative oracle REJECT
49+
oracle alignment CONTRADICTED
50+
product-level claim CONTRADICTED
51+
```
52+
53+
This artifact is intentionally labeled `REVIEWER_SUPPLIED_EXTERNAL_ORACLE`.
54+
55+
CounterProof did **not** independently execute the Claude Code binary for this
56+
publication artifact. The proof is therefore about the documented oracle
57+
contradiction and its provenance, not a claim that CounterProof reproduced the
58+
product runtime itself.
59+
60+
## Why this belongs beside Bluefin
61+
62+
Bluefin demonstrates the positive case:
63+
64+
```text
65+
same oracle
66+
CONTROL == REVERT != BAD
67+
→ causal witness
68+
```
69+
70+
Claude #89404 demonstrates the equally important negative case:
71+
72+
```text
73+
submitted judge green
74+
product oracle rejects
75+
→ do not promote green evidence into product truth
76+
```
77+
78+
Together they define the product boundary more clearly than either example alone.

‎tests/unit/test_claim_matrix.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -598,3 +598,30 @@ def test_scancode_example_binds_db_backed_behavior_receipt():
598598
assert payload["claims"][0]["oracle_applicability"] == "APPLICABLE"
599599
assert payload["claims"][0]["oracle_alignment"] == "ALIGNED"
600600
assert payload["claims"][0]["overall_claim"] == "PROVEN"
601+
602+
603+
def test_claude_oracle_disagreement_binds_external_oracle_receipt():
604+
from skill_factory.evolution.claim_matrix import load_claim_matrix
605+
606+
manifest = load_claim_matrix(Path("examples/claim_matrix/claude-code-89404.yml"))
607+
payload = claim_matrix_to_dict(manifest)
608+
609+
first = payload["claims"][0]
610+
assert first["overall_claim"] == "CONTRADICTED"
611+
assert first["oracle_alignment"] == "CONTRADICTED"
612+
assert first["receipt_observed_verdict"] == "ORACLE_CONTRADICTED_BY_PRODUCT"
613+
assert first["receipt_case"] == "anthropics/claude-code#89404"
614+
615+
receipt_path = Path(
616+
"examples/claim_matrix/receipts/claude-code-89404-oracle.json"
617+
)
618+
receipt = json.loads(receipt_path.read_text(encoding="utf-8"))
619+
assert receipt["evidence_kind"] == "REVIEWER_SUPPLIED_EXTERNAL_ORACLE"
620+
assert receipt["candidate"]["head_sha"] == (
621+
"0989f29b0bca412edc58018c462cae7344bb1a2f"
622+
)
623+
assert receipt["authoritative_oracle"]["command"] == (
624+
"claude plugin validate --json"
625+
)
626+
assert receipt["authoritative_oracle"]["result"] == "REJECT"
627+
assert receipt["does_not_claim"]

0 commit comments

Comments
 (0)