Skip to content

Say "low confidence" in the report, not just in the number - #397

Merged
PabloCodes7 merged 1 commit into
mainfrom
feat/low-confidence-framing
Jul 22, 2026
Merged

PabloCodes7 merged 1 commit into
mainfrom
feat/low-confidence-framing

Conversation

@PabloCodes7

Copy link
Copy Markdown
Contributor

v0.7.4 stopped a class carried by nothing but tool invocations from claiming diagnosis-level confidence — but the reports still printed it in the exact shape of a confident verdict:

Root cause: ruby_bundle_failure
Confidence: 0.3
Subsystem: Ruby dependency installation or test lifecycle

A maintainer reads "root cause" and goes to debug a Gemfile that is fine. The number dropped; the message did not.

What changes

Both human-facing renderers (--format text and --format markdown) add an explicit caveat when the confidence falls under LOW_CONFIDENCE_THRESHOLD (0.35 — one constant next to the existing report constants): that the verdict is a hint rather than a proven cause, the honest mechanism behind it (the signals only prove a tool ran, not that it failed), and the next useful step — the raw log, plus the existing CI failure fixture issue template if the real cause is something no class covers.

After, on rails/rails run 29648807728 (--format markdown):

Low confidence (0.3). Treat this as a hint, not a proven cause: the signals that carried this class only show that a tool ran, not that it failed, so the real failure may be something PatchRail does not recognize yet.

Read the raw log before changing anything — and if the cause is clear there, open a CI failure fixture issue with a sanitized log so PatchRail learns it.

Scope

  • The guard keys on confidence alone, never on a class or an ecosystem: 0.35 sits between the invocation-only cap (0.3) and the lowest confidence any rule that actually watched something fail can earn (0.53).
  • unknown declines at 0.15 and already has its own "help improve PatchRail" message, so it is excluded rather than told the same thing twice.
  • Presentation only. No change to classify.py, to the class, the confidence, the exit code, or the --format json payload the GitHub Action consumes. A high-confidence report is byte-identical to before.

Verification

  • New tests: text and markdown over the committed tests/data/realworld/rails-29648807728.log (the real 0.3 case), the JSON contract asserted clean, and springboot-29780604983.log asserted to keep its 0.89 verdict unqualified in both renderers.
  • ci benchmark examples/ci-triage → 223/223, top-1 1.0 (unchanged).
  • Full suite 1008 passed on Python 3.12; ruff check and ruff format --check clean.

0.7.4 capped a verdict carried by nothing but tool invocations at 0.3, but
`_render_text` and `_render_markdown` still printed it in the exact shape of a
0.95: `Root cause: ruby_bundle_failure` / `Confidence: 0.3`. A maintainer reads
"root cause" and goes to debug a Gemfile that is fine. The number told the
truth; the message did not.

Both human-facing renderers now add a caveat when the confidence falls under
`LOW_CONFIDENCE_THRESHOLD` (0.35, one constant next to the existing report
constants): that the verdict is a hint rather than a proven cause, the honest
mechanism behind it (the signals only prove a tool RAN, not that it failed),
and the next useful step -- the raw log, and the existing CI failure fixture
issue template if the real cause is something no class covers.

The guard is on the confidence, never on a class or an ecosystem: 0.35 sits
between the invocation-only cap (0.3) and the lowest confidence any rule that
actually watched something fail can earn (0.53). `unknown` declines at 0.15 and
already has its own message, so it is excluded rather than told twice.

Presentation only. Classification is untouched -- no change to `classify.py`,
to the class, the confidence, the exit code, or the `--format json` payload the
Action consumes -- and a high-confidence report is byte-identical to before.

Tests: text and markdown reports over the committed rails/rails 29648807728
log (the real 0.3 case), the JSON contract asserted clean, and spring-boot
29780604983 asserted to keep its 0.89 verdict unqualified in both renderers.
Benchmark still 223/223 top-1 1.0; suite 1008 passed on 3.12; ruff clean.
@PabloCodes7
PabloCodes7 merged commit b66fb4c into main Jul 22, 2026
5 checks passed
@PabloCodes7
PabloCodes7 deleted the feat/low-confidence-framing branch July 22, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant