Repository navigation
Say "low confidence" in the report, not just in the number - #397
Merged
Merged
Conversation
0.7.4 capped a verdict carried by nothing but tool invocations at 0.3, but `_render_text` and `_render_markdown` still printed it in the exact shape of a 0.95: `Root cause: ruby_bundle_failure` / `Confidence: 0.3`. A maintainer reads "root cause" and goes to debug a Gemfile that is fine. The number told the truth; the message did not. Both human-facing renderers now add a caveat when the confidence falls under `LOW_CONFIDENCE_THRESHOLD` (0.35, one constant next to the existing report constants): that the verdict is a hint rather than a proven cause, the honest mechanism behind it (the signals only prove a tool RAN, not that it failed), and the next useful step -- the raw log, and the existing CI failure fixture issue template if the real cause is something no class covers. The guard is on the confidence, never on a class or an ecosystem: 0.35 sits between the invocation-only cap (0.3) and the lowest confidence any rule that actually watched something fail can earn (0.53). `unknown` declines at 0.15 and already has its own message, so it is excluded rather than told twice. Presentation only. Classification is untouched -- no change to `classify.py`, to the class, the confidence, the exit code, or the `--format json` payload the Action consumes -- and a high-confidence report is byte-identical to before. Tests: text and markdown reports over the committed rails/rails 29648807728 log (the real 0.3 case), the JSON contract asserted clean, and spring-boot 29780604983 asserted to keep its 0.89 verdict unqualified in both renderers. Benchmark still 223/223 top-1 1.0; suite 1008 passed on 3.12; ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.7.4 stopped a class carried by nothing but tool invocations from claiming diagnosis-level confidence — but the reports still printed it in the exact shape of a confident verdict:
A maintainer reads "root cause" and goes to debug a Gemfile that is fine. The number dropped; the message did not.
What changes
Both human-facing renderers (
--format textand--format markdown) add an explicit caveat when the confidence falls underLOW_CONFIDENCE_THRESHOLD(0.35 — one constant next to the existing report constants): that the verdict is a hint rather than a proven cause, the honest mechanism behind it (the signals only prove a tool ran, not that it failed), and the next useful step — the raw log, plus the existing CI failure fixture issue template if the real cause is something no class covers.After, on rails/rails run 29648807728 (
--format markdown):Scope
unknowndeclines at 0.15 and already has its own "help improve PatchRail" message, so it is excluded rather than told the same thing twice.classify.py, to the class, the confidence, the exit code, or the--format jsonpayload the GitHub Action consumes. A high-confidence report is byte-identical to before.Verification
tests/data/realworld/rails-29648807728.log(the real 0.3 case), the JSON contract asserted clean, andspringboot-29780604983.logasserted to keep its 0.89 verdict unqualified in both renderers.ci benchmark examples/ci-triage→ 223/223, top-1 1.0 (unchanged).ruff checkandruff format --checkclean.