Skip to content
This repository was archived by the owner on Sep 23, 2026. It is now read-only.

refactor(telemetry): track btw side question as tool_call event - #2257

Merged
RealKai42 merged 1 commit into
MoonshotAI:mainfrom
jackfish212:btw-fix
May 13, 2026
Merged

RealKai42 merged 1 commit into
MoonshotAI:mainfrom
jackfish212:btw-fix

Conversation

@jackfish212

@jackfish212 jackfish212 commented May 13, 2026 •

Copy link
Copy Markdown
Collaborator

Related Issue

N/A

Description

Replace the lightweight input_btw telemetry event with a structured tool_call event so the /btw side-question flow shows up alongside the rest of the tool-call observability (outcome, duration, error breakdown) instead of just a fire-and-forget input counter.


1. Structured telemetry for btw side questions

Problem: /btw previously emitted a single input_btw event at the Shell layer, which only told us that the user typed /btw … — not whether the side question actually returned an answer, how long it took, or why it failed. That made it impossible to spot regressions like LLM-not-set, model attempting tool calls instead of replying, empty responses, or transport exceptions.

What was done:

  • Remove the track("input_btw") call from Shell._main_loop (it was only counting input, not outcomes).
  • Wrap execute_side_question in a try/finally that emits tool_call with tool_name=btw, outcome, duration_ms, dup_type=normal, and error_type on failure.
  • Cover all error paths with distinct error_type values: LLMNotSet, ToolCallDenied (LLM tried to call tools instead of answering on both turns), NoResponse, and the exception class name for uncaught failures.

2. Tests

Problem: Telemetry regressions are silent — without explicit tests, future refactors to execute_side_question could easily drop the track call or mis-label an outcome and no test would catch it.

What was done:

  • Add TestExecuteSideQuestionTelemetry in tests/ui_and_conv/test_btw.py with one test per outcome branch: success, LLMNotSet, ToolCallDenied, NoResponse, and an exception path.
  • Each test asserts the event name, tool_name, outcome, error_type presence/absence, and that duration_ms is recorded.

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked the related issue, if any.
  • I have added tests that prove my fix is effective or that my feature works.
  • I have run make gen-changelog to update the changelog.
  • I have run make gen-docs to update the user documentation.

Open in Devin Review

…duration

- replace simple input_btw event with structured tool_call telemetry\n- record outcome, duration_ms, error_type for success and all error paths (LLMNotSet, ToolCallDenied, NoResponse, exceptions)\n- add telemetry tests covering each outcome branch

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 693fd0104e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/kimi_cli/soul/btw.py
Comment on lines 189 to +193
except Exception as e:
_error_type = type(e).__name__
logger.warning("Side question failed: {error}", error=e)
return None, str(e)
finally:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat cancelled /btw runs as cancellation, not error

When the user dismisses the /btw modal before the model replies, _run_btw_modal cancels the side-question task; this raises asyncio.CancelledError, which is not caught by except Exception here. The finally block then still emits a tool_call event with outcome="error" and no error_type, so user-initiated cancellations are counted as failures and skew tool-call error-rate/error-type telemetry. Please handle CancelledError explicitly (or skip emitting failure telemetry for that path).

Useful? React with 👍 / 👎.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@RealKai42
RealKai42 merged commit 6c0b949 into MoonshotAI:main May 13, 2026
18 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants