Repository navigation
refactor(telemetry): track btw side question as tool_call event - #2257
Conversation
…duration - replace simple input_btw event with structured tool_call telemetry\n- record outcome, duration_ms, error_type for success and all error paths (LLMNotSet, ToolCallDenied, NoResponse, exceptions)\n- add telemetry tests covering each outcome branch
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 693fd0104e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| except Exception as e: | ||
| _error_type = type(e).__name__ | ||
| logger.warning("Side question failed: {error}", error=e) | ||
| return None, str(e) | ||
| finally: |
There was a problem hiding this comment.
Treat cancelled /btw runs as cancellation, not error
When the user dismisses the /btw modal before the model replies, _run_btw_modal cancels the side-question task; this raises asyncio.CancelledError, which is not caught by except Exception here. The finally block then still emits a tool_call event with outcome="error" and no error_type, so user-initiated cancellations are counted as failures and skew tool-call error-rate/error-type telemetry. Please handle CancelledError explicitly (or skip emitting failure telemetry for that path).
Useful? React with 👍 / 👎.
Related Issue
N/A
Description
Replace the lightweight
input_btwtelemetry event with a structuredtool_callevent so the/btwside-question flow shows up alongside the rest of the tool-call observability (outcome, duration, error breakdown) instead of just a fire-and-forget input counter.1. Structured telemetry for btw side questions
Problem:
/btwpreviously emitted a singleinput_btwevent at the Shell layer, which only told us that the user typed/btw …— not whether the side question actually returned an answer, how long it took, or why it failed. That made it impossible to spot regressions like LLM-not-set, model attempting tool calls instead of replying, empty responses, or transport exceptions.What was done:
track("input_btw")call fromShell._main_loop(it was only counting input, not outcomes).execute_side_questionin a try/finally that emitstool_callwithtool_name=btw,outcome,duration_ms,dup_type=normal, anderror_typeon failure.error_typevalues:LLMNotSet,ToolCallDenied(LLM tried to call tools instead of answering on both turns),NoResponse, and the exception class name for uncaught failures.2. Tests
Problem: Telemetry regressions are silent — without explicit tests, future refactors to
execute_side_questioncould easily drop thetrackcall or mis-label an outcome and no test would catch it.What was done:
TestExecuteSideQuestionTelemetryintests/ui_and_conv/test_btw.pywith one test per outcome branch: success,LLMNotSet,ToolCallDenied,NoResponse, and an exception path.tool_name,outcome,error_typepresence/absence, and thatduration_msis recorded.Checklist
make gen-changelogto update the changelog.make gen-docsto update the user documentation.