Repository navigation
fix(core): make usage statistics fast and responsive - #47527
kitlangton wants to merge 2 commits into
Conversation
|
Real-world data point for this PR. On v2.0.19, all-time stats on my machine takes ~50s through the API, long enough that a client with a 30s read timeout (OpenChamber's stats page) reports a transport failure. I tested the index from this PR's migration on a backup copy of my real DB: 23.7 GB, 315k messages, 288k assistant steps, 4.7 GB of user/assistant message JSON.
Row count, cost, and token sums were identical before and after. The index was 33 MB and took 22s to build once. Warm cache didn't help the baseline: CPU time was ~3s, and the rest was reading message bodies from disk. Would be great to see this land. |
|
Automated PR Cleanup Thank you for contributing to opencode. Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions. This PR was closed because it matched the following cleanup criteria:
PRs created within the last month are not affected by this cleanup. If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate. Thanks again for taking the time to contribute. |
Why
/statscan sit on “Gathering your stats…” for tens of seconds on a large history. The request reads large message JSON bodies to extract a few usage fields, and its synchronous SQL/aggregation loop can also stall unrelated server work.A live request took 20.78 s. On a fixed private snapshot containing roughly 847,000 assistant steps, the original implementation took 54.93 s median. Those are separate measurements, not an end-to-end before/after pair.
What changes
The final nine runs retained the baseline response digest, with a maximum observed event-loop timer delay of 28.5 ms. Calendar batches are not a hard row-count or latency bound. The historical snapshot benchmark used base
97303c39dd; the matched video below uses the current PR base.Demo
stats-before-after.mp4
Real OpenCode TUI, recorded with OpenCode Drive 2.1.0 and Bun 1.4.2. Left:
7a4ad68af6, 2.42 s until visible; right:e94f3b7e24, 0.21 s. Both use independent copies of the same synthetic fixture: 80,000 assistant steps with 64 KiB bodies, across 100 sessions. No private conversation data, simulated provider response, or injected latency. The index migration completes before recording begins.Playback is real-time and aligned on opening
/stats. The completed after frame is held to match the before clip's duration. Native 1600×680 side-by-side export at 60 fps, inspected at the loading and completed checkpoints. The recorded revision has the same production fix; benchmark and recording tooling have since been removed from the PR.Index cost
The large snapshot's index adds 91.9 MiB and took 69.6 s to build once. SQLite maintains it as messages change. In an in-memory benchmark using the production content projector, incremental update cost was about 0.077 ms for 64 KiB and 1.15 ms for 1 MiB bodies. These are projection measurements, not a full disk-durability or streaming benchmark.
Scope
This PR contains the core stats optimization, its index migration and generated schema, and regression coverage. It optimizes the TUI's year-to-date, all-projects,
tools: "none"request. Benchmark harnesses, synthetic fixture generation, recording scripts, and the experiment log are retained locally.Verification
20 focused tests pass, including the actual covering-query plan, large bodies, indexed updates/deletes, project/fork/subagent filtering, compaction usage, exclusive date bounds, DST changes, and a quarter-hour timezone. Formatting, file-scoped lint, core typechecking, and migration validation pass. Focused tests and core checks were repeated with the required Bun 1.4.2 toolchain. The enabled pre-push hook also passed all 33 workspace typecheck tasks.