Before submitting
Area
Indexing or file inventory
Current limitation
A warm, unchanged repository can still incur full directory discovery and content hashing during background reconciliation. Native watcher admission falls back to a 30-second periodic scan when the directory limit is exceeded; native mode also has a five-minute safety reconciliation. The resource cost depends on the walked tree, including directories excluded from indexing.
A live older-version host snapshot showed one native server near 103% CPU and approximately 19,628 read syscalls/s with no inotify descriptor. Client traffic was not captured, so that snapshot does not prove the process was idle or that polling explains all of its CPU.
Use cases
A repository with large generated/dependency directory trees and little admitted source, or many simultaneous MCP sessions whose background cost needs an attributable bound.
Controlled current-release reproduction: use mcp_multiprocess_profile with --max-index-workers 1 --process-counts 1 --files 200 --functions-per-file 40 --warm-iterations 10 --idle-seconds 5 --polling-directories 50001 --polling-observation-seconds 31 --timeout-seconds 30 --output <report.json>. Set NO_COLOR=1 for this measurement because the existing diagnostics parser otherwise fails (tracked separately).
The dedicated polling fixture contains 50,001 empty directories and one source file, with no queries or source changes during observation.
Desired outcome and success criteria
Establish a reproducible resource envelope for warm, unchanged large trees before selecting an optimization:
- Attribute wall/CPU time and bytes separately to watcher admission, discovery, hashing, preparation, and publication.
- Record backend/fallback reason, walked and admitted populations, reconciliation count, CPU, RSS/PSS, and readiness/freshness.
- Compare the same pinned tree and fixed query/event schedule before and after any candidate.
- Demonstrate lower background CPU without weaker eventual reconciliation, content identity, coverage, or bounded memory.
- Document any accepted scan-duty or freshness tradeoff explicitly rather than silently stretching polling.
Invariants and constraints
Preserve the 50,000-directory native-watch bound, bounded discovery/preparation, atomic generations, cross-process single-leader ownership, cancellation, and detection of same-size/same-mtime content changes. An unchanged generation is not proof of zero discovery/hash work.
Alternatives considered
Immediately lower workers or increase the polling interval: neither identifies the phase causing high CPU, and the latter changes freshness. Ignore timestamps-only changes or trust mtime as content identity: weakens correctness. Increase the watch cap: can transfer the cost to kernel memory. Add a file-body cache: current read measurements do not justify that memory ownership.
Possible direction
First profile an approved pinned snapshot of an affected large tree. Then evaluate admission that respects the effective indexing scope, bounded reconciliation scheduling, or safe reuse of unchanged-work evidence. Keep these as hypotheses until paired measurements pass.
References
Controlled result: admission_directory_limit, ready-time extra full scans = 0; over 31 seconds, one poll and one full reconciliation consumed 1.76 CPU seconds (about 5.68% of one core). Native idle controls on the small corpus consumed 0–0.01 aggregate CPU seconds per five-second window.
On the pinned LeanToken corpus, full no-op reconciliation p95 was 100.52 ms; it preserved the generation but still walked/hashed files. These are descriptive measurements on a shared host, not a performance-promotion result.
Owners: src/watcher/runtime.rs, src/watcher/discovery.rs, src/repository/discovery.rs, src/indexer/orchestrator.rs.
Environment: Linux x86_64 (kernel 6.1), Rust 1.95.0; disposable container limited to 2 CPUs and 6 GiB, separate HOME/cache, no network. The host MCP version and configuration were unchanged.
Revision: 0.1.28, revision 05ea6a420268c68a52cce6ca74b8e97c14bf91fc; locked release build, without cfg(test).
Related: #512 and #515 fixed different watch-bound/freshness contracts; preserve those fixes.
Before submitting
Area
Indexing or file inventory
Current limitation
A warm, unchanged repository can still incur full directory discovery and content hashing during background reconciliation. Native watcher admission falls back to a 30-second periodic scan when the directory limit is exceeded; native mode also has a five-minute safety reconciliation. The resource cost depends on the walked tree, including directories excluded from indexing.
A live older-version host snapshot showed one native server near 103% CPU and approximately 19,628 read syscalls/s with no inotify descriptor. Client traffic was not captured, so that snapshot does not prove the process was idle or that polling explains all of its CPU.
Use cases
A repository with large generated/dependency directory trees and little admitted source, or many simultaneous MCP sessions whose background cost needs an attributable bound.
Controlled current-release reproduction: use
mcp_multiprocess_profilewith--max-index-workers 1 --process-counts 1 --files 200 --functions-per-file 40 --warm-iterations 10 --idle-seconds 5 --polling-directories 50001 --polling-observation-seconds 31 --timeout-seconds 30 --output <report.json>. SetNO_COLOR=1for this measurement because the existing diagnostics parser otherwise fails (tracked separately).The dedicated polling fixture contains 50,001 empty directories and one source file, with no queries or source changes during observation.
Desired outcome and success criteria
Establish a reproducible resource envelope for warm, unchanged large trees before selecting an optimization:
Invariants and constraints
Preserve the 50,000-directory native-watch bound, bounded discovery/preparation, atomic generations, cross-process single-leader ownership, cancellation, and detection of same-size/same-mtime content changes. An unchanged generation is not proof of zero discovery/hash work.
Alternatives considered
Immediately lower workers or increase the polling interval: neither identifies the phase causing high CPU, and the latter changes freshness. Ignore timestamps-only changes or trust mtime as content identity: weakens correctness. Increase the watch cap: can transfer the cost to kernel memory. Add a file-body cache: current read measurements do not justify that memory ownership.
Possible direction
First profile an approved pinned snapshot of an affected large tree. Then evaluate admission that respects the effective indexing scope, bounded reconciliation scheduling, or safe reuse of unchanged-work evidence. Keep these as hypotheses until paired measurements pass.
References
Controlled result:
admission_directory_limit, ready-time extra full scans = 0; over 31 seconds, one poll and one full reconciliation consumed 1.76 CPU seconds (about 5.68% of one core). Native idle controls on the small corpus consumed 0–0.01 aggregate CPU seconds per five-second window.On the pinned LeanToken corpus, full no-op reconciliation p95 was 100.52 ms; it preserved the generation but still walked/hashed files. These are descriptive measurements on a shared host, not a performance-promotion result.
Owners: src/watcher/runtime.rs, src/watcher/discovery.rs, src/repository/discovery.rs, src/indexer/orchestrator.rs.
Environment: Linux x86_64 (kernel 6.1), Rust 1.95.0; disposable container limited to 2 CPUs and 6 GiB, separate HOME/cache, no network. The host MCP version and configuration were unchanged.
Revision: 0.1.28, revision
05ea6a420268c68a52cce6ca74b8e97c14bf91fc; locked release build, withoutcfg(test).Related: #512 and #515 fixed different watch-bound/freshness contracts; preserve those fixes.