test(managed-agent): add FG6e hosted SSE gap gates - #12942

[Suggestion] R1-1: privateJson() never checks the proxyFailure latch that its twin evidence() checks as its first statement (line 69), and the top-level catch at line 446 never logs it, so a broker-proxy failure surfaces as an unrelated FIFO timeout with the real cause discarded.

The proxy's catch stores the cause in proxyFailure and destroys the response when the upstream status assert fires or the 30s upstream timeout trips. The CLI's broker call then fails, so the Edit tool never reaches its FIFO read — and control is sitting in the FIFO-open waitUntil at lines 288-296, which passes no timeout (30s default), carries no proxyFailure guard, and swallows ENXIO as it polls. It throws Timed out after 30000ms; the first guarded call, evidence('entered') at line 302, is never reached. The catch prints only cli.output and operations, so the broker's status and body never reach the log that HostedWorkspaceToolTurnIT.java:236 surfaces via as("Driver output: %s", Files.readString(log)), and 30s of the 130s driver budget is burned first. The same unguarded shape sits at lines 314-317 (the waitUntil on hasActivePrompt), which hosted-broker-reply-loss-driver.ts:203-206 does guard.

Witness:

broker-reply-loss (FG6a) HELPER L48  ASSERT L55  GUARD none-in-helper  GUARD L204 (inside the hasActivePrompt waitUntil)  catch L335 no proxyFailure
store-failure    (FG6b) HELPER L228 ASSERT L239 GUARD L229 + L240 (both sides)  end-of-try L414  catch L418 no proxyFailure
process-crash    (FG6c) HELPER L250 ASSERT L261 GUARD L251 + L262 (both sides) + L325 + L345  end-of-try L417  catch L424 no proxyFailure
cancellation     (FG6d) HELPER L55  ASSERT L63  GUARD L56 + L64 (both sides)  catch L408 no proxyFailure
sse-gap          (FG6e) HELPER L45  ASSERT L52  GUARD none; only guard is L69, inside evidence()  catch L446 no proxyFailure
3 of 4 siblings guard both sides of their JSON helper; the new file guards 0 of 2 and has no end-of-try guard.

Mirror hosted-cancellation-driver.ts:55-64: the guard goes first in the helper and again after the status assert, and the two waitUntil loops get guarded the way hosted-broker-reply-loss-driver.ts:204 guards its own. One premise the fix must keep: no sibling driver logs proxyFailure in its catch (0 of 4 — hosted-cancellation-driver.ts:408, hosted-store-failure-driver.ts:418, hosted-process-crash-driver.ts:424, hosted-broker-reply-loss-driver.ts:335); the family instead guarantees that the thrown cause is proxyFailure, through helper guards or an end-of-try rethrow (hosted-store-failure-driver.ts:414, hosted-process-crash-driver.ts:417), so the guards are the family-conforming fix rather than a new catch-block log line.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R1-2: The gate's central premise — that the resumed frames arrive by live after-commit hub delivery rather than SQL reconciliation — is held only by an unstated numeric relation between these two arguments and the driver's 10s receive window in another language (hosted-sse-gap-driver.ts:176), and the comment names only polling although the reconciliation cadence is min(heartbeat-interval, poll-interval).

ManagedEventStreamService.waitDuration returns min(untilHeartbeat, reconciliationInterval), and an empty delivery sets reconcile = true, so SQL is re-read on that combined cadence. Both knobs are 60s here, which is why Observer.through()'s waitUntil(..., 10_000) can today only be satisfied by hub delivery. Lower either one below 10s — the natural edit for anyone trying to shrink the teardown stall that the shutdownNow() at line 261 exists to work around, since that stall is the 60s cadence — and SQL reconciliation can deliver the frames inside the window. FG6e then keeps passing while no longer distinguishing live hub delivery from polling fallback, so a regression that breaks after-commit hub notification ships green through the very gate built to catch it. Nothing asserts the relation in either language, and the drift direction is the system default: ManagedAgentProperties.Events ships pollInterval = 5s, heartbeatInterval = 15s.

Witness:

Reflective invocation of the real private ManagedEventStreamService.waitDuration(long) on a real ManagedAgentProperties, classpath = the module's target/classes + Spring 6.2.19:
PR override poll=60s heartbeat=60s cadence= 59985ms OK > 10s window: only live hub can deliver
product DEFAULT poll=5s heartbeat=15s cadence= 5000ms LEAK <= 10s window: SQL reconcile can deliver
poll lowered poll=5s heartbeat=60s cadence= 5000ms LEAK <= 10s window: SQL reconcile can deliver
hb lowered poll=60s heartbeat=5s cadence= 4999ms LEAK <= 10s window: SQL reconcile can deliver
boundary poll=11s heartbeat=60s cadence= 11000ms OK > 10s window: only live hub can deliver
boundary poll=9s heartbeat=60s cadence= 9000ms LEAK <= 10s window: SQL reconcile can deliver
The boundary is the driver's 10s window, and lowering either knob breaches it.

Express the invariant once instead of in a comment: runDriver already writes a config object to the driver (lines 229-232), so the cadence can travel in it for the sse-gap mode and the driver can assert the relation at startup, or derive through()'s budget from the supplied interval.

// hosted-sse-gap-driver.ts, at startup, once the cadence travels in driver.json
assert(
  reconcileIntervalMs > RECEIVE_WINDOW_MS,
  `SQL reconciliation every ${reconcileIntervalMs}ms can satisfy the ${RECEIVE_WINDOW_MS}ms receive window; the gate would no longer prove live hub delivery`,
);

Two premises the fix rests on: the driver.json map at HostedWorkspaceToolTurnIT.java:229-232 is shared by all five drivers and destructured by each, so a new key must be optional or supplied for the other four modes, which do not set the events intervals; and waitUntil's default timeout is 30_000 (integration-tests/helpers/hosted-harness-process.ts:24) while through() passes 10_000 explicitly, so the assertion must key on the 10s window rather than the default. With that assertion in place, changing line 157 to --qwen.managed-agent.events.poll-interval=5s must make the driver fail immediately on it; remove the assertion and the same configuration passes silently, which is the hole being closed.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R2-1: The public reconnect's ?after=0 is the very value PublicAgentController uses for "no cursor supplied" (@RequestParam(defaultValue = "0") long after), so this gate does not pin Last-Event-ID precedence as firmly as both design docs claim. The mutation the docs do name — ignoring a resume cursor — is caught. A precedence flip of the other shape is not: rewrite the call site as long cursor = after > 0 ? after : parseSequence(lastEventId, after); (or Math.max(after, parseSequence(...))) and FG6e stays green, because with after=0 both implementations fall back to the header and publicAfter.frames is still [2]. That is not only a fixture concern. A browser EventSource auto-reconnect resends the original URL, so a real client carries a non-zero after=N together with Last-Event-ID: M where M>N; under that mutation the cursor becomes N, the client re-receives N+1..M, and the guarantee this gate exists to protect — each durable event exactly once — is violated with nothing here to catch it. The hardcoded 0 also reads like a leftover constant next to the web branch's honest afterSequence: after ?? 0 two lines below, so a future maintainer "fixing" it to the real cursor would silently remove the coverage without reddening any test.

Witness — reflective call into the real compiled PublicAgentController.parseSequence (classpath = the module's own target/classes):

loaded real method: private static long ...PublicAgentController.parseSequence(java.lang.String,long)
Last-Event-ID=1 after=0 -> shipped=1 mutationTernary=1 mutationMax=1 | ternary indistinguishable=true max indistinguishable=true
Last-Event-ID=1 after=3 -> shipped=1 mutationTernary=3 mutationMax=3 | ternary indistinguishable=false max indistinguishable=false
header=-1 -> threw ApiException: Last-Event-ID is invalid.

javap -p -v on the same class file shows parameter 3 of events carrying RequestParam(defaultValue="0"), and javap -c shows 4: invokestatic parseSequence → 7: lstore 9, after which after is never read again.

Give Observer.start a query cursor separate from the header cursor (e.g. a queryAfter?: number parameter) and have the public reconnect pass an upward-conflicting value such as after=3, above the durable watermark at that moment. Correct code returns the header value 1 and ignores after, so the replay stays exactly [2] and the assertion still passes; under the mutation the cursor becomes 3 and publicAfter.through(2) times out red after 10s. The wording in both design docs ("a conflicting after=0" / 「冲突的 after=0」) is worth correcting at the same time, since after=0 is not a conflict, and the mutation list should name this shape alongside the three it already lists.

The conflict has to come from after rather than from an illegal header: PublicAgentController.parseSequence (PublicAgentController.java:222-235) throws ApiException(BAD_REQUEST, "invalid_event_cursor") on a negative or non-numeric Last-Event-ID — measured above — and the header at line 105 is attached only when after !== undefined, so the reconnect call must keep passing a cursor. The test that has to go red without this change is assert.deepEqual(publicAfter.frames.map((frame) => frame.id), [2], 'Public cursor replay') at hosted-sse-gap-driver.ts:327-331: once after=3 is in place, swapping the precedence to after > 0 ? after : ... must fail that assertion and the preceding publicAfter.through(2); with today's after=0 neither does.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R2-2: This is the only HTTP helper in the driver that throws away the server's response body when the status is wrong — privateJson passes text + cli.output (line 53), api passes text (65), evidence passes text (80) and the proxy passes text (203) — and it is also the call most likely to receive a server-side explanation. TenantContextFilter.doFilterInternal (managed-agent-server/src/main/java/.../api/TenantContextFilter.java:51-73) answers a missing or malformed X-Qwen-Tenant-Id with 400 plus an invalid_tenant JSON envelope, and an actor/tenant mismatch with 403 plus actor_scope_mismatch. So if FG6e ever fails at stream establishment — the fixture's AuthenticatedTenantActor not taking effect, or the managed_workspace_access grant changing — the driver reports only 403 !== 200, and the top-level catch prints cli.output and operations, neither of which contains that envelope. Every red then costs a rerun with added logging just to learn whether it was 400, 403 or 500.

Witness — three arms over the verbatim Observer class, the only variable being how this line is written:

=== server=forbidden assert=pr ===
start() threw: AssertionError
  message: "Expected values to be strictly equal:\n\n403 !== 200\n"
  message carries the server envelope? false
=== server=forbidden assert=fix ===
start() threw: AssertionError
  message: "{\"error\":{\"code\":\"actor_scope_mismatch\",\"message\":\"Authenticated actor scope is invalid.\",\"requestId\":\"probe\"}}\n\n403 !== 200\n"
  message carries the server envelope? true
=== server=stream assert=naive ===
start() STILL BLOCKED after 2010ms -- await response.text() on a live text/event-stream never resolves

Read the body only on the failure path, so the success path never touches the stream:

if (response.status !== 200) {
  assert.equal(response.status, 200, await response.text());
}

The if is load-bearing, not decorative: the 200 response body here IS the SSE stream consumed at lines 122-124 via response.body!.pipeThrough(new TextDecoderStream()).getReader(), and calling .text() on a live text/event-stream reads until the stream ends — the naive arm above was still blocked after 2010ms, which would hang Observer.start forever and leave the observer receiving no frames at all.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R2-3: These ten lines are the third byte-identical copy of the FIFO park protocol in this family — hosted-cancellation-driver.ts:83-92 and hosted-process-crash-driver.ts:89-98 are character-for-character the same, and the three differ only in what they do after the park. The block encodes a non-obvious OS-level contract (O_WRONLY|O_NONBLOCK returns ENXIO when no reader is present, so a successful open is itself the proof that the tool is parked inside read), and it writes the handle into a module-level writer that each driver's finally must remember to close. Any correction to it — an errno that differs between macOS and Linux, or closing the writer immediately after a park timeout so the handle cannot leak — has to be applied by hand in three places. Fixing one leaves FG6c, FG6d and FG6e holding three implementations of the same protocol that then fail differently, and debugging means diffing them against each other. The shared module this belongs in already exists and is already imported by all six hosted drivers: hosted-harness-process.ts exports waitUntil.

Witness — mechanical byte comparison, indentation normalised and nothing else:

### diff sse-gap(287-296) vs cancellation(83-92): BYTE-IDENTICAL (modulo indent)
### diff sse-gap(287-296) vs process-crash(89-98): BYTE-IDENTICAL (modulo indent)
### diff cancellation(83-92) vs process-crash(89-98): BYTE-IDENTICAL (modulo indent)
### who imports hosted-harness-process
hosted-broker-reply-loss-driver.ts hosted-cancellation-driver.ts
hosted-process-crash-driver.ts hosted-sse-gap-driver.ts
hosted-store-failure-driver.ts hosted-workspace-tool-turn-driver.ts

Add export async function parkOnFifo(fifo: string): Promise<FileHandle> to hosted-harness-process.ts doing the waitUntil + ENXIO retry + isFIFO() assertion and returning the writer handle; the three drivers then become writer = await parkOnFifo(proof); and keep their own evidence and status assertions.

The shared helper has to preserve if ((cause as NodeJS.ErrnoException).code !== 'ENXIO') throw cause; (hosted-cancellation-driver.ts:88) exactly — swallow only ENXIO and rethrow every other errno immediately, otherwise EACCES or ENOENT from a wrong or unreadable proof path degrades into a silent 30s timeout instead of an immediate error naming the real cause.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R2-4: This four-hop teardown chain (service → provisioner → delegate → owned[*] → process, then destroyForcibly + waitFor(5s)) is character-for-character identical to HostedCancellationProbe.java:240-249 apart from the receiver variable name, one unrelated setField on the cancellation side, and brace style — and a repo-wide grep for the literal "provisioner" returns exactly these two sites. The cost is that the chain has no compile-time contract at all: if LocalProcessRuntimeProvisioner ever renames delegate, owned or process, both probes fail during cleanup with an NPE or ClassCastException rather than at compile time, so whichever gate happens to be running gets fixed while the other one rots silently. The copy also dropped the original's .as("FIFO worker cleanup") description, so an FG6e worker that outlives 5s reports a bare expected: true but was: false with nothing naming what timed out. (The separate question of an assertion throwing from the IT's finally is already covered by the sandboxed-verify report's S3 on this PR, so this comment asks only for the shared helper and the restored description.)

Witness — mechanical diff of HostedSseGapProbe.java:223-229 against HostedCancellationProbe.java:240-249; the five middle lines are identical:

1c1,2
< Object provisioner = ReflectionTestUtils.getField(brokerService, "provisioner");
---
> ReflectionTestUtils.setField(service, "transport", transport);
> Object provisioner = ReflectionTestUtils.getField(service, "provisioner");
7c8,10
< for (Process process : processes) assertThat(process.waitFor(5, TimeUnit.SECONDS)).isTrue();
---
> for (Process process : processes) {
> assertThat(process.waitFor(5, TimeUnit.SECONDS)).as("FIFO worker cleanup").isTrue();
> }

### grep -rn '"provisioner"' --include=*.java (excluding target/)
HostedCancellationProbe.java:241
HostedSseGapProbe.java:223

Extract a package-private fixture helper — both probes are constructed by HostedWorkspaceToolTurnIT, so reachability is not an issue — e.g. HostedFixtureWorkers.destroyOwned(Object brokerService), called from both close() methods, and restore the .as("FIFO worker cleanup") description inside it.

releaseTerminal.countDown(); at line 223 must still run before the worker teardown: the relay thread is parked in assertThat(releaseTerminal.await(45, TimeUnit.SECONDS)).as("terminal release").isTrue(); (line 142) and close() ends with if (relay != null) relay.get(10, TimeUnit.SECONDS); (line 230), so a shared helper that absorbs or reorders the latch release turns cleanup from "done within 5s" into a 10s TimeoutException.

— qwen3.8-max via Qwen Code /review (v0.24.6)

[Suggestion] R2-5: FG6e is the fourth driverName-derived boolean in runDriver (faults, storeFaults, cancellations, sseGaps), and adding this one gate meant editing seven places inside the same method: line 115 (the switch), 155-159 (arguments + application.initializers), 166 (the database-banner ternary), 212 (probe construction), 238 (the HOSTED_*_OK marker ternary), 246 (the report-assertion chain) and 261 (the shutdownNow cleanup). The part worth writing down is that two of those three ternary chains are prepend-only, and nothing in the file says so: because boolean faults = !driverName.equals("workspace-tool-turn") is true for every gate driver, an arm appended at the end of the chain at 238 or 246 is swallowed by the faults arm. This diff prepends correctly at all three sites, which is exactly why the rule is invisible — the next gate author has to rediscover it, and the two failure shapes are silent rather than loud.

Witness — the real expressions extracted verbatim from lines 112-116, 166, 238-240 and 246-249, compiled with javac -proc:none under JDK 21 and evaluated by a JVM for three routing variants across all six driver names:

== asShipped            driver=fg6f | banner=FG6A_DATABASE  | marker=HOSTED_REPLY_LOSS_OK | reportAssert=assertFaultLedger(...)
== fg6fAppendedAtEnd    driver=fg6f | banner=FG6F_DATABASE  | marker=HOSTED_REPLY_LOSS_OK | reportAssert=assertFaultLedger(...)
== fg6fPrependedAtFront driver=fg6f | banner=FG6F_DATABASE  | marker=HOSTED_FG6F_OK       | reportAssert=sseProbeF.assertReport

Read the two harm paths off that table. Omitting an arm entirely (asShipped) still runs green while the CI log names the wrong database — and that banner line is the only evidence of which database the gate ran on, printed immediately before the forensic assertThat(metadata...).containsAnyOf("mysql","mariadb") output. Appending instead of prepending leaves marker=HOSTED_REPLY_LOSS_OK, so the line-238 log assertion looks for a marker the new driver never prints (red, for the wrong reason), and line 246 falls through to FG6a's generic assertFaultLedger(...), meaning the new gate's own SQL assertions never execute and the generic one can pass green while proving nothing about it. All five existing drivers route identically across the three variants, so only a new arm is exposed.

Replace the four booleans with one exhaustive driver descriptor — an enum or record carrying name, okMarker and databaseBanner, with requiresMySql() derived from databaseBanner != null. Lines 166 and 238 then become field lookups and the chain at 246 becomes a switch expression over the enum, so a missing arm is a compile error instead of being silently masked by faults.

driverName doubles as the driver's filename token — "integration-tests/helpers/hosted-" + driverName + "-driver.ts" at line 232 — so the descriptor must keep handing the literal sse-gap (not SSE_GAP) to ProcessBuilder; and the marker strings are printed by each driver itself (console.log('HOSTED_SSE_GAP_OK', ...) at hosted-sse-gap-driver.ts:444), so they cannot be renamed.

— qwen3.8-max via Qwen Code /review (v0.24.6)