Skip to content

Hosted Shell: recover a Workspace lease after incomplete pipe capture #12904

Description

@doudouOUC

Problem

The explicit private Hosted Shell profile in #12848 can leave a durable Workspace execution lease held indefinitely after an ordinary backgrounded compound command. With cd . && sleep 20 >/dev/null 2>&1 & echo ok, the shell leader exits promptly, but its subshell keeps a captured pipe open past the one-second post-exit drain boundary. The capture is incomplete, the turn becomes recovery-blocked, and every new Session in the same Workspace receives 409 workspace_busy, including file-only Sessions. Real-stack review evidence observed this more than 40 minutes later and in 3 of 6 real-model trials.

The behavior is fail-closed for execution uncertainty, but there is no supported recovery path once the producer has stopped. #12848 is correcting a related misclassification that marked the incomplete pipe storage_failed and discarded its accepted prefix; that diagnostic fix does not release the lease. #12900 addresses the independent duplicate Flyway V16 migration.

Safety boundary and proposed scope

Do not release the Workspace lease merely because the shell leader exited, the original process group was killed, or stdout reached EOF. A controlled POSIX reproduction showed a detached descendant writing to the Workspace after the original process group was killed and the pipe reached EOF. A fix must either provide an execution boundary that can prove all potential writers have stopped, or provide an explicit, audited operator recovery action based on verified quiescence and a compare-and-swap on the exact held lease. Unknown execution and uncertain output must never be replayed as a new Shell invocation.

The follow-up should define the operator-facing recovery steps and avoid widening the public Shell profile until this path is safe. Keep the current fail-closed behavior in #12848 meanwhile.

Acceptance

  • Reproduce the compound-command case with the real Broker, worker, SQL Store and a second Session in the same Workspace. Capture a durable, accurate reason and make the recovery action available after verified quiescence; a new Session then proceeds without repeating the original side effect.
  • A detached descendant that can still write to the Workspace keeps the fence in place. Killing the original process group and seeing pipe EOF alone must not satisfy recovery.
  • Writer loss, Broker/worker crash, activation replacement and restart preserve the same lease identity and never silently release or replay uncertain work.
  • Other Workspaces remain usable throughout. Cover MySQL and MariaDB once fix(managed-agent): Move the Session operation migration to V17 #12900 lands.

Related: #12380, #12848, #12766, #12670.

中文说明

问题

#12848 的显式私有 Hosted Shell profile 在普通后台复合命令后可能永久占住持久化的 Workspace 执行租约。运行 cd . && sleep 20 >/dev/null 2>&1 & echo ok 时,Shell 主进程很快退出,但子 Shell 持有捕获管道超过退出后一秒的排空边界。捕获因此不完整,回合进入恢复阻断,同一 Workspace 的每个新 Session 都收到 409 workspace_busy,包括仅文件工具的 Session。真实环境评审证据观察到 40 多分钟后仍无法使用,真实模型 6 次试验中有 3 次触发。

执行不确定时阻断是安全的,但生产者停止后,目前没有受支持的恢复路径。#12848 正在修正把不完整管道误标为 storage_failed 并丢弃已接收前缀的问题;该诊断修复不会释放租约。#12900 修复独立的 Flyway V16 重复迁移。

安全边界与范围

不能仅因 Shell 主进程退出、原进程组被杀,或 stdout 到达 EOF 就释放 Workspace 租约。受控 POSIX 复现表明,一个脱离原进程组的子进程能在原进程组被杀、管道 EOF 之后继续写 Workspace。修复须建立能证明所有潜在写入者已停止的执行边界,或提供基于已核实静止状态、针对精确租约做比较并交换的显式且可审计的运维恢复操作。不确定的执行和输出绝不能作为新 Shell 调用重放。

后续工作还应提供运维恢复步骤;在此路径安全之前,不扩大公开 Shell profile。#12848 暂时保持故障阻断。

验收标准

  • 用真实 Broker、worker、SQL Store 和同一 Workspace 的第二个 Session 复现复合命令。持久记录准确原因;核实静止后可执行恢复,新 Session 随后能继续,而原副作用不重复。
  • 仍可能写入 Workspace 的脱离进程组子进程必须使围栏保持生效。只杀原进程组并看到管道 EOF 不足以恢复。
  • writer 丢失、Broker/worker 崩溃、activation 替换和重启时保留相同租约身份,不能静默释放或重放不确定工作。
  • 其他 Workspace 始终可用;fix(managed-agent): Move the Session operation migration to V17 #12900 合入后覆盖 MySQL 和 MariaDB。

相关:#12380、#12848、#12766、#12670。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions