Skip to content

fix(runtime): release failed startup lock publication - #9110

Open
loopx-agent wants to merge 4 commits into
mainfrom
codex/recover-failed-runtime-start-lock-20261120
Open

loopx-agent wants to merge 4 commits into
mainfrom
codex/recover-failed-runtime-start-lock-20261120

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Startup recovery

If writing the Effect runtime's PID to its newly created startup lock fails, the caller can leave the lock behind and then time out waiting for it on retry. If constructing the file stream fails, the caller can also leave the descriptor open.

The startup path captures the new file's identity before constructing the stream, closes the raw descriptor, and uses the existing identity check to remove its own lock on publication failure. It preserves the original error and any replacement lock. The same runtime request then starts normally on retry.

Integration and verification

Proposed head 5cb45f4f889b82979b5ac63cbc70796456a89b6d integrates main 856cd6ad8fb93bcb8e3a4a1a2f98701da2b07cdc. The diff contains the runtime repair, its existing regression cases, and the corresponding repair pattern. The caller remains the existing Effect runtime entry point.

  • The source runtime integration suite passed 118 tests. One Linux-specific case was skipped on macOS.
  • Separately built and installed Core wheels from main and the proposed head were exercised with Python 3.11 and Node 24. On main, both injected fdopen and fsync failures left the startup lock and prevented a same-process retry. On the proposal, both failures preserved their expected errno, released the descriptor and lock, and let the same process start the real TypeScript runtime. Two replaced-lock cases preserved the replacement and also closed the descriptor.
  • All 58 tests in the installed Core qualification passed on macOS, including the executable-catalog check and the context, recovery, and replay entry points. The standard premerge run passed its 13 selected checks and 6 direct checks. Ruff passed on the Python diff.
  • The previous Package Smoke rejected the literal Workbench catalog command. This head includes the catalog repair already merged in Fix Workbench command catalog and manual #9240.
  • The previous native Windows job stopped on WinError 10038 in the research test fixture's subprocess-pipe reader. fix(testing): read the canonical authority worker portably on Windows #9236 covers that fixture. Its hosted run passed the affected fixture step and later stopped at the Windows CLI-output differential. The full Windows gate and independent review of this head are pending.

The installed results apply to the local macOS test environments. The published release and signed-in hosts still need their own adoption evidence.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI (runtime_reported, low)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

effect_runtime_result(runtime.ping) → effect_runtime_request → _start_runtime 同一 PID 的 startup lock fsync 注入 ENOSPC 后,base 保留活 PID marker 且没有 locator;同进程再次 runtime.ping 在 15.059 秒后返回 startup_lock_timeout。 候选在原错误后移除自己创建的 marker,保持 runtime_request_failed 与原始 ENOSPC cause;同一进程恢复后真实 TypeScript runtime.ping 返回 ready。替换 inode 的 marker 保留。 仅验收 macOS 的隔离 source CLI、Hook 或 TypeScript runtime;未验证已安装 App、Windows 或完整 C1 资格,不推断全平台交付。 仅验收 macOS 的隔离 source CLI、Hook 或 TypeScript runtime;未验证已安装 App、Windows 或完整 C1 资格,不推断全平台交付。

改动思路 / Approach

既有 _start_runtime 持有 exclusive startup marker;file_lock.py 提供 dev/inode/birth identity 和安全退休。 沿用已有 typed owner 和既有 file/receipt identity,不另建持久状态或 Python 决策器。将 FileExistsError 的 contention 判断限制在 exclusive create;PID write/flush/fsync 失败时用既有 identity helper 清理,再原样抛错。先检查既有 owner 和完整 diff,再沿真实调用验收成功、失败及下一次恢复。

具体改动 / Changes

精确 head c6f97097cc5226dc91356ad6b464b3a9ef52c892,diff base 8e9a28a866da06cc6fd63e78709910fe35dad564,3 个文件 +80/-7。逐项复核 原问题与实际恢复、身份与副作用边界。验收约定

关键代码讲解

  • PID write/flush/fsync 进入 cleanup scope,既有竞争者读回保留。源码
  • 复用共享文件身份比较,替换文件不删除。源码
  • 测试从真实 request 触发原异常并在同进程启动 TypeScript runtime。源码
  • repair pattern 是按故障证据执行的建议,未新增阻塞决策或自动权限。源码

对主干的风险 / Risks

Effect runtime 集成套件 113 PASS / 1 Linux-only SKIP;新增两项测试在相同 base 均 FAIL。ENOSPC probe:base 留锁且重试 15.059 秒超时,head 清理自有锁后真实 ping 0.194 秒 ready。Ruff 通过。quick premerge 六项 direct 与两项 catalog 通过,semantic-vocabulary 首次 60 秒超时,保持相同 head 单独重跑通过;未把首次 quick 记为全通过。 仅验收 macOS 的隔离 source CLI、Hook 或 TypeScript runtime;未验证已安装 App、Windows 或完整 C1 资格,不推断全平台交付。

语义与 CI 对齐

未查询或等待 CI。Effect runtime 集成套件 113 PASS / 1 Linux-only SKIP;新增两项测试在相同 base 均 FAIL。ENOSPC probe:base 留锁且重试 15.059 秒超时,head 清理自有锁后真实 ping 0.194 秒 ready。Ruff 通过。quick premerge 六项 direct 与两项 catalog 通过,semantic-vocabulary 首次 60 秒超时,保持相同 head 单独重跑通过;未把首次 quick 记为全通过。

我的整体评价 / Verdict

同意这个范围明确的修复。原问题在真实入口有基线反例,当前 head 恢复了后续调用,替换文件和原错误边界保留。合并仍需维护者授权与独立 merge gate。

English verdict: APPROVE. The same-process retry now starts the real runtime after PID sync failure, while preserving a replacement lock and the original error.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · xhigh

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机 / Motivation

在同一 Python 进程中重新启动 Effect runtime 的调用方,需要在启动锁创建后遭遇 I/O 错误时可靠恢复。 旧版在 PID 刷盘失败后会遗留一个活进程的启动锁。新版能清理这一分支并允许重试,但接管原始文件描述符之前失败时仍会留下锁和描述符。 真实 TypeScript runtime 的同步故障恢复已通过,文件包装前失败的复测仍发现同一进程保有一个打开的描述符和空锁。 只检查启动锁创建、故障清理及已有运行时启动行为,不改变其他文件锁协议。

改动思路 / Approach

把独占创建失败与其后的 PID 发布分开,能避免把发布错误误判为另一个进程持锁。对成功创建的文件沿用已有 inode 身份清理。当前接管顺序让最早的发布失败缺少身份和描述符收尾,最小修补应保持原启动和竞争路径。

具体改动 / Changes

已有文件锁实现要求以创建时的文件身份清理,保护随后替换的锁文件。文件锁实现

  • LOCK-SYNC 已实现。当前提交的同步故障回归分别证明清除本进程留下的锁、真正重启 TypeScript runtime,以及保留替换锁和原始故障原因。
  • LOCK-WRAP 未满足。补充探针把错误放在文件流接管描述符之前。精确提交留下同一个描述符和空启动标记。随后只有通用的陈旧锁回收路径能移除标记,普通重试仍可能等待。
  • LOCK-REPLACEMENT 现有同步分支保持替换文件。新增包装失败探针中替换文件保持,但原始文件描述符仍开放。

对主干的风险 / Risks

P2 覆盖文件包装前的发布失败。 现在 identity 在进入 with os.fdopen 后才计算,故包装失败时清理收到空身份,也没有对象接管最初的描述符。连续故障会叠加描述符占用,空标记留到陈旧锁超时。取得原始描述符后立即记录稳定的文件身份,明确跟踪描述符所有权,失败时关闭仍由调用方持有的描述符并只移除原锁。新增包装失败用例并保留替换锁控制。

精确提交上的 Effect runtime 集成测试 113 过、1 跳过,Ruff 和 diff 检查通过。包装故障用隔离临时目录重放,记录原错误、描述符、锁和 locator。远端 CI 未检查。

我的整体评价 / Verdict

检查了真实 runtime.ping、PID 发布失败和原始错误读回,变更没有渲染页面。 实测为 macOS,Windows 和安装后长期运行未验证。 当前已独立审阅 c6f97097cc5226dc91356ad6b464b3a9ef52c892,保留变更请求。前次同头批准覆盖了 fsync 故障。本次在 fdopen 边界独立复测,发现启动锁仍可滞留并泄漏描述符。

English verdict: REQUEST_CHANGES

Comment thread loopx/control_plane/effect_runtime.py Outdated
else:
identity = None
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as lock_file:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 在 fdopen 接管前仍需保有清理信息。我对这个边界注入 I/O 错误,原始错误返回后 FD 仍打开,空的启动锁也留在目录里。请在包装前捕获原始文件身份,并在接管失败时关闭原始 FD,只清理同一把锁。保留已有的替换锁控制。

…ock-review-fix-20261122

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · xhigh

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

Effect 客户端在本地首次启动失败后,应能在同一进程再次请求运行时。 基线在锁创建后遇到 fsync 或 fdopen 失败时保留启动锁。先前受审提交清除了 fsync 失败的锁,fdopen 失败仍留下空锁和打开的描述符。 在已成功排他建锁后分别注入 fdopen 和 fsync 错误,比较基线、先前受审提交、当前源码和实际安装的 Core wheel。 当前头用一个原始描述符 owner 收尾,先取得文件身份,再创建不接管描述符的流。失败后保留原异常并清理原锁。 改动限于本地 Effect 启动锁发布阶段,复用既有文件身份检查、重试语义与诊断。

改动思路 / Approach

_start_runtime 在 os.open 成功分支捕获 identity。内部 finally 关闭原始描述符,外层失败时复用按身份删除的 helper。 只有 O_EXCL 创建成功的分支拥有该句柄。竞争方仍遵循原 locator、holder 与退避规则。 已存在的回收逻辑可以处理过期或死亡 owner。发布失败的调用者知道自己刚创建的文件,可以在重试前即时回收自己拥有的锁,不用延长超时或缩短活动 holder 的保护。

具体改动 / Changes

准确头 5cb45f4,merge base 856cd6a,3 个文件,+107/-8。三个变更文件分别是运行时补丁、集成故障测试和 self-repair 经验条目。 按修订 856cd6a 的现有规范核对各项要求。 目标规范

关键代码讲解

  • _start_runtime 的排他创建把真正的锁竞争留在原分支。 源码

  • 发布分支先从原始描述符读取身份,再 fdopen(closefd=False)。内外清理分别负责关闭和按身份移除,原异常继续向外。 发布与清理

  • 集成测试覆盖两类故障、文件身份替换控制,以及实际启动后的重试。 测试

  • io-publication-failure 同样核对两种错误的 errno、描述符、锁和 locator。

  • ownership-during-cleanup 替换文件保持其原身份及内容。

  • actual-runtime-recovery 从当前提交构建并独立安装 standalone Core wheel,以该安装启动真实 Node 进程。故障注入只涉及临时目录。

对主干的风险 / Risks

源码与隔离安装的 Core 2.4.0 wheel 均返回原错误且未发布 locator。两种故障的描述符均已关闭、自有锁已清理,同一进程下一次真实 Node runtime.ping 返回 ready,随后正常关闭。 用相同两类故障在清理前换入另一文件,源码和 wheel 的四个替换情形均保留该文件的原内容。聚焦套件还覆盖现有 locator 快路、死 holder 回收、存活 holder 保留。 118 项运行时集成测试通过,1 项 Linux 专属跳过,Ruff 通过。标准预合并的 6 项直接检查、13 项选中检查均通过。 真实进程路径和 Core wheel 检查在 macOS、Python 3.14 和 Node 24 上完成。Linux 专属用例跳过,Windows 与桌面安装版未测。未查询远端 CI。当前范围的本地严格质量回执仍缺,合并资格单独核对。

我的整体评价 / Verdict

批准当前提交,未发现阻断项。当前补丁覆盖了上一轮 fsync 修复遗漏的 fdopen 边界,正常启动和竞争分支仍遵循原路径。 该故障和恢复均在本地 Effect 进程启动层,无前端改动。

English verdict: APPROVE at 5cb45f4. The exact head closes the descriptor and cleans up its own lock after both stream-open and fsync failures. Source and isolated installed Core both retry successfully in the same process, preserving replacement files and the original error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant