Skip to content

fix(runtime,benchmark): ignore Linux zombie groups and harden worker hygiene - #9243

Open
loopx-agent wants to merge 2 commits into
mainfrom
codex/fix-runtime-zombie-and-bench-hygiene-20261122
Open

loopx-agent wants to merge 2 commits into
mainfrom
codex/fix-runtime-zombie-and-bench-hygiene-20261122

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Root-cause runtime and benchmark worker hygiene fixes:

  1. Linux zombie detection in effect_runtime._pid_is_alive:
    • Consult _linux_thread_group_is_zombie(pid) on Linux inside _pid_is_alive so unreaped all-zombie thread groups held by blocked parents do not poison runtime locator recovery or start-lock holder checks; also reap exited direct children if they transition to zombie before _stop_effect_runtime exits.
  2. Wake failure receipt persistence & core-dump / stderr observability:
    • Guard heartbeat_host_failure(wake, receipt) in benchmark/runtime/worker.py with try/except Exception (recording host_failure_record_error) so (wake / "receipt.json").write_text(...) always flushes.
    • Disable POSIX core dumps (RLIMIT_CORE soft limit 0) in scripts/external_scheduler_worker.py.
    • Retain optional bounded stderr: bytes = b"" on CappedProcessResult (loopx/extensions/process_runtime.py) and log bounded stderr_tail on wake_failed in scripts/external_scheduler_worker.py.
  3. Checkpoint artifact exclusions & SForge research hygiene:
    • Support validated component_exclusions on probe and capture in checkpoint_artifacts.ts (never on verify/materialize, keeping manifest schema invariant) and pass RESEARCH_EXCLUDE_PATTERNS (*.raw.gz, core, target/, .cargo-target/) for benchmark_research across checkpoint.py, checkpoint_wake.py, and benchmark/edgebench/checkpoint_worker.py.
    • Persist CARGO_TARGET_DIR=/tmp/sforge-cargo-target in SForge worker runtime environment (sforge.py).
    • Enforce a 10 GiB free-disk floor before and during "research" archive spooling/extraction in sforge_backend.py (recording explicit insufficient_host_disk_headroom diagnostics in artifact-collection.json and filtering ephemeral dumps).
  4. Opt-in runtime PATH & claim-TTL forwarding:
    • Preserve existing LOOPX_NODE_DIR/bin, Path(LOOPX_CLI).parent, and LOOPX_BENCHMARK_PATH_PREPEND dirs in codex.process_environment without leaking ambient host PATH.
    • Support opt-in claim_ttl_seconds / LOOPX_BENCHMARK_CLAIM_TTL_SECONDS (1..86400) in codex.py, harbor.py, sforge.py, and worker.py, forwarding --ttl-seconds guidance compatible with control_plane: add bounded --ttl-seconds on todo claim and clarify source.ref diagnostics #9225 while keeping default configs byte-identical when unset.

Verification

  • ruff check on all 19 touched Python files: passed
  • mypy --follow-imports=silent (loopx/control_plane/effect_runtime.py, loopx/extensions/process_runtime.py): passed (0 issues)
  • node --experimental-strip-types --test tests/control_plane_ts/checkpoint_artifacts.test.ts tests/control_plane_ts/host_process_group.test.ts: 24 passed, 0 failed
  • loopx check --scan-path ... (with node_modules): errors=0, warnings=1 (pre-existing non-literal match in checkpoint_worker.py:413)
  • pytest -n 4 (16 control-plane + benchmark test suites in .venv): 541 passed, 500 skipped (sforge/harbor/Docker/Linux-only cases)
  • Unskipped sforge + harbor unit check (test_sforge_artifact_spooling.py, test_shared_codex_runtime.py, test_sforge_runtime.py, test_checkpoint_worker.py exclusion test): passed
  • loopx canary premerge --from-git-diff --git-diff-base origin/main: direct checks 6/6, catalog canaries 10/10, risk-profile smokes 8/8 (failures=0, manual_review_required due to benchmark_sensitive)

…hygiene

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…bie-and-bench-hygiene-20261122

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

# Conflicts:
#	benchmark/runtime/harbor.py
@mergify

mergify Bot commented Nov 22, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added needs-rebase Mergify: the pull request has merge conflicts with its base branch and removed needs-rebase Mergify: the pull request has merge conflicts with its base branch labels Nov 22, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

本次对 loopx-project/loopx#9243 精确提交 55c8032c80f91134c2e23ff6c7b0ca73ed796025(合入基线 origin/main 0814f04e9456899600f3a3c4efccfd7775382cd8)进行独立源码审阅、反例验证与本地检验(依 wait_for_ci: false 本地先行策略);本 PR 为 loopx-agent 自提交(author-owned PR),仅发布 COMMENTED 审批结论并保持 PR 开启。

动机 / Motivation

面向在 Linux 容器与外部调度器上运行长时评测及恢复守护的运行器与控制面调用方:此前在 Linux 容器运行环境下,当短生命 CLI 派生出的守护进程退化为孤儿全僵尸线程组时,kill(pid, 0) 仍报告进程存活,从而持续阻塞 effect_runtime 定位恢复与锁回收;同时 worker.run_once 若在 heartbeat_host_failure 解析阶段抛错会阻止 (wake / receipt.json) 落盘、子进程崩溃可能在 /tmp 留下数 GB core 文件,且研究目录中的 *.raw.gz 与 target/ 目录会撑爆检查点归档及宿主机磁盘空间。 该 PR 在 Linux 上让 _pid_is_alive 将全僵尸线程组视为已终止并尽早回收已退出的直接子进程,保障异常情况下 receipt.json 必然写入,在外部调度器关闭 POSIX RLIMIT_CORE 并在 wake_failed 时保留末尾 stderr 诊断,为 checkpoint_artifacts probe/capture 引入受白名单与路径防穿越校验的 component_exclusions(保持 verify 严格全量一致)并在 SForge 容器回收中强制 10 GiB 磁盘余量下限与 CARGO_TARGET_DIR 隔离,同时提供严格默认关闭的可选 PATH 前置与 --ttl-seconds 续约参数转发。 本 PR 不修改 checkpoint 清单摘要格式、verify/materialize 校验口径,也不改变未传 claim_ttl_seconds 时的默认运行器配置与调度决策逻辑。

改动思路 / Approach

作者严格在各既有责任边界内就地加固:effect_runtime.py 只读取 /proc//task/*/stat 并在全部线程均为 Z 时判定终止(保留仍在写盘的非僵尸线程组安全拦截);checkpoint_artifacts.ts 将 component_exclusions 严格限制在 probe 与 capture 的源观察阶段,verify 与 materialize 仍禁止传入并维持 sealed manifest 签名不可变;worker.py 用显式异常捕获保证 receipt.json 持久落盘,sforge_backend.py 在 _extract_artifact_archive 抽取前与落盘流中双重检查 10 GiB 磁盘下限,codex.py 与 harbor/sforge 保持 claim_ttl_seconds 默认为 None 零行为漂移。 当前 PR 边界为运行时僵尸进程识别、worker 异常与 core-dump 诊断加固、research 检查点暂态产物过滤、10 GiB 磁盘保护及可选 PATH/TTL 透传,与其它实验调度或模型上下文 PR 零文件重叠。

具体改动 / Changes

全部 21 个文件相对 origin/main 共 +744 / -29 行(含 7 个测试文件)。未修改既有 RFC 规范文件,对应运行环境工程加固归为无独立书面规范(no written specification exists beyond existing contract invariants);已核对全部 21 个改动文件与并行中的其它工作分支无文件交集。

关键代码讲解

  1. _pid_is_alive、_linux_thread_group_is_zombie 与 _stop_effect_runtime
    在 Linux 平台读遍 /proc/<pid>/task/*/stat 状态:仅当进程组非空、全部线程状态为 Z 且无新出现线程时才返回 False,避免孤儿僵尸 leader 卡死锁/定位恢复,而仍有活线程写入的混合线程组仍会受阻。停止循环同时通过 _reap_exited_runtime_child 对已退出的直接子进程执行非阻塞 waitpid。
  2. normalizeComponentExclusions 与 checkpointArtifacts
    仅允许 probe 与 capture 携带 component_exclusions(verify 与 materialize 携带即被 hasExactAuthorityKeys 拒绝),拒绝未知组件 ID、空列表及含 //\/../NUL 的模式;Python 侧 research_component_exclusions 仅在存在 benchmark_research 组件时附带 ("*.raw.gz", "core", "target/", ".cargo-target/"),不改动清单 schema,保证捕获后 verify 原样通过。
  3. run_once 收尾写入与 _claim_ttl_guidance / _extract_artifact_archive
    run_once 的 finally 块将 heartbeat_host_failure(wake, receipt) 包裹在独立 try/except Exception 中记录 host_failure_record_error,确保 (wake / "receipt.json").write_text(...) 始终落盘;sforge_backend 在 research_cleanup=True 时于 get_archive 前、分块暂存中及解压前三重检查可用磁盘低于 10 GiB 即抛出 InsufficientDiskHeadroomError,并在归档抽取时跳过暂态构建产物。
  4. _disable_core_dumps 与 _wake_stderr_tail / CappedProcessResult
    CappedProcessResult 向后兼容地追加默认 stderr: bytes = b"",调度守护仅在 wake_failed 时记录截断末尾 stderr_tail 并在入口处把 POSIX RLIMIT_CORE 软上限设为 0。

对主干的风险 / Risks

经独立检验与反例复现,未发现阻塞合入的问题(no blocking finding)。

精确提交 55c8032c80f91134c2e23ff6c7b0ca73ed796025 实测矩阵与环境边界披露如下:

  • TypeScript 检查与类型检查:npm run typecheck:control-plane 通过(0 错误);node --no-warnings --experimental-strip-types --test tests/control_plane_ts/checkpoint_artifacts.test.ts tests/control_plane_ts/host_process_group.test.ts 24/24 passed(0 failed, 0 skipped),checkpoint_artifacts 独立对照测试脚本验证 probe/capture 过滤与 verify 拒接过滤字段及 ../ 拒绝均符合预期。
  • Python 单元测试与静态检查:在匹配工作树路径(PYTHONPATH=$WORKTREE)下执行全部 7 个受影响套件:320 passed, 456 skipped in 78.00s(0 failed)。必须如实说明:456 skipped 为 .venv 未预装可选外部包 sforge/harbor 引起的原生态 importorskip 跳过(非本 PR 删除或弱化断言;另有 test_effect_runtime_locator_recovery.py 初次未设 PYTHONPATH 时因 .venv/bin/loopx 指向主目录指纹差异失败 1 项,对齐 PYTHONPATH=$WORKTREE 后 14/14 passed in 6.25s);为弥补环境缺口,已使用内存轻量包桩实测 _filter_research_members 与 InsufficientDiskHeadroomError,并在真实 worker.run_once 下注入 heartbeat_host_failure 异常验证 receipt.json 成功写出 host_failure_record_error。ruff check 全部 .py 改动 clean;python -m mypy(17 个 strict 核心文件)0 错误;mypy --follow-imports=silent 在 effect_runtime.py 与 process_runtime.py 0 错误;loopx check 公开边界扫描 21 个文件 errors=0, warnings=0。
  • Canary Premerge:direct_checks 4/4 passed、python_compile 1/1 passed、catalog_canaries 10/10 passed、risk_profile_smokes 8/8 passed(failures=0;manual_review_required 仅因涉及 benchmark_sensitive 路径)。
  • 远端 GitHub CI:按 wait_for_ci: false 未等待远程检查。

我的整体评价 / Verdict

综合评估精确提交 55c8032c80f91134c2e23ff6c7b0ca73ed796025:在 long_horizon 与 user_experience 维度均为 improved(消除了 Linux 孤儿僵尸进程阻塞恢复与 receipt.json 丢失故障,同时避免了 core/Rust 构建缓存引发的容器与检查点存储耗尽,并保留 stderr_tail 排障信息;无前端 UI 涉及,视觉检查为 not_applicable)。所有改动正交克制、默认关闭开关零回归,本地可覆盖检查全部通过(456 项可选 harbor/sforge 跳过已由桩测 counterfactual 与源码走查补充验证并明确标注),同意合入并按 author-owned PR 规则保持开启。

English verdict: APPROVE at 55c8032. Verified Linux zombie group handling, receipt flush resilience, POSIX core dump suppression + wake stderr_tail, validated probe/capture exclusions, and 10 GiB research headroom guard. TS tests 24/24 pass, tsc/ruff/kernel-mypy/loopx-check/canary-premerge (18/18 smokes) pass, and matching-PYTHONPATH pytest passes 320 passed / 456 skipped (missing optional harbor/sforge packages in local venv, supplemented by counterfactuals).

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · xhigh
Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机 / Motivation

使用 Linux runner 做持续长时任务、要从已封存检查点找回研究依据的操作者。 原规范保留完整的研究笔记与候选历史。旧 Linux 存活判断也可能被全僵尸组阻滞。 同一份研究输入在旧版可恢复。新头完成捕获和校验,却在实际恢复时少了两个正常路径。 改动修复僵尸进程观察、保留失败回执,新增研究过滤、磁盘下限、日志和可选 TTL。 范围覆盖原 Linux 入口、runner 自有研究目录、失败回执、环境配置与整份当前提交。

改动思路 / Approach

进程存活由 Linux 原生运行时判断。检查点和终态收集的过滤由 runner 决定,通用 TypeScript 检查点负责选择后的字节和完整性校验。现有复制与封存入口可以复用。具备恢复价值的文件应由明确的生成路径或可验证类型区别于运行残留。 范围覆盖原 Linux 入口、runner 自有研究目录、失败回执、环境配置与整份当前提交。

具体改动 / Changes

核对准确提交 55c8032c80f91134c2e23ff6c7b0ca73ed796025、merge base 0814f04e9456899600f3a3c4efccfd7775382cd8,21 个文件,+744/-29。同一提交已有 Gemini 批准。本轮以真实 Linux 进程与检查点物化补验存活判断、研究证据可恢复、过滤隔离和失败诊断。 验收依据

关键代码讲解

  1. 进程存活 在 Linux 额外读线程组状态。停止循环 复用子进程回收。
  2. 研究规则 给捕获和收集传同一组名称。检查点 先按名称过滤 witness 和复制,再验证剩余数据。
  3. 失败回执 和 外部 worker 日志 沿用现有诊断通道。终态收集 在暂存和解包前估算余量。Harbor/SForge 各自转发可选 TTL 和运行目录。

对主干的风险 / Risks

[P2] 研究检查点静默排除了名为 core 的正常证据。 研究目录下有真实的 core/analysis.md 和普通文本 topic/core 时,新版同样排除这两份材料。用两版的 probe、capture、verify、materialize 实跑,基线能恢复两份材料,新头通过完整性校验但两份都缺失。候选版本在不开过滤的对照中能保留原内容。 只排除 runner 能确认属于临时产物的路径。保留普通 core 目录;同名普通文件与实际 core dump 需要可验证的区分,并在 checkpoint 与终态收集加入同一对照。

隔离 Linux 上,基线把未回收的 Z 态子进程判成活进程,新头正确判为已终止,两版均能识别还在运行的子进程。研究捕获的构建缓存与转储样例被排除,普通代码组件完整。 定向 Python 检查 320 项通过、456 项跳过。24 项 TypeScript 检查通过。标准本地 premerge 执行 18 项选中检查且无执行失败,benchmark-sensitive 人工门仍在。 使用本地隔离 Linux 容器和真实文件。未验证完整 SForge/Harbor、生产磁盘压力或远端 CI。对当前主干合并无冲突;与仍开放的 #9176 合并时三处 benchmark 调用发生冲突。

我的整体评价 / Verdict

请求修改。保留僵尸恢复、失败回执和已知临时路径隔离。请先收窄研究目录过滤,确保普通研究笔记跨 checkpoint 恢复。现有 verify 只能校验被选中归档的内容。

English verdict: REQUEST_CHANGES at 55c8032. Verified the Linux zombie fix with an actual unreaped process. The automatic research filter also drops legitimate files and directories named core before sealing, so a verified restore silently loses evidence.


RESEARCH_DIRECTORY = "/tmp/loopx-benchmark-research"
RESEARCH_COMPONENT = "benchmark_research"
RESEARCH_EXCLUDE_PATTERNS = ("*.raw.gz", "core", "target/", ".cargo-target/")

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 研究检查点静默排除了名为 core 的正常证据。研究目录下有真实的 core/analysis.md 和普通文本 topic/core 时,新版同样排除这两份材料。用两版的 probe、capture、verify、materialize 实跑,基线能恢复两份材料,新头通过完整性校验但两份都缺失。候选版本在不开过滤的对照中能保留原内容。 只排除 runner 能确认属于临时产物的路径。保留普通 core 目录;同名普通文件与实际 core dump 需要可验证的区分,并在 checkpoint 与终态收集加入同一对照。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant