Skip to content

[Feature] Retain Agent tasks across complete-request weight updates - #2297

Merged
jayhenry merged 13 commits into
xs1997zju/agent-rl-stack-20261111/03-fit-cleanupfrom
xs1997zju/agent-rl-stack-20261111/04-agent-core
Nov 20, 2026
Merged

jayhenry merged 13 commits into
xs1997zju/agent-rl-stack-20261111/03-fit-cleanupfrom
xs1997zju/agent-rl-stack-20261111/04-agent-core

Conversation

@xs1997zju

@xs1997zju xs1997zju commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Agent RL stack — bottom to top

Review each PR against its configured base. The complete delivery is split from #2257, which remains unchanged as the pre-split reference.

  1. #2295 — Preserve sampler identity and epoch on resume · xs1997zju/agent-rl-stack-20261111/02-sampler-resume → main
  2. #2296 — Release training batch references on fit failure · xs1997zju/agent-rl-stack-20261111/03-fit-cleanup → #2295’s branch (xs1997zju/agent-rl-stack-20261111/02-sampler-resume)
  3. #2297 — Retain Agent tasks, request provenance, affinity and checkpoint lifecycle · xs1997zju/agent-rl-stack-20261111/04-agent-core → #2296’s branch (xs1997zju/agent-rl-stack-20261111/03-fit-cleanup) ← you are here
  4. #2298 — Support disaggregated rollout/train debug phases · xs1997zju/agent-rl-stack-20261111/05-disagg-debug → #2297’s branch (xs1997zju/agent-rl-stack-20261111/04-agent-core)
  5. #2299 — Configure valid-attempt filtering and carry shared tests/docs · xs1997zju/agent-rl-stack-20261111/06-valid-attempts → #2298’s branch (xs1997zju/agent-rl-stack-20261111/05-disagg-debug)

Layer 3/5. Base is #2296’s branch (xs1997zju/agent-rl-stack-20261111/03-fit-cleanup). Review only this PR’s own diff.

The core layer (#2297) contains five separate topic commits that form one integration group. Shared tests/docs are carried in the final layer (#2299). Intermediate commits are review units, not independently validated feature versions.

Independent related PRs: #2252 (iterative trace traversal; overlaps with #2297 and needs reconciliation) and #2292 (native frozen MTP). They are not additional layers of this stack.

Developer guide, diagrams, scripts and validation scope.

定位

本 PR 保留 #2297 编号,内容替换为第 4 层(04-agent-core)的新实现;原实现已备份到分支
backup/agent-rl-stack-04-agent-core-20261120(6b6439f0)。

重构设计思路

目标是把第 4 层的职责收敛成 3 个有真实深度(Depth)的行为类,并让整个 fit 生命周期落在一个 asyncio loop 上。后续评审又补齐了真实取消边界与完整组交付生命周期。

flowchart TD
    A["fit(): 一次 asyncio.run"] --> B["_fit: 异步恢复与首次 I/O"]
    B --> C["初始化: pause → 恢复权重 → publish(0)"]
    C --> D["produce_batch: publish(step) → 完整组 Task 生产 → gate drain/pause"]
    D --> E["await_owned_operation(to_thread): 训练/保存/权重同步"]
    E --> D
    D --> F["末步: 不 reopen → close 取消并等待所有组 → 退出"]
    C --> G["RolloutController 请求屏障"]
    G --> H["RequestGate"]
    H --> I["SessionServer → TraceStore"]
    D --> J["SessionAffinityRouter"]
    D --> K["_PendingTasks: 组 Task 唯一 owner"]
Loading
  1. Fit 生命周期(A3):fit() 只调用一次 asyncio.run(self._fit())。ReplayBuffer、sampler、Agent 客户端与 manager 仍在 __init__ 按原流程构造(只创建同步字段和惰性绑定的 Lock/Event/Semaphore,不启动 task、不发异步 I/O);异步恢复、首次权重 I/O 与初始发布、produce、save、close 在 fit loop 中 await。阻塞训练、DCP/HF 保存、权重/IPC 与资源切换通过 asyncio.to_thread 执行,并统一用 await_owned_operation 等待“真正结束”后再传播取消。

    约束边界:同一 loop 约束的是异步激活与使用,不是所有 Python 对象的构造;因此没有把 builder 搬进 _fit,构造阶段也不做异步 warmup 或 manager.resume。

  2. 请求屏障(RolloutController 私有状态):独占 freeze → 全部 seal → 全部 drain → 暂停后端 → 训练/保存/同步 → 全部 install → 全部 resume → reopen 全协议;原独立 RequestPublication 类已删除,controller 只保留 pause_generation / continue_generation(model_step) / assert_request_gates_healthy 入口。

  3. 请求门禁(保留 RequestGate):仅当 backend_ended AND terminal_recorded 才释放 lease;未知结局(断连/取消/未识别错误)保持 active 并毒化后续 drain。LMDeploy 已识别的预生成 context rejection 视为安全失败;SettlementReceipt 只保留 backend_ended/outcome。

  4. Affinity(合并为一个 SessionAffinityRouter):线程、loop、HTTP client、bindings、退休墓碑同一 owner;session 固定到具体 DP entrypoint,失活绑定 503 不改绑,退休 session 410,POST 不重放;probe 使用自有 session,发布后执行。

  5. 跨窗口 Agent 生命周期:_PendingTasks 仍是唯一 live task owner;策略以 retain_live_tasks 选择“跨窗口保留 + claim ready”或原 pause/drain;close 只 cancel 一次并等待原 Runner.finally 清理。

  6. 真实取消边界(F1/F2):新增 await_cancellable_tasks,持有原执行(本地 Task 或 Ray ObjectRef)。取消时本地 task.cancel()、远端 ray.cancel(force=False, recursive=False),等全部终态与 Runner finally 结束后再传播 CancelledError;普通异常同样等齐 sibling 再抛出。router 在远端终态后才释放 worker。

  7. 完整组交付(S2):BaseProduceContext.produce_group 拥有 sample → generate → 校验/入队或丢弃清理 → 统计,返回 bool;claim/pause 只读终态,不再二次 put。非共卡用 _produce_pass 确认真实调度轮次退出,删除 _producer_idle/_group_admission/scheduling_paused;共卡用 manager 持有的稳定 ProduceProgress 跨窗口消费增量,并动态推进 consumer_step。Sync 保留原局部 pending/补组循环,仅在 finally 结清剩余组。

  8. Provenance:lease 显式传入 on_response;export_training_trace(expected_request_ids=..., require_routed_experts=...) 集中校验 prompt-only 首节点、完整后缀版本、路径/记录 ID 子集与跨分枝提交、专家齐全;response_model_steps 只表示连续后缀版本。

  9. 共享引用:TokenizedSegment / RolloutState 增加 routed_experts_shared;共享清理只 detach(Ray 引用计数保活),legacy 保留显式 free。

  10. 生成计数:generate_group 记录本窗口 suffix 增长 new_response_tokens;provenance 已版本化全部 token,不能再由版本数组长度推导生成量。

  11. Checkpoint 格式:两种 manager 的 manager_state 写入 group_task_lifecycle_version=1,恢复前先校验;旧格式直接拒绝,不补字段、不转换。

删除/未引入:LoopThread、ColocateKeepaliveManager、AgentTurnPausePolicy、DrainPendingPausePolicy、SessionAffinityService、_producer_idle、_group_admission、scheduling_paused。

行为与配置变化

  • 配置收敛到一处:RolloutConfig.cross_weight_update_rollout = CrossWeightUpdateRolloutConfig(wait/freeze/drain_timeout) 与 RolloutConfig.session_affinity;删除 manager 的 turn_keepalive* 与 disagg 的 cross_weight_update_rollout_*;cross-weight-update 模式下 SessionServer 构造即关闭准入。
  • cross-weight-update 模式仍然拒绝在线评测(沿用 [Feature] Retain Agent tasks across complete-request weight updates #2297 的策略),构建时一次性校验固定拓扑、provenance、禁用 token partial rollout、恢复需 optimizer states。
  • _xt_request_id / _xt_model_step 不再作为控制通道;客户端传入的这些字段会被剥离,lease 由 SessionServer 显式传递。
  • RolloutController.continue_generation 在 turn 模式下必须携带真实 model_step。
  • AgentInSandboxLoopConfig 新增 require_turn_provenance / retire_route_on_completion。
  • turn 模式 checkpoint 必须包含 replay buffer;完整组 checkpoint 使用新格式标识,旧存档直接报错。

提交与阅读顺序

commit 内容
49aec162 feat(rollout): gate and publish complete-turn rollout versions 门禁、发布、affinity、trace 数据模型与 provenance 导出、共享引用
d08214c4 feat(rl): retain live agents across production windows 策略 retain/pause/close、manager admission/idle、生成 token 计数(后续提交删除 quiesce)
757016d0 feat(rl): drive the whole fit on one asyncio loop trainer 单 loop 编排、配置统一与校验、trainer 单测更新
42c02bcc refactor(rl): construct loop-owned components in init 构造位置对齐原流程:__init__ 统一构造组件,_fit 只保留异步恢复/首次 I/O/生产/保存/关闭
8ca6381e fix(rl): cancel real Agent execution and wait for all cleanup F1/F2:await_cancellable_tasks、组内 sample 全量清理、Ray ObjectRef 取消与 router 释放
9897fa22 refactor(rl): fix the strategy mode at build time and delegate pause S1/A1:retain_live_tasks 构建期确定、manager 委托 pause_produce
ce5f38dd refactor(rollout): centralize the complete-turn decision and split guards S3/S4:_is_complete_turn、两字段结算收据、长条件分阶段
345953c5 refactor(rl): stop disagg production through status and update_event S2 过渡:删除 _group_admission/scheduling_paused,复用 status/update_event,保留 _producer_idle
0856ef0f feat(rl): deliver complete groups once and settle disagg rounds by Task S2 完整:完整组 Task 一次交付、_produce_pass、共卡稳定账本、Sync 局部收尾、checkpoint 版本 1
b6a398b0 refactor: clarify the colocated progress ledger as next_consumer_step 6.1:ProduceProgress.consumer_step → next_consumer_step;ProduceContext.consumer_step 公开属性保留
3e6dc7d4 refactor: rename turn_resume to cross_weight_update_rollout 6.4:TurnResumeConfig → CrossWeightUpdateRolloutConfig,RolloutConfig/manager 成员同名替换;测试文件同步改名
8051f574 refactor: drop rollout quiesce and allow non-exact checkpoint state 6.3:删除 strategy/manager/trainer 的 quiesce 与 quiesce_timeout;turn 模式 pending 存档改为 warning,明确非严格数据恢复契约
091591db refactor: make model_step mandatory and move request barriers into the controller 6.2:continue_generation(model_step) 必填;RequestPublication 并入 controller 的 RolloutRequestState 私有协议

API 简化(2026-11-19 review 6.1–6.4)

  • 6.1:共卡账本字段改为 next_consumer_step,与“取走第 n 步 batch 后应为 n+1”语义一致;context 对 strategy 仍暴露 consumer_step。
  • 6.2:continue_generation 必填真实 rollout model_step,manager 不再区分有无参调用;请求门禁的暂停/发布状态移入 RolloutController,删除独立 RequestPublication 类。
  • 6.3:删除 quiesce:checkpoint/final 不再等待全部 live Agent;仍保留 gate drain 的 LLM 请求安全屏障。turn 模式允许保存带未完成组 Task 的 checkpoint,但明确 live Agent/工具状态不落盘,冷恢复不保证样本序列和统计严格一致。
  • 6.4:配置与同义成员统一为 cross_weight_update_rollout / CrossWeightUpdateRolloutConfig,表达“rollout 跨权重更新继续执行”;request_publication.py 相应改名为 cross_weight_update_rollout.py,控制器屏障测试改名为 test_rollout_request_barrier.py。

端到端实验(简要)

环境:8×GPU 共卡(RLColocateTrainer),3 步,GLOBAL_BATCH_SIZE=8,PROMPT_REPEAT_K=8,开启 cross_weight_update_rollout + session_affinity + require_turn_provenance + retire_route_on_completion,关闭 DCP/HF/eval。代码版本 0856ef0f(rebase 到 589fd88b),运行 20261117_140024。

以下 e2e 在 API 简化前完成;随后 4 个 6.1–6.4 提交为行为等价重构,由单测覆盖,未重跑 e2e。

  • 结果:3/3 step 完成,进程 exit 0;权重分别在 step 1/2 后同步并发布版本 1/2,末步不再 onload。
  • 逐步 rewards/mean:0.2333 / 0.2807 / 0.2075;raw_rewards/mean 0.2179 / 0.2400 / 0.2059;advantages/mean −0.0069 / −0.0030 / +0.0037;invalid_train_groups=0。
  • batch:78 / 75 / 75 样本;response_len mean 14.3k / 14.8k / 10.8k。
  • 窗口耗时:produce 780s / 498s / 1025s;training 183s / 147s / 128s;sync_weight 22.9s / 17.0s;save 15.5s / 9.2s / 8.8s;step 合计 1003s / 674s / 1167s。
  • 门禁与发布故障:Request publication deadline / Backend resume failed / Backend pause failed / Unsettled requests 均为 0。
  • 沙箱 attempt:194 completed / 38 failed(约 16.4%),错误为 agent_chat/entry_exception、rc_missing、rule_grader/judger 等,均由 producer 标记 FAILED 并补采;末步 buffer 余 4 个已完成组(超发余量,未消费)。
  • 与历史 run 对照(同一配置):rebase 前 553dce3e 为 0.4194 / 0.1000 / 0.2000;592cabd0 为 0.3871 / 0.1020 / 0.2679,均处于同一量级。
  • 单测:API 简化后 tests/rl 全量 588 passed / 19 skipped;15 个失败均因本机环境(缺 QWEN3_4B_PATH/QWEN3_5_MOE_PATH、GPU/Ray 启动、缺 pytest-asyncio 插件),其中 4 个在未合入本分支的 base 上同样失败。仓库 pre-commit(ruff/docformatter/codespell/mypy 435 files/pydantic-extra-check)通过。

已知未覆盖:非共卡 RLDisaggregatedTrainer 未跑端到端(启动、pause、发布与 close 由单测覆盖);本基线不含 frozen MTP 等 preview 后续能力;本次未产出评估集指标(turn 模式禁用在线评测)。

已知 TODO:超时语义(来自 #2297 review)

本次不改变行为,仅在代码中留下 TODO(timeout-reconciliation):

  • drain_timeout:RequestGate.drain 到期会写入永久 barrier failure 并抛错,即使已准入请求随后结算完成,install/reopen 仍失败并终止训练。后续应保持 gate 关闭、拓扑冻结,在同一 barrier 内继续等待/核对;只有确认未结算请求或后端故障才失败。
  • freeze_timeout:同时约束 health recovery、SessionServer 获取、seal,以及 install→resume→reopen 的共享预算;任一超时都会终止训练。ray.get 超时不会取消远端 RPC,seal/reopen 可能只完成部分入口。后续应区分等待预算耗尽与真实故障,保留 barrier/阶段/未完成 RPC,按入口核对状态后再推进,不能盲目继续或重跑 pause。
  • wait_timeout:只限制已准入请求在 gate 关闭期间的等待,到期只让该 Agent 请求失败,不会终止训练;后续可结合 agent/model 请求预算再校准。

已知 TODO:协议与流式(来自 #2297 review)

  • Anthropic 格式:SessionServer 当前只接受 OpenAI chat completions,Anthropic /v1/messages 被拒绝。已加 TODO(anthropic-format):扩展 gate 准入、请求归一化与 trace 组装以支持 Anthropic 格式。
  • Stream 模式:当前要求非流式 traced generation;已加 TODO(stream-support):支持流式响应,同时在释放 lease 前完成“backend ended + trace committed”的完整轮次结算。

合并方式


原始 PR 描述(保留)

原始 PR 分支已备份为 backup/agent-rl-stack-04-agent-core-20261120(6b6439f0)。


Long tool calls can span a training update. Keep unfinished Agent tasks with their existing producer, seal and settle admitted model requests before weight changes, publish the new version, then let the same Agent continue. Preserve request IDs, historical token logprobs and routed-expert data; bind external sessions to a concrete inference entrypoint. Quiesce live tasks before full checkpoints and start cold restore with admission closed.

Review scope and dependencies

Layer 4/6 of the mechanical split of #2257. Base: xs1997zju/agent-rl-stack-20261111/03-fit-cleanup; reviewed head: f5ee2bebdfd466467c032f62f246cfc72afb2cb8. Review this PR against its configured base; later layers depend on earlier layers. The original #2257 remains unchanged as the pre-split reference.

Code entry: xtuner/v1/rl/rollout/request_gate.py; controller.py; session_server.py; affinity_router.py; trace_store.py; agent_loop_manager/colocate_keepalive.py; train/rl_trainer.py.

This layer contains five separate review commits: request provenance, request settlement/version barriers, session affinity, task retention, and checkpoint/cold publication. They form one minimum integration group because the existing implementation has cross-topic enable/cleanup dependencies. Intermediate commits are not supported standalone feature versions.

Validation

  • Reassembling all six layers reproduces the approved runtime tree exactly. The only approved difference from original [Draft] Retain native Agent tasks across complete-turn weight publications #2257 is removal of the optional tokenizer-import compatibility fallback; no other runtime patch was added during splitting.
  • Existing protocol/lifecycle CPU checks: 130 passed on the final assembled tree. Shared tests/documentation are carried once in layer 6.
  • The assembled tree plus native frozen-MTP [Feature] Freeze native MTP parameters while preserving load, save and rollout publication #2292 completed an eight-H200 quick regression: two real optimizer updates, frozen dedicated MTP parameters, IPC weight synchronization, real requests at published versions 0/1/2, one controlled live task continuing across an update with the same session binding, and full Trainer DCP cold restore followed by a real request.
  • GPU data came from a compatible intact recorded training group; tool waiting was controlled. This is final-combination evidence, not an independent GPU run of this layer, natural CyberGym reward learning, or a 64-GPU acceptance run.

Reviewer guide and independent review

Developer guide, diagrams, scripts and experiment scope.
A real Kimi XT Review of the completed split and evidence is posted in this PR's comments. It found no mechanical-split blocker; original #2257 compatibility/design observations remain explicitly recorded. This Draft does not include fixes for those observations.

Related independent PRs: #2252 (iterative trace traversal; overlap must be reconciled before landing the core layer), #2292 (native frozen MTP). Lagent #385 targets v1.0.0; the CyberGym recipe is separate in CRG !78. The external MTP route #2289 is not part of this delivery. Keep Draft; no automatic merge.

@xs1997zju

Copy link
Copy Markdown
Collaborator Author

Independent Kimi XT Review — stack layer 4/6

This PR carries the 04-agent-core part of the reviewed split. Its exact head is f5ee2bebdfd466467c032f62f246cfc72afb2cb8. The report below independently reviews the completed six-layer result and its validation evidence; it is not a claim that this intermediate layer was separately GPU-tested or independently reviewed in a new Kimi session.

Real CLI model: kimi-code/kimi-for-coding; session: session_1637feb0-6b91-47ae-a09d-bb5ab269d418; process exit 0 and persisted session verified. Reviewed assembled head: 056e95c86a11318db22aafb69cfa6fbf1d9c31f6; base f1fddc506a49b7d0aa71554e26908885b37211cc. The public linear stack tip e42ad7b4705cbc9b1822f08d3f61652ed3e5c588 has the same tree as that assembled merge result.

Parent readback, separate from Kimi: the Python 3.10 compatibility observation is confirmed statically; the configured test environment uses 3.12. No long-run R3 leak was demonstrated, and shared ownership must be respected before adding explicit free. Missing-session-ID counting concerns mixed failed/successful groups. replay.full_restore_verified=false was recorded before the separate restore phase; the restore result and aggregate receipt passed, so the raw phase receipt has not been rewritten. No fixes were made in this mechanical split.

Developer guide · Retained public report copy


Public copy of the real Kimi report. Only local run-directory labels were normalized; findings and conclusions are unchanged. The original report and CLI session are retained locally.

review_2257_2026-11-11_21-26

审阅对象:#2257 本地拆分实际执行结果(base f1fddc506a49b7d0aa71554e26908885b37211cc → merged head 056e95c86a11318db22aafb69cfa6fbf1d9c31f6)。本次为独立审阅:所有机械与代码结论均由本审阅在 xtuner/ 仓库内用只读 git/源码核对得出,未继承任何前轮审阅意见;MERGED_TO_EXPECTED.diff 为空、报告中的 PASS 措辞未作为证据采信。

仓库内不存在 GLOSSARY.md 与 docs/adr/(已检查仓库根与 xtuner 子目录,明确标明:无)。架构评估使用 improve-codebase-architecture 的 module/interface/depth/seam/leverage/locality 词汇,未启动其交互 loop,未生成架构 HTML,未改 glossary/ADR。


summary

机械拆分(独立复核,全部静态确认,无阻断项)

我在 xtuner/ 内亲自执行了以下核对(非引用报告措辞):

  1. git diff cdcaefd4cbc45ae01739e6615ee8e41046af49c5 HEAD 为空 → 最终树与事先冻结的 expected_target 逐字相等。
  2. git diff 43d62b6e1cbc59aea24c3c2ac5c7d65a90c8ecf4 HEAD 只触及 xtuner/v1/datasets/mllm_tokenize_fn/__init__.py 一个文件;该文件与 base 的 diff 为空 → 唯一例外 = 批准的 tokenizer import fallback 排除,其余 xtuner/ 与原 head 无 diff(含新文件/mode)。
  3. 六个 merge commit(750b3b0a/9a32b6ed/9f892568/147c48a1/97efe2b5/056e95c8)逐一 git cat-file -p 核对:均为双 parent、no-ff,第二 parent 依次为 T1-admin、T1-sampler、T1-finally、T5-checkpoint(04 组 tip)、T6-debug、peripheral,与 MERGES.json 一致;第一 parent 链为线性 stack。线性历史上恰为 10 个主体主题 commit + 1 个集中 tests/docs commit,全部落在 base 之上。
  4. 逐 commit --stat 与 COMMITS.json/HUNK_LEDGER.json 的文件归属一致(166 个 hunk 全部有主题归属,EDITS_UNASSIGNED.json 实为归属前池,最终全部分配)。04 组五主题(T2-trace/T2-gate/T3-affinity/T4-lifecycle/T5-checkpoint)在组内是独立 review commit,但 stack 第 04 层只 merge 组 tip —— 与“中间 commit 是 review 单元、不承诺独立运行;04 组五主题共同合入”的口径一致,不是隐藏行为。
  5. 未夹带额外提交:base..HEAD 仅含上述 17 个 commit(静态确认)。
  6. 原副本保护:ORIGINAL_SOURCE_PRESERVATION.json 声明 head/refs/status/index 字节不变;本次审阅未对原副本做任何写操作(静态确认其存在且未被我触碰)。

六层 stack/合入组合理性:T1 三个小修复各自独立成层合理;T2-gate 依赖 T2-trace 的 _xt_request_id 注入与 provenance 字段,T3 的 retire 被 T2 的 probe 调用,T4 的 policy 调用 T2 的 prepare_pause/resume_requests,T5 的冷恢复发布复用 T2 barrier + T4 policy + T1-sampler —— 从实际代码调用关系看 04 组合组确有真实跨主题启用/清理依赖,不为强拆增 guard/no-op 的处理可以接受。tests/docs 集中一层携带符合“外围不要求一致、不为之增拆分负担”的约定。

主体实现(base→merged,~2600 行源码):这是原 #2257 的实现本体,拆分未增删实现。因此实现层面的缺陷绝大多数属于原 2257 已有设计,不是本轮回归;下文章节严格区分。未发现高严重度的新增实际风险。

高严重度实际风险:无(静态确认范围内)。

本轮拆分新增问题:仅 1 项低severity记录性事项(见其他 issues O1:运行级 ACCEPTANCE 内 replay.full_restore_verified=false 与顶层 full_Trainer_cold_restore=true 措辞不一致;restore 段自身证据完整,属措辞而非证据缺口)。

已有设计建议(原 2257 带入,不阻塞本轮):见“其他 issues”O2–O8,包括 Python 版本声明与 asyncio.timeout 不匹配、request-aware R3 显式 free 缺失、disagg _fit finally 行为变化、min_valid_attempts 计数口径等。

GPU/validation 证据可支撑结论的边界:GPU 实测树是 split + 固定 #2292 的叠加组合(validation-xtuner HEAD 414c0f43,我在该仓库核对 git log/diff,叠加仅限 4 个 runtime 文件 + #2292 自带 tests/docs),不冒充纯 2257 树;该证据可支撑“叠加组合上 2 次真实 optimizer 更新、版本 0→1→2 发布、affinity、完整 DCP 冷恢复”等结论,不可外推为自然 CyberGym rollout、恢复后第三次更新、64 卡规模或任意单主题 commit 的独立验收。回放 fixture 路径(兼容 PT 未改字段、native loader CPU 预检)符合快速工作流,不要求重跑完整自然 rollout 是合理的;遗漏路径已在上文与 O9 明确。


Main Flowchart before this PR

改动前(base f1fddc50):共卡/异卡 RL 训练的原生 pause/resume 流程,Agent 执行随暂停被清理。

flowchart TD
    subgraph ExternalClient["外部 Agent 客户端 (HTTP)"]
        EC1["POST /v1/chat/completions (OpenAI/Anthropic, 可 stream)"]
    end
    subgraph SessionServer["xtuner.v1.rl.rollout.session_server.SessionServer"]
        SS1["_handle_request(request)"]
        SS2["_forward_request(request)"]
        SS3["on_request / on_response hooks"]
        SS4["trace_store insert (TokenizedSegment, Trie)"]
        SS1 --> SS2 --> SS3 --> SS4
    end
    subgraph RolloutController["xtuner.v1.rl.rollout.controller.RolloutController"]
        RC1["pause_generation() / continue_generation()"]
        RC2["health_manager.pause/resume"]
        RC1 --> RC2
    end
    subgraph Producer["xtuner.v1.rl.agent_loop_manager.producer.AsyncProduceStrategy"]
        P1["_local_pending_tasks: set[asyncio.Task] (每次 produce_batch 重置)"]
        P2["pause_pending_tasks() (取消并收尾 pending)"]
    end
    subgraph ManagerC["xtuner.v1.rl.agent_loop_manager.agent_loop_manager.AgentLoopManager (共卡)"]
        MC1["produce_batch() -> continue_generation -> 生产 -> pause_generation -> pause_produce"]
    end
    subgraph ManagerD["xtuner.v1.rl.agent_loop_manager.disagg_agent_loop_manager.DisaggAgentLoopManager (异卡)"]
        MD1["produce_loop() / pause_produce() / continue_produce()"]
        MD2["DisaggAsyncProduceStrategy._pending_tasks"]
    end
    subgraph Trainer["xtuner.v1.train.rl_trainer.RLColocateTrainer / RLDisaggregatedTrainer"]
        T1["fit() -> produce -> _train_one_batch -> fit同步"]
        T2["sampler save/resume (epoch-local step 直接作 global 计数)"]
    end
    EC1 --> SS1
    MC1 --> RC1
    MD1 --> RC1
    T1 --> MC1
    T1 --> MD1
    T1 --> P2
Loading

Main Flowchart after this PR

改动后:新增 RequestGate 准入/结算屏障、affinity 路由、任务保活与 quiescent 保存恢复。突出显示为新增/修改节点(mermaid 8.8.0 classDef changed)。

flowchart TD
    subgraph ExternalClient["外部 Agent 客户端 (HTTP)"]
        EC1["POST /v1/chat/completions (非 stream, 带 session_id)"]
    end
    subgraph AffinityRouter["xtuner.v1.rl.rollout.affinity_router.SessionAffinityRouter / SessionAffinityService (新增)"]
        AR1["acquire(sid)->具体 DP entrypoint; retire_sessions(); 410 tombstone"]
    end
    subgraph RequestGate["xtuner.v1.rl.rollout.request_gate.RequestGate / SettlementReceipt (新增)"]
        RG1["request(session_id) 租约: seal -> drain -> install_version -> reopen"]
        RG2["未结算/未知结局 => poison barrier, 阻止权重更新"]
    end
    subgraph SessionServer["xtuner.v1.rl.rollout.session_server.SessionServer (修改)"]
        SS1["_handle_request -> 生成端点先进 RequestGate (修改)"]
        SS2["_forward_request(request, lease) (修改: 注入 _xt_request_id/_xt_model_step)"]
        SS3["结算: backend_ended+terminal_recorded => SettlementReceipt (修改)"]
        SS4["TokenizedSegment.request_id/model_step (修改)"]
        SS1 --> RG1
        SS1 --> SS2 --> SS3 --> SS4
    end
    subgraph RolloutController["xtuner.v1.rl.rollout.controller.RolloutController (修改)"]
        RC1["prepare_pause() freeze+seal+drain (新增)"]
        RC2["resume_requests() install->continue_generation->reopen, 单 deadline (新增)"]
        RC3["assert_request_gates_healthy() (新增)"]
        RC4["retire_route_sessions() (新增)"]
    end
    subgraph PausePolicy["xtuner.v1.rl.agent_loop_manager.agent_turn_pause_policy.AgentTurnPausePolicy (新增)"]
        PP1["pause/resume/check_requests/before_sync(quiesce)"]
    end
    subgraph Keepalive["xtuner.v1.rl.agent_loop_manager.colocate_keepalive.LoopThread / ColocateKeepaliveManager (新增)"]
        K1["专用 event loop 跨同步训练窗口保活 Agent 任务"]
    end
    subgraph Producer["xtuner.v1.rl.agent_loop_manager.producer.AsyncProduceStrategy (修改)"]
        P1["_PendingTasks 统一持有 pending; retain_live_tasks 跨窗口保留"]
    end
    subgraph ManagerD["DisaggAgentLoopManager (修改)"]
        MD1["quiesce(timeout) 关门收单; prepare_sync(checkpoint_due)"]
        MD2["scheduling_paused 停新组; close() 取消 pending"]
    end
    subgraph Trainer["RLColocateTrainer / RLDisaggregatedTrainer (修改)"]
        T1["_validate_turn_keepalive/_validate_turn_resume 配置校验 (新增)"]
        T2["_initialize_keepalive_rollout 冷启动/冷恢复发布 (新增)"]
        T3["_checkpoint_due + produce_batch(quiesce=...) (修改)"]
        T4["debug_rollout/debug_train 异卡阶段复用 (新增)"]
        T5["sampler rollout_group_count/rollout_next_epoch (修改)"]
    end
    EC1 --> AR1 --> SS1
    RG2 --> RC1
    PP1 --> RC1
    PP1 --> RC2
    K1 --> PP1
    MD1 --> PP1
    T1 --> K1
    T2 --> RG1
    T3 --> MD1
    classDef changed fill:#ffe08a,stroke:#b58900,stroke-width:2px;
    class AR1,RG1,RG2,SS1,SS2,SS3,SS4,RC1,RC2,RC3,RC4,PP1,K1,P1,MD1,MD2,T1,T2,T3,T4,T5 changed;
Loading

文字对应说明(流程 → 修改代码):

  • 外部请求先进 affinity(新增 affinity_router.py,RolloutProxyManager.replace_registered_session_urls 改注册 affinity 入口,acquire/retire_sessions)。
  • SessionServer 生成端点经 RequestGate.request 租约准入(session_server.py:_handle_request 修改;_SESSION_SERVER_ONLY_KEYS 增加 _xt_request_id/_xt_model_step;_forward_request 签名加 lease)。
  • 结算:backend_ended(完整响应或 LMDeploy 上下文拒绝)+ 终态记录 → SettlementReceipt,否则 poison(session_server.py 租约块;request_gate.py)。
  • 发布:RolloutController.prepare_pause/resume_requests(seal→drain / install→continue_generation→reopen,单 monotonic deadline;barrier 校验拓扑不变)。
  • 保活:共卡 ColocateKeepaliveManager+LoopThread 使 _PendingTasks 中的活任务跨同步窗口存活(producer.AsyncProduceStrategy._local_pending_tasks → _PendingTasks,retain_live_tasks);异卡 DisaggAgentLoopManager.quiesce + AgentTurnPausePolicy.before_sync 在 checkpoint 前关门收单。
  • 保存/恢复:Sampler 持久化全局 group 计数与逻辑 epoch;RLColocateTrainer._initialize_keepalive_rollout 先封门→恢复→同步→发布版本;RLDisaggregatedTrainer._resume_agent_loop_manager 前置 prepare_sync(False)+pause_produce。
  • 顺带小修复:proxy admin connect 重试(misc._post_proxy_admin)、fit 失败释放训练 batch(_release_train_batch_resources 进 finally)、sampler epoch 边界(T1)。
  • 可选项:min_valid_attempts 过滤与 agent_sandbox_env_id 元数据(T7)。

核心原理

  1. 准入-结算屏障:RequestGate 用 asyncio.Condition 维护 _open/_epoch/_active/_failures。每个生成请求持有 RequestLease;请求结束必须留下 SettlementReceipt(backend_ended, terminal_recorded)。任何“未知后端结局”(超时、断连、取消、trace 提交失败)都把 gate 置为失败态,此后 drain/install/reopen/新准入全部拒绝 —— 权重更新只发生在所有准入请求都“确定结束”之后。LMDeploy 的“input length >= context length”400 是唯一被白名单化的“生成前拒绝”,视为安全结束的失败尝试(session_server._is_lmdeploy_context_rejection,正则钉死 pinned LMDeploy 的 payload)。
  2. 版本发布协议:prepare_pause 冻结拓扑→向所有 entrypoint 发 seal→drain(一个 deadline);训练/同步后 resume_requests 校验 barrier 与 entrypoint 集合不变 → 所有 entrypoint install 新版本 → 所有后端 continue_generation → 所有 entrypoint reopen,全程共享一个 monotonic deadline。任一失败保留 barrier、不开门(fail-closed)。
  3. 跨窗口任务所有权:活 Agent 任务始终由 producer 既有的 _PendingTasks 拥有。共卡模式用独立线程事件循环(LoopThread)在同步 GPU 训练期间维持工具/HTTP 客户端推进;retain_pending_tasks 只收割已完成任务,不取消活的。异卡模式 quiesce 停止新组、等生产者收单、在 gate 仍开时让 episode 自然收尾。
  4. trace/R3 共享所有权:带 request_id 的 TokenizedSegment 与 rollout state 的 routed_experts 为共享借用 —— RolloutState.reset_response/free_rollout_state_refs 与 Trie.release 都跳过 request-aware 引用的显式 free,避免训练侧仍借用时释放。
  5. 冷恢复发布:冷启动/恢复的 SessionServer 以 turn_resume_initially_closed 出生(配置校验器设置),权重与 manager 状态就绪后才 publish 实际版本,杜绝“版本 0 准入旧权重引擎”。

核心实现与对应的单测

核心实现 对应单测 原理核验
request_gate.py RequestGate 租约/seal/drain/install/reopen tests/turn_resume/test_request_gate.py(7 项) 静态读码确认 poison 语义、TimeoutError 不计 inflight、epoch 幂等;与测试断言一致
session_server.py 租约接线+结算+LMDeploy 拒绝白名单 tests/turn_resume/test_forwarding.py(10 项,真实 aiohttp TestServer) 关键细节已独立复核:trace 失败时 backend_ended=complete=True(complete 不含 session_error_msg 条件),故 Receipt(True,True,"failed") 是 safe —— drain 可通过,符合“安全失败尝试”设计;response.body 覆写 id 走 aiohttp Response.body setter(同步 content-length),测试覆盖
controller.py prepare_pause/resume_requests tests/turn_resume/test_policy.py::ControllerTests(5 项) seal 全发再 drain、install→continue→open 顺序、deadline 耗尽不开门、拓扑变化拒绝发布,均有断言
affinity_router.py tests/rl/test_affinity_router.py(3 项,真实 loopback HTTP) 绑定跨 publication 不动、backend 失效 fail-closed、tombstone 410、无重放
colocate_keepalive.py LoopThread/ColocateKeepaliveManager tests/rl/test_colocate_keepalive.py::OwnerTests/StartupTests 主线程被同步训练阻塞时任务仍在 owner loop 推进;恢复/同步失败保持关门并关闭 owner
agent_loop_manager.py produce_batch quiesce 路径 test_colocate_keepalive.py::BoundaryTests quiesce 时 gate 开着收单、drain 失败不回原生训练
disagg_agent_loop_manager.py quiesce/close/prepare_sync tests/turn_resume/test_quiesce.py(4 项) 不丢不取消活 episode、_producer_idle 等待、超时安全
sampler.py 全局计数/epoch tests/turn_resume/test_sampler_resume.py 新旧 checkpoint 均恢复后缀与唯一 id(含真实 ParallelSampler shuffle)
trace_store.py request-aware 引用/三版本导出 tests/turn_resume/test_trace_provenance.py、test_main_compatibility.py 三版本 append 不替换旧 route/logprob、冲突写拒绝、release 不递归、普通 ref 释放而借用保留
misc._post_proxy_admin tests/rl/test_proxy_admin_connect_retry.py(AST 抽取真函数) 仅 connect timeout 重试 5 次、read timeout 不重试
配置校验器 _validate_turn_keepalive/_validate_turn_resume test_colocate_keepalive.py::CompatibilityTests gate/keepalive 成对、fixed topology、禁评估、resume 约束
checkpoint 物化(routed_experts/replay/sampler) CheckpointDataTests DCP payload 中 SerializedRayObjectRef、恢复后哈希一致、replay 未持久化报错

静态确认:以上测试与实现逐一对得上,且 CPU 验收(130 项,host 环境)覆盖的是这些文件。

抽象与信息隐藏评估

(按 module/interface/depth/seam 词汇)

  • RequestGate 是深 module:接口 5 个异步操作 + 1 个 snapshot,隐藏了租约字典、epoch 校验、poison 传播全部复杂度;SettlementReceipt 是不可变值对象。leverage/locality 好,测试通过接口完成。这是本次最好的抽象。
  • SessionAffinityRouter 同为深 module:acquire/set_backend/retire_sessions/replace_backends/handle/start/stop 背后隐藏 inflight/assigned 计数、tombstone、流式转发与“绝不重放 POST”的语义。SessionAffinityService 是独立事件循环 adapter,seam 清晰(proxy_manager 通过 invoke 调用,可替换)。
  • AgentTurnPausePolicy / DrainPendingPausePolicy 构成一个真实 seam(两个 adapter:原生 drain 策略与 turn-resume 策略),DisaggProduceStrategy.configure_turn_resume 以“不支持的策略在构造期拒绝”保护启用路径 —— 多态消分叉做得对(满足新增功能守则 R3)。
  • ColocateKeepaliveManager 偏浅:__getattr__ 透传 + 4 个具名方法转发(produce_batch/save/resume/prepare_initial_sync/publish_initial_version),接口几乎等于实现。但它确实是“第二个 adapter + 线程所有权”的薄壳,删除它会把线程所有权逻辑散回 trainer —— deletion test 勉强通过。可接受,见“单测建议”S4。
  • 瑕疵(低风险,既有):AgentLoopManager.produce_batch 的 keepalive 分支直接伸手 task.produce_strategy._pending_tasks(私有字段),且 ColocateKeepaliveManager cleanup 也遍历 _pending_tasks —— 生产策略的内部结构泄漏到 manager 层(违背 R2)。T4 主题内一致,非本轮新增。

新增功能守则

对既有类的接触点核查(T2–T7 相对既有类):

  • AgentInSandboxLoop(既有类):新增 3 个构造参数 + generate_sample finally 中 1 次具名调用(retire_route_sessions)+ _build_rollout_states 内 require_turn_provenance 分支(约 25 行实现细节:versions 长度校验、request_ids 匹配、trace 校验)。R1 满足(≤1 字段级接触 + 1-2 次调用),R2/R3 部分违反:_build_rollout_states 内的 provenance 校验块把字段名(token_model_steps/request_ids/routed_experts)与 trace 语义写进了既有类。伪代码改进(后续可选,非本轮):
    # 下沉到新增的 provenance 校验器,既有类只留一次调用
    if self.require_turn_provenance:
        segment_state = self._turn_provenance.attach(segment_state, data, item, rollout_state)
  • RolloutController:新增 5 个公开方法 + 2 个字段,既有方法仅 3 处小改(pause_generation 的 barrier 抛错、continue_generation 的条件 resume、shutdown finally 关 proxy)。R1/R3 满足。
  • SessionServer:构造参数 +2、_handle_request 前置 gate 分支、_forward_request 加 lease 参数与 3 处小挂钩、_on_response_worker(新 _xt_request_id 终态校验)。既有流程未按新特性分叉成两套 —— 租约为 None 时行为不变,R3 满足。
  • AsyncProduceStrategy:_local_pending_tasks set → _PendingTasks 是行为中性重构(普通路径语义不变,retain_live_tasks=False 默认),接触点合理。
  • RolloutProxyManager:affinity 分支 3 处 if/else(replace_registered_session_urls/on_worker_group_inactive/on_worker_group_active)轻微违反 R3(null object 可用“无 affinity 时直连注册”的 no-op adapter 消掉);低风险,既有设计。

公开接口的线性业务流程评估

  • RequestGate.request/seal/drain/install_version/reopen 每个方法都是单抽象层级的线性流程,无转发 wrapper;drain 的“等待→超时置失败→抛错”集中一处,locality 好。
  • RolloutController.prepare_pause/resume_requests:prepare_pause = freeze→取 entrypoints→建 barrier→seal→drain,线性;resume_requests 用内部 wait_all 闭包消除 deadline 重复,符合 Composed Method;失败路径统一 raise,无静默吞错。
  • SessionServer._forward_request 仍是长方法(base 已如此),本轮在其上叠加租约/计时/结算约 60 行,进一步拉长 —— 属既有债务加重而非新债;建议(后续)把“租约结算块”抽为 _settle_lease(lease, ...) 私有方法,可隐藏真实规则、提高 locality。
  • AgentLoopManager.produce_batch 的 keepalive 分支插入了 quiesce 内联闭包 finish_live_tasks(25 行),使该方法从“线性叙述”变成双路径 —— 建议(后续)抽 _finish_live_tasks_for_quiesce(...),与 deletion test 一致(删除后复杂度集中在 manager 一处)。

单测建议

现有测试已较完整(上表)。补充建议(均为静态推断,优先级低):

  • S1(对应 O2):若保留 requires-python>=3.10,加 sys.version_info<(3,11) 的兼容层或把声明抬到 >=3.11;最小测试 = import request_gate 后在 3.10 环境构造 RequestGate。
  • S2(对应 O5):_drop_failed_train_samples 在 valid 样本缺少 agent_session_id 且 min_valid_attempts=2 时的组命运(当前会按 1 个独立尝试整组 FILTERED)。
  • S3(对应 O4):长会话下 affinity retired tombstone 集合只增不减的容量行为(max_sessions 触顶 503)。
  • S4(对应抽象评估):为“manager 不碰 _pending_tasks 私有字段”加一个 seam 级测试,倒逼 strategy 暴露 quiesce(ctx)/claim_ready 公开接口。
  • S5:RLColocateTrainer.fit cleanup 失败 + 原始异常并存时 cleanup 异常被吞的断言(当前无测试覆盖该组合)。

其他 issues

O1(低,静态确认,本轮拆分新增):recorded-run/ACCEPTANCE.json 的 replay.full_restore_verified=false 与同文件 restore.passed=true/full_trainer_cold_restore=true 及顶层 QUICK_GPU_ACCEPTANCE.full_Trainer_cold_restore=true 措辞冲突。实际冷恢复证据在 restore 段(optimizer_step=2、管理器状态、恢复后真实 HTTP 响应)。建议把 replay 段该字段改名为 not_applicable_to_replay_phase 或统一口径,避免读者误读为“恢复未验证”。

O2(中,静态确认,原 2257 带入):Python 版本声明不匹配。pyproject.toml requires-python = ">=3.10",但 merged 树新增 3 处 asyncio.timeout(仅 3.11+):request_gate.py:61,94、disagg_agent_loop_manager.py:309(base 全仓零使用,已用 git grep 于 base 确认)。实测环境为 Python 3.12(已核对解释器版本),故未暴露;3.10 用户启用 turn-resume 即 AttributeError。建议抬声明或加 async_timeout 兼容。非本轮回归,不在拆分中修符合口径。

O3(低,静态确认,原 2257 带入):异卡 _fit finally(rl_trainer.py:2092-2098)由 await producer_task 改为 cancel + agent_loop_manager.close(),影响所有异卡运行(非仅 debug/turn-resume):普通路径 producer 若仍在批中会被即刻取消而非自然收尾;close() 抛错会掩盖原始异常。T6 主题内行为,建议后续给普通路径保留 graceful 收尾分支。

O4(低,静态确认,原 2257 带入):request-aware 的 routed_experts Ray 引用全链路无显式 free(rl_data.reset_response 跳过、free_rollout_state_refs 置 None 不 free、Trie.release 跳过 request-aware 段),依赖 Ray 分布式 GC 惰性回收。短程无感,长程 turn-resume 训练有 object-store 压力风险。建议后续在 trace 退休链路补一次 owner 明确的 free。

O5(低,静态确认,原 2257 带入):min_valid_attempts 以 distinct agent_session_id 计独立尝试;valid 样本缺该字段时全部归一为 None 只算 1 个,min_valid_attempts=2 会把本可训练的组整组 FILTERED。配置校验未把它与 require_turn_provenance 绑定。recipe 当前显式用 =2 且 provenance 开启,短期不触发;建议文档注明或校验联动。

O6(低,静态确认,原 2257 带入):produce_utils.BaseProduceContext.generate_group 全局(非 opt-in)给每个生成样本写 extra_fields["new_response_tokens"],改变 produced-token 记账口径且该内部键会随 extra_fields 进 replay/checkpoint/debug pt。有单测锁定新口径(test_complete_provenance_does_not_erase_generated_token_count),属设计选择;建议键名加 _ 前缀并文档化。

O7(信息,静态确认):agent_turn_lifecycle.md 明确声明多 entrypoint reopen 非原子、partial RPC 可能留开着的 gate、license/来源(matrix72c/sandbox_rl 参照、Lagent #385 依赖、#2289 不包含)——披露充分,与代码行为一致;维护者需按文档处理来源许可。

O8(信息,静态确认):T2-trace 与 #2252 的 Trie 遍历重叠按“标记重叠、不换实现”处理;Trie.keys/release 的迭代化改写确在 2257 主体内(已读 merged trace_store.py 232-240 行确认无递归),与 #2252 目的重叠但引用语义不同(request-aware 跳过 free),整文件互换会改变 R3 所有权 —— 计划的处理成立。

O9(验证边界汇总,静态确认 + 推断):可支撑——机械相等(我独立复核)、叠加 #2292 组合上 2 次真实 optimizer 更新/冻结 MTP 全量哈希不变/IPC 同步/版本 0→1→2/affinity 绑定/受控任务跨更新续跑/完整 DCP 冷恢复后真实 HTTP;runner(quick_gpu/gpu_jobs/config.py 等)走原生 build_config 校验器并断言 keepalive/冻结/closed gate,fixture PT 未改字段(GPU_PT_COMPATIBILITY_V3.unchanged_fields + sha256)。不可支撑/未覆盖——自然 CyberGym 工具评分路径、恢复后第三次更新、optimizer moments 逐元素审计、64 卡长程、多 DP entrypoint 下的 partial-reopen 故障注入、Python 3.10、每主题 commit 独立运行。这些已在 REPORT/ACCEPTANCE 中如实标注,本审阅认可其边界表述。


审阅执行说明:全程只读(git show/diff/cat-file、源码阅读、JSON 证据读取),唯一写入为本报告。未运行项目代码(依原文第 6 条),未改任何 git refs/index,未 push/评论/merge,未操作 RJob/SSH/权重。中间产物(prompt 快照、SCOPE、diff、日志、metadata)按要求全部保留,未做清理。

@jayhenry

Copy link
Copy Markdown
Collaborator

提出一个替代实现供评估:#2312(Draft)。它的 base 与本 PR 相同,均为 xs1997zju/agent-rl-stack-20261111/03-fit-cleanup,因此可以按完整第 4 层审阅;相对本 PR 的替换增量见 git diff xs1997zju/agent-rl-stack-20261111/04-agent-core..refactor/async-agent-rl-publication(35 files, +2476/−1208)。目前仅是提案,本 PR 与上方 stack 均未改动。@xs1997zju

if self.request_gate is None or not _is_generation_endpoint(request.method, path):
return await self._forward_request(request)
fmt = _detect_format(path)
if fmt != FMT_OPENAI:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要支持 anthropic format

Comment thread xtuner/v1/rl/rollout/session_server.py Outdated

if lease is not None:
complete = (
not is_stream

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stream 模式为什么不算完成?

async with asyncio.timeout(timeout):
await self._condition.wait_for(lambda: not self._active or bool(self._failures))
except TimeoutError:
self._failures["barrier"] = "drain_timeout"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里想确认一下 drain 超时的语义:共卡 rollout pause 时,turn_keepalive_drain_timeout 会传到这里。当前一旦等待到期,就写入永久的 drain_timeout failure 并重新抛出;异常经 prepare_pause → AgentTurnPausePolicy.pause → produce_batch 传到 fit(),执行清理并终止本次训练。即使原请求随后正常结束,该 failure 也不会自动清除,后续版本安装和 reopen 仍会失败。

仅因这次 drain 等待超时就终止整个训练,是预期行为吗?建议不要仅因此结束训练:请求可能只是生成或 trace 提交较慢,仍由原 forwarder/lease 持有。可以将这里的超时作为告警和状态检查间隔,保持 gate 关闭、拓扑冻结,在同一个 barrier 内继续等待原请求结算;全部入口确认 drained 后,再继续训练或权重同步。

真实失结算或后端故障仍需单独处理;不应在等待超时后直接继续训练、重放 POST,或清除真实 failure。

if self.barrier is None:
await controller.continue_generation.remote()
else:
await controller.resume_requests.remote(self.barrier, model_step, timeout=self.freeze_timeout)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里提醒核对 turn_keepalive_freeze_timeout 的全部使用点:它既限制 prepare_pause 中 recovery freeze、获取 SessionServer、seal 的分别等待,又在这里作为 install version → backend resume → reopen 的共享总预算(共卡默认 30s,不包含训练和权重传输)。目前这些等待超时都会向外抛错,使训练退出。

建议不要仅因控制 RPC 返回较慢就终止整训。不过处理时需要注意阶段状态:ray.get 超时不代表远端操作已经取消;seal 超时可能只有部分 gate 关闭,reopen 超时也可能已经有部分入口开放。因此不能简单 catch 后继续下一阶段,也不宜从头重跑 prepare_pause/resume_requests。

能否区分等待预算耗尽与真实故障,保留当前 barrier、执行阶段和未完成 RPC,告警后续等或核对状态,确认所有入口满足当前阶段条件后再推进?

while owner.count():
for ready in await owner.wait_and_claim(timeout_s=1):
await ctx.put_generated_group(ready.result())
await asyncio.wait_for(finish_live_tasks(), keepalive.quiesce_timeout)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里也提醒注意 turn_keepalive_quiesce_timeout 的超时语义:asyncio.wait_for 到期会取消 finish_live_tasks() 这个收单协程,再把异常传到 fit(),导致整训退出;底层活 Agent 不一定随该收单协程一起取消,随后才由训练退出时的清理处理。也就是说,checkpoint/末步时某个 Agent 或工具只是执行较慢,也可能因此结束整个训练。

建议不要仅因收单等待超时就终止整训,可以告警并继续等待。不过续等应保留同一个收单任务,例如用 asyncio.wait 轮询其完成状态,避免反复取消、重建收单过程;尤其需要注意 wait_and_claim 取走结果到 put_generated_group 完成之间的交接,避免漏收或重复入 buffer。

同时,不能 catch 超时后直接跳过 quiesce 去保存完整 checkpoint。仍需确认所有 pending Agent 已进入终态;如果选择延期 checkpoint,也需要显式调整后续保存决策。请一起确认这些场景的预期处理。

@jayhenry
jayhenry force-pushed the xs1997zju/agent-rl-stack-20261111/04-agent-core branch from f5ee2be to 6b6439f Compare November 17, 2026 06:43
@jayhenry
jayhenry added this pull request to stack #2318 November 17, 2026 06:59
@jayhenry
jayhenry force-pushed the xs1997zju/agent-rl-stack-20261111/04-agent-core branch from 6b6439f to 091591d Compare November 20, 2026 06:35
Admission barriers, one owned version publication path, session affinity and
request-exact traces now live in the rollout control plane:

- RequestGate admits one traced OpenAI generation per session and only
  releases a lease after the backend ended and the trace committed; unknown
  exits poison the barrier instead of allowing a weight update.
- RequestPublication freezes topology, seals/drains every SessionServer,
  pauses backends, installs the next version, then reopens admission. The
  controller only routes its existing pause/continue/health entrypoints.
- SessionAffinityRouter owns its thread, loop and HTTP client; sessions stay
  bound across updates, dead bindings fail closed with 503, and retired
  sessions get 410 without replaying a POST.
- Trace nodes carry their request identity and model step, and shared
  routed-expert references survive trace release through Ray reference
  counting while legacy refs keep the explicit free contract.

SessionServer no longer accepts client-forged _xt_* identity; the lease is
passed explicitly to the trace write path.
The production strategies now own the lifecycle of retained Agent tasks so a
turn can span weight updates:

- AsyncProduceStrategy and DisaggAsyncProduceStrategy keep their existing
  _PendingTasks as the single live-task owner; retain_live_tasks claims done
  groups instead of cancelling them, while legacy mode still drains per
  window.
- _PendingTasks.close cancels each task once, waits for the real Runner
  finally cleanup and reports non-cancellation errors.
- AgentLoopManager.produce_batch accepts quiesce and only opens the quiesce
  wait in turn mode; DisaggAgentLoopManager stops group admission, waits for
  the current producer put to finish, then quiesces and pauses.
- Produced tokens are counted from suffix growth recorded at generation time;
  provenance already versions every response token, so version-array length
  can no longer derive the generation count.

Cancellation still reaches the original Runner.finally and Pool cleanup; the
manager does not reach into another owner's pending set.
The synchronous fit now runs one ``asyncio.run`` for the complete training
lifetime, so Agent tasks and local clients keep the same loop while blocking
training, export and weight work move to threads:

- ReplayBuffer, samplers and Agent managers are built inside ``_fit``; resume,
  produce, save and close are awaited there instead of being wrapped in
  repeated ``asyncio_run`` windows.
- ``await_owned_operation`` is the single cancellation protocol: it waits for
  the owned thread/cleanup to really finish, reports its error, then
  propagates cancellation. It is reused by the ReplayBuffer path.
- Colocate startup publishes the restored version before any group, and the
  final window neither reopens admission nor onloads rollout weights.
- Disaggregated startup stops admission, prepares real weights and KV, then
  publishes version zero through the ordinary continue path; producer
  shutdown waits for its own task and the retained Agent cleanup.
- Turn-resume configuration lives only in ``RolloutConfig.turn_resume``; the
  trainer validates fixed topology, disabled evaluation, provenance and
  partial-rollout constraints once.

The trainer tests now assert the same public batch/save/report behavior
through the async entrypoint, and the debug replay paths wait for their
threaded file I/O.
The single-loop constraint constrains asynchronous activation and use, not
Python object construction. ReplayBuffer, samplers, Agent clients and the
manager are created only from synchronous fields and lazily bound asyncio
primitives, so they return to the original __init__ call sites:

- Both trainers rebuild the base construction order: debug-rollout branch,
  colocate normal path and disaggregated path build the ReplayBuffer and call
  _build_agent_loop_components (including total_epochs derivation) during
  initialization.
- The disaggregated should_continue_fn check moves back next to construction.
- _fit now only performs async manager resume, first weight I/O and initial
  publication, produce, save and close on the fit loop.

Trainer tests drop the fake _cfg/_build_agent_loop_components scaffold that
existed only because construction had been moved into the loop.
close() previously only waited for group-level Tasks: a group gather could
propagate cancellation before slow samples finished their Runner cleanup
(F1), and cancelling a local await on a Ray ObjectRef never stopped the
remote Agent (F2).

- Add await_cancellable_tasks: hold the original executions, send one local
  task.cancel or ray.cancel(..., recursive=False) per pending execution, wait
  for every terminal state, then propagate cancellation. Ordinary errors also
  wait for all siblings before propagating.
- AgentInSandboxLoop.generate_group merges its sample children with this
  helper, so every sample's Runner.finally completes before the group ends.
- BaseProduceContext.generate_group and RouterAgentLoop.generate_group keep
  the original worker ObjectRef and wait for its terminal state; the router
  releases the worker only afterwards.
- Widen await_owned_operation to Awaitable so the helper can protect the
  gathered Future.

Tests cover local cleanup ordering, error-after-siblings, a real CPU Ray
actor cancelled before the awaiting owner returns, router worker release
after remote cleanup, and the sandbox group waiting for every sample. The
blocking-training trainer test now drives the real _train_one_batch with a
thread-blocking train controller instead of an async stand-in.
The manager previously wrote strategy.retain_live_tasks after construction
and then repeated the strategy's retain/legacy branch when pausing. Both
places now use the existing interfaces:

- AsyncProduceStrategyConfig.build and DisaggAsyncProduceStrategyConfig.build
  accept retain_live_tasks; each strategy stores it as internal state. The two
  managers pass turn_resume is not None at build time and no longer mutate the
  strategy afterwards. SyncProduceStrategyConfig accepts the uniform argument
  and keeps its per-window ownership.
- AgentLoopManager.produce_batch always delegates to
  strategy.pause_produce(context): legacy drains its window, turn mode claims
  the groups that completed during gate drain, so live Agent results are not
  skipped. It also always passes model_step to continue_generation; the legacy
  controller accepts and ignores it.
- Disaggregated continue_produce uses the same single call shape.

Add a retain-mode behavior test: build(retain_live_tasks=True) pauses without
draining a live group, and the next quiesce delivers it once.
…ards

Lease settlement repeated the same terminal rules in several places, and a
receipt field was always true.

- SettlementReceipt now carries backend_ended and outcome only. The receipt is
  assigned after the success/failure terminal has been handled, so its
  presence already means "terminal recorded"; RequestGate releases a lease
  when a receipt exists and backend_ended is true. Remove terminal_recorded
  and safe_for_weight_update.
- Add _is_complete_turn and use it both in on_response and in the forwarding
  settlement. The lease branch no longer repeats checks already guaranteed by
  the entry path (non-stream, successful HTTP response, non-error payload);
  the response data guarantee is documented next to the decision.
- Split _is_lmdeploy_context_rejection into structure, protocol fields,
  message and length checks.
- Split install_version/reopen guards by barrier identity, installed version
  and unsettled state; split the cold-resume configuration checks into one
  named resuming fact.
This is the smaller transitional step from the review: keep the producer-idle
confirmation, but remove the duplicated admission flag and strategy pause flag.

- Delete _group_admission. pause_produce sets UPDATE_WEIGHT_AND_ABORT and
  _update_event before waiting for _producer_idle: the status stops the next
  scheduling round, and the event stops the running strategy from scheduling
  more groups. The idle event still confirms that the current produce call,
  including its result puts, has exited.
- produce_loop checks the status directly instead of waiting on an admission
  event; only a normal round that was not paused advances
  producer_future_step.
- Delete DisaggAsyncProduceStrategy.scheduling_paused; ctx.should_abort()
  (the shared update event) is the single pause signal.
- continue_produce, shutdown and resume no longer create or flip admission or
  scheduling flags.

Add a behavior test: pause waits for the in-flight round, no new round starts
while paused, and continue_produce resumes scheduling.
Implement the complete group Task design from
design_2312_disagg_group_task_lifecycle.md. Both layouts now share one group
lifetime; the disaggregated manager waits for an actual scheduling-round Task
instead of idle/admission flags.

- BaseProduceContext.produce_group owns sample -> generate -> validate/put or
  discard/cleanup -> statistics. A cancelled generation releases only the
  undelivered states after F1/F2 cleanup; a started delivery is awaited to
  completion. Strategies spawn this Task before any business await, so
  registration cannot miss a group.
- Both async strategies and SyncProduceStrategy keep complete group Tasks in
  _PendingTasks. Claim/quiesce/pause/close only read terminal states (via
  await_cancellable_tasks); they never put a result a second time. Sync keeps
  its original sampler path and gains the same registry ownership for pause
  and exception close.
- _PendingTasks.cancel_all reuses the strict close protocol; pause_pending_tasks
  takes no put callback for complete groups and keeps the legacy callback for
  any not-yet-migrated strategy.
- DisaggAgentLoopManager records _produce_pass: the actual round Task performs
  target planning, strategy scheduling, produce_time and future_step updates.
  pause_produce waits that Task; close waits it and then closes every
  strategy; resume resets it. Delete _producer_idle; the status/update_event
  pair already stops the next round.
- AgentLoopManager keeps one stable ProduceProgress ledger for its lifetime:
  windows re-target samples and move consumer_step, so a group delivered after
  a window returned contributes its increment exactly once to a later report.
  close now covers legacy too, and cold resume rebuilds the ledger with
  consumer_step = saved step + 1.

Tests cover complete-group delivery once, retain-mode pauses, the transitional
disagg round wait, and the cross-window ledger.
@jayhenry
jayhenry force-pushed the xs1997zju/agent-rl-stack-20261111/04-agent-core branch from 091591d to 61cf9cf Compare November 20, 2026 13:39
@jayhenry
jayhenry marked this pull request as ready for review November 20, 2026 13:54
@jayhenry
jayhenry merged commit 5c84edf into main Nov 20, 2026
7 of 12 checks passed
@jayhenry
jayhenry deleted the xs1997zju/agent-rl-stack-20261111/04-agent-core branch November 20, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants