Repository navigation
[Feature] Retain Agent tasks across complete-request weight updates - #2297
Conversation
Independent Kimi XT Review — stack layer 4/6This PR carries the Real CLI model: Parent readback, separate from Kimi: the Python 3.10 compatibility observation is confirmed statically; the configured test environment uses 3.12. No long-run R3 leak was demonstrated, and shared ownership must be respected before adding explicit free. Missing-session-ID counting concerns mixed failed/successful groups. Developer guide · Retained public report copy
review_2257_2026-11-11_21-26审阅对象:#2257 本地拆分实际执行结果(base 仓库内不存在 summary机械拆分(独立复核,全部静态确认,无阻断项) 我在
六层 stack/合入组合理性:T1 三个小修复各自独立成层合理;T2-gate 依赖 T2-trace 的 主体实现(base→merged,~2600 行源码):这是原 #2257 的实现本体,拆分未增删实现。因此实现层面的缺陷绝大多数属于原 2257 已有设计,不是本轮回归;下文章节严格区分。未发现高严重度的新增实际风险。 高严重度实际风险:无(静态确认范围内)。 本轮拆分新增问题:仅 1 项低severity记录性事项(见其他 issues O1:运行级 ACCEPTANCE 内 已有设计建议(原 2257 带入,不阻塞本轮):见“其他 issues”O2–O8,包括 Python 版本声明与 GPU/validation 证据可支撑结论的边界:GPU 实测树是 split + 固定 #2292 的叠加组合( Main Flowchart before this PR改动前(base flowchart TD
subgraph ExternalClient["外部 Agent 客户端 (HTTP)"]
EC1["POST /v1/chat/completions (OpenAI/Anthropic, 可 stream)"]
end
subgraph SessionServer["xtuner.v1.rl.rollout.session_server.SessionServer"]
SS1["_handle_request(request)"]
SS2["_forward_request(request)"]
SS3["on_request / on_response hooks"]
SS4["trace_store insert (TokenizedSegment, Trie)"]
SS1 --> SS2 --> SS3 --> SS4
end
subgraph RolloutController["xtuner.v1.rl.rollout.controller.RolloutController"]
RC1["pause_generation() / continue_generation()"]
RC2["health_manager.pause/resume"]
RC1 --> RC2
end
subgraph Producer["xtuner.v1.rl.agent_loop_manager.producer.AsyncProduceStrategy"]
P1["_local_pending_tasks: set[asyncio.Task] (每次 produce_batch 重置)"]
P2["pause_pending_tasks() (取消并收尾 pending)"]
end
subgraph ManagerC["xtuner.v1.rl.agent_loop_manager.agent_loop_manager.AgentLoopManager (共卡)"]
MC1["produce_batch() -> continue_generation -> 生产 -> pause_generation -> pause_produce"]
end
subgraph ManagerD["xtuner.v1.rl.agent_loop_manager.disagg_agent_loop_manager.DisaggAgentLoopManager (异卡)"]
MD1["produce_loop() / pause_produce() / continue_produce()"]
MD2["DisaggAsyncProduceStrategy._pending_tasks"]
end
subgraph Trainer["xtuner.v1.train.rl_trainer.RLColocateTrainer / RLDisaggregatedTrainer"]
T1["fit() -> produce -> _train_one_batch -> fit同步"]
T2["sampler save/resume (epoch-local step 直接作 global 计数)"]
end
EC1 --> SS1
MC1 --> RC1
MD1 --> RC1
T1 --> MC1
T1 --> MD1
T1 --> P2
Main Flowchart after this PR改动后:新增 RequestGate 准入/结算屏障、affinity 路由、任务保活与 quiescent 保存恢复。突出显示为新增/修改节点(mermaid 8.8.0 flowchart TD
subgraph ExternalClient["外部 Agent 客户端 (HTTP)"]
EC1["POST /v1/chat/completions (非 stream, 带 session_id)"]
end
subgraph AffinityRouter["xtuner.v1.rl.rollout.affinity_router.SessionAffinityRouter / SessionAffinityService (新增)"]
AR1["acquire(sid)->具体 DP entrypoint; retire_sessions(); 410 tombstone"]
end
subgraph RequestGate["xtuner.v1.rl.rollout.request_gate.RequestGate / SettlementReceipt (新增)"]
RG1["request(session_id) 租约: seal -> drain -> install_version -> reopen"]
RG2["未结算/未知结局 => poison barrier, 阻止权重更新"]
end
subgraph SessionServer["xtuner.v1.rl.rollout.session_server.SessionServer (修改)"]
SS1["_handle_request -> 生成端点先进 RequestGate (修改)"]
SS2["_forward_request(request, lease) (修改: 注入 _xt_request_id/_xt_model_step)"]
SS3["结算: backend_ended+terminal_recorded => SettlementReceipt (修改)"]
SS4["TokenizedSegment.request_id/model_step (修改)"]
SS1 --> RG1
SS1 --> SS2 --> SS3 --> SS4
end
subgraph RolloutController["xtuner.v1.rl.rollout.controller.RolloutController (修改)"]
RC1["prepare_pause() freeze+seal+drain (新增)"]
RC2["resume_requests() install->continue_generation->reopen, 单 deadline (新增)"]
RC3["assert_request_gates_healthy() (新增)"]
RC4["retire_route_sessions() (新增)"]
end
subgraph PausePolicy["xtuner.v1.rl.agent_loop_manager.agent_turn_pause_policy.AgentTurnPausePolicy (新增)"]
PP1["pause/resume/check_requests/before_sync(quiesce)"]
end
subgraph Keepalive["xtuner.v1.rl.agent_loop_manager.colocate_keepalive.LoopThread / ColocateKeepaliveManager (新增)"]
K1["专用 event loop 跨同步训练窗口保活 Agent 任务"]
end
subgraph Producer["xtuner.v1.rl.agent_loop_manager.producer.AsyncProduceStrategy (修改)"]
P1["_PendingTasks 统一持有 pending; retain_live_tasks 跨窗口保留"]
end
subgraph ManagerD["DisaggAgentLoopManager (修改)"]
MD1["quiesce(timeout) 关门收单; prepare_sync(checkpoint_due)"]
MD2["scheduling_paused 停新组; close() 取消 pending"]
end
subgraph Trainer["RLColocateTrainer / RLDisaggregatedTrainer (修改)"]
T1["_validate_turn_keepalive/_validate_turn_resume 配置校验 (新增)"]
T2["_initialize_keepalive_rollout 冷启动/冷恢复发布 (新增)"]
T3["_checkpoint_due + produce_batch(quiesce=...) (修改)"]
T4["debug_rollout/debug_train 异卡阶段复用 (新增)"]
T5["sampler rollout_group_count/rollout_next_epoch (修改)"]
end
EC1 --> AR1 --> SS1
RG2 --> RC1
PP1 --> RC1
PP1 --> RC2
K1 --> PP1
MD1 --> PP1
T1 --> K1
T2 --> RG1
T3 --> MD1
classDef changed fill:#ffe08a,stroke:#b58900,stroke-width:2px;
class AR1,RG1,RG2,SS1,SS2,SS3,SS4,RC1,RC2,RC3,RC4,PP1,K1,P1,MD1,MD2,T1,T2,T3,T4,T5 changed;
文字对应说明(流程 → 修改代码):
核心原理
核心实现与对应的单测
静态确认:以上测试与实现逐一对得上,且 CPU 验收(130 项,host 环境)覆盖的是这些文件。 抽象与信息隐藏评估(按 module/interface/depth/seam 词汇)
新增功能守则对既有类的接触点核查(T2–T7 相对既有类):
公开接口的线性业务流程评估
单测建议现有测试已较完整(上表)。补充建议(均为静态推断,优先级低):
其他 issuesO1(低,静态确认,本轮拆分新增): O2(中,静态确认,原 2257 带入):Python 版本声明不匹配。 O3(低,静态确认,原 2257 带入):异卡 O4(低,静态确认,原 2257 带入):request-aware 的 O5(低,静态确认,原 2257 带入): O6(低,静态确认,原 2257 带入): O7(信息,静态确认): O8(信息,静态确认):T2-trace 与 #2252 的 Trie 遍历重叠按“标记重叠、不换实现”处理; O9(验证边界汇总,静态确认 + 推断):可支撑——机械相等(我独立复核)、叠加 #2292 组合上 2 次真实 optimizer 更新/冻结 MTP 全量哈希不变/IPC 同步/版本 0→1→2/affinity 绑定/受控任务跨更新续跑/完整 DCP 冷恢复后真实 HTTP;runner( 审阅执行说明:全程只读(git show/diff/cat-file、源码阅读、JSON 证据读取),唯一写入为本报告。未运行项目代码(依原文第 6 条),未改任何 git refs/index,未 push/评论/merge,未操作 RJob/SSH/权重。中间产物(prompt 快照、SCOPE、diff、日志、metadata)按要求全部保留,未做清理。 |
|
提出一个替代实现供评估:#2312(Draft)。它的 base 与本 PR 相同,均为 |
| if self.request_gate is None or not _is_generation_endpoint(request.method, path): | ||
| return await self._forward_request(request) | ||
| fmt = _detect_format(path) | ||
| if fmt != FMT_OPENAI: |
|
|
||
| if lease is not None: | ||
| complete = ( | ||
| not is_stream |
| async with asyncio.timeout(timeout): | ||
| await self._condition.wait_for(lambda: not self._active or bool(self._failures)) | ||
| except TimeoutError: | ||
| self._failures["barrier"] = "drain_timeout" |
There was a problem hiding this comment.
这里想确认一下 drain 超时的语义:共卡 rollout pause 时,turn_keepalive_drain_timeout 会传到这里。当前一旦等待到期,就写入永久的 drain_timeout failure 并重新抛出;异常经 prepare_pause → AgentTurnPausePolicy.pause → produce_batch 传到 fit(),执行清理并终止本次训练。即使原请求随后正常结束,该 failure 也不会自动清除,后续版本安装和 reopen 仍会失败。
仅因这次 drain 等待超时就终止整个训练,是预期行为吗?建议不要仅因此结束训练:请求可能只是生成或 trace 提交较慢,仍由原 forwarder/lease 持有。可以将这里的超时作为告警和状态检查间隔,保持 gate 关闭、拓扑冻结,在同一个 barrier 内继续等待原请求结算;全部入口确认 drained 后,再继续训练或权重同步。
真实失结算或后端故障仍需单独处理;不应在等待超时后直接继续训练、重放 POST,或清除真实 failure。
| if self.barrier is None: | ||
| await controller.continue_generation.remote() | ||
| else: | ||
| await controller.resume_requests.remote(self.barrier, model_step, timeout=self.freeze_timeout) |
There was a problem hiding this comment.
这里提醒核对 turn_keepalive_freeze_timeout 的全部使用点:它既限制 prepare_pause 中 recovery freeze、获取 SessionServer、seal 的分别等待,又在这里作为 install version → backend resume → reopen 的共享总预算(共卡默认 30s,不包含训练和权重传输)。目前这些等待超时都会向外抛错,使训练退出。
建议不要仅因控制 RPC 返回较慢就终止整训。不过处理时需要注意阶段状态:ray.get 超时不代表远端操作已经取消;seal 超时可能只有部分 gate 关闭,reopen 超时也可能已经有部分入口开放。因此不能简单 catch 后继续下一阶段,也不宜从头重跑 prepare_pause/resume_requests。
能否区分等待预算耗尽与真实故障,保留当前 barrier、执行阶段和未完成 RPC,告警后续等或核对状态,确认所有入口满足当前阶段条件后再推进?
| while owner.count(): | ||
| for ready in await owner.wait_and_claim(timeout_s=1): | ||
| await ctx.put_generated_group(ready.result()) | ||
| await asyncio.wait_for(finish_live_tasks(), keepalive.quiesce_timeout) |
There was a problem hiding this comment.
这里也提醒注意 turn_keepalive_quiesce_timeout 的超时语义:asyncio.wait_for 到期会取消 finish_live_tasks() 这个收单协程,再把异常传到 fit(),导致整训退出;底层活 Agent 不一定随该收单协程一起取消,随后才由训练退出时的清理处理。也就是说,checkpoint/末步时某个 Agent 或工具只是执行较慢,也可能因此结束整个训练。
建议不要仅因收单等待超时就终止整训,可以告警并继续等待。不过续等应保留同一个收单任务,例如用 asyncio.wait 轮询其完成状态,避免反复取消、重建收单过程;尤其需要注意 wait_and_claim 取走结果到 put_generated_group 完成之间的交接,避免漏收或重复入 buffer。
同时,不能 catch 超时后直接跳过 quiesce 去保存完整 checkpoint。仍需确认所有 pending Agent 已进入终态;如果选择延期 checkpoint,也需要显式调整后续保存决策。请一起确认这些场景的预期处理。
f5ee2be to
6b6439f
Compare
6b6439f to
091591d
Compare
Admission barriers, one owned version publication path, session affinity and request-exact traces now live in the rollout control plane: - RequestGate admits one traced OpenAI generation per session and only releases a lease after the backend ended and the trace committed; unknown exits poison the barrier instead of allowing a weight update. - RequestPublication freezes topology, seals/drains every SessionServer, pauses backends, installs the next version, then reopens admission. The controller only routes its existing pause/continue/health entrypoints. - SessionAffinityRouter owns its thread, loop and HTTP client; sessions stay bound across updates, dead bindings fail closed with 503, and retired sessions get 410 without replaying a POST. - Trace nodes carry their request identity and model step, and shared routed-expert references survive trace release through Ray reference counting while legacy refs keep the explicit free contract. SessionServer no longer accepts client-forged _xt_* identity; the lease is passed explicitly to the trace write path.
The production strategies now own the lifecycle of retained Agent tasks so a turn can span weight updates: - AsyncProduceStrategy and DisaggAsyncProduceStrategy keep their existing _PendingTasks as the single live-task owner; retain_live_tasks claims done groups instead of cancelling them, while legacy mode still drains per window. - _PendingTasks.close cancels each task once, waits for the real Runner finally cleanup and reports non-cancellation errors. - AgentLoopManager.produce_batch accepts quiesce and only opens the quiesce wait in turn mode; DisaggAgentLoopManager stops group admission, waits for the current producer put to finish, then quiesces and pauses. - Produced tokens are counted from suffix growth recorded at generation time; provenance already versions every response token, so version-array length can no longer derive the generation count. Cancellation still reaches the original Runner.finally and Pool cleanup; the manager does not reach into another owner's pending set.
The synchronous fit now runs one ``asyncio.run`` for the complete training lifetime, so Agent tasks and local clients keep the same loop while blocking training, export and weight work move to threads: - ReplayBuffer, samplers and Agent managers are built inside ``_fit``; resume, produce, save and close are awaited there instead of being wrapped in repeated ``asyncio_run`` windows. - ``await_owned_operation`` is the single cancellation protocol: it waits for the owned thread/cleanup to really finish, reports its error, then propagates cancellation. It is reused by the ReplayBuffer path. - Colocate startup publishes the restored version before any group, and the final window neither reopens admission nor onloads rollout weights. - Disaggregated startup stops admission, prepares real weights and KV, then publishes version zero through the ordinary continue path; producer shutdown waits for its own task and the retained Agent cleanup. - Turn-resume configuration lives only in ``RolloutConfig.turn_resume``; the trainer validates fixed topology, disabled evaluation, provenance and partial-rollout constraints once. The trainer tests now assert the same public batch/save/report behavior through the async entrypoint, and the debug replay paths wait for their threaded file I/O.
The single-loop constraint constrains asynchronous activation and use, not Python object construction. ReplayBuffer, samplers, Agent clients and the manager are created only from synchronous fields and lazily bound asyncio primitives, so they return to the original __init__ call sites: - Both trainers rebuild the base construction order: debug-rollout branch, colocate normal path and disaggregated path build the ReplayBuffer and call _build_agent_loop_components (including total_epochs derivation) during initialization. - The disaggregated should_continue_fn check moves back next to construction. - _fit now only performs async manager resume, first weight I/O and initial publication, produce, save and close on the fit loop. Trainer tests drop the fake _cfg/_build_agent_loop_components scaffold that existed only because construction had been moved into the loop.
close() previously only waited for group-level Tasks: a group gather could propagate cancellation before slow samples finished their Runner cleanup (F1), and cancelling a local await on a Ray ObjectRef never stopped the remote Agent (F2). - Add await_cancellable_tasks: hold the original executions, send one local task.cancel or ray.cancel(..., recursive=False) per pending execution, wait for every terminal state, then propagate cancellation. Ordinary errors also wait for all siblings before propagating. - AgentInSandboxLoop.generate_group merges its sample children with this helper, so every sample's Runner.finally completes before the group ends. - BaseProduceContext.generate_group and RouterAgentLoop.generate_group keep the original worker ObjectRef and wait for its terminal state; the router releases the worker only afterwards. - Widen await_owned_operation to Awaitable so the helper can protect the gathered Future. Tests cover local cleanup ordering, error-after-siblings, a real CPU Ray actor cancelled before the awaiting owner returns, router worker release after remote cleanup, and the sandbox group waiting for every sample. The blocking-training trainer test now drives the real _train_one_batch with a thread-blocking train controller instead of an async stand-in.
The manager previously wrote strategy.retain_live_tasks after construction and then repeated the strategy's retain/legacy branch when pausing. Both places now use the existing interfaces: - AsyncProduceStrategyConfig.build and DisaggAsyncProduceStrategyConfig.build accept retain_live_tasks; each strategy stores it as internal state. The two managers pass turn_resume is not None at build time and no longer mutate the strategy afterwards. SyncProduceStrategyConfig accepts the uniform argument and keeps its per-window ownership. - AgentLoopManager.produce_batch always delegates to strategy.pause_produce(context): legacy drains its window, turn mode claims the groups that completed during gate drain, so live Agent results are not skipped. It also always passes model_step to continue_generation; the legacy controller accepts and ignores it. - Disaggregated continue_produce uses the same single call shape. Add a retain-mode behavior test: build(retain_live_tasks=True) pauses without draining a live group, and the next quiesce delivers it once.
…ards Lease settlement repeated the same terminal rules in several places, and a receipt field was always true. - SettlementReceipt now carries backend_ended and outcome only. The receipt is assigned after the success/failure terminal has been handled, so its presence already means "terminal recorded"; RequestGate releases a lease when a receipt exists and backend_ended is true. Remove terminal_recorded and safe_for_weight_update. - Add _is_complete_turn and use it both in on_response and in the forwarding settlement. The lease branch no longer repeats checks already guaranteed by the entry path (non-stream, successful HTTP response, non-error payload); the response data guarantee is documented next to the decision. - Split _is_lmdeploy_context_rejection into structure, protocol fields, message and length checks. - Split install_version/reopen guards by barrier identity, installed version and unsettled state; split the cold-resume configuration checks into one named resuming fact.
This is the smaller transitional step from the review: keep the producer-idle confirmation, but remove the duplicated admission flag and strategy pause flag. - Delete _group_admission. pause_produce sets UPDATE_WEIGHT_AND_ABORT and _update_event before waiting for _producer_idle: the status stops the next scheduling round, and the event stops the running strategy from scheduling more groups. The idle event still confirms that the current produce call, including its result puts, has exited. - produce_loop checks the status directly instead of waiting on an admission event; only a normal round that was not paused advances producer_future_step. - Delete DisaggAsyncProduceStrategy.scheduling_paused; ctx.should_abort() (the shared update event) is the single pause signal. - continue_produce, shutdown and resume no longer create or flip admission or scheduling flags. Add a behavior test: pause waits for the in-flight round, no new round starts while paused, and continue_produce resumes scheduling.
Implement the complete group Task design from design_2312_disagg_group_task_lifecycle.md. Both layouts now share one group lifetime; the disaggregated manager waits for an actual scheduling-round Task instead of idle/admission flags. - BaseProduceContext.produce_group owns sample -> generate -> validate/put or discard/cleanup -> statistics. A cancelled generation releases only the undelivered states after F1/F2 cleanup; a started delivery is awaited to completion. Strategies spawn this Task before any business await, so registration cannot miss a group. - Both async strategies and SyncProduceStrategy keep complete group Tasks in _PendingTasks. Claim/quiesce/pause/close only read terminal states (via await_cancellable_tasks); they never put a result a second time. Sync keeps its original sampler path and gains the same registry ownership for pause and exception close. - _PendingTasks.cancel_all reuses the strict close protocol; pause_pending_tasks takes no put callback for complete groups and keeps the legacy callback for any not-yet-migrated strategy. - DisaggAgentLoopManager records _produce_pass: the actual round Task performs target planning, strategy scheduling, produce_time and future_step updates. pause_produce waits that Task; close waits it and then closes every strategy; resume resets it. Delete _producer_idle; the status/update_event pair already stops the next round. - AgentLoopManager keeps one stable ProduceProgress ledger for its lifetime: windows re-target samples and move consumer_step, so a group delivered after a window returned contributes its increment exactly once to a later report. close now covers legacy too, and cold resume rebuilds the ledger with consumer_step = saved step + 1. Tests cover complete-group delivery once, retain-mode pauses, the transitional disagg round wait, and the cross-window ledger.
091591d to
61cf9cf
Compare
Agent RL stack — bottom to top
Review each PR against its configured base. The complete delivery is split from #2257, which remains unchanged as the pre-split reference.
xs1997zju/agent-rl-stack-20261111/02-sampler-resume→mainxs1997zju/agent-rl-stack-20261111/03-fit-cleanup→ #2295’s branch (xs1997zju/agent-rl-stack-20261111/02-sampler-resume)xs1997zju/agent-rl-stack-20261111/04-agent-core→ #2296’s branch (xs1997zju/agent-rl-stack-20261111/03-fit-cleanup) ← you are herexs1997zju/agent-rl-stack-20261111/05-disagg-debug→ #2297’s branch (xs1997zju/agent-rl-stack-20261111/04-agent-core)xs1997zju/agent-rl-stack-20261111/06-valid-attempts→ #2298’s branch (xs1997zju/agent-rl-stack-20261111/05-disagg-debug)The core layer (#2297) contains five separate topic commits that form one integration group. Shared tests/docs are carried in the final layer (#2299). Intermediate commits are review units, not independently validated feature versions.
Independent related PRs: #2252 (iterative trace traversal; overlaps with #2297 and needs reconciliation) and #2292 (native frozen MTP). They are not additional layers of this stack.
Developer guide, diagrams, scripts and validation scope.
定位
本 PR 保留 #2297 编号,内容替换为第 4 层(
04-agent-core)的新实现;原实现已备份到分支backup/agent-rl-stack-04-agent-core-20261120(6b6439f0)。xs1997zju/agent-rl-stack-20261111/03-fit-cleanup(589fd88b)xs1997zju/agent-rl-stack-20261111/04-agent-core(091591db)重构设计思路
目标是把第 4 层的职责收敛成 3 个有真实深度(Depth)的行为类,并让整个 fit 生命周期落在一个 asyncio loop 上。后续评审又补齐了真实取消边界与完整组交付生命周期。
flowchart TD A["fit(): 一次 asyncio.run"] --> B["_fit: 异步恢复与首次 I/O"] B --> C["初始化: pause → 恢复权重 → publish(0)"] C --> D["produce_batch: publish(step) → 完整组 Task 生产 → gate drain/pause"] D --> E["await_owned_operation(to_thread): 训练/保存/权重同步"] E --> D D --> F["末步: 不 reopen → close 取消并等待所有组 → 退出"] C --> G["RolloutController 请求屏障"] G --> H["RequestGate"] H --> I["SessionServer → TraceStore"] D --> J["SessionAffinityRouter"] D --> K["_PendingTasks: 组 Task 唯一 owner"]Fit 生命周期(A3):
fit()只调用一次asyncio.run(self._fit())。ReplayBuffer、sampler、Agent 客户端与 manager 仍在__init__按原流程构造(只创建同步字段和惰性绑定的 Lock/Event/Semaphore,不启动 task、不发异步 I/O);异步恢复、首次权重 I/O 与初始发布、produce、save、close 在 fit loop 中 await。阻塞训练、DCP/HF 保存、权重/IPC 与资源切换通过asyncio.to_thread执行,并统一用await_owned_operation等待“真正结束”后再传播取消。约束边界:同一 loop 约束的是异步激活与使用,不是所有 Python 对象的构造;因此没有把 builder 搬进
_fit,构造阶段也不做异步 warmup 或 manager.resume。请求屏障(
RolloutController私有状态):独占freeze → 全部 seal → 全部 drain → 暂停后端 → 训练/保存/同步 → 全部 install → 全部 resume → reopen全协议;原独立RequestPublication类已删除,controller 只保留pause_generation/continue_generation(model_step)/assert_request_gates_healthy入口。请求门禁(保留
RequestGate):仅当backend_ended AND terminal_recorded才释放 lease;未知结局(断连/取消/未识别错误)保持 active 并毒化后续 drain。LMDeploy 已识别的预生成 context rejection 视为安全失败;SettlementReceipt只保留backend_ended/outcome。Affinity(合并为一个
SessionAffinityRouter):线程、loop、HTTP client、bindings、退休墓碑同一 owner;session 固定到具体 DP entrypoint,失活绑定 503 不改绑,退休 session 410,POST 不重放;probe 使用自有 session,发布后执行。跨窗口 Agent 生命周期:
_PendingTasks仍是唯一 live task owner;策略以retain_live_tasks选择“跨窗口保留 + claim ready”或原 pause/drain;close只 cancel 一次并等待原Runner.finally清理。真实取消边界(F1/F2):新增
await_cancellable_tasks,持有原执行(本地 Task 或 Ray ObjectRef)。取消时本地task.cancel()、远端ray.cancel(force=False, recursive=False),等全部终态与 Runner finally 结束后再传播CancelledError;普通异常同样等齐 sibling 再抛出。router 在远端终态后才释放 worker。完整组交付(S2):
BaseProduceContext.produce_group拥有sample → generate → 校验/入队或丢弃清理 → 统计,返回 bool;claim/pause 只读终态,不再二次 put。非共卡用_produce_pass确认真实调度轮次退出,删除_producer_idle/_group_admission/scheduling_paused;共卡用 manager 持有的稳定ProduceProgress跨窗口消费增量,并动态推进consumer_step。Sync 保留原局部 pending/补组循环,仅在 finally 结清剩余组。Provenance:lease 显式传入
on_response;export_training_trace(expected_request_ids=..., require_routed_experts=...)集中校验 prompt-only 首节点、完整后缀版本、路径/记录 ID 子集与跨分枝提交、专家齐全;response_model_steps只表示连续后缀版本。共享引用:
TokenizedSegment/RolloutState增加routed_experts_shared;共享清理只 detach(Ray 引用计数保活),legacy 保留显式 free。生成计数:
generate_group记录本窗口 suffix 增长new_response_tokens;provenance 已版本化全部 token,不能再由版本数组长度推导生成量。Checkpoint 格式:两种 manager 的
manager_state写入group_task_lifecycle_version=1,恢复前先校验;旧格式直接拒绝,不补字段、不转换。删除/未引入:
LoopThread、ColocateKeepaliveManager、AgentTurnPausePolicy、DrainPendingPausePolicy、SessionAffinityService、_producer_idle、_group_admission、scheduling_paused。行为与配置变化
RolloutConfig.cross_weight_update_rollout = CrossWeightUpdateRolloutConfig(wait/freeze/drain_timeout)与RolloutConfig.session_affinity;删除 manager 的turn_keepalive*与 disagg 的cross_weight_update_rollout_*;cross-weight-update 模式下 SessionServer 构造即关闭准入。_xt_request_id/_xt_model_step不再作为控制通道;客户端传入的这些字段会被剥离,lease 由 SessionServer 显式传递。RolloutController.continue_generation在 turn 模式下必须携带真实model_step。AgentInSandboxLoopConfig新增require_turn_provenance/retire_route_on_completion。提交与阅读顺序
49aec162feat(rollout): gate and publish complete-turn rollout versionsd08214c4feat(rl): retain live agents across production windows757016d0feat(rl): drive the whole fit on one asyncio loop42c02bccrefactor(rl): construct loop-owned components in init__init__统一构造组件,_fit只保留异步恢复/首次 I/O/生产/保存/关闭8ca6381efix(rl): cancel real Agent execution and wait for all cleanupawait_cancellable_tasks、组内 sample 全量清理、Ray ObjectRef 取消与 router 释放9897fa22refactor(rl): fix the strategy mode at build time and delegate pauseretain_live_tasks构建期确定、manager 委托pause_producece5f38ddrefactor(rollout): centralize the complete-turn decision and split guards_is_complete_turn、两字段结算收据、长条件分阶段345953c5refactor(rl): stop disagg production through status and update_event_group_admission/scheduling_paused,复用 status/update_event,保留_producer_idle0856ef0ffeat(rl): deliver complete groups once and settle disagg rounds by Task_produce_pass、共卡稳定账本、Sync 局部收尾、checkpoint 版本 1b6a398b0refactor: clarify the colocated progress ledger as next_consumer_stepProduceProgress.consumer_step→next_consumer_step;ProduceContext.consumer_step公开属性保留3e6dc7d4refactor: rename turn_resume to cross_weight_update_rolloutTurnResumeConfig→CrossWeightUpdateRolloutConfig,RolloutConfig/manager 成员同名替换;测试文件同步改名8051f574refactor: drop rollout quiesce and allow non-exact checkpoint statequiesce与quiesce_timeout;turn 模式 pending 存档改为 warning,明确非严格数据恢复契约091591dbrefactor: make model_step mandatory and move request barriers into the controllercontinue_generation(model_step)必填;RequestPublication并入 controller 的RolloutRequestState私有协议API 简化(2026-11-19 review 6.1–6.4)
next_consumer_step,与“取走第 n 步 batch 后应为 n+1”语义一致;context 对 strategy 仍暴露consumer_step。continue_generation必填真实 rolloutmodel_step,manager 不再区分有无参调用;请求门禁的暂停/发布状态移入RolloutController,删除独立RequestPublication类。quiesce:checkpoint/final 不再等待全部 live Agent;仍保留 gate drain 的 LLM 请求安全屏障。turn 模式允许保存带未完成组 Task 的 checkpoint,但明确 live Agent/工具状态不落盘,冷恢复不保证样本序列和统计严格一致。cross_weight_update_rollout/CrossWeightUpdateRolloutConfig,表达“rollout 跨权重更新继续执行”;request_publication.py相应改名为cross_weight_update_rollout.py,控制器屏障测试改名为test_rollout_request_barrier.py。端到端实验(简要)
环境:8×GPU 共卡(
RLColocateTrainer),3 步,GLOBAL_BATCH_SIZE=8,PROMPT_REPEAT_K=8,开启cross_weight_update_rollout+session_affinity+require_turn_provenance+retire_route_on_completion,关闭 DCP/HF/eval。代码版本0856ef0f(rebase 到589fd88b),运行20261117_140024。invalid_train_groups=0。Request publication deadline/Backend resume failed/Backend pause failed/Unsettled requests均为 0。agent_chat/entry_exception、rc_missing、rule_grader/judger等,均由 producer 标记 FAILED 并补采;末步 buffer 余 4 个已完成组(超发余量,未消费)。553dce3e为 0.4194 / 0.1000 / 0.2000;592cabd0为 0.3871 / 0.1020 / 0.2679,均处于同一量级。tests/rl全量 588 passed / 19 skipped;15 个失败均因本机环境(缺QWEN3_4B_PATH/QWEN3_5_MOE_PATH、GPU/Ray 启动、缺 pytest-asyncio 插件),其中 4 个在未合入本分支的 base 上同样失败。仓库 pre-commit(ruff/docformatter/codespell/mypy 435 files/pydantic-extra-check)通过。已知未覆盖:非共卡
RLDisaggregatedTrainer未跑端到端(启动、pause、发布与 close 由单测覆盖);本基线不含 frozen MTP 等 preview 后续能力;本次未产出评估集指标(turn 模式禁用在线评测)。已知 TODO:超时语义(来自 #2297 review)
本次不改变行为,仅在代码中留下
TODO(timeout-reconciliation):drain_timeout:RequestGate.drain到期会写入永久barrierfailure 并抛错,即使已准入请求随后结算完成,install/reopen 仍失败并终止训练。后续应保持 gate 关闭、拓扑冻结,在同一 barrier 内继续等待/核对;只有确认未结算请求或后端故障才失败。freeze_timeout:同时约束 health recovery、SessionServer 获取、seal,以及 install→resume→reopen 的共享预算;任一超时都会终止训练。ray.get超时不会取消远端 RPC,seal/reopen 可能只完成部分入口。后续应区分等待预算耗尽与真实故障,保留 barrier/阶段/未完成 RPC,按入口核对状态后再推进,不能盲目继续或重跑 pause。wait_timeout:只限制已准入请求在 gate 关闭期间的等待,到期只让该 Agent 请求失败,不会终止训练;后续可结合 agent/model 请求预算再校准。已知 TODO:协议与流式(来自 #2297 review)
SessionServer当前只接受 OpenAI chat completions,Anthropic/v1/messages被拒绝。已加TODO(anthropic-format):扩展 gate 准入、请求归一化与 trace 组装以支持 Anthropic 格式。TODO(stream-support):支持流式响应,同时在释放 lease 前完成“backend ended + trace committed”的完整轮次结算。合并方式
04-agent-core;[Feature] Support rollout-only and train-only debug in disaggregated RL #2298/[Feature] Configure minimum valid Agent attempts and sandbox metadata #2299 已按栈顺序 rebase 到新的04-agent-core/05-disagg-debug(head 分别为c6e6455c/063d4022)。backup/agent-rl-stack-04-agent-core-20261120(6b6439f0)。原始 PR 描述(保留)
Long tool calls can span a training update. Keep unfinished Agent tasks with their existing producer, seal and settle admitted model requests before weight changes, publish the new version, then let the same Agent continue. Preserve request IDs, historical token logprobs and routed-expert data; bind external sessions to a concrete inference entrypoint. Quiesce live tasks before full checkpoints and start cold restore with admission closed.
Review scope and dependencies
Layer 4/6 of the mechanical split of #2257. Base:
xs1997zju/agent-rl-stack-20261111/03-fit-cleanup; reviewed head:f5ee2bebdfd466467c032f62f246cfc72afb2cb8. Review this PR against its configured base; later layers depend on earlier layers. The original #2257 remains unchanged as the pre-split reference.Code entry:
xtuner/v1/rl/rollout/request_gate.py; controller.py; session_server.py; affinity_router.py; trace_store.py; agent_loop_manager/colocate_keepalive.py; train/rl_trainer.py.This layer contains five separate review commits: request provenance, request settlement/version barriers, session affinity, task retention, and checkpoint/cold publication. They form one minimum integration group because the existing implementation has cross-topic enable/cleanup dependencies. Intermediate commits are not supported standalone feature versions.
Validation
Reviewer guide and independent review
Developer guide, diagrams, scripts and experiment scope.
A real Kimi XT Review of the completed split and evidence is posted in this PR's comments. It found no mechanical-split blocker; original #2257 compatibility/design observations remain explicitly recorded. This Draft does not include fixes for those observations.
Related independent PRs: #2252 (iterative trace traversal; overlap must be reconciled before landing the core layer), #2292 (native frozen MTP). Lagent #385 targets v1.0.0; the CyberGym recipe is separate in CRG !78. The external MTP route #2289 is not part of this delivery. Keep Draft; no automatic merge.