Repository navigation
feat(benchmark): batch authoring for initial task planning - #9090
loopx-agent wants to merge 1 commit into
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.2-sol · OpenAI (runtime_reported, max)
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
使用 loopx-planned 启动评测的 Harbor 和 SForge 调用方,需要先得到可执行的规划。原流程由 planner 分多次调用声明、写入并绑定 Todo。新入口让空计划可用一份 JSON 提交这些决定。实际 Todo 和需求链接仍由 LoopX 写入与读回,写入中断后可用原提案继续。本次交付范围是规划写入和恢复路径。模型采纳率、出首个评分候选的时间与评分收益需要实际跑分证据。
改动思路 / Approach
规划入口只补充可选命令说明。命令先核对任务摘要和 acceptance revision,再交由现有 Goal acceptance 和 Todo 命令写入、返回真实 ID 并绑定工作。变化集中在共享 benchmark wrapper、批量命令、测试和运行说明。执行仍由调用方走原配额 guard。 现有 effect 各自保留回执。丢失响应时,原提案继续补齐剩余工作,worker 仍以当前状态为准。
具体改动 / Changes
精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b,diff base 1c540c3e5ae059f974abdfcfe27b42f5cad9e992,5 个文件 +371/-8。逐项复核 owner-backed-initial-plan、readback-and-execution、interrupted-replay。验收约定
关键代码讲解
- 只有显式
plan阶段出现批量写入提示。两版的未指定阶段与execute文案一致,新增提示后的 checkpoint 也逐字相同。源码 - 首次通过 provider revision 声明需求。固定的声明 ID 与 Todo operation ID 将重试约束在原提案;绑定前再次读回 Todo 的归属和可运行状态。源码
- worker 按任务摘要、实际 Todo 和 requirement links 独立确认 ready。规划无扣费,执行入口可另行取得对应 Todo 的 quota guard。源码
对主干的风险 / Risks
最关键的是绑定已经成功但 caller 没收到响应。两种存储均复现了该断点,再次执行后仍为原来的三个 Todo 和同一 acceptance revision,绑定只写入一次。原测试也在第二个 Todo 的成功响应丢失后拒绝了改写的提案。
语义与 CI 对齐
基线和 head 各通过 11 个原有 task-entry 测试。head 新增的四个批量路径以及独立的两个最后绑定断点均通过,改动文件 Ruff 与 diff 检查通过。既有 Complete Turn 执行提示测试的一个 --complete-turn 断言在两版同样失败。验证使用隔离的实际 File/SQLite 与本地模拟 planner。
我的整体评价 / Verdict
通过本次精确提交的评审。当前实现减少了初始计划的写入交互,保留了既有权威数据、恢复和执行准入。线上模型、Harbor 容器矩阵,以及时间和评分收益仍待实测。
English verdict: APPROVE. The optional initial authoring path uses the existing authorities, recovers after Todo and final link response loss on File and SQLite, and retains independent execution admission. Live model and outcome effects remain unmeasured.
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google (self_reported)
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
在使用 loopx-planned 模式启动评测任务时,Harbor 与 SForge 调用方及规划器模型需要在正式执行前完成任务目标声明与可执行 Todo 的初始化。 改动前,规划器必须分多次模型工具调用依次执行需求声明、逐条写入 Todo 并绑定 Todo ID,在真正进入解题前产生额外的多轮交互开销与中断重试成本;改动后,模型在空前沿下可用一份结构化 JSON 提案通过单条 python -m benchmark.runtime.planning_batch 命令一次性完成需求声明、有序 Todo 创建、真实 ID 读回与绑定,中断后凭原提案重跑即可安全恢复。 全新空前沿(尚无现有 Todo 与 outcome 声明的 Goal)借助已有 Goal acceptance 与 Todo 所有者写入并读回校验通过的真实 Todo ID 与需求关联,保持 execution_handoff.owner 为调用方且规划期消耗 0 quota。 现有规划工作流、既有 Todo 修改、需求变更、阻塞处理以及模型采纳率、出首个评分候选时间与最终评分收益属于非目标范围,仍需后续实跑评测验证。
改动思路 / Approach
本 PR 在 benchmark/runtime/planning_batch.py 中新增无状态 CLI 适配器 apply_initial_plan,通过子进程调用既有公开控制面所有者(task_plan_packet、loopx goal-acceptance refine 与 loopx todo add),把原有多轮模型工具写入收敛为脚本内部可幂等重放的顺序事务,同时维持写入前后的权威读回校验与执行交接边界不变。将可选批量编排入口留在 benchmark/runtime/planning_batch.py 并直接经由公共 CLI 复用 goal-acceptance refine 与 todo add,把 benchmark 启动便利性与核心控制面权威读写、CAS 事务及执行准入边界彻底分离,无需新增任何共享持久化状态或新 schema,改动内聚且可独立回滚。 当前边界声明:仅限 LOOPX_TASK_STAGE=plan 下针对空工作与空需求前沿的初始 JSON 提案写入及原提案中断重放,非目标路径维持原有规划步骤,真实模型采纳率与跑分对比留待后续实测。
| 阶段 / 路径 | 基线 origin/main (1c540c3e5a) |
本 PR Head (64204a6c72) |
权威状态与恢复约束 |
|---|---|---|---|
LOOPX_TASK_STAGE=plan 空前沿 |
仅提供分步 goal-acceptance refine 与 todo add 指引 |
额外追加可选 planning_batch 命令提示(planning_batch_guidance) |
仅写入现有 goal_outcomes 与 todo;execution_handoff.owner 保持 caller 且 0 quota |
| 写入中断后重放 | 需模型自行人工对齐已写入 Todo 与声明 | 保留原 proposal.json 重跑,自动以固定 operation_id 重放 declare 与 todo:N 并补齐 bind |
提案被篡改(outcome_id 哈希变化)时 _declared_outcome 在写前直接拒绝 |
非 plan 阶段 / 非空 Goal / 阻塞 |
走原有规划与执行流程 | 与基线完全一致(render_planning_prompt 逐字不变;apply_initial_plan 显式拒绝) |
不改变 Core todo plan、/loopx、seeded-todo 或执行期提示 |
具体改动 / Changes
精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b,对比基线 1c540c3e5ae059f974abdfcfe27b42f5cad9e992,共修改 5 个文件(+371 / -8):生产运行时 2 个文件(benchmark/runtime/planning.py +6/-0、benchmark/runtime/planning_batch.py +205/-0)、公开运行文档 1 个文件(benchmark/runtime/RUNTIME.md +16/-7)、测试与夹具 2 个文件(benchmark/tests/test_task_entry.py +3/-1、tests/control_plane/test_benchmark_planning_batch.py +141/-0)。
关键代码讲解
render_planning_prompt:仅当env.get("LOOPX_TASK_STAGE") == "plan"时拼接planning_batch_guidance(),确保在未设置该阶段或execute阶段时对提示词正文零侵入,与基线输出逐字节相等;由benchmark/runtime/worker.py在stage == "plan"时调用,非死代码路径。apply_initial_plan:入口严格校验LOOPX_TASK_STAGE == "plan"、_request(proposal)结构、实时task_plan_packet的input_digest匹配、且无goal_waiting_on或blocking_todo_ids。首次执行时在provider_revision匹配下要求existing_todos与_outcomes(acceptance)均为空;以benchmark-plan:<input_digest>为前导操作 ID,outcome_id绑定整份规范化提案的 SHA-256 摘要,再经goal-acceptance refine声明并通过todo add --operation-id逐条写入advancement_taskTodo。_declared_outcome与绑定前读回校验:当provider_revision已推进(断点重试场景)时,要求current_lifetime is True且已声明 outcome 的outcome_id、owner_agent_id、objective与全部requirements与原提案一致,杜绝同一input_digest下混用另一份提案的 Todo 写入;全部 Todo 写完后再次调用task_plan_packet验证全部todo_ids均仍在runnable_todo_ids中且无新阻塞,仅当linked为空时执行最终绑定,否则直接确认一致性并返回符合TASK_PLAN_RESULT_SCHEMA的就绪结果。planning_batch_guidance:使用当前sys.executable构造-m benchmark.runtime.planning_batch调用指令,清晰声明仅面向空工作与空需求前沿的可选入口、中断原样重跑规则,以及既有工作/需求修订/阻塞回退普通规划的边界。
规范依据与准则映射
依据修改前 origin/main 修订 1c540c3e5ae059f974abdfcfe27b42f5cad9e992 的既有契约文档 https://github.com/loopx-project/loopx/blob/1c540c3e5ae059f974abdfcfe27b42f5cad9e992/benchmark/runtime/RUNTIME.md#task-entry-and-planning-ablation 逐项映射(PR 对 RUNTIME.md 第 392-407 行的同步增补仅说明该可选批量提案提交方式及非目标边界,未放松任何现有校验门禁):
BENCH-PLAN-OWNERS(复用既有goal-acceptance refine与todo add所有者,不创建规划 Todo 或宿主循环):implemented — 由benchmark/runtime/planning_batch.pyapply_initial_plan实现,经tests/control_plane/test_benchmark_planning_batch.py真实 File/SQLite 写入与读回验证。BENCH-PLAN-READBACK(worker 独立读回input_digest、身份、需求绑定与runnable_todo_ids,保持execution_handoff.owner == "caller"且 0 quota 消耗):implemented — 由benchmark/runtime/planning.pyvalidate_plan_readback与benchmark/runtime/worker.pyrun_once联合验证,test_benchmark_planning_batch.py验证通过。BENCH-PLAN-RECOVERY(写入中断后原提案安全重放且不重复产生 Todo,变更提案在写前拒绝):implemented — 由apply_initial_plan与_declared_outcome实现,经test_benchmark_planning_batch.py的:todo:1丢包测试及本次评审补充的:declare与:bind丢包反事实探针验证。
对主干的风险 / Risks
- 活跃调用路径与默认关闭隔离:
planning_batch_guidance()仅在worker.pystage == "plan"(即LOOPX_TASK_ENTRY=loopx-planned且LOOPX_TASK_STAGE=plan)通过render_planning_prompt注入;对照验证确认无环境变量或LOOPX_TASK_STAGE=execute时输出与基线逐字节一致,apply_initial_plan对非plan阶段直接报错拒绝,Coreloopx todo plan与seeded-todo完全隔离。 - 失败场景与反事实核验:针对最易出错的中间态丢包场景,实测分别在 File 与 SQLite 后端上拦截
:declare首次声明回执、:todo:1第二条 Todo 创建回执及:bind最终绑定回执,原样重跑均精确恢复原有 3 条 Todo 与唯一绑定(不产生重复 Todo 或二次绑定);修改input_digest、替换提案内容、在非空 Goal 下用当前版本重提或处于阻塞态均触发前置拒绝且不污染状态。 - 权限、类型规则与领域中立性:
apply_initial_plan仅调用现有控制面 CLI 且保持规划期 0 quota 消耗与caller执行准入权,不新增任何角色或生命周期状态;规则检查均为结构化字典/字段集合校验,未引入脆弱子串 denylist;新增代码位于benchmark/runtime/,未向核心控制面引入领域特定耦合。
语义与 CI 对齐
本 PR 仅新增 Benchmark 运行时辅助入口,未改动共享控制面状态词表或 RFC 契约(semantic_alignment: not_applicable)。本地直接校验结果(未等待或参考远程 CI,wait_for_ci: false):
tests/control_plane/test_benchmark_planning_batch.py:PASS(4/4 passed,覆盖 File/SQLite 正常路径、:todo:1丢包恢复、改写提案拒绝、SForgeprepare_entry与独立quota should-run准入)。benchmark/tests/test_task_entry.py:59 passed, 54 skipped, 8 failed — 54 项跳过均为缺少可选harbor依赖;8 项失败均为test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input中complete=True时断言"--complete-turn" in body,已在合并基线1c540c3e5ae059f974abdfcfe27b42f5cad9e992上复现完全相同的AssertionError失败签名,归类为pre_existing_unrelated。- 独立反事实探针(File/SQLite
:declare与:bind丢包重放、stage 关闭一致性、错误input_digest与非空 Goal 拒绝):PASS。 ruff check(全部改动 Python 文件)、git diff --check、scripts/check_rfc_bilingual.py、loopx check --scan-path(公开边界扫描 5 个改动文件):PASS。loopx canary premerge --tier quick --from-git-diff --git-diff-base 1c540c3e5ae059f974abdfcfe27b42f5cad9e992:4 项直接检查、py_compile与 3 项目录金丝雀(含维持性棘轮、语义漂移与 benchmark 权限策略 smoke)全部 PASS(failures=0;因涉及benchmark_sensitive路径返回manual_review_required,已由本次精确头审查覆核)。mypy(仓库配置的 17 个 strict 文件):报 1 项与本 PR 无关的既有错误(未改动的interaction_contract.py:1004),已在精确基线1c540c3e5ae059f974abdfcfe27b42f5cad9e992上验证完全一致(pre_existing_unrelated);对改动文件单独执行--strict仅报告未标注的测试/内部辅助参数注解提示,未发现类型逻辑错误。
我的整体评价 / Verdict
从 outcome_impact 评估:long_horizon 为 improved(初始规划从多轮工具写收敛为单命令可重放提交,中断后可沿固定 operation_id 与摘要绑定原提案续接并防止重复创建 Todo 或漂移绑定),user_experience 为 improved(减少规划阶段交互轮次的同时保留清晰读回与错误提示)。本 PR 仅涉及 benchmark 运行时 CLI/prompt 与测试文档,不涉及任何渲染界面改动,visual_review 为 not_applicable(已审阅全部 5 个改动文件,未验证界面范围 unverified_scope 为 none)。代码体积与架构完全合比例(code_volume: necessary、change_proportionality: proportionate、default_off_isolation: isolated、authority_semantics: not_applicable、local_only 无词表扩张)。残余风险在于本次本地验证基于隔离 File/SQLite 夹具与模拟 planner 进程,未运行真实远端 LLM 或可选 harbor 容器集成,实际模型采纳率与跑分收益有待后续基准对照测量。当前远程 PR 仍位于目标精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b。
英文简短结论:
English verdict: APPROVE — Exact head 64204a6c7215241c1a7253f146e12640ca1fc14b adds an optional initial-planning batch adapter in benchmark/runtime/planning_batch.py that reuses goal-acceptance refine and todo add with digest-bound idempotent replay on File and SQLite, preserves stage-off parity and caller-owned execution handoff, passes test_benchmark_planning_batch.py (4/4), non-Harbor test_task_entry.py checks (with pre-existing --complete-turn failures verified identically on base 1c540c3e5a), counterfactual drop probes, Ruff, bilingual/boundary scans, and quick premerge canaries (failures=0). Live model adoption and Harbor container matrix remain unmeasured.
Goal And Delivered Outcome
During a planned benchmark startup, the model declares requirements, authors Todos and binds the resulting IDs before the worker can hand off to execution. Doing those writes in separate model tool calls adds interaction overhead before solving starts.
The shared planning wake now offers a single optional command for a fresh Goal. The model supplies the outcome and ordered work. The command uses the existing Goal acceptance and Todo operations, binds the proposal to the task input digest and acceptance provider revision, and reuses original write receipts on retry. The worker still reads back current requirements and runnable Todo IDs independently. Execution still enters its own quota guard.
The intended base is
mainat1c540c3e5ae059f974abdfcfe27b42f5cad9e992.Author Declaration
loopx-plannedtask entry contract.benchmark/runtime/planning_batch.pycalls the existing operations.Scope And Continuation
The existing planned wake advertises the shortcut. It applies to an empty initial frontier and an unchanged retry of that plan. Existing work, edits to requirements and blockers use the existing planner steps. The shared worker serves Harbor and SForge. The native SForge preparation entry is covered with the real worker and an isolated Goal.
The batch groups independently recorded operations. A failed call can leave already-written Todos, which the unchanged request reuses. Runner readback controls whether a planning result is ready. The next execution admission remains owned by the caller. A matched live run would be needed to estimate model cost and time to the first evaluated candidate.
Validation
64204a6c7215241c1a7253f146e12640ca1fc14b.real_entrypointpassedtests/control_plane/test_benchmark_planning_batch.pyuse the advertised command through the shared worker on real File and SQLite authority. They cover fresh application, a lost Todo response, refusal of a changed proposal, recovery, replay and independent admission. Planning spent zero quota.regression_paritypassedbenchmark/tests/test_task_entry.pycover ordinary planning, SForge preparation, missing requirements, stale readback and clock-independent input. The new end-to-end oracle fails on the historical runner because it does not advertise the batch command.staticpassedintegrationfailedstaticfailedintegrationnot_runFrontend / Visual Evidence
UI impact none. The change is in the benchmark worker's planner input and reuses the existing execution handoff.
Type of Change
LoopX Area
Shared-authority RFC fixture impact
The writer reuses the existing authority commands and schemas. It adds no shared authority state or execution policy.