Skip to content

feat(benchmark): batch authoring for initial task planning - #9090

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-planning-batch
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-planning-batch

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

During a planned benchmark startup, the model declares requirements, authors Todos and binds the resulting IDs before the worker can hand off to execution. Doing those writes in separate model tool calls adds interaction overhead before solving starts.

The shared planning wake now offers a single optional command for a fresh Goal. The model supplies the outcome and ordered work. The command uses the existing Goal acceptance and Todo operations, binds the proposal to the task input digest and acceptance provider revision, and reuses original write receipts on retry. The worker still reads back current requirements and runnable Todo IDs independently. Execution still enters its own quota guard.

The intended base is main at 1c540c3e5ae059f974abdfcfe27b42f5cad9e992.

Author Declaration

Criterion Disposition Evidence
Use Goal acceptance and Todo owners for initial work implemented benchmark/runtime/planning_batch.py calls the existing operations.
Preserve input, links and execution ownership implemented The benchmark worker consumes the same planning result schema and independently validates source state.
Recover an interrupted write without duplicating Todo effects implemented File and SQLite fixtures drop a successful write response and replay the original request. A changed proposal is rejected before creating additional work.

Scope And Continuation

The existing planned wake advertises the shortcut. It applies to an empty initial frontier and an unchanged retry of that plan. Existing work, edits to requirements and blockers use the existing planner steps. The shared worker serves Harbor and SForge. The native SForge preparation entry is covered with the real worker and an isolated Goal.

The batch groups independently recorded operations. A failed call can leave already-written Todos, which the unchanged request reuses. Runner readback controls whether a planning result is ready. The next execution admission remains owned by the caller. A matched live run would be needed to estimate model cost and time to the first evaluated candidate.

Validation

  • Tested revision: 64204a6c7215241c1a7253f146e12640ca1fc14b.
  • Run state: finished.
  • Input classes: synthetic, public_fixture.
Check kind Result Public-safe evidence
real_entrypoint passed Four cases in tests/control_plane/test_benchmark_planning_batch.py use the advertised command through the shared worker on real File and SQLite authority. They cover fresh application, a lost Todo response, refusal of a changed proposal, recovery, replay and independent admission. Planning spent zero quota.
regression_parity passed Eight focused cases in benchmark/tests/test_task_entry.py cover ordinary planning, SForge preparation, missing requirements, stale readback and clock-independent input. The new end-to-end oracle fails on the historical runner because it does not advertise the batch command.
static passed Changed-file Ruff, diff hygiene, DCO and bilingual RFC check. Quick premerge ran five direct and three selected checks successfully. Repository review remains required.
integration failed A broader exploratory run observed failures in the existing Complete Turn execute-prompt flag assertion. A representative failure reproduces on the exact target base. The aggregate was stopped after the focused paths were isolated.
static failed Repository-wide Ruff found the same unused import present on the target base. The upstream fix is tracked in #9082.
integration not_run No live model or judge was used. Container tests requiring the optional Harbor package were skipped in the broader exploratory run.

Frontend / Visual Evidence

UI impact none. The change is in the benchmark worker's planner input and reuses the existing execution handoff.

Type of Change

  • New feature
  • Test update

LoopX Area

  • Benchmark boundary

Shared-authority RFC fixture impact

The writer reuses the existing authority commands and schemas. It adds no shared authority state or execution policy.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI (runtime_reported, max)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用 loopx-planned 启动评测的 Harbor 和 SForge 调用方,需要先得到可执行的规划。原流程由 planner 分多次调用声明、写入并绑定 Todo。新入口让空计划可用一份 JSON 提交这些决定。实际 Todo 和需求链接仍由 LoopX 写入与读回,写入中断后可用原提案继续。本次交付范围是规划写入和恢复路径。模型采纳率、出首个评分候选的时间与评分收益需要实际跑分证据。

改动思路 / Approach

规划入口只补充可选命令说明。命令先核对任务摘要和 acceptance revision,再交由现有 Goal acceptance 和 Todo 命令写入、返回真实 ID 并绑定工作。变化集中在共享 benchmark wrapper、批量命令、测试和运行说明。执行仍由调用方走原配额 guard。 现有 effect 各自保留回执。丢失响应时,原提案继续补齐剩余工作,worker 仍以当前状态为准。

具体改动 / Changes

精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b,diff base 1c540c3e5ae059f974abdfcfe27b42f5cad9e992,5 个文件 +371/-8。逐项复核 owner-backed-initial-plan、readback-and-execution、interrupted-replay。验收约定

关键代码讲解

  • 只有显式 plan 阶段出现批量写入提示。两版的未指定阶段与 execute 文案一致,新增提示后的 checkpoint 也逐字相同。源码
  • 首次通过 provider revision 声明需求。固定的声明 ID 与 Todo operation ID 将重试约束在原提案;绑定前再次读回 Todo 的归属和可运行状态。源码
  • worker 按任务摘要、实际 Todo 和 requirement links 独立确认 ready。规划无扣费,执行入口可另行取得对应 Todo 的 quota guard。源码

对主干的风险 / Risks

最关键的是绑定已经成功但 caller 没收到响应。两种存储均复现了该断点,再次执行后仍为原来的三个 Todo 和同一 acceptance revision,绑定只写入一次。原测试也在第二个 Todo 的成功响应丢失后拒绝了改写的提案。

语义与 CI 对齐

基线和 head 各通过 11 个原有 task-entry 测试。head 新增的四个批量路径以及独立的两个最后绑定断点均通过,改动文件 Ruff 与 diff 检查通过。既有 Complete Turn 执行提示测试的一个 --complete-turn 断言在两版同样失败。验证使用隔离的实际 File/SQLite 与本地模拟 planner。

我的整体评价 / Verdict

通过本次精确提交的评审。当前实现减少了初始计划的写入交互,保留了既有权威数据、恢复和执行准入。线上模型、Harbor 容器矩阵,以及时间和评分收益仍待实测。

English verdict: APPROVE. The optional initial authoring path uses the existing authorities, recovers after Todo and final link response loss on File and SQLite, and retains independent execution admission. Live model and outcome effects remain unmeasured.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google (self_reported)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

在使用 loopx-planned 模式启动评测任务时,Harbor 与 SForge 调用方及规划器模型需要在正式执行前完成任务目标声明与可执行 Todo 的初始化。 改动前,规划器必须分多次模型工具调用依次执行需求声明、逐条写入 Todo 并绑定 Todo ID,在真正进入解题前产生额外的多轮交互开销与中断重试成本;改动后,模型在空前沿下可用一份结构化 JSON 提案通过单条 python -m benchmark.runtime.planning_batch 命令一次性完成需求声明、有序 Todo 创建、真实 ID 读回与绑定,中断后凭原提案重跑即可安全恢复。 全新空前沿(尚无现有 Todo 与 outcome 声明的 Goal)借助已有 Goal acceptance 与 Todo 所有者写入并读回校验通过的真实 Todo ID 与需求关联,保持 execution_handoff.owner 为调用方且规划期消耗 0 quota。 现有规划工作流、既有 Todo 修改、需求变更、阻塞处理以及模型采纳率、出首个评分候选时间与最终评分收益属于非目标范围,仍需后续实跑评测验证。

改动思路 / Approach

本 PR 在 benchmark/runtime/planning_batch.py 中新增无状态 CLI 适配器 apply_initial_plan,通过子进程调用既有公开控制面所有者(task_plan_packet、loopx goal-acceptance refine 与 loopx todo add),把原有多轮模型工具写入收敛为脚本内部可幂等重放的顺序事务,同时维持写入前后的权威读回校验与执行交接边界不变。将可选批量编排入口留在 benchmark/runtime/planning_batch.py 并直接经由公共 CLI 复用 goal-acceptance refine 与 todo add,把 benchmark 启动便利性与核心控制面权威读写、CAS 事务及执行准入边界彻底分离,无需新增任何共享持久化状态或新 schema,改动内聚且可独立回滚。 当前边界声明:仅限 LOOPX_TASK_STAGE=plan 下针对空工作与空需求前沿的初始 JSON 提案写入及原提案中断重放,非目标路径维持原有规划步骤,真实模型采纳率与跑分对比留待后续实测。

阶段 / 路径 基线 origin/main (1c540c3e5a) 本 PR Head (64204a6c72) 权威状态与恢复约束
LOOPX_TASK_STAGE=plan 空前沿 仅提供分步 goal-acceptance refine 与 todo add 指引 额外追加可选 planning_batch 命令提示(planning_batch_guidance) 仅写入现有 goal_outcomes 与 todo;execution_handoff.owner 保持 caller 且 0 quota
写入中断后重放 需模型自行人工对齐已写入 Todo 与声明 保留原 proposal.json 重跑,自动以固定 operation_id 重放 declare 与 todo:N 并补齐 bind 提案被篡改(outcome_id 哈希变化)时 _declared_outcome 在写前直接拒绝
非 plan 阶段 / 非空 Goal / 阻塞 走原有规划与执行流程 与基线完全一致(render_planning_prompt 逐字不变;apply_initial_plan 显式拒绝) 不改变 Core todo plan、/loopx、seeded-todo 或执行期提示

具体改动 / Changes

精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b,对比基线 1c540c3e5ae059f974abdfcfe27b42f5cad9e992,共修改 5 个文件(+371 / -8):生产运行时 2 个文件(benchmark/runtime/planning.py +6/-0、benchmark/runtime/planning_batch.py +205/-0)、公开运行文档 1 个文件(benchmark/runtime/RUNTIME.md +16/-7)、测试与夹具 2 个文件(benchmark/tests/test_task_entry.py +3/-1、tests/control_plane/test_benchmark_planning_batch.py +141/-0)。

关键代码讲解

  1. render_planning_prompt:仅当 env.get("LOOPX_TASK_STAGE") == "plan" 时拼接 planning_batch_guidance(),确保在未设置该阶段或 execute 阶段时对提示词正文零侵入,与基线输出逐字节相等;由 benchmark/runtime/worker.py 在 stage == "plan" 时调用,非死代码路径。
  2. apply_initial_plan:入口严格校验 LOOPX_TASK_STAGE == "plan"、_request(proposal) 结构、实时 task_plan_packet 的 input_digest 匹配、且无 goal_waiting_on 或 blocking_todo_ids。首次执行时在 provider_revision 匹配下要求 existing_todos 与 _outcomes(acceptance) 均为空;以 benchmark-plan:<input_digest> 为前导操作 ID,outcome_id 绑定整份规范化提案的 SHA-256 摘要,再经 goal-acceptance refine 声明并通过 todo add --operation-id 逐条写入 advancement_task Todo。
  3. _declared_outcome 与绑定前读回校验:当 provider_revision 已推进(断点重试场景)时,要求 current_lifetime is True 且已声明 outcome 的 outcome_id、owner_agent_id、objective 与全部 requirements 与原提案一致,杜绝同一 input_digest 下混用另一份提案的 Todo 写入;全部 Todo 写完后再次调用 task_plan_packet 验证全部 todo_ids 均仍在 runnable_todo_ids 中且无新阻塞,仅当 linked 为空时执行最终绑定,否则直接确认一致性并返回符合 TASK_PLAN_RESULT_SCHEMA 的就绪结果。
  4. planning_batch_guidance:使用当前 sys.executable 构造 -m benchmark.runtime.planning_batch 调用指令,清晰声明仅面向空工作与空需求前沿的可选入口、中断原样重跑规则,以及既有工作/需求修订/阻塞回退普通规划的边界。

规范依据与准则映射

依据修改前 origin/main 修订 1c540c3e5ae059f974abdfcfe27b42f5cad9e992 的既有契约文档 https://github.com/loopx-project/loopx/blob/1c540c3e5ae059f974abdfcfe27b42f5cad9e992/benchmark/runtime/RUNTIME.md#task-entry-and-planning-ablation 逐项映射(PR 对 RUNTIME.md 第 392-407 行的同步增补仅说明该可选批量提案提交方式及非目标边界,未放松任何现有校验门禁):

  • BENCH-PLAN-OWNERS(复用既有 goal-acceptance refine 与 todo add 所有者,不创建规划 Todo 或宿主循环):implemented — 由 benchmark/runtime/planning_batch.py apply_initial_plan 实现,经 tests/control_plane/test_benchmark_planning_batch.py 真实 File/SQLite 写入与读回验证。
  • BENCH-PLAN-READBACK(worker 独立读回 input_digest、身份、需求绑定与 runnable_todo_ids,保持 execution_handoff.owner == "caller" 且 0 quota 消耗):implemented — 由 benchmark/runtime/planning.py validate_plan_readback 与 benchmark/runtime/worker.py run_once 联合验证,test_benchmark_planning_batch.py 验证通过。
  • BENCH-PLAN-RECOVERY(写入中断后原提案安全重放且不重复产生 Todo,变更提案在写前拒绝):implemented — 由 apply_initial_plan 与 _declared_outcome 实现,经 test_benchmark_planning_batch.py 的 :todo:1 丢包测试及本次评审补充的 :declare 与 :bind 丢包反事实探针验证。

对主干的风险 / Risks

  • 活跃调用路径与默认关闭隔离:planning_batch_guidance() 仅在 worker.py stage == "plan"(即 LOOPX_TASK_ENTRY=loopx-planned 且 LOOPX_TASK_STAGE=plan)通过 render_planning_prompt 注入;对照验证确认无环境变量或 LOOPX_TASK_STAGE=execute 时输出与基线逐字节一致,apply_initial_plan 对非 plan 阶段直接报错拒绝,Core loopx todo plan 与 seeded-todo 完全隔离。
  • 失败场景与反事实核验:针对最易出错的中间态丢包场景,实测分别在 File 与 SQLite 后端上拦截 :declare 首次声明回执、:todo:1 第二条 Todo 创建回执及 :bind 最终绑定回执,原样重跑均精确恢复原有 3 条 Todo 与唯一绑定(不产生重复 Todo 或二次绑定);修改 input_digest、替换提案内容、在非空 Goal 下用当前版本重提或处于阻塞态均触发前置拒绝且不污染状态。
  • 权限、类型规则与领域中立性:apply_initial_plan 仅调用现有控制面 CLI 且保持规划期 0 quota 消耗与 caller 执行准入权,不新增任何角色或生命周期状态;规则检查均为结构化字典/字段集合校验,未引入脆弱子串 denylist;新增代码位于 benchmark/runtime/,未向核心控制面引入领域特定耦合。

语义与 CI 对齐

本 PR 仅新增 Benchmark 运行时辅助入口,未改动共享控制面状态词表或 RFC 契约(semantic_alignment: not_applicable)。本地直接校验结果(未等待或参考远程 CI,wait_for_ci: false):

  • tests/control_plane/test_benchmark_planning_batch.py:PASS(4/4 passed,覆盖 File/SQLite 正常路径、:todo:1 丢包恢复、改写提案拒绝、SForge prepare_entry 与独立 quota should-run 准入)。
  • benchmark/tests/test_task_entry.py:59 passed, 54 skipped, 8 failed — 54 项跳过均为缺少可选 harbor 依赖;8 项失败均为 test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input 中 complete=True 时断言 "--complete-turn" in body,已在合并基线 1c540c3e5ae059f974abdfcfe27b42f5cad9e992 上复现完全相同的 AssertionError 失败签名,归类为 pre_existing_unrelated。
  • 独立反事实探针(File/SQLite :declare 与 :bind 丢包重放、stage 关闭一致性、错误 input_digest 与非空 Goal 拒绝):PASS。
  • ruff check(全部改动 Python 文件)、git diff --check、scripts/check_rfc_bilingual.py、loopx check --scan-path(公开边界扫描 5 个改动文件):PASS。
  • loopx canary premerge --tier quick --from-git-diff --git-diff-base 1c540c3e5ae059f974abdfcfe27b42f5cad9e992:4 项直接检查、py_compile 与 3 项目录金丝雀(含维持性棘轮、语义漂移与 benchmark 权限策略 smoke)全部 PASS(failures=0;因涉及 benchmark_sensitive 路径返回 manual_review_required,已由本次精确头审查覆核)。
  • mypy(仓库配置的 17 个 strict 文件):报 1 项与本 PR 无关的既有错误(未改动的 interaction_contract.py:1004),已在精确基线 1c540c3e5ae059f974abdfcfe27b42f5cad9e992 上验证完全一致(pre_existing_unrelated);对改动文件单独执行 --strict 仅报告未标注的测试/内部辅助参数注解提示,未发现类型逻辑错误。

我的整体评价 / Verdict

从 outcome_impact 评估:long_horizon 为 improved(初始规划从多轮工具写收敛为单命令可重放提交,中断后可沿固定 operation_id 与摘要绑定原提案续接并防止重复创建 Todo 或漂移绑定),user_experience 为 improved(减少规划阶段交互轮次的同时保留清晰读回与错误提示)。本 PR 仅涉及 benchmark 运行时 CLI/prompt 与测试文档,不涉及任何渲染界面改动,visual_review 为 not_applicable(已审阅全部 5 个改动文件,未验证界面范围 unverified_scope 为 none)。代码体积与架构完全合比例(code_volume: necessary、change_proportionality: proportionate、default_off_isolation: isolated、authority_semantics: not_applicable、local_only 无词表扩张)。残余风险在于本次本地验证基于隔离 File/SQLite 夹具与模拟 planner 进程,未运行真实远端 LLM 或可选 harbor 容器集成,实际模型采纳率与跑分收益有待后续基准对照测量。当前远程 PR 仍位于目标精确 head 64204a6c7215241c1a7253f146e12640ca1fc14b。

英文简短结论:
English verdict: APPROVE — Exact head 64204a6c7215241c1a7253f146e12640ca1fc14b adds an optional initial-planning batch adapter in benchmark/runtime/planning_batch.py that reuses goal-acceptance refine and todo add with digest-bound idempotent replay on File and SQLite, preserves stage-off parity and caller-owned execution handoff, passes test_benchmark_planning_batch.py (4/4), non-Harbor test_task_entry.py checks (with pre-existing --complete-turn failures verified identically on base 1c540c3e5a), counterfactual drop probes, Ruff, bilingual/boundary scans, and quick premerge canaries (failures=0). Live model adoption and Harbor container matrix remain unmeasured.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant