Skip to content

Verify the historical Short benchmark carrier before launch - #8989

Merged
huangruiteng merged 3 commits into
mainfrom
codex/c1-benchmark-carrier-20261118t042507
Nov 18, 2026
Merged

huangruiteng merged 3 commits into
mainfrom
codex/c1-benchmark-carrier-20261118t042507

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

The benchmark worker accepted an explicit historical Short arm even when the pinned product delivered the complete C1 contract. A source-pinned reproduction through actual worker-generated guards returned raw, v0 and v1 on the historical revision and v1 for all three requests on current main. The worker now verifies v0 before dispatching a model and records the expected and observed versions and eligibility in the wake receipt. An unavailable or incompatible preview fails at that boundary. See Review-A's finding on #8956.

Ordinary and explicit Complete wakes use the shared C1 source. Historical Short uses its recorded product and runner pins. A fresh-replan checkpoint preloads a distinct contract, so that combination fails before a Short launch.

Review follow-up

The Short preview now preserves the existing phase stop when its subprocess uses the phase's remaining allowance. The runner returns exit 75 with the original budget receipt, so the existing scheduler can admit a later phase. A timeout against the preview's independent 120-second limit still produces the Short qualification error, even when the phase has elapsed by the time the error is handled.

The timeout regression failed against the previous head and passes after the fix. With a source-pinned historical v0 CLI and an intentionally delayed preview, the actual worker and scheduler exercised isolated File and SQLite stores. Both returned exit 75 and created a successor from the unchanged predecessor. Each run recorded zero Host launches and zero new admissions. The focused regression also covers the independent timeout. Review finding

Author Declaration

Written by: model_agent, OpenAI GPT-6 family (Codex).

Implemented against

C1 contract and consumer migration at the PR base

Criterion Disposition Implementation Validation
Ordinary and Complete use the shared source through the real worker and checkpoint boundary Implemented Worker display receipt and current native fixtures Current default and Complete guard, unchanged wake scope and original capture with one settlement
Historical comparison records the actual carrier and refuses an unsupported source before the model Implemented Worker quota preview and wake receipt Pinned CLI, File and SQLite, plan and execute stages, v0 positive and v1 refusal
Preserve frozen controls and qualify outcomes and complete costs separately Deferred for the parent C1 acceptance Frozen study sources and historical captures Current work measures source and transport behavior with local fixtures

Scope And Continuation

The change is limited to benchmark startup, its receipt and documentation, and the affected real-path validation fixtures. The historical preview adds one quota CLI read per Short wake. Report that read in the total cost of a new attempt. This proof covers the requested carrier and transport. Matched outcomes, complete costs and installed rollout qualification remain in the C1 program.

Current validation

Tested head 0237fa838196e670a70e778454a46a4dea7c3ff7, integrated with main at 28ae33cf7efbfdc1f0c424b061db5af6d0f441db.

Check Result
Relevant Python regressions 311 passed. 220 skipped for optional Harbor, native Codex, Docker or Linux dependencies. The previous installed rollout-profile limitation remains open.
Actual local worker and scheduler File and SQLite reproduced the phase stop and verified continuation from the original receipt. The independent preview timeout stays a refusal.
Static, public-boundary and RFC checks The changed Python paths, repository-declared Ruff scope, configured mypy, signed diff and paired RFC checks passed. An extra all-path Ruff scan found 110 identical diagnostics on the immutable base and head in unchanged files.
Local premerge All five direct and 19 selected checks passed. The exact change-quality receipt is valid. The benchmark maintainer review hold remains.

Previous candidate validation

The changed paths are identical between tested revision 0364b9762ce38827fa57bd13266d77a681eca6d3 and submitted revision 5831cb867dd307e2aea3e813a3a4ccfd9ea61ca6. The rebase brings in separate manager-inbox changes. The six startup tests, repository Ruff and paired RFC diff checks were rerun on the submitted revision.

Check kind Result Public-safe evidence / limitation
Real entrypoint and backend Passed Four C1 rejection cases across File and SQLite fail before Host preparation, leave state unchanged and create zero admission receipts. Regressions failed at an independent Host sentinel before the repair. Two scripted legacy and ordinary receipt cases also pass.
Historical CLI Passed A recorded v0 product reaches the Host-preparation boundary in both phases; its fresh-replan combination is refused. The historical and current worker guard captures verify the actual source versions with zero model calls.
Checkpoint and deadline transport Passed Eight cases cover ordinary and explicit Complete with fresh and resumed execution and local observations. The current original checkpoint capture is v1 in both settings.
Native guard and recovery Passed Eight local loopback cases on Codex 0.169 cover current default, Complete, resume, hook loss and checkpoint handoff. Four history replacement and profile-loss cases pass on the already qualified Codex 0.162.1 profile.
Installed history profile Failed prerequisite Codex 0.169 rollout metadata is outside the current replay profile. Four history replacement/profile-loss cases could not establish a qualified anchor on that version. The qualification boundary stays active.
Broader local regression run Mixed 250 passed and 197 optional cases skipped. Four old checkpoint test expectations failed on the new v1 default, then passed in the checkpoint rerun.
Static and scope Passed Changed Python and product Ruff checks, 17-file mypy, paired RFC validation, public-boundary scan and signed diff hygiene pass.
Repository-wide Ruff Failed The same F811 diagnostic occurs on the immutable PR base and proposed head in the untouched steward adoption test. This inherited repository validation failure remains a merge hold.
Premerge gate Held All five direct and 19 selected checks pass after installing the checkout's locked TypeScript development dependency. The change-quality receipt and benchmark maintainer review still hold merge.

Standalone EdgeBench help needs the optional SForge and Harbor launcher dependencies. The worker and checkpoint tests exercised the real product CLI with synthetic data, using local scripted inference for native Host checks.

Frontend / Visual Evidence

The changed caller surfaces are benchmark CLI help and private wake receipts.

Review And Merge

Please verify the exact head, historical v0 qualification, ordinary C1 path parity and the current validation holds before integration.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

[P2] 保留 Short 预演耗尽阶段预算时的正常退出。 同一有效历史 CLI 只将预演读操作延后。阶段在预演前已耗尽时,worker 以 75 正常结束并写入 budget_exhausted=true。阶段在预演中耗尽时,worker 以 1 退出、标准输出为空,回执把失败写成 host_failure.kind=unknown。两条路径均未启动 Host,状态和准入数不变。调度器只为带 75 和原截止回执的前置停止建立后续阶段。 请保留预演所用的是阶段剩余时间还是独立的 CLI 超时。阶段到期按原预算截止退出,独立超时继续拒绝启动。超时捕获 原阶段续跑

评审提交 5831cb867dd307e2aea3e813a3a4ccfd9ea61ca6,基线 a79e78134063c28cbb4598625b06a0f5a25f6c3f。精确差异

动机 / Motivation

复现历史 Short 处理的 benchmark 操作者需要核对模型实际会使用哪种输入。同一个 Short 设置在旧产品上收到 v0,在当前产品上会收到完整 C1,原 worker 仍会进入 Host 准备。候选用真实产品 CLI 的只读预演阻止错误载体进入规划和执行模型,并记录版本与启动资格。这解决 #8956 已复现的 Short 错标。阶段预算在 Short 预演中耗尽时会误报为来源不可用,续跑路径随之失去依据。实际模型效果、完整成本、正式安装端的历史回放和基准发布资格仍待独立验证。原评审

改动思路 / Approach

本片覆盖 benchmark worker、局部 wake 回执、帮助与对应回归测试。保留 product CLI 的版本和结算决定权,由现有 worker 执行启动前校验并复用调度器的阶段截止语义。同一选定 CLI 先以 dry-run 输出版本,执行唤醒仍由原 renderer 生成正式 guard,当前普通与 Complete 路径跳过这个预演。按版本名直接放行无法发现产品已将兼容参数映射为 C1。保留现有相位判断即可修复本次停止回归。启动和原输入

具体改动 / Changes

实际 11 个路径 +226/-79,文件、对象与 GitHub 比较一致。两个生产路径是 worker 与 EdgeBench 帮助,三处测试覆盖前置拒绝、checkpoint、原生送达与恢复,六处说明反映当前默认和冻结历史。

验收依据为 C1 调用方替换与迁移,版本 a79e78134063c28cbb4598625b06a0f5a25f6c3f。

  • 真实调用与历史版本 已验证。隔离 File 的实际执行和 SQLite 的规划前置均接受历史 v0。当前 v1 在 Host 准备前被拒绝。将历史 v0 与 fresh-replan 组合也在同一步拒绝。预演保持原状态、零准入。历史与当前入口
  • 原捕获与结算 已验证。真实 renderer 生成的显式 guard 在历史产品返回 v0,在基线及本头 C1 返回 v1。正式 guard 各有一份原捕获与一次准入,当前产品走 v2 结算 plan。源码测试另覆盖未设 Short 的默认和显式 Complete、checkpoint 原捕获与重复结算拒绝。原 checkpoint
  • 失败及阶段停止 未满足。已验证的格式拒绝继续适用。只有预算在预演中用尽时,当前启动生命周期破坏原阶段续跑证明。阶段收据筛选
  • 匹配成果和完整成本 留在原 C1 验收。每次 Short 预演增加一次 CLI 读取,纳入新试验成本。度量和冻结控制

关键代码讲解

  • _qualify_short_envelope 在准备 Host 前比较真实产品版本。现有 _phase_timeout 已提供阶段限额标记,异常分支需使用这个信息。预演与限额
  • run_once 向决策记录加入局部资格字段,使用同一启动收据及原 guard。阶段进入时已过期的提前结束路径给出了本次可复用的结果。正常截止
  • heartbeat_body 从同一 CLI 生成绑定原 Turn 的命令。resume_state_file 只用成功读回的原阶段截止和未启动证据来决定是否建立新阶段。生成入口 接续入口

对主干的风险 / Risks

失败集中在显式历史 Short。只有实际压到阶段末尾,操作者才会看到错误的固定来源诊断,并失去正常续跑依据。修复应覆盖真正的阶段到期和独立 CLI 超时。两种错误都应保留零 Host、零准入及原状态。

PASS。独立运行 24 项启动、checkpoint 与原结算组合用例,真实本机 Codex 使用隔离脚本推理的 8 项用例通过。变更 Python lint 和双语 RFC 验证通过。5 项直接检查通过,19 项选定检查经两项独立重跑后全部通过。首次语义词表检查超时,quota smoke 在隔离参数下无法导入相邻 fixture。 基线跑同一组 checkpoint 测试时有四处旧断言期待空版本,实际是 v1。候选更新断言后四例都通过。FAIL。同一仓库 lint 命令在基线和候选均只报未改的管家测试 F811,路径、行号、代码和消息相同。此合并门须单独清理。 UNVERIFIED。本机 Codex 0.169 的一项独立替换历史探针在读取 native_resume 时失败。正式分页回放仍待验证。 此轮按 wait_for_ci=false 使用本地检查。库内检查命令

语义与 CI 对齐

版本比较复用已有枚举,资格只进入 worker 的诊断收据。拦截不兼容输入没有扩大原读写归属。保留阶段停止要复用现有 scheduler 收据和退出码。用同一历史 CLI 分别触发阶段和独立预演超时,验收两种结果以及正常和显式 Complete 的接续。 预算回归覆盖

我的整体评价 / Verdict

请求修改这个精确提交。历史 Short 的实际 v0 资格和当前 C1 保留路径都有独立证据。保留 product CLI 的版本和结算决定权,由现有 worker 执行启动前校验并复用调度器的阶段截止语义。本片覆盖 benchmark worker、局部 wake 回执、帮助与对应回归测试。修复后的 worker 应在阶段到期时保留正常续跑证据,启动资格错误保留清楚的独立诊断。

English verdict: REQUEST_CHANGES at 5831cb8. v0 qualification and current C1 paths are independently verified. A phase-limited Short preview exits 1 with an unknown Host failure instead of the existing budget stop, preventing normal continuation.

Comment thread benchmark/runtime/worker.py Outdated
}
receipt.update(quota_input_contract=contract, host_invoked=False)
deadline = env.get("LOOPX_PHASE_DEADLINE_EPOCH")
timeout, _ = _phase_timeout(120, float(deadline) if deadline else None)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里丢弃了 _phase_timeout 的阶段限额标记,并将 TimeoutExpired 转成来源错误。在真实 worker CLI 中,只延迟有效历史 pin 的只读预演,阶段到期后便返回 1,回执是 host_failure.kind=unknown。同一夹具在预演前用尽阶段预算则返回 75、budget_exhausted=true,两次均未调用 Host。后续调度依赖 75 及原截止回执。请在阶段限额选中的超时时沿用现有预算停止,独立的 CLI 超时仍保持资格失败,并用这两种超时复验。

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…-benchmark-carrier-20261118t042507

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

评审提交 0237fa838196e670a70e778454a46a4dea7c3ff7,基线 28ae33cf7efbfdc1f0c424b061db5af6d0f441db。精确差异

动机 / Motivation

用历史 Short 读取方式做匹配比较的 benchmark 操作者,要核对实际输入及预算耗尽后的续跑。旧评审发现 Short 预演期间用尽阶段预算时,worker 误报来源故障,调度器因此无法建立下一阶段。候选保留正常预算停止回执,调度器可从原状态续跑;独立预演超时仍报来源资格失败。本次覆盖真实本地 worker、产品 CLI、现有调度器和对应的用法与验证。实际安装端的历史回放、匹配成果和完整成本仍由 C1 原验收跟踪。 原 P2

改动思路 / Approach

版本读取仍由固定的 product CLI 提供,worker 保留选中超时的原因,已有调度器负责读取预算停止回执并续跑。改动覆盖 benchmark 启动、私有 wake 回执、CLI 帮助及对应回归与协议。 用已知有效的历史 v0 CLI,只延后预演读。在 File 和 SQLite 的真实 worker 中,阶段限额选中时返回 75。独立的 120 秒预演限额选中时仍返回 1。两条路径的实际耗时和最终退出回执各自可核对,调用前的选择决定错误类别。原 phase 的状态不被替换。超时分支 调度读回

具体改动 / Changes

12 个文件,+269/-79。验收依据为 C1 当前契约,版本 28ae33cf7efbfdc1f0c424b061db5af6d0f441db。

  • 当前默认与原身份。实际 worker renderer 的默认和 Complete 均绕过 Short 预演,生成 guard 的原捕获均为 v1、结算计划 v2,各一次准入和捕获。入口
  • 历史 Short 资格。固定历史 v0 可达到 Host 准备前边界,随后生成外部 guard 仍是 v0。当前 v1 及历史 v0 加 fresh_replan 均在启动模块前拒绝,结果零准入。预演资格
  • 阶段停止及独立超时。File、SQLite 的阶段超时均为 75、budget_exhausted=true,零 Host、零准入。用真实回执和隔离调度器状态调用现有续跑入口,创建同一后继,重放复用它且原文件不变。独立 120 秒超时为 1,续跑入口保留原状态。原停止回执
  • 安装和匹配成本 沿原 C1 验收。实际启动有一条新增只读 CLI 请求,须计入针对历史 Short 的完整成本。

对主干的风险 / Risks

本地定向测试 159 过、120 项可选依赖场景跳过。仓库规定 Ruff、改动文件 Ruff、DCO 与双语 RFC 检查通过。standard premerge 首次为 5 项直接和 18 项选中通过,quota plan smoke 因助手进程的安全导入路径找不到同目录夹具。恢复普通脚本导入路径后,同头该项通过。benchmark 维护者的当前评审门和实际安装回放资格仍保留。

我的整体评价 / Verdict

原阶段截止 P2 已在两种后端复验修复。当前和历史输入回执保持明确,后续 phase 可用。实际入口是 benchmark CLI 和私有回执,差异未涉及渲染页面。

English verdict: APPROVE. The selected phase timeout preserves the original budget stop and successor. An independent source timeout still fails qualification.

@huangruiteng
huangruiteng merged commit a61777e into main Nov 18, 2026
25 of 38 checks passed
@huangruiteng
huangruiteng deleted the codex/c1-benchmark-carrier-20261118t042507 branch November 18, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants