Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 22 additions & 24 deletions benchmark/edgebench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1003,30 +1003,28 @@ budget or submissions. Freeze it with exact sources under a new run ID; do not
change live workers. Receipts qualify requested configuration, while actual
review, proposal adoption and task gains require trajectory evidence.

## Optional short-context treatment

Add `--turn-envelope` to an otherwise identical `heartbeat-resume` or
`heartbeat-explore` command to use the product's short normal packet and local
same-invocation full detail. Other workers reject the option before creating a
trial. The default remains off; the runtime and worker-profile receipts record
it explicitly. Use a new run ID and freeze this as a separate context treatment,
keeping objective guidance, task/scorer version, model and budgets matched.
Omit the flag on the next run to disable it. No active experiment is changed by
installing this code. Inspect wake receipts and the saved decisions before
claiming adoption; score countability still requires the usual integrity review.

For an explicitly complete v1 Host contract, choose `--complete-turn` instead
on either heartbeat profile. The shared worker forwards that choice through
the real heartbeat renderer and guard; it retains the original ready claim,
receipt and single-command settlement. It cannot be combined with the legacy
short-envelope option. Context delivery remains independent, and unselected
runs keep their existing behavior. See the [Host protocol](../../docs/reference/protocols/turn-envelope-v0.md#explicit-complete-v1-host-entry--显式完整-v1-host-入口)
for source readback, failure recovery and rollback. User task time requirements
remain readable; private Host deadlines stay outside model context.

中文:heartbeat profile 可显式选择 `--complete-turn` 以消费完整 v1,不能与旧短包
选项同时启用;共用原 ready claim、回执和单条结算 owner,不改变未选择的运行。
按原 source reader 核实采用与恢复;该开关不新增模型、子 worker、时钟或权限。
## Quota input and historical Short

Current C1 sources give both heartbeat profiles the complete v1 input by
default. `--complete-turn` explicitly requests the complete contract, including
on supported historical product revisions. Context delivery remains a separate
setting. The [Host protocol](../../docs/reference/protocols/turn-envelope-v0.md#explicit-complete-v1-host-entry--显式完整-v1-host-入口)
documents original captures, readback and settlement.

For a historical Short benchmark arm, use its recorded product and runner
revisions. With `--turn-envelope`, the current worker verifies the selected CLI's
actual v0 response using a read-only preview before starting a model. C1 or an
unavailable preview rejects the request. A `fresh_replan` checkpoint preloads a
different view and is incompatible with Short. Inspect the wake's
`quota_input_contract` eligibility and the actual saved decision before using
a result in a comparison. Include startup and wake costs under the same study
protocol.

中文:当前 C1 的两种 heartbeat profile 默认交付完整 v1,`--complete-turn`
也可在支持的历史产品版本中显式选择完整输入。旧 Short 组应按记录的产品和 runner
版本复现。当前 worker 在启动模型前用只读预演核实 v0,回执
`quota_input_contract` 记录预期、实测版本与资格。C1 或无法核实的来源会拒绝请求。
`fresh_replan` 会预载另一种输入,因此不能与 Short 合用。

## Close a managed native attempt

Expand Down
2 changes: 1 addition & 1 deletion benchmark/edgebench/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ def main(argv=None):
help="Opt-in private binding for native terminal proof, consumed by the controller")
turn_view = parser.add_mutually_exclusive_group()
turn_view.add_argument("--turn-envelope", action="store_true",
help="Opt-in short heartbeat context with same-invocation full captures")
help="Historical Short envelope treatment; the worker requires a verified v0 quota response")
turn_view.add_argument("--complete-turn", action="store_true",
help="Explicit complete v1 Turn contract with shared exact-Turn settlement; heartbeat workers only")
parser.add_argument("--context-delivery", choices=("off", "on"),
Expand Down
27 changes: 16 additions & 11 deletions benchmark/runtime/RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,17 +81,22 @@ agents:
replan_after_turns: 6
```

For an explicitly selected **heartbeat-only** context experiment, add
`turn_envelope: true` to `kwargs` (default `false`). Each wake creates a private
capture root and asks the product renderer for a short TurnEnvelope dispatcher.
Full decisions remain available through `detail_ref.full_decision` from the same
guard invocation. Selection/reentry still evaluate current authority; generated
selection commands retain the capture root. When enabled, the setting appears in install,
execution and wake receipts as `turn_envelope: true`. When disabled, the field
and worker environment override are absent, preserving the pre-opt-in shape. Invalid native-Goal/plain/Turn-driver combinations
fail before execution. Set it back to `false` for the next run to roll back;
do not change a running trial's treatment. Captures contain private Goal context.
Transport tests do not establish lower token cost or better model decisions.
Historical Short trials request `turn_envelope: true` in heartbeat `kwargs`.
Before planning or execution starts a model, the worker previews the selected
product's quota response through the same CLI. Its wake receipt records the
expected and observed versions and eligibility in `quota_input_contract`. C1
sources return v1 and the Short request fails before Host preparation. A
`fresh_replan` checkpoint preloads a distinct view and also prevents a Short
launch. The preview has no Turn identity and creates no admission or capture.
Ordinary and explicit Complete wakes proceed without this Short preview.

Use the recorded product and runner revisions to reproduce a historical
treatment. On a qualified v0 source, an execution wake records
`turn_envelope_source=explicit` once it renders the Short bootstrap. Its
same-invocation full decision remains available through `detail_ref.full_decision`.
The planning prompt has its own input. Include the preview in startup
and wake cost when measuring a new attempt with this runner. Keep captures
private and preserve the settings and receipts of existing attempts.

New native heartbeat trials default to context delivery `on`. Add
`context_delivery: "off"` to the same heartbeat agent `kwargs` to disable it.
Expand Down
16 changes: 9 additions & 7 deletions benchmark/runtime/SETTINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ use matched repetitions to test its effect.
| --- | --- | --- |
| Task entry | LoopX modes: loopx-planned | Record planned or seeded for every arm |
| Explore | Off in heartbeat-resume; on in heartbeat-explore | Keep resume as the reference |
| Turn envelope | Off; a checkpoint handoff enables capture on fresh_replan execute wakes and receipts record `turn_envelope_source: checkpoint_handoff` | Enable only in its ablation; read the effective receipt, not the opt-in flag |
| Quota input | Current C1 product sources return the complete v1 contract on ordinary and explicit Complete wakes | Historical Short needs a verified v0 product response before model launch |
| Native-Goal continuation | Off: a native Goal that declares completion ends the worker; a Goal still active at the budget edge is recorded as `native_goal_outcome: budget_exhausted` with exit 0 | `native_goal_continue: true` / `--native-goal-continue` starts continuation Goals until 300 s before the budget edge (native-goal profile only) |
| Replan cadence | 6 settled effective work Turns | Use explicit completed-Todo cadence or another Turn count for ablation |
| Iteration context | Harbor: fresh; EdgeBench heartbeat: resume | Freeze the provider and context within a comparison |
Expand Down Expand Up @@ -48,7 +48,7 @@ These are recommendations, not automatically launched experiments.
| Reference | `--worker heartbeat-resume --task-entry loopx-planned --replan-after-turns 6` | Planned entry with effective-turn replanning |
| Seeded | Replace only `--task-entry` with `seeded-todo` | Does initial task decomposition help? |
| Explore | Replace only `--worker` with `heartbeat-explore` | Are recorded evidence and subsequent route choices useful? |
| Short envelope | Add `--turn-envelope` | Does progressive context loading reduce overhead without losing decisions? |
| Historical Short envelope | Use the recorded product and runner revisions with `--turn-envelope`; the current worker verifies v0 before launch | Does short input with reachable original detail reduce overhead while preserving decisions? |
| Todo cadence | Replace `--replan-after-turns 6` with `--replan-after-todos 3` | How do units and thresholds change interruptions? This is a combined ablation. |
| No LoopX | `--worker official`; omit LoopX-specific flags | What is the net effect of the whole LoopX treatment? |

Expand Down Expand Up @@ -93,11 +93,13 @@ configuration unchanged.

## Readback and interpretation

TurnEnvelope projections reuse the validated checkpoint declaration. An explicit
opt-in records `turn_envelope_source=explicit`; a known `fresh_replan` declaration
for heartbeat records `checkpoint_handoff`. Omitting `trigger` means after-wake
capture and does not imply TurnEnvelope. A programmatic guest path without
declaration bytes records capture configuration only: it cannot prove a trigger.
The Short setting in the install profile records the request. Before a model
starts, the worker records the read-only quota preview's expected version,
observed version and eligibility in `quota_input_contract`. The wake records
`turn_envelope_source=explicit` after v0 qualification. A `fresh_replan` checkpoint
preloads a different view and is incompatible with this Short launch. An omitted
trigger means after-wake capture. Interpret saved attempts against their pinned
source and recorded carrier.
For the Harbor adapter, pass the same validated `checkpoint_request` bytes with
their `checkpoint_request_path` when projecting that known configuration. SForge
forwards the exact bytes it stages. The worker's individual wake receipt remains
Expand Down
68 changes: 63 additions & 5 deletions benchmark/runtime/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,13 @@
from dataclasses import replace
from pathlib import Path

from benchmark.runtime.codex import Execution, explicit_turn_display, prepare_codex_home, process_environment
from benchmark.runtime.codex import (
Execution,
explicit_turn_display,
prepare_codex_home,
process_environment,
turn_envelope_receipt,
)

from benchmark.runtime.scheduler import PHASE_WORK_RESERVE_SECONDS as PHASE_WORK_RESERVE_SECONDS
from benchmark.runtime.scheduler import phase_work_reserve
Expand Down Expand Up @@ -523,6 +529,57 @@ def _phase_timeout(timeout: float | None, deadline: float | None) -> tuple[float
return timeout, False


def _qualify_short_envelope(env: dict[str, str], receipt: dict) -> bool:
"""Verify the pinned Short carrier, returning False if the phase expires."""
from loopx.control_plane.quota.turn_envelope import (
COMPLETE_TURN_DECISION_VERSION,
TURN_ENVELOPE_SCHEMA_VERSION,
)

contract = {
"expected_schema_version": TURN_ENVELOPE_SCHEMA_VERSION,
"observed_schema_version": None,
"eligible": False,
}
receipt.update(quota_input_contract=contract, host_invoked=False)
# This preview runs the same product projection without a Turn, capture,
# or admission. Keep optional delivery out of the carrier observation.
command = loopx_command(env) + [
"quota", "should-run", "--dry-run", "--runtime-profile", "generic_cli",
"--goal-id", env["LOOPX_GOAL_ID"], "--agent-id", env["LOOPX_AGENT_ID"],
"--available-capability", "shell", "--available-capability", "filesystem_write",
"--turn-envelope", "--context-delivery", "off",
]
deadline = env.get("LOOPX_PHASE_DEADLINE_EPOCH")
timeout, phase_limited = _phase_timeout(120, float(deadline) if deadline else None)
try:
result = subprocess.run(command, cwd=env["LOOPX_PROJECT"], env=env,
capture_output=True, text=True, timeout=timeout, check=True)
payload = read_json(result.stdout)
except (OSError, ValueError, subprocess.SubprocessError) as exc:
if phase_limited and isinstance(exc, subprocess.TimeoutExpired):
receipt.update(ok=True, budget_exhausted=True)
return False
raise ValueError("Short envelope requires a successful v0 quota preview from the pinned product.") from exc
if payload.get("ok") is not True:
raise ValueError("Short envelope requires a successful v0 quota preview from the pinned product.")
observed = payload.get("schema_version")
if observed in (TURN_ENVELOPE_SCHEMA_VERSION, COMPLETE_TURN_DECISION_VERSION):
contract["observed_schema_version"] = observed
if observed != TURN_ENVELOPE_SCHEMA_VERSION:
raise ValueError("Short envelope requires a verified v0 response from the pinned product. "
"Use the recorded historical source pins for that arm.")
if checkpoint_path := env.get("LOOPX_CHECKPOINT_REQUEST"):
from .checkpoint import read_checkpoint_request

request, _ = read_checkpoint_request(checkpoint_path)
if request.get("trigger") == "fresh_replan":
raise ValueError("Short envelope requires the direct v0 guard. "
"A fresh_replan checkpoint preloads a different view.")
contract["eligible"] = True
return True


def run_once(env: dict[str, str]) -> dict:
if env.get("LOOPX_CONTEXT_DELIVERY", "off") not in {"off", "on"}:
raise ValueError("LOOPX_CONTEXT_DELIVERY supports only off or on")
Expand Down Expand Up @@ -585,10 +642,7 @@ def run_once(env: dict[str, str]) -> dict:
"task_entry": execution.task_entry,
**({"execution_observation": observation_producer}
if observation_producer is not None else {}),
**({"turn_envelope": True, "turn_envelope_source": "explicit"}
if execution.turn_envelope else {}),
**({"complete_turn": True, "complete_turn_source": "explicit"}
if execution.complete_turn else {}),
**turn_envelope_receipt(explicit=False, complete=execution.complete_turn),
**({"context_delivery": "on"} if execution.context_delivery != "off" else {}),
**({"native_child_limit": execution.native_child_limit}
if execution.native_child_limit is not None else {}),
Expand Down Expand Up @@ -629,6 +683,8 @@ def run_once(env: dict[str, str]) -> dict:
execution, timeout_seconds=(remaining - reserve if execution.timeout_seconds is None
else min(execution.timeout_seconds, remaining - reserve))
)
if execution.turn_envelope and not _qualify_short_envelope(env, receipt):
return receipt
prepare_codex_home(
home,
execution=execution,
Expand Down Expand Up @@ -678,6 +734,8 @@ def run_once(env: dict[str, str]) -> dict:
elif execution.mode in {"heartbeat", "loopx-goal"}:
body = heartbeat_body(env, turn_id, native_goal=execution.native_goal,
decision_handoff_socket=checkpoint_wake.socket_path)
if execution.turn_envelope:
receipt.update(turn_envelope_receipt(explicit=True))
elif execution.mode == "plain":
body = Path(env["LOOPX_TASK_DOC"]).read_text(encoding="utf-8")
session_wake = None
Expand Down
68 changes: 68 additions & 0 deletions benchmark/tests/test_shared_codex_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -990,18 +990,86 @@ async def task_exec(**kwargs):
assert receipt["turn_envelope"] is True


@pytest.mark.parametrize("stage", ["plan", "execute"])
@pytest.mark.parametrize("provider", ["file", "sqlite"])
def test_short_wake_rejects_c1_before_host_preparation(tmp_path, monkeypatch, stage, provider):
from tests.control_plane.canonical_authority_fixture import isolate_sqlite_runtime
from tests.control_plane.test_quota_settlement_cli import (
AGENT_ID, GOAL_ID, _heartbeat_receipt_count, _write_fixture,
)

isolate_sqlite_runtime(tmp_path, monkeypatch)
project, runtime, registry = _write_fixture(tmp_path, provider=provider)
state = project / f".codex/goals/{GOAL_ID}/ACTIVE_GOAL_STATE.md"
original_state = state.read_bytes()
wakes = tmp_path / "wakes"
home = tmp_path / "codex-home"
source = Path(__file__).resolve().parents[2]
env = dict(
os.environ,
PYTHONPATH=str(source), LOOPX_PYTHON=sys.executable, LOOPX_USAGE_PING="0",
LOOPX_CLI=str(Path(sys.executable).parent / "loopx"),
LOOPX_PROJECT=str(project), LOOPX_REGISTRY=str(registry), LOOPX_RUNTIME_ROOT=str(runtime),
LOOPX_GOAL_ID=GOAL_ID, LOOPX_AGENT_ID=AGENT_ID, LOOPX_WAKE_LOG_DIR=str(wakes),
LOOPX_CODEX_HOME=str(home), LOOPX_EXECUTION_MODE="heartbeat",
LOOPX_SHARED_SKILLS=str(tmp_path / "shared-skills"),
LOOPX_TASK_STAGE=stage, LOOPX_TURN_ENVELOPE="1", LOOPX_CONTEXT_DELIVERY="off",
MODEL_NAME="fixture", REASONING_EFFORT="low",
)

def unexpected_host_preparation(*args, **kwargs):
pytest.fail("A C1 response cannot start the Short envelope arm")

monkeypatch.setattr(worker, "prepare_codex_home", unexpected_host_preparation)
with pytest.raises(ValueError, match="Short envelope requires"):
run_once(env)
receipt_path, = wakes.glob("*/receipt.json")
receipt = json.loads(receipt_path.read_text())
assert receipt["ok"] is False and receipt["host_invoked"] is False
assert receipt["quota_input_contract"] == {
"expected_schema_version": "loopx_turn_envelope_v0",
"observed_schema_version": "loopx_turn_envelope_v1",
"eligible": False,
}
assert "turn_envelope_source" not in receipt
assert _heartbeat_receipt_count(runtime, receipt["turn_id"]) == 0
assert state.read_bytes() == original_state
assert not home.exists()


@pytest.mark.parametrize("enabled", [False, True])
def test_wake_receipt_preserves_opt_out(tmp_path, monkeypatch, enabled):
from benchmark.tests.test_resume_sessions import resume_env
env = resume_env(tmp_path, monkeypatch)
env["LOOPX_TURN_ENVELOPE"] = "1" if enabled else "0"
original_run = subprocess.run
previews = []

def legacy_preview(argv, **kwargs):
if argv[0] != "loopx" or "--dry-run" not in argv:
return original_run(argv, **kwargs)
previews.append(argv)
return subprocess.CompletedProcess(argv, 0, json.dumps({
"ok": True, "schema_version": "loopx_turn_envelope_v0",
}), "")

monkeypatch.setattr(worker.subprocess, "run", legacy_preview)
receipt = run_once(env)
assert receipt["ok"], receipt
assert ("turn_envelope" in receipt) is enabled
assert ("turn_envelope_source" in receipt) is enabled
assert ("quota_input_contract" in receipt) is enabled
assert len(previews) == int(enabled)
if enabled:
assert receipt["turn_envelope"] is True
assert receipt["turn_envelope_source"] == "explicit"
assert receipt["quota_input_contract"] == {
"expected_schema_version": "loopx_turn_envelope_v0",
"observed_schema_version": "loopx_turn_envelope_v0",
"eligible": True,
}
assert "should-run" in previews[0] and "--turn-envelope" in previews[0]
assert "--turn-instance-id" not in previews[0] and "--decision-output-root" not in previews[0]


@pytest.mark.parametrize("observed", [
Expand Down
Loading
Loading