You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(server): Codex runs sharing a session no longer fall behind together - #39136
Codex runs every thread of a provider instance on one provider session, so every live Codex run receives every thread's
events. A run dropped another thread's event only when it processed it (routeProviderEvent), after consumeProviderEvents had handled it: another thread's snapshots were queued as jobs in the shared commit, and another
thread's usage report waited for those commits. With a root thread and a few delegated children running, each run
waited for every thread's commits, so on a busy writer they all fell behind together and the lag kept growing. This is
still the case after #38544. Details in #39135.
Change
RunExecutionService now routes the subscription's events as they arrive, on its own copy of the routing state, and
drops before the commit queue the ones routing rejects without changing that state. routeProviderEvent is a pure
state machine and only it sets the run's routing state, so the run would reject exactly those events when processing
them: it still accepts the same events in the same order. Before the root turn ends, a rejected event writes nothing and
its stop check returns false; past the root turn every event still reaches the run, so a run waiting on background work
rechecks as before.
vp test run src/orchestration-v2/RunExecutionService.test.ts -t "keeps every run on a shared provider session"
(from apps/server): nine runs through the real RunExecutionService on one shared ProviderCommitBatch; the writer
is a FIFO one-permit lock that ten other writers keep busy (~1 s per commit); every 10 s each thread reports usage,
writes two turn items and reports usage again. It requires every run to be at most 2 rounds behind after 5 minutes. On main it fails by assertion (6 rounds behind); with this change it passes.
Runtime transcript: the same model with 1-12 runs for 10 minutes, on main and on this branch (rounds behind; commits
and queued snapshot jobs after 10 minutes):
runs
rounds behind at 1 / 5 / 10 min, main
this PR
commits / snapshot jobs in 10 min, main
this PR
1
0 / 0 / 0
0 / 0 / 0
59 / 118
59 / 118
5
0 / 0 / 0
0 / 0 / 0
472 / 2,950
118 / 590
9
0-1 / 4-5 / 9-10
0 / 0 / 0
590 / 8,022
118 / 1,062
12
1-2 / 8-9 / 18
0 / 0 / 0
596 / 11,830
118 / 1,416
On main the snapshot jobs grow with the square of the runs and every run waits for one commit per thread each
round; with this change a run queues only its own two snapshots per round. The output is identical on Linux x86_64
(Intel Xeon, Debian 13, Node 26.13.1) and macOS (Apple M5 Max). To reproduce from a checkout of this PR with
dependencies installed: bash reproduce.sh <checkout> from https://gist.github.com/navidnadali/6571419c289985057c67f09a8adc9f91 (it appends the model to RunExecutionService.test.ts, runs it with main's and this PR's RunExecutionService.ts, prints the table and
restores both files).
vp run check:changed passes; vp test run src/orchestration-v2 (with the provider replay fixtures) 3,152/3,152 (124 files); vp test run integration from apps/server (all 26 *.integration.test.ts files) 394/394.
Not checked: a live Codex session; the model stands in for the writer load.
Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.
Codex runs every thread of an instance on one provider session, so each
run receives every thread's events. A run dropped another thread's events
only when it processed them: their snapshots were queued as commit jobs
and their usage reports waited for those commits, so with several runs
live each one waited for every thread's commits and all fell behind.
A run now routes its events as they arrive and drops, before the commit
queue, the ones it would reject without a routing change. Past the root
turn every event still reaches the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing files that changed from the base of the PR and between ef47de4 and a00b167.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
No actionable comments were generated in the recent review.
Walkthrough
The provider-event stream now applies routing before commit queuing and filters certain events before the root turn ends. A shared-session test exercises event ingestion while competing writers hold a semaphore.
Changes: Provider event routing
Layer / File(s)
Summary
Route events before commit queuing apps/server/src/orchestration-v2/RunExecutionService.ts
Before the root turn ends, the stream filters events that are neither accepted nor routing-state updates. After it ends, the stream passes all events downstream.
Test shared-session ingestion under contention apps/server/src/orchestration-v2/RunExecutionService.test.ts
The background scenario can use a supplied shared commit coordinator. A new test runs nine consumers with ten competing writers and asserts that maximum ingestion lag stays within two rounds after five simulated minutes.
This change stops runs that share a provider session from queuing work for events they will reject, so they no longer fall behind together under writer load. The supplied evidence shows no concrete regression, and the new test covers the contention scenario. Merge risk is low.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: Codex runs that share a provider session no longer fall behind together. It uses a conventional commit prefix and is specific. The changeset routes provider events ahe…
Description check
The description follows the template. It has Problem, Change, Scope, and Verification sections. The Problem and Change sections explain the lag cause and the routing fix. Scope links #39135. Verificat…
Linked Issues check
Directly linked issue #39135 asks that runs sharing a provider session drop other threads' events without waiting on the shared writer. The PR summary says routedAhead in RunExecutionService.ts ap…
Out of Scope Changes check
The changes touch two files: RunExecutionService.ts (about +11 lines, the pre-queue routing step) and RunExecutionService.test.ts (about +95/-2, the shared-coordinator test support and the new reg…
Full details: Title check
Explanation
The title names the main change: Codex runs that share a provider session no longer fall behind together. It uses a conventional commit prefix and is specific. The changeset routes provider events ahead of the commit queue and drops rejected events, which addresses the lag described in the title.
Full details: Description check
Explanation
The description follows the template. It has Problem, Change, Scope, and Verification sections. The Problem and Change sections explain the lag cause and the routing fix. Scope links #39135. Verification gives focused test commands, a benchmark table, and the full suite results, and it states what was not checked, a live Codex session. The author also notes the model used, as the template asks. The Scope section links the issue but does not explain a reason for a bug fix that lacks an issue, which is not needed because an issue exists. The description is thorough and complete.
Full details: Linked Issues check
Explanation
Directly linked issue #39135 asks that runs sharing a provider session drop other threads' events without waiting on the shared writer. The PR summary says routedAhead in RunExecutionService.ts applies routeProviderEvent before the commit queue and drops events that routing rejects without changing routing state. Before the root turn ends, rejected events write nothing. After the root turn ends, all events still pass through, so background-work checks continue. The reviewed head confirms that routeProviderEvent is a pure function over ProviderEventRoutingState that returns [boolean, state], which supports the claim that the pre-queue check matches the in-run check. The test keeps every run on a shared provider session in RunExecutionService.test.ts models nine runs on one shared ProviderCommitBatch with a busy writer and asserts at most two rounds of lag after five minutes. The PR reports that this test fails on main (6 rounds behind) and passes with the change. The issue's Codex-only framing is broader in the code: the fix sits in the generic routing path, so it covers any provider with supportsMultipleProviderThreadsPerSession. No acceptance criteria beyond the reported behavior are specified. Limits: the diff hunks for RunExecutionService.ts could not be read (the diff tool reported missing blobs, and the full-file read was truncated), so the routedAhead logic rests on the PR summary and the test description. Test results are author-reported and were not re-run.
Full details: Out of Scope Changes check
Explanation
The changes touch two files: RunExecutionService.ts (about +11 lines, the pre-queue routing step) and RunExecutionService.test.ts (about +95/-2, the shared-coordinator test support and the new regression test). Both serve the objective in #39135: the source change implements the routing, and the test checks the lag behavior. The test support adds Effect Context and Semaphore imports, which the new test needs. No unrelated files or API declarations change (the summary reports no changes to exported declarations). The PR description's verification runs (check:changed, the orchestration-v2 and integration suites) are validation of this change, not separate scope. Limit: the exact RunExecutionService.ts hunks could not be read, so this verdict relies on the change summary.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.
The automated pipeline stopped on this pull request and will not merge it:
the capture step could not show the problem on the base commit, so there is nothing to prove fixed.
Read the scenario guidance, which excludes timing and provider-specific performance proof. Started the head app and completed a real Codex turn. Requested eight concurrent native children from one parent. Only three launched. Configured provider launch arguments with agents.max_threads=12 and repeated; Codex again reported three successful launches and five errors: collab spawn failed: agent thread limit reached. Uploaded the scripted diagnostic. Stopped the server and removed the worktree and scratch files. The base app was not run.
We ran this shape live with Codex: twelve T3 threads on one Codex provider instance (so one shared provider
session), all started at once, each asked for ten rounds of "stream a 100-line numbered list, then run a
checkpoint command" (gpt-6-luna, effort low, a fresh database per run, one server at a time; macOS arm64,
Node 26.13.1, codex-cli 0.169.0). Codex's own item/completed frames from the provider logs were compared with read-only database samples every 250 ms.
When the last of those lists finished in Codex: lines Codex had finished / lines stored
1,000 / 1,000
1,000 / 1,000
Rounds behind (Codex's completed checkpoints minus stored), every 15 s
0
0
Shared-commit jobs over the turns (per stored list line)
41,228 (34.4)
5,936 (4.2)
Commit batch wait p90 / max
15 / 63 ms
4 / 38 ms
So this live run does not reproduce the lag on main. Every run did queue every thread's snapshots (about
eight times this PR's commit jobs for the same output), but with no other writer a commit waited milliseconds,
so no thread fell behind. The lag needs commits that wait about a second, as in the issue's model, and a fresh
dev database does not get there. The model also ended most turns after the first list, so the threads
streamed together for about 15 s rather than minutes. Driver, analysis and steps: https://gist.github.com/navidnadali/18d79e2ae998c645a43f1904c7d3243d
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
size:S10-29 changed lines (additions + deletions).takeover:blockedTakeover blocked; reason in the PR bodyvouch:trustedAuthor is trusted by repo permissions or the VOUCHED list.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Written by Claude Opus 5.5 on behalf of Navid
Problem
Codex runs every thread of a provider instance on one provider session, so every live Codex run receives every thread's
events. A run dropped another thread's event only when it processed it (
routeProviderEvent), afterconsumeProviderEventshad handled it: another thread's snapshots were queued as jobs in the shared commit, and anotherthread's usage report waited for those commits. With a root thread and a few delegated children running, each run
waited for every thread's commits, so on a busy writer they all fell behind together and the lag kept growing. This is
still the case after #38544. Details in #39135.
Change
RunExecutionServicenow routes the subscription's events as they arrive, on its own copy of the routing state, anddrops before the commit queue the ones routing rejects without changing that state.
routeProviderEventis a purestate machine and only it sets the run's routing state, so the run would reject exactly those events when processing
them: it still accepts the same events in the same order. Before the root turn ends, a rejected event writes nothing and
its stop check returns false; past the root turn every event still reaches the run, so a run waiting on background work
rechecks as before.
Scope
Fixes #39135.
Verification
vp test run src/orchestration-v2/RunExecutionService.test.ts -t "keeps every run on a shared provider session"(from
apps/server): nine runs through the realRunExecutionServiceon one sharedProviderCommitBatch; the writeris a FIFO one-permit lock that ten other writers keep busy (~1 s per commit); every 10 s each thread reports usage,
writes two turn items and reports usage again. It requires every run to be at most 2 rounds behind after 5 minutes. On
mainit fails by assertion (6 rounds behind); with this change it passes.Runtime transcript: the same model with 1-12 runs for 10 minutes, on
mainand on this branch (rounds behind; commitsand queued snapshot jobs after 10 minutes):
mainmainOn
mainthe snapshot jobs grow with the square of the runs and every run waits for one commit per thread eachround; with this change a run queues only its own two snapshots per round. The output is identical on Linux x86_64
(Intel Xeon, Debian 13, Node 26.13.1) and macOS (Apple M5 Max). To reproduce from a checkout of this PR with
dependencies installed:
bash reproduce.sh <checkout>from https://gist.github.com/navidnadali/6571419c289985057c67f09a8adc9f91 (it appends the model toRunExecutionService.test.ts, runs it withmain's and this PR'sRunExecutionService.ts, prints the table andrestores both files).
vp run check:changedpasses;vp test run src/orchestration-v2(with the provider replay fixtures) 3,152/3,152 (124 files);vp test run integrationfromapps/server(all 26*.integration.test.tsfiles) 394/394.Not checked: a live Codex session; the model stands in for the writer load.
Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.
🤖 Generated with Claude Code