You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(server): interrupted compaction no longer freezes the prune boundary - #38818
ProjectionMaintenance.pruneEventStore prunes events through the mark the previous pass left once that mark is a day
old, then records a new mark (the head at pass start). It recorded the new mark only after the deletes finished. The
pass runs every 6 hours at 10% duty, so on a large event log a due pass can outlast the time between restarts (an
update, a quit, a crash). A pass cut short never writes its mark: the old mark stays due, every later pass re-runs the
same boundary, and the boundary never moves past it. Before its first delete each of those passes also reads the
reference range above the boundary, which grows every day the boundary is stuck, so they reach fewer deletes until they
reach none and the log keeps growing.
We found it on a large install where event-log-pruned-through stayed at the same sequence for days while compaction
kept running.
Change
Write the next mark as a due pass starts: after the boundary is published, before any delete. The mark says every
event at or below the head was committed by then, which already holds at pass start, so the retained day is unchanged
and a pass that completes leaves the same rows and metadata as before. A pass cut short now still moves the boundary a
day later. What it did not reach is deleted by that next due pass, which prunes through the newer mark.
Scope
A small fix of an obvious ordering bug (one write moved before the deletes), so no issue. It does not make a pass
resume where it stopped; each pass still scans from the start of the log.
Verification
vp test run src/orchestration-v2/ProjectionMaintenance.test.ts (from apps/server): the existing "prunes events
the projections cover once the retained day has passed" case now also cuts a pass. A day after its prune, a due pass
is interrupted at its first betweenPages, and a day later a full pass must publish the boundary at the head the cut
pass started with. On main it fails by assertion (expected 11 to equal 12: the boundary stays at the previous
mark); with this change 3/3 pass.
Runtime transcript: a script on an in-memory database (the real migrations and pruneEventStore), 20,000 events on
day 0 and 5,000 more a day, four passes a day 6 hours apart, each on a fresh service and interrupted once it has read
12,000 rows (as if a restart arrived). Counts at the end of each day:
day
boundary, main
boundary, this PR
rows left, main
rows left, this PR
1
20,000
20,000
18,280
18,280
2
20,000
25,000
17,063
16,829
3
20,000
30,000
22,063
15,636
4
20,000
35,000
27,063
14,691
8
20,000
55,000
47,063
13,196
On main, from day 3 the reference range above the frozen boundary (15,000 rows and growing) uses up a pass's whole
budget, so no pass deletes anything again. With this change the range stays at one day (5,000) and each due pass keeps
deleting.
vp run check:changed passes; from apps/server, vp test run src/orchestration-v2 (with the provider replay
fixtures) 3,134/3,134 and vp test run integration (all 26 *.integration.test.ts files) 387/387.
Not checked: a real large database across real restarts.
Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.
Event-log compaction prunes through the mark the previous pass left once that mark is a day old,
but it wrote the next mark only after the deletes finished. A due pass cut short by a restart left
the old mark in place, so every later pass re-scanned the same range and the boundary never moved.
The next mark is now written as a due pass starts, before any delete. Its claim, that every event
at or below the head is committed, already holds then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing files that changed from the base of the PR and between 754e947 and 802fc43.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
No actionable comments were generated in the recent review.
Walkthrough
When pruning is due, the pass now records the current event-store head before pruning. A test covers an interrupted pass and verifies that a later pass advances the pruning boundary to the recorded head.
Changes: Prune mark handling
Layer / File(s)
Summary
Record the head before pruning apps/server/src/orchestration-v2/ProjectionMaintenance.ts, apps/server/src/orchestration-v2/ProjectionMaintenance.test.ts
The pruning pass writes the current head to EVENT_LOG_PRUNE_MARK before pruning. The test interrupts a due pass and checks that a later pass advances the boundary to the recorded head.
This change stops an interrupted pruning pass from leaving the old mark due indefinitely, so later passes advance instead of rescanning the same range. No actionable merge-blocking risk was found.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: an interrupted compaction pass no longer leaves the prune boundary frozen. This matches the objectives and the summary, which move the next mark write to the start of …
Description check
The description follows the template sections: Problem, Change, Scope, and Verification. The Problem section explains the stuck prune boundary and its effect on the event log. The Change section expla…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: an interrupted compaction pass no longer leaves the prune boundary frozen. This matches the objectives and the summary, which move the next mark write to the start of a due pass. The title uses the conventional commit format from the template.
Full details: Description check
Explanation
The description follows the template sections: Problem, Change, Scope, and Verification. The Problem section explains the stuck prune boundary and its effect on the event log. The Change section explains the reordered mark write and why it keeps the retained day unchanged. The Scope section gives a reason for having no linked issue. The Verification section gives focused test results, a runtime transcript, and states what was not checked (a real large database across real restarts). The template's conventional-commit title guidance is also satisfied by the title. The description is complete.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.
…-pass-start
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
size:XS0-9 changed lines (additions + deletions).takeover:in-progressA takeover lead is working on this PRvouch:trustedAuthor is trusted by repo permissions or the VOUCHED list.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Written by Claude Opus 5.5 on behalf of Navid
Problem
ProjectionMaintenance.pruneEventStoreprunes events through the mark the previous pass left once that mark is a dayold, then records a new mark (the head at pass start). It recorded the new mark only after the deletes finished. The
pass runs every 6 hours at 10% duty, so on a large event log a due pass can outlast the time between restarts (an
update, a quit, a crash). A pass cut short never writes its mark: the old mark stays due, every later pass re-runs the
same boundary, and the boundary never moves past it. Before its first delete each of those passes also reads the
reference range above the boundary, which grows every day the boundary is stuck, so they reach fewer deletes until they
reach none and the log keeps growing.
We found it on a large install where
event-log-pruned-throughstayed at the same sequence for days while compactionkept running.
Change
Write the next mark as a due pass starts: after the boundary is published, before any delete. The mark says every
event at or below the head was committed by then, which already holds at pass start, so the retained day is unchanged
and a pass that completes leaves the same rows and metadata as before. A pass cut short now still moves the boundary a
day later. What it did not reach is deleted by that next due pass, which prunes through the newer mark.
Scope
A small fix of an obvious ordering bug (one write moved before the deletes), so no issue. It does not make a pass
resume where it stopped; each pass still scans from the start of the log.
Verification
vp test run src/orchestration-v2/ProjectionMaintenance.test.ts(fromapps/server): the existing "prunes eventsthe projections cover once the retained day has passed" case now also cuts a pass. A day after its prune, a due pass
is interrupted at its first
betweenPages, and a day later a full pass must publish the boundary at the head the cutpass started with. On
mainit fails by assertion (expected 11 to equal 12: the boundary stays at the previousmark); with this change 3/3 pass.
Runtime transcript: a script on an in-memory database (the real migrations and
pruneEventStore), 20,000 events onday 0 and 5,000 more a day, four passes a day 6 hours apart, each on a fresh service and interrupted once it has read
12,000 rows (as if a restart arrived). Counts at the end of each day:
mainmainOn
main, from day 3 the reference range above the frozen boundary (15,000 rows and growing) uses up a pass's wholebudget, so no pass deletes anything again. With this change the range stays at one day (5,000) and each due pass keeps
deleting.
vp run check:changedpasses; fromapps/server,vp test run src/orchestration-v2(with the provider replayfixtures) 3,134/3,134 and
vp test run integration(all 26*.integration.test.tsfiles) 387/387.Not checked: a real large database across real restarts.
Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.
🤖 Generated with Claude Code