Skip to content

fix(server): interrupted compaction no longer freezes the prune boundary - #38818

Open
navidnadali wants to merge 2 commits into
pingdotgg:mainfrom
navidnadali:up/compaction-mark-at-pass-start
Open

navidnadali wants to merge 2 commits into
pingdotgg:mainfrom
navidnadali:up/compaction-mark-at-pass-start

Conversation

@navidnadali

Copy link
Copy Markdown
Contributor

Note

Written by Claude Opus 5.5 on behalf of Navid

Problem

ProjectionMaintenance.pruneEventStore prunes events through the mark the previous pass left once that mark is a day
old, then records a new mark (the head at pass start). It recorded the new mark only after the deletes finished. The
pass runs every 6 hours at 10% duty, so on a large event log a due pass can outlast the time between restarts (an
update, a quit, a crash). A pass cut short never writes its mark: the old mark stays due, every later pass re-runs the
same boundary, and the boundary never moves past it. Before its first delete each of those passes also reads the
reference range above the boundary, which grows every day the boundary is stuck, so they reach fewer deletes until they
reach none and the log keeps growing.

We found it on a large install where event-log-pruned-through stayed at the same sequence for days while compaction
kept running.

Change

Write the next mark as a due pass starts: after the boundary is published, before any delete. The mark says every
event at or below the head was committed by then, which already holds at pass start, so the retained day is unchanged
and a pass that completes leaves the same rows and metadata as before. A pass cut short now still moves the boundary a
day later. What it did not reach is deleted by that next due pass, which prunes through the newer mark.

Scope

A small fix of an obvious ordering bug (one write moved before the deletes), so no issue. It does not make a pass
resume where it stopped; each pass still scans from the start of the log.

Verification

  • vp test run src/orchestration-v2/ProjectionMaintenance.test.ts (from apps/server): the existing "prunes events
    the projections cover once the retained day has passed" case now also cuts a pass. A day after its prune, a due pass
    is interrupted at its first betweenPages, and a day later a full pass must publish the boundary at the head the cut
    pass started with. On main it fails by assertion (expected 11 to equal 12: the boundary stays at the previous
    mark); with this change 3/3 pass.

  • Runtime transcript: a script on an in-memory database (the real migrations and pruneEventStore), 20,000 events on
    day 0 and 5,000 more a day, four passes a day 6 hours apart, each on a fresh service and interrupted once it has read
    12,000 rows (as if a restart arrived). Counts at the end of each day:

    day boundary, main boundary, this PR rows left, main rows left, this PR
    1 20,000 20,000 18,280 18,280
    2 20,000 25,000 17,063 16,829
    3 20,000 30,000 22,063 15,636
    4 20,000 35,000 27,063 14,691
    8 20,000 55,000 47,063 13,196

    On main, from day 3 the reference range above the frozen boundary (15,000 rows and growing) uses up a pass's whole
    budget, so no pass deletes anything again. With this change the range stays at one day (5,000) and each due pass keeps
    deleting.

  • vp run check:changed passes; from apps/server, vp test run src/orchestration-v2 (with the provider replay
    fixtures) 3,134/3,134 and vp test run integration (all 26 *.integration.test.ts files) 387/387.

Not checked: a real large database across real restarts.

Made with Claude Opus 5.5 via Claude Code. Reviewed by GPT-6.1 Sol via Codex.

🤖 Generated with Claude Code

Event-log compaction prunes through the mark the previous pass left once that mark is a day old,
but it wrote the next mark only after the deletes finished. A due pass cut short by a restart left
the old mark in place, so every later pass re-scanned the same range and the boundary never moved.

The next mark is now written as a due pass starts, before any delete. Its claim, that every event
at or below the head is committed, already holds then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XS 0-9 changed lines (additions + deletions). labels Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 754e947 and 802fc43.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

When pruning is due, the pass now records the current event-store head before pruning. A test covers an interrupted pass and verifies that a later pass advances the pruning boundary to the recorded head.

Changes: Prune mark handling

Layer / File(s) Summary
Record the head before pruning
apps/server/src/orchestration-v2/ProjectionMaintenance.ts, apps/server/src/orchestration-v2/ProjectionMaintenance.test.ts
The pruning pass writes the current head to EVENT_LOG_PRUNE_MARK before pruning. The test interrupts a due pass and checks that a later pass advances the boundary to the recorded head.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Possibly related PRs

Suggested reviewers: juliusmarminge, t3dotgg, macroscopeapp

Merge Risk: Minimal · previous review 754e9

This change stops an interrupted pruning pass from leaving the old mark due indefinitely, so later passes advance instead of rescanning the same range. No actionable merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: an interrupted compaction pass no longer leaves the prune boundary frozen. This matches the objectives and the summary, which move the next mark write to the start of …
Description check Passed The description follows the template sections: Problem, Change, Scope, and Verification. The Problem section explains the stuck prune boundary and its effect on the event log. The Change section expla…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: an interrupted compaction pass no longer leaves the prune boundary frozen. This matches the objectives and the summary, which move the next mark write to the start of a due pass. The title uses the conventional commit format from the template.

Full details: Description check

Explanation

The description follows the template sections: Problem, Change, Scope, and Verification. The Problem section explains the stuck prune boundary and its effect on the event log. The Change section explains the reordered mark write and why it keeps the retained day unchanged. The Scope section gives a reason for having no linked issue. The Verification section gives focused test results, a runtime transcript, and states what was not checked (a real large database across real restarts). The template's conventional-commit title guidance is also satisfied by the title. The description is complete.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between cbd8267 and 754e947.

Files selected for processing: 2

  • apps/server/src/orchestration-v2/ProjectionMaintenance.test.ts
  • apps/server/src/orchestration-v2/ProjectionMaintenance.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@juliusmarminge juliusmarminge added the takeover:in-progress A takeover lead is working on this PR label Nov 19, 2026
…-pass-start

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS 0-9 changed lines (additions + deletions). takeover:in-progress A takeover lead is working on this PR vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants