Skip to content

fix(server): preserve data across rollback and update recovery - #162

Merged
ScriptedAlchemy merged 2 commits into
mainfrom
fix/deep-recovery-audit-2026-11-18
Nov 19, 2026
Merged

ScriptedAlchemy merged 2 commits into
mainfrom
fix/deep-recovery-audit-2026-11-18

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

The deeper audit found three recovery paths that could damage data outside the intended operation:

  • Rollback of a managed leftover worktree removed its preserved folder as soon as Git recreation succeeded. A later provider, checkpoint, or cancellation failure then lost the original uncommitted files.
  • Checkpoint capture and stale-checkpoint deletion followed an existing symbolic checkpoint ref. If it pointed at a branch, these operations could move or delete that branch instead of changing only the checkpoint ref. T3 normally creates direct refs; this defect requires an existing symbolic ref.
  • Offline service-update backup appended SQLite sidecar suffixes to the configured symlink path instead of the resolved database path. After an unclean stop, committed rows in the real database's WAL could be omitted from backup and lost during failed-trial restoration.

The preserved worktree now lives through rollback completion. Cleanup distinguishes a durable rollback commit from a cancelled caller, restoring the original folder on earlier failure and retaining the committed workspace after a later cancellation. Checkpoint mutations use --no-deref. Offline backup enumeration, free-space accounting, and clone probing use the resolved database path consistently with online backup and restore.

The worktree defect was introduced in #124. The symbolic-ref and offline-WAL defects predate that merge and were discovered while tracing its affected recovery paths.

Verification

Each production defect has a failing regression before its fix. The tests use owned temporary repositories, files, real SQLite, and an owned child process for an unclean database stop.

  • The worktree cases cover provider failure, checkpoint failure, cancellation before commit, successful cleanup, and cancellation after the real event and projection commit. The last case verifies that the filesystem still matches the committed projection and that no preserved aside remains.
  • Both symbolic-ref cases fail on the original code: capture moves the referent branch and deletion removes it. The fixed cases preserve the branch.
  • The WAL fixture committed 2,131 rows before its unclean stop; the old backup/restore recovered only 1,486. The corrected backup retains every committed row.
  • Final owning suites: 18 rollback, 114 Git-driver, and 47 launcher tests. Another 94 checkpoint-safety, pinned-runtime, and desktop-update boundary cases passed.
  • Independent review covered the original fixes and the durable-commit cleanup hardening.

On the final tree, vp run check:changed 68193a2d1c69d3e237bae0f279228135af1afebe --concurrency 1 passed the native build, formatting, lint, both Knip scans, generated GraphQL checks, and 181 tests across the three owning files plus Agent Context Budget. The cached typecheck launcher is blocked by seccomp_unotify in this environment, so that gate invocation exits nonzero. All three configured effect-tsgo tsrs --noEmit package checks passed directly for server, evals, and scripts; the server check passed alone after an attempt during concurrent validation was killed.

GitHub Fork CI, including final Fork Check, Agent Context Budget, and every other applicable check must pass on the exact published head before merge.

Native Windows/macOS service managers and live providers were not exercised; their relevant process/filesystem interfaces use the existing fixtures. No CI or budget gate was weakened.

Additional recovery audit

Two more failure paths are fixed in this update:

  • Later checkpoint refs were retired before the rollback event transaction committed. A SQLite failure left ready checkpoint projections pointing at deleted Git refs. Ref retirement now follows the durable event commit under the existing thread lock; a cleanup failure remains best effort. The real Git/SQLite regression verifies the later saved work stays recoverable on transaction failure, normal retry retires it, and an externally locked ref does not turn a committed rollback into failure.
  • An incomplete update backup directory was treated as a usable backup even if its required main database file was missing. Restore could then delete the live database through the optional-sidecar cleanup path. Backup reuse and restore now reject that layout before starting a trial or touching the live database, preserving pending recovery state and reporting the backup path. Both pending-trial and restore-marker replay cases use real temporary SQLite data. This checks the required file's presence; it does not claim to reconstruct missing backups or detect every kind of corruption.

Both defects were reproduced on unchanged production source before their fixes. They predate the earlier changes in this PR.

Validation of the updated tree

The updated tree integrates reviewed main 09efb428b74ef6a1d22be25bdba9c62f8a180669. The required changed-file gate was run. Native build, formatting, lint, Knip file scan and GraphQL generation checks passed. Initial export scan/test/compiler processes were killed or returned without diagnostics under resource pressure; their isolated reruns passed: 185 tests across four files, the full configured Knip export scan, and the full server compiler. The configured evals and scripts compilers also passed. A strict changed-file compiler run found and led to correction of a duplicated Effect-provide fixture warning; the final strict check passed.

The cached typecheck launcher still cannot use seccomp_unotify here. All three affected packages were checked directly with the configured effect-tsgo tsrs --noEmit; no check settings or CI gates were weakened. GitHub must pass all applicable checks on the new exact head before merge.

The launcher regression constructs the filesystem layout left by an incomplete backup operation; it does not claim an actual launcher crash during directory removal. Native service managers and live providers remain outside local validation.

Implemented and verified with GPT-6 Astra Pro / ChatGPT.

@changeset-bot

changeset-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 48ba512

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Integrate reviewed main and preserve Git recovery refs until the event transaction commits. Refuse incomplete database backups before trial startup or restoration can mutate the live database.
@ScriptedAlchemy
ScriptedAlchemy merged commit 8e81612 into main Nov 19, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant