Skip to content

fork-follow: fetch the rebase's files first, name every stop (#475) - #496

Merged
scuttletoby merged 3 commits into
mainfrom
wff/fork-follow-partial-clone-rebase
Nov 18, 2026
Merged

scuttletoby merged 3 commits into
mainfrom
wff/fork-follow-partial-clone-rebase

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Refs #475

Root cause

The Plan job checks out with --filter=blob:none (a partial clone), so during the rebase git fetches each missing file from origin the first time it reads it. The runners moved to ubuntu-26.04 with Git 2.56.1. While applying pick 28/232, Git 2.56.1 asked origin, in one batch, for the blob that this pick had itself just merged (apps/server/src/cloud/selfUpdate.ts). No server has that object, so origin answered not our ref and the pick died with fatal: could not fetch … from promisor remote. That is a git error, not a conflict: there is no REBASE_HEAD and no unmerged file. The old report took the patch name only from REBASE_HEAD (|| echo unknown) and never quoted git's output, so it said "stopped at: unknown" with no files and no reason. Git 2.43 does not ask for the merged blob, so the same rebase passes locally.

I reproduced this with Git 2.56.1 (alpine:edge) on a git init + fetch --filter=blob:none checkout, as in the workflow, and read the packet trace. With this branch's script on the same setup, the plan follows on its own: action=rebase, the old main → 4877, the same tree as a full-clone rebase, and 1110 blobs fetched in one batch before the rebase.

Changes (.github/scripts/fork-follow.sh)

  • Prefetch before the rebase. In a partial clone (the promisor is extensions.partialclone, or the remote marked promisor = true), the script now fetches in one batch every blob the rebase reads that is not already here: both sides of each patch, the series' own objects, and the old base's side of every file upstream changed. The rebase then never needs a lazy fetch. Only ids origin can have are asked for (no gitlinks, no merge results). If that fetch fails, the plan stops before rebasing and quotes git.
  • Every stop is named. The patch comes from REBASE_HEAD, else from the last pick in rebase-merge/done, else the report says git did not begin the rebase. The report says whether it was a conflict (and lists the files) or a git error, and always quotes git's own output: the last 20 lines of the last rebase command, without progress lines. "stopped at: unknown" with no reason can no longer be produced.

Tests (.github/scripts/fork-follow.test.sh)

The 4 new checks fail on the old script (run with FORK_FOLLOW_DIR set to main's scripts) and pass now. Totals: 163 ok here; on the old script, 159 ok and 4 FAIL.

  • A partial clone whose origin refuses any fetch while a rebase runs (an upload-pack wrapper) follows with no refused fetch (Fork cannot follow upstream #475).
  • A pick stopped by a git error (an untracked file in the way) names the patch, says "not a conflict" and quotes git.
  • A rebase git refuses to begin says so and quotes git.
  • A conflict report quotes git's CONFLICT ( lines.
  • One existing assertion was narrowed to the list line (^- .github/workflows/ci.yml), because git's quoted output now legitimately names that file.

fork-contracts.test.py and fork-temp-files.test.sh also pass.

Fix round (after the trust and state lenses)

Rebased onto main (after #497). Each item has a check that fails on the previous head (db683d82); with FORK_FOLLOW_DIR set to that head's scripts, 11 checks fail (10 new, plus the narrowed listed-file assertion, whose list is now indented).

  1. Quoting. Git's output, the conflicting-file list, the prefetch log and the other quoted logs are now indented code blocks (a blank line, then four spaces on every line), so no line in them can end the block. Upstream's fork-named files, the series' kept listed files, the outside-startable workflows and a workflow name with a newline are quoted the same way. Every @ in a report gets a zero-width space after it, so no name mentions anyone. Checks: the trust lens's repro (a file name with a newline and a fence line) and a conflicting file named @octo.txt.
  2. Done-list fallback. A partial clone where a later pick dies on a refused fetch (prefetch skipped by a test git) is named from rebase-merge/done. This check passes on the old head too, since the fallback was already there; with picked="" it is the one check that fails.
  3. Prefetch refused. Origin refuses every fetch: the report says how many files could not be fetched, quotes git, says nothing was rebased, and the head is unchanged with no rebase left.
  4. Die before the first pick. With a rebase dir but no done list, the report says git stopped while it checked out the target, before it applied any patch.
  5. Promisor booleans. git config --type=bool, so a remote marked yes, on or 1 is prefetched for (one check each).
  6. Abort output. A failed git rebase --abort is quoted in the report and says the checkout is still rebasing.
  7. Ended by --continue/--skip. When that step fails and git leaves no rebase state, the report names the patch whose listed files were deleted and the command that failed, not "git did not begin the rebase".

Found while writing check 4: the prefetch asked only for the base side of upstream's changes, on the assumption that the upstream fetch sends every target blob. A server with reachability bitmaps (GitHub) does not send a blob that the base's history already has, for example when upstream reverts a file to an earlier content. The checkout of the target then fetched it while the rebase ran. The prefetch now asks for both sides; whatever is missing after the upstream fetch is in the base's history, which origin has. Check: a revert with a bitmapped upstream follows with no fetch during the rebase (it fails on the old head).

Not done: each prefetch source is still not pinned on its own (nit 3).

Totals: fork-follow.test.sh 174 ok (was 163). fork-temp-files.test.sh and fork-contracts.test.py pass.

General vs project

General: prefetching a partial clone's rebase inputs and naming any git stop with git's own words are generic git-follow mechanics. Nothing in them depends on this repository. Project: none. The script still reads its repo, upstream and branch from the workflow's existing env, as before.

🤖 Generated with Claude Code

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: trust verdict: SAFE (head db683d8)

Read-only review (trust/safety lens). The change cannot make fork-follow release something wrong: rebase_ok is still set only by a rebase/continue/skip that git reports as finished, every other end goes to fail before any output line is written, and a failed prefetch stops the plan before the rebase. Dropped-patch detection is unchanged. Two minor findings and three nits below; none blocks the merge.

Checked

  • Tests: 163 ok on this head. With FORK_FOLLOW_DIR set to main's scripts: 159 ok and exactly the 4 new checks FAIL. The claim holds.
  • Mutations on this head (each one run against the full suite): with the prefetch_rebase_blobs call removed, 1 FAIL (the partial-clone check); with git_said emptied, 3 FAIL. Removing any single prefetch source (rev-list, log --raw or diff-tree) leaves 0 FAIL (nit 3). With the done-list fallback emptied (picked=""), 0 FAIL (minor 2).
  • Prefetch runs no upstream-controlled code. fetch with no refspec destination and --no-write-fetch-head writes no ref and runs no hook, and .gitattributes cannot define a filter or merge driver without local config. The fetch goes to the promisor remote (origin, the fork's public URL, no token). FORK_PUSH_TOKEN is not in the plan step's environment (the check step keeps it in GIT_CONFIG_* env for its own process only), so no token can reach prefetch.log or the rebase output.
  • Stop naming on a real die: on git 2.43 I made a partial clone with origin gone, had pick 1 conflict and resolved it, then let pick 3 die with fatal: could not fetch … from promisor remote. Git deletes REBASE_HEAD at each pick, so no earlier patch is named by mistake, and the last line of rebase-merge/done is the pick that died. The logic is right; it just has no test (minor 2).

Minor 1: upstream file names can break out of the "What git said" fence (markdown and live @mention in the issue) (.github/scripts/fork-follow.sh:466-469, used at :513)
ort prints paths raw in its CONFLICT lines. The git diff --name-only conflict list is C-quoted, but git's own output is not. An upstream path that contains a newline and a ``` line closes the fence, and the text after it renders as markdown in the "Fork cannot follow upstream" issue comment, including an @mention that notifies. Before this PR, git's rebase output never reached the report, so this exposure is new.
Repro: copy fork-follow.test.sh, change line 609 to `git mv .github/workflows/ci.yml "$(printf ".github/workflows/x\n\140\140\140\n**@octocat injected**\n\140\140\140\ny.yml")"` and stop after the `plan` on line 612. The resulting `report.md`:

What git said:
```
CONFLICT (rename/delete): .github/workflows/ci.yml renamed to .github/workflows/x
```
**@octocat injected**
```
y.yml in HEAD, but deleted in 2a9ed81 (fork: delete ci.yml).

Impact is low: upstream already ships code to every install, the fork's release is unaffected, and the plan still fails closed. Fix: indent each quoted line by four spaces instead of fencing it (sed 's/^/ /'; an injected line is indented too, so it cannot leave the block), or fence with one more backtick than the longest backtick run in the text. Add the repro above as a check. The same applies to the prefetch fail at :459-460, but prefetch.log holds only fetch errors with object ids.

Minor 2: the #475 path itself (REBASE_HEAD absent while rebasing) is untested (.github/scripts/fork-follow.sh:499-502)
The new git-error test (an untracked late.txt) goes through REBASE_HEAD: git writes REBASE_HEAD for that error (checked on 2.43, where REBASE_HEAD named "fork: add late"). So picked="" passes all 163 checks. The prefetch-failure report (:459-460) is also unpinned. One way to test both: in partial_fork, add a variant whose upload-pack wrapper refuses every fetch. That covers the prefetch fail. For the done fallback, a fetch that dies mid-rebase is needed, for example the existing while-rebasing wrapper plus a test-only way to skip the prefetch.

Nits
3. Each prefetch source on its own is unpinned (see the mutations above). The partial-clone test adds only fork-new files, so the base side of a patch that edits an existing upstream file, and blobs that upstream renames read, are not exercised. This is fail-closed: a missed blob now ends as a named git-error stop.
4. stopped at: its start: git did not begin the rebase is chosen whenever no rebase dir is left. If a --continue/--skip in the listed-file loop fails and git also removes its state (steps > 0), the text is wrong. I did not reproduce this; it is only a wording risk. Suggest a different text when steps > 0.
5. Pre-existing, not from this PR: the - path conflict list sits outside any fence, so a plain path with @name in it is a live mention. The same indent fix would cover it.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state verdict: SAFE (head db683d8)

read at: db683d8 (the head did not move during the review)

Lens: state. This was a read-only review: nothing was pushed, and no branch, tag, release or workflow was touched. My clone and containers are deleted. I ran everything on Git 2.43.0 (the host) and Git 2.56.1 (alpine:edge in docker).

In short: the prefetch is complete for the real #475 case. On Git 2.56.1, with origin refusing any fetch while the rebase runs, the PR's script follows with no fetch reaching origin. The old script stops with "stopped at: unknown". Every exit path of the rebase step that I could reach names a patch or the start, and quotes git. I found no blocker and no major. There are two minors, two nits and two infos.

Evidence

# Run Result
1 fork-follow.test.sh on Git 2.43.0 and on Git 2.56.1 163/163 on both
2 Real #475 case, Git 2.56.1: blob:none checkout of old main b86c4a13 (on 4864), plan → 4877, GitHub as origin, PR script action=rebase, candidate tree d80e41a3 (the same tree as the hand candidate in #475), 1110 blobs prefetched, plan 69 s (the old script took 164 s)
3 The same, old script (from b86c4a13) passed in my run: action=rebase, tree d80e41a3, after 209 lazy fetches mid-rebase. The original stop did not happen on its own here, so it is not deterministic in this setup.
4 Real case with a full local copy of the fork as origin, whose upload-pack refuses while rebase-merge exists (the PR's own test, at full size). Upstream had moved to 4890 by then. PR script: action=rebase, 0 refused. Old script: refused once, report "stopped at: unknown … (none listed; see the run log)", which is the #475 report
5 Toy with content-merging picks, partial clone, origin refusing while rebasing (2.43 and 2.56.1) no prefetch: dies "could not fetch … from promisor remote"; prefetch: no fetch, correct result

State answers

  • Run again after a stop: the plan writes nothing outside $logs, aborts the rebase and pushes nothing. In CI each run starts from a fresh checkout, so a re-run is the same plan. A run killed during the prefetch leaves at most a temporary pack in a runner that is thrown away.
  • Old/empty state: when the upstream tag has already been followed, the plan exits before the prefetch (the target_tag == base_tag block), so the common run costs nothing. An empty series still prefetches the base side of upstream's changes, but those blobs are already in HEAD's checked-out tree, comm drops them, and nothing is fetched. A full clone has no promisor, so the step returns at once. In a sparse checkout, the third list (the base side of every upstream change) is what keeps the prefetch complete.
  • Completeness: in a non-sparse checkout every blob in the base tree is present: it is either in HEAD's checked-out tree or on the old side of a series patch. The target side comes from the plain upstream fetch. Rename detection on either side reads only P-side blobs (base or series) and ours-side blobs (present after the checkout of the target, or merged). The PR excludes gitlinks. No -X or autostash is used. I found no object the rebase reads that the lists miss, and runs 2 and 4 confirm this on the real series.
  • Exit paths (tested on both versions):
    • A resolved listed-file conflict, then a git error on a later pick: --continue clears the old REBASE_HEAD. The failing pick either writes its own (an untracked file in the way) or leaves none, and then done names it (lazy-fetch die). A stale patch name is not possible.
    • Git refuses to begin: the report says "its start" (the PR's test).
    • Schema/seam check after the rebase: unchanged, and named by fail.
    • Git's words: progress lines end in \r, so git_said keeps the fatal: line that follows Rebasing (n/m) on 2.56.1 too.

Findings

  • minor: fork-follow.sh:500-502. If git dies while it checks out the target, before the first pick (a lazy-fetch die or a kill during checkout onto), rebase-merge exists with no done file. The report then says "stopped at: a patch git did not name", although no patch was applied. Repro, both versions: a partial clone where the series edits a.txt and origin refuses during the rebase, with no prefetch (so this is not reachable through the prefetch any more, only through another die or a kill). After git rebase exits with 128, .git/rebase-merge holds git-rebase-todo onto orig-head … and no done. Suggested fix: when rebasing and done is missing or empty, say "its start: git stopped while checking out $target_tag, before the first patch".
  • minor (test gap): the prefetch refusal (fork-follow.sh:459-461, "Could not fetch the N files … Nothing was rebased") has no test. Suggested test: partial_fork, then make origin refuse every fetch after the checkout. Expect the report to say "Could not fetch the" and "Nothing was rebased", no rebase-merge directory, and the head unchanged.
  • nit: fork-follow.sh:444 recognizes only promisor = true. A remote marked 1, yes or on (all valid git booleans) skips the prefetch silently, which brings back the old lazy behaviour. git config --type=bool --get-regexp … would normalize the value.
  • nit: fork-follow.sh:506 now discards the abort's output. A failed abort has no effect in CI, but a local run would not see why the tree is left mid-rebase. Appending the output to $last_log would keep it.
  • info: on Git 2.56.1, in the real case with the PR script, the rebase still starts 5 git fetch children (shown by trace2: fetch → remote-https → rev-list --exclude-promisor-objects). None reaches origin: there is no packet traffic after the prefetch, and with a refusing file:// origin there were 0 refusals. Git asks for objects that are already local and finds them there before it talks to origin. This fits the PR's explanation (2.56 asks for blobs it has just written). It also means the PR's small test (toy series) does not exercise this 2.56 path. The evidence that the fix holds there is runs 2 and 4, not the test.
  • info: because run 3 passed, the CI stop needs something my container did not have (most likely a batch that mixes a missing blob with a just-written one). The prefetch removes that precondition: after it, no blob the rebase reads is missing.

scuttletoby and others added 3 commits November 18, 2026 16:30
The Plan job checks out with --filter=blob:none, so the rebase fetched each
missing file from origin on first read. Git 2.56 (ubuntu-26.04 runners) asked,
while applying a pick, for the blob that pick had just merged; origin has no
such object ("not our ref"), so the pick died with a git error. There was no
REBASE_HEAD and no unmerged file, so the report said "stopped at: unknown"
with no reason.

- Before the rebase, fetch in one batch every blob it reads that is missing
  here (each patch's sides, the series' own, the old base's side of upstream's
  changes). The promisor is extensions.partialclone or the remote marked
  promisor = true. The rebase then never fetches.
- A stop names the patch from REBASE_HEAD, else the last pick in the done
  list, else says git did not begin; it says whether it was a conflict or a
  git error and quotes git's own output (last 20 lines, no progress lines).

Tests (red on the old script, green now): a partial clone whose origin refuses
fetches while a rebase runs follows; a pick stopped by a git error names the
patch and quotes git; a rebase git refuses to begin says so; a conflict report
quotes git.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check that the partial clone lacks a blob piped rev-list into grep -q,
which fork-ci-tests.test.cjs refuses (SIGPIPE under pipefail). grep -c reads
everything.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…op (#475)

- Git's words, the conflict list and upstream file names go into the
  report as indented code blocks that no line can end, and every @ in a
  report gets a zero-width space, so no name mentions anyone.
- The prefetch also asks for the target side of upstream's changes: a
  server with reachability bitmaps does not send a blob the base's
  history already has (an upstream revert), and the checkout read it.
- A promisor remote marked yes/on/1 is found (git config --type=bool).
- A stop before the first pick, and a --continue/--skip that fails and
  ends the rebase, are named as such; a failed rebase --abort is quoted.
- Tests: fence in a file name, @ file, failed abort, continue ending the
  rebase, refused prefetch, mid-pick die (done list), die at checkout,
  revert prefetch, promisor spellings.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby force-pushed the wff/fork-follow-partial-clone-rebase branch from db683d8 to 089e2d2 Compare November 18, 2026 16:43
@scuttletoby
scuttletoby merged commit 508ef56 into main Nov 18, 2026
29 checks passed
scuttletoby added a commit that referenced this pull request Nov 18, 2026
…496)

* fork-follow: fetch the rebase's files first, name every stop (#475)

The Plan job checks out with --filter=blob:none, so the rebase fetched each
missing file from origin on first read. Git 2.56 (ubuntu-26.04 runners) asked,
while applying a pick, for the blob that pick had just merged; origin has no
such object ("not our ref"), so the pick died with a git error. There was no
REBASE_HEAD and no unmerged file, so the report said "stopped at: unknown"
with no reason.

- Before the rebase, fetch in one batch every blob it reads that is missing
  here (each patch's sides, the series' own, the old base's side of upstream's
  changes). The promisor is extensions.partialclone or the remote marked
  promisor = true. The rebase then never fetches.
- A stop names the patch from REBASE_HEAD, else the last pick in the done
  list, else says git did not begin; it says whether it was a conflict or a
  git error and quotes git's own output (last 20 lines, no progress lines).

Tests (red on the old script, green now): a partial clone whose origin refuses
fetches while a rebase runs follows; a pick stopped by a git error names the
patch and quotes git; a rebase git refuses to begin says so; a conflict report
quotes git.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fork-follow test: read the whole rev-list, not grep -q (lint)

The check that the partial clone lacks a blob piped rev-list into grep -q,
which fork-ci-tests.test.cjs refuses (SIGPIPE under pipefail). grep -c reads
everything.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fork-follow: quote git safely, prefetch reverted blobs, name every stop (#475)

- Git's words, the conflict list and upstream file names go into the
  report as indented code blocks that no line can end, and every @ in a
  report gets a zero-width space, so no name mentions anyone.
- The prefetch also asks for the target side of upstream's changes: a
  server with reachability bitmaps does not send a blob the base's
  history already has (an upstream revert), and the checkout read it.
- A promisor remote marked yes/on/1 is found (git config --type=bool).
- A stop before the first pick, and a --continue/--skip that fails and
  ends the rebase, are named as such; a failed rebase --abort is quoted.
- Tests: fence in a file name, @ file, failed abort, continue ending the
  rebase, refused prefetch, mid-pick die (done list), die at checkout,
  revert prefetch, promisor spellings.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: t <t@t>
scuttletoby added a commit that referenced this pull request Nov 18, 2026
…496)

* fork-follow: fetch the rebase's files first, name every stop (#475)

The Plan job checks out with --filter=blob:none, so the rebase fetched each
missing file from origin on first read. Git 2.56 (ubuntu-26.04 runners) asked,
while applying a pick, for the blob that pick had just merged; origin has no
such object ("not our ref"), so the pick died with a git error. There was no
REBASE_HEAD and no unmerged file, so the report said "stopped at: unknown"
with no reason.

- Before the rebase, fetch in one batch every blob it reads that is missing
  here (each patch's sides, the series' own, the old base's side of upstream's
  changes). The promisor is extensions.partialclone or the remote marked
  promisor = true. The rebase then never fetches.
- A stop names the patch from REBASE_HEAD, else the last pick in the done
  list, else says git did not begin; it says whether it was a conflict or a
  git error and quotes git's own output (last 20 lines, no progress lines).

Tests (red on the old script, green now): a partial clone whose origin refuses
fetches while a rebase runs follows; a pick stopped by a git error names the
patch and quotes git; a rebase git refuses to begin says so; a conflict report
quotes git.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fork-follow test: read the whole rev-list, not grep -q (lint)

The check that the partial clone lacks a blob piped rev-list into grep -q,
which fork-ci-tests.test.cjs refuses (SIGPIPE under pipefail). grep -c reads
everything.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fork-follow: quote git safely, prefetch reverted blobs, name every stop (#475)

- Git's words, the conflict list and upstream file names go into the
  report as indented code blocks that no line can end, and every @ in a
  report gets a zero-width space, so no name mentions anyone.
- The prefetch also asks for the target side of upstream's changes: a
  server with reachability bitmaps does not send a blob the base's
  history already has (an upstream revert), and the checkout read it.
- A promisor remote marked yes/on/1 is found (git config --type=bool).
- A stop before the first pick, and a --continue/--skip that fails and
  ends the rebase, are named as such; a failed rebase --abort is quoted.
- Tests: fence in a file name, @ file, failed abort, continue ending the
  rebase, refused prefetch, mid-pick die (done list), die at checkout,
  revert prefetch, promisor spellings.

Refs #475

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: t <t@t>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant