Skip to content

fork CI: FORK_CHANGES.md checks move to the workflows job so fork-only PRs skip app tests (#259) - #389

Merged
scuttletoby merged 1 commit into
mainfrom
fork/fork-changes-workflows-job
Nov 17, 2026
Merged

scuttletoby merged 1 commit into
mainfrom
fork/fork-changes-workflows-job

Conversation

@scuttletoby

Copy link
Copy Markdown
Owner

What and why

The fork-only skip (F2) rarely fired. Each fork feature PR changes its line in FORK_CHANGES.md, and that file was not fork-owned because an app test (packages/shared/src/forkDocs.test.ts, in the app Test job) read it. So every app job ran.

This PR:

  • Moves the FORK_CHANGES.md checks to .github/scripts/fork-changes-list.test.cjs (with fork-changes-list.cjs). The checks are: every fork setting is named, and no setting is named that does not exist. They run in Fork tests' workflows job, which runs on every pull request, and they read packages/contracts/src/forkDefaults.ts as text. The test is listed in .github/fork-ci-tests.txt. The parse fails closed when the table is missing or empty, when a scope: has no key: it can read, or when a scope is unknown.
  • Removes the same two checks from forkDocs.test.ts. That file still checks docs/fork/README.md, which is not fork-owned.
  • Marks FORK_CHANGES.md fork-owned in .github/fork-owned-paths.txt. .github/fork-fleet.json stays excluded.
  • Changes fork-owned-reads.test.py. It used to drop every .md name, so it could never have seen the vitest reader. It now counts a read of a .md file that the list owns by its own line (FORK_CHANGES.md). Runbook pointers matched by a wider line (.github/fork-*.md) still do not count. A new planted case covers this. The vitest reader is gone, so the real scan is clean.
  • Updates fork-owned-paths.test.py (FORK_CHANGES.md is now expected to be fork-owned). Updates the two docs that named the old test (FORK_CHANGES.md, docs/fork/updates.md).

No workflow edit needed

The workflows job's sparse checkout leaves out FORK_CHANGES.md and forkDefaults.ts. In a sparse checkout (core.sparseCheckout=true), and only there, the test reads a missing file from the checked-out commit with git show HEAD:<file>. actions/checkout's sparse fetch is partial (blob:none), so git fetches that blob on demand. A file missing from a full checkout, or from the commit, fails the test.

Optional, not required: to skip the on-demand fetch, add these two lines to the workflows job's sparse-checkout: list in .github/workflows/fork-tests.yml. This agent's token cannot push workflow files.

            /FORK_CHANGES.md
            /packages/contracts/src/forkDefaults.ts

Proof

Same failures as the old test. All runs locally with node --test .github/scripts/fork-changes-list.test.cjs:

State Result
Current files 6 pass. The text parse reads the same 22 settings as the table (22 rows).
A setting dropped from FORK_CHANGES.md (server.agentSlots un-backticked) red: "names every fork setting", actual ['server.agentSlots']
An unknown setting added (server.notASetting) red: "names no setting that does not exist", actual ['server.notASetting']
Empty FORK_CHANGES.md red: every setting missing
FORK_CHANGES.md deleted both checks red ("missing from the checkout")
A new row in forkDefaults.ts only (server.brandNew) red: "names every fork setting"

Each guard was mutation-checked red. The checks: always fall back to the commit, never fall back, no scope/key count guard, no unknown-scope guard, no empty-table guard, table end ignored.

fork-owned-reads.test.py:

  • Red with FORK_CHANGES.md fork-owned and the vitest reader still there: reads: [('packages/shared/src/forkDocs.test.ts', 'FORK_CHANGES.md')].
  • Green after the reader moved.
  • The planted read goes red without the new .md rule.

fork-owned-paths.test.py: red ({"FORK_CHANGES.md": true}) before its expectation was updated, then green.

F2 decision. The list's own judge(), main vs this branch:

Changed paths main this PR
#238's paths not fork-only (fork-fleet.json, FORK_CHANGES.md) not fork-only (fork-fleet.json only, correct)
fork-kit/x/y.py + FORK_CHANGES.md not fork-only fork-only
FORK_CHANGES.md only not fork-only fork-only
forkDefaults.ts only not fork-only not fork-only

vp run check:changed passes: format, lint, knip, typecheck, forkDocs.test.ts (24 pass) and the new node test.

State question

  • A PR that edits only FORK_CHANGES.md: fork-only, so the app jobs are skipped. The workflows job still runs fork-changes-list.test.cjs, so a missing or unknown setting is red there.
  • A PR that edits forkDefaults.ts and not FORK_CHANGES.md: forkDefaults.ts is not fork-owned, so every job runs. The workflows job (Workflow lint and upstream drift, step "Fork script tests") catches the missing line, as the forkDefaults.ts-only row above shows. That job runs on every same-repo pull request whatever F2 decides.
  • An empty FORK_CHANGES.md: it names nothing, so "names every fork setting" fails with the full list. An empty or unreadable forkDefaults.ts table throws ("no row read"). It does not pass on empty state.
  • Runs again or stops half-way: the test only reads files. The sparse fallback reads the checked-out commit, the same content the job tests, so a re-run gives the same answer. A failed on-demand fetch is red, never a pass.

Runner-minutes saved

In #238's run (44033525268), the jobs a fork-only PR skips (Build and Typecheck, Test 1-2, Test Web 1-3, Test Server 1-8, Rust, Release Smoke) took 2406 s together. A typical fork feature PR that changes only fork-owned paths plus its FORK_CHANGES.md line now saves about 40 runner-minutes. Wall-clock time drops by about 7 minutes: Test 1 alone was 403 s, and that is the issue's figure. The new test adds well under a second to the workflows job.

General vs project

  • General: a "skip unrelated CI" switch is only as good as the files it can own. A check that reads a shared file belongs in a job that always runs, so the file can be owned. A reads scanner that ignores a file type (.md) needs an exception for owned files of that type, or it can never see their readers.
  • Project: the fork's .github/fork-owned-paths.txt, the workflows job's sparse checkout, FORK_CHANGES.md and forkDefaults.ts.

Fixes #259

🤖 Generated with Claude Code

…y PRs skip app tests (#259)

The FORK_CHANGES.md checks (every fork setting named, no unknown setting) move from the
vitest file packages/shared/src/forkDocs.test.ts to .github/scripts/fork-changes-list.test.cjs,
which reads forkDefaults.ts as text and runs in Fork tests' `workflows` job on every pull
request. FORK_CHANGES.md is now fork-owned, so a fork feature pull request that only adds its
line there skips the app jobs. fork-owned-reads.test.py now counts a read of a `.md` file the
list owns by its own line, so an app test reading FORK_CHANGES.md again is caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: skip-safety verdict: SAFE (head 2cca645)
read at: 2cca645

Read-only review: skip safety and correctness. The head did not move while I reviewed.

Skip safety: no PR can skip app tests it needs

  • The list comes from the merge commit, and a change to it never counts as fork-only. fork-owned-paths.cjs reads the list from the checked-out merge commit, so the PR's own copy. But .github/fork-owned-paths.txt and the script are on the NEVER list. A PR that widens the list to skip its own tests therefore runs every job. I checked this with the real list: FORK_CHANGES.md + .github/fork-owned-paths.txt gives forkOnly=false.
  • Path edges, checked with judge() on the real list. Only FORK_CHANGES.md on its own is fork-only. Each of these runs every job: FORK_CHANGES.md + forkDefaults.ts, + forkDocs.test.ts or + README.md. So do fork_changes.md, Fork_Changes.md, ./FORK_CHANGES.md, FORK_CHANGES.md/x, docs/FORK_CHANGES.md and FORK_CHANGES.mdx. The line compiles to the exact regex ^FORK_CHANGES\.md$.
  • Renames show both paths. The diff uses --no-renames, so a rename into or out of FORK_CHANGES.md lists both names. The other name is not owned, so every job runs.
  • A symlink at FORK_CHANGES.md is fork-only but goes red. In the sparse job, git show HEAD: returns the link target as text, which names no setting, so "names every fork setting" fails. In a full checkout, the file read follows the link and checks the target.
  • Nothing in the app jobs still reads it. git grep finds FORK_CHANGES.md outside the docs only in .github/scripts and in the root README.md (a link). The lint job (vp check and the reads scan) has no fork_only gate, so the format check and the scan still run on fork-only PRs.
  • The workflows job runs on every same-repo PR, whatever F2 decides, and fork-ci-result needs it.

The moved checks still fail closed

  • The on-demand fetch works in CI. This PR's run 44177849530, job "Workflow lint and upstream drift": checkout uses --filter=blob:none --depth=2 and core.sparseCheckout true. fork-changes-list.test.cjs ran there and passed 6 of 6. Its first read took 617 ms, which fits an on-demand blob fetch.
  • A failed fetch is red. I built a local blob:none sparse clone that leaves out both files, then removed its remote. readRepoFile throws Command failed: git show HEAD:FORK_CHANGES.md. node --test showed both checks ✖, "skipped 0", exit 1.
  • Missing or empty state is red, and no case skips. An empty FORK_CHANGES.md fails with every setting missing. A deleted one fails both checks in a full checkout and in a sparse one. A deleted or empty table throws.
  • State question:
    • A PR opened before this merges is judged on its merge commit with this list and test. A stale forkDefaults.ts change is not fork-only.
    • A stale branch that brings back the old vitest reader is red in lint. Test: I restored main's forkDocs.test.ts. fork-owned-reads.test.py reported [('packages/shared/src/forkDocs.test.ts', 'FORK_CHANGES.md')], exit 1.
    • A re-run reads the same commit and gives the same answer.

Findings

minor: .github/scripts/fork-changes-list.cjs:30-33. The text parse silently drops rows that do not contain scope:. The count guard only counts the literal scope:, so a row with no such text is skipped without a throw. I tested each shape, appended after one normal row:

Shape Result
...EXTRA_ROWS, (spread) missed, no error
row("server", "x") (helper call) missed, no error
{ scope, key: "x" } (shorthand) missed, no error
{ "scope": "server", "key": "x" } missed, no error
{ scope : "server", ... } missed, no error
a column-0 ]; inside a row cuts the table short

The always-on formatter would rewrite the quoted and spaced forms. It would leave the spread, the helper call and the shorthand alone. The old vitest check read the runtime array, so it saw them. In those shapes, a setting missing from FORK_CHANGES.md now passes.

These shapes throw, which is correct: key before scope, as const, a template-literal key, a comment between scope and key, and multi-line values (these parse). A commented-out row counts as pinned, which errs red.

Today's table is uniform (22 scope:, 22 read), so nothing is missed now. Possible fix: also count the table's row-level entries, for example the lines matching /^ \S/m that are not { or },, and throw on any other entry. A spread or call row would then throw, the way a row in the wrong order already does.

nit: .github/scripts/fork-owned-reads.test.py:62. The "*" not in l filter does nothing. name in OWNED_DOCS compares exact strings, and a scanned name never contains *. It is harmless. The comment could say that exact-name equality is what keeps the .github/fork-* runbook pointers out.

nit (not this PR): two result: checks in fork-owned-paths.test.py need jq on the host. They fail locally where jq is missing, and are fine in CI.

Check 3: the new .md rule cannot hide other reads. The change only adds or name in OWNED_DOCS to the include condition. That makes the scan see more names, never fewer. The real scan is clean (8000 files scanned).

Tests run (as the workflow runs them)

  • node --test .github/scripts/fork-changes-list.test.cjs: 6 pass.
  • python3 -I -B .github/scripts/fork-owned-reads.test.py: all passed.
  • python3 -B .github/scripts/fork-owned-paths.test.py: 48 ok. The 2 failures are the jq-only result: checks above; every list: and script: check passed.
  • I did not run forkDocs.test.ts locally: it needs the workspace install. CI Test 1 passed on this head.

Mutation results

I broke one line at a time; each run went red (exit 1), and I restored each file afterwards.

# Mutation Red test
M1 readRepoFile always falls back to the commit "readRepoFile reads the working tree, and the commit only for…"
M2 no rows.length !== scopes guard "pinnedIds fails closed…"
M3 no unknown-scope guard "pinnedIds fails closed…"
M4 compare never reports missing "namedIds … compare finds missing…"
M5 table end ignored (slice to end of file) "reads each row's scope and key, and nothing outside the table" and "fails closed"
M6 reads scan drops every .md again (the new rule removed) "a planted read is found…"
M7 FORK_CHANGES.md removed from .github/fork-owned-paths.txt fork-owned-paths.test.py "list: the fork's own folders and scripts are fork-owned…"
M8 real data: every `server.agentSlots` un-backticked in FORK_CHANGES.md "names every fork setting", actual ['server.agentSlots']
M9 stale branch: main's forkDocs.test.ts restored fork-owned-reads.test.py lists the read

@scuttletoby
scuttletoby marked this pull request as ready for review November 17, 2026 21:32
@scuttletoby
scuttletoby merged commit f59d8ad into main Nov 17, 2026
35 checks passed
scuttletoby added a commit that referenced this pull request Nov 18, 2026
…y PRs skip app tests (#259) (#389)

The FORK_CHANGES.md checks (every fork setting named, no unknown setting) move from the
vitest file packages/shared/src/forkDocs.test.ts to .github/scripts/fork-changes-list.test.cjs,
which reads forkDefaults.ts as text and runs in Fork tests' `workflows` job on every pull
request. FORK_CHANGES.md is now fork-owned, so a fork feature pull request that only adds its
line there skips the app jobs. fork-owned-reads.test.py now counts a read of a `.md` file the
list owns by its own line, so an app test reading FORK_CHANGES.md again is caught.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
scuttletoby added a commit that referenced this pull request Nov 18, 2026
…y PRs skip app tests (#259) (#389)

The FORK_CHANGES.md checks (every fork setting named, no unknown setting) move from the
vitest file packages/shared/src/forkDocs.test.ts to .github/scripts/fork-changes-list.test.cjs,
which reads forkDefaults.ts as text and runs in Fork tests' `workflows` job on every pull
request. FORK_CHANGES.md is now fork-owned, so a fork feature pull request that only adds its
line there skips the app jobs. fork-owned-reads.test.py now counts a read of a `.md` file the
list owns by its own line, so an app test reading FORK_CHANGES.md again is caught.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fork-only skip (F2) rarely fires: FORK_CHANGES.md is read by an app test

1 participant