Repository navigation
3724 lines (3514 loc) · 206 KB
/
Copy pathci.yml
File metadata and controls
3724 lines (3514 loc) · 206 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
name: CI
on:
push:
branches:
- main
# ⭐ NO `branches:` filter here, deliberately (#16482). A `pull_request:`
# trigger filtered to `branches: [main]` does not run at all when the PR's
# base is a feature branch — and a workflow that does not run publishes NO
# check run, which is an ABSENCE, not a skip. Branch protection reads that as
# permanently pending, so every required context this file publishes reports
# nothing on such a PR. Measured on the #14478 stack: six card PRs showed
# 8-11 light checks and ZERO of the six required contexts; the first real
# signal arrived at the trunk PR, after every card was written, reviewed and
# merged. Correctness rested on local runs quoted into PR bodies — a
# discipline, not a gate.
#
# Runner cost of removing the filter, measured 2026-09-08 over the 30 days
# 2026-08-09 → 2026-09-08 (4000 PRs read, `sort=created&direction=desc` paged
# until the window closed): 6 of 4000 PRs had a base other than `main`
# (0.15%), all six the #14478/#15679 stack. At the window's measured 7008
# pull_request runs per workflow, 0.15% is ~10.5 extra runs per workflow per
# 30 days ≈ 2.5/week, or ~57 job-runs/week across both files at 23 job-runs
# per PR event. Even the pessimistic bound — one uncancelled run per one of
# those PRs' 115 commits — is 1.6% of current pull_request spend. Small, so
# the trigger is widened unconditionally rather than gated on a label or on
# "the base is an open PR's head" (#16482 records the arithmetic).
#
# ⛔ Do NOT mirror this widening onto the `push:` trigger above. `push:` is
# per-branch, not per-PR: unfiltering it would run this workflow on every
# push to every agent branch in the repo, which is a different and far larger
# population than the 6 PRs measured here.
pull_request:
# Merge queue: the queue builds each PR as speculatively merged onto the
# current main and only lands it if this workflow is green on that result —
# the race-free version of the "pull main and re-verify before merging"
# discipline (AGENTS.md multi-agent §7/§10). Every workflow that produces a
# branch-protection-required check MUST carry this trigger, or queue builds
# wait forever on a check that never reports.
merge_group:
# ── THE HOURLY FULL RUN (#16467) ─────────────────────────────────────────
# `push` above now computes the Test Core package set with `--affected`
# against `github.event.before`, so a merge no longer re-tests the whole
# workspace it just tested in the queue. Something still has to run the FULL
# battery on `main`, on a clock, or "main is green" stops being a statement
# about the workspace and becomes a statement about the last diff.
#
# This is that run. Minute 0 rather than an offset minute ON PURPOSE, and
# against the sibling convention in test-nightly-tiers.yml: the reading this
# feeds — `scripts/ci/select-shard-timings-run.mjs` — wants one complete run
# per hour far more than it wants a short queue wait, and a run that starts
# a few minutes late is still the hour's run. What it must NOT do is collide
# with the next hour's, which the concurrency group below decides.
#
# ⛔ This is a NEW TRIGGER KEY, not a widening of `push:`. The ⛔ above still
# binds: `push:` stays filtered to `main`.
schedule:
- cron: '0 * * * *'
# The same full battery, on demand: an operator who has just landed a fix for
# a red hourly run should not have to wait up to an hour to see it answered.
# Treated as `full` by the selection script for the same reason `schedule` is.
workflow_dispatch:
# Superseded runs on the same PR waste runners and delay feedback; cancel them.
# A push to `main` is the opposite case, and gets a group of its OWN (#25373).
#
# ⭐ `github.event.after` IS LOAD-BEARING IN THIS KEY. A push run is the only
# run that tests ITS OWN merge: Test Core computes `--affected` against
# `github.event.before` (see `on.schedule` above), so a push run that does not
# finish leaves that merge's diff with no Test Core verdict on `main` until the
# next hourly run. Main merges land every ~9–16 min and a push run takes ~27,
# so under the old shared key `ci-CI-push-refs/heads/main` the next merge
# cancelled it: 31 of 40 push runs on 2026-10-23 01:03Z–11:03Z, most of them
# inside "Run this shard's tests". (The Turbo seed is NOT what was lost: the
# "Save Turbo cache (main only)" steps are `if: always()` and run on a
# cancelled job too.) `cancel-in-progress: false` alone does NOT fix it:
# GitHub keeps ONE pending run per group and cancels it when a third arrives;
# replayed on the same 40 arrivals, 18 still die. So the push segment is the
# pushed commit. `github.event.after` is set on a push and absent on
# `schedule`, `workflow_dispatch` and `merge_group`, which still fall through
# to `github.ref`; on `pull_request` the PR number wins before it is reached.
# Two push runs never share a group, so every one runs to completion. The seed
# key carries `github.sha`, so two overlapping main runs never write the same
# key.
#
# ⭐ `github.event_name` IS LOAD-BEARING TOO (#16467). A `schedule` and a
# `workflow_dispatch` run on `main` both fall through to `refs/heads/main`;
# without the event name they share one group and an on-demand run cancels the
# hourly one (and, under any key that drops `github.event.after`, the next
# merge cancels both, which is how 36 of 60 hourly-era push runs died).
#
# ⛔ Do not "simplify" either segment away. `scripts/ci/scheduled-full-run.mjs
# --check-concurrency` evaluates this expression against push, schedule,
# dispatch, PR and queue contexts on two different commits. It reds when two
# pushes share a group, when a push shares one with a schedule or dispatch run,
# or when two schedule, dispatch or same-PR runs stop sharing one (a run-unique
# key such as `github.sha` or `github.run_id` would switch cancellation off for
# those events too).
#
# `cancel-in-progress` stays `true`: with a per-commit push group it never
# fires on a push, and for everything else it is the policy. Two hourly runs
# overlapping means one of them is over an hour old, and an hour-old
# measurement of `main` is strictly the worse of the two. Test Core's wall clock
# is well inside the cadence (~39 shard-minutes across the shards), so this is
# the exceptional path, not the normal one.
concurrency:
group: ci-${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.event.after || github.ref }}
cancel-in-progress: true
# ── THE TWO SPEEDS (#22998) ──────────────────────────────────────────────────
# A `pull_request` or `merge_group` run carries the FAST SET: `Build Core`,
# `Test Core` (affected-only) and `Build Docs`. `main`'s hourly `schedule` run,
# its `push` runs and an on-demand `workflow_dispatch` carry the FULL BATTERY:
# the fast set plus the three jobs below that spell
#
# github.event_name != 'pull_request' && github.event_name != 'merge_group'
#
# in their `if:` — `temporal-conformance`, `dogfood` (three shards) and
# `dogfood-verify`. Nothing is deleted: the jobs moved off the
# per-PR path. A red in one of them is a main-red for the owning lane, and the
# publish lane reads the full battery before its approval (release.yml's
# `full-battery` job, `scripts/ci/scheduled-full-run.mjs --release-gate`).
#
# Spelled as a DENY-list of the two per-PR events on purpose — the filter
# contract's half 1 below, applied to events: a trigger added to `on:` later
# RUNS these jobs, and the failure direction of a wrong guess is a slower PR,
# never a full battery that silently stopped running. `dogfood-gate` keeps
# `if: always()` (check:shard-attestation pins it): with both legs skipped it
# counts zero expected attestations and concludes success, exactly as on a
# filter-skipped diff.
jobs:
filter:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
# On the `schedule` / `workflow_dispatch` full runs (#16467) everything
# counts as changed: dorny/paths-filter is skipped there (no diff to
# resolve), a skipped step's output is the empty string (falsy), and
# `|| 'true'` supplies the value without touching PR/push behavior.
#
# On merge_group the step RUNS (the diff is the queue base_sha against
# the group head, see the step below), but only the three ui outputs
# read it. `docs`, `core` and `crosspkg` stay 'true' there through the
# `github.event_name != 'merge_group' &&` guard: the queue build is the
# last validation before main, and those jobs run on every entry, as
# they did before the step ran on this event. On any other event the
# guard is true and the expression reads exactly as `<output> ||
# 'true'` did.
docs: ${{ github.event_name != 'merge_group' && steps.changes.outputs.docs || 'true' }}
core: ${{ github.event_name != 'merge_group' && steps.changes.outputs.core || 'true' }}
# Gates the `test` job ONLY, ORed with `core` — see the `crosspkg`
# filter below and the `test` job's `if:` (#9829, #10015).
crosspkg: ${{ github.event_name != 'merge_group' && steps.changes.outputs.crosspkg || 'true' }}
# The objectui tree under `ui/` (objectstack#25677). Three outputs, read
# together by the four `* UI` jobs at the bottom of this file:
# ui — a path under ui/, or a root file the ui jobs build
# from (the lockfile, the workspace, turbo.json, ci.yml);
# ui_consumers — one of the seven packages objectui installs from this
# tree changed, so the Console is compiled (Type Check
# UI) and built (Build & E2E UI) against the new shape in
# the PR that moves it;
# ui_tree — `ui/package.json` exists in this checkout.
# On merge_group `ui` and `ui_consumers` come from the queue diff
# (objectstack#25867): an entry whose merged diff against
# `merge_group.base_sha` touches no ui path and none of the seven
# packages skips the four ui jobs, exactly as the same diff does on its
# PR head; an entry that touches one runs them. If the step fails (the
# base cannot be fetched, the diff errors), it is `continue-on-error`
# there, its outputs are empty and `|| 'true'` runs every ui job.
#
# `ui_tree` is what keeps the ui jobs SKIPPED, not failed, on a tree that
# does not carry ui/ yet: `ui` and `ui_consumers` fall back to 'true' on
# schedule / dispatch (half 1 below), and on push to main a spec change
# sets `ui_consumers`, so neither path output alone can tell a job there
# is nothing to build. Its step runs on every event, so the `|| 'true'`
# fallback is reached only when this job died before it — and then THE
# FILTER CONTRACT's "run everything" is the intended reading.
ui: ${{ steps.changes.outputs.ui || 'true' }}
ui_consumers: ${{ steps.changes.outputs.ui_consumers || 'true' }}
ui_tree: ${{ steps.ui-tree.outputs.present || 'true' }}
# ── THE FILTER CONTRACT, both halves (#4928) ──────────────────────────
#
# Half 1 is the `|| 'true'` above: when in doubt, RUN EVERYTHING. It
# covers a missing output VALUE.
#
# Half 2 is every downstream `if:` in this file, and it used to be
# missing. GitHub wraps an `if:` that names no status function in an
# IMPLICIT success(), so the original `if: needs.filter.outputs.core ==
# 'true'` had TWO independent paths to "skip" when this job DIED
# (checkout flake, a dorny/paths-filter fault, the 10-minute timeout):
# the implicit success() is false, and the output is no longer a
# trustworthy 'true'. Both roads led to skipping test / dogfood /
# build-core — and skipped counts as SUCCESS in branch protection, so a
# single flake here produced a fully green, zero-test-run, mergeable PR
# with no red signal anywhere.
#
# So every downstream job now spells the contract as "skip only when the
# filter EXPLICITLY said false":
#
# if: ${{ !cancelled() && needs.filter.outputs.<name> != 'false' }}
#
# `!cancelled()` displaces the implicit success() (a genuine run-level
# cancellation still skips — #3668's lifecycle reasoning); `!= 'false'`
# makes the empty string mean "run", exactly like `|| 'true'` does. This
# is deliberately robust to BOTH readings of what a failed job's outputs
# are — null/empty or the `|| 'true'` fallback — because `'' != 'false'`
# and `'true' != 'false'` are both true. The old form was broken under
# both, because the implicit success() dominates either way.
#
# Third occurrence of this GitHub semantic in one audit; the other two
# were release.yml's publish-integrity guard and its docker job (#4900).
steps:
- name: Checkout repository
uses: actions/checkout@v7
# No `if:` on purpose: every event needs this answer, and a skipped step
# would hand `ui_tree` the `|| 'true'` fallback on a tree without ui/.
- name: Detect the imported ui/ tree
id: ui-tree
run: |
if [ -f ui/package.json ]; then
echo 'present=true' >> "$GITHUB_OUTPUT"
echo 'ui/package.json is present: the ui jobs are scheduled by the ui / ui_consumers outputs.'
else
echo 'present=false' >> "$GITHUB_OUTPUT"
echo 'ui/package.json is absent: every ui job skips on this run.'
fi
# ⭐ SPELLED AS AN ALLOW-LIST, never as a deny-list (#16467). This
# action reads a diff, and only three events hand it one it can resolve
# unaided: `pull_request` (base vs head), `push` (`event.before` vs
# `after`) and `merge_group` (the payload's `base_sha` vs `head_sha`; at
# the pinned v4.0.3, src/main.ts reads both when `base` / `ref` are
# unset, and src/git.ts fetches the base sha at depth 1 and diffs the two
# trees, `base..head`). `schedule` and `workflow_dispatch` hand it
# neither, and under a deny-list spelling they would RUN the action with
# no resolvable base.
#
# That failure direction is the dangerous one: every filter output would
# come back `false`, `|| 'true'` supplies nothing (the output exists and
# says `false`), and every downstream `!= 'false'` guard SKIPS — so the
# hourly full run would have been an entirely green, entirely empty run
# of nothing at all. An allow-list makes each new event fail toward
# THE FILTER CONTRACT's half 1 instead: the step is skipped, its outputs
# are the empty string, `|| 'true'` fills them in, and everything runs.
#
# `merge_group` joined the list for the ui outputs alone (#25867; the
# three core-side outputs above ignore the step on that event). Its
# `continue-on-error` is the fail-open half there: a queue build whose
# base cannot be fetched reads as "run every ui job", never as a red
# `filter` job. On pull_request and push a failure stays red, as before.
- uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
id: changes
if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'merge_group'
continue-on-error: ${{ github.event_name == 'merge_group' }}
with:
filters: |
docs:
- 'apps/docs/**'
- 'content/**'
- 'pnpm-lock.yaml'
- '.github/workflows/ci.yml'
core:
- 'packages/**'
- 'examples/**'
- 'apps/!(docs)/**'
- 'package.json'
- 'pnpm-lock.yaml'
- 'tsconfig.json'
- '.github/workflows/ci.yml'
# BUILD INPUTS outside every package (#21202): turbo.json itself,
# its `globalDependencies`, and each `$TURBO_ROOT$/…` input of a
# build task Build Core's `pnpm build` runs. A diff confined to
# one of them moves the build hashes it reaches, and before these
# entries it matched nothing here, so Build Core skipped and the
# merge queue ran that build first (PR #21199, turbo.json only).
# Every core-gated job starts on such a diff, not Build Core
# alone: each one builds, and turbo re-runs every task downstream
# of a moved build hash.
#
# scripts/check-ci-filter-parity.mjs DERIVES the required set from
# turbo.json and the root `build` script, and reds while a declared
# build input is uncovered here, or while a literal entry beside
# them covers none of them and none of Build Core's guards below
# (a leftover). `tsconfig.json` and the
# `packages/cli/...` input are covered by the entries above;
# `@objectstack/docs#build` (its input is `content/**`) is the
# docs filter's, since `pnpm build` excludes that package.
#
# WIDTH: literal files, not `scripts/**`. Over the 3288
# first-parent commits of `main` in the 30 days to `1371dc980c`,
# through picomatch 2.3.1 and 4.0.5 (they agree on every row),
# `core` matched 2247; these entries add 28, `scripts/**` with
# turbo.json and tsup.config.ts would add 593, 516 of them diffs
# touching a `scripts/pm/` tool or a `scripts/check-*` gate.
- 'turbo.json'
- 'tsup.config.ts'
# `@objectstack/skills#build` copies the repository root's
# `skills/**` into its publish tree, so that catalog is a build
# input of a task `pnpm build` runs (turbo.json declares it as
# `$TURBO_ROOT$/skills/**`; check-ci-filter-parity holds it here).
- 'skills/**'
- 'scripts/build-input-hash.mjs'
- 'scripts/check-dev-prereqs.mjs'
- 'scripts/check-dts-emitted.mjs'
- 'scripts/check-dts-references.mjs'
- 'scripts/check-regen-pending.mjs'
- 'scripts/cli-build-prerequisite.mjs'
- 'scripts/git-env.mjs'
- 'scripts/import-prerequisite.mjs'
- 'scripts/invoked-as.mjs'
- 'scripts/js-comment-mask.mjs'
- 'scripts/regen-artifacts.mjs'
- 'scripts/sync-scaffold-emission-policy.mjs'
- 'scripts/sync-template-versions.mjs'
- 'scripts/ts-parse.mjs'
- 'scripts/tsup-drop-sources-content.mjs'
- 'scripts/workspace-enumerator.mjs'
# BUILD CORE'S GUARDS (#21341): every repository script the
# build-core job's own steps run (named by a `run:` step or
# reached through its `pnpm check:*` alias) and each one's
# first-party imports. Build Core is the only job that runs
# them, so before these rows a diff confined to one started no
# Build Core and the changed guard ran nowhere before the merge
# queue (PR #17100 changed only check-lean-entry-closure.mjs;
# Build Core skipped). A change to the guard runs the guard.
#
# Not a hand-kept list: scripts/check-ci-filter-parity.mjs
# DERIVES the set from the job's steps and reds while a script a
# step runs is uncovered here, or while a literal row covers
# neither a build input nor such a script (a leftover of a
# deleted step). Scripts that are also build inputs
# (check-dts-emitted.mjs and the helpers the guards import) are
# covered by the rows above and are not repeated.
# population-floor.mjs is here because check-dual-build-cjs-
# loads.mjs imports it.
#
# WIDTH: over the 3287 first-parent commits of `main` in the 30
# days to `96b12b589f`, through picomatch 2.3.1 and 4.0.5 (they
# agree on every row), `core` matched 2276; these rows add 2.
- 'scripts/check-dts-closure.mjs'
- 'scripts/check-dual-build-cjs-loads.mjs'
- 'scripts/check-lean-entry-closure.mjs'
- 'scripts/check-sourcemap-no-sources-content.mjs'
- 'scripts/ci/assert-turbo-remote-cache.mjs'
- 'scripts/ci/prune-turbo-cache.mjs'
- 'scripts/population-floor.mjs'
# Test inputs that live OUTSIDE every package (#9829, #10015). Packages
# whose suites read across their own boundary declare that radius in
# scripts/check-cross-package-test-inputs.mjs, and turbo.json mirrors each
# as a `$TURBO_ROOT$/…` input. The step that pulls those packages back into
# the affected set, `--union-into` in the `test` job below, is a step
# INSIDE that job — so when NO filter matches the changed path the job
# never starts, the union never runs, and the declaration buys nothing at
# PR time: the merge queue is the first signal, which is the #7802 failure
# this whole mechanism exists to prevent.
#
# Measured by instantiating every one of turbo.json's `$TURBO_ROOT$`
# declarations to a real tracked file and running it through these filters
# with picomatch — the matcher dorny/paths-filter uses — under BOTH 2.3.1
# (what the action's own lockfile resolves and ncc-bundles) and 4.0.5 (what
# this tree resolves); the two agree on every row. On `0bfbeb70c`, 19 of 64
# declarations were core=false. #9829 closed the nine rooted at `scripts/`;
# the four roots added below are the remaining ten (#10015).
#
# Deliberately a SEPARATE output rather than a widening of `core`, because
# it gates ONE job — the `test` job ORs it with `core`, while Build Core,
# Dogfood and Temporal Conformance stay filtered exactly as they are today.
# That separation is what makes `content/**` affordable at all. Widening
# `core` would put the whole core pipeline on every docs PR; here a
# docs-only diff starts Test Core, whose package set is `turbo ls
# --affected` (empty for such a diff) UNIONED with the packages whose
# declarations matched — so it runs the declaring package, not the
# workspace. Measured through the real `--union-into` on all three
# content-only commits in the window below: `create-objectstack` alone,
# all three times, against a positive control where a
# `packages/lint/src/**` diff unions in @objectstack/spec and
# @objectstack/core.
#
# WIDTH, per entry — each is exactly what some package declared, never a
# root rounded up:
# `scripts/**` and `content/**` are declared verbatim, by
# @objectstack/spec and create-objectstack respectively, so the
# narrower filter does not exist and a hand-kept list of the paths
# inside them is the failure mode #7802 recorded ("a list you must
# remember to update"). Eight further single-file `scripts/`
# declarations and six narrower `content/docs/…` ones (spec, cli,
# dogfood) sit inside the two.
# The last four are a single file, a single subtree, a single file and a
# single file, because that is what the declarations say — NOT `docs/**`,
# `skills/**` or `.github/workflows/**`. Over the last 100 first-parent
# commits of `main` at `0bfbeb70c`, three of those roots would newly have
# started Test Core on 5, 2 and 3 commits; the declared paths on ZERO.
# `skills/objectstack-automation/SKILL.md` joined later (#10178) and was
# NOT in that window's measurement — it is declared by @objectstack/rest,
# one file, and the `skills/**` root it sits in is still declined here.
#
# PRICE, re-measured over that same 100-commit window through this matcher:
# Test Core scheduling goes 80/100 → 83/100. All three additions are
# content-only docs commits, and each runs one package's suite.
#
# This list is a SECOND recognizer of those declarations and is kept in
# step with them BY HAND: nothing fails today when a declaration lands in a
# root no entry here covers. Closing that is #10379.
crosspkg:
- 'scripts/**'
- 'content/**'
- 'docs/audits/2026-07-unknown-key-strictness-ledger.md'
- 'skills/objectstack-formula/**'
- 'skills/objectstack-automation/SKILL.md'
# @objectstack/spec's scripts/export-list.test.ts corpus gate reads
# the whole published catalog (#12201). Verbatim per the declaration;
# it subsumes the two narrower skills entries above, which are left
# as the packages that declared them spelled them.
- 'skills/**'
- '.github/workflows/scaffold-e2e.yml'
- '.claude/skills/spec-property-retirement/SKILL.md'
# @objectstack/lint's production-witness suite reads the repo-root
# SDUI manifest (#12924) — verbatim per the declaration in
# scripts/cross-package-test-inputs.mjs (check:ci-filter-parity
# holds this list to the declarations; the examples/ page files
# the same suite declares are covered by entries the core filter
# already carries).
- 'sdui.manifest.json'
# @objectstack/rest's meta-state-route-doc-spelling test discovers
# its population instead of listing it (#14561): it asks git for
# the authored-prose corpus under content/, docs/ and skills/ and
# judges every file that mentions the route. Verbatim per that
# declaration. `content/**` and `skills/**` above already cover
# two of the three roots; this is the third, and the FIRST entry
# to open the `docs/` root that the WIDTH note above records as
# declined — declined then because no declaration reached past
# `docs/audits/...`, and reached now because one does. The
# narrower audits entry is left as the package that declared it
# spelled it, exactly as the two narrower `skills/` entries were.
- 'docs/**'
# The objectui tree and the root files the ui jobs build from. A
# change to the guard runs the guard, so ci.yml itself is listed.
ui:
- 'ui/**'
- 'pnpm-lock.yaml'
- 'pnpm-workspace.yaml'
- 'turbo.json'
- '.github/workflows/ci.yml'
# The seven packages objectui installs from this tree (the set its
# `.objectstack-pin` used to pack). A change here compiles and builds
# the Console in the same PR — the check the retired Spec Shape Gate
# ran hourly against a clone of objectui main.
ui_consumers:
- 'packages/spec/**'
- 'packages/client/**'
- 'packages/core/**'
- 'packages/formula/**'
- 'packages/lint/**'
- 'packages/sdui-parser/**'
- 'packages/types/**'
test:
# Sharded 8-way BY PACKAGE (6-way until #25376): a core-touching PR ran
# the affected suite ~11½ min on one 4-vCPU runner — the longest pole in
# the whole workflow — and at 2 shards the slower shard still ran ~10 min,
# keeping merge-queue builds at ~11 min end-to-end (#4859).
# scripts/partition-test-shards.mjs splits the package list into eight
# deterministic bins balanced on MEASURED per-package test duration
# (scripts/test-shard-timings.json, generated — see #10472 below) and each
# shard runs its slice through turbo. NOT the dogfood
# job's vitest --shard passthrough, deliberately: that works for dogfood
# because it is ONE package with ~60 files, but applied workspace-wide,
# vitest 4 hard-fails every package with fewer test files than the shard
# count — and `--passWithNoTests` converts the failure into running NOTHING
# on either shard (three packages have exactly one test file today). See
# the script header for the verification.
#
# ── WHY MORE THAN THREE (#10227) ──────────────────────────────────────
#
# The per-package durations this job's logs print are CONTENDED WALL
# CLOCK, not package cost. `--concurrency=4` on a 4-vCPU runner means up
# to four suites share four cores, so each averages about one core, and
# the contention factor differs per package (measured spread across six
# packages: 0.69x-2.05x). Read as cost, those numbers said the two long
# poles were inherently over #4859's <=7min (420s) line and that no shard
# count could help. Measured like-for-like against the same suites run
# ALONE on an idle 4-core box (#10152 / PR #10258), both clear it:
#
# @objectstack/cli 548.6s contended vs 337.13s alone
# @objectstack/spec 496.4s contended vs 325.31s alone
#
# So the thing over the line is the co-tenancy, not the suites. More
# shards means fewer suites per runner AND four more vCPUs per added
# shard, because every shard gets its own runner.
#
# EIGHT specifically (#25376; six from #22075 until then): sharding is BY
# PACKAGE, so no shard can finish faster than the heaviest serial task it
# carries. `@objectstack/cli` (2899.02s on the 2026-10-23 dataset) runs as 3
# file-level slices whenever a run's own list makes it the serial floor
# (FILE_SHARDED_PACKAGES and planShards() in the script), each slice placed
# at its PLAIN weight so every bin keeps the whole-package split's density,
# and no slicing may put a shard past that split's densest bin (the density
# cap, densityCap() in the script; the shard-timing drift check below is
# why). Re-derived with the partitioner's own planShards() and balanceOf()
# on scripts/test-shard-timings.json, bins of predicted test windows:
#
# 4 shards -> max 3733s mean 3732s ratio 1.00x (the CLI runs whole)
# 6 shards -> max 2489s mean 2488s ratio 1.00x
# 7 shards -> max 2136s mean 2133s ratio 1.00x
# 8 shards -> max 1868s mean 1866s ratio 1.00x
# 9 shards -> max 1661s mean 1659s ratio 1.00x
# 10 shards -> max 1607s mean 1493s ratio 1.08x
#
# Bins are not wall. A shard's whole packages run four at a time while a
# CLI slice runs as a serial leg after them, so its tests step reads close
# to wholeSum / 2.9 + slice. On merge_group run 40155484566 (59 packages
# executed, the CLI as 3 slices) that model lands within 6.1% of the
# measured step on the four slowest shards (the two light ones never set
# the wall); replayed on that run's own list it puts the slowest shard
# at ~1696s at six (measured 1794s, a slice shard) and ~1436s at eight,
# -15%. Full dataset list -14%; a 21-package merge_group run (40157571602)
# -10%, its slice shards at eight carrying almost nothing else. The
# partitioner's SHARD_COUNT header carries the arithmetic.
#
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins
# 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now
# made in measured seconds, because #10472 replaced the weight input: the
# file count was a proxy that ran ~2.8x off on @objectstack/cli, and the
# six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/
# 9.0 min in run 32428961038. Weights now come from
# scripts/test-shard-timings.json — see partition-test-shards.mjs.
#
# AND WHY NOT MORE THAN EIGHT. #10472 asked for 6 -> 8 and was refused
# while spec's 1146.69s was the floor; #25376 widened once the CLI was
# sliced and spec's 842s repo file was split (#25372). Two floors remain.
# In bins, spec's whole 1606.55s binds at ten shards (above), and past it
# max/mean gets WORSE with every shard. In wall, a CLI slice (~960-1010s
# measured) is serial: at eight the slice shards of a small run already
# carry little else, so a ninth shard mostly adds a runner. The
# partitioner's self-test pins the half of this that can fail: at
# SHARD_COUNT the heaviest item must sit within 1.3x the mean, and when it
# stops fitting, slicing it below package granularity is the remedy that
# failure names.
#
# ⚠ THE COST, stated because it is real: per-shard fixed overhead is paid
# once PER SHARD, and the merge queue multiplies it by the speculative
# build count. Measured over the 512 `Test Core (N/6)` jobs of the window
# below, job start to the tests step is 1.6 min median (5.5 p90) and the
# tail after it 0.2 min, so two more shards cost ≈ +3.6 runner-min per run
# at the median, accepted by the maintainer directive on #25376.
# `--concurrency=4` is deliberately UNCHANGED here: narrower shards may
# well want a lower value, but moving both at once makes the next
# measurement unreadable.
#
# Branch protection requires the bare "Test Core" context, which a matrix
# can never publish again — the test-gate job below carries that name
# (the #3622 lesson; see dogfood-gate).
name: Test Core (${{ matrix.shard }}/8)
needs: filter
# "Skip only when the filter EXPLICITLY said no core paths changed" — see
# THE FILTER CONTRACT on the filter job's outputs (#4928). `scripts` is ORed
# in, never ANDed: two independent reasons to run this job, and it skips only
# when BOTH filters explicitly said false, so the contract's "when in doubt,
# RUN EVERYTHING" posture is unchanged (#9829).
if: ${{ !cancelled() && (needs.filter.outputs.core != 'false' || needs.filter.outputs.crosspkg != 'false') }}
runs-on: ubuntu-latest
# Backstop only — the stall guard on the test steps is the primary
# detector for a #4250-style hang and fires well before this. 30 min was
# chosen as ~5× a normal sharded run when a normal run was ~4-6 min, with
# margin for a cold Turbo cache; the old 45 left a hung job "running" for
# half an hour past any plausible healthy finish.
#
# History: RAISED 30 -> 45 (#16173) while one shard ran the whole CLI and
# was killed at the 30-minute wall; held at 45 until #22075 cut the CLI
# into 3 slices (5919483472).
#
# ⚠ RE-SIZED 45 -> 50 (#22075), derived so that BOTH stall-guard verdicts
# land before this wall on every observed run. Window: every completed,
# non-cancelled CI run whose tested tree carries 5919483472, runs
# 37917885267 to 37956350887 (2026-10-09 10:29Z-16:03Z): 5 schedule,
# 6 push, 30 merge_group and 25 pull_request runs, 396 `Test Core (N/6)`
# jobs; 24 cancelled runs excluded. Slowest job wall per event, in
# minutes: schedule 21.0, push 20.9, merge_group 24.1, pull_request 27.1;
# p90 of all 396 is 22.0. Measured with
# `node scripts/measure-stall-guard-headroom.mjs` over those jobs, the
# worst p + s (job start to the end of "Run this shard's tests") is
# 26m38s, on run 37923333449 `Test Core (3/6)`. Three readings set the
# value, and the largest answer wins:
# * the window verdict, a silent freeze killed 10m after the last
# output: p + s + 10m = 36m38s;
# * the cap-deferred verdict, when the liveness probe sees the suite
# still writing: p + s + 20m = 46m38s, inside 50 by 3m22s. It missed
# 45 on 8 of the 396 jobs and would miss 40 on 60;
# * the 80% wall margin: the slowest job wall, 27m05s on that same job
# (70 packages executed, the CLI as all 3 slices), is within 80% of
# 50 (40.0).
#
# ⚠ RE-SIZED 50 -> 60 (#25376), with the matrix widened 6 -> 8. The same
# tool over the 89 completed, non-cancelled CI runs created 2026-10-24
# 03:00Z-2026-10-25 03:45Z (23 merge_group, 39 pull_request, 16 push,
# 11 schedule; 512 `Test Core (N/6)` jobs) found 50 already short at six:
# worst p + s 40m27s (run 40149547507 `Test Core (1/6)`, a red PR shard;
# healthy worst 37m22s, run 40125226571 `Test Core (4/6)`), so the
# cap-deferred path needed 60.45m. The slowest shard is the one carrying
# a CLI slice; eight shards take it down by 10-15% on the replays in
# partition-test-shards.mjs's SHARD_COUNT header, so s is scaled by 0.90
# (the smallest gain, the safe side) until a window of eight-shard runs
# exists. Five readings, the largest answer wins:
# * the window verdict: p + s + 10m = 2m03s + 34m34s + 10m = 46m37s;
# * the cap-deferred verdict: p + s + 20m = 56m37s, inside 60 by 3m23s;
# * the 80% wall margin: the healthy worst job wall, 37m41s at six, is
# ~34m40s at eight (7m28s + 26m55s + the 0.3m tail), within 80% of 60
# (48.0);
# * the #16465 red line, the slowest shard at or under 70% of its
# timeout: ~34m40s against 42.0;
# * the drift ceiling: that healthy worst step running 1.5x its
# prediction (MAX_MEASURED_OVER_PREDICTED, the DRIFT verdict) lands at
# 7m28s + 26m55s x 1.5 = 47.8m, still a reading before the wall rather
# than a kill.
# The first full-battery run after landing re-takes these on eight-shard
# jobs and is reported on #25376.
#
# ⛔ REVERT CONDITION, restated against measurement, not a card: a value
# below 60 is allowed only when, on a re-measured window of Test Core
# runs (`node scripts/measure-stall-guard-headroom.mjs` over the jobs
# payloads of every completed pull_request, merge_group, push and
# schedule run in it), BOTH guard verdicts -- the window path
# (p + s + 10m) and the cap-deferred path (p + s + 20m) -- land before the
# new value on every observation, AND the slowest `Test Core (N/8)` job
# wall is at or under 80% of it. Read the pull_request and merge_group
# runs, not the hourly schedule alone: scheduled runs mostly replay the
# cache their commit's push run saved. It carries no other expiry — a
# raise with no revert condition beside it becomes permanent by
# forgetting — and the stall guard stays the primary hang detector.
timeout-minutes: 60
permissions:
contents: read
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
- name: Checkout repository
uses: actions/checkout@v7
with:
# Full history so `turbo --affected` can diff against the PR base.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: '22'
- name: Setup pnpm
uses: ./.github/actions/setup-pnpm
- name: Get pnpm store directory
shell: bash
run: |
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v6
with:
path: ${{ env.STORE_PATH }}
key: ${{ runner.os }}-pnpm-store-v3-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-v3-
# Restore-only on PRs: PR-side saves (~5 turbo entries per push) churned
# the repo's 10 GB Actions cache pool and evicted the main-branch seeds —
# observed as sudden cold-cache spikes (Build Core 51s → 4m30s). PRs fall
# back to main's entries via the prefix restore-keys; only main pushes
# save (the "Save Turbo cache" step at the end of the job). Shard-scoped
# key: each shard builds/tests a different half of the workspace (same
# pattern as dogfood). Jobs that used to fall back to this job's
# namespace fall back to Build Core now — neither single shard builds a
# superset anymore.
- name: Restore Turbo cache
uses: actions/cache/restore@v6
with:
path: .turbo/cache
key: ${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-${{ github.ref_name }}-
${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-
${{ runner.os }}-turbo-${{ github.job }}-
- name: Install dependencies
run: pnpm install --frozen-lockfile
# PRs and merge-queue builds: only test packages affected by the diff —
# against the PR base on pull_request, against the group's own
# `base_sha` on merge_group (#16453). spec sits at the root of the
# dependency graph, so spec-touching diffs still run (close to)
# everything — but the many that don't touch spec skip the bulk of the
# 75-package matrix, and at ~120 queue merges a day that is the bulk of
# the queue's test compute. Push to main partitions the FULL package
# list: it gates nobody, and its full run is the ground truth the
# shard-timings refresh reads. (Spec's suite runs here plain /
# uninstrumented; the coverage-instrumented pass lives in the nightly
# coverage-nightly workflow.)
#
# !@objectstack/dogfood: the ~7½-minute dogfood suite is the dedicated
# Dogfood job's whole purpose, and both jobs run under the same `core`
# filter — without the exclusion every core PR executed the suite twice
# in parallel, and it dominated this job's critical path.
#
# `turbo ls` is experimental; the partition script asserts its output
# shape loudly so an upgrade that changes it turns into a red step
# naming the cause, not a silently empty shard. An EMPTY shard file must
# short-circuit the test step below: `turbo run test` with zero --filter
# args runs the entire workspace.
#
# WHERE THE AFFECTED DIFF STARTS (#6195, the #6129 family's third
# consumer). The one thing this base must never be is
# `github.event.pull_request.base.sha`.
#
# The payload's `base.sha` is frozen when the PR is OPENED and does not
# move on `synchronize`. HEAD, meanwhile, is the merge ref
# (`refs/pull/N/merge`) that the checkout above resolves by default on a
# `pull_request` event — no `ref:` is given, so this job stands on a merge
# commit containing everything main has today. Everything main gained
# while the PR sat open therefore lands between the two, and
# `turbo ls --affected` reads it as this PR's own changes: packages only
# SOMEBODY ELSE's merged PR touched get tested on this shard.
#
# The direction is conservative — the frozen base is an ancestor of HEAD,
# so its file set is a strict SUPERSET of this PR's own. Nothing that
# should run is skipped; what degrades is the optimisation itself, and it
# degrades with how long the PR has been open. At ~18 merges a day an
# affected-only shard drifts back toward a full run.
#
# Measured on turbo 2.10.7 against a real merge-ref fixture (base branch
# moved 1 commit touching pkg-b; this PR touched pkg-a only):
# TURBO_SCM_BASE=<frozen base.sha> -> pkg-a, pkg-b
# TURBO_SCM_BASE=merge-base(origin/main,HEAD) -> pkg-a
#
# Four spellings that look like the fix and are not:
# - `base.sha...HEAD` (three dots). Three-dot means
# `merge-base(base.sha, HEAD)..HEAD`, and the frozen sha is ALREADY an
# ancestor of HEAD, so it IS its own merge base and the set does not
# move. Measured: still both packages. (pr-automation.yml records the
# same result for its own diff — same fact, one family.)
# - `HEAD^1`. Correct on a merge ref and silently wrong the day someone
# gives this checkout a `ref:`, where parent^1 becomes the PR's
# previous commit. `merge-base` is right under BOTH checkouts.
# - Dropping the variable and letting turbo default to `main`. Measured
# in a CI-shaped clone: `fetch-depth: 0` populates
# `refs/remotes/origin/*`, NOT local heads, so there is no local `main`
# — and turbo does not error, it silently returns the ENTIRE
# workspace, untouched packages included. Safe, and the whole
# optimisation gone.
# - `TURBO_SCM_BASE=origin/$BASE_REF`, letting turbo resolve the ref.
# Measured correct today, but it rests the shard's package set on
# `turbo ls`'s internal choice of dot-ness — undocumented, and
# `turbo ls` is experimental (see above). Resolving to a commit here
# leaves turbo no choice to make.
# The selection itself lives in scripts/ci/select-shard-packages.sh:
# environment in, `$RUNNER_TEMP/turbo-ls.json` out. This step only
# exports the event's fields, so the same script -- every branch of
# it -- runs locally and under its self-test (#16453). Per event:
# pull_request affected set against merge-base(origin/<base>, HEAD)
# merge_group affected set against the group's `base_sha`, unioned
# with the same cross-package scans. An EMPTY set is a
# legitimate docs-only group: it reaches the "No packages
# on this shard" exit below, every shard still attests,
# and Test Core is an honest green. It is NOT the #10057
# case, which is pull_request-only (the script says why).
# push affected set against `github.event.before`, the commit
# `main` was on before this merge landed (#16467). A zero
# sha -- a first push, or a force-push that rewrote
# history -- falls back to the FULL list, loudly.
# schedule the FULL list: the hourly run IS the full battery, and
# workflow_ the on-demand rerun of it. Neither carries a diff base,
# dispatch and neither should: they are the run that says whether
# the whole workspace is green on `main`.
- name: Compute this shard's package set
env:
OS_SHARD_EVENT_NAME: ${{ github.event_name }}
OS_SHARD_PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
OS_SHARD_PR_PINNED_BASE_SHA: ${{ github.event.pull_request.base.sha }}
OS_SHARD_MERGE_GROUP_BASE_SHA: ${{ github.event.merge_group.base_sha }}
OS_SHARD_PUSH_BEFORE_SHA: ${{ github.event.before }}
run: |
bash scripts/ci/select-shard-packages.sh
node scripts/partition-test-shards.mjs "$RUNNER_TEMP/turbo-ls.json" \
--shard ${{ matrix.shard }}/8 --exclude @objectstack/dogfood \
> "$RUNNER_TEMP/shard-packages.txt"
echo 'Items on this shard (a package name, or a package plus a k/n file-level slice):'
cat "$RUNNER_TEMP/shard-packages.txt"
# ⭐ A SHARD WHOSE TESTS READ NO `.d.ts` BUILDS ITS CLOSURE WITHOUT THEM
# (#25375). vitest loads JavaScript, so on most shards every declaration
# the closure emits -- spec's 6 GB-heap pass, cli's type-checking `tsc`
# -- is written and never read. `OS_SKIP_DTS=1` goes to $GITHUB_ENV, so
# both closure builds below, the test step and the shard's main seed all
# hash in one namespace (turbo.json `globalEnv`).
#
# ⛔ EXCEPT a shard carrying a package in DTS_READERS, whose tests compile
# against BUILT declarations and go red without them (TS7016). Measured
# on 830a63a, the whole Test Core list under `OS_SKIP_DTS=1`: red only in
# @objectstack/cli published-subpath-console/-hook-body.pin (its
# own dist/*.d.ts), scaffold-emission-typechecks,
# scaffold-test-script-passes (spec's .d.ts)
# @objectstack/metadata typescript-serializer-annotation (spec's .d.ts)
# A new test that reads a built `.d.ts` reds the same loud way on a
# skipping shard: add its package here. Every other package's counts
# matched the DTS-on run but one: spec's root-entry-type-nameability
# pin skips (by its own design) and runs in lint.yml's built-dist step.
#
# Local cold closure, 65 packages, 4 cores, `--force`: 5m59s with
# declarations, 1m24s without. The cost is replay: a skipping shard
# cannot replay Build Core's DTS-on remote artifacts, only the DTS-off
# ones the queue and main runs of these shards write.
- name: Choose this shard's declaration build
run: |
DTS_READERS='@objectstack/cli @objectstack/metadata'
KEEP=""
while read -r PKG _; do
for R in $DTS_READERS; do
[ "$PKG" = "$R" ] && KEEP="$KEEP $PKG"
done
done < "$RUNNER_TEMP/shard-packages.txt"
if [ -n "$KEEP" ]; then
echo "Declarations kept: this shard tests$KEEP, which read built .d.ts."
else
echo "OS_SKIP_DTS=1" >> "$GITHUB_ENV"
echo "Declarations skipped: OS_SKIP_DTS=1 for the rest of this shard."
fi
# ⛔ THIS SHARD'S `^build` CLOSURE IS BUILT HERE, ON THE TURBO REMOTE
# CACHE (#22077), so `Run this shard's tests` REPLAYS it from the local
# cache instead of building it. Before this step the closure was built
# inside the test step's own turbo run (the slice step below ran zero
# iterations from #21487 to #22075). Merge-queue run 37623284168, `Test Core
# (3/6)`: `Tasks: 72 successful, 72 total` / `Cached: 1 cached, 72
# total` / `Time: 11m43.373s` for the tests AND their closure in one
# run. That run cannot read the remote, because `test` / `test:repo`
# stay off it (Build Core's PINS), so the closure moves here.
#
# `--filter=PKG^...` selects PKG's dependencies without PKG: the
# `^build` closure that `test` and `test:repo` schedule. Measured on
# 3d9188502e, turbo 2.11.5, all six shards of the local partition: this
# plan equals the test plan's build tasks, 32 to 63 per shard, with
# identical hashes, 0 differing and 0 extra, and every one of them has
# Build Core's hash -- unless the step above set OS_SKIP_DTS (#25375). A `test` task that `dependsOn: ["build"]` (cli and
# metadata in turbo.json) also needs its OWN package's build. This filter
# leaves that out on purpose, so a shard never builds a package its tests
# do not need; the test step builds it as before. Today that is
# `cli#build`, on each shard carrying a CLI slice: the slice step below
# builds it there.
#
# Its own guarded step, for the reason the slice step below gives: a
# guarded SITE is (file, job, step). `.turbo/runs/` stays the test step's
# alone, because that is what the drift check and the timings capture
# read: the `--summarize` here is for the remote-cache assertion after
# this step (#25374), which moves the summary it read out to
# $RUNNER_TEMP before the tests run. The four TURBO_* lines are Build Core's,
# verbatim; its "Turbo remote cache (#21186)" comment holds the write
# rule, the signing and the PINS. ⛔ An empty FILTERS must exit here:
# `turbo run build` with no filter builds the whole workspace.
- name: Build this shard's dependency closure
env:
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
TURBO_CACHE: ${{ (github.event_name == 'merge_group' || ((github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main')) && 'local:rw,remote:rw' || 'local:rw,remote:r' }}
TURBO_TOKEN: ${{ (github.event_name == 'merge_group' || github.ref == 'refs/heads/main') && secrets.TURBO_REMOTE_CACHE_SIGNATURE_KEY != '' && secrets.TURBO_TOKEN || '' }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
TURBO_REMOTE_CACHE_SIGNATURE_KEY: ${{ (github.event_name == 'merge_group' || github.ref == 'refs/heads/main') && secrets.TURBO_REMOTE_CACHE_SIGNATURE_KEY || '' }}
run: |
FILTERS=""
if [ -s "$RUNNER_TEMP/shard-packages.txt" ]; then
while read -r PKG _; do
[ -n "$PKG" ] || continue
FILTERS="$FILTERS --filter=$PKG^..."
done < "$RUNNER_TEMP/shard-packages.txt"
fi
if [ -z "$FILTERS" ]; then
echo "No packages on this shard — nothing to build."
exit 0
fi
mkdir -p "$RUNNER_TEMP/stall-reports"
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/test-core-closure-build.log" --stall-minutes 10 \
--report-dir "$RUNNER_TEMP/stall-reports" -- \
pnpm turbo run build $FILTERS --concurrency=4 --log-order=stream \
--summarize --log-file=.turbo/remote-cache/test-core-closure.json
# Build Core's "Turbo remote cache (#21186)" comment, ASSERTED paragraph.
# `--if-ran`: an empty shard exits the step above before turbo runs.
- name: The Turbo remote cache was on for this shard's closure build
run: |
node scripts/ci/assert-turbo-remote-cache.mjs --site 'Test Core (${{ matrix.shard }}/8) closure' \
--log .turbo/remote-cache/test-core-closure.json --if-ran \
--move-summary-to "$RUNNER_TEMP/turbo-remote-cache"
# ⛔ A FILE-LEVEL SLICE BUILDS ITS DEPENDENCY CLOSURE HERE, IN ITS OWN
# GUARDED STEP, so the slice leg in the next step REPLAYS it. Since #19278
# that leg carries its slice in `OS_TEST_SHARD`, which only the `test`
# task declares, so its build tasks hash exactly as they do here (61 of 61
# identical, measured) and it schedules its own closure: this step is no
# longer what makes the leg correct, only what keeps the closure build off
# the test step's stall-guard site (see the last paragraph below).
#
# Why the step first existed (#16395, when the slice was still a
# passthrough and the leg was `--only`): turbo folds a run-level
# passthrough into the hash of EVERY task in the run, not only the task
# that receives it -- and `-- "--shard=k/n"` was then the whole reason a
# slice got its own invocation at all. Measured on turbo
# 2.10.10, `--filter=@objectstack/cli`, `turbo run test ... --dry=json`
# (60 tasks: 59 `build` + 1 `test`):
#
# plain vs plain 60 identical, 0 changed <- control, fires
# plain vs -- --shard=1/2 0 identical, 60 changed
# --shard=1/2 vs 2/2 0 identical, 60 changed
#
# So the sliced leg could hit NEITHER the main-seeded Turbo cache restored
# above NOR the builds the shared leg ran seconds earlier in the SAME job:
# it re-executed the closure every run, and shards 5/6 and 6/6 -- the two
# that carry a slice -- paid that closure twice per job. Live reading,
# `Test Core (5/6)` of run 34193080219 (a `packages/spec` PR, so the
# affected set reaches cli): the slice leg reported
# `Cached: 2 cached, 58 total` / `Time: 9m18.941s` while
# `@objectstack/cli:test` itself measured `Duration 187.49s` -- six of those
# nine minutes were the duplicate rebuild, on a job that was then standing
# against a 30-minute wall (#16395's measurement).
#
# `turbo run build --filter=$PKG` is that closure and nothing more:
# measured 59 build tasks, all 59 hash-IDENTICAL to the ones in the
# passthrough-free test plan (0 differing, 0 extra, and 0 missing against
# the test's own `^build` closure), so they REPLAY rather than re-execute.
# `--filter=...^$PKG` was measured too and schedules 12 packages this
# closure does not need. Locally, back-to-back invocations of this exact
# command measured `57 cached, 57 total` / `Time: 153ms >>> FULL TURBO` on
# the second, against `5 cached, 57 total` / `3m45.918s` on the first.
#
# ⚠ THIS IS ITS OWN STEP, not a second guarded run inside the step below,
# because a guarded SITE is the triple (file, job, step) --
# `measure-stall-guard-headroom` REFUSES to report a verdict when two
# guarded runs share one, and refusing is right: the two would be judged
# against the worst reading of their union. `pnpm check:stall-guard-budget`
# and `pnpm check:stall-guard-headroom` both read this step, so it keeps
# its own `--stall-minutes` and its own headroom row.
#
# A shard with no slice runs zero iterations here. Since #22075 a run
# whose own list makes `@objectstack/cli` its serial floor cuts it into 3
# slices on 3 shards (partition-test-shards.mjs planShards(), which
# prints its decision in "Compute this shard's package set"), so those
# three shards run one iteration each. Every shard still reaches the
# step, so its name is a stable site for those two gates.
- name: Build the sliced package's dependency closure
env:
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
run: |
if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ]; then
echo "No packages on this shard — nothing to build."
exit 0
fi
mkdir -p "$RUNNER_TEMP/stall-reports"
while read -r PKG SLICE; do
[ -n "$PKG" ] || continue
[ -n "$SLICE" ] || continue
echo "Slice $PKG=$SLICE — building its dependency closure with no passthrough."
LOG="$RUNNER_TEMP/test-core-slice-build-$(printf '%s' "$PKG" | tr -c 'A-Za-z0-9' '-').log"
node scripts/run-with-stall-guard.mjs --log "$LOG" --stall-minutes 10 \
--report-dir "$RUNNER_TEMP/stall-reports" -- \
pnpm turbo run build "--filter=$PKG" --concurrency=4 --log-order=stream
done < "$RUNNER_TEMP/shard-packages.txt"
# ── Test leftovers, step 1 of 2: the snapshot (#23880, ruling 乙) ──────
#
# REPORT ONLY, never a gate; the rationale, the listing and the closed
# exclusion list are in the `scripts/ci/report-test-leftovers.mjs` header.
# Taken HERE, after both dependency-closure builds above and before the
# tests, so the report after the tests can tell what the test run added
# from what the builds wrote. A package's OWN build (a test task that
# turbo.json makes depend on `build` without `^`) still runs inside the
# test step; the report excludes its declared outputs, read from
# turbo.json on every run. The snapshot goes to $RUNNER_TEMP, outside the
# tree, or the report would count it.
#
# ⛔ No `if:` and no `continue-on-error`: the script exits 0 on every
# path, by construction (a failure prints one NOT MEASURED warning), so
# this step cannot fail the shard or withhold its attestation.
- name: Snapshot the working tree before this shard's tests
run: node scripts/ci/report-test-leftovers.mjs snapshot --out "$RUNNER_TEMP/test-leftovers/before.json"
# --concurrency=4: turbo's default (10) oversubscribes the 4-vCPU
# hosted runner; matching the core count bounds peak memory and the
# job is CPU-bound anyway.
# run-with-stall-guard replaces the old `… 2>&1 | tee $log` +
# `set -o pipefail` idiom: the guard tees combined output to the log
# itself and propagates the suite's real exit status, so there is no
# pipe whose status tee could mask (do not reintroduce `| tee`). Its
# actual job is #4250: a run whose output freezes mid-suite while the
# job sits in_progress. Silence past --stall-minutes is declared a
# stall — a labeled red naming the last output line — instead of a
# 20-minute wait for a human (or the job timeout) to notice. 10 min is
# ~5× the longest healthy quiet gap.
#