Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/fork-actions-policies.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"about": "The one place the repository's Actions policy for pull_request_target is defined (issue #401). From 2026-12-02 GitHub blocks pull_request_target unless a repository Actions policy (Settings > Actions > Policies, rule 'Restrict events') allows it for the workflow file. pull_request_target.allowed: the fork workflows the policy must allow, exactly these. pull_request_target.upstream_blocked: upstream's workflows with that trigger; the fork keeps them disabled and the policy does not name them, so they stay blocked. fork-contracts.test.py (section policies) checks every workflow with the trigger is in one list. fork_checks: the check run fork-tests.yml's fork_checks_ran job must find on a pull request's head (fork-checks-ran.py), so a blocked or renamed fork-checks.yml fails the required fork-ci-result instead of passing silently.",
"about": "The one place the repository's Actions policy for pull_request_target is defined (issue #401). From 2026-12-02 GitHub blocks pull_request_target unless a repository Actions policy (Settings > Actions > Policies, rule 'Restrict events') allows it for the workflow file. pull_request_target.allowed: the fork workflows the policy must allow, exactly these. pull_request_target.upstream_blocked: upstream's workflows with that trigger; the fork keeps them disabled and the policy does not name them, so they stay blocked. fork-contracts.test.py (section policies) checks every workflow with the trigger is in one list. fork_checks: the job fork-tests.yml's fork_checks_ran job requires to have passed in the newest pull_request_target run of that workflow on a pull request's head (fork-checks-ran.py), so a blocked, failed or renamed fork-checks.yml fails the required fork-ci-result instead of passing silently. fork-checks-ran.py reads this section from the pull request's base commit (#430), so a pull request that renames the workflow or the job is judged by the names that are live; outside_check names the job that refuses outside pull requests, pinned by the contract test.",
"pull_request_target": {
"allowed": [".github/workflows/fork-checks.yml"],
"upstream_blocked": [
Expand Down
241 changes: 179 additions & 62 deletions .github/scripts/fork-checks-ran.py

Large diffs are not rendered by default.

474 changes: 391 additions & 83 deletions .github/scripts/fork-checks-ran.test.py

Large diffs are not rendered by default.

130 changes: 118 additions & 12 deletions .github/scripts/fork-contracts.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,16 +45,20 @@
can merge.
policies every workflow with a pull_request_target trigger is listed in
.github/fork-actions-policies.json (#401): allowed (the fork workflows the repository's
Actions policy allows; each there, with the trigger) or upstream_blocked; fork-checks'
check names are its jobs' names, and fork-tests.yml's fork_checks_ran runs
fork-checks-ran.py with actions: read.
Actions policy allows; each there, with that trigger and no other) or upstream_blocked;
fork-checks' check names are its jobs' names, and fork-tests.yml's fork_checks_ran is
exactly the pinned job (#430): its if:, permissions, the shared checkout and the step
running fork-checks-ran.py with the pinned env (PR_NUMBER too, #433), no
continue-on-error, and a timeout that holds the worst case: the checkout, the wait cap (or
the script's git calls, if longer) and one last poll, from the script's own limits.

python3 .github/scripts/fork-contracts.test.py

Needs PyYAML, git and bash. Each check prints ok/FAIL; any FAIL exits 1.
"""

import copy
import importlib.util
import json
import os
import pathlib
Expand Down Expand Up @@ -1629,9 +1633,65 @@ def result_mutant(mutate):
# (fork-tests.yml's fork_checks_ran) proves the effect on each pull request.
policies_path = root / ".github/fork-actions-policies.json"
POLICY_SCRIPT = "python3 -I .github/scripts/fork-checks-ran.py"
# fork_checks_ran as it must be (#430): any other env value could turn it into a silent pass (EVENT_NAME
# from another expression, OUTSIDE_PR hard-coded, the base or head from elsewhere).
POLICY_ENV = {
"EVENT_NAME": "${{ github.event_name }}",
"HEAD_SHA": "${{ github.event.pull_request.head.sha }}",
"BASE_SHA": "${{ github.event.pull_request.base.sha }}",
"PR_NUMBER": "${{ github.event.pull_request.number }}",
"OUTSIDE_PR": "${{ github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository }}",
"DEFAULT_BRANCH": "${{ github.event.repository.default_branch }}",
"GITHUB_TOKEN": "${{ github.token }}",
}
POLICY_CHECKOUT_ENV = {"REF": "${{ github.sha }}", "HISTORY": "tip", "SPARSE": "/.github/scripts/"}
POLICY_JOB_KEYS = {"name", "if", "runs-on", "timeout-minutes", "permissions", "steps"}
POLICY_CHECKOUT_BUDGET = 180 # seconds for the token-free sparse checkout of one commit (a few seconds in CI)
_fcr_spec = importlib.util.spec_from_file_location("fork_checks_ran", root / ".github/scripts/fork-checks-ran.py")
_fcr = importlib.util.module_from_spec(_fcr_spec)
sys.dont_write_bytecode = True # no __pycache__ in the tree
_fcr_spec.loader.exec_module(_fcr)
# The script's own limits (fork-checks-ran.test.py checks it keeps to them): its cap counts its git calls.
POLICY_LIMITS = {k: getattr(_fcr, k) for k in ("GIT_TIMEOUT", "GIT_CALLS", "API_TIMEOUT", "API_CALLS_PER_POLL")}


def policy_worst_case(cap, limits):
"""Seconds fork_checks_ran can take: the checkout, then the cap (the git calls run inside it, or past it if they
take longer), then one last poll after the deadline."""
return (POLICY_CHECKOUT_BUDGET + max(cap, limits["GIT_TIMEOUT"] * limits["GIT_CALLS"])
+ limits["API_TIMEOUT"] * limits["API_CALLS_PER_POLL"])


def policy_job_problems(job, fc, limits=POLICY_LIMITS):
"""fork-tests.yml's fork_checks_ran: exactly the pinned shape (the shared checkout step itself is held by the
checkout section)."""
problems = []
where = "fork-tests.yml: job fork_checks_ran"
if set(job) != POLICY_JOB_KEYS:
problems.append(f"{where} has keys {sorted(set(job))}, not {sorted(POLICY_JOB_KEYS)} (continue-on-error, needs, env ...)")
if expression(job.get("if", "")) != PR_GUARD:
problems.append(f"{where}: if: is not the pull request guard alone")
if job.get("permissions") != {"actions": "read", "contents": "read"}:
problems.append(f"{where}: permissions are not exactly actions: read, contents: read")
steps_ = job.get("steps") or []
if len(steps_) != 2:
problems.append(f"{where}: not exactly two steps (checkout, script)")
else:
checkout, script = steps_
if "git init" not in str(checkout.get("run", "")) or checkout.get("env") != POLICY_CHECKOUT_ENV:
problems.append(f"{where}: the first step is not the shared checkout of github.sha with only the script")
if set(script) != {"name", "env", "run"} or str(script.get("run", "")).strip() != POLICY_SCRIPT \
or script.get("env") != POLICY_ENV:
problems.append(f"{where}: the second step is not `{POLICY_SCRIPT}` with the pinned env and nothing else")
timeout, cap = job.get("timeout-minutes"), fc.get("wait_seconds")
if not isinstance(timeout, int) or not isinstance(cap, int) or policy_worst_case(cap, limits) > timeout * 60:
problems.append(f"{where}: its worst case (checkout {POLICY_CHECKOUT_BUDGET} s, fork_checks.wait_seconds "
f"{cap!r} or the git calls {limits}, one last poll) does not fit the timeout ({timeout!r} min), "
"so a timeout would cancel the job before its message")
return problems


def policy_problems(policies, wfs, tests_wf):
def policy_problems(policies, wfs, tests_wf, limits=POLICY_LIMITS):
"""wfs: tree path -> parsed workflow (every workflow file, upstream's too)."""
prt = policies.get("pull_request_target") or {}
allowed, blocked = prt.get("allowed") or [], prt.get("upstream_blocked") or []
Expand All @@ -1642,8 +1702,9 @@ def policy_problems(policies, wfs, tests_wf):
for path in allowed:
if path not in wfs:
problems.append(f"allowed {path} is not in the tree (renamed? the Actions policy names the old path)")
elif "pull_request_target" not in triggers(wfs[path]):
problems.append(f"allowed {path} has no pull_request_target trigger")
elif triggers(wfs[path]) != {"pull_request_target"}:
# Another trigger runs the pull request's own copy of the file: a same-named job of no worth (#430).
problems.append(f"allowed {path} has triggers {sorted(triggers(wfs[path]))}, not pull_request_target alone")
if not path.startswith(".github/workflows/fork-"):
problems.append(f"allowed {path} is not a fork workflow")
for path in blocked:
Expand All @@ -1656,10 +1717,7 @@ def policy_problems(policies, wfs, tests_wf):
names = {str(j.get("name")) for j in (wfs.get(fc["workflow"], {}).get("jobs") or {}).values()}
problems += [f"{fc['workflow']} has no job named {fc.get(k)!r} (fork_checks.{k})"
for k in ("check", "outside_check") if fc.get(k) not in names]
job = (tests_wf.get("jobs") or {}).get("fork_checks_ran") or {}
if not any(str(s.get("run", "")).strip() == POLICY_SCRIPT for s in job.get("steps") or []) \
or (job.get("permissions") or {}).get("actions") != "read":
problems.append(f"fork-tests.yml: job fork_checks_ran does not run `{POLICY_SCRIPT}` with actions: read")
problems += policy_job_problems((tests_wf.get("jobs") or {}).get("fork_checks_ran") or {}, fc, limits)
return problems


Expand All @@ -1671,10 +1729,10 @@ def policy_problems(policies, wfs, tests_wf):
"names are its jobs'", not problems, "; ".join(problems))


def policy_mutant(mutate):
def policy_mutant(mutate, limits=POLICY_LIMITS):
p, w, t = copy.deepcopy(policies), copy.deepcopy(all_wfs), copy.deepcopy(fork_tests)
mutate(p, w, t)
return bool(policy_problems(p, w, t))
return bool(policy_problems(p, w, t, limits))


FC = ".github/workflows/fork-checks.yml"
Expand All @@ -1695,8 +1753,56 @@ def policy_mutant(mutate):
"fork_checks_ran without actions: read": lambda p, w, t: t["jobs"]["fork_checks_ran"]["permissions"].pop("actions"),
"fork_checks_ran not running the script": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1].__setitem__(
"run", POLICY_SCRIPT + " || true"),
"fork-checks.yml also on push": lambda p, w, t: w[FC][True].__setitem__("push", None),
"fork-checks.yml also on pull_request": lambda p, w, t: w[FC][True].__setitem__("pull_request", None),
"fork_checks_ran job may fail quietly": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__(
"continue-on-error", True),
"fork_checks_ran step may fail quietly": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1].__setitem__(
"continue-on-error", True),
"fork_checks_ran with another if": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__(
"if", "${{ " + PR_GUARD + " && github.actor != 'x' }}"),
"fork_checks_ran with OUTSIDE_PR hard-coded": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.__setitem__("OUTSIDE_PR", "false"),
"fork_checks_ran with EVENT_NAME from elsewhere": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.__setitem__("EVENT_NAME", "push"),
"fork_checks_ran with the base from the head": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.__setitem__("BASE_SHA", "${{ github.event.pull_request.head.sha }}"),
"fork_checks_ran with an extra env": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.__setitem__("BASH_ENV", "x"),
"fork_checks_ran with a job env": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__("env", {"X": "1"}),
"fork_checks_ran with a step between": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"].insert(
1, {"run": "echo 'EVENT_NAME=push' >> $GITHUB_ENV"}),
"fork_checks_ran checking out with actions/checkout": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"]
.__setitem__(0, {"uses": "actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803"}),
"fork_checks_ran with write permissions": lambda p, w, t: t["jobs"]["fork_checks_ran"]["permissions"].__setitem__(
"actions", "write"),
"a wait cap equal to the job timeout": lambda p, w, t: p["fork_checks"].__setitem__(
"wait_seconds", t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60),
"a job timeout cut below the cap": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__("timeout-minutes", 10),
"a cap that leaves no room for the last poll": lambda p, w, t: p["fork_checks"].__setitem__(
"wait_seconds", t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60 - POLICY_CHECKOUT_BUDGET - 1),
"fork_checks_ran without PR_NUMBER": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.pop("PR_NUMBER"),
"fork_checks_ran with PR_NUMBER from elsewhere": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.__setitem__("PR_NUMBER", "${{ github.run_number }}"),
# The checkout's env (#433 fix round 1, state finding 5): only github.sha, depth 1, only the scripts.
"fork_checks_ran checking out the pull request's head": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]
["env"].__setitem__("REF", "${{ github.event.pull_request.head.sha }}"),
"fork_checks_ran checking out the whole tree": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"]
.__setitem__("SPARSE", ""),
"fork_checks_ran checking out the full history": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"]
.__setitem__("HISTORY", "full"),
"fork_checks_ran's checkout with an extra env": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"]
.__setitem__("BASH_ENV", "x"),
}.items():
check(f"policies: {label} fails", policy_mutant(mutate))
# The script's limits, as if it changed them: the budget must still fit (#433 fix round 1, nit of both lenses).
for label, change in {
"git calls that outlast the job": {"GIT_TIMEOUT": 300},
"more git calls than the budget holds": {"GIT_CALLS": 20},
"a slower last poll than the budget holds": {"API_TIMEOUT": 200},
}.items():
check(f"policies: {label} fails", policy_mutant(lambda p, w, t: None, {**POLICY_LIMITS, **change}))

shutil.rmtree(work, ignore_errors=True)
if failures:
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/fork-follow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -176,7 +176,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -337,7 +337,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -443,7 +443,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/fork-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -171,7 +171,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -405,7 +405,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down Expand Up @@ -545,7 +545,7 @@ jobs:
# .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git
# submodule foreach`, which fails on them, sparse checkout or not. The repository is public,
# so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout.
# same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran.
- name: Checkout
shell: bash
env:
Expand Down
Loading
Loading