Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions .github/scripts/fork-checks-ran.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,24 @@
lists on a run the open pull requests whose head SHA or head branch matches it, so a second pull request from
another branch at the same commit may be listed on this one's runs; its entry names the other branch, so it is not
counted. GitHub leaves pull_requests empty on runs of a pull request from another repository, so such a run is never
counted (fail closed). Two pull requests from the same head branch may both be listed on each other's runs; both
come from someone who can push to this repository. It waits while that job is queued or running; failed,
cancelled or skipped fails at once, and so does a finished run without the job (a blocked event leaves one). The
Actions API needs only actions: read, which every caller of fork-tests.yml grants.
counted (fail closed). Two pull requests from the same head branch may both be listed on each other's runs, and so
may a pull request closed and a new one opened from the same branch at the same commit (the old pull request's runs
list the new one, and decide until its own run exists); either needs someone who can push to this repository. It
waits while that job is queued or running; failed, cancelled or skipped fails at once, and so does a finished run
without the job (a blocked event leaves one). The Actions API needs only actions: read, which every caller of
fork-tests.yml grants.

The config is read from the pull request's base commit (BASE_SHA, fetched when the checkout lacks it), not from the
checkout: pull_request_target runs the base branch's copy of fork-checks.yml, so a pull request that renames its job
or file is judged by the names the base branch's copy produces, and cannot edit the config its own check reads.
Missing or undecodable there: fail.

It polls every poll_seconds for at most wait_seconds, counted from before the config is read (its git calls count),
then fails. The job's timeout leaves room for the checkout, the cap and one last poll: fork-contracts.test.py checks
it with GIT_TIMEOUT, GIT_CALLS, API_TIMEOUT and API_CALLS_PER_POLL below, so the message shows. An API error fails
closed: a rate limit or a server error is retried within the same cap (and named in the failure if it lasts), any
other error (403, 404 ...) fails at once. A pull request from another fork fails at once (the job's if: skips those;
fork-checks' only job for them always fails).
then fails. The job's timeout leaves room for the checkout, the cap, one last poll and a minute for the job's
set-up: fork-contracts.test.py checks it with GIT_TIMEOUT, GIT_CALLS, API_TIMEOUT and API_CALLS_PER_POLL below, so
the message shows. An API error fails closed: a rate limit or a server error is retried within the same cap (and
named in the failure if it lasts), any other error (403, 404 ...) fails at once. A pull request from another fork
fails at once (the job's if: skips those; fork-checks' only job for them always fails).

Not a pull_request event: a run that another workflow called (GITHUB_WORKFLOW_REF names the caller) has nothing to
check, since its check runs are named "<caller job> / ... / fork-ci-result", never the required "fork-ci-result". A
Expand Down
10 changes: 10 additions & 0 deletions .github/scripts/fork-checks-ran.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,16 @@ def test_an_entry_whose_head_is_another_repository_does_not_count(self):
self.assertEqual(code, 1, out)
self.assertIn('fork-checks did not run', out)

def test_a_run_whose_head_repository_has_another_name_does_not_count(self):
# The id match alone already ties the entry to the run; the run's head repository must also be this one by
# name (GITHUB_REPOSITORY), kept as defence in depth (#449). A foreign name with this repository's id fails.
run = {**wr(5, [job(CFG['check'])]), 'head_repository': {'full_name': 'x/r', 'id': REPO_ID}}
fake = Fake([[run]])
code, out, _ = check(fake)
self.assertEqual(code, 1, out)
self.assertIn('fork-checks did not run', out)
self.assertNotIn('actions/runs/5/jobs', ' '.join(fake.paths))

def test_missing_head_fields_fail_closed(self):
# Without the head branch and repository on both sides the entry cannot be tied to this run: never counted.
entry, run = pr(7), wr(5, [job(CFG['check'])])
Expand Down
48 changes: 39 additions & 9 deletions .github/scripts/fork-contracts.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
running fork-checks-ran.py with the pinned env (PR_NUMBER too, #433), no
continue-on-error, a checkout with its own timeout, and a job timeout that holds the worst
case: the checkout's timeout, the wait cap (or the script's git calls, if longer) and one
last poll, from the script's own limits.
last poll, from the script's own limits, with a minute to spare for the job's set-up.

python3 .github/scripts/fork-contracts.test.py

Expand Down Expand Up @@ -1664,6 +1664,9 @@ def result_mutant(mutate):
_fcr_spec.loader.exec_module(_fcr)
# The script's own limits (fork-checks-ran.test.py checks it keeps to them): its cap counts its git calls.
POLICY_LIMITS = {k: getattr(_fcr, k) for k in ("GIT_TIMEOUT", "GIT_CALLS", "API_TIMEOUT", "API_CALLS_PER_POLL")}
# Room the worst case leaves in the job timeout for the job's own set-up and the script's start (#449): a worst case
# of exactly the timeout would be cancelled before its message.
POLICY_MARGIN = 60


def policy_worst_case(checkout, cap, limits):
Expand Down Expand Up @@ -1702,10 +1705,11 @@ def policy_job_problems(job, fc, limits=POLICY_LIMITS):
problems.append(f"{where}: the second step is not `{POLICY_SCRIPT}` with the pinned env and nothing else")
timeout, cap = job.get("timeout-minutes"), fc.get("wait_seconds")
if not isinstance(timeout, int) or not isinstance(cap, int) or checkout_minutes is None \
or policy_worst_case(checkout_minutes * 60, cap, limits) > timeout * 60:
or policy_worst_case(checkout_minutes * 60, cap, limits) + POLICY_MARGIN > timeout * 60:
problems.append(f"{where}: its worst case (checkout {checkout_minutes!r} min, fork_checks.wait_seconds "
f"{cap!r} or the git calls {limits}, one last poll) does not fit the timeout ({timeout!r} min), "
"so a timeout would cancel the job before its message")
f"{cap!r} or the git calls {limits}, one last poll) plus {POLICY_MARGIN} s for the job's "
f"set-up does not fit the timeout ({timeout!r} min), so a timeout would cancel the job before "
"its message")
return problems


Expand Down Expand Up @@ -1739,18 +1743,32 @@ def policy_problems(policies, wfs, tests_wf, limits=POLICY_LIMITS):
return problems


def policy_problems_or_crash(policies, wfs, tests_wf, limits=POLICY_LIMITS):
"""policy_problems; a crash (a contract bug, or a mutation that cannot apply to the file) is one named problem,
so the later checks, the failure count and the clean-up still run (#449)."""
try:
return policy_problems(policies, wfs, tests_wf, limits)
except Exception as e:
return [f"the policy check crashed: {e!r}"]


all_wfs = {str(p.relative_to(root)): load(p) for p in sorted(workflows_dir.glob("*.y*ml"))}
policies = json.loads(policies_path.read_text()) if policies_path.is_file() else {}
check("policies: .github/fork-actions-policies.json is there", bool(policies))
problems = policy_problems(policies, all_wfs, fork_tests)
problems = policy_problems_or_crash(policies, all_wfs, fork_tests)
check("policies: every pull_request_target workflow is allowed or upstream_blocked, and fork-checks' check "
"names are its jobs'", not problems, "; ".join(problems))


def policy_mutant(mutate, limits=POLICY_LIMITS):
"""(caught, detail): caught when the mutated files give a problem. A crash, in the mutation or in the check, is
not a catch but a named FAIL with its error (#449)."""
p, w, t = copy.deepcopy(policies), copy.deepcopy(all_wfs), copy.deepcopy(fork_tests)
mutate(p, w, t)
return bool(policy_problems(p, w, t, limits))
try:
mutate(p, w, t)
return bool(policy_problems(p, w, t, limits)), ""
except Exception as e:
return False, f"crashed: {e!r}"


FC = ".github/workflows/fork-checks.yml"
Expand Down Expand Up @@ -1811,6 +1829,10 @@ def policy_mutant(mutate, limits=POLICY_LIMITS):
.__setitem__("timeout-minutes", 0),
"a checkout timeout that breaks the budget": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]
.__setitem__("timeout-minutes", (t["jobs"]["fork_checks_ran"]["steps"][0].get("timeout-minutes") or 0) + 2),
# Worst case exactly the job timeout leaves nothing for the job's set-up: cancelled before the message (#449).
"a checkout timeout that fills the job timeout exactly": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]
.__setitem__("timeout-minutes", (t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60
- policy_worst_case(0, p["fork_checks"]["wait_seconds"], POLICY_LIMITS)) // 60),
"fork_checks_ran without PR_NUMBER": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
.pop("PR_NUMBER"),
"fork_checks_ran with PR_NUMBER from elsewhere": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"]
Expand All @@ -1825,14 +1847,22 @@ def policy_mutant(mutate, limits=POLICY_LIMITS):
"fork_checks_ran's checkout with an extra env": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"]
.__setitem__("BASH_ENV", "x"),
}.items():
check(f"policies: {label} fails", policy_mutant(mutate))
check(f"policies: {label} fails", *policy_mutant(mutate))
# A crash in a mutation or in the check is a contract bug: a named FAIL, so the later checks, the count and the
# clean-up still run (#449).
crash = policy_mutant(lambda p, w, t: 1 / 0)
check("policies: a mutant that crashes is a named FAIL, not a traceback",
crash[0] is False and "ZeroDivisionError" in crash[1], repr(crash))
crash = policy_problems_or_crash(policies, all_wfs, None) # no fork-tests.yml to read: the check itself crashes
check("policies: a crash in the check on the real files is a named problem, not a traceback",
len(crash) == 1 and crash[0].startswith("the policy check crashed: AttributeError"), repr(crash))
# The script's limits, as if it changed them: the budget must still fit (#433 fix round 1, nit of both lenses).
for label, change in {
"git calls that outlast the job": {"GIT_TIMEOUT": 300},
"more git calls than the budget holds": {"GIT_CALLS": 20},
"a slower last poll than the budget holds": {"API_TIMEOUT": 200},
}.items():
check(f"policies: {label} fails", policy_mutant(lambda p, w, t: None, {**POLICY_LIMITS, **change}))
check(f"policies: {label} fails", *policy_mutant(lambda p, w, t: None, {**POLICY_LIMITS, **change}))

shutil.rmtree(work, ignore_errors=True)
if failures:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/fork-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,8 @@ jobs:
# with persist-credentials: false fails on upstream's gitlinks); the script fetches the base commit.
# fork-contracts.test.py pins this job (if, env, steps, no continue-on-error) and checks its worst case
# fits the job timeout: the checkout step's own timeout + the wait cap (or the script's git calls, if
# longer) + one last poll (today 3 + 10 + 1 = 14 min of 15). The checkout's timeout makes a slow fetch fail
# longer) + one last poll + 1 min for the job's set-up (today 3 + 10 + 1 + 1 = 15 min of 15, #449; a worst case
# of exactly the timeout would be cancelled before the message). The checkout's timeout makes a slow fetch fail
# as a timed-out Checkout step, not as a job cancelled before the script's message.
# The checkout and the base fetch use no token, so this job works only in a public repository.
fork_checks_ran:
Expand Down
Loading