Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions fork-ci/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,23 +21,34 @@ reason) when that config:

- does not parse (`config.parse`: not JSON, empty, an unknown or bad field, a key written twice, a bad `review` block
in any mode), or is in `enforce` with a rule order `enforce` refuses;
- is deleted, or is no longer a file;
- is deleted, or is no longer a regular file (git mode 100644 or 100755): a folder, a symlink or a submodule entry
is refused, even one whose object is the config's own (the queue opens the config from a checkout, where none of
them is a file);
- is over the caps on a config a pull request writes: 32 KiB, 128 characters per glob in `review.rules[].paths`, and
2,048 glob characters in all;
- cannot be read or checked at all (any error is a refusal that names it, never a crash or a pass).

Merged, any of these would leave a base config that refuses every pull request after it, the fix too. A pull request
that leaves the config exactly as the base has it is not read again, so the caps never apply to a base config that is
already bigger. In bare mode, a head that does not merge cleanly into the base is refused too (GitHub cannot merge it
either).
either), and so is a `git merge-tree` that takes over 300 seconds.

This narrows the race of two pull requests that each leave a good config but merge into a bad one; it does not close
it. The check judges a head on the base as of its run, and no trigger runs it again when new commits land on the base
(`edited` fires on a retarget or a title or body edit), so a pass from before the first merge still stands for the
second. Only the ruleset's "Require branches to be up to date before merging" closes it: the second pull request is
then behind, and the update that brings it up to date is a new head, judged on the merged config.

**Upgrade:**

- To delete the queue config, first take its checks out of the branch ruleset; a pull request that deletes it is
refused.
- A config change that needs more than the caps has to be pushed by an owner.
- After an owner fixes a refused base config by hand, open pull requests no longer need to merge the base first: they
are judged on what they would merge.
- After an owner fixes a refused base config by hand, open pull requests no longer need to merge the base first: a
new run judges them on what they would merge. In checkout mode that takes a new event (a push, a reopen, a title or
body edit): a re-run of the old `pull_request` job checks out the old merge commit again, which holds the old base.
- A base whose config is a symlink or a submodule entry (only an owner's push can leave one now) refuses every pull
request until one makes it a regular file again; that one is judged as a new config.

### pr-queue and head-check: stricter rule order in `review` (#378)

Expand Down
16 changes: 12 additions & 4 deletions fork-ci/actions/head-check/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ make that job a required check (the config's `head_check`):
```yaml
on:
pull_request:
# edited: a base change (a stacked pull request retargeted) runs it again
# edited: a retarget to another base branch runs it again (new commits on the base do not)
types: [opened, synchronize, reopened, ready_for_review, edited]
permissions: {}
jobs:
Expand All @@ -114,9 +114,17 @@ jobs:
The check reads the comments when it runs. After a lens or an override is
posted for the head, run it again (re-run the job, or let a tool the project
uses for its ready command do that); a new push runs it anyway. The `edited`
trigger runs it again when the base branch changes: the answer depends on the
base (its config, and which merged-in commits are the base's). A title or body
edit also runs it; it only reads, so that costs one short job.
trigger runs it again when the pull request is retargeted to another base
branch: the answer depends on the base (its config, and which merged-in
commits are the base's). A title or body edit also runs it; it only reads, so
that costs one short job. Nothing runs it when new commits land on the base,
so a pass can be older than the base: two pull requests that each leave a good
config can still merge into one that does not parse. Turn on "Require branches
to be up to date before merging" in the ruleset (the queue needs it anyway):
the second pull request is then behind, and its update is a new head, judged
on the merged config. A re-run of an old job checks out the old merge commit
again; after the base changed, a new event (a push, a reopen, a title edit)
judges the head on the new base.

**Pin it by the release sha, always.** Use the full sha a `ci/vN` release
names (the release sha of ci/vN), never a branch, a tag alone or a local path
Expand Down
67 changes: 48 additions & 19 deletions fork-ci/actions/head-check/head_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,16 @@ def __init__(self, path, run=None, bare=None, remote='origin'):
# Bare, git logs in the way the queue does (gh's login, GH_TOKEN), never with a helper of the machine's config.
self.auth = pr_queue.GIT_AUTH if self.bare else ()

def git(self, *a, errors='strict'):
"""errors: how git's output is decoded; 'surrogateescape' for file names, which need not be UTF-8."""
def git(self, *a, errors='strict', timeout=None):
"""errors: how git's output is decoded; 'surrogateescape' for file names, which need not be UTF-8. timeout:
seconds, after which git is killed and subprocess.TimeoutExpired raised."""
if self.run:
return self.run(*a)
with tempfile.TemporaryDirectory(prefix='head-check-') as empty:
return subprocess.run(['git', *GIT_SAFE, *self.auth, '--git-dir', self.git_dir, *a], cwd=empty, text=True,
errors=errors, capture_output=True, env={**os.environ, 'GIT_TERMINAL_PROMPT': '0',
'GIT_ATTR_NOSYSTEM': '1', **pr_queue.GIT_ENV})
errors=errors, capture_output=True, timeout=timeout,
env={**os.environ, 'GIT_TERMINAL_PROMPT': '0', 'GIT_ATTR_NOSYSTEM': '1',
**pr_queue.GIT_ENV})

def problem(self):
"""Why the repository cannot be read as git made it, or None: not a git folder, shallow (the base's history is
Expand Down Expand Up @@ -398,6 +400,10 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
MAX_CONFIG_BYTES = 32 * 1024
MAX_GLOB_LENGTH = 128 # characters of one glob in review.rules[].paths
MAX_GLOB_TOTAL = 2048 # characters of all of them together
# The queue opens the config from a checkout (config.load): only a regular file is one there. git keeps a symlink
# (120000) as a blob of its target text, and a submodule (160000) can name any object, so the object alone says nothing.
FILE_MODES = ('100644', '100755')
MERGE_TREE_SECONDS = 300 # a merge that takes longer is refused; the gate's job has 15 minutes in all
LOCKS = ('the base config would refuse every pull request after it, the one that fixes it too, and the queue would '
'stop')

Expand Down Expand Up @@ -432,9 +438,10 @@ def merged_config_problem(checkout, path):
"""Why the config this pull request would merge must not merge, or None. The config that judges a head is the
base's, so a head is judged by the old config and could merge a new one that does not parse (or is in enforce with
a rule order enforce refuses, or is deleted); after it, the base config refuses every head, the one that fixes it
too, and the queue stops. So the config the merge leaves must parse (config.parse), under the caps above. It is
the merge's, never the head's own copy: two pull requests that each leave a good config can merge into a bad one,
and an older head's stale copy says nothing once an owner fixed the base. Checkout mode reads GitHub's merge commit
too, and the queue stops. So the config the merge leaves must parse (config.parse), under the caps above, and stay
a regular file (FILE_MODES: the queue opens it from a checkout). It is the merge's, never the head's own copy: two
pull requests that each leave a good config can merge into a bad one, and an older head's stale copy says nothing
once an owner fixed the base. Checkout mode reads GitHub's merge commit
(HEAD), bare mode `git merge-tree` of the base and the head (a head that does not merge cleanly is refused: the
merged config is unknown, and GitHub cannot merge it either). The merged config is never used for anything else:
the rules that judge the head stay the base's. Any exception is a refusal that says why, never a crash or a pass."""
Expand All @@ -448,12 +455,32 @@ def merged_config_problem(checkout, path):
f'{type(e).__name__}: {review_judge.shown(str(e))[:300]}')


def tree_entry(checkout, tree, rel):
"""(mode, type, oid) of the entry `rel` in `tree` as git ls-tree lists it, or None when there is none. Raises
ValueError when git fails or lists anything but that one entry (the caller refuses)."""
r = checkout.git('ls-tree', '-z', tree, '--', rel, errors='surrogateescape')
records = [x for x in (r.stdout or '').split('\0') if x] if r.returncode == 0 else None
if records == []:
return None
meta, _, name = records[0].partition('\t') if records else ('', '', '')
fields = meta.split(' ')
if len(records or ()) != 1 or name != rel or len(fields) != 3 or not re.fullmatch(r'[0-7]{6}', fields[0]) \
or not SHA.fullmatch(fields[2]):
said = review_judge.shown((r.stderr if r.returncode else r.stdout or '').strip()[:200])
raise ValueError(f'git ls-tree {tree[:12]} -- {rel} did not list one entry: {said}')
return tuple(fields)


def _merged_config_problem(checkout, path, base, head):
rel = repo_path(path)
merged = 'HEAD'
if checkout.bare:
m = checkout.git('-c', 'merge.directoryRenames=true', 'merge-tree', '--write-tree', base, head,
errors='surrogateescape')
try:
m = checkout.git('-c', 'merge.directoryRenames=true', 'merge-tree', '--write-tree', base, head,
errors='surrogateescape', timeout=MERGE_TREE_SECONDS)
except subprocess.TimeoutExpired:
return (f'git merge-tree of the base {base[:12]} and the head {head[:12]} took over {MERGE_TREE_SECONDS} '
'seconds, so the config it would merge is unknown')
tree = (m.stdout or '').split()[:1]
if m.returncode == 1:
return (f'the head {head[:12]} does not merge cleanly into the base {base[:12]}, so the config it would '
Expand All @@ -462,18 +489,20 @@ def _merged_config_problem(checkout, path, base, head):
said = review_judge.shown((m.stderr or m.stdout or '').strip()[:200])
return f'git merge-tree of the base {base[:12]} and the head {head[:12]} failed: {said}'
merged = tree[0]
new = checkout.git('rev-parse', '--verify', '-q', f'{merged}:{rel}')
if new.returncode != 0:
new = tree_entry(checkout, merged, rel)
if new is None:
return (f'merged, this pull request deletes {path}: {LOCKS}. To stop using the queue, take its checks out of '
'the branch ruleset first')
old = checkout.git('rev-parse', '--verify', '-q', f'{base}:{rel}')
if old.returncode == 0 and old.stdout.strip() == new.stdout.strip():
return None # the merge leaves the base's own config, which base_config parsed
oid = new.stdout.strip()
kind = checkout.git('cat-file', '-t', oid).stdout.strip()
if kind != 'blob':
what = review_judge.shown(kind[:20]) or 'thing'
return f'merged, this pull request leaves {path} as a {what}, not a file: {LOCKS}'
mode, kind, oid = new
if kind == 'tree':
return f'merged, this pull request leaves {path} as a tree, not a file: {LOCKS}'
if mode not in FILE_MODES or kind != 'blob': # even when the base holds the same: unknown never passes
what = {'120000': 'symlink', '160000': 'submodule'}.get(mode) or review_judge.shown(kind[:20])
return (f'merged, this pull request leaves {path} as a {what} (mode {mode}), not a file (mode 100644 or '
'100755): the queue reads the config from a checkout, where it is no file, and would stop. Make it a '
'file again')
if tree_entry(checkout, base, rel) == new:
return None # the merge leaves the base's own config, mode and text, which base_config parsed
size = checkout.git('cat-file', '-s', oid).stdout.strip()
if not size.isdigit() or int(size) > MAX_CONFIG_BYTES:
return config_size_problem(path, size)
Expand Down
Loading
Loading