Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions fork-ci/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,25 @@ nothing.

## After ci/v14

### pr-queue: reads its config with git, never from the checkout's file (#424)

The queue now reads its config as the commit checked out at `config` holds it (HEAD): the blob, with `git ls-tree` and
`git cat-file`, never the file on disk. A checkout writes that file through `.gitattributes` (a
`working-tree-encoding=UTF-16` made it UTF-16 and stopped the queue with a `UnicodeDecodeError`; `ident` rewrote it),
and a pull request could add such an attribute without changing the config's own entry, so the head check passed it.
No checkout filter or attribute changes what the queue reads now. A config that is not UTF-8 is a refusal that says so
(exit 2), never a crash.

The head check (and the gate) also refuses a pull request that adds a name differing from the config's path only in
case (`.github/PR-QUEUE.json`, `.GitHub/pr-queue.json`): a checkout on a case-insensitive file system keeps one of the
two, and which one is up to git. A base that already holds one is not a lock: pull requests that leave or delete it
pass.

**Upgrade:** the action's `config` must be a path inside a git checkout whose HEAD commit holds the config as a regular
file (the reusable workflow's checkout does). The queue refuses to start (exit 2) when the path is not in a git
checkout, when the checked-out commit has no config (an edit or a file that is not committed is not read), or when the
config there is a symlink, a submodule entry or a folder.

### head-check: the config a pull request would merge must parse (#388)

The head check (and the gate) now reads the queue config as the merge would leave it: GitHub's merge commit in
Expand All @@ -22,8 +41,7 @@ reason) when that config:
- does not parse (`config.parse`: not JSON, empty, an unknown or bad field, a key written twice, a bad `review` block
in any mode), or is in `enforce` with a rule order `enforce` refuses;
- is deleted, or is no longer a regular file (git mode 100644 or 100755): a folder, a symlink or a submodule entry
is refused, even one whose object is the config's own (the queue opens the config from a checkout, where none of
them is a file);
is refused, even one whose object is the config's own (the queue reads only a regular file's config);
- is over the caps on a config a pull request writes: 32 KiB, 128 characters per glob in `review.rules[].paths`, and
2,048 glob characters in all;
- cannot be read or checked at all (any error is a refusal that names it, never a crash or a pass).
Expand All @@ -48,7 +66,10 @@ then behind, and the update that brings it up to date is a new head, judged on t
new run judges them on what they would merge. In checkout mode that takes a new event (a push, a reopen, a title or
body edit): a re-run of the old `pull_request` job checks out the old merge commit again, which holds the old base.
- A base whose config is a symlink or a submodule entry (only an owner's push can leave one now) refuses every pull
request until one makes it a regular file again; that one is judged as a new config.
request. A pull request that makes it a regular file again passes (judged as a new config) only when the link's text
is itself a config that parses. A real symlink's text is a path and a submodule entry names a commit, so the check
refuses the fix too ("not JSON"), and an owner has to push the fix by hand, as for any base config that does not
parse.

### pr-queue and head-check: stricter rule order in `review` (#378)

Expand Down
36 changes: 31 additions & 5 deletions fork-ci/actions/head-check/head_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -400,8 +400,8 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
MAX_CONFIG_BYTES = 32 * 1024
MAX_GLOB_LENGTH = 128 # characters of one glob in review.rules[].paths
MAX_GLOB_TOTAL = 2048 # characters of all of them together
# The queue opens the config from a checkout (config.load): only a regular file is one there. git keeps a symlink
# (120000) as a blob of its target text, and a submodule (160000) can name any object, so the object alone says nothing.
# The queue reads the config only from a regular file's blob (pr_queue.committed_config). git keeps a symlink (120000)
# as a blob of its target text, and a submodule (160000) can name any object, so the object alone says nothing.
FILE_MODES = ('100644', '100755')
MERGE_TREE_SECONDS = 300 # a merge that takes longer is refused; the gate's job has 15 minutes in all
LOCKS = ('the base config would refuse every pull request after it, the one that fixes it too, and the queue would '
Expand Down Expand Up @@ -439,7 +439,7 @@ def merged_config_problem(checkout, path):
base's, so a head is judged by the old config and could merge a new one that does not parse (or is in enforce with
a rule order enforce refuses, or is deleted); after it, the base config refuses every head, the one that fixes it
too, and the queue stops. So the config the merge leaves must parse (config.parse), under the caps above, and stay
a regular file (FILE_MODES: the queue opens it from a checkout). It is the merge's, never the head's own copy: two
a regular file (FILE_MODES: the queue reads no other). It is the merge's, never the head's own copy: two
pull requests that each leave a good config can merge into a bad one, and an older head's stale copy says nothing
once an owner fixed the base. Checkout mode reads GitHub's merge commit
(HEAD), bare mode `git merge-tree` of the base and the head (a head that does not merge cleanly is refused: the
Expand Down Expand Up @@ -471,6 +471,28 @@ def tree_entry(checkout, tree, rel):
return tuple(fields)


def case_matches(checkout, tree, rel):
"""The paths in `tree` that name `rel` ignoring case, `rel` itself too: at each level the entries whose name
casefolds to the same as `rel`'s, the last of any type (a folder there collides too). Raises ValueError when git
fails (the caller refuses)."""
parts = rel.casefold().split('/')
level = [('', tree)]
for i, want in enumerate(parts):
found = []
for prefix, oid in level:
r = checkout.git('ls-tree', '-z', oid, errors='surrogateescape')
if r.returncode != 0:
said = review_judge.shown((r.stderr or '').strip()[:200])
raise ValueError(f'git ls-tree {oid[:12]} failed: {said}')
for record in filter(None, (r.stdout or '').split('\0')):
meta, _, name = record.partition('\t')
fields = meta.split(' ')
if name.casefold() == want and (i == len(parts) - 1 or fields[1:2] == ['tree']):
found.append((prefix + name, fields[-1]))
level = [(p + '/', oid) for p, oid in found] if i < len(parts) - 1 else found
return {p for p, _ in level}


def _merged_config_problem(checkout, path, base, head):
rel = repo_path(path)
merged = 'HEAD'
Expand Down Expand Up @@ -499,8 +521,12 @@ def _merged_config_problem(checkout, path, base, head):
if mode not in FILE_MODES or kind != 'blob': # even when the base holds the same: unknown never passes
what = {'120000': 'symlink', '160000': 'submodule'}.get(mode) or review_judge.shown(kind[:20])
return (f'merged, this pull request leaves {path} as a {what} (mode {mode}), not a file (mode 100644 or '
'100755): the queue reads the config from a checkout, where it is no file, and would stop. Make it a '
'file again')
'100755): the queue reads only a regular file\'s config and would stop. Make it a file again')
added = sorted(case_matches(checkout, merged, rel) - case_matches(checkout, base, rel))
if added: # checked before the skip below: a twin leaves the config's own entry as it was
return (f'merged, this pull request adds {review_judge.shown(added[0])[:200]}, which differs from {path} only '
'in case: a checkout on a case-insensitive file system (macOS, Windows) keeps one of the two, and '
'which one is up to git. Rename or delete it')
if tree_entry(checkout, base, rel) == new:
return None # the merge leaves the base's own config, mode and text, which base_config parsed
size = checkout.git('cat-file', '-s', oid).stdout.strip()
Expand Down
34 changes: 33 additions & 1 deletion fork-ci/actions/head-check/head_check.test.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Tests of head_check.py (run: python3 fork-ci/actions/head-check/head_check.test.py).

The check fails a pull request head that is not the change its review read, with the merge queue's own rules
Expand Down Expand Up @@ -1009,7 +1009,7 @@
code, out = self.check(base, self.entry(docs, '100755', oid, base), EVERY_LENS)
self.assertEqual(code, 0, out)

def test_a_base_that_holds_a_symlink_config_refuses_heads_until_one_makes_it_a_file(self): # old state: no lock
def test_a_base_that_holds_a_symlink_config_refuses_heads_until_one_makes_it_a_file(self): # old state
m = self.base(mode=None)
oid = self.w.git(self.w.work, 'rev-parse', f'{m}:{CONFIG_PATH}')
base = self.entry(m, '120000', oid, m) # the gate reads the link's text; the queue's open() cannot
Expand All @@ -1022,6 +1022,14 @@
fix = self.w.commit(base_files(cfg_raw=cfg, **PROMPTS, **self.STORE), [base])
code, out = self.check(base, fix, EVERY_LENS)
self.assertEqual(code, 0, out)
for name, text in (('a link to a path', 'pr-queue.real.json'), ('a submodule', m)): # #424 item 2: a lock
with self.subTest('the fix is refused too when the base holds ' + name):
mode = '160000' if text == m else '120000'
locked = self.entry(m, mode, text if text == m else self.blob(text), m)
fix = self.w.commit(base_files(**PROMPTS, **self.STORE), [locked])
code, out = self.check(locked, fix, EVERY_LENS)
self.assertEqual(code, H.REFUSED, out) # only an owner's push makes it a file again
self.assertIn(f'{CONFIG_PATH} at the base {locked[:12]}: not JSON', out)
big = json.dumps({**CFG_RAW, 'lens_format_doc': 'd' * H.MAX_CONFIG_BYTES}) # the gate reads it, uncapped
base = self.entry(m, '120000', self.blob(big), m)
fix = self.w.commit(base_files(cfg_raw=json.loads(big), **PROMPTS, **self.STORE), [base])
Expand All @@ -1030,6 +1038,30 @@
self.assertEqual(code, H.REFUSED, out)
self.assertIn('bytes the check reads', out)

def test_a_name_that_differs_from_the_config_only_in_case_is_refused(self): # #424 item 3
# A case-insensitive checkout (a macOS or Windows runner) keeps one of the two, and which one is up to git.
base = self.base(mode=None)
for twin in ('.github/PR-QUEUE.json', '.GitHub/pr-queue.json', '.GITHUB/Pr-Queue.JSON/x.json'):
with self.subTest(twin):
code, out = self.check(base, self.head(base, mode=None, **{twin: 'not json\n'}), EVERY_LENS)
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'merged, this pull request adds {twin.split("/x.json")[0]}, which differs from '
f'{CONFIG_PATH} only in case', out)
for name in ('.github/pr-queue.json.bak', '.github/other.json', 'pr-queue.json', '.GITHUB'): # others pass
with self.subTest(name):
self.assertEqual(self.check(base, self.head(base, mode=None, **{name: 'x\n'}), EVERY_LENS)[0], 0)
old = self.base(mode=None, **{'.github/PR-QUEUE.json': 'x\n'}) # old state: a base that holds one already
for name, files, drop in (('a head that leaves it', {'docs/a.md': 'a\n'}, ()),
('a head that deletes it', {}, ('.github/PR-QUEUE.json',))):
with self.subTest(name):
head = self.head(old, mode=None, drop=drop, **{'.github/PR-QUEUE.json': 'x\n', **files})
code, out = self.check(old, head, EVERY_LENS)
self.assertEqual(code, 0, out)
code, out = self.check(old, self.head(old, mode=None, **{'.github/PR-QUEUE.json': 'x\n',
'.github/pr-QUEUE.json': 'y\n'}), EVERY_LENS)
self.assertEqual(code, H.REFUSED, out) # a second one is still new
self.assertIn('adds .github/pr-QUEUE.json, which differs', out)

def test_the_caps_hold_at_their_edge(self): # #413 item 4: the documented values, exactly at and one over
base = self.base('shadow')
at = 'd/' * 62 + '*.py' # 128 characters
Expand Down
14 changes: 11 additions & 3 deletions fork-ci/actions/pr-queue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,17 @@ only for a head that holds exactly the change the review read.

## Config

One JSON file in the project's repository. Unknown keys, a key written twice, a
One JSON file in the project's repository, UTF-8. Unknown keys, a key written twice, a
missing required key, an empty `required_checks` or a value of the wrong type is
an error: the queue then exits non-zero and touches no pull request.

The queue reads the config with git, as the commit checked out at its path
(HEAD) holds it, never the file on disk: a checkout writes that file through
`.gitattributes` (an encoding, `ident`), which a pull request can change
without touching the config itself. So the path must be inside a git checkout,
and the config must be committed there as a regular file: an edit that is not
committed is not read, and a symlink, a submodule entry or a folder is refused.

```json
{
"required_checks": ["build", "test", "review-check"],
Expand Down Expand Up @@ -161,8 +168,9 @@ head, the one that fixes it too, and the queue stops (an owner then has to push
the fix by hand). The gate therefore refuses a pull request whose merged config
(GitHub's merge commit, or `git merge-tree` of base and head in the gate's bare
mode) does not parse, is deleted, is no regular file (a folder, a symlink or a
submodule entry), or is in `enforce` with a rule order `enforce` refuses (the
order checks above), and caps what it reads of a config a pull request changes
submodule entry), gains a twin whose name differs from it only in case, or is
in `enforce` with a rule order `enforce` refuses (the order checks above), and
caps what it reads of a config a pull request changes
(32 KiB, 128 characters per glob, 2,048 in all). That one check reads the
merged config; everything else, the rules that judge the head too, the base's.
It judges the merge with the base of its run: two pull requests that each pass
Expand Down
4 changes: 3 additions & 1 deletion fork-ci/actions/pr-queue/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ inputs:
required: false
default: ${{ github.repository }}
config:
description: The queue's JSON config (README.md, "Config").
description: >-
Path of the queue's JSON config inside a git checkout; the queue reads it as the checked-out commit (HEAD) holds
it, never the file on disk (README.md, "Config").
required: true
state-file:
description: Absolute path, outside the job's workspace, of the plan a stopped run leaves for the next one.
Expand Down
16 changes: 7 additions & 9 deletions fork-ci/actions/pr-queue/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,13 @@ def _review(v):


def parse(text, where='config'):
"""A Config from the JSON text, or ConfigError naming every problem at once."""
"""A Config from the JSON text (str, or bytes that must be UTF-8), or ConfigError naming every problem at once."""
if isinstance(text, bytes):
try:
text = text.decode('utf-8')
except UnicodeDecodeError as e: # a ValueError, but not JSON's: say what it is, never crash on it
raise ConfigError(f'{where}: not UTF-8 text ({e.reason} at byte {e.start}): commit the config as UTF-8 '
'JSON') from None
try:
raw = json.loads(text, object_pairs_hook=_no_duplicates)
except ConfigError as e:
Expand Down Expand Up @@ -140,11 +146,3 @@ def parse(text, where='config'):
lens_author_associations=tuple(raw.pop('lens_author_associations')),
override_prefix=override['prefix'], override_association=override['author_association'], **raw)


def load(path):
try:
with open(path, encoding='utf-8') as f:
text = f.read()
except OSError as e:
raise ConfigError(f'{path}: cannot read it: {e}') from None
return parse(text, path)
5 changes: 0 additions & 5 deletions fork-ci/actions/pr-queue/lens.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -664,11 +664,6 @@ def test_not_json_or_not_an_object(self):
with self.assertRaises(config.ConfigError):
config.parse(text)

def test_an_unreadable_file_is_an_error(self):
with self.assertRaises(config.ConfigError) as e:
config.load(os.path.join(HERE, 'testdata', 'no-such-config.json'))
self.assertIn('cannot read it', str(e.exception))


if __name__ == '__main__':
unittest.main()
Loading
Loading