Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions fork-ci/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,26 @@ nothing.

## After ci/v14

### pr-queue and head-check: follow-ups of the git config read (#435)

- The queue's config read fetches nothing and is timed. In a partial clone (actions/checkout's sparse checkout) a
config whose blob the sparse-checkout pattern did not check out was fetched from origin with the token the checkout
stored, with no timeout. It is now a refusal (exit 2) that names the cause; each git call of the read has 60
seconds.
- Every git call of the queue and the head check drops the job's `GIT_*` variables that point git at another
repository, object store, index, config or pathspec rule (`GIT_DIR`, `GIT_WORK_TREE`, `GIT_OBJECT_DIRECTORY`,
`GIT_ALTERNATE_OBJECT_DIRECTORIES`, `GIT_INDEX_FILE`, `GIT_NAMESPACE`, `GIT_CONFIG_PARAMETERS`, `GIT_CONFIG_COUNT`
with its `GIT_CONFIG_KEY_*`/`GIT_CONFIG_VALUE_*`, and the like: `pr_queue.GIT_DROP`).
- A base config that is not UTF-8 is a refusal in the head check (exit 2, "not UTF-8 text", as the queue says), not
a crash (exit 1).
- The case-twin rule also refuses a file, symlink or submodule entry that differs only in case from one of the
config's folders (`.GitHub` next to `.github/`), and it counts Unicode NFC and NFD forms of a name as one.

**Upgrade:** a caller whose checkout leaves the config's blob out must check the config out. The reusable workflow's
sparse pattern is the config's path, which matches itself unless the name holds `[ ] * ?`, a leading `!` or `#`, a
trailing space or a backslash: rename such a config. A base that already holds a twin of a config folder is not a
lock: pull requests that leave or delete it pass.

### pr-queue: reads its config with git, never from the checkout's file (#424)

The queue now reads its config as the commit checked out at `config` holds it (HEAD): the blob, with `git ls-tree` and
Expand Down
36 changes: 22 additions & 14 deletions fork-ci/actions/head-check/head_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
head come from differs (Checkout).
`--required-lenses` takes one lens name per line (a name may hold a comma: `data-integrity (DI-1, DI-2)`).
"""
import argparse, json, os, re, subprocess, sys, tempfile, time
import argparse, json, os, re, subprocess, sys, tempfile, time, unicodedata

sys.dont_write_bytecode = True # no __pycache__ next to the scripts
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'pr-queue'))
Expand Down Expand Up @@ -81,7 +81,7 @@ def git(self, *a, errors='strict', timeout=None):
with tempfile.TemporaryDirectory(prefix='head-check-') as empty:
return subprocess.run(['git', *GIT_SAFE, *self.auth, '--git-dir', self.git_dir, *a], cwd=empty, text=True,
errors=errors, capture_output=True, timeout=timeout,
env={**os.environ, 'GIT_TERMINAL_PROMPT': '0', **pr_queue.GIT_ENV})
env=pr_queue.git_env(os.environ))

def problem(self):
"""Why the repository cannot be read as git made it, or None: not a git folder, shallow (the base's history is
Expand Down Expand Up @@ -422,12 +422,12 @@ def base_config(checkout, path, problems):
return None
# Bare, the base is the tip the gate read from GitHub's API, fetched by sha: what the pull request's branch says
# about its own config is never read.
r = checkout.git('show', f'{parents[0]}:{repo_path(path)}')
r = checkout.git('show', f'{parents[0]}:{repo_path(path)}', errors='surrogateescape')
if r.returncode != 0:
problems.append(f'the base commit {parents[0][:12]} has no config {path}')
return None
try:
return config.parse(r.stdout, f'{path} at the base {parents[0][:12]}')
try: # as bytes: config.parse refuses one that is not UTF-8, as the queue does
return config.parse(r.stdout.encode('utf-8', 'surrogateescape'), f'{path} at the base {parents[0][:12]}')
except config.ConfigError as e:
problems.append(str(e))
return None
Expand Down Expand Up @@ -470,12 +470,17 @@ def tree_entry(checkout, tree, rel):
return tuple(fields)


def folded(name):
"""`name` as a case-insensitive file system compares it: case folded, NFC and NFD as one (APFS, HFS+)."""
return unicodedata.normalize('NFD', unicodedata.normalize('NFD', name).casefold())


def case_matches(checkout, tree, rel):
"""The paths in `tree` that name `rel` ignoring case, `rel` itself too: at each level the entries whose name
casefolds to the same as `rel`'s, the last of any type (a folder there collides too). Raises ValueError when git
fails (the caller refuses)."""
parts = rel.casefold().split('/')
level = [('', tree)]
"""The paths in `tree` that name `rel` or one of its folders ignoring case (folded), `rel` itself too: at each level
the entries of any type whose name folds to the same as `rel`'s there (a file, a symlink or a submodule next to the
folder collides with it too); only folders are read on. Raises ValueError when git fails (the caller refuses)."""
parts = [folded(p) for p in rel.split('/')]
level, matches = [('', tree)], set()
for i, want in enumerate(parts):
found = []
for prefix, oid in level:
Expand All @@ -486,10 +491,13 @@ def case_matches(checkout, tree, rel):
for record in filter(None, (r.stdout or '').split('\0')):
meta, _, name = record.partition('\t')
fields = meta.split(' ')
if name.casefold() == want and (i == len(parts) - 1 or fields[1:2] == ['tree']):
found.append((prefix + name, fields[-1]))
level = [(p + '/', oid) for p, oid in found] if i < len(parts) - 1 else found
return {p for p, _ in level}
if folded(name) == want:
if i < len(parts) - 1 and fields[1:2] == ['tree']:
found.append((prefix + name + '/', fields[-1]))
else:
matches.add(prefix + name)
level = found
return matches


def _merged_config_problem(checkout, path, base, head):
Expand Down
86 changes: 79 additions & 7 deletions fork-ci/actions/head-check/head_check.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -972,24 +972,34 @@ def test_a_base_config_over_the_caps_still_judges_heads_that_leave_it_alone(self
self.assertEqual(code, H.REFUSED, out) # a change to it is read under the caps
self.assertIn('bytes the check reads', out)

def entry(self, commit, mode, oid, parent):
"""`commit`'s tree with the config's entry made (mode, oid), committed on `parent`: a symlink (120000) or a
submodule (160000), which World.commit (it writes files) cannot make."""
def entry(self, commit, mode, oid, parent, path=CONFIG_PATH):
"""`commit`'s tree with the entry `path` (the config's) made (mode, oid), committed on `parent`: a symlink
(120000) or a submodule (160000), which World.commit (it writes files) cannot make."""
env = {**os.environ, **GIT_ENV, 'GIT_INDEX_FILE': os.path.join(self.w.dir, 'entry.index')}

def git(*a):
return subprocess.run(['git', *a], cwd=self.w.work, env=env, text=True, capture_output=True,
check=True).stdout.strip()
git('read-tree', commit)
git('update-index', '--add', '--cacheinfo', f'{mode},{oid},{CONFIG_PATH}')
git('update-index', '--add', '--cacheinfo', f'{mode},{oid},{path}')
sha = git('commit-tree', git('write-tree'), '-p', parent, '-m', 'entry')
self.w.n += 1
git('push', '-q', 'origin', f'{sha}:refs/keep/{self.w.n}')
return sha

def blob(self, text):
return subprocess.run(['git', 'hash-object', '-w', '--stdin'], cwd=self.w.work, input=text, text=True,
capture_output=True, check=True).stdout.strip()
data = text.encode() if isinstance(text, str) else text
return subprocess.run(['git', 'hash-object', '-w', '--stdin'], cwd=self.w.work, input=data,
capture_output=True, check=True).stdout.decode().strip()

def test_a_base_config_that_is_not_utf8_is_a_refusal_never_a_crash(self): # #435 item 3: as the queue says
m = self.base(mode=None)
oid = self.blob(json.dumps(CFG_RAW).encode('utf-16'))
base = self.entry(m, '100644', oid, m)
head = self.entry(self.head(base, mode=None, **{'docs/a.md': 'a\n'}), '100644', oid, base)
code, out = self.check(base, head, EVERY_LENS)
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'{CONFIG_PATH} at the base {base[:12]}: not UTF-8 text', out)

def test_a_config_that_is_no_regular_file_is_refused(self): # #413 item 1: the queue open()s it and would stop
base = self.base(mode=None)
Expand Down Expand Up @@ -1047,7 +1057,7 @@ def test_a_name_that_differs_from_the_config_only_in_case_is_refused(self): #
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'merged, this pull request adds {twin.split("/x.json")[0]}, which differs from '
f'{CONFIG_PATH} only in case', out)
for name in ('.github/pr-queue.json.bak', '.github/other.json', 'pr-queue.json', '.GITHUB'): # others pass
for name in ('.github/pr-queue.json.bak', '.github/other.json', 'pr-queue.json', '.GITHUBX'): # others pass
with self.subTest(name):
self.assertEqual(self.check(base, self.head(base, mode=None, **{name: 'x\n'}), EVERY_LENS)[0], 0)
old = self.base(mode=None, **{'.github/PR-QUEUE.json': 'x\n'}) # old state: a base that holds one already
Expand All @@ -1062,6 +1072,43 @@ def test_a_name_that_differs_from_the_config_only_in_case_is_refused(self): #
self.assertEqual(code, H.REFUSED, out) # a second one is still new
self.assertIn('adds .github/pr-QUEUE.json, which differs', out)

def test_git_variables_of_the_job_change_nothing_the_check_reads(self): # #435 item 6: pr_queue.GIT_DROP
base = self.base(mode=None)
head = self.head(base, mode=None, **{'docs/a.md': 'a\n'})
path = self.w.checkout(base, head)
other = os.path.join(self.w.dir, 'other')
os.makedirs(os.path.join(other, 'objects'))
hostile = {'GIT_OBJECT_DIRECTORY': os.path.join(other, 'objects'), 'GIT_INDEX_FILE': os.path.join(other, 'i'),
'GIT_CONFIG_COUNT': '1', 'GIT_CONFIG_KEY_0': 'core.repositoryFormatVersion',
'GIT_CONFIG_VALUE_0': '99', 'GIT_NAMESPACE': 'x', 'GIT_ICASE_PATHSPECS': '1'}
with unittest.mock.patch.dict(os.environ, hostile):
code, out = run_cli(path, fake_gh((0, json.dumps(pr([self.p.lens(n, head) for n in EVERY_LENS],
head=head)))))
made, why = head_gate.bare_repository(self.w.dir, 'hostile')
self.assertEqual(code, 0, out)
self.assertIsNone(why)
self.assertTrue(os.path.isdir(os.path.join(made, 'objects', 'info')), 'git init wrote its objects elsewhere')
self.assertEqual(os.listdir(os.path.join(other, 'objects')), [])

def test_a_twin_of_a_parent_folder_is_refused_whatever_its_type(self): # #435 item 4
# A case-insensitive checkout holds one of `.GitHub` and the folder `.github/`: a link there could lead the
# config's path anywhere.
base = self.base(mode=None)
docs = self.head(base, mode=None, **{'docs/a.md': 'a\n'})
for kind, mode, oid in (('a file', '100644', self.blob('x\n')), ('a symlink', '120000', self.blob('/tmp')),
('a submodule', '160000', base)):
for twin in ('.GitHub', '.GITHUB'):
with self.subTest(kind, twin=twin):
code, out = self.check(base, self.entry(docs, mode, oid, base, path=twin), EVERY_LENS)
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'merged, this pull request adds {twin}, which differs from {CONFIG_PATH} only in '
'case', out)
old = self.entry(docs, '120000', self.blob('/tmp'), base, path='.GitHub') # old state: the base holds one
keeps = self.entry(self.head(old, mode=None, **{'docs/b.md': 'b\n'}), '120000', self.blob('/tmp'), old,
path='.GitHub')
code, out = self.check(old, keeps, EVERY_LENS)
self.assertEqual(code, 0, out) # a head that leaves it

def test_the_caps_hold_at_their_edge(self): # #413 item 4: the documented values, exactly at and one over
base = self.base('shadow')
at = 'd/' * 62 + '*.py' # 128 characters
Expand Down Expand Up @@ -1238,6 +1285,31 @@ def test_text_that_breaks_the_parser_is_a_refusal(self):
with self.subTest(name):
self.assertIn(why, H.config_text_problem(text, CONFIG_PATH))

def test_a_case_twin_listing_that_fails_is_a_refusal(self): # #435 item 5: fewer twins would pass
text = json.dumps(CFG_RAW)

def ls_tree(*a):
if '--' in a: # tree_entry: the config is a file
return types.SimpleNamespace(returncode=0, stdout=f'100644 blob {"2" * 40}\t{a[-1]}\0', stderr='')
return types.SimpleNamespace(returncode=128, stdout='', stderr='fatal: not a tree object')
answers = {('cat-file', '-s'): (0, f'{len(text)}\n'), ('cat-file', 'blob'): (0, text)}
why = H.merged_config_problem(reading({**answers, 'ls-tree': ls_tree}), CONFIG_PATH)
self.assertIn('could not be checked, so it is not passed: ValueError: git ls-tree', why or '')

def test_case_twins_fold_nfc_and_nfd(self): # #435 item 4: macOS file systems hold both forms as one name
sub = 'e' * 40
trees = {C: f'040000 tree {sub}\t.github\0',
sub: ''.join(f'100644 blob {B}\t{n}\0' for n in ('caf\u00e9.json', 'cafe\u0301.json', 'CAFE\u0301.json',
'cafe.json'))}

class Listing:
def git(self, *a, **kw):
return types.SimpleNamespace(returncode=0, stdout=trees[a[2]], stderr='')
for rel in ('.github/caf\u00e9.json', '.github/cafe\u0301.json'):
with self.subTest(rel=ascii(rel)):
self.assertEqual(H.case_matches(Listing(), C, rel), {'.github/caf\u00e9.json', '.github/cafe\u0301.json',
'.github/CAFE\u0301.json'})

def test_a_config_that_is_not_utf8_is_a_refusal(self): # strict: a config is text, and base_config reads it so
def blob(*a):
b'{"x": "\xff"}'.decode()
Expand Down
2 changes: 1 addition & 1 deletion fork-ci/actions/head-check/head_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -651,7 +651,7 @@ def bare_repository(work, name):
except OSError as e:
return None, f'could not make a folder in {work}: {e}'
r = subprocess.run(['git', *pr_queue.GIT_SAFE, 'init', '-q', '--bare', '--template=', path], text=True,
capture_output=True, env={**os.environ, **pr_queue.GIT_ENV})
capture_output=True, env=pr_queue.git_env(os.environ))
if r.returncode != 0:
return None, f'git init failed: {r.stderr.strip()[:300]}'
with open(os.path.join(path, 'config'), 'w', encoding='utf-8') as f:
Expand Down
7 changes: 6 additions & 1 deletion fork-ci/actions/pr-queue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ The queue reads the config with git, as the commit checked out at its path
without touching the config itself. So the path must be inside a git checkout,
and the config must be committed there as a regular file: an edit that is not
committed is not read, and a symlink, a submodule entry or a folder is refused.
The read fetches nothing: in a partial clone (a sparse checkout) the config's
blob must be checked out, so the sparse-checkout pattern has to match its name.
A name with `[ ] * ?`, a leading `!` or `#`, a trailing space or a backslash
does not match itself as a pattern, and the queue then refuses to start.

```json
{
Expand Down Expand Up @@ -168,7 +172,8 @@ head, the one that fixes it too, and the queue stops (an owner then has to push
the fix by hand). The gate therefore refuses a pull request whose merged config
(GitHub's merge commit, or `git merge-tree` of base and head in the gate's bare
mode) does not parse, is deleted, is no regular file (a folder, a symlink or a
submodule entry), gains a twin whose name differs from it only in case, or is
submodule entry), gains a twin whose name differs from it or one of its
folders only in case (Unicode NFC and NFD forms count as one name), or is
in `enforce` with a rule order `enforce` refuses (the order checks above), and
caps what it reads of a config a pull request changes
(32 KiB, 128 characters per glob, 2,048 in all). That one check reads the
Expand Down
Loading
Loading