Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions fork-ci/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,34 @@ nothing.

## After ci/v14

### pr-queue and head-check: the cache's objects are checked before each merge check (#457)

- Before its merge check the queue runs `git fsck` on its cache. git checks an object's hash on only some reads,
never of a tree or blob `merge-tree` reads: a tree or blob an earlier job stored under one side's name made a real
conflict merge clean. In an update the check runs in the background, while GitHub updates the branch. A cache
`git fsck` finds broken, or one a fetch fails on for a reason that is not the network or the login, is downloaded
from scratch once and checked again; one still broken is not merged in (the run keeps `STATE`, and the PR comment
says the cache fails `git fsck`). Old formats `git fetch` takes (time zone `+99999`, a tree out of order, an odd
`.gitmodules`) are ignored; a hash mismatch, a missing object or a corrupt pack never is. A cache that fails the same
way as right after its download from scratch is not downloaded again: a person must look. A fetch's `git gc` runs
in the foreground. `git fsck` reads every object on one core: about 28 seconds for 270,000 objects, at most once
per run.
- The queue's cache and the gate's repository are made with `GIT_DEFAULT_HASH=sha1` and
`GIT_DEFAULT_REF_FORMAT=files` (every git call gets both), and `HEAD` is written by the queue whatever branch git's
`init` names (git 3.0 plans `main`). A git that still makes another format (an `[extensions]` section) is refused
with the extension named: no cache, nothing fetched or merged; in the gate the head's check run fails.
- git calls that reach no remote (the queue's, the gate's `merge-tree`, and `head_check.py` in bare mode) drop
`GIT_TRACE*` and `GIT_CURL_VERBOSE`, as the fetches from GitHub do; `HTTPS://` counts as https in any case. The
head check's checkout mode keeps them, as before (a partial clone may fetch from `origin`).
- A plain `http://` remote, a `transport::address` remote and one that starts with `-` are refused: GitHub redirects
http to https, where git hands the token over, a remote helper runs any program, and git reads `-...` as an
option. `head_check.py --bare --remote ''` is refused (exit 2) instead of becoming `origin`.
- The gate checks git (2.43 or newer) and its remote before it makes its repository: either problem fails the head's
check run ("not judged: ..."), with nothing made or fetched.

**Upgrade:** nothing to do. A kept cache is checked on its next merge check (it was made by the same code, so it
passes); a broken one is downloaded from scratch once.

### pr-queue and head-check: no attributes from HEAD; local git calls run no job program (#450)

- Every git call of the queue, the gate and the head check reads attributes from the empty tree
Expand All @@ -31,8 +59,10 @@ nothing.
for the gate).

**Upgrade:** the queue, the gate and the head check need git 2.43 or newer (`GIT_ATTR_SOURCE` came in 2.41, and
`merge-tree` with it crashed before 2.43); an older git is refused (exit 2), never run without it. A kept queue cache
is rebuilt once if its `HEAD` is not the queue's; nothing else to do.
`merge-tree` with it crashed before 2.43); an older git never merges or judges anything. The queue refuses to start
(exit 2; it reads its config with git first and decides nothing on it), the head check refuses (exit 2), and the gate
fails the head's check run before it makes its repository (#457). A kept queue cache is rebuilt once if its `HEAD` is
not the queue's; nothing else to do.

### pr-queue and head-check: GIT_ATTR_SOURCE and the job's git programs (#442)

Expand Down
13 changes: 11 additions & 2 deletions fork-ci/actions/head-check/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,10 @@ alternates, `info/attributes`). A refusal is a red required check too. git
runs with no hooks, no fsmonitor, no replace refs and no attributes (they are
read from the empty tree): neither the pull request's `.gitattributes` nor a
bare repository's `HEAD` or `attr.tree` can turn a conflict into a clean merge.
`--remote` (where a missing read commit is fetched from) needs `--bare`.
`--remote` (where a missing read commit is fetched from) needs `--bare`, and
is refused when it is empty, plain `http://` (GitHub would redirect it to
https, and git would hand the token over), a `transport::address` remote
helper, or starts with `-` (git would read it as an option).

It reads only and posts nothing. The one write is a fetch, into the job's own
checkout, of a read commit the head dropped (a rebase or force-push; GitHub
Expand Down Expand Up @@ -220,7 +223,13 @@ How it works:
from the caller's tree. Nothing of the pull request is checked out: the
head and the base tip are read from the API and fetched by sha into a new
bare repository, and `head_check.decide` (the same rule code as the action,
in bare mode) judges them. The config is read from the base tip.
in bare mode) judges them. The config is read from the base tip. Before it
makes that repository the gate checks its git (2.43 or newer) and its
remote (https, a path or a remote name): either problem fails the head's
check run ("not judged: ..."), with no repository made and nothing fetched.
The repository is made as the queue makes its cache (SHA-1 objects and files
refs, whatever git's defaults): a git that makes another format fails the
head too, never judged in a repository its config misnames.
- **A check run belongs to a commit, not to a pull request.** The newest check
run of the name on a head counts for every pull request with that head. So
the gate judges a head, never one pull request alone:
Expand Down
7 changes: 5 additions & 2 deletions fork-ci/actions/head-check/head_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,8 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
p.add_argument('--bare', action='store_true', help='read a bare repository (head_gate.py) instead of a checkout')
p.add_argument('--base', default='', help='with --bare: the base tip the config is read from')
p.add_argument('--head', default='', help='with --bare: the head GitHub named when the base was read')
p.add_argument('--remote', help='with --bare: where a missing read commit is fetched from (default origin)')
p.add_argument('--remote', help='with --bare: where a missing read commit is fetched from (default origin): a '
'remote name, a path or a URL; not plain http, not empty')
a = p.parse_args(argv)
required, problems = argument_problems(a.repo, a.pr, a.required_lenses)
bare = None
Expand All @@ -387,7 +388,9 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
problems.append('--base and --head need --bare (a checkout names its own through its merge commit)')
if a.remote is not None and not a.bare:
problems.append("--remote needs --bare (a checkout fetches from its own origin)")
checkout = Checkout(a.path, git, bare=bare, remote=a.remote or 'origin')
elif a.remote is not None and (why := pr_queue.remote_problem(a.remote)): # an empty one is no "origin"
problems.append(f'--remote: {why}')
checkout = Checkout(a.path, git, bare=bare, remote='origin' if a.remote is None else a.remote)
if git is None:
problems += filter(None, [pr_queue.git_problem(), checkout.problem()])
code, ok, reason, rows = decide(a.repo, a.pr, a.config, checkout, required, a.pass_drafts, gh, sleep, out, problems)
Expand Down
36 changes: 36 additions & 0 deletions fork-ci/actions/head-check/head_check.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -597,6 +597,19 @@ def test_remote_needs_the_bare_mode(self): # #450 item 3: checkout mode fetche
self.assertEqual(code, H.REFUSED)
self.assertIn('--remote needs --bare', out[0])

def test_a_remote_that_is_no_url_or_path_is_refused(self): # #457 item 4: never "origin" for an empty one
path = self.w.checkout(self.m1, self.read)
for remote, says in (('', 'the remote is empty'), ('http://github.com/o/r.git', 'plain http'),
('ext::sh -c id', 'remote helper'), ('--upload-pack=touch x', 'starts with -')):
with self.subTest(remote):
for _ in range(2):
calls, out = [], []
code = H.cli(['--repo', REPO, '--pr', '7', '--config', CONFIG_PATH, '--bare', '--path', path,
'--base', self.m1, '--head', self.read, '--remote', remote], env={},
gh=fake_gh((0, json.dumps(pr([])))), out=out.append)
self.assertEqual((code, calls), (H.REFUSED, []), out)
self.assertIn(says, '\n'.join(out))

def test_base_and_head_need_the_bare_mode(self): # a checkout names its own: never two answers to one question
path = self.w.checkout(self.m1, self.read)
out = []
Expand Down Expand Up @@ -1314,6 +1327,29 @@ def run(argv, **kw):
for env in seen:
self.assertEqual((env['LC_ALL'], env['GIT_ATTR_SOURCE']), ('C', '4b825dc642cb6eb9a060e54bf8d69288fbee4904'))

def test_the_gates_local_calls_and_github_fetch_write_no_trace(self): # #457 item 4: merge-tree's stderr is
github = f'https://github.com/{REPO}.git' # put into the public summary
trace = {'GIT_TRACE': '1', 'GIT_TRACE_PACKET': '1', 'GIT_TRACE2_EVENT': '1', 'GIT_CURL_VERBOSE': '1',
'GIT_TRACE_REDACT': '0'}
for name, call, kept in (
('git init', lambda d: head_gate.bare_repository(d, 'gate'), False),
('merge-tree', lambda d: H.Checkout(d, bare=(C, A), remote=github).git('merge-tree', C, A), False),
('the gate fetch', lambda d: head_gate.fetch(H.Checkout(d, bare=(C, A), remote=github), C, A), False),
('the checkout mode', lambda d: H.Checkout(d).git('merge-base', C, A), True)): # its own origin
with self.subTest(name):
seen = []

def run(argv, **kw):
seen.append(kw['env'])
return types.SimpleNamespace(returncode=0, stdout='', stderr='')
work = tempfile.mkdtemp(prefix='head-check-env-')
self.addCleanup(shutil.rmtree, work, True)
with unittest.mock.patch.object(subprocess, 'run', run), unittest.mock.patch.dict(os.environ, trace):
call(work)
self.assertTrue(seen)
got = {k: seen[0][k] for k in trace if k in seen[0]}
self.assertEqual(got, trace if kept else {})

def test_the_checkout_mode_keeps_them_as_before(self): # old callers: its origin is the caller's
for name, call in (('merge-base', lambda d: H.Checkout(d).git('merge-base', C, A)),
('fetch', lambda d: H.Checkout(d).fetch(A))):
Expand Down
18 changes: 8 additions & 10 deletions fork-ci/actions/head-check/head_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -645,19 +645,15 @@ def finish(repo, sha, run, verdict_path, gh, out, env, sleep=time.sleep):

# --- judge ----------------------------------------------------------------------------------------------------------
def bare_repository(work, name):
"""(path, None): a new, empty bare repository in `work` with the queue's own config (pr_queue.CACHE_CONFIG: no
remote, no hook path, no helper), never one an earlier run left; or (None, why it could not be made)."""
"""(path, None): a new, empty bare repository in `work` made as the queue makes its cache (pr_queue.init_bare:
CACHE_CONFIG, no remote, no hook path, no helper, SHA-1 objects and files refs), never one an earlier run left; or
(None, why it could not be made)."""
try:
path = tempfile.mkdtemp(prefix=name + '-', dir=work)
except OSError as e:
return None, f'could not make a folder in {work}: {e}'
r = subprocess.run(['git', *pr_queue.GIT_SAFE, 'init', '-q', '--bare', '--template=', path], text=True,
capture_output=True, env=pr_queue.git_env(os.environ))
if r.returncode != 0:
return None, f'git init failed: {r.stderr.strip()[:300]}'
with open(os.path.join(path, 'config'), 'w', encoding='utf-8') as f:
f.write(pr_queue.CACHE_CONFIG)
return path, None
problem = pr_queue.init_bare(path)
return (None, problem) if problem else (path, None)


def fetch(checkout, base, head):
Expand Down Expand Up @@ -712,6 +708,8 @@ def judge(a, gh, out, env, decide=head_check.decide, sleep=time.sleep):
state, why, _, base, numbers = preflight(a, gh)
if state != 'go':
ok, reason = False, f'not judged: {why}'
elif problem := pr_queue.git_problem() or pr_queue.remote_problem(a.remote): # before git runs at all
ok, reason = False, f'not judged: {problem}'
else:
path, problem = bare_repository(a.work, f'head-{a.sha[:12]}')
checkout = head_check.Checkout(path or a.work, bare=(base, a.sha), remote=a.remote)
Expand All @@ -722,7 +720,7 @@ def judge(a, gh, out, env, decide=head_check.decide, sleep=time.sleep):
and (problem := empty_lenses_problem(checkout, a.config)):
ok, reason = False, f'not judged: {problem}'
else:
shared = list(filter(None, [pr_queue.git_problem(), checkout.problem()]))
shared = list(filter(None, [checkout.problem()]))
required, _ = head_check.required_names(a.required_lenses)
for n in numbers: # every one: the strictest verdict is the head's
code, good, why, rows = decide(a.repo, str(n), a.config, checkout, required, False, gh, sleep, out,
Expand Down
46 changes: 45 additions & 1 deletion fork-ci/actions/head-check/head_gate.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
T = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(T) # first: its guard puts a fake gh on PATH, drops tokens and isolates git

import argparse, base64, io, json, shutil, tempfile, types, unittest, unittest.mock # noqa: E402
import argparse, base64, io, json, shutil, subprocess, tempfile, types, unittest, unittest.mock # noqa: E402

sys.path.insert(0, HERE)
import head_check as H # noqa: E402
Expand Down Expand Up @@ -968,6 +968,50 @@ def test_a_fetch_that_fails_is_a_failed_verdict(self): # GitHub dropped a comm
self.assertEqual((code, v['conclusion']), (0, 'failure'), out)
self.assertIn('could not fetch the base and the head', v['title'])

def test_an_old_git_fails_the_head_before_anything_is_made_or_fetched(self): # #457 item 4: as the notes say
view = T.pr([self.p.lens('correctness', self.read)], head=self.read)
old = 'git 2.43 or newer is needed (GIT_ATTR_SOURCE); found: git 2.42'
with unittest.mock.patch.object(G.pr_queue, 'git_problem', lambda *a: old), \
unittest.mock.patch.object(G, 'fetch', side_effect=AssertionError('fetched with an old git')):
for _ in range(2): # runs again: the same failure
code, out, _ = self.judge(self.routes(self.read), view)
v = self.verdict()
self.assertEqual((code, v['conclusion']), (0, 'failure'), out)
self.assertIn('not judged: git 2.43 or newer', v['title'])
self.assertEqual(os.listdir(self.work), [], 'no repository made')

def test_a_remote_that_is_no_url_or_path_fails_the_head(self): # #457 item 4: http hands the token over
view = T.pr([self.p.lens('correctness', self.read)], head=self.read)
for remote, says in (('http://github.com/o/r.git', 'plain http'), ('ext::sh -c id', 'remote helper')):
with self.subTest(remote), \
unittest.mock.patch.object(G, 'fetch', side_effect=AssertionError('fetched')):
code, out, _ = self.judge(self.routes(self.read), view, remote=remote)
v = self.verdict()
self.assertEqual((code, v['conclusion']), (0, 'failure'), out)
self.assertIn('not judged', v['title'])
self.assertIn(says, v['title'] + v['summary'])
self.assertEqual(os.listdir(self.work), [])

def test_a_git_that_makes_another_format_fails_the_head(self): # #457 item 3: git 3.0 defaults ignored
view = T.pr([self.p.lens('correctness', self.read)], head=self.read)
real = subprocess.run

def run(cmd, **kw):
r = real(cmd, **kw)
if 'init' in cmd and r.returncode == 0:
with open(os.path.join(cmd[-1], 'config'), 'a') as f:
f.write('[extensions]\n\trefStorage = reftable\n')
return r
with unittest.mock.patch.object(subprocess, 'run', run):
for _ in range(2):
code, out, _ = self.judge(self.routes(self.read), view)
v = self.verdict()
self.assertEqual((code, v['conclusion']), (0, 'failure'), out)
self.assertIn('git init made a repository with extensions.refStorage=reftable',
v['title'] + v['summary'])
code, out, _ = self.judge(self.routes(self.read), view) # a git that keeps the formats: judged again
self.assertEqual(self.verdict()['conclusion'], 'success', out)

def test_a_fetch_with_no_login_says_so_by_name(self): # #450 item 7: a private repository and no GH_TOKEN
class NoLogin:
remote = f'https://github.com/{REPO}.git'
Expand Down
28 changes: 25 additions & 3 deletions fork-ci/actions/pr-queue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,9 +389,31 @@ a merge reads a file), and a cache that is not that user's alone (mode
`0700`), has a config or a `HEAD` the queue did not write, has a branch at
`HEAD`, or holds a file that changes what its objects say or how a merge reads
them (`info/grafts`, `shallow`, `objects/info/alternates`,
`objects/info/http-alternates` or `info/attributes`) is rebuilt, never used. A job of the same user
can still change the cache between that check and git's next read: only the
runner rule above stops that.
`objects/info/http-alternates` or `info/attributes`) is rebuilt, never used.
git checks an object's hash on only some reads (never a tree or a blob
`merge-tree` reads), so the queue runs `git fsck` on the cache before its merge
check: an object stored under another object's name (a tree or a blob that
makes a conflict merge clean) is found there. In an update it runs in the
background right after the first fetch, while GitHub updates the branch, and
the queue waits for it before it fetches the new head; in a run that finishes
a stopped one it runs right after the fetch. Old formats that `git fetch`
takes (a commit with time zone `+99999`, a tree out of order, an odd
`.gitmodules`) are no problem (`FSCK_FORMAT_IDS`); an object that does not
hash to its name, a missing or unreadable one, or a corrupt pack is. A cache
`git fsck` finds broken, or one a fetch fails on for a reason that is not the
network or the login, is downloaded from scratch once and checked again; one
still broken after that is not merged in (STATE stays, and the PR comment says
the cache fails `git fsck`). If a later run's `git fsck` gives the same lines
as right after that download, the history GitHub serves is what fails: the
queue does not download it again, and a person must look. A fetch's automatic
`git gc` runs in the foreground, so no repack races the check. `git fsck`
reads every object on one core: about 30 seconds for 270,000 objects, at most
once per run (the queue makes at most one merge check per run). The cache is
made with SHA-1 objects and files refs whatever git's defaults
(`GIT_DEFAULT_HASH`, `GIT_DEFAULT_REF_FORMAT`; git 3.0 plans others); a git
that makes another format is refused, never written over. A job of the same
user can still change the cache between that check and git's next read: only
the runner rule above stops that.

| Exit | Meaning |
| ---- | ----------------------------------------------------------------------------------------------------------- |
Expand Down
Loading
Loading