Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/fork-ci-tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
.github/scripts/fork-fleet-desk-seed.test.py ~10s
.github/scripts/fork-fleet-desk.test.py ~8s
.github/scripts/fork-fleet.test.py ~47s
.github/scripts/fork-fleet-refresh.test.py ~13s
.github/scripts/fork-fleet-refresh.test.py ~46s
.github/scripts/fork-fleet-walls.test.py ~75s
.github/scripts/fork-fleet-walls-contract.test.py ~100s
.github/scripts/fork-fleet-walls-wsl2.test.py ~36s
Expand Down
2 changes: 1 addition & 1 deletion .github/fork-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -1015,7 +1015,7 @@ dispatcher keeps only pacing in memory, so a restart starts over cleanly.
| Launcher | Template configured (`.env` and the like, even a symlink to nothing), a symlink, or writable by others | It refuses to start a runner, and the dispatcher logs the reason. |
| Refresh | The template's folder holds anything else (a host set up before it had a folder of its own) | Refused before any download; `start` and `ready` keep working. Host step 7 gives the one-time move. |
| Refresh | The newest release is below the template's floor (`.fork-fleet-floor`, or one major version down) or `RUNNER_MIN` | Refused; the template stays. Host step 7 says how to take it by hand. |
| Refresh | A hostile tarball: fanned-out symlinks, a huge long-name or pax header, a sparse file, a name over 4096 | Refused quickly, before the header is read into memory. The service also has `MemoryMax=1G`, `CPUQuota=100%`, `TasksMax=64`. |
| Refresh | A hostile tarball: fanned-out symlinks, a huge long-name or pax header, a sparse file of any format, a name over 4096 | Refused at its header, before it is read or inflated; so are long-name and pax headers, or names, past 64 MiB in all, over 64 global pax keys or one set per entry, a size its data lacks, special files, over 4 GiB of files, a long header chain. Unit: `MemoryMax=1G`, `CPUQuota=100%`, `TasksMax=64`. |
| Dispatcher | Template rolled back (a withdrawn release) | The dispatcher may still hold the withdrawn version for up to an hour: no runner starts on that pool until its next hourly read. Heals by itself. |
| Launcher | `launch.conf` or the hook writable by anyone but root | It refuses. |
| Launcher | Bad input (an extra line, a key twice, a malformed value) | It refuses before touching anything. |
Expand Down
102 changes: 73 additions & 29 deletions .github/scripts/fork-fleet-launch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -302,19 +302,32 @@ EOF
# $4 the floor (the oldest version a later refresh may roll this template back to).
# The tarball is treated as hostile (a compromised release passes the sha256 check): read once, checked whole, then
# written by this same reader (no second parser to disagree with). Refused: an absolute name, a "..", a name twice, a
# name under a symlink or a file, a symlink or hard link that leads outside the folder, a device, FIFO, sparse or other
# special file, a .fork-fleet-* name of its own, more than MAX_ENTRIES entries or MAX_BYTES of files, a long-name or pax
# header over MAX_HEADER bytes (refused before it is read into memory), a name or link target over MAX_NAME, symlinks
# that take more than MAX_STEPS steps to follow (fanned-out links would take hours). Stripped (as tar's
# --no-same-owner --no-same-permissions would): owner (all root's), setuid, setgid, sticky and group/other write, and
# every directory is 0755. Files are created with O_EXCL|O_NOFOLLOW and symlinks last, so nothing is ever written
# through a link; the version and floor files too.
# name under a symlink or a file, a symlink or hard link that leads outside the folder, a device, FIFO or other special
# file, a sparse file of any format (GNU or pax: its map is read into memory), a .fork-fleet-* name of its own, more
# than MAX_ENTRIES entries or MAX_BYTES of files (the type and size refused at each header, before the data is
# decompressed), a file whose size its data does not have (a pax GNU.sparse.realsize: gzip would decompress again from
# the start for each file after it), a long-name or pax header over MAX_HEADER bytes or such headers over MAX_NAMES
# bytes read in all (each refused before it is read into memory), names and link targets over MAX_NAMES bytes in all
# (as kept in memory), global pax headers of more than MAX_GLOBAL keys (tarfile copies them into every entry after
# them) or with any key tarfile sets on each entry, more headers in a row than tarfile can follow, a name or link
# target over MAX_NAME, symlinks that take more than MAX_STEPS steps to follow (fanned-out links would take hours).
# Stripped (as tar's --no-same-owner --no-same-permissions would): owner (all root's), setuid, setgid, sticky and
# group/other write, and every directory is 0755. Files are created with O_EXCL|O_NOFOLLOW and symlinks last, so nothing
# is ever written through a link; the version and floor files too.
read -r -d '' UNPACK_PY <<'EOF' || true
# fork-fleet-unpack
import os, posixpath, sys, tarfile, zlib
src, dest, version, floor = sys.argv[1:5]
MAX_ENTRIES, MAX_BYTES = 100000, 4 << 30
MAX_HEADER, MAX_NAME, MAX_STEPS = 65536, 4096, 1000000
# MAX_NAMES bounds both the long-name and pax headers read (counted in the 512-byte blocks read, so empty ones cost
# too) and the names and link targets kept for all MAX_ENTRIES entries. A runner release reads about 1.1 MB of such
# headers (GNU long names; no pax and no global header) and keeps about 0.9 MB of names.
MAX_NAMES, MAX_GLOBAL = 64 << 20, 64
# The global pax keys tarfile sets on every entry after them (a 4300-digit uid is a new int in each).
PER_ENTRY = {*tarfile.PAX_FIELDS, *(f'GNU.sparse.{k}' for k in ('name', 'size', 'realsize', 'major', 'minor', 'map',
'offset', 'numblocks'))}
headers = names = total = 0
NEW = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC
os.umask(0o022)

Expand All @@ -323,6 +336,13 @@ def no(why):
sys.exit(f'refused: {why}')


def kept(s):
"""The bytes a name takes in memory: a str is 1, 2 or 4 bytes a character, by its widest one (one emoji makes
4096 characters 16 KiB, from 4099 bytes of header)."""
w = max(s, default='')
return len(s) * (1 if w <= '\xff' else 2 if w <= '\uffff' else 4)


def parts(name):
if name.startswith('/'):
no(f'{name!r} is an absolute path')
Expand All @@ -333,17 +353,35 @@ def parts(name):


class Capped(tarfile.TarInfo):
"""Refuses a GNU long name or pax header over MAX_HEADER bytes before tarfile reads it whole into memory, and a
GNU sparse file (its chained headers are read into memory too; a runner release has none)."""
"""Refuses a GNU long name or pax header over MAX_HEADER bytes, or past MAX_NAMES bytes of them read in all,
before tarfile reads it whole into memory; global pax keys past MAX_GLOBAL, or one tarfile sets on each entry,
before tarfile copies them into the next entry; and a sparse file of any format before tarfile reads its map (a
runner release has none)."""

def _proc_member(self, tar):
global headers
if len(tar.pax_headers) > MAX_GLOBAL:
no(f'more than {MAX_GLOBAL} global pax keys')
if bad := sorted(tar.pax_headers.keys() & PER_ENTRY): # a new copy in each entry, or a size it does not have
no(f'a global pax header with {bad[0]}')
if self.type == tarfile.GNUTYPE_SPARSE:
no(f'{self.name!r} is a sparse file')
if self.type in (tarfile.GNUTYPE_LONGNAME, tarfile.GNUTYPE_LONGLINK, tarfile.XHDTYPE, tarfile.XGLTYPE,
tarfile.SOLARIS_XHDTYPE) and self.size > MAX_HEADER:
no(f'a header of {self.size} bytes (more than {MAX_HEADER})')
tarfile.SOLARIS_XHDTYPE):
if self.size > MAX_HEADER:
no(f'a header of {self.size} bytes (more than {MAX_HEADER})')
headers += tarfile.BLOCKSIZE + self._block(self.size)
if headers > MAX_NAMES:
no(f'more than {MAX_NAMES} bytes of long names and pax headers')
return super()._proc_member(tar)

# A pax header's sparse map, in any of its three formats, is read (from the header or the entry's data) into
# lists with no bound; called once the next header is read, before the map is.
def _proc_gnusparse_00(self, *_):
no('a pax sparse file')

_proc_gnusparse_01 = _proc_gnusparse_10 = _proc_gnusparse_00


try:
tar = tarfile.open(src, 'r:gz', tarinfo=Capped)
Expand All @@ -352,26 +390,36 @@ try:
# Each name as it is read: many long names would fill memory before a check after the loop.
if len(m.name) > MAX_NAME or len(m.linkname) > MAX_NAME:
no(f'a name or link target longer than {MAX_NAME}: {m.name[:100]!r}')
names += kept(m.name) + kept(m.linkname)
if names > MAX_NAMES:
no(f'more than {MAX_NAMES} bytes of names and link targets')
# A size its data does not have (a pax GNU.sparse.realsize sets it, and tarfile skips the data by the old
# one): each file after it would seek back, and gzip seeks back by decompressing from the start again.
if m.isreg() and tar.offset != m.offset_data + m._block(m.size):
no(f'{m.name[:100]!r} has a size its data does not have')
if 'GNU.sparse.realsize' in m.pax_headers:
no(f'{m.name[:100]!r} has a pax GNU.sparse.realsize')
# The type and the size at the header, before tarfile decompresses past the data (a gzip bomb).
if not (m.isdir() or m.issym() or m.islnk() or m.isreg()):
what = 'device' if m.ischr() or m.isblk() else 'FIFO' if m.isfifo() else 'special'
no(f'{m.name!r} is a {what} file')
if m.isreg():
total += m.size
if total > MAX_BYTES:
no(f'more than {MAX_BYTES} bytes of files')
m.pax_headers = {} # its copy of the global pax keys (and its own): not used after this
members.append(m)
if len(members) > MAX_ENTRIES:
no(f'more than {MAX_ENTRIES} entries')
except (tarfile.TarError, OSError, EOFError, zlib.error) as e:
except RecursionError: # tarfile recurses on each header in a row (pax, long name) before an entry
no('more headers in a row than tarfile can follow')
except (tarfile.TarError, OSError, EOFError, zlib.error, ValueError) as e: # ValueError: int() of 4301 digits
sys.exit(f'not a gzip tarball: {e}')

kinds, links, order, plan, total = {}, {}, {}, [], 0
kinds, links, order, plan = {}, {}, {}, []
for i, m in enumerate(members):
p = parts(m.name)
if m.isdir():
kind = 'dir'
elif m.issym():
kind = 'symlink'
elif m.islnk():
kind = 'hardlink'
elif m.isreg():
kind = 'file'
else:
what = 'device' if m.ischr() or m.isblk() else 'FIFO' if m.isfifo() else 'special'
no(f'{m.name!r} is a {what} file')
kind = 'dir' if m.isdir() else 'symlink' if m.issym() else 'hardlink' if m.islnk() else 'file' # no other
if not p:
if kind != 'dir':
no(f'{m.name!r}, the top, is not a directory')
Expand All @@ -382,11 +430,7 @@ for i, m in enumerate(members):
if path in kinds and not (kind == 'dir' == kinds[path]):
no(f'{path!r} twice')
kinds[path], order[path] = kind, i
if kind == 'file':
total += m.size
if total > MAX_BYTES:
no(f'more than {MAX_BYTES} bytes of files')
elif kind == 'symlink':
if kind == 'symlink':
links[path] = m.linkname
plan.append((path, kind, m))

Expand Down
4 changes: 2 additions & 2 deletions .github/scripts/fork-fleet-refresh.service
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Fork only: the fleet host's runner template refresh (fork-fleet-launch.sh refresh). Install as
# /etc/systemd/system/t3code-fleet-refresh.service with its timer (.github/fork-fleet.md, host step 7); not as
# fork-fleet-*, the runner units' prefix. Safe to start by hand: a second refresh waits for the first (refresh.lock),
# then finds the template new already.
# fork-fleet-*, the runner units' prefix. Safe to start by hand: a second refresh waits for the first (a lock on the
# template's own folder), then finds the template new already.
#
# Root, sandboxed. Root because the template must be root's (each job hard-links it, and start refuses one anyone
# else owns or may write). The sandbox lets it write only the template's own folder (/opt/fork-fleet/template, where
Expand Down
Loading
Loading