Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion fork-kit/golive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,28 @@ start: the new launcher then idled and verify waited for minutes.
(`handover`: `launcher`, `timeout`, `start failed`, `before the start ...`), for `launcher` the measured delay
(`handoverMs`: at most this many ms from the claim to the let-go; with `handoverLate` when it reached 250 ms, so
a server that idled on a very busy host is named), and any marker renewal that failed (`renewFailed`).
- **Root cause, a run that ends with T3 stopped (#530).** A W1 (and a switch) may end, fail or be killed with T3
stopped on purpose (the home may be half-restored). A lock held by the run's own process would be gone with it,
and the next SSH launch would start a server on that home. So right after the stop the kit starts a holder, a
transient unit of its own (`t3-golive-hold-<hash of the home>`, running `t3-golive hold --home <home>`), that
takes the owner lock and keeps it until it is stopped; the run adopts it as its own lock. Every start the kit makes
goes through the same handover, so the launcher gets the home from the holder. It goes only to a launcher that
holds its serving lock, or once the service runs: a failed start, a start that returns without T3 running (`t3
update` exits non-zero or times out, a build that never comes up), a failure before the start, or a kill leaves
the holder holding. The next run of the same `back` takes the holder over (no free moment, never a second one:
there is one unit name per home); a resumed verify stops one that is still there once the service runs T3.
`status` and `check` name the holder (do not start T3 by hand while it holds: its server would idle); `settle`
stops it (the owner says the home is right). It is found by its live process in `/proc/locks`, never by a file,
so nothing stale outlives it. Where `/proc/locks` cannot be read (the kit could never find the holder again) or
no transient unit can be started, the run holds the lock itself, as before, and the stop step says so (`holder:
null`, `holderError`); `status` and `check` name an active holder unit they cannot see there. A copy needs none:
its guard starts T3 again as soon as the copy run dies.
- **Known gap: a reboot ends the holder.** The service stays disabled, so nothing starts T3 at boot, but a T3
desktop app connected over SSH that reconnects after the reboot finds the home free and may start its own server
on it while it may be half-restored. A holder that outlives a reboot would be an enabled unit on disk, state that
can outlive its reason (what finding the holder by its live process avoids), so the kit keeps the gap and says
it: `status` then fails `home holder` (none, but a run that stopped and disabled T3 has not ended ok) and says to
quit or disconnect that app until the same `back` ends ok.
- **Backstop, a foreign server.** An SSH client older than 0.1.10 reads neither, and a server started by hand
ignores both. When the runtime record names a live process that the boot service did not start and that itself
holds `userdata/server-owner.sqlite` (a stale record whose pid was reused holds nothing and is never named), for
Expand Down Expand Up @@ -402,7 +424,7 @@ empty state?
| `copy --here` from a T3 terminal | Refused: stopping T3 would kill it half-way |
| A desktop app's SSH launch while T3 is stopped (the gap) | It waits: the kit's marker is valid from before the stop until the new server is verified, and the kit holds the owner lock until the service's main process (the new launcher) holds its serving lock (an SSH launch's own probe of that lock, from another process, is not the launcher). No server of the old build takes the home (`tests/gap.test.py`, failing before the fix) |
| ... the marker cannot be written | Refused before T3 stops, and for W1 before it says it stops T3 (`the update marker ... cannot be written; T3 was not stopped`; W2/W3: `... not restarted`) |
| ... the run is killed with T3 stopped | Nothing renews the marker: it runs out within 60 s and launches go on as without it. The dead run holds no lock. A rerun is refused (`Unfinished`) as for any killed run |
| ... the run is killed with T3 stopped | Nothing renews the marker: it runs out within 60 s. A dead copy holds no lock (its guard starts T3 at once). A W1 or switch that stopped T3 leaves the kit's holder (above): the home stays owned and launches keep waiting until the same `back` (or the switch guard) hands it to T3, `settle` stops it, or the machine reboots. A rerun is refused (`Unfinished`) as for any killed run; a W1's own rerun follows it and takes the holder over |
| ... a newer marker (upstream's own update) is there at the end | Left alone: the kit removes only the bytes it wrote last (`marker: left: ...` in the verify step) |
| ... the target build takes no serving lock (an older build) | Older than `0.1.10-nightly.20261114.4695`: the kit lets go of the owner lock just before the start, as before; the marker still holds fresh launches |
| ... an SSH client older than 0.1.10, or a server started by hand, takes the home | Backstop: verify stops within about 10 s and names the server (pid, command) and the remedy. The copy reports T3 did not come back. The switch's W1 refuses too, so nothing is put back: the switch ends failed, says that the target waits for the home unverified, and ends the remedy with `t3-golive back --way w1 --lose-work-since <time>` (`tests/gap.test.py` SwitchBackstop); the kit never stops it |
Expand Down
22 changes: 19 additions & 3 deletions fork-kit/golive/lib/golive/back.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,12 +284,14 @@ def stop_restore_update():
'jobs stay paused (on purpose: the home may be half-restored); if it stops or fails, run the same back '
'again')
stopped.append(True)
with run.step('stop-service', pid=server and server['pid'], marker=handoff.marker_path(cfg.home)):
with run.step('stop-service', pid=server and server['pid'], marker=handoff.marker_path(cfg.home)) as facts:
adapter.disable() # first: from here T3 must not start at boot until the update enables it again
adapter.stop()
coldcopy._wait_stopped(adapter, server['pid'] if server else 0)
handoff.adopt(cfg.home, coldcopy.hold_owner_lock(os.path.join(cfg.home, 'userdata'),
coldcopy.STOP_WAIT)) # kept until the update starts T3
# kept until the update starts T3; by the holder, so a failed or killed W1 leaves the home owned (#530)
lock, held = coldcopy.hold_home(adapter, cfg.home, coldcopy.STOP_WAIT)
handoff.adopt(cfg.home, lock)
facts.update(held)
with run.step('restore', copy=copy_dir) as facts:
facts.update(coldcopy.restore(run, copy_dir, cfg.home, from_home=cfg.home, version=old,
floor_bytes=cfg.free_floor_mb * MB))
Expand Down Expand Up @@ -425,6 +427,10 @@ def _verify(run, cfg, adapter, version, old_pid, say):
(_w1_scan). A kill between the failure and its mark reads as cut off: the safe side."""
try:
with run.step('verify') as facts:
# a holder a killed W1 left after its update restarted T3 (#530): the running T3 needs the home
let_go = handoff.find_holder(cfg.home) and handoff.holder_to_service(cfg.home, adapter.service())
if let_go:
facts['holderLetGo'] = let_go
again = coldcopy._wait_started(cfg, adapter, version, old_pid)
if live.service_state(cfg.home)[0] != version or not adapter.service()['enabled']:
raise RuntimeError(f't3 {version} runs, but the service state does not name it or {osadapter.UNIT} '
Expand Down Expand Up @@ -773,6 +779,11 @@ def _rerun_hint(runs):
return f'W1 did not stop T3; run {again} again to finish it'


NO_HOLDER = ('none, but a run that stopped and disabled T3 has not ended ok (a reboot ends the kit\'s holder): the '
'home is free, and a T3 desktop app connected over SSH may start its own server on it while it may be '
'half-restored; quit or disconnect that app until the same `t3-golive back` ends ok')


def _status(cfg, adapter, add, notes, seen):
"""The checks; returns 'up', 'restarting', 'down' or 'unknown' (the live server goes in seen['server'])."""
try:
Expand All @@ -782,13 +793,18 @@ def _status(cfg, adapter, add, notes, seen):
return 'unknown'
add('service', svc['loaded'] and svc['active'] and svc['enabled'] and check_mod._same_dir(svc['home'], cfg.home),
f'{osadapter.UNIT} {svc.get("state")}, enabled: {svc["enabled"]}, serves {svc["home"]}')
holder = handoff.holder_seen(cfg.home, adapter.unit_active)
if holder is not None: # #530
add('home holder', False, holder)
try:
runs = journal.read(cfg.state_dir)[0]
except journal.Damaged as e: # a journal from a newer kit included
runs = []
add('journal', False, str(e))
else:
open_run = runs[-1] if runs and journal.needs_settling(runs[-1]) else None
if holder is None and open_run is not None and not svc['active'] and not svc['enabled']:
add('home holder', False, NO_HOLDER) # a reboot ended the holder a W1 or switch left (#530)
if open_run is None:
add('journal', True, 'no golive run was left half-way')
elif open_run['command'] in FOLLOWS:
Expand Down
16 changes: 14 additions & 2 deletions fork-kit/golive/lib/golive/check.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import argparse, hashlib, json, os, sys

import machine_file
from golive import config as config_mod, journal, live, osadapter
from golive import config as config_mod, handoff, journal, live, osadapter

MB = 1024 * 1024

Expand Down Expand Up @@ -125,6 +125,10 @@ def all_ok():
else:
add('boot service', True, f'{osadapter.UNIT} serves this home')

holder = handoff.holder_seen(cfg.home, adapter.unit_active)
if holder is not None: # #530: a W1 or switch left the home held on purpose
add('home holder', False, holder)

server = None
try:
server = live.live_server(cfg.home, svc['mainPid'])
Expand Down Expand Up @@ -308,11 +312,15 @@ def _print(code, result, out):


def _settle(cfg, reason, out, err):
stopped = [] # the owner says the home is right: a holder a W1 or switch left (#530) has nothing to keep
try:
what, value = journal.settle(cfg.state_dir, reason)
what, value = journal.settle(cfg.state_dir, reason, then=lambda: stopped.append(handoff.stop_holder(cfg.home)))
except (journal.Busy, journal.Newer, ValueError) as e:
print(f't3-golive: {e}', file=err)
return 1
if stopped and stopped[0] is not None:
print(f'stopped the kit\'s holder {handoff.holder_unit(cfg.home)} (pid {stopped[0]}): the home is not kept '
'owned any more, and T3 is left as it is', file=out)
if what == 'settled':
print(f'settled: {journal.describe(value)}; reason: {reason.strip()}', file=out)
elif what == 'moved':
Expand Down Expand Up @@ -363,6 +371,8 @@ def main(argv, out=sys.stdout, err=sys.stderr):
p.add_argument('--json', action='store_true', help='print the result as JSON')
p = sub.add_parser('guard', help=argparse.SUPPRESS)
p.add_argument('--run', required=True)
p = sub.add_parser('hold', help=argparse.SUPPRESS) # the holder's unit (handoff.serve_holder)
p.add_argument('--home', required=True)
p = sub.add_parser('switch', help='move the live T3 to the target build, verify it, go back on its own if not',
description='Needs a passed, fresh rehearsal of this staged build (its archive sha256) from '
'the live build. Keeps the old runtime outside the home, then at a quiet point: '
Expand All @@ -386,6 +396,8 @@ def main(argv, out=sys.stdout, err=sys.stderr):
help="after a reboot, to let it run W1: the switch copy's time, as `t3-golive back` prints it")
p.add_argument('--wait', action='store_true', help=argparse.SUPPRESS)
args = parser.parse_args(argv)
if args.command == 'hold': # it only holds the lock of the home it was started for: no machine file needed
return handoff.serve_holder(args.home)
try:
cfg = config_mod.load()
except machine_file.MachineFileError as e:
Expand Down
Loading
Loading