Repository navigation
golive: keep the home owned while T3 is stopped (update marker + lock handover) - #531
scuttletoby wants to merge 3 commits into
Conversation
… handover) A desktop app's SSH launch started its own t3 serve of the old build in the gap the kit left between stopping and starting t3code.service (#513, and the go-live switch on 11-19): the kit wrote no update marker, and let go of the owner lock before the start, so a launch saw the home free for a moment. Root cause (lib/golive/handoff.py): before every stop (copy, switch, W1 and its restore, W2/W3, the guard's starts) the kit writes upstream's desktop-update-in-progress marker exactly as DesktopUpdateMarker.publish does (path, {startedAt, expiresAt} ms, 60 s, .tmp + rename), renews it every 15 s, and removes it (only its own bytes) once the new server is verified. A run that dies leaves a marker that runs out by itself. The owner lock taken after the stop is kept until the new launcher holds its serving lock (builds from 0.1.10-nightly.20261114.4695 on), then let go; older builds get the old order. Backstop: a server the boot service did not start (an SSH client older than 0.1.10, or one started by hand) makes verify stop within about 10 s with the remedy, and W1/W2/W3 refuse before changing anything. The kit never kills it. tests/gap.test.py models upstream's launch script and launcher and reproduces the race (fails on the old kit, passes now), runs upstream's own node reader on the kit's marker, and contract.test.py pins the upstream names. CI's sparse checkouts get the pinned files; README gets the gap section and failure rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review (independent, Opus)Head What I checked
1. SHOULD-FIX: the handover can mistake an SSH launch's own lock probe for the new launcher
What goes wrong. Failure scenario. A desktop app is waiting in the launch script's
Measured. I ran a node prober using upstream's exact probe every 50 ms against Fix. A real launcher holds the lock from 2. SHOULD-FIX: the refusal's remedy is wrong after a W1 that stopped T3
Failure scenario.
Fix. When the journal shows a stop-service with no finished back, the remedy should be: kill it, then run the same Same text, also racy (NIT-level, any state). The remedy is "kill, then restart", but the kit has closed its gap by then. A connected desktop relaunches within about 1 s of the kill. It finds no marker and a free home, and starts the next hand server before 3. SHOULD-FIX: README says the switch goes back with W1 on a foreign server, but W1 refuses
Failure scenario.
No test covers switch plus foreign server; Fix. Correct the README row. Have the switch's error say where T3 stands, and that the way back still has to be run after the kill. Add a SwitchBackstop test. 4. NIT:
|
Review (independent, Opus)2 blocking findings: one race that can still free the home in the middle of 1. BLOCKING: an SSH launch's lock probe looks like a serving launcher, so the kit lets go of the owner lock too early
Measured with the kit's own
Failure scenario (switch):
Fix:
2. BLOCKING: the backstop's remedy says to restart the service on a half-restored home
Failure scenario:
Fix: when the journal shows a W1 3. SHOULD-FIX:
|
- handover: count the launcher as serving only when service-serving.lock is held on every poll for >= 100 ms and >= 3 polls; an SSH launch's own home-owned probe no longer frees the home mid `t3 update` - backstop: name a foreign server only while that pid itself holds userdata/server-owner.sqlite (/proc/locks); switch verify passes old_pid - remedy: always quit the desktop app first, then kill; behind an unfinished way-back stop (or a disabled service) run the same back again and never start the service; the switch says its W1 refused too and where T3 stands - W1/W2/W3: a marker that cannot be written is a clear refusal before "stopping T3" - verify facts: handover (launcher/timeout/...) and renewFailed - tests: probe-loop handover (unit, copy, switch), stale pid, W1 rerun remedy, switch backstop, W1 marker refusal, renewal failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
t3-goliveleft a gap between stopping and startingt3code.service. A T3 desktop app connected over SSH runs upstream's launch script as soon as its server goes away. In that gap the script found the home free and started its ownt3 serveof the old build. The new launcher then idled and verify stalled. This is what happened in #513 and in the go-live switch on 11-19.The fix closes the gap with upstream's own mechanism, the one its self-update uses. A small backstop covers the clients that mechanism cannot reach.
Root cause vs backstop
Root cause, part 1: the update marker (
lib/golive/handoff.py,Gap.open/served)runtime/desktop-update-in-progressmarker exactly asDesktopUpdateMarker.publish(used bycloud/selfUpdate.ts) does:{"startedAt":ms,"expiresAt":ms}, withexpiresAt = startedAt + 60_000;<marker>.tmp, then renamed into place.markerfact saysremoved,goneorleft: ....Root cause, part 2: the owner lock handover (
Gap.adopt/starting)runtime/service-serving.lock, so the home is owned (__ssh-helper home-owned) the whole time.t3 update. The launcher holds it continuously from#claimServingon, and its server waits 250 ms for the owner lock, so the handover still comes well in time.0.1.10-nightly.20261114.4695take no serving lock (checked against upstream tags). For them the kit keeps the old order: it lets go just before the start, because their server would otherwise meet the kit's lock and idle for a minute.Backstop: foreign-server detection, refuse and report only (
foreign,ForeignWatch)userdata/server-owner.sqlite(read from/proc/locks), for more than 10 s. A stale record whose pid was reused holds nothing and is never named. Switch verify passes the stopped server's pid asold_pid.kill <pid>. Then it depends on where T3 stands:systemctl --user restart t3code.service, thent3-golive status(or the refused back again).stop-service, or the service is disabled): run the samet3-golive backagain, and never start the service (the home may be half-restored).t3-golive back --way w1 --lose-work-since <the copy's time>.Every stop and restart site
copy(coldcopy_run_copy)stop-service; lock adopted_restarthands over viastarting; verify removes the markerrestoreswitch(_switch_on)stopping; lock adopted (the earlylock.close()is gone)_update(t3 update --yes+ fallback start) viastarting; verify removes the marker_put_backback-start)startingwith the old version; removes the markerbackW1stop-service; lock adopted after the stop_updateviastarting;_verifyremoves the markerbackW2/W3plan; marker before the updateMarker format checked against the build being stopped
__ssh-helper update-remaining), not by the stopped server.tests/gap.test.pyruns upstream's own node reader, taken fromtunnel.ts, on the kit's marker: a fresh marker means the launch waits, an expired one means it does not.tests/contract.test.pynow pins the file name, TTL, schema,.tmp+rename,SERVICE_SERVING_LOCK_FILE,#claimServing, and the two__ssh-helpercalls in the launch script. The CI sparse checkouts include those files.Tests
tests/gap.test.py(26 tests, ~30 s after this round). It models upstream's launch script (a fresh launch, a launch that waited through the stop, and a pre-0.1.10 client) and the service launcher, then runs the kit's fakes through the race. It is added to.github/fork-ci-tests.txt.common/tests/machine_file.test.py.switch.test.pynow expects W1's verify to reportmarker: removed.fork-ci-tests.test.cjs,fork-changes-list.test.cjs,fork-tests-drift.test.cjsand its check,fork-contracts.test.py,fork-owned-reads.test.py,fork-owned-paths.test.py.Review fixes (round 1)
Both reviews, all items:
f88941c)Gap.startinglets go only once the serving lock was held on every poll for >= 100 ms and >= 3 polls (HANDOVER_HOLD,HANDOVER_HOLD_POLLS)Marker.test_a_probe_loop_on_the_serving_lock_is_not_the_launcher(probe loop duringstarting(): owner lock held until a real launcher takes the serving lock);CopyRace.test_ssh_launches_probing_the_serving_lock_during_the_start_never_free_the_home(old head: a hand server takes the home);SwitchRace.test_ssh_launches_probing_through_the_update_never_free_the_homeback.refuse_foreign: behind an unfinished way-back stop (journal) or a disabled service, the remedy is quit the desktop app,kill <pid>, the samet3-golive backagain, never start the serviceW1Gap.test_a_w1_rerun_behind_its_own_stop_never_says_to_start_t3;Foreign.test_the_remedy_follows_where_t3_standsforeign()trusted a stale pidserver-owner.sqlite(handoff.holds_lock,/proc/locks; falls back to "someone holds it" when/proc/lockscannot be read); switch verify passesold_pidForeign.test_a_stale_record_naming_a_reused_pid_is_not_foreign,Foreign.test_a_hand_server_holding_the_home_is_foreign_and_the_stopped_server_is_notswitch.HomeHeld: the switch says W1 was refused too, where T3 stands, and the remedy ending int3-golive back --way w1 --lose-work-since <time>; the switch guard says the same; README rows fixedSwitchBackstop.test_verify_stops_on_it_w1_refuses_and_the_switch_says_where_t3_stands(also runs the remedy: kill, then W1 by hand goes back)kill, and a connected desktop starts the next server within a secondOSError; W2/W3 rawOSErrorRefused: the update marker ... cannot be written; T3 was not stopped(W2/W3:... not restarted)W1Gap.test_a_w1_whose_marker_cannot_be_written_refuses_before_saying_it_stops_t3handover(launcher,timeout,start failed,before the start ...) andrenewFailed(count and last error) next tomarkerMarker.test_a_failed_renewal_is_journalled;CopyRacecheckshandover: launcherserved()docstring names the read-to-unlink window (microseconds; upstream'sreleaseUpdateMarkerhas the same)contract.test.py: the pin of upstream's 250 ms owner-lock wait now also nameshandoff.HANDOVER_HOLD.switch.test.py: W1's verify facts now includehandover.Round 1 test runs (this host):
f88941clib + thisgap.test.py): the probe-loop copy test fails on behaviour (a hand server takes the home:pid N runs .../python3, not a runtime of this home), and so do the probe-loop unit test, the stale-pid test, the W1 rerun remedy test and the switch backstop test. The rest fail on the new API.common/tests/machine_fileexit 0. gap: 24 of 26 pass; the 2SwitchRacetests (and 1 backStatustest, and the HTTP-checking switch tests) fail only on "nothing answers HTTP": this host blocks loopback TCP (a bare socket connect to 127.0.0.1 times out). The old head fails exactly the same tests the same way here, apart from the oneswitch.test.pyexpectation updated above. CI runs them with loopback.State question (runs again / stops half-way / old or empty state)
For this round's changes:
starting()call; a run killed while it waits for the serving lock to hold drops the owner lock with the process, and the marker runs out within 60 s. A launcher that holds the lock only briefly and then lets it go is not counted; after 20 s the kit lets go anyway (handover: timeout)./proc/locksit falls back to "someone holds the owner lock" (never weaker than before).failedwith nothing restored; the home needs settling.t3-golive back --way w1 --lose-work-since <time>follows it (tested). Killed at any point, the switch guard says the same when its W1 is refused.plan,runtime) are done, so the run endsrefused(orfailedwhen it follows an unfinished run, as any refusal there). Running again retries the write.handover,renewFailed) are only written, never read; old journals read as before.Earlier rounds:
back) reuses the outer gap and does not close it.Unfinished) as before.runtime/(its service state lives there). Ifruntime/is missing or cannot be written, the marker write fails and the run refuses before the stop. The kit does not create the folder.Not covered here (follow-ups)
backorsettletakes over.Fork features on after go-live (task 4)
checkrefuses a switch unless every fork feature is off, and so do the switch'sbeforestep,verifyandrehearseR3. The design (section 4.7) turnsagentSlots,waitForandusagePacingGateon after day 0. From then on:checkrefuses every latert3-golive switch;Later updates need one of two things:
featuresOn. The rehearsal must also force features off in the copy only, so thatwaitForcannot send messages from the rehearsal.t3 update/automaticUpdatesinstead of the kit.This is opened as #529. This PR does not change it.
General vs project
🤖 Generated with Claude Code