Skip to content

golive: keep the home owned while T3 is stopped (update marker + lock handover) - #531

Open
scuttletoby wants to merge 3 commits into
mainfrom
golive-gap-handoff
Open

scuttletoby wants to merge 3 commits into
mainfrom
golive-gap-handoff

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

t3-golive left a gap between stopping and starting t3code.service. A T3 desktop app connected over SSH runs upstream's launch script as soon as its server goes away. In that gap the script found the home free and started its own t3 serve of the old build. The new launcher then idled and verify stalled. This is what happened in #513 and in the go-live switch on 11-19.

The fix closes the gap with upstream's own mechanism, the one its self-update uses. A small backstop covers the clients that mechanism cannot reach.

Root cause vs backstop

Root cause, part 1: the update marker (lib/golive/handoff.py, Gap.open / served)

  • Before every stop, the kit writes upstream's runtime/desktop-update-in-progress marker exactly as DesktopUpdateMarker.publish (used by cloud/selfUpdate.ts) does:
    • same path;
    • compact JSON {"startedAt":ms,"expiresAt":ms}, with expiresAt = startedAt + 60_000;
    • written to <marker>.tmp, then renamed into place.
  • A thread renews it every 15 s.
  • It is removed only after the new server is verified serving. The verify step's marker fact says removed, gone or left: ....
  • It is removed only if it still holds the kit's own last bytes, so a newer marker (upstream's own update) is left alone.
  • A run that dies stops renewing, and its marker runs out by itself within 60 s. No desktop app waits forever.
  • If the marker cannot be written, the run refuses before T3 stops.

Root cause, part 2: the owner lock handover (Gap.adopt / starting)

  • The old kit closed its owner lock before starting the service. A launch that had waited through the copy then saw the home free for a moment before the new launcher took it.
  • Now the kit keeps the owner lock until the new launcher holds runtime/service-serving.lock, so the home is owned (__ssh-helper home-owned) the whole time.
  • "Holds" means held on every poll for at least 100 ms and 3 polls in a row. An SSH launch waiting for the home probes the same lock for a moment once a second; one poll that met such a probe used to free the home in the middle of t3 update. The launcher holds it continuously from #claimServing on, and its server waits 250 ms for the owner lock, so the handover still comes well in time.
  • If no launcher takes the lock within 20 s of the start returning, the kit lets go anyway.
  • Builds older than 0.1.10-nightly.20261114.4695 take no serving lock (checked against upstream tags). For them the kit keeps the old order: it lets go just before the start, because their server would otherwise meet the kit's lock and idle for a minute.

Backstop: foreign-server detection, refuse and report only (foreign, ForeignWatch)

  • It covers what the marker and lock cannot: an SSH client older than 0.1.10 (it reads no marker and probes no lock), or a server started by hand.
  • It fires when the runtime record names a live process that is not a child of the service's main pid and that process itself holds userdata/server-owner.sqlite (read from /proc/locks), for more than 10 s. A stale record whose pid was reused holds nothing and is never named. Switch verify passes the stopped server's pid as old_pid.
  • Then verify stops waiting at once; W1, W2 and W3 refuse before changing anything.
  • The remedy always starts with: quit or disconnect the T3 desktop app connected over SSH (an old one starts the next server within a second of the kill), then kill <pid>. Then it depends on where T3 stands:
    • T3 enabled, no way back holding it stopped: systemctl --user restart t3code.service, then t3-golive status (or the refused back again).
    • A way back stopped T3 and has not ended ok (the journal shows its stop-service, or the service is disabled): run the same t3-golive back again, and never start the service (the home may be half-restored).
    • Switch verify: the switch's W1 refuses too, so nothing is put back. The switch says that the target is installed, its service waits for the home, never verified, and the copy is kept, and the remedy ends with t3-golive back --way w1 --lose-work-since <the copy's time>.
  • The kit never kills a server it did not start.

Every stop and restart site

Where Before the stop / start After
copy (coldcopy _run_copy) marker before stop-service; lock adopted _restart hands over via starting; verify removes the marker
copy guard start marker left to run out (the guard does not wait)
restore reuses the run's held lock, or takes one the lock goes back to the run's gap
switch (_switch_on) marker before stopping; lock adopted (the early lock.close() is gone) _update (t3 update --yes + fallback start) via starting; verify removes the marker
switch way back (_put_back back-start) (gap still open) starting with the old version; removes the marker
switch guard: journal unreadable / killed run marker before the start or put-back closed; marker runs out
back W1 foreign refusal; marker before stop-service; lock adopted after the stop _update via starting; _verify removes the marker
back W2/W3 foreign refusal before plan; marker before the update same

Marker format checked against the build being stopped

  • The marker is read by the SSH client's runtime (__ssh-helper update-remaining), not by the stopped server.
  • The reader and writer are byte-identical in upstream v0.1.10 (where they first appeared) and in the 0.1.11 nightly that is live now.
  • 0.1.9 has neither. A 0.1.9 client is covered only by the backstop.
  • tests/gap.test.py runs upstream's own node reader, taken from tunnel.ts, on the kit's marker: a fresh marker means the launch waits, an expired one means it does not.
  • tests/contract.test.py now pins the file name, TTL, schema, .tmp+rename, SERVICE_SERVING_LOCK_FILE, #claimServing, and the two __ssh-helper calls in the launch script. The CI sparse checkouts include those files.

Tests

  • New tests/gap.test.py (26 tests, ~30 s after this round). It models upstream's launch script (a fresh launch, a launch that waited through the stop, and a pre-0.1.10 client) and the service launcher, then runs the kit's fakes through the race. It is added to .github/fork-ci-tests.txt.
    • Fails before the fix: the old kit with this test file gives all 9 race tests (CopyRace, SwitchRace, W1Gap) FAILED/ERROR.
    • Passes after the fix.
    • It also covers: a killed copy leaves a valid marker that runs out and holds no lock; an unwritable marker refuses before the stop; a newer marker is left alone; the old-build order; the backstop message; W1 refusing with a foreign server and leaving the snapshot unchanged.
  • Kit suites, all exit 0: journal, check, contract, gap, copy (48 s), back (89 s), rehearse (55 s), switch (88 s), and common/tests/machine_file.test.py. switch.test.py now expects W1's verify to report marker: removed.
  • CI script tests, all exit 0: fork-ci-tests.test.cjs, fork-changes-list.test.cjs, fork-tests-drift.test.cjs and its check, fork-contracts.test.py, fork-owned-reads.test.py, fork-owned-paths.test.py.

Review fixes (round 1)

Both reviews, all items:

Item Fix Test (fails on f88941c)
BLOCKING: handover phantom (an SSH probe of the serving lock taken for the launcher) Gap.starting lets go only once the serving lock was held on every poll for >= 100 ms and >= 3 polls (HANDOVER_HOLD, HANDOVER_HOLD_POLLS) Marker.test_a_probe_loop_on_the_serving_lock_is_not_the_launcher (probe loop during starting(): owner lock held until a real launcher takes the serving lock); CopyRace.test_ssh_launches_probing_the_serving_lock_during_the_start_never_free_the_home (old head: a hand server takes the home); SwitchRace.test_ssh_launches_probing_through_the_update_never_free_the_home
BLOCKING: refusal remedy said "restart the service" on a half-restored home back.refuse_foreign: behind an unfinished way-back stop (journal) or a disabled service, the remedy is quit the desktop app, kill <pid>, the same t3-golive back again, never start the service W1Gap.test_a_w1_rerun_behind_its_own_stop_never_says_to_start_t3; Foreign.test_the_remedy_follows_where_t3_stands
foreign() trusted a stale pid Named only while that pid itself holds server-owner.sqlite (handoff.holds_lock, /proc/locks; falls back to "someone holds it" when /proc/locks cannot be read); switch verify passes old_pid Foreign.test_a_stale_record_naming_a_reused_pid_is_not_foreign, Foreign.test_a_hand_server_holding_the_home_is_foreign_and_the_stopped_server_is_not
README / message: the switch "goes back with W1" on a foreign server, but W1 refuses switch.HomeHeld: the switch says W1 was refused too, where T3 stands, and the remedy ending in t3-golive back --way w1 --lose-work-since <time>; the switch guard says the same; README rows fixed SwitchBackstop.test_verify_stops_on_it_w1_refuses_and_the_switch_says_where_t3_stands (also runs the remedy: kill, then W1 by hand goes back)
Nit: the remedy began with kill, and a connected desktop starts the next server within a second Every remedy begins with quitting or disconnecting the desktop app (or updating it to 0.1.10+) the tests above check the order
Nit: W1 announced "stopping T3" before a marker write that may fail, with a raw OSError; W2/W3 raw OSError The marker is written first; failure is Refused: the update marker ... cannot be written; T3 was not stopped (W2/W3: ... not restarted) W1Gap.test_a_w1_whose_marker_cannot_be_written_refuses_before_saying_it_stops_t3
Nit: renewal failures and handover-vs-timeout never journalled The verify step's facts now carry handover (launcher, timeout, start failed, before the start ...) and renewFailed (count and last error) next to marker Marker.test_a_failed_renewal_is_journalled; CopyRace checks handover: launcher
Nit: "a newer marker is left alone" overclaimed The served() docstring names the read-to-unlink window (microseconds; upstream's releaseUpdateMarker has the same) -
Nit: diagnosis said "most likely an old SSH client" only Says "or someone started it by hand"; with the handover fix the probe phantom no longer produces foreign servers -

contract.test.py: the pin of upstream's 250 ms owner-lock wait now also names handoff.HANDOVER_HOLD. switch.test.py: W1's verify facts now include handover.

Round 1 test runs (this host):

  • The new tests on the old head (f88941c lib + this gap.test.py): the probe-loop copy test fails on behaviour (a hand server takes the home: pid N runs .../python3, not a runtime of this home), and so do the probe-loop unit test, the stale-pid test, the W1 rerun remedy test and the switch backstop test. The rest fail on the new API.
  • Head: journal, check, contract, copy, rehearse, common/tests/machine_file exit 0. gap: 24 of 26 pass; the 2 SwitchRace tests (and 1 back Status test, and the HTTP-checking switch tests) fail only on "nothing answers HTTP": this host blocks loopback TCP (a bare socket connect to 127.0.0.1 times out). The old head fails exactly the same tests the same way here, apart from the one switch.test.py expectation updated above. CI runs them with loopback.

State question (runs again / stops half-way / old or empty state)

For this round's changes:

  • Handover: per starting() call; a run killed while it waits for the serving lock to hold drops the owner lock with the process, and the marker runs out within 60 s. A launcher that holds the lock only briefly and then lets it go is not counted; after 20 s the kit lets go anyway (handover: timeout).
  • Foreign check: read only. A stale record, an empty home (no record, no lock file), or a server the service started are never named. Without /proc/locks it falls back to "someone holds the owner lock" (never weaker than before).
  • Refusal remedy: read from the journal each time; an unreadable journal takes the safe side (never start the service). Running the same back again after the kill continues the stopped back as before.
  • Switch backstop: the switch ends failed with nothing restored; the home needs settling. t3-golive back --way w1 --lose-work-since <time> follows it (tested). Killed at any point, the switch guard says the same when its W1 is refused.
  • Marker write refused in W1/W2/W3: journalled steps before it (plan, runtime) are done, so the run ends refused (or failed when it follows an unfinished run, as any refusal there). Running again retries the write.
  • Old journals: the new verify facts (handover, renewFailed) are only written, never read; old journals read as before.

Earlier rounds:

  • Runs again: the gap is per home and per run. A nested run in the same process (the guard firing a back) reuses the outer gap and does not close it.
  • Stops half-way: the marker runs out within 60 s, and the dead process's lock is gone. A rerun is refused (Unfinished) as before.
  • Old state: a leftover marker from an earlier run is simply overwritten. A newer one is never removed by the kit. An old target build gets the old lock order. An old SSH client gets the backstop.
  • Empty state: a boot-service home always has runtime/ (its service state lives there). If runtime/ is missing or cannot be written, the marker write fails and the run refuses before the stop. The kit does not create the folder.

Not covered here (follow-ups)

  • t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530: a W1 (or restore) that fails leaves T3 stopped and disabled on purpose. Once the kit exits, the marker runs out and the lock is gone, so an SSH launch can start a server on the half-restored home. The proposal there is a small holder unit that keeps the owner lock until back or settle takes over.
  • SSH clients older than 0.1.10: only the backstop covers them. The kit cannot make them wait.
  • The milliseconds between the old server exiting and the kit taking the owner lock: only the marker covers them. That is enough, because every fresh launch reads it first.

Fork features on after go-live (task 4)

check refuses a switch unless every fork feature is off, and so do the switch's before step, verify and rehearse R3. The design (section 4.7) turns agentSlots, waitFor and usagePacingGate on after day 0. From then on:

  • check refuses every later t3-golive switch;
  • a verify with features on can never pass and would go back with W1.

Later updates need one of two things:

  1. Change the rule from "all off" to "unchanged": compare against the live fork build's featuresOn. The rehearsal must also force features off in the copy only, so that waitFor cannot send messages from the rehearsal.
  2. Or decide that fork-to-fork updates go through upstream t3 update / automaticUpdates instead of the kit.

This is opened as #529. This PR does not change it.

General vs project

  • General (public fork, any OS with a systemd user unit): everything here. It uses only upstream's names (the marker, the serving lock, the owner lock, the SSH helper commands), with no path, port, name or id of any machine. The version threshold is an upstream build fact.
  • Project/machine: nothing. No machine config changes are needed. The marker lives in the T3 home the config already names.

🤖 Generated with Claude Code

… handover)

A desktop app's SSH launch started its own t3 serve of the old build in the
gap the kit left between stopping and starting t3code.service (#513, and the
go-live switch on 11-19): the kit wrote no update marker, and let go of the
owner lock before the start, so a launch saw the home free for a moment.

Root cause (lib/golive/handoff.py): before every stop (copy, switch, W1 and
its restore, W2/W3, the guard's starts) the kit writes upstream's
desktop-update-in-progress marker exactly as DesktopUpdateMarker.publish does
(path, {startedAt, expiresAt} ms, 60 s, .tmp + rename), renews it every 15 s,
and removes it (only its own bytes) once the new server is verified. A run that
dies leaves a marker that runs out by itself. The owner lock taken after the
stop is kept until the new launcher holds its serving lock (builds from
0.1.10-nightly.20261114.4695 on), then let go; older builds get the old order.

Backstop: a server the boot service did not start (an SSH client older than
0.1.10, or one started by hand) makes verify stop within about 10 s with the
remedy, and W1/W2/W3 refuse before changing anything. The kit never kills it.

tests/gap.test.py models upstream's launch script and launcher and reproduces
the race (fails on the old kit, passes now), runs upstream's own node reader on
the kit's marker, and contract.test.py pins the upstream names. CI's sparse
checkouts get the pinned files; README gets the gap section and failure rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (independent, Opus)

Head f88941c. No blocking findings. 3 SHOULD-FIX, 3 NIT.

What I checked

  • All kit suites pass on the PR head: journal, check, contract, gap (16), copy, back, rehearse, switch, and common/tests/machine_file.test.py.
  • The tests really fail on the old kit. I ran gap.test.py against the base lib/ for CopyRace, SwitchRace and W1Gap: 2 failures and 7 errors out of 9. These are not import errors. Hand servers really take the home (T3 did not come back ... pid N runs .../python3, not a runtime of this home), and W1 is not refused.
  • The version threshold is right. Upstream v0.1.10-nightly.20261114.4682 has no #claimServing, while ...4695 and v0.1.10 have it. compare_versions puts 0.1.10 after the 0.1.10-nightly.* builds, and puts *.fork.N builds after their base build.
  • The kit uses the right launcher version. t3 update installs the unit from the target's binary (cli/update.ts target.install), so starting(..., target_version) names the launcher that will really run.
  • The handover has room. The server's own owner-lock probe waits 250 ms (serverOwnership.ts:192), far more than the 20 ms handover poll.
  • Marker format and lock probe match upstream. The marker matches DesktopUpdateMarker.publish / remainingWait byte for byte. _lock_held matches serviceLauncherIsRunning.
  • The launch-script model matches tunnel.ts. An SSH launch that waited and finds the home still owned gives up after the ready timeout ("no second server was started"), so a long stop with the lock held is safe.

1. SHOULD-FIX: the handover can mistake an SSH launch's own lock probe for the new launcher

fork-kit/golive/lib/golive/handoff.py:173-178 (watch in Gap.starting)

What goes wrong. __ssh-helper home-owned (and the launch script's node fallback) probes service-serving.lock with BEGIN EXCLUSIVE and then closes it. While it holds that lock for a moment, the kit's 20 ms probe sees "held". One such hit is enough for the kit to decide the launcher serves and to let go of the owner lock. No launcher may be running yet: during t3 update this window lasts the whole install, and during any start it lasts up to 20 s after fn returns.

Failure scenario. A desktop app is waiting in the launch script's home_owned loop. It probes once a second, and the copy is long enough that its marker wait is used up. Then:

  1. One of its probes coincides with the kit's poll during _update.
  2. The kit closes the owner lock.
  3. The next probe, 1 s later, sees neither lock held.
  4. The desktop app starts t3 serve of the old build. This is the Owner question: one short T3 restart so the boot service owns the server? #513 incident again, now caught only by the backstop.

Measured. I ran a node prober using upstream's exact probe every 50 ms against handoff.launcher_serving polled at HANDOVER_POLL for 30 s: 1 false "serving" in about 600 probes (about 0.2% per probe). At one probe per second through a t3 update plus up to 20 s, that is a few percent per waiting desktop per switch.

Fix. A real launcher holds the lock from #claimServing until its server is up, which takes far longer than a probe. So require the lock to be held on several polls in a row covering at least about 100 ms before letting go. Add a gap test with a prober that flickers the serving lock.

2. SHOULD-FIX: the refusal's remedy is wrong after a W1 that stopped T3

fork-kit/golive/lib/golive/back.py:269 (W1 rerun) and :643 (W2/W3 behind a stuck W1). The text is at handoff.py:321-325.

Failure scenario.

  1. A W1 fails after its stop. T3 is disabled and stopped, and the home is half-restored, on purpose.
  2. Once the kit exits, an SSH launch takes the home. This is the t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530 window, and it is the most likely way a foreign server shows up at all.
  3. The owner reruns t3-golive back --way w1. It refuses with "kill , then at once systemctl --user restart t3code.service, then t3-golive status".
  4. restart starts the disabled unit on the half-restored home. The kit's own rule for this state is the opposite: W1 says "T3 stays stopped and is not started at boot (on purpose: the home may be half-restored) until the same back ends ok".

Fix. When the journal shows a stop-service with no finished back, the remedy should be: kill it, then run the same t3-golive back again, and do not start the service.

Same text, also racy (NIT-level, any state). The remedy is "kill, then restart", but the kit has closed its gap by then. A connected desktop relaunches within about 1 s of the kill. It finds no marker and a free home, and starts the next hand server before restart. For a client older than 0.1.10, nothing but quitting it helps. The remedy should begin with "quit or disconnect the T3 desktop app connected over SSH (or update it to 0.1.10+)".

3. SHOULD-FIX: README says the switch goes back with W1 on a foreign server, but W1 refuses

fork-kit/golive/README.md:383; code path switch.py:290, then _put_back at switch.py:421, then back.py:269

Failure scenario.

  1. The switch's verify stops on the foreign server.
  2. _put_back runs W1, and W1's own foreign check raises Refused.
  3. The switch ends with "verify failed: ...; then putting T3 back failed: Refused: ... nothing was changed". The target's launcher stays active, idle behind the hand server.
  4. If the owner follows the remedy, the unverified target is what goes live, not the old build. Nothing tells them that the switch still needs settling (W1 by hand, or a settle).

No test covers switch plus foreign server; CopyBackstop covers only copy.

Fix. Correct the README row. Have the switch's error say where T3 stands, and that the way back still has to be run after the kill. Add a SwitchBackstop test.

4. NIT: foreign() trusts the pid in the record without checking what runs there

fork-kit/golive/lib/golive/handoff.py:312-319

Failure scenario. A server killed with SIGKILL leaves its server-runtime.json, and the pid is later reused by an unrelated process. The W1/W2/W3 pre-checks then refuse (with no grace period) and tell the owner to kill that unrelated pid. live.live_server already checks that the exe is a runtime of this home.

Fix. Require the exe to be a t3 runtime (any runtime/versions/*/t3, or a t3/node binary) before calling the process foreign.

5. NIT: a W1 whose marker cannot be written announces a stop that never happens

fork-kit/golive/lib/golive/back.py:281-284

What goes wrong. W1 prints "stopping T3 for W1: ... T3 stays stopped ..." and only then calls handoff.open_gap. If the write fails, a bare OSError comes out, with no "T3 was not stopped" note and no Refused, while T3 is in fact still running. The README promises "Refused before T3 stops (the update marker ... cannot be written)", which is true for copy and switch (they go through coldcopy.open_gap) but not for W1, W2 or W3 (back.py:382 is the same).

Fix. Open the gap before the say, and map the OSError to Refused with the same text copy and switch use.

6. NIT: what the gap learns never reaches the journal

fork-kit/golive/lib/golive/handoff.py:125-127, 147-149, 190

What goes wrong. Gap.error (a failed renewal) and handed_to_launcher (a handover vs a 20 s timeout) are written but never read or journalled. If renewal keeps failing, the marker runs out silently mid-stop. The owner lock still covers waiting launches, but the journal cannot tell this apart from a good run, and it cannot show whether the handover or the timeout let go.

Fix. Add both to the verify step's facts next to marker.

State question, the rest of it.

  • Killed between steps: the marker runs out within 60 s and the OS drops the lock.
  • Stale or foreign marker: a leftover marker is overwritten, and a newer one is left in place (byte compare; the read-to-unlink window is negligible).
  • Old builds: they get the old order.
  • Missing runtime/: the run refuses before the stop.
  • Nested gap: a W1 inside the switch's _put_back reuses the switch's gap and ends it on its verify. The guards run as separate processes, so their handoff.close cannot end a run's gap.

I found no path that deletes a marker the kit did not write last, and none that stops or kills a process it did not start.

@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (independent, Opus)

2 blocking findings: one race that can still free the home in the middle of t3 update, and one refusal message whose remedy is wrong in the one state it matters most. The rest is SHOULD-FIX or NIT. The overall design is sound: upstream's marker plus keeping the owner lock until the launcher holds its serving lock.

1. BLOCKING: an SSH launch's lock probe looks like a serving launcher, so the kit lets go of the owner lock too early

fork-kit/golive/lib/golive/handoff.py:88-107 (_lock_held / launcher_serving), used by handoff.py:172-178 (Gap.starting.watch)

  • launcher_serving() treats any momentary holder of runtime/service-serving.lock as "the launcher serves".
  • Upstream's home_owned probe also takes that same lock for a moment, once a second, for every launch still in its owner-wait loop (packages/ssh/src/tunnel.ts: the node held() fallback, and serviceLauncherIsRunning via __ssh-helper home-owned). That probe does BEGIN EXCLUSIVE, then closes.
  • When the kit's 20 ms poll overlaps one of those probes, the watcher leaves its loop and calls let_go() with no launcher running.

Measured with the kit's own handoff.launcher_serving, HANDOVER_POLL polling against a probe that runs like upstream's:

  • about 0.67 % of SSH probes are seen as "launcher serving";
  • that is about 18 % over 30 probes and 33 % over 60, i.e. one waiting launch during a 30–60 s t3 update.

Failure scenario (switch):

  1. A desktop app retries during the copy. One launch has used up its 60 s marker wait and is in the owner loop, probing once a second.
  2. _update (switch.py:380) runs t3 update, which installs first and restarts T3 later.
  3. The watcher sees the phantom hold and lets go of the owner lock while t3 update is still installing.
  4. One second later, the same launch's home_owned finds both locks free. It leaves the loop and starts its own t3 serve. A waited launch does not read the marker again.
  5. The new launcher's server meets that server and idles. This is the Owner question: one short T3 restart so the boot service owns the server? #513 / 11-19 incident again.
  6. The backstop then fails verify. The automatic W1 is refused by the same foreign check (see 4), so the switch ends failed with the hand server owning the home.
  7. W1's _update has the same exposure (back.py:387).

Fix:

  • Count the launcher as serving only when the lock is held on several polls in a row across more than an SSH probe's hold time (for example 3 polls over at least 100 ms). Upstream's launcher holds it continuously; a probe holds it for microseconds.
  • Optionally also require adapter.service()['active'].
  • Add a gap test that runs a probe loop on the serving lock during starting(), and assert the owner lock is still held until a real launcher takes it.

2. BLOCKING: the backstop's remedy says to restart the service on a half-restored home

handoff.py:321-325, raised by back.py:269 (W1) and back.py:643 (W2/W3 following a stuck W1)

Failure scenario:

  1. A W1 (by hand, or the switch's automatic one) stops and disables T3, then fails at restore.
  2. The kit exits. Its marker runs out and its lock is gone (t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530).
  3. An SSH launch starts a server on the half-restored home.
  4. The owner reruns t3-golive back --way w1, as the earlier error told them to.
  5. That rerun is refused with kill <pid>, then at once systemctl --user restart t3code.service.
  6. Restarting starts whatever build service-state.json names (after a switch, the target) on a half-restored home. Holding T3 stopped and disabled exists to prevent exactly that: "T3 stays stopped and is not started at boot (on purpose: the home may be half-restored)".

Fix: when the journal shows a W1 stop-service that did not end verified, or the service is disabled, the remedy should be: kill the pid, then run the same t3-golive back again (do not start the service). Add a test for that state.

3. SHOULD-FIX: foreign() trusts a possibly stale pid and tells the owner to kill it

handoff.py:304-319; switch.py:282 builds ForeignWatch(cfg.home) without old_pid

  • foreign() takes the pid from server-runtime.json and checks only that it is alive and not a child of mainPid. It does not check that the pid is a T3 server or that it holds the home. live.live_server checks the exe; this does not.
  • server-runtime.json is left stale when a server is SIGKILLed, for example when systemd escalates at the stop timeout. A long cold copy plus install gives that pid time to be reused.

Failure scenario:

  1. In switch verify, the new server has not yet published its record, and the stale pid now belongs to an unrelated process.
  2. After 10 s, verify fails with "kill ".
  3. The switch goes back with W1 over a target that may have been fine.

Fix: count a server as foreign only while userdata/server-owner.sqlite is actually held (the same probe as _lock_held); a stale record never holds it. Also pass the stopped server's pid as old_pid in switch verify.

4. SHOULD-FIX: the README and the message describe a switch backstop that cannot happen, and no test covers it

README.md ("The gap" table: "The switch goes back with W1"), switch.py:682-690, back.py:269

  • When the backstop fires in switch verify, _put_back calls restore_and_downgrade. That refuses at once on the same foreign() check.
  • The switch therefore does not go back. It ends with "verify failed: ; then putting T3 back failed: Refused: ; nothing was changed".
  • Not restoring under a live server is arguably right, but the README says otherwise, and gap.test.py has no switch-backstop test that would have caught this.
  • The message's diagnosis ("most likely ... an SSH client older than 0.1.10") is also not the only cause, because of finding 1.

5. NIT: a failed marker renewal is never reported

handoff.py:125,149

  • Gap.error is recorded but never read.
  • A renewal that fails, for example ENOSPC while the cold copy fills the disk, lets the marker run out without a word. The owner lock still covers the home, but the verify marker fact should say so, for example left/renew failed: ....

6. NIT: a marker-write failure in W2/W3 is an error, not a refusal

back.py:381-382, back.py:281-284

  • For W2/W3, a marker that cannot be written raises a raw OSError after the plan step was journalled. The run then needs settling although nothing changed, while the PR says "refuses before T3 stops".
  • W1 says "stopping T3 for W1 ..." before the marker write that may fail.

7. NIT: a newer marker can still be removed in a tiny window

handoff.py:207-217

  • served() reads the marker, compares it, then unlinks it. A newer marker renamed in between would be removed.
  • The window is tiny, and upstream's DesktopBackendPool.releaseUpdateMarker has the same pattern, so this is acceptable. Mention it in the docstring rather than claim "left alone".

Checked and fine

  • Marker format: the writer matches DesktopUpdateMarker.publish (compact JSON, 60 s TTL, .tmp then rename). The kit's remaining_ms matches remainingWait and the launch script's node reader.
  • Lock order: the launcher calls #claimServing before it starts its child, and the child waits 250 ms (busy_timeout) for the owner lock. So with a real launcher, the 20 ms handover is safe.
  • Holding the lock through t3 update: upstream t3 update (update.ts, install.ts, servicePreflight.ts) takes neither the owner lock nor a stateDirOwned check, so keeping the lock through it does not make the update fail.
  • Killed run: the daemon thread dies, the marker runs out within 60 s, and the SQLite lock dies with the process. Nested runs keep the outer gap. restore reuses the lock the gap holds and gives it back.
  • Nothing kills or stops a foreign process. The guards' fresh marker is left to run out.

Tests run (fresh clone, PR head)

  • Pass: journal 29, check 70, contract 5, copy 108, rehearse 87, common/tests/machine_file 27.
  • gap 15/16 and back 105/106. The failing test in each (SwitchRace…verifies, Status.test_up…) and the errors in switch.test.py all fail on "nothing answers HTTP" on this host. The same tests fail the same way on unchanged main, so this host's loopback HTTP is the cause, not the PR. I could not get a clean switch run here.
  • The new tests on the old code (main plus the new gap.test.py): CopyRace (waited launch), SwitchRace and both W1Gap tests fail on behaviour. A hand server takes the home, there is no marker, and W1 is not refused. The other tests fail only on importing handoff. So the race tests are real regression tests. None of them exercises finding 1 or finding 2.

scuttletoby and others added 2 commits November 19, 2026 14:20
- handover: count the launcher as serving only when service-serving.lock
  is held on every poll for >= 100 ms and >= 3 polls; an SSH launch's own
  home-owned probe no longer frees the home mid `t3 update`
- backstop: name a foreign server only while that pid itself holds
  userdata/server-owner.sqlite (/proc/locks); switch verify passes old_pid
- remedy: always quit the desktop app first, then kill; behind an
  unfinished way-back stop (or a disabled service) run the same back again
  and never start the service; the switch says its W1 refused too and
  where T3 stands
- W1/W2/W3: a marker that cannot be written is a clear refusal before
  "stopping T3"
- verify facts: handover (launcher/timeout/...) and renewFailed
- tests: probe-loop handover (unit, copy, switch), stale pid, W1 rerun
  remedy, switch backstop, W1 marker refusal, renewal failure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant