Skip to content

fix(client-runtime): an unexpected failure no longer stops the reconnect loop for good - #41

Closed
viktorce wants to merge 6 commits into
mainfrom
fix/reconnect-loop-crash
Closed

viktorce wants to merge 6 commits into
mainfrom
fix/reconnect-loop-crash

Conversation

@viktorce

@viktorce viktorce commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Fixes #16.

Filed upstream as pingdotgg#39420, for the upstream issue pingdotgg#39418. The follow-up #39 is upstream as pingdotgg#39419, and a maintainer's PR, pingdotgg#39431, fixes it.

The connection supervisor keeps a connection alive with one loop: try to connect, wait, try again. That loop ran as a single background fiber with nothing around it. A defect or a stray interruption anywhere in it ended the fiber silently. The app kept showing its last state, and Retry did nothing, because nothing was left to hear it.

The issue knew of no real trigger. There is one. rpc/session.ts fails a session's closed with whatever cause ended the config subscription, unless that cause holds interruptions only. A cause that holds a failure and an interruption gets through, and exitUnlessInterrupted turned any cause holding an interruption into an interruption of the loop.

What changes

  • An attempt reports its own unexpected failures. runAttempt uses Effect.exit where it used exitUnlessInterrupted, and that helper is gone. A stray interruption becomes a failed attempt that keeps what the attempt knows: whether the session was live and whether it was stable. A typed error mixed with an interruption keeps its typed error, so a blocked host stays blocked. Effect does not run the continuation of a fiber that is itself interrupted, so a real shutdown still ends the loop.
  • A defect elsewhere in an attempt is caught around the attempt. It is logged and counts as a failed attempt on the normal backoff ladder. If the session had gone live, it counts as a lost session. The catch does not know how long the session lived, so it never resets the ladder.
  • Closing an attempt's scope is separate from the attempt. The scope is opened and closed with Effect.acquireUseRelease. A defect from closing it is logged as a failed release, and the attempt's outcome stands. If it replaced the outcome, a release that throws after a requested disconnect would show backoff, and a Retry would wait out the ladder's current delay.
  • The generation number advances where the session is published as connected. The counter lives outside the loop. No lost outcome and no restart gives two sessions the same number.
  • A defect in the loop outside an attempt is the last resort. It comes from the loop's own bookkeeping, or from a logger that throws inside the handlers above. It is logged, the state shows backoff, offline or available, and the loop restarts after the longest retry delay or the next signal: Retry, connect, disconnect, a network change or a wakeup. The wait is the ladder's ceiling because a restarted loop has lost its retry counters, so a shorter wait would let a repeating defect reconnect with no backoff. This layer is coarse on purpose. The restarted loop has also lost the last failure, and it retries a host that was blocked.
  • The shutdown finalizer is registered before the background fibers. Closing a supervisor then interrupts them before the signal queue shuts down. Otherwise the loop would see the closed queue as an unexpected interruption and log an error on every shutdown.

The issue proposed only the restart where the loop is forked. That alone would send every attempt defect, the issue's own example included, through the five-minute wait and reset the retry ladder. So attempts handle their own failures, and the restart is the last resort.

New log lines

One error line per unexpected failure: Connection <attempt | release | loop> failed with an unexpected <defect | interruption>. It carries the environment and a reason count. For a defect it also carries the first defect's name and stack, through the existing safe-attributes helper. It never carries the error's message.

Main had only the attempt-defect line, and only while connecting. New here: a defect in the connected phase, a defect beside a typed failure in the same cause, every interruption line, and the release and loop lines. A release defect while a supervisor shuts down is logged too.

Proof

The real route, in the running app

reconnect-survives-config-stream-failing-while-interrupted.e2e.ts is part of this PR. It opens a thread, then hands the app one server frame that ends the config subscription with a typed failure and an interruption in the same cause. The frame is injected from the page. I found no way to make a real server send it. Everything after the frame is the app's own code.

$ vp run scenarios:prove scripts/scenarios/reconnect-survives-config-stream-failing-while-interrupted.e2e.ts --base origin/main

Proven: the scenario fails on base 57db750b8f and passes on head 89017c15a0.

base 57db750b8f
  ASSERTION_FAILED: expect.toBeVisible failed
  locator: getByText("The command printed one JSON line.").nth(1)
  observed: no node (match count 0)

Five seconds after the frame, the scenario types a message and presses Enter. These are three seconds later:

Base 57db750b8f Head 89017c15a0
Base: the message is still in the composer Head: the message was sent and answered
No banner and no error. The message stays in the composer. The app reconnected. The message was sent and answered.

Both runs side by side:

real-route-before-after.mp4

The issue's own example, in the running app

A second scenario does what the issue describes. It closes the app's socket from the page, as a network drop would, and makes the next read of navigator.onLine throw once. No real browser throws there, so this scenario is not part of the PR. Its code and the reproduction on main are in this comment on the issue.

scenarios:prove gives the same verdict: it fails on base 57db750b8f and passes on head 89017c15a0. Ten seconds after the drop, the scenario presses Retry now if the button is there:

Base 57db750b8f Head 89017c15a0
Base: still reconnecting after Retry now Head: reconnected, the message was answered
The banner still says "is reconnecting" five seconds after Retry now. Reconnected by itself. There was no Retry now button to press, and a new message was answered.
network-status-before-after.mp4

On head the page logged one line: ERROR: Connection attempt failed with an unexpected defect. On base it logged nothing. A control run of the same scenario with no injected throw reconnected on main's supervisor, so the stuck state comes from the defect and not from the dropped socket.

Tests

The new and changed cases, run with main's supervisor.ts and with this branch:

$ vp test run src/connection/supervisor.test.ts

base 57db750b8f supervisor.ts
 × retries after the attempt's network-status read dies
 × retries after an attempt interrupts itself
 × reconnects soon after a stable session is interrupted
 × reconnects soon after a stable session fails and dies while interrupted
 × reconnects soon after a stable session fails while interrupted
 × reconnects soon after a stable session closes while its release defects
 × does not reuse a live session's generation when its attempt defects after its session went live
 × does not reuse a live session's generation when logging its release defect dies
 × keeps a released session's outcome
 × restarts the loop after a defect outside an attempt
 × keeps a blocked host that fails while interrupted idle until an external signal requests another attempt
 Tests  11 failed | 79 passed (90)

89017c15a0
 Tests  90 passed (90)

All eleven fail on base because the loop is dead and the state never changes. The two "stops attempting when its scope closes" cases pass on base on purpose. They guard the finalizer order, and they fail if the finalizer is registered after the fibers again.

vp test run src/connection passes, 214 of 214. vp run check:changed passes.

Not covered

Implemented by Claude Fable 5.2 and Claude Opus 5.5 in Claude Code (T3 Code). Reviewed by Claude Opus 5.5 and GPT-6.1 Sol.

viktorce and others added 2 commits November 19, 2026 22:48
…ect loop for good

The connection supervisor's reconnect loop ran as one forked fiber with
nothing around it. A defect or a stray interruption in it ended the fiber
silently. The state stayed where it was and Retry did nothing.

One route real code takes: a session's `closed` can fail with a cause that
mixes a failure with an interruption, and `exitUnlessInterrupted` turned any
cause holding an interruption into an interruption of the loop.

The loop now survives all of that:

- `runAttempt` uses `Effect.exit` where it used `exitUnlessInterrupted`, so a
  stray interruption becomes a failed attempt that keeps whether the session
  was live and stable.
- A defect elsewhere in an attempt is caught around the attempt, logged, and
  retried on the normal backoff ladder.
- A defect from closing an attempt's scope is logged as a failed release, and
  the attempt's outcome stands.
- The generation number advances where the session is published as connected,
  so no lost outcome or restart gives two sessions the same number.
- A defect in the loop outside an attempt is logged, and the loop restarts
  after the longest retry delay or the next signal.
- The shutdown finalizer is registered before the fibers, so closing a
  supervisor does not look like a failure to the loop.

A failure inside the restart handler itself still ends the loop, after one
log line. The supervisor's other background fibers are unchanged.
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:L labels Nov 19, 2026
@github-actions github-actions Bot added size:XXL and removed size:L labels Nov 19, 2026
@viktorce

Copy link
Copy Markdown
Owner Author

Closing without merging here. The fix landed upstream as pingdotgg#39420, squash-merged as 38bac7091a, with the tests trimmed by the maintainer. This fork gets it the next time its main syncs with upstream.

@viktorce viktorce closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A crash inside the reconnect loop stops it for good, and nothing is logged

2 participants