Skip to content

fix(zero-cache): don't crash when a client resets during a WebSocket handoff - #112

Merged
tmm merged 2 commits into
mainfrom
zero-cache-handoff-reset
Nov 6, 2026
Merged

tmm merged 2 commits into
mainfrom
zero-cache-handoff-reset

Conversation

@tmm

@tmm tmm commented Nov 6, 2026

Copy link
Copy Markdown
Member

zero-cache crashes when a client resets its TCP connection while a WebSocket handoff is pending. Found by the Swift chaos fuzz (#111).

What happens. The handoff handler takes the raw socket from the HTTP upgrade event, or from the parent over IPC, and nothing listens for its errors until ws takes it over. The runner's handoff waits for getWorker() first. A client RST in that window raises read ECONNRESET as an uncaughtException, and the process exits:

pid=4963,worker=dispatcher uncaughtException {"code":"ECONNRESET","syscall":"read",...}
zero-cache (4963) exited with code (13)

In #111's Linux end-to-end job, every end-to-end test after that point was refused on port 4848. The same crash happened in the runner on a local orb, where the service supervisor restarted zero-cache and hid it.

Fix. Until ws takes the socket, the handler destroys it on an error. The listener is removed just before handleUpgrade, which adds its own.

Checked.

  • The new websocket-handoff.test.ts sends an upgrade, then resets the connection while the handoff waits. It fails without the fix with that same read ECONNRESET, and passes with it.
  • Locally, zero-cache with this fix ran the Swift end-to-end suite and 40 chaos seeds without exiting.
  • Upstream (rocicorp/mono main) has the same code, so this is worth sending upstream.

No changeset: like #63, it's a zero-cache fix that this fork doesn't publish.

ampagent and others added 2 commits November 6, 2026 09:23
…handoff

Nothing listened for errors on an upgraded socket while the runner waited
for a worker to hand it to, or on the socket a worker received over IPC
before ws took it. A client that reset the connection in that window
raised read ECONNRESET as an uncaught exception, which exits the process
and takes zero-cache down. The Swift chaos fuzz hit it in CI: zero-cache
exited with code 13 and every later end-to-end test was refused.

The handoff handler now destroys the socket on an error until ws takes
it over.

Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859
Co-authored-by: tmm <tmm@tmm.dev>
@tmm
tmm merged commit 62886b4 into main Nov 6, 2026
10 checks passed
@tmm
tmm deleted the zero-cache-handoff-reset branch November 6, 2026 09:33
tmm added a commit that referenced this pull request Nov 10, 2026
…ay (#129)

When a connection closes while the runner sends it to the dispatcher, Node
delivers the handoff message without the socket. Since #112 the dispatcher
calls socket.on('error', …) as soon as the message arrives, so a missing
socket threw a TypeError, an uncaught exception that exits zero-cache with
code 13. main's Swift end-to-end job hit it at a0df53f: the dispatcher died
at websocket-handoff.ts:80 right after the chaos fuzz cut two clients, and
every later end-to-end test timed out.

The dispatcher now drops such a message, as installWebSocketReceiver already
does in the syncer.

Amp-Thread-ID: https://ampcode.com/threads/T-0d2ed206-a5dd-4f42-9b68-c56e6b23e8f6

Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants