[Bug]: Durable SSH terminal remains falsely Attached and non-interactive after RPC stream timeout #3439

Activity

1. CitrusYL commented on Jul 22, 2026

   

   CitrusYL

   on Jul 22, 2026

   More actions

   Same issue on Wave v0.14.5, macOS arm64. Durable SSH + tmux + codex becomes non-interactive, and Force Restart Controller restores it.
2. YVin3D commented on Aug 28, 2026

   

   YVin3D

   on Aug 28, 2026

   More actions

   fwiw it's not codex only. i get the same issue (Wave v0.14.5, macOS arm64) in codex, claude, and other durable SSH terminal blocks.

   tail -f on remote logs that are rapidly outputting lines seems to trigger it a lot more consistently if anyone wants to try and replicate it. (helps if your remote server's network connection is bad)

   the one block will freeze while other blocks on the same SSH connection and/or the same tab are just fine.

   remote process stays alive, and the stalled job-manager log prints:

   SendData: error sending stream data: timeout sending request;

   Reattaching that job’s stream restores it. Seems to be from Wave’s per-job durable stream/RPC
3. kieled commented on Oct 13, 2026

   

   kieled

   on Oct 13, 2026

   Last edited by kieled

   More actions

   Also hitting this on v0.14.5 (macOS arm64 → Linux x64 over Tailscale, fish, term:durable: true), several times a day. Each frozen job's remote log has exactly one SendData: error sending stream data: timeout sending request and nothing after it, while the SSH connection stays healthy (no stalled/disconnected events).

   I traced it in the source, and I think the issue is slightly more specific than "the error is swallowed": a single failed send permanently deadlocks the stream, because a dropped packet is never retransmitted and the receiver never ACKs past the gap. (Links are to current main; these files are unchanged since v0.14.5.)

   1. prepareNextPacket advances sentNotAcked before the send happens (streammanager.go#L409-L410).
   2. If the enqueue in SendComplexRequest blocks for more than DefaultTimeoutMs (5s), the packet is dropped (wshrpc.go#L802-L808). routedDataSender.SendData only logs this (mainserverconn.go#L44-L51), so senderLoop carries on as if the bytes had been delivered.
   3. The sender has no retransmit timer. RecvAck only advances on cumulative ACKs.
   4. On the main-server side, streamclient.Reader.RecvData parks every later packet (Seq > nextSeq) in oooPackets and returns without sending an ACK (streamreader.go#L71-L74). nextSeq can never move past the lost packet.
   5. The sender keeps sending until sentNotAcked reaches the 64 KB window, then waits on drainCond forever (streammanager.go#L386-L390). The job route is still up, so handleRouteDownEvent / auto-reconnect never fire, and the shield keeps saying Attached.

   This fits the reports here and in #2985: only the affected block freezes, other blocks on the same connection keep working, the remote process keeps running, and a full reattach (app restart, or Disconnect → Reconnect) restores everything with no data loss, since ClientDisconnected() resets sentNotAcked and PrepareConnect resumes from the client's nextSeq. Bursty TUI output (Claude Code, Codex, tail -f) makes a 5s enqueue stall more likely, which would explain why those workloads trigger it.

   Also worth noting for anyone who lands here: Advanced → Force Restart Controller does not reattach. It terminates the job manager and starts a new shell (my log shows terminating job manager followed by RemoteStartJobCommand), so the session is lost. Use Disconnect → Reconnect instead.

   On #3460: I think its job-manager half (return the error from SendData, then ClientDisconnected() + close the stale client so the route drops and the existing auto-reconnect runs) is enough to fix this on its own. I'm less sure about the 45s output watchdog in jobcontroller. A shell idle at a prompt produces no output, so it would get a disconnect/reattach cycle every ~45s. Also, runOutputLoop breaks and closes the reader before handleStalledStream decides whether to act. When that returns early (cooldown, connection reported down, the remote-disconnect RPC timing out), nothing restarts the output loop, and the block freezes again with no recovery. Raising the timeout (as #3551 does) makes drops rarer but doesn't remove the deadlock.

   I just opened #3553 with just the job-manager-side fix: on a send error the stream drops back to buffering mode (only if the failing sender is still the current one) and the stale connection is closed, so the existing route-down auto-reconnect reattaches from the client's last received seq. It includes streammanager_test.go cases with a failing DataSender, which fail on main. On the main-server side it only changes the auto-reconnect cooldown, so a second failure inside it is deferred instead of dropped. There's no new watchdog.