Repository navigation
agent: reconnect to the servers when the network changes - #1927
Draft
Narasimha-sc wants to merge 2 commits into
Draft
Narasimha-sc wants to merge 2 commits into
Narasimha-sc wants to merge 2 commits into
Conversation
This was referenced Oct 5, 2026
Draft
Draft
Narasimha-sc
force-pushed
the
nd/net-close-clients
branch
from
October 5, 2026 10:46
2eae233 to
a1cf376
Compare
Narasimha-sc
force-pushed
the
nd/net-socket-timeout
branch
2 times, most recently
from
October 5, 2026 16:13
4ee5896 to
ad94f82
Compare
Narasimha-sc
force-pushed
the
nd/net-close-clients
branch
from
October 5, 2026 16:13
a1cf376 to
d4c68c9
Compare
… data takes to fail TCP keep-alive probes are only sent while there is no unacknowledged data, so a connection that died with data in flight is only reported as failed when the OS gives up retransmitting - ~15 minutes on Linux and Android. When no network change is reported to the agent (a silent route change, a client without a network observer) that is how long delivery keeps failing on the dead client. KeepAliveOpts gets unackedDataTimeout: when it is set, the sockets that set TCP keep-alive also set TCP_USER_TIMEOUT (TCP_RXT_CONNDROPTIME on Darwin, not set on Windows) to the keep-alive detection time, so that such a connection fails after the same time as an idle one. On Linux this timeout also bounds the zero-window persist timer, so a server that stops reading for that long then fails the connection. The option is off unless the client sets it, so that it can be enabled by the users who test it before it becomes the default. It is a Maybe to keep the clients that do not send it working. As it is derived from the keep-alive options where they are set, the two detection times cannot diverge and lowering the keep-alive settings lowers both. It is removed for the connections via SOCKS proxy - the socket is then connected to the proxy, its acknowledgements only mean that the proxy buffered the data, and a slow circuit should not fail the connection.
After the network changes (e.g. the IP address changed after internet loss) the connections to the servers are dead, but the OS reports it only after TCP retransmissions time out - ~15 minutes on Linux/Android while the agent keeps writing to the dead socket. Until then message delivery, subscriptions and file transfers keep failing on the same client. setUserNetworkInfo on every online network event (the clients report one when the network type changes and when it goes offline to online): - closes the connected SMP and NTF clients and fails their pending requests, so that the workers that resume do not wait for the responses that will not arrive - the requests are failed only once the client's threads terminated, which the client now records, as its session var is also removed when a resubscription fails (each client type bounded by 1 second; the requests of a client that did not terminate in this time are left to time out as before); - discards the errors of the clients that failed to connect, cached for persistErrorInterval, and removes a client that is still connecting, in the transaction that reads it, so that the resumed workers create a new client instead of waiting for a connection made via the previous network - that client is closed by newProtocolClient when it finds it is no longer in the map; - closes the XFTP clients asynchronously, removing them from the map. A client that is no longer the one in the map is rejected in the same transaction that would publish it, so that the requests made on it do not wait for the timeout, and it is closed in a forked thread, as closing an XFTP client waits for its thread to terminate; it sets its session id while its var is the one in the map and updates its service only if it was. The clients are closed in a forked thread, so that the API does not block, and the closes of an event that a newer one superseded are skipped. When the agent was offline the reported network info is applied when the clients are closed, so that the workers waiting for the network do not resume before then; when it was online it is applied immediately. As the clients are closed, the network change now results in DISCONNECT and DOWN events followed by CONNECT and UP, as any other reconnection. The retry loops are not restarted, so a worker already waiting out its retry interval resumes only when it expires.
Narasimha-sc
force-pushed
the
nd/net-socket-timeout
branch
from
October 5, 2026 21:20
ad94f82 to
656df39
Compare
Narasimha-sc
force-pushed
the
nd/net-close-clients
branch
from
October 5, 2026 21:20
d4c68c9 to
8e3f9a4
Compare
Narasimha-sc
force-pushed
the
nd/net-socket-timeout
branch
from
October 8, 2026 20:49
656df39 to
84381e0
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After the network changes (e.g. the IP address changed after internet loss) the connections to the servers are dead, but the OS reports it only after TCP retransmissions time out - ~15 minutes on Linux/Android while the agent keeps writing to the dead socket. In a measured trace the online event woke the sender immediately and it then retried
SENDon the dead client 35 times, every ~23 s, until the kernel gave up.setUserNetworkInfoon every online network event now:persistErrorInterval, and removes a client that is still connecting, in the transaction that reads it, so that resumed workers create a new client instead of waiting for a connection made via the previous network;A client that is no longer the one in the map is rejected in the same transaction that would publish it, so the requests made on it do not wait for the timeout. The clients are closed in a forked thread so the API does not block, and the closes of an event that a newer one superseded are skipped. When the agent was offline the reported info is applied when the clients are closed, so the workers waiting for the network do not resume before then.
As the clients are closed, a network change now produces DISCONNECT/DOWN followed by CONNECT/UP, as any other reconnection.
The retry loops are not restarted here, so a worker already waiting out its retry interval resumes only when it expires - that is a separate PR in this stack.
Stack
Reducing reconnection delay after a network change. Each PR is based on the previous one and can be merged on its own once the ones below it are in.
mastermaster