Skip to content

agent: restart the retry loops when the network changes - #1929

Closed
Narasimha-sc wants to merge 1 commit into
nd/net-session-racesfrom
nd/net-retry-epoch
Closed

Narasimha-sc wants to merge 1 commit into
nd/net-session-racesfrom
nd/net-retry-epoch

Conversation

@Narasimha-sc

@Narasimha-sc Narasimha-sc commented Oct 5, 2026 •

Copy link
Copy Markdown

Closing the clients when the network changes makes the workers fail their current attempt, but a worker already waiting out its retry interval resumes only when it expires - up to the maximum interval, so delivery and subscriptions can take minutes after the network is usable again. This matters most when the IP changes without an offline event (a roam to another network, or a lease renewed while still associated): the workers are not parked on waitForUserNetwork, they are in backoff, so closing the clients does not wake them.

The network epoch is incremented when the clients are closed, and the agent retry loops restart from the loop's initial interval when it changes - for delivery and XFTP workers resumed with persisted retry state that is the persisted interval (withRetryEpoch).

  • message delivery restarts the fast interval only, the slow interval for QUOTA errors is unchanged;
  • the subscription worker keeps its foreground and online resets;
  • retry attempt counters are not reset;
  • a loop that was waiting for the network sees the epoch change after its first attempt, so if that attempt fails it retries once more without delay.

With this, a reported IP change recovers in 3-4 s in the harness, against 808 s before the stack.

135 of the added lines are RetryIntervalTests covering the epoch, including mixed slow/fast intervals and an epoch change during the action.

Rebased onto master

master had refactored withRetryLock2 into a generalised internal withRetryWait2, and moved the command worker from a single retry interval to two (loop RISlow/loop RIFast). Rather than duplicate master's loop, withRetryWait2 is generalised once more: it takes the value to sample before the action - sampling before matters, so that an epoch change during the action also restarts the interval - and a wait that receives the RetryIntervalMode and returns whether the interval restarts. One loop then serves the plain, lock and epoch variants. RISlow retries in STM and falls through to the plain wait, so the slow quota interval is byte-for-byte unchanged in behaviour.

withRetryEpoch2 is added for the command worker: withRetryLock2 without the lock. It is covered by master's own parameterised 2-mode tests, which now run against all three variants.

Retry interval is 15 examples, 0 failures (was 11 before the rebase, as master's parameterisation adds cases); user network info is 5/5.


Stack

Reducing reconnection delay after a network change. Each PR is based on the previous one and can be merged on its own once the ones below it are in.

PR base
#1926 transport: bound unacknowledged-data failure time master 894 s -> 95 s, no app change, independent
#1927 agent: reconnect to the servers when the network changes #1926 the 15-minute stall
#1928 agent: fix three races closing clients makes reachable #1927 correctness guard for #1927
#1929 agent: restart the retry loops when the network changes #1928 retry interval -> ~3 s
#1930 agent: restart retry loops when reconnecting all servers #1929 manual reconnect only
simplex-chat/simplex-chat#7650 android, ios: pass the chat controller to the network info command master crash fix, independent
simplex-chat/simplex-chat#7651 android, ios: report a network change when the network identity changes #7650 app side of simplexmq#1927
simplex-chat/simplex-chat#7652 desktop: report network changes to the core #7651 new capability

Closing the clients when the network changes makes the workers fail their
current attempt, but a worker already waiting out its retry interval resumes
only when it expires - up to the maximum interval, so delivery and
subscriptions can take minutes after the network is usable again.

The network epoch is incremented when the clients are closed, and the agent
retry loops restart from the loop's initial interval when it changes - for
delivery and XFTP workers resumed with persisted retry state that is the
persisted interval (withRetryEpoch). Message delivery restarts the fast
interval only, the slow interval for QUOTA errors is unchanged; the
subscription worker keeps its foreground and online resets; retry attempt
counters are not reset. A loop that was waiting for the network sees the epoch
change after its first attempt, so if that attempt fails it retries once more
without delay.

withRetryWait2 now takes the value to sample before the action and a wait that
receives the retry interval mode and returns whether the interval restarts, so
one loop serves the plain, the lock and the epoch variants - the slow interval
retries in STM and falls through to the plain wait, so it is unchanged.
withRetryEpoch2 is withRetryLock2 without the lock, for the command worker,
which retries with two intervals.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant