agent: choose another proxy when the chosen proxy keeps failing

[Narasimha-sc]

Not ready: needs Android testing with the core library built from simplex-chat/simplex-chat#7739. Desktop testing is done.

Problem

With private routing, a connection can stay at "connecting…" until the app is restarted, and joining a link can fail on every "Retry". The agent keeps one proxy per destination relay. When that proxy can't forward to the relay, every retry goes through it again. Seen with public servers on 2026-10-14 (smp4 → smp10).

The existing delayed resets don't help here. A failed connection is retried after 3 s (persistErrorInterval) to the same remembered proxy. A connection is dropped only after 3 timeouts and 15 minutes with no response, and a proxy that answers with errors keeps its connection alive. When the proxy's own connection to the relay is dropped, the client gets NO_SESSION and by design (#1178) makes a new session through the same proxy.

Fix

Forget the chosen proxy when it keeps failing, so the next attempt chooses again:

1. The connection to the proxy fails (proxy offline or unreachable): forget it at once. Without a live connection, this proxy normally hasn't received the destination, so choosing again adds no proxy that learns it. A request that only waits for another request's connection doesn't count its own wait timing out.
2. A forwarded command fails with PROXY (BROKER e) other than a host error (e.g. the proxy times out reaching the relay): the first failure starts a failure streak, stored next to the chosen proxy. A forward that reaches the relay clears it: a success, an error from the destination relay inside the response, or PROXY (PROTOCOL _). A failure at least proxyFailureInterval (new AgentConfig field, 30 s) after the start of the streak forgets the proxy. So a single timeout, or a burst of errors while the relay restarts, doesn't move the destination to another proxy.

Updates are guarded: an entry is changed only while it still names the proxy that failed or succeeded (also in smpClientDisconnected and newProxiedRelay), so a late result from the old proxy can't change a new choice. After PROXY NO_SESSION, the dead relay session is removed from the proxy client that handled the request. The retry through the same proxy (#1178) doesn't record that proxy as the choice when there is none, so a proxy forgotten concurrently isn't put back.

Details: plans/2026-10-15-proxy-repick-on-failure.md.

Tests

• New SMPProxyTests:
  • "chooses another proxy when the chosen proxy is offline";
  • "chooses another proxy when the chosen proxy cannot forward to destination relay";
  • "keeps the chosen proxy until it fails for proxyFailureInterval", a direct test of the streak rule (needs proxiedRelayFailed and proxiedRelaySucceeded exported).
• Mutation checks: with the 30 s forget disabled, "cannot forward" fails (2/2). With the connect-failure forget disabled, "offline" fails (2/2). With the fix, all 3 pass (3/3 runs).
• SMP proxy tests: 52 examples, 0 failures.
• Full suite: 896 examples, 8 failures, none caused by this change. 2 pass when re-run alone. 3 are environment failures on this machine that also fail on the base code (batching 200 subscriptions, XRCP multicast, SMP server CLI static files). The other 3 also fail on the base-code binary when run alone: delivery receipts concurrently (2/2), XFTP resume deleting after restart (2/2) and join when reply queue creation fails (1/2).

Desktop app (simplex-chat with this branch, local servers)

Destination relay R; proxy P1 whose forwarding to R is black-holed (a SOCKS relay passes the handshakes and then drops all bytes); good proxy P2. The app's message servers are only P1 and P2. Trials where the first pick was P1:

Scenario	master	this branch
The contact joins the app's 1-time link (the app sends its reply in the background): "connecting…"	not connected after 10 min (2/2)	connected after 64, 64, 129 and 192 s, no user action (4/4)
The app joins a link, user taps "Retry" after each error	not connected after 6 Retries (2/2)	connected after 3, 4 and 6 Retries (86–177 s), and once after 61 s without a Retry (4/4)

The longer trials picked P1 again after forgetting it: the failed proxy isn't excluded, so with two proxies that happens half of the time.

Recordings (the app's proxies: 127.0.0.1 = P1, broken, and 127.0.0.2 = P2; the red box marks what to look at):

"connecting…" after the contact joins the app's link. master stays "connecting…":

This branch connects once P1 has failed for 30 s, with no user action:

Joining a link with Retry. master fails on every Retry:

This branch: the first "Private routing timeout" starts the streak, the next one 30 s later forgets P1, and the next Retry connects:

Android

Pending: needs the core library built from simplex-chat/simplex-chat#7739. The previous version of this PR, which forgot the proxy on the first failure, passed fixed-outcome checks on Android.

Open questions

• Interactive recovery takes a few Retries. Client timeouts don't count, so a join needs the proxy's report twice, 30 s apart. In the trials that took 3–6 Retries, up to about 3 minutes.

• The failed proxy can be chosen again. Excluding it from the next pick would make recovery take one streak instead of about two on average with two proxies.

• The streak doesn't expire. With rare sends, one timeout, then another hours later with nothing reaching the relay in between, forgets the proxy.

• Not covered by tests:

  • not counting a waiter's own timeout;
  • clearing on relay errors;
  • the deleteRelaySession change;
  • the disconnect and PRXY guards;
  • not recording an explicit proxy.

  The re-pick tests also don't assert that the bad proxy was tried first; they rely on the seeded pick.

• Unchanged: a host error that the proxy reports about the relay keeps the entry (e.g. an onion-only relay through a proxy without Tor).