Skip to content

fix(server): stale route probes no longer fail resumed turns - #171

Merged
ScriptedAlchemy merged 2 commits into
mainfrom
fix/inference-restart-resume
Nov 20, 2026
Merged

ScriptedAlchemy merged 2 commits into
mainfrom
fix/inference-restart-resume

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Why

After updating T3 Code, a turn that resumed on restart failed with "503 No inference host is currently available for model gpt-6.2-astra" even though the host had the model. A route probe read CLIProxyAPI's /v1/models right after the proxy started, before the list was complete. That result was cached for 15 seconds, so every retry of the resumed turn got the same stale answer.

Server trace from the affected machine, times relative to server start:

  • -0.40s: the probe reads /v1/models (200) from the proxy that just started. The model is not listed yet.
  • +1.76s to +9.81s: ten requests to the linked route return 503 in 1 to 3 ms each, all served from the cached probe.
  • +15.52s: the cache expires and a fresh probe sees the model.
  • +36.49s: the same route returns 200, with no route-state change in between.

The proxy was ready 2 seconds before the first resumed request, so waiting longer at startup would not have helped. The same staleness can block a model that returns from a cooldown inside the cache window.

What changed

  • When the cached probes leave no host for the requested model, routeCandidates refreshes the matching probes once and recomputes before returning empty.
  • Requests that find a host keep the cached fast path.
  • No refresh happens when every matching host is in a request cooldown. Those cooldowns come from real recent failures.

Blast Radius

Inference routing only. A request that would have failed now spends one probe round first, which is at most the 3-second probe timeout per unreachable host.

Verification

  • The new assertion in InferenceProxy.test.ts fails on main (expected [] to have a length of 1 but got +0) and passes with the fix. The test commit lands before the fix.
  • Focused inference suites: 11 passed. vp run check:changed passes.
  • Not reproduced live yet. It needs a restart that resumes a turn while CLIProxyAPI is still warming up.

Opus 5.5 (Claude Code in T3 Code), fix implemented by gpt-6.2-sol (Codex).

🤖 Generated with Claude Code

ScriptedAlchemy and others added 2 commits November 19, 2026 17:34
A probe cached while CLIProxyAPI was warming up keeps returning no host.
This assertion fails on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a restart, a route probe read CLIProxyAPI's model list before it
was complete and cached that for 15 seconds. Every request in that
window got "No inference host is currently available", so the resumed
turn spent its retries on a stale answer.

When the cached probes leave no host for the requested model, routing
now refreshes the matching probes once before giving up. Requests that
find a host keep the cached path, and hosts in a request cooldown are
not re-probed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 300ea8b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy merged commit 7f3a54d into main Nov 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant