Repository navigation
🤖 fix: bound a stored bind target, and keep AgentNotReady across failed reads (#153 plan section 16) - #464
Merged
Conversation
… reads (#153) A pool member raised its bind extension to a stored status.bindExtendDeadline with no fresh bound, so a status writer could turn a one-hour TTL into a one-year Coder deadline. A stored target later than a fresh extension now counts as missing: the member stores the fresh target before the request and asks Coder for that. A failed Coder read, a wrong answer, or a control plane without operator access set Ready=True on an AgentNotReady member and restarted its 5-minute AgentLost timer. Such a step now keeps AgentNotReady and its transition time. Plan section 16 items (#339 and #343 final assessors), found by the #463 verifier. Change-Id: Ic260b6bc198fd41efe79fed8b6ce51cc3af1ee2e Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Member
Author
|
@codex security review |
3 tasks done
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two pool member fixes from plan section 16 that never landed. The #463 verifier found both with overlay probes.
status.bindExtendDeadlinewith no fresh bound. A status writer could turn a one-hour template TTL into a one-year Coder deadline, so the workspace never stopped. Now a stored target later than a fresh extension (now + max(template default TTL, 30 min), capped by the max deadline) counts as missing.AgentNotReady. Before this PR, a Coder5xxor429, a wrong answer, or a control plane without operator access setReady=Trueon a member whose agent was not ready. That restarted the 5-minuteAgentLosttimer, and a claim could take the dead workspace in the meantime. Now such a step keepsReady=False/AgentNotReadyand its transition time.Refs #153
#463 (the #153 docs,
Fixes #153) waits for this PR.Background
Plan section 16 lists both items as "before PR 12 registers the reconciler":
status.bindExtendDeadlineandstatus.lastExtendTimebefore use, to at most a freshnow + max(template default TTL, 30 min), or treat a larger stored value as missing, with an envtest for each.AgentLosttimer. Keep theAgentNotReadystart time across failed reads, with an envtest.The verifier's probes on main reproduced both: a one-year Coder deadline from a one-year stored target with a 1 h TTL, and
Ready=Truewith reasonCoderUnavailableafter a503.Implementation
coderworkspacepoolmember_extend.go: a bound member already stores its target before the extend request, so a later read proves whether the extension happened. A stored target later than the fresh extension now takes the same path as a missing one. The member stores the fresh target, then asks Coder for it. The proof still holds, because the stored target is never more than the request.status.lastExtendTimehas no reader in the controller. The new test pins that a future value stops no extension and is replaced by the real time.coderworkspacepoolmember_watch.go:poolMemberKeepAgentNotReadykeeps the reasonAgentNotReadyon a wait step that read no agent, for a ready member whoseReadycondition isFalse/AgentNotReady.applyPoolMemberStepthen keepsReady=False, andmeta.SetStatusConditionkeeps itslastTransitionTime, which is the timer. The message names the failure. The429backoff is kept.memberWaitOnWrongAnswernow runs inwatchReadytoo), and a control-plane wait inreadWorkspace. A step after a successful read, such as a failed extend, still setsReady=True, because that read proved the agent ready.docs/how-to/troubleshooting.md: theAgentNotReadyrow says that a failed Coder read keeps the reason and the timer.Validation
Both tests failed on main first, for the reported reason:
TestPoolMemberBoundsStoredExtendTimes(new)lastExtendTimeon an unclaimed member: the member still extends 14 min before the deadline and stores the real time.TestPoolMemberWatchRules(3 new failed reads)AgentLost, a503, a wrong workspace answer, and a control plane without operator access each keepAgentNotReadywith the samelastTransitionTime. The member still fails withAgentLost5 min after the first not-ready read.On main, the first test fails with Coder deadline
2027-11-01, and the second with reasonCoderUnavailable. The verifier's two probes (closure-probes_test.gofrom the #463 evidence) pass on this branch.Mutation checks. Each mutation removes one control, and the named test fails (7 of 7 killed):
TestPoolMemberBoundsStoredExtendTimesTestPoolMemberBoundsStoredExtendTimeslastExtendTimeskips the extensionTestPoolMemberBoundsStoredExtendTimesAgentNotReadyTestPoolMemberWatchRulesTestPoolMemberWatchRulesTestPoolMemberWatchRulesTestPoolMemberWatchRulesLocal gates on the head:
make verify-vendor test build lint test-integration docs-reference-checkandmkdocs build --strictpass.CI on 10789ac: every required check passes. The E2E job passed the pool phases P1 to P12 and the lifecycle phases.
Risks
status.bindExtendeditself.AgentNotReadymember stays not ready and never fails, because only a successful read can find the agent lost. Once Coder answers, a member whose agent is still not ready fails at once if its 5 minutes are up.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh