Skip to content

🤖 fix: bound a stored bind target, and keep AgentNotReady across failed reads (#153 plan section 16) - #464

Merged
ThomasK33 merged 1 commit into
mainfrom
fix/pool-member-bounds
Nov 1, 2026
Merged

ThomasK33 merged 1 commit into
mainfrom
fix/pool-member-bounds

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Two pool member fixes from plan section 16 that never landed. The #463 verifier found both with overlay probes.

  1. A stored bind target no longer sets the Coder deadline. Before this PR, a member raised its one bind extension to the stored status.bindExtendDeadline with no fresh bound. A status writer could turn a one-hour template TTL into a one-year Coder deadline, so the workspace never stopped. Now a stored target later than a fresh extension (now + max(template default TTL, 30 min), capped by the max deadline) counts as missing.
  2. A failed read no longer ends AgentNotReady. Before this PR, a Coder 5xx or 429, a wrong answer, or a control plane without operator access set Ready=True on a member whose agent was not ready. That restarted the 5-minute AgentLost timer, and a claim could take the dead workspace in the meantime. Now such a step keeps Ready=False/AgentNotReady and its transition time.

Refs #153

#463 (the #153 docs, Fixes #153) waits for this PR.

Background

Plan section 16 lists both items as "before PR 12 registers the reconciler":

The verifier's probes on main reproduced both: a one-year Coder deadline from a one-year stored target with a 1 h TTL, and Ready=True with reason CoderUnavailable after a 503.

Implementation

  • coderworkspacepoolmember_extend.go: a bound member already stores its target before the extend request, so a later read proves whether the extension happened. A stored target later than the fresh extension now takes the same path as a missing one. The member stores the fresh target, then asks Coder for it. The proof still holds, because the stored target is never more than the request.
  • status.lastExtendTime has no reader in the controller. The new test pins that a future value stops no extension and is replaced by the real time.
  • coderworkspacepoolmember_watch.go: poolMemberKeepAgentNotReady keeps the reason AgentNotReady on a wait step that read no agent, for a ready member whose Ready condition is False/AgentNotReady. applyPoolMemberStep then keeps Ready=False, and meta.SetStatusCondition keeps its lastTransitionTime, which is the timer. The message names the failure. The 429 backoff is kept.
  • It runs on the three steps that read no agent: a failed workspace or operator-user read, a wrong answer (memberWaitOnWrongAnswer now runs in watchReady too), and a control-plane wait in readWorkspace. A step after a successful read, such as a failed extend, still sets Ready=True, because that read proved the agent ready.
  • docs/how-to/troubleshooting.md: the AgentNotReady row says that a failed Coder read keeps the reason and the timer.

Validation

Both tests failed on main first, for the reported reason:

Test Proves
TestPoolMemberBoundsStoredExtendTimes (new) A one-year stored bind target with a 1 h TTL: Coder's deadline and the stored target become now + 1 h, and the bind is extended. A one-year lastExtendTime on an unclaimed member: the member still extends 14 min before the deadline and stores the real time.
TestPoolMemberWatchRules (3 new failed reads) Between the first not-ready read and AgentLost, a 503, a wrong workspace answer, and a control plane without operator access each keep AgentNotReady with the same lastTransitionTime. The member still fails with AgentLost 5 min after the first not-ready read.

On main, the first test fails with Coder deadline 2027-11-01, and the second with reason CoderUnavailable. The verifier's two probes (closure-probes_test.go from the #463 evidence) pass on this branch.

Mutation checks. Each mutation removes one control, and the named test fails (7 of 7 killed):

# Mutation Killed by
B1 Trust the stored target again TestPoolMemberBoundsStoredExtendTimes
B2 Request the stored target before it is replaced TestPoolMemberBoundsStoredExtendTimes
B3 A recent lastExtendTime skips the extension TestPoolMemberBoundsStoredExtendTimes
F1 A failed workspace read resets AgentNotReady TestPoolMemberWatchRules
F2 A wrong answer resets it TestPoolMemberWatchRules
F3 A control-plane wait resets it TestPoolMemberWatchRules
F4 The kept step restarts the timer TestPoolMemberWatchRules

Local gates on the head: make verify-vendor test build lint test-integration docs-reference-check and mkdocs build --strict pass.

CI on 10789ac: every required check passes. The E2E job passed the pool phases P1 to P12 and the lifecycle phases.

Risks

  • A stored bind target in the past still counts as "the extension happened" once Coder's deadline passes it, as before. A status writer gains nothing from that, because it can set status.bindExtended itself.
  • While Coder is unreachable, an AgentNotReady member stays not ready and never fails, because only a successful read can find the agent lost. Once Coder answers, a member whose agent is still not ready fails at once if its 5 minutes are up.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

… reads (#153)

A pool member raised its bind extension to a stored
status.bindExtendDeadline with no fresh bound, so a status writer could
turn a one-hour TTL into a one-year Coder deadline. A stored target later
than a fresh extension now counts as missing: the member stores the fresh
target before the request and asks Coder for that.

A failed Coder read, a wrong answer, or a control plane without operator
access set Ready=True on an AgentNotReady member and restarted its
5-minute AgentLost timer. Such a step now keeps AgentNotReady and its
transition time.

Plan section 16 items (#339 and #343 final assessors), found by the #463
verifier.

Change-Id: Ic260b6bc198fd41efe79fed8b6ce51cc3af1ee2e
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T04:59:27.418730Z 10789ac PR opened
🔒 Security Review ✅ Completed 2026-11-01T04:59:58.806660Z 10789ac Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 10789ac535

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit 8296834 Nov 1, 2026
12 of 13 checks passed
@ThomasK33
ThomasK33 deleted the fix/pool-member-bounds branch November 1, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant