Repository navigation
🤖 feat: replace old unclaimed pool workspaces and reap idle claimed ones, both opt-in (#153 PR 15a) - #447
Conversation
…aces (#153 PR 15) Change-Id: I09245332cca3086a4ace22b0a00e2511e0243b04 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…kspace pool how-to Change-Id: Icce5678e04430e30f78a6f00b4ad1d2a3eaeb87d Signed-off-by: Thomas Kosiewski <tk@coder.com>
…without the pool, one clock reading per pool reconcile Change-Id: I70f6e4ddba4e44f456c0b7f212a30b7076201b4c Signed-off-by: Thomas Kosiewski <tk@coder.com>
…s its workspace Change-Id: Ib066d695208202dc191eb1b265efbd24727be993 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I3075167eddd9a920362b432857ae8650de84068e Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I374499ecf1cfbaa92dd4ca5c91655cfcbbfea3e6 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…and name a kept workspace in the claim message Change-Id: I2eecb71f24b498d924ecf80c6292cce4efb5c3ba Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Summary
CoderWorkspacePoolgets two opt-in fields.spec.maxUnclaimedAgeSecondsreplaces unclaimed workspaces that have been ready too long, new before old.spec.claims.idleTimeoutSecondsdeletes a claimed workspace once both its bind and its Coderlast_used_atare older than the timeout, and the claim becomesLostwith reasonIdleTimeout. Without either field, nothing changes.Refs #153
This is plan PR 15a, the first half of a split. PR 15 grew past the plan's 550-line limit, so the Kind phases P10 to P12 (scale to 0, KEDA, version rollout) and their offline stubs follow in PR 15b. 15b waits for this PR and for PR 14 (#446, Share mode). Of this PR and #446, the one that lands second regenerates the generated files.
Background
Plan 6.6 and decision D9 (in the #153 plan):
version.active: truealready replaces outdated members new before old.maxUnclaimedAgeSecondsmakes "ready for too long" one more reason to count as outdated, so the same order applies.Implementation
All new logic is in
internal/controller/coderworkspacepool_refresh.go.poolMemberCurrentadds the age check to the existing "pins the resolved inputs" check. Counting, binding (current before outdated), and the delete order use it. Each pool reconcile reads the clock once and passes that time to all three, so a member that ages in the middle of a reconcile counts the same in every step.poolRefreshWake). The old tailswitchnow computes one requeue and takes the earlier of it and the wake. The existing polls are unchanged.reapIdleClaimsonly looks at claim holders whose label, owner, and control plane match the pool. A candidate's bind must be older than the timeout first. Then the pool reads the workspace in Coder with the member's identity guard. A failed read, a foreign or deleted workspace, or a zerolast_used_atreaps nothing.last_used_atit checked on the member (annotationcoder.com/idle-check-last-used-at).Lost/IdleTimeout.Lostis final.finishIdleReapsdeletes the holder of anyLost/IdleTimeoutclaim after a crash or a 409. It also runs on the orphan-claim path, so a pool delete between step 2 and step 3 strands nothing. A holder without the recorded check is never deleted.last_used_at, or none at all (zero or left out), releases the member without a delete build, and the workspace stays in Coder, as for unclaimed members since CoderWorkspacePool: use after the last read can still be bound or deleted on a delete retry #409.Lostis final, so the claim message written before the delete covers both outcomes: the pool deletes the workspace unless its last check before the delete sees a use, and a kept workspace stays in Coder as<kwp- name>, owned by<owner>(the owner name comes from the deciding Coder read). The baseline counts only while the claim, read without the cache, isLost/IdleTimeoutand not being deleted. So a recorded check whose claim write failed, a claim deletion, or a person's delete of the member deletes the workspace as before.Validation
New envtests (
coderworkspacepool_refresh_test.go):TestPoolRefreshesOldUnclaimedMembersTestPoolAgesOnOneClockReadingTestPoolReapsIdleClaimsLost/IdleTimeout, member deleted, workspace deleted. A recent bind with an oldlast_used_atkeeps a claim. An idle unclaimed member stays.LoststaysIdleTimeoutafter the member is gone.TestPoolIdleReapAfterCrashTestPoolIdleReapKeepsLaterUseTestPoolIdleCleanupNeedsLastUsedAt(zero, omitted)last_used_at: no delete build, and the workspace stays.TestPoolIdleCheckNeedsEvidencelast_used_atreaps nothing. A recorded check whose claim write failed deletes nothing, and a later delete of the member deletes the workspace as usual.TestPoolIdleReapSkipsForeignMemberTestCoderWorkspacePoolAPIValidation(3 new rows)Mutation checks. Each mutation removes one control, and the named test fails (19 of 19 killed):
TestPoolRefreshesOldUnclaimedMembersTestPoolRefreshesOldUnclaimedMembersTestPoolRefreshesOldUnclaimedMembersTestPoolReapsIdleClaimslast_used_atcheck droppedTestPoolReapsIdleClaimsboundTimecheck droppedTestPoolReapsIdleClaimsTestPoolReapsIdleClaimsTestPoolIdleReapAfterCrashTestPoolIdleReapKeepsLaterUseTestPoolIdleCheckNeedsEvidenceTestPoolIdleCheckNeedsEvidenceTestPoolIdleReapKeepsLaterUselast_used_atcounts as oldTestPoolIdleCheckNeedsEvidenceTestPoolAgesOnOneClockReadingTestPoolIdleReapSkipsForeignMemberlast_used_atdeletesTestPoolIdleCleanupNeedsLastUsedAtTestPoolIdleReapKeepsLaterUseLocal gates on the head:
make verify-vendor test build lint test-integration docs-reference-check,mkdocs build --strict, markdownlint-cli2 and cspell on the changed docs.make codegen manifests build-installer docs-referenceleaves no diff.A root verifier blocked 6fe263e on two findings, both fixed in 8daca14 with tests that failed first: a last read without
last_used_atstill deleted, and the final claim message of a kept workspace said it was deleted. The verifier's own probes (crashes at every idle write, use at the last read, refresh with bound and used members, pool gone before the commit, foreign control plane, orphanLostwithout evidence) pass on the fix.A gpt-6.1-sol reviewer reviewed the code before this PR opened. Its five findings shaped the design above: the use check at cleanup, the orphan path, the owner and control-plane match, one clock reading per reconcile, and a zero
last_used_at.Kind: this PR adds no Kind phase. The CI E2E job must still pass P1 to P8 and the lifecycle phases on the final head. PR 15b adds P10 to P12.
Risks
last_used_atabout once a minute, so a session that starts less than a minute before the delete build can still be lost. This is the same limit as the unclaimed last-use check (plan 6.5).last_used_at, keeps the workspace in Coder, and the claim staysLost. The claim message names the workspace and its owner. The workspace is then not managed, and the how-to points admins at thekwp-cleanup./scale-only access cannot enable it.maxWorkspacesfor the replacement. At the cap, an old member stays claimable and is not replaced, as for a version rollout.countPoolMembers,bindClaims, anddeleteUnclaimedMemberstake the reconcile's single clock reading. Without the new fields,poolMemberCurrentequals the old input check, and the requeue result is the same as before. The existing pool envtests pass unchanged.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh