Skip to content

🤖 feat: replace old unclaimed pool workspaces and reap idle claimed ones, both opt-in (#153 PR 15a) - #447

Merged
ThomasK33 merged 7 commits into
mainfrom
feat/workspace-pool-refresh-idle
Nov 1, 2026
Merged

ThomasK33 merged 7 commits into
mainfrom
feat/workspace-pool-refresh-idle

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Summary

CoderWorkspacePool gets two opt-in fields. spec.maxUnclaimedAgeSeconds replaces unclaimed workspaces that have been ready too long, new before old. spec.claims.idleTimeoutSeconds deletes a claimed workspace once both its bind and its Coder last_used_at are older than the timeout, and the claim becomes Lost with reason IdleTimeout. Without either field, nothing changes.

Refs #153

This is plan PR 15a, the first half of a split. PR 15 grew past the plan's 550-line limit, so the Kind phases P10 to P12 (scale to 0, KEDA, version rollout) and their offline stubs follow in PR 15b. 15b waits for this PR and for PR 14 (#446, Share mode). Of this PR and #446, the one that lands second regenerates the generated files.

Background

Plan 6.6 and decision D9 (in the #153 plan):

  1. Unclaimed workspaces never age on their own. A pool that follows version.active: true already replaces outdated members new before old. maxUnclaimedAgeSeconds makes "ready for too long" one more reason to count as outdated, so the same order applies.
  2. A claimed workspace keeps a normal autostop deadline, but a template with a default TTL of 0 has none (risk R12). Idle reaping is the only automatic limit there. It deletes a claimant's workspace and its data, so it is opt-in (D9 option B), and it never touches unclaimed workspaces, which are idle by design.

Implementation

All new logic is in internal/controller/coderworkspacepool_refresh.go.

  • poolMemberCurrent adds the age check to the existing "pins the resolved inputs" check. Counting, binding (current before outdated), and the delete order use it. Each pool reconcile reads the clock once and passes that time to all three, so a member that ages in the middle of a reconcile counts the same in every step.
  • The pool requeues when the next member ages (poolRefreshWake). The old tail switch now computes one requeue and takes the earlier of it and the wake. The existing polls are unchanged.
  • reapIdleClaims only looks at claim holders whose label, owner, and control plane match the pool. A candidate's bind must be older than the timeout first. Then the pool reads the workspace in Coder with the member's identity guard. A failed read, a foreign or deleted workspace, or a zero last_used_at reaps nothing.
  • A reap runs in this order, with one write per step:
    1. The pool stores the last_used_at it checked on the member (annotation coder.com/idle-check-last-used-at).
    2. The pool sets the claim to Lost/IdleTimeout. Lost is final.
    3. The pool deletes the member with UID and resourceVersion preconditions.
  • finishIdleReaps deletes the holder of any Lost/IdleTimeout claim after a crash or a 409. It also runs on the orphan-claim path, so a pool delete between step 2 and step 3 strands nothing. A holder without the recorded check is never deleted.
  • In the member cleanup, the delete steps of a reaped member compare their own last Coder read with the recorded value. A newer last_used_at, or none at all (zero or left out), releases the member without a delete build, and the workspace stays in Coder, as for unclaimed members since CoderWorkspacePool: use after the last read can still be bound or deleted on a delete retry #409.
  • Lost is final, so the claim message written before the delete covers both outcomes: the pool deletes the workspace unless its last check before the delete sees a use, and a kept workspace stays in Coder as <kwp- name>, owned by <owner> (the owner name comes from the deciding Coder read). The baseline counts only while the claim, read without the cache, is Lost/IdleTimeout and not being deleted. So a recorded check whose claim write failed, a claim deletion, or a person's delete of the member deletes the workspace as before.
  • Docs: the how-to describes both fields, the data-loss warning, the R12 cost note, and the extra Coder reads.

Validation

New envtests (coderworkspacepool_refresh_test.go):

Test Proves
TestPoolRefreshesOldUnclaimedMembers Nothing ages without the field. With it, two replacements are built first, old members go oldest first only as replacements get ready, claimable ready members stay at 2 throughout, and the pool wakes at the next expiry (exactly 1 h).
TestPoolAgesOnOneClockReading A clock that jumps 1 h per reading: a member that ages mid-reconcile still leaves room for its replacement after a bind.
TestPoolReapsIdleClaims No reap without the field. A use within the timeout keeps a claim (wake at 50 min). Both timestamps old: Lost/IdleTimeout, member deleted, workspace deleted. A recent bind with an old last_used_at keeps a claim. An idle unclaimed member stays. Lost stays IdleTimeout after the member is gone.
TestPoolIdleReapAfterCrash A failed member delete after the claim write is finished by the next reconcile, even with the timeout removed.
TestPoolIdleReapKeepsLaterUse A crash, then a pool delete: the orphan path deletes the member. A use after the check keeps the workspace in Coder, with no delete build. The claim message names the kept workspace and its owner.
TestPoolIdleCleanupNeedsLastUsedAt (zero, omitted) The identity guard reads a full answer, but the delete step's own read has no last_used_at: no delete build, and the workspace stays.
TestPoolIdleCheckNeedsEvidence A zero last_used_at reaps nothing. A recorded check whose claim write failed deletes nothing, and a later delete of the member deletes the workspace as usual.
TestPoolIdleReapSkipsForeignMember A same-name pool with another owner never reaps the old pool's bound member, even with a drifted label.
TestCoderWorkspacePoolAPIValidation (3 new rows) Bounds: 3600 and 2592000 accepted, 3599 and 599 refused.

Mutation checks. Each mutation removes one control, and the named test fails (19 of 19 killed):

# Mutation Killed by
M1 Age ignored TestPoolRefreshesOldUnclaimedMembers
M2 Aged members deleted at once TestPoolRefreshesOldUnclaimedMembers
M3 No wake at the next expiry TestPoolRefreshesOldUnclaimedMembers
M4 Reap without the field (default 600) TestPoolReapsIdleClaims
M5 last_used_at check dropped TestPoolReapsIdleClaims
M6 boundTime check dropped TestPoolReapsIdleClaims
M7 Unclaimed members reaped TestPoolReapsIdleClaims
M8 No crash recovery TestPoolIdleReapAfterCrash
M9 No cleanup use check TestPoolIdleReapKeepsLaterUse
M10, M10b The recorded check alone counts, or the claim state is ignored TestPoolIdleCheckNeedsEvidence
M10c The finish step ignores the claim state TestPoolIdleCheckNeedsEvidence
M11 No finish on the orphan path TestPoolIdleReapKeepsLaterUse
M12 Zero last_used_at counts as old TestPoolIdleCheckNeedsEvidence
M13 Bind reads its own clock TestPoolAgesOnOneClockReading
M14 Owner check dropped TestPoolIdleReapSkipsForeignMember
M15 A last read without last_used_at deletes TestPoolIdleCleanupNeedsLastUsedAt
M16 The claim message hides the kept workspace TestPoolIdleReapKeepsLaterUse

Local gates on the head: make verify-vendor test build lint test-integration docs-reference-check, mkdocs build --strict, markdownlint-cli2 and cspell on the changed docs. make codegen manifests build-installer docs-reference leaves no diff.

A root verifier blocked 6fe263e on two findings, both fixed in 8daca14 with tests that failed first: a last read without last_used_at still deleted, and the final claim message of a kept workspace said it was deleted. The verifier's own probes (crashes at every idle write, use at the last read, refresh with bound and used members, pool gone before the commit, foreign control plane, orphan Lost without evidence) pass on the fix.

A gpt-6.1-sol reviewer reviewed the code before this PR opened. Its five findings shaped the design above: the use check at cleanup, the orphan path, the owner and control-plane match, one clock reading per reconcile, and a zero last_used_at.

Kind: this PR adds no Kind phase. The CI E2E job must still pass P1 to P8 and the lifecycle phases on the final head. PR 15b adds P10 to P12.

Risks

  • Idle reaping deletes a claimant's workspace and its data. It is opt-in, and the docs warn about it. Coder writes last_used_at about once a minute, so a session that starts less than a minute before the delete build can still be lost. This is the same limit as the unclaimed last-use check (plan 6.5).
  • A use after the idle check, or a last read without last_used_at, keeps the workspace in Coder, and the claim stays Lost. The claim message names the workspace and its owner. The workspace is then not managed, and the how-to points admins at the kwp- cleanup.
  • Anyone who can edit a pool can enable idle reaping for its existing claims. The same people can already delete the pool's claims, and with them the workspaces. /scale-only access cannot enable it.
  • Refresh needs room under the owner's maxWorkspaces for the replacement. At the cap, an old member stays claimable and is not replaced, as for a version rollout.
  • Changes to existing paths: countPoolMembers, bindClaims, and deleteUnclaimedMembers take the reconcile's single clock reading. Without the new fields, poolMemberCurrent equals the old input check, and the requeue result is the same as before. The existing pool envtests pass unchanged.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

…aces (#153 PR 15)

Change-Id: I09245332cca3086a4ace22b0a00e2511e0243b04
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…kspace pool how-to

Change-Id: Icce5678e04430e30f78a6f00b4ad1d2a3eaeb87d
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…without the pool, one clock reading per pool reconcile

Change-Id: I70f6e4ddba4e44f456c0b7f212a30b7076201b4c
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…s its workspace

Change-Id: Ib066d695208202dc191eb1b265efbd24727be993
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I3075167eddd9a920362b432857ae8650de84068e
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I374499ecf1cfbaa92dd4ca5c91655cfcbbfea3e6
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T23:29:07.149052Z 8daca14 Manual request
🔒 Security Review ✅ Completed 2026-10-31T23:29:27.628098Z 8daca14 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6fe263e533

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…and name a kept workspace in the claim message

Change-Id: I2eecb71f24b498d924ecf80c6292cce4efb5c3ba
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 8daca14538

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8daca14538

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant