[ThomasK33]

Summary

The Kind E2E pool step now also runs phases P10 to P12 of the #153 plan:

• P10: a pool at replicas: 0 makes no workspaces, and a claim still gets one: the pool builds a member on demand and binds it.
• P11: KEDA v2.21.0, installed from a pinned manifest with a checked SHA-256, moves a pool from 0 to 2 ready workspaces and back to 0 with a cron ScaledObject. The HPA that KEDA makes reports ScalingActive=True (risk R4).
• P12: a new active template version replaces both members new before old, and readyReplicas never drops below 2.

Every new phase has offline stub scenarios in hack/e2e-workspace-pool_test.sh. No controller code changes.

Refs #153

This is plan PR 15b, the second half of a split. PR 15 grew past the plan's 550-line limit. PR 15a (#447, merged as 19041e8) added refresh and idle reaping. This PR adds the Kind phases P10 to P12, the KEDA install, and their offline stubs. It is based on main after PR 14 (#446, Share mode, merged as 30e9cf8), so P9 runs too.

Background

Plan 10.4 lists P10 to P12 for PR 15, and the plan's risk table names R4: the HPA that KEDA makes can fail on a pool whose status.selector matches no pods. Only a real HPA catches selector and metric-type mistakes, so decision D8 (in the #153 plan) puts a KEDA cron phase in the Kind job (option B). If the whole job grows by more than 10 minutes over main, P11 moves to a nightly job (option C). The measurement is in Validation below.

Implementation

hack/e2e-workspace-pool.sh, after P9 and before the cleanup:

1. P10 creates the pool at replicas: 0 and checks that it is Ready with no members. A claim then binds to a member built on demand, and only that member exists. Deleting the claim deletes the member and its workspace in Coder.
2. P11 downloads the KEDA v2.21.0 manifest, refuses it unless its SHA-256 matches, applies it server-side, and waits for KEDA's deployments and its v1beta1.external.metrics.k8s.io APIService. A cron ScaledObject (minReplicaCount: 0, maxReplicaCount: 2, cooldownPeriod: 30, desiredReplicas: 2) opens a 3-minute window at the next minute. The driver waits for spec.replicas 2 with 2 ready members, then for KEDA's HPA with ScalingActive=True. It checks that the HPA targets the CoderWorkspacePool with an External AverageValue metric. Then it waits for 0 and checks that the scaled-up workspaces are deleted in Coder. It removes the ScaledObject, the APIService first, then the rest of KEDA. A stale APIService stalls namespace deletion.
3. P12 scales the pool to 2 and adds a comment to spec.files["main.tf"] of the fixture CoderTemplate. That update imports and activates a new version. The driver samples readyReplicas every 2 s and fails at once if a sample is below 2. It passes when both live members pin the new version and the pool is ready. The driver then checks that the old workspaces are deleted in Coder, deletes the pool, and checks that the new workspaces are deleted too.
4. The receipt adds on_demand_member, keda_up_seconds, keda_down_seconds, keda_hpa, and rollout_ready_samples. sha256sum joins the tool check.

hack/e2e-workspace-pool_test.sh: the fake pool controller now follows KEDA's cron window, builds a member on demand for a waiting claim, and replaces outdated members new before old. The pass scenario must run P1 to P12 and leave no ScaledObject or KEDA behind. Six new scenarios each break one behavior, and the driver must fail with the check that covers it: no-on-demand, keda-tampered, hpa-not-active, keda-stays-up, rollout-dips, rollout-stays-old.

.github/workflows/ci.yaml: the pool step name lists the new phases, and the failure dump adds the ScaledObjects, the HPAs, and the KEDA pods.

docs/how-to/workspace-pools.md: one paragraph on the autoscaler contract. The pool's selector matches no pods, so an HPA needs External or Object metrics with AverageValue. KEDA's HPAs do that, and the Kind test proves 0 to 2 to 0. PR 16 owns the rest of the pool docs.

Validation

CI on 88767b9 (E2E job): P1 to P12 pass, and every required check passes.

Phase	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12
Seconds	5	6	6	7	13	49	6	33	96	13	294	48

• P10: the claim bound e2e-pool-v2pz9, a member built on demand.
• P11 (risk R4): KEDA scaled the pool 0 to 2 in 67 s and 2 to 0 in 189 s. Its HPA targets the CoderWorkspacePool with an AverageValue metric and reports ScalingActive=True with reason ValidMetricFound.
• P12: all 16 readyReplicas samples during the rollout were 2.

Job growth (D8): the E2E job took 22m54s. The #446 E2E job on its final PR head took 15m29s, and its merge-queue E2E job took 15m45s. That is 7m25s and 7m09s of growth. The pool step grew by 6m00s (3m36s to 9m36s), and P10 to P12 account for 5m55s of it. The growth is under 10 minutes, so P11 stays in the PR job (D8 option B), and no nightly job is needed.

Local Kind replica of the CI job on the same tree (image built from this tree): P1 to P12 pass.

Phase	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12
Seconds	5	12	11	10	15	109	12	44	148	24	276	103

In that run, KEDA scaled the pool 0 to 2 in 57 s and 2 to 0 in 179 s. Its HPA reported ScalingActive=True with reason ValidMetricFound and an AverageValue target. Every readyReplicas sample during the rollout was 2.

• make test-scripts passes (the pool driver: all scenarios, including the 6 new ones).
• make verify-vendor test build lint test-integration docs-check pass on the head.
• shellcheck -S warning on both scripts, actionlint on ci.yaml, markdownlint-cli2 and cspell on the changed doc: clean.

Dogfood (plan row 15): the KEDA cron window with kubectl get hpa every 15 s, and the pool's workspaces in the Coder UI.

Risks

• P11 downloads the KEDA manifest from GitHub releases at job time. A GitHub outage fails the step. The checked SHA-256 refuses a changed file.
• P11 waits for a real cron window, so it takes about 5 minutes (294 s in CI). The job grew by about 7 minutes in total, under the D8 limit of 10.
• If the step fails in P10 to P12, the driver leaves its objects and KEDA in place for the failure dump, as the earlier phases do. The job stops there, and the Kind cluster is discarded.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T02:06:41.751598Z	88767b9	PR opened
🔒 Security Review	✅ Completed 2026-11-01T02:03:34.413304Z	88767b9	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[ThomasK33]

@codex security review

[ThomasK33]

Dogfood (plan row 15)

I ran plan 14.2 steps 2 to 5, 7, and 8 by hand on a Kind cluster with the operator image built from this PR's tree (Coder, the agent fixture template e2e-pool-agent, the KEDA v2.21.0 manifest). The recording is 90 s, sped up from about 6 minutes.

1. A password user dogfood-owner owns the pool workspaces. The admin allows that user and the template in spec.workspacePools.
2. A pool ci-runners with replicas: 2 becomes Ready. kubectl scale --replicas=3 gives 3 ready members, and the /scale subresource reports 3.
3. A Reserve claim build-1234 binds at once. coder ssh dogfood-owner/<kwp- name> with the owner's token prints hello from coder-… from the workspace pod.
4. The Coder UI shows the owner's 4 running kwp- workspaces: 3 unclaimed and 1 claimed.

5. KEDA's cron window: I installed KEDA from the pinned manifest (SHA-256 checked), scaled the pool to 0, and created a cron ScaledObject (minReplicaCount: 0, maxReplicaCount: 2, cooldownPeriod: 30, window minutes 46 to 49 UTC, desiredReplicas: 2). The recording prints the pool and kubectl get hpa every 15 s:
   • 01:45:30 to 01:45:46: DESIRED 0, the claimed workspace stays bound (BOUND 1). KEDA's HPA keda-hpa-ci-runners-cron targets CoderWorkspacePool/ci-runners with <unknown>/1 (avg).
   • 01:46:01: KEDA moves the pool from 0 to 1. At 01:46:17 the HPA reads 2/1 (avg) and moves it to 2. From 01:46:32 the pool is Ready with 2 ready members and the HPA reads 1/1 (avg).
   • 01:49:07: the window closed, and the HPA reads 0/1 (avg). At 01:49:38, after the 30 s cooldown, KEDA moves the pool back to 0.
   • The HPA JSON taken after the window shows the External AverageValue target. Its ScalingActive is False/ScalingDisabled at that point because KEDA disables its HPA while the target is at 0. Inside the window, the HPA computed a scale from its metric (the TARGETS column above). The Kind phase P11 in CI checks ScalingActive=True inside the window.
6. Cleanup: I deleted the ScaledObject, KEDA (APIService first), the claim, and the pool. No pool objects remain, and the owner has no workspaces in Coder.

The recording:

pr15-dogfood.mp4

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 88767b946b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.