Repository navigation
fix(kubernetes): delete sandboxes by UID, not revision - #5843
Merged
Merged
Conversation
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
4 tasks
TaylorMutch
approved these changes
Nov 20, 2026
matthewgrossman
deleted the
fix/k3s-ha-delete-recreate/matthewgrossman
branch
November 20, 2026 18:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Kubernetes driver could report a Sandbox CR as absent after a rejected DELETE, allowing the gateway to release the name while Kubernetes still held it. Delete by the original Kubernetes UID, not by its mutable
resourceVersion, and surface conflicts instead of treating them as absence.User scenario: A team runs a CI job in an OpenShell sandbox named
ci-buildon a Kubernetes gateway. The next run wants a clean sandbox but reusesci-buildso its scripts don’t have to change. It deletes the old sandbox, waits for its ID to disappear, then creates a new sandbox with the same name. This is replacement, not a rename. Before this fix, the wait could succeed but the new create could fail withAlreadyExistsbecause the old sandbox was still there.TL;DR: A Kubernetes status update could make DELETE return
409 Conflict. The driver called that “already gone,” so the gateway freed the name even though recreating it could still hit409 AlreadyExists. The fix deletes by the original UID only and surfaces any remaining conflict.Related Issue
Fixes the k3s-ha SSH host identity conformance flake seen in
v0.1.10-pre.3(job, artifact). Stop/start and SSH succeeded. Delete oftest-ixd1xhjz-hkreturned success at11:20:54.383593Z; after the old-ID SDK wait returned, recreation started at11:20:54.415222Zand got KubernetesAlreadyExistsfordefault--test-ixd1xhjz-hk. Attempt 2 passed on the same SHA (93761e1578).The driver looked up the CR by OpenShell sandbox ID, then sent DELETE with both UID and
resourceVersionpreconditions. A controller status update between lookup and DELETE can produce a 409 without deleting the original CR. The old driver turned that conflict intodeleted=false, which the gateway treated as proof of absence. This mechanism is reproduced by the regression; the artifact does not record the original Kubernetes DELETE response, so the CI diagnosis is inferred. This is distinct from #5265 (stale start, already in the tested SHA) and #5666 (a gateway row lingering after Kubernetes removal). Open #3001 adds a CLI wait on gateway state and would not detect false absence.Scope and follow-up
This PR fixes one Kubernetes driver path. It does not change the gateway or compute-driver contract.
DeleteSandboxResponse.deleteddescribes whether the request deleted a resource, but the gateway interpretsfalseas proof that the original sandbox is absent and releases its name without checking. Follow-up: #5844 tracks distinguishing confirmed absence from an ambiguous result and auditing other drivers, including Podman and VM.Changes
wait_deletedsees that gateway state, not the CR UID. Polling the same API again in testkit would not fix false absence.SandboxGuard::cleanup(); that separate async gap and the existing testkit name-only cleanup hooks are unchanged. Replacement-safe shared cleanup would require an atomic expected-ID delete, not a read followed by delete-by-name.Testing
cargo test --locked -p openshell-driver-kubernetes --lib(338 passed).--all-targets -- -D warnings), formatting, andgit diff --check./dev/kvm.Checklist