Repository navigation
feat(arc): add an optional reaper for runner pods that outlive their job - #68
Merged
Merged
Conversation
Move the JWT signing and token minting out of the pull Secret renewal into a sourced file in the role's files, so another in-cluster job can mint a token restricted to its own permissions the same way. The renewer image carries it beside the renewal script.
One pass lists a namespace's EphemeralRunners and, for each one ARC has assigned a job, reads the job's run from GitHub with an actions: read installation token. It deletes the runner when the run or its own job is completed, or when the job has run past a configured bound, and never touches an idle runner or one it could not judge.
…amespace Off by default. When on, each scale-set profile's namespace gets a CronJob that runs the reaper from a ConfigMap in the pull Secret renewal's image, with a Role that may only list and delete EphemeralRunners there and the App Secret mounted as a volume. Turning it off removes all of it.
github_runner_arc_runner_active_deadline_seconds puts the deadline on every runner pod as a backstop that needs no GitHub call. Unset leaves the runner pod spec as it was.
Applies ARC's CRDs and the role's reaper manifests to a kind cluster, records jobs on EphemeralRunners as ARC's listener does, and runs a pass against a stand-in for GitHub: the runner past the bound and the cancelled run's runner go, the idle and busy runners stay, the ServiceAccount can do nothing else, and a refused token deletes nothing.
Mearman
marked this pull request as ready for review
October 8, 2026 13:43
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🎉 This PR is included in version 1.26.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the optional runner reaper from the issue to the
github_runner_arcrole, plus theactiveDeadlineSecondsbackstop.With
github_runner_arc_runner_reaper_enabledon, each runner namespace gets a CronJob that runsfiles/reap-runners.shfrom a ConfigMap, in the pull Secret renewal's image (it already has bash, curl, jq, kubectl and openssl). Each pass mints an installation token from the App Secret restricted toactions: read, lists the namespace's EphemeralRunners, and for each runner ARC has recorded a workflow run on, reads that run. It deletes the runner when the run is completed (cancelled included), when the runner's own job in a still-running run is completed, or when that job has been running longer thangithub_runner_arc_runner_reaper_max_job_seconds. Completed runs and jobs only count once they have been completed for a grace period, so a runner exiting normally is left to ARC. Every deletion is logged with the run, the repository and the reason.A runner with no recorded run is idle and is never looked up or deleted, and nor is one already being deleted. A runner is only deleted on a positive answer from GitHub: if the token cannot be minted or the runners cannot be listed, the pass deletes nothing and fails; if one runner's run or jobs cannot be read, that runner is left alone, the others are still judged, and the pass fails at the end. The Role allows only
listanddeleteonephemeralrunners; the App Secret reaches the pod as a volume.The App needs the Actions repository permission (read) on the repositories whose jobs the runners take. The role's default App manifest only asks for organisation self-hosted runners (write), so an existing App needs that permission added and approved on its installation; until then each pass fails with a message naming it and deletes nothing.
The JWT signing and token minting moved out of the renewal script into
files/github-app-token.sh, which the renewer image and the reaper's ConfigMap both carry.One thing worth knowing: ARC 0.15 keeps the pod of a runner GitHub still reports as running a job, even after the EphemeralRunner is deleted, until GitHub releases the job. So for a job past the bound, the deletion takes effect once GitHub ends the job, while for a completed or cancelled run it takes effect straight away.
github_runner_arc_runner_active_deadline_secondsis the backstop that stops the pod regardless.Closes #67