feat(arc): add an optional reaper for runner pods that outlive their job

.github/workflows/runner-reaper.yml

@@ -0,0 +1,43 @@
+name: Runner reaper
+
+# Runs the ARC role's runner reaper in a kind cluster (tests/runner_reaper/run.sh): ARC's EphemeralRunner CRD, the role's rendered CronJob, RBAC and ConfigMap, the pull Secret renewal's image built from this checkout, and a stand-in for GitHub's API. Shows a runner whose job ran past the bound and one whose run was cancelled being deleted, and an idle runner and a busy one being left alone. Only on changes that can affect the reaper.
+#
+# GitHub-hosted ubuntu-latest has Docker, kind, kubectl, helm and jq already.
+
+on:
+  pull_request:
+    paths:
+      - roles/github_runner_arc/**
+      - pull-secret-renewer/**
+      - tests/runner_reaper/**
+      - .github/workflows/runner-reaper.yml
+  workflow_dispatch:
+
+concurrency:
+  group: runner-reaper-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  reaper:
+    name: Reaper in kind
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Install Ansible
+        run: |
+          set -euo pipefail
+          if ! python3 -m venv "${RUNNER_TEMP}/venv" 2>/dev/null; then
+            sudo apt-get update -qq
+            sudo apt-get install -y -qq python3-venv
+            python3 -m venv "${RUNNER_TEMP}/venv"
+          fi
+          "${RUNNER_TEMP}/venv/bin/pip" install --quiet --prefer-binary ansible-core
+          echo "${RUNNER_TEMP}/venv/bin" >> "$GITHUB_PATH"
+
+      - name: Run the reaper test
+        run: tests/runner_reaper/run.sh

pull-secret-renewer/Dockerfile

@@ -4,6 +4,7 @@ FROM alpine:3.20
 RUN apk add --no-cache bash curl jq kubectl openssl
 
 COPY scripts/renew-pull-secret.sh /app/renew-pull-secret.sh
+COPY roles/github_runner_arc/files/github-app-token.sh /app/github-app-token.sh
 RUN chmod +x /app/renew-pull-secret.sh
 
 # A non-root user by number, so the CronJob's runAsNonRoot check can verify it without reading /etc/passwd.

roles/github_runner_arc/README.md

@@ -44,6 +44,8 @@ Upgrading ARC on a live fleet. The new controller deletes every scale set whose
 - `github_runner_arc_heartbeat_gist_id`: install the fleet-health platform from this host, refreshing this gist. `github_runner_arc_heartbeat_bootstrap_gist`, `github_runner_arc_heartbeat_gist_description` and `github_runner_arc_heartbeat_gist_consumer` control the bootstrap described above.
 - `github_runner_arc_autoscaler_usable_budget_gi`, `github_runner_arc_autoscaler_max_ceiling`, `github_runner_arc_autoscaler_floor`: required when a profile sets `autoscale: true` and the platform is installed. The floor must equal the autoscaled profile's `maxRunners`, which every Helm upgrade reverts to. The other `github_runner_arc_autoscaler_*` and `github_runner_arc_heartbeat_*` settings have defaults; see `defaults/main.yml`.
 - `github_runner_arc_node_recovery_enabled` (default `true`), `github_runner_arc_node_recovery_after_seconds` (default 120), `github_runner_arc_node_recovery_poll_seconds` (default 15), `github_runner_arc_node_recovery_image`: the node recovery watcher (see [Node recovery](#node-recovery)).
+- `github_runner_arc_runner_reaper_enabled` (default `false`), `github_runner_arc_runner_reaper_max_job_seconds` (default 432000, five days), `github_runner_arc_runner_reaper_completed_grace_seconds` (default 300), `github_runner_arc_runner_reaper_schedule` (default every 5 minutes), `github_runner_arc_runner_reaper_dry_run` (default `false`), `github_runner_arc_runner_reaper_image` (default the renewal's image): the runner reaper (see [Runner reaper](#runner-reaper)).
+- `github_runner_arc_runner_active_deadline_seconds` (default unset): `activeDeadlineSeconds` on every runner pod (see [Runner reaper](#runner-reaper)).
 - `github_runner_arc_app_setup_*`, `github_runner_arc_app_manifest_code`, `github_runner_arc_app_private_key_path`: inputs to `playbooks/github_app_setup.yml` (see below), including `github_runner_arc_app_setup_write_secret`, `github_runner_arc_app_setup_secret_namespaces`, `github_runner_arc_app_setup_secret_name`, `github_runner_arc_app_setup_keep_private_key` and `github_runner_arc_app_setup_command`.
 
 ## Sizing
@@ -332,6 +334,30 @@ github_runner_arc_orgs:
 
 Every host that installs anything also installs a small watcher, the `node-recovery` Deployment in `github_runner_arc_platform_namespace`, so that a node that stops does not leave its pods `Terminating` for ever. ARC waits for a listener's old pod to go before starting a new one, so a listener on a stopped node otherwise stalls its scale set until the pod is force-deleted by hand. Once a node's `Ready` condition has been `Unknown` for `github_runner_arc_node_recovery_after_seconds`, the watcher gives it the `node.kubernetes.io/out-of-service=nodeshutdown:NoExecute` taint of Kubernetes' non-graceful node shutdown, which makes the control plane evict the node's pods and force-delete the terminating ones; it removes the taint when the node reports `Ready` again. It never acts on a node reporting `NotReady`, never on its own node, and never removes a taint it did not apply. Its ClusterRole allows `get`, `list` and `patch` on nodes and nothing else. `github_runner_arc_node_recovery_enabled: false` removes the watcher, its RBAC and any taint it left behind. Its image is pulled without a pull Secret, so it must be public. The repository README's Node recovery section explains the default threshold.
 
+## Runner reaper
+
+A runner pod can outlive its job: a step hangs and the run is cancelled but the runner never exits, or the run ends while the pod keeps its slot. With `github_runner_arc_runner_reaper_enabled: true` each scale-set profile's namespace gets a `runner-reaper` CronJob that finds such runners and deletes their EphemeralRunner, which the scale set then replaces. It is off by default.
+
+Each pass mints an installation token from the profile's App Secret restricted to `actions: read`, lists the namespace's EphemeralRunners, and reads the workflow run that ARC's listener recorded on each runner when it assigned it a job (`status.workflowRunId` and `status.jobRepositoryName`). It deletes the runner when:
+
+- the run is completed, whatever its conclusion, cancelled included;
+- the run is still going but the runner's own job (the job of any attempt of the run whose `runner_name` is the runner's name) is completed; or
+- that job has been running for longer than `github_runner_arc_runner_reaper_max_job_seconds`, counted from the job's `started_at`.
+
+A run or job only counts as completed once it has been completed for `github_runner_arc_runner_reaper_completed_grace_seconds`, since ARC removes a runner that finished normally within seconds and the reaper should not race it. The bound is the longest a job may legitimately run: its default is GitHub's own limit on a job's execution time on a self-hosted runner, five days, which nothing can exceed, so a fleet whose jobs are shorter should lower it to a little above its longest job's `timeout-minutes`. Every deletion is logged on the Job's pod with the runner, the run, the repository and the reason, and each pass ends with a count of runners deleted, kept, idle and not readable. `github_runner_arc_runner_reaper_dry_run: true` logs what a pass would delete and deletes nothing.
+
+Its safety properties:
+
+- A runner with no recorded run has not been assigned a job, and an ephemeral runner never takes a second one, so an idle runner is never looked up or deleted, and neither is a runner already being deleted. A runner whose job has been assigned but has not started is kept.
+- A runner is deleted only on a positive answer from GitHub. When the token cannot be minted (GitHub unreachable, the App Secret unusable or the permission missing) or the runners cannot be listed, the pass deletes nothing and its Job fails with the reason. When one runner's run or jobs cannot be read, that runner is left alone, the rest are still judged, and the Job fails at the end. A failed pass is not retried; the next scheduled one tries again.
+- Its Role allows `list` and `delete` on `ephemeralrunners` in its own namespace and nothing else. The App Secret reaches the pod as a volume, the token never appears in a command's arguments or the log, and the pod runs as a non-root user with a read-only root filesystem.
+
+The reaper needs the org's GitHub App to have the **Actions** repository permission set to **Read**, approved on its installation, and the installation must include every repository whose jobs the runners take. The App the role's [setup flow](#github-app-setup) creates asks only for organisation self-hosted runners (write), so add `actions: read` to `github_runner_arc_app_setup_permissions` for a new App, or add the permission to an existing App in its settings and have an organisation owner approve it. Until then every pass fails saying the App needs the `actions: read` permission, and deletes nothing; a run in a repository the installation does not include is reported as unreadable, and its runner is left alone.
+
+The script (`files/reap-runners.sh`, with the token minting it shares with the pull Secret renewal) is mounted from a ConfigMap and runs in the renewal's image, which already has bash, curl, jq, kubectl and openssl, so a change to it reaches the cluster on the next run of the role without a new image. Its image is pulled without a pull Secret, so it must be public. `github_runner_arc_runner_reaper_enabled: false` removes the CronJob, its ConfigMap and its RBAC.
+
+ARC keeps the pod of a runner that GitHub still reports as running a job, even once its EphemeralRunner is being deleted, until GitHub releases the job. A runner whose run is completed is therefore removed straight away, while one whose job is still running past the bound is removed once GitHub ends that job, by its `timeout-minutes` or a cancellation. `github_runner_arc_runner_active_deadline_seconds` is a backstop that needs no GitHub call: it sets `activeDeadlineSeconds` on every runner pod, after which the kubelet stops the pod whatever it is doing and ARC handles it as a failed runner. It counts from when the pod starts, including any time the runner waits idle for a job, so set it above the longest a runner may wait plus the longest job the fleet runs. Unset, the runner pods are rendered without it.
+
 ## GitHub App setup
 
 `playbooks/github_app_setup.yml` creates the App the scale sets authenticate as, through GitHub's manifest flow. The first run renders a local form that posts the manifest (organisation self-hosted runner write access, no webhook, private) to GitHub; an organisation owner submits it and copies the one-time code GitHub returns. The second run, with that code and a path for the key, exchanges the code, writes the private key there, waits while the App is installed on the organisation, and prints the `app_id` and `installation_id` to record.

roles/github_runner_arc/defaults/main.yml

@@ -154,7 +154,22 @@ github_runner_arc_node_recovery_poll_seconds: 15
 # Image the watcher runs. Pulled without a pull Secret, so it must be public.
 github_runner_arc_node_recovery_image: "ghcr.io/exadev/github-runner-node-recovery:1.25.0"
 
-# GitHub App creation through the manifest flow (tasks/app_setup.yml, run by playbooks/github_app_setup.yml). The App needs only organisation self-hosted runner write access; add packages: read to the permissions if its installation token should also pull private packages.
+# Runner reaper (see the role README's Runner reaper section, tasks/install_runner_reaper.yml and files/reap-runners.sh): a CronJob in each runner namespace that deletes an EphemeralRunner whose job's workflow run is completed or cancelled, whose own job is completed, or whose job has been running for longer than github_runner_arc_runner_reaper_max_job_seconds; the scale set replaces it. It reads the runs with an installation token from the org's GitHub App restricted to actions: read, so the App needs the Actions repository permission (read) on every repository whose jobs the runners take, approved on its installation. It never deletes a runner that has not been assigned a job, and deletes nothing when GitHub or the token is unavailable. false removes the CronJob, its RBAC and its ConfigMap.
+github_runner_arc_runner_reaper_enabled: false
+# A job running for longer than this has its runner deleted. The default is GitHub's own limit on a job's execution time on a self-hosted runner, five days, past which GitHub ends the job itself, so no job can legitimately run longer; a fleet whose jobs are all shorter sets it a little above its longest job's timeout-minutes.
+github_runner_arc_runner_reaper_max_job_seconds: 432000
+# How long a run or job must have been completed before its runner is deleted. ARC removes a runner whose job finished normally within seconds, so this keeps the reaper clear of a runner on its way out, at the cost of at most one more pass at the default schedule.
+github_runner_arc_runner_reaper_completed_grace_seconds: 300
+github_runner_arc_runner_reaper_schedule: "*/5 * * * *"
+# true logs what each pass would delete and deletes nothing, to see the reaper's verdicts on a fleet before trusting it.
+github_runner_arc_runner_reaper_dry_run: false
+# Image the CronJob runs the reaper in. It needs bash, curl, jq, kubectl and openssl, which the pull Secret renewal's image has; the reaper itself comes from a ConfigMap, so the image need not be rebuilt when it changes. Pulled without a pull Secret, so it must be public.
+github_runner_arc_runner_reaper_image: "{{ github_runner_arc_image_pull_secret_renewer_image }}"
+
+# A backstop that needs no GitHub call: activeDeadlineSeconds on every runner pod, after which the kubelet stops the pod whatever it is doing and ARC treats it as a failed runner. It counts from when the pod starts, including any time the runner waits idle for a job, so set it above the longest a runner may wait plus the longest job a fleet runs. Unset leaves the runner pods without one.
+github_runner_arc_runner_active_deadline_seconds:
+
+# GitHub App creation through the manifest flow (tasks/app_setup.yml, run by playbooks/github_app_setup.yml). The App needs only organisation self-hosted runner write access; add packages: read to the permissions if its installation token should also pull private packages, and actions: read if the runner reaper is on.
 github_runner_arc_app_setup_org: ""
 github_runner_arc_app_setup_name: ""
 github_runner_arc_app_setup_description: "Registers self-hosted GitHub Actions runners through Actions Runner Controller."

roles/github_runner_arc/files/github-app-token.sh

@@ -0,0 +1,59 @@
+# shellcheck shell=bash
+# Sourced, not run: mints a GitHub App installation token restricted to the permissions a caller names. Shared by the pull Secret renewal (scripts/renew-pull-secret.sh, which its image carries beside this file) and the runner reaper (files/reap-runners.sh, which its ConfigMap carries beside this file).
+#
+# The caller defines fail MESSAGE, which must not return, and sets umask 077 before calling. The JWT never appears in a command's arguments: curl reads it from a config file under the caller's private working directory.
+
+# GitHub rejects an App JWT whose exp is more than ten minutes after iat; iat is backdated a minute to absorb clock drift.
+GITHUB_APP_JWT_BACKDATE_SECONDS=60
+GITHUB_APP_JWT_LIFETIME_SECONDS=540
+
+github_app_b64url() {
+  openssl base64 -A | tr '+/' '-_' | tr -d '='
+}
+
+# Writes a curl config line for a header or user value. curl config values are double-quoted strings in which a backslash escapes the next character.
+curl_config() {
+  local option="$1" value="$2"
+  value="${value//\\/\\\\}"
+  value="${value//\"/\\\"}"
+  printf '%s = "%s"\n' "$option" "$value"
+}
+
+# github_app_mint_token APP_DIR API_URL PERMISSIONS WORK
+#
+# APP_DIR holds the App Secret's keys as files (github_app_id, github_app_installation_id, github_app_private_key). PERMISSIONS is the JSON object of permissions to restrict the token to, such as {"packages":"read"}. On success it writes the token, on one line, to WORK/token and its expiry to WORK/expires_at. Every refusal calls fail, naming the permissions the App needs when GitHub refuses them.
+github_app_mint_token() {
+  local app_dir="$1" api_url="$2" permissions="$3" work="$4"
+  local field app_id installation_id now header claims signature status wanted
+  for field in github_app_id github_app_installation_id github_app_private_key; do
+    [ -s "$app_dir/$field" ] || fail "the App Secret has no $field"
+  done
+  app_id="$(tr -d '[:space:]' < "$app_dir/github_app_id")"
+  installation_id="$(tr -d '[:space:]' < "$app_dir/github_app_installation_id")"
+  [[ "$app_id" =~ ^[0-9]+$ ]] || fail "the App Secret's github_app_id is not a number"
+  [[ "$installation_id" =~ ^[0-9]+$ ]] || fail "the App Secret's github_app_installation_id is not a number"
+  wanted="$(jq -r 'to_entries | map("\(.key): \(.value)") | join(", ")' <<< "$permissions")" || fail "the permissions to request are not a JSON object"
+
+  now="$(date +%s)"
+  header="$(printf '{"alg":"RS256","typ":"JWT"}' | github_app_b64url)"
+  claims="$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$((now - GITHUB_APP_JWT_BACKDATE_SECONDS))" "$((now + GITHUB_APP_JWT_LIFETIME_SECONDS))" "$app_id" | github_app_b64url)"
+  signature="$(printf '%s.%s' "$header" "$claims" | openssl dgst -sha256 -sign "$app_dir/github_app_private_key" -binary | github_app_b64url)" \
+    || fail "could not sign the App JWT with the App Secret's private key"
+  curl_config header "Authorization: Bearer ${header}.${claims}.${signature}" > "$work/github-app.curl"
+
+  status="$(curl -sS -o "$work/mint.json" -w '%{http_code}' -K "$work/github-app.curl" -X POST \
+    -H 'Accept: application/vnd.github+json' -H 'X-GitHub-Api-Version: 2022-11-28' \
+    --data "$(jq -c '{permissions: .}' <<< "$permissions")" \
+    "${api_url}/app/installations/${installation_id}/access_tokens")" || fail "could not reach the GitHub API"
+  rm -f "$work/github-app.curl"
+  if [ "$status" != "201" ]; then
+    fail "GitHub refused to mint an installation token restricted to ${wanted} (HTTP ${status}: $(jq -r '.message // empty' "$work/mint.json" 2>/dev/null)). The App needs the ${wanted} permission, approved on this installation"
+  fi
+  jq -r '.token // empty' "$work/mint.json" > "$work/token"
+  jq -r '.expires_at // empty' "$work/mint.json" > "$work/expires_at"
+  rm -f "$work/mint.json"
+  if [ ! -s "$work/token" ] || [ "$(wc -l < "$work/token")" -gt 1 ]; then
+    fail "GitHub's response held no token"
+  fi
+  [ -s "$work/expires_at" ] || fail "GitHub's response held no expiry"
+}