Repository navigation
feat(arc): add an optional pull-through registry cache for dind runners - #65
Merged
Merged
Conversation
github_runner_arc_registry_cache_enabled installs a Deployment with one CNCF Distribution container per upstream registry, in proxy mode, behind one Service, and points every dind scale set at it without any change on the workflow side. dockerd reads a hosts.toml per registry from /etc/docker/certs.d, which lists the cache as a pull and resolve mirror ahead of the registry, and buildx reads the same mirrors from buildkitd.default.toml under BUILDX_CONFIG when it creates a docker-container builder, as docker/setup-buildx-action does. Both resolvers move on to the registry itself when the mirror refuses the connection or answers with an error, so a cache that is down slows a pull rather than failing it. The chart's dind container cannot take an extra mount, so with the cache on a dind profile has the chart's dind wiring written out in its values. Credentials are only read from Secrets in the cache's namespace, and a NetworkPolicy admits only the runner namespaces. With the cache off the values render exactly as before and the run removes what it added.
… kind tests/registry_cache/run.sh installs the cache as the role renders it in a kind cluster, with Docker Hub, ghcr.io and a private registry behind basic auth, and runs the chart's runner pod template as the role renders a dind profile. The step pulls from all three and builds through a docker-container builder created as setup-buildx-action creates one, and the cache's access logs must show every pull; with the cache scaled to nothing a second pod must still pull and build from the registries directly.
The role README describes the variables, how runner pods use the cache, which pull paths go through it and which bypass it, the written-out dind wiring and what a credential on the cache exposes; the top-level README summarises it and lists its integration test.
Runs tests/registry_cache/run.sh in kind on GitHub-hosted runners when the ARC role, its filters or the test change, as capabilities.yml does for capabilities.
…ered chart Helm 4 prints its Pulled and Digest notice for an OCI chart on standard output, which parses as a YAML mapping with no kind and stopped the AutoscalingRunnerSet lookup with a KeyError.
dockerd looks up an image's referrers on every pull, and containerd sends that lookup only to hosts with the referrers capability, so with a pull and resolve mirror it went to the registry alone. For a registry that needs a login the job has none, and the whole pull failed although the cache had served every manifest and blob.
The private registry's image is pushed with crane from inside the cluster rather than through a port-forward from the host, which a Docker daemon in a VM cannot reach, the platform priority class the cache runs at is created as the role creates it, Helm 4's pull notice is skipped when reading the chart, and the docker-container builder also builds from the private image.
Mearman
marked this pull request as ready for review
October 8, 2026 10:56
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🎉 This PR is included in version 1.25.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
github_runner_arc_registry_cache_enabledto thegithub_runner_arcrole: a pull-through cache that every Docker-in-Docker scale set pulls through, with nothing to change in workflows, Dockerfiles or jobs. It is off by default, and with it off the role renders exactly what it rendered before (the unit tests compare against the earlier rendering written out in full); turning it off again removes the cache, its storage, the per-namespace ConfigMaps and the written-out dind wiring.The cache is one Deployment with a CNCF Distribution (
registry:3) container per upstream registry in proxy mode, each on its own port of one Service, with persistent (PVC, size and class) or ephemeral (capped emptyDir) storage. Registries are data (github_runner_arc_registry_cache_registries, Docker Hub, ghcr.io and quay.io by default); each can name a Secret in the cache's namespace holding a read-only credential, which the container reads throughsecretKeyRef, so jobs pull that registry's private images without logging in. Inline credentials are rejected at validation, the role only compares the Secret's key names, and a NetworkPolicy admits only the runner namespaces.Runner pods use it through mirrors rather than a proxy. dockerd 29 (the chart's
docker:dind, which uses the containerd image store) resolves every registry through containerd'sConfigureHostswith/etc/docker/certs.das its hosts directory (mobydaemon/hosts.go), so ahosts.tomlper registry listing the cache ahead of the registry covers non-Hub registries too, for the daemon's pulls and for its built-in BuildKit. Adocker-containerbuilder created bydocker/setup-buildx-actiondoes its own pulls and inherits none of the daemon's settings, but buildx loadsbuildkitd.default.tomlfrom$BUILDX_CONFIGwhenbuildx createis given no config (buildxbuilder/builder.go), and setup-buildx-action passes none unless asked (src/context.ts), so the runner container getsBUILDX_CONFIGpointing at a writable directory holding the same mirrors. Both resolvers try the mirror, then the registry when the mirror refuses the connection or answers with an error, which is what makes it fail open. The mirror is also given thereferrerscapability: dockerd looks up an image's referrers on every pull, and with only pull and resolve containerd sent that lookup straight to the registry, which for a registry that needs a login failed the whole pull even though the cache had served every manifest and blob (the kind test caught this).The chart's dind container cannot take an extra mount, so with the cache on, a
container_mode: dindprofile has the chart's own dind wiring (Kubernetes 1.29 and later, dind as a native sidecar) written out in its values andcontainerModedropped. A values file that already writes out adindcontainer just gets the mount.I tested the mechanisms against real
docker:dind29.8.2 daemons with local pull-through caches, reading each cache's access log for the image's manifest requests (a fresh daemon per row, so nothing came from a local store):registry-mirrorsonlydocker pull, Hubdocker pull, ghcr.ioDOCKER_BUILDKIT=0), Hub / ghcr.iodocker build(buildx, docker driver), Hub / ghcr.iodocker buildx build --builder default, Hub / ghcr.ioFail-open with the mirror design: with the cache containers stopped, and with them replaced by a server answering every request with 503, every path above still pulled from both registries. A cache that accepts connections and never answers held one pull until containerd's own timeout before it fell back, which is why each cache container has a readiness probe: once it fails, the Service has no endpoints, connections are refused, and pulls fall back at once. The proxy design, by contrast, failed every pull once the proxy was stopped (
proxyconnect tcp: ... no route to host), so it would need a pod-local forwarder to fail open, would put a MITM CA in every runner pod, and still would not reach the docker-container builder, whose container environment carried no proxy settings. The mirror design needs no CA, no proxy and no forwarder, so I went with it.What bypasses the cache (and still works, uncached): registries not in the list; a
docker-containerbuilder created with its own--config/buildkitd-configinput, or one created withDOCKER_CONFIG/BUILDX_CONFIGchanged by the job; Docker run inside a job container rather than through the pod's daemon; a dind daemon switched back to the legacy graph-driver store, where only Docker Hub could be mirrored; and buildx versions that predate the default config file.Security notes are in the role README: anything reachable from the runner namespaces can pull what the configured credentials can read, so they should be read-only and scoped, and the cache holds those images on its volume.
tests/registry_cache/run.sh(run by the newregistry-cache.ymlworkflow on pull requests) installs the cache as the role renders it in kind, with Docker Hub, ghcr.io and an in-cluster registry behind basic auth whose credential the cache reads from a Secret, starts a pod from the chart's runner template as the role renders a dind profile with the cache on, pulls from all three withdocker pulland builds from all three through a docker-container builder created exactly as setup-buildx-action creates one, with no login anywhere, and checks the cache's access log for each; then it scales the cache to zero and checks a second pod still pulls and builds directly. It passes locally on arm64 (kind 0.33, Helm 4). The capabilities test had the same Helm 4 problem reading the rendered chart (Helm 4 prints its pull notice on stdout), fixed in its own commit.Closes #64.