Skip to content

Commit 834b14f

Browse files
matthewgrossmanmatthewgrossman
authored
ci(helm): retry chart pushes when GHCR briefly misses a new manifest (#4578)
helm push uploads the chart manifest by digest, then fetches it back to apply the version tag. GHCR sometimes returns 404 for a manifest it just accepted, failing the push with 'failed to perform "Tag" on destination: <digest>: not found'. This failed Release Dev on Oct 5 and the v0.1.5-pre.2 Release Tag run after qualification passed. Retry each chart push up to four times with a short backoff. A retry finds the blobs and manifest already present and only applies the tag. Closes #4577 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent c38e279 commit 834b14f

1 file changed

Lines changed: 32 additions & 2 deletions

File tree

‎.github/actions/release-helm-oci/action.yml‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,8 +94,24 @@ runs:
9494
shell: bash
9595
run: |
9696
set -euo pipefail
97+
# Helm pushes the manifest by digest, then fetches it back to tag it.
98+
# GHCR can briefly return 404 for a manifest it just accepted, so
99+
# retry. Re-pushing the same archive is idempotent.
100+
push_chart() {
101+
local attempt
102+
for attempt in 1 2 3 4; do
103+
if helm push "$1" oci://ghcr.io/nvidia/openshell; then
104+
return 0
105+
fi
106+
if [ "${attempt}" -lt 4 ]; then
107+
echo "::warning::helm push of $(basename "$1") failed (attempt ${attempt}); retrying in $((attempt * 10))s."
108+
sleep $((attempt * 10))
109+
fi
110+
done
111+
return 1
112+
}
97113
for archive in /tmp/helm-charts/*.tgz; do
98-
helm push "${archive}" oci://ghcr.io/nvidia/openshell
114+
push_chart "${archive}"
99115
done
100116
101117
- name: Push SHA-pinned chart
@@ -107,6 +123,20 @@ runs:
107123
shell: bash
108124
run: |
109125
set -euo pipefail
126+
# See "Push Helm chart to GHCR OCI" for why pushes retry.
127+
push_chart() {
128+
local attempt
129+
for attempt in 1 2 3 4; do
130+
if helm push "$1" oci://ghcr.io/nvidia/openshell; then
131+
return 0
132+
fi
133+
if [ "${attempt}" -lt 4 ]; then
134+
echo "::warning::helm push of $(basename "$1") failed (attempt ${attempt}); retrying in $((attempt * 10))s."
135+
sleep $((attempt * 10))
136+
fi
137+
done
138+
return 1
139+
}
110140
mkdir -p /tmp/sha-pin
111141
for source_dir in "${GATEWAY_CHART_DIR}" "${WORKSPACE_CHART_DIR}"; do
112142
chart_name="$(basename "${source_dir}")"
@@ -117,5 +147,5 @@ runs:
117147
helm package "${sha_chart_dir}" --destination /tmp/sha-pin
118148
done
119149
for archive in /tmp/sha-pin/*.tgz; do
120-
helm push "${archive}" oci://ghcr.io/nvidia/openshell
150+
push_chart "${archive}"
121151
done

0 commit comments

Comments
 (0)