Skip to content

CI: enforce patched Python floor in setup-python and simplify harden-pip #3259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
juenglin wants to merge 9 commits into from
Closed

CI: enforce patched Python floor in setup-python and simplify harden-pip #3259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
40a5fb0
CI: fail jobs that run an unpatched CPython interpreter
juenglin Oct 27, 2026
fa74283
CI: simplify harden-pip — drop hash-pinned bootstrap
juenglin Oct 28, 2026
4db603e
CI: force setup-python to resolve latest patch in pre-commit
juenglin Oct 28, 2026
9a474d5
CI: enforce patched Python floor in setup-python; drop guard script
juenglin Oct 28, 2026
0734199
CI: floor Python in the remaining harden-pip jobs
juenglin Oct 28, 2026
0e852e6
CI: resolve latest Python patch in matrix setup-python steps
juenglin Oct 28, 2026
cb8baed
CI: use check-latest instead of tilde-range for Windows setup-python
juenglin Oct 28, 2026
dd043cd
CI: reinstate interpreter guard inline in harden-pip
juenglin Oct 29, 2026
680f0dc
CI: floor Python to ~3.14.10 in lychee.yml and build-docs.yml
juenglin Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 34 additions & 9 deletions .github/actions/harden-pip/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@

name: Harden pip
description: >-
Install the security-fixed pip (26.2.2) from a hash-pinned wheel so that later
pip invocations do not fall back on the vendored certifi bundle that trusts the
compromised eMudhra/emSign intermediate CAs. The sha256 fragment guarantees the
bootstrap download's integrity even if the bootstrapping pip's trust store is
compromised. See https://github.com/NVIDIA/cuda-python/issues/3195 and
https://discuss.python.org/t/109576.
Update pip. Call sites must use setup-python with a floor spec that ensures
the interpreter's ssl module rejects the compromised eMudhra/emSign CA keys
(CVE-2026-115557; first fixed releases: 3.11.18, 3.12.16, 3.13.18, 3.14.10,
3.15.2). With a patched ssl module, pip install -U pip is safe and picks up
the latest pip, which uses truststore (the OS certificate store) by default.
See https://github.com/NVIDIA/cuda-python/issues/3195 and
https://discuss.python.org/t/109678.

inputs:
python:
Expand All @@ -19,10 +20,34 @@ inputs:
runs:
using: composite
steps:
- name: Install hash-pinned pip 26.2.2
- name: Check Python interpreter is patched (CVE-2026-115557)
shell: bash --noprofile --norc -euo pipefail {0}
env:
HARDEN_PIP_PYTHON: ${{ inputs.python }}
run: |
"${HARDEN_PIP_PYTHON}" -c '
import sys
v = sys.version_info
print("sys.version:", sys.version)
FIXED = {(3,11):18,(3,12):16,(3,13):18,(3,14):10,(3,15):2}
s = (v.major, v.minor)
p = str(v.major)+"."+str(v.minor)+"."+str(v.micro)
if s in FIXED:
fl = str(v.major)+"."+str(v.minor)+"."+str(FIXED[s])
if v.micro < FIXED[s]:
print("::error::Python "+p+" is older than "+fl+", the first "+str(v.major)+"."+str(v.minor)+" release that rejects the compromised eMudhra CA keys (CVE-2026-115557).")
sys.exit(1)
Comment on lines +37 to +39

@coderabbitai coderabbitai Bot Oct 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Stability & Availability · Heavy lift

Supported by static analysis

Script executed:

rg -n -C4 'harden-pip|python-version|check-latest|allow-prereleases' .github/workflows/build-wheel.yml .github/workflows/test-sdist-windows.yml .github/workflows/test-wheel-windows.yml
sed -n '1,70p' .github/actions/harden-pip/action.yml

Repository: NVIDIA/cuda-python

Length of output: 12178

Script executed:

set -u
printf '%s\n' '--- relevant commit messages ---'
git log -8 --format='%H%n%s%n%b' -- .github/actions/harden-pip/action.yml .github/workflows/build-wheel.yml .github/workflows/test-sdist-windows.yml
printf '%s\n' '--- actions/python-versions manifest: Windows 3.12 entries ---'
python3 - <<'PY'
import json
import urllib.request
url = "https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json"
try:
    with urllib.request.urlopen(url, timeout=15) as r:
        data = json.load(r)
except Exception as exc:
    print(f"manifest fetch failed: {exc}")
else:
    found = []
    for item in data:
        version = str(item.get("version", ""))
        if version.startswith("3.12"):
            files = item.get("files", [])
            windows = [f for f in files if str(f.get("platform", "")).lower().startswith("win")]
            if windows:
                found.append((version, windows))
    print(json.dumps(found, indent=2))
PY

Repository: NVIDIA/cuda-python

Length of output: 23827

important: Restore the Windows CI path.

Both Windows workflows request Python 3.12 with check-latest: true and run harden-pip. The Windows manifest resolves that request to Python 3.12.10, but the guard rejects every 3.12 interpreter below 3.12.16. These jobs therefore fail before their build or test steps. The fail-closed behavior is intentional, but it remains a mergeability problem.

Provide patched Windows interpreters for the required minor versions, or retain a secure hash-pinned bootstrap for affected versions.

print("Python "+p+": ssl module carries the fix (first fixed: "+fl+").")
elif s > max(FIXED):
print("::warning::Python "+p+": no fixed release known for "+str(v.major)+"."+str(v.minor)+" yet.")
else:
print("::error::Python "+str(v.major)+"."+str(v.minor)+" has no release that rejects the compromised eMudhra CA keys (CVE-2026-115557).")
sys.exit(1)
'

- name: Update pip
shell: bash --noprofile --norc -xeuo pipefail {0}
env:
HARDEN_PIP_PYTHON: ${{ inputs.python }}
run: |
"${HARDEN_PIP_PYTHON}" -m pip install \
"pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
"${HARDEN_PIP_PYTHON}" -m pip install -U pip

@coderabbitai coderabbitai Bot Oct 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical · Security & Privacy · Quick win

important: Keep the verified bootstrap for unpatched Python. .github/workflows/build-wheel.yml (Line 106) and .github/workflows/test-sdist-windows.yml (Line 58) request 3.12 without a floor. The current manifest resolves Windows to 3.12.10, below the 3.12.16 fix. (raw.githubusercontent.com)

With an older bundled pip, a network attacker can use a compromised CA key to replace the unpinned download and execute code in CI. Restore the hash-pinned bootstrap and ensure callers use a patched interpreter or fail closed. (discuss.python.org)

As per path instructions: “focus on … security boundaries.”

Source: Path instructions

2 changes: 1 addition & 1 deletion .github/workflows/build-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ jobs:
if: ${{ !inputs.is-release && startsWith(github.ref_name, 'pull-request/') }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
python-version: "~3.14.10" # floor: first 3.14 with eMudhra ssl fix (CVE-2026-115557)

- name: Check rendered documentation links through pre-commit
if: ${{ !inputs.is-release && startsWith(github.ref_name, 'pull-request/') }}
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/build-wheel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ jobs:
# WAR: setup-python is not relocatable, and cibuildwheel hard-wires to 3.12...
# see https://github.com/actions/setup-python/issues/871
python-version: "3.12"
# Resolve newest patch from manifest (not stale tool cache); tilde-range
# (~3.12.16) is not used because the Windows manifest may lag behind Linux.
check-latest: true

@coderabbitai coderabbitai Bot Oct 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Security & Privacy · Heavy lift

important: Keep an explicit patched-version floor in these setup steps. check-latest: true selects the newest version available in the actions/python-versions manifest. It does not require 3.12.16 or the corresponding floor for each matrix series. If the Windows manifest lags during a security release, setup-python can install the previous patch, and harden-pip then runs with the pre-fix ssl module. Retain a floor range such as ~3.12.16 for fixed-version jobs, and add an offline version-floor assertion before harden-pip for matrix jobs and the other affected workflows. (github.com)

As per path instructions, this CI change must preserve the security boundary before networked package installation.

Also applies to: 454-454

Source: Path instructions

architecture: ${{ ((inputs.host-platform == 'linux-aarch64' || inputs.host-platform == 'win-arm64') && 'arm64') || 'x64' }}

- name: Harden pip
Expand Down Expand Up @@ -445,6 +448,10 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: ${{ startsWith(matrix.python-version, '3.16') }}
# Resolve the newest patch from the manifest rather than a stale tool
# cache; every matrix series' newest patch carries the eMudhra ssl fix
# (CVE-2026-115557), which a single floor spec cannot express here.
check-latest: true

- name: Harden pip
if: ${{ env.TEST_BINDINGS == 'true' || env.TEST_CORE == 'true' }}
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/ci-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ jobs:
with:
fetch-depth: 1

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "~3.12.16" # floor: first 3.12 with eMudhra ssl fix (CVE-2026-115557)

- name: Harden pip
uses: ./.github/actions/harden-pip

Expand Down
12 changes: 8 additions & 4 deletions .github/workflows/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ on:

env:
PY_VER: "3.14"
# Floor spec for setup-python: first 3.14 whose ssl module rejects the
# compromised eMudhra keys (CVE-2026-115557). PY_VER stays minor-only for
# env-vars/naming; PY_VER_FLOOR is used only as the setup-python spec.
PY_VER_FLOOR: "~3.14.10"
LOCAL_CTK: "1"
GPU: "a100"
DRIVER: "latest"
Expand Down Expand Up @@ -104,7 +108,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PY_VER }}
python-version: ${{ env.PY_VER_FLOOR }}
env:
# we use self-hosted runners on which setup-python behaves weirdly...
AGENT_TOOLSDIRECTORY: "/opt/hostedtoolcache"
Expand Down Expand Up @@ -252,7 +256,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PY_VER }}
python-version: ${{ env.PY_VER_FLOOR }}

- name: Set up MSVC
uses: step-security/msvc-dev-cmd@22c98154b708dbd743e6f27a933cf6ceba3305c4 # v1.13.1
Expand Down Expand Up @@ -391,7 +395,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PY_VER }}
python-version: ${{ env.PY_VER_FLOOR }}

- name: Set up mini CTK
uses: ./.github/actions/fetch_ctk
Expand Down Expand Up @@ -501,7 +505,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PY_VER }}
python-version: ${{ env.PY_VER_FLOOR }}

- name: Harden pip
uses: ./.github/actions/harden-pip
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/lychee.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
kind: [authored, rendered]
env:
PYTHONUTF8: "1"
# Reuse the same minor version for Python setup and cache fallback.
# Minor version used as the cache restore-key prefix (see setup-python step).
PYTHON_VERSION: "3.14"
steps:
- name: Checkout checked revision
Expand All @@ -70,7 +70,10 @@ jobs:
id: python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PYTHON_VERSION }}
# Floor spec ensures the interpreter's ssl module rejects the compromised
# eMudhra keys (CVE-2026-115557). PYTHON_VERSION stays minor-only ("3.14")
# for the cache restore-key prefix below.
python-version: "~3.14.10"

- name: Set pre-commit cache location
run: echo "PRE_COMMIT_HOME=${RUNNER_TEMP}/pre-commit" >> "${GITHUB_ENV}"
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/pre-commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
# install a Git commit hook. All other hooks, including Lychee, run.
SKIP: check-precommit-installed
PYTHONUTF8: "1"
# Minor version used by setup-python and as the cache restore-key prefix.
# Minor version used as the cache restore-key prefix (see setup-python step).
PYTHON_VERSION: "3.14"
steps:
- name: Enable Git symlinks (Windows, must precede checkout)
Expand All @@ -65,7 +65,10 @@ jobs:
id: python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ env.PYTHON_VERSION }}
# Floor spec ensures the interpreter's ssl module rejects the compromised
# eMudhra keys (CVE-2026-115557). PYTHON_VERSION stays minor-only ("3.14")
# for the cache restore-key prefix below.
python-version: "~3.14.10"

- name: Set pre-commit cache location
run: echo "PRE_COMMIT_HOME=${RUNNER_TEMP}/pre-commit" >> "${GITHUB_ENV}"
Expand Down
10 changes: 9 additions & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
python-version: "~3.12.16" # floor: first 3.12 with eMudhra ssl fix (CVE-2026-115557)

- name: Check versioned release notes exist
id: release-notes
Expand Down Expand Up @@ -342,6 +342,14 @@ jobs:
EXTRA_INDEX_URL: https://pypi.org/simple/
INDEX_NAME: TestPyPI
steps: &verify-pathfinder-steps
- name: Set up Python
# Floor the interpreter so the venv below inherits an ssl module that
# rejects the compromised eMudhra keys (CVE-2026-115557). setup-python
# puts its shim first on PATH, so `python3 -m venv` uses it.
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "~3.12.16" # floor: first 3.12 with eMudhra ssl fix

- name: Create verification environment
run: |
[[ -n "$VERSION" ]] || { echo "::error::Empty version output"; exit 1; }
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-sdist-linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
python-version: "~3.12.16" # floor: first 3.12 with eMudhra ssl fix (CVE-2026-115557)

- name: Harden pip
uses: ./.github/actions/harden-pip
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/test-sdist-windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ jobs:
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
# Resolve newest patch from manifest (not stale tool cache); tilde-range
# (~3.12.16) is not used because the Windows manifest may lag behind Linux.
check-latest: true

- name: Harden pip
uses: ./.github/actions/harden-pip
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/test-wheel-linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,10 @@ jobs:
with:
python-version: ${{ matrix.PY_VER }}
allow-prereleases: ${{ startsWith(matrix.PY_VER, '3.16') }}
# Resolve the newest patch from the manifest rather than a stale tool
# cache; every matrix series' newest patch carries the eMudhra ssl fix
# (CVE-2026-115557), which a single floor spec cannot express here.
check-latest: true
env:
# we use self-hosted runners on which setup-python behaves weirdly (Python include can't be found)...
AGENT_TOOLSDIRECTORY: "/opt/hostedtoolcache"
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/test-wheel-windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,10 @@ jobs:
with:
python-version: ${{ matrix.PY_VER }}
allow-prereleases: ${{ startsWith(matrix.PY_VER, '3.16') }}
# Resolve the newest patch from the manifest rather than a stale tool
# cache; every matrix series' newest patch carries the eMudhra ssl fix
# (CVE-2026-115557), which a single floor spec cannot express here.
check-latest: true

- name: Harden pip
# $/ resolves at the workflow's commit. ./ would resolve against the
Expand Down
Loading