[threexc]

Instead of hard-coding our target versions, change our code in ci_scripts/, .github/workflows, skills/, and docs/development.md to support dynamically deriving a target Python build matrix for each package, which can be rolled out gradually as updates are performed (instead of having to bulk rewrite and trigger all existing workflows). Now, when a workflow sets the python-matrix parameter to true, use ci_scripts/pending_versions.py to derive the list based on what the release targets upstream. Also allow us to extend/exclude the lists on a per-package basis by setting the python-extend and python-exclude inputs.

Modify build-falcon.yml as a test change. The skills files indicate that the rollout should be gradual, so our existing automation can convert workflows over time as new releases are built.

[luhenry]

Can't we get the list of python configuration we should build in check_versions.py and store that information in the docs/packages/<pkg>.yml per-version, like:

- version: 1.2.3
  python-versions: [3.10, 3.11, 3.14, 3.14t] // the versions that upstream builds for that package

Then in .github/workflows/_setup.yml we still only have docs/packages/<pkg>.yml to check. I like the idea of python-exclude to make sure we can still only have a subset for things that don't work. I don't get the need for python-extend: though as there shouldn't be anything we build that upstream doesn't build already.

[justeph]

Can't we get the list of python configuration we should build in check_versions.py and store that information in the docs/packages/<pkg>.yml per-version, like:

- version: 1.2.3
  python-versions: [3.10, 3.11, 3.14, 3.14t] // the versions that upstream builds for that package

Then in .github/workflows/_setup.yml we still only have docs/packages/<pkg>.yml to check. I like the idea of python-exclude to make sure we can still only have a subset for things that don't work. I don't get the need for python-extend: though as there shouldn't be anything we build that upstream doesn't build already.

I really like the idea of having that info in docs/packages/<pkg>.yml.

[justeph]

why false by default?

[threexc]

The idea was to set it to false by default as part of a gradual rollout to our workflows - the skills/ directory gives the LLM instructions for converting workflows to use a dynamic matrix as it updates them to build new versions. That way existing workflows keep working as expected until there's a reason to trigger them again.

[luhenry]

For the transition, can't we simply generate it all the time and simply ignore the output in the workflows that haven't migrated yet?

[threexc]

@luhenry I've reworked it so that a matrix is always generated, but it's ignored if python-versions is missing because we're manually specifying a version list.

[threexc]

Can't we get the list of python configuration we should build in check_versions.py and store that information in the docs/packages/<pkg>.yml per-version, like:

- version: 1.2.3
  python-versions: [3.10, 3.11, 3.14, 3.14t] // the versions that upstream builds for that package

I like that idea. I'll work on a change to do that instead.

Then in .github/workflows/_setup.yml we still only have docs/packages/<pkg>.yml to check. I like the idea of python-exclude to make sure we can still only have a subset for things that don't work. I don't get the need for python-extend: though as there shouldn't be anything we build that upstream doesn't build already.

python-extend was meant as an opposite to python-exclude, e.g. for if we wanted to build free-threaded wheels for a project which doesn't, or something like that. It's unnecessary future-proofing at this point though, so I'll remove it.

[threexc]

Updated to address most feedback. Still left the questions about default versions to exclude and default python-matrix state.

[luhenry]

@threexc LGTM. Only question is around how do we migrate the build-<pkg>.yml files to this new way of working.

[threexc]

@threexc LGTM. Only question is around how do we migrate the build-<pkg>.yml files to this new way of working.

There's a section added in skills/python-project-porting/references/workflow-anatomy.md to teach the LLM what to do as it upgrades packages:

+**Converting old workflows is gradual.** Most `build-<pkg>.yml` still hard-code `python:
+[...]` next to `version:`. When a workflow is touched to build a new release, switch it:
+make sure the new version's entry has `python-versions`, add `python-matrix: true` to the
+`setup` call, and replace the `version:` + `python:` vectors
+of the build job(s) with `include: ${{ fromJSON(needs.setup.outputs.matrix) }}` (see
+`build-falcon.yml`). The matrix may grow (cp310/cp311, cp315/cp315t): exclude what fails for
+riscv64-specific reasons rather than reverting to a hard-coded list. Workflows whose extra
+axes (`libc`, `build`) don't fit a `{version, python}` include can stay hard-coded.
+
+A JSON `include:` can't carry per-entry keys (`test_skip`, `pytest_k`, …) or be trimmed by
+`exclude:`, so compute per-interpreter values in expressions instead, e.g.
+`CIBW_TEST_SKIP: ${{ endsWith(matrix.python, 't') && '*' || '' }}`.

Do you think more is needed? I wanted to avoid an immediate rewrite to all of the existing workflows.

[threexc]

Rebased the PR and fixed conflicts.

[threexc]

GitHub says this was updated ~3h ago, but I see nothing new... rebasing anyway.

[luhenry]

Gave a try to build-falcon.yml here. Matrix output is:

Matrix: [{"version":"4.4.0","python":"cp312"},{"version":"4.4.0","python":"cp313"},{"version":"4.4.0","python":"cp314"},{"version":"4.4.0","python":"cp314t"},{"version":"4.4.0","python":"cp315"},{"version":"4.4.0","python":"cp315t"}]