Skip to content

ci: select pytest worker count automatically - #5132

Merged
KSchlobohm merged 1 commit into
github:mainfrom
KSchlobohm:kschlobohm-oct-26-pytest-auto-upstream
Oct 26, 2026
Merged

KSchlobohm merged 1 commit into
github:mainfrom
KSchlobohm:kschlobohm-oct-26-pytest-auto-upstream

Conversation

@KSchlobohm

Copy link
Copy Markdown
Contributor

Description

Use uv run pytest -n auto instead of -n 2 in the existing six-job OS/Python matrix. Let pytest-xdist select the worker count for each runner. Keep the full suite, matrix, dependencies, fixture behavior, and separate Ruff job unchanged.

Builds on #5051, which already added pytest-xdist and faster test-server shutdown.

In a three-attempt fork experiment, auto selected four workers on Ubuntu and Windows and five on macOS. Ubuntu test steps were approximately 23% faster than two workers. Windows differences were within runner noise; macOS coverage was insufficient for a speed claim. This is observed evidence, not a guaranteed speedup.

Testing

  • Tested locally with uv run specify --help
  • Ran existing tests with uv sync && uv run pytest
  • Tested with a sample project (if applicable)

Hosted experiment at source commit 380663f9474cd1257a440813dc9d63229cea9c57:

  • uv run pytest -n auto: Ubuntu and Windows passed on Python 3.13 and 3.14 in all three attempts. macOS/Python 3.14 passed once; the other five macOS auto jobs were cancelled without running tests because hosted runners could not be acquired.
  • Across all worker configurations (2, 4, auto), 42 test jobs succeeded and 12 were cancelled. No completed test job failed. The overall workflow was not green because of the cancellations.
  • These are experiment results, not a full-matrix validation of this rebased one-line patch. The PR's normal CI run must provide that evidence.

Local Windows/Python 3.14.8 checks on this patch:

uv sync --extra test
uv run specify --help
.venv\Scripts\python -m pytest tests\test_security_workflow.py tests\specify_cli\integrations\test_installed_adapters.py -k 'test_security_workflow or test_round9_optional_metadata' -n auto --tb=short -q
git diff --check

Results: dependency setup, CLI help, and whitespace checks passed; 80 focused tests passed with eight auto-selected workers, including valid and invalid adapter metadata cases.

The broader command .venv\Scripts\python -m pytest tests\test_security_workflow.py tests\specify_cli\integrations\test_installed_adapters.py -n auto -q was interrupted after failures and stalled progress. A repeat with -x --tb=short ended with 302 passed, 2 skipped, and 3 failed. All three captured failures were WinError 1314 (missing symlink privilege). Repeating those three cases with upstream's -n 2 reproduced all three failures:

.venv\Scripts\python -m pytest tests\specify_cli\integrations\test_installed_adapters.py::test_same_adapter_key_in_different_projects_uses_its_own_paths tests\specify_cli\integrations\test_installed_adapters.py::test_symlinked_storage_never_follows_target 'tests\specify_cli\integrations\test_installed_adapters.py::test_output_paths_are_validated_against_selected_project[False]' -n 2 --tb=short -q

No local full-suite pass is claimed. Sample-project testing is not applicable to this CI-only change.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Prepared with GitHub Copilot in Auto mode; the underlying model and reasoning effort were not exposed. The agent autonomously prepared the one-line workflow change, ran local checks, inspected hosted results, and drafted this description within a human-approved scope. Earlier experimental work was AI-assisted as disclosed in the linked fork experiment.

Use pytest-xdist auto selection within the existing OS/Python matrix.

Assisted-by: GitHub Copilot (model: unknown, Auto mode, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@KSchlobohm
KSchlobohm requested a review from mnriem as a code owner October 26, 2026 19:04
Copilot AI max review requested due to automatic review settings October 26, 2026 19:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Full six-job validation of the rebased patch remains unverified, particularly macOS/Python 3.13 with automatic workers.

No open issues 🎉

What changed in this PR

Updates Spec Kit’s existing parallel test setup from #5051 so pytest-xdist selects the worker count automatically for each CI runner.

Changes:

  • Replace pytest -n 2 with pytest -n auto.
  • Keep the full suite, six-job OS/Python matrix, dependencies, fixtures, and separate Ruff job unchanged.
File Description
.github/​workflows/​test.yml Enable automatic pytest worker selection.

Review effort: Max

@KSchlobohm
KSchlobohm merged commit 0bd4773 into github:main Oct 26, 2026
16 checks passed
@KSchlobohm
KSchlobohm deleted the kschlobohm-oct-26-pytest-auto-upstream branch October 26, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants