[JacobSzwejbka]

About 45 pull.yml jobs had no if: and ran on every PR push regardless of what
changed. pull.yml is the single largest CI cost (~1,740 self-hosted minutes per
run, ~1,500 runs a month). This gates the backend-specific ones on their own
paths for pull_request events, using the same
github.event_name != 'pull_request' || contains(changed-files, ...) pattern
already used by test-arduino-library, test-minimal-wheel-linux and
test-pybindings-no-aten. Pushes to main and release branches still run every
job, so anything a path filter misses is caught post-merge.

Gated groups (approx. minutes per PR run today):

• QNN (11 jobs, ~290 min): backends/qualcomm, examples/qualcomm, QNN CI
  scripts; the LLM ones also on examples/models/llama and extension/llm; the
  testsuite also on backends/test/{suite,harness}.
• Arm (3 jobs, ~118 min) and test-mcu-cortex-m-backend (~33 min):
  backends/arm, examples/arm, backends/cortex_m, setup-arm-baremetal-tools.sh.
• NXP (3 jobs, ~122 min): backends/nxp, examples/nxp.
• Vulkan (2 jobs, ~45 min): backends/vulkan, setup-vulkan-linux-deps.sh.
• WASM (2 jobs, ~28 min): extension/wasm, examples/wasm, tools/cmake.
• OpenVINO (1 job): backends/openvino, examples/openvino.
• LLM export/runner jobs (phi-3-mini, eval_llama, llama_runner_eager, lora,
  lora-multimethod; ~78 min on 94-vCPU boxes): examples/models/{llama,qwen3,
  phi-3-mini}, extension/llm, backends/xnnpack and their CI scripts.

Every gate also includes .github/workflows/pull.yml, so this PR itself runs
all of them.

The five LLM jobs also move from mt-l-x86iavx512-94-192 to
mt-l-x86iavx512-32-256: more memory (256 vs 192 GB) at roughly half the
price. They run 11-25 minutes today; this PR's own CI validates they fit.

Binary-size jobs (test-arm-cortex-m-size-test, test-binary-size-*) are left
ungated on purpose since they exist to catch core-runtime size regressions.

Context

Part of a set of CI cost reductions for self-hosted (OSDC / AWS) runners. Estimates come from the last 30 days of GitHub Actions run counts times per-job runtimes of a sample run, priced at AWS on-demand rates as a proxy. Independent of the other PRs in the set; they all merge cleanly together.

Validation

Edited YAML parses, and actionlint 1.7.12 reports no diagnostics introduced by this change compared with the base commit. Pre-existing diagnostics were recorded separately. GitHub CI is pending.

Authored with AI assistance; this handoff was reviewed and the PR was opened with Codex.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24673

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b625b54 with merge base d1f597e ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[psiddh]

@claude review this pr and check if there are any potential regressions / issues

[claude]

Claude finished @psiddh's task in 4m 10s —— View job

---

The gating logic itself works (pushes to main and release still run every job, contains() only ever over-matches, every path exists, and the new runner label is already used in the repo), but the path lists are narrower than what these jobs depend on, so PRs that change shared or setup code will now skip these jobs and show green; I recommend Request Changes until the gates are widened.

Detailed review

PR Review: #24673

Summary

This PR adds PR-only path gates to about 30 backend and LLM jobs in pull.yml, and moves the five LLM jobs to mt-l-x86iavx512-32-256.

Three independent reviewers found the same main problem. Each gate lists only its backend's own directory. Shared setup code, the CI Docker images, the PyTorch pin and the core runtime are left out. A PR that breaks QNN, Arm, NXP, Vulkan or the LLM jobs from outside those directories now skips them and passes. pull is the required check in .github/merge_rules.yaml, so the breakage only shows up on main, after merge.

Infrastructure

• Major: shared setup files are missing from every new gate. (pull.yml QNN gates at :40-49, Arm at ~:1031/1092/1138, Vulkan at :1986/2033, NXP, OpenVINO, LLM.)

  • The job in the same file that already does this well, test-arm-backend-docgen (:969-987), lists .ci/docker/, .ci/scripts/utils.sh, install_executorch.{sh,py}, install_requirements.{sh,py} and pyproject.toml. None of the new gates include any of these.
  • Every gated job installs through these files and runs inside the qnn-sdk, arm-sdk or clang12 images. Several scripts also source utils.sh directly: test_qnn_static_llm.sh, test_lora*.sh and setup-openvino.sh.
  • Example: a PR that bumps .ci/docker/ci_commit_pins/pytorch.txt or changes the QNN or Arm SDK Docker image skips every QNN, Arm, Vulkan, NXP and LLM job. Those are the changes most likely to break backends.
  • Suggested fix: add this shared set to each gate. At minimum, consider exir/ (or exir/passes, exir/backend), backends/transforms, CMakePresets.json and tools/cmake/ for the backends and LLM jobs that build with CMake presets. test_lora.sh:17, test_lora_multimethod.sh:17 and test_phi_3_mini.sh all use --preset llm-release.
  • Fix this →

• Major: some gates leave out files their own job runs. A PR that edits only these files skips the job that tests them:

  • test-vulkan-models-linux (:1986-1990) runs python -m examples.vulkan.export (:2022, :2028), but examples/vulkan is not in its gate.
  • test-qnn-models-linux (:1312-1320) runs .ci/scripts/test_model.sh on examples/models/*. Neither is in its gate. The gate instead lists test_wheel_package_qnn.sh, which this job doesn't use; it looks copied from the wheel job.
  • test-qnn-python-imports-linux (~:1544-1552) runs .ci/scripts/test_backend_imports.py and test_example_imports.py. Neither is in its gate.
  • test-qnn-testsuite-linux (:1376-1383) goes through .ci/scripts/test_backend.sh, which sources build-qnn-sdk.sh and pytest-parallelism.sh. Neither is in its gate. More importantly, .github/workflows/test-backend-qnn.yml already runs this same _test_backend.yml with flows: ["qnn"] on every PR, with no gate and on the same mt-l-x86iavx512-32-256 runner. This pull.yml job duplicates it and could simply be deleted for a larger saving.

• Major: the Arm gates leave out backends/cortex_m, but the dependency only runs one way. backends/arm/quantizer/arm_quantizer.py and arm_quantizer_utils.py import executorch.backends.cortex_m. The cortex-m gate includes backends/arm, but the three Arm gates don't include backends/cortex_m. A PR that changes only backends/cortex_m can break the Arm quantizer without running any Arm job. The same applies to backends/transforms, which is imported widely by Arm, Vulkan, NXP and QNN.

• Minor: PRs where every file is deleted skip every gated job. _get-changed-files.yml drops removed files (select(.status != "removed")). If a PR only deletes files, the output is '*', and contains('*', ...) is false, so every new gate skips. Deleting a still-imported file under backends/qualcomm would pass the PR. The docgen job handles this with needs.changed-files.outputs.changed-files == '*' ||. The new gates should add the same clause.

• Minor: there is no way to force these jobs to run on a PR. pull.yml has no ciflow/* trigger, and the new gates don't check run-decision.outputs.is-full-run. The header comment at :21-25 says path-filtered jobs rely on that output. For these jobs, an author's only options are editing pull.yml or using workflow_dispatch. Consider adding needs.run-decision.outputs.is-full-run == 'true' || to the gates, as llm-server already does.

• Minor: the Arm op-support doc check is no longer triggered by doc edits. test-arm-backend-no-driver runs git diff --exit-code on the generated docs/source/backends/arm-*/*_op_support.md files, but docs/source/backends/arm-* is not in its gate. A hand edit to those files won't be caught until it reaches main.

• Minor: the phi-3-mini gate leaves out the optimum-executorch pin. examples/models/phi-3-mini/install_requirements.sh installs the version in .ci/docker/ci_commit_pins/optimum-executorch.txt, and the job exports through optimum-cli. A PR that only bumps that pin skips the one job that uses it. Also, the PR description says the LLM gates include examples/models/llama, but the phi-3 gate (:1505-1512) doesn't.

Performance

• Minor, unconfirmed: the LLM jobs drop from 94 to 32 vCPUs with the same 90-minute timeout.
  • The jobs are test-phi-3-mini-runner-linux, eval_llama, llama_runner_eager, lora and lora-multimethod.
  • test_phi_3_mini.sh builds with -j$(nproc), and the lora jobs do full llm-release CMake builds, so their build phases could take roughly 3x longer.
  • The extra memory (256 GB) helps, but there is no timing data yet. Please check the run times on this PR's CI before merging, especially for lora and lora-multimethod.

Recommendation

Request Changes. The gating approach is reasonable, but before merging, the gates should at least:

• include the shared setup and Docker paths that the existing docgen gate already lists;
• cover the scripts and example directories each job runs (examples/vulkan, test_model.sh, test_*_imports.py, backends/cortex_m for the Arm jobs, backends/transforms);
• add the == '*' fallback for PRs with only deletions.

Separately, the redundant test-qnn-testsuite-linux job could be removed, since test-backend-qnn.yml already runs the same suite on every PR.