Skip to content

test(bindings): cap memory pools missed in #3140 - #3290

Merged
juenglin merged 1 commit into
NVIDIA:mainfrom
juenglin:fix/mempool-cap-missed-in-3140
Oct 30, 2026
Merged

juenglin merged 1 commit into
NVIDIA:mainfrom
juenglin:fix/mempool-cap-missed-in-3140

Conversation

@juenglin

Copy link
Copy Markdown
Contributor

Description

Follows up on #3140, which capped the memory pools created by test_cuda_memPool_attr and test_cudart_MemPool_attr in cuda_bindings_13. Three sites with the same uncapped-pool (maxSize 0) pattern were missed and are fixed here.

A pool with maxSize 0 asks CUDA to reserve virtual address space of roughly twice the device memory. On riscv64 (Sv39, 39-bit VA space) this fails with CUDA_ERROR_OUT_OF_MEMORY on GPUs with enough on-device memory, which is exactly the error observed on the riscv64 test runner.

cuda_bindings_13/tests/test_cuda.py

  • test_ffi_coverage_new_mempool_attributes (v2 driver API): no max_size was set on the MemPoolProps_v1 before calling mem_pool_create. This is the test that actively failed in the riscv64 CI run that prompted this PR.

cuda_bindings_12/tests/test_cuda.py

cuda_bindings_12/tests/test_cudart.py

  • test_cudart_MemPool_attr: same as above for the runtime API pool. Added POOL_SIZE and set poolProps.maxSize.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

NVIDIA#3140 capped the memory pools created by test_cuda_memPool_attr and
test_cudart_MemPool_attr but missed three more sites with the same
uncapped-pool (maxSize 0) pattern:

* cuda_bindings_13/tests/test_cuda.py: test_ffi_coverage_new_mempool_attributes
  (v2 driver API, no max_size set).
* cuda_bindings_12/tests/test_cuda.py: test_cuda_memPool_attr (no POOL_SIZE
  constant or maxSize) and test_cuda_graphMem_attr (no xfail_on_graph_mempool_oom
  fixture; cuda_bindings_12 was added after NVIDIA#3140).
* cuda_bindings_12/tests/test_cudart.py: test_cudart_MemPool_attr (same).

A pool with maxSize 0 reserves virtual address space of about twice the device
memory; on riscv64 Sv39 (39-bit VA space) this fails with
CUDA_ERROR_OUT_OF_MEMORY on GPUs with enough device memory.
@juenglin juenglin added this to the cuda.bindings next milestone Oct 29, 2026
@juenglin juenglin added bug Something isn't working cuda.bindings Everything related to the cuda.bindings module labels Oct 29, 2026
@juenglin juenglin self-assigned this Oct 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin
juenglin requested a review from rwgk October 29, 2026 22:53
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 991ea99

@juenglin
juenglin marked this pull request as ready for review October 29, 2026 22:55
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Tests
    • CUDA memory-pool tests now use bounded allocations across supported versions. Graph-memory tests handle out-of-memory conditions as expected skips and verify resource cleanup when allocation setup fails. Successful allocation paths continue to be validated, making test outcomes more consistent across devices with different amounts of available memory.

Tests cap created memory pools at 2 MiB. The graph memory attribute test cleans up after an out-of-memory allocation result and marks that result as an expected failure.

Changes: Memory-pool test behavior

Layer / File(s) Summary
Cap test memory pools
cuda_bindings_12/tests/test_cuda.py, cuda_bindings_12/tests/test_cudart.py, cuda_bindings_13/tests/test_cuda.py
Test-created memory pools use a 2 MiB maximum size.
Handle graph allocation out of memory
cuda_bindings_12/tests/test_cuda.py
If cuGraphAddMemAllocNode returns CUDA_ERROR_OUT_OF_MEMORY, the test destroys the graph and stream, verifies both operations, and marks the test as expected to fail.

Possibly related issues

  • NVIDIA/cuda-python#2362: This PR sets maximum sizes on test-created pools. It does not cover the issue’s separate location-routing or cuda.core API work.

Priority: Normal

Change: Bug fix

Merge Risk: Low · up to 991ea

The expected-failure path can leave a CUDA context live in the test process; clean it up before merging.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3cc4edb and 991ea99.

Files selected for processing: 3
  • cuda_bindings_12/tests/test_cuda.py
  • cuda_bindings_12/tests/test_cudart.py
  • cuda_bindings_13/tests/test_cuda.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +592 to +597
if err == cuda.CUresult.CUDA_ERROR_OUT_OF_MEMORY:
(destroy_err,) = cuda.cuGraphDestroy(graph)
assert destroy_err == cuda.CUresult.CUDA_SUCCESS
(destroy_err,) = cuda.cuStreamDestroy(stream)
assert destroy_err == cuda.CUresult.CUDA_SUCCESS
xfail_on_graph_mempool_oom(err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

Supported by static analysis

Script executed:

sed -n '520,625p' cuda_bindings_12/tests/test_cuda.py
rg -n 'def test_cuda_graphMem_attr|cuCtxDestroy|xfail_on_graph_mempool_oom|yield|autouse|context' cuda_bindings_12/tests/test_cuda.py cuda_bindings_12/tests/conftest.py cuda_bindings_12/tests

Repository: NVIDIA/cuda-python

Length of output: 12879


Script executed:

set -u
printf '%s\n' '--- test files and conftest files ---'
rg --files cuda_bindings_12/tests -g 'conftest.py' -g 'test_cuda.py'
printf '%s\n' '--- applicable pytest configuration and teardown hooks ---'
rg -n -F --glob 'pyproject.toml' --glob 'pytest.ini' --glob 'setup.cfg' --glob 'tox.ini' --glob 'conftest.py' --glob '*.py' \
  -e 'pytest_sessionfinish' -e 'pytest_runtest_teardown' -e 'pytest_runtest_protocol' -e 'autouse=True' -e 'cuCtxDestroy' -e 'cuda.cuCtxDestroy' \
  . || test "$?" -eq 1
printf '%s\n' '--- complete graph test through its cleanup ---'
nl -ba cuda_bindings_12/tests/test_cuda.py | sed -n '545,640p'

Repository: NVIDIA/cuda-python

Length of output: 11345


Script executed:

printf '%s\n' '--- cuda_bindings_12/tests/conftest.py ---'
nl -ba cuda_bindings_12/tests/conftest.py
printf '%s\n' '--- repository pytest configuration files ---'
for f in pyproject.toml pytest.ini setup.cfg tox.ini; do
  if [ -f "$f" ]; then
    printf '%s\n' "--- $f ---"
    nl -ba "$f" | rg -n -C 4 -e 'pytest|testpaths|addopts|python_files|plugins|conftest'
  fi
done
printf '%s\n' '--- parent and root conftest files ---'
find . -maxdepth 3 -name conftest.py -print

Repository: NVIDIA/cuda-python

Length of output: 3334


Script executed:

printf '%s\n' '--- registered pytest plugin files ---'
rg --files cuda_python_test_helpers | rg '(^|/)_pytest_plugin(\.py|/)|pytest'
printf '%s\n' '--- plugin teardown and CUDA cleanup references ---'
rg -n -C 6 -F --glob '*.py' \
  -e 'pytest_runtest_teardown' -e 'pytest_runtest_protocol' -e 'pytest_sessionfinish' \
  -e 'pytest.fixture' -e 'cuCtxDestroy' -e 'cuda.cuCtxDestroy' -e 'context' \
  cuda_python_test_helpers

Repository: NVIDIA/cuda-python

Length of output: 20168


Script executed:

nl -ba cuda_python_test_helpers/cuda_python_test_helpers/_pytest_plugin.py
printf '%s\n' '--- imports and pytest hook declarations in helper package ---'
rg -n -C 3 --glob '*.py' \
  -e '^from ' -e '^import ' -e '^def pytest_' -e '^class ' -e 'pytest_plugins' \
  cuda_python_test_helpers/cuda_python_test_helpers

Repository: NVIDIA/cuda-python

Length of output: 29198


important: Destroy ctx before xfail_on_graph_mempool_oom(err).

The OOM branch destroys graph and stream, then pytest.xfail exits before the later cuCtxDestroy(ctx) call. Add context destruction to this branch and assert success before calling the xfail helper.

@github-actions

Copy link
Copy Markdown
Contributor

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, with optional suggestions.

BTW, the cuda_bindings_12/ code isn't currently exercised on main (or in the CI here), but please merge: it will then get exercised in my PR #3182 stack.

I asked codex about helper consolidation. This is the full review, with an extra remark to answer that question.

codex GPT-6.1-Sol ultra:

I found two P2 issues at 991ea99:

  • Destroy the context before xfail. The new OOM branch destroys the graph and stream, then exits through pytest.xfail, bypassing cuCtxDestroy(ctx). CUDA 12 owns this context directly; CUDA 13 has fixture teardown. Destroy it before xfail or use finally. A forced-OOM probe confirms the leak.

  • Version-gate the CUDA 12 cap. The driver assignment and runtime assignment are unconditional, while the tests allow older drivers. maxSize occupies bytes that CUDA 12.1 requires to remain zero; CUDA 12.2 introduces the field. Preserve the uncapped path on older drivers. This finding follows from the documented ABI; I haven’t reproduced it on an old driver.

On consolidation, I’d keep it optional. Each OOM fixture has only about five executable lines and one caller; calling pytest.xfail(...) directly inside the existing OOM branch would be simpler. A shared POOL_SIZE constant is reasonable cleanup.

For preventing further misses, I’d suggest extending the existing memory-pool lint, which currently recognizes only cuda_core constructors. Consolidating a constant won’t detect a missing maxSize assignment.

@juenglin
juenglin merged commit 7743220 into NVIDIA:main Oct 30, 2026
225 of 227 checks passed
@juenglin

Copy link
Copy Markdown
Contributor Author

The two P2 issues and the lint extension are addressed in #3298. The version-gate for CUDA 12.2 was dropped after checking CI: no test matrix row runs on a driver older than 12.2, so the gate would never fire.

juenglin added a commit that referenced this pull request Oct 30, 2026
…3298)

* test: consolidate duplicated memory-pool test helpers

Follow-up to #3290. The same memory-pool helpers were copied across the
cuda_bindings_12, cuda_bindings_13, and cuda_core test suites. Move them
into one shared module and delete the copies. No test outcome changes: the
conditions behind every skipif and skip are the same.

New cuda_python_test_helpers/memory.py:

* POOL_SIZE (9 copies, including the cuda_core constant, which now
  re-exports it).
* supports_memory_pools() and supports_managed_memory() (13 copies,
  including the two Cython test modules).
* xfail_graph_mempool_oom() (3 copies of the xfail_on_graph_mempool_oom
  fixture). Each call site keeps its own out-of-memory check and cleanup.

cuda_bindings_12/tests/test_cuda.py also uses the shared
driver_version_less_than() instead of its own copy, which behaves the same.
cuda_bindings_12/tests/test_cudart.py keeps its local driverVersionLessThan:
it does not call cuInit, and the shared one would turn collection-time skips
into errors on machines with a driver but no device.

cuda_core/tests/helpers/memory.py gains skip_if_mempool_unsupported(),
skip_if_managed_mempool_unsupported(), and is_pool_resource(), which replace
the per-module _skip_if_no_mempool (6 copies), _skip_without_mempools,
_gpu_with_mempool_or_skip, _skip_if_no_managed_mempool (2 copies), and
_is_pool (3 copies). Two skip reason strings change wording; their
conditions do not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.bindings Everything related to the cuda.bindings module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants