You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(bindings): cap memory pools missed in #3140 - #3290
Follows up on #3140, which capped the memory pools created by test_cuda_memPool_attr and test_cudart_MemPool_attr in cuda_bindings_13. Three sites with the same uncapped-pool (maxSize 0) pattern were missed and are fixed here.
A pool with maxSize 0 asks CUDA to reserve virtual address space of roughly twice the device memory. On riscv64 (Sv39, 39-bit VA space) this fails with CUDA_ERROR_OUT_OF_MEMORY on GPUs with enough on-device memory, which is exactly the error observed on the riscv64 test runner.
cuda_bindings_13/tests/test_cuda.py
test_ffi_coverage_new_mempool_attributes (v2 driver API): no max_size was set on the MemPoolProps_v1 before calling mem_pool_create. This is the test that actively failed in the riscv64 CI run that prompted this PR.
test_cuda_graphMem_attr: same root cause for the graph memory pool (which cannot be right-sized); added the xfail_on_graph_mempool_oom fixture and the OOM-cleanup branch, matching the pattern from test(bindings): cap the pools in the memPool attribute tests #3140.
cuda_bindings_12/tests/test_cudart.py
test_cudart_MemPool_attr: same as above for the runtime API pool. Added POOL_SIZE and set poolProps.maxSize.
NVIDIA#3140 capped the memory pools created by test_cuda_memPool_attr and
test_cudart_MemPool_attr but missed three more sites with the same
uncapped-pool (maxSize 0) pattern:
* cuda_bindings_13/tests/test_cuda.py: test_ffi_coverage_new_mempool_attributes
(v2 driver API, no max_size set).
* cuda_bindings_12/tests/test_cuda.py: test_cuda_memPool_attr (no POOL_SIZE
constant or maxSize) and test_cuda_graphMem_attr (no xfail_on_graph_mempool_oom
fixture; cuda_bindings_12 was added after NVIDIA#3140).
* cuda_bindings_12/tests/test_cudart.py: test_cudart_MemPool_attr (same).
A pool with maxSize 0 reserves virtual address space of about twice the device
memory; on riscv64 Sv39 (39-bit VA space) this fails with
CUDA_ERROR_OUT_OF_MEMORY on GPUs with enough device memory.
CUDA memory-pool tests now use bounded allocations across supported versions. Graph-memory tests handle out-of-memory conditions as expected skips and verify resource cleanup when allocation setup fails. Successful allocation paths continue to be validated, making test outcomes more consistent across devices with different amounts of available memory.
Tests cap created memory pools at 2 MiB. The graph memory attribute test cleans up after an out-of-memory allocation result and marks that result as an expected failure.
Changes: Memory-pool test behavior
Layer / File(s)
Summary
Cap test memory pools cuda_bindings_12/tests/test_cuda.py, cuda_bindings_12/tests/test_cudart.py, cuda_bindings_13/tests/test_cuda.py
Test-created memory pools use a 2 MiB maximum size.
Handle graph allocation out of memory cuda_bindings_12/tests/test_cuda.py
If cuGraphAddMemAllocNode returns CUDA_ERROR_OUT_OF_MEMORY, the test destroys the graph and stream, verifies both operations, and marks the test as expected to fail.
Possibly related issues
NVIDIA/cuda-python#2362: This PR sets maximum sizes on test-created pools. It does not cover the issue’s separate location-routing or cuda.core API work.
Priority: Normal
Change: Bug fix
Merge Risk: Low · up to 991ea
The expected-failure path can leave a CUDA context live in the test process; clean it up before merging.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Reviewing files that changed from the base of the PR and between 3cc4edb and 991ea99.
Files selected for processing: 3
cuda_bindings_12/tests/test_cuda.py
cuda_bindings_12/tests/test_cudart.py
cuda_bindings_13/tests/test_cuda.py
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
important: Destroy ctx before xfail_on_graph_mempool_oom(err).
The OOM branch destroys graph and stream, then pytest.xfail exits before the later cuCtxDestroy(ctx) call. Add context destruction to this branch and assert success before calling the xfail helper.
The reason will be displayed to describe this comment to others. Learn more.
Approving, with optional suggestions.
BTW, the cuda_bindings_12/ code isn't currently exercised on main (or in the CI here), but please merge: it will then get exercised in my PR #3182 stack.
I asked codex about helper consolidation. This is the full review, with an extra remark to answer that question.
Destroy the context before xfail. The new OOM branch destroys the graph and stream, then exits through pytest.xfail, bypassing cuCtxDestroy(ctx). CUDA 12 owns this context directly; CUDA 13 has fixture teardown. Destroy it before xfail or use finally. A forced-OOM probe confirms the leak.
On consolidation, I’d keep it optional. Each OOM fixture has only about five executable lines and one caller; calling pytest.xfail(...) directly inside the existing OOM branch would be simpler. A shared POOL_SIZE constant is reasonable cleanup.
For preventing further misses, I’d suggest extending the existing memory-pool lint, which currently recognizes only cuda_core constructors. Consolidating a constant won’t detect a missing maxSize assignment.
The two P2 issues and the lint extension are addressed in #3298. The version-gate for CUDA 12.2 was dropped after checking CI: no test matrix row runs on a driver older than 12.2, so the gate would never fire.
…3298)
* test: consolidate duplicated memory-pool test helpers
Follow-up to #3290. The same memory-pool helpers were copied across the
cuda_bindings_12, cuda_bindings_13, and cuda_core test suites. Move them
into one shared module and delete the copies. No test outcome changes: the
conditions behind every skipif and skip are the same.
New cuda_python_test_helpers/memory.py:
* POOL_SIZE (9 copies, including the cuda_core constant, which now
re-exports it).
* supports_memory_pools() and supports_managed_memory() (13 copies,
including the two Cython test modules).
* xfail_graph_mempool_oom() (3 copies of the xfail_on_graph_mempool_oom
fixture). Each call site keeps its own out-of-memory check and cleanup.
cuda_bindings_12/tests/test_cuda.py also uses the shared
driver_version_less_than() instead of its own copy, which behaves the same.
cuda_bindings_12/tests/test_cudart.py keeps its local driverVersionLessThan:
it does not call cuInit, and the shared one would turn collection-time skips
into errors on machines with a driver but no device.
cuda_core/tests/helpers/memory.py gains skip_if_mempool_unsupported(),
skip_if_managed_mempool_unsupported(), and is_pool_resource(), which replace
the per-module _skip_if_no_mempool (6 copies), _skip_without_mempools,
_gpu_with_mempool_or_skip, _skip_if_no_managed_mempool (2 copies), and
_is_pool (3 copies). Two skip reason strings change wording; their
conditions do not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bugSomething isn't workingcuda.bindingsEverything related to the cuda.bindings module
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follows up on #3140, which capped the memory pools created by
test_cuda_memPool_attrandtest_cudart_MemPool_attrincuda_bindings_13. Three sites with the same uncapped-pool (maxSize 0) pattern were missed and are fixed here.A pool with
maxSize 0asks CUDA to reserve virtual address space of roughly twice the device memory. On riscv64 (Sv39, 39-bit VA space) this fails withCUDA_ERROR_OUT_OF_MEMORYon GPUs with enough on-device memory, which is exactly the error observed on the riscv64 test runner.cuda_bindings_13/tests/test_cuda.pytest_ffi_coverage_new_mempool_attributes(v2 driver API): nomax_sizewas set on theMemPoolProps_v1before callingmem_pool_create. This is the test that actively failed in the riscv64 CI run that prompted this PR.cuda_bindings_12/tests/test_cuda.pytest_cuda_memPool_attr:cuda_bindings_12was added after test(bindings): cap the pools in the memPool attribute tests #3140 and itsCUmemPoolPropshad nomaxSize. Added thePOOL_SIZEconstant and setpoolProps.maxSize.test_cuda_graphMem_attr: same root cause for the graph memory pool (which cannot be right-sized); added thexfail_on_graph_mempool_oomfixture and the OOM-cleanup branch, matching the pattern from test(bindings): cap the pools in the memPool attribute tests #3140.cuda_bindings_12/tests/test_cudart.pytest_cudart_MemPool_attr: same as above for the runtime API pool. AddedPOOL_SIZEand setpoolProps.maxSize.Checklist