Add CUDA version compatibility check #1412

Andy-Jost · 2025-12-18T19:06:15Z

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
Exported from cuda.bindings.utils
Added comprehensive unit tests in tests/test_version_check.py

cuda-core

Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver 
only supports CUDA 12.8. Some features may not work correctly. Consider updating 
your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
Invoked by cuda-core: Called when Device first triggers CUDA initialization
Runs once: Uses a module-level flag to ensure the check runs only once per process
Non-blocking: Warning only, does not prevent operation
Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
Major version only: No warning for minor version differences (handled by graceful degradation per PR Handle unsupported device attributes gracefully #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

No warning when driver is newer
No warning when same major version
Warning when compile major > driver major
Warning only issued once per process
Suppression via environment variable
Silent handling of driver errors
Silent handling of missing CUDA_VERSION attribute

Related Work

Complements PR Handle unsupported device attributes gracefully #1409 which handles graceful degradation for minor version mismatches

copy-pr-bot · 2025-12-18T19:06:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Andy-Jost · 2025-12-18T19:06:28Z

/ok to test 7ce325c

Andy-Jost · 2025-12-18T19:08:17Z

/ok to test 1962e35

github-actions · 2025-12-18T19:20:22Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1412/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

cuda_core/cuda/core/_utils/cuda_utils.pyx

cuda_bindings/docs/source/environment_variables.rst

cuda_core/cuda/core/_utils/cuda_utils.pyx

Andy-Jost · 2025-12-19T00:10:25Z

/ok to test 73e5a43

Andy-Jost · 2025-12-19T00:48:39Z

/ok to test ddd92bd

cuda_bindings/cuda/bindings/utils/_version_check.py

rparolin

We should issue a warning to the user if we can't fetch the driver version.

cuda_bindings/cuda/bindings/utils/_version_check.py

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

# Conflicts: # cuda_core/pixi.lock

Andy-Jost · 2026-01-07T21:11:36Z

/ok to test 62dfcca

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Andy-Jost · 2026-01-08T18:00:34Z

/ok to test b2083ed

Extract Device.__new__ logic into cdef helper functions: - Device_ensure_cuda_initialized(): cuInit + version check - Device_resolve_device_id(): resolve None to current device or 0 - Device_ensure_tls_devices(): create thread-local singletons Reduces Device.__new__ from ~60 lines to ~12 lines. Helpers placed after Device class following memory module pattern.

Andy-Jost · 2026-01-08T18:19:03Z

/ok to test fdb3a7e

rparolin · 2026-01-09T21:08:17Z

cuda_bindings/tests/test_version_check.py

+
+    def setup_method(self):
+        """Reset the version compatibility check flag before each test."""
+        _version_check._major_version_compatibility_checked = False


This is effectively going to reset if the version was checked already and after the unit tests executes the check could potentially execute again. Shouldn't we cache the current state and before setting it to false for the unit tests and then restore to its previous value on unit test teardown?

Someone with a dodgy build might get multiple warnings during testing. Not sure it's worth fixing.

We should either mock it or restore whatever the original value was during teardown, I agree with Rob.

Andy-Jost · 2026-01-09T21:30:52Z

/ok to test acadb31

Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost · 2026-01-13T19:42:51Z

/ok to test 3a5c210

Andy-Jost · 2026-01-14T01:12:48Z

/ok to test 424a113

Andy-Jost · 2026-01-14T01:42:35Z

/ok to test 6071609

leofang

Sorry for my late review.

leofang · 2026-01-14T03:17:22Z

cuda_bindings/cuda/bindings/utils/_version_check.py

+    global _major_version_compatibility_checked
+    if _major_version_compatibility_checked:
+        return
+    _major_version_compatibility_checked = True


need a lock + double check pattern

move _major_version_compatibility_checked = True to the code later because we could raise below

Added a lock

This was intentional. If the code attempting to perform the check fails for some reason, we would never want to retry it.

leofang · 2026-01-14T03:20:20Z

cuda_bindings/docs/source/environment_variables.rst


 - ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

+- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.


TBH I am not sure if this env var is really useful, because the added API is entirely opt-in?

The version check can be triggered indirectly from the end user's POV. For example, by using cuda-core. Without this, each library that calls the version-check function would need its own opt-out, and users might need to set them all. To me it makes sense to place the opt-out alongside the implementation and there is little/no cost.

leofang · 2026-01-14T03:21:16Z

cuda_bindings/tests/test_version_check.py

+
+    def setup_method(self):
+        """Reset the version compatibility check flag before each test."""
+        _version_check._major_version_compatibility_checked = False


We should either mock it or restore whatever the original value was during teardown, I agree with Rob.

leofang · 2026-01-14T03:23:11Z

cuda_core/cuda/core/_device.pyx

        return GraphBuilder._init(stream=self.create_stream(), is_stream_owner=True)
+
+
+cdef inline void Device_ensure_cuda_initialized() except *:


We prefer cdef inline int Device_ensure_cuda_initialized() except? -1: to avoid an unconditional exception check (which Cython warns at build time).

leofang · 2026-01-14T03:25:23Z

cuda_core/cuda/core/_device.pyx

+        try:
+            from cuda.bindings.utils import warn_if_cuda_major_version_mismatch
+        except ImportError:
+            pass
+        else:
+            warn_if_cuda_major_version_mismatch()


We want to move imports like this to the top so as to avoid the little overheads in a hot loop. Please benchmark the constructor to see the perf diff before/after this PR. As per @mdboom such imports, despite they are cached, have a noticeable perf overhead (on the order that we care about).

I agree with the general advice. Note that in this case, the block is guarded by _is_cuInit so the import is only attempted once. I decided to bury this import because 1) it would otherwise add clutter to the top-of-file import list, which human readers parse relatively often, 2) an import failure is ignored, so it can't hurt anything, and 3) there is no performance impact, since the import is only attempted once.

Andy-Jost · 2026-01-14T17:49:45Z

/ok to test 96532b2

Andy-Jost · 2026-01-14T17:51:15Z

/ok to test 973af9b

Replace setup_method/teardown_method with a pytest fixture that uses monkeypatch to properly save and restore the original value of _major_version_compatibility_checked after each test. Minor change to Cython cdef inline helper function signature.

Andy-Jost · 2026-01-14T18:03:55Z

/ok to test 7f23b08

Andy-Jost added this to the cuda.core beta 11 milestone Dec 18, 2025

Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 18, 2025

Andy-Jost self-assigned this Dec 18, 2025

Andy-Jost requested review from leofang, rparolin and rwgk December 18, 2025 19:06

Andy-Jost force-pushed the runtime-version-check branch from 7ce325c to 1962e35 Compare December 18, 2025 19:08

mdboom approved these changes Dec 18, 2025

View reviewed changes

Andy-Jost mentioned this pull request Dec 18, 2025

fix: derive CUDA major version from headers for build #1395

Merged

2 tasks

leofang requested changes Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_bindings/docs/source/environment_variables.rst Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

Andy-Jost force-pushed the runtime-version-check branch 2 times, most recently from 8a0d248 to 73e5a43 Compare December 19, 2025 00:09

Andy-Jost force-pushed the runtime-version-check branch from 73e5a43 to ddd92bd Compare December 19, 2025 00:47

leofang self-requested a review December 19, 2025 01:22

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin requested changes Dec 20, 2025

View reviewed changes

rwgk reviewed Dec 26, 2025

View reviewed changes

Andy-Jost added 2 commits January 7, 2026 13:09

Add CUDA major version compatibility check

d999f40

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

Merge remote-tracking branch 'origin/main' into runtime-version-check

62dfcca

# Conflicts: # cuda_core/pixi.lock

Andy-Jost requested a review from rparolin January 8, 2026 17:48

Andy-Jost added 2 commits January 8, 2026 09:59

Move version check import to local scope

73611ed

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Merge remote-tracking branch 'origin/main' into runtime-version-check

b2083ed

Andy-Jost force-pushed the runtime-version-check branch from 07bf4a4 to b2083ed Compare January 8, 2026 18:00

Andy-Jost force-pushed the runtime-version-check branch from a92ed3b to fdb3a7e Compare January 8, 2026 18:18

rparolin reviewed Jan 9, 2026

View reviewed changes

rparolin approved these changes Jan 9, 2026

View reviewed changes

kkraus14 approved these changes Jan 13, 2026

View reviewed changes

Merge branch 'main' into runtime-version-check

3a5c210

Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost force-pushed the runtime-version-check branch from acadb31 to 3a5c210 Compare January 13, 2026 19:42

Merge branch 'main' into runtime-version-check

424a113

Merge branch 'main' into runtime-version-check

6071609

rwgk approved these changes Jan 14, 2026

View reviewed changes

leofang requested changes Jan 14, 2026

View reviewed changes

Merge branch 'main' into runtime-version-check

7925693

Andy-Jost force-pushed the runtime-version-check branch from 96532b2 to 973af9b Compare January 14, 2026 17:51

Andy-Jost force-pushed the runtime-version-check branch from 973af9b to 27732f2 Compare January 14, 2026 17:56

Andy-Jost force-pushed the runtime-version-check branch from 27732f2 to 7f23b08 Compare January 14, 2026 18:03


		- ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

		- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.

		return GraphBuilder._init(stream=self.create_stream(), is_stream_owner=True)


		cdef inline void Device_ensure_cuda_initialized() except *:

Add CUDA version compatibility check #1412

Are you sure you want to change the base?

Add CUDA version compatibility check #1412

Conversation

Andy-Jost commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

cuda-bindings

cuda-core

Rationale

Design

Future Work

Test Coverage

Related Work

Uh oh!

copy-pr-bot bot commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

rparolin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Jan 7, 2026

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Jan 9, 2026

Uh oh!

Andy-Jost commented Jan 13, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Dec 18, 2025 •

edited

Loading

Andy-Jost Jan 14, 2026 •

edited

Loading

Andy-Jost Jan 14, 2026 •

edited

Loading