cuda on windows not working

[alosslessdev]

I downloaded the latest prebuilt cuda version from the releases page, and when i try to run it on windows, it doesn't seem to recognize the gpu even though i have the nvidia driver installed. I even tried downloading the latest cuda toolkit but nothing. How can i make cuda work with llama.cpp? I don't mind using the linux version...

[alosslessdev]

solved by using the winget version but it uses vulkan

[alosslessdev]

actually this is not a solution. gpu only got 26 tok/s. The cpu was faster at 36 tok/s

[CISC]

Did you also download cudart-llama-bin-win-cuda-12.4-x64.zip to get the required DLLs?

[alosslessdev]

no

[DocShotgun]

@alosslessdev What GPU are you using? If you're on a Blackwell GPU, the prebuilt Windows releases won't work because they are built against CUDA 12.4, while 12.8+ is required for Blackwell.

[alosslessdev]

I am on an rtx 3060 mobile so pre blackwell

[CISC]

Would a CUDA 13 release help? #17839

[alosslessdev]

ok i just tried the latest release from the releases page using cuda 13, now i get this, tok/s increased from 20 to 30 something on gpu:

[alosslessdev]

i have now downloaded cudart-llama-bin-win-cuda-13.1-x64.zip , unpacked it and placed the DLLs in the same folder as llama cli. and the issue persists. however my tok/s on gpu increased from 30 to 60 something. For now only linux can get 126 tok/s on gpu

[CISC]

What driver version do you have installed?