CUDA boost

[neocoretechs]

I am finishing up custom CUDA kernels and device helpers to boost model runner performance. I am using FFI to call pinpointed crosscuts that move the critical data up, leave it there, process, then move it down with minimal traffic: device helpers: GGUF dequant, sdot. kernels: matmul, softmax, rmsnorm. I am adding a USE_CUDA flag to turn the features on or off. Will provide DLL and so for aarch64 and windoze. Testbed: windoze 11/ Nvidia A2000 CUDA 13, Jetson Orin Nano, CUDA 13 Ubuntu.

[mikepapadim]

hello @neocoretechs, did you try the https://github.com/beehive-lab/GPULlama3.java

[neocoretechs]

I am aware of that effort. Not really interested in committing to entire gargantuan ecosystem of unknown provenance. Dirty little secret of GPU programing is its all application specific. General purpose frameworks quagmire you in a deep dark thicket of data transfer and bus bottlenecks. My effort is targeted crosscuts, custom kernels and device helpers, small footprint, entirely integrated with original effort, drop in replacement for forward inference.