Summary : illegal memory access with RTX5090 with 570.86.16 Relevant Area: usage detected in llama.cpp NVIDIA GPU or System: GeForce RTX 5090 CUDA Version: 12.8 NVIDIA Software Version: NVIDIA-SMI 570.86.16 OS: Leap15.6 kernel 6.4 x86_64 Other Details: tested ok on Quadro P1000 4GB I have done so…

Can you add cudaDeviceSynchronize within CUDA_CHECK after each command? And also test its return value? Just to identify with certainty the failing instruction.

aginies@ryzen9:~/testcuda> cuda-gdb --args ./a.out -f qwen2.5-coder-3b-instruct-q4_0.gguf NVIDIA (R) cuda-gdb 12.8 .... Reading symbols from ./a.out... (cuda-gdb) run Starting program: /home/aginies/testcuda/a.out -f qwen2.5-coder-3b-instruct-q4_0.gguf [Thread debugging using libthread_db enabled] …

Okay, so definitely within the cudaMemset and both times at the same address within the data structure. What is the size of ggufData? Is it an issue with memory above 2 GiB? You could try to do the CudaMemset in parts with several calls in a for loop and pointers within the array to find out, whi…

The size of the gguf is fine, only 2Gb. I have done some testing with different size of gguf, always the same error. I really think this is due to the driver as this is not reproductible on win10 with version 572. I can try to detect when the issue occur with CudaMemset but i am quite sure this star…

Well, no need to check if the file is too big, It fails at first Mb … aginies@ryzen9:~/testcuda> cuda-gdb --args ./a.out -f qwen2.5-coder-3b-instruct-q4_0.gguf NVIDIA (R) cuda-gdb 12.8 .... Reading symbols from ./a.out... (cuda-gdb) run Starting program: /home/aginies/testcuda/a.out -f qwen2.5-cod…

suggestions: simplify your code. If the problem is related to cudaMemset there should be no need to load a file. Hardcode the sizes. File a bug

Could it also be a defect in the video RAM? Can you clear the memory with a kernel setting to 0 instead of cudaMemset? Even if it happens in the first MB, does it also happen if you allocate much less than 2 GB? 2^31 does not fit into signed 32 bit integers and can lead to bugs or limitations (e.…

Hmm, i doubt → Why this is working under Windows? I can load very big gguf without any issue (using ollama).

Just FYI: after 2 months, the bug is still present, even with latest beta version released today and cuda 12.9. Nvidia was not able to find the root cause.

For others who are reading these things: [image] aginies: Nvidia was not able to find the root cause. Specifically, NVIDIA has not been able to reproduce or observe the issue in-house with production level 5080 and 5090 boards. 5112735 [image] aginies: with latest beta version released …

cudaMemset: illegal memory access with RTX5090 with 570.86.16

Accelerated Computing CUDA CUDA Programming and Performance

Robert_Crovella February 14, 2025, 2:32pm 7

suggestions:

simplify your code. If the problem is related to cudaMemset there should be no need to load a file. Hardcode the sizes.
File a bug

Topic		Replies	Views
Consistent "CUDA error: an illegal memory access was encountered" Error Compute Sanitizer cudnn , cublas , cusolver , cufft , cusparse , curand	7	2057	January 20, 2026
Cuda Driver (an illegal memory access was encountered) DeepStream SDK	8	2403	May 4, 2024
GPU Illegal Memory Access when using pycuda/tensorrt with FP16 input TensorRT	3	1489	December 1, 2022
Illegal access memory error Linux cuda	0	489	August 19, 2020
Misaligned access violation CUDA Programming and Performance	15	4600	July 7, 2017
S1070 device 0 broken Test case provided CUDA Programming and Performance	10	4474	June 9, 2009
Titan RTX memory access error/malfunction/bug Linux	0	648	March 6, 2020
cudaErrorIllegalAddress Encountered: "CUDA error: an illegal memory access was encountered" CUDA Programming and Performance cudnn , cublas , cusolver , cufft , cusparse , curand	1	1870	January 20, 2026
Cuda libraries have memory errors CUDA Programming and Performance	0	3659	August 20, 2011
cuDNN crashes ever since an error during training cuDNN	6	6412	July 28, 2020

cudaMemset: illegal memory access with RTX5090 with 570.86.16

Related topics