`cuCtxCreate` and `cuCtxDestroy` pairs have a memory leak

Accelerated Computing CUDA CUDA Programming and Performance
,
1

image
image573×582 35.5 KB

Here is my test code. I use memory profile of visual studio performance profiler for take snapshot of heap.

image

The results suggest that there is a leak.

My graphic card and driver version is NVIDIA RTX 3080 Ti and 537.13

and Here is CUDA Toolkit version information.

{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "12.1.1"
   },
   "cuda_cccl" : {
      "name" : "CUDA C++ Core Compute Libraries",
      "version" : "12.1.109"
   },
   "cuda_cudart" : {
      "name" : "CUDA Runtime (cudart)",
      "version" : "12.1.105"
   },
   "cuda_cuobjdump" : {
      "name" : "cuobjdump",
      "version" : "12.1.111"
   },
   "cuda_cupti" : {
      "name" : "CUPTI",
      "version" : "12.1.105"
   },
   "cuda_cuxxfilt" : {
      "name" : "CUDA cu++ filt",
      "version" : "12.1.105"
   },
   "cuda_demo_suite" : {
      "name" : "CUDA Demo Suite",
      "version" : "12.1.105"
   },
   "cuda_nvcc" : {
      "name" : "CUDA NVCC",
      "version" : "12.1.105"
   },
   "cuda_nvdisasm" : {
      "name" : "CUDA nvdisasm",
      "version" : "12.1.105"
   },
   "cuda_nvml_dev" : {
      "name" : "CUDA NVML Headers",
      "version" : "12.1.105"
   },
   "cuda_nvprof" : {
      "name" : "CUDA nvprof",
      "version" : "12.1.105"
   },
   "cuda_nvprune" : {
      "name" : "CUDA nvprune",
      "version" : "12.1.105"
   },
   "cuda_nvrtc" : {
      "name" : "CUDA NVRTC",
      "version" : "12.1.105"
   },
   "cuda_nvtx" : {
      "name" : "CUDA NVTX",
      "version" : "12.1.105"
   },
   "cuda_nvvp" : {
      "name" : "CUDA NVVP",
      "version" : "12.1.105"
   },
   "cuda_opencl" : {
      "name" : "CUDA OpenCL",
      "version" : "12.1.105"
   },
   "cuda_sanitizer_api" : {
      "name" : "CUDA Compute Sanitizer API",
      "version" : "12.1.105"
   },
   "libcublas" : {
      "name" : "CUDA cuBLAS",
      "version" : "12.1.3.1"
   },
   "libcufft" : {
      "name" : "CUDA cuFFT",
      "version" : "11.0.2.54"
   },
   "libcurand" : {
      "name" : "CUDA cuRAND",
      "version" : "10.3.2.106"
   },
   "libcusolver" : {
      "name" : "CUDA cuSOLVER",
      "version" : "11.4.5.107"
   },
   "libcusparse" : {
      "name" : "CUDA cuSPARSE",
      "version" : "12.1.0.106"
   },
   "libnpp" : {
      "name" : "CUDA NPP",
      "version" : "12.1.0.40"
   },
   "libnvjitlink" : {
      "name" : "JIT Linker Library",
      "version" : "12.1.105"
   },
   "libnvjpeg" : {
      "name" : "CUDA nvJPEG",
      "version" : "12.2.0.2"
   },
   "libnvvm_samples" : {
      "name" : "NVVM Samples",
      "version" : "12.1.105"
   },
   "nsight_compute" : {
      "name" : "Nsight Compute",
      "version" : "2023.1.1.4"
   },
   "nsight_vse" : {
      "name" : "Nsight Visual Studio Edition (VSE)",
      "version" : "2023.1.1.23089"
   },
   "nvidia_driver" : {
      "name" : "NVIDIA Windows Driver",
      "version" : "531.14"
   }
}
2

Please don’t post code as a picture on these forums.

When people make reports like this, I usually suggest to re-verify results on the latest CUDA version and driver available.

If the results are confirmed, you may wish to file a bug.

1 Like
3

Since this has host memory (usage) in view, its reasonable to conjecture that it might be OS-specific. But FWIW on linux I did not observe any change in process memory usage using ps -eo size,pid over 2500 loops of the above code, on CUDA 12.2. (Likewise there was no gradual increase in device memory usage as reported by nvidia-smi.)

4

I updated CUDA version to 12.2 and i ran over 100 loops of the above code.

image

This is result and still show memory leak.

and I don’t think it’s a matter of OS, because of profiler shows unresolved allocations heap memory allocated by nvcuda64.dll.

This is 4 allocations call stack.

5,720bytes

ntdll!0x7ffe8886d041()
nvcuda64!0x7ffe3035240f()
nvcuda64!0x7ffe30233d2d()
nvcuda64!0x7ffe2fe99282()
nvcuda64!0x7ffe2fe996a0()
nvcuda64!0x7ffe2fe986cf()
nvcuda64!0x7ffe2ff47856()

5,720bytes

ntdll!0x7ffe8886d041()
nvcuda64!0x7ffe3035240f()
nvcuda64!0x7ffe30233d2d()
nvcuda64!0x7ffe2fe99282()
nvcuda64!0x7ffe2fe997b3()
nvcuda64!0x7ffe2fe986cf()
nvcuda64!0x7ffe2ff47856()

320bytes

ntdll!0x7ffe8886d041()
nvcuda64!0x7ffe3035240f()
nvcuda64!0x7ffe300aaa2c()
nvcuda64!0x7ffe2ff57a2f()
nvcuda64!0x7ffe2ffd682c()
nvcuda64!0x7ffe2ff49591()
nvcuda64!0x7ffe2ff4a399()

320bytes

ntdll!0x7ffe8886d041()
nvcuda64!0x7ffe3035240f()
nvcuda64!0x7ffe300aaa2c()
nvcuda64!0x7ffe300aae9f()
nvcuda64!0x7ffe2ffd6aa6()
nvcuda64!0x7ffe2ff49591()
nvcuda64!0x7ffe2ff4a399()

6

Referring this to NVBUG 4308741
We will keep you updated . The engineering team will investigate this .