cudnnDestroy does not release CPU memory

I am using Titan XP and trying to check whether cudnnDestroy or cudaDeviceReset releases CPU memory. The cudaDeviceReset releases GPU memory but both function do not release (or only release very little) CPU memory. In fact, cudnnDestroy doesn’t release any CPU memory. Here is the simple test code. I also put the measured CPU memory usages in the comment after each cuda function was run for reference.
So what should I expect from cudnnDestroy and cudaDeviceReset?

I am using the nvidia docker at nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04 and my nvidia driver is 418.87.01

#include <stdlib.h>
#include <unistd.h>
#include <cudnn.h>
#include <cuda.h>

int main(){
  
    // CPU RAM usage at 1.16 G
    cudnnHandle_t hdl(0);
    cudnnCreate(&hdl);
    // CPU RAM usage at 1.87 G
    printf("got cudnn, sleep for 5 sec\n");
    sleep(5);
    cudaDeviceSynchronize();
    printf("Destroy status %u\n", cudnnDestroy(hdl));
    printf("free cudnn destory sleep for 5 sec\n");
    // CPU RAM usage at 1.87 G
    sleep(5);
    cudaDeviceReset();
    printf("reset device sleep for 5 sec\n");
    // CPU RAM usage at 1.76 G
    sleep(5);

}```

Hi @mingt,
I am not able to reproduce the issue on system with CUDA 10.2 & cuDNN 7.6.5.
Could you please retry this on latest cuDNN release?
Thanks!

I am still seeing the same problem in which cudnnDestroy does not release any CPU memory.

To test:
Following your reply, I upgraded my driver to 440.82 so that I could use cuda 10.2

I am using the docker container nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04. From the Dockerfile, it is said the docker is using cudnn 7.6.5.32

The docker command to run the docker container:

docker run --gpus all --rm -ti nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04 bash

nvcc command to compile (the code snippet is stored in test.cu)

nvcc test.cu -o test_out -gencode=arch=compute_61,code=sm_61 -I/usr/local/cuda/include -lcudart -lcudnn -L/usr/local/cuda/lib

Please verify. Thanks

Hi @mingt,
Driver may load the libs if the program doesn’t exit and during this time memory loss in GB’s is expected.
Can you please try checking it after exiting the program.
Also, there are some code fixes in cuDNN 8.0, can you please try this on cuDNN 8.0.
Thanks!