In the production environment, I want to find the peak gpu memory usage of my cuDNN based algorithm by running it every day as part of the test automation setup. Can I use CUPTI to extract the peak gpu memory usage of the CUPTI session? I wish to retrieve the peak gpu memory usage information in my test application rather than visualizing the profiler logs.
CUPTI can be used to track the user allocations done using CUDA APIs like cudaMalloc, cudaMallocManaged etc. But it doesn’t track allocations done by the CUDA driver. CUDA API cudaMemGetInfo or NVML API nvmlDeviceGetMemoryInfo should give better picture of total and available memory.