CUDA memory unavailable despite clearing cache and garbage

I am using AirSim with an ESPNet segmentation inference model on a GeForce RTX 2060 SUPER. It was working until CUDA started showing the OOM error. Despite nvitop and nvidia-smi showing free GPU memory, torch.cuda.mem_get_info() is showing that atleast 70% of my memory is filled (even going above 90% in sometimes).
I tried torch.cuda.empty_cache and gc.collect() but they aren’t helping me.

What can be the reason torch.cuda.mem_get_info() is showing filled memory? Please suggest some solutions to free up GPU memory.

my general suggestion for people asking pytorch questions here is that you may get better help by asking on a pytorch forum. This isn’t a pytorch forum. The control of CUDA memory in pytorch is handled by pytorch.

Sure, Robert, I’ve put it up on the PyTorch forum.
Here it is in case someone stumbles here: