this bug happens on multi-gpu machines with cuda 12 and v525 or 528. I solved it by sudo rmmod nvidia_uvm from the host. Looks like if 2 processes access this file at the same time, 1st one blocks it and 2nd one hangs and after that any call to cuda gives an error for everybody. I thought it was related to power management, but looks like it’s not. Because after restart if the nvidia_uvm file is there and 2 proc access it, I get immediately that error. On Ubuntu 20.04. So something is wrong with this. Hope that helps trace it further