Gromacs stalls on T4

Hi Everyone,

We have encountered another issue with GROMACS 2018.8 on T4s with openmpi 2.1.2.
When we try to run some analysis, this happens on a single node with single or multiple gpus, some of the processes (gmx_mpi) will stall on the gpus (they do nothing, no output, not error). We did recently updated the driver and cuda to the latest version.

We are wondering if anyone had run into the same issue or anyone has any insight on what could be the cause.

Thanks!
Delilah