I was using a system with 4 RTX GPU cards trying to run an MPI program and overloading the GPUs with more than 1 process per card using MPS. This program runs with a single process per GPU and MPS turned on. When I try to run more than 1 process per GPU, the simulation will hit a segmentation fault and crash. This occurs about 80% of the time.
Note that this does not occur a multi-v100 system running the same driver and OS.
The driver I was 440.82
The system was running CentOS