Hi everyone,
today I tried to split up some A100 40GB PCIe using MIG. I set up the GPU instance and the compute instance it it works fine for training some PyTorch models. However, I am using Mujoco (https://mujoco.org/) to generate data for my models and suddenly, Mujoco performs a magnitude worse (i.e. I had 500 frames/s before, now 40 frames/s).
If I disable MIG, it is all fine again and I am at full performance. I was wondering whether MIG has some restrictions on driver capabilities, since I used to have problems with Mujoco not using EGL properly, which resulted in similar problems.
Best,
Tim