Isaac Sim Multi GPUs get slower proformance than one GPU

Hi everyone,
I am very confused about why multi gpus get slower proformance.
Our company bought two RTX A6000 gpus for Isaac Sim issue.
I run the same project by RXT 5000 and RTX A6000 in real time mode,
RTX 5000 can run about 14-24 fps, but with two RTX A6000 s only has lower than 4 fps.

Are there any properties that I need to setup? Anyone can give some good suggestions?

Can you run the nvidia-smi to make sure the Isaac Sim is using the both GPUs?

Hi @ltorabi ,
Following is my GPU usage memory, but I’m not sure if it uses multi GPUs.
My Isaac Sim version is 2022.2.0.


And following is one GPU usage memory picture.

Hi @ltorabi ,
The following pictures is the same project run at the Isaac Sim with different GPU.
As you see, the fps in RTX 5000 is faster than A6000. And I’m also confused at the
memory usage. Why the same project run at two GPUs use more memories?
image
image

@vic-chen , this is strange, would you please do full profiling, so we can see if this is caused by cpu limitation

Hi @vic-chen - Are you seeing similar issues on the latest Isaac Sim 2022.2.1 release as well?

Hi @ltorabi ,
The attachments are the profiles of RTX 5000 and RTX A6000.
They both run at Isaac Sim 2022.2.0 version with the same project.
image
image

RTX5000.gz (60.4 MB)
RTXA6000.gz (28.0 MB)

Hi @rthaker ,
Yes, they have the same issues on Isaac Sim 2022.2.1.

Hi @rthaker and @ltorabi ,
Because of the RTX A6000 computer ran my project with low FPS, so I edited my project on RTX 5000’s computer.
And after finished the edit,I copy my project to RTX A6000’s computer. And the following picture you can see, the second GPU was stopped. I didn’t change any settings but it was be closed!
And you can also see that the FPS become faster! I have no idea to deal with the problem.
Can you give some advices?

image

Hi. Can you try the latest production branch 525 drivers instead?

Try creating a second viewport to see if the second GPU is used.

Are able to reproduce this issue with the latest Create or Code?

Hi @Sheikh_Dawood ,
My OS system is windows, because of some company’s policy that I need use windows system. But I’ll try to change the display driver with 525.89.02 version is that OK?
This issue have been run at Create 2022.3.1 and have the same problem. I’ll try it at Cate 2022.3.3 and see if it has the same issue.
Thanks for your advice, have a nice day!

Hi @Sheikh_Dawood ,
I can’t find windows version 525.89.02 on official website, could you give me a download link?

You can download the latest driver from here or using the Geforce Experience app.

Can you please provide the log files for Create too. We will try to reproduce the issue internally.

Hi @Sheikh_Dawood ,
I can only download version 528.89 on official website. I can’t find 525.89.02 version.
And new version driver didn’t change anything.


Hi @Sheikh_Dawood ,
I reinstalled Isaac Sim 2022.2.1 and Create 2022.3.3, and following picture is current test result. As you see, new version fps faster than before.But I’m not sure if it uses 2 GPUs. And the attachments are Create log and Isaac Sim’s profile.


Create_log.log (50.9 MB)
IsaacSimProfile.gz (52.9 MB)

Hi. The latest 528 or 531 drivers should be fine.
Looks like Create is also showing a greyed out text and using both GPUs. I think this is fine.
In Isaac Sim, try running a demo scene and create a Viewport 2 with a different camera view. You should see the second GPU being used.

Hi @rthaker @ltorabi @Sheikh_Dawood ,
I found why my project run at RTX 5000 can faster than RTX A6000.
The following picture is the really FPS when I run my project. RTX 5000
can faster than RTX A6000 is due to extend module not be autoload.
But two RTX A6000 gpus only can faster 0.5 fps was disappointing.
image
image

Sorry, I need update new information. I found my project lost some materials.
After fixed it, the really FPS was RTX 5000 faster than RTX A6000 about 1 FPS.

p1

Thanks for the update @vic-chen

@vic-chen Have you tried the above tests with two viewports enabled?
Adding the second GPU may not necessarily increase FPS. It will be useful for multiple viewports/sensors/robots scenarios.

Hi @Sheikh_Dawood ,
I have been tried use two viewports, it really helpful for view scenarios.
My project is tried to build a digit twins system by use omniverse.
According to nvidia offical in Taiwan suggests us to upgard hardware so that we bought two RTX A6000 gpus.
But as you said, it seems even though we add more gpus, it may not increase the performance. I appreciate you taking the time to answer this question.
So that I can tell my boss the proejct seems can’t keep going.