Cart delivery in Omniverse Isaac Sim + Isaac SDK almost works but fails at dolly docking stage

[infox22oo]

OK we got further than some others who tried this example in Omniverse.

Following the documentation here:
https://docs.nvidia.com/isaac/doc/simulation/ovkit.html#cart-delivery

The summary is that everything seems to work as expected, with the STR transporter recognizing the dolly, approaching it, showing the bounding box, creating a goal to go under the dolly, and then moving towards the goal.

However, viewing this in sight, it appears that as the STR transporter robot is getting closer to the dolly, the goal itself is not stationary and instead also moves towards the STR. So it reaches the success status of being at the goal position before the STR is under the dolly.

Then the lifter is activated, and the spherical model shape of the robot changes as if the dolly was in fact docked, and the STR moves to the destination.

Here are some screenshots:

Omniverse view of STR approaching dolly, everything looks ok:

Sight view of STR approaching dolly, everything looks ok, blue robot shape in the planner view is moving towards the green goal:

Now the new goal is generated with the goal position initially under the dolly, centered between the 4 wheel obstacles. But almost immediately it seems that the goal position is moving towards the STR just as the STR is approaching the goal, and this shouldn’t happen

Now the STR has reached its goal and changed its shape, but the dolly is not on the STR

It looks like we’re very close in seeing this example of cooperation between Isaac SDK and Omniverse Isaac Sim working.

We do understand that this example was originally developed in Unity3D and all the training was probably done in Unity3D. And it’s amazing that Nvidia managed to create a simulation engine that has much superior rendering and physics and yet compatible enough that the simulation almost works the same.

So, 3 questions:

1. Is the assumption correct that this example was fully functional with Unity3D ? The only system we have that still runs Unity3D doesn’t have enough VRAM to run this example.

2. was this ever fully functional with Omniverse Isaac Sim ?

3. we didn’t modify any source code, and in fact re-loaded the Omniverse Isaac SDK assets so everything is as it was distributed. Any idea why the example isn’t working now ?

Turning anti-aliasing on makes the image more pleasing without change in behavior. Turning reflections and translucency off increases frame rate but that also doesn’t change the behavior.

The results are repeatable, and moving things around for a slightly different initial position also doesn’t change the behavior.

Looks like the RTX2060 is not running out of resources:

We tried upgrading to RTX3060 but that causes TensorRT errors, crashing the Isaac SDK process.

Thanks for any insight

[infox22oo]

Found that if you have a slow PC+GPU, odometry in Omniverse Isaac Sim gets confused. We were able to get the example running with some small changes to the source code.

Also ordered additional new PCs with old graphics cards. Cannot use new graphics cards because then TensorRT complains about serialization error even if “force engine update” is selected. Don’t know if there’s a solution for that.

Also new cards need newer drivers that cause upgrades of CUDA beyond 10.2, then there’s a mismatch with real robots that are running CUDA10.2 with Jetpack 4.51 (currently latest supported by Isaac SDK)

[hshah]

Team, i am not the right person to help with this. Looks like you have chosen wrong name. I am from IT. However, if you still need help let me know.

[infox22oo]

Thank you, this was posted in the Omniverse Isaac Sim forum without referencing anyone specific.

You are right, this one is not an IT question. It is more related to the implementation details of how Isaac SDK interacts with Omniverse Isaac Sim, especially under non ideal conditions of a PC and graphics card being slower than the required minimum and apparently confusing the odometry calculation.

At the moment we’re all set with this question. A variety of upgraded PCs and Nvidia GPU cards have been ordered. If the issue persists we’ll open a new topic.