Recommended versions w/ RTX3070 to run SIM, transfer learning, Jetson

I have Isaac Sim running via Omniverse. Just trained object detection using the given docker. OOPS - the RTX3070 isn’t supported on the tensorRT7.0 in the docker (tltexport). Forum claims I also need to upgrade CUDA to 11.2.

Any recommendations to run both Isaac SIM as well as the TLT toolkit on a desktop with an RTX3070?

Like what are the real versions that Isaac SIM will work with - the recommended ones (version_checker.py) are too low for TLT.

Thanks!

p.s. Jetson is next - after all this is working…

Hello,

For TLT you require a driver newer than 455 ( TLT Quick Start Guide — Transfer Learning Toolkit 3.0 documentation (nvidia.com)), while for Isaac sim Omniverse there is a strict requirement of 460.67: Isaac Sim Requirements — Omniverse Robotics documentation (nvidia.com)

For your use case, I would only use 460.67 and install properly docker for TLT. Now, you should not need to worry about cuda, or TensorRT version, since those will be handled by docker in the background. This webinar walks you through TLT and how to deploy on jetson: Webinar: Create Gesture-Based Interactions with a Robot | NVIDIA Developer Blog

One thing to be cautious when moving to jetson is the versions of Jetpack, that will determine your TRT version, which in the case of jetson matters unlike in the workstation. The thing you should make sure is that the tlt-converter version matches your jetpack version.

Let me know if this answers your question :)

Good answer - though more than I had asked for!!

I guess I’m more disappointed with the range of documentation and examples. Some don’t work, some are partially out’o’date (like the object detection docker via Isaac sim - which dies on RTX3070 due to too old versions), to the conflicting versions ‘required’ for sim/sdk and TLT and perhaps what the forums have discovered.

I get this from a github. But NVidia has more resources and reason to do a good job in this area than anyone!

Thanks again - sorry for the rant.

btw, I still see a looming challenge. Jetpack 4.5 (and the rest) use tensorRT 7.1.3. This version doesn’t support the desktop RTX3070 card.

So I guess the key issue to me is resolving the tltconverter - can it handle the different version. I’m not there yet, may be peppering the forum soon!

Hello,
Don’t worry about the TRT version. Just get to an etlt model and then make sure to have the proper tlt-converter (exact version match to the jetpack) and the converter will worry about the right TRT, not you.

Hi Peter,

You mentioned “object detection docker via Isaac sim” , if this (Training Object Detection from Simulation in Docker — ISAAC 2021.1 documentation) is what you are using, then that is not Omniverse Isaac Sim, is the old Unity version.

Kindly,
Liila

Okay - thanks! But on the website, it sure purports to be active and usable. And more easily found than the one a previous person now has thankfully pointed me to.

And I really want to do 3D Object Pose prediction. That would seem to be an older Unity thing - and TLT doesn’t have a model in that space. Ughh. I’m going to have to get something working in that space.

Thanks again for the response! And I really wish NVidia would clean up there training/educational resources for num-nums like me…

This is what I want to do - 3D Object Pose Estimation with Pose CNN Decoder — ISAAC 2021.1 documentation

While clearly for Unity Sim, this document has at least been updated for Omniverse - a little. In the text - around 5 paragraphs down, it says “However, the training application is agnostic to the simulation platform used to generate the data and can be replaced with any other simulation engines like Isaac Sim based on NVIDIA’s Omniverse Kit.”.

Should I just try this and flail? Again - noting that the embedded versions are probably too old for the RTX3070 GPUs? Or is there an update/workaround?

(Perhaps I should make this a top-level forum topic, eh?)

Thanks again for your response!

Hi Peter,

Sorry for the confusion.
Here is the summary, it might clear stuff a bit for you.

  • You can use Isaac SDK with Isaac Sim Unity or Omniverse Isaac Sim
  • Unity Isaac Sim is the older version, and although it is still supported in the recent Isaac SDK release, but this is the last release that support Unity Isaac Sim. From next release is going to be only Omniverse Isaac Sim
  • Please go here and download Omniverse launcher and from there install and launch Isaac Sim . I am sure you will love it
  • RTX 3070 is an RTX GPU therefore perfect for Omniverse Isaac Sim
  • Unfortunately there is no docker version of Omniverse Isaac Sim for object detection
  • As pointed in the Isaac SDK doc which you also mentioned , you can do object detection and pose estimation in Omniverse IsaacSim. please try it out and ask in the forum if you hit any issue

kindly,
Liila

Thanks for your response - and I am exploring this path.

The key question is whether there are updated learning resources to help, in say - 3D Object Pose Estimation with Pose CNN Decoder — ISAAC 2021.1 documentation

This version requires the Isaac unity sim3d version, prior to the omniverse version. e.g., the first command to execute is…
sktop:~/isaac_sim_unity3d$ ./builds/factory_of_the_future.x86_64 --scene Factory01 --scenario 8

Any help available on the learning resources? Will they be updated? Or are we on our own here?

Thanks - p

Hi Peter,

For NVIDIA the path forward is Omniverse. which is NVIDIA’s own engine instead of using (Unity).
So Unity Isaac Sim will not be supported in future, but there will be a lot of development in Omniverse apps including Isaac Sim.
Please do read the following examples in the Isaac Sim docs and see if you find it useful:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/sample_syntheticdata.html#offline-training-with-tlt
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/sample_syntheticdata.html#online-training

Kindly,
Liila

Thank you. Perfect.

Hi Peter,
Have you been able to train or test 3D pose estimation or object detection in Isaac Sim on Omniverse?
Thank you.

I was very successful in creating simulated data (non-programmatic way) and then using TLT to train object detection. Happy with that - see image below!

Creating simulated data the programmatic way is the current task at hand. a) documentation is wrong, e.g., debugging code in vs code. b) still have to figure out how to save pose - I’ll probably do that myself and not use the built-in visualization tools - or some combination. Still open as to how to do the ML pose estimation - I’ll try what they have (I think there are 3 potential approaches on the NVidia website - there is no TLT solution), but I feel I may have to do my own ML, based on something like PoseCNN or whatever.

004609

1 Like

So have you used the synthetic data generation for your data collection and used TLT to train your model?
how do you test the trained model in the warehouse scene?
Also I have a GPU of RTX 2070. That wouldn’t work for TLT ?

Yes to first question. I think your second question is whether this will work in the real world. The assumption of course is yes - that’s why I’m doing this, but proving will require more work.

RTX2070. Got me, it probably will work. TLT uses a docker setup w/ the right versions. Make sure you get the most recent one - the previous one died on the RTX class of GPUs. You can always use Keras/whatever and not use TLT. Then you can train on any GPU, even non-GPU.

Thank you for your response.
For my second question I wanted to know if you modified any scripts or did you make your own object detection code and used that with the robot engine bridge to get the box around the detected object?

That is provided code in the Jupyter notebook I used (from NVidia)

Could you please share the link to the instruction if there is any?
is it this: Object Detection with DetectNetv2 — ISAAC 2021.1 documentation

I used the first link back on the first response in this thread - TLT Quick Start Guide — Transfer Learning Toolkit 3.0 documentation

Not sure, but the link you have may be the one that didn’t work for me…

1 Like