Failed to execute synthetic data generation for pose estimation in IsaacSim 2022.2.0

I’ve been trying to run offline pose generation example in the IsaacSim, however, I’m facing the following

[Warning] [carb.flatcache.plugin] Type tag does not have a corresponding USD type

[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /OmniverseKit_Front

[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /OmniverseKit_Persp

[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /OmniverseKit_Right

[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /OmniverseKit_Top

[Warning] [carb.flatcache.plugin] UsdRelationship /Replicator/SDGPipeline/OgnGroup_02.inputs:prims has multiple targets, which is not supported

[Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0xf2392ef8 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0

[Warning] [omni.synthetic data.plugin] OgnSdInstanceMapping missing valid input renderVar InstanceSemanticMapSD

[gpu.foundation.plugin] Requesting texture to use texture streaming, but the service isn’t available. The texture will be created as a regular resource.

As soon as the script runs, the Sim opens and scene loads but abruptly stops responding. It gets stuck and prompts every time to Force Quit.

Please help me with this persistent issue.

Hi there,

the warnings you are getting can be ignored, the script should still run. The first time you run the script, depending on your hardware, you might need to wait a couple of minutes for the shaders to compile and to download the environment.

Best,
Andrei

Hello!

Could you provide more information about your system properties? Could you also attach a full log of what you see on the terminal when you run Isaac Sim?

Like Andrei mentioned, the warnings you listed above can all be ignored. It should not affect the script. On the first run, it usually takes 5 minutes or so to compile all of the GPU shaders.

  1. System specs:

  2. Ubuntu - 20.04

  3. CPU - Intel Core i7

  4. Cores - 24

  5. RAM - 32 GB

  6. GeForce RTX 3060-Ti

  7. VRAM - 16GB

  8. Disk - 2TB SSD

this is the system spec of the customer.

Thanks @mukundang.

Seeing that you have a 3060Ti, I expect that the GPU shaders will take a few minutes to compile. @deveshkumar21398, how long do you usually wait before force quitting the application?

Hi @andrew_g I kept it running for around 20-30 minutes, however, still it remains stuck and forces to quit. I have also uploaded the screenshot where the freezing is observed in Isaac Sim 2022.1.

Additionally, I have also tried to run the same example in Isaac Sim 2022.2.0. Here as well the screen freezes after following errors and warnings:

Errors:

[Error] [omni.graph.core.plugin] /Replicator/SDGPipeline/OgnWritePrimAttribute_02: OmniGraph Error: WritePrimAttribute Error: ‘xformOp:orient’
(from compute() at line 237 in /home/labuser/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/omni/replicator/core/ogn/python/_impl/nodes/OgnWritePrimAttribute.py)

Warnings:

[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /Replicator/Camera_Xform
[Warning] [omni.hydra.scene_delegate.plugin] cannot find xform op xformOp:rotateXYZ for /Replicator/Camera_Xform
[Warning] [carb.flatcache.plugin] UsdRelationship /Replicator/SDGPipeline/OgnGroup_04.inputs:prims has multiple targets, which is not supported
[Warning] [carb.flatcache.plugin] PathToAttributesMap (0x17e12890) contains attributes with duplicate name “camera” with different types but same per-element size. Data may become corrupted during request to move elements between buckets!

The screen freezes as soon as it tries to update:

Loading materials. Will generate data soon…
Start timestamp: 02/14/2023, 12:33:55
Starting dataset generation of 10 frames…
Starting MESH dataset generation of 5 frames…
ID: 0/9

I did try to debug in Isaac Sim 2022.2.0 and was able to find that it freezes on world.render() and rep.orchestrator.step(). Specifically, whenever omni.kit.app.get_app().update() is run within above mentioned methods. I have tried re-installing both the versions of Isaac Sim as well. However, the result remains unchanged.

Hi @andrew_g yeah sure, please find attached the logs:

For Isaac Sim 2022.2.0 -

kit_20230214_123344.log (888.3 KB)
kit_20230214_121705.log (723.6 KB)

For Isaac Sim 2022.1 -

kit_20230214_134047.log (1.1 MB)
kit_20230214_122441.log (1.1 MB)

Please let me know if anything else is required. I have attached screenshot as well in your other comment.

Lastly, @mukundang have already attached the specifications.

Hi @deveshkumar21398,

Thanks for the logs. Could you try to generate MESH examples only and DOME examples only? I’m curious if this issue is specific to either DOME or MESH.

You can run with the options --num_dome 0 --num_mesh 5 and --num_dome 5 --num_mesh 0.

Hi @andrew_g I just tried what you had suggested in both versions 2022.1.1 and 2022.2.0. However, it is still stuck after waiting for around 10 minutes and forces to quit. Here are the screenshots for --num_dome 5 --num_mesh 0 :

Fig 1:

Fig 2:

For the --num_dome 0 --num_mesh 5 the output is similar to FIg 1 in both versions.

Please let me know if you need logs or terminal output for above executions because those are exactly similar to as shared earlier.

Hi @andrew_g @mukundang could you please let me know whether there is any update on the query? Because I have trying to run other scripts related to offline generation, online generation and pose estimation example and all of them get stuck after a point and remain as it is. I believe there is some fundamental issue because all above examples are using somewhat similar domain of APIs. Please help me with it.

@deveshkumar21398 Our team is able to run the demos without any issues. Please run nvidia-smi while running the script and post the screenshot. This will help us identify more around whats happening

Sure, @mukundang please find below the screenshot of nvidia-smi.

Hi @deveshkumar21398, it looks like you don’t have the Nvidia driver properly installed. Can you follow Q1 on this guide to reinstall your driver?

Let me know if you have any trouble re-installing the driver

Troubleshooting.md (16.5 KB)

Hi @andrew_g thanks for your response. Which driver version need to be installed particularly and whether do I need to reinstall IsaacSim?

Specifications:

Ubuntu - 20.04
GeForce RTX 3060-Ti
Isaac Sim 2022.2.0

Hi, if you visit the link in the document, you can select the right Driver version for your machine. I don’t think you need to reinstall Isaac Sim but if you encounter difficulties, it might be worth to try reinstalling.

Hi @andrew_g I tried installing the driver according to the instructions you had provided earlier. However, post downloading the run file and installing the drivers whenever I reboot the system it does not boot and gets stuck at the manufacturer’s and Ubuntu logo. Subsequently, as soon as I uninstall that driver it does boot and I am at the home screen.

As a matter of fact, the reboot runs successfully whenever I install nvidia drivers either from “Terminal” or from “Software and Updates”, which are not advised according to the instructions.

Moreover, I am trying to install nvidia-530 and Cuda 12.1.

Could you please suggest on what exactly is the issue? The executable driver files does not let the machine reboot whereas PPA drivers do.

Hi @andrew_g @mukundang there’s an update on this issue:

I tried the same example on recently launched new version of IsaacSim, i.e., 2022.2.1 and it worked after waiting for around 10 minutes (additional to downloading time of ray shaders) with PPA drivers itself. Because (as previously commented) I couldn’t workout with .run executable drivers as the Ubuntu wouldn’t boot after that.

However, I guess with previous versions it still remains to be an issue with PPA drivers.

Thank you so much @mukundang and @andrew_g for your help. I am now moving ahead with the latest version of the IsaacSim.

I don’t know if this has something to do with your problem, but when I am using a script for the first time and I need to download props from my nucleus server I need to log in to my nucleus server as admin in my web browser. Until I do that my window is not responding aswell and stuck.
Glad to hear that it works for you now, just wanted to share my experience with frozen windows.

In addition to updating the driver. Try cleaning the cache folder as discussed in this forum post. It might help: Isaac Sim recompiling ray tracing shaders every time it is opened

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.