Can't render first frames with manual World stepping

Omniverse Synthetic Data Generation (SDG)
1

I’m following this example Frequently Used Python Snippets — Omniverse Robotics documentation. It works fine but if I change the for loop (from line 62 to 75) with the following code:

for i in range(10):
    world.step(render=True, step_sim=True)
    write_rgb_data(rgb_annot.get_data(), f"{out_dir}/Cube_{i}_step_rgb")

It throws several errors:

Traceback (most recent call last):
  File "example.py", line 73, in <module>
    write_rgb_data(rgb_annot.get_data(), f"{out_dir}/Cube_{i}_step_rgb")
  File "example.py", line 21, in write_rgb_data
    *rgb_data.shape, -1
ValueError: cannot reshape array of size 0 into shape (0,0,4,newaxis)
2023-05-05 10:42:24 [46,063ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,063ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,063ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,063ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,063ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,064ms] [Warning] [omni.usd] Warning: in operator() at line 95 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyFunction.h -- Tried to call a method on an expired python instance

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Warning] [omni.usd] Warning: in operator() at line 95 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyFunction.h -- Tried to call a method on an expired python instance

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,065ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Warning] [omni.usd] Warning: in operator() at line 95 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyFunction.h -- Tried to call a method on an expired python instance

2023-05-05 10:42:24 [46,066ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-05-05 10:42:24 [46,066ms] [Warning] [omni.usd] Warning: in operator() at line 95 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyFunction.h -- Tried to call a method on an expired python instance

2023-05-05 10:42:24 [46,226ms] [Warning] [carb.audio.context] 1 contexts were leaked
/home/federico/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh: line 41: 2315329 Segmentation fault      (core dumped) $python_exe "$@" $args
There was an error running python

It can’t render the image (it renders a 0x0x4 image).
It also raise a segmentation fault.

I solved it with a warmup:

for _ in range(5):
    world.step(render=True, step_sim=True)

for i in range(10):
    world.step(render=True, step_sim=True)
    write_rgb_data(rgb_annot.get_data(), f"{out_dir}/Cube_{i}_step_rgb")

I don’t know why this works. Moreover I encountered a couple of problems:

  • it is not clear how many warmup steps are needed
  • sometimes the behavoiur is not deterministic, for example with 3 warmup steps, sometimes it works and sometimes it throws the errors shows above

I’m using Isaac 2022.2.1.

2

Hi @federico.domeniconi. This is focused on Replicator code so I’m moving it to the SDG forum.

3

Crash seems unrelated to the empty array error. If you comment out simulation_app.close(), you will get a crash. As for the warmup frames, they are always needed because the renderer already renders some frames when the rgb color is not enabled. When you run the script, the first couple of frames you get is when rgb is not enabled. 3 calls of omni.kit.app.get_app_interface().update() should be enough to get rid of this problem. @mati-nvidia Can you let the isaac sim team know if it is possible to add this warmup frames into the function World.step()?

4

And all these warnings are benign.