Rl python samples issue of jetbot_train.py usd_plugin: Used expired 'Material' prim </World/Env/Light/OmniverseKitViewportLightMesh/Looks/GizmoTex>

Autonomous Machines Isaac Omniverse Isaac Sim
, ,
#1

I was trying to run the RL Python Sample (jebot_train.py), and got into the following issue, the app just crashed when loading.

the logs as following.
2021-01-25 02:17:22 [33,309ms] [Error] [omni.client.plugin] Main: usd_plugin: Type mismatch for </World/Env/Tiles/Tile_29/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’
2021-01-25 02:17:22 [33,309ms] [Warning] [omni.usd] Coding Error: in _SetValueImpl at line 5716 of /opt/buildagent-share/work/da639afa0455b478/USD/pxr/usd/lib/usd/stage.cpp – Type mismatch for </World/Env/Tiles/Tile_29/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’

2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa57c2d1ac0 released with 11 waiters; spinning until waiters finish
2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa37e37bd90 released with 20 waiters; spinning until waiters finish
2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5bcb0e660 released with 78 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,117ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5b4b2f650 released with 11 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,117ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5a00a8570 released with 10 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,118ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5981ba2b0 released with 8 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,122ms] [Warning] [carb.tasking.plugin] Counter 0x7fa590003d10 released with 7 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,209ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5d05393b0 released with 18 waiters; spinning until waiters finish
2021-01-25 02:17:23 [34,250ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5a01c6330 released with 18 waiters; spinning until waiters finish
2021-01-25 02:17:23 [34,488ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5cd0409e0 released with 19 waiters; spinning until waiters finish
2021-01-25 02:17:24 [35,239ms] [Warning] [carb.tasking.plugin] Counter 0x7fa58cb0f6d0 released with 9 waiters; spinning until waiters finish
2021-01-25 02:17:25 [36,397ms] [Error] [omni.client.plugin] Main: usd_plugin: Used expired ‘Material’ prim </World/Env/Light/OmniverseKitViewportLightMesh/Looks/GizmoTex>
2021-01-25 02:17:25 [36,397ms] [Error] [omni.client.plugin] Main: usd_plugin: Fatal error in void pxrInternal_v0_19__pxrReserved__::Usd_IssueFatalPrimAccessError(const pxrInternal_v0_19__pxrReserved__::Usd_PrimData*)
Aborted (core dumped)

nvidia-smi
Mon Jan 25 10:29:19 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 TITAN RTX Off | 00000000:17:00.0 Off | N/A |
| 41% 37C P8 14W / 280W | 814MiB / 24220MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 3090 Off | 00000000:65:00.0 On | N/A |
| 82% 70C P2 195W / 350W | 1417MiB / 24265MiB | 35% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

#2

Hello, the patch for this issue is available at jetbot/isaacsim_patch at master · hailoclu/jetbot · GitHub
You can follow the instructions on how to apply the patch Here

for future Isaac Sim releases, the patch won’t be necessary.

#3

Already applied the patch.
when i run it in headless mode, it crashed, as following.
2021-01-27 01:36:43 [9,026ms] [Error] [omni.client.plugin] Main: usd_plugin: Type mismatch for </World/Env/Tiles/Tile_03/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’
2021-01-27 01:36:43 [9,026ms] [Warning] [omni.usd] Coding Error: in _SetValueImpl at line 5716 of /opt/buildagent-share/work/da639afa0455b478/USD/pxr/usd/lib/usd/stage.cpp – Type mismatch for </World/Env/Tiles/Tile_03/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’

2021-01-27 01:36:46 [12,139ms] [Warning] [carb.datasource-omniclient.plugin] carb::extras::HandleDatabase<TrueType, HandleType>::~HandleDatabase() [with TrueType = carb::datasource::Subscription; HandleType = long unsigned int]: had 13 outstanding handle(s) at shutdown
2021-01-27 01:36:46 [12,139ms] [Warning] [carb.audio.context] 1 contexts were leaked
terminate called without an active exception
Aborted (core dumped)

And with some file generated:
-rw------- 1 terry terry 1983064 1月 27 09:36 9a885403-f4b3-42f8-2fbc8580-92ee42f0.dmp
-rw-r–r-- 1 terry terry 593 1月 27 09:36 crash_2021-01-27_09-36-47_11998.txt

cat crash_2021-01-27_09-36-47_11998.txt
libpthread.so.0!funlockfile
libc.so.6!gsignal
libc.so.6!abort
libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler()
libcarb.crashreporter-breakpad.plugin.so!carb::crashreporter::Context::_terminateHandler()
libstdc++.so.6!__cxxabiv1::__terminate(void (*)())
libstdc++.so.6!std::terminate()
libcarb.profiler-cpu.plugin.so!std::unique_ptr<carb::profiler::EventProcessorcarb::profiler::Provider, std::default_delete<carb::profiler::EventProcessorcarb::profiler::Provider > >::~unique_ptr()
libc.so.6!__libc_secure_getenv
libc.so.6!exit
libc.so.6!__libc_start_main
python!_start

#4

@terryzhv83 can you confirm that you created/imported the tile assets to the correct location?
Instructions for that are here:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/rl_samples.html#acquiring-road-tile-assets

Can you open the individual tile assets once imported?

#5

YES. all the assets as imported as document.
Actually, I can train the model from the original py files instead of the patch. the patch could crash. and I notice one of the patch is to fix the render performance issue, that one is applied anyway.

#6

In the System Monitor, could you watch out for remnant python processes that take up lots of Memory > 1GB? I.e Kill all the old python processes before launching a new training again.

After that, can you try reducing the PPO’s batch_size? Thanks.

#7

no remnant processes at all.
I reviewed the patch code, and found out that the default location of the checkpoint is not reset. so i set it to empty, and it works now. the code is as following.

if name == “main”:

parser = argparse.ArgumentParser()
parser.add_argument(
    "loaded_checkpoint",
    help="path to checkpoint to be loaded",
    #default="/home/hmazhar/repos/omni_isaac_sim/source/python_samples/eval_log/best_model.zip",
    default="",
    nargs="?",
    type=str,
)
#8

Thanks. I updated the file on github.