Rl python samples issue of jetbot_train.py usd_plugin: Used expired 'Material' prim </World/Env/Light/OmniverseKitViewportLightMesh/Looks/GizmoTex>

I was trying to run the RL Python Sample (jebot_train.py), and got into the following issue, the app just crashed when loading.

the logs as following.
2021-01-25 02:17:22 [33,309ms] [Error] [omni.client.plugin] Main: usd_plugin: Type mismatch for </World/Env/Tiles/Tile_29/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’
2021-01-25 02:17:22 [33,309ms] [Warning] [omni.usd] Coding Error: in _SetValueImpl at line 5716 of /opt/buildagent-share/work/da639afa0455b478/USD/pxr/usd/lib/usd/stage.cpp – Type mismatch for </World/Env/Tiles/Tile_29/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’

2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa57c2d1ac0 released with 11 waiters; spinning until waiters finish
2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa37e37bd90 released with 20 waiters; spinning until waiters finish
2021-01-25 02:17:22 [33,589ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5bcb0e660 released with 78 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,117ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5b4b2f650 released with 11 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,117ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5a00a8570 released with 10 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,118ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5981ba2b0 released with 8 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,122ms] [Warning] [carb.tasking.plugin] Counter 0x7fa590003d10 released with 7 waiters; spinning until waiters finish
2021-01-25 02:17:22 [34,209ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5d05393b0 released with 18 waiters; spinning until waiters finish
2021-01-25 02:17:23 [34,250ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5a01c6330 released with 18 waiters; spinning until waiters finish
2021-01-25 02:17:23 [34,488ms] [Warning] [carb.tasking.plugin] Counter 0x7fa5cd0409e0 released with 19 waiters; spinning until waiters finish
2021-01-25 02:17:24 [35,239ms] [Warning] [carb.tasking.plugin] Counter 0x7fa58cb0f6d0 released with 9 waiters; spinning until waiters finish
2021-01-25 02:17:25 [36,397ms] [Error] [omni.client.plugin] Main: usd_plugin: Used expired ‘Material’ prim </World/Env/Light/OmniverseKitViewportLightMesh/Looks/GizmoTex>
2021-01-25 02:17:25 [36,397ms] [Error] [omni.client.plugin] Main: usd_plugin: Fatal error in void pxrInternal_v0_19__pxrReserved__::Usd_IssueFatalPrimAccessError(const pxrInternal_v0_19__pxrReserved__::Usd_PrimData*)
Aborted (core dumped)

nvidia-smi
Mon Jan 25 10:29:19 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 TITAN RTX Off | 00000000:17:00.0 Off | N/A |
| 41% 37C P8 14W / 280W | 814MiB / 24220MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 3090 Off | 00000000:65:00.0 On | N/A |
| 82% 70C P2 195W / 350W | 1417MiB / 24265MiB | 35% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

Hello, the patch for this issue is available at jetbot/isaacsim_patch at master · hailoclu/jetbot · GitHub
You can follow the instructions on how to apply the patch Here

for future Isaac Sim releases, the patch won’t be necessary.

Already applied the patch.
when i run it in headless mode, it crashed, as following.
2021-01-27 01:36:43 [9,026ms] [Error] [omni.client.plugin] Main: usd_plugin: Type mismatch for </World/Env/Tiles/Tile_03/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’
2021-01-27 01:36:43 [9,026ms] [Warning] [omni.usd] Coding Error: in _SetValueImpl at line 5716 of /opt/buildagent-share/work/da639afa0455b478/USD/pxr/usd/lib/usd/stage.cpp – Type mismatch for </World/Env/Tiles/Tile_03/Looks/Dark_Grey_material/Dark_Grey_material.inputs:emissive_color>: expected ‘SdfAssetPath’, got ‘GfVec3f’

2021-01-27 01:36:46 [12,139ms] [Warning] [carb.datasource-omniclient.plugin] carb::extras::HandleDatabase<TrueType, HandleType>::~HandleDatabase() [with TrueType = carb::datasource::Subscription; HandleType = long unsigned int]: had 13 outstanding handle(s) at shutdown
2021-01-27 01:36:46 [12,139ms] [Warning] [carb.audio.context] 1 contexts were leaked
terminate called without an active exception
Aborted (core dumped)

And with some file generated:
-rw------- 1 terry terry 1983064 1月 27 09:36 9a885403-f4b3-42f8-2fbc8580-92ee42f0.dmp
-rw-r–r-- 1 terry terry 593 1月 27 09:36 crash_2021-01-27_09-36-47_11998.txt

cat crash_2021-01-27_09-36-47_11998.txt
libpthread.so.0!funlockfile
libc.so.6!gsignal
libc.so.6!abort
libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler()
libcarb.crashreporter-breakpad.plugin.so!carb::crashreporter::Context::_terminateHandler()
libstdc++.so.6!__cxxabiv1::__terminate(void (*)())
libstdc++.so.6!std::terminate()
libcarb.profiler-cpu.plugin.so!std::unique_ptr<carb::profiler::EventProcessorcarb::profiler::Provider, std::default_delete<carb::profiler::EventProcessorcarb::profiler::Provider > >::~unique_ptr()
libc.so.6!__libc_secure_getenv
libc.so.6!exit
libc.so.6!__libc_start_main
python!_start

@terryzhv83 can you confirm that you created/imported the tile assets to the correct location?
Instructions for that are here:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/rl_samples.html#acquiring-road-tile-assets

Can you open the individual tile assets once imported?

YES. all the assets as imported as document.
Actually, I can train the model from the original py files instead of the patch. the patch could crash. and I notice one of the patch is to fix the render performance issue, that one is applied anyway.

In the System Monitor, could you watch out for remnant python processes that take up lots of Memory > 1GB? I.e Kill all the old python processes before launching a new training again.

After that, can you try reducing the PPO’s batch_size? Thanks.

no remnant processes at all.
I reviewed the patch code, and found out that the default location of the checkpoint is not reset. so i set it to empty, and it works now. the code is as following.

if name == “main”:

parser = argparse.ArgumentParser()
parser.add_argument(
    "loaded_checkpoint",
    help="path to checkpoint to be loaded",
    #default="/home/hmazhar/repos/omni_isaac_sim/source/python_samples/eval_log/best_model.zip",
    default="",
    nargs="?",
    type=str,
)

Thanks. I updated the file on github.