Omniverse Kit Remote Client show python script visuals

I am pretty new to Omniverse toolkit. I am tryin to reproduce this reinforcement learning example with my own tweaks: Reinforcement Learning — Omniverse Robotics documentation
I am able the set up the remote client since my container is running AWS EC2 instance, and able to run all of the interactive examples successfully. I am also able to run headless python examples. I am wondering, however, is there a way to launch a python example such as the following:

./python.sh python_samples/jetbot_soccer/jetbot_train.py 

And show the visual through remote client? Because I want to observe my RL model at the beginning of the training to make sure it is behaving as expected. Any help is appreciated!

Hello @yuxiny5n6lk! I have directed this post to the development team. Thank you for reaching out to us!

Hi Yuxin,

Did you try these steps to connect the Remote Client to the AWS and not passed in the headless option when you train if you want to see?
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/setup.html#kit-remote-client-requirements

parser.add_argument("-H", “–headless”, help=“run in headless mode (no GUI)”, action=“store_true”)

This talk shows the steps to set up AWS for streaming Isaac Sim 2020: Deep Dive - YouTube

If I run without --headless argument, the training script fails with the following script. Here is the step I followed:

  1. SSH into my instance which is running my container with remote client entry point.
  2. Bash into this container:
docker exec -it 7cd7a685240c bash
  1. Run my python training scrip by running:
./python.sh python_samples/jetbot/jetbot_train.py
  1. The script exits with the following error message:
2021-08-23 19:42:25 [402ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [402ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:25 [402ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [402ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:25 [403ms] [Error] [omni.appwindow.plugin] Failed to acquire IWindowing interface
[0.417s] [ext: omni.kit.renderer.core-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 460.73      | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|---------------------------------------------------------------------------------------------|
| 0   | Tesla T4                         | Yes    |     | 15606   MB | 10de      | 0          |
|     |                                  |        |     |            | 1eb8      | b58943f9.. |
|=============================================================================================|
| OS: Linux ip-172-31-21-181, Version: 5.4.0-1048-aws
| Processor: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 255000 | Free Memory: 194038
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2021-08-23 19:42:25 [714ms] [Warning] [carb.cudainterop.plugin] NVML could not be loaded. GPU Utilization stats will be unavailable. Error: libnvidia-ml.so: cannot open shared object file: No such file or directory
2021-08-23 19:42:25 [724ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [724ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,066ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:26 [1,066ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,067ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:26 [1,067ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,492ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2021-08-23 19:42:26 [1,493ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::*)(unsigned int, int, carb::cpp20::latch*), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2021-08-23 19:42:26 [1,496ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-08-23 19:42:26 [1,497ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.appwindow.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)
2021-08-23 19:42:26 [1,497ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-08-23 19:42:26 [1,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.ext.plugin.so!carbOnPluginPostShutdown
2021-08-23 19:42:26 [1,500ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,502ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!carbOnPluginShutdown
2021-08-23 19:42:26 [1,503ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)
2021-08-23 19:42:26 [1,505ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,506ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,507ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::operator=(std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> > const&)
2021-08-23 19:42:26 [1,509ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2021-08-23 19:42:26 [1,511ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!PyInit__app
2021-08-23 19:42:26 [1,513ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-08-23 19:42:26 [1,516ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMethodDef_RawFastCallKeywords
2021-08-23 19:42:26 [1,520ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyCFunction_FastCallKeywords
2021-08-23 19:42:26 [1,523ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,526ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-08-23 19:42:26 [1,529ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,532ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-08-23 19:42:26 [1,535ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallDict
2021-08-23 19:42:26 [1,538ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyObject_Call_Prepend
2021-08-23 19:42:26 [1,541ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyType_LookupId
2021-08-23 19:42:26 [1,544ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyType_IsSubtype
2021-08-23 19:42:26 [1,547ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyObject_FastCallKeywords
2021-08-23 19:42:26 [1,550ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,553ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-08-23 19:42:26 [1,556ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,559ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-08-23 19:42:26 [1,562ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCodeEx
2021-08-23 19:42:26 [1,565ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCode
2021-08-23 19:42:26 [1,568ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_FileExFlags
2021-08-23 19:42:26 [1,571ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_SimpleFileExFlags
2021-08-23 19:42:26 [1,574ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMainInterpreterConfig_Read
2021-08-23 19:42:26 [1,577ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_Py_UnixMain
2021-08-23 19:42:26 [1,580ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2021-08-23 19:42:26 [1,580ms] [Fatal] [carb.crashreporter-breakpad.plugin] python3!_start
./python.sh: line 27:  4758 Segmentation fault      (core dumped) $python_exe $@
There was an error running python

I think the fatal message comes from:

[omni.appwindow.plugin] Failed to acquire IWindowing interface

Seems like it is able to connect to my remote client server and launch the window there?

Sorry, i mixed this up. Remote client still works with headless on.

Could you add these extra lines to your jetbot_train.py after creating the OmniKitHelper, with headless on?

Start the omniverse application

kit = OmniKitHelper(config=CONFIG)

Enable Livestream extension

ext_manager = omni.kit.app.get_app().get_extension_manager()
kit.set_setting("/app/window/drawMouse", True)
kit.set_setting("/app/livestream/proto", “ws”)
ext_manager.set_extension_enabled_immediate(“omni.kit.livestream.core”, True)
ext_manager.set_extension_enabled_immediate(“omni.kit.livestream.native”, True)

https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/sample_python_basic.html#livestream-py

We have an example livestream.py there as well.

I am getting following error when I run the livestream example:

[7.160s] [ext: omni.kit.window.console-0.0.0] startup
[7.163s] [ext: omni.isaac.motion_planning-0.1.4] startup
[7.172s] [ext: omni.isaac.occupancy_map-0.1.0] startup
[7.179s] [ext: omni.kit.widget.prompt-1.0.0] startup
[7.183s] [ext: omni.kit.asset_converter-1.1.22] startup
[7.220s] [ext: omni.kit.window.script_editor-1.5.0] startup
[7.221s] [ext: omni.kit.window.status_bar-0.0.0] startup
[7.223s] [ext: omni.rtx.window.settings-0.6.0] startup
[7.230s] [ext: omni.isaac.synthetic_utils-0.1.0] startup
[7.244s] [ext: omni.kit.widget.graph-1.1.2] startup
[7.252s] [ext: omni.kit.window.profiler-1.0.0] startup
[7.258s] [ext: omni.rtx.settings.core-0.5.4] startup
[7.264s] [ext: omni.isaac.dr-0.1.6] startup
[7.271s] [ext: omni.kit.window.extensions-1.0.2] startup
[7.277s] [ext: omni.isaac.urdf-0.1.4] startup
[7.289s] [ext: omni.isaac.contact_sensor-0.1.0] startup
[7.295s] [ext: omni.isaac.python_app-0.0.0] startup
[7.297s] [ext: omni.isaac.sim.python-2021.1.1] startup
[7.297s] app started
OmniKitHelper Starting up …
OmniKitHelper Startup Complete
[96.676s] [ext: omni.kit.livestream.core-0.0.0] startup
[96.685s] [ext: omni.kit.livestream.native-0.0.0] startup

Active user not found. Using default user [kiosk]2021-08-23 21:40:31 [96,894ms] [Error] [carb.livestream.plugin] Stream Server: starting the server failed, 0x800B1002
2021-08-23 21:40:31 [96,894ms] [Error] [carb.livestream.plugin] Could not initialize streaming components

And it is stuck here.

Have you seen this error before?

Any updates?

Hi. The livestream server error could be caused by a running process of already opened ports.
Can you try kill the docker container and try again or reboot your EC2 instance?

1 Like

Yes! I was able to run this after I stopped the ./runheadless.kitremote.sh remote client script. It is running smoothly now.

Thanks a lot for your help!

1 Like