If I run without --headless argument, the training script fails with the following script. Here is the step I followed:
- SSH into my instance which is running my container with remote client entry point.
- Bash into this container:
docker exec -it 7cd7a685240c bash
- Run my python training scrip by running:
./python.sh python_samples/jetbot/jetbot_train.py
- The script exits with the following error message:
2021-08-23 19:42:25 [402ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [402ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:25 [402ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [402ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:25 [403ms] [Error] [omni.appwindow.plugin] Failed to acquire IWindowing interface
[0.417s] [ext: omni.kit.renderer.core-0.0.0] startup
|---------------------------------------------------------------------------------------------|
| Driver Version: 460.73 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | Tesla T4 | Yes | | 15606 MB | 10de | 0 |
| | | | | | 1eb8 | b58943f9.. |
|=============================================================================================|
| OS: Linux ip-172-31-21-181, Version: 5.4.0-1048-aws
| Processor: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 255000 | Free Memory: 194038
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2021-08-23 19:42:25 [714ms] [Warning] [carb.cudainterop.plugin] NVML could not be loaded. GPU Utilization stats will be unavailable. Error: libnvidia-ml.so: cannot open shared object file: No such file or directory
2021-08-23 19:42:25 [724ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:25 [724ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,066ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:26 [1,066ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,067ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2021-08-23 19:42:26 [1,067ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v0.1],[carb::windowing::IWindowing v1.1]) (impl: carb.windowing-glfw.plugin)
2021-08-23 19:42:26 [1,492ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2021-08-23 19:42:26 [1,493ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::*)(unsigned int, int, carb::cpp20::latch*), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2021-08-23 19:42:26 [1,496ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-08-23 19:42:26 [1,497ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.appwindow.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)
2021-08-23 19:42:26 [1,497ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-08-23 19:42:26 [1,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-08-23 19:42:26 [1,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.ext.plugin.so!carbOnPluginPostShutdown
2021-08-23 19:42:26 [1,500ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,502ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!carbOnPluginShutdown
2021-08-23 19:42:26 [1,503ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)
2021-08-23 19:42:26 [1,505ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,506ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string>>::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2021-08-23 19:42:26 [1,507ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::operator=(std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> > const&)
2021-08-23 19:42:26 [1,509ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2021-08-23 19:42:26 [1,511ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!PyInit__app
2021-08-23 19:42:26 [1,513ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-08-23 19:42:26 [1,516ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMethodDef_RawFastCallKeywords
2021-08-23 19:42:26 [1,520ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyCFunction_FastCallKeywords
2021-08-23 19:42:26 [1,523ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,526ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-08-23 19:42:26 [1,529ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,532ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-08-23 19:42:26 [1,535ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallDict
2021-08-23 19:42:26 [1,538ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyObject_Call_Prepend
2021-08-23 19:42:26 [1,541ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyType_LookupId
2021-08-23 19:42:26 [1,544ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyType_IsSubtype
2021-08-23 19:42:26 [1,547ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyObject_FastCallKeywords
2021-08-23 19:42:26 [1,550ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,553ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-08-23 19:42:26 [1,556ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-08-23 19:42:26 [1,559ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-08-23 19:42:26 [1,562ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCodeEx
2021-08-23 19:42:26 [1,565ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCode
2021-08-23 19:42:26 [1,568ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_FileExFlags
2021-08-23 19:42:26 [1,571ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_SimpleFileExFlags
2021-08-23 19:42:26 [1,574ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMainInterpreterConfig_Read
2021-08-23 19:42:26 [1,577ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_Py_UnixMain
2021-08-23 19:42:26 [1,580ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2021-08-23 19:42:26 [1,580ms] [Fatal] [carb.crashreporter-breakpad.plugin] python3!_start
./python.sh: line 27: 4758 Segmentation fault (core dumped) $python_exe $@
There was an error running python