Isaac Sim Headless native fails to initialize streaming

I’m running the Isaac Sim container version 4.0.0 in a podman container on a SLURM compute cluster node with L4 GPUs. I can run nvidia-smi with no problem inside the container, but when I run ./runheadless.native.sh --/app/livestream/logLevel=verbose Kit fails to setup the streaming components and it spits out the following error:

2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #8(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&1768021A&<VersionDump>      Name: unnamed
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #9(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&CF6CAAC5&<VersionDump>      DVS Branch: gcomp/rel/gs_04_59
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #0(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&EA649458&<VersionDump>      DVS P4VirtualChangelist: 33332890
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #3(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&7DF5E4E1&<Server>           Using: OpenSSL 1.1.1q  5 Jul 2022
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&55B93E2E&<Server>           RND is initialized
2024-07-25 15:56:38 [11,267ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #1(E)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&168334B0&<NvNetworkCommon>  Network Address To String: invalid parameters 0x7ff7835fd9a0, 0x7ff7835fd9c0, 48
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #5(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&AD28CD28&<Server>           --- 42:01:0A:A4:0F:DD: 10.164.15.221 / invalid
2024-07-25 15:56:38 [11,267ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #0(E)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&168334B0&<NvNetworkCommon>  Network Address To String: invalid parameters 0x7ff7835fd9a0, 0x7ff7835fd9c0, 48
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&AD28CD28&<Server>           --- 42:01:0A:A4:0F:DD: 10.164.15.221 / invalid
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: NvEncUtil: #0(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&CC7C5461&<NvEncUtil>        Computing SPS/PPS headers.
2024-07-25 15:56:38 [11,268ms] [Warning] [carb.livestream.plugin] StreamSDK: VideoFrameCtxt: #1(W)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&0564BB28&<VideoFrameCtxt>   Couldn't find VideoFrameContext for this thread.
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #3(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&1E959A86&<AdapterContext>   Initializing Adapter context
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #6(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&F0E847F4&<AdapterContext>   Initialized context for adapter 0: 1280 x 720 @ 60.0 Hz
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #8(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&9A64D99C&<AdapterContext>   System is NOT co-proc.
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #9(M)[2024-07-25 15:56:38,204]=15:56:38=*00124*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #0(M)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&DD40CDEB&<AdapterContext>   Initialized CUDA for device 'NVIDIA L4' (SM 8.9) in compute mode 'CU_COMPUTEMODE_DEFAULT'.
2024-07-25 15:56:38 [11,392ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #1(E)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&5FA2F6E4&<NvEncUtil>        Failed to allocate encoder for RTSP SPS/PPS header generation
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #2(M)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,459ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #5(E)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&099DD006&<NvEncUtil>        Error computing SPS/PPS headers.
2024-07-25 15:56:38 [11,459ms] [Warning] [carb.livestream.plugin] StreamSDK: Server: #6(W)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&F0CB1C29&<Server>           NvEncodeAPI Not Supported, falling back to software encode
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: NvEncUtil: #7(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&CC7C5461&<NvEncUtil>        Computing SPS/PPS headers.
2024-07-25 15:56:38 [11,459ms] [Warning] [carb.livestream.plugin] StreamSDK: VideoFrameCtxt: #8(W)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&0564BB28&<VideoFrameCtxt>   Couldn't find VideoFrameContext for this thread.
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #0(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&1E959A86&<AdapterContext>   Initializing Adapter context
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #3(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&F0E847F4&<AdapterContext>   Initialized context for adapter 0: 1280 x 720 @ 60.0 Hz
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #5(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&9A64D99C&<AdapterContext>   System is NOT co-proc.
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #6(M)[2024-07-25 15:56:38,392]=15:56:38=*00120*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #7(M)[2024-07-25 15:56:38,392]=15:56:38=*00000*{16272343718602006736}&DD40CDEB&<AdapterContext>   Initialized CUDA for device 'NVIDIA L4' (SM 8.9) in compute mode 'CU_COMPUTEMODE_DEFAULT'.
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: VideoEncoder: #0(M)[2024-07-25 15:56:38,392]=15:56:38=*00000*{16272343718602006736}&8BAD092A&<VideoEncoder>     Initializing OpenH264VideoEncoder.
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: GsLoadLibrary: #2(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&649F1C44&<GsLoadLibrary>    Failed to load library libopenh264.so due to error "libopenh264.so: cannot open shared object file: No such file or directory"
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: OpenH264VideoEncoder: #3(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&E18798CF&<OpenH264VideoEnc> Failed to load OpenH264 DLL from 'libopenh264.so'.
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #4(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&A32B5425&<NvEncUtil>        Failed to initialize encoder for RTSP SPS/PPS header generation.
2024-07-25 15:56:38 [11,581ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #5(M)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,651ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #9(E)[2024-07-25 15:56:38,463]=15:56:38=*00000*{16272343718602006736}&099DD006&<NvEncUtil>        Error computing SPS/PPS headers.
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] StreamSDK: RunSdkCommand: #0(E)[2024-07-25 15:56:38,464]=15:56:38=*00000*{16272343718602006736}&D291FF46&<RunSdkCommand>    nvstCreateVersionedServer failed: StreamSdkException 800e8401 [NVST_DISCONN_SERVER_VIDEO_ENCODER_INIT_DLL_LOAD_FAILED] Software video encoder not supported
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] Stream Server: Net Stream Creation failed, 0x800E8401
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] Could not initialize streaming components
2024-07-25 15:56:38 [11,653ms] [Error] [carb.livestream.plugin] Couldn't initialize the capture device.

could anyway provide any pointers to what is wrong?
here is the full output for completeness:


The NVIDIA Omniverse License Agreement (EULA) must be accepted before
Omniverse Kit can start. The license terms for this product can be viewed at
https://docs.omniverse.nvidia.com/app_isaacsim/common/NVIDIA_Omniverse_License_Agreement.html

Omniverse Software collects installation and configuration details about your software, hardware, and network
configuration (e.g., version of operating system, applications installed, type of hardware, network speed, IP
address) based on our legitimate interest in improving your experience. To improve performance, troubleshooting
and diagnostic purposes of our software, we also collect session behavior, error and crash logs.

Data Collection in container mode is completely anonymous unless specified. You may opt-out of this collection
anytime by not setting the OMNI_ENV_PRIVACY_CONSENT environment variable.

To opt-in set the OMNI_ENV_PRIVACY_CONSENT environment variable when running the container. Set the 
OMNI_ENV_PRIVACY_USERID environment variable tag the telemetry data with a user ID.
Loading user config located at: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/user.config.json'
[Info] [carb] Logging to file: /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/4.0/kit_20240725_155626.log
[0.281s] [ext: omni.kit.async_engine-0.0.0] startup
[0.835s] [ext: omni.stats-0.0.0] startup
[0.837s] [ext: omni.client-1.1.0] startup
[0.881s] [ext: omni.datastore-0.0.0] startup
[0.882s] [ext: omni.blobkey-1.1.0] startup
[0.882s] [ext: omni.ujitso.default-1.0.0] startup
[0.883s] [ext: omni.hsscclient-0.0.0] startup
[0.884s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.885s] [ext: omni.assets.plugins-0.0.0] startup
[0.887s] [ext: omni.gpu_foundation-0.0.0] startup
[0.904s] [ext: carb.windowing.plugins-1.0.0] startup
2024-07-25 15:56:27 [863ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:27 [863ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[0.905s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-07-25 15:56:27 [889ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.
2024-07-25 15:56:27 [889ms] [Warning] [gpu.foundation.plugin] nvidia-smi not found in '/usr/bin/'
2024-07-25 15:56:27 [1,069ms] [Warning] [carb.cudainterop.plugin] CUDA_VISIBLE_DEVICES environment variable is set.
2024-07-25 15:56:27 [1,069ms] [Warning] [carb.cudainterop.plugin] Note CUDA device enumeration and Omniverse device enumeration are different.
2024-07-25 15:56:27 [1,069ms] [Warning] [carb.cudainterop.plugin] Setting CUDA_VISIBLE_DEVICES can lead to undesired behavior or crashes.

|---------------------------------------------------------------------------------------------|
| Driver Version: 550.90.07     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA L4                        | Yes: 0 |     | 23034   MB | 10de      | 0          |
|     |                                  |        |     |            | 27b8      | 8b83db1e.. |
|     |                                  |        |     |            | 80        |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.9.8-1-insait
| Processor: Intel(R) Xeon(R) CPU @ 2.20GHz | Cores: 48 | Logical: 96
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 386948 | Free Memory: 355304
| Total Page/Swap (MB): 32767 | Free Page/Swap: 32744
|---------------------------------------------------------------------------------------------|
2024-07-25 15:56:28 [1,760ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
2024-07-25 15:56:28 [1,760ms] [Warning] [gpu.foundation.plugin] validateIommu. A failure occurred when cheking IOMMU status.
[2.033s] [ext: omni.kit.pipapi-0.0.0] startup
[2.035s] [ext: omni.kit.pip_archive-0.0.0] startup
[2.035s] [ext: omni.pip.compute-1.3.1] startup
[2.036s] [ext: omni.mtlx-0.1.0] startup
[2.036s] [ext: omni.usd.config-1.0.3] startup
[2.041s] [ext: omni.gpucompute.plugins-0.0.0] startup
[2.042s] [ext: omni.usd.libs-1.0.0] startup
[2.167s] [ext: omni.pip.cloud-1.1.3] startup
[2.172s] [ext: omni.isaac.core_archive-2.3.0] startup
[2.172s] [ext: omni.kit.telemetry-0.5.0] startup
[2.211s] [ext: omni.kit.loop-isaac-1.2.0] startup
[2.214s] [ext: omni.kit.test-0.0.0] startup
[2.293s] [ext: omni.isaac.ml_archive-2.0.1] startup
[2.293s] [ext: omni.appwindow-1.1.8] startup
2024-07-25 15:56:29 [2,253ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,253ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[2.297s] [ext: omni.kit.renderer.core-1.0.1] startup
2024-07-25 15:56:29 [2,259ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,259ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-07-25 15:56:29 [2,270ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,270ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[2.314s] [ext: omni.kit.renderer.capture-0.0.0] startup
[2.318s] [ext: omni.kit.renderer.imgui-1.0.1] startup
2024-07-25 15:56:29 [2,281ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,281ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-07-25 15:56:29 [2,283ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,283ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-07-25 15:56:29 [2,284ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:29 [2,284ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[2.470s] [ext: omni.ui-2.23.11] startup
[2.487s] [ext: omni.kit.mainwindow-1.0.3] startup
[2.489s] [ext: carb.audio-0.1.0] startup
[2.493s] [ext: omni.uiaudio-1.0.0] startup
[2.495s] [ext: omni.kit.uiapp-0.0.0] startup
[2.495s] [ext: omni.usd.schema.metrics.assembler-106.0.1] startup
[2.653s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.664s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[2.668s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.695s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.719s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.724s] [ext: omni.anim.navigation.schema-106.0.2] startup
[2.729s] [ext: omni.usd.schema.isaac-2.0.2] startup
[2.739s] [ext: omni.usd.schema.physx-106.0.16] startup
[2.806s] [ext: omni.anim.graph.schema-106.0.2] startup
[2.817s] [ext: omni.usd.schema.scene.visualization-2.0.2] startup
[2.819s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[2.825s] [ext: omni.graph.exec-0.9.3] startup
[2.826s] [ext: omni.kit.actions.core-1.0.0] startup
[2.829s] [ext: omni.kit.usd_undo-0.1.8] startup
[2.830s] [ext: omni.kit.exec.core-0.13.0] startup
[2.833s] [ext: omni.kit.commands-1.4.9] startup
[2.838s] [ext: omni.usd_resolver-1.0.0] startup
[2.845s] [ext: omni.resourcemonitor-105.0.1] startup
[2.848s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
[2.854s] [ext: omni.activity.core-1.0.1] startup
[2.857s] [ext: omni.usd.core-1.2.11] startup
[2.862s] [ext: omni.timeline-1.0.10] startup
[2.865s] [ext: omni.kit.widget.nucleus_connector-1.1.5] startup
[2.868s] [ext: usdrt.scenegraph-7.4.8] startup
[2.942s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[2.945s] [ext: omni.hydra.usdrt_delegate-7.4.7] startup
[2.975s] [ext: omni.hydra.scene_delegate-0.3.3] startup
[2.986s] [ext: omni.iray.libs-0.0.0] startup
[2.997s] [ext: omni.usd-1.10.39] startup
[3.070s] [ext: omni.mdl.neuraylib-0.2.4] startup
[3.072s] [ext: omni.kit.asset_converter-2.1.10] startup
2024-07-25 15:56:30 [3,221ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 0 (NVIDIA L4): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
[3.268s] [ext: omni.usd.schema.forcefield-106.0.16] startup
[3.278s] [ext: omni.kit.collaboration.telemetry-1.0.0] startup
[3.279s] [ext: omni.kit.collaboration.channel_manager-1.0.11] startup
[3.281s] [ext: omni.kit.widget.path_field-2.0.9] startup
[3.282s] [ext: omni.kit.usd.layers-2.1.30] startup
[3.295s] [ext: omni.kit.clipboard-1.0.3] startup
[3.297s] [ext: omni.kit.widget.browser_bar-2.0.9] startup
[3.298s] [ext: omni.kit.notification_manager-1.0.8] startup
[3.301s] [ext: omni.kit.collaboration.presence_layer-1.0.8] startup
[3.303s] [ext: omni.kit.helper.file_utils-0.1.7] startup
[3.306s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[3.307s] [ext: omni.kit.widget.options_menu-1.1.4] startup
[3.311s] [ext: omni.kit.widget.filebrowser-2.3.47] startup
[3.317s] [ext: omni.kit.search_core-1.0.5] startup
[3.319s] [ext: omni.kit.widget.options_button-1.0.2] startup
[3.320s] [ext: omni.kit.widget.versioning-1.4.6] startup
[3.322s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[3.324s] [ext: omni.kit.widget.context_menu-1.2.1] startup
[3.325s] [ext: omni.convexdecomposition-106.0.16] startup
[3.333s] [ext: omni.ui.scene-1.9.3] startup
[3.342s] [ext: omni.kit.menu.utils-1.5.23] startup
[3.360s] [ext: omni.kit.window.filepicker-2.10.32] startup
[3.374s] [ext: omni.kvdb-106.0.16] startup
[3.377s] [ext: omni.physx.foundation-106.0.16] startup
[3.378s] [ext: omni.kit.widget.searchable_combobox-1.0.5] startup
[3.379s] [ext: omni.usdphysics-106.0.16] startup
[3.384s] [ext: omni.localcache-106.0.16] startup
[3.387s] [ext: omni.kit.widget.prompt-1.0.7] startup
[3.388s] [ext: omni.kit.widget.settings-1.1.0] startup
[3.390s] [ext: omni.physx.cooking-106.0.16] startup
[3.408s] [ext: omni.kit.context_menu-1.8.0] startup
[3.410s] [ext: omni.kit.window.preferences-1.5.1] startup
[3.416s] [ext: omni.kit.widget.live_session_management.ui-1.0.1] startup
[3.419s] [ext: omni.kit.widget.filter-1.1.3] startup
[3.420s] [ext: omni.inspect-1.0.1] startup
[3.423s] [ext: omni.kit.window.file_exporter-1.0.29] startup
[3.426s] [ext: omni.kit.hotkeys.core-1.3.3] startup
[3.428s] [ext: omni.kit.widget.graph-1.12.6] startup
[3.436s] [ext: omni.graph.core-2.170.0] startup
[3.441s] [ext: omni.kit.widget.stage-2.10.26] startup
[3.455s] [ext: omni.kit.widget.live_session_management-1.2.18] startup
[3.459s] [ext: omni.graph.tools-1.77.0] startup
[3.490s] [ext: omni.kit.window.extensions-1.4.7] startup
[3.499s] [ext: omni.ujitso.processor.texture-1.0.0] startup
[3.501s] [ext: omni.kit.window.file_importer-1.1.11] startup
[3.503s] [ext: omni.kit.primitive.mesh-1.0.16] startup
[3.508s] [ext: omni.volume-0.5.0] startup
[3.513s] [ext: omni.ujitso.client-0.0.0] startup
[3.514s] [ext: omni.mdl-52.0.1] startup
[3.542s] [ext: omni.kit.stage_template.core-1.1.20] startup
[3.543s] [ext: omni.kit.widget.highlight_label-1.0.1] startup
[3.545s] [ext: omni.kit.stage_templates-1.2.2] startup
[3.549s] [ext: omni.hydra.rtx-0.2.0] startup
[3.582s] [ext: omni.kit.material.library-1.4.0] startup
[3.594s] [ext: omni.hydra.engine.stats-1.0.1] startup
[3.600s] [ext: omni.kit.widget.searchfield-1.1.5] startup
[3.602s] [ext: omni.kit.window.file-1.3.52] startup
[3.606s] [ext: omni.kit.hydra_texture-1.2.6] startup
[3.617s] [ext: omni.kit.window.drop_support-1.0.1] startup
[3.618s] [ext: omni.kit.widget.viewport-106.0.2] startup
[3.623s] [ext: omni.kit.viewport.registry-104.0.6] startup
[3.624s] [ext: omni.kit.raycast.query-1.0.5] startup
[3.636s] [ext: omni.kit.window.content_browser_registry-0.0.4] startup
[3.637s] [ext: omni.kit.viewport.window-106.0.8] startup
[3.677s] [ext: omni.kit.window.property-1.11.1] startup
[3.681s] [ext: omni.kit.window.content_browser-2.9.13] startup
[3.708s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[3.712s] [ext: omni.kit.viewport.utility-1.0.17] startup
[3.714s] [ext: omni.kit.property.usd-3.21.26] startup
[3.727s] [ext: omni.graph-1.135.0] startup
[3.947s] [ext: omni.physx-106.0.16] startup
[3.977s] [ext: omni.graph.ui-1.67.1] startup
[4.011s] [ext: omni.graph.action_core-1.1.4] startup
[4.023s] [ext: omni.graph.image.core-0.3.2] startup
[4.029s] [ext: omni.graph.action_nodes-1.21.3] startup
[4.043s] [ext: omni.graph.image.nodes-1.0.2] startup
[4.047s] [ext: omni.kit.numpy.common-0.1.2] startup
[4.051s] [ext: omni.graph.nodes-1.141.2] startup
[4.083s] [ext: omni.physics.tensors-106.0.16] startup
[4.115s] [ext: omni.physx.stageupdate-106.0.16] startup
[4.165s] [ext: omni.graph.action-1.101.1] startup
[4.167s] [ext: omni.warp.core-1.1.0] startup
Warp 1.1.0 initialized:
   CUDA Toolkit 11.5, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA L4" (22 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.1.0
[4.405s] [ext: omni.isaac.dynamic_control-1.3.7] startup
[4.430s] [ext: omni.warp-1.1.0] startup
[4.451s] [ext: omni.kit.viewport.actions-105.2.8] startup
[4.513s] [ext: omni.kit.widget.imageview-1.0.3] startup
[4.521s] [ext: omni.debugdraw-0.1.3] startup
[4.553s] [ext: omni.physx.tensors-106.0.16] startup
[4.574s] [ext: omni.kit.window.cursor-1.1.2] startup
[4.600s] [ext: omni.isaac.version-1.1.0] startup
[4.606s] [ext: omni.kit.widget.toolbar-1.6.2] startup
[4.852s] [ext: omni.kit.manipulator.transform-104.7.5] startup
[4.881s] [ext: omni.kit.viewport.menubar.core-107.0.0] startup
[4.931s] [ext: omni.isaac.nucleus-0.1.2] startup
[4.933s] [ext: omni.kit.manipulator.tool.snap-1.4.5] startup
[4.942s] [ext: omni.kit.viewport.menubar.display-106.0.0] startup
[4.946s] [ext: omni.kit.window.quicksearch-2.4.4] startup
[4.953s] [ext: omni.physx.commands-106.0.16] startup
[4.961s] [ext: omni.usdphysics.ui-106.0.16] startup
[5.004s] [ext: omni.isaac.core-3.12.0] startup
[6.710s] [ext: omni.syntheticdata-0.6.7] startup
[6.755s] [ext: omni.kit.property.material-1.9.3] startup
[6.767s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[6.774s] [ext: omni.physx.ui-106.0.16] startup
[6.830s] [ext: omni.fabric.commands-1.1.4] startup
[6.839s] [ext: omni.kit.manipulator.viewport-107.0.0] startup
[6.842s] [ext: omni.kit.viewport.manipulator.transform-106.0.0] startup
ViewportTransformManipulatorExt on_startup
[6.845s] [ext: omni.kit.property.physx-106.0.16] startup
[6.957s] [ext: omni.physx.demos-106.0.16] startup
[7.006s] [ext: omni.ui_query-1.1.2] startup
[7.011s] [ext: omni.kit.manipulator.prim.core-106.0.2] startup
[7.021s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[7.037s] [ext: omni.physx.vehicle-106.0.16] startup
[7.077s] [ext: omni.kit.ui_test-1.2.18] startup
[7.084s] [ext: omni.kit.manipulator.prim.fabric-106.0.1] startup
[7.085s] [ext: omni.kit.manipulator.prim.usd-106.0.1] startup
[7.089s] [ext: omni.physx.camera-106.0.16] startup
[7.105s] [ext: omni.physx.cct-106.0.16] startup
[7.121s] [ext: omni.kit.manipulator.selection-104.0.9] startup
[7.131s] [ext: omni.kit.manipulator.prim-106.0.0] startup
[7.131s] [ext: omni.kit.window.toolbar-1.6.1] startup
[7.142s] [ext: omni.kit.widget.layers-1.7.9] startup
[7.169s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[7.172s] [ext: omni.physx.graph-106.0.16] startup
[7.195s] [ext: omni.graph.ui_nodes-1.24.1] startup
[7.205s] [ext: omni.physx.supportui-106.0.16] startup
[7.265s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[7.273s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[7.295s] [ext: omni.physx.telemetry-106.0.16] startup
[7.307s] [ext: omni.graph.bundle.action-2.0.4] startup
[7.310s] [ext: omni.command.usd-1.0.3] startup
[7.313s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[7.321s] [ext: omni.isaac.debug_draw-0.7.1] startup
[7.332s] [ext: omni.kit.window.material_graph-1.8.15] startup
[7.369s] [ext: omni.sensors.tiled-0.0.3] startup
[7.375s] [ext: omni.physx.bundle-106.0.16] startup
[7.376s] [ext: omni.graph.scriptnode-1.18.2] startup
[7.378s] [ext: omni.isaac.ui-0.15.1] startup
[7.396s] [ext: omni.replicator.core-1.11.8] startup
2024-07-25 15:56:34 [7,393ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[7.582s] [ext: omni.sensors.nv.common-1.0.1] startup
[7.612s] [ext: omni.isaac.range_sensor-3.0.1] startup
[7.713s] [ext: omni.isaac.core_nodes-1.15.4] startup
[7.740s] [ext: omni.sensors.nv.wpm-1.0.0] startup
[7.746s] [ext: omni.sensors.nv.materials-1.0.0] startup
[7.756s] [ext: omni.isaac.lula-3.0.1] startup
[7.783s] [ext: omni.isaac.surface_gripper-1.0.1] startup
[7.803s] [ext: omni.isaac.motion_generation-7.0.1] startup
[7.892s] [ext: omni.isaac.manipulators-2.1.0] startup
[7.914s] [ext: omni.sensors.nv.radar-1.0.1] startup
[7.929s] [ext: omni.sensors.nv.lidar-1.0.1] startup
[7.944s] [ext: omni.isaac.franka-0.4.1] startup
[7.953s] [ext: omni.isaac.dofbot-0.3.2] startup
[7.965s] [ext: omni.isaac.sensor-11.3.0] startup
[8.124s] [ext: omni.isaac.cortex-0.3.8] startup
[8.131s] [ext: omni.isaac.universal_robots-0.3.5] startup
[8.139s] [ext: omni.isaac.quadruped-1.4.5] startup
[8.186s] [ext: omni.isaac.cortex.sample_behaviors-1.0.5] startup
[8.193s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[8.216s] [ext: omni.kit.graph.delegate.modern-1.10.6] startup
[8.225s] [ext: omni.isaac.wheeled_robots-2.1.3] startup
[8.231s] [ext: omni.graph.window.core-1.107.1] startup
[8.259s] [ext: omni.isaac.examples-2.0.0] startup
[8.479s] [ext: omni.importer.mjcf-1.1.0] startup
[8.537s] [ext: omni.graph.window.generic-1.23.1] startup
[8.580s] [ext: omni.importer.onshape-0.7.1] startup
[8.636s] [ext: omni.isaac.manipulators.ui-1.2.1] startup
[8.643s] [ext: omni.importer.urdf-1.14.1] startup
[8.728s] [ext: omni.kit.streamsdk.plugins-3.2.1] startup
[8.733s] [ext: omni.isaac.robot_description_editor-2.4.0] startup
[8.780s] [ext: omni.isaac.examples_nodes-0.1.3] startup
[8.785s] [ext: omni.isaac.cloner-0.8.1] startup
[8.797s] [ext: omni.isaac.extension_templates-1.7.0] startup
[8.803s] [ext: omni.isaac.gain_tuner-1.1.1] startup
[8.818s] [ext: omni.isaac.gym-0.11.3] startup
[8.828s] [ext: omni.isaac.import_wizard-0.5.2] startup
[8.844s] [ext: omni.isaac.kit-1.10.0] startup
[8.853s] [ext: omni.kit.mesh.raycast-105.4.0] startup
[8.862s] [ext: omni.anim.curve.core-1.1.13] startup
[8.892s] [ext: omni.isaac.block_world-1.0.0] startup
[8.903s] [ext: omni.anim.shared.core-106.0.0] startup
[8.917s] [ext: omni.anim.curve.ui-1.3.16] startup
[8.944s] [ext: omni.isaac.occupancy_map-1.0.1] startup
[8.980s] [ext: omni.isaac.merge_mesh-1.0.1] startup
[8.997s] [ext: omni.kit.widget.stage_icons-1.0.3] startup
[9.011s] [ext: omni.anim.curve.bundle-1.2.3] startup
[9.012s] [ext: omni.isaac.occupancy_map.ui-1.1.0] startup
[9.022s] [ext: omni.kit.widget.zoombar-1.0.5] startup
[9.033s] [ext: omni.kit.window.stage-2.5.9] startup
[9.045s] [ext: omni.kit.usd.collect-2.2.21] startup
[9.054s] [ext: omni.kit.browser.core-2.3.11] startup
[9.068s] [ext: omni.kit.menu.stage-1.2.5] startup
[9.074s] [ext: omni.kit.tool.collect-2.2.12] startup
[9.084s] [ext: omni.kit.browser.folder.core-1.9.12] startup
[9.098s] [ext: omni.kit.search.files-1.0.4] startup
[9.104s] [ext: omni.isaac.physics_inspector-0.1.4] startup
[9.111s] [ext: omni.isaac.asset_browser-0.4.6] startup
[9.127s] [ext: omni.kit.usda_edit-1.1.11] startup
[9.150s] [ext: omni.scene.optimizer.core-106.0.4] startup
[9.175s] [ext: omni.isaac.scene_blox-0.1.1] startup
[9.178s] [ext: omni.isaac.surface_gripper.ui-1.0.0] startup
[9.187s] [ext: omni.isaac.synthetic_recorder-1.7.3] startup
[9.202s] [ext: omni.kit.converter.common-201.0.2] startup
[9.214s] [ext: omni.isaac.throttling-1.1.0] startup
[9.218s] [ext: omni.isaac.utils-1.0.1] startup
[9.227s] [ext: omni.isaac.wheeled_robots.ui-1.1.1] startup
[9.235s] [ext: omni.kit.converter.cad_core-201.0.2] startup
[9.258s] [ext: omni.kit.converter.jt_core-201.0.2] startup
[9.294s] [ext: omni.kit.tool.asset_importer-2.5.5] startup
[9.311s] [ext: omni.kit.actions.window-1.1.1] startup
[9.325s] [ext: omni.isaac.window.about-1.1.0] startup
[9.333s] [ext: omni.kit.browser.asset-1.3.9] startup
[9.339s] [ext: omni.kit.selection-0.1.4] startup
[9.344s] [ext: omni.kit.converter.cad-201.0.2] startup
[9.356s] [ext: omni.kit.hotkeys.window-1.4.5] startup
[9.376s] [ext: omni.kit.manipulator.camera-105.0.5] startup
[9.387s] [ext: omni.kit.menu.aov-1.1.3] startup
[9.392s] [ext: omni.kit.menu.edit-1.1.19] startup
[9.402s] [ext: omni.kit.property.audio-1.0.11] startup
[9.409s] [ext: omni.hydra.scene_api-0.1.2] startup
[9.422s] [ext: omni.kit.property.camera-1.0.6] startup
[9.428s] [ext: omni.kit.property.geometry-1.3.0] startup
[9.443s] [ext: omni.kit.property.light-1.0.8] startup
[9.448s] [ext: omni.kit.property.render-1.1.1] startup
[9.455s] [ext: omni.kit.property.transform-1.5.1] startup
[9.474s] [ext: omni.kit.menu.file-1.1.9] startup
[9.492s] [ext: omni.kit.core.collection-0.1.7] startup
[9.503s] [ext: omni.kit.preferences.animation-1.1.7] startup
[9.514s] [ext: omni.isaac.menu-0.4.1] startup
[9.527s] [ext: omni.kit.property.bundle-1.2.11] startup
[9.540s] [ext: omni.kit.menu.create-1.0.12] startup
[9.548s] [ext: omni.isaac.physics_utilities-0.1.4] startup
[9.560s] [ext: omni.kit.property.collection-0.1.17] startup
[9.573s] [ext: omni.kit.property.isaac-0.2.3] startup
[9.582s] [ext: omni.mdl.usd_converter-1.0.18] startup
[9.595s] [ext: omni.kit.property.layer-1.1.5] startup
[9.612s] [ext: omni.kit.stagerecorder.core-105.0.5] startup
[9.626s] [ext: omni.kit.scripting-106.0.1] startup
[9.649s] [ext: semantics.schema.property-1.0.2] startup
[9.655s] [ext: omni.kit.stage.copypaste-1.2.2] startup
[9.661s] [ext: omni.kit.stage.mdl_converter-1.0.5] startup
[9.665s] [ext: omni.kit.widgets.custom-1.0.8] startup
[9.987s] [ext: omni.kit.stagerecorder.ui-105.0.6] startup
[10.121s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[10.214s] [ext: omni.kit.stage_column.payload-2.0.0] startup
[10.323s] [ext: omni.isaac.range_sensor.examples-1.0.0] startup
[10.426s] [ext: omni.graph.window.action-1.25.2] startup
[10.535s] [ext: omni.kit.variant.editor-106.0.0] startup
[10.722s] [ext: omni.kit.stagerecorder.bundle-105.0.2] startup
[10.724s] [ext: omni.kit.thumbnails.usd-1.0.9] startup
[10.838s] [ext: omni.isaac.range_sensor.ui-1.0.0] startup
[10.843s] [ext: omni.kit.variant.presenter-105.1.2] startup
[10.851s] [ext: omni.kit.viewport.menubar.camera-105.1.8] startup
[10.861s] [ext: omni.kit.viewport.menubar.lighting-106.0.2] startup
[10.866s] [ext: omni.rtx.window.settings-0.6.15] startup
[10.877s] [ext: omni.kit.viewport.menubar.render-106.1.2] startup
[10.883s] [ext: omni.kit.widget.live-2.1.6] startup
[10.887s] [ext: omni.kit.widget.calendar-1.0.8] startup
[10.890s] [ext: omni.kit.viewport.menubar.settings-106.0.0] startup
[10.897s] [ext: omni.kit.widget.collection-0.1.18] startup
[10.902s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[10.904s] [ext: omni.kit.viewport_widgets_manager-1.0.8] startup
[10.906s] [ext: omni.kit.widget.cache_indicator-2.0.8] startup
2024-07-25 15:56:37 [10,938ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.
[10.982s] [ext: omni.kit.widget.extended_searchfield-1.0.27] startup
[10.994s] [ext: omni.kit.widget.timeline-105.0.1] startup
[11.002s] [ext: omni.kit.window.collection-0.1.22] startup
[11.006s] [ext: omni.kit.window.commands-0.2.5] startup
[11.010s] [ext: omni.kit.window.console-0.2.12] startup
[11.027s] [ext: omni.kit.window.imageviewer-1.0.7] startup
[11.034s] [ext: omni.kit.window.privacy-0.1.0] startup
[11.038s] [ext: omni.kit.tool.asset_exporter-1.3.3] startup
[11.042s] [ext: omni.kit.window.script_editor-1.7.6] startup
[11.046s] [ext: omni.kit.window.status_bar-0.1.6] startup
[11.054s] [ext: omni.kit.window.usd_paths-1.0.6] startup
[11.058s] [ext: omni.usd.metrics.assembler-106.0.1] startup
[11.070s] [ext: omni.replicator.isaac-1.15.0] startup
[11.094s] [ext: omni.kit.browser.material-1.5.2] startup
[11.105s] [ext: omni.replicator.replicator_yaml-2.0.5] startup
[11.118s] [ext: omni.kit.ui.actions-1.0.1] startup
[11.130s] [ext: omni.rtx.settings.core-0.6.0] startup
[11.148s] [ext: omni.simready.explorer-1.0.26] startup
[11.180s] [ext: omni.usd.metrics.assembler.physics-106.0.2] startup
[11.189s] [ext: omni.kit.tool.remove_unused.core-0.1.2] startup
[11.192s] [ext: omni.kit.window.material-1.5.7] startup
[11.202s] [ext: omni.usd.metrics.assembler.ui-106.0.2] startup
[11.211s] [ext: omni.kit.menu.common-1.1.5] startup
[11.215s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[11.218s] [ext: omni.kit.quicklayout-1.0.6] startup
[11.221s] [ext: omni.kit.window.title-1.1.3] startup
2024-07-25 15:56:37 [11,182ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:37 [11,182ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-07-25 15:56:37 [11,184ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-07-25 15:56:37 [11,184ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[11.227s] [ext: semantics.schema.editor-0.3.4] startup
[11.232s] [ext: omni.kit.tool.remove_unused.controller-0.1.3] startup
[11.235s] [ext: omni.isaac.robot_assembler-1.5.0] startup
[11.240s] [ext: omni.isaac.camera_inspector-0.2.1] startup
[11.245s] [ext: omni.kit.livestream.core-3.2.0] startup
[11.252s] [ext: omni.kit.window.stats-0.1.6] startup
[11.255s] [ext: omni.isaac.app.setup-0.13.2] startup
[11.259s] Isaac Sim Headless Native Version: 4.0.0-rc.21
[11.272s] [ext: omni.isaac.assets_check-0.3.4] startup
[11.275s] [ext: omni.kit.profiler.window-2.2.1] startup
2024-07-25 15:56:38 [11,239ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead
[11.286s] [ext: omni.isaac.sim.base-4.0.0] startup
[11.291s] [ext: omni.kit.livestream.native-4.1.0] startup

Active user not found. Using default user [kiosk]2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #8(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&1768021A&<VersionDump>      Name: unnamed
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #9(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&CF6CAAC5&<VersionDump>      DVS Branch: gcomp/rel/gs_04_59
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #0(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&EA649458&<VersionDump>      DVS P4VirtualChangelist: 33332890
2024-07-25 15:56:38 [11,266ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #3(M)[2024-07-25 15:56:38,078]=15:56:38=*00000*{16272343718602006736}&7DF5E4E1&<Server>           Using: OpenSSL 1.1.1q  5 Jul 2022
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&55B93E2E&<Server>           RND is initialized
2024-07-25 15:56:38 [11,267ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #1(E)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&168334B0&<NvNetworkCommon>  Network Address To String: invalid parameters 0x7ff7835fd9a0, 0x7ff7835fd9c0, 48
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #5(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&AD28CD28&<Server>           --- 42:01:0A:A4:0F:DD: 10.164.15.221 / invalid
2024-07-25 15:56:38 [11,267ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #0(E)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&168334B0&<NvNetworkCommon>  Network Address To String: invalid parameters 0x7ff7835fd9a0, 0x7ff7835fd9c0, 48
2024-07-25 15:56:38 [11,267ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-25 15:56:38,079]=15:56:38=*00000*{16272343718602006736}&AD28CD28&<Server>           --- 42:01:0A:A4:0F:DD: 10.164.15.221 / invalid
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: NvEncUtil: #0(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&CC7C5461&<NvEncUtil>        Computing SPS/PPS headers.
2024-07-25 15:56:38 [11,268ms] [Warning] [carb.livestream.plugin] StreamSDK: VideoFrameCtxt: #1(W)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&0564BB28&<VideoFrameCtxt>   Couldn't find VideoFrameContext for this thread.
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #3(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&1E959A86&<AdapterContext>   Initializing Adapter context
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #6(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&F0E847F4&<AdapterContext>   Initialized context for adapter 0: 1280 x 720 @ 60.0 Hz
2024-07-25 15:56:38 [11,268ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #8(M)[2024-07-25 15:56:38,080]=15:56:38=*00000*{16272343718602006736}&9A64D99C&<AdapterContext>   System is NOT co-proc.
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #9(M)[2024-07-25 15:56:38,204]=15:56:38=*00124*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #0(M)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&DD40CDEB&<AdapterContext>   Initialized CUDA for device 'NVIDIA L4' (SM 8.9) in compute mode 'CU_COMPUTEMODE_DEFAULT'.
2024-07-25 15:56:38 [11,392ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #1(E)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&5FA2F6E4&<NvEncUtil>        Failed to allocate encoder for RTSP SPS/PPS header generation
2024-07-25 15:56:38 [11,392ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #2(M)[2024-07-25 15:56:38,204]=15:56:38=*00000*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,459ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #5(E)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&099DD006&<NvEncUtil>        Error computing SPS/PPS headers.
2024-07-25 15:56:38 [11,459ms] [Warning] [carb.livestream.plugin] StreamSDK: Server: #6(W)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&F0CB1C29&<Server>           NvEncodeAPI Not Supported, falling back to software encode
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: NvEncUtil: #7(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&CC7C5461&<NvEncUtil>        Computing SPS/PPS headers.
2024-07-25 15:56:38 [11,459ms] [Warning] [carb.livestream.plugin] StreamSDK: VideoFrameCtxt: #8(W)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&0564BB28&<VideoFrameCtxt>   Couldn't find VideoFrameContext for this thread.
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #0(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&1E959A86&<AdapterContext>   Initializing Adapter context
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #3(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&F0E847F4&<AdapterContext>   Initialized context for adapter 0: 1280 x 720 @ 60.0 Hz
2024-07-25 15:56:38 [11,459ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #5(M)[2024-07-25 15:56:38,271]=15:56:38=*00000*{16272343718602006736}&9A64D99C&<AdapterContext>   System is NOT co-proc.
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #6(M)[2024-07-25 15:56:38,392]=15:56:38=*00120*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #7(M)[2024-07-25 15:56:38,392]=15:56:38=*00000*{16272343718602006736}&DD40CDEB&<AdapterContext>   Initialized CUDA for device 'NVIDIA L4' (SM 8.9) in compute mode 'CU_COMPUTEMODE_DEFAULT'.
2024-07-25 15:56:38 [11,580ms] [Fatal] [carb.livestream.plugin] StreamSDK: VideoEncoder: #0(M)[2024-07-25 15:56:38,392]=15:56:38=*00000*{16272343718602006736}&8BAD092A&<VideoEncoder>     Initializing OpenH264VideoEncoder.
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: GsLoadLibrary: #2(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&649F1C44&<GsLoadLibrary>    Failed to load library libopenh264.so due to error "libopenh264.so: cannot open shared object file: No such file or directory"
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: OpenH264VideoEncoder: #3(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&E18798CF&<OpenH264VideoEnc> Failed to load OpenH264 DLL from 'libopenh264.so'.
2024-07-25 15:56:38 [11,581ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #4(E)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&A32B5425&<NvEncUtil>        Failed to initialize encoder for RTSP SPS/PPS header generation.
2024-07-25 15:56:38 [11,581ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #5(M)[2024-07-25 15:56:38,393]=15:56:38=*00000*{16272343718602006736}&21C9B327&<AdapterContext>   GPU memory usage: used = 527.375000, free = 21950.937500 MB, total = 22478.312500 MB
2024-07-25 15:56:38 [11,651ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #9(E)[2024-07-25 15:56:38,463]=15:56:38=*00000*{16272343718602006736}&099DD006&<NvEncUtil>        Error computing SPS/PPS headers.
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] StreamSDK: RunSdkCommand: #0(E)[2024-07-25 15:56:38,464]=15:56:38=*00000*{16272343718602006736}&D291FF46&<RunSdkCommand>    nvstCreateVersionedServer failed: StreamSdkException 800e8401 [NVST_DISCONN_SERVER_VIDEO_ENCODER_INIT_DLL_LOAD_FAILED] Software video encoder not supported
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] Stream Server: Net Stream Creation failed, 0x800E8401
2024-07-25 15:56:38 [11,652ms] [Error] [carb.livestream.plugin] Could not initialize streaming components
2024-07-25 15:56:38 [11,653ms] [Error] [carb.livestream.plugin] Couldn't initialize the capture device.
[11.695s] [ext: omni.isaac.sim.headless.native-4.0.0] startup
[11.837s] [ext: omni.isaac.ros2_bridge-2.26.4] startup
2024-07-25 15:56:38 [11,798ms] [Warning] [omni.isaac.ros2_bridge.scripts.extension] ROS_DISTRO env var not found, Please source ROS2 Foxy, or Humble, before enabling this extension
[11.840s] Using backup internal ROS2 humble distro
Checking to see if RMW can be loaded:
failed to get symbol 'rmw_init_options_init' due to Environment variable 'AMENT_PREFIX_PATH' is not set or empty, at /workspace/humble_ws/src/rmw_implementation/src/functions.cpp:171, at /workspace/humble_ws/src/rcl/rcl/src/rcl/init_options.c:75
RMW was not loaded

[11.853s] To use the internal libraries included with the extension please set the following environment variables to use with FastDDS (default) or CycloneDDS (ROS2 Humble only): 
RMW_IMPLEMENTATION=rmw_fastrtps_cpp
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/isaac-sim/exts/omni.isaac.ros2_bridge/humble/lib

OR 

RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/isaac-sim/exts/omni.isaac.ros2_bridge/humble/lib
Before starting Isaac Sim
2024-07-25 15:56:38 [11,812ms] [Error] [omni.isaac.ros2_bridge.scripts.extension] ROS2 Bridge startup failed
2024-07-25 15:56:38 [11,904ms] [Warning] [carb.audio.device] audio device is misconfigured or broken {deviceIndex = 0, name = 'default'} (The device is likely misconfigured, check your $HOME/.asoundrc)
2024-07-25 15:56:38 [11,904ms] [Warning] [carb.audio.output] failed to retrieve the capabilities for device 0 {result = eDeviceLost (2)}
2024-07-25 15:56:38 [11,904ms] [Warning] [carb.audio.context] failed to set the requested output during context creation.  Using a null streamer instead {result = eDeviceLost (2)}
[14.707s] [ext: omni.isaac.ros2_bridge-2.26.4] shutdown
2024-07-25 15:56:41 [14,734ms] [Warning] [omni.ext._impl._internal] omni.isaac.ros2_bridge-2.26.4 -> <class 'omni.isaac.ros2_bridge.scripts.og_shortcuts.og_shortcut_menu.Extension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140714443649600", "[1]:type: <class 'method'>, id: 140714444847296", "[2]:type: <class 'method'>, id: 140714441828992", "[3]:type: <class 'method'>, id: 140714455573184", "[4]:type: <class 'method'>, id: 140714454599168", '[*] ...']
[17.009s] app ready
2024-07-25 15:56:44 [17,356ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
2024-07-25 15:56:44 [17,356ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
[17.568s] Isaac Sim Headless Native App is loaded.
^C2024-07-25 15:59:48 [201,589ms] [Warning] [carb] [Plugin: omni.sensors.tiled.plugin] Module /isaac-sim/extscache/omni.sensors.tiled-0.0.3+106.0.0.lx64.r/bin/libomni.sensors.tiled.plugin.so remained loaded after unload request
2024-07-25 15:59:48 [201,600ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /isaac-sim/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2024-07-25 15:59:48 [201,662ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.

I somehow got it to run by installing libopenh264-dev inside the container, but it’s painfully slow. How do I get hardware encoding to work?

2 Likes

This is the output of nvidia-smi:

Fri Jul 26 06:59:23 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:80:00.0 Off |                    0 |
| N/A   53C    P8             17W /   72W |       1MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

TL;DR: the drivers version is 550. I’ve just noticed that in the Isaac Sim documentation it says:

  • The latest NVIDIA drivers may not be fully supported for some features like livestreaming.

So could the problem be that my drivers are too new?

also evaluating moving to an NVIDIA L4 GPU. any word on the driver version / gpu compatibility for isaacsim? thanks for any pointers!