Isaac Sim container failed to load (native & WebRTC) livestreaming server due to Net Stream Creation failed

Te77iiiii_Ho · May 11, 2023, 3:16am

when I was using Omniverse Streaming Client to connect to my remote server(isaac sim headless container), I met the following error “unknown error occured (0x100)” with black screen. Then I opened the log it shows 3 errors:

- ERROR [Geronimo::Analytics] {197A5140} - Failed to load dll with error: /home/stardust/.local/share/ov/pkg/kit_remote-103.1.1//NvTelemetryAPI.so: cannot open shared object file: No such file or directory

- ERROR [BifrostClient: Streamer] {197A5140} -  updateVideoSettingsForNVbProfile: profile 8 is not handled

- ERROR [NVST:RtspClient] {0E728700} -  Failed to create RTSP Session: Timeout: connect timed out

I can handle the 3rd error with open the target port which is 48010, how to solve the first two errors?

thanks.

Te77iiiii_Ho · May 12, 2023, 3:25am

after checked the server side I found the following errors on server:

[Error] [carb.l
ivestream.plugin] Stream Server: Net Stream Creation failed

[Error] [carb.livestream.plugin] Could not initialize streaming
components

[Error] [carb.livestream.plugin] Couldn't initialize the capture
 device.

then I search the forums for helps, found some related topics about this issue(Webrtc is not working, "Couldn't initialize the capture device" when using webrtc streaming), but all of them were remain unsolved.

any suggestions for this issue?

my station:

nvidia A10 * 2
ubuntu 20.04
driver version: 525.85.05
CUDA Version: 12.0
run on isaac sim container(https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-sim)
Isaac Sim Headless Native Version: 2022.2.1-rc.14
./runheadless.native.sh

rthaker · May 15, 2023, 9:39pm

Hi @Te77iiiii_Ho - Can you please confirm that you have reviewed this documentation and followed the instructions for native and WebRTC?
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/manual_livestream_clients.html#isaac-sim-setup-kit-remote

rthaker · May 15, 2023, 9:43pm

In addition, can you run the commands suggested by Sheikh in this previous post and let us know.

Te77iiiii_Ho · May 16, 2023, 2:19am

Yes, I did follow the instructions from official docs. Just like I said in the 2nd post, its more likely to be a server issue. Also, I run docker with the command just the same with Sheikh’s, the only difference is my driver is 525.85.05, and the original post suggest 525.89.02. I don’t think its a driver version issue cause I can run headless.websocket isaac sim but have problems with headless.WebRTC and headless.Native.

Sheikh_Dawood · May 16, 2023, 4:19pm

Hi, either of the 525 drivers should work fine.
Try running Isaac Sim with the “-v” flag to see more info in the logs.
Please open all required ports for native livestreaming to work.

Which OS are you running Omniverse Streaming Client on?

Te77iiiii_Ho · May 17, 2023, 3:47am

Hi, @Sheikh_Dawood . the required ports were all open; for the “-v” flag, I used to check the log file for server, I think it works the same with -v flag; For the client side, I tested ubuntu18.04 and ubuntu20.04, both using omniverse streaming client;
here’s the log with “-v” flag:

2023-05-17 03:32:22 [10,915ms] [Info] [carb] Registered plugin: carb.livestream.plugin (interfaces: [carb::livestream::Vision v0.4]) (impl: carb.livestream.plugin)
2023-05-17 03:32:22 [10,916ms] [Info] [omni.ext._impl._internal] Searching for classes derived from 'omni.ext.IExt' in 'omni.kit.livestream.native' ('/isaac-sim/kit/exts/omni.kit.livestream.native')
2023-05-17 03:32:22 [10,916ms] [Info] [omni.ext._impl._internal] Found class `NativeStreamingExtension` derived from `omni.ext.IExt` in module: 'omni.kit.livestream.native' in '/isaac-sim/kit/exts/omni.kit.livestream.native'. Calling `on_startup`.
2023-05-17 03:32:22 [10,916ms] [Info] [carb] Initializing plugin: omni.kit.livestream.plugin (interfaces: [omni::kit::livestream::ILivestream v0.1]) (impl: omni.kit.livestream.plugin)
2023-05-17 03:32:22 [10,917ms] [Info] [carb] Initializing plugin: carb.livestream.plugin (interfaces: [carb::livestream::Vision v0.4]) (impl: carb.livestream.plugin)
2023-05-17 03:32:22 [10,917ms] [Info] [carb.livestream.plugin] Attempting to start Nvst Server...
2023-05-17 03:32:22 [10,917ms] [Info] [carb.livestream.plugin] Stream Server: Video stream created
2023-05-17 03:32:22 [10,917ms] [Info] [carb.livestream.plugin] Stream Server: input stream created

Active user not found. Using default user [kiosk]2023-05-17 03:32:22 [11,104ms] [Error] [carb.livestream.plugin] Stream Server: Net Stream Creation failed
2023-05-17 03:32:22 [11,104ms] [Error] [carb.livestream.plugin] Could not initialize streaming components
2023-05-17 03:32:22 [11,104ms] [Error] [carb.livestream.plugin] Couldn't initialize the capture device.
2023-05-17 03:32:22 [11,105ms] [Info] [omni.kit.livestream.native.scripts.extension] The Streaming Manager extension was not available when enabling native streaming features.
2023-05-17 03:32:22 [11,106ms] [Info] [omni.ext.plugin] [ext: omni.kit.livestream.native-0.1.5] started, startup time: 197 (ms)
2023-05-17 03:32:22 [11,106ms] [Info] [omni.ext.plugin] About to startup: [ext: omni.services.client-0.4.0] (order: 0) Triggered by: omni.kit.thumbnails.images-0.2.3, omni.kit.window.movie_capture-2.1.8, omni.services.starfleet.auth-0.1.4, omni.services.transport.client.http_async-1.3.2 (path: /isaac-sim/extscache/omni.services.client-0.4.0+cp37)
2023-05-17 03:32:22 [11,106ms] [Info] [omni.kit.app.plugin] [11.140s] [ext: omni.services.client-0.4.0] startup
[11.140s] [ext: omni.services.client-0.4.0] startup
2023-05-17 03:32:22 [11,114ms] [Info] [omni.ext._impl._internal] Searching for classes derived from 'omni.ext.IExt' in 'omni.services.client' ('/isaac-sim/extscache/omni.services.client-0.4.0+cp37')
2023-05-17 03:32:22 [11,114ms] [Info] [omni.ext.plugin] [ext: omni.services.client-0.4.0] started, startup time: 8 (ms)
2023-05-17 03:32:22 [11,115ms] [Info] [omni.ext.plugin] About to startup: [ext: omni.kit.viewport.rtx-104.0.0] (order: 0) Triggered by: omni.isaac.sim.base-2022.2.1, omni.kit.thumbnails.mdl-1.0.20 (path: /isaac-sim/kit/exts/omni.kit.viewport.rtx)
2023-05-17 03:32:22 [11,115ms] [Info] [omni.kit.app.plugin] [11.149s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[11.149s] [ext: omni.kit.viewport.rtx-104.0.0] startup
2023-05-17 03:32:22 [11,115ms] [Info] [omni.ext.plugin] [ext: omni.kit.viewport.rtx-104.0.0] started, startup time: 0 (ms)
2023-05-17 03:32:22 [11,115ms] [Info] [omni.ext.plugin] About to startup: [ext: omni.services.transport.server.base-1.1.0] (order: 0) Triggered by: omni.services.transport.server.http-1.1.2 (path: /isaac-sim/extscache/omni.services.transport.server.base-1.1.0)
2023-05-17 03:32:22 [11,115ms] [Info] [omni.kit.app.plugin] [11.149s] [ext: omni.services.transport.server.base-1.1.0] startup
[11.149s] [ext: omni.services.transport.server.base-1.1.0] startup
2023-05-17 03:32:22 [11,116ms] [Info] [omni.ext._impl._internal] Searching for classes derived from 'omni.ext.IExt' in 'omni.services.transport.server.base' ('/isaac-sim/extscache/omni.services.transport.server.base-1.1.0'

Sheikh_Dawood · May 24, 2023, 2:06pm

Hi. Can you share the full logs and the commands you run for both native and webrtc livestreaming?
Also please share the output of nvidia-smi.

Te77iiiii_Ho · May 25, 2023, 7:45am

Hi, @Sheikh_Dawood , Thanks for the reply.
Here’s the nvidia-smi info:

Thu May 25 03:42:26 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.05    Driver Version: 525.85.05    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A10          Off  | 00000000:4B:00.0 Off |                    0 |
|  0%   35C    P8    15W / 150W |      6MiB / 23028MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A10          Off  | 00000000:CA:00.0 Off |                    0 |
|  0%   37C    P8    16W / 150W |      5MiB / 23028MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1779      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      1779      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

And here’s the log, and the command which I ran headless isaac-sim is simply ./runheadless.native.sh in docker container:
kit_20230525_072732.log (1.2 MB)

Sheikh_Dawood · May 30, 2023, 5:01pm

Hi. thanks for the logs. I believe this could be a driver version support issue for livestreaming.
Try updating to the latest production branch drivers ( 525.116.04) or the recommended 525.60.11 drivers.

You could also see more info on the livestream server log with this flag when running Isaac Sim: --/app/livestream/logLevel=verbose

akash.parikh · August 6, 2024, 11:51am

I am having similar error.

Active user not found. Using default user [kiosk]2024-08-06 10:35:01 [11,436ms] [Error] [carb.livestream-rtc.plugin] Stream Server: Net Stream Creation failed, 0x800E8401
2024-08-06 10:35:01 [11,436ms] [Error] [carb.livestream-rtc.plugin] Could not initialize streaming components
2024-08-06 10:35:01 [11,436ms] [Error] [carb.livestream-rtc.plugin] Couldn't initialize the capture device.

At the end it shows that the app has been loaded but I see only gray screen when I open it on web browser.

I have attached a file containing the log after running “./isaac-sim.headless.webrtc.sh” command.
isaac_webrtc_error.txt (37.8 KB)

This is my nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.28.03              Driver Version: 560.28.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:21:00.0  On |                  N/A |
| 57%   73C    P2            302W /  350W |    2610MiB /  24576MiB |     95%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2172      G   /usr/lib/xorg/Xorg                            198MiB |
|    0   N/A  N/A      3181      G   /usr/lib/xorg/Xorg                            138MiB |
|    0   N/A  N/A      3392      G   /usr/bin/gnome-shell                           55MiB |
|    0   N/A  N/A      4443      G   ...ures=SpareRendererForSitePerProcess         42MiB |
|    0   N/A  N/A     10463    C+G   ...e/ov/pkg/isaac_sim-2023.1.0/kit/kit       2102MiB |
+-----------------------------------------------------------------------------------------+

What can be the reason for gray screen? Should I downgrade my driver version? Which version?

Topic		Replies	Views
Connect Isaac Sim on a docker container through Omniverse streaming client remotely Isaac Sim ubuntu , docker , hpc , containers , isaacsim , kit-remote , remote-client , isaac-sdk , a100	3	1469	May 3, 2023
Error occured during the installation of IsaaC SIM Isaac Sim setup	10	2852	April 5, 2024
Isaac Sim (2021.1.1) Wont Launch (Omniverse 1.3.4) Isaac Sim aws , setup	23	4160	April 5, 2024
IsaacSim livestreaming container with 525.125 driver Isaac Sim	4	492	October 23, 2023
Issues with starting Isaac Sim headless in a docker container and Workstation installation Isaac Sim containers , installation , setup	32	3885	March 4, 2024
Unable to use webrtc when I run runheadless.webrtc.sh in remote headless container Isaac Sim	28	2967	October 20, 2023
Livestream.py is not working for me for 2023.1.0 Isaac Sim	19	722	November 28, 2023
Isaac SIM in omniverse not launching Isaac Sim setup	62	8965	August 31, 2022
Isaac Sim Headless native fails to initialize streaming Isaac Sim gstreamer	3	217	November 5, 2024
Omniverse Streaming Client Blanck screen error Isaac Sim	8	982	October 6, 2023

Isaac Sim container failed to load (native & WebRTC) livestreaming server due to Net Stream Creation failed

Related topics