Omniverse Remote Kit Remote Client Issues

I am attempting to run ISAAC SIM headless using on my remote machine.

sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm -p 47995-48012:47995-48012/udp -p 47995-48012:47995-48012/tcp -p 49isaac-sim:2020.1_ea --network host

On my ubuntu 18.04 client, I am getting the following issue

Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  perform() failed: 28
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspPocoEvent] {D8A7E700} -  RTSP-XNvEvent Polling failed: 28, rc: 408
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Cannot get server options - vinny@192.168.6.141:48010 via RTSP, 28:NOT FOUND
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  rtsp perform() returnCode:Host not found, [N4Poco3Net21HostNotFoundExceptionE]
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  perform() failed: 28
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspPocoEvent] {D8A7E700} -  RTSP-XNvEvent Polling failed: 28, rc: 408
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Failed to connect to server using RTSP handshake (error codes: 0X8008201C 0X8008201C)
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Failed to create RTSP connection with error 0X8008201C
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientLibraryWrapper] {FD3B4700} -  Could not create initial streaming connection : 0x8008201c
Mon Oct 12 20:53:27:824   ERROR [NVST:ConvertNvstResultToString] {FD3B4700} -  Unhandled NvstResult -2146951140
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: Streamer] {FD3B4700} - Failed to initiate connection to rtsp://vinny@192.168.6.141:48010: Error: Unknown (0x8008201c)
Mon Oct 12 20:53:27:824   ERROR [NVST:ConvertNvstResultToString] {FD3B4700} -  Unhandled NvstResult -2146951140
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: Streamer] {FD3B4700} - Streamer connection failure. Error: Unknown (0x8008201c)
Mon Oct 12 20:53:27:825   ERROR [RemoteGridApp] {FC3B2700} - STREAMING: FAILED: NVB_SN_STREAMING_START_FAILED (8008201c)
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: SessionController] {FD3B4700} - Failed to start the streamer for session dab9131b-f033-4e3c-b2be-3fc2630131a2. Aborting the session.

On my remote client, this just produces a blank window in which functionality does not work. Is this a video-related issue or a network related one? I can SSH on the server machine just fine.

Just to close the loop on this, it seems to have been a combination of docker networking and VPN networking issues on our side (blocked ports).
It is also important to know that the server only supports one client

Are you using AWS? I am still struggling to get my client to connect to my AWS server instance so any more details you have on what you did to resolve the issue would be great.

No this was outside of AWS.
One change we found that helped was this:
sudo docker run --gpus all -e "ACCEPT_EULA=Y" --network=host --rm -p 47995-48012:47995-48012/udp -p 47995-48012:47995-48012/tcp -p 49 isaac-sim:2020.1_ea
Apparently the placement of --network=host mattered, still not sure why.

We also tried paying special attention to the ports. I got this message in the above log which I should have paid more attention to:
Mon Oct 12 20:53:27:824 ERROR [BifrostClient: Streamer] {FD3B4700} - Failed to initiate connection to rtsp://vinny@192.168.6.141:48010: Error: Unknown (0x8008201c)

Which made us look into specific ports.

Sorry for these issues. Yes, please use --network=host when running docker for now. We are investigating the livestream issues and will be updating the documentation with the upcoming release.

2 Likes

@Sheikh_Dawood

For further information, I ran into a similar issue.

At our lab, the server running Isaac Sim sits inside a different VLAN, say VLAN 2, than my notebook, VLAN 1. Furthermore, our VPN (openVPN based as far as I can tell) tunnels me into VLAN 3. Between VLAN 1 and 2, and VLAN 2 and 3, ports 80, 3000, 3009, 3030, 8080, 47995-48012, and 49000-49007 are towards VLAN 2, both for udp, as well as tcp/ip.
Now, running the Isaac Sim docker on the server explicitly with the ports from the documentation, i.e. 47995-48012 and 49000-49007 for udp and tcp/ip, I can connect from VLAN 1 using the client no problem. However, I cannot connect via VPN, except if I run the Isaac Sim docker using --network=host.
Attached are some connection logs, the first is from connecting from VLAN 1 to the server, the second from VLAN 3 (VPN) to the server. I noticed how the client tries to open ports which are outside the range specified in the docker run command. Nevertheless, it’s working fine.

If you have any more questions regarding our setup and what works/does not work, feel free to contact me.

Thank you,

Johann

PS: Not sure if it helps, but I didn’t want to open yet another issue for the same problem just to get the same info.

VLAN 1 to VLAN 2 (successfull)

2020-10-15 11:44:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63052 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63051 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:32 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63056 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:58 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63062 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63063 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63068 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63069 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63070 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63071 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:27 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63072 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:31 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63073 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=47998/udp src_port=49005 dst_port=47998 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=47999/udp src_port=49006 dst_port=47999 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48000/udp src_port=49003 dst_port=48000 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63074 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63075 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48199/udp src_port=49006 dst_port=48199 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48200/udp src_port=49003 dst_port=48200 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48198/udp src_port=49005 dst_port=48198 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:35 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48000/udp src_port=49003 dst_port=48000 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127

VLAN 3 (VPN) to VLAN 2 (failed)

2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8061 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8062 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8063 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8064 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:23 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8065 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:27 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8066 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:30 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=47999/udp src_port=49006 dst_port=47999 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:46:30 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8067 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:30 Gateway Deny src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48299/udp src_port=49006 dst_port=48299 src_intf=VLAN3 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
1 Like

Thanks for the feedback, this is helpful. 48199-48200 and 48299 is not the port range that the client should use. I’ll get this to the team.

1 Like