Omniverse Remote Kit Remote Client Issues

vinny.ruia · October 12, 2020, 8:55pm

I am attempting to run ISAAC SIM headless using on my remote machine.

sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm -p 47995-48012:47995-48012/udp -p 47995-48012:47995-48012/tcp -p 49isaac-sim:2020.1_ea --network host

On my ubuntu 18.04 client, I am getting the following issue

Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  perform() failed: 28
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspPocoEvent] {D8A7E700} -  RTSP-XNvEvent Polling failed: 28, rc: 408
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Cannot get server options - vinny@192.168.6.141:48010 via RTSP, 28:NOT FOUND
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  rtsp perform() returnCode:Host not found, [N4Poco3Net21HostNotFoundExceptionE]
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspSessionPocoBase] {D8A7E700} -  perform() failed: 28
Mon Oct 12 20:53:27:824   ERROR [NVST:RtspPocoEvent] {D8A7E700} -  RTSP-XNvEvent Polling failed: 28, rc: 408
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Failed to connect to server using RTSP handshake (error codes: 0X8008201C 0X8008201C)
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientSession] {FD3B4700} -  Failed to create RTSP connection with error 0X8008201C
Mon Oct 12 20:53:27:824   ERROR [NVST:ClientLibraryWrapper] {FD3B4700} -  Could not create initial streaming connection : 0x8008201c
Mon Oct 12 20:53:27:824   ERROR [NVST:ConvertNvstResultToString] {FD3B4700} -  Unhandled NvstResult -2146951140
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: Streamer] {FD3B4700} - Failed to initiate connection to rtsp://vinny@192.168.6.141:48010: Error: Unknown (0x8008201c)
Mon Oct 12 20:53:27:824   ERROR [NVST:ConvertNvstResultToString] {FD3B4700} -  Unhandled NvstResult -2146951140
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: Streamer] {FD3B4700} - Streamer connection failure. Error: Unknown (0x8008201c)
Mon Oct 12 20:53:27:825   ERROR [RemoteGridApp] {FC3B2700} - STREAMING: FAILED: NVB_SN_STREAMING_START_FAILED (8008201c)
Mon Oct 12 20:53:27:824   ERROR [BifrostClient: SessionController] {FD3B4700} - Failed to start the streamer for session dab9131b-f033-4e3c-b2be-3fc2630131a2. Aborting the session.

On my remote client, this just produces a blank window in which functionality does not work. Is this a video-related issue or a network related one? I can SSH on the server machine just fine.

vinny.ruia · October 15, 2020, 8:00pm

Just to close the loop on this, it seems to have been a combination of docker networking and VPN networking issues on our side (blocked ports).
It is also important to know that the server only supports one client

jservos · October 15, 2020, 8:03pm

Are you using AWS? I am still struggling to get my client to connect to my AWS server instance so any more details you have on what you did to resolve the issue would be great.

vinny.ruia · October 15, 2020, 8:11pm

No this was outside of AWS.
One change we found that helped was this:
sudo docker run --gpus all -e "ACCEPT_EULA=Y" --network=host --rm -p 47995-48012:47995-48012/udp -p 47995-48012:47995-48012/tcp -p 49 isaac-sim:2020.1_ea
Apparently the placement of --network=host mattered, still not sure why.

We also tried paying special attention to the ports. I got this message in the above log which I should have paid more attention to:
Mon Oct 12 20:53:27:824 ERROR [BifrostClient: Streamer] {FD3B4700} - Failed to initiate connection to rtsp://vinny@192.168.6.141:48010: Error: Unknown (0x8008201c)

Which made us look into specific ports.

Sheikh_Dawood · October 16, 2020, 4:10pm

Sorry for these issues. Yes, please use --network=host when running docker for now. We are investigating the livestream issues and will be updating the documentation with the upcoming release.

johann.lange · October 19, 2020, 8:02am

@Sheikh_Dawood

For further information, I ran into a similar issue.

At our lab, the server running Isaac Sim sits inside a different VLAN, say VLAN 2, than my notebook, VLAN 1. Furthermore, our VPN (openVPN based as far as I can tell) tunnels me into VLAN 3. Between VLAN 1 and 2, and VLAN 2 and 3, ports 80, 3000, 3009, 3030, 8080, 47995-48012, and 49000-49007 are towards VLAN 2, both for udp, as well as tcp/ip.
Now, running the Isaac Sim docker on the server explicitly with the ports from the documentation, i.e. 47995-48012 and 49000-49007 for udp and tcp/ip, I can connect from VLAN 1 using the client no problem. However, I cannot connect via VPN, except if I run the Isaac Sim docker using --network=host.
Attached are some connection logs, the first is from connecting from VLAN 1 to the server, the second from VLAN 3 (VPN) to the server. I noticed how the client tries to open ports which are outside the range specified in the docker run command. Nevertheless, it’s working fine.

If you have any more questions regarding our setup and what works/does not work, feel free to contact me.

Thank you,

Johann

PS: Not sure if it helps, but I didn’t want to open yet another issue for the same problem just to get the same info.

VLAN 1 to VLAN 2 (successfull)

2020-10-15 11:44:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63052 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63051 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:32 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63056 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:44:58 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63062 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:09 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63063 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63068 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63069 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63070 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:24 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63071 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:27 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63072 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:31 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63073 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=47998/udp src_port=49005 dst_port=47998 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=47999/udp src_port=49006 dst_port=47999 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48000/udp src_port=49003 dst_port=48000 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63074 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=63075 dst_port=48010 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48199/udp src_port=49006 dst_port=48199 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48200/udp src_port=49003 dst_port=48200 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:34 Gateway Deny src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48198/udp src_port=49005 dst_port=48198 src_intf=VLAN1 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127
2020-10-15 11:45:35 Gateway Allow src_ip=10.1.0.1 dst_ip=10.2.0.1 pr=48000/udp src_port=49003 dst_port=48000 src_intf=VLAN1 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127

VLAN 3 (VPN) to VLAN 2 (failed)

2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8061 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8062 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8063 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:19 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8064 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:23 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8065 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:27 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8066 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:30 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=47999/udp src_port=49006 dst_port=47999 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=47 ttl=127
2020-10-15 11:46:30 Gateway Allow src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48010/tcp src_port=8067 dst_port=48010 src_intf=VLAN3 dst_intf=VLAN2 msg=Allowed pckt_len=52 ttl=127
2020-10-15 11:46:30 Gateway Deny src_ip=10.3.0.1 dst_ip=10.2.0.1 pr=48299/udp src_port=49006 dst_port=48299 src_intf=VLAN3 dst_intf=VLAN2 msg=Denied pckt_len=47 ttl=127

Sheikh_Dawood · October 19, 2020, 3:04pm

Thanks for the feedback, this is helpful. 48199-48200 and 48299 is not the port range that the client should use. I’ll get this to the team.

johann.lange · November 13, 2020, 7:45am

@Sheikh_Dawood
Any news on the issue?
Using 2020.2_ea I’m still running into this issue.
I’m asking because our server is meant to serve multiple people. But using --network=host I can only spin up one instance at a time :/

Sheikh_Dawood · November 13, 2020, 3:49pm

Just checking, are you using the latest Kit Remote client version (2020.2.1022)?

Does it work if you expose the port 48299 in the container and VLAN/VPN?

We might need to get more information on how your VPN was setup for me to file a bug.

johann.lange · November 13, 2020, 4:00pm

My bad!
Didn’t realize the client got an update too.

Works like charm now, thank you!

newuhe · January 19, 2021, 9:09am

How to open multiple docker instances and let multiple remote kits connect to them individually?

toni.sm · January 19, 2021, 10:34am

Hi,

There is not possible to have multiples clients connected to multiples dockers in the same remote workstation at the moment. However, I agree with you about the relevance of this topic. I had reported this suggestion in the Omniverse developer section. Hope this feature be released soon.

toni.sm · February 11, 2021, 9:43pm

Hi @newuhe,

Do you still interested in running multiple docker instances on the same remote workstation?

Using socat I was able to map the remote kit (fixed ports) with different Isaac Sim containers (each one with different exposed ports) running on the same workstation…

So, one kit for multiples containers (this is the restriction, but at least I can run several simulations at the same time)

N.sannomi · March 8, 2021, 9:21am

Hi @toni.sm .

I just interested in running multiple docker instances.

I have successfully launched multiple Docker containers.
However, I couldn’t connect to them remotely due to a bad socat configuration.

That is docker launch command.

sudo docker run --gpus all -e “ACCEPT_EULA=Y” -e “OMNI_SERVER=IP_ADDRESS” -e “OMNI_USER=user” -e “OMNI_PASS=user” --rm -p 37995-38012:47995-48012/udp -p 37995-38012:47995-48012/tcp -p 39000-39007:49000-49007/tcp -p 39000-39007:49000-49007/udp nvcr.io/nvidia/isaac-sim:2020.2.2_ea

And I use this socat command. (Only the ports that were used by the remote Kit.)

socat udp4-listen:49003 udp4:IP_ADDRESS:39003 & socat udp4-listen:49005 udp4:IP_ADDRESS:39005 & socat udp4-listen:49006 udp4:IP_ADDRESS:39006 & socat udp4-listen:49007 udp4:IP_ADDRESS:39007 & socat tcp4-listen:48010 tcp4:IP_ADDRESS:38010

I started Remote Kit with localhost specified, but could not connect.
Could you tell us how you used socat?

thanks.

toni.sm · March 8, 2021, 5:29pm

Hi @N.sannomi

I have identified as active the next 4 ports: TCP (1): 48010 and UDP (3): 47998-48000. You can check them using ss command
I use (I recommend you) alpine/socat, a docker version for socat. It is useful for non-root users
Launch Isaac-Sim container. Change CUSTOM_TCP (1 port) and CUSTOM_UDP_RANGE (3 ports). Where the CUSTOM… ports are unique for each container…

docker run -it --gpus all -e "ACCEPT_EULA=Y" \ 
    --entrypoint /bin/bash \
    -p CUSTOM_TCP:48010 \
    -p CUSTOM_UDP_RANGE:47998-48000/udp \
    nvcr.io/nvidia/isaac-sim:2020.2.2_ea

Launch alpine/socat. Change CUSTOM_TCP, CUSTOM_UDP_RANGE (CUSTOM_UDP1, CUSTOM_UDP2, CUSTOM_UDP3), and HOST_IP

docker run \
    --detach \
    --publish 48010:CUSTOM_TCP \
    --publish 47998-48000:CUSTOM_UDP_RANGE/udp \
    alpine/socat \
    tcp-listen:CUSTOM_TCP ,fork,reuseaddr tcp-connect:HOST_IP:CUSTOM_TCP  | \
    udp-listen:CUSTOM_UDP1,fork,reuseaddr udp-connect:HOST_IP:CUSTOM_UDP1 | \
    udp-listen:CUSTOM_UDP2,fork,reuseaddr udp-connect:HOST_IP:CUSTOM_UDP2 | \
    udp-listen:CUSTOM_UDP3,fork,reuseaddr udp-connect:HOST_IP:CUSTOM_UDP3

I hope this solve your question :)

N.sannomi · March 9, 2021, 5:59am

@toni.sm

Thanks!
Now, I have access to multiple Omniverse containers in your way.
(I removed the “fork” and “reuseaddr” options from the udp-listen because they caused errors.)

Once a connection was made, it was not a problem to terminate the socat container, so I could have two or more connections at the same time in one remote environment.

There were a few limitations.

We can only access one Omniverse at a time on a NucleusServer, so if we want to use multiple containers, we need to launch multiple NucleusServer.
My environment meets the ideal environment for Omniverse, but it still drops below 30FPS when running 4 samples.(If I run two SRTs and two Carters in one environment, I could get more than 90FPS.)
When I ran an environment that uses ROS and Lidar at the same time, I could only get 10-20 FPS.

Thanks

P.S.
I’ve tried running ROS in multiple Omniverse containers, but only the first container I run can use the /OmniIsaacRosBridge node.
It would be nice to have the option to change the ROS node name in the next update.

toni.sm · March 9, 2021, 8:58am

Hi @N.sannomi

I am glad to hear about you running multiples instances…

Also, I have some comments about the last 2 points:

It is possible to connect multiples containers to the same Nucleus server at the same time… Are you using the “OMNI_SERVER=IP_ADDRESS” -e “OMNI_USER=user” -e “OMNI_PASS=user” configuration to launch the containers?.. Maybe it could be the problem (I am guessing)
There are some tricks to increase the FPS:

Use a low rendering resolution:
You can modify the resolution in the Viewport Settings
- Render Resolution Multiplier: Allows you to up/down sample your viewport based off Render Resolution (below)
- Render Resolution: Base Resolution Rendered in the Viewport (Pixels x Pixels)
White mode
Enable the “White mode” in RTX Renderer Render Settings