Omniverse Kit Remote Client can't connect to AWS

Omniverse Isaac Sim
,
1

I’m trying to setup an Isaac Sim instance on AWS but I have not been able to connect to it with the Omniverse Kit Remote Client.

I am running the client on Ubuntu 18.04.

When I run the client I get:

jservos@CPR02011L:/mnt/data/nvidia/omniverse-kit-remote$  ./omniverse-kit-remote.sh -s  3.128.236.182 /mnt/data/nvidia/omniverse-kit-remote/bin /mnt/data/nvidia/omniverse-kit-remote
I-!$> Sun Oct 11 17:26:54:898    INFO [BifrostClient: Scheduler] {691B6700} - Successfully set TLS with value 0x15f12d0
I-!$> Sun Oct 11 17:26:54:898    INFO [BifrostClient: Scheduler] {699B7700} - Successfully set TLS with value 0x15f12d0
I-!$> Sun Oct 11 17:26:54:898    INFO [BifrostClient: Scheduler] {689B5700} - Successfully set TLS with value 0x15f12d0
I-!$> Sun Oct 11 17:26:54:898    INFO [BifrostClient: Scheduler] {6A1B8700} - Successfully set TLS with value 0x15f12d0
Sun Oct 11 17:26:55:480   ERROR [BifrostClient: Streamer] {74830180} -  updateVideoSettingsForNVbProfile: profile 8 is not handled

Sun Oct 11 17:26:55:583   ERROR [NVST:ClientSession] {699B7700} -  Server URL is invalid 3.128.236.182
Sun Oct 11 17:26:55:888   ERROR [NVST:ClientSession] {699B7700} -  Number of channels(2) is not valid for surround configuration
Sun Oct 11 17:26:55:888   ERROR [NVST:ClientSession] {699B7700} -  Either in stereo or error in receiving opus information from server
Sun Oct 11 17:26:55:889   ERROR [NVST:ClientSession] {699B7700} -  Server URL is invalid 3.128.236.182
Sun Oct 11 17:26:56:804   ERROR [NVST:NattHolePunch] {37FFF700} -  NattHolePunch: Failed to receive hole punching packet
Sun Oct 11 17:26:56:804   ERROR [NVST:ClientSession] {699B7700} -  Control channel hole punching failed.
Sun Oct 11 17:27:02:814   ERROR [NVST:EnetClientSocket] {699B7700} -  Failed to connect to specified host
Sun Oct 11 17:27:02:814   ERROR [NVST:ServerControlEnet] {699B7700} -  Could not create control connection to server 172.17.0.2 port 47999
Sun Oct 11 17:27:02:815   ERROR [NVST:ClientSession] {699B7700} -  Failed to connect to server using RTSP handshake (error codes: 0X800C0014 0X800C0014)
Sun Oct 11 17:27:02:815   ERROR [NVST:ClientSession] {699B7700} -  Failed to create RTSP connection with error 0X800C0014
Sun Oct 11 17:27:02:815   ERROR [NVST:ClientLibraryWrapper] {699B7700} -  Could not create initial streaming connection : 0x800c0014
Sun Oct 11 17:27:02:815   ERROR [BifrostClient: Streamer] {699B7700} - Failed to initiate connection to rtsp://3.128.236.182:48010: Error: NVST_R_SERVER_CONTROL_HOLE_PUNCH_FAILURE (0x800c0014)
Sun Oct 11 17:27:02:815   ERROR [BifrostClient: Streamer] {699B7700} - Streamer connection failure. Error: NVST_R_SERVER_CONTROL_HOLE_PUNCH_FAILURE (0x800c0014)
Sun Oct 11 17:27:02:815   ERROR [RemoteGridApp] {691B6700} - STREAMING: FAILED: NVB_SN_STREAMING_START_FAILED (800c0014)
Sun Oct 11 17:27:02:815   ERROR [BifrostClient: SessionController] {699B7700} - Failed to start the streamer for session f1d5bec4-3140-45e4-8f3e-979a83521f11. Aborting the session.
Sun Oct 11 17:27:03:016   ERROR [NVST:RtspSessionPocoBase] {456FF700} -  rtsp perform() returnCode:No message received, [N4Poco3Net18NoMessageExceptionE]
Sun Oct 11 17:27:03:017   ERROR [NVST:RtspSessionPocoBase] {456FF700} -  perform() failed: 28
Sun Oct 11 17:27:03:017   ERROR [NVST:RtspPocoEvent] {456FF700} -  RTSP-XNvEvent Polling failed: 28, rc: 408
I-!#>
 Sun Oct 11 17:27:03:019    INFO [BifrostClient: Interface] {74830180} - nvbDestroyClient(). Successfully destroyed NVbClient
./omniverse-kit-remote.sh: line 18: 31492 Segmentation fault      (core dumped) ./omniverse-kit-remote $@

I can ping/ssh to the server just fine. Everything on the server side seems to be running just fine (no errors showing up that I can see). I followed the instructions here (https://docs.omniverse.nvidia.com/exp_robotics/exp_robotics/Robotics_QuickStart.html#aws-deployment) exactly.

Any help or suggestions would be appreciated.

2

Ah I just saw this note at the top of the AWS instructions:

Live-streaming to Isaac Sim on an AWS instance is not working for some cases. This user guide will be updated when the live-streaming feature is fixed. There are no issues with live-streaming Isaac Sim on remote workstation within the same network.

So I’m guessing AWS deployment is broken at the moment… any ETA on a fix?

3

Yes, the livestreaming from AWS is not working for some instances. Apologies, there is no ETA on a fix yet.
You could try using the --network host flag when running the docker container and have all ports open from your instance to the client IP address.

4

Thanks for the response.

I tried with --network host but did not seem to improve things.

On my AWS instance I beleive I have all the ports open as per the instructions:

Inbound rules
|Port range|Protocol|Source|Security groups|
|All|All|xx.xxx.xx.xxx/32|ISSAC_GPU_ACCESS|

Outbound rules
Port range|Protocol|Destination|Security groups
All|All|0.0.0.0/0|ISSAC_GPU_ACCESS

I unfortunatly don’t have a RTX graphics card locally so I can’t test it on my local network.

If there is a “Known Good” configuration for AWS that works I would be happy to change anything.

5

Hi,

I’m facing the same exact problem. Could let me know when you can have a fix for this.

6

Hi. We have a fix for this in the upcoming 2020.2 EA release this week. Please stay tuned.

1 Like
7

Yes, those Security Group settings are recommended. You can open all traffic to your public IP address. We will have a fix for this in the next release, in a couple of days.

1 Like
9

Hey,

I have the same issue as above. When the issue will be solved? Or does it exist some settings how i can skip this errors? Thank you in advance.

10

Hi, have you tried allowing all traffic to your IP address in the inbound rules?

11

Yes, I tried the same as above (see jservos message).

12

Please post your Isaac Sim and Kit Remote logs. Are you using the latest 2020.2.2 EA release and 2020.2.1022 Kit Remote Client?

13

Here my Kit Remote logs:

~/Downloads/omniverse-kit-remote-2020.2.1022-linux-x86_64/bin ~/Downloads/omniverse-kit-remote-2020.2.1022-linux-x86_64
I-!$> [2021-05-05 16:03:30,881]=18:03:30= INFO [BifrostClient: Scheduler] {77FFF700} - Successfully set TLS with value 0x14141d0
I-!$> [2021-05-05 16:03:30,881]=18:03:30= INFO [BifrostClient: Scheduler] {7F910700} - Successfully set TLS with value 0x14141d0
I-!$> [2021-05-05 16:03:30,881]=18:03:30= INFO [BifrostClient: Scheduler] {80912700} - Successfully set TLS with value 0x14141d0
I-!$> [2021-05-05 16:03:30,881]=18:03:30= INFO [BifrostClient: Scheduler] {80111700} - Successfully set TLS with value 0x14141d0
I-!$> [2021-05-05 16:03:30,886]=18:03:30= DEBUG [BifrostClient: Interface] {8D604200} - nvbCreateClient( 0x1709a00 )
[2021-05-05 16:03:31,635]=18:03:31= ERROR [BifrostClient: Streamer] {8D604200} - updateVideoSettingsForNVbProfile: profile 8 is not handled

[2021-05-05 16:03:37,770]=18:03:37= ERROR [NVST:RtspSessionPocoBase] {591AA700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:37,770]=18:03:37= ERROR [NVST:RtspSessionPocoBase] {591AA700} - perform() failed: 28
[2021-05-05 16:03:37,770]=18:03:37= ERROR [NVST:RtspPocoEvent] {591AA700} - RTSP-XNvEvent Polling failed: 28, rc: 408
[2021-05-05 16:03:37,771]=18:03:37= ERROR [NVST:RtspSessionPocoBase] {80912700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:37,771]=18:03:37= ERROR [NVST:RtspSessionPocoBase] {80912700} - perform Request failed with Bad request (408)…Retrying in 3000ms
[2021-05-05 16:03:41,738]=18:03:41= ERROR [BifrostClient: Streamer] {80111700} - *** NVSC LIBRARY method ‘nvstConnectToServer’ has not returned in 10 seconds.
[2021-05-05 16:03:44,776]=18:03:44= ERROR [NVST:RtspSessionPocoBase] {591AA700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:44,776]=18:03:44= ERROR [NVST:RtspSessionPocoBase] {591AA700} - perform() failed: 28
[2021-05-05 16:03:44,776]=18:03:44= ERROR [NVST:RtspPocoEvent] {591AA700} - RTSP-XNvEvent Polling failed: 28, rc: 408
[2021-05-05 16:03:46,776]=18:03:46= ERROR [NVST:RtspSessionPocoBase] {80912700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:46,776]=18:03:46= ERROR [NVST:RtspSessionPocoBase] {80912700} - perform Request failed with Bad request (408)…Retrying in 6000ms
[2021-05-05 16:03:51,738]=18:03:51= ERROR [BifrostClient: Streamer] {80111700} - *** NVSC LIBRARY method ‘nvstConnectToServer’ has not returned in 20 seconds.
[2021-05-05 16:03:51,780]=18:03:51= ERROR [NVST:RtspSessionPocoBase] {591AA700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:51,780]=18:03:51= ERROR [NVST:RtspSessionPocoBase] {591AA700} - perform() failed: 28
[2021-05-05 16:03:51,780]=18:03:51= ERROR [NVST:RtspPocoEvent] {591AA700} - RTSP-XNvEvent Polling failed: 28, rc: 408
[2021-05-05 16:03:58,784]=18:03:58= ERROR [NVST:RtspSessionPocoBase] {591AA700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:58,784]=18:03:58= ERROR [NVST:RtspSessionPocoBase] {591AA700} - perform() failed: 28
[2021-05-05 16:03:58,784]=18:03:58= ERROR [NVST:RtspPocoEvent] {591AA700} - RTSP-XNvEvent Polling failed: 28, rc: 408
[2021-05-05 16:03:58,784]=18:03:58= ERROR [NVST:RtspSessionPocoBase] {80912700} - rtsp perform() returnCode:Timeout, [N4Poco16TimeoutExceptionE]
[2021-05-05 16:03:58,784]=18:03:58= ERROR [NVST:RtspSessionPocoBase] {80912700} - timed out. Retrying in 3000ms
[2021-05-05 16:04:01,739]=18:04:01= ERROR [BifrostClient: Streamer] {80111700} - *** NVSC LIBRARY method ‘nvstConnectToServer’ has not returned in 30 seconds.
[2021-05-05 16:04:01,787]=18:04:01= ERROR [NVST:ClientSession] {80912700} - Cannot get server options - ec2-xxxxxxxxxxx.eu-west-1.compute.amazonaws.com:48010 via RTSP, 2148016156:NOT FOUND
[2021-05-05 16:04:05,790]=18:04:05= ERROR [NVST:RtspSessionPocoBase] {591AA700} - rtsp perform() returnCode:I/O error, [N4Poco11IOExceptionE]
[2021-05-05 16:04:05,790]=18:04:05= ERROR [NVST:RtspSessionPocoBase] {591AA700} - perform() failed: 28
[2021-05-05 16:04:05,790]=18:04:05= ERROR [NVST:RtspPocoEvent] {591AA700} - RTSP-XNvEvent Polling failed: 28, rc: 408
[2021-05-05 16:04:06,791]=18:04:06= ERROR [NVST:ClientSession] {80912700} - Failed to connect to server using RTSP handshake (error codes: 0X8008201C 0X8008201C)
[2021-05-05 16:04:06,791]=18:04:06= ERROR [NVST:ClientSession] {80912700} - Failed to create RTSP connection with error 0X8008201C
[2021-05-05 16:04:06,791]=18:04:06= ERROR [NVST:ClientLibraryWrapper] {80912700} - Could not create initial streaming connection : 0x8008201c
[2021-05-05 16:04:06,791]=18:04:06= ERROR [BifrostClient: Streamer] {80912700} - Failed to initiate connection to rtsp://ec2-xxxxxxxxxx.eu-west-1.compute.amazonaws.com:48010: Error: NVSTE_RTSP_OPTIONS_RTSPE_OPERATION_TIMEDOUT (0x8008201c)
[2021-05-05 16:04:06,792]=18:04:06= ERROR [BifrostClient: Streamer] {80912700} - Streamer connection failure. Error: NVSTE_RTSP_OPTIONS_RTSPE_OPERATION_TIMEDOUT (0x8008201c)
[2021-05-05 16:04:06,792]=18:04:06= ERROR [BifrostClient: SessionController] {80912700} - Failed to start the streamer for session 206ad6a5-1978-43c6-9954-8d3c3082fcb2. Aborting the session.
[2021-05-05 16:04:06,792]=18:04:06= ERROR [RemoteGridApp] {77FFF700} - STREAMING: FAILED: NVB_SN_STREAMING_START_FAILED (8008201c)
I-!#>
[2021-05-05 16:04:06,995]=18:04:06= INFO [BifrostClient: Interface] {8D604200} - nvbDestroyClient(). Successfully destroyed NVbClient
Active user not found. Using default user [kiosk]~/Downloads/omniverse-kit-remote-2020.2.1022-linux-x86_64

Yes, I use the latest 2020.2.2 EA release and 2020.2.1022 Kit Remote Client version.

I do not get any system logs from my EC2 instance (Instances/Actions/Instance Settings/Get System Log). I took “IsaacSim-Ubuntu-18.04-GPU-2020-10-22 - ami-0200d75a4d6d97bb7” with g4dn.xlarge Type and eu-west-1a as AZ.

Best regards,
Dima

14

It doesn’t look like you have a NAT issue on the Client.
Here’s a guide to read the Isaac Sim logs and check if the streaming server is running: https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/setup.html#reading-the-logs-in-a-container
Do you also set the SSH and ICMP only to your public IP address of the Client and see if you can ping the Instance?

15

Hi,

here are the logs:
x86_64/release/data/Kit/isaac-sim-headless/2020.2.28460# grep “Stream Server” omniverse-kit.log
2021-05-18 11:35:39 [22,597ms] [Info] [carb.livestream.plugin] Stream Server: Video stream created
2021-05-18 11:35:39 [22,597ms] [Info] [carb.livestream.plugin] Stream Server: input stream created
2021-05-18 11:35:42 [25,002ms] [Info] [carb.livestream.plugin] Stream Server: streaming instance started - waiting for a client…
root@ip-172-31-35-44:/isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-
x86_64/release/data/Kit/isaac-sim-headless/2020.2.28460#

Yes, i can ping the Instance and it works:

ubuntu@ip-172-31-35-44:~$ ping -c5 172.31.35.44
PING 172.31.35.44 (172.31.35.44) 56(84) bytes of data.
64 bytes from 172.31.35.44: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 172.31.35.44: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from 172.31.35.44: icmp_seq=3 ttl=64 time=0.040 ms
64 bytes from 172.31.35.44: icmp_seq=4 ttl=64 time=0.034 ms
64 bytes from 172.31.35.44: icmp_seq=5 ttl=64 time=0.034 ms

hmm, it seems everything is fine, but it does not work.

Best regards,
Dima