Isaac Sim first run on AWS is too much slow

[bigbellmercy]

Hi~

Though I succeeded in running Isaac Sim on AWS by following exactly on the procedure of the docs guide and a workaround for a problem, but it is too slow.

Problem
Its performance is too slow that the robot and mouse cursor does not move when an Isaac Example for a robot is executed.

Environment
Server: AWS EC2
AMI: IsaacSim-Ubuntu-18.04-GPU-2021-11-12
Instance: g4dn.4xlarge (4 vCPU, 64 GB RAM) as recommended on the guide

Workaround
So, I found a work around below, but it may not be the correct solution and it’s not perfrect:

As we know, the option --rm, below in the Docker container running command for Isaac Sim, causes the container to be removed after Isaac Sim is exited:

$ sudo docker run --name isaac-sim --entrypoint bash -it --gpus all -e “ACCEPT_EULA=Y” –rm --network=host
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw
-v ~/docker/isaac-sim/config:/root/.nvidia-omniverse/config:rw
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw
-v ~/docker/isaac-sim/documents:/root/Documents:rw
nvcr.io/nvidia/isaac-sim:2021.2.1

But, after removing the ‘–rm’ option and executing the command as below,

$ sudo docker run --name isaac-sim --entrypoint bash -it --gpus all -e “ACCEPT_EULA=Y” --network=host
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw
-v ~/docker/isaac-sim/config:/root/.nvidia-omniverse/config:rw
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw
-v ~/docker/isaac-sim/documents:/root/Documents:rw
nvcr.io/nvidia/isaac-sim:2021.2.1

by exiting the first slow running container of Isaac Sim and running it again, its performance is so improved that we can see the Isaac Example for a robot works but it’s a little slow.

Question
So, worrying that Isaac Sim could be used in usual workstations or servers,
it will be better if there is a correct solution for the slow performance problem.

Thanks,

[bigbellmercy]

Hi~

Today, the speed has been increased normall onetime, but it happens again several tens of minutes passed. I’m using ‘RoboParty’ example for speed checking. But, after that its speed is normal and here is the console output of Isaac Sim:
console.txt (54.0 KB)

So, it may be related to the problem of AWS service performance, but I don’t know.

So, I’ll observe the speed performance on and report in this post.

And the other problem of the WebSocket black screen flicking issue is reported here:

Problem
The WebSocket screen is flickered with black screen continuously.

Symptom 1

When WebSocket flickering happens, the warning below is repeated and the symptom is decreased after re-run of Isaac Sim in the same container:

[2022-04-18 07:44:58] [connect] WebSocket Connection 118.42.61.188:62372 v13 “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36” / 101
2022-04-18 07:44:58 [241,313ms] [Warning] [omni.livestream-websocket.plugin] Opening connection: “54.180.150.203” (2 total)
2022-04-18 07:44:59 [241,703ms] [Warning] [omni.livestream-websocket.plugin]
NVENC: found 1 CUDA device(s):

---

active 0 Tesla T4

---

2022-04-18 07:44:59 [241,773ms] [Warning] [omni.livestream-websocket.plugin] Encode settings:
2022-04-18 07:44:59 [241,773ms] [Warning] [omni.livestream-websocket.plugin] bitrate = 32777216
2022-04-18 07:44:59 [241,773ms] [Warning] [omni.livestream-websocket.plugin] iframe_interval = 1
[2022-04-18 07:44:59] [connect] WebSocket Connection 118.42.61.188:62373 v13 “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36” / 101
2022-04-18 07:44:59 [241,796ms] [Warning] [omni.livestream-websocket.plugin] Opening connection: “54.180.150.203” (3 total)
2022-04-18 07:44:59 [241,937ms] [Warning] [omni.livestream-websocket.plugin]
NVENC: found 1 CUDA device(s):

---

active 0 Tesla T4

---

Symptom 2
The error below is repeated when the WebSocket is disconnected abnormally:

2022-04-18 08:24:40 [1,440,394ms] [Error] [omni.livestream-websocket.plugin] Error while sending buffer: invalid state

Thanks.

[bigbellmercy]

Hi, I report the current status of the problems.

Currently, there is no slowness and no flickering though I didn’t do anything.
The problems disappeared.

So, I suspect that the AWS EC2 service was unstable in my first a few days usage for that service for the first time.

Even in g4dn.xlarge (4 vCPU) virtual machine, the Isaac Example, RoboParty works well though g4dn.4xlarge (16 vCPU) was recommended by Nvidia.

So, if somebody undergo the slowness and flicking symptoms in AWS EC2 service, follow the method in the post above (written again below):

@bigbellmercy

But, after removing the ‘–rm’ option and executing the command as below,

$ sudo docker run --name isaac-sim --entrypoint bash -it --gpus all -e “ACCEPT_EULA=Y” --network=host
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw
-v ~/docker/isaac-sim/config:/root/.nvidia-omniverse/config:rw
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw
-v ~/docker/isaac-sim/documents:/root/Documents:rw
nvcr.io/nvidia/isaac-sim:2021.2.1

by exiting the first slow running container of Isaac Sim and running it again, its performance is so improved that we can see the Isaac Example for a robot works but it’s a little slow.

And if not resolved, then ask AWS service team or wait a few days for AWS to be stable in your first use of the service.

Bye~