Installation gets hung up on 'Checking for Isaac on nucleus servers' on aws

schmiJo · July 23, 2021, 3:01pm

Hello Everyone,
So I want to use isaac sim remotely using the remote client and an installation on aws.

I have followed this guide:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/setup.html#cloud-deployment

Everything worked fine until the installation of isaac sim onto the aws instance (g4dn.xlarge).
I have entered the following command to install isaac sim:
sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.1.1 as instructed by the documentation.

And after about two minutes it gets hung up and the last line reads: “Checking for Isaac Sim assets on nucleus”.
I have waited for about two and a half hours and nothing happened since.
There are no errors in the console either.

On the AWS console I can see, that the cpu runs at 50% and there is no network or disk activity going on.

Any Ideas what I might have done wrong?
(I have tried it three times now)

Thank you very much in Advance.

Sheikh_Dawood · July 23, 2021, 4:19pm

Hi. Please share the full log file. There might clues in it to why it hangs.

schmiJo · July 23, 2021, 4:47pm

log-isaac-aws-install.txt (213.1 KB)

Yes sure, I have attached it. This log is from a rerun from right now, but the issue is the same.

Thank you for taking a look at this

Sheikh_Dawood · July 23, 2021, 4:59pm

The log looks OK. Have you tried connecting to Isaac Sim using a browser (since it is in WebSocket mode).

Next, you should install the Nucleus server.

schmiJo · July 23, 2021, 6:04pm

Thank you very much Sheikh
I have tried connecting to the Isaac remote kit, but that did not work.
I have not tried connecting to Isaac sim in the browser, because I did not know that this was possible. (The Websocket worked. Awesome!)

I thought that the installation did not work, because the terminal was unresponsive afterwards (ctrl + c did not work) and when I tried to install the nucleus server. I got this error message:

Error: Could not find file '/home/ubuntu/.nvidia-omniverse/config/omniverse.toml'.
AppPath /home/ubuntu/nucleus_installer/setup
ConfigPath /home/ubuntu/nucleus_installer/setup/configs
RootPath /home/ubuntu/nucleus_installer
System Monitor not found

Thank you very much for taking a look at this

Sheikh_Dawood · July 23, 2021, 6:09pm

That message should be fine.

Cool! Next, try to connect to localhost and upload the assets.

Currently, the app cannot be exited with CTRL-C on the container. To exit it, use sudo docker kill <container_id>.

schmiJo · July 23, 2021, 7:30pm

Okay, Thank you so much Sheikh

john.a.sun · July 29, 2021, 11:36pm

I’m running it thru AWS. Any idea why the red button on 8211/streaming/client doesn’t do anything after I clicked on it? The interface is just the red button and the stop button, unlike in the video under user manual here: Livestream WebSocket — Omniverse Extensions documentation (nvidia.com)

Sheikh_Dawood · July 29, 2021, 11:59pm

Hi, please share your full log. The live-streaming will once Isaac Sim is fully loaded. Look for:

Isaac Sim Headless Websocket App is loaded.

john.a.sun · July 30, 2021, 12:02am

2021-07-29 21:25:09 [17,402ms] [Info] [omni.ui] Added dataSource connection for prefix ‘/’ 2021-07-29 21:25:09 [17,449ms] [Info] [carb.python] Started server process [9] 2021-07-29 21:25:10 [18,926ms] [Info] [rtx.neuraylib.plugin] MDL SDK: Loading module “::MdlStates” from “file:/isaac-sim/kit/mdl/rtx/MdlStates.mdl”. 2021-07-29 21:25:10 [18,926ms] [Info] [rtx.neuraylib.plugin] Loaded MdlModule to DB (rtx_scope): ::MdlStates 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Streaming::Streaming((3)) 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Failed::CompileFailed((4)) 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Deprecated::Deprecated((5)) 2021-07-29 21:25:10 [18,942ms] [Info] [rtx.neuraylib.plugin] MDL SDK: Loading module “::UsdPreviewSurface” from “file:/isaac-sim/kit/mdl/rtx/UsdPreviewSurface.mdl”. 2021-07-29 21:25:10 [18,942ms] [Info] [rtx.neuraylib.plugin] Loaded MdlModule to DB (rtx_scope): ::UsdPreviewSurface 2021-07-29 21:25:10 [18,943ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*UsdPreviewSurface::UsdPreviewSurface((6)) 2021-07-29 21:25:11 [19,485ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,485ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,486ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,486ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,801ms] [Info] [omni.isaac.dynamic_control.plugin] Acquired interface ‘omni::physx::IPhysx’, version 1.0 2021-07-29 21:25:11 [19,801ms] [Info] [omni.isaac.dynamic_control.plugin] Acquired interface ‘omni::physx::IPhysxSceneQuery’, version 1.0 2021-07-29 21:25:11 [19,805ms] [Info] [carb.graphics-vulkan.plugin] Active GPU: Tesla T4 2021-07-29 21:25:11 [19,882ms] [Info] [carb] Initializing plugin: omni.gpucompute-d3dvk.plugin (interfaces: [omni::gpucompute::ComputeCompiler v0.1],[omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-d3dvk.plugin) 2021-07-29 21:25:11 [19,882ms] [Info] [carb] Initializing plugin: omni.gpucompute-cuda.plugin (interfaces: [omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-cuda.plugin) 2021-07-29 21:25:11 [19,882mIsaac Sim Headless Websocket App is loaded. Checking for Isaac Sim assets on nucleus server

Sheikh_Dawood · July 31, 2021, 3:48am

That looks Ok.
Which OS and browser are you running on?
Try enter CTRL+SHIFT+I if you are on Chrome browser to open the browser console. Please post any errors you see there.

john.a.sun · August 2, 2021, 5:59pm

Failed to load resource: the server responded with a status of 404 (Not Found)
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:242 [view 0]: resize: 1228 943
client.js:242 [view 0]: {“command”:“video_resize”,“video_resize”:{“video_width”:1228,“video_height”:943}}
DevTools failed to load source map: Could not load content for http://3.92.30.20:8211/static/css/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
(anonymous) @ client.js:101
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
(anonymous) @ client.js:101
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899

john.a.sun · August 2, 2021, 6:32pm

Update: I was guessing it’s the inbound rule in my AWS that refused the connection, so I modified my port range. Now everything seems fine.

system · October 1, 2021, 6:33pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.