Installation gets hung up on 'Checking for Isaac on nucleus servers' on aws

Hello Everyone,
So I want to use isaac sim remotely using the remote client and an installation on aws.

I have followed this guide:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/setup.html#cloud-deployment

Everything worked fine until the installation of isaac sim onto the aws instance (g4dn.xlarge).
I have entered the following command to install isaac sim:
sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.1.1 as instructed by the documentation.

And after about two minutes it gets hung up and the last line reads: “Checking for Isaac Sim assets on nucleus”.
I have waited for about two and a half hours and nothing happened since.
There are no errors in the console either.

On the AWS console I can see, that the cpu runs at 50% and there is no network or disk activity going on.

Any Ideas what I might have done wrong?
(I have tried it three times now)

Thank you very much in Advance.

Hi. Please share the full log file. There might clues in it to why it hangs.

log-isaac-aws-install.txt (213.1 KB)

Yes sure, I have attached it. This log is from a rerun from right now, but the issue is the same.

Thank you for taking a look at this

The log looks OK. Have you tried connecting to Isaac Sim using a browser (since it is in WebSocket mode).

Next, you should install the Nucleus server.

Thank you very much Sheikh
I have tried connecting to the Isaac remote kit, but that did not work.
I have not tried connecting to Isaac sim in the browser, because I did not know that this was possible. (The Websocket worked. Awesome!)

I thought that the installation did not work, because the terminal was unresponsive afterwards (ctrl + c did not work) and when I tried to install the nucleus server. I got this error message:

Error: Could not find file '/home/ubuntu/.nvidia-omniverse/config/omniverse.toml'.
AppPath /home/ubuntu/nucleus_installer/setup
ConfigPath /home/ubuntu/nucleus_installer/setup/configs
RootPath /home/ubuntu/nucleus_installer
System Monitor not found

Thank you very much for taking a look at this

That message should be fine.

Cool! Next, try to connect to localhost and upload the assets.

Currently, the app cannot be exited with CTRL-C on the container. To exit it, use sudo docker kill <container_id>.

Okay, Thank you so much Sheikh

1 Like

I’m running it thru AWS. Any idea why the red button on 8211/streaming/client doesn’t do anything after I clicked on it? The interface is just the red button and the stop button, unlike in the video under user manual here: Livestream WebSocket — Omniverse Extensions documentation (nvidia.com)

Hi, please share your full log. The live-streaming will once Isaac Sim is fully loaded. Look for:

Isaac Sim Headless Websocket App is loaded.

2021-07-29 21:25:09 [17,402ms] [Info] [omni.ui] Added dataSource connection for prefix ‘/’ 2021-07-29 21:25:09 [17,449ms] [Info] [carb.python] Started server process [9] 2021-07-29 21:25:10 [18,926ms] [Info] [rtx.neuraylib.plugin] MDL SDK: Loading module “::MdlStates” from “file:/isaac-sim/kit/mdl/rtx/MdlStates.mdl”. 2021-07-29 21:25:10 [18,926ms] [Info] [rtx.neuraylib.plugin] Loaded MdlModule to DB (rtx_scope): ::MdlStates 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Streaming::Streaming((3)) 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Failed::CompileFailed((4)) 2021-07-29 21:25:10 [18,927ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*MdlStates-Deprecated::Deprecated((5)) 2021-07-29 21:25:10 [18,942ms] [Info] [rtx.neuraylib.plugin] MDL SDK: Loading module “::UsdPreviewSurface” from “file:/isaac-sim/kit/mdl/rtx/UsdPreviewSurface.mdl”. 2021-07-29 21:25:10 [18,942ms] [Info] [rtx.neuraylib.plugin] Loaded MdlModule to DB (rtx_scope): ::UsdPreviewSurface 2021-07-29 21:25:10 [18,943ms] [Info] [rtx.neuraylib.plugin] Stored MdlEntitySnapshot to DB: usd::rtx_scope::*UsdPreviewSurface::UsdPreviewSurface((6)) 2021-07-29 21:25:11 [19,485ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,485ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,486ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,486ms] [Info] [carb.settings.python] ISettings.get(…) was called on element that has no value (from python). Use set_default(…) before to set default value. (path = ‘ext/omni.kit.widget.stage/exclusion/types’) 2021-07-29 21:25:11 [19,801ms] [Info] [omni.isaac.dynamic_control.plugin] Acquired interface ‘omni::physx::IPhysx’, version 1.0 2021-07-29 21:25:11 [19,801ms] [Info] [omni.isaac.dynamic_control.plugin] Acquired interface ‘omni::physx::IPhysxSceneQuery’, version 1.0 2021-07-29 21:25:11 [19,805ms] [Info] [carb.graphics-vulkan.plugin] Active GPU: Tesla T4 2021-07-29 21:25:11 [19,882ms] [Info] [carb] Initializing plugin: omni.gpucompute-d3dvk.plugin (interfaces: [omni::gpucompute::ComputeCompiler v0.1],[omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-d3dvk.plugin) 2021-07-29 21:25:11 [19,882ms] [Info] [carb] Initializing plugin: omni.gpucompute-cuda.plugin (interfaces: [omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-cuda.plugin) 2021-07-29 21:25:11 [19,882mIsaac Sim Headless Websocket App is loaded. Checking for Isaac Sim assets on nucleus server

That looks Ok.
Which OS and browser are you running on?
Try enter CTRL+SHIFT+I if you are on Chrome browser to open the browser console. Please post any errors you see there.

Failed to load resource: the server responded with a status of 404 (Not Found)
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:242 [view 0]: resize: 1228 943
client.js:242 [view 0]: {“command”:“video_resize”,“video_resize”:{“video_width”:1228,“video_height”:943}}
DevTools failed to load source map: Could not load content for http://3.92.30.20:8211/static/css/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899
client.js:39 [model]: document is now hidden; disconnecting
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
(anonymous) @ client.js:101
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:109 WebSocket connection to ‘ws://3.92.30.20:8899/’ failed:
connectToServer @ client.js:109
(anonymous) @ client.js:101
client.js:242 [view 0]: error: undefined
client.js:242 [view 0]: disconnected
client.js:39 [model]: document is now visible; reconnecting
client.js:39 [model]: Connecting to: ws://3.92.30.20:8899

Update: I was guessing it’s the inbound rule in my AWS that refused the connection, so I modified my port range. Now everything seems fine.

1 Like