Isaac Sim (2021.1.1) Wont Launch (Omniverse 1.3.4)

When I click “Launch” in Omniverse for Isaac sim it 1/2 opens a window and then crashes immediately. (I am assuming it crashes, there is no error displayed it just doesn’t open anything.)

Details about my installation:

I have installed Omniverse (1.3.4) and set up a local Nucleus Service (2021.2.5). I used the Omniverse Echange to install Isaac Sim (2021.1.1).

I am using the Ubuntu 18.04 NVIDIA RTX image on EC2 with the g4dn.xlarge instance. I have installed a vnc server to support the GUI. It is set up with 500 extra GB of storage.

I was looking at this thread but running ./isaac-sim.sh -v also results in an error.

2021-11-15 19:56:55 [5,229ms] [Info] [omni.client.plugin]  Tick: provider_nucleus: Connected to localhost:3009
2021-11-15 19:56:55 [5,230ms] [Warning] [carb.python] default server omniverse://localhost does not have /Isaac folder
2021-11-15 19:56:55 [5,230ms] [Warning] [carb.python] /isaac/nucleus/default not specified in json config or via --/isaac/nucleus/default="omniverse://my-nucleus-server" command line
2021-11-15 19:56:55 [5,231ms] [Warning] [carb.python] Attempting to locate server from previously saved servers...
2021-11-15 19:56:55 [5,231ms] [Info] [carb.settings.python] ISettings.get(...) was called on element that has no value (from python). Use set_default(...) before to set default value. (path = '/persistent/app/omniverse/savedServers')
2021-11-15 19:56:55 [5,231ms] [Warning] [carb.python] /persistent/app/omniverse/savedServers setting not found
2021-11-15 19:56:55 [5,231ms] [Warning] [carb.python] /persistent/app/omniverse/mountedDrives setting not found
2021-11-15 19:56:55 [5,231ms] [Warning] [carb.python] No saved servers
2021-11-15 19:56:55 [5,231ms] [Error] [carb.python] Could not find nucleus server with /Isaac folder
.
.
.
2021-11-15 19:56:56 [6,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-11-15 19:56:56 [6,503ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-11-15 19:56:56 [6,506ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-11-15 19:56:56 [6,510ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2021-11-15 19:56:56 [6,515ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2021-11-15 19:56:56 [6,520ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<omni::kit::RunLoopThread::run()::{lambda()#1}> > >::~_State_impl()
2021-11-15 19:56:56 [6,525ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2021-11-15 19:56:56 [6,530ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!carbOnPluginPostShutdown
2021-11-15 19:56:56 [6,531ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
2021-11-15 19:56:56 [6,536ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2021-11-15 19:56:56 [6,536ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
Floating point exception (core dumped)

Is there a way to create this missing Isaac folder manually?

Hi, here are the instructions to add the /Isaac folder.
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/setup.html#adding-sample-assets

Does Create or other Omniverse apps works?

Unfortunately I cannot run Isaac Sim at all so I cant follow the instructions you linked. I have not tried to run a different app. I will install and try Create now.

Thank you for your swift reply!

The issue is not specific to Isaac Sim. When I hit launch on Create it also does nothing. I am assuming this is a crash

Yes, Vulkan is not supported over VNC without a physical display. The recommended way to run on an EC2 instance is with Isaac Sim headless mode and livestream using WebSocket.

1 Like

Should Isaac Sim (Headless WebSocket) be installed using the Omniverse launcher? I do not see it pop up in the Exchange tab when I search for Isaac.

It seems like it is supposed to be selected in this Launcher window, but the window never loads and crashes immediately.

When you run Isaac Sim for the first time, an Isaac Sim App Launcher should pop up and you can select Websocket mode. This may or may not work.

It is recommended to run Isaac Sim using containers on EC2. See Native Workstation Deployment — Omniverse Robotics documentation

1 Like

I have restarted my install process following these instructions from NVIDIA..

The EC2 instance is using this community AMI as suggested in the above documentation: IsaacSim-Ubuntu-18.04-GPU-2021-05-25

This has the same hardware setup as the original.

Unfortunately when I try to install & run the docker image it hangs. Here is the command that it fails on:

sudo docker run --gpus all -e “ACCEPT_EULA=Y” --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.1.1

Hang Location:

2021-11-16 20:39:24 [101,365ms] [Info] [carb.python] Handle changed 2 prims for layer anon:0xd093630:World0.usd.
2021-11-16 20:39:24 [101,366ms] [Info] [carb.python] Load prim spec tree rooted from /World
2021-11-16 20:39:24 [101,366ms] [Info] [carb.python] Handle changed prims for layer anon:0xd093630:World0.usd done.
2021-11-16 20:39:24 [101,366ms] [Info] [carb.python] Handle changed 4 prims for layer aIsaac Sim Headless Websocket App is loaded.
Checking for Isaac Sim assets on nucleus server

Should I have installed anything else on the instance before the Isaac container? There is nothing else currently on it.

Looks like the app is loaded but may hang when looking for the assets on Nucleus.

You can install Nucleus on the EC2 instance. Native Workstation Deployment — Omniverse Robotics documentation

I have followed these directions. I have uploaded the nucleus .zip file via scp and then tried to install via the
setup/nucleus-setup -i step in the instructions. It failed the first time with the error so I re-ran it and (I think) it succeeded.

ubuntu@ip:~/nucleus_installer$ setup/nucleus-setup -i
install
Error: Could not find file '/home/ubuntu/.nvidia-omniverse/config/omniverse.toml'.
AppPath /home/ubuntu/nucleus_installer/setup
ConfigPath /home/ubuntu/nucleus_installer/setup/configs
RootPath /home/ubuntu/nucleus_installer
System Monitor not found
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Discovery Service
Killing Process - omni-discovery-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Discovery Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Discovery Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Discovery Service
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Thumbnail
Killing Process - omni-thumbnails-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Thumbnail/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Thumbnail/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Thumbnail
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Search Service
Killing Process - omni-search-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Search Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Search Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Search Service
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Snapshot
Killing Process - omni-snapshot-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Snapshot/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Snapshot/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Snapshot
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Auth
Killing Process - omni-auth-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Auth/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Auth/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Auth
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Web
Killing Process - omni-web-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Web/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Web/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Web
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/System Monitor
Killing Process - omni-system-monitor
Copying  - /home/ubuntu/nucleus_installer/setup/configs/System Monitor/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/System Monitor/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/System Monitor
config.toml found, copying to /home/ubuntu/.nvidia-omniverse/config/System Monitor/config.toml
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Nucleus
Killing Process - omni.server.app
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Nucleus/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Nucleus/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Nucleus
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Tagging Service
Killing Process - omni-tagging-service
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Tagging Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Tagging Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Tagging Service
Starting System Monitor: /home/ubuntu/nucleus_installer/System Monitor/omni-system-monitor

When the nucleus install succeeded it produced this result:

ubuntu@ip:~/nucleus_installer$ setup/nucleus-setup -i
install
AppPath /home/ubuntu/nucleus_installer/setup
ConfigPath /home/ubuntu/nucleus_installer/setup/configs
RootPath /home/ubuntu/nucleus_installer
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Discovery Service
Killing Process - omni-discovery-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Discovery Service/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Discovery Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Discovery Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Discovery Service
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Thumbnail
Killing Process - omni-thumbnails-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Thumbnail/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Thumbnail/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Thumbnail/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Thumbnail
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Search Service
Killing Process - omni-search-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Search Service/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Search Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Search Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Search Service
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Snapshot
Killing Process - omni-snapshot-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Snapshot/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Snapshot/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Snapshot/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Snapshot
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Auth
Killing Process - omni-auth-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Auth/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Auth/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Auth/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Auth
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Web
Killing Process - omni-web-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Web/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Web/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Web/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Web
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/System Monitor
Killing Process - omni-system-monitor
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/System Monitor/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/System Monitor/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/System Monitor/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/System Monitor
config.toml found, copying to /home/ubuntu/.nvidia-omniverse/config/System Monitor/config.toml
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Nucleus
Killing Process - omni.server.app
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Nucleus/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Nucleus/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Nucleus/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Nucleus
UpdateComponentToml - /home/ubuntu/nucleus_installer/setup/configs/Tagging Service
Killing Process - omni-tagging-service
Deleting existing - /home/ubuntu/.nvidia-omniverse/config/Tagging Service/Omniverse.toml
Copying  - /home/ubuntu/nucleus_installer/setup/configs/Tagging Service/Omniverse.toml to /home/ubuntu/.nvidia-omniverse/config/Tagging Service/Omniverse.toml
Setting InstallDir - /home/ubuntu/nucleus_installer/Tagging Service
Starting System Monitor: /home/ubuntu/nucleus_installer/System Monitor/omni-system-monitor

Unfortunately, when I run the original Headless Isaac installation docker command I get this error now:

ubuntu@ip:~$ sudo docker login nvcr.io
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
ubuntu@:~$ sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.1.1
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
ubuntu@ip:~$ sudo docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Next I thoroughly deleted the half built docker image and container just to run it again and get the same error:

a2016a6ce947: Pull complete                                                                                                                                                                                                    0e566ff3e71c: Pull complete                                                                                                                                                                                                    Digest: sha256:1d4bc215354ca1e47223bad3d4323b8d19f3867d9a18abb332d37f838ab367f2
Status: Downloaded newer image for nvcr.io/nvidia/isaac-sim:2021.1.1
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.

There may be an issue with the drivers. This may happen after an update. Please try running nvidia-smi to check if the drivers are working.
You can reinstall the drivers if nvidia-smi failes. Download the .run drivers via wget.

For others who run into this issue:
I started from scratch and discovered that before running setup/nucleus-setup -i there are nvidia drivers and no issues. After running that command there nvidia-smi doesnt work because it cant find nvidia.

ubuntu@ip-10-85-153-189:~/nucleus_installer$ sudo nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

So then I ran sudo apt install nvidia-driver-460 since that is what the instance had originally. Then I re-ran the docker command. Now it is sitting on a new line:

2021-11-18 15:50:20 [129,782ms] [Info] [carb.python] [py stdout]: Checking for Isaac Sim assets on nucleus server

2021-11-18 15:50:20 [129,795ms] [Warning] [carb.python] default server omniverse://localhost does not have /Isaac folder
2021-11-18 15:50:20 [129,795ms] [Warning] [carb.python] /isaac/nucleus/default not specified in json config or via --/isaac/nucleus/default="omniverse://my-nucleus-server" command line
2021-11-18 15:50:20 [129,795ms] [Warning] [carb.python] Attempting to locate server from previously saved servers...
2021-11-18 15:50:20 [129,795ms] [Info] [carb.settings.python] ISettings.get(...) was called on element that has no value (from python). Use set_default(...) before to set default value. (path = '/persistent/app/omniverse/savedServers')
2021-11-18 15:50:20 [129,795ms] [Warning] [carb.python] /persistent/app/omniverse/savedServers setting not found
2021-11-18 15:50:20 [129,795ms] [Info] [carb.python] Testing omniverse://localhost Server for /Isaac folder
2021-11-18 15:50:20 [129,795ms] [Warning] [carb.python] Server omniverse://localhost does not have /Isaac folder
2

Was I supposed to pre-configure nucleus somehow?

Hi @altimm doesn’t seem that the instance is crashing anymore right?
You should be able to access it now via one of the Livestream Clients. If you followed the default instructions you are likely using the Websocket

The logs also indicate that you have not added the Isaac asset folder. Did you do that after you install Nucleus? Check instructions here Adding Sample Assets to Headless Nucleus

I had not followed those instructions when I made original post. Since then, I did find and follow those instructions but I get an error. Unfortunately I did not take a screenshot.

Was this nucleus issue solved in Isaac 2021.2.0? I noticed that it listed these services as coming pre-installed.

I ran the new 2021.2.0 docker file using this command:

sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.2.0

It has stopped on this output:

A few errors and warnings occurred before this so I am unsure if this is an intended outcome or something is broken. Trying to connect to the websocket in my local browser does not work but I expect that since this is an EC2 instance I have some networking/permissions issues to work out before that works.

In conclusion: Is Omniverse System Monitor with Nucleus and Cache started a success state or an intermediate step?

Hi, that output looks correct. Isaac Sim should start after that line. I would recommend following the steps here: 3. Advanced Installation — Omniverse Robotics documentation

See 6. Setup FAQ — Omniverse Robotics documentation if you have issues with starting Nucleus.

@altimm you can check if your Nucleus and Cache are running by going to http://<ip_address_of_instance>:8080 (Nucleus) or http://<ip_address_of_instance>:3080 (see all services that are running, including Cache)

After Nucleus is setup you can proceed to run Isaac SIM:

./isaac-sim.headless.websocket.sh --allow-root

It should startup in headless mode. Once it is ready it will output something like “Isaac Sim Headless Websocket App is loaded.”

After this step, you can connect via browser on http://<ip_address_of_instance>:8211/streaming/client

I have changed my version to isaac-sim:2021.2.0 and am still trying to implement the websocket setup that was described above.

To this point on the EC2 instance I have:

  1. Updated the Driver to from 460 to 470: sudo apt install nvidia-driver-470
  2. Rebooted
  3. Logged into docker with NGC API Key
  4. Built the docker container and got the sucessful " Isaac Sim Headless Websocket App is loaded" message
> sudo docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
> -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
> -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
> -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
> -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
> -v ~/docker/isaac-sim/config:/root/.nvidia-omniverse/config:rw \
> -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
> -v ~/docker/isaac-sim/documents:/root/Documents:rw \
> nvcr.io/nvidia/isaac-sim:2021.2.0
  1. Open bash in the docker container sudo docker exec -it <containter name> /bin/bash
  2. Start nucleus: ./start_nucleus.sh
  3. Start Isaac Sim: ./isaac-sim.headless.websocket.sh --allow-root

This all works as expected and as described in the Advanced Installation tutorial.

output1

Unfortunately, when visiting http://IP:8080it did not work as expected. The connection was partially successful as there was an NVIDIA Window displayed. However, the connection to Nucleus failed.

failed to connect

Additionally, when trying to access http://<IP>:8211/streaming/client to view Isaac Sim itself the window with the red play button appeared, but clicking it did nothing.

red play button

Are there some logs somewhere that I can dig into to find the root cause?

You can take a look at the Nucleus logs for any errors at ~/docker/isaac-sim/logs/Nucleus/omni.server.app.log

Please also try manually starting up Nucleus instead of the start_nucleus.sh script.
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/install_faq.html#manual-install-nucleus-in-container

Try running just the Nucleus services first to get port 8080 running, then run Isaac Sim.

I followed the instructions for manually starting nucleus. However, it is sitting on the “Listen” line. When checking the logs you mentioned, I did see that there were connection refused errors when running it the ./start_nucleus.sh way. (This error message is included below). When I ran with the manual nucleus instructions I didn’t see any errors appear in the logs, but I am getting the same error when trying to acces the webpage.

I have opened access through port 8080 and 8211. This line in the manual build makes me think I may need to open 3085 and 3080 as well?

:~/docker/isaac-sim/logs/Nucleus$ cat omni.server.app.log
2021-12-13 15:43:42 [0ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread)
2021-12-13 15:43:42 [0ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core)
2021-12-13 15:43:42 [0ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.objects (interfaces: [omni::server::IObjects v1.0]) (impl: omni.objects)
2021-12-13 15:43:42 [0ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.messaging (interfaces: [omni::server::IMessaging v1.0]) (impl: omni.messaging)
2021-12-13 15:43:42 [1ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler)
2021-12-13 15:43:42 [1ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.2]) (impl: carb.launcher.plugin)
2021-12-13 15:43:42 [1ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.resolver (interfaces: [omni::IResolver v5.2]) (impl: omni.resolver)
2021-12-13 15:43:42 [1ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server)
2021-12-13 15:43:42 [2ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v1.6],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin)
2021-12-13 15:43:42 [2ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib)
2021-12-13 15:43:42 [2ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb)
2021-12-13 15:43:42 [2ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v1.0]) (impl: omni.meta-upgrade)
2021-12-13 15:43:42 [3ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.eventlogger (interfaces: [omni::server::IEventLogger v1.0]) (impl: omni.eventlogger)
2021-12-13 15:43:42 [3ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.metrics-prometheus (interfaces: [omni::server::IMetrics v1.0]) (impl: omni.metrics-prometheus)
2021-12-13 15:43:42 [3ms] [Info] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:registerPlugin():351: Registered plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem)
2021-12-13 15:43:42 [4ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.signalhandler (interfaces: [omni::server::ISignalHandler v2.0]) (impl: omni.signalhandler)
2021-12-13 15:43:42 [4ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.wakeupthread (interfaces: [omni::server::IWakeupThread v1.0]) (impl: omni.wakeupthread)
2021-12-13 15:43:42 [4ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.eventsystem (interfaces: [omni::server::IEventSystem v1.0]) (impl: omni.eventsystem)
2021-12-13 15:43:42 [4ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.server (interfaces: [omni::server::IDeltaContentServer v1.0],[omni::server::IServer v1.1]) (impl: omni.server)
2021-12-13 15:43:42 [4ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: carb.tasking.plugin (interfaces: [carb::tasking::ITasking v1.6],[carb::tasking::IThreadPool v1.0]) (impl: carb.tasking.plugin)
2021-12-13 15:43:42 [4ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():220: Launching C++ Omniverse API Server version (1.0.98a7aeec) API version(1.20)
2021-12-13 15:43:42 [4ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniverseServer.cpp:create():222:       Build from commit 98a7aeec on Tue Sep 28 10:48:50 2021 -0700
2021-12-13 15:43:42 [4ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniverseServer.cpp:OmniverseServer():114: Starting up Omniverse Server.
2021-12-13 15:43:42 [5ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniverseServer.cpp:loadSettings():742: Setting Log directory to: /root/.nvidia-omniverse/logs/Nucleus

2021-12-13 15:43:42 [5ms] [Info] [omni.server] [56] ../../../source/plugins/common/ResolvedConfig.h:dumpResolvedConfig():46: Loaded settings:
2021-12-13 15:43:42 [5ms] [Info] [omni.server] [56] ../../../source/plugins/common/ResolvedConfig.h:operator()():42: {
    "LFT": {
        "INTERFACE": "0.0.0.0"
    },
    "log": {
        "fileLogLevel": "verbose",
        "outputStream": "",
        "file": "/root/.nvidia-omniverse/logs/Nucleus/omni.server.app.log",
        "debugConsoleLevel": "verbose",
        "includeFunctionName": true,
        "includeTimeStamp": true,
        "sources": {
            "omni.server.RECORD_LOCKS": {
                "enabled": false
            },
            "omni.server.DELTA_MANAGER": {
                "enabled": false
            },
            "omni.server.API_SERVER": {
                "enabled": false
            },
            "omni.server.SHUTDOWN": {
                "enabled": false
            },
            "omni.server.RECORD_SAVES": {
                "enabled": false
            },
            "omni.server.COPY_API": {
                "enabled": false
            },
            "omni.server.SYMLINK": {
                "enabled": false
            },
            "omni.server.DELTA_CONTENT": {
                "enabled": false
            },
            "omni.server.STATE_MACHINE": {
                "enabled": false
            },
            "omni.server.LOCK_SYSTEM": {
                "enabled": false
            },
            "omni.server.FRAMING": {
                "enabled": false
            },
            "omni.server.MESSAGING_SYSTEM": {
                "enabled": false
            },
            "omni.server.SERVER_ECHO": {
                "enabled": false
            },
            "omni.server.FST": {
                "enabled": false
            },
            "omni.server.DATABASE_API": {
                "enabled": false
            },
            "omni.server.OMNI_OBJECTS": {
                "enabled": false
            },
            "omni.server.KILL_CONTENT": {
                "enabled": false
            },
            "omni.server.HASH_BIN": {
                "enabled": false
            },
            "omni.server.DBCONTENT": {
                "enabled": false
            },
            "omni.server.LIST_EVENTS": {
                "enabled": false
            },
            "omni.server.BENCHMARK": {
                "enabled": false
            },
            "omni.server.USER_ACCOUNTS": {
                "enabled": false
            },
            "omni.server.META_DIRECTORY": {
                "enabled": false
            },
            "omni.server.CONNECTIONS": {
                "enabled": false
            },
            "omni.server.SERVER_SLEEP": {
                "enabled": false
            },
            "omni.server.VALIDATE": {
                "enabled": false
            },
            "omni.server.FOLDER": {
                "enabled": false
            },
            "omni.server.VALIDATE_API": {
                "enabled": false
            },
            "omni.server.PUBSUB": {
                "enabled": false
            },
            "omni.server.EVENT_SYSTEM": {
                "enabled": false
            },
            "omni.server.READ_EVENT": {
                "enabled": false
            },
            "omni.server.API_REFERENCE": {
                "enabled": false
            },
            "omni.server.CONTENT_HASHES": {
                "enabled": false
            },
            "omni.server.CONTENT_MANAGER": {
                "enabled": false
            },
            "omni.server.COPY_MANAGER": {
                "enabled": false
            },
            "omni.server.SNAPSHOT": {
                "enabled": false
            },
            "omni.server.GLOBAL_LOCKS": {
                "enabled": false
            },
            "omni.server.META_CACHE": {
                "enabled": false
            },
            "omni.server.API_CLIENT": {
                "enabled": false
            },
            "omni.server.CONTENT_REF_COUNT": {
                "enabled": false
            }
        },
        "fileFlushLevel": "verbose",
        "includeFilename": true,
        "setElapsedTimeUnits": "milliseconds",
        "enableColorOutput": true,
        "includeLineNumber": true,
        "level": "Info",
        "outputStreamLevel": "verbose",
        "enableDebugConsoleOutput": true,
        "includeThreadId": true,
        "enableStandardStreamOutput": true,
        "enabled": true,
        "async": false,
        "fullDetail": false,
        "fileAppend": true,
        "flushStandardStreamOutput": false,
        "processGroupId": 0,
        "includeSource": true,
        "includeChannel": true,
        "includeProcessId": false,
        "detail": false
    },
    "APILOGGING": 1,
    "crashreporter": {
        "dumpDir": ".",
        "alwaysUpload": false,
        "data": {
            "buildBranch": "master",
            "buildNumber": "18991",
            "buildMr": "0",
            "buildHash": "98a7aeec",
            "buildVersion": "111.5.0",
            "buildMinor": "5",
            "buildMajor": "111",
            "buildPatch": "0",
            "buildCi": "gitlab"
        },
        "version": "111.5+master.gitlab.18991.98a7aeec",
        "enabled": "true",
        "product": "OmniverseNucleus",
        "log": "true",
        "url": "https://services.nvidia.com/submit"
    },
    "plugins": {
        "omni.metrics-file": {
            "dumpPath": "metrics.csv",
            "flushTimeInSeconds": 5
        },
        "omni.server": {
            "metadata": {
                "plugin": "omni.keyvaluedatabase.leveldb",
                "options": {
                    "async": true,
                    "root": "",
                    "subdir": "meta.1.1",
                    "delete_on_startup": "false"
                }
            },
            "run": {
                "db_content": true,
                "copy_manager": true,
                "delta_content_server": true,
                "delta_manager": true,
                "versioning": "true"
            },
            "delta_manager": {
                "ignorelist_retry_delay_ms": 5000,
                "num_retries_until_ignored": 10,
                "delta_apply_limit_mb": 128,
                "ping_timeout_ms": 1000,
                "delta_apply_throttle_ms": 500,
                "delta_apply_timeout_ms": 5000,
                "reconnect_timeout_s": 20,
                "ping_interval_on_startup_ms": 100,
                "bw_estimate_bps": 256,
                "ping_interval_ms": 1000
            },
            "removed_content_pending_time_sec": 5,
            "endpoints": {
                "api": [
                    {
                        "interface": "0.0.0.0",
                        "lft_address": "http://*:3030",
                        "port": "3009",
                        "secure": false
                    }
                ],
                "lft": [
                    {
                        "interface": "127.0.0.1",
                        "port": "3007",
                        "secure": false
                    }
                ],
                "service": [
                    {
                        "interface": "127.0.0.1",
                        "port": "3006",
                        "secure": false
                    }
                ]
            },
            "allowed_users": {},
            "default_folders": [
                [
                    "/",
                    {
                        "users": "r",
                        "gm": "rwa"
                    }
                ],
                [
                    "/Users/",
                    {
                        "users": "r",
                        "gm": "rwa"
                    }
                ],
                [
                    "/Projects/",
                    {
                        "users": "rw",
                        "gm": "rwa"
                    }
                ],
                [
                    "/Library/",
                    {
                        "users": "rw",
                        "gm": "rwa"
                    }
                ]
            ],
            "allowed_users_legacy": {},
            "max_full_path_length": "16384",
            "auth": {
                "connection_id_salt": "7b6a5f4e0a1b2c3d",
                "jwt_rs256_public_key": "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJFEw8/EqME3ZB1dGk/fvIP6UTzrWA7d\ny3dLlSYkTI0IO7rhia55RVsuxVLuQ4QO1jsC6YF/jw8hqMdyXQkCetUCAwEAAQ==\n-----END PUBLIC KEY-----"
            },
            "mounts": {
                "default_mounts": [
                    {
                        "path": "/NVIDIA",
                        "options": "{\"timeout\": 600, \"proxy\":\"localhost:8891\", \"host\":\"dcb18d6mfegct.cloudfront.net\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dcb18d6mfegct.cloudfront.net/\"}",
                        "resolver": "omniverse_resolver_s3",
                        "override": true
                    },
                    {
                        "path": "/Isaac",
                        "options": "{\"timeout\": 600, \"proxy\":\"localhost:8891\", \"host\":\"ov-isaac.s3.us-west-1.amazonaws.com\",\"service\":\"s3\",\"secure\":false,\"redirection\":\"http://dpcil1p2xgdf8.cloudfront.net/\"}",
                        "resolver": "omniverse_resolver_s3",
                        "override": true
                    }
                ],
                "allowed_mount_resolvers": [
                    "omniverse_resolver_file",
                    "omniverse_resolver_s3"
                ],
                "allowed_file_resolver_roots": {}
            },
            "content_kv_database": false,
            "disable_passwords": false,
            "content_flush_enabled": true,
            "force_sync": false,
            "delta_content_bw_estimate": 0.5,
            "content_flush_time_sec": 10,
            "check_permissions": true,
            "peer_messaging": {
                "channel": "omni_server_messages",
                "enable": "false",
                "application_name": "Nucleus API",
                "exit_on_connection_failure": "false",
                "payloads": {
                    "create_event": {
                        "name": "create"
                    },
                    "list_event": {
                        "name": "list"
                    },
                    "servernotification_event": {
                        "name": "servernotification"
                    }
                }
            },
            "console_enabled": true,
            "discovery": {
                "port_based_default_port": 3333,
                "deployment_string": "{ \"reg\": [ { \"name\": \"external\", \"transport\": { \"type\": \"connlib\", \"params\": { \"url\": \"ws://127.0.0.1:3009\" } } } ] }",
                "enabled": "true",
                "token": "test",
                "host": "localhost"
            },
            "master_password": "omniverse",
            "omniobjects": {
                "content_mem_limit": 268435456,
                "mem_vs_file_threshold": 262144,
                "max_objects_in_cache": 1024,
                "options": {
                    "async": true,
                    "root": "",
                    "subdir": "omniobjects.1.0",
                    "delete_on_startup": "false"
                },
                "max_content_file_cache_size": 1048576000,
                "max_content_mem_cache_size": 104857600,
                "unpack_object_buffer_size": 33554432,
                "plugin": "omni.keyvaluedatabase.leveldb",
                "lft_read_timeout_seconds": 3600,
                "lft_multi_content_threshold": 0,
                "ext_value_min_size": 262144
            },
            "omniobjects.cache": {
                "resolver_kind": "omniverse_resolver_file",
                "resolver_options": {
                    "async": true,
                    "root": ""
                }
            },
            "api_acls": {
                "unlock": "w",
                "lock": "w",
                "rename": "a",
                "set_path_options": "w",
                "set_acl_v2": "a",
                "delete": "a",
                "read": "r",
                "mount": "w",
                "copy": "w",
                "create_object": "w",
                "list": "r",
                "create_asset": "w",
                "get_checkpoints": "r",
                "update": "w",
                "get_acl_v2": "r",
                "create_directory": "w",
                "unmount": "w",
                "checkpoint_version": "w"
            },
            "omniobjects.values": {
                "resolver_kind": "omniverse_resolver_file",
                "resolver_options": {
                    "async": true,
                    "root": ""
                }
            },
            "content_resolver": {
                "delete_on_startup": "false",
                "resolver": "omniverse_resolver_composite",
                "options": {
                    "total_size_high": 6442450944,
                    "total_size_low": 4294967296,
                    "number_elems_high": 120000,
                    "persistent_options": {
                        "async": true,
                        "root": ""
                    },
                    "persistent_kind": "omniverse_resolver_file",
                    "temporal_options": {},
                    "number_elems_low": 100000,
                    "elem_size_max": 1048576,
                    "temporal_kind": "omniverse_resolver_memory"
                }
            },
            "store_crash_repro": true,
            "stream_copy_size_mb": 32,
            "file_and_username_filter_exclude_regex": "",
            "timeouts": {
                "create": 600,
                "update": 600,
                "default": 10,
                "list": 120,
                "mount": 60,
                "read": 600
            },
            "default_acls": {
                "owner": "rwa",
                "rest": {
                    "gm": "rwa"
                }
            },
            "home_folder_default_acls": {
                "owner": "rwa",
                "rest": {
                    "users": "",
                    "gm": "rwa"
                }
            },
            "dirs": {
                "omniobjects_dir": "/root/.local/share/ov/data/server",
                "usergroups_dir": "/root/.local/share/ov/data/server",
                "log_dir": "/root/.nvidia-omniverse/logs/Nucleus",
                "metric_dir": "/root/.local/share/ov/data/server",
                "content_dir": "/root/.local/share/ov/data/server",
                "record_dir": "",
                "meta_dir": "/root/.local/share/ov/data/server"
            },
            "password_recovery": true,
            "non_utf8_paths_allowed": "false",
            "admin_folders": [
                "/Users/omniverse/",
                "/Users/deeptag_service/",
                "/Users/search_service/",
                "/Users/tags_service/",
                "/Users/thumbnails_service/",
                "/Users/index/",
                "/.system/"
            ],
            "test_token_prefix": "",
            "usergroups": {
                "plugin": "omni.keyvaluedatabase.leveldb",
                "options": {
                    "async": true,
                    "root": "",
                    "subdir": "usergroups.1.0",
                    "delete_on_startup": "false"
                }
            },
            "notifications": {},
            "file_system_table": true,
            "meta_directory_scan": true,
            "lft_content_expiration_time": "600",
            "delta_content_hang_timeout": 2.0,
            "api_logging": {
                "json_outputs": {},
                "enable_legacy_system": "1",
                "source_base_name": "api-server",
                "event_buffer_min_size_bytes": 1024,
                "tsv_outputs": [
                    {
                        "omit_unknown_columns": true,
                        "print_header_on_new_field": true,
                        "column_order": [
                            "source",
                            "time",
                            "id",
                            "type",
                            "requestname",
                            "status",
                            "reqid",
                            "username",
                            "connectionid",
                            "clientip",
                            "uri",
                            "eventsize",
                            "optionalinfo",
                            "branch",
                            "checkpoint",
                            "multiplepath"
                        ],
                        "files": [
                            "eventlog.tsv"
                        ],
                        "omit_columns": {}
                    }
                ],
                "source_instance_name": "",
                "event_buffer_max_size_bytes": 1048576,
                "file_settings": [
                    {
                        "filename": "eventlog.tsv",
                        "flush_frequency": 1
                    }
                ],
                "event_buffer_try_shrink_interval_sec": 10,
                "enable": false,
                "omit_event_types": {}
            },
            "lft": {
                "launch": "true",
                "check_alive_interval": 10,
                "threshold": "65536"
            },
            "database_threading": true,
            "auto_create_users": true,
            "content_type": "",
            "concurrent_mount_commands": true,
            "metric_period": 10.0,
            "token_expire_time": 86400,
            "max_path_piece_length": "255",
            "max_parallel_copies": 8,
            "crash_record_recovery": true,
            "mount_management_requires_admin": true,
            "record_integrity_check": false,
            "metacache_flush_count": 1000,
            "delta_content_ping_interval": 0.0,
            "list_batch_size_limit": 100,
            "default_branch_name": "default"
        },
        "carb.tasking.plugin": {
            "debugLocks": false,
            "debugTaskBacktrace": false,
            "threadCount": "1",
            "fiberCount": 4095,
            "stuckCheckSeconds": 1,
            "resetFiberStack": true,
            "debugWaitingTasks": false
        },
        "omni.metrics-prometheus": {
            "predefined_labels": {
                "omni_instance": ""
            },
            "model": "pull",
            "push": {
                "host": "",
                "port": 9091,
                "flushTimeInSeconds": 5
            },
            "metricsPrefix": "omni_server_api",
            "pull": {
                "host": "127.0.0.1",
                "endpoint": "/",
                "port": 3010
            }
        },
        "omni.delta-applier": {
            "ports": {
                "server_port": 3001
            }
        },
        "omni.meta-upgrade": {
            "interactive": false,
            "exit_on_any_failure": false,
            "enabled": "true",
            "repair": {
                "enabled": "true",
                "only_after_upgrade": "true"
            },
            "report_progress_every": 1000,
            "convertors": {
                "meta": {
                    "dry_run_since_version": -1
                }
            },
            "remove_deduplicated_content": true,
            "content_resolver": {
                "resolver": "omniverse_resolver_file",
                "options": {
                    "async": true,
                    "root": ""
                }
            }
        },
        "omni.resolver": {
            "directory": ".",
            "libraries": [
                "omniverse_resolver_file",
                "omniverse_resolver_memory",
                "omniverse_resolver_composite",
                "omniverse_resolver_leveldb",
                "omniverse_resolver_s3"
            ]
        },
        "omni.websocketslib": {
            "tracing": false,
            "defaults": {
                "tcp_keepalive_interval": 2,
                "tcp_keepalive_probes": 2,
                "tcp_keepalive_time": 10,
                "http_max_header_length": 61440,
                "tcp_timeout_seconds": 10
            }
        },
        "omni.messaging-redis": {
            "password": "yBGfkZ0iRT",
            "reconnect_interval_ms": 1000,
            "port": "6379",
            "timeout_ms": 500,
            "host": "kube-msk-redis.nvidia.com",
            "max_reconnects": 5
        }
    },
    "privacy": {
        "performance": "false"
    },
    "SERVER": "localhost",
    "LOG": {
        "DIR": "/root/.nvidia-omniverse/logs/Nucleus"
    },
    "omni.common.settings": {
        "env.var.prefix": "OMNI_"
    },
    "S3": {
        "RESOLVER": {
            "CACHE": "localhost:8891"
        }
    },
    "USER": "admin",
    "pluginSearchPaths": [
        ".",
        "."
    ],
    "PASS": "admin",
    "NUCLEUS": {
        "SERVER": {
            "INTERFACE": "0.0.0.0"
        }
    },
    "pluginsLoaded": [
        "carb.dictionary*.plugin",
        "carb.settings.plugin",
        "carb.launcher.plugin",
        "carb.tasking.plugin",
        "carb.crashreporter-breakpad.plugin",
        "omni.metrics-prometheus",
        "omni.signalhandler",
        "omni.wakeupthread",
        "omni.eventsystem",
        "omni.keyvaluedatabase.memdb",
        "omni.keyvaluedatabase.leveldb",
        "omni.server",
        "omni.websocketslib",
        "omni.resolver",
        "omni.core",
        "carb.omnitrace.plugin",
        "omni.objects",
        "omni.meta-upgrade",
        "omni.messaging-redis",
        "omni.eventlogger"
    ],
    "omni.server.app": {}
}
2021-12-13 15:43:42 [6ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/ApiServerCarbLogging.cpp:omniToggleLogChannel():143: Logging enabled for channel: omni.server.BENCHMARK
2021-12-13 15:43:42 [6ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.websocketslib (interfaces: [omni::server::IWebSockets v1.1]) (impl: omni.websocketslib)
2021-12-13 15:43:42 [18ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/ApiConnection.cpp:ApiConnectionManagerImpl():372: ServerVersion: 111.5-98a7aeec
2021-12-13 15:43:42 [18ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.resolver (interfaces: [omni::IResolver v5.2]) (impl: omni.resolver)
2021-12-13 15:43:42 [19ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.keyvaluedatabase.leveldb (interfaces: [omni::server::IKeyValueDatabase v1.0]) (impl: omni.keyvaluedatabase.leveldb)
2021-12-13 15:43:42 [19ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/mounts/MountsConfig.cpp:loadKeyValueDatabase():622: Successfully loaded MetaDataPlugin(omni.keyvaluedatabase.leveldb)
2021-12-13 15:43:42 [26ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.meta-upgrade (interfaces: [omni::server::IMetaUpgrade v1.0]) (impl: omni.meta-upgrade)
2021-12-13 15:43:42 [27ms] [Info] [omni.meta-upgrade] [56] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:getCurrentDbVersion():546: Database version record not found, key: service://metaDatabaseVersion
2021-12-13 15:43:42 [27ms] [Info] [omni.meta-upgrade] [56] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:upgrade():2451: Fresh database, nothing to upgrade
2021-12-13 15:43:42 [27ms] [Info] [omni.meta-upgrade] [56] ../../../source/plugins/omni.meta-upgrade/MetaUpgrade.cpp:setVersion():524: Meta database upgraded
2021-12-13 15:43:42 [27ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/mounts/MountsConfig.cpp:loadUserGroupsDatabase():579: Successfully loaded UserGroupsDatabasePlugin(omni.keyvaluedatabase.leveldb)
2021-12-13 15:43:42 [34ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/mounts/MountsConfig.cpp:loadOmniObjectsDatabase():536: Successfully loaded OmniObjectsDatabase plugin(omni.keyvaluedatabase.leveldb)
2021-12-13 15:43:42 [41ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniversePlugins.cpp:getOmniResolverInstance():42: Successfully created OmniResolver instance of type(omniverse_resolver_file)
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/OmniversePlugins.cpp:getOmniResolverInstance():42: Successfully created OmniResolver instance of type(omniverse_resolver_composite)
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/FileSystemTable.cpp:deleteOldFileSystemTable():375: Deleting any old FileSystemTable entries that might have previously existed
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/FixupRecords.cpp:fixupRecords():333: No non-flushed pending content records found.
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/MetaCache.cpp:firstTimeScan():485: First time scan of the database for records.
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/MetaCache.cpp:firstTimeScan():497: Initial scan of database records complete.
2021-12-13 15:43:42 [42ms] [Info] [omni.server] [56] ../../../source/plugins/omni.server/MetaCache.cpp:loadInitialMetaRecords():512: Loading 0 initial meta records.
2021-12-13 15:43:42 [42ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.metrics-prometheus (interfaces: [omni::server::IMetrics v1.0]) (impl: omni.metrics-prometheus)
2021-12-13 15:43:42 [43ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: omni.core (interfaces: [omni::server::ICore v1.0],[omni::server::ICoreMetaContent v1.0],[omni::server::ICoreService v1.0],[omni::server::IServerFeatures v1.0]) (impl: omni.core)
2021-12-13 15:43:42 [43ms] [Warning] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():702: Acquiring non optional plugin interface which is not listed as dependency: [carb::dictionary::IDictionary v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2021-12-13 15:43:42 [43ms] [Warning] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():702: Acquiring non optional plugin interface which is not listed as dependency: [carb::dictionary::IDictionary v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2021-12-13 15:43:42 [43ms] [Warning] [carb] [56] ../../../source/framework/FrameworkImpl.cpp:acquireInterface():702: Acquiring non optional plugin interface which is not listed as dependency: [carb::dictionary::IDictionary v1.0] (plugin: (null)), by client: omni.core. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2021-12-13 15:43:42 [43ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: carb.launcher.plugin (interfaces: [carb::launcher::ILauncher v1.2]) (impl: carb.launcher.plugin)
2021-12-13 15:43:42 [43ms] [Info] [carb.launcher.plugin] [56] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:ProcessImpl():177: created the process object 0x208e9c0 for './omni-lft-service/omni-lft-service'.
2021-12-13 15:43:42 [43ms] [Info] [carb.launcher.plugin] [56] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():427: launching the process './omni-lft-service/omni-lft-service' from process 56.
2021-12-13 15:43:42 [43ms] [Info] [carb] [56] ../../../source/framework/PluginImpl.cpp:initialize():406: Initializing plugin: carb.threadutil-framework.plugin (interfaces: [carb::thread::IThreadUtil v1.0]) (impl: carb.threadutil-framework.plugin)
2021-12-13 15:43:42 [43ms] [Info] [carb] [56] ../../../source/framework/ThreadUtil.cpp:runRelayTask():199: starting the relayed task thread.
2021-12-13 15:43:42 [44ms] [Info] [carb.launcher.plugin] [88] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():663: [child]: starting the child process 88 from the parent 56.
2021-12-13 15:43:42 [44ms] [Info] [carb.launcher.plugin] [88] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():672: [child]: setting the parent death signal for the child process 88.
2021-12-13 15:43:42 [656ms] [Info] [carb.launcher.plugin] [87] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:forkProcess():834: [parent]: successfully launched the child process 88.
2021-12-13 15:43:42 [656ms] [Info] [carb.launcher.plugin] [56] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():555: [parent]: launched the child process 88 from the parent 56.
2021-12-13 15:43:42 [656ms] [Info] [carb.launcher.plugin] [56] ../../../source/plugins/carb.launcher/ProcessLinux.cpp:launch():578: [parent]: successfully launched the child process 88 {process = 0x208e9c0}
2021-12-13 15:43:42 [656ms] [Info] [omni.core] [56] ../../../source/plugins/common/ChildProcess.cpp:launch():108: Successfully launched the child server Process (/isaac-sim/installers/nucleus/Nucleus/./omni-lft-service/omni-lft-service)
2021-12-13 15:43:42 [656ms] [Info] [carb.launcher.plugin] [116] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():410: starting the read thread for 'launcher stdout reader' for the child process 88.
2021-12-13 15:43:42 [656ms] [Info] [carb.launcher.plugin] [117] ../../../source/plugins/carb.launcher/ProcessCommon.cpp:readFromHandleThread():410: starting the read thread for 'launcher stderr reader' for the child process 88.
2021-12-13 15:43:42 [656ms] [Info] [omni.core] [56] ../../../source/plugins/common/ChildProcess.cpp:launch():111: Sleeping for 2 seconds to get the child server time to spin up
2021-12-13 15:43:43 [917ms] [Error] [omni.core] [117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:DATA_DIR: /root/.local/share/ov/data/serverBundled - using PyInstaller's runtime dir with resolvers

2021-12-13 15:43:43 [947ms] [Error] [omni.core] [117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2021-12-13 15:43:43,196 - system_log - INFO - reconnecting ...

2021-12-13 15:43:43 [950ms] [Error] [omni.core] [117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:2021-12-13 15:43:43,199 - system_log - INFO - [Errno 111] Connect call failed ('127.0.0.1', 3007)

2021-12-13 15:43:44 [1,952ms] [Error] [omni.core] [117] ../../../source/plugins/common/ChildProcess.cpp:stderrReadCallback():30: From process ./omni-lft-service/omni-lft-service:Traceback (most recent call last):
  File "idl/connection/transport/omni.py", line 110, in prepare
  File "websockets/legacy/client.py", line 622, in __await_impl__
  File "asyncio/base_events.py", line 794, in create_connection
  File "asyncio/base_events.py", line 781, in create_connection
  File "asyncio/selector_events.py", line 439, in sock_connect
  File "asyncio/selector_events.py", line 469, in _sock_connect_cb
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 3007)