Nucleus error - failed connection to Delta server

Omniverse General Discussion
#1

Hi! New user here, excited to dive into Omniverse!

But facing a connection issue that I can’t sort on my own.

I’m on a single local machine, 2080 Ti.
Installed the Nucleus Collaboration Service and I’m able to log in to its config page, as well as add a server.
But no app or connector relying on a connection works.

I then realized the Nucleus app (v.109.8-1935) shows as not running, with an error log.

From:
[Info] [omni.server] Successfully launched the DeltaContent Server Process

It leads to:
[...]
[Info] [omni.server] Attempt to connect Delta server failed, retry after 2000 ms. (10/10)
[Fatal] [omni.server] Failed to connect to the DeltaContentServer, can't continue

Since it doesn’t display much more, does anybody have an idea what it could be?

I tried whitelisting all the ports mentioned in the documentation, as well as reinstalling the Nucleus service on varied locations. No change.

Any insight greatly appreciated!

#2

Hi phscatena, sorry you’re having issues with Nucleus. Unfortunately I can’t repro your issue. Can you provide some more info to help us resolve this issue please?

  • What is your environment OS?
  • Do you still repro the issue when you kill the omni.server.app process and restart it?
  • Can you share your full Nucleus server log? It can be found in the .nvidia-omniverse directory. It can also be found in System Monitor (localhost:3080), under Logs.
#3

Also, one more question to tack on to @j-san’s requests :

Can you check to see if omni.delta-server.app.exe is actually running in your Task Manager Details tab? (CTRL+SHIFT+ESCAPE, click More Details at the bottom if you don’t have a Details tab, I find sorting by name helps quickly find processes in the list).

#4

Thank you for the quick replies!

Indeed, it sounds more like a specific local issue. I had a colleague with similar specs try a fresh install and it all ran smoothly. So I’m trying to narrow it down…

I am running Win 10 v1909, and he is running v20H2. Maybe that could be a thing? I’m queuing up an update anyways.

For the process,omni.delta-server.app.exe runs while the Nucleus attempt to use it (i.e., when I set it to “start” from the system monitor). When it fails, the process is killed. If I start the process on my own, nothing changes.

Log is in attachment.
Nucleus.log (24.7 KB)

#5

Thank you for the logs phscatena. I’ve opened up an internal ticket to track this. OM-25729

I’ve tried a few more times to repro this and no dice. For what it’s worth, these are the exact steps I’m using:

  1. Install Launcher on Windows. I’m on Version 10.0.19041.685
  2. Set Library and Data paths to external D: drive
  3. Install Local Collaboration Service (version 2020.3.0)
  4. Verify that everything is running (no DeltaServer issues at this point). I used localhost:3080 in web browser to check.
  5. Install and launch Create
  6. In Create content browser, try to add ‘localhost’ as a connection
  7. Open up omniverse://localhost/NVIDIA/Samples/OldAttic/Attic_NVIDIA.usd

I did not encounter any issue when following these steps.

1 Like
#6

Thank you for the continued support!

I did update Windows to its latest version, to no changes. Tried fresh re-installs of launcher and nucleus service on different drives. Always the same undesired error, in what would be step 4.

It’s sounding like it is a problem very specific to this machine. I wonder why.
I will update here if I manage to do a full system reinstall.

If it’s of any worth, logs and configs for the latest install can be found in attachment:
monitor_logs.zip (28.9 KB)
user_nvidia-omniverse.zip (20.6 KB)

#7

Hi @phscatena, it’s no problem. I’m grateful for your patience while we try and get this issue resolved.

Checking Search Service logs, it looks like it can’t communicate with the Nucleus Server.

A couple more questions

  • Is the Nucleus Server itself running? It is omniverse.server.app.exe in task manager
  • If so, can you try running this command in Windows and dump the output here please? netstat -nao

It’s possible another process is running on the default Nucleus Server and Delta Server ports. Nucleus Server uses 3009, and Delta Server uses 3001. Checking the netstat output should help verify that.

1 Like
#8

Hi @j-san, thanks again!

The nucleus server process runs only while attempting to connect. When the error happens, it gets killed, together with the Delta Server.

I had formerly used the Resource Monitor to check if something else was listening at said ports, and it didn’t seem so.
I believe Netstat points in the same direction.

For better review, in attachment are the output of netstat in four different moments: before the Omniverse services were started, during the attempted connection, shortly after, and a couple of minutes after, when the processes are fully terminated again.
Apparently the Omni servers are the only ones using such ports.

netstat_scatena.zip (9.7 KB)

#9

We have forwarded that information to our developers and will get back to you.

#10

@phscatena Sorry for the late reply. We’ve identified the issue. A fix will be available in an upcoming release.

In the mean time, there is a workaround that you can try:

  1. Stop the Nucleus Server
  2. Find the directory that has the omni.server.app executable
  3. Open omni.server.app.config.json in a text editor (and create a backup copy just in case)
  4. Find these three fields:
  • delta_content_hang_timeout
  • delta_content_ping_interval
  • delta_connect_retry_count
  1. Set them as follows:
  • “delta_content_hang_timeout”: 10.0,
  • “delta_content_bw_estimate”: 0.5,
  • “delta_content_ping_interval”: 30.0,
  • “delta_connect_retry_count”: 20,
  1. Restart the Nucleus Server
#11

@j-san, thanks for the info and suggestion.
Sorry for the late reply. I had tried the suggestion at the time, to no effect, but ended up forgetting to write here.
Today I tried it again with the current release: Nucleus v.109.9 / nucleus-workstation-2020.3.1
Still same error, regardless if I increase the timeout or retries tolerance.
It still looks very specific to this one workstation. But I appreciate the support!

#12

Hello phscatena, I wanted to check in with you. Are you still having a connection issue?

#13

Hi there,
its been a while this thread was active. So, I have the same issues, omni.delta-server.app.exe is beeing killed after a while, so I don’t have access to the localhost. Any news on that?

#14

Hi, I have the exact same problem as the people above.
Nucleus service gives error and stops running after 15-20 secs in Local Nucleus Service 2022.3.0 so I get “[127.0.0.1] (NUCLEUS) service is not accessible. Please check your network connection and the server status with administrator.”
If the service is restarted manually through the localhost:3080 page+start button, same thing happens, the process gives error and stops (gets killed) after a while.
Tried running manually omni.delta-server.app.exe, high CPU usage for about 20-30 secs, then the cmd window closes and the process gets killed.
I have been trying about a month now, with multiple installations-uninstallations, different installation paths & drives, using the clean-up tool, deleting every leftover trace I can find, with and without cache, disabling firewall, updating graphics card drivers and windows updates with no luck. The problem remains every single time.
I can replicate this error only on one specific machine. Installation and nucleus service on another pc (Windows 10 Pro) within the same network, same firewall, same antivirus works just fine.
Processor Intel(R) Core™ i9-9900X CPU / 64GB ram
GPU RTX 3090
OS: Windows 10 Pro 21H2
Nucleus.log (39.6 KB)