Unable to connect to worker (aodt-1.2.0)

[sxn0581]

I am facing an issue when I am trying to connect to woker, could you please help me with this.

Debugging console:
[INFO] Trying to add live to root layer
[DEBUG] primvars:mobility_type not found
[DEBUG] num indoor triangles: 0, num mobility indices: 0
[DEBUG] Adding /opt/nvidia/aodt_sim/build/channel_predictor to python path.

[ERROR] Couldn’t find/read ‘Thickness’ value of material: concrete
[DEBUG] sending message: “log” to: 324bc1ec-09ff-4107-93f5-f0c9496c2bbd with payload:
{
** “body”: “Couldn’t find/read ‘Thickness’ value of material: concrete”,**
** “level”: “[ERROR]”**
}
[ERROR] simulation state constructor threw exception error in buildMaterialDictionary
[DEBUG] sending message: “log” to: 324bc1ec-09ff-4107-93f5-f0c9496c2bbd with payload:
{
** “body”: “simulation state constructor threw exception error in buildMaterialDictionary”,**
** “level”: “[ERROR]”**
}
[DEBUG] sending message: “open_scene_reply” to: 324bc1ec-09ff-4107-93f5-f0c9496c2bbd with payload:
{
** “open_success”: false**
}
[DEBUG] sending message: “heartbeat” to: 324bc1ec-09ff-4107-93f5-f0c9496c2bbd with payload:
{
** “count”: 0.0,**
** “periodicity_in_seconds”: 30.0**
}
[DEBUG] Join Channel Callback called
** Result: 0**
** User Data Ptr: 0x58c2f2620be0**
** From: 73J0L94G75M7F8ND**
** Event: Message**
[DEBUG] Got message type: “heartbeat_reply”, payload: [73J0L94G75M7F8ND]: {
** “from”: “324bc1ec-09ff-4107-93f5-f0c9496c2bbd”,**
** “message_len”: 0.0,**
** “message_payload”: {**
** “count”: 0.0**
** },**
** “message_type”: “heartbeat_reply”,**
** “to”: “9b1eb84f-3c8a-457d-e8a1-6428d6a6799d”**
}
[DEBUG] sending message: “heartbeat” to: 324bc1ec-09ff-4107-93f5-f0c9496c2bbd with payload:
{
** “count”: 1.0,**
** “periodicity_in_seconds”: 30.0**
}
[DEBUG] Join Channel Callback called
** Result: 0**
** User Data Ptr: 0x58c2f2620be0**
** From: 73J0L94G75M7F8ND**
** Event: Message**
[DEBUG] Got message type: “heartbeat_reply”, payload: [73J0L94G75M7F8ND]: {
** “from”: “324bc1ec-09ff-4107-93f5-f0c9496c2bbd”,**
** “message_len”: 0.0,**
** “message_payload”: {**
** “count”: 1.0**
** },**
** “message_type”: “heartbeat_reply”,**
** “to”: “9b1eb84f-3c8a-457d-e8a1-6428d6a6799d”**
}

This is the image of console displaying the errors in UI

List of Containers running:

Could you please help me with this.

Thank you

[kpasad1]

@sxn0581

1. Can you please share the output of nvidia-smi
2. Can you please share the snapshot of configuration tab? (you can black out any IP address if you wish)

[sxn0581]

[kpasad1]

Can you share the bottom half of configuration? We want to ensure the correct assets paths

[sxn0581]

[kpasad1]

Thanks. That looks OK.
Did you update from 1.1? If so, how?
Please try the following:

1. delete the .live folder under kyoto map
2. Close and reopen UI
3. Restart Backend.

[sxn0581]

I installed aodt-1.2.0 from scratch by deleting all the files inside .local/share/ov/pkg/.
I removed all the containers after stopping them, and then I started the installation of aodt-1.2.0.

I also removed all the files of aodt-1.1.1

I followed the above steps of removing .live files of kyoto and restarted the backend, but still not able to connect to worker

If there is something wrong in these steps please do let me know.

[kpasad1]

Was there a reason to install AODT 1.2 this way, vs upgrading using our scripts?

[sxn0581]

No particular reason, thought any error would result in crashing of the application.

[sxn0581]

Would there be any files from the previous version which are causing this error. If so please do let know what are the factors that could lead to this error.

[kpasad1]

yes, thats what we think. The asset files could be mixed up with previous version. Do you see anything under omniverse://<ip_address>/Library/ folder in the content tab?

[sxn0581]

This is the image of assets folder.

[kpasad1]

Are you able to resolve? There is a mix up of 1.1 assets and 1.2 assets.
If not, copy the content of Library/aodt_1.2.0 into Users/aerial.

[sxn0581]

This step resulted in the loss of worker.

[kpasad1]

@sxn0581 Please restart the container

[kpasad1]

@sxn0581 Did you resolve the issue?

[sxn0581]

No that process of restarting container did not work.

[sxn0581]

I tried this way of running the build application with command
OMNI_USER=omniverse OMNI_PASS=aerial_123456 ./build/aodt_sim --nucleus omniverse://omniverse-server --broadcast broadcast --log debug
and then tried to connect with worker using UI and this method was working fine until the build file was running inside the dev container.

[kpasad1]

@sxn0581 Please check the following:

1. In the file /etc/hosts, omniverse-server is mapped to the correct ip address.
2. In the file /opt/ove/base_stack/nucleus-stack.env, check if SERVER_IP_OR_HOST is set correctly.
3. Please share the backend console output when you try to make a connection to the worker