Issue with AODT 1.2: DU Asset Not Found

Accelerated Computing Aerial Forum
1

I’m currently working on deploying Aerial Omniverse Digital Twin (AODT) 1.2 on my local machine, and I’ve run into a couple of issues during the installation and connection process.

During the deployment of the local Nucleus Server, I encountered the following error:

image
image1884×1058 222 KB

Despite this, I proceeded with the setup for now.

However, when I try to connect AODT to the Nucleus server and Attach Worker, I get an error stating that the DU asset is not found.

image
image1920×1193 109 KB

Is it related to the earlier Nucleus error I ignored? Any guidance or suggestions would be greatly appreciated!

Thanks in advance.

2

Hi @fltang
what GPU models are used for front and backends?
Also did the installation finish successfully?
can you share the output of the “docker ps” command in your terminal?

3

For the GPU models:

  • Frontend: A4000 (16G)
  • Backend: A100 (40G)

There were no issues during the AODT installation.

Below is the output of docker ps:

Backend (A100):

CONTAINER ID   IMAGE                                                     COMMAND                   CREATED      STATUS                PORTS                                                                                                                             NAMES
a07154650da4   nvcr.io/esee5uzbruax/aodt-sim:1.2.0_runtime               "/opt/nvidia/nvidia_…"   7 days ago   Up 6 days                                                                                                                                               backend-connector-1
86cc87cd0d8e   nvcr.io/esee5uzbruax/aodt-gis:1.2.0                       "python3 aodt_py/aod…"   7 days ago   Up 6 days                                                                                                                                               backend-gis-1
a7fb94d679f0   clickhouse/clickhouse-server:24.7.4.51-alpine             "/entrypoint.sh"          7 days ago   Up 6 days             8123/tcp, 9009/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                                                     backend-clickhouse-1
aef97baad8b9   aodt_sim-notebook                                         "tini -g -- start.sh…"   7 days ago   Up 6 days (healthy)   0.0.0.0:8888->8888/tcp, :::8888->8888/tcp                                                                                         aodt_sim-notebook-1
7caa3c404eba   nvcr.io/nvidia/omniverse/nucleus-auth:1.5.2               "/root/eula.sh /bin/…"   7 days ago   Up 6 days             0.0.0.0:3100->3100/tcp, :::3100->3100/tcp, 0.0.0.0:3180->3180/tcp, :::3180->3180/tcp, 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp   base_stack-nucleus-auth-1
dc693eef1cd6   nvcr.io/nvidia/omniverse/nucleus-api:1.14.37              "/root/eula.sh ./lau…"   7 days ago   Up 6 days             0.0.0.0:3009->3009/tcp, :::3009->3009/tcp, 0.0.0.0:3106->3106/tcp, :::3106->3106/tcp, 3019/tcp                                    base_stack-nucleus-api-1
3b4e8f8e072e   nvcr.io/nvidia/omniverse/nucleus-search:3.2.9             "/root/eula.sh /bin/…"   7 days ago   Up 6 days             0.0.0.0:3400->3400/tcp, :::3400->3400/tcp                                                                                         base_stack-nucleus-search-1
aeeb5ad5406a   nvcr.io/nvidia/omniverse/nucleus-navigator:3.3.4          "/root/eula.sh /entr…"   7 days ago   Up 6 days             443/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp                                                                                    base_stack-nucleus-navigator-1
438be4f0a335   nvcr.io/nvidia/omniverse/nucleus-tagging:3.1.11           "/omni/docker-entryp…"   7 days ago   Up 6 days             0.0.0.0:3020->3020/tcp, :::3020->3020/tcp                                                                                         base_stack-nucleus-tagging-1
9d6a78948b60   nvcr.io/nvidia/omniverse/nucleus-discovery:1.5.2          "/root/eula.sh pytho…"   7 days ago   Up 6 days             0.0.0.0:3333->3333/tcp, :::3333->3333/tcp                                                                                         base_stack-nucleus-discovery-1
46351d154cad   nvcr.io/nvidia/omniverse/utl-monpx:1.14.37                "/root/eula.sh pytho…"   7 days ago   Up 6 days             0.0.0.0:3010->8080/tcp, :::3010->8080/tcp                                                                                         base_stack-utl-monpx-1
7b1e545fbb8f   nvcr.io/nvidia/omniverse/nucleus-thumbnails:1.5.10        "/omni/docker-entryp…"   7 days ago   Up 6 days                                                                                                                                               base_stack-nucleus-thumbnails-1
a42338dd5562   nvcr.io/nvidia/omniverse/nucleus-lft-lb:1.14.37           "/root/eula.sh /omni…"   7 days ago   Up 6 days             0.0.0.0:3030->3030/tcp, :::3030->3030/tcp                                                                                         base_stack-nucleus-lft-lb-1
c0c2961ad597   nvcr.io/esee5uzbruax/aodt-sim:1.0.0_runtime_SM89          "/opt/nvidia/nvidia_…"   9 days ago   Up 6 days                                                                                                                                               backend_bundle-connector-1
b75ae61fcaf6   nvcr.io/nvidia/omniverse/nucleus-lft:1.14.37              "/root/eula.sh pytho…"   9 days ago   Up 6 days             3030/tcp                                                                                                                          base_stack-nucleus-lft-1
154d0e17b304   nvcr.io/nvidia/omniverse/nucleus-log-processor:1.14.37    "/root/eula.sh pytho…"   9 days ago   Up 6 days                                                                                                                                               base_stack-nucleus-log-processor-1
42bc90b7122c   nvcr.io/nvidia/omniverse/nucleus-resolver-cache:1.14.37   "/root/eula.sh /omni…"   9 days ago   Up 6 days             8891/tcp                                                                                                                          base_stack-nucleus-resolver-cache-1
73ea905f20ab   graffic/logrotate:1.4                                     "/usr/local/bin/entr…"   9 days ago   Up 6 days                                                                                                                                               base_stack-tagging-logrotate-1

Frontend (A4000):

CONTAINER ID   IMAGE                                                     COMMAND                   CREATED       STATUS                PORTS                                                                                                                                   NAMES
231ad9899ec7   nvcr.io/esee5uzbruax/aodt-sim:1.2.0_runtime               "/opt/nvidia/nvidia_…"   6 days ago    Up 6 days                                                                                                                                                     backend-connector-1
76cc8b52a1f6   nvcr.io/esee5uzbruax/aodt-gis:1.2.0                       "python3 aodt_py/aod…"   6 days ago    Up 6 days                                                                                                                                                     backend-gis-1
d5121bdd2921   clickhouse/clickhouse-server:24.7.4.51-alpine             "/entrypoint.sh"          6 days ago    Up 6 days             8123/tcp, 9009/tcp, 0.0.0.0:9000->9000/tcp, [::]:9000->9000/tcp                                                                         backend-clickhouse-1
5d342aa62b63   aodt_sim-notebook                                         "tini -g -- start.sh…"   6 days ago    Up 6 days (healthy)   0.0.0.0:8888->8888/tcp, [::]:8888->8888/tcp                                                                                             aodt_sim-notebook-1
d2564ad0ecef   nvcr.io/nvidia/omniverse/nucleus-api:1.14.37              "/root/eula.sh ./lau…"   6 days ago    Up 6 days             0.0.0.0:3009->3009/tcp, [::]:3009->3009/tcp, 0.0.0.0:3106->3106/tcp, [::]:3106->3106/tcp, 3019/tcp                                      base_stack-nucleus-api-1
aa8403b80529   nvcr.io/nvidia/omniverse/nucleus-auth:1.5.2               "/root/eula.sh /bin/…"   6 days ago    Up 6 days             0.0.0.0:3100->3100/tcp, [::]:3100->3100/tcp, 0.0.0.0:3180->3180/tcp, [::]:3180->3180/tcp, 0.0.0.0:8000->8000/tcp, [::]:8000->8000/tcp   base_stack-nucleus-auth-1
0ac284f1dfcf   nvcr.io/nvidia/omniverse/nucleus-navigator:3.3.4          "/root/eula.sh /entr…"   6 days ago    Up 6 days             443/tcp, 0.0.0.0:8080->80/tcp, [::]:8080->80/tcp                                                                                        base_stack-nucleus-navigator-1
7211405c11e8   nvcr.io/nvidia/omniverse/nucleus-thumbnails:1.5.10        "/omni/docker-entryp…"   6 days ago    Up 6 days                                                                                                                                                     base_stack-nucleus-thumbnails-1
184476e21467   nvcr.io/nvidia/omniverse/nucleus-log-processor:1.14.37    "/root/eula.sh pytho…"   6 days ago    Up 6 days                                                                                                                                                     base_stack-nucleus-log-processor-1
1d450b97247b   nvcr.io/nvidia/omniverse/nucleus-lft-lb:1.14.37           "/root/eula.sh /omni…"   6 days ago    Up 6 days             0.0.0.0:3030->3030/tcp, [::]:3030->3030/tcp                                                                                             base_stack-nucleus-lft-lb-1
fd642127e68f   nvcr.io/nvidia/omniverse/nucleus-lft:1.14.37              "/root/eula.sh pytho…"   6 days ago    Up 6 days             3030/tcp                                                                                                                                base_stack-nucleus-lft-1
590c1d7e7ffd   nvcr.io/nvidia/omniverse/nucleus-tagging:3.1.11           "/omni/docker-entryp…"   6 days ago    Up 6 days             0.0.0.0:3020->3020/tcp, [::]:3020->3020/tcp                                                                                             base_stack-nucleus-tagging-1
aa7ede0b94d1   nvcr.io/nvidia/omniverse/nucleus-discovery:1.5.2          "/root/eula.sh pytho…"   6 days ago    Up 6 days             0.0.0.0:3333->3333/tcp, [::]:3333->3333/tcp                                                                                             base_stack-nucleus-discovery-1
d8365ef81ef8   nvcr.io/nvidia/omniverse/utl-monpx:1.14.37                "/root/eula.sh pytho…"   6 days ago    Up 6 days             0.0.0.0:3010->8080/tcp, [::]:3010->8080/tcp                                                                                             base_stack-utl-monpx-1
cba875e29ff7   nvcr.io/nvidia/omniverse/nucleus-search:3.2.9             "/root/eula.sh /bin/…"   6 days ago    Up 6 days             0.0.0.0:3400->3400/tcp, [::]:3400->3400/tcp                                                                                             base_stack-nucleus-search-1
b4572d791203   graffic/logrotate:1.4                                     "/usr/local/bin/entr…"   6 days ago    Up 6 days                                                                                                                                                     base_stack-tagging-logrotate-1
80dcd5ecc801   nvcr.io/nvidia/omniverse/nucleus-resolver-cache:1.14.37   "/root/eula.sh /omni…"   6 days ago    Up 6 days             8891/tcp                                                                                                                                base_stack-nucleus-resolver-cache-1
55a4278578a6   nvcr.io/esee5uzbruax/aodt-sim:1.0.0_runtime_SM89          "/opt/nvidia/nvidia_…"   2 weeks ago   Up 6 days                                                                                                                                                     backend_bundle-connector-1

Let me know if you need anything else!

4

@fltang
Please confirm if this is a co-located setup?
how did you install and if it was installed on a new OS?
when installing using the recommended method in the documentation, the user does not need to install Nucleus separately.
Please also provide the installation logs “install_date.log” in the ~/aodt_1.2.0 folder and console logs here:
~/.local/share/ov/pkg/aodt-1.2.0/logs/Kit/aodt/1.2

5

We tested both co-located and non co-located setups and saw the same error in both cases. By default, it tries to connect to a Nucleus server, but as shown in the screenshot, the connection fails.

image
image3056×852 100 KB

Due to this, we tried setting up a local Nucleus server, which connects successfully, but we encountered the issue described in this post.

Attached logs show:

  • kit_20250418_190830.log: no config changes, stuck at “checking nucleus server accessibility” when attaching worker.
  • kit_20250418_190945.log: with config pointing to local Nucleus server, ends with: “Cannot attach worker because the following assets do not exist:”

install_20250418_185427.log (31.3 KB)
kit_20250418_190830.log (464.9 KB)
kit_20250418_190945.log (465.2 KB)

6

Discovery service failure could be due to mismatch in IP between OV on UI and IP in the Nucleus.

Please try to see if you are able to reach the nucleus server via the browser at ipaddress:3180
Also try this from different browsers: Chromium/chrome/firefox. Requests could be blocked due to CORS (Cross-Origin Request Sharing)- a configuration in the browser.

7

Yes, it was an IP issue — I’ve fixed that. However, the DU asset is still missing. For now, I’ve manually copied the DU asset file from the backend directory to the Nucleus server. It seems to be working fine at the moment. Thanks!