The failure about importing USD file in AODT

Hi, community:
I am using AODT v1.0 and our team is interested in indoor scenario.

First, I found kitchen.usd from Downloads and Videos — Universal Scene Description 24.11 documentation
Then, it was uploaded to server as:

However, the error is shown like:


and

How to solve it?

Then, I followed user guide **on v1.0 ** aodt gis container to convert gml file to usd file but the error came up as:

I found some files generated in


Then, I used docker cp to host, but the error again:

My questions are:

  1. How to fix the two problem?
  2. Is there any available indoor usd file for AODT v1.0?

Thank you.

I met the same problem when I followed user guide

docker run -it --name aodt-gis --entrypoint /bin/bash  nvcr.io/esee5uzbruax/aodt-gis:1.0.0

cd src/
chmod 777 aodt_gis

cd src/aodt_gis/samples/input_data
./get_tokyo.sh

export OMNI_USER=omniverse
export OMNI_PASS=aerial_123456

cd /src/aodt_gis/build

./aodt_gis \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393599_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393690_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393589_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393680_bldg_6697_2_op.gml \
--output omniverse://omniverse-server/Users/aerial/plateau/sample.usd --epsg_in EPSG:6697 --epsg_out EPSG:32654

Then,

root@77690c2b1298:/src/aodt_gis/build# ./aodt_gis \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393599_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393690_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393589_bldg_6697_2_op.gml \
../samples/input_data/13100_tokyo23-ku_2022_citygml_1_2_op/udx/bldg/53393680_bldg_6697_2_op.gml \
--output omniverse://omniverse-server/Users/aerial/plateau/sample.usd --epsg_in EPSG:6697 --epsg_out EPSG:32654
[ERROR] Failed connection, exiting.
terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
Aborted (core dumped)

@guofachang What version is this? Please update to AODT 1.1. The GIS processing is a part of OV Console and easier to use.

@kpasad1 , the version is v1.0.0 on A6000. Since our team are working on it, the risks of the upgrade need to be assessed.

Could you please give any suggestion on the above core dump? This seems unrelated to the GPU type.

If I want to upgrade the software, do I need to completely uninstall the previous version first?

@guofachang

  1. What is your deployment? Is it colocated (both UI/Backend on same machine)?
  2. Can you share the output of nvidia-smi when you run the pipeline?
  3. Please stop the backend docker process and try again.

@kpasad1

Yes, it is colocated on the same machine.

The output was shown in host machine like:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:04:00.0 Off |                  Off |
| 35%   66C    P2            195W /  300W |    6055MiB /  49140MiB |     48%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A6000               Off |   00000000:82:00.0 Off |                  Off |
| 30%   33C    P8             26W /  300W |    3795MiB /  49140MiB |     21%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX A6000               Off |   00000000:83:00.0 Off |                  Off |
| 30%   32C    P8             17W /  300W |    3799MiB /  49140MiB |     21%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     27766      C   ./aodt_sim                                   1218MiB |
|    0   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       4811MiB |
|    1   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       3775MiB |
|    2   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       3779MiB |
+-----------------------------------------------------------------------------------------+

Besides, there is no nvidia-smi in the container:

root@77690c2b1298:/src/aodt_gis/build# nvidia-smi
bash: nvidia-smi: command not found

I have stopped the backend container using docker compose down and the nvidia-smi in host terminal was

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:04:00.0 Off |                  Off |
| 43%   71C    P2            201W /  300W |    4834MiB /  49140MiB |     58%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A6000               Off |   00000000:82:00.0 Off |                  Off |
| 30%   33C    P8             25W /  300W |    3795MiB /  49140MiB |     23%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX A6000               Off |   00000000:83:00.0 Off |                  Off |
| 30%   32C    P8             18W /  300W |    3799MiB /  49140MiB |     26%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       4811MiB |
|    1   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       3775MiB |
|    2   N/A  N/A   1859299    C+G   ...cal/share/ov/pkg/asim-1.0.0/kit/kit       3779MiB |
+-----------------------------------------------------------------------------------------+

However, the same coredump came up again AND the watch -n 1 nvidia-smi kept the same

Then, I stopped all runnping process:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:04:00.0 Off |                  Off |
| 30%   43C    P0             72W /  300W |       1MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A6000               Off |   00000000:82:00.0 Off |                  Off |
| 30%   49C    P0             81W /  300W |       1MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX A6000               Off |   00000000:83:00.0 Off |                  Off |
| 30%   44C    P0             73W /  300W |       1MiB /  49140MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

The result was still:

I guess this is unrelated to nvidia-smi since there is no driver brought into container.
Is the GML file type an independent type that doesn’t require other files?

Thank you for your help.

  1. Make sure you copied template.usd from omniverser server location at Users//assets into omniverse://omniverse-server/Users/aerial/plateau/sample.usd
  2. Please run the aodt_gis command with -v info

@kpasad1

  1. yes, I copied template.usd from the target path
  2. with ‘-v info’, the info was
root@77690c2b1298:/src/aodt_gis/build# ./aodt_gis -v info ../samples/53393599_bldg_6697_2_op.gml ../samples/53393690_bldg_6697_2_op.gml ../samples/53393589_bldg_6697_2_op.gml ../samples/53393680_bldg_6697_2_op.gml --output omniverse://omniverse-server/Users/aerial/plateau/sample.usd --epsg_in EPSG:6697 --epsg_out EPSG:32654 
[INFO] Scaling: 100
[INFO] Translation: (0,0,0)
[INFO] Initializing Version 2.38.2-hotfix.5050+tc.742a8f8d
[INFO] Initialized Nucleus API v11.20-2ef23c40
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 173 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 528 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 1198 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 1939 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 2601 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 3800 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 5169 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 6495 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Will attempt to reconnect in 8215 milliseconds
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] libcurl error (6): Couldn't resolve host name for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Connection to omniverse://omniverse-server failed: Timed Out
[INFO] Stopped automatic reconnects after 120 seconds (9 attempts)
[ERROR] Failed connection, exiting.
terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
Aborted (core dumped)

I have done the following before

export OMNI_USER=omniverse
export OMNI_PASS=aerial_123456

Besides, with the command docker compose -f docker-compose-sm80.yml up, the log at start was

connector-1   | [INFO] OVLOG:[Info] Initializing Version 2.38.2-hotfix.5050+tc.742a8f8d
connector-1   | [INFO] Attempting to connect to nucleus url: omniverse://omniverse-server/broadcast
connector-1   | [INFO] OVLOG:[Info] Initialized Nucleus API v11.20-2ef23c40
connector-1   | [INFO] join request id is 1
connector-1   | [DEBUG] Connection Status: Attempting to connect [omniverse://omniverse-server]
connector-1   | [INFO] OVLOG:[Info] libcurl error (7): Couldn't connect to server for https://omniverse-server/omni/auth/api/device/authorize
connector-1   | [INFO] OVLOG:[Info] Using username/password from environment variables for omniverse://omniverse-server
connector-1   | [INFO] OVLOG:[Info] Auth: Retrieved token for user "omniverse"
connector-1   | [INFO] OVLOG:[Info] Server is running version 1.14.37+tag-1.14.37.gitlab.12309863.0959a9e9
connector-1   | [INFO] OVLOG:[Info] Server omniverse://omniverse-server supports atomic checkpoints
connector-1   | [INFO] OVLOG:[Info] Server omniverse://omniverse-server supports omni-objects and it is enabled
connector-1   | [INFO] OVLOG:[Info] Server omniverse://omniverse-server supports checkpoints and it is enabled
connector-1   | [INFO] OVLOG:[Info] Connected to omniverse://omniverse-server
connector-1   | [DEBUG] Connection Status: Successfully connected [omniverse://omniverse-server]

It seems that the version 2.38.2-hotfix.5050+tc.742a8f8d fails to connect while 1.14.37+tag-1.14.37.gitlab.12309863.0959a9e9 succeeds

Thank you for your kindly reply.

@guofachang Thanks. Lets try two solutions:

  1. Add the --network=host option to docker e.g.
    docker run -it --network=host --name aodt-gis --entrypoint /bin/bash nvcr.io/esee5uzbruax/aodt-gis:1.0.0
    Make sure the exisiting docker session is stopped

  2. In the .aodt_gis command, replace the omniverse server with the actual IP addrress e.g. ./aodt_gis -v info *** --output omniverse://10.123.456.789/Users/aerial/plateau/sample.usd --epsg_in EPSG:6697 --epsg_out EPSG:32654

Please try these two fixes.

@kpasad1 Thank you very much. I fixed it with rebuild the container with --network=host.

The actual ip fails but using omniverse-server succeeds with log:
root@ubuntu22:/src/aodt_gis/build# ./aodt_gis -v info …/samples/53393690_bldg_6697_2_op.gml
–output omniverse://omniverse-server/Users/aerial/plateau/sample.usd

[INFO] Scaling: 100
[INFO] Translation: (0,0,0)
[INFO] Initializing Version 2.38.2-hotfix.5050+tc.742a8f8d
[INFO] Initialized Nucleus API v11.20-2ef23c40
[INFO] libcurl error (7): Couldn't connect to server for https://omniverse-server/omni/auth/api/device/authorize
[INFO] Using username/password from environment variables for omniverse://omniverse-server
[INFO] Auth: Retrieved token for user "omniverse"
[INFO] Server is running version 1.14.37+tag-1.14.37.gitlab.12309863.0959a9e9
[INFO] Server omniverse://omniverse-server supports atomic checkpoints
[INFO] Server omniverse://omniverse-server supports omni-objects and it is enabled
[INFO] Server omniverse://omniverse-server supports checkpoints and it is enabled
[INFO] Connected to omniverse://omniverse-server
[INFO] Successfully created S3 provider for https://omniverse-content-production.s3.us-west-2.amazonaws.com
[INFO] Using CloudFront http://dcb18d6mfegct.cloudfront.net/, List = No
[INFO] Stage UUID: 4f4c0a8c-53eb-4f5c-8899-f1815a672652
[INFO] No ground plane geometry
[INFO] apply projection
......
1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.