AODT 1.2 Fails Abruptly

Accelerated Computing Aerial Forum
1

AODT 1.2 fails with CUDA error.

2025-04-16 18:03:42 [277,384ms] [Error] [omni.kit.app._impl] [py stderr]: Traceback (most recent call last):
2025-04-16 18:03:42 [277,384ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/kit/python/lib/python3.10/contextlib.py", line 153, in __exit__
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:     self.gen.throw(typ, value, traceback)
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.usd.libs-1.0.1+3a7abd1c.lx64.r.cp310/pxr/Trace/__init__.py", line 41, in TraceScope
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:     yield
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.usd.libs-1.0.1+3a7abd1c.lx64.r.cp310/pxr/Trace/__init__.py", line 77, in invoke
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:     return func(*args, **kwargs)
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.kit.widget.layers-1.7.9+3a7abd1c/omni/kit/widget/layers/selection_watch.py", line 86, in _on_kit_selection_changed
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:     edit_target, selection = self._tree_view.model.find_all_specs(sdf_paths)
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.usd.libs-1.0.1+3a7abd1c.lx64.r.cp310/pxr/Trace/__init__.py", line 77, in invoke
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:     return func(*args, **kwargs)
2025-04-16 18:03:42 [277,385ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.kit.widget.layers-1.7.9+3a7abd1c/omni/kit/widget/layers/layer_model.py", line 936, in find_all_specs
2025-04-16 18:03:42 [277,386ms] [Error] [omni.kit.app._impl] [py stderr]:     return edit_target, edit_target.find_all_specs(paths)
2025-04-16 18:03:42 [277,386ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.usd.libs-1.0.1+3a7abd1c.lx64.r.cp310/pxr/Trace/__init__.py", line 77, in invoke
2025-04-16 18:03:42 [277,386ms] [Error] [omni.kit.app._impl] [py stderr]:     return func(*args, **kwargs)
2025-04-16 18:03:42 [277,386ms] [Error] [omni.kit.app._impl] [py stderr]:   File "/home/aerial/.local/share/ov/pkg/aodt-1.2.0/extscache/omni.kit.widget.layers-1.7.9+3a7abd1c/omni/kit/widget/layers/layer_item.py", line 962, in find_all_specs
2025-04-16 18:03:42 [277,387ms] [Error] [omni.kit.app._impl] [py stderr]:     prim_spec = self._layer.GetPrimAtPath(path)
2025-04-16 18:03:42 [277,387ms] [Error] [omni.kit.app._impl] [py stderr]: Boost.Python.ArgumentError: Python argument types in
    Layer.GetPrimAtPath(Layer, Path)
did not match C++ signature:
    GetPrimAtPath(pxrInternal_v0_22__pxrReserved__::SdfLayer {lvalue}, pxrInternal_v0_22__pxrReserved__::SdfPath)
2

Hi @subhams
please share some details regarding your simulation setup and also the event/user action that caused this issue.

3

Here are the details of the system:

  1. Server: Dell PowerEdge R750

  2. GPUs: 2x NVIDIA RTX 6000 Ada

  3. Driver version: 560.35.03

  4. AODT version: 1.2.0 (installed using recommended installation script)

I was using the to Start UE Mobility function.

4

Hi @subhams
which map are you using? number of RUs and UEs?
did you define any waypoints for UEs?
is this RAN simulation or EM solver only?

5

I am attaching the map here. mit.zip (13.9 MB)

No, we did not define any waypoints for the UEs.

We only tried the EM solver.

Thank you for your help in advance.

6

@subhams
Please provide full console log, located at ~/.local/share/ov/pkg/aodt-1.2.0/logs/Kit/aodt/1.2
Also try to install USD composer (from omniverse launcher) and check if you can open the map in the composer.

7

Here are the logs:
logs.zip (142.7 KB)

I have attached two logs, since it crashed twice with similar error report.

I will check the USD composer.

8

USD composer does not launch from the Omniverse launcher. Is there a different way other than this?

image
image1264×703 113 KB

9

Hi @subhams
We were able to run a simulation successfully using the mit.usd map.
Please try the following to troubleshoot your issue:

  1. First try to open tokyo.usd map and verify you are able to run a basic simulation.
  2. check the proper access rights on your mit.usd map as we noticed this error in the logs:
    “Layer @omniverse://omniverse-server/Users/aerial/mit/mit.usd@ is not editable.”
    also try to place your map in the same folder as other .usd maps.
  3. delete the .live folder using the configuration tab
  4. open the mit.usd map again and try a basic simulation without any waypoint for the UE initially.
10

Hi,

We could use the mit.usd map. But while running the simulation, it intermittently stops and GPUs don’t show up. The captured error logs shows when the GPU error occur. However, from the user perspective, the error is impossible to diagnose.

11

Can you please share the size of your simulation:

  • number of UEs/RUs/antennas per pannel
  • length of simulation
12

Hi @rfarhoodi,

We have used 2 UEs and 1 RU. Here is the scene:

image
image1919×1058 109 KB

The simulation duration is 10s with 0.1s interval.

14

@subhams For the record

  1. The GPU ram was showing a very utilization despite the fact that the aodt container was not running.
  2. The GIS container was hogging up the VRAM, probably because of failed OSM to USD conversion.
  3. Stopping the GIS container, freed up the memory
  4. Now you are able to scale up the AODT.
    I will mark this as closed.