AODT 1.5.1: insertCIRs scans the full RayPath vector per link, CIR export never completes on a 42 RU / 285 UE scene (insertRaypaths is fine)

Accelerated Computing Aerial Forum
1

Worker: nvcr.io/nvidia/aerial/aodt-sim-runtime:1.5.1 (aodt 1.5.1, CUDA 12.9.1), Driver 595.71.05, Ubuntu 22.04 host, single H100. I know from the earlier thread that H100 is not a qualified GPU for 1.5.1; both issues below are CPU-side and do not touch the GPU. Client: dt-client 0.1.0Editable install from the 1.5 client repo, remote gRPC transport.

Batched EM run, 42 RUs, 285 UEs, 1 tile (35,688 buildings, 4.1M triangles), 2 time steps, duration/interval timeline, non-RAN mode. Two runs: first with cirs and raypaths opted in, second with raypaths (full) only.

Run 1: cirs + raypaths. Write phase never finished

EM solve was fine: ~13.3 s and ~13.8 s per time step, ~12.05M paths per step, 57 GB on the GPU.

Log timeline:

14:04:22  [time_sample=0] Adding 12054318 cir results to cirs table
14:04:43  [time_sample=1] Adding 12054318 cir results to cirs table
14:04:29  Finalizing simulation and writing results to DB
14:04:29  Waiting for all DB writes to complete before Parquet flush...

After that, only the 30 s NVML heartbeat for two hours. Two DB worker processes (one per time step), each at 100% of one core, ~117 min CPU each when I killed them. Zero rows in cirs or raypaths in ClickHouse, no INSERT in system.query_log, no socket writes in strace. Not the database.

perf top on a DB worker:

95.51%  libclickhouse.so  [.] 0x2c868
 2.89%  libclickhouse.so  [.] 0x2c883
 0.50%  libclickhouse.so  [.] 0x2c87d

objdump resolves that to insertCIRs(db_settings const&, std::vector<emsolver::RayPath> const&, std::vector<TXInfo> const&, std::vector<RXInfo> const&, std::vector<unsigned> const&, std::vector<std::vector<unsigned>> const&, bool, int), hot loop:

2c868:  add    $0x228,%rbx          ; next RayPath, sizeof = 552
2c86f:  cmp    %rbx,%r14            ; end?
2c878:  cmp    %eax,(%rbx)          ; field @0x00
2c88a:  cmp    %eax,0x4(%rbx)       ; field @0x04
2c893:  cmp    0x8(%rbx),%eax       ; field @0x08
2c89c:  cmp    0xc(%rbx),%eax       ; field @0x0c
2c8a5:  cmp    0x10(%rbx),%eax      ; field @0x10
2c8ae:  cmp    0x14(%rbx),%eax      ; field @0x14

A linear scan over the entire RayPath vector matching six integer fields, once per link and antenna pair. For this scene, that is 42 x 285 x 4 element pairs = ~48k full passes over a 12M element, 6.6 GB array per time step. At that memory bandwidth, that is many hours per step, which matches what I saw. The bool argument (first antenna pair only) divides the pass count by 4, making it unusable for bulk export.

Run 2: raypaths (full) only. Write phase completes in ~90 s

Same scene, cirs removed from opt_in_tables, raypaths: full kept.

17:24:19  Finalizing simulation and writing results to DB
17:24:19  [time_sample=0] Adding 12054318 ray results to raypaths table
17:24:19  Waiting for all DB writes to complete before Parquet flush...
17:24:19  [time_sample=1] Adding 12054318 ray results to raypaths table
17:25:48  [time_sample=0] Rays insert took 89035.6 ms
17:25:48  [time_sample=1] Rays insert took 81367.2 ms
17:26:02  Exported raypaths t0-2: 24108636 rows in 6260.7ms

Both steps inserted in parallel, ~85 s each for 12.05M rows. Parquet export of all 24,108,636 rows took 6.3 s and produced a 983 MiB file with per-path points, normals, interaction_types, prim_ids, and complex amplitude per (ru_ant_el, ue_ant_el) pair. So insertRaypaths is linear and the problem is confined to insertCIRs.

The second arm of the same tile (different tilts, otherwise identical) went load 21.7 s, solve 146 s, insert ~85 s per step, export 23,785,742 rows in 6.1 s. About 3 minutes end-to-end, whereas the first attempt did not finish in 2 hours.

For my purposes, this is the workaround: delay and gain per path are already in the raypaths Parquet, so I can bin CIRs myself in pandas without touching insertCIRs. I will cross-check against Interactive Mode get_cirs on a small subset.

Secondary issue, both runs: 30 min scenario load is node placement

client.start() took 1.955e6 ms (32.6 min) on run 1 and 1.811e6 ms (30.2 min) on run 2 for a 146 KB YAML. Main worker process at 100% of one core the whole time, ~140 other threads busy-waiting. Meshes then loaded in 4.4 s.

Three py-spy dumps of the main worker process during the load, 90 s apart, all show the same stack:

_find_downward_projection     aodt/stage/world_reader.py:328
get_z_from_x_y_per_tile       aodt/stage/world_reader.py:574
get_z_from_x_y                aodt/stage/world_reader.py:480
proj_point_to_cartesian       aodt/fetcher/world_prim.py:130
deploy_nodes_from_sim_config  aodt/fetcher/node_prim.py:439

Locals in get_z_from_x_y_per_tile: include_buildings: True, buildings = a list of every building prim in the tile. For each 2D node, it shoots one downward ray and calls _find_downward_projection once per building prim through a BBoxCache, with no spatial index. That is 35,688 Python-level calls per node. 42 RUs + 285 UEs = 327 nodes, so ~11.7M calls, about 5.5 s per node. The busy threads are the TBB pool under BBoxCache computing bounds on the large meshes.

Workaround, confirmed on two loads: pass alt on every georeferenced node. That takes the dim: 3 branch in proj_point_to_cartesian, which still calls get_z_from_x_y but with include_buildings=False, so only the ground plane is tested. Same 42 RU / 285 UE scene, same tile, load went from 1.81e6 ms to 17,468 ms and 21,661 ms on two runs. RU z in rus.parquet is identical (1250 stage units) either way, so the fast path lands nodes in the same place; alt must be 0.0 because the node height attributes (aerial_gnb_height, set_ues_height) are applied on top of it. Cartesian points with z skip projection entirely.

Suggested fix: build a BVH or a 2D grid over building footprints per tile once at scene load and query that per node.

Questions

  1. Is the insertCIRs pattern known, and does 1.6 change the CIR export path? A single bucketing pass by (link, antenna pair) before the per-link loop would make it linear.
  2. Is Interactive Mode (allocate_cirs_memory / get_cirs / to_numpy_all_cir / export_results) the intended route for bulk CIR extraction at this scale in the meantime, or is reconstructing from the raypaths Parquet the expected approach?

Full worker logs, perf / objdump output, and the py-spy dumps available on request.

2

@aditya.chaudhuri1 in gRPC mode insertCIRS() reads out of the GPU memory, so there is a GPU dependency. How many time steps worth of CIRs are you requesting via gRPC? I will need to look at this in detail, but it looks like the gRPC read might too large. Will get back in a few days.

3

Thanks for looking into this. I checked the submitted config and logs from that run.

The client didn’t request any time steps. dt_server.log shows “42 RUs, 285 UEs, 0 CFR + 0 CIR allocations”, so there was no allocate_cirs_memory or get_cirs call. CIRs came only from the batched opt-in: opt_in_tables was [raypaths, cirs] with raypaths: full. The worker negotiated REMOTE (grpc) transport.

The scenario itself was sim_duration 1.0 s and sim_interval 1.0 s, so 2 time steps. Both panels are dual-polarised: the RU panel is 2x1, and the UE panel is 1x2, so 4 elements each and 16 antenna pairs per link. That comes to 42 x 285 x 16 = 191,520 CIRs per step and 383,040 for the run, with about 12.05M paths per step from the solver.

One correction to my original post: I wrote about 48k passes per step and said the bool divides by 4. With 16 pairs per link, it is about 190k passes per step, and the bool divides by 16.

On the GPU read, both “Adding 12054318 cir results to cirs table” lines were logged within 21 s of each other. After that, the two DB worker processes spent 2 hours at 100% of one core, with perf samples on the loop in insertCIRs that steps through std::vectoremsolver::RayPath. GPU memory stayed at 57 GB, strace showed no socket writes, and ClickHouse received no INSERT. To me, that looks like the data was already on the host and the time went into the per-link scan. If gRPC mode reads from the GPU inside that loop, per link or per antenna pair, that would change the picture, and I would like to know.

With cirs removed, the same scene over the same transport wrote 12.05M raypath rows per step in about 85 s.

I can rerun with 1 time step or a 2 RU subset, with cirs opted in, if that helps.