Error running inference in nms layers with CUDA-PointPillars

[omeedj02]

I have been attempting to run inference with CUDA-PointPillars, but I run into the following error:

Lidar points count: 201600
==================PointPillars===================
[⏰ [NoSt] CopyLidar]: 1.13382 ms
[⏰ Lidar Voxelization]: 1.00134 ms
[⏰ Lidar Backbone & Head]: 16.67251 ms
[⏰ Lidar Decoder + NMS]: 2.26544 ms
Total: 19.939 ms

Detections after NMS: 4
1722282619196
Saved prediction in: …/data/benewake/1722282619196
Saved results to 1722282619196
Lidar points count: 201600
==================PointPillars===================
[⏰ [NoSt] CopyLidar]: 1.12464 ms
[⏰ Lidar Voxelization]: 0.99965 ms
[⏰ Lidar Backbone & Head]: 16.67779 ms
[⏰ Lidar Decoder + NMS]: 2.86938 ms
Total: 20.547 ms

Detections after NMS: 2
1722282619295
Saved prediction in: …/data/benewake/1722282619295
Saved results to 1722282619295
Lidar points count: 201600
==================PointPillars===================
[⏰ [NoSt] CopyLidar]: 1.10653 ms
[⏰ Lidar Voxelization]: 0.98237 ms
[⏰ Lidar Backbone & Head]: 16.35475 ms
[⏰ Lidar Decoder + NMS]: 2.28144 ms
Total: 19.619 ms

Detections after NMS: 5
1722282619396
Saved prediction in: …/data/benewake/1722282619396
Saved results to 1722282619396
Lidar points count: 201600
==================PointPillars===================
[⏰ [NoSt] CopyLidar]: 1.10365 ms
[⏰ Lidar Voxelization]: 0.99091 ms
[⏰ Lidar Backbone & Head]: 16.34883 ms
CUDA Runtime error nmslaunch(bndbox_num, bndbox, param.nmsthresh, h_mask, _stream) # invalid configuration argument, code = cudaErrorInvalidConfiguration [ 9 ] in file /home/omeed/CUDA-PointPillars/src/pointpillar/lidar-postprocess.cu:470
Aborted

It seems to be triggered every time there are 0 detections from the Lidar Backbone and Head layers going into the Lidar Decoder + NMS layers. I am running everything on CUDA 11.4 on the Jetson Orin with Jetpack 5.1.1. Could there be a dependency issue? It’s also strange that the back-to-back frames are nearly identical but one ends up having 0 detections.

[AastaLLL]

Hi,

The GitHub comes with a Dockerfile.
Could you try to run the inference with the container to see it works?

Thanks.

[omeedj02]

I believe the Dockerfile is only for the steps where you would want to convert a trained model to onnx. The Dockerfile doesn’t seem to match the runtime environment that was specified for inference and I’m having a hard time finding a docker container with both CUDA 11.4 and TensorRT 8.6.11

[AastaLLL]

Hi,

We will give it a try and provide more info to you.

Thanks.

[AastaLLL]

Hi,

Please set up Orin with JetPack 6.0 GA and apply following change to build the repo:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ae8f01b..34e1182 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -22,11 +22,11 @@ set(arch ${CMAKE_HOST_SYSTEM_PROCESSOR})
 
 find_package(CUDA REQUIRED)
 
-set(CMAKE_CXX_FLAGS_RELEASE "-std=c++14 -Wextra -Wall -Wno-missing-field-initializers -Wno-deprecated-declarations -O3")
-set(CMAKE_CXX_FLAGS_DEBUG   "-std=c++14 -O0 -g")
+set(CMAKE_CXX_FLAGS_RELEASE "-std=c++17 -Wextra -Wall -Wno-missing-field-initializers -Wno-deprecated-declarations -O3")
+set(CMAKE_CXX_FLAGS_DEBUG   "-std=c++17 -O0 -g")
 
-set(CUDA_NVCC_FLAGS_RELEASE "-Werror=all-warnings -Xcompiler -std=c++14,-Wextra,-Wall,-Wno-deprecated-declarations,-O3")
-set(CUDA_NVCC_FLAGS_DEBUG   "-std=c++14 -O0 -g")
+set(CUDA_NVCC_FLAGS_RELEASE "-Werror=all-warnings -Xcompiler -std=c++17,-Wextra,-Wall,-Wno-deprecated-declarations,-O3")
+set(CUDA_NVCC_FLAGS_DEBUG   "-std=c++17 -O0 -g")
 
 set(CUDA_NVCC_FLAGS ${CUDA_NVCC_FLAGS} 
     -gencode arch=compute_$ENV{CUDASM},code=compute_$ENV{CUDASM}
diff --git a/tool/environment.sh b/tool/environment.sh
index 771570a..6334e50 100644
--- a/tool/environment.sh
+++ b/tool/environment.sh
@@ -26,10 +26,10 @@ export TensorRT_Lib=/lib/aarch64-linux-gnu/
 export TensorRT_Inc=/usr/include/aarch64-linux-gnu/
 export TensorRT_Bin=/usr/src/tensorrt/bin/
 
-export CUDA_Lib=/usr/local/cuda-11.4/targets/aarch64-linux/lib/
-export CUDA_Inc=/usr/local/cuda-11.4/targets/aarch64-linux/include/
-export CUDA_Bin=/usr/local/cuda-11.4/bin/
-export CUDA_HOME=/usr/local/cuda-11.4/
+export CUDA_Lib=/usr/local/cuda/targets/aarch64-linux/lib/
+export CUDA_Inc=/usr/local/cuda/targets/aarch64-linux/include/
+export CUDA_Bin=/usr/local/cuda/bin/
+export CUDA_HOME=/usr/local/cuda/
 
 export CUDNN_Lib=/lib/aarch64-linux-gnu/

To verify, please follow our steps below to test it with the Kitti dataset:
https://github.com/NVIDIA-AI-IOT/CUDA-PointPillars/blob/main/tool/eval/README.md

Thanks.