CUDA memory errors on Xavier when running Isaac

Autonomous Machines Robotics - Isaac Isaac SDK
,
1

Hello,

I’ve written an Isaac interface library using Isaac’s C API that is built and deployed on Xavier using Isaac’s deploy.sh script. I link to this library in my existing application for Xavier. My application’s basic functionality is to interface with Argus and use the CUDA Driver API and EGL Stream framework to consume and process Argus-produced frames in CUDA and stream over the network. In parallel to that pipeline, I am trying to publish CUDA-processed images to Isaac to detect fiducials via the April Tags GEM.

The Isaac integration appears to working fine inside my application, behaving normally until I try to use the April Tags GEM. Shortly after successfully publishing an image to the AprilTags GEM, I get several cudaErrorMemoryAllocation errors in “/home/nvidia/AprilTagging/april_tagging/corner_detection/corner_detector.cpp”, and then Isaac terminates with the following error:

2020-04-23 15:33:52.508 PANIC ./engine/gems/image/utils.hpp@100: row count mismatch: 540 vs 0

| Isaac application terminated unexpectedly |

There are several GBytes worth of memory space on the device still, so lack of memory is not the issue. I am wondering if there is some sort of conflict between the CUDA Driver API usage in my application, and the AprilTags CUDA code that is trying to run. Any advice would be greatly appreciated.

P.s. I should note that although my interface library compiles and deploys without any errors, when I first tried to run my application, I got en error that Isaac couldn’t find “libAprilTagsDetection_module.so”. I am not sure why it is looking for that filename, as the deployed library is “libapril_tags_module.so”. In any case, I created a symlink and this appeared to solve the issue. I don’t think it’s related to the problem above, but wanted to mention it anyway.

2

Hi,

I’ve done some more testing using a simple C++ application I wrote that links my Isaac interface library. It loads an image from disk using openCV, which I then copy to a uint8_t buffer and publish to Isaac. I am seeing the same problem as above, both on the Xavier and on my host machine. So it appears NOT to be related to my CUDA usage as mentioned above.

Perhaps there is a problem with my Isaac interface library? I should note that I publish the image to isaac.viewers.ColorCameraViewer/color_listener, as well as isaac.perception.AprilTagsDetection/image, and the image displays fine on the Tag in Sight that is connected to isaac.viewers.ColorCameraViewer/Color – I used sample code from the c_api_viewer example to build the message w/ appended image buffer. I should note that for some reason, the ColorCameraViewer Tag on Sight does not refresh after displaying the first frame/image, however I see the same problem when running the c_api_viewer example directly (not sure what the issue is).

3

Just checking in… any suggestions would be appreciated.