2019.3 Kaya example run failure on fresh install

Autonomous Machines Robotics - Isaac Isaac SDK
1

We are evaluating the 2019.3 release and our first kaya test fails to run on fresh install. Has something to do with the distance map. It fails on our teams machines and I tried it on my own machine, so its not an isolated failure.

2020-01-22 13:14:54.719 PANIC engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 38

| Isaac application terminated unexpectedly |

#01 engine/alice/tools/main(+0x880fa) [0x5581c94c70fa]
#02 engine/alice/tools/main(+0xf97d0) [0x5581c95387d0]
#03 engine/alice/tools/main(+0xf9b40) [0x5581c9538b40]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fcf40986890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fcf40020e97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fcf40022801]
#07 /home/andrew/.cache/bazel/_bazel_andrew/167f886c5ee5ea2e6c1de6c7748c623c/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/kaya/navigate.runfiles/com_nvidia_isaac//packages_x86_64/navigation/libnavigation_module.so(+0x47c8d6) [0x7fcf3d8b68d6]
#08 isaac::navigation::DistanceMap::setDistanceMap(isaac::partitions::GridMap<float, float> const&) /home/andrew/.cache/bazel/_bazel_andrew/167f886c5ee5ea2e6c1de6c7748c623c/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/kaya/navigate.runfiles/com_nvidia_isaac//packages_x86_64/navigation/libnavigation_module.so(_ZN5isaac10navigation11DistanceMap14setDistanceMapERKNS_10partitions7GridMapIffEE+0x21e) [0x7fcf3d67d8be]
#09 engine/alice/tools/main(+0xa4daf) [0x5581c94e3daf]
#10 engine/alice/tools/main(+0xa6ac2) [0x5581c94e5ac2]
#11 engine/alice/tools/main(+0xa7bfd) [0x5581c94e6bfd]
#12 engine/alice/tools/main(+0xa7d25) [0x5581c94e6d25]
#13 engine/alice/tools/main(+0xea4c6) [0x5581c95294c6]
#14 engine/alice/tools/main(+0x1c61a) [0x5581c945b61a]
#15 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd66f) [0x7fcf406a866f]
#16 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fcf4097b6db]
#17 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fcf4010388f]

2

Hello Fedoralabs, just confirmed that we are able to run all kaya apps with 2019.3 release. Can you check your set up?

Seems like others are able to run Kaya app for e.g. https://devtalk.nvidia.com/default/topic/1066964/isaac-sdk-isaac-sim-for-navigation/robot-not-moving-during-kaya-waste-sorting-/post/5420001/#5420001

The reported error: 2020-01-22 13:14:54.719 PANIC engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 38; could be caused by a low-memory condition on Jetson Nano. Please check the amount of free ram before running the app.

3

Hi Shri,

It turns out it was a kernel issue. Currently L4T is running a 4.9 kernel version. Ubuntu 18.04 LTS is currently coming with a 5.0+ kernel (currently ~5.3). When testing from a workstation (as the team was in this case) this causes this issue but makes it appear as a low memory issue as you had pointed out. Rolling the kernel back to 4.15 on the workstation fixed the issue.

I would also like to note for other users that the 5.0+ kernel does cause issues with other hardware components that are very difficult to debug as the error output is irrational (such as in this case showing as a memory issue).

If you can please note this in the isaac setup documentation to match the worksation kernel with the L4T kernel version or less this will help a lot of people running into these strange error outputs.

4

Thank you FedoraLabs.