Hello , i am making a Custom Codelet and codelet builds successfully using Bazel , when deployted on Jetson Device , i am geting Core Dump ( Segmentation Fault )
i have tried analysing the minidump file and did not get any meaningful result .
What are my debug options ?
bazel build -c dbg gives "Couldn’t build file external/redis/_objs/redis-server-lib/evict.o " error and build fails.
If you run the deploy.sh script with -s, you should be able to get a symbolicated minidump stack trace on crash. When you say meaningful result from minidump, is it that it showed threads were doing nothing interesting or the result was obfuscated?
You could try starting the program on Jetson using gdb to catch the segfault live and get a stack trace, or worse but effective, resort to some printf debugging. If you think the segfault is in your codelet, have you tried running it in isolation on x86 in a unit test?
hi , i am trying to do bazel build … -c dbg --cpu -x86 for library generation , which can be debugged using vs code !! But i am getting an error no X86 target found !! Pls let me know,how can i debug the codelet , using IDE.
The “dbg” build configuration is likely going to turn off most of the compiler optimizations, add in memory guard instructions, and avoid stripping symbols from the binary. That will all help when debugging, especially with a debugger, and may catch your segfault earlier (at the time of corruption rather than at a subsequent read). VS Code has a number of tutorials for how to configure debugging, but it will invoke gdb or lldb in the end, so it might be easier to just run the debugger yourself from shell. For CPU spec to bazel, “x86_64” is probably what the target is named rather than “x86”
by going through the terminal do you mean i should use standard debug tools build into Isaac SDK ?
like :- gdb – args bazel-bin/engine/alice/tools/main --app " Application Path" ?
problem is when i am trying to build my package using bazel build -c dbg (App Path ) , i am getting these errors ( Something to do with Redis Server, so i am stuck , i now have no clue , how to proceed. :-
ERROR: /home/octobotics/.cache/bazel/_bazel_octobotics/5372c8bdc184531a09ffefb7ae658daa/external/redis/BUILD.bazel:77:1: Couldn’t build file external/redis/_objs/redis-server-lib/evict.o: C++ compilation of rule ‘@redis//:redis-server-lib’ failed (Exit 1) crosstool_wrapper_driver_is_not_gcc_host.py failed: error executing command external/toolchain/crosstool/scripts/crosstool_wrapper_driver_is_not_gcc_host.py -MD -MF bazel-out/k8-dbg/bin/external/redis/_objs/redis-server-lib/evict.d … (remaining 46 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
external/redis/src/evict.c: In function ‘evictionPoolPopulate’:
external/redis/src/evict.c:188:18: error: ‘o’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
idle = estimateObjectIdleTime(o);
~^
external/redis/src/evict.c: In function ‘freeMemoryIfNeeded’:
external/redis/src/evict.c:554:27: error: ‘bestdbid’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
db = server.db+bestdbid;
^
cc1: all warnings being treated as errors
ERROR: /home/octobotics/.cache/bazel/_bazel_octobotics/5372c8bdc184531a09ffefb7ae658daa/external/redis/BUILD.bazel:77:1: Couldn’t build file external/redis/_objs/redis-server-lib/geo.o: C++ compilation of rule ‘@redis//:redis-server-lib’ failed (Exit 1) crosstool_wrapper_driver_is_not_gcc_host.py failed: error executing command external/toolchain/crosstool/scripts/crosstool_wrapper_driver_is_not_gcc_host.py -MD -MF bazel-out/k8-dbg/bin/external/redis/_objs/redis-server-lib/geo.d … (remaining 46 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
external/redis/src/geo.c: In function ‘georadiusGeneric’:
external/redis/src/geo.c:654:26: error: ‘zs’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
serverAssert(dictAdd(zs->dict,gp->member,&znode->score) == DICT_OK);
^~~~~~~
cc1: all warnings being treated as errors
Target //apps/samples/zed_camera:zed_camera failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.386s, Critical Path: 0.15s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
I have tried doing the Minidump Crash Analysis , by generating symbols and then reading the crash file using the script provided. :-
Operating system: Linux
0.0.0 Linux 4.9.140-tegra #1 SMP PREEMPT Thu Jun 25 21:22:12 PDT 2020 aarch64
CPU: arm64
8 CPUs
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x6f8
Process uptime: not available
Thread 8 (crashed)
0 libpthread.so.0 + 0x96f0
x0 = 0x00000000000006e8 x1 = 0x0000000000000000
x2 = 0x0000007f9a3377f0 x3 = 0x0000000000005cc8
x4 = 0x00000000ffffffff x5 = 0x0000007f7a4ed1c0
x6 = 0x0000000000000004 x7 = 0x0000007f9a3384d0
x8 = 0x0000007f9a337950 x9 = 0x0000007f9a3383d0
x10 = 0x0101010101010101 x11 = 0x0000000000000000
x12 = 0x0000005588a8f1c0 x13 = 0x0000000000000438
x14 = 0x0000007fa1fee848 x15 = 0x0000007fa1fd5510
x16 = 0x0000007fa549e430 x17 = 0x0000007fae30f6f0
x18 = 0x0000000000000fbe x19 = 0x0000007fa5498000
x20 = 0x0000000000000000 x21 = 0x0000000000000000
x22 = 0x0000007f9a337950 x23 = 0x0000007f7a4ed1c0
x24 = 0x00000000000006e8 x25 = 0x0000000000000001
x26 = 0x0000007f7a4f2b10 x27 = 0x0000007f9a337630
x28 = 0x0000007f9a337950 fp = 0x0000007f9a336cb0
lr = 0x0000007fa21aa3f4 sp = 0x0000007f9a336cb0
pc = 0x0000007fae30f6f0
Found by: given as instruction pointer in context
1 libsl_zed.so!sl::ZEDmcu::getIMUPose(int) + 0x38
fp = 0x0000007f9a337290 lr = 0x0000007fa2a79198
sp = 0x0000007f9a336cc0 pc = 0x0000007fa21aa3f4
Found by: previous frame’s frame pointer
2 libsl_zed.so!sl::Camera::getSensorsData(sl::SensorsData&, sl::TIME_REFERENCE) + 0x50
x19 = 0x0000007f7808a820 x20 = 0x0000007f9a337dc0
x21 = 0x0000007fa59fb000 x22 = 0x0000005588add230
x24 = 0x0000007f7808a820 fp = 0x0000007f9a337ab0
sp = 0x0000007f9a337870 pc = 0x0000007fa2a52dac
Found by: call frame info
3 libzed_module.so + 0x12f868
x19 = 0x0000007f9a337f88 x20 = 0x0000007f9a337dc0
x21 = 0x0000007fa59fb000 x22 = 0x0000005588add230
x24 = 0x0000007f7808a820 fp = 0x0000007f9a337ad0
sp = 0x0000007f9a337ad0 pc = 0x0000007fa568586c
Found by: call frame info
Thread 0
0 libpthread.so.0 + 0x11584
x0 = 0x0000007ffc599f28 x1 = 0x0000007ffc599f28
x2 = 0x0000000000000000 x3 = 0x0000007fae4d73c0
x4 = 0x00000000ffffffbb x5 = 0x0000007fae4d6bc0
x6 = 0x0000000000000000 x7 = 0x0000005588ae3fa0
x8 = 0x0000000000000065 x9 = 0x000000558885cbf0
x10 = 0x000000558885cc48 x11 = 0x000000558885cc70
x12 = 0x00000055888571a0 x13 = 0x000000558885cc48
x14 = 0x0000000000000008 x15 = 0x0000007fae306e00
x16 = 0x00000055791309f0 x17 = 0x0000007fae317528
x18 = 0x0000000000000139 x19 = 0x0000007ffc599f28
x20 = 0x0000007ffc599f28 x21 = 0x0000005579131000
x22 = 0x0000007ffc59a010 x23 = 0x0000000000000000
x24 = 0x0000000000000000 x25 = 0x0000000000000000
x26 = 0x0000000000000000 x27 = 0x0000000000000000
x28 = 0x0000000000000000 fp = 0x0000007ffc599ee0
lr = 0x0000007fae317574 sp = 0x0000007ffc599ee0
pc = 0x0000007fae317584
Found by: given as instruction pointer in context
1 libpthread.so.0 + 0x11570
fp = 0x0000007ffc599f00 lr = 0x0000005579026c48
sp = 0x0000007ffc599ef0 pc = 0x0000007fae317574
Found by: previous frame’s frame pointer
2 main + 0x14cc44
fp = 0x0000007ffc599f40 lr = 0x0000005578f2a328
sp = 0x0000007ffc599f10 pc = 0x0000005579026c48
Found by: previous frame’s frame pointer
3 main + 0x50324
fp = 0x0000007ffc599f80 lr = 0x0000005578efd754
sp = 0x0000007ffc599f50 pc = 0x0000005578f2a328
Found by: previous frame’s frame pointer
4 main + 0x23750
fp = 0x0000007ffc59a2f0 lr = 0x0000007fae0156e0
sp = 0x0000007ffc599f90 pc = 0x0000005578efd754
Found by: previous frame’s frame pointer
5 libc.so.6 + 0x206dc
fp = 0x0000000000000000 lr = 0x0000005578f09ab8
sp = 0x0000007ffc59a300 pc = 0x0000007fae0156e0
Found by: previous frame’s frame pointer
Thread 1
0 libpthread.so.0 + 0xd22c
x0 = 0x00000055889ef388 x1 = 0x0000000000000080
x2 = 0x0000000000000000 x3 = 0x0000000000000000
x4 = 0x00000000ffffffbb x5 = 0x0000007f9f09ce10
x6 = 0x0000000000000000 x7 = 0x0000000000000000
x8 = 0x0000000000000062 x9 = 0x0000000000000000
x10 = 0x0101010101010101 x11 = 0x0000000000000000
x12 = 0x0000007f9f09cf20 x13 = 0x000000000000270f
x14 = 0x0000007fae308c90 x15 = 0x0000007fae306e00
x16 = 0x0000007fae312ff8 x17 = 0x0000007f9f260170
x18 = 0x00000000000000e4 x19 = 0x00000055889ef370
x20 = 0x0000007fae32c000 x21 = 0x00000055889ef2f8
x22 = 0x0000000000000000 x23 = 0x0000007fae312f40
x24 = 0x0000000000000000 x25 = 0x0000000000000000
x26 = 0x00000055889ef388 x27 = 0x0000000000000000
x28 = 0x00000055889ef360 fp = 0x0000007f9f09c5a0
lr = 0x0000007fae313210 sp = 0x0000007f9f09c5a0
pc = 0x0000007fae31322c
Found by: given as instruction pointer in context
1 libpthread.so.0 + 0xd20c
fp = 0x0000007f9f09c720 lr = 0x0000007f9f238430
sp = 0x0000007f9f09c5b0 pc = 0x0000007fae313210
Found by: previous frame’s frame pointer
2 libdrm.so.2 + 0x642c
fp = 0x0000000000000000 lr = 0x0000007fae0c54ec
sp = 0x0000007f9f09c730 pc = 0x0000007f9f238430
Found by: previous frame’s frame pointer
Thread 2
0 libpthread.so.0 + 0xd22c
x0 = 0x00000055889ef358 x1 = 0x0000000000000080
x2 = 0x0000000000000000 x3 = 0x0000000000000000
x4 = 0x00000000ffffffbb x5 = 0x0000007f9e89be10
x6 = 0x0000000000000000 x7 = 0x0000000000000000
x8 = 0x0000000000000062 x9 = 0x0000000000000000
x10 = 0x0000007fae30cfd8 x11 = 0x00000000003d0f00
x12 = 0x0000007f9e89bf20 x13 = 0x000000000000270f
x14 = 0x0000000000000002 x15 = 0x0000000000000000
x16 = 0x0000007f9f260170 x17 = 0x0000007fae312ff8
x18 = 0x0000007fae148a70 x19 = 0x00000055889ef340
x20 = 0x0000007fae32c000 x21 = 0x00000055889ef2f8
x22 = 0x0000000000000000 x23 = 0x0000007fae312f40
x24 = 0x0000000000000000 x25 = 0x0000000000000000
x26 = 0x00000055889ef358 x27 = 0x0000000000000000
x28 = 0x00000055889ef330 fp = 0x0000007f9e89b5b0
lr = 0x0000007fae313210 sp = 0x0000007f9e89b5b0
pc = 0x0000007fae31322c
Found by: given as instruction pointer in context
1 libpthread.so.0 + 0xd20c
fp = 0x0000007f9e89b720 lr = 0x0000007f9f238700
sp = 0x0000007f9e89b5c0 pc = 0x0000007fae313210
Found by: previous frame’s frame pointer
2 libdrm.so.2 + 0x66fc
fp = 0x0000000000000000 lr = 0x0000007fae0c54ec
sp = 0x0000007f9e89b730 pc = 0x0000007f9f238700
Found by: previous frame’s frame pointer
Loaded modules:
0x5578eda000 - 0x5579117fff main ??? (main) (WARNING: No symbols, main, 7809BF768A2695408006CB5E02AA31CE0)
0x7f9ab86000 - 0x7f9bc15fff libnvidia-eglcore.so.32.4.3 ???
0x7f9f152000 - 0x7f9f158fff libffi.so.6 ???
0x7f9f16a000 - 0x7f9f178fff libwayland-server.so.0 ???
0x7f9f18b000 - 0x7f9f195fff libwayland-client.so.0 ???
0x7f9f1a8000 - 0x7f9f1affff libnvidia-egl-wayland.so.1 ???
0x7f9f1c3000 - 0x7f9f1d0fff libdrm.so.2 ???
0x7f9f1e2000 - 0x7f9f1e5fff libnvimp.so ???
0x7f9f1f7000 - 0x7f9f204fff libnvdc.so ???
0x7f9f216000 - 0x7f9f21cfff libnvll.so ???
0x7f9f232000 - 0x7f9f24ffff libdrm.so.2 ??? (WARNING: No symbols, libdrm.so.2, 87BC7ECA7E6EAB7E6AD7C00674EAE46F0)
0x7f9f266000 - 0x7f9f28bfff libnvidia-rmapi-tegra.so.32.4.3 ???
0x7f9f2a8000 - 0x7f9f31ffff libnvidia-glsi.so.32.4.3 ???
0x7f9f355000 - 0x7f9f422fff libEGL_nvidia.so.0 ???
0x7f9f480000 - 0x7f9f486fff gconv-modules.cache ???
0x7f9f497000 - 0x7f9f4a6fff libbsd.so.0 ???
0x7f9f4b9000 - 0x7f9f58cfff libGLdispatch.so.0 ???
0x7f9f5e5000 - 0x7f9f5e8fff libXdmcp.so.6 ???
0x7f9f5fa000 - 0x7f9f5fbfff libXau.so.6 ???
0x7f9f60d000 - 0x7f9f618fff libnvos.so ???
0x7f9f62b000 - 0x7f9f62cfff libnvbuf_fdmap.so.1.0.0 ???
0x7f9f63e000 - 0x7f9f648fff libnvddk_vic.so ???
0x7f9f65c000 - 0x7f9f66bfff libEGL.so.1 ???
0x7f9f67d000 - 0x7f9f69bfff libxcb.so.1 ???
0x7f9f6ad000 - 0x7f9f6ccfff libv4lconvert.so.0 ???
0x7f9f732000 - 0x7f9f77bfff libnvidia-fatbinaryloader.so.440.18 ???
0x7f9f7a1000 - 0x7f9f7aefff libnvrm_graphics.so ???
0x7f9f7c1000 - 0x7f9f7f1fff libnvrm.so ???
0x7f9f804000 - 0x7f9f832fff libnvrm_gpu.so ???
0x7f9f847000 - 0x7f9f872fff libgomp.so.1 ??? (WARNING: No symbols, libgomp.so.1, DCACB4B9DD72EB93D1EC984AC0C106B40)
0x7f9f884000 - 0x7f9f88bfff libnvbuf_utils.so.1.0.0 ???
0x7f9f89e000 - 0x7f9f9b1fff libX11.so.6 ???
0x7f9f9c7000 - 0x7f9f9d0fff libv4l2.so.0 ???
0x7f9fadc000 - 0x7f9faeffff libusb-1.0.so.0 ???
0x7f9fb02000 - 0x7f9fb40fff libturbojpeg.so.0 ???
0x7f9fb53000 - 0x7f9fb8afff libjpeg.so.8 ???
0x7f9fb9d000 - 0x7f9fbb8fff libz.so.1 ???
0x7f9fbca000 - 0x7f9fbf3fff libpng16.so.16 ???
0x7f9fc05000 - 0x7f9fc64fff libnppisu.so.10 ???
0x7f9fc79000 - 0x7fa1049fff libnppist.so.10 ???
0x7fa1060000 - 0x7fa1dd1fff libcuda.so.1 ??? (WARNING: No symbols, libcuda.so.1, 8D5899C979F71EDF7D1E3E63486961A30)
0x7fa1fa3000 - 0x7fa1fbbfff libudev.so.1 ???
0x7fa1fcd000 - 0x7fa5463fff libsl_zed.so ???
0x7fa5556000 - 0x7fa59ddfff libzed_module.so ??? (WARNING: No symbols, libzed_module.so, C2262875176E0C6431F8F1760DF2B6430)
0x7fa5a06000 - 0x7fa6004fff libsight_module.so ??? (WARNING: No symbols, libsight_module.so, 4693A586400F1374799769D4185A3B2A0)
0x7fa6042000 - 0x7fa60a3fff libnppc.so.10 ???
0x7fa60b9000 - 0x7fa69b9fff libnpps.so.10 ???
0x7fa69ef000 - 0x7fa8542fff libnppig.so.10 ???
0x7fa855e000 - 0x7fab91cfff libnppif.so.10 ???
0x7fab93c000 - 0x7fac0effff libnppidei.so.10 ???
0x7fac124000 - 0x7facc05fff libnppial.so.10 ???
0x7facc1b000 - 0x7fad0b2fff libnppicc.so.10 ???
0x7fad0cb000 - 0x7fad8b6fff libperception_module.so ???
0x7fad8e0000 - 0x7fad8e1fff libi2c.so.0 ???
0x7fad8f3000 - 0x7fadcb8fff libimu_module.so ???
0x7fadcdf000 - 0x7fadfbcfff locale-archive ???
0x7fadfbd000 - 0x7fadfc6fff libnss_files.so.2 ???
0x7fadfde000 - 0x7fadfe3fff librt.so.1 ???
0x7fadff5000 - 0x7fae134fff libc.so.6 ??? (WARNING: No symbols, libc.so.6, 1D0CF453372FAE39017DCDCEF1A173140)
0x7fae14e000 - 0x7fae160fff libgcc_s.so.1 ???
0x7fae172000 - 0x7fae2e6fff libstdc++.so.6 ??? (WARNING: No symbols, libstdc++.so.6, AF20639DACBBC08654B0FB6C107DE2940)
0x7fae306000 - 0x7fae31cfff libpthread.so.0 ??? (WARNING: No symbols, libpthread.so.0, 73F347944609C11C5FB1915BC198D7180)
0x7fae332000 - 0x7fae3dafff libm.so.6 ???
0x7fae3ec000 - 0x7fae3eefff libdl.so.2 ???
0x7fae425000 - 0x7fae484fff libcudart.so.10.2 ???
0x7fae499000 - 0x7fae49ffff libnvToolsExt.so.1 ???
0x7fae4b2000 - 0x7fae4cefff ld-linux-aarch64.so.1 ???
0x7fae4d2000 - 0x7fae4d3fff nvhost-ctrl-gpu ???
0x7fae4dd000 - 0x7fae4ddfff linux-gate.so ???
This is helpful to narrow it down. It looks like the crash is in the Zed library which could be for a variety of reasons including some reference given to the function call was deallocated or something worse. The call itself is likely issued by line 60 in //packages/zed/components/ZedImuReader.cpp which you should have source to in 2020.1NX. The crash itself seems to be further into that call where the Zed library is pulling IMU data. Of course, the error or memory corruption could have been further back somewhere and just happen to get noticed by the OS when calling the Zed library.
Does this happen repeatably? Anything unusual in how you are using the ZedImuReader codelet?