Unable to debug and profile on Jetson TX2 from remote

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi all,

I have a Jetson TX2 and have installed all the things from SDK Manager in my host (Ubuntu 16.04).

After flashing TX2 and setting cross-compile by the instruction below:

https://devblogs.nvidia.com/cuda-jetson-nvidia-nsight-eclipse-edition/

I am not able to debug any cuda code on remote TX2(using USB port to connect) while strangely I can run the sample code(box filter) without any problem or error on TX2.

Every time I want to step over or step into a cuda code, the debugger hangs on that line, showing log in console below:

Coalescing of the CUDA commands output is off.
warning: "remote:" is deprecated, use "target:" instead.
warning: sysroot set to "target://".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib/ld-linux-aarch64.so.1 from remote target...
Reading /lib/ld-2.27.so from remote target...
Reading /lib/.debug/ld-2.27.so from remote target...
0x0000007fb7fd31c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
$1 = 0xff
The target endianness is set automatically (currently little endian)
Reading /usr/lib/aarch64-linux-gnu/libGL.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libglut.so.3 from remote target...
Reading /lib/aarch64-linux-gnu/librt.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libpthread.so.0 from remote target...
Reading /lib/aarch64-linux-gnu/libdl.so.2 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libstdc++.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libgcc_s.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libGLX.so.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libX11.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libm.so.6 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXi.so.6 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXxf86vm.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libxcb.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXext.so.6 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXau.so.6 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libbsd.so.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/6599f15de1756576e5dcebe6ddf7753c82b62d.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/6599f15de1756576e5dcebe6ddf7753c82b62d.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/e7dfc219d479cecadf71c362dce2f8b0491105.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/e7dfc219d479cecadf71c362dce2f8b0491105.debug from remote target...
Reading /lib/aarch64-linux-gnu/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/libdl-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libdl-2.27.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/a6cec032b9969b1d556b4ebee1400fabda2fdc.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/a6cec032b9969b1d556b4ebee1400fabda2fdc.debug from remote target...
Reading /lib/aarch64-linux-gnu/0a79672e8a81d551322b2912578e0ea9cef6e9.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/0a79672e8a81d551322b2912578e0ea9cef6e9.debug from remote target...
Reading /lib/aarch64-linux-gnu/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libc-2.27.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/f3526220bed678e0de3905b82c8bdeaa972d59.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/f3526220bed678e0de3905b82c8bdeaa972d59.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/8d4674bb6aba390eaccfcbb65d74b1bdd5a1c1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/8d4674bb6aba390eaccfcbb65d74b1bdd5a1c1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/c016049d83424eaa17139fafa41725c498d95c.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/c016049d83424eaa17139fafa41725c498d95c.debug from remote target...
Reading /lib/aarch64-linux-gnu/libm-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libm-2.27.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/d5aee57ec08ec3ed92c14224870c310c6e1f11.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/d5aee57ec08ec3ed92c14224870c310c6e1f11.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/8b97873ad999a4bfd6b95252e51a1186e0cf03.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/8b97873ad999a4bfd6b95252e51a1186e0cf03.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/02e52775ea6a51790d71afd04aebe619fd8e0f.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/02e52775ea6a51790d71afd04aebe619fd8e0f.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXext.so.6.4.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/libXext.so.6.4.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libXau.so.6.0.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/libXau.so.6.0.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/59a8e9cae54ff668fbfd38cb24e2e7eb04a79c.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/59a8e9cae54ff668fbfd38cb24e2e7eb04a79c.debug from remote target...
Reading /lib/aarch64-linux-gnu/16fc2b21aec24cd436caae6372b6376fa69c6a.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/16fc2b21aec24cd436caae6372b6376fa69c6a.debug from remote target...

Temporary breakpoint 1, main (argc=1, argv=0x7ffffff3f8) at ../src/boxFilter.cpp:548
548	{
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.2.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvos.so.debug from remote target...

And I do follow this instruction to set sshd_config:
https://devtalk.nvidia.com/default/topic/1052253/jetson-agx-xavier/no-timeline-for-profiler-xavier-nvvp-and-nsight-compute-is-not-working-in-jetson-xavier/post/5347671/#534767

But nothing’s changed after doing all those settings.

And also when I’m tring to see timeline from profiling toos in Nsight Eclipse, it turns out

showing nothing on the screen.

Can anybody give me some help to solve this problem? Thank you so much.

2

Hi,

You will need the root authority to make profiler work.
Have you login with root first?

Thanks.

3

Hi,

I do have login with root, and I also have run nsight eclipse by using sudo nsight.

Do I have to do any extra works?

Also, I am not able to debug as long as debugger steps into any cuda code such as cudaFree.

Thanks.

4

Hi,

This is a known issue.
Please check this topic for details:
https://devtalk.nvidia.com/default/topic/1056303/jetson-tx2/cuda-api-calls-throw-exceptions-in-remote-debugger-/

The root cause is from cpu-gdb rather than cuda-gdb.
We are going to check this issue after upgrading gdb into v8.2, whcih is planed for CUDA 11.0.

For workaround, please press the “Resume” button to continue the debugging.

Thanks.