Many issues with nsight nvidia

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello,

I cope with many issues with nvidia nsight.

I am working from linux 18.04.4 LTS host. I’ve and install SDK on jetson TX2 from that host.
I’ve connected TX2 and host via SSH.

I tried to develop code from the host, using synchronisation, not cross compilation.
I build in debug/release mode, and run on the target.
It is impossible to use debugger, profiler and even add libraries. In fact, I can’t do anything.

Here information about my jetson TX2 2 :
NVIDIA Jetson TX2
L4T 32.3.1 [ JetPack 4.3 ]
Ubuntu 18.04.3 LTS
Kernel Version: 4.9.140-tegra
CUDA 10.0.326

I run the example AddVector.
I’ve enable root access for ssh for profiler, and I have the following message :

Result windows :
No timeline
Application timeline is required for the analysis

Console windows :
/bin/sh: 1: cd: can’t cd to /home/nl37/RemoteSystemsTempFiles/192.168.20.21/home/jetson/Desktop/Test_Remote/Debug

Moreother, for example, when I tried to include opencv, I do :
1 - libaries
jetson@jetson-desktop:~$ locate highgui.so
/usr/lib/aarch64-linux-gnu/libopencv_highgui.so
/usr/lib/aarch64-linux-gnu/libopencv_highgui.so.4.1
/usr/lib/aarch64-linux-gnu/libopencv_highgui.so.4.1.1

properties->Build->Setting->Tool Setting
Libraries (-l) : add highgui
Library search path (-L) : /usr/lib/aarch64-linux-gnu
2-Headers
jetson@jetson-desktop:~$ locate highgui.hpp
/usr/include/opencv4/opencv2/highgui.hpp
/usr/include/opencv4/opencv2/highgui/highgui.hpp

properties->C/C++ General->Path and Symbol->Include
Then, I add /usr/include/opencv4, for all language.

I add the following header in the code :
#include<opencv2/highgui/highgui.hpp>
When I build the sample, I obtain the following error :
/usr/local/cuda-10.0/bin/nvcc --cudart static -L/usr/lib/aarch64-linux-gnu --relocatable-device-code=false -gencode arch=compute_60,code=compute_60 -gencode arch=compute_60,code=sm_60 -m64 -ccbin aarch64-linux-gnu-g++ -link -o “Test” ./src/vectorAdd.o -lhighgui
/usr/bin/ld: cannot find -lhighgui

About the debugger, I heard that it is because of cuda version, I have to wait version 10.1. When will you release that version for TX2 ?

Thank you,

3

Hi,

I am going to find an Ubuntu18.04 host and give it a try.
Will update more information with you later.

Thanks.

4

Hi AastaLLL,

Many thanks for your help !

5

Hello AastaLLL,

Did have time to test with an Ubuntu 18.04 host ?

Thank you

6

Hi,

Sorry that it took me some time to setup the Ubuntu host.

Confirm that the vectorAdd sample can run with Nsight EE without issue.
Here are the detail steps for your reference:

1. Setup device

2. Setup host

  • Please install CUDA toolkit from JetPack 4.3.
  • Please install OpenJDK 8 from this document.
  • Launch nsight with this commnad:
    $ /usr/local/cuda-10.0/bin/nsight -vm /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

3. Create a vectorAdd project

  • File → New → CUDA C/C++ Project → Type project name → Import CUDA sample → Next
  • vectorAdd → Next
  • Generate PTX code → select 5.3(Nano), 6.2(TX2), 7.2(Xavier)
    Generate GPU code ->select 5.3(Nano), 6.2(TX2), 7.2(Xavier)
    → Next
  • Select remote connection … → Manage → Add
    Host name: IP ex.192.168.1.xxx
    User name: root
    → Finish
  • In the root@192.168.1.xxx window
    Project Path → Browse → Enter device password → /home/nvidia/cuda-workspace
    Toolkit → Manage → Detect → /home/nvidia/cuda-workspace → Finish
    CPU Architecture → AArch64
    → Finish

4. Compile

5. Run

  • Run as → Run as → 2. Remote C/C++ application → Finish
    IMG_7145
    IMG_71452016×1512 1.46 MB
  • You should be able to see the running log like this
echo $PWD'>'
/bin/sh -c "cd \"/home/nvidia/cuda-workspace/Debug\";export LD_LIBRARY_PATH=\"/usr/local/cuda-10.0/lib64\":\${LD_LIBRARY_PATH};export NVPROF_TMPDIR=\"/tmp\";\"/home/nvidia/cuda-workspace/Debug/VectorAdd\"";exit
root@tx2:~# echo $PWD'>'
/root>
root@tx2:~#
/bin/sh -c "cd \"/home/nvidia/cuda-workspace/Debug\";export LD_LIBRARRY_PATH=\"/usr/local/cuda-10.0/lib64\":\${LD_LIBRARY_PATH};export NVPROF_TMPDIR=\\"/tmp\";\"/home/nvidia/cuda-workspace/Debug/VectorAdd\"";exit
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
Logout

Thanks.

1 Like
7

Hi,

Thank you for your post.
Actually, I can run. My questions was about :
-debugger ;
-profiler.

As I said, I’ve already enabled root, but the profiler doesn’t work. Can you profile ?
And about the debugger ?

Thank you

8

Hi,

YES.

Debugger

Coalescing of the CUDA commands output is off.
warning: "remote:" is deprecated, use "target:" instead.
warning: sysroot set to "target://".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib/ld-linux-aarch64.so.1 from remote target...
Reading /lib/ld-2.27.so from remote target...
Reading /lib/.debug/ld-2.27.so from remote target...
0x0000007fb7fd31c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
$1 = 0xff
The target endianness is set automatically (currently little endian)
Reading /lib/aarch64-linux-gnu/librt.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libpthread.so.0 from remote target...
Reading /lib/aarch64-linux-gnu/libdl.so.2 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libstdc++.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libgcc_s.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libm.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/libdl-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libdl-2.27.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/d7646e96801c7eed3642d3c10e301e0f3ea553.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/d7646e96801c7eed3642d3c10e301e0f3ea553.debug from remote target...
Reading /lib/aarch64-linux-gnu/4bfa7077953acb0e38a1039923ecb5fe9f6a62.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/4bfa7077953acb0e38a1039923ecb5fe9f6a62.debug from remote target...
Reading /lib/aarch64-linux-gnu/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/libm-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libm-2.27.so from remote target...
Temporary breakpoint 2, main () at ../src/vectorAdd.cu:48
48 ../src/vectorAdd.cu: No such file or directory.
Breakpoint 1, main () at ../src/vectorAdd.cu:58
58 in ../src/vectorAdd.cu
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.3.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvos.so.debug from remote target...
[Inferior 1 (process 10721) exited normally]
Quit

Profile

Thanks.

9

Hi,

Many thanks for that !!

Ok so the problem is from the configuration …
I followed the same procedure, and that doesn’t work.

Here, the sshd_config of the jetson :

> #$OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp$
> #This is the sshd server system-wide configuration file.  See
> #sshd_config(5) for more information.
> #This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
> #The strategy used for options in the default sshd_config shipped with
> #OpenSSH is to specify options with their default value where
> #possible, but leave them commented.  Uncommented options override the
> #default value.
> #Port 22
> #AddressFamily any
> #ListenAddress 0.0.0.0
> #ListenAddress ::
> #HostKey /etc/ssh/ssh_host_rsa_key
> #HostKey /etc/ssh/ssh_host_ecdsa_key
> #HostKey /etc/ssh/ssh_host_ed25519_key
> #Ciphers and keying
> #RekeyLimit default none
> #Logging
> #SyslogFacility AUTH
> #LogLevel INFO
> #Authentication:
> #LoginGraceTime 2m
> #PermitRootLogin prohibit-password
> PermitRootLogin yes
> #StrictModes yes
> #MaxAuthTries 6
> #MaxSessions 10
> #PubkeyAuthentication yes
> #Expect .ssh/authorized_keys2 to be disregarded by default in future.
> #AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2
> #AuthorizedPrincipalsFile none
> #AuthorizedKeysCommand none
> #AuthorizedKeysCommand none
> #AuthorizedKeysCommandUser nobody
> #For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
> #HostbasedAuthentication no
> #Change to yes if you don't trust ~/.ssh/known_hosts for
> #HostbasedAuthentication
> #IgnoreUserKnownHosts no
> #Don't read the user's ~/.rhosts and ~/.shosts files
> #IgnoreRhosts yes
> #To disable tunneled clear text passwords, change to no here!
> #PasswordAuthentication yes
> PermitEmptyPasswords no
> #Change to yes to enable challenge-response passwords (beware issues with
> #some PAM modules and threads)
> ChallengeResponseAuthentication no
> #Kerberos options
> #KerberosAuthentication no
> #KerberosOrLocalPasswd yes
> #KerberosTicketCleanup yes
> #KerberosGetAFSToken no
> #GSSAPI options
> #GSSAPIAuthentication no
> #GSSAPICleanupCredentials yes
> #GSSAPIStrictAcceptorCheck yes
> #GSSAPIKeyExchange no
> #Set this to 'yes' to enable PAM authentication, account processing,
> #and session processing. If this is enabled, PAM authentication will
> #be allowed through the ChallengeResponseAuthentication and
> #PasswordAuthentication.  Depending on your PAM configuration,
> #PAM authentication via ChallengeResponseAuthentication may bypass
> #the setting of "PermitRootLogin without-password".
> #If you just want the PAM account and session checks to run without
> #PAM authentication, then enable this but set PasswordAuthentication
> #and ChallengeResponseAuthentication to 'no'.
> UsePAM yes
> #AllowAgentForwarding yes
> #AllowTcpForwarding yes
> #GatewayPorts no
> X11Forwarding yes
> X11DisplayOffset 10
> #X11UseLocalhost yes
> #PermitTTY yes
> PrintMotd no
> #PrintLastLog yes
> #TCPKeepAlive yes
> #UseLogin no
> #PermitUserEnvironment no
> #Compression delayed
> #ClientAliveInterval 0
> #ClientAliveCountMax 3
> #UseDNS no
> #PidFile /var/run/sshd.pid
> #MaxStartups 10:30:100
> #PermitTunnel no
> #ChrootDirectory none
> #VersionAddendum none
> #no default banner path
> #Banner none
> #Allow client to pass locale environment variables
> AcceptEnv LANG LC_*
> #override default of no subsystems
> Subsystem       sftp    /usr/lib/openssh/sftp-server
> #Example of overriding settings on a per-user basis
> #Match User anoncvs
> #X11Forwarding no
> #AllowTcpForwarding no
> #PermitTTY no
> #ForceCommand cvs server

When I launch the profiler, I have nothing :

Profile
Profile1913×1049 102 KB

Moreother, the debugger doesn’t work for the GPU.
For example, when I resume, I can’t stop the debugging. Nsight is blocked on this page :

Debugger
Debugger1919×1039 160 KB

Moreother, I have the following lines :
Coalescing of the CUDA commands output is off.
warning: “remote:” is deprecated, use “target:” instead.
warning: sysroot set to “target://”.
Reading /lib/ld-linux-aarch64.so.1 from remote target…
warning: File transfers from remote targets can be slow. Use “set sysroot” to access files locally instead.

Thanks a lot for your help !

10

please someone help me

11

Hi,

Based on topic 119676, it looks like you are already fixed this issue.
Let us track the following update directly on the topic 119676.

Thanks