cuda remote debug error & problem creating executable

Hi.
I have two problems.
The first is that if you try to run cudaSetDevice or cudaGetDeviceCount, debugging will not proceed.

I installed JetPack 3.3 on Jetson Tx2 and my host PC and ran remote debugging with Nsight Eclipse.

However, I tried to remote debug with Jetson Tx2 using Nsight Eclipse on the host PC, but the JetPack versions were different so I could not debug.

So I installed JetPack 4.2.1 on Jetson Tx2 and host PC and tried to remote debug with Jetson Tx2, but if I try to run cudaSetDevice or cudaGetDeviceCount when debugging is executed, the debugging does not proceed.

However, when you run the executable file in Nsight Eclipse or execute the executable file using Terminal on the Jetson board, the program will run without any problem.

Why can’t I debug when I try to run cudaSetDevice or cudaGetDeviceCount on the host PC?

The second problem is that when you build a Jetson board from the host PC, you get a shared library instead of an executable.

After installing JetPack 3.3 on Jetson Tx2 and building with a Jetson board using Nsight Eclipse, you have an executable file.
However, if you build after installing JetPack 4.2.1, you will have a Shared Library.
Can’t I have an executable file instead of a shared library?

Please help me
Thank you.

Hi,

1.
May I know have you setup the remote connection with root authority?
Please noticed it’s required to execute the profiler as root after CUDA 10.0.

If the environment is setup correctly, you may meet the similar issue of this topic:
https://devtalk.nvidia.com/default/topic/1056303/jetson-tx2/cuda-api-calls-throw-exceptions-in-remote-debugger-/

The latest progress is that we found this issue comes from GDB CPU debugger rather than cuda-gdb.
Due to the GDB upgrade plan of our profiler, we will check this directly on our next CUDA release.

2.
Suppose you should also get an executable from Nsight. Maybe file path is changed?
Would you mind to check the steps in this document and try it again?
https://docs.nvidia.com/cuda/nsight-eclipse-edition-getting-started-guide/index.html#remote-development

Thanks.

https://devtalk.nvidia.com/default/topic/1052253/jetson-agx-xavier/no-timeline-for-profiler-xavier-nvvp-and-nsight-compute-is-not-working-in-jetson-xavier/post/5347671/#5347671

I tried to set the root password as in this topic, but I’m not sure how to edit / etc / ssh / sshd_config.
The contents of my Jetson Tx2 board is / etc / ssh / sshd_config.

#       $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no

#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem       sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server

Is there something wrong here?
If something is wrong, please let me know how to fix it.

Thank you.

I can’t answer all of your questions, but some information on Ubuntu and the root account and ssh might help. This is a lot, but if you go with keys, then development life will be much easier.

On Ubuntu “root” has no password, nor login ability…at least by default. You are expected to use “sudo”. However, as you probably found out, this can be a real pain. Unless you unlock root you cannot directly login as root. If you can avoid logging in as root you are better off, but if you are on a trusted network (for example my embedded systems all sit on their own private network), then you could unlock root.

The other side of the coin is that by default most Linux distributions (including Ubuntu) don’t allow root to log in via ssh unless you’ve set this up. The changes mentioned for “/etc/ssh/sshd_config” are for that purpose. Combining unlocking of root with allowing root login can do what you want.

Personally, even on a private network (and especially when I want convenience), I find I don’t trust passwords for root login. I instead add ssh keys, and although I allow root login, I only allow via a key, not via password. As a result I can enter commands with an ssh prefix from my host PC to the Jetson more or less as if I am sitting directly at the Jetson terminal.

Unlocking root is fairly easy and painless. Just unlocking root via ssh is also fairly painless, but do realize if you change certain things (such as the IP address or host keys), then you’ll have to fix that when ssh thinks there is a “man in the middle” attack.

So to start with, to unlock root you must log in locally to the Jetson. Set a password for root, and make it a very strong password:

sudo passwd root
# Follow prompts to enter pass twice.

Test the root login. Locally at the Jetson:

su -

Note that “su -” requires root’s password, not the account password you are currently logged in to. Nothing is inherited from the previous login when using this method. If the password works, then you can go to ssh setup. The original URL on ssh setup will do this, but briefly:

  • Edit "/etc/ssh/sshd_config"
  • Look for "PermitRootLogin", and make sure it is:
    # No '#' commenting this out:
    PermitRootLogin yes
    
  • Depending on o/s restarting sshd might differ, but this is probably correct for your version:
    sudo systemctl restart sshd.service
    

    …or for some flavors or versions of Linux:

    sudo service ssh restart
    
  • At this point use ssh from your host PC to see if you can login to root at the Jetson's address. You'll need to verify you trust the connection.
  • Here it can get a bit more confusing, but I recommend installing keys from host to Jetson for both your regular user and from root on the PC to the root user of the Jetson. This will allow you to disable password logins, and will also enable you to run ssh without being prompted for passwords. Security will improve with key-only access, but this is not required...you could stop here.
  • If you already have keys on your host for each user, then you don't have to generate keys. However, most people won't have keys unless they've used key-based logins before, so I'll assume you don't have custom keys for each user you plan to access the Jetson from. Here is a URL on ssh-keygen (part of the "openssh-client" package) which is fairly good: https://www.thegeekdiary.com/using-the-ssh-keygen-command-in-linux/ Note: I like using RSA 2048-bit or better. "man ssh-keygen" and searching for the "-b bits" option will tell you more, but I believe 2048 is still default. You can set higher values in multiples of 2, e.g., you could also use 4096 bit (double 2048). I do not set a password to the key itself. If you do, then you'll be prompted for the key's password, not the remote system's password.
  • After you generate a key, then you put this in the right place with the right permissions. There is a ".pub" key which is public and anyone can read this...this is for encrypting, e.g., "id_rsa.pub" is a traditional name. This must be set to be readable by anyone. The private key, "id_rsa" (again, this is a traditional name for RSA keys) must not be readable or in any way touchable by anyone but your account. These keys must go in the "~/.ssh/" directory, which in turn must be readable and executable by anyone, but cannot be writable by anyone but you (ssh will refuse to work if permissions are wrong).
  • Next you need to copy your key to the Jetson using the "ssh-copy-id" command. You will log in via the password, and then after the successful login, the Jetson will remember the public key and allow this as a login. You'd use this to copy to both your regular developer account on the Jetson, and again to root on the Jetson. Do not do the reverse...do not give the Jetson permission to access your host. As an example, if your remote developer account name is "ubuntu", with password "ubuntu", and you have unlocked root to have password "maybeImCrazy", and if the remote host has IP address "192.168.1.2", then from your PC:
    # Logged in as your regular user who will do the debugging...this is regular user of the host PC,
    # not the Jetson:
    # Regular login only just to make sure you have accepted for normal ssh:
    ssh ubuntu@192.168.1.2
    exit
    ssh root@192.168.1.2
    exit
    ssh-copy-id ubuntu@192.168.1.2
    # Enter the "ubuntu" password.
    exit
    ssh-copy-id root@192.168.1.2
    # Enter the "maybeImCrazy" password.
    exit
    # Now disable root login via password, but keep non-password login. Edit "/etc/ssh/sshd_config" again,
    # and this is what you will have:
    PermitRootLogin without-password
    sudo systemctl restart sshd.service
    # (or perhaps "sudo service ssh restart" on some systems)
    # Now go back and see if you can log in without entering the password. I'll just send the "whoami"
    # command instead of actual login (no password should be required):
    ssh ubuntu@192.168.1.2
    exit
    ssh root@192.168.1.2
    exit
    # You might still have a step with root if this doesn't work,
    

From then on this PC, if keys do not change, and if you access using the same IP address, should not prompt for a password. The public/private key pair will instead do the job. No system should be able to log in to “root” via any password.

Now to relock root (you can still login via key, but password will be removed…“su -” will not work after that, but “sudo -s” will, and ssh won’t need a password):

sudo passwd -d -l root

If you wish to add other accounts to log in to root via ssh using keys at a later date, then you need to unlock root password again and tell sshd to allow root password login while adding the key. Then lock it back up.

Always always always back up any keys in your host’s “/etc/ssh/” (the naming will make it obvious), and also host keys of the Jetson. Always back up the “~/.ssh/” of your host PC. You didn’t add keys to the individual login accounts of the Jetson, so any “~/.ssh/” doesn’t need backup. If for some reason you lost your Jetson, then there are no keys to worry about. Even if the Jetson were lost, if the IP address changed, then anyone having the lost system would still be unable to get in. Always keep any copy of private keys safe from anyone other than you. Public keys can be handed out any time and will not be a risk. “ssh-copy-id” hands out only public keys.

If for some reason you are debugging with software which mandates using a password, then having keys might not do the job. You can still have the keys, but you’d end up keeping root unlocked and keeping ssh to allow login via password (a bad idea if it can be avoided…also a bad idea if it can’t be avoided). However, there is no technical reason you can’t have keys for most things and still allow a password if you are on a private network or if for some reason you think you can trust this.

Note: There is a case where sometimes you need a second step beyond ssh-copy-id for root logins. If you run into this just post the error message here for more info.

Example of host PC to Jetson things you can do conveniently with keys and no password, while remaining more secure than with passwords (uses example account names and IP addresses…adjust for your case):

ssh ubuntu@192.168.1.2 df -H -T /
ssh root@192.168.1.2 shutdown -r now
ssh root@192.168.1.2 shutdown -h now
ssh ubuntu@192.168.1.2 sha1sum -c /etc/nv_tegra_release
ssh ubuntu@192.168.1.2 head -n 1 /etc/nv_tegra_release
ssh ubuntu@192.168.1.2 cat /etc/issue
ssh root@192.168.1.2

Note: You can copy keys from “/etc/ssh/” to the sample “Linux_for_Tegra/rootfs/etc/ssh/” before a flash, and you can copy individual user keys to the “Linux_for_Tegra/rootfs/home/…some user…/.ssh/” before a flash, and they will “just work” after done. Don’t forget to also copy the sshd_config edits. After flash this will “just work” if IP addresses did not change.

Thanks, linuxdev!

Hi, jiyang1,
Please try the suggestion from linuxdev to setup the root remote access.

Thanks.

Thanks, linuxdev!

But I still don’t have remote debugging.
I set the root password using “sudo passwd root” and proceeded with the example below and was able to log in without entering the password.

# Logged in as your regular user who will do the debugging...this is regular user of the host PC,
# not the Jetson:
# Regular login only just to make sure you have accepted for normal ssh:
ssh ubuntu@192.168.1.2
exit
ssh root@192.168.1.2
exit
ssh-copy-id ubuntu@192.168.1.2
# Enter the "ubuntu" password.
exit
ssh-copy-id root@192.168.1.2
# Enter the "maybeImCrazy" password.
exit
# Now disable root login via password, but keep non-password login. Edit "/etc/ssh/sshd_config" again,
# and this is what you will have:
PermitRootLogin without-password
sudo systemctl restart sshd.service
# (or perhaps "sudo service ssh restart" on some systems)
# Now go back and see if you can log in without entering the password. I'll just send the "whoami"
# command instead of actual login (no password should be required):
ssh ubuntu@192.168.1.2
exit
ssh root@192.168.1.2
exit
# You might still have a step with root if this doesn't work,

After completing all the tests, I was able to log in as Tx2 as root when I started the debugger.

However, there is still no remote debugging.
The console output is as follows:

Coalescing of the CUDA commands output is off.
warning: "remote:" is deprecated, use "target:" instead.
warning: sysroot set to "target://".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib/ld-linux-aarch64.so.1 from remote target...
Reading /lib/ld-2.27.so from remote target...
Reading /lib/.debug/ld-2.27.so from remote target...
0x0000007fb7fd31c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
$1 = 0xff
The target endianness is set automatically (currently little endian)
No source file named SobelFilter.cpp.
No source file named SobelFilter.cpp.
No line 25 in file "../helloCUDA.cu".
No source file named helper_cuda.h.
No source file named PointCloud_Main.cpp.
No source file named SobelFilter.cpp.
No source file named SobelFilter.cpp.
No source file named SobelFilter.cpp.
No source file named SobelFilter.cpp.
No source file named SobelFilter.cpp.
No line 58 in file "../main.cpp".
Reading /lib/aarch64-linux-gnu/librt.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libpthread.so.0 from remote target...
Reading /lib/aarch64-linux-gnu/libdl.so.2 from remote target...
Reading /usr/lib/aarch64-linux-gnu/libstdc++.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libgcc_s.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libm.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/librt-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/47f37309461cc15fb1915bc198d718017a1f87.debug from remote target...
Reading /lib/aarch64-linux-gnu/libdl-2.27.so from remote target...[img][/img]
Reading /lib/aarch64-linux-gnu/.debug/libdl-2.27.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/a6cec032b9969b1d556b4ebee1400fabda2fdc.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/.debug/a6cec032b9969b1d556b4ebee1400fabda2fdc.debug from remote target...
Reading /lib/aarch64-linux-gnu/0a79672e8a81d551322b2912578e0ea9cef6e9.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/0a79672e8a81d551322b2912578e0ea9cef6e9.debug from remote target...
Reading /lib/aarch64-linux-gnu/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/libm-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libm-2.27.so from remote target...

13	../main.cpp: No such file or directory.
Temporary breakpoint 15, main (argc=1, argv=0x7ffffff368) at ../main.cpp:13
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.32.2.0 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_graphics.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvidia-fatbinaryloader.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvos.so.debug from remote target...

Is there anything I missed when connecting remotely with root authority?

https://devtalk.nvidia.com/default/topic/1056303/jetson-tx2/cuda-api-calls-throw-exceptions-in-remote-debugger- /
Is this the same issue?
If the same problem, how do you solve it?

log in.PNG

I can’t answer all of the details of the remote debugging itself, but some of the failed file reads were unrelated to login. For example, “SobelFilter.cpp” is just a file the debugger couldn’t find. If on the TX2 the file didn’t exist, or if configuration did not tell where to find the file, then there is no way the file could ever be read. Consider that in some cases you need to install both source packages and debug symbol packages for various libraries if you want to single step through library calls. Your actual project must also have the source files accessible, and the executable built with debug symbols. This is a very general setup topic I probably can’t help much with.

If you look specifically at this, then you will note an option:

warning: "remote:" is deprecated, use "target:" instead.
warning: sysroot set to "target://".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Note that when doing work on a foreign architecture (such as working on an X86_64 PC using cross tools to deal with non-native arm64/aarch64/ARMv8-a), then when a file or debug symbol is referenced, you will need the foreign system’s file or symbol, not the native x86_64 file or symbol. This is what the “sysroot” is. If you were to clone your root filesystem of the Jetson, then loopback mount this on the host PC, the result would be the most perfect sysroot possible: An exact clone. Even this wouldn’t help if you needed files or symbols you had never added to the Jetson. However, check out cloning:
https://devtalk.nvidia.com/default/topic/1000105/jetson-tx2/tx2-cloning/
(instructions can differ based on release, so there are changes over time)

So as a step in understanding what is going on, can you use just the regular “gdb” application running natively on the Jetson, and not have the failure to find those files? You may need to add source files and/or debug symbol packages. After that a clone (or copy of individual content) to the PC would increase what files/libraries you can single step into.


Sort of a separate issue related to passwords (not about files and clones) may be hitting you. It may be the debug application itself, run from the host, may not have taken into account having no-password key-based login; ssh has the option for key based login, but does the application know this? Maybe not. You could leave the root account unlocked and try to enter the root password. Or, it could be it expects login as your regular user, and then the password of that regular user for the purpose of running sudo. I don’t know what the NVIDIA Eclipse release is asking for there, but someone from NVIDIA can probably tell you which user and password is being requested.

warning: "remote:" is deprecated, use "target:" instead.
warning: sysroot set to "target://".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Does this message occur when you are not connected to root authority?

If you connect remotely with root authority, does debugging proceed immediately without logging in to the remote device?
How can I tell if I made a remote connection with root authority from nsight eclipse?

Hello AastaLLL
Thank you for answering my question.

As you answered, I’ve tried https://docs.nvidia.com/cuda/nsight-eclipse-edition-getting-started-guide/index.html#remote-development again with the steps in this article.

In this document, the Run / Debug Settings method is called ‘Upload local executable’. So I set it like <Attachment 1> and local like <Attachment 2>.

As a result, when you build with the ‘Upload local executable’ method, you get an executable file (attachment 3), but it doesn’t run on Jetson.
(Attachment 4: Error when running)

So I tried again with the ‘Run remote executable’ method (attachment 5) and as a result it runs but still has the Shared library (attachment 6).

<Attachment 7> is an executable file properties(cuda 9.0) when JetPack 3.3 is installed.
<Attachment 8> is an propertiesof the presently created file (Shared library). (JetPack 4.2.1 installation, cuda 10.0)

I want to have an executable like .
What should I do?

  • Reference *

3.PNG

4.PNG

6.PNG

7.PNG

8.PNG

It is easier to first comment on something regarding local debug (sorry, you probably know this already, but I’m reiterating as the easiest way of explaining), and then extend it to remote debug (but it won’t answer all of your questions, I’m only commenting on a small piece of the puzzle)…

When you debug a program the debugger needs access to the source code. Debug symbols are required in order to relate particular parts of a program to particular source code. Often people think of source code and debug symbols in the program they are working on, but seldom think about what happens when you single step into a linked library.

So if you want to actually see the library, then you need the library itself to have the debug symbols installed. Most people don’t install symbol packages for extra libraries, and I’m a bit odd, so for example I have source and debug for libc on every system I have (plus a number of others). If I run into a library call with some sort of error, it then becomes possible to single step into the library and see specific lines of code and data.

When you remote debug, then even the source code of the program you are working on becomes similar to a linked library in that unless you’ve pointed your debugger to a remote set of code the code will never be found. As an alternative you can put the code and debug symbols on your host PC. Even if you do point your debugger at a remote location with the correct source and have debug symbols the operation will probably be slow: The remote disk has to retrieve this, and then it has to be transferred over the network connection…each time you go back to some particular code location the same download over the network has to happen again. You can instead put the code on your host PC and point your debugger at this for reference to the source code and not have it download each time you step through some code location.

Now if you add all of the source and debug packages you are interested in to the Jetson, then a single clone operation will make all of that infrastructure available via loopback mount on your host PC. This greatly speeds up debugging. The trick is that you (or the debugger) must either copy what is on your Jetson to the host PC, or download it each time over the network link (which is slow and inefficient), but if you have added all of the debug code and source onto the Jetson, a loopback mounted clone is quick and easy just prior to starting debug. If you have five or ten embedded systems, imagine copying all of that code manually to the host PC…it becomes a mess. Instead you can have one standard mount location and loopback mount the clone there while using it, or umount the partition when not using it…you can switch back and forth to different systems, and you are absolutely guaranteed that what you copied is an exact match to what is being debugged. This is the “sysroot” content.

If you choose to do this though, be certain you have all of the desired source and debug packages on the Jetson prior to cloning. You can update a clone with rsync, but that’s another learning curve.


Sometimes a file just isn’t there, and thus you can’t use it to debug. Sometimes you can’t reach a file, or can’t perform an operation, due to permissions. The error message you had received was one of a missing file, not one of permission. “SobelFilter.cpp” is an example of something which is missing. Even if the file is on the Jetson, the debugger didn’t know how to find it. I find it easiest to add the source to the Jetson, and then point the debugger at the clone which has the file, or you could directly copy the file to the host PC. FYI, that file is part of the “/usr/local/cuda-something/” samples, in particular within “3_Imaging/”. The debugger must find this either on the Jetson or the host PC, but the debugger does not know the path to the source code (or the source code isn’t installed).

Another reason to clone: If something fails you have a full copy capable of being flashed onto a new Jetson.

Bigger NOTE: The exec format error means you are trying to execute code for the wrong architecture. Either you are incorrectly trying to run arm64 on x86_64, or trying to run x86_64 on arm64. Regular tools create and run the same architecture (native); cross tools run on one architecture, but create code for another architecture (foreign). When you see exec format error, think about finding the same code, but compiled for the other architecture.

Thanks, linuxdev.
I used this article to succeed in remote debugging.
But another problem arises.

An error occurs at ‘glutInit (argc, argv);’ and the consol message is shown below…

No protocol specified
freeglut (/ root / cudaSample / Debug / cudaSample): failed to open display ': 0'

I changed the code to: 1 instead of: 0 and got the same error message.
How can I solve this?

Should I add “xhosts +”?
If so, can you tell me where to add it?
I do not know in detail because I am a beginner.

Thank you.

Sorry, I can’t answer the specific debugger questions, but you’ll need this long background to understand what is going on. If you understand this prior to looking at the documents on debugger setup you should get further along. You can always ask more questions.

Any time you run a program with X11 protocols, or the GPU when the driver loads through the X server, you will need to tell the program which display to use. Literally, there is the environment variable “DISPLAY” which names this. Assuming the program is using the GPU from the Jetson, then the Jetson needs a running GUI (it can be a virtual GUI or a real keyboard/monitor login). When you log in to this, then for the local machine’s purposes, this tells you which DISPLAY:

echo $DISPLAY

…which will be the “:0” or “:1” style display.

Remote display is a different level of difficulty. One topic is which DISPLAY the program you are debugging will run on. Presumably the DISPLAY of the Jetson. The remote debugger itself must run on the host PC’s DISPLAY. From your host PC command line in the GUI, if you run the “echo $DISPLAY”, then you will see the DISPLAY for what is local to the host PC.

There is a subtle point I’m going to describe: You will want the program on the Jetson to use the Jetson’s DISPLAY, and you’ll want the debugger to use the PC’s DISPLAY. Instead of giving the full description I am going to just demonstrate something first. With the Jetson booted (I’ll refer to it’s network address as “jetson”, and pretend the user login name is “nvidia”…maybe it is, but adjust for your case…and will pretend the PC’s login name is “dev”, with address “pc”).

  1. From the Jetson, see what it says for "echo $DISPLAY".
  2. From the PC, see what it says for "echo $DISPLAY".
  3. Note that even if the two say the same thing, that the context of which computer it runs on matters. "xhost" and some other tools more or less extend the syntax to something like "pc:0" or "jetson:1". I'm assuming "pc" and "jetson" are the IP addresses, or else that you've added those to "/etc/hosts".
  4. From the host PC, connect to the Jetson with "ssh nvidia@jetson". Find this causes an error (any GUI program will work, I'm just using "lxterm" as an example):
    lxterm
    
  5. Since the connection is really at the Jetson, look at the DISPLAY from the Jetson. Now type this (I'm pretending it is ":0", but change to whatever it really is):
    DISPLAY=':0' lxterm
    

    …lxterm should pop up in the running GUI of the Jetson.

  6. Now do this instead, from the PC, carefully noting the option to ssh (which forwards to the PC):
    ssh -Y nvidia@jetson
    lxterm
    

    …you’ll see the lxterm pop up on your PC instead of the Jetson.

So when you don’t have a DISPLAY set, it means the program doesn’t know which display to use. You must name it in DISPLAY. The CUDA program must use the DISPLAY of the Jetson. The debugger must use the DISPLAY of the host PC. “ssh -Y” is a way to automatically forward from Jetson to PC without fiddling with the DISPLAY. When this is done automatically, then “xhost” is not required. “xhost” is a way to manually add permissions for a remote system or a different user to display on your personal DISPLAY.

You can override “-Y” for individual applications. So you can make it default that you are forwarding GUI to the host PC:

ssh -Y nvidia@jetson
# Now display on the Jetson's DISPLAY even though you said to forward...override forward:
DISPLAY=:0 lxterm
# Kill that, and now display again to the PC's DISPLAY:
lxterm

Both lxterm applications run on the Jetson. All that differs is where they display, as set by DISPLAY. Without xhost or an ssh option you will find security gets in the way. If operating purely from a command line DISPLAY will always be missing unless you add it.

I don’t know from your logs which DISPLAY isn’t set. However, keep in mind that when CUDA or any hardware accelerated video is forwarded, that it is no longer the Jetson’s GPU doing the work. This would mean your host, if it happens to have all the right versions of CUDA and other libraries, will make your Jetson appear rather fast…but in reality it would be that RTX2080 doing the work, not the Jetson. The CUDA and anything meant to run on the Jetson should use the Jetson’s DISPLAY. The remote debugger is intended to display on the PC, so the debugger uses the host’s DISPLAY. Debugging a program running on the Jetson while watching from the host means you need both DISPLAYs to be in use simultaneously depending on whether the software is meant to run on the Jetson versus the PC.

glutInit() in particular is a GUI API call for initializing the video hardware. It needs to know which DISPLAY context to use. If glutInit() is for a program running on the Jetson, then it needs the Jetson’s DISPLAY. Likely the debugger GUI already knows correctly to run on the PC. I can’t tell you how to set up the debugger specifically for your case, but if you go through the setup docs and carefully note the “echo $DISPLAY” of both while both have a logged in GUI, then you are doing ok.

FYI, two users on the same computer would need permission to display on the other user’s DISPLAY. If a program is logged in as root, but the GUI is logged in as nvidia, then there might be a security issue blocking display. If sudo is used from the nvidia user’s GUI, then it automatically gets the equivalent of either the “xhost” or “ssh -Y” automatic pass-through. I rarely ever use xhost since mostly I use “ssh -Y”. Some people use “ssh -X”.

Thank you for answering my question in detail.

I checked DISPLAY on hostPC as below.
And setenv (“DISPLAY”, “: 0”, 0) was changed to setenv (“DISPLAY”, “: 10”, 0).

ssh -Y nvidia @ jetson
echo $ DISPLAY
localhost: 10.0

As a result I was able to run my program on the host PC.

But I still can’t run cudaSample (sobelFilter, OceanFFT, etc).
An error occurred when cudaSample was run.
The error message is shown below.

CUDA error at ../src/SobelFilter.cpp:314 code = 30 (cudaErrorUnknown) "cudaGraphicsGLRegisterBuffer (& cuda_pbo_resource, pbo_buffer, cudaGraphicsMapFlagsWriteDiscard)"

How can I run the sample?

Where is the physical display the program is displaying to? Is it on the host PC (ssh redirects), or is it on the Jetson (real or virtual X server does not matter)?

Does the program work when run locally directly on the Jetson? And then does it fail only via the ssh redirect?

The current error occurs when DISPLAY is on the host PC.

When you run the sobelFilter sample, it displays on the host PC like <Attachment 1>, but the program immediately exits with an error.

Running sobelFilter in Jetson works just like <Attachment 2>.


When DISPLAY is on the Jetson, then the libraries and software required for CUDA or hardware accelerated rending depend on what is installed on the Jetson. When you remote DISPLAY to a different computer, then the Jetson no longer provides any GPU or display function…this is then offloaded to the remote PC. If you have valid libraries and support for some hardware-dependent operation on the Jetson, but not on the PC (including version mismatches), then it will fail because the PC lacks the infrastructure requirements.

The actual debugger is ok on the PC, but the CUDA program itself requires DISPLAY to the Jetson. Is the CUDA or graphical program (not the debugger) running entirely on the Jetson? From what I can tell you are putting both on the PC, which is the error.

Note that you can log in locally to a Jetson with the GUI, and then set DISPLAY for the program on the Jetson while the debugger runs on the PC (the CUDA program would show up only on the Jetson, the debugger would show up only on the PC). The distinction is that as Eclipse starts on the PC, that the “run program” names a remote target and the DISPLAY to use on that target of a logged in user. In your case you have redirected everything to the PC.

Hi.
I have a same problem in remote debug.When i trace the program remotely, if findCudaDevice function is executed the next error will be occurred:
fatal: The CUDA driver initialization failed. (error code = CUDBG_ERROR_INITIALIZATION_FAILURE(0x14)
Next attachment specifies this error:


My host PC is ubuntu 18.04 run on a virtual machine and the target is jetson tx2.
Can anybody help me to solve this problem.

me too,do you find solution?