Hi,
I am doing some project that requires me to use openDDS on jetson tx2 board to send/receive data but I am not sure how am I going to compile it into Jetson TX2 board as it is linux ARM. I read from its instruction is that I need to use some tool chain to cross compile it.
Does anyone have experience compiling it or cross compile?
Thanks!
If you were to compile natively on the TX2, then you might want to mount an SD card and build from there so you don’t use up the eMMC space. This would be your simplest approach.
For cross compile you have basically two types of builds: Bare metal, or user space applications.
Bare metal would include a kernel or bootloader, and these do not require external libraries and do not use a linker. This is the simple case.
User space implies you need a linker, and you also need whatever libraries are linked to. As it turns out some libraries link to other libraries, so there can be a lot of libraries. This is your case.
Either scenario requires tools to cross compile, but the linker and extra libraries (the “sysroot”) are what you are probably confused by. If you go to direct download of what you need you’ll find this is provided by Linaro. Linaro builds PC style compiler tools for cross compile…meaning they run on x86_64 PC, but produce code for arm64/aarch64. When you go to somewhere like Linaro for downloading what you need you will see it is broken down into the three packages: Cross compiler, linker tools, and the sysroot.
Most everyone will have the same cross compiler and linker tools. What differs (and is most painful) is getting your sysroot.
Different Linux PC distributions will have variations on directory layout, but take a Jetson on Ubuntu as an example. If you go to “/usr/lib/”, then you’ll also notice architecture-specific subdirectories, e.g., “/usr/lib/aarch64-linux-gnu/”. Files in “/usr/lib/” are intended for the host. Perhaps files in “/usr/lib/aarch64-linux-gnu/” are also for the host, but these files have gone out of their way to declare they are for a specific architecture. When a package manager on a distribution downloads a cross tool or cross architecture file (“foreign architecture” versus “native”) the subdirectory they go to will go out of its way to declare architecture.
Under that scheme a cross tool on x86_64 for building ARMv8-a will search for either the bare metal version (example for 32-bit ARM bare metal hard float calling convention: “arm-none-eabi”…emphasis “none” for bare metal) or the user space version (example for a TX2: “aarch64-linux-gnu”…emphasis “linux” for user space Linux applications). You want the user space version. Note: The user space compiler can build Linux kernels and bootloaders.
I wish to emphasize that a TX2 is not “arm” (which is 32-bit)…a TX2 is “arm64” (a.k.a., “aarch64” or “ARMv8-a”). The TX2’s arm64 has a compatibility mode which can execute 32-bit “arm”, but this is a foreign architecture and not normally done without good reason (performance sucks and you need all the linker and other support to do it).
Keep in mind while looking at this that Ubuntu and various Linux flavors make cross packages and you might not need to download and install manually from Linaro. However, to illustrate, if you go to this URL you’ll see various packages:
https://releases.linaro.org/components/toolchain/binaries/
Let’s say you’ve selected a recent version, release 7.3 (dated 2018.05). Within that the “arm” versions are 32-bit ARMv7, and the “aarch64” versions are 64-bit ARMv8-a (which is what a TX2 is). These files run on a desktop PC, but produce and work with code for a Jetson’s architecture. “aarch64-linux-gnu” is what you want, so go to that. Within this you will see a gcc package (the compiler), a runtime (the linker and library support), and sysroot. The sysroot is once again the actual libraries which the linker will work with…the libraries are 64-bit ARM, and thus would go into a directory such as “/usr/lib/aarch64-linux-gnu/” (or similar) instead of into the native locations on the desktop PC.
However, the sysroot is very minimal. If you use this sysroot, then for example if you need a library for networking, then you have only the basics and must build the library for networking. Only the networking probably requires some wrappers, so you have to build those wrappers. But the wrappers might require a utility library, so you must build the utility library before you can build the wrapper needed to build the network library needed to build what you really want to build. Whew :(
Ok, so to avoid sysroot hell, you can get the prebuilt version directly off of your Jetson if the Jetson has this. Sysroot files can come from the Jetson itself since they are already the right architecture and built to interoperate with each other.
The offical docs from the L4T version you are using tell you which packages you can download onto an Ubuntu PC to cross compile the kernel or various apps. Use that to start with. Find out where the foreign architecture files are (remember, we’re on a PC so aarch64 is foreign and x86_64 is native). Try to build something…when a library is missing, copy it from the correct location on the Jetson to the foreign architecture location on the PC.
Just FYI, you can clone a Jetson and loopback mount the clone on your PC host. The correct directory of the loopback mounted clone can be used on the PC via a symbolic link and suddenly you have 100% of your Jetson available to the PC for the best sysroot you’ll ever find. Cloning takes time, but it sure beats the work. Plus the fact that you have a backup now which can be restored to a Jetson. The clone also can be rsync updated as the actual Jetson is updated. It wouldn’t hurt to install a sysroot package just to see where everything is, but I like to cheat by using the loopback system.
Another possibility is that if your Jetson is set up to export the right directories via NFS you could NFS mount those directories from a running system to the foreign architecture parts of the PC. NFS has its flaws though and I tend to avoid it, e.g., if you do something to lock up one system it can lock both.
If interested in cloning understand that exact procedures change depending on which release you have. However, see this:
https://devtalk.nvidia.com/default/topic/1000105/jetson-tx2/tx2-cloning/
Also, beware that if you didn’t install the needed library or dev package on your Jetson, then the clone also won’t have this. If you go to cross compile and have the foreign architecture library on your PC, but not on the Jetson, then you end up having to install that library on the Jetson anyway. You will probably benefit by having your Jetson set up for full development.
One way to do this is to try to build your target on the Jetson natively (and if that works there really isn’t any need to cross compile…unless it is setting up for future cross compile…then all the pain is gone on the second build).
As an example, I downloaded OpenDDS 3.13, and ran this on a Jetson Xavier (same architecture as a TX2):
./configure --prefix=/usr/local
It told me some libraries were missing, e.g., wireshark, and so it wasn’t going to build those features. When you see this you could install wireshark, and then run the configure again. Note what is missing and add what you want each time. Then I set the Xavier up for max performance and built using all cores…this is a big package even with Xavier’s 8 cores…but it worked. The temp location consumes nearly 4GB, so you could do this on an SD card, run the “sudo make install”, and then umount the SD to avoid clutter on the eMMC.
Once you are done you probably should clone since you could lose valuable work if something fails. If you plan to do more builds of various software, then you might experiment with loopback mounting the clone (probably read-only for safety while experimenting) and using this to build the same lib from the cross tools on the PC.
Here is a similar note on setting up for cross compile:
https://devtalk.nvidia.com/default/topic/1043115/jetson-tx2/cross-compiler-version-problem/post/5292337/#5292337
Almost forgot: To see what libraries any Linux install has available by default you can run command “ldconfig -p”. Especially useful in combination with “grep”.