Hey Guys,
I am working on running a large project on TX2 but i am difficulty compiling this project in the TX2. So i am looking into options for compiling the code in a host system and transfer the executable to the TX2. But i am not able to find an guide or tutorials on how to step up a cross compilation link between my host linux system and Nvidia TX2. Could someone please suggest some links
Thanks.
You might start with setting up for kernel cross compile. This is “bare metal”, and does not have all of the requirements of user space. See the “Documentation” download under R28.1.
For kernel space there is no need for linkers or system libraries. This is something you might consider “kernel space”, versus “user space” sysroot (e.g., this is how Linaro tool chains word it). Beyond the sysroot (e.g., the basics of libc and the linker) you need everything the application will link against. I actually suggest that if you have the ability to add a foreign architecture of aarch64/arm64 to your host this might be the easiest way to go, but unless your host and Jetson are both Ubuntu 16.04 this might get complicated. I use Fedora, so I can’t go this route…but basically you should research adding a foreign architecture to your host, and then adding this foreign architecture for any files you need, e.g., libc and the linker.
As an alternative, you can clone the rootfs of your Jetson and mount this on loopback on the host, then sym link where you need the links on the host (this is what I usually do, Ubuntu installer/UEFI/BIOS bugs don’t allow me to add Ubuntu to my hardware without wiping out everything else).
So set up your build tools. Some are available with the above mentioned Documentation. If your host is the same version of Ubuntu as the Jetson is, then learn about adding a foreign architecture and do this, followed by whatever user space packages you need. If you can’t do this, clone your Jetson and substitute sym links to the loopback mounted image whenever you need the aarch64 infrastructure.
If it turns out that your trouble for compiling on a Jetson is due to missing libraries, then cross compile will also fail unless you add those libraries. Cross compile complicates things, it doesn’t get around any of the base requirements.
Hi,
I successfully installed Nvidia’s Nsight Eclipse for cross compilation.
I was able to link OpenCV as well. I tested it on various sample codes and my individual codes. It works fine.
The problem occurs when I have a huge project like SLAM where I have lots of include files, lots of thirdparty files and lots of other software files. I was still able to link the whole project. The problem occurred during its execution when its trying to make all the src files executable. THe error looks like this
Finished building: /home/apoorva/ORB-SLAM2-GPU2016-final/gpu/tx1.cpp
Building target: libtest_cuda_SLAM
Invoking: NVCC Linker
/usr/local/cuda-8.0/bin/nvcc --cudart shared -L"/usr/local/cuda-8.0/samples/common/inc" -L"/usr/local/cuda-8.0/samples/common/lib/linux/aarch64" -L"/usr/local/cuda-8.0/targets/aarch64-linux/lib" -L"/home/apoorva/ORB-SLAM2-GPU2016-final/cmake_modules" -L"/usr/local/lib" -shared -lGL -lGLU -lglut -Xlinker -unresolved-symbols=ignore-in-shared-libs --relocatable-device-code=true -gencode arch=compute_60,code=compute_60 -gencode arch=compute_60,code=sm_60 -m64 -ccbin aarch64-linux-gnu-g++ -link -o “libtest_cuda_SLAM” ./src/Allocator_gpu.o ./src/Converter.o ./src/Cuda.o ./src/Fast_gpu.o ./src/Frame.o ./src/FrameDrawer.o ./src/Initializer.o ./src/KeyFrame.o ./src/KeyFrameDatabase.o ./src/LocalMapping.o ./src/LoopClosing.o ./src/Map.o ./src/MapDrawer.o ./src/MapPoint.o ./src/ORBextractor.o ./src/ORBmatcher.o ./src/Optimizer.o ./src/Orb_gpu.o ./src/PnPsolver.o ./src/Sim3Solver.o ./src/System.o ./src/Tracking.o ./src/Viewer.o ./gpu/tx1.o /usr/local/cuda-8.0/samples/common/lib/linux/x86_64/libGLEW.a -lcudart -lopencv_core -ltbb -lz
/usr/lib/gcc-cross/aarch64-linux-gnu/5/…/…/…/…/aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libopencv_core.so when searching for -lopencv_core
makefile:60: recipe for target ‘libtest_cuda_SLAM’ failed
/usr/lib/gcc-cross/aarch64-linux-gnu/5/…/…/…/…/aarch64-linux-gnu/bin/ld: skipping incompatible //usr/local/lib/libopencv_core.so when searching for -lopencv_core
/usr/lib/gcc-cross/aarch64-linux-gnu/5/…/…/…/…/aarch64-linux-gnu/bin/ld: cannot find -lopencv_core
collect2: error: ld returned 1 exit status
make: *** [libtest_cuda_SLAM] Error 1
I haven’t built this, but it seems like it is a matter of using a different “-L” linker path and avoiding the x86_64 paths. This suggests “/usr/local/lib/” is not valid (and it is probably x86_64) and rejected by the aarch64 linker:
/usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: <b>skipping incompatible</b> /usr/local/lib/libopencv_core.so when searching for -lopencv_core
Basically the files for linking against on your host are for the host, not for a Jetson. You’ll need find where the aarch64 versions are and set the “-L” option to that. Alternatively, you may find some of the settings for environment in eclipse offer clues and changing those paths might be as simple as changing a GUI menu. Of course the aarch64 versions of files do need to be wherever you point the linker. This is where I add those files to the Jetson and clone it, then aim linkers at the loopback mounted clone (either directly with “-L” or indirectly via symbolic links).
@linuxdev When you say ‘clone rootfs’ does it mean copy the contents of rootfs into a location on the host?
Secondly, if I have a copy of the rootfs on host can’t I simply point my compiler and linker to the paths within that rootfs folder? why do I need to loopback mount?
When you look at a hard drive directly as bits the method of using it is different than when looking at a file system and using the file system to access it. During normal run the rootfs is accessed by the methods of an ext4 file system, and this in turn talks to the driver which directly accesses as bits. If you were to read bits, then there is no such thing as having any meaning, e.g., permissions, users, files versus directories, so on. Copying a mounted file system’s content causes interpretation of what is there to become permissions, files, directories, so on…on the other hand, copy of raw bits means the partition and the file system itself are also copied.
One can manipulate partitions on a real disk, and as a result of a clone being a bit-for-bit exact copy (versus files and directories), one can use loopback to make those bits appear as a disk and allow the bits to be manipulated exactly the same as if it is a raw disk or partition (you can use gparted to resize a disk partition…and to resize a loopback covered clone). When a Jetson is flashed the driver package has no concept of ext4…it merely copies bits which by coincidence were previously formatted as an ext4 partition. If you copy files directly, you can’t flash it. If you copy bits, then the file system and every aspect is an exact mirror, and this can flash to become a complete restoration down to the last bit.
Because clone is a bit-for-bit exact copy, this means your entire file system can be mounted on your host the same as if it were an SD card or external disk drive…you just have to cover it first with loopback so the kernel thinks it is a disk instead of a file. A clone is better than a file copy since it has the file system itself copied into it. Loopback mounting this implies you can use rsync from your Jetson to update the loopback partition as if it were a real disk. Future flashes can use this. This is the usual reason for cloning. See:
https://devtalk.nvidia.com/default/topic/1000105/jetson-tx2/tx2-cloning/
You can also use “dd” to clone a partition. However, the partition should not be mounted (because you don’t want it changing in the middle of a copy) when using dd. A clone works with the Jetson in recovery mode, and thus the operating system is not running and the clone is guaranteed to be an exact copy of your system. You could use dd correctly if and only if you boot to alternate media and do not mount the partition, e.g., by having an SD card rescue boot set up.
In the above URL pay close attention to file names. If you clone “backup.img”, then you will have created both “backup.img” and “backup.img.raw”. Only the raw image is useful…the “backup.img” would be a “sparse” file which is essentially a cheap kind of compression…and it can be flashed…but it can never be updated or examined without restoring it onto a Jetson. I throw away the “backup.img” and keep only the raw “backup.img.raw” (as a file system gets filled with content the size of the sparse image will approach the size of the raw image…after a fresh flash a raw image is about 15GB, and a sparse image is about 2GB).
When the driver package running on the host PC flashes a Jetson (which can be done on command line or via JetPack running the driver package) it can use either the raw format or the sparse format…the driver package doesn’t care and both will correctly flash the Jetson (the Jetson knows how to unpack raw or sparse ext4 images). Originally there was only the raw version, and sparse had no support. Sparse was added to speed up flash time. Sparse is much smaller than raw if the raw is mostly empty…raw is the exact and full size of a partition. If your Jetson had been flashed with “-S 14GiB” size argument, then this makes the rootfs partition “1410241024*1024” bytes (15032385536 bytes). That’s a huge file. Transferring it over USB takes longer than something which is perhaps 20% of that size. Your host would have to be able to hold files in the 15GB range (the host needs a lot of spare disk space for clones) for the raw image, and perhaps 2 or 3GB for the raw image.
If you take a raw image, e.g., named “backup.img.raw”, then you can explore it or use it like a real partition:
sudo -s
mkdir /usr/local/my_arm64
mount -o loop /where/ever/it/is/backup.img.raw /usr/local/my_arm64
cd /usr/local/my_arm64
ls
cd /usr/lib
ln -s /usr/local/my_arm64/usr/lib/aarch64-linux-gnu .
# Now your PC has the entire arm64 library at "/usr/lib/aarch64-linux-gnu/".
# The content on the host is a 100% exact match of your complete Jetson "/usr/lib/aarch64-linux/gnu/".
cd
umount /usr/local/my_arm64
# Now the aarch64 is empty.
# Have a different clone from a different Jetson with some difference? Mount it at
# the same location...your build environment is now an exact match to that other system.
# Have other people who need to share builds? Pass around the clones and mount this way...
# you'll now have an exact duplicate build environment.
# Not enough space? Put a clone on an SD card or external USB disk...or even an NFS server...
# your environment can now become any image you want without modifying the host (as you go
# through lots of embedded devices and lots of cross-tool versions this becomes more valuable).
exit
There is a subtle detail to think about when comparing dd or clones to rsync or other file copy mechanisms. If you were to mount another file system on top of the original file system, e.g., some subdirectory, then file copy will only see what is mounted over the top…it can’t see what was originally underneath that new mount. As an example, when you first boot, there is content in “/dev” which is used for booting. Later on at some stage of boot, sysfs and udev mounts a pseudo file system on top of “/dev” and creates its own content. rsync and other backup methods are generally designed to not cross file systems, and thus would leave “/dev” empty. If you get around this by file copy, then you’ve just contaminated your backup image with files not on the original partition…those files are hidden by being mounted over. What you really want is the partition content, not the pseudo file content. Clone does this without any rescue disk. dd does this if you have a rescue disk. File copy can never see or copy the full partition when a pseudo file system covers up parts.
For the purpose of developing you could copy every file which you know is needed (e.g., all of the “/usr/lib/aarch64-linux-gnu/” and nothing else). When you do a system update of the Jetson, then you will need to repeat this if you want the build to be for that version. If you have two Jetsons with different versions, then you need to do this again each time you build for the different version. Every…single…time.
Or…you could mount a clone which is sym linked and be done. And have the added security that if you lose the content on a Jetson you can absolutely restore the rootfs.
Do beware that restoring and cloning may have differences depending on which release you use. rsync doesn’t care about that, so there rsync and file copy wins.
If you do clone, then I suggest you save a pristine copy. I use “bzip2 -9” on images I’m not currently using…this takes a lot of time. Any image mounted with “mount -o loop,ro …” is read-only and protected…a good way to mount to library directories you don’t expect to modify. Any “restore from clone” directions do eventually have you copy the image to “Linux_for_Tegra/bootloader/system.img”, and in that case I would never ever put my original there…it would get overwritten the first time you forget to use the “-r” option to flash.sh.
Note that you could use file copy to an SD card instead of cloning and it would probably be good for that purpose. You could then mount the SD card somewhere and sym link to the lib directory the same as if it were a clone. I prefer the clone because it doesn’t matter how I might mess up the copy…it’ll always be perfect, and there all kinds of things which can go wrong with recursive file copies.
Tip: If your host PC already has “/usr/lib/aarch64-linux-gnu/”, then it means you’ve added foreign architecture environment for cross compile. Like algebra, there are a lot of ways to avoid conflicts…some are more work than others and some have more problems than others. You could use symbolic links to individual files in the loopback mount location’s aarch64-linux-gnu instead of linking the entire directory. Or you could add “/usr/lib/alt/” to your cross-linker path, and then sym link aarch64-linux-gnu in there. You can always ask if you see alternatives and want help picking one.
Caution: You can never restore a Jetson with a clone from a different L4T version than what is on the rest of the Jetson. However, a clone makes it much safer and easier to restore content onto a newly upgraded flash and take your time with it.
Hi @linuxdev,
As always your posts are very educational. Thanks a lot for taking the time to explain in detail. I understand most of it.
I have one question, you mentioned that if I create a copy (not clone) and use that for cross compile environment then every time I update Jetson with new libs, I will have to copy there over to host. How is it different in case I use a clone mounted on host? If I create a .img.raw clone and after that I install new libs on Jetson, I will still need to create a new clone for use on host for cross compile. In fact it will be more time taking to create a clone whenever an update is made on Jetson. I think you mentioned using rsync from Jetson to update the loopback mount of the image on host. If I do that, would it actually update the underlying image file or the updates remain supercial on mount level?
Thanks
Nouman
If you have a clone, but only one clone, then it isn’t much different until you try to use it for a restore. As soon as you need to migrate to a new JetPack/L4T release, and want to keep the old environment for other use, even if just comparisons, the clone becomes easier. I suppose you could use copies to separate disks as well and mount the partition with the copies the same way as a clone (e.g., external USB or SD card).
In this case it won’t matter to you, but if you need to preserve the file system itself, a clone becomes superior. If you are not intending to have something to restore from in case of a system failure, then file copies are just as good.
FYI, after a clone is made you don’t have to re-clone each update. You can use rsync if you want (which is just a smart file copy).
When rsync updates a clone the clone must first be loopback mounted. The content within the file gets updated. If you were to flash a new system with an rsync updated clone, then you would get a 100% exact match. If you were to flash from a file copy, then mostly it would match…but not necessarily (e.g., the real files in “/dev” which exist prior to udev). Any copy of that image file sent to another developer would be a perfect copy of your environment on that Jetson. The host itself would have no knowledge of the content of that image unless it is loopback mounted…all hosts with that image would have the same exact content.
great! Clone is much better choice and since we are in prototype stage, it is good to keep exact copies of the system, these will come in handy if we decide to scale to production with multiple Jetsons operational.
With that said, I would like to have your input on another method of making the Jetson environment available to host for cross-compile. Some folks mount the actual filesystem on Jetson over the network at host for cross-compile. This also gives the benefit of automatically getting the updated environment for cross-compile whenever new libs are installed on Jetson but obviously lack the liberty of restoring the image to multiple Jetsons. What do you think about this method? Any downsides of this for example.
One more question, after mounting the clone, can’t I just point the linker to the mount and lib path in there. I didn’t understand the symlink part properly. Care to explain please? I have some idea, like if host has a foreign architecture added and has most of what is needed and rest can be replaced in their with symlink to the clone mount. This raises another questions, how can we know which files need to be replaced with symlinks to the clone mount
Some people use NFS export to make a file system available. I never use that between any development system and host because normal configuration means that if one system locks, then probably so does the other. You can make a soft mount version which won’t lock one system when the other has serious panic issues, but you still get a sudden failure. Having a file system available to the host which won’t have any effect on the development host when the remote end crashes and burns will leave you far less frustrated (unless your modifications never fail).
Add to this that you’d need to make sure your kernel config and other files on the exporter are actually configured for NFS export (most distributions are set up by default to support NFS import, but not export). A clone has no requirements at all on the Jetson. NFS sounds great, but the first time you have issues that a loopback mounted clone does not have you’ll probably wish you used a clone or copy instead of NFS. YMMV, it depends on your circumastances, e.g., you could loopback mount a read-only clone on a dedicated NFS server and then each developer can use that and there would be no issue of a failed Jetson interfering with the host machine.
You could just point the cross linker to an alternate search path, but you’d need to make sure the linker path of the cross linker (not your operating system linker) has that path show up prior to any other standard system path (perhaps you’ve added an aarch64-linux-gnu library due to installing the cross linker…you’d want your version to occur prior to that other version…an update of a system package provided version which you did not expect could replace the expected version in some cases with an incorrect search path order). If you sym link, then you have no need to mess with the cross linker environment…it’ll “just work” if the files are in the default location for any reason (the reason in this case being due to symbolic links).
cool thanks got a clone and mounted it loopback on host. Now the problem is that when I installed the cross linker (Cuda toolkit) on host it has created a folder /usr/aarch64-linux-gnu and within this folder there are include and lib sub-folders. When liking the app using nsight IDE it automatically picks up libs from this path. I have my clone mounted and its /usr/local/my_jetson/usr/lib/aarch64-linux-gnu symklinked as /usr/lib/aarch64-linux-gnu.
Build was throwing errors upon not finding some OpenGL .so files it needed so I copied these over from my mount to /usr/aarch64-linux-gnu/lib and it is now linking fine. However what I really wanted was to somehow make the linker look into my symlinked folder. What do you think is the proper solution to this?
One option: Create “/usr/local/aarch64-linux-gnu/”, with that being a symbolic link to the loopback mounted “aarch64-linux-gnu”. Then tell eclipse to use that instead.
There is more than one directory involved in many apps, e.g., OpenGL for many GUI apps. Those will exist on your Jetson, but they may be spread out (this is one reason I suggest a clone…they are there, and you won’t have to build them yourself).
If you look at the Jetson’s “/etc/nv_tegra_release” you’ll see many of the involved files. Most are originally in “/usr/lib/aarch64-linux-gnu/”. For the GUI you will also see “/usr/lib/xorg/modules/extensions/libglx.so” and one other (in reality the two are the same file, just in different locations…think of the “/usr/lib/aarch64-linux-gnu/tegra/libglx.so” as the master copy). Basically you can create symbolic links and any parent directory needed somewhere under “/usr/local/aarch64-linux-gnu/”. Example if you have your clone mounted at “/some/where/”:
sudo -s
cd /usr/local
mkdir lib
cd lib
ln -s /some/where/usr/lib/aarch64-linux-gnu .
ln -s /some/where/usr/lib/xorg .
exit
There are a lot of files in “/usr/lib/” of any Linux distribution. The symbolic link provides all of them without manual copy.
Since I don’t use the system provided files on my host for cross compile I just created my own “/usr/lib/aarch64-linux-gnu/” (there wasn’t anything for me to destroy by sym linking directly).
isn’t it better to symlink the lib folder and be done with it and tell the linker to search in /usr/local/lib for all dependencies instead of creating a symlink to each different lib folder, like aarch64 and xorg in above example?
First, you never want a foreign architecture in the native architecture location. The “aarch64-linux-gnu” is important on a desktop PC, but “/usr/local/lib/” on a PC is for x86_64/amd64 only. However, you could use any combination of sym links and/or link location naming (within eclipse) which you think is convenient. I have many images going all the way back to 32-bit arm soft flow and 32-bit armhf, and this is how I find it least confusing. Other than always explicitly using an aarch64-linux-gnu name it’s just a case of what you find least confusing.
ah ok got it thanks. As I mentioned while installing cross toolkit I got a folder /usr/aarch64-linux-gnu that linker is currently pointing to for includes and libs. I hope it is safe to just replace this one with a symlink into my Jetson’s aarch64-linux-gnu folder? Since the mount version will have everything that linker needs plus additional libs that were installed on Jetson.
It is safe to do so…but beware that if some content there is via a system package (dpkg or apt-get), then the package manager won’t deal well with it. I have no packages there, everything is from my clones.
If you want to see which package (if any) owns a file, then:
dpkg -S /where/ever/it/is/filename
For the case of sym linking the entire directory you could remove any package providing a file there before putting your own in there (if you were to sym link only individual files, then you’d have to remove any individual conflicting file instead of the whole directory…it is much easier to just sym link the entire directory and not individual files).
thanks @linuxdev for the amazing help you provide on these forums. I have benefited from your posts many times over. cheers!!